12.3. Sine Wave Fitting Method (fitHifiFringe)

12.3.1. Introduction to fitHifiFringe

Periodic signals in HIFI Level 2 spectra can be removed with the sine wave fitting tool fitHifiFringe. This tool allows for automatic or manual masking of lines before fitting a user-defined number of sine waves. As it makes use of the general task fitFringe, details on the algorithms can be found in the FitFringe manual .

The cookbooks (Chapter 2) demonstrate use of the tool on specific observations.

12.3.2. Running fitHifiFringe

FitHifiFringe is automatically registered to ObservationContexts and HifiTimelineProducts (i.e. when clicking on such variables in the Variable window of HIPE, fitHifiFringe shows up as an applicable task). However, you can also process SpectrumDatasets, SimpleSpectrums, and HIFI spectral cubes (SpectralSimpleCubes) by opening the GUI (see Figure 12.1) under All Tasks and then dragging the variable to the appropriate bullet. Alternatively, run it on the command line as follows:

FitHifiFringe GUI

Figure 12.1. FitHifiFringe GUI


For ObservationContexts and HifiTimelineProducts:

result=fitHifiFringe(data=obs_in,nfringes=2,typicalPeriod=150., product='WBS-H-USB')

or for SpectrumDatasets:

result=fitHifiFringe(data=sds_in,nfringes=2,typicalPeriod=150.)

or for SpectralSimpleCubes:

result=fitHifiFringe(data=ss_in,nfringes=2,typicalPeriod=150.)

or for SimpleSpectrums:

result=fitHifiFringe(data=ss_in,nfringes=2,typicalPeriod=150.)

Note that in the latter case, the SimpleSpectrum input can be constructed from any spectrum of type Spectrum1d as follows:

ss_in=SimpleSpectrum(spectrum1d)

This allows FitHifiFringe to be applied to a Single Sideband spectrum taken from the doDeconvolution output product and to a spectrum extracted from a data cube using the extractRegionSpectrum task.

For HifiTimelineProducts, SpectrumDatasets, SimpleSpectrums, and SpectralSimpleCubes the output result is a list of products: a product that is identical to the input, but with the fitted sine waves subtracted from the flux columns (result[0]) and the summary standing wave tables (result[1]). These products are easily retrieved, for example:

result=fitHifiFringe(data=obs)
myNewObs=result[0]
mySwTable=result[1]

Remember that because of the nature of an ObservationContext the original input will have the sine waves subtracted as well. If you wish to compare before and after results, you will need to read the input data separately into a different variable name.

When subBase=1 is selected, fitHifiFringe subtracts the fitted baseline (shape+continuum level) together with the (multi-)sine fringe model. At the same time, a median baseline level is computed (per spectrometer subband, or as a whole, depending on the input on doglue) and reported in the output table swTab (see Figure 12.3). The parameter addMedianContinuum=True allows you to re-add the median baseline value computed by the task in the baseline subtraction process. This will allow you to benefit from baseline correction (shape) but preserve the overall continuum.

By default, fitHifiFringe produces two plots for each fitted spectrum. The first plot (not shown here) shows the chi-square values as a function of sine wave period. The second plot (see Figure 12.2) shows the input data, fitted sine waves, baseline, line mask, channel flags and output data for the user to inspect. The orange line represents all data ignored in the fitting process: values of 0.0 for line masks and values of 0.5 for channel flags. Before processing the next spectrum, you are asked for confirmation through a pop-up GUI. For products with many spectra, you can also decide to continue non-stop (without further plotting and pop-up GUIs).

Fitted sine waves, baseline, line mask, channel flags, and output data

Figure 12.2. Fitted sine waves, baseline, line mask, channel flags, and output data


The amplitudes, periods, phases, and chi-squares of the fitted sine waves are printed on the HIPE console. In case the input was an ObservationContext, the table is saved in the context as indicated in the screenshot below (see Figure 12.3):

Table containing information resulting from the fitting

Figure 12.3. Table containing information resulting from the fitting


The following input parameters are allowed:

  • data: input can be ObservationContexts (Level 1 or 2), HifiTimelineProducts (Level 1 or 2), SpectrumDatasets (WBS or HRS), SimpleSpectrums (Waveunit needs to be in GHz), and spectral cubes (SpectralSimpleCubes) (Waveunit needs to be in GHz).

  • product: if the input is an ObservationContext, indicate which Level 2 product needs to be processed. Options are: 'WBS-H-USB' (default), 'WBS-H-LSB', 'WBS-V-USB', 'WBS-V-LSB', 'HRS-H-USB', 'HRS-H-LSB', 'HRS-V-USB', 'HRS-V-LSB'.

    Note that standing waves are automatically corrected in both sidebands of whichever spectrometer you select so, for example, if you select 'WBS-H-USB' the standing waves will also be corrected in the WBS-H-LSB.

  • sds_index: index of a single SpectrumDataset in a product to process. The index can be determined by listing the product in the Context viewer. This option is irrelevant if the input is an sds. [DEFAULT: sds_index=-1, which means all SpectrumDatasets in a product will be processed].

  • nfringes: number of sine waves to be fitted. By default, nfringes=2 for HIFI bands 2,5, nfringes=3 for bands 1,3,4,6,7, and nfringes=1 if the HIFI band cannot be determined.

  • startPeriod: shortest-period standing wave (in MHz) to search for. By default, startPeriod=80 MHz for HIFI bands 1,2,3,4,5,6,7, and startPeriod=20 MHz if the HIFI band cannot be determined.

  • endPeriod: longest-period standing wave (in MHz) to search for. By default, endPeriod=120 MHz for HIFI bands 2,5, endPeriod=200 MHz for HIFI band 1, endPeriod=1000 MHz for bands 3,4,6,7, and endPeriod=3000 MHz if the HIFI band cannot be determined.

  • typicalPeriod: typical standing wave period (in MHz) in the data. This is used for the baseline determination. Features with much longer periods are considered baseline structure and will not be removed with sine waves. By default, typicalPeriod=95 MHz for HIFI bands 1,2,3,4,5, typicalPeriod=600 MHz for band 6, typicalPeriod=625 MHz for band 7, and typicalPeriod=150 MHz if the HIFI band cannot be determined.

  • plot=...: Sets if and how the results should be plotted.

    • plot=0: no plotting at all.

    • plot=1: two plots per scan: (1) period versus Chi^2 (2) the before/after plot and the subtracted sine wave and the line mask. WARNING: If the observation contains many scans, many plot windows will appear on the screen!

    • plot=2: as plot=1, but after each scan, show a pop-up window to process the next scan and delete previous plot windows. One can also opt to continue without further plotting.

    • [DEFAULT: plot=2]

  • doglue=False, Determine SW on individual subband spectrum. This is desired for HRS, but often not for WBS, as long period SW can only be determined on the combined spectrum. [DEFAULT: doglue=True]

  • automask=True: automatically mask datapoints using the sigma-clip algorithm described in the SmoothBaseline manual. This mask is added to any user-defined mask ('usermask') that is provided and any line flags (LINE or BRIGHT_LINE) that were defined before fitHifiFringe was started (e.g., using flagTool). [DEFAULT: automask=True]

  • usermask=..., mask frequency ranges in addition to the automatically determined mask ('automask') and any line flags (LINE or BRIGHT_LINE) that were defined before fitHifiFringe was started (e.g., using flagTool). Example: usermask=[(537.0,538.0), (539,539.5)] marks the ranges 537-538 GHz and 539-539.5 GHz [DEFAULT: only use automatically determined mask]

  • avermask=True, determine the mask on the average of all scans and then apply this mask to each scan. This reduces the amount of work for large maps a lot. It also should give more accurate masks as the average spectrum has better signal-to-noise. Spectra with different LO settings will not be averaged, so this option does not work for spectral scans. [DEFAULT: avermask=False, i.e. determine mask on each indvidual spectrum]

  • subBase=True, subtract the smooth baseline that was determined in the sine wave fitting process from the input data (i.e., in addition to subtracting the fitted sine waves). Note that this can be dangerous as the smooth baseline may include weak lines or wings of bright lines that were not properly masked. [DEFAULT: do not subtract the baseline]

  • flags=128, data points with these flag values will be excluded from the calculations. Multiple flags are added, e.g., flags=384 means that data points with flag values 128 and 256 will be excluded. Note that data with flag values 1, 2, 8, 64, and 1073741824 (=2**30) will ALWAYS be ignored. [DEFAULT: 128 (=SPUR_CANDIDATE)]

  • resolution=10, used to internally rebin the spectra in order to speed up the task. The default of 10 MHz is sufficient for HIFI spectra, but should be reduced for processing of non-HIFI spectra with standing wave periods less than about 30 MHz. [DEFAULT: 10 MHz]

  • addMedianContinuum= True, allow the median continuum to be added back into the baseline fit flux. [DEFAULT: False]

  • saveInput=True, will save the fitHifiFringe parameter inputs to a calibration product with a table dataset and metadata, and store this in the pipeline-out product, i.e. obs.refs["calibration"].product.refs["pipeline-out"].product.refs["fitHifiFringe"].product. This product can be generated only when the input data is an ObservationContext. The calibration product may be used in later applications of fitHifiFringe to fix the task parameters, using the cal input parameter. [DEFAULT: False]

  • cal: An optional calibration product containing a fitHifiFringe product saved from a previous application of fitHifiFringe with saveInput = True. When a calibration product (e.g. cal = obs.refs["calibration"].product) is supplied by drag and drop into the GUI, or at the command line, the task parameters are fixed by the contents of the fitHifiFringe product (changes to task parameters in the GUI or command line will have no effect). If the calibration product lacks the fitHifiFringe product, a warning message will pop-up when using the GUI, prior to executing the task. If a calibration product is provided in a command line application of the task, and it is missing the fitHifiFringe product, only a warning message is given, the cal variable is then ignored, and the task will run with the default or current task parameter inputs. [DEFAULT: Null (no cal input)]

12.3.3. Example of using fitHifiFringe

In Level 2 data, standing waves are mixed with astronomical signals, and separating them from each other requires the astronomer's judgment. fitHifiFringe is an interactive user tool to do this, and a typical way of using it is described here.

To get started, the most important parameter to supply to fitHifiFringe is typicalPeriod. This sets the shape of the baseline for the standing waves. Its value depends on the HIFI band that is being processed. A reasonable default is automatically selected by fitHifiFringe. To optimise the value for a particular observation, you have to either inspect the plot before applying fitHifiFringe, or run fitHifiFringe once to get an estimate of the typical period(s). As an example, see the next plot. This is a Load Chop observation in band 3A taken without a sky reference. The typical period is about 100 MHz.

Example of a typical period (at about 100 MHz) in a Load Chop observation in band 3A (taken without a sky reference)

Figure 12.4. Example of a typical period (at about 100 MHz) in a Load Chop observation in band 3A (taken without a sky reference)


Subsequently, run the tool by clicking on Accept. Alternatively, you can enter the following command line:

result = fitHifiFringe(data=obs,product="WBS-H-USB",nfringes=1,typicalPeriod=100.0)

Here, only one sine wave is fitted to the data (nfringes=1). If one does not give this input parameter, nfringes is automatically adjusted depending on the HIFI band. Two plots are generated: one with the sine wave period Figure 12.5) as a function of chi-square, and one with before/after spectra Figure 12.6).

Sine wave fitted using only one sine wave (i.e. nfringes=1)

Figure 12.5. Sine wave fitted using only one sine wave (i.e. nfringes=1)


Resulting fit showing the before and after spectra

Figure 12.6. Resulting fit showing the before and after spectra


To optimise the fit, the following questions need to be answered by inspecting the plots:

  • Does the smooth baseline (blue curve in the spectrum plot) look reasonable as a baseline for the standing waves? You may need to zoom in to see, do this by drawing a box around the region of interest with the left mouse button. In this example it looks OK. If it does not (e.g., if it follows the waves), run fitHifiFringe again with a different value of typicalPeriod (if the input was an ObsContext, you will need to read the original data from disk again).

  • Are any lines properly masked? In this example, no lines are masked (the orange mask curve only shows values of 1), but there is clearly a spectral line between 835.04 and 835.24 GHz. You can mask this line by hand using the usermask input. Line masks are shown with values of 0.0 in the orange line, and channel flags taken into account with values of 0.5. Only channels where the orange line is 1.0 are taken into account in the calculations.

  • How many peaks are present in the chi-square versus period plot? You may need to zoom in by drawing a box with the left mouse button. In this case, a strong 96 MHz standing wave is found (indicated by a vertical red line), but the spectrum shows, especially near the IF band edges, that the fit is not so good. The chi-square plot does not show additional peaks, but the main peak is rather broad. Increase the number of sine waves to be fitted (nfringes). Also, sometimes minima are found at periods that are irrelevant for the science, e.g., with very long periods. The fit solutions can be influenced by limiting the period search range better than the default 20-3000 MHz range. Use the startPeriod and endPeriod parameters for this. Do not choose a very narrow range, or otherwise the chi-square peaks cannot be determine accurately enough, or you might miss waves that are hard to see by eye in the spectrum.

  • The smooth baseline (blue line in the spectrum plot) is only used as a baseline for the standing waves. It is not actually subtracted from the data. In some cases, such as in this example which shows a negative baseline level, it is actually a very reasonable baseline choice. You can subtract it using the subBase=True option. Then you can omit fitting polynomial baselines at a later stage (using fitBaseline).

  • By default the WBS subbands are fitted at once. Sometimes the fit improves by fitting them individually. In this case use the doglue=False option.

Taking the above considerations into account, the optimum fitHifiFringe parameters are:

result = fitHifiFringe(data=obs,product="WBS-H-USB",nfringes=3,startPeriod=80.0,endPeriod=300.0,\
typicalPeriod=100.0,subBase=True, doglue=False, usermask=[(835.04, 835.24)])

The end result is shown here Figure 12.7):

Resulting fit using the optimum fitHifiFringe parameters

Figure 12.7. Resulting fit using the optimum fitHifiFringe parameters


If it is desired to save these input parameter values, the option saveInput can be used, and a fitHifiFringe product will be saved in the calibration tree of the observation context. In the same example:

result = fitHifiFringe(data=obs,product="WBS-H-USB",nfringes=3,startPeriod=80.0, endPeriod=300.0, \
typicalPeriod=100.0,subBase=True, doglue=False, usermask=[(835.04, 835.24)], saveInput=True)

Later, if the output product result (or mynewObs = result[0]) is saved, the same fitHifiFringe parameters can be retrieved from the data tree and reapplied, either to the same, or a different observation with similar standing wave characteristics:

myCal = obs.refs["calibration"].product
result2 = fitHifiFringe(data=obs2,cal=myCal)

Other considerations for removing standing waves with fitHifiFringe:

  • The band 6 and 7 "HEB" waves are not pure sine waves: the phase drifts over the IF band pass. You may still approximate fits by using a combination of 2-3 sine waves with periods around 320 MHz. For the WBS backends, fitting the waves per subband may give better results (doglue=False). The next Section 12.4 deals with the Electrical Standing Wave correction in HEB bands.

  • Waves in bands 3 and 4 show increasing amplitudes toward the IF band edges. Approximate fits can be obtained by fitting a combination of sine waves, and sometimes by fitting per WBS subband (doglue=False).