Periodic signals in HIFI Level 2 spectra can be removed with the sine wave fitting tool
This tool allows for automatic or manual masking of lines before fitting a user-defined number of sine waves. As it makes
of the general task
fitFringe, details on the algorithms can be found in the
FitFringe manual .
The cookbooks (Chapter 2) demonstrate use of the tool on specific observations.
FitHifiFringe is automatically registered to
HifiTimelineProducts (i.e. when clicking on such variables in the Variable window
fitHifiFringe shows up as an applicable task). However, you can also process
SimpleSpectrums, and HIFI spectral cubes
SpectralSimpleCubes) by opening the GUI (see
Figure 12.1) under All Tasks and then
dragging the variable to the appropriate bullet. Alternatively, run it on the command line as follows:
Note that in the latter case, the
SimpleSpectrum input can be constructed from any spectrum of type
Spectrum1d as follows:
FitHifiFringe to be applied to a Single Sideband spectrum taken from the
doDeconvolution output product and to a spectrum extracted from a data cube using the
SpectralSimpleCubes the output result is a list of products: a product that is identical to the input,
but with the fitted sine waves subtracted from the flux columns (result) and the summary standing wave tables (result).
These products are easily retrieved, for example:
result=fitHifiFringe(data=obs) myNewObs=result mySwTable=result
Remember that because of the nature of an
ObservationContext the original input will have the sine waves
subtracted as well. If you wish to compare before and after results, you will need
to read the input data separately into a different variable name.
subBase=1 is selected,
fitHifiFringe subtracts the fitted baseline
(shape+continuum level) together with the (multi-)sine fringe model. At the same time, a median baseline level is computed
(per spectrometer subband, or as a whole, depending on the input on
doglue) and reported in the output
table swTab (see Figure 12.3). The parameter
addMedianContinuum=True allows you to re-add the median baseline value computed by the task in the
baseline subtraction process. This will allow you to benefit from baseline correction (shape) but preserve the overall continuum.
fitHifiFringe produces two plots for each fitted spectrum. The first plot (not shown here) shows
the chi-square values as a function of sine wave period. The second plot (see Figure 12.2)
shows the input data, fitted sine waves, baseline, line mask, channel flags and output data for the user to inspect. The orange
represents all data ignored in the fitting process: values of 0.0 for line masks and values of 0.5 for channel flags. Before
the next spectrum, you are asked for confirmation through a pop-up GUI. For products with many spectra, you can also decide
non-stop (without further plotting and pop-up GUIs).
The amplitudes, periods, phases, and chi-squares of the fitted sine waves are printed on the HIPE console. In case the input was an ObservationContext, the table is saved in the context as indicated in the screenshot below (see Figure 12.3):
The following input parameters are allowed:
data: input can be
ObservationContexts (Level 1 or 2),
(Level 1 or 2),
SpectrumDatasets (WBS or HRS),
SimpleSpectrums (Waveunit needs to be in GHz), and
spectral cubes (
SpectralSimpleCubes) (Waveunit needs to be in GHz).
product: if the input is an
ObservationContext, indicate which Level 2 product needs to be processed.
Options are: 'WBS-H-USB' (default), 'WBS-H-LSB', 'WBS-V-USB', 'WBS-V-LSB', 'HRS-H-USB', 'HRS-H-LSB', 'HRS-V-USB', 'HRS-V-LSB'.
Note that standing waves are automatically corrected in both sidebands of whichever spectrometer you select so, for example, if you select 'WBS-H-USB' the standing waves will also be corrected in the WBS-H-LSB.
sds_index: index of a single
SpectrumDataset in a product to process. The index can be determined
by listing the product in the Context viewer. This option is irrelevant if the input is an sds. [DEFAULT: sds_index=-1, which
SpectrumDatasets in a product will be processed].
nfringes: number of sine waves to be fitted. By default, nfringes=2 for HIFI bands 2,5, nfringes=3
for bands 1,3,4,6,7, and nfringes=1 if the HIFI band cannot be determined.
startPeriod: shortest-period standing wave (in MHz) to search for. By default, startPeriod=80 MHz for HIFI
bands 1,2,3,4,5,6,7, and startPeriod=20 MHz if the HIFI band cannot be determined.
endPeriod: longest-period standing wave (in MHz) to search for. By default, endPeriod=120 MHz for HIFI bands 2,5,
endPeriod=200 MHz for HIFI band 1, endPeriod=1000 MHz for bands 3,4,6,7, and endPeriod=3000 MHz if the HIFI band cannot be
typicalPeriod: typical standing wave period (in MHz) in the data. This is used for the baseline determination. Features
with much longer periods are considered baseline structure and will not be removed with sine waves. By default, typicalPeriod=95
MHz for HIFI
bands 1,2,3,4,5, typicalPeriod=600 MHz for band 6, typicalPeriod=625 MHz for band 7, and typicalPeriod=150 MHz if the HIFI
band cannot be
plot=...: Sets if and how the results should be plotted.
plot=0: no plotting at all.
plot=1: two plots per scan: (1) period versus Chi^2 (2) the before/after plot and the subtracted sine wave and the line mask. WARNING: If the observation contains many scans, many plot windows will appear on the screen!
plot=2: as plot=1, but after each scan, show a pop-up window to process the next scan and delete previous plot windows. One can also opt to continue without further plotting.
doglue=False, Determine SW on individual subband spectrum. This is desired for HRS, but often
not for WBS, as long period SW can only be determined on the combined spectrum. [DEFAULT: doglue=True]
automask=True: automatically mask datapoints using the sigma-clip algorithm described in the
SmoothBaseline manual. This mask is added to any user-defined mask ('usermask') that is provided and any
line flags (LINE or BRIGHT_LINE) that were defined before
fitHifiFringe was started
flagTool). [DEFAULT: automask=True]
usermask=..., mask frequency ranges in addition to the automatically determined mask ('automask')
and any line flags (LINE or BRIGHT_LINE) that were defined before
started (e.g., using
flagTool). Example: usermask=[(537.0,538.0), (539,539.5)] marks
the ranges 537-538 GHz and 539-539.5 GHz [DEFAULT: only use automatically determined mask]
avermask=True, determine the mask on the average of all scans and then apply this mask to each scan.
This reduces the amount of work for large maps a lot. It also should give more accurate masks as the average
spectrum has better signal-to-noise. Spectra with different LO settings will not be averaged, so this option
does not work for spectral scans. [DEFAULT: avermask=False, i.e. determine mask on each indvidual spectrum]
subBase=True, subtract the smooth baseline that was determined in the sine wave fitting process from
the input data (i.e., in addition to subtracting the fitted sine waves). Note that this can be dangerous as
the smooth baseline may include weak lines or wings of bright lines that were not properly masked. [DEFAULT:
do not subtract the baseline]
flags=128, data points with these flag values will be excluded from the calculations. Multiple flags
are added, e.g., flags=384 means that data points with flag values 128 and 256 will be excluded. Note that data
with flag values 1, 2, 8, 64, and 1073741824 (=2**30) will ALWAYS be ignored. [DEFAULT: 128 (=SPUR_CANDIDATE)]
resolution=10, used to internally rebin the spectra in order to speed up the task. The default of 10
MHz is sufficient for HIFI spectra, but should be reduced for processing of non-HIFI spectra with standing wave
periods less than about 30 MHz. [DEFAULT: 10 MHz]
addMedianContinuum= True, allow the median continuum to be added back into the baseline fit flux. [DEFAULT: False]
saveInput=True, will save the
fitHifiFringe parameter inputs to a calibration
product with a table dataset and metadata, and store this in the pipeline-out product, i.e.
obs.refs["calibration"].product.refs["pipeline-out"].product.refs["fitHifiFringe"].product. This product can
generated only when the input
data is an ObservationContext. The
calibration product may be used in later applications of
fitHifiFringe to fix the task
parameters, using the
cal input parameter. [DEFAULT: False]
cal: An optional calibration product containing a
fitHifiFringe product saved
from a previous application of
= True. When a calibration product (e.g.
cal = obs.refs["calibration"].product) is supplied
by drag and drop into the GUI, or at the command line, the task parameters are fixed by the
contents of the
fitHifiFringe product (changes to task parameters in the GUI or command
line will have no effect). If the calibration product lacks the
fitHifiFringe product, a
warning message will pop-up when using the GUI, prior to executing the task. If a calibration product is provided
in a command line application of the task, and it is missing the
fitHifiFringe product, only
a warning message is given, the
cal variable is then ignored, and the task will run with
the default or current task parameter inputs. [DEFAULT: Null (no cal input)]
In Level 2 data, standing waves are mixed with astronomical signals, and separating them from each other requires the astronomer's
fitHifiFringe is an interactive user tool to do this, and a typical way of using it is described
To get started, the most important parameter to supply to
This sets the shape of the baseline for the standing waves. Its value depends on the HIFI band that is being processed. A
default is automatically selected by fitHifiFringe. To optimise the value for a particular observation, you have to either
plot before applying
fitHifiFringe, or run
fitHifiFringe once to get an estimate of the
typical period(s). As an example, see the next plot. This is a Load Chop observation in band 3A taken without a sky reference.
typical period is about 100 MHz.
Figure 12.4. Example of a typical period (at about 100 MHz) in a Load Chop observation in band 3A (taken without a sky reference)
Subsequently, run the tool by clicking on Accept. Alternatively, you can enter the following command line:
result = fitHifiFringe(data=obs,product="WBS-H-USB",nfringes=1,typicalPeriod=100.0)
Here, only one sine wave is fitted to the data (nfringes=1). If one does not give this input parameter, nfringes is automatically adjusted depending on the HIFI band. Two plots are generated: one with the sine wave period Figure 12.5) as a function of chi-square, and one with before/after spectra Figure 12.6).
To optimise the fit, the following questions need to be answered by inspecting the plots:
Does the smooth baseline (blue curve in the spectrum plot) look reasonable as a baseline for the standing waves? You may need
to zoom in to see, do this by drawing a box around the region of interest with the left mouse button. In this example it looks
OK. If it does not (e.g., if it follows the waves), run
fitHifiFringe again with a different value of
typicalPeriod (if the input was an ObsContext, you will need to read the original data from disk again).
Are any lines properly masked? In this example, no lines are masked (the orange mask curve only shows values of 1), but there
clearly a spectral line between 835.04 and 835.24 GHz. You can mask this line by hand using the
input. Line masks are shown with values of 0.0 in the orange line, and channel flags taken into account with values of 0.5.
channels where the orange line is 1.0 are taken into account in the calculations.
How many peaks are present in the chi-square versus period plot? You may need to zoom in by drawing a box with the left mouse
In this case, a strong 96 MHz standing wave is found (indicated by a vertical red line), but the spectrum shows, especially
IF band edges, that the fit is not so good. The chi-square plot does not show additional peaks, but the main peak is rather
Increase the number of sine waves to be fitted (
nfringes). Also, sometimes minima are found at periods that
are irrelevant for the science, e.g., with very long periods. The fit solutions can be influenced by limiting the period search
better than the default 20-3000 MHz range. Use the
parameters for this. Do not choose a very narrow range, or otherwise the chi-square peaks cannot be determine accurately enough,
you might miss waves that are hard to see by eye in the spectrum.
The smooth baseline (blue line in the spectrum plot) is only used as a baseline for the standing waves. It is not actually
from the data. In some cases, such as in this example which shows a negative baseline level, it is actually a very reasonable
choice. You can subtract it using the
subBase=True option. Then you can omit fitting polynomial baselines at
a later stage (using
By default the WBS subbands are fitted at once. Sometimes the fit improves by fitting them individually. In this case use
Taking the above considerations into account, the optimum
fitHifiFringe parameters are:
result = fitHifiFringe(data=obs,product="WBS-H-USB",nfringes=3,startPeriod=80.0,endPeriod=300.0,\ typicalPeriod=100.0,subBase=True, doglue=False, usermask=[(835.04, 835.24)])
The end result is shown here Figure 12.7):
If it is desired to save these input parameter values, the option
saveInput can be used, and
fitHifiFringe product will be saved in the calibration tree of the observation context.
In the same example:
result = fitHifiFringe(data=obs,product="WBS-H-USB",nfringes=3,startPeriod=80.0, endPeriod=300.0, \ typicalPeriod=100.0,subBase=True, doglue=False, usermask=[(835.04, 835.24)], saveInput=True)
Later, if the output product result (or mynewObs = result) is saved, the same
parameters can be retrieved from the data tree and reapplied, either to the same, or a different observation with similar
standing wave characteristics:
myCal = obs.refs["calibration"].product result2 = fitHifiFringe(data=obs2,cal=myCal)
Other considerations for removing standing waves with
The band 6 and 7 "HEB" waves are not pure sine waves: the phase drifts over the IF band pass. You may still approximate fits
using a combination of 2-3 sine waves with periods around 320 MHz. For the WBS backends, fitting the waves per subband may
better results (
doglue=False). The next Section 12.4 deals
with the Electrical Standing Wave correction in HEB bands.
Waves in bands 3 and 4 show increasing amplitudes toward the IF band edges. Approximate fits can be obtained by fitting a
combination of sine waves, and sometimes by fitting per WBS subband (