Determine RMS noise of HIFI data (Level 2 and above) for both WBS and HRS.
An essential step is the determination of the baseline and the line masks which is delegated to the SmoothBaselineTask. Once mask and baseline are determined the baseline(s) is (are) subtracted from the input spectra (if 'subtractBase' has been set to True) and the line mask(s) is (are) set, accordingly. Optionally, the results can be smoothed (by default in accordance to smoothing widths configured in the uplink product by the noiseMaxWidth-, noiseMinWidth-parameters). After this optional smoothing the statistics are computed on a per spectrum basis (across the channels) by ignoring the line masked regions.
From this raw output which is found in the output product with integer keys, we also prepare suitable overviews:
index table: summarises the information of all the raw statistics output datasets, by giving specifying the index (the integer key in the output product, an identifier of the dataset used as basis for computing the statistics, the associated LO frequency if available, the smoothing applied, etc. and the average, standard deviation, min and max of the statistics quantities reported in the raw statistics output datasets.
summary table (per smoothing): in contrast to the index table, the summary table reports the average, stdev, median of the all the statistics found in all the datasets for a given smoothing.
For point modes, index and summary tables contain the same information. However, for scan modes or mapping modes it differs. When computing the statistics, in particular, when determining baseline and mask, spectra with different LO frequency or associated with a different point in sky should be treated separately. Spectra treated separately in this sense are combined in groups. Hence, we arrive at many different raw statistics datasets in the output product. The raw statistics datasets included in the output product may contain many rows. This is possible, when the average at the end of the Level 2 pipeline is skipped, and the corresponding timeline product is passed as input to the mkRms task. Then, the index table contains the summary statistics (e.g. average of the standard deviations found in the different rows of the raw statistics dataset) - whereas the summary table contains the the summary statistics over all quantities found in the raw statistics datasets for given smoothing.
Various different input data can be passed - and, for a given input data type, different behaviour results according to the observing mode. In the following, we describe the default behaviour as intended for the pipeline.
By setting the parameters backend and sideband, the associated Level 2 timeline product is selected. See further details on how this data type is handled.
Point Mode: We expect that at Level 2 we see a single dataset. In case the average in Level 2 pipeline has been skipped, the mask is computed from an average over all these spectra. Then, one raw statistics dataset is included in the output product with the statistics for the individual scans.
Mapping Mode: We expect that at Level 2 we see for each point in sky a seperate dataset. By default, for each of these points mask and baseline are computed separately and the statistics will end up in a separate raw statistics dataset. In case the average in Level 2 pipeline has been skipped, the mask is computed from an average over all these spectra per point in sky. Then, per point in sky one raw statistics dataset is included in the output product with the statistics for the individual scans.
Scan Mode: We expect that at Level 2 we see for each LO setting a separate dataset. If the parameter 'scanModeBehavior' has been set to True (which is the default) the statistics is computed only for at most three datasets with LO frequency around noiseRefFrequency (as found in the uplink product). Here, the datasets with LO frequencies closest to noiseRefFrequency, noiseRefFrequency + 4GHz, noiseRefFrequency - 4GHz are selected. Three datasets are selected so that in case the dataset closest to noiseRefFrequency has a large fraction line masked, an alternative dataset can be picked. The average fraction of the total spectrum / spectra that is line-masked is reported in the metadata of the raw statistics datasets as 'fractionMasked_avg'. Finally, in the summary table, only one single raw statistics dataset is used (usually, the one closest to noiseRefFrequency). If the parameter 'scanModeBehavior' is set to False, for all LO settings mask and baseline are computed separately and the raw statistics computed. Then, in the summary table, the e.g. average over all the stdev values found in the raw statistics quantities is reported. In case the average in Level 2 pipeline has been skipped, the mask is computed from an average over all these spectra LO setting. Then again, per LO setting one raw statistics dataset is included in the output product which contains the statistics for the individual scans. - Here, the behaviour is somewhat different when 'scanModeBehavior' is set to True: Single datasets in the input htp are selected and other datasets that belong to the same group are not included in the calculations.
Note that, generally, when more than one dataset is found in the original htp for a given group a single identifier is used for the group data in the htp. This is of the form 'htp[k]' where k enumerates the groups. If we apply mkRms to a htp with average applied at the end of the Level2 pipeline this matches the dataset ids of the original htp. However, if the average has been skipped this no longer holds true.
For these data (such as found at Level 2.5) raw statistics are computed for each spaxel separately - including the computation of mask and baseline. Here the identifier for the data used for the computation of a raw statistics dataset is denoted by 'spaxel(j,k)' where j,k denote integer row and column number within the raster map.
Deconvolved Spectrum Dataset (Spectrum1D, 'scanModeBehavior=True'):
For this spectrum (such as found at Level 2.5) raw statistics are computed for suitable ranges. These ranges are determined as follows: Ranges are picked around the noiseRefFrequency - with a similar rule as adopted for spectral scan htps - ranges are selected around noiseRefFrequency, noiseRefFrequency + 4GHz and noiseRefFrequency - 4GHz. These ranges are selected only provided that at least 50% overlap with the input spectrum is obtained. The width of the ranges to select depends on:
Band: 4GHz for the SIS bands and 2.6 GHZ for the HEP bands
oneGhzReference-parameter: if set to True 1GHz
The range is then defined such that the reference frequency is in the center of the range.
General Spectrum Container (such as Spectrum2d etc):
A single raw statistics dataset is computed based on a single baseline and mask calculation.
stats = mkRms(input=obs, backend="WBS-H", sideband="USB", domask=1, base=80.0, rebin=2.0, doglue=False, \ uplink=None, preserveGroups=True, maskFromAvg=False, percentiles=, segmentIndex=2, smoothing=0.0, \ stitchSegments=False, plot=1, subtractBase=False)
htp = obs.refs["level2"].product.refs["WBS-H-USB"].product stats = mkRms(input=htp, domask=1, base=80.0, rebin=2.0, doglue=False, uplink=None, preserveGroups=True, \ maskFromAvg=False, percentiles=, segmentIndex=2, stitchSegments=False, plot=1, subtractBase=False)
ds = obs.refs["level2"].product.refs["WBS-H-USB"].product.refs["box_001"].product["0001"] stats = mkRms(input=ds, domask=1, base=80.0, rebin=2.0, doglue=False, uplink=None, maskFromAvg=False, \ percentiles=, segmentIndex=2, stitchSegments=False, plot=1, subtractBase=False)
MkRms is a special branch/script of HifiPipeline. You can find the script in the HIPE Pipelines menu: Pipelines --> HIFI --> RMS Pipeline Algorithm
The output is stored in the Statistics node of the Trend Analysis product.