10.2. Channel flags

Channel (or pixel) flags apply to individual pixels and are added as a column in the HTP. Their names are also added to the metadata of a dataset during processing, and this is used for the history of the pipeline. It also means that you can tell that, e,g., the WBS pipeline has been applied if you see things like "isMasked" and "checkZero" in the metadata.

For each pixel, there are 32 flags which can be set. The definition of the currently used mask bits and values in HIFI data is given below. For bit n, the value is computed according to value=2n. For scripting, it is recommended that instead of using the bit values, you avail yourself of the software names defined in the HifiMask class. The section on Chapter 11 has more details.

Flag Name Bit Software Name Description
Bad pixel 0 HifiMask.BAD_PIXEL Indicates a failed pixel on a WBS CCD. Not used in HRS.
Saturated pixel 1 HifiMask.SATURATED If this bit is set, the sample was saturated.
Not observed 2 HifiMask.NOT_OBSERVED If this bit is set, the sample is not observed. See Note 1.
Not Calibrated 3 HifiMask.NOT_CALIBRATED If this bit is set, the sample is not calibrated. It is set on pixels that also have the SATURATION bit set. See Note 2.
Not used 4   Not used
Glitch detected 5 HifiMask.GLITCHED Indicates a WBS data point affected by a cosmic ray.
Dark pixel 6 HifiMask.DARK_PIXEL If this bit is set, the sample is used to measure the dark.
Spur candidate 7 HifiMask.SPUR_CANDIDATE If this bit is set, the sample is a candidate to be a spur. It is a 'candidate' since not all things flagged by the spurfinder are necessarily spurs.
Spur warning 8 HifiMask.SPUR_WARNING This bit is set to indicate that spurs have been observed at similar LO tunings for those channels, although it does not necessarily imply that a spur is actually present in this particular observation. This flag is essentially a warning, and therefore is not honoured by the fitHifiFringe, fitBaseline, deconvolution or gridding algorithms.
line 28 HifiMask.LINE User set flag to denote a line.
Bright line 29 HifiMask.BRIGHT_LINE User set flag to denote a bright line.
Ignore data 30 HifiMask.IGNORE_DATA User set flag to ignore data.

NOTE 1 : The NOT_OBSERVED bit is typically used to represent data computed via interpolation or extrapolation. At Level 2, this is common at the end-points of a subband where the resampler has to extrapolate beyond the observed frequency range by under 1 pixel in order to ensure that the data sets are all the same size. It is also used at Level 0 and 0.5 to represent data that really are not observed. This occurs when the IF of HIFI covers a region narrower than the 8192 pixels that the CCD does. By design, the IF is well matched to the CCD, hence the number of unobserved pixels is very small. They are culled from the spectra in Level 1 and beyond.

NOTE 2 : The NOT_CALIBRATED bit is set for saturations in addition to the SATURATED bit. Saturated regions (such as strong spurs) can migrate in IF over time. This leads to a possible scenario where, when averaging data, a 'good' region from one spectra overlaps with a 'bad' region from another. The NOT_CALIBRATED flag is logically 'or-ed' between the good and the bad data, which means that the final output will properly reflect that the data in this overlap region should not be trusted.

NOTE 3 : Bits 28-30 are special because they are set by you and not the pipeline. The 'line' and 'bright line' will be used by fitBaseline, fitHifiFringe, and the doDeconvolution tools. The ignore data flag allows you to mask out bad spectral regions which can then be ignored by the pipeline.

NOTE 4 : It is possible to replace a flagged value with a NaN (Not a Number) or another value (e.g. fitted with an algorithm) using the removePixel task. In addition, if the mask is homogeneous for all scans within a dataset, it is possible to remove the channel(s) from the HTP.

from herschel.hifi.pipeline.product import HifiMask
from herschel.hifi.pipeline.util.tools import RemoveFlaggedPixelsTask

# replace bad pixels with NaN (default behaviour)
removePixel(htp=htp, mask=HifiMask.BAD_PIXEL)

# interpolate over bad and saturated pixels
removePixel(htp=htp, mask=[HifiMask.BAD_PIXEL,HifiMask.SATURATED],mode='interpolate')

# remove flagged channels (
removePixel = RemoveFlaggedPixelsTask()