Channel (or pixel) flags apply to individual pixels and are added as a column in the HTP. Their names are also added to the metadata of a dataset during processing, and this is used for the history of the pipeline. It also means that you can tell that, e,g., the WBS pipeline has been applied if you see things like "isMasked" and "checkZero" in the metadata.
For each pixel, there are 32 flags which can be set. The definition of the currently used mask bits and values in HIFI
data is given below. For bit n, the value is computed according to
For scripting, it is recommended that instead of using the bit values, you avail yourself of the software names defined
in the HifiMask class. The section on Chapter 11 has more details.
|Flag Name||Bit||Software Name||Description|
|Bad pixel||0||HifiMask.BAD_PIXEL||Indicates a failed pixel on a WBS CCD. Not used in HRS.|
|Saturated pixel||1||HifiMask.SATURATED||If this bit is set, the sample was saturated.|
|Not observed||2||HifiMask.NOT_OBSERVED||If this bit is set, the sample is not observed. See Note 1.|
|Not Calibrated||3||HifiMask.NOT_CALIBRATED||If this bit is set, the sample is not calibrated. It is set on pixels that also have the SATURATION bit set. See Note 2.|
|Not used||4||Not used|
|Glitch detected||5||HifiMask.GLITCHED||Indicates a WBS data point affected by a cosmic ray.|
|Dark pixel||6||HifiMask.DARK_PIXEL||If this bit is set, the sample is used to measure the dark.|
|Spur candidate||7||HifiMask.SPUR_CANDIDATE||If this bit is set, the sample is a candidate to be a spur. It is a 'candidate' since not all things flagged by the spurfinder are necessarily spurs.|
|Spur warning||8||HifiMask.SPUR_WARNING||This bit is set to indicate that spurs have been observed at similar LO tunings for those channels, although it does not necessarily imply that a spur is actually present in this particular observation. This flag is essentially a warning, and therefore is not honoured by the fitHifiFringe, fitBaseline, deconvolution or gridding algorithms.|
|line||28||HifiMask.LINE||User set flag to denote a line.|
|Bright line||29||HifiMask.BRIGHT_LINE||User set flag to denote a bright line.|
|Ignore data||30||HifiMask.IGNORE_DATA||User set flag to ignore data.|
NOTE 1 : The NOT_OBSERVED bit is typically used to represent data computed via interpolation or extrapolation. At Level 2, this is common at the end-points of a subband where the resampler has to extrapolate beyond the observed frequency range by under 1 pixel in order to ensure that the data sets are all the same size. It is also used at Level 0 and 0.5 to represent data that really are not observed. This occurs when the IF of HIFI covers a region narrower than the 8192 pixels that the CCD does. By design, the IF is well matched to the CCD, hence the number of unobserved pixels is very small. They are culled from the spectra in Level 1 and beyond.
NOTE 2 : The NOT_CALIBRATED bit is set for saturations in addition to the SATURATED bit. Saturated regions (such as strong spurs) can migrate in IF over time. This leads to a possible scenario where, when averaging data, a 'good' region from one spectra overlaps with a 'bad' region from another. The NOT_CALIBRATED flag is logically 'or-ed' between the good and the bad data, which means that the final output will properly reflect that the data in this overlap region should not be trusted.
NOTE 3 : Bits 28-30 are special because they are set by you and not the pipeline. The 'line' and 'bright line' will be used
fitHifiFringe, and the
tools. The ignore data flag allows you to mask out bad spectral regions which can then be ignored by the pipeline.
NOTE 4 : It is possible to replace a flagged value with a NaN (Not a Number) or another value (e.g. fitted with an algorithm)
removePixel task. In addition, if the mask is homogeneous for all scans within a dataset, it
is possible to remove the channel(s) from the HTP.
from herschel.hifi.pipeline.product import HifiMask from herschel.hifi.pipeline.util.tools import RemoveFlaggedPixelsTask # replace bad pixels with NaN (default behaviour) removePixel(htp=htp, mask=HifiMask.BAD_PIXEL) # interpolate over bad and saturated pixels removePixel(htp=htp, mask=[HifiMask.BAD_PIXEL,HifiMask.SATURATED],mode='interpolate') # remove flagged channels ( removePixel = RemoveFlaggedPixelsTask()