SPIRE data contains various kinds of (Boolean) masks, used to indicate problems or caveats associated with the data. For consistency, SPIRE Masks follow the rule that the Boolean false state represents that a mask condition is not applicable.
The SPIRE data processing supports three general categories of masks.
Much of the sample mask in the data is populated during the Level 0 - Level 0.5 processing pipeline using 2 calibration products (Channel Mask Calibration Product and the Instrument Mode Mask Calibration Product). All masks fall into 1 of 3 categories depending on their impact and severity;
Note that in addition to the individual mask types, there is also a Master Bit to provide a quick reference as to whether a data sample is or is not scientifically valid which is set if any unusable flag is raised. Masks are stored as a bit mask and are described below in Table 8.1 and Table 8.2 (detailed information can be found in the "SPIRE Pipeline Mask Policy" document SPIRE-BSS-DOC-003127);
Table 8.1. Description of SPIRE masks
|Bit no.||Bit value||Mask Name||Mask Type||Description|
|0||1||MASTER||unusable||If set, then the data should not be used|
|1||2||INVALID_TIME||unusable||Set in Level 0-0.5 pipeline for sample has an invalid time|
|2||4||ADC_LATCH||unusable||Set in Level 0-0.5 pipeline for possible ADC latchup error|
|3||8||TRUNCATED||correctable||Set in Level 0-0.5 pipeline for ADC conversion truncated (saturated)|
|4||16||TRUNCATED_ UNCORR||correctable||Set in Clipping Correction module for ADC truncation not corrected|
|7||128||ISDEAD||unusable||Set in Level 0-0.5 pipeline for dead detector|
|7||128||UNRELIABLE_ PARAM||unusable||Set in Level 0-0.5 pipeline for unreliable telemetry parameter|
|8||256||ISNOISY||informational||Set in Level 0-0.5 pipeline for noisy detector|
|9||512||ISNOCHOPSKY||informational||Set in Level 0-0.5 pipeline for channel chopped off array FoV|
|10||1024||VOLTAGE_OOL||informational||Set in Flux-Conversion (phot) or Non-linearity (spec) module for a voltage value outside range of fitted bolometer responsivity curve|
|11||2048||GLITCH_FIRST_LEVEL||correctable||Glitch detected in 1st level deglitching|
|12||4096||GLITCH_FIRST_ LEVEL_UNCORR||correctable||Glitch detected in 1st level deglitching not removed|
Table 8.2. Description of SPIRE masks (.. continued)
|Bit no.||Bit value||Mask Name||Mask Type||Description|
|13||8192||GLITCH_SECOND_ LEVEL||correctable||Glitch detected in 2nd level deglitching|
|14||16384||GLITCH_SECOND_ LEVEL_UNCORR||correctable||Glitch detected in 1st level deglitching not removed|
|15||32768||ISSLOW||informational||Set in Level 0-0.5 pipeline if detector deemed as slow|
|16||65536||VOLTAGE_BELOW_K3||unusable||Flux conversion (Photometer) or Non-linearity correction (Spectrometer) module encountered a Voltage sample with value < K3 calibration factor|
|17||131072||NO_RESP_DATA||unusable||Set in Level 0-0.5 pipeline if flux conversion (Photometer) or linearization (Spectrometer) is not possible|
|18||262144||TSIGNAL_HDV||informational||thermistor/DP signal deviations larger than expected from Temperature Drift Correction module|
|19||524288||BSM_CHOP_OOL||informational||Set in BSM module if chop motion outside soft limits|
|20||1048576||BSM_JIGG_OOL||informational||Set in BSM module if jiggle motion outside soft limits|
|21||2097152||JUMP_THERMISTORS_ DARKS_SIGNAL||informational||Set by Jump Detection module for sudden signal jumps in the timelines of thermistors or dark channels. Used by the Temperature Drift Correction module|
|22||4194304||NO_THERMISTOR_ AVAILABLE||informational||Set by Temperature Drift Correction module if all available thermistors are affected by saturation or signal jump. Temperature Drift Correction module will try to use the dark channels.|
|23||8388608||NON_NOMINAL_ VELOCITY||informational||Set by Associate Sky Position module for all data between nominal scan legs|
|24||16777216||NO_DARKCHANNEL_ AVAILABLE||informational||Set by Temperature Drift Correction module if all available dark channels are affected by saturation or signal jump. In this case the module will not be able to correct the corresponding building block|
The mask and the masked data can be investigated in detail - either by using the Detector Timeline Viewer (see Section 8.2), the SPIRE Mask Editor Viewer, or by writing a script to examine the Mask Table of the detector timelines, etc.
From HIPE v9 onwards, two SpireMask editors, "Level0_5 SpireMaskEditor" and "Level1 SpireMaskEditor", are available. The "Level0_5 SpireMaskEditor" works for Level-0.5 data and "Level1 SpireMaskEditor" works for Photometer Level-1 data. From HIPE v11 onwards, the Spectrometer "Level1 MaskEditor" is also available, and it works for Spectrometer Level-1 interferogram data. The viewers are launched from HIPE by highlighting the Observation Context in the Variables tab, and then selecting the "open with" menu item by right clicking the mouse button. The Level0_5 SpireMaskEditor, Level1 SpireMaskEditor or Spectrometer Level1 MaskEditor should be available in the list. Note that the Level-0.5 or Level-1 context must have been loaded into HIPE for the mask editor to appear in the list. The GUI will pop up in the HIPE editor tab, for example, see Figure 8.37 and Figure 8.38. The viewers list all the masks from Table 8.1 and Table 8.2.
The circular radio buttons on the left select which mask bit will be used with the buttons in the right column of the panel
(Show/Edit, Set All, etc.). The squares will light up green if the corresponding mask bit in the selected read-out (lower
right selector) is set and they can also be edited (toggled) by a mouse click to switch them on and off. The white fields
show the number of masks that have the corresponding bit set for the selected building block. The individual mask bit for
each read-out can be edited by mouse click. The whole mask array for a selected mask bit and a selected channel can be edited
Unset All, and
Invert All buttons. The same changes can be also achieved by using
Show/Edit button through marking an area to select/de-select.
Note that the viewers are no longer accessible by right clicking the mouse on the Observation Context if the level-x data is not loaded (if the context has not been loaded, it will appear in red in the Observation Context Viewer).
The example shown in Figure 8.37 is for a SPIRE Spectrometer mapping observation (OBSID 1342187088 or 0x50002650 in hexadecimal). In this observation some
of the detectors have been saturated such that some clipping has occured. In order to show the samples that have been flagged,
the procedure in the spireMaskEditor is: select the FTS scan building block (0xa106000x) in the drop down menu of BBIDs at
the top of the editor window, select the the
truncated mask bit by clicking on the radio button next to it, click
Show/Edit on the right hand side of the window – this will pop up a Table Plotter window, in which the detector plotted can be selected.
Masked data samples are indicated with a red cross. In the example in Figure 8.39 it is clear that both the maximum and minimum of the interferogram have been flagged as clipped (the red crosses are the
flagged data points). In the Table Plotter, mask bits can be set by clicking the
Hide button followed by dragging a rectangle around the signals where the mask bit should be set. Mask bits can be unset by using
Unhide button instead. Upon closing the Table Plotter the changes will already be reflected in the main
MaskEditorTool panel, but they will only be committed to the observation context if the
Save button is used to finally close the
Note that the SPIRE Mask Editor Tool is not the only way to view and manipulate masks. The Detector Timeline Viewer (See
Section 8.2) can also be used to display flagged samples. For the above example, to open this tool, right click on the Spectrometer Detector
Timeline (SDT, or equivalent PDT for the photometer) in the
Variables tab of HIPE and
openWith Detector Timeline Viewer. Mask bits can be selected using the
Color&Mask Preferences button, and the flagged samples are indicated in the plot.
The same information can be obtained in a script by accessing the mask table of each dataset. The method of accessing the mask table for our example of the Spectrometer Detector Timeline (SDT) is shown below but the prinicipal applies equally to photometer observations. The following lines of code operate on the SDT to create a list of detectors with clipped samples:
clipped =  notclipped =  for chan in sdt.channelNames: clippedSamples = sdt['mask'][chan].data.where(SpireMask.TRUNCATED) if clippedSamples.length(): clipped.append(chan) else: notclipped.append(chan)
A similar method can be used to examine each scan in the SDI product – for example, to find the clipped samples in scan 0001 for detector SSWD4: