8.4. SPIRE Mask Handling

8.4.1. SPIRE Mask Formalization

SPIRE data contains various kinds of (Boolean) masks, used to indicate problems or caveats associated with the data. For consistency, SPIRE Masks follow the rule that the Boolean false state represents that a mask condition is not applicable.

The SPIRE data processing supports three general categories of masks.

  • Channel Masks: are applied to all data from a given channel (bolometer, thermistor, resistor, etc.) and thus affects all time samples for that channel. The Channel Masks are contained within the Channel Mask Table calibration product (see RD1 for definition). They indicate whether a signal channel is useful or not, such as whether that particular channel is dead.
  • Instrument Mode Masks: are effectively channel masks that are applied only to data from specific observing modes. The observing mode masks are contained within the Instrument Mode Mask Table calibration product (see RD1 for definition). Following the same procedure as for Channel Masks, they indicate whether a signal channel is useful or not for that specific observing mode
  • Sample Masks: All data products contain a sample mask table added at the Reformat Level 0 Product stage. In this table, a 32- bit integer is reserved for each sample for each detector, referred to as a Sample Mask. Mask information is represented by bits in a Sample Mask, with different bits representing different mask conditions.

Much of the sample mask in the data is populated during the Level 0 - Level 0.5 processing pipeline using 2 calibration products (Channel Mask Calibration Product and the Instrument Mode Mask Calibration Product). All masks fall into 1 of 3 categories depending on their impact and severity;

  • Unusable: reserved for data samples that are afflicted with critical problems to the degree that the data samples should not be considered scientifically valid and are ignored by the pipeline modules.
  • Informational: represent non-critical problems with the detector data samples for which there exists no correcting data processing module. Data processing modules that encounter a Information condition should process the sample as normal and propagate the Sample Mask Table.
  • Correctable: conditions for which a data processing module exists that may be able to correct the condition. These masks always come in pairs: one mask will denote the identification of the condition; the second will denote whether the condition has been corrected.

Note that in addition to the individual mask types, there is also a Master Bit to provide a quick reference as to whether a data sample is or is not scientifically valid which is set if any unusable flag is raised. Masks are stored as a bit mask and are described below in Table 8.1 and Table 8.2 (detailed information can be found in the "SPIRE Pipeline Mask Policy" document SPIRE-BSS-DOC-003127);

Table 8.1. Description of SPIRE masks

Bit no. Bit value Mask Name Mask Type Description
0 1 MASTER unusable If set, then the data should not be used
1 2 INVALID_TIME unusable Set in Level 0-0.5 pipeline for sample has an invalid time
2 4 ADC_LATCH unusable Set in Level 0-0.5 pipeline for possible ADC latchup error
3 8 TRUNCATED correctable Set in Level 0-0.5 pipeline for ADC conversion truncated (saturated)
4 16 TRUNCATED_ UNCORR correctable Set in Clipping Correction module for ADC truncation not corrected
5 32 GLITCH ....... not used
6 64 GLITCH_UNCORR ....... not used
7 128 ISDEAD unusable Set in Level 0-0.5 pipeline for dead detector
7 128 UNRELIABLE_ PARAM unusable Set in Level 0-0.5 pipeline for unreliable telemetry parameter
8 256 ISNOISY informational Set in Level 0-0.5 pipeline for noisy detector
9 512 ISNOCHOPSKY informational Set in Level 0-0.5 pipeline for channel chopped off array FoV
10 1024 VOLTAGE_OOL informational Set in Flux-Conversion (phot) or Non-linearity (spec) module for a voltage value outside range of fitted bolometer responsivity curve
11 2048 GLITCH_FIRST_LEVEL correctable Glitch detected in 1st level deglitching
12 4096 GLITCH_FIRST_ LEVEL_UNCORR correctable Glitch detected in 1st level deglitching not removed


Table 8.2. Description of SPIRE masks (.. continued)

Bit no. Bit value Mask Name Mask Type Description
13 8192 GLITCH_SECOND_ LEVEL correctable Glitch detected in 2nd level deglitching
14 16384 GLITCH_SECOND_ LEVEL_UNCORR correctable Glitch detected in 1st level deglitching not removed
15 32768 ISSLOW informational Set in Level 0-0.5 pipeline if detector deemed as slow
16 65536 VOLTAGE_BELOW_K3 unusable Flux conversion (Photometer) or Non-linearity correction (Spectrometer) module encountered a Voltage sample with value < K3 calibration factor
17 131072 NO_RESP_DATA unusable Set in Level 0-0.5 pipeline if flux conversion (Photometer) or linearization (Spectrometer) is not possible
18 262144 TSIGNAL_HDV informational thermistor/DP signal deviations larger than expected from Temperature Drift Correction module
19 524288 BSM_CHOP_OOL informational Set in BSM module if chop motion outside soft limits
20 1048576 BSM_JIGG_OOL informational Set in BSM module if jiggle motion outside soft limits
21 2097152 JUMP_THERMISTORS_ DARKS_SIGNAL informational Set by Jump Detection module for sudden signal jumps in the timelines of thermistors or dark channels. Used by the Temperature Drift Correction module
22 4194304 NO_THERMISTOR_ AVAILABLE informational Set by Temperature Drift Correction module if all available thermistors are affected by saturation or signal jump. Temperature Drift Correction module will try to use the dark channels.
23 8388608 NON_NOMINAL_ VELOCITY informational Set by Associate Sky Position module for all data between nominal scan legs
24 16777216 NO_DARKCHANNEL_ AVAILABLE informational Set by Temperature Drift Correction module if all available dark channels are affected by saturation or signal jump. In this case the module will not be able to correct the corresponding building block


8.4.2. SPIRE Mask Editor Viewer

The mask and the masked data can be investigated in detail - either by using the Detector Timeline Viewer (see Section 8.2), the SPIRE Mask Editor Viewer, or by writing a script to examine the Mask Table of the detector timelines, etc.

From HIPE v9 onwards, two SpireMask editors, "Level0_5 SpireMaskEditor" and "Level1 SpireMaskEditor", are available. The "Level0_5 SpireMaskEditor" works for Level-0.5 data and "Level1 SpireMaskEditor" works for Photometer Level-1 data. From HIPE v11 onwards, the Spectrometer "Level1 MaskEditor" is also available, and it works for Spectrometer Level-1 interferogram data. The viewers are launched from HIPE by highlighting the Observation Context in the Variables tab, and then selecting the "open with" menu item by right clicking the mouse button. The Level0_5 SpireMaskEditor, Level1 SpireMaskEditor or Spectrometer Level1 MaskEditor should be available in the list. Note that the Level-0.5 or Level-1 context must have been loaded into HIPE for the mask editor to appear in the list. The GUI will pop up in the HIPE editor tab, for example, see Figure 8.37 and Figure 8.38. The viewers list all the masks from Table 8.1 and Table 8.2.

The circular radio buttons on the left select which mask bit will be used with the buttons in the right column of the panel (Show/Edit, Set All, etc.). The squares will light up green if the corresponding mask bit in the selected read-out (lower right selector) is set and they can also be edited (toggled) by a mouse click to switch them on and off. The white fields show the number of masks that have the corresponding bit set for the selected building block. The individual mask bit for each read-out can be edited by mouse click. The whole mask array for a selected mask bit and a selected channel can be edited by using Set All, Unset All, and Invert All buttons. The same changes can be also achieved by using Show/Edit button through marking an area to select/de-select.

Note that the viewers are no longer accessible by right clicking the mouse on the Observation Context if the level-x data is not loaded (if the context has not been loaded, it will appear in red in the Observation Context Viewer).

The SPIRE Mask Editor

Figure 8.37. The SPIRE Mask Editor


The SPIRE Level 1 Sprectrometer Mask Editor

Figure 8.38. The SPIRE Level 1 Sprectrometer Mask Editor


The example shown in Figure 8.37 is for a SPIRE Spectrometer mapping observation (OBSID 1342187088 or 0x50002650 in hexadecimal). In this observation some of the detectors have been saturated such that some clipping has occured. In order to show the samples that have been flagged, the procedure in the spireMaskEditor is: select the FTS scan building block (0xa106000x) in the drop down menu of BBIDs at the top of the editor window, select the the truncated mask bit by clicking on the radio button next to it, click Show/Edit on the right hand side of the window – this will pop up a Table Plotter window, in which the detector plotted can be selected. Masked data samples are indicated with a red cross. In the example in Figure 8.39 it is clear that both the maximum and minimum of the interferogram have been flagged as clipped (the red crosses are the flagged data points). In the Table Plotter, mask bits can be set by clicking the Hide button followed by dragging a rectangle around the signals where the mask bit should be set. Mask bits can be unset by using the Unhide button instead. Upon closing the Table Plotter the changes will already be reflected in the main MaskEditorTool panel, but they will only be committed to the observation context if the Save button is used to finally close the MaskEditorTool.

Clipped (truncated) samples masked in the interferogram

Figure 8.39. Clipped (truncated) samples masked in the interferogram


8.4.3. Other ways to view SPIRE masks

Note that the SPIRE Mask Editor Tool is not the only way to view and manipulate masks. The Detector Timeline Viewer (See Section 8.2) can also be used to display flagged samples. For the above example, to open this tool, right click on the Spectrometer Detector Timeline (SDT, or equivalent PDT for the photometer) in the Variables tab of HIPE and openWith Detector Timeline Viewer. Mask bits can be selected using the Color&Mask Preferences button, and the flagged samples are indicated in the plot.

The same information can be obtained in a script by accessing the mask table of each dataset. The method of accessing the mask table for our example of the Spectrometer Detector Timeline (SDT) is shown below but the prinicipal applies equally to photometer observations. The following lines of code operate on the SDT to create a list of detectors with clipped samples:

clipped = []
notclipped = []
for chan in sdt.channelNames:
  clippedSamples = sdt['mask'][chan].data.where(SpireMask.TRUNCATED) 
  if clippedSamples.length():
    clipped.append(chan)
  else:
    notclipped.append(chan)

A similar method can be used to examine each scan in the SDI product – for example, to find the clipped samples in scan 0001 for detector SSWD4:

clippedSamples=sdi['0001']['SSWD4']['mask'].data.where(SpireMask.TRUNCATED)