11.6. Setting and Clearing Flags with command line tools

When you set or clear flags with the Spectrum Explorer, the command-line version appears in the console window and can be cut and paste for inclusion in a script. An example is shown below. We recommend that you use the following four flags, and that you use the flag name (e.g. HifiMask.BRIGHT_LINE.value):

Flag Name Bit Software Name / Recommended syntax Description Flag value
Spur candidate 7 HifiMask.SPUR_CANDIDATE / HifiMask.SPUR_CANDIDATE.value If this bit is set, the sample is a candidate to be a spur. It is a 'candidate' since not all things flagged by the spurfinder are necessarily spurs. 128
line 28 HifiMask.LINE / HifiMask.LINE.value User set flag to denote a line 268435456
Bright line 29 HifiMask.BRIGHT_LINE / HifiMask.BRIGHT_LINE.value User set flag to denote a bright line 536870912
Ignore data 30 HifiMask.IGNORE_DATA / HifiMask.IGNORE_DATA.value User set flag to ignore data. 1073741824

# You cannot work from the observationContext therefore, you must create a variable when working
# with flagPixels.
# Here we first extract the HTP
htp = obs.refs["level2"].product.refs["WBS-V-USB"].product

# and then extract dataset 0004 from the HTP i.e. this would correspond to
# obs.refs["level2"].product.refs["WBS-V-USB"].product.refs["box_001"].product["0004"]
sds=htp.get(0004)

# Flag some data where mask is mask={subband:[channel_numbers]} with channel numbers listed one by one (note
# that you cannot give a range of channels)
# and where selection[3] represents the third spectrum (row) of the spectrum datasets i.e.
# obs.refs["level2"].product.refs["WBS-V-USB"].product.refs["box_001"].product["0004"].getPointSpectrum(3)
flagPixels(ds=sds, selection=[3], mask={3:[1239,1240,1241]}, flag=HifiMask.BRIGHT_LINE.value)

# Re-insert flagged dataset sds back into htp
htp.set(sds,4)

# Followed by a re-insertion of the htp into the obsContext
obs.level["level2"].setProduct("WBS-V-USB",htp)

Command-line example if we choose flag & remove:

flagPixels(ds=sds,selection=[3], mask={3:[1679]}, setFluxToNaN=1, flag=HifiMask.BRIGHT_LINE.value)

This flagging task is not very user friendly. The mask represents pixel values within the spectrum. Because it is easier to identify regions by frequency ranges, we recommend that you use the new task flagTool (see Section 11.5).

Alternatively, you may wish to flag many spectra within an HTP where you have identified a problem at a specific frequency range.

# We first define the flagging routine using, as an example, the recommended "ignore" flag
# "HifiMask.IGNORE_DATA" (see table above)

def flagData(htp,htpindex,freq1,freq2) :

   spectrum=htp.get(htpindex)
   spectrumType=spectrum.meta["sds_type"].value # determine type (COMB, HC, SCIENCE, TUNE)

   nrows=spectrum.getLength()  # get number of spectrum in this dataset
   for dsidx in range(nrows) :  # loop through them
     for sb in (spectrum.getSegmentIndices()): # loop through the subbands
       segment = spectrum.getPointSpectrum(dsidx).getSegment(sb)
       flag=segment.getFlag() # fetch the flags
       freq=segment.getWave() # fetch the frequencies
       idx=freq.where((freq>=freq1).and(freq<=freq2))
       if idx.length()>0 :
         flag[idx]=HifiMask.IGNORE_DATA.set(flag[idx])  # set the flag
         segment.setFlag(flag)  # reinsert into the spectrum
   htp.set(htpindex,spectrum,spectrumType) # reinsert modified spectra back into the HTP
   return htp

#  START OF SCRIPT

# read in your data using obs = getObservation("1342xxxxxx")
obs = getObservation("1342205481")

# extract the Level 2 tree
lev2=obs.getProduct("level2")

# from that, extract the HTP for the WBS-H backend
htpwbsh=lev2.getProduct("WBS-H-USB")

# At this stage, you have a HTP that needs some cleaning.
#
# Inspect the data and identify those affected datasets, then run the cleaning function flagData.

# In our example, we wish to flag the spectra from 3 to 6 in the frequency range 1900.49 to 1900.63

htpwbsh=flagData(htpwbsh,3,1900.49,1900.63)
htpwbsh=flagData(htpwbsh,4,1900.49,1900.63)
htpwbsh=flagData(htpwbsh,5,1900.49,1900.63)
htpwbsh=flagData(htpwbsh,6,1900.49,1900.63)

# Now with everything flagged, re-insert the data (i.e. the HTP) back into the observation context "obs"

lev2.setProduct("WBS-H-USB",htpwbsh)
obs.setProduct("level2",lev2)

# Using Spectrum Explorer, verify, in the obsContext, that the HTP was re-inserted into the obsContext and that
# the flags were applied accordingly (see Section 11.2).

Level 2.5 'stitched' cube

Command line example to set a flag in a given spectrum of a Level 2.5 WBS 'subbands stitched' cube (pipeline product). Here, the parameter selection is the pixel number within the cube. Pixel numbers are labelled from 0 (lower-left corner), and increase from left to right and from bottom to top. Please be aware, however, that flags with values above 32767 (short integer limit) will not be honoured by the task in cubes as those latter are set as short integer arrays.

flagPixels(ds=obs.refs["level2_5"].product.refs["cubesContext"].product.refs["cubesContext_WBS-V-USB"].product.refs["cube_WBS_V_USB_1"].product, \
selection=[31], mask={0:[1268]}, flag=HifiMask.SPUR_CANDIDATE.value)