11.5. Setting flags interactively for many spectra

The flagTool task allows you to interactively flag (via a graphical interface) rows and channels in HIFI for a given spectrometer. The task provides a user friendly environment and allows you to easily rerun the task and modify your masks. With the concept of an interactive table of datasets provided with the plot (left side of the plot), it is visually easy to see where your flags are (per dataset and per subband), and to choose which dataset to examine.

The flagTool task is applied on an ObservationContext (recommended) or a HifiTimelineProduct (htp) and once selected, you will find flagTool in the Applicable Tasks menu where you can open the GUI from there (see Figure 11.7). The ObservationContext (or htp) will then be loaded into the product bullet. You can also open the GUI from the HIFI list of tasks under By Category. Please note that we recommend using the flagTool task on an ObservationContext as opposed to a htp. This is because the modifications applied on a htp is not conserved unless you re-insert the htp into the ObservationContext. This can be achieved by using the following command line:

obs.getProduct(level).setProduct(backend, htp)

Capabilities.

(Mac users: If you are using a trackpad instead of a 3-button mouse, you need to configure your trackpad for One Finger and Two Fingers. And, in order to emulate the middle-click mouse button, you just use Ctrl + two-finger tap.)

You can select the spectrometer (HRS or WBS), the polarisation (H or V), and the sideband (LSB or USB) to be processed by using the backend drop-down menu (e.g. WBS-H-USB, WBS-V-LSB, HRS-V, WBS-H, etc...) in the GUI (for the command line syntax, see examples at the end of this section). The level of the data to be processed must also be selected using the drop-down menu called level. Note that for Level 1, the backend choices should be HRS-H, HRS-V, WBS-H, or WBS-V i.e. spectrometer and polarisation. For Level 2, you must add the sideband choice. If your selection is wrong, e.g. if you choose backend=WBS-H-USB, and level= level1, the task will end and you will see a descriptive error message in the Console window. Simply reselect the proper level, are restart the task.

If you select the HRS backend, note that prior to HIPE 9.0, HRS data did not contain a flag column, making it impossible to flag HRS spectra. If the flag column does not exist, the task will send a comment Column name 'flag_1' not found in the console window and the task will not start. Therefore, if you wish to flag HRS data, check that the SPG version (you can find this information in the ObservationContext metadata) begins with 9.0 or higher. If it does not, you will need to reprocess your observation from Level 0 using the pipeline from HIPE 9.0 onwards (see Chapter 5).

You are also able to choose to run flagTool with the flags category set to either channel only, rowflags only, or with both categories (the default setting), i.e. by selecting both.

FlagTool task GUI

Figure 11.7. FlagTool task GUI


By default, flagTool display the first dataset, and all subbands at once, thus allowing you to process all subbands of a dataset in one go. This feature allows you to process your data with significantly reduced time, and to flag regions where two subbands connect. But you can also choose to plot one subband at a time for a given dataset with one right-click (or left-click) in a subband box. It is fast and easy to choose which dataset to process by selecting (right-click or left-click in the ds column) a dataset in the interactive table on the left side of the plot (see Figure 11.8). Note that you work on one dataset at a time i.e. that flagging ds 2, subband 3 will not be propagated to the other datasets. Therefore, you need to examine all datasets to establish which ones need flagging.

Once the task is invoked, you will be presented with a datasets table containing, at least, column 1: ds (dataset), column 2: row, and column 3: LO value associated with the dataset. Then, prior to running the task, if you chose category = channel, you will see additional columns associated with each subband. If you chose category = rowflag, you will see column 1: ds (dataset), column 2: row, column 3: LO value associated with the dataset, and additional columns with the name of all rowflags raised (in at least one spectrum) in your datasets except UNALIGNED_HK since this rowflag is very common in most datasets. In addition to the rowflags raised in your data, you will also see the two rowflags SUSPECT_LO and IGNORE_DATA. Note that these two rowflags are the only rowflags that you are permitted to modify. The rowflags raised by the pipeline are read only and will not be editable. A cross (or a marker) will indicate that the rowflag is present in a given dataset. If you chose category = both, you will see all columns from channel and rowflag displayed.

FlagTool datasets table and plot showing two flagged regions (coloured 'curtains') using two different flags

An interactive table listing all the datasets of an HTP is provided on the left side of the plot. This example shows the category = channel table.

Figure 11.8. FlagTool datasets table and plot showing two flagged regions (coloured 'curtains') using two different flags


FlagTool messages in the console

Summary of the HTP being processed and a list of available actions appearing in the console.

Figure 11.9. FlagTool messages in the console


Interactive and Expert modes.

By default the interactive option is set to 'True'. If you wish to use flagTool in batch mode, you can unselect ('False') this option. The interactive mode defines the verbosity of flagTool: if set to 'True', all actions on the flags are shown in the Console window.

By default, the expert option is set to 'False'. Expert mode set to 'True' affects only the row flags by focusing only on the IGNORE_DATA flags. If combined with Interactive = 'False', the text in the Console window is kept to its minimum.

Details of channel flagging.

The channel flagging technique consist of 1] choosing a flag type, and 2] defining a mask region. By default, the flag will be set to LINE but you can choose a flag from a sets of four types of masks via four clickable choices situated at the top of the plot. You have a choice of SPUR_CANDIDATE (value= 2^7 Orange), LINE (value=2^28 Pink), BRIGHT_LINE (value=2^29 Blue), and IGNORE_DATA (value=2^30 Green). Once you start the task, the console will show a summary of the HTP being processed, and will show a list of available actions (see Figure 11.9).

You define a mask region (i.e. a frequency range) by holding shift and left-click on the left border of a region and by dragging to the right border of the region. The mask is then visible as a coloured 'curtain' (see Figure 11.8). To undo the most recent flag, use the Undo button at the bottom of the plot. Note that the Undo button only works right after you have assigned a mask, and if you have not moved to a new dataset. But, if you need to undo any masks, you can use one left-click within the coloured 'curtain'. If you choose a flag type and apply it but then decide that you need to re-assign a different type to the region, or if you need to modify the region, you must first disable the mask with a one left-click within the coloured 'curtain', choose a flag type, and then re-define the mask region.

To understand the implication of those flags, we strongly suggest that you consult the chapter covering flags in HIFI (see Chapter 10).

You may also zoom in and out by drawing a box around the region of interest or by using the mouse wheel (Mac trackpad: two-finger pinch, or two-finger drag across the pad). Using the right-click will allow you to use the default menu of PlotXY.

If a flag is set, flagTool will:

Note that SPUR_CANDIDATE and IGNORE_DATA prevail upon LINE and BRIGHT_LINE. As a consequence, the flag value of the channels you flagged as both SPUR_CANDIDATE (or IGNORE_DATA) and LINE (or BRIGHT_LINE) will only be SPUR_CANDIDATE (or IGNORE_DATA).

As you progress in flagging your data, please pay attention to the comments available from the Console window. Some warnings may be important about the choice of flags you may have used. Once you are done flagging your datasets, you may quite the task (Quit option at the bottom of the plot).

Handling rowflags.

The purpose of listing, in the interactive table, all rowflags raised by the pipeline (except for UNALIGNED_HK), is to allow you to have a clear overview of potential problems in your data. As mention earlier, the only two rowflags that you should, if necessary, modify are SUSPECT_LO and IGNORE_DATA. You may raise a flag, or remove one raised by the pipeline if you judge that the quality of the dataset is good. A simple click will allow you to either select or deselect a rowflag in a given dataset.

MaskTables: Linemasks and Rowmasks files.

FlagTool stores the channel flags masks and row flags masks in mask tables. Channel flags masks are saved in the variable Linemasks and row flag masks are saved in the variable Rowmasks. You can find both of them in the HTP you processed, and view your masks by selecting the variable (see Figure 11.10). Mask tables have the same format and the same characteristics as TableDatasets so you should not be disoriented. They are specific data structures to handle masks and provide useful functions as we will see.

FlagTool Linemasks table

Figure 11.10. FlagTool Linemasks table


If the Linemasks and Rowmasks products are already present in an ObservationContext, you will see these masks already in the data and you can modify them (remove or select a different mask) and/or add new masks by just running the task again.

FlagTool also automatically creates a backup FITS file in your working directory from a channel flag mask table for every 5 masks that you set. This feature was implemented in order to avoid losing an extensive flagging session due to an unexpected termination of HIPE. The backup file will be named backup_obsid_backend.fits. If your flagging session was less than 5 masks then you will not see this backup file.

[Note] Note

You may not be able to use mask tables created from older HIPE versions due to an incompatibility in the table format. Work is in progress to implement a functionality to allow compatibility between the old and the new format.

While you can handle the mask tables like any other TableDataset (drag and drop, Send to FITS file...) flagTool provides you with specific functions to add, load, remove, and save channel flag and row flag mask tables.

The three operations that alter the mask tables (add, load, remove) use the same syntax:

<operation> MaskTables({backend: maskTable})

They should be called before the process method because they take into account the mask table provided, and analyze it before flagging the data.

Add new masks from a mask table

    ft = FlagTool()
    ft.addMaskTables({backend: maskTable})
    ft.process(obs, backend)

Only the difference between the mask table in the data and the mask table provided is taken into account. If the new mask table contains a mask that is not in the data, this mask will be added to the data:

How addMaskTables work

In the data In the maskTable Masks to be added Final maskTable
A A   A
B     B
  C C C
D     D
E     E

Load all the masks from a mask table

    ft = FlagTool()
    ft.loadMaskTables({backend: maskTable})
    ft.process(obs, backend)

All the previous user row flags (if maskTable contains row flags) or all the channel flags (if maskTable contains channel flags) will be removed and replaced by the new mask table provided:

How loadMaskTables work

In the data In the maskTable Final maskTable
A A A
B    
  C C
D D D
E    

Remove the masks contained in a mask table

    ft = FlagTool()
    ft.removeMaskTables({backend: maskTable})
    ft.process(obs, backend)

All the masks in your data that match the masks in maskTable will be removed:

How removeMaskTables work

In the data In the maskTable Masks to be removed Final maskTable
A A A  
B     B
  C    
D     D
E     E
[Note] Note
  • The rule about row flags: You can add or remove user row flags but not row flags set by the pipeline (see Section 10.3).

  • The rule about channel flags: You can remove any channel flag but you can only add user channel flags (see Section 10.2).

Save the masks to a FITS file

The last operation (save) should be called after the process method, when the flagging is done, and all the masks are updated in the mask table. It allows you to save your channel flag or row flag mask tables to a FITS file:

    ft = FlagTool()
    ft.process(obs, backend)
    ft.saveMaskTables(filename)

saveMaskTables looks for Linemasks and Rowmasks in the HTP. If any of these 2 mask tables is found, it is saved to a FITS file. Note that filename is optional. If it is provided, the file will be suffixed by _Linemasks or _RowMasks. If you do not provide a filename, the file will be saved in your working directory under the name <obsid>_<backend>_Linemasks.fits or <obsid>_<backend>_Rowmasks.fits.

A simple use case to handle mask tables

An example will make things clearer. Let's imagine you were setting channel flags with flagTool on the obsid 1342191559, on WBS-H-USB but something went wrong and your session was closed. Don't panic. flagTool made a backup of your channel masks in your working directory. You just have to reload your obsid and add these masks back into WBS-H-USB:

    # Load the observation context
    obs = getObservation(1342191559)

    # Create a FlagTool object
    ft = FlagTool()

    # Add the mask table named "backup_1342191559_WBS-H-USB.fits" into WBS-H-USB
    ft.addMaskTables({"WBS-H-USB": "backup_1342191559_WBS-H-USB.fits"})

    # Process WBS-H-USB
    ft.process(obs, "WBS-H-USB", level=2, category="channel")

FlagTool will read all the masks in the mask table backup_1342191559_WBS-H-USB.fits and set the corresponding flags. Messages in the HIPE console will list all the masks added.

After you set all the flags you wanted, you will probably want to save your masks:

    ft.saveMaskTables(filename="finished_1342191559_WBS-H-USB.fits")

FlagTool appended _Linemasks to your filename because it found only Linemasks in WBS-H-USB.

One backend is finished. Let's do the same for WBS-V-USB. But this time, we want exactly the same masks as in WBS-H-USB. You cannot use addMaskTables because it will only add the masks that are not already in WBS-V-USB, which means that the masks you removed in WBS-H-USB will not be removed in WBS-V-USB. What you want is to load all the masks as if no other masks previously existed in the data. This is when loadMaskTables comes in handy:

    ft = FlagTool()
    ft.loadMaskTables({"WBS-V-USB": "finished_1342191559_WBS-H-USB_Linemasks.fits"})
    ft.process(obs, "WBS-V-USB", level=2, category="channel")

All the channel flags existing in WBS-V-USB are now removed and replaced by the masks in finished_1342191559_WBS-H-USB_Linemasks.fits.

[Note] Note

As you can imagine, loadMaskTables can be dangerous and must be used carefully because it removes all the channel flags including those set by the pipeline. However, this behaviour is coherent with what you can do interactively in flagTool because it will only remove or add user row flags, preventing you from altering row flags set by the pipeline.

You are done! But if, for some reason, you had to remove some masks in WBS-V-USB, and let's say that these masks are stored in spurs_in_1342191559_WBS-V-USB.fits, there is a command for that:

    ft.removeMaskTables({"WBS-V-USB": "spurs_in_1342191559_WBS-V-USB.fits"})
    ft.process(obs, "WBS-V-USB", level=2, category="channel")

FlagTool will read the masks in spurs_in_1342191559_WBS-V-USB.fits and remove them from WBS-V-USB. Now you know everything about mask tables and how to handle them in flagTool.

FitHifiFringe and FitBaseline.

For a given dataset of Level 2, you may also choose to perform simultaneously fitHifiFringe and/or fitBaseline. The default values are to NOT perform those options. FlagTool allows you to define some of the main options of these two tools:

All other options relative to these two tools are set to their default value. This means that the option doglue (used in both fitHifiFringe and fitBaseline) will be set to the flagTool default i.e. to True. Some data show offsets between the subbands. If this is the case in your data, it is then unwise to use fitBaseline within the flagTool task because you will want to fit your baselines one subband at a time.

In order to apply fitHifiFringe or fitBaseline to the current dataset, select the corresponding button at the bottom of the GUI. FlagTool will run fitHifiFringe or fitBaseline with the options provided in the GUI. Note that the two fitting tools work exactly the same way as in standalone mode. Please refer to the corresponding documentation: Chapter 12 and Chapter 13.

flagTool using fitHifiFringe as an option

Figure 11.11. flagTool using fitHifiFringe as an option


flagTool using fitBaseline as an option

Figure 11.12. flagTool using fitBaseline as an option


FlagTool can also be used from the command line:

# Flag WBS-H data at Level 1 - by default, the category is set to "channel" flags
flagTool(obs, "WBS-H", level=1)

# Flag WBS-H-USB from the default Level 2
flagTool(obs, "WBS-H-USB")

# Flag WBS-H-USB data using a polynomial of order=3 to fit the baseline
flagTool(obs, "WBS-H-USB", order=3)

# Flag WBS-H-USB and use 2 sine waves for fitHifiFringe
flagTool(obs, "WBS-H-USB", nfringes=2)

# Flag WBS-H-USB but do not display the plot for FitHifiFringe
flagTool(obs, "WBS-H-USB",fhfPlot=0)

# Flag WBS-H-USB with the category rowflag
flagTool(obs, "WBS-H-USB", category="rowflag")

# Flag both channel flags and rowflags in WBS-H-USB
flagTool(obs, "WBS-H-USB", category="both")