5.2. How to run the HIFI Pipeline

[Warning] Warning

Although it is possible to modify parameters values and add certain tasks to be processed by the hifiPipeline, via the Customize Pipeline and the Interactive Level 2.5 Pipeline, please note that the order in which the tasks are to be performed is very important. The expert panel, accessed by pressing the go to expert mode... button at the top of the GUI also contains other options for running the pipeline. These are usually only of interest to HIFI ICC calibration scientists and are not typically relevant in the context of reprocessing data for science purposes. The order of the tasks shown in the GUI (and the flow diagrams shown in Chapter 4) constitute the default flow of the tasks. You may change the order only if you have a deep understanding of the pipeline. Wrongly changing the order may lead to obscure error messages in the Console window, and the pipeline not completing to the end. The tasks of the pipeline are described in details in the HIFI Pipeline Specification Document if you wish to learn in-depth details of each tasks.

The HIFI ICC recommends that the data should be processed using the HIPE version that you are using for data analysis to ensure that you do not run into data-software compatibility problems. However, it is not anticipated that you will have problems if you keep within one HIPE version, e.g. perform data analysis in HIPE 13 on data that was pipelined with HIPE 12.

The hifiPipeline task links together the five stages of the pipeline described above and it can be used to reprocess Observation Contexts up to any level, from any level, for any choice of spectrometer(s) and polarisation(s). The pipeline can be run via a GUI or the command line.

Opening the hifiPipeline GUI:

HIFI pipeline task: default view

Figure 5.1. HIFI pipeline task: default view


Running the hifiPipeline task:

The default (or basic) dialogue in the GUI allows you to re-process data in an observation context. It also allows you to:

The default set-up of the pipeline is to reprocess data from Level 0 to 2.5 for all four spectrometers (or as many as were used in the observation), using the calibration that can be found in the Observation Context and the default pipeline algorithms.

The expert panel, accessed by pressing the go to expert mode... button at the top of the GUI contains other options for running the pipeline. These are usually only of interest to HIFI ICC calibration scientists and are not typically relevant in the context of reprocessing data for science purposes.

Below are some examples of running the hifiPipeline task from the command line. It is assumed that an Observation Context called Myobs has been loaded into the session.

# Reprocess an ObservationContext up to Level 2 for all spectrometers,
# without writing to memory to decrease processing time:
MyNewobs = hifiPipeline(obs=Myobs, save=False)
#
# Reprocess Myobs from Level 0.5 to Level 1, for all spectrometers:
MyNewobs = hifiPipeline(obs=Myobs, fromLevel=0.5, upToLevel=1, save=False)
#
# Now reprocess MyNewobs (which now contains data only up to Level 1) but only for the WBS.
# WBS-H and WBS-V are the horizontal and vertical polarisations, respectively:
MyEvenNewerobs = hifiPipeline(obs=MyNewobs, apids=['WBS-H', 'WBS-V'])
#
# What is an apid?  "Application Program IDentifier":  it is what the pipeline calls spectrometers.
#
# You can retain Level 0.5
obs = hifiPipeline(obs=obs, upToLevel=2, removeLevel0_5=False)

Pipeline behaviour to note:

Saving the output:

There are several methods you can use to save your reprocessed observation in a pool (also known as a local store). See the Data Analysis Guide, chapter 1 for more information about pools.)

Running the pipeline for multiple observations:

The script below will retrieve a list of Observation Contexts from pools, reprocess them, then save the reprocessed data in another pool.

list = [1342xxxxxx, 1342yyyyyy, 1342zzzzzz]
for obsid in list:
    obs=getObservation(obsid)
    obs = hifiPipeline(obs=obs, save=False)
    pool = '%i_new' %(obsid)
    saveObservation(obs, poolName=pool)

The script assumes that the original Observation Contexts are stored in pools with the name of the observation number, i.e., observation 1342190798 is stored in a pool with name 1342190798, and in this case, the output pool in 1342190798_new.