Although it is possible to modify parameters values and add certain tasks to be processed by the
The HIFI ICC recommends that the data should be processed using the HIPE version that you are using for data analysis to ensure that you do not run into data-software compatibility problems. However, it is not anticipated that you will have problems if you keep within one HIPE version, e.g. perform data analysis in HIPE 13 on data that was pipelined with HIPE 12.
hifiPipeline task links together the five stages of the pipeline described above and it
can be used to reprocess Observation Contexts up to any level, from any level, for any choice of spectrometer(s) and
polarisation(s). The pipeline can be run via a GUI or the command line.
Opening the hifiPipeline GUI:
Click once on an observation context in the Variables pane and the
task will appear in the Applicable Tasks folder, double click on it to open the Task dialogue in the
Alternatively, open the
hifiPipeline task by double-clicking on it under the Hifi Category in the
Running the hifiPipeline task:
The default (or basic) dialogue in the GUI allows you to re-process data in an observation context. It also allows you to:
Pass any version calibration to the pipeline, see Section 5.3 for details on passing a new calibration version to the pipeline.
Change the order of tasks and introduce new ones in the Level 2.5 pipeline with the Interactive Pipeline, see Section 5.4.1.
Enable or disable the algorithms to process the OFF spectra in the observation or calculate the rms noise in the Level 2 and/or Level 2.5vdata.
The default set-up of the pipeline is to reprocess data from Level 0 to 2.5 for all four spectrometers (or as many as were used in the observation), using the calibration that can be found in the Observation Context and the default pipeline algorithms.
The way the data is to be reprocessed is defined in the Inputs section:
hifiPipeline task was opened from the Applicable Tasks folder then
the Observation Context selected in the Variables View will automatically be loaded into the Task
dialogue, and you will see its name by the observation context bullet, which will be green. Alternatively, drag the
name of the observation context to be reprocessed from the Variables view to the ObservationContext bullet.
Select the spectrometers you wish to process data for by checking the desired combination of instrument(s) and polarisation(s). By default, the box for a given spectrometer is checked only if data is present at Level 0.
Select which levels to (re-)process 'from and to' via the drop-down menus. By default the pipeline will process Level 0 data up to Level 2.5 but you can choose any start and end point you wish. The lowest level that you can re-process from is Level 0. An option to process from Level -1 is available but it is meant for use by the HSA and the ICC. Processing from Level -1 to Level 0 (the Level 0 pipeline) involves the combination of the unsorted raw data and housekeeping frames, which you do not have access to.
Enable or disable the algorithms to process the OFF spectra in the observation with the processOff parameter. If enabled, the OFF spectra will be processed up to Level 2 (sky frequency and antenna temperature) and stored in the Calibration Tree in the Pipeline-out node. By default the algorithm is enabled, to disable it, uncheck the processOff box.
Enable or disable the algorithm to calculate the rms noise in the Level 2 and/or Level 2.5 data with the
computeRms option. If the computeRms option is enabled,
mkRms task is enabled in both the Level 2 and the Level 2.5. You may choose to leave both actions
enabled, or disable one of them. The
mkRms task can be run in the Level 2 pipeline
mkRms algorithm can be run in the Level 2.5 pipeline. The
resulting tables are stored in the Trend Analysis node in the Observation Context. By default, this task
is run during bulk reprocessing at the HSA but is turned off in HIPE because it can be time intensive. To enable the algorithm,
check the computeRms box.
In the Outputs section, choose the name of the observation context that will be produced or use the HIPE default,
Click on Accept to run the pipeline. The status ("running" if all is well, error messages if not) and the progress of the pipeline are given in the Info section at the bottom of the Task dialogue. The command to run the pipeline is echoed in the console. You will also see more informative messages about the status of the pipeline written in the console and terminal.
The expert panel, accessed by pressing the go to expert mode... button at the top of the GUI contains other options for running the pipeline. These are usually only of interest to HIFI ICC calibration scientists and are not typically relevant in the context of reprocessing data for science purposes.
Below are some examples of running the
hifiPipeline task from the command line. It is assumed that an Observation
Myobs has been loaded into the session.
# Reprocess an ObservationContext up to Level 2 for all spectrometers, # without writing to memory to decrease processing time: MyNewobs = hifiPipeline(obs=Myobs, save=False) # # Reprocess Myobs from Level 0.5 to Level 1, for all spectrometers: MyNewobs = hifiPipeline(obs=Myobs, fromLevel=0.5, upToLevel=1, save=False) # # Now reprocess MyNewobs (which now contains data only up to Level 1) but only for the WBS. # WBS-H and WBS-V are the horizontal and vertical polarisations, respectively: MyEvenNewerobs = hifiPipeline(obs=MyNewobs, apids=['WBS-H', 'WBS-V']) # # What is an apid? "Application Program IDentifier": it is what the pipeline calls spectrometers. # # You can retain Level 0.5 obs = hifiPipeline(obs=obs, upToLevel=2, removeLevel0_5=False)
Pipeline behaviour to note:
The original ObservationContext (in HIPE, not on your hard disk) is overwritten with the results of running the pipeline task. As a consequence, levels higher than those you reprocessed to will be removed from the original Observation Context. In contrast, HTPs for all spectrometers are retained in the observation context regardless of what what was selected but only the data for spectrometer(s) you selected will be modified.
When using the GUI, a new Observation Context variable is always created, even if you choose the output to be identically named to the input as HIPE will automatically append _1 to the repeated variable name. However, in the command line you can ensure that no new variable is created in one of two ways:
# Assuming an observationContext called 'obs' # # 1. Choose the output name to be the same as the input obs = hifiPipeline(obs=obs) # # 2. Do not specify an output variable name at all hifiPipeline(obs=myObs) #
Both approaches have the same result, just pick the one that makes most sense to you.
If you want to compare the effects of running the pipeline you will either need to reload the original Observation context or make a copy of products (spectra or HTP) before running the pipeline:
# Extract an HTP (here WBS-H-USB) and make a copy for comparison htp = htp = obs.refs["level2"].product.refs["WBS-H-USB"].product htp_orig = htp.copy()
If you try to re-process from a higher level data than exists in the Observation Context, then the
hifiPipelineTask will automatically select the highest existing level. For example, if you try to
re-process from Level 0.5 to 1 but the ObservationContext only contains a Level 0 product then the pipeline will automatically
run from Level 0 to Level 1.
By default, the pipeline runs without saving the new observation context on disk. This is done to reduce the amount of time it takes to run the pipeline but is at the cost of forcing the pipeline to use more memory in HIPE in order to avoid writing to disk.
If you prefer to save the output automatically then you can switch the
save option on by checking the
box in the expert mode, or in the command line:
obs = hifiPipeline(obs=obs, save=True)
Note that unless you specify a pool to write to, using the
palStore option (see below), then the
observation will be written to .hcss/lstore/pipeline-out.
When running the pipeline from the command line, the exact ordering of the parameters does not matter.
If the pipeline is not behaving as you expect (keeping old values, for example) try resetting it:
hifiPipeline = hifiPipelineTask()
Please note that the addition of Level 2.5 to the pipeline has made it more memory intensive. We have found that a 2 Gb machine is capable only of processing small observations (and data reduction may have to be carried out in a fresh HIPE session). Even a 4 Gb machines may have trouble processing larger maps and Spectral Scans. A more realistic memory allocation would be 6 Gb. We remind you of the option to request processing on demand from the HSA. See also Chapter 27 for hints on dealing with memory problems.
Saving the output:
There are several methods you can use to save your reprocessed observation in a pool (also known as a local store). See the Data Analysis Guide, chapter 1 for more information about pools.)
Right click on the output ObservationContext
and select "Send to Local store".
Alternatively, you can use
saveProduct in the command line:
# #To save
MyNewObsto pool "reprocessed": bg("saveProduct(product=myObs, pool='reprocessed', tag='My reprocessed data')")
Tagging the observation can help to quickly search for the correct version of the observation for a future HIPE session.
In addition, when you run the pipeline, you can specify which pool the output should be written to. In the console type,
pool to the palStore bullet in the GUI.
Or completely in the command line:
# Specify the pool to which the pipeline should write output:
name)) MyNewobs = hifiPipeline(obs=Myobs, palStore=
pool, save=True) #
Using this option will results in the pipeline take longer to run.
Running the pipeline for multiple observations:
The script below will retrieve a list of Observation Contexts from pools, reprocess them, then save the reprocessed data in another pool.
list = [1342xxxxxx, 1342yyyyyy, 1342zzzzzz] for obsid in list: obs=getObservation(obsid) obs = hifiPipeline(obs=obs, save=False) pool = '%i_new' %(obsid) saveObservation(obs, poolName=pool)
The script assumes that the original Observation Contexts are stored in pools with the name of the observation number, i.e., observation 1342190798 is stored in a pool with name 1342190798, and in this case, the output pool in 1342190798_new.