1.90. hifiPipeline

Full Name: herschel.hifi.pipeline.HifiPipelineTask
Alias: hifiPipeline
Type: Java Task - Java Task
Import: from herschel.hifi.pipeline import HifiPipelineTask



Task to run the HIFI pipeline.

It wraps together the three stages of the pipeline: the spectrometer branch (WBS and HRS pipelines), the level 1 pipeline and the level 2 pipeline

You will find the gui for this task (hifiPipeline) under applicable tasks of a ObservationContext. Double-click on it to open it. In the basic mode it allows you to re-run the pipeline for each of the backends individually and from any (present) level to some higher level, note that processing from scratch (level -1) is only possible for ICC members with access to the ICC databases. You can use check boxes to select which spectrometers to re-run the pipeline on. It is also possible to supply your own algorithm for each part of the HIFI pipeline, note that this requires detailed knowledge of the pipeline and it is recommended to read the HIFI Pipeline Specification Document before attempting this. You can more easily customize the pipeline using the doPipelineConfiguration tab, with this you can omit pipeline steps and change default parameters.

Another useful feature is the option to re-pipeline data with the latest calibration, by checking the cal box. This option requires that you configure the HIFI pipeline and pass the location of the calibration to the pipeline using the pal option. There is a task called configureHifiPipeline to help with this setup, see the reference below for a link to this task. Also see the HIFI Data Reduction Guide for more details.


Example 1: PipelineTask
# from  herschel.hifi.pipeline import HifiPipelineTask
# Example 1: Having an existing observation context called obs you can reprocess it.
# Note that all level 0, calibration and other products (auxiliary etc) are not replaced.
obsid = 1342191578L
obs = getObservation( obsid, useHsa=1)
hifiPipeline(obs=obs, fromLevel=0)
# Note: you can use the Observation Viewer to inspect the resulting observation context
# Example 2: Reprocess only WBS products from level 1 to level 2
hifiPipeline(obs=obs, fromLevel=1, upToLevel=2, apids=["WBS-H", "WBS-V"])
# Example 3: Reprocess to level 1 and re-install the calibration tree. See the next example
# to learn how to use a different calibration tree.
# Note the level 2 is removed.
hifiPipeline(obs=obs, fromLevel=0, upToLevel=1, cal=1)
# Example 4: like 3 now using a new calibration from the HSA.
cal = configureHifiPipeline(useHsa=True)
obs = hifiPipeline(obs=obs,fromLevel=0.0,upToLevel=2.0, cal=1, palStore=cal, save=False)
# Example 5: Reprocess only from level 1 to level 2, keeping the products in the level 0.5 context.
hifiPipeline( obs=obs, fromLevel=0, removeLevel0_5=0 )
# Example 6: Reprocess to level 2, repopulate cal and aux, redo the quality pipeline and save in palStore
hifiPipeline( obs=obs, fromLevel=0, cal=1, aux=1, save=1, quality=1 )
Example 2: PipelineTask
# Example 7, like ex. 1 now using your own algorithm for wbs, hrs, level1 and/or level2:
# one can find these algos by clicking in HIPE on pipeline -> HIFI and the select one
# to look / edit / use.
hifiPipeline( obs, wbsAlgo=myWbsAlgo, hrsAlgo=myHrsAlgo, level1Algo=mylevel1Algo, level2Algo=mylevel2Algo)
# The following examples need telemetry access. In general this is NOT available outside the ICCs.
# Example 9, like ex. 8, but with different datasets per box
obs = hifiPipeline( obsid=1342190191L, datasetsPerBox=1 )

API details


String[] apids [INPUT, OPTIONAL, default=all apids]

The spectrometers (apids) that have to be processed

ObservationContext obs [INOUT, OPTIONAL, default=No default value]

A pre-existing observation context as input and also the result of run this task.

Long obsid [INPUT, OPTIONAL, default=No default value]

ICC use only: Alternative input for the observation to be processed. ObservationContext created on the fly

Long datasetsPerBox [INPUT, OPTIONAL, default=100]

Defines the number of datasets per box. datasetsPerBox should be > 0. The value 0 will return the old structure (before v3.0). This use is deprecated.

boolean cal [INPUT, OPTIONAL, default=false]

If true repopulate the calibration context from the palStore.

String calVersion [INPUT, OPTIONAL, default=Latest calibration release]

The version of calibration data to use. The default of the system is to always use the latest calibration version contained in the supplied calibration pool [palStore = xxx]. This parameter can only be used in conjunction with the task parameter cal=True.

boolean aux [INPUT, OPTIONAL, default=false]

If true repopulate the auxillary context from the palStore.

PipelineConfiguration params [INPUT, OPTIONAL, default=No default value]

A map of level 1 and 2 pipeline tasks and their parameters

ProductStorage palStore [INPUT, OPTIONAL, default=No default value]

The store where the result will be stored and where the calibration product can be retrieved.

boolean save [INPUT, OPTIONAL, default=false]

If true save the products in the palStore.

PyFunction wbsAlgo [INPUT, OPTIONAL, default=wbsPipelineAlgo]

The algorithm for the wbs pipeline.

PyFunction hrsAlgo [INPUT, OPTIONAL, default=hrsPipelineAlgo]

The algorithm for the hrs pipeline.

PyFunction level1Algo [INPUT, OPTIONAL, default=level1PipelineAlgo]

The algorithm for the level1 pipeline.

PyFunction level2Algo [INPUT, OPTIONAL, default=level2PipelineAlgo]

The algorithm for the level2 pipeline.

PyFunction level2_5Algo [INPUT, OPTIONAL, default=level2_5PipelineAlgo]

The algorithm for the level2_5 pipeline.

Boolean removeLevel0_5 [INPUT, OPTIONAL, default=true]

Level 0.5 is removed when valid Level 1 data can be formed.

Float fromLevel [INPUT, OPTIONAL, default=0]

The starting point level for the pipeline. If no ObservationContext is passed in the input parameter "obs" the level will be always = -1 Values used: LEVEL_RAW=-1 LEVEL0=0 LEVEL0_5=0.5 LEVEL1=1 LEVEL=2

Float upToLevel [INPUT, OPTIONAL, default=2.5]

The final level of the pipeline processed. Values used: LEVEL0=0 LEVEL0_5=0.5 LEVEL1=1 LEVEL2=2 LEVEL=2.5

ProcessEnvironment env [INPUT, OPTIONAL, default=automatic]

The process environment for the specified obsid.

boolean quality [INPUT, OPTIONAL, default=true]

Run the quality pipeline too.

boolean browseProduct [INPUT, OPTIONAL, default=true]

Run the browse product pipeline.

String obsMode [INPUT, OPTIONAL, default=automatic]

Expert option. The observation mode to be used in the pipeline. Overwrite the obsMode present in the ObservationContext. This option is only relevant for ILT data where the obsMode was not always present.

String tmVersion [INPUT, OPTIONAL, default=automatic]

Expert option. The telemetry version to be used in the pipeline. This option is only relevant for ILT data where the tmVersion was not always correct.

Boolean gui [INPUT, OPTIONAL, default=false]

Deprecated parameter. If the progress bar is displayed or not.

See also


  • 2008-02-12 - PZ: Add JTAGs for this task
  • 2008-03-07 - PZ: Add update example for this task
  • 2008-03-28 - PZ: make pipelineTask with API for CS
  • 2010-03-17 - DK: update of documentation.
  • 2012-01-12 - KE: Add a level 2.5 step to the task
  • 2015-05-20 - KE: References to databases removed.