1.81. hifiPipeline

Full Name:	herschel.hifi.pipeline.HifiPipelineTask
Alias:	hifiPipeline
Type:	Java Task -
Import:	from herschel.hifi.pipeline import HifiPipelineTask
Category:	HIFI/Pipeline

Description

Task to run the HIFI pipeline.

It wraps together the three stages of the pipeline: the spectrometer branch (WBS and HRS pipelines), the level 1 pipeline and the level 2 pipeline

You will find the gui for this task (hifiPipeline) under applicable tasks of a ObservationContext. Double-click on it to open it. In the basic mode it allows you to re-run the pipeline for each of the backends individually and from any (present) level to some higher level, note that processing from scratch (level -1) is only possible for ICC members with access to the ICC databases. You can use check boxes to select which spectrometers to re-run the pipeline on. It is also possible to supply your own algorithm for each part of the HIFI pipeline, note that this requires detailed knowledge of the pipeline and it is recommended to read the HIFI Pipeline Specification Document before attempting this. You can more easily customize the pipeline using the doPipelineConfiguration tab, with this you can omit pipeline steps and change default parameters.

Another useful feature is the option to re-pipeline data with the latest calibration, by checking the cal box. This option requires that you configure the HIFI pipeline configureHifiPipeline and pass the location of the calibration to the pipeline using the pal option. See the HIFI Data Reduction Guide for more details.

The expert mode is intended for ICC use only.

Examples

Example 1: PipelineTask

# from  herschel.hifi.pipeline import HifiPipelineTask
#
# Example 1: Having an existing observation context called obs you can reprocess it.
# Note that all level 0, calibration and other products (auxiliary etc) are not replaced.
#
obsid = 1342191578L
obs = getObservation( obsid, useHsa=1)
hifiPipeline(obs=obs, fromLevel=0)
#
# Note: you can use the Observation Viewer to inspect the resulting observation context
#
#
# Example 2: Reprocess only WBS products from level 1 to level 2
#
hifiPipeline(obs=obs, fromLevel=1, upToLevel=2, apids=["WBS-H", "WBS-V"])
#
#
# Example 3: Reprocess to level 1 and re-install the calibration tree. See the next example
# to learn how to use a different calibration tree.
# Note the level 2 is removed.
#
hifiPipeline(obs=obs, fromLevel=0, upToLevel=1, cal=1)
#
#
# Example 4: like 3 now using a new calibration from the HSA.
#
cal = configureHifiPipeline(useHsa=True)
obs = hifiPipeline(obs=obs,fromLevel=0.0,upToLevel=2.0, cal=1, palStore=cal, save=False)
#
#
# Example 5: Reprocess only from level 1 to level 2, keeping the products in the level 0.5 context.
#
hifiPipeline( obs=obs, fromLevel=0, removeLevel0_5=0 )
#
# Example 6: Reprocess to level 2, repopulate cal and aux, redo the quality pipeline and save in palStore
#
hifiPipeline( obs=obs, fromLevel=0, cal=1, aux=1, save=1, quality=1 )
#
#

Example 2: PipelineTask

# Example 7, like ex. 1 now using your own algorithm for wbs, hrs, level1 and/or level2:
# one can find these algos by clicking in HIPE on pipeline -> HIFI and the select one 
# to look / edit / use.
#
hifiPipeline( obs, wbsAlgo=myWbsAlgo, hrsAlgo=myHrsAlgo, level1Algo=mylevel1Algo, level2Algo=mylevel2Algo)
#
# The following examples need telemetry database access. In general this is NOT available outside the ICCs.
#
# Example 8, generate an observation context import scratch
# the default db = retrieved import the property: var.database.devel, check with propgen.
#
obs = hifiPipeline( obsid=1342190191L, db='hifi_icc_ops_1@iccdb1.sron.rug.nl 0 READ' )
#
# Note: you can use the Observation Viewer to inspect the outcoming observation context
#
#
# Example 9, like ex. 8, but with different datasets per box
#
obs = hifiPipeline( obsid=1342190191L, db='hifi_icc_ops_1@iccdb1.sron.rug.nl 0 READ', datasetsPerBox=1 )
#

API Summary

Properties
`String[] apids [INPUT, OPTIONAL, default=all apids]`
`ObservationContext obs [INOUT, OPTIONAL, default=No default value]`
`Long obsid [INPUT, OPTIONAL, default=No default value]`
`String db [INPUT, OPTIONAL, default=Configuration.getProperty("var.database.devel")]`
`Long datasetsPerBox [INPUT, OPTIONAL, default=100]`
`boolean cal [INPUT, OPTIONAL, default=false]`
`String calVersion [INPUT, OPTIONAL, default=Latest calibration release]`
`boolean aux [INPUT, OPTIONAL, default=false]`
`PipelineConfiguration params [INPUT, OPTIONAL, default=No default value]`
`ProductStorage palStore [INPUT, OPTIONAL, default=No default value]`
`boolean save [INPUT, OPTIONAL, default=false]`
`PyFunction wbsAlgo [INPUT, OPTIONAL, default=wbsPipelineAlgo]`
`PyFunction hrsAlgo [INPUT, OPTIONAL, default=hrsPipelineAlgo]`
`PyFunction level1Algo [INPUT, OPTIONAL, default=level1PipelineAlgo]`
`PyFunction level2Algo [INPUT, OPTIONAL, default=level2PipelineAlgo]`
`Boolean removeLevel0_5 [INPUT, OPTIONAL, default=true]`
`Float fromLevel [INPUT, OPTIONAL, default=0]`
`Float upToLevel [INPUT, OPTIONAL, default=2]`
`Boolean upToLevel0 [INPUT, OPTIONAL, default=false]`
`Boolean upToLevel0_5 [INPUT, OPTIONAL, default=false]`
`Boolean upToLevel1 [INPUT, OPTIONAL, default=false]`
`ProcessEnvironment env [INPUT, OPTIONAL, default=automatic]`
`boolean quality [INPUT, OPTIONAL, default=true]`
`boolean browseProduct [INPUT, OPTIONAL, default=true]`
`String obsMode [INPUT, OPTIONAL, default=automatic]`
`String tmVersion [INPUT, OPTIONAL, default=automatic]`
`String execMode [INPUT, OPTIONAL, default=interactive]`
`Boolean reprocessAllLevels [INPUT, OPTIONAL, default=false]`
`Boolean gui [INPUT, OPTIONAL, default=false]`

API details

Properties

`String[] apids [INPUT, OPTIONAL, default=all apids]`
	The spectrometers (apids) that have to be processed

`ObservationContext obs [INOUT, OPTIONAL, default=No default value]`
	A pre-existing observation context as input and also the result of run this task.

`Long obsid [INPUT, OPTIONAL, default=No default value]`
	ICC use only: Alternative input for the observation to be processed. ObservationContext created on the fly

`String db [INPUT, OPTIONAL, default=Configuration.getProperty("var.database.devel")]`
	ICC use only: The database containing the specified obsid.

`Long datasetsPerBox [INPUT, OPTIONAL, default=100]`
	Defines the number of datasets per box. datasetsPerBox should be > 0. The value 0 will return the old structure (before v3.0). This use is deprecated.

`boolean cal [INPUT, OPTIONAL, default=false]`
	If true repopulate the calibration context from the palStore.

`String calVersion [INPUT, OPTIONAL, default=Latest calibration release]`
	The version of calibration data to use. The default of the system is to always use the latest calibration version contained in the supplied calibration pool [palStore = xxx]. This parameter can only be used in conjunction with the task parameter cal=True.

`boolean aux [INPUT, OPTIONAL, default=false]`
	If true repopulate the auxillary context from the palStore.

`PipelineConfiguration params [INPUT, OPTIONAL, default=No default value]`
	A map of level 1 and 2 pipeline tasks and their parameters

`ProductStorage palStore [INPUT, OPTIONAL, default=No default value]`
	The store where the result will be stored and where the calibration product can be retrieved.

`boolean save [INPUT, OPTIONAL, default=false]`
	If true save the products in the palStore.

`PyFunction wbsAlgo [INPUT, OPTIONAL, default=wbsPipelineAlgo]`
	The algorithm for the wbs pipeline.

`PyFunction hrsAlgo [INPUT, OPTIONAL, default=hrsPipelineAlgo]`
	The algorithm for the hrs pipeline.

`PyFunction level1Algo [INPUT, OPTIONAL, default=level1PipelineAlgo]`
	The algorithm for the level1 pipeline.

`PyFunction level2Algo [INPUT, OPTIONAL, default=level2PipelineAlgo]`
	The algorithm for the level2 pipeline.

`Boolean removeLevel0_5 [INPUT, OPTIONAL, default=true]`
	Level 0.5 is removed when valid Level 1 data can be formed.

`Float fromLevel [INPUT, OPTIONAL, default=0]`
	The starting point level for the pipeline. If no ObservationContext is passed in the input parameter "obs" the level will be always = -1 Values used: LEVEL_RAW=-1 LEVEL0= 0 LEVEL0_5=0.5 LEVEL1= 1

`Float upToLevel [INPUT, OPTIONAL, default=2]`
	The final level of the pipeline processed. Values used: LEVEL0=0 LEVEL0_5=0.5 LEVEL1=1 LEVEL2=2

`Boolean upToLevel0 [INPUT, OPTIONAL, default=false]`
	The final level of the pipeline processed. Same as upToLevel=0.

`Boolean upToLevel0_5 [INPUT, OPTIONAL, default=false]`
	The final level of the pipeline processed. Same as upToLevel=0.5.

`Boolean upToLevel1 [INPUT, OPTIONAL, default=false]`
	The final level of the pipeline processed. Same as upToLevel=1.

`ProcessEnvironment env [INPUT, OPTIONAL, default=automatic]`
	The process environment for the specified obsid.

`boolean quality [INPUT, OPTIONAL, default=true]`
	Run the quality pipeline too.

`boolean browseProduct [INPUT, OPTIONAL, default=true]`
	Run the browse product pipeline.

`String obsMode [INPUT, OPTIONAL, default=automatic]`
	Expert option. The observation mode to be used in the pipeline. Overwrite the obsMode present in the ObservationContext. This option is only relevant for ILT data where the obsMode was not always present.

`String tmVersion [INPUT, OPTIONAL, default=automatic]`
	Expert option. The telemetry version to be used in the pipeline. This option is only relevant for ILT data where the tmVersion was not always correct.

`String execMode [INPUT, OPTIONAL, default=interactive]`
	Expert option. The execution mode. Expert option, irrelevant to users.

`Boolean reprocessAllLevels [INPUT, OPTIONAL, default=false]`
	Deprecated parameter. In case you pass an (adopted) observation context and you want to reprocess level 0 and following switch this one to TRUE

`Boolean gui [INPUT, OPTIONAL, default=false]`
	Deprecated parameter. If the progress bar is displayed or not.

History

2008-02-12 - PZ: Add JTAGs for this task
2008-03-07 - PZ: Add update example for this task
2008-03-28 - PZ: make pipelineTask with API for CS
2010-03-17 - DK: update of documentation.