Chapter 5. Running the HIFI pipeline

Table of Contents

5.1. Introduction to the Pipeline
5.2. How to run the HIFI Pipeline
5.3. How to process with new (or different) calibration data
5.4. Modifying the pipeline
5.4.1. Using the interactive Level 2.5 pipeline
5.4.2. Customising the Level 1 and 2 pipelines
5.4.3. Editing the pipeline algorithms
5.4.4. Running the Pipeline step by step

Last updated: 4 December, 2015

5.1. Introduction to the Pipeline

HIFI data is automatically processed through the HIFI pipeline before it can be accessed from the the Herschel Science Archive (HSA). The HIFI pipeline is used for processing data received from one or more of the four HIFI spectrometers into calibrated spectra or spectral cubes, and comprises five stages of processing:

  1. Manipulate data taken from the satellite into time ordered Data Frames (a HifiTimelineProduct, or HTP) for each spectrometer. This is done in the Level 0 pipeline and results in the Level 0 data Product, which is the least processed data available to the scientific community.

  2. Remove backend instrumental effects - essentially a frequency calibration. There are separate pipelines for the WBS and HRS spectrometers, and the result is a Level 0.5 Product. You will not see this Product in the Observation Context unless the generation of a Level 1 product fails. However, you can generate it for yourself when reprocessing.

  3. Application of observing mode specific calibrations, i.e., subtraction of reference and OFF positions, and intensity calibration using Hot/Cold loads. This is done by the Level 1 pipeline, and resulting Level 1 Products are sets of frequency (IF) and intensity (TA') calibrated spectra.

  4. The Level 2 pipeline removes further instrumental effects by converting to antenna temperature (TA*), applying sideband gain corrections, and converting to sky frequency. Spectra at Level 2 are averaged according to LO setting and position.

  5. The Level 2.5 pipeline combines the Level 2 products into final products for each observing mode:

    • Point mode spectra are stitched, folded (in the case of Frequency Switch), and converted to SimpleSpectrum format. HRS data are stitched only in the case that subbands overlap in frequency.

    • Mapping mode Level 2.5 HTP contain spectra that are stitched and folded (in the case of Frequency Switch), a cubesContext is also produced at Level 2.5 containing cubes constructed from the stitched and folded (if applicable) spectra. Stitching of HRS data is done as for point mode observations. In the case of mapping observations carried out with a non-zero rotation angle, a cubesContextRotated is also produced, which contains cubes generated from the Level 2.5 HTP and with the rotation angle applied.

    • The deconvolved single sideband spectra is produced at Level 2.5 for Spectral Scan observations.

There are several reasons why you may wish to run data through the HIFI pipeline yourself. For this reason, the HIFI pipeline is available in HIPE.

In theory, Level 2 and Level 2.5 products can immediately be used for scientific analysis. However, they are produced without correction for baseline issues (see Chapter 13) or standing waves (see Chapter 12). Furthermore, you may wish to change the temperature scale or reference frame of the data (see Section 18.1). You can introduce these steps into the pipeline using the Interactive Level 2.5 pipeline, see Section 5.4.1.

In some cases, data may need to be looked at more carefully before scientific analysis can be done. You may wish to take advantage of the Customiseable Pipeline, see Section 5.4.2, to re-run all, or part of the pipeline in order to omit tasks or to change task defaults. Alternatively, you may wish to provide your own pipeline algorithm, or examine the results of each step of processing in order to improve data quality. Finally, you may wish to reprocess data in order to take advantage of new developments to the pipeline or calibration. To these ends, the Observation Context that is obtained from the HSA contains, along with the Level 0-2.5 data Products, everything you need to reprocess your observations - calibration products, satellite data - as well as quality, logging, and history products, which you can use to identify any problems with your data or its processing.

The following sections explain how to re-run the pipeline using the hifiPipeline task. For a full review of all tasks used by the pipeline, please see Chapter 4.