3.4. The pipeline menu

3.4.1. Where are the scripts?

In the following chapters we describe how to run each of the spectroscopy pipelines. A summary of the various pipeline scripts can be found in the Spectroscopy Launch Pad (Section 2.2.2). The pipelines take you from Level 0 (raw) to Level 2 (fully-processed) or Level 2.5 (for background subtracted for unchopped rangeScan AOTs). The pipelines can be found in the HIPE Pipeline menu, as these figures show:

Pipeline menu 1: chopNod line scan

Figure 3.1. Pipeline menu 1: chopNod line scan


Pipeline menu 2: chopNod rangeScan

Figure 3.2. Pipeline menu 2: chopNod rangeScan


Pipeline menu 3: unchopped line scan

Figure 3.3. Pipeline menu 3: unchopped line scan


Pipeline menu 4: unchopped rangeScan

Figure 3.4. Pipeline menu 4: unchopped rangeScan


Pipeline menu 5: wavelength switching

Figure 3.5. Pipeline menu 5: wavelength switching


Select the pipeline script you want and it will open in the Editor pane (noting that the SPG scripts are not intended for interactive use, they are provided only for completeness). From there you can edit it and run it, but if you want to save your edits you should save it to a different location on disk—otherwise you are changing the HIPE copy of that pipeline script.

The scripts start with an explanation and some set-up. They can be run all in one go or line-by-line, to be used variously depending on your level of experience. The scripts contain all the pipeline tasks, pipeline helper plotting tasks, and basic descriptions of what the tasks are doing; the individual pipeline tasks are more fully described in the PACS URM.

The "SPG Scripts" sub-menu is the PACS standard product generation [SPG] pipeline which is run, in automatic mode, by the HSC: those in HIPE 14 will produce "SPG 14" products. These pipelines scripts do not differ much from the interactive ones which are described in this PDRG.

We remind you here that you should consult PACS documentation web-page, hosted on the Herschel site (here), for information about the calibration of PACS data such as spectral leakages, sensitivity, uncertainties, ghosts, saturation limits, and PSFs.

3.4.2. What are the differences between the pipeline scripts?

As shown in the figures in the previous section,

  • In the "Chopped line scan and short rangeScan" menu are several interactive pipeline scripts: "Telescope normalisation", "Calibration sources and RSRF", "Telescope normalisation drizzled maps" and "Pointing offset correction (point sources)". This menu is for chopNod observations of line scan and short rangeScan (<5 microns) mode. The additional script "Split On-Off" is a helper script, which can be used to produce cubes of on-source and off-source data to check for off-source contamination.

  • In the "Chopped large rangeScan SED" menu are: "Telescope normalisation", "Calibration sources and RSRF", "Single obs", "Background Normalisation", "Pointing offset correction (point sources)". The additional helper scripts are: "Combine observations for a full SED", to run the chosen pipeline script on two or more observations and combine the final cubes and any point-source calibrated spectra into a single product, and "Split On-Off", which can be used to produce cubes of on-source and off-source data to check for off-source contamination. This menu is for chopNod rangeScan AOTs (from 5 microns up to the full SED).

  • In the "Unchopped line" menu: "Calibration sources and RSRF" and "...with transient correction". This menu is for line scan unchopped observations. The first script includes the original transients correction task written for this mode, which corrects the transient that can occur at the beginning of an observation. The second script is new to HIPE 13, and includes a more complete transients correction task.

    This new pipeline with the new transient correction tasks should not be used blindly: test the results carefully.

  • In the "Unchopped range" menus: "Calibration sources and RSRF"" , "...with transient correction", and "Combine on-off". This menu is for all rangeScan unchopped observations, no matter how long or short the range. The first and second scripts reduce any single observation, and the third is to subtract the off-source observation results (the background) from the on-source observation results. There is also a PACS script do this subtraction: Scripts→PACS useful scripts→Spectroscopy: off-subtraction and post-processing in unchopped range spectroscopy. The first script does not include any transients correction. The second script was new to HIPE 13, and applies a new and comprehensive transients correction.

    This new pipeline with the new transient correction tasks should not be used blindly: test the results carefully.

3.4.2.1. The SPG scripts vs the interactive scripts

The SPG scripts are provided in the pipeline menus for each AOT, however they are not intended for interactive use. They were provided in previous versions of HIPE for users to check what had been done on their ObservationContext by the SPG of that version of HIPE, as in previous versions of HIPE there were some tasks not run by the SPG. From HIPE 14 onwards, all tasks of the standard pipeline are those of the SPG pipeline.

3.4.2.2. When to use line and when to use range

For chopNod AOTs, the "line scan" menu scripts are to be used with Line spectroscopy AOTs and Range spectroscopy AOTs with a range of less than about 5 microns. The flatfielding from the line scan script performs better for these short ranges than the flatfield of the rangeScan script. This is what is also done in the SPG scripts.

For unchopped AOTs, you always have to use scripts from the "line scan" menu for Line spectroscopy AOTs and scripts from the "rangeScan" pipeline for Range spectroscopy AOTs, even for short range scans. Later we demonstrate how to use the flatfielding task of the line scan pipeline script for the shorter range scans. This is what is also done in the SPG scripts.

More on the differences in flatfielding for line and rangeScan AOTs can be found in Section 7.4.

3.4.2.3. The pipeline scripts "Telescope normalisation" and "Calibration sources and RSRF"

This "Telescope normalisation" pipeline script has been the default and the SPG script since HIPE 13. Before that it was the "Calibration sources and RSRF" script. These two methods differ in the way the telescope background is handled and the data are flux calibrated.

Both scripts are valid, they are just different. This earlier script uses the data from the calibration block, taken in the beginning of all observations at key wavelengths, together with the relative spectral response function (RSRF), to flux calibrate the science spectra. The RSRF was computed before the launch of Herschel; the RSRF and the spectra of the calibration blocks were monitored throughout the mission with observations of calibration sources.

However, this pipeline script was never the preferred script for sources with low continuum flux levels (less than about 10 Jy) or of long duration. To deal better with these observations, the "Telescope normalisation" script was developed, and this one is now preferred for all chopNod AOTs. Instead of using the RSRF and the calibration block to flux calibrate the spectra, it uses the off-source spectra to background subtract and to also relatively spectrally response-correct the on-source spectra, resulting in a spectrum corrected for the background but in units of "telescopes". These units are then turned into Jy using a calibration file that contains the spectrum of the telescope in Jy. Since the off-source datapoints and the on-source datapoints are taken right next to each other, any detector response drifts or jumps will be better 'corrected out' than with the calibration block method. It produces results that are cleaner (especially for broad-band features), and better for the long-duration and faint observations. For all observations, the band-matching is also better.

Note that the absolute flux levels of the two methods have been calibrated to return the same results. The spectra produced by the calibration block and the background normalisation scripts may show slight differences in the contiuum shape, but the spectral line (integrated) fluxes should not differ strongly. If they do, it could be that you have contamination in the background (off-source) pointings. In this case, you should not only check for the contamination but also prefer the calibration block result: this script cannot remove the contamination, but it will not also be incorrected in the flux calibration it applies.

To measure the fluxes of spectral lines longer than 190 μm, you must use the "Calibration sources and RSRF" script: see Section 5.4 (chopNod) or Section 6.4 (unchopped).

3.4.2.4. Telescope normalisation drizzle maps

New to HIPE 14 and higher: because of an oversight that lead to an incorrect calibration of drizzle cubes created by HIPE version 13 (see the HSC pages which are currently at herschel.esac.esa.int/twiki/bin/view/Public/DpKnownIssues and herschel.esac.esa.int/twiki/bin/view/Public/PacsCalibrationWeb) when using the Telescope normalisation method, it is now necessary to calibrate the data when creating drizzle cubes slightly differently to when calibrating any other type of cube. The drizzle cubes require a calibration applied to the Level 1 PacsCubes, for which one particular calibration file has to be used. All other cubes (rebinned, projected, and interpolated) require a calibration applied to the final PacsRebinnedCubes of Level 2, for which a slighty different calibration file is necessary. The two calibrations differ only very slightly: the noise level is very slightly higher for the drizzle cubes, but the overall flux levels, line fluxes, and continuum slopes, will be the same. However, this means that for mapping line scan chopNod observations for which drizzle cubes are required, the "Telescope normalisation drizzle maps" pipeline script needs to be used rather than the "Telescope normalisation" script.

If using the "Calibration sources and RSRF" pipeline script, no change from previous HIPE versions for drizzled cubes is necessary, and these can be created from within the pipeline script itself.

3.4.2.5. The pipeline scripts "Pointing offset correction (point sources)"

This script is for chopNod pointed observations of bright point sources (continuum ≻10 Jy) which are located close to the central spaxel (certainly well with the central 3 spaxels). It uses the Telescope normalisation method to do the flux calibration, but in addition applies a wavelength-dependent correction to the spectral fluxes to correct for the effect of pointing jitter; the spectra of the central spaxels (where the point source is expected to be located) are corrected. The correction is done by comparing the fluxes of the point source in the central spaxels of the cubes to the corresponding fluxes of the PACS beams, and for this reason it works best on brighter sources located closer to the centre of the FoV. The science end-product of this pipeline is an extracted, point source calibrated spectrum. Its purpose is to produce spectra that are cleaner, and hence if you are looking for broad features it is worth trying this pipeline.

We recommend you experiment with the pointing corrections of this pipeline, since they can have an important effect on the result. It is also recommended to compare the pipeline end-result spectrum with that from the "Telescope normalisation" script, e.g. using the Spectrum Explorer (see the DAG chap. 6). They should not differ too markedly.

3.4.2.6. The unchopped pipeline scripts with the old and the new transient correction

A "transient" is any temporary impact on the detector while an observation was being performed, and which affected the response detector for that of that time-line data-stream. An example would be a cosmic ray, or observing something faint immediately after something very bright. Since a transient changes the response of the detector, the flux calibration will be incorrect for the transient-affected datapoints. The usual appearance of a transient event, in the time-line spectrum, is a spike (up or down) followed by a decay back to the normal count levels. The decay can have a long or a short timescale. Transients often occurred at the beginning of an observation, immediately after the (bright) calibration block was "observed". For the chopNod AOTs, the signal we deal with is differential: subsequent data-points are subtracted from each other by the pipeline before the flux calibration is done. As the datapoints are taken very close in time to each other, transients generally have a negligible effect and are dealt with by the glitch detection pipeline tasks. But for the unchopped AOTs, this is not the case and the transients need to be dealt with in another manner.

Before HIPE 13, a pipeline task to correct for the transients that occurred in the beginning of the observation (after the calibration block) was done only for line scan AOTs. In HIPE 13 we provided two new pipeline scripts, which both apply a new, more comprehensive set of tasks for correcting transients occurring at any point in an observation, for line and rangeScan AOTs. These new tasks are presented in the menu as "...with transient correction". The transient corrections tasks should not be run blindly: the plots produced should be checked and the results should be compared to the SPG scripts: they should not differ too much.

While no transient correction is done in the SPG scripts for unchopped observations, a flatfielding is done. It turns out that the correction performed by the flatfielding task is similar to that performed by some of the new transient correction tasks. Hence, a level of transient correction is, effectively, done for the unchopped observations.

3.4.2.7. The pipeline script "Combine off-source with on-source" (unchopped range)

This pipeline script is for rangeScan unchopped AOTs. For this observing mode, the off-pointings are a separate observation to the on-source pointings. Hence, to fully reduce such data it is necessary to reduce first the on-source and then the off-source observations, and then subtract the off-source (background) from the on-source. The scripts starts with some set up, including setting the obsids of the on- and off-source observations, and runs the main pipeline script to reduce the data, and then does the subtraction.

There is also a PACS script Scripts→PACS useful scripts→Spectroscopy: off-subtraction and post-processing in unchopped range spectroscopy. The same tasks are run in the pipeline and the useful script, but the first does it in a hands-off loop while the second is run line-by-line.

3.4.2.8. "Split On-Off" testing script

This is a script offered for all chopNod AOTs. Its aim is to produce separate cubes for the on-source pointings and the off-source pointings, which can be compared to see if the off-source spectra have significant spectral features (lines) that will affect their use in subtracting the telescope background from the on-source spectra. After checking that all is well, you then need to run one of the actual pipeline scripts.

Since this script is just to check the on and off-source spectra to each other, you could run it almost blindly, i.e. just set the obsid and the camera you want to reduce, and run the rest all in one go.

3.4.2.9. The bulk-processing script "Combine observations for full SED"

For chopNod rangeScans (but aimed at full SEDs) we offer a "Combine observations for full SED" script that can be used for point sources, where at the end of the pipeline the point source spectrum is extracted, and multiple spectra arising from separate observations and from the red and blue camera are concatenated into a single spectral product.

In this script you will run a pipeline script for each observation and camera—whichever pipeline script you want to run: you should copy to a directory and edit is as necessary (setting the parameters of tasks, making the appropriate saving and plotting, etc.), since the script will then be run in one go. At the end of each running of the single observation script, it will save the various cubes and then run the point source extraction task extractCentralSpectrum on them, and save the final extracted spectra concatenated into a single Spectrum1d.