Data that you retrieve from the HSA is processed with calibration data appropriate for the version of the pipeline used to populate the HSA during Standard Product Generation (SPG). You can identify the pipeline version by looking at the 'creator' field in the ObservationContext metadata; the SPG version corresponds to the version of HIPE used. Note that the ObservationContext metadata is not updated when you reprocess yourself but you can look at the HistoryTasks of the History Product to check the last version of the pipeline applied to the data.
Updates the HIFI calibration data are generally concurrent with the release of each major version of the HCSS-HIFI software. It is possible to have updates to the calibration data in between major releases of the software as the software and data are independent of each other. Information about calibration versions and their contents can be found at this page.
Prior to HIPE 8.0, you can find the calibration
version number by looking at the "appliedTagToLoad" meta data item in the Calibration Context and the version number
has the format IA_CAL_USER_, where
version_numberversion_number is the version number with a format like 1_0.
From HIPE 8, the metadata item calVersion can be found in all calibrated products (level 1 and 2) and the version
number has format HIFI_CAL_version_number
While it is normally possible to use processed data created with a previous version of the software in a more recent major release, one will find more consistent behaviour in HIPE if data version, calibration version and software version are compatible with each other. This means that reprocessing of data should be considered normal when there are data and software version differences; some examples follow, in all cases the data is retrieved from a local pool.
An internet connection is required to access the HSA for the calibration files.
Reprocess all spectrometers up to level 2 using the calibration present in ObservationContext
obs = getObservation("1342205520")
obs_1 = hifiPipeline(obs=obs)
Here we are not accessing the calibration from the HSA so a network connection is not required.
Reprocess all spectrometers ip to level 2 using newest calibration available in the HSA
obs = getObservation("1342205520")
cal = configureHifiPipeline(useHsa=True)
obs_1 = hifiPipeline(obs=obs, cal=True, aux=True, palStore=cal)
You must first configure the pipeline to go to the HSA for the calibration data. If you do not configure the pipeline but set cal=True then the pipeline will send a message to the console warning you that new calibration could not be updated and reprocessing commences using the calibration available in the Observation Context as above.
Note that you can choose to connect to the HSA for the latest calibration each time you reprocess, as here, or you can download the calibration files and store them on disk, see below for information about this.
Configuring the pipeline as above also configures the pipeline to save output after each level is completed to a default location , which is .hcss/pipeline-out, when the save=True option is used:
obs = getObservation("1342205520")
cal = configureHifiPipeline(useHsa=True)
obs_1 = hifiPipeline(obs=obs,fromLevel=0.0,upToLevel=2.0, palStore=cal, save=True)
If you forget to configure the pipeline before running with save=True and also do not provide a pool yourself for the pipeline to write to, then the pipeline runs to completion but without saving anything. You can then save the output yourself
You can also specify the calibration version to use with the calVersion option. If you do not set this the latest version is taken by default.
obs = getObservation("1342205520")
cal = configureHifiPipeline(useHsa=True)
obs_1 = hifiPipeline(obs=obs, cal=True, aux=True, palStore=cal, calVersion="HIFI_CAL_6_0")
You do not need to download different versions of the calibration tree in order to apply different calibration versions to your data, the calibration tree contains all the past calibration versions already.
This functionality is only available from the command line at present.
Repipelining with new calibration using the GUI.
In order to repipeline data with new calibration via the GUI you must still first configure the pipeline with:
cal = configureHifiPipeline(useHsa=True)
You should then drag cal from the Variables view into the palStore bullet on the GUI, and then
check the hifical box. You can then set up and run the pipeline as desired.
Downloading the calibration pool from the HSA.
It is also possible to download the latest calibration available at the HSA into a local pool, which can be stored on your disk. This avoids the need for an internet connection every time you run the pipeline and makes reprocessing of multiple observations much more efficient.
The following command will get the latest calibration from the HSA and store it in a pool called hifi-cal (location .hcss/lstore/hifi-cal), which is the default location for the pipeline to obtain calibration data when not using the HSA. You only need to do this again when you want to update the calibration from the HSA.
hifical = getHifiCal(useHsa=True)
To pass this pool for use in the pipeline:
store=ProductStorage(["pipeline-out","hifi-cal"]) obs_new = hifiPipeline(obs=obs, palStore=store, cal=True)
The "pipeline-out" is required as a precaution because of a subtlety in how HIPE deals with pools; the first pool registered is writeable. By default, the pipeline will write its output to pool and you do not want to risk polluting your calibration pool with a processed observation. Therefore we specify a pool for the pipeline to write out to. In fact it is the same output pool that is used in the pipeline configuration above.
To do this in the GUI you can create the variable store as above and drag that from the Variables view to
the palStore bullet in the GUI. You should then check the hifical box beofre running the pipeline.
Getting calibration products.
The calibration contained within an ObservationContext can be obtained with the following:
cal = obs.getCalibration()
These calibration products will only apply to this observation and only represent a part of the HIFI calibration tree. The contents of the calibration tree are described in Section 3.1
To get the full HIFI calibration tree from the hifi-cal pool:
from herschel.ia.obs.cal import CalTreeFactoryManager
from herschel.hifi.cal import HifiCalTreeFactoryPalImpl
factory = CalTreeFactoryManager.getInstance()
store = ProductStorage("hifi-cal")
registry = factory.getRegistry()
# user release version of HIFI calibration tree
calroot = registry.checkout()
# all versions of the calibration tree contained in the storage
state = registry.state