Data that you retrieve from the HSA is processed with calibration data appropriate for the version of the pipeline used to populate the HSA during Standard Product Generation (SPG). You can identify the pipeline version by looking at the SPG Version in the Observation Context Summary; the SPG version corresponds to the version of HIPE used. Note that when you reprocess yourself, the SPG Version gets updated. You can also look at the build version of the last pipeline task reported in the HistoryTasks of the History Product to check the last version of the pipeline applied to the data.
Where can I find information about calibration updates?
Updates to the HIFI calibration data are generally concurrent with the release of each major version of the HCSS-HIFI software. It is possible to have updates to the calibration data in between major releases of the software as the software and data are independent of each other. Information about calibration versions and their contents can be found on the HIFI Instrument and Calibration page , which is updated whenever new calibration is available.
How do I find the calibration version used to process my data?
The metadata item
calVersion can be found in all calibrated products (Level 1 and 2),
and the version number has the format HIFI_CAL_
Which calibration version should I use?
There is no need to use different calibrations for different science goals with HIFI data. Instead, it is recommended to keep data version, calibration version, and software version compatible with each other. For example, this means using HIPE 13.0 to work with data that has been processed in HIPE 13.0, with calibration that was released with HIPE 13.0. This can be done by downloading HIPE and data from the HSA following bulk reprocessing, or by reprocessing the data yourself with the latest calibration version available after each HIPE release.
However, you may wish to use an older calibration version, for example, in order to more directly compare recently acquired data with older data, and it is possible to pass older versions of the calibration to the pipeline. Consult the HIFI Instrument and Calibration page to understand the differences between the different calibration versions. However, whilst every effort is made to ensure backwards compatibility, using old calibration with newer software can cause inconsistent results.
How do I re-process with a new (or different) calibration version?
The default action of the pipeline is to reprocess the observation using the latest calibration that is already stored in the observation context. However, you can configure the pipeline to allow you to pass an older calibration version to it, or to access the latest calibration version held in the HSA, which requires an internet connection. This can be done with the following command (there is no way to do this in the GUI):
In the GUI, you should then:
from the Variables pane into the
palStore bullet in the GUI,
check the hifical box,
select the calibration version you want from the drop-down menu, which automatically selects the latest calibration version available from the HSA from the list of all available calibration versions
You can then set up and run the pipeline as desired.
A completely command line example is below:
obs = getObservation("1342205520") cal = configureHifiPipeline(useHsa=True) obs_1 = hifiPipeline(obs=obs, cal=True, palStore=cal, calVersion="HIFI_CAL_18_0")
If you do not configure the pipeline but set cal=True then the pipeline will send a message to the console warning you that
new calibration could not be updated, and reprocessing commences using the calibration available in the Observation Context.
If you omit the
calVersion, then the latest calibration version at the HSA will be used.
Configuring the pipeline as above also configures the pipeline to save output after each level is completed to a default location,
which is .hcss/pipeline-out, when the
save=True option is used (or the save box is
checked in the expert pipeline panel):
obs = getObservation("1342205520") cal = configureHifiPipeline(useHsa=True) obs_1 = hifiPipeline(obs=obs,fromLevel=0.0,upToLevel=2.0, palStore=cal, save=True)
If you forget to configure the pipeline before running with save=True, and also do not provide a pool yourself for the pipeline to write to, then the pipeline runs to completion but without saving anything. You can then save the output yourself.
Downloading the calibration pool from the HSA
Accessing the HSA every time you wish to reprocess data with a new calibration version may not be very efficient and requires that you have an internet connection. It is also possible to download the latest calibration available at the HSA into a local pool, which can be stored on your disk. This avoids the need for an internet connection every time you run the pipeline and makes reprocessing of multiple observations much more efficient.
The following command will get the latest calibration from the HSA and store it in a pool called hifi-cal (location .hcss/lstore/hifi-cal), which is the default location for the pipeline to obtain calibration data when not using the HSA. You only need to do this again when you want to update the calibration from the HSA. The length of time it takes to download the calibration pool depends on your internet connection; be aware that it is not a quick process.
hifical = getHifiCal(useHsa=True)
To pass this pool for use in the pipeline:
store=ProductStorage(["pipeline-out","hifi-cal"]) obs_new = hifiPipeline(obs=obs, palStore=store, cal=True)
The "pipeline-out" is required as a precaution because of a subtlety in how HIPE deals with pools; the first pool registered is writeable. By default, the pipeline will write its output to pool and you do not want to risk polluting your calibration pool with a processed observation. Therefore, we specify a pool for the pipeline to write out to. In fact, it is the same output pool that is used in the pipeline configuration above.
Note that you do not need to download different versions of the calibration tree in order to apply different calibration versions to your data, the calibration tree contains all the past calibration versions already.
Getting calibration products
The calibration contained within an ObservationContext can be obtained with the following:
cal = obs.getCalibration()
These calibration products will only apply to this observation and only represent a part of the HIFI calibration tree. The contents of the calibration tree are described in Section 3.1.
To get the full HIFI calibration tree from the hifi-cal pool:
from herschel.ia.obs.cal import CalTreeFactoryManager from herschel.hifi.cal import HifiCalTreeFactoryPalImpl factory = CalTreeFactoryManager.getInstance() store = ProductStorage("hifi-cal") factory.setProductStorage(store) registry = factory.getRegistry() # user release version of HIFI calibration tree calroot = registry.checkout() # all versions of the calibration tree contained in the storage state = registry.state
Re-pipelining with a new Auxiliary product
An additional option,
aux=True, allows to reprocess data with updated auxiliary information, e.g. data
from the satellite regarding timing and position of the observation. This option is intended for the use of HIFI calibration
scientists and is described here for the sake of completeness. This option requires connection to a dedicated data-base that
is distinct from the HSA, which hosts the same version of the auxiliary products used in the latest bulk reprocessing.
obs = getObservation("1342205520") cal = configureHifiPipeline(useHsa=True) obs_1 = hifiPipeline(obs=obs, cal=True, aux=True, palStore=cal)