r83 - 03 Mar 2015 - 14:35:43 - DavidTeyssierYou are here: TWiki >  Public Web  > HifiCalibrationWeb

HIFI instrument and calibration web pages

NEW NEW NEW Latest updates -- 19 September 2014 NEW NEW NEW

  • New beam and coupling efficiency release - see the "HIFI performance and calibration" section

Observing with HIFI

  • HIFI calibration data
    • A significant number of the data taken by HIFI in the framework of the Performance Validation (PV) and routine phase are now publicly available. There are particular caveats that apply to those data. They are described in this disclaimer note

  • See also
    • HIFI fact sheet A 2-page guide to the HIFI instrument
    • A&A paper describing the method used to measure stability performance with HIFI (added 4 Oct 2011)
    • A&A paper about the optimization of mapping modes for heterodyne instruments (added 18 March 2014)]

Reducing HIFI data

Recommended User release

Mac OS X and Java 7 Issues : please consult the HIPE Known Issues page regarding important information on issues with Java for different versions of HIPE.

Which data would most benefit from being reprocessed in HIPE 12.1?

If your data have been processed with the latest bulk reprocessing (11.1):

  • Rotated maps
    • Rotated cubes are now produced in the Level 2.5 product by default for maps carried out with a non-zero position angle, a non-rotated cube is also produced.

  • Solar System Object ephemerides
    • A bug (see here) was fixed in the calculation of SSO ephemeris positions (ra_centre/dec_centre) that led to offsets of up to ~10". This bug affected users that use cubes in a co-moving frame (including the standard level 2.5 cubes), use the doOffset task, or make explicit use of ra_centre / dec_centre in their HIPE scripts. The offset is practically constant over a map and could, e.g., make the emission of a comet appear to be off-center when it really isn't. The fixed bug is in the level 0 pipeline, which users cannot run easily. The issue is fixed by the HIPE 11.1 bulk reprocessing.

  • Improved pointing reconstruction for observations taken before OD 320

If your data have been processed with a version earlier than 10.3:

  • DBS Observations If you have data processed with < HIPE 10
    • A new pipeline step, mkDbsReference, calculates the differences the chop positions in all DBS observations and after applying the band-pass correction stores them in a product in calibration->pipeline-out called ReferenceSpectra. This allows you to check for contamination in chop positions for all types of DBS observations.

  • Improved pointing reconstruction for observations taken between OD 320 and 761 If you have data processed with < HIPE 9
    • The pointing information attached to the data for observations taken between OD 320 and OD761 did not use the most accurate representation of the star tracker focal length. This will be done for the bulk reprocessing with HIPE 9. As a consequence, some observations will experience a shift in astrometry, that can be as high as 8 arcsec. All the details about the consequences for a particular obsid, and recipes to reconstruct yourself the improved pointing can be found at http://herschel.esac.esa.int/twiki/bin/view/Public/HowToUseImprovedPointingProducts as well as in the level0 section of the Hifi Data Reduction Guide

  • Backfilling of observational parameters If you have data processed with < HIPE 9
    • From 9.1 onwards, most of the observational parameters that got optimised in HSpot will be propagated into the observation context, under a new product called HifiUplinkProduct (in the auxiliary product branch). Some of these parameters will be used in the pipeline to e.g. estimate the dimension of the cubes in the mapping and have them more representative of how the map was really obtained.

  • Solar System Object maps If you have data processed with < HIPE 9
    • Cubes for moving targets are now created in the comoving frames in HIPE 9. In order to benefit from that, you should re-pipeline from level 0 up to level 2.5. Note the hifiPipeline task should be called with the option "Aux=True". Please check the Pipeline chapter of the HIFI Data Reduction Guide for more details about how to do that.

Documentation and Cookbooks

  • The HIFI Launch Pad is intended to help you quickly off the ground with HIFI data reduction

  • If you are working in HIPE, the HIFI Data Reduction Guide will be your prime resource for all things HIFI but you should also look at the Herschel Data Analysis Guide for information about general tools, such as those for viewing and manipulating spectra and spectral cubes, and for obtaining data from the archive.
    • [http://herschel.esac.esa.int/hcss-doc-12.0/load/dag/html/dag.html Herschel Data Analysis Guide]]

  • The full set of online documentation for the current user release also contains detailed information about the HIFI pipeline, information about scripting in HIPE, and reference manuals for command listings. New users of HIPE are recommended to read through the Quick Start Guide and the HIPE Owner's Guide

  • The full set of Herschel documentation for the developer track is also available. This documentation contains revisions and updates to the documentation associated with the current User Release but you should also be aware that it may describe functionality not available in the current User Release

Typical Data Reduction Workflow

  • Inspect data quality. Both point spectra and spectral cubes can be viewed in the SpectrumExplorer. In addition to inspecting the quality of your level 2 data, you should always check the level 1 data for any bad scans. If data at level 2 appears strange then looking at level 1 data may provide insight to the problem, see the Data Primer chapter of the HIFI Data Reduction Guide for more information about levels of data

  • Assess whether data should be re-pipelined.
    • Look at the section above and also at the What's New in the latest User Release to decide if you should re-pipeline.
    • Note that the ICC recommends that data processing should not be allowed to fall more than one version behind the HIPE version being used.
    • Everything you need to re-pipeline data is available in the ObservationContext, and the instructions for using the pipeline, including tips on re-pipelining using new calibration data and customising the pipeline, can be found in the Pipeline chapter of the Hifi Data Reduction Guide

  • Flag data. A new user-friendly task has been developed in order to easily flag your data: FlagTool. It can be used both as a GUI (similar concept as for the FitBaseline task) or in command line

  • Remove standing waves. Standing waves are a common problem in HIFI data, particularly in bands 3, 4, 6, and 7. The HIFI Data Reduction Guide provides some information about typical standing waves in HIFI data and describes how to remove them in HIPE using fitHifiFringe and the modified band pass technique

  • Correct baseline drift. Baselines can be flattened by subtracted, or division in the case of real continuum. This can be done in HIPE with the fitBaseline task

  • Fold frequency switch data. Frequency switch data is not folded in the pipeline and can be done with the doFold task.

  • Average together H and V polarisation.
    • The noise estimates given by HSpot assume that the H and V polarisations are averaged together. This can be done in HIPE with the PolarPair task, which resamples the spectra to the same frequency scale (they are frequency calibrated with different comb measurements) and then averages them. Alternatively, you can use the Accumulate task, which allows you to specify the resampling width if you desire.
    • Note that differences may be seen in H and V profiles, see the note on H and V positions. If you are particularly interested in the spatial structure of your source you may prefer not to average the H and V polarisations together.
    • You can also average maps taken with the H and V polarisations:

  • Fit lines. The SpectrumFitterGUI, which is also a part of the Spectral Toolbox, can be used to fit line profiles.

  • You may wish to export data as FITS, ASCII or use the HiClass task to export data as CLASS readable FITs files.
    • Saving as FITS
    • Saving as ASCII
    • HiClass task.
      • warning WARNING! When converting to velocity, CLASS corrects the reference frequency (the frequency at the reference channel) for the Doppler shift, but not the channel width so there is a very small mismatch between the channel width before and after Doppler correction. Lines are shifted from the correct velocity as a consequence, with an increasing shift away from the reference channel.. The ICC recommends that line frequencies are checked in HIPE before exporting to CLASS.

Dedicated data reduction tips for Spectral Scan data

  • The data needs to be deconvolved to a single sideband solution. This can be done with the doDeconvolution task, which works with an ObservationContext.

  • It is important to clean the data before deconvolving it, this means removing standing waves, correcting baselines and flagging out any spurs or bad data that the pipeline missed.

  • If you are planning to deconvolve your data with CLASS, please contact the Helpdesk, or Claudia Comito (email:ccomito@ph1.uni-koeln.de) if you need further details.

Dedicated data reduction tips for Mapping observations

  • Spectral cubes from all HIFI mapping observations are produced as part of the SPG pipeline (that is, the pipeline run for the HSA) and are now found in the level 2.5 product. However, baseline and standing wave corrections are not done automatically in the pipeline because of the risk of harming the scientific content of the data. This can now be customised in the level 2.5 step and be done prior to the cube creation.

  • It is strongly recommended that you inspect your level 2 HTPs for baseline drift and residual standing waves and decide if some, or all, datasets need to be cleaned up before re-running the gridding task.

  • Regridding in HIPE > 9 will automatically use the comoving frame for SSOs. See the section above about the benefits from reprocessing with HIPE 11 for more details on how to do that

  • You can then grid your cleaned data into a spectral cube using the doGridding task, you can also use this task to customise the cube creation to best suit your science goals.

Dedicated data reduction tips for Strong continuum observations

  • Strong continuum sources
    • An optional step in the pipeline can be used to remove standing waves arising in the loads, this is particularly effective for strong continuum sources. A report describing the technique can be found here. Instructions for using this modified passband technique are available in the Standing Wave Removal chapter of the HIFI Data Reduction Guide:

HIFI performance and calibration


Calibration error budget

Line Intensity Calibration Accuracy

The following table provides the percentage flux error associated with each component of the error budget (from Roelfsema et al. 2012).

Error source Bands 1/2 Bands 3/4 Band 5 Bands 6/7
Side-band ratio* 3-4 4-6 4-6 5-8
Hot load coupling < 1 < 1 < 1 < 1
Cold load coupling < 1 < 1 < 1 < 1
Hot load temperature < 1 < 1 < 1 < 1
Cold load temperature < 1 < 1 < 1 < 1
Planetary model error < 3 < 3 < 3 < 3
Beam efficiency < 5 < 5 < 10 < 5
Pointing < 1 < 2 < 2 < 4
Opt. standing waves 4 4 3 3

*Note that the "side-band gain ratio" convention used in the HIFI pipeline does not correspond to the ratio between the respective side-band gains (equal to 1 for a balanced system), but to the gain in one side-band normalised by the combined gain of the two side-band. Consequently, perfectly-balanced side-bands will feature a side-band ratio of 0.5 (see also section 5.3.1 of the Observers' Manual, and the Intensity Calibration Framework document).

Frequency Calibration Accuracy

Spectrometer Resolution Bandwidth (kHz) Accuracy (kHz)
HRS (high-res. mode, Hann appodization) 207 - 364 20
WBS 1100 100

Dominant Contributions:

  • HRS accuracy: Master Oscillator accuracy (1 part in 10^8)
  • HRS resolution: LO signal frequency profile, as determined by the LO Source Unit mixers; band-dependent
  • WBS accuracy: COMB algorithm and interpolation interval
  • WBS resolution: optical alignment & thermal expansion effects within spectrometer

HIFI beams

HIFI beam coupling efficiencies

A revision of the HIFI beam analysis has been performed, based on a multi-epoch set of measurements on Mars. As a result, a more accurate description of the HIFI beams has been obtained, leading to some change in the beam coupling efficiencies, typically a drop. The details of the analysis and the revised coupling efficiencies are given in this release note. For the record, the first release of the HIFI beam efficiencies was described in this document (17 Nov 2010). Please note that these updates will only be featured in the HIFI calibration tree in HIPE 13.

HIFI beam maps and Encircled Energy Fraction

In addition to this, a detailed description of the HIFI beam models is provided as 2-D beam maps, 1-D azimuthal beam averages, and Encircled Energy Fractions (EEF). Each are provided at two frequencies per mixer band and polarisation, and can be extrapolated (within the LO frequencies of a given band) as function of wavelength. The release note provides further details about the delivered files and how to use them. In particular, some useful functions are provided in this Jython script.

The beam material is concatenated as single FITS file per mixer, frequency and polarisation. Each file has three extensions table: one for the 2-D beam model maps (2705x2705 pixels), one for the EEF, and one for the 1-D azimuthal beam averages. Note that 2-D beam model of smaller size (201x201 pixels) will also be made available in the calibration tree (not before HIFI 13), and used in the generation of level 2.5 cubes by the pipeline. The following page provides links to each individual file (27 Mb each). Plots of the 1-D azimuthal beam averages and EEF for each band can be found in the Appendix B of the release note.

Access to the calibration tree:

  • Information about what calibration is found in the HIFI ObservationContext is found in the Data Primer of the HIFI Data Reduction Guide

  • Information about how to find what calibration version was used on your data, how to get the latest calibration and reprocess observations with it is found in the Pipeline chapter of the HIFI Data reduction Guide

  • the latest calibration tree pool can be also retrieved directly here (2 July 2014)

  • Updates to the HIFI calibration data are generally concurrent with the release of each major version of the HCSS-HIFI software. However, it is possible to have updates to the calibration data in between major releases of the software as the software and data are independent of each other, in the table below the HIPE or the OD from which the calibration updates apply are given. The calibration versions available since HIPE 5 are listed in the table below

Calibration version number Release date OD HIPE version Changes
IA_CAL_USER_ or HIFI_CAL_ dd-mm-yy      
2_0 29-11-10   5.0 Beam efficiency parameters introduced, updates to spur table
3_0 11-01-11   5.1 Beam efficiency parameters updated
4_0 18-02-11 645   Spur table update
5_0 13-04-11   6.1 Smoothing widths of OFF positions updated
6_0 21-06-11 779   Sideband ratios in band 2a, addition of strong spur at 1108 GHz in band 5a to spur table
7_0 12-12-11   8.0 Prevention of unnecessary Quality Flags by correction and addition of units in calibration products and correction of thresholds, addition of Quality Flags when LO multiplier currents are out of limits, improved saturated pixel flagging, addition to spur table, uplink product
8_0 03-02-12 995 8.1 Update of a priori table of IF saturations, used in order to flag bad LO data to be discarded in the deconvolution
9_0 24-07-12   9.0 Sideband ratios in bands 5a and 5b, update of a number of engineering threshold for more accurate quality flagging related to hardware housekeeping, introduction of a list of a priori known corrupted data-frames for dedicated flagging in level0 data
10_0 08-11-12   9.1 Introduction of a new HifiUplink product to back-fill most the observational parameters as of HSpot optimisation back into the Uplink product, update of the list of corrupted data-frames for flagging at Level 0, update of a number of engineering threshold for more accurate quality flagging related to hardware housekeeping
11_0 22-01-13   10.0 Update of the list of corrupted data-frames for flagging at Level 0, Quality flags created for SEUs (Singe Event Upsets leading to on-board software corruption)
12_0 10-05-13   10.1 Update of the list of corrupted data-frames for flagging at Level 0, updates to quality flag meta-data and uplink information for mapping modes
13_0 30-05-13   11.0 Update of the list of corrupted data-frames for flagging at Level 0, updates to spur table to account for band 3B purification
14_0 04-07-13   11.0 Update of the list of corrupted data-frames for flagging at Level 0
15_0 18-03-14   12.0 Update of the list of corrupted data-frames for flagging at Level 0, ingestion of the Electrical Standing Wave correction reference table
18_0 02-07-14   12.1 Update of the list of corrupted data-frames for flagging at Level 0, Absolute Pointing Error (APE) added to allow a quality flag to be raised in future releases in case of large pointing offsets, updates to allow IF-dependent corrections to side band ratio, update to spur tables to store entry of spurs found by the ICC or a Key Programme, addition of an instrument uncertainty table (currently only placeholder)

  • The side-band ratio assumed in the pipeline as of the latest calibration tree pool are illustrated in this plot.

  • The HIFI beam release #1 (see section above) including updated aperture efficiencies, main-beam efficiencies, and HPBW is not yet reflected in the HIFI caltree.

Outstanding calibration issues

  • Standing waves:
    • there are several standing waves that can affect the HIFI data at various processing levels. Their nature and impact on the HIFI calibration are described in the Standing Wave technical note and, more briefly, in the Standing Wave removal chapter of the HIFI Data Reduction Guide
    • There are various techniques currently offered to clean these baseline distortions:
      • Remove sine waves, or combinations of sine waves. Check the bullet about usage of the fitHifiFringe task in the data reduction section above.
      • Electrical Standing Wave (aka Matching technique): to correct for the IF standing wave present in bands 6 and 7
      • in case of strong source continuum some standing waves can be enhanced, as described in the Alternative Calibration Scheme report. Such standing waves can be significantly reduced in amplitude using an alternative pipeline algorithm known as the Modified Passband Technique , which is described in the Standing Wave Removal chapter of the HIFI Data Reduction Guide. Alternatively, you can run the Level1PipelineAlgo_hc_filtered_v1.py script.

  • Baseline removal: imperfect ON-OFF calibration scheme can result in residual baseline distortion (not necessarily standing waves). Check the bullet about usage of the fitBaseline task in the data reduction section above.

  • Spectral purity: there are places over the HIFI frequency range where the Local Oscillator does not offer a single frequency tone, so that spurious spectral line may end up in the data, and the calibration of the targeted line gets affected. A dedicated cookbook on the matter is in preparation. In the meantime, we refer to section 5.4.6 of the Observer's Manual, and to the release notes provided above.

  • Spurs: Spurs are reported in HSpot when using the frequency editor and general information is also provided in the HIFI Observer's Manual (see above link). Spurs are also being checked in the pipeline and any detected feature is reported in the TrendAnalysis product of an observation context. The list of currently known spurs can be found in section 5.4.6 of the Observer's Manual. However not all possible spurs can be predicted or caught by the data processing, so that Users are invited to inform the Helpdesk about any features they found in their data and may not have been properly flagged by the system.

Interest Groups and Scripts

  • HIPI: a library of HIFI Plug-In's has been prepared by the NHSC, and is regularly upgraded with new features - check the HIPI website for more information
  • The following interest groups relate to processing of observations taken with HIFI. The links provided allow subscription to these interest groups.
    • subscribe to PACS, SPIRE and HIFI spectral maps interest group
    • subscribe to HIFI point source and spectral scan interest group
  • User scripts: Users are welcome to submit scripts they believe could be of general interest to the community to the Herschel Helpdesk.

Further Information

-- DavidTeyssier , CarolynMcCoey and SylvieFBeaulieu - 03 March 2015

toggleopenShow attachmentstogglecloseHide attachments
Topic attachments
I Attachment Action Size Date Who Comment
pdfpdf aa14698-10.pdf manage 767.0 K 21 Jan 2011 - 09:38 AnthonyMarston de Graauw et al 2010 -- HIFI instrument paper
pdfpdf calibframework.pdf manage 303.5 K 08 Jun 2011 - 08:18 DavidTeyssier  
pdfpdf spatial_response_framework.pdf manage 581.8 K 08 Jun 2011 - 08:18 DavidTeyssier  
pdfpdf freq_framework.pdf manage 181.3 K 08 Jun 2011 - 08:18 DavidTeyssier  
pdfPDF freq_vel_3.PDF manage 305.2 K 08 Jun 2011 - 08:19 DavidTeyssier  
txttxt Level1PipelineAlgo_hc_filtered_v1.py.txt manage 10.6 K 09 Jun 2011 - 09:08 DavidTeyssier  
pdfpdf Sherry_et_al_AAS217_poster.pdf manage 101.6 K 26 Jun 2011 - 16:00 DavidTeyssier  
pdfpdf thesis_ronan_higgins_nuim.pdf manage 18922.3 K 26 Jun 2011 - 16:10 DavidTeyssier  
pdfpdf freq_vel_1.1.pdf manage 255.4 K 26 Jun 2011 - 17:55 DavidTeyssier  
pdfpdf calibration_hifi_stdw_2011_v0_1.pdf manage 4022.0 K 10 Aug 2011 - 20:47 CarolynMcCoey  
pdfpdf HifiObservingModesPerformance_110926a.pdf manage 4324.4 K 30 Sep 2011 - 13:26 JeanMatagne  
pdfpdf Ossenkopf_2008_AllanVariance.pdf manage 1500.3 K 04 Oct 2011 - 09:27 DavidTeyssier  
pdfpdf aa15120-10.pdf manage 377.9 K 01 Dec 2011 - 17:21 DavidTeyssier  
pdfpdf Alternative_Calibration_Schemes_v1.0.pdf manage 2815.8 K 14 Dec 2011 - 08:57 DavidTeyssier  
pngpng HIFI_SBR_Summary_V9.0.png manage 86.0 K 23 Aug 2012 - 11:18 DavidTeyssier  
pdfPDF calibration_hifi_stdw_2011_v0_1.PDF manage 2897.1 K 10 Jan 2013 - 10:58 DavidTeyssier  
pdfpdf HIFI_Calibration_ReleaseNotes.pdf manage 174.5 K 31 Jan 2013 - 12:56 DavidTeyssier  
pdfpdf HIFI_SBR_Cookbook_V1.0.pdf manage 1440.2 K 27 Jun 2013 - 13:16 SylvieFBeaulieu HIFI_SBR_Cookbook_V1.0
pdfpdf spatial_response_framework_v1.9.pdf manage 589.0 K 10 Feb 2014 - 09:11 DavidTeyssier  
txttxt hebCorrection_calTree_example.py.txt manage 2.4 K 18 Mar 2014 - 12:57 DavidTeyssier  
txttxt hebCorrection_externalModels_example.py.txt manage 2.4 K 18 Mar 2014 - 12:57 DavidTeyssier  
elsefits hebCorrectionModels_HICAL15.fits manage 241.9 K 18 Mar 2014 - 12:57 DavidTeyssier  
pdfpdf HIFISubbands.pdf manage 8.5 K 23 Jul 2014 - 18:52 CarolynMcCoey  
pdfpdf HifiBeamReleaseNote_Sep2014.pdf manage 1513.2 K 01 Oct 2014 - 12:10 DavidTeyssier  
txttxt 16293_SIS.txt manage 171.7 K 09 Oct 2014 - 14:50 SylvieFBeaulieu CASSIS lineList for the idenfifyLines task (HDRG, chapter 'The HIFI line identification tool')
elsefits Orion_S_baselined_485_490.fits manage 168.8 K 09 Oct 2014 - 14:51 SylvieFBeaulieu Spectrum1d (fits) for the idenfifyLines task (HDRG, chapter 'The HIFI line identification tool')
txttxt line_identification_guide.py.txt manage 12.0 K 09 Oct 2014 - 14:52 SylvieFBeaulieu Script for the idenfifyLines task (HDRG, chapter 'The HIFI line identification tool')
txttxt getHifiBeam.py.txt manage 5.1 K 01 Dec 2014 - 15:06 DavidTeyssier  
pdfpdf ESWCorrection.pdf manage 640.3 K 21 Jan 2015 - 12:49 DavidTeyssier  
Edit | Attach | Printable | Raw View | Backlinks: Web, All Webs | History: r83 < r82 < r81 < r80 < r79 | More topic actions
Herschel Twiki