• HIPE 14.0 has been released
• Check the important calibration updates and associated improvements expected in the HIPE 14 reprocessed data- this includes in particular updated sideband gain ratio in bands 1 to 4
• The HIFI products now come with a frequency-dependent intensity calibration uncertainty budget
• Channel flags for spurs are now populated in all HIFI observations (this was limited to spectral scan up to HIPE 13)
• Re-organisation of the HIFI quality flags for a better assessement of issues being relevant or not to the end products

• • The HIFI observer's manual version 2.4 (version for OT-2) (HTML) or PDF (5.7 Mb)

• • Technical note on HIFI mapping modes (30 June 2010)

• • HIFI fact sheet A 2-page guide to the HIFI instrument
  • A&A paper describing the method used to measure stability performance with HIFI (added 4 Oct 2011)
  • A&A paper about the optimisation of mapping modes for heterodyne instruments (added 18 March 2014)

• HIPE 14.0

Which data would most benefit from being reprocessed in HIPE 14.0?

• Intensity calibration uncertainty
  
  • HIPE 14 introduces a new product providing an estimated breakdown of the intensity calibration uncertainty. The uncertainties are given separately for each component, as well as in a quadratic sum fashion for all statistically independent elements. This sum applies to the calibration of data in the Ta* scale. Further systematic and random error apply for data converted into different intensity scales (e.g. Tmb or Jy). See further details in section 9 of the DRG.

• Pointing reconstruction
  
  • In HIPE 14, those observations where the interlacing mode was activated (i.e. using more than just 9 guide stars in the star tracker) will benefit from an more accurate reconstructed astrometry and so positions may change in those cases. On top of that, the quality figure associated to the new gyro-propagated method introduced back in HIPE 13 (see also the following Pointing Information page) is now computed in a slightly different fashion. Since the application or not of the gyro-propagated pointing is, for HIFI, conditional upon a certain threshold on this quality (avoiding to apply the new pointing reconstruction to under-performing case), the number of case making us of one or the other approach will differ in HIPE 14 - we recall that in case the gyro-propagated method is discarded, the pointing is the same as that used back in HIPE 12.For the HIFI data such a correction is applied in a conditional fashion depending on a quality figure computed for each individual observation. The new pointing reconstruction will not apply to under-performing cases, and those latter will still use the pointing files used back in HIPE 12. Details about the new attitude reconstruction, and the way it is approached and may impact the HIFI data, can be found in this memo.

• • From HIPE 13 onward observations in bands 6 and 7 have been automatically corrected from the Electrical Standing Wave affecting those data. The correction is based on an optimised fit to the baseline artefact stored in the HIFI calibration files and applied by the pipeline. In HIPE 14, some of those stored solutions have been refined and so the resulting products will be improved. For instructions on how to benefit from this reprocessing and see the typical improvement expected in the data please refer to this section from the HIFI Data Reduction Guide (DRG).

• • The reprocessing of spectral scans in HIPE 13 made use of optimised mask tables for spurs and unruly baseline ranges, resulting in improved deconvolved solutions at the Level 2.5 (an example of such improvement between HIPE 12 (left - red) and HIPE 13 (right - black) is shown in this plot - see also section 5.4 of the DRG). In HIPE 14, a dozen of additional spectral scans (mostly from the calibration programme) have been added to this list.In order to benefit from this update you should reprocess your data from Level 0 with the new calibration tree.

• Spur warning flags
  
  • In HIPE 14, a new channel flag has been introduced ("warning") that will be assigned to the data based on a knowledge base of spurious features built from the spectral scan flagging (see bullet above). These flags will be applied to all point and mapping observations. They should be taken as indicative as they not necessarily accurately match spurious features in the data where they got applied. For this reason, this new flag is not honoured by any of the standard interactive post-processing tasks.

• Flags in OFF positions.
  
  • In HIPE 13 the data used in the OFF positions were already processed up to an equivalent Level 2 calibration (both in intensity and frequency) in order to be directly comparable to the ON-target data. In HIPE 14, those OFF spectra will also hold spur and warning flags that will be propagated from the ON-targert ones. This feature is for example very interesting in order to perform a deconvolution of the OFF spectras in a spectral scan and be able to compare the OFF spectra to the level 2.5 deconvolved products of those observations.

• • A bug (see here) was fixed in the calculation of SSO ephemeris positions (ra_centre/dec_centre) that led to offsets of up to ~10". This bug affected users that use cubes in a co-moving frame (including the standard Level 2.5 cubes), use the doOffset task, or make explicit use of ra_centre / dec_centre in their HIPE scripts. The offset is practically constant over a map and could, e.g., make the emission of a comet appear to be off-center when it really isn't. The fixed bug is in the Level 0 pipeline, which users cannot run easily. The issue was fixed in HIPE 11.1.

• DBS Observations If you have data processed with < HIPE 10
  • A new pipeline step, mkDbsReference, calculates the differences in the chop positions in all DBS observations, and after applying the band-pass correction, stores them in a product in calibration->pipeline-out called ReferenceSpectra. This allows you to check for contamination in chop positions for all types of DBS observations.

• Improved pointing reconstruction for observations taken between OD 320 and 761 If you have data processed with < HIPE 9
  • The pointing information attached to the data for observations taken between OD 320 and OD761 did not use the most accurate representation of the star tracker focal length. This was done for the bulk reprocessing with HIPE 9. As a consequence, some observations will experience a shift in astrometry, that can be as high as 8 arcsec. Details about the consequences for a particular obsid, and recipes to reconstruct the improved pointing yourself can be found at http://herschel.esac.esa.int/twiki/bin/view/Public/HowToUseImprovedPointingProducts as well as in the Level 0 section of the HIFI Data Reduction Guide.

• Backfilling of observational parameters If you have data processed with < HIPE 9

• Solar System Object maps If you have data processed with < HIPE 9
  • Cubes for moving targets are now created in the comoving frames in HIPE 9. In order to benefit from that, you should re-pipeline from Level 0 up to Level 2.5. Note that the hifiPipeline task should be called with the option "Aux=True". Please check the Pipeline chapter of the HIFI Data Reduction Guide for more details about how to do that. <!--* Calibration of data in bands 2a, 5a and 5b If you have data processed with < HIPE 8
    * Un-balanced sideband gains will apply for some ranges of these bands (see also the calibration section). Reprocessing data from band 2a is recommended if your data are still processed with HIPE 6. Reprocessing data from band 5a and 5b is recommended if your data are processed with HIPE 8.
    
    * Spectral Scan Data in bands 6 and 7 If you have data processed with < HIPE 8
    * It is recommended to reprocess spectral scan data taken in band 6 and 7 if they are still processed with HIPE <8. This is not needed in data processed with HIPE 8 onward.
    * Reprocessing with HIPE 9 onward will create a new Level 2.5 product containing a deconvolved single-sideband spectrum for each applicable spectrometer
    
    * Maps If you have data processed with < HIPE 8
    * Spectral cubes from mapping observations after OD 835 gridded using actual scan line and readout used in observation, earlier maps use a better approximation than in HIPE 7. These improvements are a consequence of changes in the Level 0 pipeline to read in information about how the observation was performed and require the pipeline to be re-run after running the doUplink step:
    
    <verbatim>
    from herschel.hifi.pipeline.level0 import DoUplink
    doUplink=DoUplink()
    backend = "WBS-V"
    htp=obs.refs["level1"].product.refs[backend].product
    htp1=doUplink(htp=htp.copy(), auxiliary=obs.getAuxiliary())
    l1=obs.refs["level1"].product
    l1.setProduct(backend,htp1)
    obs.setProduct("level1",l1)
    obs = hifiPipeline( obs=obs, fromLevel=1, upToLevel=2.5,save=0,apids=[backend])
    #
    # Now repeat replacing backend with "WBS-H", "HRS-H", "HRS-V"
    </verbatim> -->

• See the What's New page for more details of improvements in HIPE 14.0
  • http://herschel.esac.esa.int/twiki/bin/view/Public/HipeWhatsNew14x

• Other details about the benefits of reprocessing with HIPE 14.0 are also compiled in the HIFI Data Reduction Guide .

• • HIFI Launch Pad

• • HIFI Cookbooks

• • HIFI Data Reduction Guide

• A series of useful scripts can be found following the HIPE menu Scripts -> HIFI Useful Scripts

• • HIFI Pipeline Specification Document

• If you are working in HIPE, the HIFI Data Reduction Guide will be your prime resource for all things HIFI but you should also look at the Herschel Data Analysis Guide for information about general tools, such as those for viewing and manipulating spectra and spectral cubes, and for obtaining data from the archive.
  • Herschel Data Analysis Guide

• • online documentation for the current User Release

• • full set of Herschel documentation for the developer track

• • Documentation about downloading data

• • Viewing spectra in Spectrum Explorer
  • Viewing cubes in Spectrum Explorer
  • Data Primer

• • Look at the section above and also at the What's New in the latest User Release to decide if you should re-pipeline.

• • • Pipeline chapter

• Flag data. A user-friendly task has been developed in order to easily flag your data: flagTool. It can be used both as a GUI (similar concept as for the fitBaseline task) or in command line
  • Flagging data interactively

• • Standing Wave Removal

• • Baseline Removal

• • doFold documentation.

• • • PolarPair
    • Accumulate

• • • note on H and V positions

• • • Combining cubes

• • Spectral Toolbox documentation for spectra
  • Spectral Toolbox documentation for spectral cubes

• • Spectral Fitting

• • Saving data to a pool
  • Storing your HIPE session

• You may wish to export data as FITS, ASCII or use the hiClass task to export data as CLASS readable FITs files.
  • Saving as FITS
  • Saving as ASCII
  • HiClass task - Note, however, that GILDAS/Class can now directly read the HCSS-generated FITS files (as of release October 2015) and so it is no longer required to convert the HCSS FITS files into a dedicated format. More details about this new FITS reader can be found in this report.

• Line identification. IdentifyLines and exportLines are two new tasks that allow you to identify and export lines in your spectrum, and then run a comparison of known lines with a Linelist. To learn how to use the tasks identifyLines and exportLines, you will need the following three files, and we direct you to HIFI Data Reduction Guide for further details on how to run the tasks:

• • 16293_SIS.txt (the CASSIS lineList) Note that a copy of the linelist is now part of the HIPE build.

• • Deconvolving HIFI data

• It is important to clean the data before deconvolving it: this means removing standing waves, correcting baselines, and flagging out any spurs or bad data that the pipeline missed. The calibration tree for HIPE 14 provides optimised mask tables for each spectral scan obsids so it is recommended that your make use of those, typically be reprocessing your data with HIPE 14 and this calibration file (see above).

• Note that deconvolution does not work properly on data taken in the Frequency Switching mode (see also the Data Products Known Issues Page)

• Regridding in HIPE > 9 will automatically use the comoving frame for SSOs. Considering the section above, mapping observations of SSOs will benefit from reprocessing in HIPE > 11.1.

• • Creating Spectral Cubes

• • • Modified Passband Technique

• • http://arxiv.org/abs/1404.2806: "The effect of sideband ratio on line intensity for Herschel/HIFI", Experimental Astronomy.
  • After an extensive investigation, the HIFI ICC has introduced updated sideband gain ratio in the calibration tree. A final documentation is still in preparation but the overview of the sideband gain ratio frequency dependence in each band is illustrated in this plot.

• • ZIP file containing information and data needed for accessing HIFI H and V beam positions from within an observation context in HIPE. (3 Dec 2010)

• This FTP repository of system temperature plots across the IF for each HIFI subband (updated 24 Jan 2011) inform about the variation in system temperature (sensitivity) across the HIFI intermediate frequency band at various frequencies.

• Information about what calibration is found in the HIFI ObservationContext is found in the Data Primer of the HIFI Data Reduction Guide.

• Information about how to find what calibration version was used on your data, how to get the latest calibration, and reprocess observations with it is found in the Pipeline chapter of the HIFI Data reduction Guide.

• The latest calibration tree pool can also be retrieved directly here (10 December 2015).

• An overview of the data product known issues can be found Data Known Issues Page

• • • Standing Wave Removal

• • • Remove sine waves, or combinations of sine waves. Check the bullet about usage of the fitHifiFringe task in the data reduction section above.

• • • • A new task hebCorrection has been implemented in HIPE 12.0 and is being used by default by the pipeline in HIPE 13 (see also the Reducing HIFI data section above). We warn that the task does not guarantee an accurate correction in all circumstances .
      • The task is described in the following document. The mathematical background can be found in this proceeding. Unlike the script made available in previous version of HIPE, this task is applicable to any HIFI observing mode.

• • • • A thorough description of the technique, and the underlying causes, is in Ronan Higgins' thesis: Advanced optical calibration of the Herschel HIFI heterodyne spectrometer
    • in case of strong source continuum some standing waves can be enhanced, as described in the Alternative Calibration Scheme report. Such standing waves can be significantly reduced in amplitude using an alternative pipeline algorithm known as the Modified Passband Technique , which is described in the Standing Wave Removal chapter of the HIFI Data Reduction Guide. Alternatively, you can run the Level1PipelineAlgo_hc_filtered_v1.py script.
      

• • • • Modified Passband Technique.

• Baseline removal: imperfect ON-OFF calibration scheme can result in residual baseline distortion (not necessarily standing waves). Check the bullet about usage of the fitBaseline task in the data reduction section above.

• Spectral purity: there are places over the HIFI frequency range where the Local Oscillator does not offer a single frequency tone, so that spurious spectral line may end up in the data, and the calibration of the targeted line gets affected. A dedicated cookbook on the matter is in preparation. In the meantime, we refer to section 5.4.6 of the Observer's Manual, and to the release notes provided above.

• Spurs: Spectral Spurs are narrow and broad spectral features due to instabilities in the Local Oscillator. Up to HIPE 12, spurs have been assigned by the pipeline in an automatic fashion. In HIPE 13, this is no longer the case and the masks are assigned based on a priori tables provided by the ICC in the calibration tree. As of now, all spectral scans benefit from such tables, but only a handful of single point and spectral mapping observations. This limitation has little impact on the top level data though since no pipeline module makes use of these masks in those modes. A list of the spurs registered up to HIPE 12 can be found in section 5.4.6 of the Observer's Manual. This table will be updated soon based on the new statistics derived from the HIPE 13 mask tables.

• HIPI: a library of HIFI Plug-In's has been prepared by the NHSC - check the HIPI website for more information. From HIPE 13 onwards, most of those utilities have been ported to Useful scripts (Menu Scripts > HIFI Useful Scripts)

• • subscribe to PACS, SPIRE and HIFI spectral maps interest group
  • subscribe to HIFI point source and spectral scan interest group
• User scripts: Users are welcome to submit scripts they believe could be of general interest to the community to the Herschel Helpdesk.

• More information -- particularly for US users -- can be found at the HIFI home page at the NHSC NASA Herschel Science Center
• HIFI science pages
• A&A Herschel First Science Result issue (Vol. 518) and A&A Herschel/HIFI Special issue (Vol. 521)