(r91) HifiCalibrationWeb < Public

Public Web>HifiCalibrationWeb (revision 91) (raw view)~~EditAttach~~
---+ HIFI instrument and calibration web page

%TOC{depth="2"}%

%N% %N% %N% %RED% *Latest updates -- 14 April 2015* %ENDCOLOR% %N% %N% %N%
   * %BLUE%HIPE 13.0 has been released%ENDCOLOR%
   * %BLUE%Check below the important calibration updates and associated improvements expected in the HIPE 13 reprocessed data %ENDCOLOR%
   * %BLUE%HIFI now feature a series of useful scripts %ENDCOLOR%

---++ Observing with HIFI
   * Useful Documentation
      * The HIFI observer's manual version 2.4 (version for OT-2) [[http://herschel.esac.esa.int/Docs/HIFI/html/hifi_om.html  (HTML)]] or [[http://herschel.esac.esa.int/Docs/HIFI/pdf/hifi_om.pdf PDF (5.7 Mb)]]
      * [[%ATTACHURL%/aa14698-10.pdf The Herschel-Heterodyne Instrument for the Far-Infrared (HIFI) -- PDF (0.8MB)]], de Graauw et al., 2010, A&A, 518, L6, in-flight scientific capabilities of the HIFI instrument.
      * [[%ATTACHURL%/HifiObservingModesPerformance_110926a.pdf    HIFI AOT Observing Mode Release and Performance Notes v3.0 (24 September 2011)]]
      * [[http://herschel.esac.esa.int/Docs/TechnicalNotes/HIFI_Mapping_Modes_note_30Jun2010.pdf Technical note on HIFI mapping modes (30 June 2010)]]
      * [[%ATTACHURL%/HIFISubbands.pdf  Table of frequency ranges of HIFI subbands]]

   * HIFI calibration data
      * A significant number of the data taken by HIFI in the framework of the Performance Validation (PV) and routine phase are now publicly available. There are particular caveats that apply to those data. They are described in this [[%ATTACHURL%/HIFI_Calibration_ReleaseNotes.pdf disclaimer]] note. 
 
   * See also
      * [[http://herschel.esac.esa.int/Docs/Flyers/HIFI_flyer_20May2010.pdf HIFI fact sheet]] A 2-page guide to the HIFI instrument
      * [[http://cdsads.u-strasbg.fr/abs/2008A%26A...479..915O A&A paper]] describing the method used to measure stability performance with HIFI (added 4 Oct 2011)
      * [[http://cdsads.u-strasbg.fr/abs/2009A%26A...495..677O A&A paper]] about the optimisation of mapping modes for heterodyne instruments (added 18 March 2014)]

---++ Reducing HIFI data

---+++ Recommended User release
   * [[http://herschel.esac.esa.int/HIPE_download.shtml#HIPE_download HIPE 13.0]]

*%RED%HIPE 13 and Java 8%ENDCOLOR%* : please consult the [[http://herschel.esac.esa.int/twiki/bin/view/Public/HipeKnownIssues HIPE Known Issues]] page regarding important information on issues with Java for different versions of HIPE.

---+++ Which data would most benefit from being reprocessed in HIPE 13.0?

%BLUE% *If your data have been processed with the latest bulk reprocessing (12.1):* %ENDCOLOR%

   * *Pointing in all observations* 
      * A new approach to pointing reconstruction has been implemented in HIPE 13 by introducing additional correction based on the gyroscope information (see the following [[http://herschel.esac.esa.int/twiki/bin/view/Public/ImprovedPointingGyro Pointing Information page]]). For the HIFI data such a correction will be applied in a conditional fashion depending on a quality figure computed for each individual observation. The new pointing reconstruction will not apply to under-performing cases, and those latter will still use the pointing files used back in HIPE 12. Details about the new attitude reconstruction, and the way it is approached and may impact the HIFI data, can be found in [[%ATTACHURL%/HIFI_PointingAccuracyNote.pdf this memo]].

   * *Band 6 and 7 observations*
      * From HIPE 13 onward observations in bands 6 and 7 will automatically be corrected from the Electrical Standing Wave affecting those data. The correction is based on an optimised fit to the baseline artefact stored in the HIFI calibration files and applied by the pipeline. This also means that the optimum solution may not necessarily completely correct the standing wave. Note that the continuum present in band 6 and 7 data can only be fully trusted if the Electrical Standing Wave is corrected (see also the Outstanding Calibration Issues section below). Refined solutions will be provided on a case by case basis in HIPE 14. For instructions on how to benefit from this reprocessing and see the typical improvement expected in the data please refer to this [[http://herschel.esac.esa.int/hcss-doc-13.0/index.jsp#hifi_um:sw_removal_heb2 section]] from the HIFI Data Reduction Guide (DRG).

   * *Spectral Scan observations*
      * The reprocessing of spectral scans in HIPE 13 will make use of optimised mask tables for spurs and unruly baseline ranges. This will result in improved deconvolved solutions at the Level 2.5. An example of such improvement between HIPE 12 (left - red) and HIPE 13 (right - black) is shown in this [[%ATTACHURL%/1342180554_band4b.png plot]]. In order to benefit from this update you should reprocess your data from Level 0 with the new calibration tree (see [[http://herschel.esac.esa.int/hcss-doc-13.0/load/hifi_um/html/pipeline_cal.html section 5.4]] of the DRG).

   * *Calibrated OFF position spectra (any mode)*
      * In HIPE 13 the data used in the OFF positions will be processed up to an equivalent Level 2 calibration (both in intensity and frequency) and can therefore be directly compared to the ON-target data. Details about the peculiarity of each OFF spectra can be found in the cookbooks (see next section). In order to generate those spectra you need to reprocess the data from the Level 1, and can work from the calibration tree present in the data processed with HIPE 12.1 - see the [[http://herschel.esac.esa.int/hcss-doc-13.0/index.jsp#hifi_um:runninghifipipeline pipeline section]] of the DRG for more details.

   * *Single Point and Spectral Mapping modes*
      * The Level 2.5 products in these modes have been optimised by stitching the respective spectrometer subbands. In maps, for example, this will result in a smaller number of cubes. More details can be found in [[http://herschel.esac.esa.int/hcss-doc-13.0/load/hifi_um/html/hdrg_what_done_2_5.html section 4.6]] of the DRG.

%BLUE% *If your data have been processed with a version earlier than 11.1:* %ENDCOLOR%

   * *Solar System Object ephemerides*
      * A bug [[http://herschel.esac.esa.int/twiki/bin/view/Public/HipeKnownIssues#Known_issues_in_HIPE_10_1 (see here)]] was fixed in the calculation of SSO ephemeris positions (ra_centre/dec_centre) that led to offsets of up to ~10".  This bug affected users that use cubes in a co-moving frame (including the standard Level 2.5 cubes), use the doOffset task, or make explicit use of ra_centre / dec_centre in their HIPE scripts. The offset is practically constant over a map and could, e.g., make the emission of a comet appear to be off-center when it really isn't.  The fixed bug is in the Level 0 pipeline, which users cannot run easily. The issue was fixed in HIPE 11.1. 

%BLUE% *If your data have been processed with a version earlier than 10.3:* %ENDCOLOR%

   * *DBS Observations* %BLUE% If you have data processed with < HIPE 10 %ENDCOLOR%
      * A new pipeline step, mkDbsReference, calculates the  differences in the chop positions in all DBS observations, and after applying the band-pass correction, stores them in a product in calibration->pipeline-out called _ReferenceSpectra._  This allows you to check for contamination in chop positions for all types of DBS observations.

   * *Improved pointing reconstruction for observations taken between OD 320 and 761*  %BLUE% If you have data processed with < HIPE 9 %ENDCOLOR%
      * The pointing information attached to the data for observations taken between OD 320 and OD761 did not use the most accurate representation of the star tracker focal length. This was done for the bulk reprocessing with HIPE 9. As a consequence, some observations will experience a shift in astrometry, that can be as high as 8 arcsec. Details about the consequences for a particular obsid, and recipes to reconstruct the improved pointing yourself can be found at [[http://herschel.esac.esa.int/twiki/bin/view/Public/HowToUseImprovedPointingProducts]] as well as in the [[http://herschel.esac.esa.int/hcss-doc-13.0/index.jsp#hifi_um:what_l0 Level 0 section]] of the HIFI Data Reduction Guide.

   * *Backfilling of observational parameters* %BLUE% If you have data processed with < HIPE 9 %ENDCOLOR%
      * From 9.1 onwards, most of the observational parameters that got optimised in !HSpot will be propagated into the observation context, under a new product called !HifiUplinkProduct (in the auxiliary product branch). Some of these parameters will be used in the pipeline to e.g. estimate the dimension of the cubes in the mapping and have them more representative of how the map was really obtained.

   * *Solar System Object maps* %BLUE% If you have data processed with < HIPE 9 %ENDCOLOR%
      * Cubes for moving targets are now created in the comoving frames in HIPE 9. In order to benefit from that, you should re-pipeline from Level 0 up to Level 2.5. Note that the hifiPipeline task should be called with the option "Aux=True". Please check the [[http://herschel.esac.esa.int/hcss-doc-13.0/index.jsp#hifi_um:runninghifipipeline Pipeline chapter]] of the HIFI Data Reduction Guide for more details about how to do that.

   <!--* *Calibration of data in bands 2a, 5a and 5b*  %BLUE% If you have data processed with < HIPE 8 %ENDCOLOR%
      * Un-balanced sideband gains will apply for some ranges of these bands (see also the calibration section). Reprocessing data from band 2a is recommended if your data are still processed with HIPE 6. Reprocessing data from band 5a and 5b is recommended if your data are processed with HIPE 8.

   * *Spectral Scan Data in bands 6 and 7* %BLUE% If you have data processed with < HIPE 8 %ENDCOLOR%
      * It is recommended to reprocess spectral scan data taken in band 6 and 7 if they are still processed with HIPE <8. This is not needed in data processed with HIPE 8 onward.
      * Reprocessing with HIPE 9 onward will create a new Level 2.5 product containing a deconvolved single-sideband spectrum for each applicable spectrometer

   * *Maps* %BLUE% If you have data processed with < HIPE 8 %ENDCOLOR%
      *  Spectral cubes from mapping observations after OD 835 gridded using actual scan line and readout used in observation, earlier maps use a better approximation than in HIPE 7.  These improvements are a consequence of changes in the Level 0 pipeline to read in information about how the observation was performed and require the pipeline to be re-run after running the _doUplink_ step:

<verbatim>
from herschel.hifi.pipeline.level0 import DoUplink
doUplink=DoUplink()
backend = "WBS-V"
htp=obs.refs["level1"].product.refs[backend].product
htp1=doUplink(htp=htp.copy(), auxiliary=obs.getAuxiliary())
l1=obs.refs["level1"].product
l1.setProduct(backend,htp1)
obs.setProduct("level1",l1)
obs = hifiPipeline( obs=obs, fromLevel=1, upToLevel=2.5,save=0,apids=[backend])
#
# Now repeat replacing backend with "WBS-H", "HRS-H", "HRS-V"
</verbatim> -->


   * See the What's New page for more details of improvements in HIPE 13.0
      * [[http://herschel.esac.esa.int/twiki/bin/view/Public/HipeWhatsNew13x]]
      
   * Other details about the benefits of reprocessing with HIPE 13.0 are also compiled in the [[http://herschel.esac.esa.int/hcss-doc-13.0/index.jsp#hifi_um:what_l0 HIFI Data Reduction Guide]] .

---+++ Documentation and Cookbooks
   * The HIFI Launch Pad is intended to help you quickly off the ground with HIFI data reduction
      * [[http://herschel.esac.esa.int/hcss-doc-13.0/load/hifi_um/html/hificookbook.html HIFI Launch Pad]]
 
   * Cookbooks now exist for each HIFI observing mode
      * [[http://herschel.esac.esa.int/hcss-doc-13.0/load/hifi_um/html/hdrg_cookbooks.html HIFI Cookbooks]]

   * The _HIFI Data Reduction Guide_ and the _HIFI AOT Observing Mode Release and Performance Notes_ contain information you need to know about HIFI data, regardless of the software you use for data reduction
      * [[http://herschel.esac.esa.int/hcss-doc-13.0/index.jsp#hifi_um:hifi-um HIFI Data Reduction Guide]] 
      * [[%ATTACHURL%/HifiObservingModesPerformance_110926a.pdf    HIFI AOT Observing Mode Release and Performance Notes v3.0 (24 September 2011)]]

   * A series of useful scripts can be found following the HIPE menu Scripts -> HIFI Useful Scripts

   *  If you want to know more about the details of the HIFI pipeline, you should look at the _HIFI Pipeline Specification Document_ 
      * [[http://herschel.esac.esa.int/hcss-doc-13.0/load/hifi_pipeline/html/hifi_pipeline.html HIFI Pipeline Specification Document]]

   * If you are working in HIPE, the [[http://herschel.esac.esa.int/hcss-doc-13.0/index.jsp#hifi_um:hifi-um HIFI Data Reduction Guide]] will be your prime resource for all things HIFI but you should also look at the _Herschel Data Analysis Guide_ for information about general tools, such as those for viewing and manipulating spectra and spectral cubes, and for obtaining data from the archive.
      * [[http://herschel.esac.esa.int/hcss-doc-13.0/load/dag/html/dag.html Herschel Data Analysis Guide]]

   * The full set of online documentation for the current user release also contains detailed information about the HIFI pipeline, information about scripting in HIPE, and reference manuals for command listings. New users of HIPE are recommended to read through the Quick Start Guide and the HIPE Owner's Guide
      * You can create pdfs of each document by clicking on the pdf symbol at the top of the main page. 
      * [[http://herschel.esac.esa.int/hcss-doc-13.0/ online documentation for the current User Release]]

   * The full set of Herschel documentation for the developer track is also available.  This documentation contains revisions and updates to the documentation associated with the current User Release but you should also be aware that it may describe functionality not available in the current User Release
      * [[http://herschel.esac.esa.int/hcss-doc-14.0/  full set of Herschel documentation for the developer track]]  

   * The General HCSS Public Twiki page provides general framework information and updates: 
      * http://herschel.esac.esa.int/twiki/bin/view/Public/WebHome
 


---+++ Typical Data Reduction Workflow
   * _Download data from the HSA_.  If you are downloading several observations it is better to use the tar ball.
      * [[http://herschel.esac.esa.int/hcss-doc-13.0/load/dag/html/Dag.DataIO.Hsa.html Documentation about downloading data]]

   * _Inspect data quality_. Both point spectra and spectral cubes can be viewed in the !SpectrumExplorer.  In addition to inspecting the quality of your Level 2 data,  you should always check the Level 1 data for any bad scans.  If data at Level 2 appears strange then looking at Level 1 data may provide insight to the problem, see the Data Primer chapter of the HIFI Data Reduction Guide for more information about levels of data
      * [[http://herschel.esac.esa.int/hcss-doc-13.0/load/dag/html/Dag.Sec.SpectrumViewing.html Viewing spectra in Spectrum Explorer]]
      * [[http://herschel.esac.esa.int/hcss-doc-13.0/load/dag/html/Dag.Sec.CubeSpectralData.html Viewing cubes in Spectrum Explorer]]
      * [[http://herschel.esac.esa.int/hcss-doc-13.0/index.jsp#hifi_um:dataprimer Data Primer]]

   * _Assess whether data should be re-pipelined_. 
      * Look at the section above and also at the  [[http://herschel.esac.esa.int/twiki/bin/view/Public/HipeWhatsNew What's New]] in the latest User Release to decide if you should re-pipeline.  
      * Note that  the ICC recommends that data processing should not be allowed to fall more than one version behind the HIPE version being used.  
      * Everything you need to re-pipeline data is available in the !ObservationContext, and the instructions for using the pipeline, including tips on *re-pipelining using new calibration data* and *customising the pipeline*, can be found in the Pipeline chapter of the HIFI Data Reduction Guide
         * [[http://herschel.esac.esa.int/hcss-doc-13.0/index.jsp#hifi_um:runninghifipipeline Pipeline chapter]]

   * _Flag data_. A new user-friendly task has been developed in order to easily flag your data: *FlagTool*. It can be used both as a GUI (similar concept as for the *FitBaseline* task) or in command line
      * [[http://herschel.esac.esa.int/hcss-doc-13.0/load/hifi_um/html/hdrg_flags_flagtool.html Flagging data interactively]]

   * _Remove standing waves_. Standing waves are a common problem in HIFI data, particularly in bands 3, 4, 6, and 7.  The HIFI Data Reduction Guide provides some information about typical standing waves in HIFI data and describes how to remove them in HIPE using *fitHifiFringe* and the modified band pass technique
      * [[http://herschel.esac.esa.int/hcss-doc-13.0/index.jsp#hifi_um:sw_removal_hifi2 Standing Wave Removal]] 

   * _Correct baseline drift_. Baselines can be flattened by subtracted, or division in the case of real continuum.  This can be done in HIPE with the *fitBaseline* task
      * [[http://herschel.esac.esa.int/hcss-doc-13.0/index.jsp#hifi_um:hifium_bl1 Baseline Removal]]

   * _Fold frequency switch data_.  Frequency switch data is not folded in the pipeline and can be done with the *doFold* task.
      * [[http://herschel.esac.esa.int/hcss-doc-13.0/load/hifi_pipeline/html/pipespecl25.html#dofold2_5 doFold documentation]].

   * _Average together H and V polarisation_.  
      * The noise estimates given by HSpot assume that the H and V polarisations are averaged together.   This can be done in HIPE with the *PolarPair* task, which resamples the spectra to the same frequency scale (they are frequency calibrated with different comb measurements) and then averages them.  Alternatively, you can use the *Accumulate* task, which allows you to specify the resampling width if you desire.
         * [[http://herschel.esac.esa.int/hcss-doc-13.0/index.jsp#hifi_um:avhv PolarPair]] 
         * [[http://herschel.esac.esa.int/hcss-doc-13.0/load/dag/html/Dag.Sec.SpectrumWorking.html#Dag.Sec.SpectrumStats.SEII Accumulate]]
      * Note that differences may be seen in H and V profiles, see the _note on H and V positions_.  If you are particularly interested in the spatial structure of your source you may prefer not to average the H and V polarisations together.
         * [[http://herschel.esac.esa.int/Docs/TechnicalNotes/HIFI_HandVPositions_03Dec2010.zip note on H and V positions]]
      * You can also average maps taken with the H and V polarisations:
         * [[http://herschel.esac.esa.int/hcss-doc-13.0/load/hifi_um/html/hum_dogridding_common.html#hum_dogrid_combine Combining cubes]]

   * _Spectral manipulation_. The *Spectral Toolbox* in HIPE allows you to perform typical spectral arithmetic tasks on spectra and spectral cubes  
      * [[http://herschel.esac.esa.int/hcss-doc-13.0/load/dag/html/Dag.Sec.SpectrumWorking.html Spectral Toolbox documentation for spectra]]
      * [[http://herschel.esac.esa.int/hcss-doc-13.0/load/dag/html/Dag.Sec.CubeSpectrumWorking.html Spectral Toolbox documentation for spectral cubes]]

   * _Fit lines_. The *SpectrumFitterGUI*, which is also a part of the Spectral Toolbox, can be used to fit line profiles.
      *  [[http://herschel.esac.esa.int/hcss-doc-13.0/load/dag/html/Dag.Ch.SpectralFitting.SEII.html Spectral Fitting]]

   * _Save modified data_ You can save data to a pool, or  store your session in order to continue working in HIPE.
      * [[http://herschel.esac.esa.int/hcss-doc-13.0/load/dag/html/Dag.DataIO.SaveEntireObservation.html Saving data to a pool]]
      * [[http://herschel.esac.esa.int/hcss-doc-13.0/load/hipeowner/html/Hipeowner.Sec.SavingRestoring.html Storing your HIPE session]]

   * You may wish to _export data_ as FITS, ASCII or use the *HiClass* task to export data as CLASS readable FITs files.
      * [[http://herschel.esac.esa.int/hcss-doc-13.0/load/dag/html/Dag.DataIO.Fits.html Saving as FITS]]
      * [[http://herschel.esac.esa.int/hcss-doc-13.0/load/dag/html/Dag.DataIO.Ascii.html Saving as ASCII]]
      * [[http://herschel.esac.esa.int/hcss-doc-13.0/index.jsp#hifi_um:hiclass HiClass task]].
         * %ICON{"warning"}% %RED% *WARNING!* %ENDCOLOR%  When converting to velocity, CLASS corrects the reference frequency (the frequency at the reference channel) for the Doppler shift, but not the channel width so there is a very small mismatch between the channel width before and after Doppler correction. Lines are shifted from the correct velocity as a consequence, with an increasing shift away from the reference channel. The ICC recommends that line frequencies are checked in HIPE before exporting to CLASS.

   * _Line identification_. *IdentifyLines* is a new task that allows you to identify lines in your spectrum and then run a comparison of known lines with a Linelist. To learn how to use the task *identifyLines* you will need the following three files: 
      * [[%ATTACHURL%/line_identification_guide.py.txt line_identification_guide.py.txt (the script - please remove extension '.txt' prior to using the script)]]
      * [[%ATTACHURL%/16293_SIS.txt 16293_SIS.txt (the CASSIS lineList)]] 
      * [[%ATTACHURL%/Orion_S_baselined_485_490.fits Orion_S_baselined_485_490.fits (the spectrum1d)]]

---+++Dedicated data reduction tips for *Spectral Scan* data
   * The data needs to be deconvolved to a single sideband solution. This can be done with the *doDeconvolution* task, which works with an !ObservationContext. 
      *  [[http://herschel.esac.esa.int/hcss-doc-13.0/index.jsp#hifi_um:decon Deconvolving HIFI data]]

   * It is important to clean the data before deconvolving it: this means removing standing waves, correcting baselines, and flagging out any spurs or bad data that the pipeline missed. The calibration tree for HIPE 13 provides optimised mask tables for each spectral scan obsids so it is recommended that your make use of those, typically be reprocessing your data with HIPE 13 and this calibration file (see above).

   * If you are planning to deconvolve your data with CLASS, please contact the Helpdesk, and cc to Claudia Comito (email:ccomito@ph1.uni-koeln.de) if you need further details.

   * Note that deconvolution does not work properly on data taken in the Frequency Switching mode (see also the [[http://herschel.esac.esa.int/twiki/bin/view/Public/DpKnownIssues Data Products Known Issues Page]])

---+++Dedicated data reduction tips for *Mapping* observations
   *  Spectral cubes from all HIFI mapping observations are produced as part of the SPG pipeline (that is, the pipeline run for the HSA) and are now found in the Level 2.5 product. However, baseline and standing wave corrections are not done automatically in the pipeline because of the risk of harming the scientific content of the data. This can now be customised in the Level 2.5 step and be done prior to the cube creation.

   * *It is strongly recommended that you inspect your Level 2 HTPs for baseline drift and residual standing waves, and decide if some, or all, datasets need to be cleaned up before re-running the gridding task.*     

   * Regridding in HIPE > 9 will automatically use the comoving frame for SSOs. Considering the section above, mapping observations of SSOs will benefit from reprocessing in HIPE > 11.1.

   *  You can then grid your cleaned data into a spectral cube using the *doGridding* task, you can also use this task to customise the cube creation to best suit your science goals.
      *  [[http://herschel.esac.esa.int/hcss-doc-13.0/index.jsp#hifi_um:otfmapping Creating Spectral Cubes]] 

---+++Dedicated data reduction tips for *Strong continuum* observations
   * *Strong continuum sources* 
      *  An optional step in the pipeline can be used to remove standing waves arising in the loads, this is particularly effective for strong continuum sources.  A report describing the technique can be found [[%ATTACHURL%/calibration_hifi_stdw_2011_v0_1.PDF here]]. Instructions for using this modified passband technique are available in the _Standing Wave Removal_ chapter of the HIFI Data Reduction Guide:
         * _[[http://herschel.esac.esa.int/hcss-doc-13.0/load/hifi_um/html/sw_removal_passb2.html  Modified Passband Technique]]_ 


---++ HIFI performance and calibration

---+++Documentation
   * General: [[%ATTACHURL%/aa15120-10.pdf In-orbit performance of Herschel-HIFI, Roelfsema et al. 2012]]
   * Intensity calibration: [[%ATTACHURL%/calibframework.pdf Intensity Calibration Framework document]] 
   * Sideband ratio
      * [[%ATTACHURL%/HIFI_SBR_Cookbook_V1.0.pdf Sideband ratio technical note]]: This note describes the general calibration problem associated with the sideband gain imbalance existing in double-sideband heterodyne systems such as HIFI, and provides some recipes and calibration tables applicable to the particular case of the HIFI mixers.
      * [[http://arxiv.org/abs/1404.2806]]: "The effect of sideband ratio on line intensity for Herschel/HIFI", Experimental Astronomy.
   * Beam calibration (%RED% see also HIFI beam sub-section thereafter %ENDCOLOR%)
      * [[%ATTACHURL%/spatial_response_framework_v1.9.pdf Spatial Calibration Framework document]]
      * [[%ATTACHURL%/HifiBeamReleaseNote_Sep2014.pdf Measured beam efficiencies on Mars. (revision v1.1, 1 October 2014)]]
      * [[http://herschel.esac.esa.int/Docs/TechnicalNotes/HIFI_HandVPositions_03Dec2010.zip ZIP file containing information and data needed for accessing HIFI H and V beam positions from within an observation context in HIPE. (3 Dec 2010)]]
   *  Spectral calibration
      *  [[%ATTACHURL%/freq_framework.pdf Spectral Calibration Framework document]]
      *  [[%ATTACHURL%/freq_vel_1.1.pdf HIFI frequency/velocity transform document]]: a document describing the frequency and velocity handling algorithms adopted in the HIFI pipeline.
   * This  [[ftp://ftp.sciops.esa.int/pub/hsc-calibration/HIFI/TSYS_PLOTS/ FTP repository]] of system temperature plots across the IF for each HIFI subband (updated 24 Jan 2011) inform about the variation in system temperature (sensitivity) across the HIFI intermediate frequency band at various frequencies.
   
---+++Calibration error budget

*<u>Line Intensity Calibration Accuracy</u>*

The following table provides the percentage flux error associated with each component of the error budget (from Roelfsema et al. 2012).

| *Error source*               | *Bands 1/2*  | *Bands 3/4*  | *Band 5* | *Bands 6/7* |
| Sideband ratio*            | 3-4               |    4-6            |      4-6    |     5-8          |
| Hot load coupling          | < 1               |    < 1        |      < 1   |     < 1       |
| Cold load coupling       | < 1               |    < 1        |      < 1   |     < 1       |
| Hot load temperature    | < 1               |    < 1        |      < 1   |     < 1       |
| Cold load temperature   | < 1               |    < 1        |      < 1   |     < 1       |
| Planetary model error | < 3               |    < 3        |      < 3   |     < 3       |
| Beam efficiency              | < 5               |    < 5        |      < 10   |     < 5       |
| Pointing                         | < 1               |    < 2        |      < 2   |     < 4       |
| Opt. standing waves      | 4               |    4            |      3    |     3          |

**Note that the "sideband gain ratio" convention used in the HIFI pipeline does not correspond to the ratio between the respective sideband gains (equal to 1 for a balanced system), but to the gain in one sideband normalised by the combined gain of the two sidebands. Consequently, perfectly-balanced sidebands will feature a sideband ratio of 0.5* (see also section 5.3.1 of the [[http://herschel.esac.esa.int/Docs/HIFI/html/hifi_om.html Observers' Manual]], and the [[%ATTACHURL%/calibframework.pdf Intensity Calibration Framework document]]).

*<u>Frequency Calibration Accuracy</u>*

| *Spectrometer*                  |  *Resolution Bandwidth (kHz)*  | *Accuracy (kHz)*  |
|HRS (high-res. mode, Hann appodisation) |  207 - 364                     | 20                |
|WBS                                     |  1100                          | 100               |

<u>Dominant Contributions</u>:
   * _HRS accuracy_: Master Oscillator accuracy (1 part in 10^8)
   * _HRS resolution_: LO signal frequency profile, as determined by the LO Source Unit mixers; band-dependent
   * _WBS accuracy_: COMB algorithm and interpolation interval
   * _WBS resolution_: optical alignment & thermal expansion effects within spectrometer

---+++HIFI beams

*<u>HIFI beam coupling efficiencies</u>*

A revision of the HIFI beam analysis has been performed, based on a multi-epoch set of measurements on Mars. As a result, a more accurate description of the HIFI beams has been obtained, leading to some change in the beam coupling efficiencies, typically a drop. The details of the analysis and the revised coupling efficiencies are given in this [[%ATTACHURL%/HifiBeamReleaseNote_Sep2014.pdf release note]]. For the record, the first release of the HIFI beam efficiencies was described in this [[http://herschel.esac.esa.int/Docs/TechnicalNotes/HIFI_Beam_Efficiencies_17Nov2010.pdf document]] (17 Nov 2010). %RED%Please note that these updates will only be featured in the HIFI calibration tree in HIPE 13%ENDCOLOR%.

*<u>HIFI beam maps and Encircled Energy Fraction</u>*

In addition to this, a detailed description of the HIFI beam models is provided as 2-D beam maps, 1-D azimuthal beam averages, and Encircled Energy Fractions (EEF). Each are provided at two frequencies per mixer band and polarisation, and can be extrapolated (within the LO frequencies of a given band) as a function of wavelength. The [[%ATTACHURL%/HifiBeamReleaseNote_Sep2014.pdf release notes]] provides further details about the delivered files and how to use them. In particular, some useful functions are provided in this [[%ATTACHURL%/getHifiBeam.py.txt Jython script]].

The beam material is concatenated as a single FITS file per mixer, frequency, and polarisation. Each file has three extensions tables: one for the 2-D beam model maps (2705x2705 pixels), one for the EEF, and one for the 1-D azimuthal beam averages. Note that 2-D beam model of smaller size (201x201 pixels) will also be made available in the calibration tree (not before HIPE 13), and used in the generation of Level 2.5 cubes by the pipeline. The following [[ftp://ftp.sciops.esa.int/pub/hsc-calibration/HIFI/BEAM_FILES/][page]] provides links to each individual file (27 Mb each). Plots of the 1-D azimuthal beam averages, and EEF for each band can be found in the [[%ATTACHURL%/HifiBeamReleaseNote_Sep2014.pdf Appendix B]] of the release note.

---+++Access to the calibration tree:
   * Information about what calibration is found in the HIFI !ObservationContext is found in the [[http://herschel.esac.esa.int/hcss-doc-13.0/index.jsp#hifi_um:hum_tour_cal Data Primer]] of the HIFI Data Reduction Guide.

   * Information about how to find what calibration version was used on your data, how to get the latest calibration, and reprocess observations with it is found in the [[http://herschel.esac.esa.int/hcss-doc-13.0/index.jsp#hifi_um:pipeline_cal Pipeline]] chapter of the HIFI Data reduction Guide.

   * The latest calibration tree pool can also be retrieved directly [[ftp://ftp.sciops.esa.int/pub/hsc-calibration/latest_cal_tree/][here]] (11 April 2015).

   *  Updates to the HIFI calibration data are generally concurrent with the release of each major version of the HCSS-HIFI software. However, it is possible to have updates to the calibration data in between major releases of the software as the software and data are independent of each other, in the table below the HIPE or the OD from which the calibration updates apply are given. The calibration versions available since HIPE 5 are listed in the table below: 


| *Calibration version number* | *Release date*  | *OD*  | *HIPE version* | *Changes* |
| IA_CAL_USER_ or HIFI_CAL_    | dd-mm-yy        |           |                        |                  |
| 2_0                                        | 29-11-10         |           | 5.0                  | Beam efficiency parameters introduced,  updates to spur table |
| 3_0                                        | 11-01-11         |           | 5.1                  | Beam efficiency parameters updated |
| 4_0                                        | 18-02-11         | 645    |                        | Spur table update |
| 5_0                                        | 13-04-11         |           | 6.1                  | Smoothing widths of OFF positions updated |
| 6_0                                        | 21-06-11         | 779    |                        | Sideband ratios in band 2a, addition of strong spur at 1108 GHz in band 5a to spur table |
| 7_0                                        | 12-12-11         |           | 8.0                  | Prevention of unnecessary Quality Flags by correction and addition of units in calibration products and correction of thresholds, addition of Quality Flags when LO multiplier currents are out of limits, improved saturated pixel flagging, addition to spur table, uplink product| 
| 8_0                                        | 03-02-12         | 995    | 8.1                  | Update of a priori table of IF saturations, used in order to flag bad LO data to be discarded in the deconvolution| 
| 9_0                                        | 24-07-12         |          | 9.0                | Sideband ratios in bands 5a and 5b, update of a number of engineering threshold for more accurate quality flagging related to hardware housekeeping, introduction of a list of a priori known corrupted data-frames for dedicated flagging in Level 0 data | 
| 10_0                                        | 08-11-12         |          | 9.1                | Introduction of a new !HifiUplink product to back-fill most the observational parameters as of !HSpot optimisation back into the Uplink product, update of the list of corrupted data-frames for flagging at Level 0, update of a number of engineering threshold for more accurate quality flagging related to hardware housekeeping  | 
| 11_0                                      | 22-01-13          |          | 10.0               | Update of the list of corrupted data-frames for flagging at Level 0, Quality flags created for SEUs (Singe Event Upsets leading to on-board software corruption) |
| 12_0                                      | 10-05-13          |          | 10.1               | Update of the list of corrupted data-frames for flagging at Level 0, updates to quality flag meta-data and uplink information for mapping modes |
| 13_0                                      | 30-05-13          |          | 11.0              | Update of the list of corrupted data-frames for flagging at Level 0, updates to spur table to account for band 3B purification |
| 14_0                                      | 04-07-13          |          | 11.0              | Update of the list of corrupted data-frames for flagging at Level 0 |
| 15_0                                      | 18-03-14          |          | 12.0              | Update of the list of corrupted data-frames for flagging at Level 0, ingestion of the Electrical Standing Wave correction reference table |
| 18_0                                      | 02-07-14          |          | 12.1              | Update of the list of corrupted data-frames for flagging at Level 0,  Absolute Pointing Error (APE) added to allow a quality flag to be raised in future releases in case of large pointing offsets, updates to allow IF-dependent corrections to sideband ratio, update to spur tables to store entry of spurs found by the ICC or a Key Programme, addition of an instrument uncertainty table (currently only placeholder) |
| 22_0                                      | 11-04-15          |          | 13.0              | Update of the beam parameters (coupling efficiencies, HPBW, 2-D beam maps - see also the beam section above), addition of a new table of expected spectral spurs to be masked by the pipeline in Spectral Scans, addition of a database of Electrical Standing Wave correction solution for observations in bands 6 and 7, addition of a new Error table that will be propagated into an error budget in HIPE 14 (currently only placeholder), introduction of IF-dependent sideband gain ratio (currently only true for the lower end of band 2a), update of the list of corrupted data-frames for flagging at Level 0|


   * The sideband ratio assumed in the pipeline as of the latest calibration tree pool are illustrated in this [[%ATTACHURL%/HIFI_SBR_Summary_V9.0.png plot]]. 

   * The HIFI beam release #1 (see section above) including updated aperture efficiencies, main-beam efficiencies, and HPBW is not yet reflected in the HIFI caltree.

---+++Outstanding calibration issues
   * An overview of the data product known issues can be found [[http://herschel.esac.esa.int/twiki/bin/view/Public/DpKnownIssues Data Known Issues Page]]

   * _Standing waves_:
      *  there are several standing waves that can affect the HIFI data at various processing levels.  Their nature and impact on the HIFI calibration are described in the Standing Wave technical note and, more briefly, in the Standing Wave removal chapter of the HIFI Data Reduction Guide
         * [[%ATTACHURL%/calibration_hifi_stdw_2011_v0_1.PDF Standing Wave technical note]] 
         * [[http://herschel.esac.esa.int/hcss-doc-13.0/index.jsp#hifi_um:sw_removal_hifi2 Standing Wave Removal]] 
      * There are various techniques currently offered to clean these baseline distortions:
         * *Remove sine waves, or combinations of sine waves*.  Check the bullet about usage of the *[[http://herschel.esac.esa.int/hcss-doc-13.0/index.jsp#hifi_um:sw_removal_hifi2 fitHifiFringe]]* task in the data reduction section above.
         * *Electrical Standing Wave (aka _Matching technique_)*: to correct for the IF standing wave present in bands 6 and 7 
            * A new task hebCorrection has been implemented in HIPE 12.0 and is being used by default by the pipeline in HIPE 13 (see also the Reducing HIFI data section above). %RED% *We warn that the task does not guarantee an accurate correction in all circumstances*. %ENDCOLOR%
            * The task is described in the following *[[http://herschel.esac.esa.int/hcss-doc-13.0/index.jsp#hifi_um:sw_removal_heb2 document]]*. The mathematical background can be found in this [[%ATTACHURL%/ESWCorrection.pdf proceeding]]. Unlike the script made available in previous version of HIPE, this task is applicable to any HIFI observing mode.
            * The task needs to work from Level 1 data, then reprocess the corrected data up to Level 2.5. Two example codes are provided here:
               * Example script for usage with stand-alone data-base:  [[%ATTACHURL%/hebCorrection_externalModels_example.py.txt hebCorrection_externalModels_example.py]] and the data-base as a FITS file: [[%ATTACHURL%/hebCorrectionModels_HICAL15.fits hebCorrectionModels_HICAL15.fits]]
               * Example script for usage with the latest calibration tree: [[%ATTACHURL%/hebCorrection_calTree_example.py.txt hebCorrection_calTree_example.py]]
            * A thorough description of the technique, and the underlying causes, is in Ronan Higgins' thesis:
                [[http://eprints.nuim.ie/2584/1/thesis_ronan_higgins_nuim.pdf _Advanced optical calibration of the Herschel HIFI heterodyne spectrometer_]]
         *  in case of *strong source continuum* some standing waves can be enhanced, as described in the Alternative Calibration Scheme report.  Such standing waves can be significantly reduced in amplitude using an alternative pipeline algorithm known as the _Modified Passband Technique_ , which is described in the Standing Wave Removal chapter of the HIFI Data Reduction Guide.  Alternatively, you can run the Level1PipelineAlgo_hc_filtered_v1.py script. <!--Note that in 9.0, a bug still affects the doFilterLoad task in such a way that it will ignore the user-input parameters, resulting in the task always applying the "cubic_spline" method with default parameters - i.e. the "fft" method will not be applied despite being set in the task inputs. This will be fixed in 9.1.-->
            * [[%ATTACHURL%/Alternative_Calibration_Schemes_v1.0.pdf Alternative Calibration Scheme Report]]    
            * [[http://herschel.esac.esa.int/hcss-doc-13.0/load/hifi_um/html/sw_removal_passb2.html Modified Passband Technique]].  
            * [[%ATTACHURL%/Level1PipelineAlgo_hc_filtered_v1.py.txt Level1PipelineAlgo_hc_filtered_v1.py  script]] 

   * _Baseline removal_: imperfect ON-OFF calibration scheme can result in residual baseline distortion (not necessarily standing waves). Check the bullet about usage of the *[[http://herschel.esac.esa.int/hcss-doc-13.0/index.jsp#hifi_um:hifium_bl1 fitBaseline]]* task in the data reduction section above.

   * _Spectral purity_: there are places over the HIFI frequency range where the Local Oscillator does not offer a single frequency tone, so that spurious spectral line may end up in the data, and the calibration of the targeted line gets affected. A dedicated cookbook on the matter is in preparation. In the meantime, we refer to section [[http://herschel.esac.esa.int/Docs/HIFI/html/ch05s04.html#N11370 5.4.6]] of the Observer's Manual, and to the release notes provided above. 

   * _Spurs_: Spectral Spurs are narrow and broad spectral features due to instabilities in the Local Oscillator. Up to HIPE 12, spurs have been assigned by the pipeline in an automatic fashion. In HIPE 13, this is no longer the case and the masks are assigned based on a priori tables provided by the ICC in the calibration tree. As of now, all spectral scans benefit from such tables, but only a handful of single point and spectral mapping observations. This limitation has little impact on the top level data though since no pipeline module makes use of these masks in those modes. A list of the spurs registered up to HIPE 12 can be found in section [[http://herschel.esac.esa.int/Docs/HIFI/html/ch05s04.html#N11370 5.4.6]] of the Observer's Manual. This table will be updated soon based on the new statistics derived from the HIPE 13 mask tables. 

---++ Interest Groups and Scripts

   * %RED%HIPI%ENDCOLOR%: a library of HIFI Plug-In's has been prepared by the NHSC - check the [[https://nhscsci.ipac.caltech.edu/sc/index.php/Hifi/HIPI][HIPI website]] for more information. %RED% In HIPE 13, most of those utilities have been ported to Useful scripts (Menu Scripts > HIFI Useful Scripts) %ENDCOLOR%
   * The following interest groups relate to processing of observations taken with HIFI. The links provided allow subscription to these interest groups.
      * [[http://www.sciops.esa.int/mailman/listinfo/herschel_dp_specmap/ subscribe]] to PACS, SPIRE and HIFI spectral maps interest group
      * [[http://www.sciops.esa.int/mailman/listinfo/herschel_dp_hifi/ subscribe]] to HIFI point source and spectral scan interest group
   * User scripts: Users are welcome to submit scripts they believe could be of general interest to the community to the Herschel [[http://herschel.esac.esa.int/esupport/ Helpdesk]].

---++ Further Information

   * More information -- particularly for US users -- can be found at the HIFI home page at the [[https://nhscsci.ipac.caltech.edu/sc/index.php/Hifi/HomePage NHSC NASA Herschel Science Center]]
   * [[http://www.sron.nl/index.php?option=com_content&task=view&id=1963&Itemid=1795 HIFI science pages]]
   * [[http://www.aanda.org/index.php?option=com_toc&url=/articles/aa/abs/2010/10/contents/contents.html A&A Herschel First Science Result issue (Vol. 518)]] and [[http://www.aanda.org/index.php?option=com_toc&url=/articles/aa/abs/2010/13/contents/contents.html A&A Herschel/HIFI Special issue (Vol. 521)]]
   * Further details about some of the standing waves at play in HIFI and the matching technique can be found in the [[%ATTACHURL%/thesis_ronan_higgins_nuim.pdf thesis of Ronan Higgins (18.5 Mb)]]

-- Main.DavidTeyssier , Main.CarolynMcCoey, Main.SylvieFBeaulieu and Main.RengelMiriam - 14 Apr 2015-
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
txt	hebCorrection_calTree_example.py.txt	r1	manage	2.4 K	2014-03-18 - 12:57	DavidTeyssier
txt	hebCorrection_externalModels_example.py.txt	r1	manage	2.4 K	2014-03-18 - 12:57	DavidTeyssier
txt	getHifiBeam.py.txt	r2 r1	manage	5.1 K	2014-12-01 - 15:06	DavidTeyssier
pdf	HIFISubbands.pdf	r1	manage	8.5 K	2014-07-23 - 18:52	CarolynMcCoey
txt	Level1PipelineAlgo_hc_filtered_v1.py.txt	r1	manage	10.6 K	2011-06-09 - 09:08	DavidTeyssier
txt	line_identification_guide.py.txt	r1	manage	12.0 K	2014-10-09 - 14:52	SylvieFBeaulieu	Script for the idenfifyLines task (HDRG, chapter 'The HIFI line identification tool')
png	1342180554_band4b.png	r1	manage	54.9 K	2015-04-07 - 21:23	DavidTeyssier
png	HIFI_SBR_Summary_V9.0.png	r1	manage	86.0 K	2012-08-23 - 11:18	DavidTeyssier
pdf	Sherry_et_al_AAS217_poster.pdf	r1	manage	101.6 K	2011-06-26 - 16:00	DavidTeyssier
fits	Orion_S_baselined_485_490.fits	r1	manage	168.8 K	2014-10-09 - 14:51	SylvieFBeaulieu	Spectrum1d (fits) for the idenfifyLines task (HDRG, chapter 'The HIFI line identification tool')
txt	16293_SIS.txt	r1	manage	171.7 K	2014-10-09 - 14:50	SylvieFBeaulieu	CASSIS lineList for the idenfifyLines task (HDRG, chapter 'The HIFI line identification tool')
pdf	HIFI_Calibration_ReleaseNotes.pdf	r1	manage	174.5 K	2013-01-31 - 12:56	DavidTeyssier
pdf	freq_framework.pdf	r1	manage	181.3 K	2011-06-08 - 08:18	DavidTeyssier
fits	hebCorrectionModels_HICAL15.fits	r1	manage	241.9 K	2014-03-18 - 12:57	DavidTeyssier
pdf	freq_vel_1.1.pdf	r1	manage	255.4 K	2011-06-26 - 17:55	DavidTeyssier
pdf	calibframework.pdf	r1	manage	303.5 K	2011-06-08 - 08:18	DavidTeyssier
PDF	freq_vel_3.PDF	r1	manage	305.2 K	2011-06-08 - 08:19	DavidTeyssier
pdf	HIFI_PointingAccuracyNote.pdf	r2 r1	manage	373.2 K	2015-04-10 - 08:14	RengelMiriam
pdf	aa15120-10.pdf	r1	manage	377.9 K	2011-12-01 - 17:21	DavidTeyssier
pdf	spatial_response_framework.pdf	r1	manage	581.8 K	2011-06-08 - 08:18	DavidTeyssier
pdf	spatial_response_framework_v1.9.pdf	r1	manage	589.0 K	2014-02-10 - 09:11	DavidTeyssier
pdf	ESWCorrection.pdf	r1	manage	640.3 K	2015-01-21 - 12:49	DavidTeyssier
pdf	aa14698-10.pdf	r1	manage	767.0 K	2011-01-21 - 09:38	AnthonyMarston	de Graauw et al 2010 -- HIFI instrument paper
pdf	HIFI_SBR_Cookbook_V1.0.pdf	r1	manage	1440.2 K	2013-06-27 - 13:16	SylvieFBeaulieu	HIFI_SBR_Cookbook_V1.0
pdf	Ossenkopf_2008_AllanVariance.pdf	r1	manage	1500.3 K	2011-10-04 - 09:27	DavidTeyssier
pdf	HifiBeamReleaseNote_Sep2014.pdf	r2 r1	manage	1513.2 K	2014-10-01 - 12:10	DavidTeyssier
pdf	Alternative_Calibration_Schemes_v1.0.pdf	r1	manage	2815.8 K	2011-12-14 - 08:57	DavidTeyssier
PDF	calibration_hifi_stdw_2011_v0_1.PDF	r1	manage	2897.1 K	2013-01-10 - 10:58	DavidTeyssier
pdf	calibration_hifi_stdw_2011_v0_1.pdf	r1	manage	4022.0 K	2011-08-10 - 20:47	CarolynMcCoey
pdf	HifiObservingModesPerformance_110926a.pdf	r1	manage	4324.4 K	2011-09-30 - 13:26	JeanMatagne
pdf	thesis_ronan_higgins_nuim.pdf	r1	manage	18922.3 K	2011-06-26 - 16:10	DavidTeyssier
Topic revision: r91 - 2015-05-07 - SylvieFBeaulieu
Public
Webs
Public
TWiki