Difference: HifiCalibrationWeb (56 vs. 57)

Revision 572013-07-29 - AnthonyMarston

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

HIFI instrument and calibration web pages

Changed:
<
<
NEW NEW NEW Latest updates -- 27 June 2013 NEW NEW NEW
  • HIPE 10.3 has been released !
>
>
NEW NEW NEW Latest updates -- 26 July 2013 NEW NEW NEW
  • HIPE 11.0 has been released !
 
  • HIPI (a library of Hifi Plug-In's) - see Interest Groups and Scripts section
  • New matching technique script is available in the Expert Version of HIPE - see the "Outstanding calibration issues" section
  • A large amount of calibration data has been released ! - see the "Observing with HIFI" section
Line: 25 to 26
 

Reducing HIFI data

Recommended User release

Changed:
<
<
>
>
 
Changed:
<
<

Which data would most benefit from being reprocessed in HIPE 10.3?

>
>

Which data would most benefit from being reprocessed in HIPE 11.0?

 
  • DBS Observations
    • A new pipeline step, mkDbsReference, calculates the differences the chop positions in all DBS observations and after applying the band-pass correction stores them in a product in calibration->pipeline-out called ReferenceSpectra. This allows you to check for contamination in chop positions for all types of DBS observations.
Line: 36 to 37
 
    • Rotated cubes are now produced in the Level 2.5 product by default for maps carried out with a non-zero position angle, a non-rotated cube is also produced.

  • Improved pointing reconstruction for observations taken between OD 320 and 761 If you have data processed with < HIPE 9
Changed:
<
<
    • The pointing information attached to the data for observations taken between OD 320 and OD761 did not use the most accurate representation of the star tracker focal length. This will be done for the bulk reprocessing with HIPE 9. As a consequence, some observations will experience a shift in astrometry, that can be as high as 8 arcsec. All the details about the consequences for a particular obsid, and recipes to reconstruct yourself the improved pointing can be found at http://herschel.esac.esa.int/twiki/bin/view/Public/HowToUseImprovedPointingProducts as well as in the level0 section of the Hifi Data Reduction Guide
>
>
    • The pointing information attached to the data for observations taken between OD 320 and OD761 did not use the most accurate representation of the star tracker focal length. This will be done for the bulk reprocessing with HIPE 9. As a consequence, some observations will experience a shift in astrometry, that can be as high as 8 arcsec. All the details about the consequences for a particular obsid, and recipes to reconstruct yourself the improved pointing can be found at http://herschel.esac.esa.int/twiki/bin/view/Public/HowToUseImprovedPointingProducts as well as in the level0 section of the Hifi Data Reduction Guide
 
  • Backfilling of observational parameters If you have data processed with < HIPE 9
    • From 9.1 onwards, most of the observational parameters that got optimised in HSpot will be propagated into the observation context, under a new product called HifiUplinkProduct (in the auxiliary product branch). Some of these parameters will be used in the pipeline to e.g. estimate the dimension of the cubes in the mapping and have them more representative of how the map was really obtained.

  • Solar System Object maps If you have data processed with < HIPE 9
Changed:
<
<
    • Cubes for moving targets are now created in the comoving frames in HIPE 9. In order to benefit from that, you should re-pipeline from level 0 up to level 2.5. Note the hifiPipeline task should be called with the option "Aux=True". Please check the Pipeline chapter of the HIFI Data Reduction Guide for more details about how to do that.
>
>
    • Cubes for moving targets are now created in the comoving frames in HIPE 9. In order to benefit from that, you should re-pipeline from level 0 up to level 2.5. Note the hifiPipeline task should be called with the option "Aux=True". Please check the Pipeline chapter of the HIFI Data Reduction Guide for more details about how to do that.

  • Solar System Object ephemerides If you have data processed with < HIPE 11
    • A bug (see here) was fixed in the calculation of SSO ephemeris positions (ra_centre/dec_centre) that led to offsets of up to ~10". This bug affected users that use cubes in a co-moving frame (including the standard level 2.5 cubes), use the doOffset task, or make explicit use of ra_centre / dec_centre in their HIPE scripts. The offset is practically constant over a map and could, e.g., make the emission of a comet appear to be off-center when it really isn't. The fixed bug is in the level 0 pipeline, which users cannot run easily. The issue will be fixed by the HIPE 11 bulk reprocessing. In the meantime, you can use a HIPE snippet such as the one attached here.
 
  • Strong continuum sources
    • An optional step in the pipeline can be used to remove standing waves arising in the loads, this is particularly effective for strong continuum sources. A report describing the technique can be found here. Instructions for using this modified passband technique are available in the Standing Wave Removal chapter of the HIFI Data Reduction Guide:
Changed:
<
<
>
>
 
Changed:
<
<
>
>
 
Changed:
<
<
>
>
 

Documentation and Cookbooks

  • The HIFI Launch Pad is intended to help you quickly off the ground with HIFI data reduction
Changed:
<
<
>
>
 
  • There is a cookbook for point mode DBS observations, other observing mode cookbooks are in progress
Changed:
<
<
>
>
 
  • The HIFI Data Reduction Guide and the HIFI AOT Observing Mode Release and Performance Notes contain information you need to know about HIFI data, regardless of the software you use for data reduction
Changed:
<
<
>
>
 

  • If you want to know more about the details of the HIFI pipeline, you should look at the HIFI Pipeline Specification Document
Changed:
<
<
>
>
 
Changed:
<
<
  • If you are working in HIPE, the HIFI Data Reduction Guide will be your prime resource for all things HIFI but you should also look at the Herschel Data Analysis Guide for information about general tools, such as those for viewing and manipulating spectra and spectral cubes, and for obtaining data from the archive.
>
>
  • If you are working in HIPE, the HIFI Data Reduction Guide will be your prime resource for all things HIFI but you should also look at the Herschel Data Analysis Guide for information about general tools, such as those for viewing and manipulating spectra and spectral cubes, and for obtaining data from the archive.
 
  • The full set of online documentation for the current user release also contains detailed information about the HIFI pipeline, information about scripting in HIPE, and reference manuals for command listings. New users of HIPE are recommended to read through the Quick Start Guide and the HIPE Owner's Guide
    • You can create pdfs of each document by clicking on the pdf symbol at the top of the main page.
Changed:
<
<
>
>
 
  • The full set of Herschel documentation for the developer track is also available. This documentation contains revisions and updates to the documentation associated with the current User Release but you should also be aware that it may describe functionality not available in the current User Release
Changed:
<
<
>
>
 
Line: 109 to 113
 

Typical Data Reduction Workflow

  • Download data from the HSA. If you are downloading several observations it is better to use the tar ball.
Changed:
<
<
>
>
 
  • Inspect data quality. Both point spectra and spectral cubes can be viewed in the SpectrumExplorer. In addition to inspecting the quality of your level 2 data, you should always check the level 1 data for any bad scans. If data at level 2 appears strange then looking at level 1 data may provide insight to the problem, see the Data Primer chapter of the HIFI Data Reduction Guide for more information about levels of data
Changed:
<
<
>
>
 
  • Assess whether data should be re-pipelined.
    • Look at the section above and also at the What's New in the latest User Release to decide if you should re-pipeline.
    • Note that the ICC recommends that data processing should not be allowed to fall more than one version behind the HIPE version being used.
    • Everything you need to re-pipeline data is available in the ObservationContext, and the instructions for using the pipeline, including tips on re-pipelining using new calibration data and customising the pipeline, can be found in the Pipeline chapter of the Hifi Data Reduction Guide
Changed:
<
<
>
>
 
  • Flag data. A new user-friendly task has been developed in order to easily flag your data: FlagTool. It can be used both as a GUI (similar concept as for the FitBaseline task) or in command line
Changed:
<
<
>
>
 
  • Remove standing waves. Standing waves are a common problem in HIFI data, particularly in bands 3, 4, 6, and 7. The HIFI Data Reduction Guide provides some information about typical standing waves in HIFI data and describes how to remove them in HIPE using fitHifiFringe and the modified band pass technique
Changed:
<
<
>
>
 
  • Correct baseline drift. Baselines can be flattened by subtracted, or division in the case of real continuum. This can be done in HIPE with the fitBaseline task
Changed:
<
<
>
>
 
  • Fold frequency switch data. Frequency switch data is not folded in the pipeline and can be done with the doFold task.
Changed:
<
<
>
>
 
  • Average together H and V polarisation.
    • The noise estimates given by HSpot assume that the H and V polarisations are averaged together. This can be done in HIPE with the PolarPair task, which resamples the spectra to the same frequency scale (they are frequency calibrated with different comb measurements) and then averages them. Alternatively, you can use the Accumulate task, which allows you to specify the resampling width if you desire.
Changed:
<
<
>
>
 
    • Note that differences may be seen in H and V profiles, see the note on H and V positions. If you are particularly interested in the spatial structure of your source you may prefer not to average the H and V polarisations together.

  • Spectral manipulation. The Spectral Toolbox in HIPE allows you to perform typical spectral arithmetic tasks on spectra and spectral cubes
Changed:
<
<
>
>
 
  • Fit lines. The SpectrumFitterGUI, which is also a part of the Spectral Toolbox, can be used to fit line profiles.
Changed:
<
<
>
>
 
  • Save modified data You can save data to a pool, or store your session in order to continue working in HIPE.
Changed:
<
<
>
>
 
  • You may wish to export data as FITS, ASCII or use the HiClass task to export data as CLASS readable FITs files.
Changed:
<
<
>
>
 
      • Warning, important WARNING! When converting to velocity, CLASS corrects the reference frequency (the frequency at the reference channel) for the Doppler shift, but not the channel width so there is a very small mismatch between the channel width before and after Doppler correction. Lines are shifted from the correct velocity as a consequence, with an increasing shift away from the reference channel.. The ICC recommends that line frequencies are checked in HIPE before exporting to CLASS.

Dedicated data reduction tips for Spectral Scan data

  • The data needs to be deconvolved to a single sideband solution. This can be done with the doDeconvolution task, which works with an ObservationContext.
Changed:
<
<
>
>
 
  • It is important to clean the data before deconvolving it, this means removing standing waves, correcting baselines and flagging out any spurs or bad data that the pipeline missed.
Line: 174 to 178
 
  • It is strongly recommended that you inspect your level 2 HTPs for baseline drift and residual standing waves and decide if some, or all, datasets need to be cleaned up before re-running the gridding task.
Changed:
<
<
  • Regridding in HIPE > 9 will automatically use the comoving frame for SSOs. See the section above about the benefits from reprocessing with HIPE 10 for more details on how to do that
>
>
  • Regridding in HIPE > 9 will automatically use the comoving frame for SSOs. See the section above about the benefits from reprocessing with HIPE 11 for more details on how to do that
 
  • You can then grid your cleaned data into a spectral cube using the doGridding task, you can also use this task to customise the cube creation to best suit your science goals.
Changed:
<
<
>
>
 

HIFI performance and calibration

Line: 234 to 238
 
  • Information about how to find what calibration version was used on your data, how to get the latest calibration and reprocess observations with it is found in the Pipeline chapter of the HIFI Data reduction Guide
Changed:
<
<
  • the latest calibration tree pool can be also retrieved directly here (16 May 2013)
>
>
  • the latest calibration tree pool can be also retrieved directly here (4 Jul 2013)
 
  • Updates to the HIFI calibration data are generally concurrent with the release of each major version of the HCSS-HIFI software. However, it is possible to have updates to the calibration data in between major releases of the software as the software and data are independent of each other, in the table below the HIPE or the OD from which the calibration updates apply are given. The calibration versions available since HIPE 5 are listed in the table below
Line: 252 to 256
 
10_0 08-11-12   9.1 Introduction of a new HifiUplink product to back-fill most the observational parameters as of HSpot optimisation back into the Uplink product, update of the list of corrupted data-frames for flagging at Level 0, update of a number of engineering threshold for more accurate quality flagging related to hardware housekeeping
11_0 22-01-13   10.0 Update of the list of corrupted data-frames for flagging at Level 0, Quality flags created for SEUs (Singe Event Upsets leading to on-board software corruption)
12_0 10-05-13   10.1 Update of the list of corrupted data-frames for flagging at Level 0, updates to quality flag meta-data and uplink information for mapping modes
Added:
>
>
13_0 30-05-13   11.0 Update of the list of corrupted data-frames for flagging at Level 0, updates to spur table to account for band 3B purification
14_0 04-07-13   11.0 Update of the list of corrupted data-frames for flagging at Level 0
 
  • The side-band ratio assumed in the pipeline as of the latest calibration tree pool are illustrated in this plot.
Line: 263 to 269
 
  • Standing waves:
    • there are several standing waves that can affect the HIFI data at various processing levels. Their nature and impact on the HIFI calibration are described in the Standing Wave technical note and, more briefly, in the Standing Wave removal chapter of the HIFI Data Reduction Guide
Changed:
<
<
>
>
 
    • There are various techniques currently offered to clean these baseline distortions:
Changed:
<
<
      • Remove sine waves, or combinations of sine waves. Check the bullet about usage of the fitHifiFringe task in the data reduction section above.
>
>
      • Remove sine waves, or combinations of sine waves. Check the bullet about usage of the fitHifiFringe task in the data reduction section above.
 
      • Matching technique: for the strong IF standing waves seen in bands 6 and 7.
        • A script is available in the source directory of the 'expert' install of HIPE: scripts/hifi/scripts/users/engineering/HEBStWvCatalogCorrection.py It is extensively commented. Alternatively, the latest version can be found here
Changed:
<
<
        • At the HIPE Forum 2012, a demo of a pre-release version of the script provided in HIPE 10.0 was given. The presentation, that version of the script, a prepared input catalog for use by the script, and some result plots can be found on the twiki page:
>
>
        • At the HIPE Forum 2012, a demo of a pre-release version of the script provided in HIPE 11.0 was given. The presentation, that version of the script, a prepared input catalog for use by the script, and some result plots can be found on the twiki page:
 
Line: 279 to 285
 
        • for more information please contact Ian Avruch (I.Avruch@sron.nl) at the HIFI ICC. You can also check the progress on this effort in the following internal ICC pages.
      • in case of strong source continuum some standing waves can be enhanced, as described in the Alternative Calibration Scheme report. Such standing waves can be significantly reduced in amplitude using an alternative pipeline algorithm known as the Modified Passband Technique , which is described in the Standing Wave Removal chapter of the HIFI Data Reduction Guide. Alternatively, you can run the Level1PipelineAlgo_hc_filtered_v1.py script.
        <--Note that in 9.0, a bug still affects the doFilterLoad task in such a way that it will ignore the user-input parameters, resulting in the task always applying the "cubic_spline" method with default parameters - i.e. the "fft" method will not be applied despite being set in the task inputs. This will be fixed in 9.1.-->
Changed:
<
<
>
>
 
Changed:
<
<
  • Baseline removal: imperfect ON-OFF calibration scheme can result in residual baseline distortion (not necessarily standing waves). Check the bullet about usage of the fitBaseline task in the data reduction section above.
>
>
  • Baseline removal: imperfect ON-OFF calibration scheme can result in residual baseline distortion (not necessarily standing waves). Check the bullet about usage of the fitBaseline task in the data reduction section above.
 
  • Spectral purity: there are places over the HIFI frequency range where the Local Oscillator does not offer a single frequency tone, so that spurious spectral line may end up in the data, and the calibration of the targeted line gets affected. A dedicated cookbook on the matter is in preparation. In the meantime, we refer to section 5.4.6 of the Observer's Manual, and to the release notes provided above.
Changed:
<
<
  • Spurs: Spurs are reported in HSpot when using the frequency editor and general information is also provided in the HIFI Observer's Manual (see above link). Spurs are also being checked in the pipeline and any detected feature is reported in the TrendAnalysis product of an observation context. The list of currently known spurs can be found in section 5.4.6 of the Observer's Manual. However not all possible spurs can be predicted or caught by the data processing, so that Users are invited to inform the Helpdesk about any features they found in their data and may not have been properly flagged by the system.
>
>
  • Spurs: Spurs are reported in HSpot when using the frequency editor and general information is also provided in the HIFI Observer's Manual (see above link). Spurs are also being checked in the pipeline and any detected feature is reported in the TrendAnalysis product of an observation context. The list of currently known spurs can be found in section 5.4.6 of the Observer's Manual. However not all possible spurs can be predicted or caught by the data processing, so that Users are invited to inform the Helpdesk about any features they found in their data and may not have been properly flagged by the system.
 

Interest Groups and Scripts

Line: 303 to 309
 
Changed:
<
<
-- DavidTeyssier , CarolynMcCoey and SylvieFBeaulieu - 27 June 2013
>
>
-- DavidTeyssier , CarolynMcCoey and SylvieFBeaulieu - 26 July 2013
 
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1307610505" name="Level1PipelineAlgo_hc_filtered_v1.py.txt" path="Level1PipelineAlgo_hc_filtered_v1.py.txt" size="10864" user="Main.DavidTeyssier" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1307521151" name="freq_vel_3.PDF" path="freq_vel_3.PDF" size="312508" user="Main.DavidTeyssier" version="1"
 
This site is powered by the TWiki collaboration platform Powered by Perl