Difference: SpireCalibrationWeb (61 vs. 62)

Revision 622012-07-30 - IvanV

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

SPIRE instrument and calibration web pages

Added:
>
>
Back to main Herschel Public TWiki page
 

Introduction

This page provides up-to-date information about using the SPIRE instrument: from preparing observations to reducing your data. This page also provides you with the latest calibration accuracies and known SPIRE calibration issues.

Observing with SPIRE

Changed:
<
<
Warning, important The most up to date information on instrument calibration and performance is given in the SPIRE Observers' Manual. This is the reference document used by all the rest of the SPIRE user guides (eg data reduction guide, cookbooks etc). Sometimes it may happen that outdated values are quoted in some of the documents. In such a case use the values given in the SPIRE Observers' Manual.
>
>
The most up to date information on instrument calibration and performance is given in the SPIRE Observers' Manual. This is the reference document used by all the rest of the SPIRE user guides (eg data reduction guide, cookbooks etc). Sometimes it may happen that outdated values are quoted in some of the documents. In such a case use the values given in the SPIRE Observers' Manual.
 
Changed:
<
<
  • The SPIRE Observer's Manual (HTML) and PDF (15.5 MB).
    Info We recommend using the PDF version (search and hyperlinks work) because the HTML version has some formatting problems due to the conversion from LaTeX to HTML.
  • A quick guide to the SPIRE instrument: SPIRE fact sheet
  • Summary papers from the A&A Special Issue: Warning, important Some values on the performance are now outdated. Please consult the SPIRE Observers' Manual for most up to date information.
>
>
 
Changed:
<
<
>
>
 

AOT release notes

Deleted:
<
<

Photometer

Spectrometer

 
Changed:
<
<

>
>
Photometer  
Photometer point source mode SPIRE photometer point source release note (30 April 2010)
Photometer scan map mode SPIRE scan map release note (20 October 2009), ADDENDUM to scan map mode (2 June 2010)
Photometer small scan map mode SPIRE small scan map mode (17 March 2010)

Spectrometer  
FTS point source mode (sparse) Spectrometer point source mode release note (21 May 2010)
FTS mapping mode (intermediate, full) Spectrometer mapping mode release note (30 April 2010))
FTS bright source modes Spectrometer bright source release note (7 September 2010)
 

Reducing SPIRE data

Deleted:
<
<
Info In order to obtain the best possible Level 2 SPIRE photometry or spectroscopy data, the observations might have to be reprocessed with the latest HIPE User Release.
 

Software and documentation

Changed:
<
<
  • HIPE (Herschel Interactive Processing Environment): The latest User Release HCSS version that you should use for reducing SPIRE data is HIPE v9.0.0. It can be downloaded from: http://herschel.esac.esa.int/HIPE_download.shtml. FYI: this corresponds to the so-called CIB (continuous integration build) HIPE 9.0 build 2974.
>
>
  • HIPE (Herschel Interactive Processing Environment): The latest User Release HCSS version that you should use for reducing SPIRE data is HIPE v9.0.0. It can be downloaded from: http://herschel.esac.esa.int/HIPE_download.shtml. This corresponds to the developer's CIB (Continuous Integration Build) HIPE 9.0 build 2974.

  • We also provide access to the latest stable developer build (latest stable CIB).
    • Beware These developer builds do not undergo the same in-depth testing as the user releases do. The latest developer build can be found here.
 
Changed:
<
<
  • We also provide access to the latest stable developer build (a.k.a latest stable CIB), used by the instrument experts at the ICC.
    • Beware _These developer builds do not undergo the same in-depth testing as the user releases do. The current latest stable developer build can be found here.
>
>
 
Changed:
<
<
  • Within HIPE you can access all the SPIRE data reduction and HIPE-use documentation. For those who wish to read the SPIRE Data Reduction Guide (SDRG) in PDF form, we provide that here: SDRG version 2.1. This version can be used with HIPE v9.0.0 as well as all track 9 of the CIBs. (Note that within the PDF version, document links will not work.) The SDRG follows the pipeline scripts (see "Cookbooks" below) and also explains what you are doing as you pipeline process.
>
>
  • SPIA: The SPIRE Photometer Interactive Analysis (SPIA) package is available as a plug-in for HIPE. SPIA provides a structured GUI based access to the more intricate parts of the scan map photometer pipeline for SPIRE without the immediate need to resort to scripts. More information can be found in the SDRG or on the SPIA web page
 
Deleted:
<
<
 
Deleted:
<
<
  • SPIA: The SPIRE Photometer Interactive Analysis (SPIA) is available as a plug-in for HIPE. SPIA provides a structured GUI based access to the more intricate parts of the scan map photometer pipeline for SPIRE without the immediate need to resort to scripts. More information can be found on the SPIA web page
 

The SPIRE Launch Pads

  • The SPIRE Launch Pads are single sheet quick entries (like a cheat sheet) into SPIRE data reduction and providing quick references to the relevant sections in the SPIRE Data Reduction Guide. There are launch pads for Data Access, SPIRE Photometer and Spectrometer data reduction.
Changed:
<
<
DocumentSorted ascending Description Download Version
SPIRE Launch Pad Data Launch Pad PDF version 1.5
SPIRE Launch Pad Photometer Launch Pad PDF version 1.6
SPIRE Launch Pad Spectrometer Launch Pad PDF version 1.6
>
>
Document Version
SPIRE Data Launch Pad version 1.5
SPIRE Photometer Launch Pad version 1.6
SPIRE Spectrometer Launch Pad version 1.6
 

Photometer data reduction

Changed:
<
<
Maps
>
>
The best source of information for reducing SPIRE Photometer data is the SPIRE Data Reduction Guide available through the HIPE help. This runs through the User Pipeline scripts step by step, describes several other Useful Scripts, and offers advice for specific issues that might be encountered.
 
Deleted:
<
<
  • Note that SPIRE maps are in units of Jy/beam, and are calibrated in the assumption of a point source having a spectral index equal to -1, i.e. νSν = const. To calibrate your data for other cases or convert to e.g. Jy/sr, please refer to section 5.2 of the SPIRE Observers' Manual .
  • By default, the SPIRE pipeline uses a näive map-maker. In this case, the error map is simply the standard deviation of all the data points falling into a given pixel. As a consequence, error maps contain increased errors associated with binning data from Gaussian sources, producing a torus shape; this is an artefact of the map-making process.
 
Changed:
<
<
Level 2.5
>
>
  • Note that SPIRE maps are in units of Jy/beam, and are calibrated in the assumption of a point source having a spectral index equal to -1, i.e. νSν = const. To calibrate your data for other cases or convert to e.g. Jy/sr, please refer to section 5.2 of the SPIRE Observers' Manual .
 
Deleted:
<
<
  • As of HIPE 6.1.1, SPIRE observations may include a new Level 2.5. This product includes maps obtained merging all contiguous observations belonging to the same program and having same observing mode (i.e. small map, large map or Parallel Mode). Maps are produced using the standard pipeline, i.e.:
    1. query the database to retrieve all the required observations
    2. merge all Level 1s
    3. remove the baseline using a median fit from each scan
    4. build the maps using the näive map-maker
  • All the photometer known issues applies to these maps as well. Moreover, note that:
    • no astrometry fix is applied, so sources may be blurred/_duplicated_ if the shift between 2 or more observations is big (>5 arcsec);
    • in merging together multiple observations of the same field, you may not notice anymore some artifacts such as undetected glitches, temperature drifts or detectors jumps. In both cases, you need to re-reduce the data with the tips suggested below.
  • The list of observations used to build the Level 2.5 maps are included in the observations' metadata.
 
Changed:
<
<
Solar System objects
>
>
The main issues that you might find in your data are: undetected glitches, thermistor or detector jumps, and bad baseline removal.
 
Deleted:
<
<
  • When the target is a Solar System Object (SSO) having a proper motion, the spacecraft re-adjusts its position after each scan, in order to always be centred on the target. However, the products retrieved from the Herschel Science Archive have been reduced using a standard pipeline which does not correct for the target's proper motion. As a result, the background of SSO observations will be focused while the SSO itself will appear blurred (for short observations or slow objects) or as a streak (for longer ones or faster objects).
  • As of HIPE 8.0, a new script named SSO_MotionCorrection is available under the SPIRE Useful Scripts menu. The script computes the SSO speed in RA & Dec coordinates, applies the required shift to Level 1 timelines, computes the corrected maps and eventually saves the modified products to a local pool. The results are maps centred on the SSO (i.e. the target will appear focused) with a smooth/blurred background. More details can be found in the SPIRE Data Reduction Guide.

Data processing known issues
 
  • Stripes in PSW, PMW and/or PLW (Level 2) maps
Changed:
<
<
    • All SPIRE photometry pipelines now use by default the destriper, which improves the issue of striping in level 2 maps. Hence observers should expect potential improvements in that respect with version 9.

<--   * Stripes in PSW, PMW and/or PLW (Level 2) maps 
    • Most of the stripes that are present in the final maps are due to a combination of thermal drifts (which in few cases are not efficiently removed) and median baseline subtraction. A similar effect is caused by very bright sources: in this case, the problem resides in the median baseline subtraction only.
    • Suggested solutions:
      • switch to a baseline subtraction using a polynomial fitting using the optional task baselineRemovalPolynomial. If there are no jumps in the timelines, you may also try to run the baseline removal on the entire timeline;
      • in the case of bright sources, you may try to mask them before running the baseline removal (either median or polynomial): you can use this script as a template.;
      • use the SPIRE Destriper: this new task is giving good results in most cases, especially for diffuse emission and extended sources. In the case of Parallel Mode observations, although the destroyer will work on single scans it is always better to merge 2 or more observations together using the map merge script within HIPE. The destriper documentation can be found on the NHSC website -->
>
>
    • All SPIRE photometry pipelines now use the destriper by default, which improves the issue of stripes in Level 2 maps. There should be noticeable improvements in that respect with HIPE version 9. The destriper documentation can be found on the NHSC website
 
  • De-glitcher masks faint sources
Changed:
<
<
    • The de-glitcher is a very delicate process. In particular, for data taken in Parallel Mode (sampling at 10Hz) and at high speed (60"/s) the de-glitcher with standard parameters may flag very faint sources as glitches. Bright sources are different from glitches in that they have a Gaussian (i.e. beam/PSF) shape. For faint sources, the sampling rate could be not high enough and hence they have a "delta" shape, which is similar to a small glitch. The user might try to modify the correlation parameter to 0.95: this will decrease the number of detected glitches. In case of high scan rate and low sampling speed one may want to stay with a limited level 1 detection rate and defer to Level 2 deglitching.
  • Some sources have saturated the ADC and the corresponding data have been masked
    • There is nothing a user can do: the source was simply too bright. If the user has other sources still not observed and of the same intensity, it is suggested to change the AORs to use the bright source mode.

  • Thermistor jumps
    • As of HIPE 6.0.3, a new module together called signalJumpDetector in place to identify the jump and to exclude the affected thermistor(s).
    • This module should not be run with observations in bright mode as it can lead to too many unnecessarily excluded scans. Jumps seem not to occur in bright mode.

  • Cooler temperature variations
    • After the end of the SPIRE cooler recycle, the temperature is few mK below the plateau (i.e. the most stable value which lasts for about 40h): it takes about 7h to reach it. Between 6 to 7h after the cooler recycle ends, its temperature raises steeply and reaches the plateau. At present, the pipeline is not able to cope with these strong temperature variations (although a correction script is planned for HIPE v.9), hence observations taken during such times may exhibit stripes in the final maps (especially for extragalactic fields). To solve this, the user can try a baseline polynomial fit of order >2 on the entire baseline - or use the destriper.

  • NaNs pixels present in the PSW, PMW and/or PLW (Level 2) maps
    • This effect, related to data masked for various reasons and poor coverage (not enough redundancy), is more evident in single fast-scan Parallel Mode maps. To avoid NaNs, increase the pixel's dimension (i.e., decrease the map's resolution).
    • This effect can also happen with destriped maps. In this case check if increasing the sigma or switching off the Level 2 deglitcher helps.
<-- Especially the HIPE 8 destriper should not be currently used with the Level 2 deglitcher active. -->

<--   * WCS in 3-colour images 
    • In all observation reduced with HIPE 8, the task createRgbImage puts wrong WCS in the output. Instead of using the WCS provided by the WCS input parameter, this task uses the WCS of one of the input images. This has been fixed in HIPE 9
-->

  • Quality flags in the quality context
    • Currently, the quality flags at the quality context inside the observation context are just meant for HSC/ICC internal evaluation of the quality of the products and not for the users. In case the data had some serious quality problem, the PI of the program has been contacted about it. Otherwise, only information in the quality summary, when available, should concern the observers.

Tips to re-reduce your data

  • Always remember to update to the latest calibration tree compatible with the HIPE built you are using (See the SPIRE Data Reduction Guide, Chapter 3 for a detailed explanation and examples). Assuming the observation is loaded into HIPE as a variable named obs:

cal = spireCal(calTree="spire_cal")
obs.calibration.update(cal)

  • If the observation you retrieved from HSA has been reduced with SPG v. 2.x or less, than start reprocessing from level 0 (i.e., run again the engineering conversion level 0 -> 0.5). In addition, if you want to apply the extended gains correction then reprocessing of the data through the User Pipeline is required for all photometer data processed with HIPE versions earlier than v8.

  • Main issues you might find in your data are: undetected glitches, thermistor or detector jumps, bad baseline removal.
    • Undetected glitches: you may try to play with the parameters of the waveletDeglitcher, in particular changing correlationThreshold parameter; other solution is to use the alternative sigmaKappaDeglitcher
<--      * Thermistor jumps: this should be automatically solved re-reducing your observation as of HIPE v.6. If this is not the case, you must exclude the affected thermistor when running the temperatureDriftCorrection adding e.g. pswThermistorSelect='T1' 
    • Failure of Temperature Drift Correction: Due to an update of the Temperature Drift Correction task in the pipeline, the pipeline may fail with an Index argument 0 is out of range error if run with Calibration Tree spire_cal_6_0. Please update to at least use spire_cal_6_1 to solve the problem (See the SPIRE Data Reduction Guide, Chapter 3). -->
  • Bad baseline removal (see also above) as of Hipe v. 6.x, a new polynomial fit (in comparison to the standard median) for baseline removal has been added as a prototype. Assuming that your Observation Context is stored in a variable named obs, you can call it as e.g.:
  • from herschel.spire.ia.pipeline.phot.baseline import BaselineRemovalPolynomialTask
    baselineRemovalPolynomial = BaselineRemovalPolynomialTask()
    
    scansBaseline = baselineRemovalPolynomial(input=obs.level1, polyDegree=3)
    mapBaseline   = naiveScanMapper(input=scansBaseline, array="PLW")
    
    >
    >
      • For data taken in Parallel Mode in particular (sampling at 10Hz, at high speed 60"/s), the de-glitcher may flag very faint sources as glitches when it is run with standard parameters. Faint sources may have a "delta function" shape due to the low sampling rate, which looks similar to a small glitch. Try modifying the "correlation parameter" to 0.95: this will decrease the number of detected glitches - it may be better to have a limited detection rate in first level deglitching and defer to Level 2 deglitching.
     
    Changed:
    <
    <
     

    Spectrometer data reduction

    Line: 196 to 106
     
    • Faint (<10 Jy) and medium (<100 Jy) strength sources
    • Bright sources (>500 Jy)
    • Extended sources
    Added:
    >
    >
    • Observations with few repetitions
     
    • H+L observations

    For faint sources, the subtraction of instrument, telescope and background emission is particularly important. Optimum subtraction can be performed in several ways (read the SPIRE Data Reduction Guide for details):

    1. Subtract the Dark Sky spectrum closest to your observation (use the "Background Subtraction" script in HIPE)
    2. Subtract the spectrum of surrounding detectors (use the "Background Subtraction" script in HIPE)
    Deleted:
    <
    <
    <--  3. Substitute the standard teleRsrf calibration file for one derived specifically for the OD of your observation (in the User Pipeline Script in HIPE) -->
      Dark Sky observations are observed on every SPIRE Spectrometer OD, and are all public in the Archive.
    Changed:
    <
    <
    A listing of the available Dark Sky observations can be found here.

    Info SPIRE FTS known data processing issues are given here.

    <-- In order to obtain the teleRsrf calibration file derived for the OD of your observation and valid for your HIPE/calibration tree version, please raise a Helpdesk ticket (select the "SPIRE FTS" department) specifying the observation that you are trying to process. -->

    >
    >
    Spectral cubes are produced by the Spectrometer pipeline for mapping observations. Some tips, suggestions and examples of spectral cube analysis for SPIRE data are provided here.
     

    Cookbooks

    Changed:
    <
    <

    SPIRE photometry cookbook.

    >
    >
    Cookbooks are provided inside the SPIRE Data Reduction Guide (see above).
     
    Changed:
    <
    <
    The current version of the cookbook is available here and provides practical guidelines on how to do photometry with SPIRE. HELP The cookbook is not HIPE specific.
    >
    >
    The standalone "Photometry Cookbook", is no longer maintained - it is being incorporated into the SPIRE DRG - please see the SDRG for photometry cookbook information, and raise a Helpdesk ticket if you find something missing.
     

    SPIRE calibration file versions

    Changed:
    <
    <
    Calibration files for SPIRE can be obtained here:

    The available calibration trees for SPIRE are listed below (with the current operational version at the top). More details of the changes in each version are given here.

    >
    >
    The available calibration trees for SPIRE are listed below (with the current operational version at the top).
     
    SPIRE Calibration Tree Applicable HIPE Version Comment
    Line: 255 to 151
     
    SPIRE_CAL_1_2 HIPE v1 Phot flux conv based on Ceres
    (SPIRE_CAL_1_1) HIPE v1 Pre-launch dummy values
    Added:
    >
    >
    More details of the changes in each version are given here.
     Any of the calibration trees can be retrieved in HIPE from the HSA using (e.g.)
    Changed:
    <
    <
    cal = spireCal(calTree="spire_cal_9_1") etc. The default (applicable to the HIPE version) can be obtained with cal = spireCal(calTree="spire_cal")
    >
    >
    cal = spireCal(calTree="spire_cal_9_1") etc. The default (applicable to the Hipe version) can be obtained with cal = spireCal(calTree="spire_cal") The latest calibration context for SPIRE can be obtained as a .zip file from Latest calibration trees. The zipped file is a pool, so it should be unpacked in the local store folder and then loaded in HIPE with cal = spireCal(pool="spire_cal_9_1").
     
    Changed:
    <
    <
    See the SPIRE Data Reduction Guide for more details.
    >
    >
    See the SPIRE Data Reduction Guide for more details.
     

    SPIRE calibration and performance

    Photometer calibration

    Changed:
    <
    <
    • SPIRE Photometer Beams: These are available in the SPIRE calibration context, at the standard map pixel size of (6,10,14) arcsec/pixel for (250,350,500) µm bands, and can be accessed in HIPE after a calibration context has been loaded (see above):

    BeamPSW = cal.phot.getBeamProf('PSW')
    BeamPMW = cal.phot.getBeamProf('PMW')
    BeamPLW = cal.phot.getBeamProf('PLW')
    
    >
    >
    • SPIRE Photometer Beams:
      • These are available in the SPIRE calibration context, at the standard map pixel size of (6,10,14) arcsec/pixel for (250,350,500) µm bands, and can be accessed in HIPE after a calibration context has been loaded (see above).
      • Tip, idea The observed beams at much finer scale of 1 arcsec/pixel, as well as the theoretical ones, are available from here . Please read the release note for more details.
     
    Changed:
    <
    <
    Tip, idea The observed beams at much finer scale of 1 arcsec/pixel, as well as the theoretical ones, are available from here . Please read the release note for more details.
    >
    >
    • SPIRE Photometer filter transmission curves:
      • These are also available in the SPIRE calibration context (photRsrf) and can be accessed in HIPE after a calibration context has been loaded (See above).
     
    Changed:
    <
    <
    • SPIRE Photometer filter transmission curves: You can access the filter transmission curves (also known as Relative Spectral Response Function, RSRF) from here. These are also available in the SPIRE calibration context and can be accessed in HIPE after a calibration context has been loaded (See above):

    rsrf = cal.phot.rsrf
    

    • Neptune and Uranus models used for the SPIRE flux calibration: the ESA2 models currently used in the SPIRE calibration are available here.
    >
    >
    • Neptune and Uranus models used for the SPIRE photometer flux calibration:
      • The ESA2 models currently used in the SPIRE calibration are available here.
     

    Spectrometer calibration

    Changed:
    <
    <
    Important FTS information, including calibration, point source and extended source calibration etc, is available in the SPIRE Observers' Manual, Sections 4.2 and 5.3. These two sections are a must-read for anybody processing SPIRE FTS data.
    >
    >
    Important FTS information, including details of the calibration, point source and extended source calibration etc, is available in the SPIRE Observers' Manual, Sections 4.2 and 5.3. These two sections are a must-read for anybody processing SPIRE FTS data.
     

    Interest groups and scripts

    Line: 297 to 188
     

    Further information

    Changed:
    <
    <
    >
    >
    Contact the Helpdesk
     
    META FILEATTACHMENT attr="" autoattached="1" comment="SPIRE Spectrometer Launch Pad v1.6" date="1328388627" name="SPIRE_SpectrometerLaunchPad_v1.6.pdf" path="SPIRE_SpectrometerLaunchPad_v1.6.pdf" size="196595" user="Main.ChrisPearson" version="1"
    META FILEATTACHMENT attr="" autoattached="1" comment="" date="1306492467" name="maskTimelines.py.txt" path="maskTimelines.py.txt" size="5945" user="Main.LucaConversi" version="1"
     
    This site is powered by the TWiki collaboration platform Powered by Perl