TWiki> Public Web>SpireCalibrationWeb (revision 34)EditAttach

SPIRE instrument and calibration web pages

Introduction

This page provides up-to-date information about using the SPIRE instrument: from preparing observations to reducing your data. This page also provides you with the latest calibration accuracies and known SPIRE calibration issues.

Observing with SPIRE

AOT release notes

Photometer

FTS Spectrometer

Reducing SPIRE data

Software and Documentation

  • HIPE (Herschel Interactive Processing Environment): The latest User Release HCSS version that you should use for reducing SPIRE data is HIPE v7.0.0. It can be downloaded from: http://herschel.esac.esa.int/HIPE_download.shtml. FYI: this corresponds to the so-called CIB (continuous integration build) HIPE 7.0 build 1931.

  • We also provide access to the latest stable developer build (a.k.a latest stable CIB), used by the instrument experts at the ICC.
    • Beware These developer builds do not undergo the same in-depth testing as the user releases do. The current latest stable developer build is identical to the latest User Release (HIPE 7.0 build 1931).

  • Within HIPE you can access all the SPIRE data reduction and HIPE-use documentation. For those who wish to read the SPIRE Data Reduction Guide (SDRG) in PDF form, we provide that here: SDRG version 1.5. This version can be used with HIPE v7.0.0 as well as all track 7 and track 8 of the CIBs. (Note that within the PDF version, document links will not work.) The SDRG follows the pipeline scripts (see "Cookbooks" below) and also explains what you are doing as you pipeline process.

Photometer

Maps

  • Note that SPIRE maps are in units of Jy/beam, and are calibrated in the assumption of a point source having a spectral index equal to -1. To calibrate your data for other cases or convert to e.g. Jy/sr, please refer to section 5.2 of the SPIRE Observers' Manual .
  • By default, the SPIRE pipeline uses a nšive map-maker. In this case, the error map is simply the standard deviation of all the data points falling into a given pixel. As a consequence, error maps contain increased errors associated with binning data from Gaussian sources, producing a torus shape; this is an artefact of the map-making process.

Level 2.5

  • As of HIPE 6.1.1, SPIRE observations may include a new Level 2.5. This product includes maps obtained merging all contiguous observations belonging to the same program and having same observing mode (i.e. small map, large map or Parallel Mode). Maps are produced using the standard pipeline, i.e.:
    1. query the database to retrieve all the required observations
    2. merge all Level 1s
    3. remove the baseline using a median fit from each scan
    4. build the maps using the nšive map-maker
  • All the photometer known issues applies to these maps as well. Moreover, note that:
    • no astrometry fix is applied, so sources may be blurred/_duplicated_ if the shift between 2 or more observations is big (>5 arcsec);
    • in merging together multiple observations of the same field, you may not notice anymore some artifacts such as undetected glitches, temperature drifts or detectors jumps. In both cases, you need to re-reduce the data with the tips suggested below.
  • The list of observations used to build the Level 2.5 maps are included in the observations' metadata.

Data processing known issues

  • In order to obtain the best possible Level 2 SPIRE photometry data, the observations might have to be reprocessed with the latest HIPE User Release (see above).

  • Stripes in PSW, PMW and/or PLW (Level 2) maps
    • Most of the stripes that are present in the final maps are due to a combination of thermal drifts (which in few cases are not efficiently removed) and median baseline subtraction. A similar effect is caused by very bright sources: in this case, the problem resides in the median baseline subtraction only.
    • Suggested solutions:
      • switch to a baseline subtraction using a polynomial fitting using the optional task baselineRemovalPolynomial. If there are no jumps in the timelines, you may also try to run the baseline removal on the entire timeline;
      • in the case of bright sources, you may try to mask them before running the baseline removal (either median or polynomial): you can use this script as a template.;
      • use the SPIRE Destriper: this new task is giving good results in most cases, especially for diffuse emission and extended sources. Note that it only works if you have enough coverage redundancy, i.e. cross-scans and/or multiple repetitions. Hence, in the case of Parallel Mode observations, you will need to merge 2 or more observations. The destriper documentation can be found on the NHSC website

  • De-glitchter masks faint sources
    • The de-glitcher is a very delicate process. In particular, for data taken in Parallel Mode (sampling at 10Hz) and at high speed (60"/s) the de-glitcher with standard parameters may flag very faint sources as glitches. Bright sources are different from glitches in that they have a Gaussian (i.e. beam/PSF) shape. For faint sources, the sampling rate could be not high enough and hence they have a "delta" shape, which is similar to a small glitch. The user might try to modify the correlation parameter to 0.95: this will decrease the number of detected glitches.

  • Some sources have saturated the ADC and the corresponding data have been masked
    • There is nothing a user can do: the source was simply too bright. If the user has other sources still not observed and of the same intensity, it is suggested to change the AORs to use the bright source mode.

  • Thermistor jumps
    • As of HIPE 6.0.3, a new module together called signalJumpDetector in place to identify the jump and to exclude the affected thermistor(s).

  • NaNs pixels present in the PSW, PMW and/or PLW (Level 2) maps
    • This effect, related to data masked for various reasons and poor coverage (not enough redundancy), is more evident in single fast-scan Parallel Mode maps. To avoid NaNs, increase the pixel's dimension (i.e., decrease the map's resolution)

  • Quality flags in the quality context
    • Currently, the quality flags at the quality context inside the observation context are just meant for HSC/ICC internal evaluation of the quality of the products and not for the users. In case the data had some serious quality problem, the PI of the program has been contacted about it. Otherwise, only information in the quality summary, when available, should concern the observers.

Tips to re-reduce your data

  • Always remember to update to the latest calibration tree compatible with the HIPE built you are using (See the SPIRE Data Reduction Guide, Chapter 3 for a detailed explanation and examples). Assuming the observation is loaded into HIPE as a variable named obs:

cal = spireCal(calTree="spire_cal")
obs.calibration.update(cal)

  • If the observation you retrieved from HSA has been reduced with SPG v. 2.x or less, than start reprocessing from level 0 (i.e., run again the engineering conversion level 0 -> 0.5)

  • Main issues you might find in your data are: undetected glitches, thermistor or detector jumps, bad baseline removal.
    • Undetected glitches: you may try to play with the parameters of the waveletDeglitcher, in particular changing correlationThreshold parameter; other solution is to use the alternative sigmaKappaDeglitcher
    • Thermistor jumps: this should be automatically solved re-reducing your observation as of HIPE v. 6. If this is not the case, you must exclude the affected thermistor when running the temperatureDriftCorrection adding e.g. pswThermistorSelect='T1'
    • Failure of Temperature Drift Correction: Due to an update of the Temperature Drift Correction task in the pipeline, the pipline may fail with an Index argument 0 is out of range error if run with Calibration Tree spire_cal_6_0. Please update to use spire_cal_6_1 to solve the problem (See the SPIRE Data Reduction Guide, Chapter 3).
    • Bad baseline removal (see also above) as of Hipe v. 6.x, a new polynomial fit (in comparison to the standard median) for baseline removal has been added as a prototype. Assuming that your Observation Context is stored in a variable named obs, you can call it as e.g.:

from herschel.spire.ia.pipeline.phot.baseline import BaselineRemovalPolynomialTask
baselineRemovalPolynomial = BaselineRemovalPolynomialTask()

scansBaseline = baselineRemovalPolynomial(input=obs.level1, polyDegree=3)
mapBaseline   = naiveScanMapper(input=scansBaseline, array="PLW")

Source extraction

Tests have demonstrated that a source fitter working on the detectors' timeline works better than the map-based, such as sourceExtractorDaophot or sourceExtractorSussex. The algorithm will be included in future Hipe releases in the form of a task.

For the time being, you can use the jython script written by G. Bendo bendoSourceFit_v0.9.py: it will fit a Gaussian function to the baseline-subtracted SPIRE timelines in a SpireListContext.

Example Use

This example is based on fitting the peak of Gamma Dra in ObsID 0x50005984 in the PSW band.

fitter=bendoSourceFit(inputContext)
param=fitter.fit("PSW", 269.1515617, 51.488894, 200)
param=fitter.fit("PSW", 269.1515617, 51.488894, 22, 300, 350)

The first line defines an instance of the fitter object.

The second line calls a method in which the data within a 200 arcsec circle centered on RA=269.1515617 and Dec=51.488894 is fit with a Gaussian function. The default is to fit an elliptical Gaussian function with a variable background. The first parameter will be the peak flux density.

The third line calls a methods in which a background is measured within an annulus between radii of 300 and 350 arcsec and then a Gaussian function is fit to both the central 22 arcsec and the background annulus. The default function, an elliptical Gaussian function with a variable background, is still used in this case.

See the comments at the beginning of the code to learn how to select optional functions, set parameters for the fits, or get additional data based on the resulting fits (e.g. uncertainties in the best fitting parameters).

FTS Spectrometer

Telescope RSRFs (daily dark sky observation) are available here for different versions of HIPE. These can be used directly in the user processing script. For best results, one should use the telescope RSRF derived from a daily dark taken in the day of the observation. The RSRF file must match the spectral resolution of the observations, the links with "aH, aM, and aL" are for the apodized case.

Cookbooks

SPIRE Photometry cookbook.

The current version of the cookbook is available here and provides practical guidelines on how to do photometry with SPIRE. HELP The cookbook is not HIPE specific.

SPIRE calibration file versions

Calibration files for SPIRE can be obtained here:

The available calibration trees for SPIRE are listed below (with the current operational version at the top).

SPIRE Calibration Tree Applicable Hipe VersionSorted descending Comment
SPIRE_CAL_7_0 Hipe v7 Final v7 cal tree, currently used in operations.
SPIRE_CAL_6_1 Hipe v6 Final v6 cal tree
(SPIRE_CAL_6_0) Hipe v6 Spec major update
SPIRE_CAL_5_2 Hipe v5 Final v5 cal tree
(SPIRE_CAL_5_1) Hipe v5  
(SPIRE_CAL_5_0) Hipe v5 Phot flux conv. based on Neptune. Spec major update
SPIRE_CAL_4_0 Hipe v4 Spec point source flux conv based on Uranus
SPIRE_CAL_3_2 Hipe v3  
(SPIRE_CAL_3_1) Hipe v3  
(SPIRE_CAL_3_0) Hipe v3  
SPIRE_CAL_2_1 Hipe v2 Spec point source flux conv based on Vesta
(SPIRE_CAL_2_0) Hipe v2  
SPIRE_CAL_1_2 Hipe v1 Phot flux conv based on Ceres
(SPIRE_CAL_1_1) Hipe v1 Pre-launch dummy values

More details of the changes in each version are given here. Any of the calibration trees can be retrieved in HIPE from the HSA using (e.g.) cal = spireCal(calTree="spire_cal_7_0") etc. The default (applicable to the Hipe version) can be obtained with cal = spireCal(calTree="spire_cal")

See the SPIRE Data Reduction Guide for more details.

SPIRE calibration and performance

Photometer

  • SPIRE Photometer Beams: The theoretical and the observed SPIRE photometer beams are available from here . Please read the release note for more details. These are also available in the SPIRE calibration context and can be accessed in HIPE after a calibration context has been loaded (See above):

BeamPSW = cal.phot.getBeamProf('PSW')
BeamPMW = cal.phot.getBeamProf('PMW')
BeamPLW = cal.phot.getBeamProf('PLW')

  • SPIRE Photometer filter transmission curves: You can access the filter transmission curves (also known as Relative Spectral Response Function, RSRF) from here. These are also available in the SPIRE calibration context and can be accessed in HIPE after a calibration context has been loaded (See above):

rsrf = cal.phot.rsrf

  • Neptune and Uranus models used for the SPIRE flux calibration: the ESA2 models currently used in the SPIRE calibration are available here.

FTS Spectrometer

Important FTS information, including calibration, point source and extended source calibration etc, is available in the SPIRE Observers' Manual, Sections 4.2 and 5.3. These two sections are a must-read for anybody processing SPIRE FTS data.

Interest Groups and Scripts

  • The following interest groups relate to processing of observations taken with SPIRE. The links provided allow subscription to these interest groups.
    • subscribe to the SPIRE Photometer interest group
    • subscribe to the SPIRE and PACS parallel mode large map and point source extraction interest group
    • subscribe to the SPIRE Spectrometer interest group
    • subscribe to the PACS, SPIRE and HIFI spectral maps interest group
  • User contributed scripts: Users are welcome to submit scripts and software that they believe could be of general interest to the community to the Herschel Helpdesk.

Further Information

Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf Phot_Pipeline_Issue7.pdf r1 manage 453.9 K 2011-05-06 - 11:07 LucaConversi  
PDFpdf Pipe_Description_2.1.pdf r1 manage 2078.9 K 2011-05-06 - 11:06 LucaConversi  
PDFpdf SPIRE-BSS-DOC-002966_SPIRE_Spectrometer_pipeline_description_Draft_3_1.pdf r1 manage 756.5 K 2011-05-06 - 11:12 LucaConversi  
PDFpdf SPIREPhotometryCookbook_may2011_1.pdf r1 manage 61.5 K 2011-06-24 - 09:20 IvanV SPIRE Photo cookbook
PDFpdf aa14519-10.pdf r1 manage 1321.5 K 2011-01-24 - 09:51 AnthonyMarston SPIRE A&A paper, Griifin et al 2010
PDFpdf aa14605-10.pdf r1 manage 264.1 K 2011-01-24 - 16:48 IvanV SPIRE in-flight calibration
PDFpdf beam_release_note_v1-1.pdf r1 manage 4044.5 K 2011-05-11 - 09:44 LucaConversi  
Texttxt bendoSourceFit_v0_9.py.txt r1 manage 25.7 K 2011-08-10 - 09:03 IvanV  
Texttxt maskTimelines.py.txt r1 manage 5.8 K 2011-05-27 - 10:34 LucaConversi  
Edit | Attach | Watch | Print version | History: r151 | r36 < r35 < r34 < r33 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r34 - 2011-08-10 - IvanV
 
This site is powered by the TWiki collaboration platform Powered by Perl