Difference: SpireCalibrationWeb (90 vs. 91)

Revision 912013-07-26 - EdwardPolehampton

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

SPIRE instrument and calibration web pages

Line: 46 to 46
 

Software and documentation

Changed:
<
<
  • HIPE (Herschel Interactive Processing Environment): The latest User Release HCSS version that you should use for reducing SPIRE data is HIPE v10.3. It can be downloaded from: http://herschel.esac.esa.int/HIPE_download.shtml.
    • Warning, important Please note that there was a bug in the destriper task included in HIPE 9.0 that may affect your final map, especially if there are bright objects in the observed field. This has been corrected since HIPE 9.1. If your observation falls in the mentioned category, you are strongly advised to update your HIPE installation.
>
>
 
  • We also provide access to the latest stable developer build (latest stable CIB).
    • Beware These developer builds do not undergo the same in-depth testing as the user releases do. The latest developer build can be found here.
      Info Please contact the Herschel helpdesk if you plan to use a developer build as there may be some additional information needed in order for you to properly make use of it.

  • Within HIPE you can access all the SPIRE data reduction and HIPE-user documentation. The SPIRE Data Reduction Guide (SDRG) follows the user pipeline scripts and also explains the details of pipeline processing and data analysis. It is also available online here:
Changed:
<
<
>
>
 
  • SPIA: The SPIRE Photometer Interactive Analysis (SPIA) package is available as a plug-in for HIPE. SPIA provides a structured GUI based access to the more intricate parts of the scan map photometer pipeline for SPIRE without the immediate need to resort to scripts. More information can be found in the SDRG or on the SPIA web page
Line: 76 to 75
 
Overview
The best source of information for reducing SPIRE Photometer data is the SPIRE Data Reduction Guide available through the HIPE help. This runs through the User Pipeline scripts step by step, describes several other Useful Scripts, and offers advice for specific issues that might be encountered.
Changed:
<
<
New definition of Leve2 products
>
>
New definition of Level-2 products
 
  • For versions of the HCSS prior to HIPE 10.0, a single point source calibrated (Jy/beam) map was provided in the Level 2 product for each of the PSW, PMW, PLW bands. However, for observations processed with HIPE 10.0 or later, more than one map calibration is made available within the Level 2 product. Maps are provided for the following scenarios for post HIPE v10.0 processing:
Line: 89 to 88
 
Extended Emission Destriper Diagnostic extdPSWdiag -
     
Changed:
<
<
  • psrcPxW are the previous PxW maps, calibrated for point source and in units of Jy/beam. Note that to do aperture photometry on such maps you'll first need to convert them to surface brightness (Jy/pixel, MJy/sr, etc.), although it is suggested to directly use the already extended emission calibrated extdPxW maps. Finally, bear in mind that SPIRE itself cannot measure the absolute sky flux, hence psrcPxW maps have an arbitrary offset having zero median.
>
>
  • psrcPxW are the previous PxW maps, calibrated for point source and in units of Jy/beam. Note that to do aperture photometry on such maps you'll first need to convert them to surface brightness (Jy/pixel, MJy/sr, etc.), although it is suggested to directly use the already extended emission calibrated extdPxW maps. Finally, bear in mind that SPIRE itself cannot measure the absolute sky flux, hence psrcPxW maps have an arbitrary offset having zero median.
 
  • ssoPxW maps are corrected for SSO proper motion: maps are in Jy/beam and they are subject to the same photometry rules of the psrcPxW maps.

  • extdPxW maps are calibrated for extended emission and provided in units of MJy/sr. These maps are provided with an estimation of the absolute offset via cross-calibration with Planck data.
Changed:
<
<
  • In all cases, SPIRE data is calibrated in the assumption of source having a spectral index equal to -1, i.e. νSν = const. To calibrate your data for other cases or convert to e.g. Jy/sr, please refer to section 5.7 of the SPIRE Data Reduction Guide.
>
>
  • In all cases, SPIRE data is calibrated in the assumption of source having a spectral index equal to -1, i.e. νSν = const. To calibrate your data for other cases or convert to e.g. Jy/sr, please refer to section 5.7 of the SPIRE Data Reduction Guide.
 
  • The SPIRE Photometer filter transmission curves, also known as Relative Spectral Response Functions (RSRF) are available here. For more details, please read the .readme file in this ftp folder.
Line: 105 to 104
 The main issues that you might find in your data are: undetected glitches, thermistor or detector jumps, and bad baseline removal.

  • Stripes in PSW, PMW and/or PLW (Level 2) maps
Changed:
<
<
    • All SPIRE photometry pipelines now use the destriper by default, which improves the issue of stripes in Level 2 maps. There should be noticeable improvements in that respect with HIPE version 9. The destriper documentation can be found on the NHSC website
>
>
    • All SPIRE Photometer pipelines now use the destriper by default, which improves the issue of stripes in Level 2 maps. There should be noticeable improvements in that respect from HIPE version 9 onwards. The destriper documentation can be found on the NHSC website
 
  • De-glitcher masks faint sources
    • For data taken in Parallel Mode in particular (sampling at 10Hz, at high speed 60"/s), the de-glitcher may flag very faint sources as glitches when it is run with standard parameters. Faint sources may have a "delta function" shape due to the low sampling rate, which looks similar to a small glitch. Try modifying the "correlation parameter" to 0.95: this will decrease the number of detected glitches - it may be better to have a limited detection rate in first level deglitching and defer to Level 2 deglitching.
Line: 117 to 116
 
    • This effect, related to data masking or poor coverage, is more evident in single fast-scan Parallel Mode maps. To avoid NaNs, increase the pixel size (i.e., decrease the map's resolution).
    • This effect can also occur with destriped maps. In this case check if increasing the sigma parameter or switching off the Level 2 deglitcher helps.
Deleted:
<
<
  • WCS in 3-colour images
    • Problems with the wrong WCS in the output RGB images in all observations reduced with HIPE 8 have been fixed in HIPE 9
 
  • Quality flags in the quality context
    • Currently, the quality flags at the quality context inside the observation context are just meant for HSC/ICC internal evaluation of the quality of the products and not for the users. In case the data had some serious quality problem, the PI of the program has been contacted about it. Otherwise, only information in the quality summary, when available, should concern the observers.

Known Issues in ODs 1304 & 1305
Changed:
<
<
For (yet) unknown reasons, the three detectors PSW-B5, PSW-E9 and PSW-F8 - that use to behave well during the entire mission - were noisy during the two operational days 1304 and 1305. The result are stripes visible in the final PSW map which the current (HIPE 10) pipeline is not able to correct. The solution is to mask and exclude these detectors from the analysis. This could be done in 2 ways:
  1. You can use the SpireMaskEditor GUI as described in sec. 7.4.2 of the SPIRE Data Reduction Guide: write-click on your observation context variable and then select Level1_SpireMaskEditor and set to Master all samples in all scans (listed as BBID) for the detectors mentioned above.
  2. Alternatively, you can use the following line of codes:

##### EDIT MASK SCRIPT BEGINS #####

# List of detectors to be masked
bolos = ['PSWB5', 'PSWE9', 'PSWF8']

# Level1 of your observation, assuming the observation context varible is named 'obs'
level1 = obs.level1

# Create new level 1
new_l1 = Level1Context()

for scan in range(0, level1.getCount()):
   #
   # Load level 1 product, scan by scan
   data = level1.refs[scan].product
   #
   # Change mask for selected detectors in all scans, setting it to MASTER
   for bolo in bolos:
      data['mask'][bolo].data[:] = 1
   #
   new_l1.addProduct(data)

##### EDIT MASK SCRIPT ENDS #####
>
>
For (yet) unknown reasons, the three detectors PSW-B5, PSW-E9 and PSW-F8 - that use to behave well during the entire mission - were noisy during the two operational days 1304 and 1305. The result are stripes visible in the final PSW map which the current (HIPE 11) pipeline is not able to correct. The solution is to mask and exclude these detectors from the analysis. This could be done in 2 ways:
  1. You can use the SpireMaskEditor GUI as described in sec. 7.4.2 of the SPIRE Data Reduction Guide: write-click on your observation context variable and then select Level1_SpireMaskEditor and set to Master all samples in all scans (listed as BBID) for the detectors mentioned above.
  2. Alternatively, you can use these lines of code
 
Changed:
<
<
  • After either of those cases, you must then re-run level 1 to 2 steps on the newly modified level1 product. If your observation has been already re-reduced with HIPE 10, original and new level1s are already destriped, so you can directly run the naive map-maker on the new level1. Otherwise, you must run the destriper step: check the pipeline script for details.
>
>
  • After either of those cases, you must then re-run level 1 to 2 steps on the newly modified level1 product. If your observation has been already re-reduced with HIPE 11, original and new level1s are already destriped, so you can directly run the naive map-maker on the new level1. Otherwise, you must run the destriper step: check the pipeline script for details.
 

Planck-HFI & Herschel-SPIRE cross calibration: absolute offset re-processing

Changed:
<
<
Herschel-SPIRE detectors are only sensitive to relative variations, as a consequence the absolute brightness of the observed region is unknown and maps are constructed such that they have zero median. Planck-HFI detectors are similar to the SPIRE ones, however its observing strategy allows it to (almost) observe a sky's great circle every minute (having a 1 rpm spinning rate). By comparing the sky brightness as measured by COBE-FIRAS at the galactic poles (where the dust emission is lower), HFI is capable of setting an absolute offset to its maps. SPIRE and HFI share two channels with overlapping wavebands: SPIRE-PMW and HFI-857 have a similar filter profile, while SPIRE-PLW and HFI-545 are shifted by ~10%.
>
>
Herschel-SPIRE detectors are only sensitive to relative variations, and so as a consequence, the absolute brightness of the observed region is unknown and maps are constructed such that they have zero median. The Planck-HFI detectors are similar to the SPIRE ones, but the Planck observing strategy allowed it to (almost) observe a great circle on the sky every minute (having a 1 rpm spinning rate). By comparing the sky brightness as measured by COBE-FIRAS at the galactic poles (where the dust emission is lower), HFI is capable of setting an absolute offset to its maps. SPIRE and HFI share two channels with overlapping wavebands: SPIRE-PMW and HFI-857 have a similar filter profile, while SPIRE-PLW and HFI-545 are shifted by ~10%.
 
Changed:
<
<
As of HCSS 10, a new task named zeroPointCorrection is available: this task calculates the absolute offset for a SPIRE map based on cross-calibration with HFI-545 and HFI-857 maps, colour-correcting HFI to SPIRE wavebands assuming a grey body function with fixed beta. At first, Planck data needed by the task were delivered to HSC under special agreement: as a consequence, Herschel users were not able to re-process the absolute offset calculation. However, Planck data became public in April 2013 and it is now possible to exectue the zeroPointCorrection.
>
>
As of HCSS 10, a new task named zeroPointCorrection is available: this task calculates the absolute offset for a SPIRE map based on cross-calibration with HFI-545 and HFI-857 maps, colour-correcting HFI to SPIRE wavebands assuming a grey body function with fixed beta. At first, Planck data needed by the task were delivered to HSC under special agreement: as a consequence, Herschel users were not able to re-process the absolute offset calculation. However, Planck data became public in April 2013 and it is now possible to exectue the zeroPointCorrection task.
  Files needed:
Changed:
<
<
  • Download the HFI-545 and HFI-857 maps from the HSC/SPIRE FTP area. These maps are derived from the ones available in the Planck Legacy Archive, but convolved with an 8 arcmin Gaussian beam in order to circularize the effective maps' beams, plus the maps absolute offset as estimated by the Planck-HFI team via cross-calibration with FIRAS (see Planck Collaboration VIII. 2013, In preparation)
>
>
  • Download the HFI-545 and HFI-857 maps from the HSC/SPIRE FTP area. These maps are derived from the ones available in the Planck Legacy Archive, but convolved with an 8 arcmin Gaussian beam in order to circularize the effective map beams, plus the maps absolute offset as estimated by the Planck-HFI team via cross-calibration with FIRAS (see Planck Collaboration VIII. 2013, In preparation)
 
Changed:
<
<
The offsets are computed on extdPxW maps, calibrated for extended emission, with extended gain correction applied and in units of MJy/sr (as explained in the section 5.7 of the SPIRE Data Reduction Guide). Hence, the re-processing will start from a level-1 context (which may be the result of merging multiple observations, see e.g. the Photometry Map Merging scirpt available in HIPE under the menu ScriptsSPIRE Useful script) and then executing the zeroPointCorrection task with one of the following methods:
>
>
The offsets are computed on extdPxW maps, calibrated for extended emission, with extended gain correction applied and in units of MJy/sr (as explained in the section 5.7 of the SPIRE Data Reduction Guide). Hence, the re-processing will start from a level-1 context (which may be the result of merging multiple observations, see e.g. the Photometry Map Merging scirpt available in HIPE under the menu ScriptsSPIRE Useful script) and then executing the zeroPointCorrection task with one of the following methods:
 
  1. Run the zeroPointCorr.py script. It assumes that a Level1Context and Level2Context are already defined and named level1 and level2, respectively. It also sets three required properties needed by the zeroPointCorrection task, i.e. the location of two HFI maps and the colour correction table: please modify the PATH_TO_FILE accordingly to your set-up.
  2. Alternatively, run the correction using the SPIA interface (SPIRE Photometer Interactive Analysis). In order to be able to run the zeroPointCorrection task, the user.props file present (by default) in you $HOME/.hcss directory must be modified and the following lines added (please modify the PATH_TO_FILE accordingly to your set-up):
    • spire.spg.hfi.545map = PATH_TO_FILE/DX9_map_545_smooth_8arcmin.fits
Line: 216 to 186
 
SPIRE Calibration Tree Applicable HIPE Version Comment
Changed:
<
<
SPIRE_CAL_10_1 HIPE v10 Calibration tree currently used in operations
>
>
SPIRE_CAL_11_0 HIPE v11 Calibration tree currently used in operations
SPIRE_CAL_10_1 HIPE v10 Final v10 cal tree
 
SPIRE_CAL_9_1 HIPE v9 Final v9 cal tree
SPIRE_CAL_8_1 HIPE v8 Final v8 cal tree
SPIRE_CAL_7_0 HIPE v7 Final v7 cal tree.
Line: 236 to 207
  More details of the changes in each version are given here.
Changed:
<
<
  • Any of the calibration trees can be retrieved in HIPE from the HSA using (e.g.) cal = spireCal(calTree="spire_cal_10_1") etc. The default (applicable to the HIPE version in use) can be obtained with cal = spireCal(calTree="spire_cal"). It can then be saved to a local pool right-clicking on the cal variable and then selecting from the context menu Send To -> Local Pool.
>
>
  • Any of the calibration trees can be retrieved in HIPE from the HSA using (e.g.) cal = spireCal(calTree="spire_cal_11_0") etc. The default (applicable to the HIPE version in use) can be obtained with cal = spireCal(calTree="spire_cal"). It can then be saved to a local pool right-clicking on the cal variable and then selecting from the context menu Send To -> Local Pool.
 
  • Alternatively, the latest calibration tree for SPIRE can be obtained as a jar file from Latest calibration trees. Then, you have to possibilities to read and save:
Changed:
<
<
    1. The jar file can be load directly into HIPE with the command: cal = spireCal(jarFile="PATH_TO_FILE/spire_cal_10_1.jar"). To save it to a local pool, proceed as described above, right-clicking on the cal variable and then selecting from the context menu Send To -> Local Pool.
    2. The jar file can also be saved directly to a local pool without opening HIPE, running the following command in the terminal command line: cal_import PATH_TO_FILE/spire_cal_10_1.jar. Then, to load the calibration tree in HIPE, simply type: cal = spireCal(pool="spire_cal_10_1")
>
>
    1. The jar file can be load directly into HIPE with the command: cal = spireCal(jarFile="PATH_TO_FILE/spire_cal_11_0.jar"). To save it to a local pool, proceed as described above, right-clicking on the cal variable and then selecting from the context menu Send To -> Local Pool.
    2. The jar file can also be saved directly to a local pool without opening HIPE, running the following command in the terminal command line: cal_import PATH_TO_FILE/spire_cal_11_0.jar. Then, to load the calibration tree in HIPE, simply type: cal = spireCal(pool="spire_cal_11_0")
 
Changed:
<
<
See the SPIRE Data Reduction Guide for more details.
>
>
See the SPIRE Data Reduction Guide for more details.
 

SPIRE calibration and performance

Line: 296 to 267
 
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1304679998" name="Pipe_Description_2.1.pdf" path="Pipe_Description_2.1.pdf" size="2128829" user="Main.LucaConversi" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1330422558" name="SPIRE-BSS-DOC-002966_SPIRE_Spectrometer_pipeline_description_Issue_4_0_1.pdf" path="SPIRE-BSS-DOC-002966_SPIRE_Spectrometer_pipeline_description_Issue_4_0_1.pdf" size="812580" user="Main.EdwardPolehampton" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="SPIRE Photometer Launch Pad v1.7" date="1345816708" name="SPIRE_PhotometerLaunchPad_v1.7.pdf" path="SPIRE_PhotometerLaunchPad_v1.7.pdf" size="210362" user="Main.ChrisPearson" version="1"
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Script to correct photometer data in ODs 1304/1305" date="1374833050" name="correct_od1304_1305.py.txt" path="correct_od1304_1305.py.txt" size="610" user="Main.EdwardPolehampton" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="SPIRE A&A paper, Griifin et al 2010" date="1295862676" name="aa14519-10.pdf" path="aa14519-10.pdf" size="1353253" user="Main.AnthonyMarston" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1370344174" name="SpireHfiColourCorrTab_v1.1.fits" path="SpireHfiColourCorrTab_v1.1.fits" size="20160" user="Main.LucaConversi" version="1"
 
This site is powered by the TWiki collaboration platform Powered by Perl