SPIRE Data Reduction Guide

For HIPE 15 version 5.2, Document Number: SPIRE-RAL-DOC 003248
23 Nov 2016

Table of Contents

1. Versioning
1.1. Changelog
1. Introduction
1.1. Scope and Structure of this Data Reduction Guide
1.2. SPIRE Observing Modes
1.3. SPIRE Observation ID Numbers - converting hexadecimal to decimal
1.4. List of Useful Documentation
2. SPIRE Launch Pad: Data Reduction Overview
2.1. Data Launch Pad
2.1.1. Obtaining SPIRE data from the archive and importing into HIPE
2.1.2. Looking at your Data
2.2. Photometer Launch Pad
2.2.1. Does the Photometer observation data need re-processing?
2.2.2. Re-processing with the User Pipeline Scripts
2.2.3. Further Analysis
2.3. Spectrometer Launch Pad
2.3.1. Does the observation data need re-processing?
2.3.2. Re-processing with the User Pipeline Scripts
2.3.3. Further Analysis
3. Overview of Scripts in HIPE
4. SPIRE Observation Context Data Structure
4.1. Accessing SPIRE Data
4.1.1. Understanding how HIPE accesses data
4.1.2. Accessing data directly from the Herschel Science Archive
4.1.3. Querying the SPIRE Observation Log
4.2. SPIRE Observation Context Data Structure
4.2.1. Anatomy of a SPIRE Observation: Products, Pools, Storage, and Building Blocks
4.2.2. Linking it altogether: Introducing the Context
4.2.3. The Level 2.5 Data in the Herschel Science Archive
4.2.4. The Level 3. Data in the Herschel Science Archive
4.2.5. Looking at your Observation Context in HIPE
4.2.6. What the Observation Context looks like on your hard disk
4.3. Pipelines, versions and the archive
5. SPIRE Calibration Data
5.1. The SPIRE Calibration Context
5.2. The SPIRE Calibration Tree
5.3. SPIRE Calibration Product Editions
5.4. Updating a Calibration Tree
5.5. The SPIRE Calibration Task GUI and Calibration Viewer
5.6. Updating Individual Calibration Products
5.7. Removing Calibration Products from the Tree
5.8. Which Calibration Tree Version was used to Process my Data?
5.9. Further Information
6. SPIRE Photometer Mode Cookbook
6.1. SPIRE Scan Map and Data Structure
6.1.1. A first look at your image maps (The Level 2 Data Product)
6.1.2. Saving a map as a FITS file and reading it in again
6.1.3. Looking at observations with Level 2.5 and Level 3 products
6.1.4. Looking at the Level 1 Timeline Data
6.1.5. SPIRE Serendipity Data at Level 1
6.1.6. Looking at the Level 0.5 Timeline Data
6.1.7. Looking at the Raw Level 0 Data
6.2. SPIRE Point Source Mode Data Structure
6.2.1. Looking at the Point Source Mode Data
6.2.2. Reading the JPP into memory and saving it as a FITS file and reading it in again
6.2.3. Looking at the Level 1 Data for Point Source Observations
6.2.4. Looking at the Level 0.5 Timeline Data for Point Source Observations
6.2.5. Looking at the Raw Level 0 Data
6.3. Memory Requirements
6.4. Processing SPIRE Observations with the 2-Pass Pipeline
6.4.1. The SPIRE 2-Pass User Pipeline Script
6.5. Recipe for Photometer Large Map and Parallel Mode
6.5.1. Large Map and Parallel Mode User Pipeline Script
6.5.2. Large / Parallel Mode Map Troubleshooting and Tips
6.6. Recipe for Photometer Small Map Mode
6.6.1. Small Map Mode User Pipeline Script
6.6.2. Small Map Troubleshooting and Tips
6.7. Recipe for Photometer Point Source Mode
6.7.1. Point Source Mode User Pipeline Script
6.7.2. Point Source Mode Troubleshooting and Tips
6.8. Destriping and Baseline Removal
6.8.1. Improving Maps with Baseline Removal and Destriping Tools
6.9. Recipe for SPIRE Photometry
6.9.1. SPIRE Source Extraction & Photometry in HIPE
6.10. Absolute Calibration of SPIRE Maps
6.10.1. SPIRE Map Zero Point from Herschel-Planck Calibration
6.11. Tools for Manipulating SPIRE Maps
6.11.1. SPIRE Mapping World Coordinate System
6.11.2. Processing more than one SPIRE observation together
6.11.3. Merging two or more SPIRE Maps together
6.11.4. Aligining SPIRE Maps
6.11.5. Creating Superresolution Maps for SPIRE Observations
6.12. Tools for Moving Objects
6.12.1. Correcting observations of Solar System Objects
6.12.2. Locating a Solar System Object in a map
7. SPIRE Spectroscopy Mode Cookbook
7.1. Introduction to processing FTS data
7.1.1. Basics of Fourier transform spectroscopy
7.1.2. Detection
7.1.3. Fourier Transformation
7.1.4. Efficiency Losses
7.1.5. Interferogram Asymmetries - Spectral Phase
7.2. SPIRE Spectroscopy Data Structure
7.2.1. Introduction to FTS data
7.2.2. The SPIRE FTS Observation Context
7.2.3. The Final FTS Spectral Data Products (Level-2)
7.2.4. The Level-1 Interferogram Data Products
7.2.5. The Spectrometer Level-0.5 Data Products
7.2.6. The Spectrometer Level-0 Data
7.3. Spectroscopy Pipeline Step-by-step
7.3.1. Reprocessing SPIRE spectrometer data
7.3.2. Memory requirements
7.3.3. Introduction to the User Pipeline Scripts
7.3.4. User Selectable Options
7.3.5. Preparing to process
7.3.6. Start processing from the Level 0.5 products
7.3.7. Removing unnecessary channels
7.3.8. Identify the jiggle position
7.3.9. First level deglitching
7.3.10. Account for non-linearities
7.3.11. Correct the detector signals for clipping
7.3.12. Correct time domain phase in the detector signals
7.3.13. Create a SPIRE Pointing product
7.3.14. Interpolate SDT and SMECT to create interferograms
7.3.15. Subtract the interferogram baseline
7.3.16. Apply second level deglitching
7.3.17. Correct interferogram phase
7.3.18. Fourier transform the interferograms
7.3.19. Fetch calibration files
7.3.20. Remove out-of-band power
7.3.21. Apply bright gain correction
7.3.22. Correct for instrument emission
7.3.23. Apply the extended source flux conversion
7.3.24. Correct for telescope emission
7.3.25. Apply the point-source flux calibration
7.3.26. Apply the low resolution correction
7.3.27. Average the spectra
7.3.28. Apodization
7.3.29. Correct the frequency axis to LSR
7.3.30. Sort the meta data order
7.3.31. Apply the extended calibration correction
7.3.32. Add overlapping photometer observation IDs
7.3.33. Save the Level-2 products as FITS
7.3.34. Create spectral cubes (mapping script)
7.3.35. Update and store the Observation Context
7.4. Pointing Considerations
7.4.1. SPIRE BSM Position
7.4.2. Correcting the pointing
7.5. Recipes for faint and medium-strength sources
7.5.1. Comparing point-source and extended-source calibration
7.5.2. Optimising background subtraction
7.5.3. Checking Spectral Noise
7.5.4. Comparing with the SPIRE Photometer
7.6. Recipes for semi-extended sources
7.6.1. Identifying partially-extended sources
7.6.2. Does my spectrum need correcting?
7.6.3. Using the SECT GUI
7.6.4. Pointing considerations
7.6.5. Scripting with SECT
7.6.6. How much can I believe the SECT results?
7.7. Recipes for bright sources (above 500 Jy)
7.8. Recipes for mapping observations
7.8.1. Understanding the SPIRE beam and how it relates to mapping observations
7.8.2. Check clipping
7.8.3. Pointing Information
7.8.4. Restricting or expanding the data for gridding
7.8.5. The gridding algorithms
7.8.6. Holes in the map: examining the coverage
7.8.7. Maps with faint continuum levels
7.9. Observations with few repetitions
7.9.1. Removing data
7.9.2. Correcting data
7.10. Comparison with the SPIRE photometer
7.10.1. The FTS Synthetic Photometry task
7.10.2. Point source observations
7.10.3. Mapping observations
7.11. Spectral Analysis
7.11.1. Local Standard of Rest
7.11.2. Joining SLW and SSW data
7.11.3. Visualising SPIRE spectra
7.11.4. Spectral fitting
7.11.5. Sinc model
7.11.6. Gaussian model for apodized data
7.11.7. Partially resolved lines
7.11.8. Interactive line fitting using the Spectrum Fitter GUI
7.11.9. SPIRE Spectrometer Line Fitting script
7.11.10. Calculating line fitting errors
7.11.11. Line identification
7.11.12. Joining SPIRE and PACS point spectra
7.11.13. Comparing SPIRE and HIFI spectra
7.12. Cube Analysis
7.12.1. SPIRE FTS cubes
7.12.2. Extracting spectra from a spectral cube
7.12.3. Extracting a point source from a spectral cube
7.12.4. Matching spectra from SSW and SLW cubes
7.12.5. Convolving a cube to a different beam size
7.12.6. Cube fitting with the Spectrum Fitter GUI
7.12.7. Cube fitting and line intensity/velocity maps
7.12.8. Publication quality plots for mapping data
7.12.9. Comparison with photometer maps
8. SPIRE Visualisation Tools
8.1. SDI/SDS Explorer
8.1.1. Starting SDI/SDS Explorer
8.1.2. SDI/SDS Explorer Layout
8.1.3. Example 1: Plotting and Overplotting
8.1.4. Example 2: Making a Thumbnail Image
8.1.5. Example 3: Plotting complex data: real, imaginary, absolute, and phase
8.2. Detector Timeline Explorer (DTE)
8.2.1. Starting DTE
8.2.2. DTE Layout
8.2.3. DTE Preferences
8.2.4. Example 1: Plotting functions
8.2.5. Example 2: Browse through time across the timeline.
8.2.6. Example 3: Visualize the flagged detectors in the Array Display
8.3. SPIRE Bolometer Finder
8.3.1. Purpose
8.3.2. The BoloFinderTool
8.3.3. Interacting with the BoloFinder via Scripting
8.3.4. Specifying the map pixel via the command line
8.3.5. Visualising timelines with the plotAllData() method
8.3.6. Identifying outliers with the Bolometer Finder Useful Script and the SPIRE Mask Editor
8.4. SPIRE Mask Handling
8.4.1. SPIRE Mask Formalization
8.4.2. SPIRE Mask Editor Viewer
8.4.3. Other ways to view SPIRE masks
8.5. Quality Control for SPIRE pipeline processed data
8.5.1. Quality Control for SPIRE pipeline processed data
9. Advanced HIPE Tips
9.1. Saving an observation context to a pool on disk
9.2. Using TEMP STORAGE to overcome memory problems with HIPE
9.3. Filtering variables in the HIPE variable list
10. Glossary
11. Reprocessing with the SPIA
11.1. Overview
11.2. Why reprocess with the SPIA?
11.3. SPIA Setup and Workflows
11.4. Locating the SPIA tasks within HIPE
11.5. Retrieving observations from the Herschel Science Archive into local pools
11.6. Loading your observation into your HIPE session
11.7. Downloading the SPIRE calibration tree
11.8. Loading SPIRE calibration information into HIPE
11.9. Reprocessing your data to calibrated timelines
11.10. Making maps from one or more observations
11.10.1. Removing Serendipity Scans from the Level 1 Context
11.11. Inspecting your reprocessed maps
11.12. Inspecting concatenated calibrated timelines
11.13. Correcting artifacts via manual inspection
11.13.1. Processing to Level0.5 and detecting artifacts
11.13.2. Correcting artifacts via manual inspection
11.13.3. Repairing artifacts and reprocessing to Level 1
11.14. Saving the reprocessed results
11.15. Worked Examples
11.15.1. Map-making with turnaround data and destriping
11.15.2. Destriping two large maps
12. References
12.1. SPIRE DRG References

List of Figures

3.1. Menu access to scripts within HIPE
4.1. How HIPE performs data access
4.2. Accessing the Herschel Science Archive through the HIPE menus
4.3. Accessing the Herschel Science Archive through the HIPE menus
4.4. The Observation Context available via the HIPE variable window
4.5. Query the SPIRE Onsevation Log on source name.
4.6. General structure of a SPIRE data Product
4.7. General structure of the Local Store
4.8. The Context structure within HCSS
4.9. The complete Observation Context of a SPIRE observation
4.10. Level 2.5 map created from 3 individual observations
4.11. The Observation Context within HIPE
4.12. Inside the Observation Context within HIPE
4.13. Inside the Observation Context within physical Structure of Observation Context.
5.1. The SPIRE calibration context.
5.2. SPIRE calibration editions.
5.3. The SPIRE Calibration Automatic Updater GUI.
5.4. The SPIRE Calibration Automatic Updater GUI.
5.5. The SPIRE calibration GUI.
5.6. The SPIRE calibration view GUI.
5.7. The preference panel for the SPIRE Calibration view.
5.8. Cal versions from the product History dataset.
6.1. Loading and viewing the Observation Context for the Large Map Observation
6.2. Accessing the final Level 2 Product maps
6.3. Viewing the Level 2 Image Maps
6.4. Viewing the Level 2 Image Array Datasets
6.5. Exporting Image Maps as FITS files
6.6. Observation Context showing Level 2, Level 2.5 and Level 3 products (the veiwer shows the Level 2)
6.7. Level 2.5 product constructed from 2 parallel mode scan map observations
6.8. Level 3 product constructed from contiguous observations conected to obsid=1342189081.
6.9. Mosaic Level 2 maps to create a Level 3 map.
6.10. Creating a new WCS for input into the mosaic task.
6.11. Viewing the Level 1 Photometer Scan Products
6.12. Plotting Level 1 Photometer Scan Product Timeline Data
6.13. Final Level 1 Photometer Scan Product Timeline Data variable list
6.14. Serendipity Scan in the Level 1 data.
6.15. Serendipity Scan motion across sky to make a small map.
6.16. Serendipity Scan processed as an image.
6.17. Anatomy of Level 0.5 Building Block structure
6.18. Inside the Level 0.5 Building Block structure for a Large Map observation
6.19. Plotting the Level 0.5 data for a Large Map observation
6.20. The Level 0 Raw Data within the Observation Context
6.21. Loading and viewing the Observation Context for the Photometer Point Source Observation.
6.22. Accessing the final Level 2 Product Jiggled Photometer Product
6.23. Exporting the JPP as a FITS file
6.24. Viewing the Level 1 Averaged Pointed Photometer Product
6.25. Plotting Level 1 APPP Data Product
6.26. Anatomy of Level 0.5 Building Block structure for a Point Source observation
6.27. Inside the Level 0.5 Building Block structure for a Point Source observation
6.28. Plotting the Level 0.5 data for a 7-point Jiggle Point Source observation
6.29. The Level 0 Raw Data within the Observation Context
6.30. Improvements with the SPIRE 2-Pass Pipeline
6.31. The SPIRE Photometer 2-Pass Pipeline.
6.32. Selecting the Photometer 2-Pass User pipeline script within HIPE
6.33. Selecting the Photometer Large Map User Pipeline pipeline script
6.34. The SPIRE Photometer Large Map pipeline.
6.35. The Essence of the SPIRE Photometer mapping Pipeline (applicable to Large Map, Parallel and Small Map) Script.
6.36. Prerformance of the Concurrent Deglitcher.
6.37. Edge ringing in the final scan line.
6.38. Selecting the Product Viewer
6.39. The NGC 5315 PSW Level 2 image map
6.40. The NGC 5315 PSW Level 2 error map
6.41. The NGC 5315 PSW Level 2 coverage map
6.42. Thumbnail Summary of Map Artefacts.
6.43. PLW, PMW and PSW maps for a single scan observation.
6.44. Artifacts in the map due to bright source effects.
6.45. The effect of stray light from a planet on a map.
6.46. Failure of baseline removal example.
6.47. The effect of jumps in thermistor channel signal.
6.48. The effect of a saturated thermistor.
6.49. Bolometer signal jumps in the map.
6.50. Correction of bolometer signal jumps.
6.51. Artefacts due to Cooler Burp effect.
6.52. Detection and correction of Cooler Burp effects.
6.53. Temperature variation in long observations.
6.54. Bright pixels in the final map caused by multiples glitches.
6.55. Holes represented as NaNs in the final signal map.
6.56. Edge effects in maps.
6.57. Donut shaped artefacts in the error maps at the positions of sources.
6.58. Anomolous errors from binning data from Gaussian sources.
6.59. OD1304-OD1305 maps before and after masking Noisy bolometers.
6.60. Large undetected glitch in the turnaround data.
6.61. Example of failed Temperature-drfit correction in early SPIRE observation.
6.62. Selecting the Product Viewer
6.63. The Gamma Draconis PSW Level 2 image map
6.64. The Gamma Draconis PSW Level 2 error map
6.65. The Gamma Draconis PSW Level 2 coverage map
6.66. The SPIRE POF2 Photometer Point Source pipeline.
6.67. The Essence of the POF2 Point Source Mode Pipeline Script.
6.68. PSW Sparse Map
6.69. The sparse maps for the Point Source mode.
6.70. The contribution of the background to chopped observations.
6.71. The effect of pointing jitter and nod uncertainties on the measured signal.
6.72. Selecting the Photometer Baseline Removal and Destriper Script
6.73. Summary of different baseline removal performance
6.74. Destriper GUI interface
6.75. Destriper Diagnostic Product
6.76. Setting the Destriper Bright Source Flag
6.77. Setting the Destriper Bright Source Flag
6.78. Median Baseline Removal Task
6.79. Examples of Baseline Removal and Destriping results
6.80. Summary of different baseline removal performance
6.81. Summary of Source Extraction and Photometry.
6.82. Summary of Point Source Photometry methods in HIPE. The necessary inputs are summarised in the table below.
6.83. List of parameters for the two source extraction tasks.
6.84. The list of sources shown in the Product Viewer, with the internal dataset highlighted.
6.85. GUI for the Timeline Fitter
6.86. Summary of Photometry methods for Extended Emission in HIPE. The necessary inputs are summarised in the table below.
6.87. Results from HIPE source fitting task
6.88. Summary of Source Extraction and Photometry.
6.89. Measuring source fluxes with list driven photometry.
6.90. sourceExtractorSimultaneous GUI
6.91. Zero point correction task dialogue
6.92. Description of thezeroPointCorrection
6.93. Output maps from the Zero Point Correction Task
6.94. spireMultiObs task GUI in HIPE
6.95. Map Merging of 2 scans from Parallel Mode
6.96. Map Merging of 4 Small Map Mode maps
6.97. Outputs of Superresolution Mapping script
6.98. Telescope tracking an SSO during a mapping observation.
6.99. SSO obsevation before and after SSO correction script.
6.100. Editing SSO moving object meta data.
6.101. Marking the start and end position of an SSO on a map.
7.1. The optical layout of the SPIRE imaging FTS.
7.2. The centre burst of a sample interferogram I(x).
7.3. 1/f-like variation of the baseline of eight repeated scans during the observation of a very strong point source.
7.4. Sample interferograms from SPIRE taken in High (black) and Low (red) resolution mode.
7.5. The imaginary part of five double-sided spectra (left) and the corresponding phase (right).
7.6. SPIRE FTS observing modes
7.7. Products in the Level-2 Context for a HR sparse observation (left), a HR mapping observation (middle) and a H+LR sparse observation (right).
7.8. Inspecting spectral data from a Level-2 product in table form
7.9. Plot of the centre detectors of a point-source calibrated Level-2 product, which was generated using the example code given in the main text.
7.10. Viewing the SPIRE Level-1 Context. The interferogram is selected in the Data panel, which opens it in the SDS Explorer in the Display Panel tab.
7.11. Anatomy of Level-0.5 Building Block structure for the Observation Context of observation 1342227790.
7.12. Inside the Level-0.5 Building Block structure for a spectrometer observation
7.13. Plotting the Level-0.5 data for a Spectrometer observation
7.14. The Level-0 raw data within the Observation Context
7.15. The SPIRE Spectrometer pipeline. Note that the "Bright Gain" correction is only applied to Bright-mode observations and the "LR Correction" is only applied to low resolution observations.
7.16. The SLWC3 detector timeline for the 10 scan example observation of NGC 7027 . The timeline can be visualised by double-clicking the sdt variable.
7.17. The interference pattern for one scan can be inspected more closely by using the zoom functionality of the plot.
7.18. SCALTEMP viewed as a function of observation time. This housekeeping product is a good indication of the temperature of the optical bench.
7.19. The position of the stage mechanism as a function of time during the observation.
7.20. A glitch in the signal timeline of SLWC5 about 645.5s into the observation.
7.21. The spectrum will show a sinusoidal artefact if a glitch remains uncorrected. The frequency of the sinusoidal artefact is higher the further away from ZPD the glitch occurs. As an example, see the interferograms without any deglitching and resulting spectra for scans 2 (blue), 3 (black), and 8 (green) of SLWC5.
7.22. The signal timeline of SLWC5 before (blue) and after (red) executing the wavelet deglitcher with the settings given above.
7.23. The original signal timeline of SSWD4 (blue) shows damaged interference fringes after executing the wavelet deglitcher with correlationThreshold = 0.5 (deglitched and damaged timeline in red).
7.24. Thumbnail plot (Section 8.1.4) showing the interferogram for all SSW detectors after subtracting a constant from each one. The curvature of the baseline increases away from the centre of the array.
7.25. An interferogram with a glitch-like feature before removing the baseline and after removing the baseline with Fourier filtering and fitting a 4th order polynomial function.
7.26. The phase-corrected spectrum measured by the centre detectors in a single scan.
7.27. The SSWD4 spectrum processed with the full user script (blue) compared to the results after processing the data with the baseline correction module disabled (turquoise).
7.28. Averaged SSWD4 spectrum showing little systematic power outside of the optical passband.
7.29. Spectra for the centre detectors before (blue) and after (red) instrument correction.
7.30. The telescope model subtracted from the spectra in Figure 7.31.
7.31. Spectra for the centre detectors before (blue) and after (red) telescope correction.
7.32. Point source calibrated spectra from the centre detectors.
7.33. The difference between the HR and LR point-source calibrated spectra for the centre detectors of the H+LR observation 1342253971. SSW is essentially flat, whereas there is a bumpy distortion in the difference for SLW. The peaks seen at 500 and 900 GHz are introduced by artefacts in the LR data. The LR data were interpolated onto the HR frequency scale before subtraction from the HR data.
7.34. The standard FTS line shape (dark blue) compared to the "smoothed" Gaussian-like shape after applying the adjusted Norton-Beer 1.5 apodization function (light blue).
7.35. Comparison of the average photometer surface brightness to synthetic FTS photometry ratios for 24 spatially flat sources (see Valtchanov et al. 2016). The blue symbols are the median ratio of SPIRE photometer vs spectrometer. The error bars include the 10% uncertainty of the Planck zero offset and the median absolute deviation for the ratio of the 24 flat sources. The green circles are ground based measurements from Chattopadhyay et al. (2003). The far-field feedhorn coupling efficiency is shown for the two FTS frequency bands (dashed red lines). The SSW level is increased by 10% with respect to Wu et al. (2013), to minimise discontinuity in the overlap region for extended calibrated spectra and to better match with the photometer.
7.36. The different elements involved in calculating the pointing of each detector. The array shown is SSW.
7.37. Comparison of the 2σ APE distribution, shown as a blue cloud, before (left) and after (right) OD 1011.
7.38. Plots produced by the Pointing Offset Corrector script for the observation of Uranus given in the script. On the left, the grid of pointing offset is plotted as a function of overlap ratio. The green square marks the final pointing offset. On the right, the input data is compared to the data corrected, output by the script.
7.39. The point-source calibrated continuum offset, which is a systematic additive uncertainty associated to residual telescope and instrument emission. The offset is relatively flat in SSW, away from the noisy band edges, whereas SLW sees a steep increase above and below 700 GHz. The difference between the two frequency bands arises as contribution from the instrument is only significant below 600 GHz and telescope emission is more significant for SLW, due to a larger beam size compared to SSW.
7.40. Four point-source calibrated spectra, processed using the standard HIPE pipeline (SLW in red, SSW in blue) in comparison to the dark sky observation taken on the same operational day (black). The source brightness increases from faint to medium through observations (A) to (D). Source (A) is compared to a reduction with HIPE version 10 (grey) to illustrate the improvement from HIPE version 11 (red and blue).
7.41. Plots produced at the end of the HIPE "Background Subtraction" script for example source (A). They show the smoothed off-axis detectors (black) and the mean used for subtraction (dashed blue) for both SLW (left) and SSW (right). The red curves for SSW show outliers that were excluded from the mean.
7.42. Plots produced by the HIPE "Spectrometer Array Footprint Plot" script for example sources (A) and (C), using the corresponding SPIRE photometer PSW maps. (A) is point-like in the map and the off-axis detectors are not shown to be contaminated by nearby sources or high background. The apparent offset of the source centre is within the absolute pointing errors for the spectrometer and photometer. Although (C) is also point-like, it sits within high background emission, which can be subtracted using the Background Subtraction useful script.
7.43. Plots resulting from running the HIPE "Spectrometer Background Subtraction" script for the four example sources. The original pipeline spectrum is shown in red, the dark sky subtracted spectrum in black and the off-axis detector subtracted spectrum in green.
7.44. The GUI layout for the HIPE task spiaFtsBackgroundRemoval. This task performs a background subtraction for sparse pointed observations and works with full Observation Contexts.
7.45. Noise estimates for observations (A) to (D) are shown for the reduced data (red) and the dark sky subtracted spectrum outputted from the Background Subtraction Script (black). All strong lines were subtracted for observations (A), (C) and (D) before assessing the noise. The noise for the off-axis detector subtracted spectrum, obtained from the same script and shown in Figure 7.43, is not included as the error measured sees no significant change after this subtraction. The "error" column for the reduced data is shown in grey and predicted noise from HSpot in blue. The blue dots are those taken directly from the HSpot tool, which it rounds to one decimal place. While the solid blue line represents the values given in the SPIRE Handbook, after they have been scaled.
7.46. The GUI layout for the spiaFtsNoiseEstimate task in HIPE.
7.47. On the left, the off-axis subtracted spectrum is shown for observation (C) and the standard reduction for observation (D) is plotted on the right. The blue points are extracted directly from SPIRE photometer maps using aperture photometry and show there is a good agreement with the respective continuum.
7.48. The beam diameter in arcseconds, plotted from the SCalSpecBeamParam calibration product.
7.49. The spectra of three sources, each with a different angular size, which increases left-to-right. On the left is a point source (red), in the middle a partially-extended source (blue) and a fully-extended source on the right (grey). The data are processed with the extended-source calibration (top row) and the point-source calibration (bottom row). When the appropriate calibration is used (bottom left and top right) a good result is achieved. For the other cases fringing and features from the RSRF are introduced, as well as changes in beam size leading to discontinuity between the spectral bands. The observations used are (left-to-right) CRL618 (1342214858), M83 (1342212345) and the Orion Bar (1342204919).
7.50. Three different route causes that manifest as similar discontinuity between SSW and SLW, illustrated by standard pipeline point-source calibrated spectra plotted on the left and the corresponding corrected data on the right. The observations used are 1 AFGL4106 (1342253667), 2 Uranus (1342259588) and 3 NGC6302 (1342268288). Issue 1 can be corrected using the Spectrometer Background Subtraction useful script, whilst issues 2 and 3 can be corrected using SECT.
7.51. Two examples showing the improvement in calibration for faint observations processed with HIPE version 13 compared to HIPE version 12. The step between the bands is eliminated or significantly reduced in HIPE version 13.
7.52. The SECT GUI in HIPE.
7.53. The plots produced by SECT when doPlots is set to True (and optimiseDiameter is False) for the observation of Saturn with observation ID 1342198279.
7.54. Diagram of an example source model distribution and the reference beam, with the source distribution assumed independent of frequency. The final spectrum is normalised to include only emission inside the reference beam. If the FWHM of the reference beam is increased, more flux density is encompassed within the beam. And if set large enough, the entire source distribution will effectively be included inside the reference beam.
7.55. On the left, the Level-0.5 data for a clipped detector (SSWD1) in jiggle position 13 of obsid 1342192173 are plotted in the Detector Timeline Explorer. On the right, the final extended-source calibrated SSWD1 spectrum is compared to the unclipped SSWC1 spectrum, plotted using the SDS Explorer. The interferogram data shows truncation both near to zero path difference, which has been corrected by the pipeline, and at high OPD, which has been removed by pipeline. The reduction in OPD range causes the final spectrum to be reduced in spectral resolution (as if it had been apodized).
7.56. Displaying the Spectrum2d dataset extracted from a preCube, in the Spectrum Explorer. The RA and Dec for each spectrum is shown, and individual spectra can be plotted by clicking in the second column labeled "0". Double-clicking on the "0" column title will deselect or select all spectra.
7.57. The actual positions observed on the sky for the fully sampled example observation of the Orion Bar. SSW is on the left and SLW on the right.
7.58. The observed SSW sky positions for the example Orion Bar observation (Figure 7.57), overlaid with a WCS grid to illustrate how the individual spectra are gridded to create a spectral cube, where each grid square corresponds to a cube spectral pixel.
7.59. The standard pipeline creates two sets of spectral cubes. One set uses the Naive projection algorithm, which takes the mean sum of spectra falling within a grid square. The other set uses the convolution projection algorithm, which for each grid square, sums the Gaussian weighted contribution from spectra falling within the area of the weight kernel. This figure illustrates the final Naive cube. For the convolution projected cube: the "flux" is the weighted mean of spectra within the kernel; the "error" is the weighted error; the "coverage" is likewise weighted and uses the default minimum limit of 0.1.
7.60. SSW cube maps at 1200 GHz. Both cubes are overlaid with the FTS footprint for the central Jiggle position (pink circles) and all the positions of the ungridded spectra (green pluses). On the left is the naive projected cube and on the right is the convolution projected cube, which is smoother with no "holes"."
7.61. SSW cubes of the example Orion Bar observation. From left-to-right, these were created using the Nearest Neighbour, Naive, Convolution and Gridding Projection tasks. The corners are filled with adjacent values when using the Nearest Neighbour task, but the noise levels in the spaxels are also higher as there is no averaging. There are NaN spaxels within the map for the Naive projection, but no holes when a convolution is applied.
7.62. The coverage of the Naive projected (naive) and Convolution projected (cp) SSW cube of the example Orion Bar observation. Each spaxel contains data from 0 (black), or 4 (orange) to 12 (white) spectral scans.
7.63. All scans from all SSW unvignetted detectors for an example observation. The green scan in the centre detector (SSWD4) is an outlier.
7.64. All scans from detector SSWD4 for the example observation. Scan 3 (green) is clearly an outlier when compared to the other three scans.
7.65. All interferograms from detector SSWD4 in an area of the central burst region. The green line deviates from the other data.
7.66. Data samples from detector SSWD4.
7.67. All interferograms from detector SSWD4 in an area of the central burst region after fixing scan 3.
7.68. All spectra from detector SSWD4 after fixing two samples in scan 3 in an area of the central burst region.
7.69. Example plot generated by spireSynthPhotometry for a sparse HR observation of CRL618. The input spectra, calculated synthetic photometry and photometer values provided are all included. The SPIRE photometer RSRFs are also plotted by the task for reference.
7.70. The ratio of the intensity in extended-source calibrated photometer maps and the synthetic photometry from extended-source calibrated spectra.
7.71. A full SPSS product opened in the SDS Explorer (left) and the plot produced (right) after selecting several detectors from the "honeycomb" display of the FTS detector arrays. If additional products are opened with the SDS Explorer, any detectors selected will be added to the same plot. There are options to manage the plot or plot thumbnails in the lower panels. And the metadata can be examined by selecting the "Meta Data" tab."
7.72. A full SDS product open in the Spectrum Explorer (left) and the right-click route to open a single SpireSpectrum1d with the Spectrum Explorer (right).
7.73. Ratio of the integrated line flux in sinc-Gauss to sinc function profiles for various intrinsic source widths.
7.74. Normalised sincGauss profiles with various widths. There is a cutoff in the wings of the function that gets closer to the central peak as the line width decreases.
7.75. The Spectrum Fitter GUI with a polynomial and multiple sinc profile model fitted to a point-source calibrated spectrum. One line, the corresponding best fit model and the model subtracted residual are shown in the left panel to illustrate the line asymmetry present in FTS lines and how this compares to a fitted sinc profile.
7.76. The Spectrum Fitter GUI with a polynomial and multiple Gaussian profiles fitted to an apodized point-source calibrated spectrum. The close-up of one fitted line is shown in comparison to the fitted model and model subtracted residual. The apodized line asymmetry is highlighted by the fitted Gaussian profile.
7.77. Example of the plots produced by the Spectrometer Line Fitting script.
7.78. The combined SPIRE/PACS Spectrum1d produced by the Combine PACS and SPIRE spectra Useful script, for the example source CRL618. From left-to-right, the segments are SPIRE SLW, SPIRE SSW, PACS R1, PACS B2B, PACS B2A. The log-log scale highlights the good agreement between the instrument calibrations.
7.79. Viewing the naive projected SSW cube from the Orion Bar observation 1342204919 in the Spectrum Explorer. One frequency layer is displayed, which corresponds to the red line in the top plot and the bar to the bottom of the display panel, where the layer selected can be changed. There are several options for spaxel selection. Pressing the plus box will allow one or more single spaxels to be selected. Rectangular, circular and line regions can also be selected. The corresponding error or coverage layer can be displayed via the pulldown menu to the bottom right of the cube display window. Additional cubes can be added for comparison by dragging them onto the top plot panel. The Cube Toolbox can be opened by clicking the cube, hammer and spanner icon, which provides a number of tasks that can be applied to an FTS cube.
7.80. A SPIRE photometer observation of L1521F (left) with the matched WCS grid from the spectral cubes marked in pink. The right hand plot shows the spectra from the SLW and SSW cubes after matching the WCS (green) and after matching the WCS and convolving to an 80" beam (SLW in red, SSW in blue).
7.81. The integrated intensity plot produced by the Spectrometer Cube Fitting script after running it on the SSW CP cube of an observation of the Orion Bar, 1342228734.
7.82. Plots produced by the Spectrometer Cube Fitting script after being run on the SSW CP cube of the Orion Bar observation 1342228734. The initial fit to the central SSW spaxel is show to the left, along with the model fitted and the residual. To the right is a plot of the residuals from all spaxels in the cube. The 12CO lines are well fitted for the central spaxel, whereas some spaxels show more significant residual.
7.83. The results of overplotting contours on the output of the cube line fitting script.
8.1. Starting the SDI/SDS Explorer via HIPE.
8.2. SDI/SDS Explorer Graphical User Interface.
8.3. Control Panel Thumbnails selection menu.
8.4. SDI/SDS Explorer - Preferences Panel.
8.5. Preference panel to edit plot titles.
8.6. If a dataset has non-zero error values, both data and error are plotted when the error checkbox is ticked.
8.7. Single plot of the reverse scans 1 and 3.
8.8. Overplot of data from two different detectors in two different detector arrays.
8.9. Thumbnail images of the unvignetted SSW detectors in the spectral overlap region selected in the main plot window.
8.10. Viewing the phase of a complex spectrum.
8.11. Overplotting the real and the imaginary part of a complex spectrum.
8.12. Starting the DTE.
8.13. DTE Layout.
8.14. Array Display: on the left the visualization for the photometer using the "heat" colour scheme, on the right for the spectrometer using the "grey" colour scheme.
8.15. Plot visualized in the "Quick View Area" (left) and in the floating windows (right).
8.16. Control Panel.
8.17. Drop-down menu to select the array (on the left) and the dataset (on the right).
8.18. Slider for browsing through time across the timeline.
8.19. Plot Visualization: Overplot (Left) and Table Plotter (Right).
8.20. Color&Mask Preferences panel.
8.21. Color scale Min/Max value example.
8.22. Mask preferences: the selection of a specific mask value cause an update of the Array Display.
8.23. Mask preferences: the mask value is overplotted on the data.
8.24. Mask preferences: displaying and plotting of a bolometer when multiple mask bits are set at the same time.
8.25. Mask preferences: displaying and plotting of a bolometer when multiple mask bits are set at the same time.
8.26. Radio buttons to select the plotting options.
8.27. Mask preferences: two mask values with associated checkboxes and colour buttons.
8.28. Mask preferences: colour selector to change the colour associated with the mask value.
8.29. Selecting the Bolometer Finder Tool from an Observation variable.
8.30. Bolometer Finder Set Up Tab.
8.31. Bolometer Finder Image Tab.
8.32. Bolometer Finder Timeline Sample Tab.
8.33. Interactive Deglitching using the Bolometer Finder.
8.34. BoloFinder built-in display
8.35. Plot of all the timelines contributing to a particular map pixel
8.36. Plot of all the samples contributing to a particular map pixel where a glitch is visible in the error map
8.37. The SPIRE Mask Editor
8.38. The SPIRE Level 1 Sprectrometer Mask Editor
8.39. Clipped (truncated) samples masked in the interferogram
9.1. Overview of code required for Temporal Pool in User Scripts
9.2. Control of variable visibilty in HIPE
11.1. SPIA Workflows
11.2. Flow of data through SPIA tasks
11.3. SPIA Tasks
11.4. spiaCopyHsa
11.5. spiaLoadCal
11.6. spiaLevel1 - Main tab
11.7. spiaLevel2 - Main tab
11.8. Output of the spiaPlotPosition task
11.9. Viewing spiaL1Concat output in OverPlotter
11.10. spiaLevel05
11.11. spireMaskEditor dialogue
11.12. Plots for spireMaskEditor
11.13. spiaLevel1Repair - Main tab
11.14. spiaSaveObs
11.15. spiaSaveMaps2Fits

List of Tables

1.1. List of Useful Documentation
4.1. Non-standard Level 2.5 products.
4.2. Description of Meta Data in the SPIRE Observation Context
6.1. Description of the SPIRE Level 2 Map Products (PSW example)
6.2. Description of the Building Blocks in a Large Map Level 0.5 Context
6.3. Description of the Building Blocks in a Point Source Mode Level 0.5 Context
6.4. Summary of Photometer Observation Context
6.5. Summary of Map Artefacts
6.6. Description of Destriper Parameters
6.7. Description of Destriper Call Options
6.8. Description of Destriper Diagnostic Table
6.9. SPIRE pipeline conversion factors for point and extended sources.
6.10. Beam Areas assumed by the pipeline (α = -1)
6.11. Effective Beam Area ratios (beam correction) as function of spectral index (α)
6.12. SPIRE FWHM Parameters for 1 arcsec pixels.
6.13. Parameters for the Timeline Fitter
6.14. Parameters for Aperture Photometry
6.15. Aperture Corrections for Annular Aperture Photometry
6.16. Color Corrections for Point Sources and Extended Emission
6.17. Criteria used for selecting SPIRE observations for HiResprocessing.
7.1. Description of FTS Level-1 pipeline products.
7.2. Summary of FTS Level-2 pipeline products.
7.3. Description of the Building Blocks in the Spectrometer Level-0.5 Context shown in Figure 7.11
7.4. Summary of Spectrometer Observation Context
7.5. Summary of pointing metadata.
7.6. Gridding algorithms
7.7. spececraft velocity information
8.1. Description of SPIRE masks
8.2. Description of SPIRE masks (.. continued)