The spectroscopy pipeline cube products are Frames, PacsCube, PacsRebinnedCube, and different types of SpectralSimpleCubes. The lowest-level, raw products, are of class Frames, and as the pipeline proceeds the Frames are turned into PacsCube, then PacsRebinnedCube, and finally a projected cube (SpectralSimpleCube), a drizzled cube (a SpectralSimpleCube) or an interpolated cube (a SpectralSimpleCube). The science-quality products are the rebinned, projected, interpolated or drizzled cubes: which cube is the most suitable for science analysis depends on the type of observation. More guidance can be found in Chapter 6 and in the PDRG chp. 1 and chp. 9.
In addition to the cubes, the SPG also produces spectrum tables, containing the data of (i) the rebinned cubes, and (ii) the central spaxel spectrum and point-source calibrated spectra. These are also explained here. These products are not part of the interactive pipeline processing, although there is a useful script in HIPE to re-create them (see Chapter 5).
The science data-products at these lower levels are Frames and PacsCubes, the HPSFIT[R|B] and HPS3D[R|B] as discussed in Section 2.3. The datasets that make up the Frames and PacsCubes can be seen by clicking on the + next to any of the HPSFIT[R|B] or HPS3D[B|R] listed in the Data tab of the Observation Viewer, and then clicking the +[number] therein:
Figure 3.1. The datasets in a Frames (SPG 13.0 and higher), and the top part of the Pacs Product Viewer is visible in the Data viewing tab
Figure 3.2. The datasets in a PacsCube (SPG 13.0 and higher) and the top part of the Spectrum Explorer is visible in the Data viewing tab
The 3D data arrays of signal (Frames) or flux (PacsCube), wave, ra, dec. You can browse through these by clicking on them in the Data tab.
The BlockTable is an organisation table of data-blocks: click on it to see a table listing chopper, nod, grating postions and other information, organised by blocks but also including data index and time information.
The Status is a table containing data about where the different parts of the instrument were set to (grating position, nod position, chop position, datapoint number, ra, dec, time, temperatures, velocity...), for each time-point in the observation. These data are used in some of the pipeline tasks.
The Masks contain various masks created by pipeline tasks that flag data-points as bad for different effects, e.g. saturation. These are 3D boolean arrays. A flag value of 0 means "is not bad" and 1 means "is bad". These masks can be viewed along with the science data using the Pacs Product Viewer, via a right-click on an individual Frames or PacsCube which contain the masks. See the PDRG (e.g. chp. 10) for more on working with masks.
To inspect one of the Frames or PacsCubes in a viewer, a single click on the product while in the Observation Viewer will open the default viewer (or double-click to get a list of viewer and action options). For the Frames this is the Pacs Product Viewer, for the PacsCube it is the Spectrum Explorer. How to use these viewers is explained in the PDRG (chp. 10): the screenshots above show the top view of both of these viewers.
At level 0 and 0.5 there are also contexts that contain engineering, housekeeping, and auxiliary data. These are used by the pipeline but are otherwise not interesting for the astronomer.
The Level 2 products are the end products for all observing modes except the unchopped range. For this mode, the on-source and off-source data come from separate observations (unless no off-source observation was requested). They are each reduced and the final products placed in their Level 2s, then the Level 2 off-source rebinned cubes are subtracted from the Level 2 on-source rebinned cubes and all subsequent cubes made from those: these are all placed in the Level 2.5 of the on-source observation. The product names have a "BS" (background subtracted) appended. For some observations (pointed, chopNod, and taken to cover the entire SED) there is also a Level 3 product.
The cubes at the higher pipeline levels are: PacsCube, PacsRebinnedCube, and the three types of mosaic cube, of class SpectralSimpleCube and called projected cubes, drizzled cubes and interpolated cubes. These are all contained in the ObservationContext in contexts called HPS3D[R|B] (SlicedPacsCube), HPS3DR[R|B] (SlicedPacsRebinnedCube), HPS3DP[R|B] (ListContext of projected cubes), HPS3DD[R|B] (ListContext of drizzled cubes), and HPS3DI[R|B] (ListContext of interpolated cubes). All except the PacsCube all also have a background subtracted version for the unchopped range scan observations (with "BS" in the name), and some also have an "equidistant" version (with "EQ" in the name): these are explained later. The PacsCube in Level 2 is that from Level 1 but with a flatfielding and a flux calibration applied—the units are Jy/pixel[which here means spaxel]: see Section 3.2.4 to learn more about this flux calibration. This cube is the only product at Level 2 that is not intended to be used for science.
Each ObservationContext gotten from the HSA with SPG version 14.x contains the PacsCubes and PacsRebinnedCubes, and depending on the AOT, the following mosaic cubes:
drizzled and projected cubes for Nyquist and oversampled mapping lineScans;
interpolated and projected cubes for Nyquist mapping rangeScans, and only projected cubes for oversampled mapping rangeScans;
projected and interpolated cubes for pointed and undersampled mapping/tiling line and rangeScans.
More detail can be found in Section 2.3.3. One type of cube (depending on the observing mode) is provided additionally with an equidistant wavelength grid (the other cubes have a grid that scales with wavelength), these "equidistant cubes" also being the PACS spectroscopy "standalone browse products" (see Chapter 5 for more detail). In the ObservationContext the letters "EQ" are added to the HPS name of the cube context.
To see the cubes contained within any ListContext, or to see the datasets within the cubes, click on the appropriate product in the Observation Viewer Data tab, e.g.,
Figure 3.3. The datasets in a Level 2 PacsRebinnedCube and a projected cube, taken from an SPG 13 observation. The Spectrum Explorer is open on the cube in Data viewing tab
The screenshot shows the datasets of the rebinned and projected cubes. The datasets of all the cubes include:
ImageIndex and WaveGrid: a table of the wavelengths. These are the same for each spaxel. Because the bin sizes scale with wavelength, this spectral information is carried in a dataset rather than via WCS keywords. This is found in the projected, interpolated, and drizzled cubes.
New to SPG 14.2 is a wcs-tab. This contains the same information as in the ImageIndex, although organised differently. It is provided to accompany the WCS (world coordinate system) of cubes that do not have an equidistant wavelength grid (i.e. all the PACS cubes except those with "EQ" in their name). If the ctype3 of the WCS is "WAVE-TAB", that means that the wavelength information is provided in the a "lookup table" called "wcs-tab". Note that this dataset is also present in the cubes that do have an equidistant wavelength grid (the "EQ" cubes); rather than remove the dataset the values in it are replaced with those of the equidistant wavelength grid ()as the cdelt3 for these cubes is "Wavelength", this lookup table is however redundant.
Image: is the flux data (units of Jy/pixel [though actually Jy/spaxel]) for the entire 3D array. This is found in the rebinned, projected, interpolated, and drizzled cubes.
Error: are the errors of the fluxes, as propagated through the pipeline. This is found in the projected, interpolated, and drizzled cubes. The source of these errors is the stddev array of the preceeding rebinned cubes (for projected or interpolated cubes) or as created from the preceeding PacsCube (for drizzled cubes). The stddev array is propagated by the same algorithm that the image dataset is subjected to by the mosaic tasks; note that for interpolated cubes the the interpolation error is not included in this value (for the other algorithms, no extra algorithmic error is necessary). See the PDRG chp. 7 for more information.
Coverage: is a 3D dataset found in the re-mapped cubes: the projected and drizzled cubes, which are created by re-mapping a raster sequence of rebinned cubes into a single cube. The dataset contains one value, per wavelength and per spaxel of the re-mapped cube, that is the sum of the weights of the spaxels of the rebinned cube(s) that contributed to each new spaxel of the re-mapped cube.
Flag: This is found in the rebinned, interpolated, and drizzled cubes. When these cubes are created, the data masked as bad in the Masks of the PacsCubes (which the rebinned and drizzled cubes are both created from) are excluded. In the flag array is information about which datapoints were affected by which flags. This can be useful if you wish to look for saturated data, since otherwise these appear as NaN (blanks) in the spectra. See the PDRG chp. 10.10 for a short summary of working with the flags.
Stddev and exposure: these datasets are found in the rebinned cubes. The PacsRebinnedCubes are created from PacsCubes: the spectrum of each spaxel of the rebinned cube is created by combining the multiple spectra contained in each spaxel of the PacsCube. The resulting spectra are combined on a wavelength grid created from the wavelength datasets in the PacsCubes. The stddev is the scatter in the PacsCube datapoints that fall into each wavelength bin of the rebinned cube. The exposure is a measure of the number of times a wavelength bin in the rebinned cube was visited.
Ra, dec: the values of RA and Dec for each spaxel and wavelength. These are found in the rebinned cubes—in the other cubes the spatial grid is regular and the sky coordinates are found the WCS.
QualityControl: to be ignored.
Three types of spectrum table are provided in the SPG-reduced data.
Rebinned cube tables. the data of the rebinned cubes, which are placed in a product of class PacsSpecTable: one table per line id and within each table you will find the data of all the raster pointings together. These tables are in Level 2 and are called HPSTBR[R|B], and those at Level 2.5 are HPSTBRBS[R|B].
Point source spectra. For all pointed observations, the spectrum taken from the central spaxel together with point source calibrated spectra created by the task extractCentralSpectrum (see chp. 8 of the PDRG) are wrapped in a table of class is PacsCentralSpectrum. Three point-source calibrated spectra are provided for chopNod observations (c1, c9, c129) and two for unchopped observations (c1 and c9). These tables are in Level 2/2.5 and are called HPSSPEC[R|B]/HPSSPECBS[R|B]. Note that no assumption is made that the target is a point source, that it is located in the central spaxel, or that the entire flux of the central region is of the point source alone—all of these are necessary for these products to be useful.
Point source SED spectra. For all chopNod pointed observations that were taken in "SED" mode (used to obtain the spectrum of the entire PACS spectral range in two or three separate observations), the red and blue PacsCentralSpectrum tables and from all observations that contributed to the coverage of the spectral range are concatenated into a single table. These tables are in Level 3 and are called HPSSPEC.
The dataset of these three tables is: are:
Spectra: in the PacsCentralSpectrum and PacsSpecTable products, this dataset (of class TableDataset: which can be exported to FITS separately) contains the spectral data given in columns. When viewing this dataset in HIPE, the column titles indicate what they are and their units.
The rebinned cube tables contain much data which we explain here. The columns include the red and blue data for all spaxels and, for mapping observations, for all raster positions in one table. A separate table is provided for each wavelength range requested for each observation. The length of the table is dictated by how many spectra there are, since all the data for each wavelength point of each spectrum fills its own row. The columns are:
Raster Line and Raster Column: the position within the raster sequence
Band: the filter band (the same for all data)
Spaxel Number, Row, and Column: spaxel index (0—25) and row (0—4) and column (0—4) coordinates
Wavelength, Flux, Flux Error
as can be seen in this figure:
So, for a cube with spectra of 10 datapoints, in a raster with 4 pointings, the order of rows is:
0—9: ra, dec, wave, flux, error for raster line 1, column 1, for spaxel row 0, column 0, index 0
10—19: ra, dec, wave, flux, error for raster line 1, column 1, for spaxel row 0, column 1, index 1
240—250: ra, dec, wave, flux, error for raster line 1, column 1, for spaxel row 4, column 4, index 24
250—260: ra, dec, wave, flux, error for raster line 1, column 2, for spaxel row 0, column 0, index 0
These tables can be saved to FITS or as an ASCII file to disk. In the FITS file, the header of extension 1 contains the names and units of the columns; in the ASCII file these information are given at the beginning of the file.
The spectrum tables are less complicated, as the screenshot shows:
Figure 3.5. The PacsCentralSpectrum product. Columns are: index, segment, wavelength, band, central spaxel flux and error, "c1" flux and error, "c129" flux and error, and "c9" flux and error. If a column is NaN it means that spectrum is not provided for this type of observation
The first column of flux data (and its adjacent error column) is taken from the central spaxel, and its units are Jy/pixel [i.e. Jy from that spaxel]. The column called "pointSourceFlux" contains "c1" from extractCentralSpectrum and is the central-spaxel spectrum with the basic point source correction applied, and its units are Jy. The column "pointSourceCen3x3Flux" is "c9" from extractCentralSpectrum and contains the point-source corrected flux of the central 9 spaxels which includes a seconder-order flux correction for pointing jitter. These two are provided for chop-nod and unchopped observations, and generally c9 is better than c1 (with the possible exception of very faint sources). The column called "pointSourceScaledFlux" is the spectrum "c129", which is c1 scaled to the flux level of c9: giving the best of both worlds (SNR and flux), and is provided for all chop-nod observations (it is not suitable for use with unchopped observations). The values in the error columns are taken from the stddev dataset of the rebinned cube from which this table was created. For more information about these point-source spectra, see the PDRG.
It is important to note that no assumption is made that the target in the observation is a point source, that it is located in the central spaxel, or that all the flux in the central region is from the point source alone—all of which are preconditions for the output of this task to be useful. It is up to the end-user to determine if these are the case.
The names of the products and their datasets found at all the levels of the pipeline are listed, in alphabetical order, in the table below.
Table 3.1. The contents of PACS pipeline products from Level 2,2.5,3
|Name||What it is / task that made it||What product it is found in|
|BlockTable||organisation table of data blocks / findBlocks and added to by a few other tasks||Frames, PacsCube|
|coverage||a measure of the amount of data from the input PacsRebinnedCubes that became data of the SpectralSimpleCube, one value per wavelength and per spaxel / specProject and drizzle||SpectralSimpleCubes (projected and drizzled)|
|error||values of the flux error (Jy/pixel), propagated from the stddev dataset of the input (1) PacsRebinnedCube and modified by the specProject (2) PacsCube and modified by drizzle||SpectralSimpleCubes: (1) projected, interpolated; (2) drizzled|
|exposure||a measure of the amount of data from the input PacsCube that became data of the output PacsRebinnedCube, one value per wavelength bin and per and spaxel / specWaveRebin||PacsRebinnedCube|
|flag||an HCSS-compliant array that contains values based on the Masks of the PacsCubes that created the (1)PacsRebinnedCube (2) SpectralSimpeCube via the pipeline task drizzle. For information on the state of these masks included in this dataset, see its Meta data / specWaveRebin||(1) PacsRebinnedCube and (2) SpectralSimpleCube|
|flux||the flux dataset in units of Jy/pixel (where a pixel is actually a spaxel) / specFrames2PacsCube||PacsCube|
|image||the flux dataset in Jy/pixel (where a pixel is actually a spaxel) / specWaveRebin||PacsRebinnedCube, SpectralSimpleCubes created by specProject, drizzle and specInterpolate|
|ImageIndex||the 1d dataset of the wavelengths in the cube / specWaveRebin||PacsRebinnedCube, SpectralSimpleCubescreated by specProject, drizzle and specInterpolate|
|Mask||contains all the individual masks, 0 and 1 values for is not bad andis bad data for different instrumental or data effects / present from the Level0 and added to as the pipeline proceeds||Frames, PacsCube|
|qualityControl||to be ignored||PacsRebinnedCube|
|ra, dec||RA and Dec datasets / specAssignRaDec||Frames, PacsCube, PacsRebinnedCube|
|signal||signal dataset, in units of counts / already there at Level0 but changed by specConvDigit2VoltsPerSecFrames and rsrfCal||Frames|
|Status||a table of the status of PACS during the entire observation / there from the Level0 and added to as pipeline proceeds||Frames, PacsCube|
|stddev||the standard deviation dataset, created from the data of the input PacsCubes / specWaveRebin||PacsRebinnedCube|
|wave||wavelength dataset in micrometres / waveCalc||Frames, PacsCube|
|waveGrid||the wavelength grid dataset created by wavelengthGrid and used to create the PacsRebinnedCube / specWaveRebin||PacsRebinnedCube|
|wcs-tab||a lookup table of the wavelengths in the cube / specWaveRebin||PacsRebinnedCube, SpectralSimpleCubescreated by specProject, drizzle and specInterpolate and present also in the equidistant versions of these cubes|
|spectra||the TableDataset of spectral data found in the spectral tables provided at Level 2/2.5/3||PacsCentralSpectrum and PacsSpecTable|
Two flux calibration schemes are used in the SPG pipeline and the telescope normalisation interactive pipeline (upon which the SPG pipeline is based) for all chop-nod observations. Both schemes are based on the "telescope normalisation" method, which uses the telescope spectrum as a flux calibrator, but the derivation of the telescope spectrum is slightly different. One scheme is used when creating drizzled cubes and the other scheme is used when creating all other cubes, the rebinned, projected, and interpolated cubes. This became necessary to correct an oversight in Track 13 that left the drizzled cubes incorrectly calibrated. The reason that this "drizzle" scheme was introduced is because to create drizzled cubes, the preceding PacsCubes need to be flux calibrated, but the original scheme did not work on these products. We therefore used a previous scheme that does work on these products, but for which the derivation of the telescope spectrum differs slightly. The two schemes produce very similar results: the absolute level is the same, the noise is only mildly higher for the "drizzled calibration", and only very slight differences in the shape of the noise and extremely faint lines may be found.
This affects chop-nod line scan AOTs taken in the Nyquist and oversampled mapping mode (see Table 2.4 for mapping mode parameters), i.e. all observations for which drizzled cubes are made by the pipeline. Unchopped observations have their own calibration scheme (that based on the internal calibration sources) are not affected. For more information on the flux calibration of PACS cubes, see the PACS Handbook (available from the HSC web-pages).
In order to provide Level 2 flux calibrate cubes for observations downloaded from the HSA for all, the Level 2 PacsCubes have had the "drizzle" calibration applied. However, the rebinned cubes, which are created from the PacsCubes, and the subsequent projected and interpolated cubes, are all are created from the PacsCubes created before that calibration is applied. At the end of the Level 2 pipeline these subsequent cubes are then flux calibrated using the standard (non-drizzled) scheme.
The consequence of this is that if you were to run the Level 1 to 2 pipeline and rebinned cubes and mosaic cubes from the PacsCubes taken directly from Level 2, you will not get exactly the same result as the rebinned cubes in the SPG Level 2. If you do want to do this, you should run the pipeline script from the beginning of Level 1 instead.
Some comments that apply to all mosaic cubes and rebinned cubes.
All rebinned and mosaic cubes will have some NaN datapoints. Here we explain where, in most cases, these comes from.
The projected and interpolated cubes are created from Level 2 rebinned cubes. The rebinned and the drizzled cubes are created from Level 2 "pacs cubes". The "pacs cubes" are hence the root of all Level 2 science cubes. Now, the "pacs cubes" carry all data-points within them, including those masked as bad. The Masks datasets in these cubes carry the information about the good/bad state of each data-point. When the "pacs cubes" are turned into rebinned cubes and from there into interpolated or projected cubes, or when they are turned directly into drizzled cubes, the data masked as bad are excluded—instead, they become NaN values in the output cubes.
In practice this means that where there are saturated data in the "pacs cubes", there will be NaNs in the spectra of the rebinned or mosaic cubes: saturated spectral lines will hence appear to have "gaps" at their peak, or very bright continuum regions will have large gaps in them. Fortunatley, very few PACS observations are saturated. NaNs will also be found in the rebinned and mosaic cubes at the edges of spectral ranges of SEDs, where the spectral sampling is poor, and for some of the blue SEDS, where the spectral sampling was commanded for the red camera and hence is poor for the blue camera. In this case every other data-point is a NaN.
To allow the user to figure out if any particular NaN data-point came from a saturated data-point, the rebinned and all mosaic cubes have a 3D "flag" dataset. The values in this dataset indicate which masks (including SATURATION and RAWSATURATION) were activated or not, for each data-point, and hence NaNs data-points which are in fact due to saturation can be identified. In practice, unless using the tools in HIPE developed to allow you to inspect this flag dataset (as explained in sec. 10.11 of the PDRG) it is very difficult to identify which particular flag was activated or deactivated for any data-point. Our recommendation is that you simply ignore this flag dataset, remembering that saturated spectral regions and regions of poor spectral sampling will be blank, and this can look like "holes" in the cube if inspecting a single wavelength layer of that cube.
NaN data are propagated into the mosaic cubes differently by specProject, drizzle, and specInterpolate because of the different way their algorithms work. For all tasks, the NaNs are propagated if one corner of a spaxel is neighbouring a NaN data-point, when the projection/drizzle/interpolation is done. The tasks specProject and drizzle have only about four neighbours for each spaxel, and hence a NaN spectral region is usually of the same spatial extent as in the input cube (e.g. just over one 9.4" spaxel). For specInterpolate, however, there are about eight neighbours and if only one is a NaN, the output spaxel is a NaN for that spectral layer. This results in larger spatial regions that are affected by the NaNs.
Finally, note that since specInterpolate uses an interpolation to create its mosaic output, the edges of the FoV are not extrapolated. The result of this is that for small spectral fields (e.g. a pointed observation) the spatial extent covered by an interpolated cube is slightly smaller than that from a projected cubes.