Data Products Known Issues

HCSS, SPG, and HIPE

The whole suite of Herschel software is known as the Herschel Common Science System (HCSS). This encompasses all the software aspects related to the mission: automatic pipelines, spacecraft calibrations, etc. As data are retrieved from the spacecraft they are ingested in the Herschel Science Archive (HSA) and processed with the current official version of the pipeline. That means that, at any given time, different data in the HSA may be reduced with different pipeline versions. The pipeline version is listed in the HSA GUI as SPG vX.X.X, where SPG stands for Standard Product Generator. The SPG version is available as the header keyword 'creator', in the data .fits files. Every HCSS version, the whole HSA is bulk re-processed with the same up-to-date version of the pipelines.

From the user point of view, the most important piece of the HCSS system is the Herschel Interactive Processing Environment (HIPE). HIPE allows the astronomer the possibility of inspecting the data and re-process them, if the results from the automatic pipeline are not good enough for his/her purposes. Because HIPE is part of the HCSS, the latest version of HIPE will have the most up-to-date pipeline, calibrations and documentation available.

For example, some data in the HSA may be marked SPG v13.0.0, which means that they were processed with that HCSS version (see metadata of the observation to find pipeline processing used for current product in the archive). If the current HCSS version is 13.0.0, HIPE is informally called HIPE 13.0. If you re-reduce the data with this HIPE version with the default parameters, you will get the result that would be produced with the official 13.0.0 pipelines at the Herschel Science Centre.

The latest version of HIPE can be obtained here. Known HIPE issues/problems/bugs are detailed here.

In what follows, we provide a summary of the known issues that you may encounter when inspecting data processed with the automatic pipelines SPG versions 6.1 to 14.2. Most can be resolved by running the pipelines within HIPE and optimizing their parameters as explained below.

Note that some of this information can also be found in the quality report of the observation (QC Report) and as metadata with the FITS keyword "PCAVEATS".

TABLE OF CONTENTS

METADATA AND FITS KEYWORDS

• Message about DATE-OBS: Some internal HCSS metadata are renamed by the HCSS software when translating to FITS. One special case is startDate which gets written to FITS as both DATE-OBS and DATE_OBS. This is done for compatibility with legacy Spitzer software (namely MOPEX).

PACS Photometer (scan mapping)

General notes

• Important note: PACS maps from any map-maker are in essence differential maps: the absolute level is undefined due to the dominating telescope background removed by map-makers. Hence it is not unusual if your background level is negative.

• To produce the highest quality maps, you may want to consider re-processing or fine-tuning the observations with the latest HIPE User Release. Maps available from the HSA are created within a bulk processing framework, and a reprocessing while fine-tuning the mapper parameters, according to the characteristics of the observed sky region, could enhance the quality of the final maps. This is especially important for faint/tricky fields (e.g. those with point and extended sources intermingled, or where there is very little "background" around the source which the mappers could use as a baseline). However, you should first check for the presence of a Level 3 (which maps are deeper than Level 2.5) or for HPDPs (which have been produced for some tricky regions).

• JScanam
  • JScanam (an HCSS implementation of the Scanamorphos IDL map-maker) maps are available in level 2.5 and level 3 as of SPG 12 onwards, for scan-mapping observations that had a scan and cross-scan observation.
  • The JScanam ipipe script to process the maps is available in Hipe 11 onwards. Significant improvements have been achieved in the latest HIPE release, in terms of memory requirement, processing speed, and final map quality. Within the HSA, the JScanam Level2.5 maps are used as the Stand-alone Browse Products for the PACS Photometer scan-maps.
  • In the JScanam pipeline for the the generation of the Level 2.5 maps, a random number generator is used for filling the masked signal pixels (within the scanamorphosNoiseSpectrum task, called internally by the scanamorphosIndividualDrifts task, just before the final projection). The random number generator is inizialized at each run and this means that calling twice the task on the same input frames (i.e. by running the same SPG version twice), the maps finally generated are slightly different. This difference can be considered negligible (remember that none PACS maps is absolute calibrated), being the percentage differences pixel-to-pixel of the order of few 0.1%, and well below the level of the standerd deviation map.
  • Non-exposed pixels in JScanam maps are given a value of zero. However, it should be born in mind that PACS maps are differential maps, where the absolute level is undefined as it is removed by the map-making process. Hence, zero as a value could appear in the maps even for exposed pixels. To avoid problems when making measurements from JScanam maps, the astronomer can make use of the coverage map to first mask out the non-exposed pixels.

• Unimap
  • Unimap (a GLS mapmaker) maps are available in level 2.5 and level 3 as of SPG 13 onwards, for scan-mapping observations that had a scan and cross-scan observation.
  • The Unimap interactive pipeline script to reprocess the maps is available in HIPE 13 onwards. Unimap is Matlab software and it is invoked by a jython task. When running the interactive pipeline script, the Unimap release must be locally installed (instructions are given at the beginning of ipipe script).
  • Unimap maps into HSA were generated with the release 6.4.4, while interactive script is compatible with Unimaps versions 6.5.3 and below.
  • Distortions introduced by the Unimap GLS algorithm are generally removed by the Unimap post-processing or by the Pixel Noise compensation (see the PACS Data Reduction Guide for photometry). If you are not happy of the SPG archival results, you can run the interactive script by fine tuning the parameters.
  • In some cases, drifts due to the calibration blocks persistence and strips due to strong saturated pixels can be observed, because not properly corrected by the Unimap pre-processing.
  • For the pairs of observations (obsIDs) reported into this list unimap_special_cases.txt, the SPG 14.2.0 fails because of a bug in the Unimap release 6.4.4. They are processed with SPG 14.2.1 by using the same Unimap release (6.4.4), but by modifying the values of some Unimap parameters (wrt the default ones) according the fourth (name of parameter) and fifth (adopted value) columns of the list unimap_special_cases.txt.
  • For very short scan-mapping observations (scan+cross-scan, but with fewer than 250 frames taken in each), no Unimap maps are created. Only HPF+PP and Jscanam maps will be found in Level 2.5

• Highpass-filtered maps (and their intrinsic limitations)
  • High-pass filtered maps in SPG14 are available at level 2 and level 2.5. The extended emission is filtered out in the maps since a rather small filter width is used to remove 1/f noise stripping: this is done to allows us to obtain the best sensitivity for point-sources. Bright sources are masked out during the highpass filtering, hence their flux is not too much affected by filtering. But faint sources are not or are inadequately masked out by the SPG processing, hence the flux loss for these point-source can reach up to 20--30%. Re-processing your data, using customised masks, is recommended here.

• MADmap
  • MADmap -- a GLS (Generalized Least Square) map-maker -- maps are no longer created by SPG13 and beyond. However, the pipeline script to do this yourself is provided.
  • The background MADmap map is sometimes affected by wiggles, arising from the drift correction, but the overall maps have been reliable since SPG 10.
  • A post-processing removes artifacts around bright sources, but slight artifacts are still seen around the very brightest sources.

• Error maps
  • The error map in the high-pass filter branch is derived from the coverage map. For this, an empirical calibration as a function of high-pass filter window size, map pixel size, and the pixfrac parameter has been worked out which also takes the effects of correlated noise into account.
  • For JScanam maps, a comprehensive error map cannot be easily constructed. However, a standard-deviation map of the flux values belonging to the same sky pixel is provided. It is obtained in the final projection of pipeline chain by taking into account the frames corrected for drift and electronic noise effects (de-striped frames)
  • Unimap takes into account the full noise propagation along the pipelines by providing a noise map that combines the contributions of the pixel noise and the electronic noise. It can be assumed as the proper error map if the pixel noise compensation is applied, although it represents a good approximation when the compensation is suppressed and the post-processing applied.

• Pointing/Astrometry issues
  • SPG14.0.1 products acquired during the Observing Day 150 are affected by major pointing issue. The problem is fixed for the SPG14.2 maps.
  • SPG14.0.1 maps in the early stage of the mission (Operational day < 113) can be affected by pointing issues because of a not properly handled of the star tracker misalignment. The problem is fixed for the SPG14.2 maps
  • The measured Herschel APE (Absolute Pointing Error) on pointed observation was improved throughout the mission down to 1 arcsec (1-sigma) on most of the fields/observations reprocessed with the latest SPG version. If your source is still severely mis-pointed you should contact the HSC helpdesk.

• Quality issues, and the quality flags
  • One matrix (fully half of) of the red channel array was lost at the end of the mission, so from OD1375 onwards this matrix is masked out automatically in the SPG processing.
  • The quality flags in the quality context ("quality" or "qualitySummary") inside the observation context are meant for HSC/ICC internal evaluation of the quality of the products: when an observation had a serious quality problem, the PI of the program would have been contacted about it. For archive users, only the information in the "qualitySummary", when available, is useful.

• Other calibration/pipeline problems/needs
  • The mosaicTask implemented into SPG 14.0.1 is affected by a mistake in the generation of the error maps. It means that the error maps belonging to the Level2.5 High Pass Filtered maps and to all the Level 3 maps are wrong. This issue was fixed for SPG 14.2 products.
  • For more information check the PACS Instrument and Calibration webpage and/or contact the HSC helpdesk.

PACS Spectroscopy

Issues affecting only one SPG/HIPE release

• Use of the Unchopped range scan "...with transient correction" interactive pipeline script.
  • The release of the interactive pipeline script in HIPE 14.2 (and possibly 14.0.1) for unchopped range scans (called "...with transient correction" in the Pipelines menu) contains an overactive smoothing that can clip away spectral lines. We strongly recommend not using this script for your reductions. The script from HIPE track 15 (which is currently the development track) can be used instead. In addition, this is only recommended for use with range scans. This does not affect the SPG reductions of these observations.

• Warning for a subset of unchopped range scan PACS spectroscopy observations in HIPE/SPG 14.0.1.
  • PACS spectroscopy observations reported in this text file are affected by a problem that occurred at end of the pipeline processing for SPG 14.0.1. For this observing mode, the off-source observation is a separate obsid to the on-source observation. The on-source and off-source obsids are processed separately and the results placed in the Level 2 of the observation data. Normally the off-source data are then subtracted from the on-source data and these results placed in the Level 2.5 of the on-source observation. Unfortunately, for a subset of unchopped range scans this did not happen, but instead the on-source data were subtracted from the off-source data and placed in the Level 2.5 of the off-source observation. However, this will be fixed in the final bulk processing (early 2017) and the SPG 14.2.2 products will be correct.

• Flat fluxes for certain spectral ranges
  • The pipeline task extractCentralSpectrum is used to created point-source calibrated spectra for pointed observations, working on the rebinned cubes to do this. This task does a spectral interpolation over NaNs (this is to avoid having jumps in summed spectra where one spaxel may have a NaN). The output spectra are improved as a result of this, however the central 9 spaxels of the input rebinned cubes are modified by this interpolation. The result of this modification is that for the rebinned cubes, the fluxes in the spectra in the ranges that were not flatfielded are nearly flat. In SPG 14.0.1-reduced observations, this affects those of the the pointed range scan mode, and specifically in the spectral ranges affected by leakage in the different bands. As an additional side effect, the interpolated cubes (which are created from the rebinned cubes) also show these extended flat-valued spectral ranges at some central locations. However, this has been fixed in SPG 14.2.

• Using the pipelines in HIPE 15 and 14 and comparing to SPG 14
  • Processing an observation in HIPE 15 and HIPE 14.2 will produce the same results. However, marginal differences may appear in these processing results compared to the SPG 14.2(.0,1,2) gotten from the HSA. There is no fix for this, but the impact on the science data is insignificant.

• SPG 14.2.0 red leak range
  • With products processed by SPG 14.2.0, for observations with a small range falling mostly in the red leak region (>190 microns) but with a short stretch shortward of 190 microns, the entire Level 2 red spectral range was lost. This has been fixed in 14.2.2 and only the spectral ranges that fall within the red leak are suppressed.

General notes

• The SPG results for most observations are very good; however....., there will always be a subset of observations (e.g. those which are very faint or extremely bright, off-centred point sources, semi-extended sources, taken in the red leak region, with contamination, mapping observations for which you wish to change the cube spaxel sizes...) which will benefit from an interactive pipeline processing in HIPE, with the latest HIPE User Release. The first two chapters of the PACS Data Reduction Guide for spectroscopy give more information about the need to reprocess, and about what to do with HSA-obtained cubes before using them for science.

• Point and semi-extended sources
  • Point source corrections must be applied for a correct spectrum of a point source to be produced. This simple correction can be done easily in HIPE.
  • The same advice also applies to semi-extended sources. However here you need a model of the source shape to proceed with the correction task in HIPE.
  • In both cases the information is provided in the PACS Data Reduction Guide and in tasks and scripts in HIPE.

• Spectral leakage
  • The order selection filters of the PACS spectrometer have a steep but not perfectly vertical transmission profile at the cut­off wavelengths of the spectral bands. PACS spectra near the band borders of bands R1, B3A and B2B are therefore affected by higher- or lower-order wavelengths leaking into the spectra -- both continuum and spectral lines! Consult the PACS Calibration Document for more information on the leakage regions.
  • In SPG 14.2 all observations will be cut off below 55 microns to avoid including this badly-calibrated spectral range in the blue.
  • In SPG 14.2 all observations will be cut off above 190 microns, to avoid including the red leak region. However, it is possible to reduce these data with small changes to the standard pipeline script, and this is explained in the PACS Data Reduction Guide. Note that after this special interactive pipeline processing, the line fluxes will be corrected for but the continuum will still be incorrect.

• Limitations on absolute spectrophotometric accuracy
  • The PACS spectrometer flux calibration accuracy is limited by detector response drifts and slight pointing offsets arising from the standard 1.2" (1-sigma) pointing error occurring within each and every observation. These limit both the absolute flux accuracy and relative accuracy within a band. Various pipelines deal better with these than others (see the advice in the PACS Data Reduction Guide for spectroscopy) but they can never be entirely negated. Hence the calibration uncertainty for any particular observation is a combination of the general calibration uncertainties (given in the PACS Observers Manual), the noise on the spectrum (explained in more detail in the PDRG chp 7.6), and the "activity" during any single observation.

• Flux calibration for all unchopped observations
  • The absolute flux calibration of the PACS spectrometer is based on observations of flux calibration standards using chopped spectroscopy modes. There are hints of systematic differences in the response scaling between chopped and unchopped mode due to response transients within the chopping pattern. The continuum level for unchopped observations has a greater uncertainty as a consequence, and could be off by 10--20 Jy.

• Unchopped Range scan observations
  • Unchopped range scan observations consist of one on-source obsid and one (or potentially more) off-source obsid. In the pipeline, one off-source obsid is processed fully to Level 2 and then subtracted from the Level 2 of the on-source obsid, to create a Level 2.5. It was up to the observer to decide where the off-source observation was on the sky, and how long and how often it was observed. Some observers chose not to have an off-source observation, which means that the telescope background continuum spectrum is not subtracted from the data and there is no Level 2.5 for the on-source obsid. The list of on-source--off-source pairs can be found on this page: http://www.cosmos.esa.int/web/herschel/data-products-overview.

• Check for contaminating flux in chop-off positions for chopNod AOTs
  • To check for the presence of contamination from unwanted astronomical sources in the off positions of chopNod mode observations, you can use a Split On-Off pipeline script to produce an off-source and on-source cube. These cubes can then be compared to each other to check for contamination in spectral lines or by strong continuum emission, e.g. by over-plotting the respective spectra. Note that the on-source and off-source cubes produced by this task will not allow you to detect faint levels of contamination because wiggles from the RSRF are not removed by this process. For faint targets (line peak-to-continuum emission ~<5-10 Jy) you should also check the differential signal between the nodA and B on-source cubes. This is documented in the PACS Data Reduction Guide for spectroscopy. In this guide you can also find advice on checking for contamination in the unchopped mode observations, which is done either by comparing companion observations (unchopped range) or with a small script provided in the PDRG (unchopped line).

• Missing data within an observation
  • Sometimes one or both cameras are missing from science-levels of an observation, or some of the spectral ranges are not provided. This occurs when entire spectral ranges have been masked out: the affected slices can no longer be processed and are discarded. If all the data within a camera have been discarded, then the entire camera will be missing. The most common reason for masking out entire products is because they fall in an "out of band" region, most commonly below 55 microns and above 190 microns. Observations which are heavily saturated may also have spectral ranges entirely masked out. More information can be found the PACS Products Explained (available with the HIPE help). The qualitySummary may also contain comments about "missing slices/cameras".

• Second-pass ghosts
  • A second pass in the optics of the PACS spectrometer can cause a ghost image to appear on most spaxels (but never in the central spaxel). If a source located in one of these "originating" spaxels shows a strong spectral line (typically an atomic fine-structure line), then a weak, broadened line can be seen at an offset wavelength in its corresponding "destination" spaxel affected by the 2nd-pass ghost. The peak flux of this line is typically ~5% of the peak of the originating line. The most prominent ghost is the 122 micron feature, which originates from the usually strong CII+ 157.7 micron line. A list of strong ghosts and an image showing the directions of the projected passes on the 5x5 IFU footprint can be found in the PACS Calibration Document.

• 62 micron dip
  • A dip can sometimes be seen at spectra between 62 and 63 microns. This is a filter feature. Its appearance depends on the angle at which the light from the sources goes through the filter, and so it depends on the source position and its spatial structure.

• Spectral line profiles: the skew
  • As for any slit-spectrograph, if the incoming light beam is neither homogeneously nor symmetrically illuminating a spaxel, then line profiles may be distorted from the ideal Gaussian shape. In case of a point-source observed with PACS, the spectral line profile develops a skew, increasing as the offset of the photocentre along the instrument Z-axis (perpendicular the slit direction) does. Examples of skewed line shapes are given in the PACS Observers Manual.

• Unreliable broad-band (dust) features
  • Broad spectral features (of a few micrometer width) and continuum shape variations can be introduced by transient effects (for chopNod mode and more so for unchopped mode observations) and by pointing offsets distorting the Relative Spectral Response Function. The "background normalisation" pipeline script for chopNod observations is recommended for observations looking for such features, as it minimises the effect -- this is the SPG pipeline for HIPE 13 and upwards. For unchopped mode observations you could reprocess the observation with the "transients correction" pipeline script that is new to HIPE 13. However, neither of these pipelines will completely negate the effect of transients and pointing jitter-induced RSRF distortions.

• NaN's in cubes
  • It is normal to have NaNs at the very edges of the spectral ranges of SED mode observations: this is due to gaps in the spectral sampling.

• Quality flags in the quality product
  • The quality flags in the quality context ("quality" or "qualitySummary") inside the observation context are meant for HSC/ICC internal evaluation of the quality of the products: when an observation had a serious quality problem, the PI of the program would have been contacted about it. For archive users, only the comment in the "qualitySummary", when available, is of interest.

• PacsObsSummary which is a text summary of an observation that can be viewed in HIPE, contains a table of the wavelength ranges covered in the observation. These refer to the user-requested ranges. The achieved ranges may be smaller if they cross the band borders. If they do, this information will be provided (eventually) in the qualitySummary (present in all observations and via in the HSA GUI: search results)

• Split On Off pipeline scripts for chopNod observations. In this script one important mask is not activated in the Level 1 to 2 part of the pipeline. The mask NOTFFED, created by the spectral flatfielding task, was left out of the call "activateMasks" in the HIPE 14 and 15 "Split On-Off" interactive pipeline scripts. If you run this script (for chopNod line or range observations) you must add this mask in to all calls on "activateMasks" occurring after the flatfielding has been done. The ipipe script ChopNodSplitOnOff.py still has the masks spelled out one by one. It needs 'NOTFFED' to be included in the list of masks to take into account the outofband.

• Interpolated cubes for tiling observations with no or little overlap between the rasters. A few observations were performed with PACS spectroscopy as tilings where the individual pointings have no, or very little, sky overlap. As with all tiling observations, the standalone browse products provided for these observations are the interpolated cubes. However, these cubes are not suitable for science, since the field mapped by the interpolated algorithm is stretched beyond recognition. It is recommended to use the projected cubes, which can be found in the Level 2 (or 2.5) of the HSA download. The postcards for these (as for all) observations are created from the projected cubes, and hence by looking at the postcards it is easy to recognise the observations with these no/little overlap pointings.

• The postcards. The postcards displayed in the HSA search results panel show the prime and parallel ranges for range scans (i.e. both those requested by the observer and those that come for free in the other camera), while for line scans usually only the prime ranges are shown (the exceptions are where the parallel ranges are longer than 2 microns, which are then also shown).

SPIRE Photometry

Special messages

Warning for pointing anomaly in HIPE/SPG 13

PACS and SPIRE photometry observations reported in this CSV file are affected by a known problem related to the reset of the Spacecraft Velocity Vector (SVV) during the upload of the star-tracker's defective pixel table. For the affected observations, the pointing of the telescope can be off along the scan direction, and shifted up to 20 arcsec. This effect is corrected in HIPE 14.1.

General notes

• The best science quality Level 2 and Level 2.5 SPIRE photometry products are created with version 14.1 of the Herschel Systematic Product Generation pipelines. These data are available in the Herschel Science Archive. Reprocessing SPIRE observations with HIPE v14.2 or above will result in identical products, except Level-3 maps (see below).

• SPIRE-P Level 2.5 and Level 3 maps
  • Level 2.5 maps combine parallel mode as well as SPIRE-only maps into larger maps using the standard two-pass pipeline, see The SPIRE Data Reduction Guide, sections 4.2.3. The list of the Level 2.5 maps is available here.
  • Level 3 maps are mosaics of overlapping Level 2 and/or Level 2.5 maps, see The SPIRE Data Reduction Guide, sections 4.2.4. This list of the Level 3 maps is available here.
  • No astrometry correction is applied on the individual maps used in Level 2.5 or Level 3 processing. This may result in broadened PSFs (or double sources in the extreme case) if some individual maps are affected by astrometry offsets.
  • Before HIPE 14.2.1, the mosaic() task calculated incorrectly the SPIRE Photometer Level-3 maps: the image and error extensions.

• Stripes in PSW, PMW and/or PLW maps
  • All SPIRE photometry pipelines use the iterative destriper method with a constant baseline (polynomial of zeroth order). This is the best method for SPIRE maps as reported in The SPIRE Map-Making Test Report, Xu et al, 2013 (arXiv:1401.2109). In some cases there could be a residual striping due to a combination of thermal drifts, bolometer jumps and/or improper baseline subtraction. Suggested solutions is to reprocess the map using a first order polynomial for the baseline estimation in the destriper.

• De-glitchter masks faint sources
  • Removing glitches from the data is a very delicate process. In particular, for data taken in Parallel Mode (sampling at 10Hz) and at high speed (60"/s) the de-glitcher with standard parameters may flag very faint sources as glitches. Bright sources are different from glitches in that they have a gaussian (i.e. beam/PSF) shape. For faint sources, the sampling rate could be not high enough and hence they have a "delta" shape, which is similar to a small glitch. The user might try to modify the correlation parameter to 0.95: this will decrease the number of detected glitches.

• Some sources have saturated the ADC and the corresponding data have been masked
  • There is nothing a user can do: the source was simply too bright.

• Cooler temperature variations
  • The cooler temperature variations, as explained in greater details in the SPIRE Data Reduction Guide, section 6.4, can affect observations performed soon after the cooler recycle. The steep rise of the sub-K detector temperature is also known as the cooler burp and there is a quality flag coolerBurpDetected (as of HIPE v11 or later) that indicates if the observation was performed during this period.
    The current list of observations known to have cooler temperature effects is here. Note that not all observations in this list raised the coolerBurpDetected flag.

• NaNs pixels present in the PSW, PMW and/or PLW maps
  • This effect, related to data masked for various reasons and poor coverage (not enough redundancy), is more evident in single fast-scan Parallel Mode maps. To avoid NaNs, increase the pixel's dimension (i.e., decrease the map's resolution)

• Quality flags
  • Currently, the quality flags at the quality context inside the observation context are just meant for HSC/ICC internal evaluation of the quality of the products and not for the users. In case the data had some serious quality problem, the PI of the program has been contacted about it. Otherwise, only information in the quality summary, when available, should concern the observers.

• Planck derived zero offsets
  • The extended calibrated maps (extdPxW in Level 2, 2.5 or 3) incorporate zero level offsets derived from Planck-HFI. For small size SPIRE maps, smaller than ~30 arcmin, the zero-offset can be rather uncertain, due to the large Planck beam (8 arcmin). In such cases the interpretation of the zero offset as the absolute zero level must to be treated with extreme caution.

• Missing keywords in calTree
  • In HIPE 14.x: trying to access beamProf meta parameter 'FWHM_majorPsw' from the spireCal tree crashes with "parameter not found" exception

    cal = spireCal(pool="spire_cal_14_3")
    print cal.phot.beamProfList[5].meta['FWHM_majorPsw']

    This works in build 15. Note that the parameter (and a few others) are actually present in the product, but in 14, they are renamed to META_1, META_2...

• Incorrect WCS in the primary header of Level 2.5 and Level 3 maps
  • The correct WCS for Level 2.5 and Level 3 maps is in the header of the image, error and coverage extensions. The WCS in the primary header corresponds to one of the individual maps and it is not updated for the combined or mosaicked map.

• Reprocessing with the level-2.5 pipeline
  • Processing a list of observations with the level25_pipeline.py SPG script may lead to tiny differences (1.0e-4, 1.e-5 Jy/beam or MJy/sr) with respect to the same products in the Herschel Science Archive. This is due to the way the current pipeline incorporates the last turnaround building block of concatenated observations in order to proceed with the the deglitching. There is currently no fix, although exactly the same results, as in the archive, can be obtain by individually processing each observation with the two-pass pipeline and then running the level25_pipeline.py script.

• Interactive analysis
  • Fixed in 15.x: SourceTimelineFitter: the errors on the fitted RA, Dec positions from the task are wrong near the poles.
  • Fixed in 15.0.1: SourceTimelineFitter: some observations have serendipity scans in level-1 context. These are telescope slews passing through the region of the map. These are usually scans from a different epoch and in many cases may have very different detector offsets than the actual scans. As a consequence, including the serendipity scans in the timeline fitter may lead to fit failure (null pointer exception message) or completely wrong timeline fitted parameters. One way to identify if the observation has serendipity scans is to check the building blocks types of the level-1 scan lines, usually the serendipity scan is the last one. Here is an example on how to check and remove such scans before running the SourceTimelineFitter:

# loop over all level-1 products references
OBSID=1342204107
obs = getObservation(OBSID, useHsa=True, instrument='SPIRE')
for ref in obs.level1.refs:
     print ref.meta["bbTypeName"] # this is the buildingBlock type
     if ("Serendipity" in ref.meta["bbTypeName"].string):
         print "Removing this scan as it is serendipity"
         obs.level1.refs.remove(ref)

Then the timeline fitter can be run using obs.level1 context as input.

• • The SPIRE Useful Scripts, Calibration Photometer Calibration bundle point/ext and semi-ext are broken in HIPE v15, they work as expected in previous versions, e.g. HIPE v14. The error message is

<type 'exceptions.TypeError'>: range() integer end argument expected, got float.

As HIPE is no longer maintained and updated, the corrected scripts are available attached at the bottom of this page. Unzip the file and and edit the following line near the top of each script: sys.path.append("path_to_scripts") and replace "path_to_scripts" with the actual path to the folder with the unzipped files. You can also comment out this line and put the scripts in your PYTHONPATH. Then you can use the two *_usage15.py scripts.

• For more information check the SPIRE Wiki page, SPIRE User's Manuals and/or contact the HSC helpdesk.

SPIRE Photometer Release Notes

SPIRE scan map release note : 20 October 2009

SPIRE Small Scan Map AOT release note: 17 Mar 2010

SPIRE Spectroscopy

• The best science quality Level 2 SPIRE spectrometer products are created with version 14.1 of the Herschel Systematic Product Generation pipelines. These data are available in the Herschel Science Archive. Reprocessing SPIRE observations with HIPE v14.2 or above will result in identical products.

• Extended calibrated spectra and spectral cubes: prior to SPG 14.0 all extended calibrated spectra, including the spectral cubes (as these are built by extended calibrated spectra) were affected by a missing correction for the far-field feed horn efficiency (η_ff ). This correction is significant - a factor of 1.3-1.5 in SSW and 1.3-2.2 in SLW, as shown in the figure, where the red dashed line shows η_ff , the blue squares are the average ratios of uncorrected FTS synthetic photometry to the corresponding photometry extracted from extended-calibrated photometer maps and the green circles are ground based measurements, taken pre-launch. More details on this critical problem will be available in a dedicated paper (Valtchanov et al, 2018).

• Bright source mode: observations in bright source mode processed with HIPE v7 or earlier result in spectra that have no scientific value. Bright source observations processed with HIPE versions 8 to 12.1 are properly calibrated. In HIPE v13 and HIPE 14.0, an out-of-date bright gain calibration product leads to poorly calibrated spectra. This calibration issue is resolved in HIPE v14.1.

• Low resolution (LR) and High + Low resolution (H+LR) observations: LR processing was improved from HIPE v9. As of HIPE v14, a correction was introduced to remove artefacts in the SLW array of LR spectra (or the LR spectra from H+LR observations). This correction is applied to point-source calibrated spectra. The following figure shows four examples of LR point-source calibrated spectra for the centre detectors, before and after the LR correction has been applied, i.e. HIPE v13 processed (blue) and HIPE v14 processed (black). Note that there is no change for SSWD4. This correction is propagated to the extended calibrated SLW spectra of LR observations as of HIPE v14.1.

• High + Low resolution (H+LR) observations: the Spectrometer user processing pipeline scripts available in HIPE handle H+LR observations in a slightly different way to the pipeline script used to process the data found in the Herschel Science Archive (HSA). This is in order to keep the user processing script straightforward to understand. However if processed with the Spectrometer user mapping pipeline script in HIPE, there may be small differences seen between the resulting LR spectral cubes compared to those taken directly from the HSA. The mapping pipeline script has been adjusted for this issue in the continuous integration builds (HIPE version 15) and will be fixed in the next public HIPE release

• Faint point source observations: for a know point-like source, if the SSW and SLW bands do not match, observers are recommended to run the background subtraction script in HIPE - more details are in the SPIRE Data Reduction Guide. If this doesn't solve the problem, the observer is encouraged to contact the HSC helpdesk or the FTS User Support Group.

• Partial spectra in convolution projected spectral cubes: as of HIPE 14, and in addition to the Naive projected cubes, convolution projected (CP) cubes are provided in the Level-2 Observation Context for all FTS intermediate or fully sampled mapping observations. For each spaxel, the convolution projection sums the Gaussian weighted contribution from all spectra in the relevant detector array that fall within the kernel, which has a FWHM of the beam diameter. This means more spatial information is used when gridding CP cubes and therefore they have more complete coverage compared to Naive cubes, which can have NaN spaxels, even at the centre of a cube. However, the minimum coverage for a CP cube is determined per frequency layer and as the FTS beam diameter is frequency dependent, this can lead to spectra that have gaps at the centre of the frequency bands. This issue is illustrated by the figure below, where a CP cube has been created with the default minimum coverage of 0.1 (with a single spaxel plotted in red) and is compared to the CP cube that results from raising the minimum coverage to 2.0 (corresponding spaxel plotted in blue), with the coverage shown in the top panel. Note that this issue only occurs for the very edge spaxels of CP cubes.

• Artefacts in the continuum for few repetitions
  • Very small repetition numbers (e.g. 2 or 4) make 2nd level deglitching, which is based on a statistical outlier criterion, more challenging. The deglitching module may either not identify a glitch at all or it may not remove it completely. In cases where the glitch is located within the double-sided portion of the interferogram, the additional energy from the glitch will translate into artefacts of the continuum level. This kind of problem can be identified by inspecting all detectors from all scans in the level-1 spectral products. For some detectors, one or several scans may appear to be outliers. As a work-around, it is recommended to reprocess the data with a lower thresholdFactor when calling deglitchIfgm(). If the problem persists, the identified detector should be removed from the applicable scan in the SDI product.
    (NB: This affects HIPE 6 and higher)

• Line fitting
  • Unresolved lines should be fitted using a sinc model with the width fixed. For the sinc model implemented in the SpectrumFitterGUI, the sinc width can be set equal to resolution/π, which, for HR is 1.2/π = 0.382 GHz. For partially resolved lines, with widths greater than 200 km/s, the sincGauss model can be used, keeping the sinc width fixed.

• Point-source and extended-source calibrated spectra
  • If your level-2 spectra show characteristic jumps at ~1250 GHz and ~750 GHz, so the spectra from the two bands SSW and SLW do not match, then your target may be extended or semi-extended in the SPIRE beam. You may need to use the semi-extended correction tool (SECT), available since HIPE v10. Check the SPIRE Data Reduction Guide (SDRG), section 7.6.2 "Does my spectrum need correcting?".

• Quality flags in the quality
  • The quality flags at the quality context inside the observation context are just meant for HSC/ICC internal evaluation of the quality of the products and not for the users. In case the data had some serious quality problem, the PI of the program has been contacted about it. Otherwise, only information in the Quality Summary, when available, should concern the observers.
  • For observations before OD1000 there could be an erroneously raised or a missing flag RADECACC. This flag is calculated as the difference between the observer requested target coordinates (kept in a metadata raNominal, decNominal) and the average pointing of the telescope during the observation (excluding the slew). This is an indicator of the pointing accuracy of the observation. If greater than 2.2" then RADECACC is raised in the QC context. The calculation of the average (ra, dec) during the observation does not take into account the known offset of 1.7" of the BSM home position for OD < 1000 and consequently observations with better pointing than 2.2" may be flagged and vice versa, observations at more than 2.2" may not be flagged. If you want to check the pointing offset with respect to the requested target coordinates then you should use the pointing information for the central detector SSWD4 in the level-2 context.
    A new quality parameter raDecOffset is introduced in HIPE v13, in level-2 products, which contains the correct offset between the raNominal, decNominal and ra,dec for the FTS central detector.

• Observations during the steep rise of the sub-K temperature
  • This issue is fixed since HIPE v13, except for a handful of observations made before OD400: contact the HSC Helpdesk if you think data processed with HIPE v13 or later still needs correcting for this issue.
  • During the first few hours after the cooler was recycled the bolometers' temperature (the sub-K temperature) undergoes a steep rise before it reaches a stable plateau. Observations during this period suffer overcorrection of the instrument/telescope emission. This is more significant for faint targets and can be identified as an unphysical slope of the SLW spectrum, with an important negative gap in the region that overlaps with SSW, i.e. a droop at the high-frequency end of SLW (see the figure below). One way to check if your observation is within this problematic category is to get the median sub-K temperature for the first building block: print MEDIAN(obs.level0_5.get(0xA1060001).nhkt['signal']['SUBKTEMP'].data), where obs is the pre-loaded observational context. If the result is less than 0.2869 K then the observation is affected by this.
  • A possible fix to this problem is to subtract the smoothed off-axis detectors, because they are also subject to the same overcorrection. The Background Subtraction script from the HIPE Useful Scripts can be used for this investigation, the output of this script is shown in the Figure below.

• Calibration/pipeline problems/needs
  • For more information check the SPIRE Manuals and/or contact the HSC helpdesk.

SPIRE Spectroscopy Release Notes

SPIRE Spectrometer Point Source AOT release note: 21 May 2010

SPIRE Spectrometer Bright Source Mode AOT release note: 7 Sep 2010

SPIRE Spectrometer Mapping AOT release note: 30 Apr 2010

HIFI observations (point mode, spectral survey and mapping observations)

In order to obtain the best possible Level 2 HIFI data the observations should be reprocessed with the latest HIPE User Release.

• Use of option avermask=True in fitHifiFringe
  • It has been discovered in 15.0.0 that the use of option avermask=True in the fitHifiFringe task coud lead to unexpected error in very rare circumstances. This occurs when the mask computed by the task happens to contain only one channel, or that it concerns the very last channel of the spectrum. As a work-around, users can always provide the mask themselves as input to the task (usermask parameter). Alternatively, the parameter resolution can be changed to a smaller value than the default (10 MHz) in order to force at least two channels to be masked (e.g. resolution = 5)

• Flagging of spectral cubes
  • Section 11.6 of the HIFI Data Reduction Guide provides examples on how to assign flags to various kinds of products. In the case of cubes, users should be aware of a limitation of the flagPixel task in that flags with values above 32767 will not be honoured and therefore be uneffective despite of the task not complaining or reporting anything about it.

• Isolated problematic processing with 14.1
  • About a dozen of observations were not properly processed in 14.1. The list of those obsids is: 1342197147, 1342210032, 1342216806, 1342219255, 1342219454, 134230216, 1342245299, 1342246507, 1342245997, 1342250406, 1342250721, 1342265972, 1342263227, 1342192673, 1342218631. These observations have been reprocessed correctly with 14.2.

• Erroneous mean and median computation in trendAnalysis tables
  
  • A bug has appeared in the HIFI 14.0.1 products, that was not present in the 13.0.0 products. The problem is in the automatic computation of the mean and median in the trendAnalysis tables. In HIPE 14.0.1 those are erroneously computed on baseline-subtracted spectra, resulting in values very close to 0 even in the event of noticeable continuum level. This problem is resolved in 14.1 and proper statistics figures will therefore be available in the products once the 14.1. Bulk Reprocessing has completed.

• Limitations in the mkRms task for automatic noise estimates
  
  • From HIPE 13 onwards, automatic estimates of the spectra noise rms are provided to the users in the trendAnalysis > Statistisc context - these are based on an upgraded version of the task mkRms (see also the HIFI pipeline specficication document). Those estimates are in particular used to raise, or not, flags informing about whether the data are under-performing in terms of radiometric noise with respect to the theoretical predictions used at the time of the proposal. Because the noise estimate relies on the efficiency of automatically masking lines that may be present in the data, there can exist circumstances in which this masking in sub-optimally performed, therefore leading to inaccurate noise estimate, and possible false positives in the raised flags. Users should be aware of this limitation and, if necessary, cross-check the reliability of certain flags related to those noise figures through their own estimates. It should be noted, however, that significant improvements on this matter have been implemented in 14.1, which should make the estimated noise closer to the ones effectively applying to the spectra.

• Reprocessing of old products with the new HIPE 14.0 calibration tree from level 1
  • Whenever you try to reprocess old products starting from levels as high as the level1 and want to apply a new calibration tree, the whole calibration products will be erased - this includes for example the System Temperature spectra stored in the pipeline-out products. As a consequence, for some modes like OTF-PSW or PSW, the OFF spectra, which relies on those Tsys spectra, will not be possible to be computed and an error message will be issued. You should either reprocess from level 0, or simply use the HIPE 14.0 products whenever they will become available.

• Strong/Weak Spurs
  • Spurs are still affecting the HIFI data at spot frequency points. They will be flagged automatically in the data using a SpurFinder task when present above a certain threshold in the internal load spectra. There are still some spur categories that are not yet properly detected but this is improving in each new versions of HIPE. Some spurs are so intense and broad that they can lead to the loss of complete WBS sub-bands. Note that the strong spur affecting the upper end of band 1a has now been completely cleaned - see the corresponding technical note: HIFI Information note on the removal of spurs in band 1a and the sampling in the mapping modes. The same is true for a weaker spur that did affect data in band 7b close to LO = 1834 GHz.

• Unpumped data
  • Due to gaps in the LO output power over the HIFI frequency range, some spectral scans will contain spectra that cannot be used due their very high noise temperature levels. These data should be discarded from further processing (e.g. deconvolution). For single frequency observations, there exist some places where the theoretical noise temperature (i.e. the one advertised by HSpot) should be nominal, but the sensitivity effectively achieved at the time of observations is noticeably worse. These are cases where the mixer tuning algorithm does not fully converge. Such cases have e.g. been reported around LO frequencies of ~ 1441.5 GHz. An overview of the expected unpumped frequency areas is given in the AOT release notes. Users observing such effects should contact the Helpdesk.

• Standing waves
  • There are a variety of standing waves known to affect the data at Level2 - see the AOT release notes for more details. At the present time, the FitHifiFringe tool usually does a decent job for most standing waves in SIS bands. In HEB bands, where the standing wave is not of optical nature and does have a strictly sinusoidal shape, FitHifiFringe can help when restricted to the frequency range where the line is present. Another alternative in the form of a dedicated algorithm has been developed (so-called "matching technique") and is offered in the hebCorrection task from HIPE 12.0 onwards (details can be found in Chapter 10 of the HIFI Data Reduction Guide). You can contact Ian Avruch (i.avruch@sron.nl) at the HIFI ICC for further details. In any case, it is advised not to use more than 3 components in FitHifiFringe. Since HIPE 5, a number of new features have been made available in FitHifiFringe. One of them ("sub_base" option) allows you to fit a baseline at the same time. Since FitHifiFringe can identify automatically the lines in order to set masks, users should be careful when treating lines with large wings. In this case, the user should set a window by hand. The automatic line masking should then be switched off.
  • In case of strong source continuum some standing waves can be enhanced. Such standing waves can be significantly reduced in amplitude using an alternative pipeline algorithm known as the Modified Passband Technique , which is described in the Standing Wave Removal chapter of the HIFI Data Reduction Guide. Note that in 9.0, a bug affected the doFilterLoad task in such a way that it ignored the user-input parameters, resulting in the task always applying the "cubic_spline" method with default parameters - i.e. the "fft" method will not be applied despite being set in the task inputs. This was fixed in 9.1 before the SPG 9.1.0 bulk reprocessing of the archive.

• Baseline distortion
  • Residual drift from the detector response can translate into imperfect baseline structures. This usually manifests as wavy structures, slopes, or overall level offsets of the spectral baselines. Such artefacts are expected to be enhanced e.g. at the borders of spectra obtained in diplexer bands (sub-bands 1 and 4 for bands 3 and 4, sub-bands 2 and 4 for bands 6 and 7). Similar artefacts are also expected when using modes without reference, such as the FreqSwitchNoRef, or the LoadChopNoRef, and are enhanced with strong continuum sources. Please refer to the HIFI Data Reduction Guide for details about the most recent improvements to the FitBaseline task.

• Saturation
  • Saturation of the WBS CCDs may be observed at spot frequencies. These are usually due to either strong and broad spurs, or to cases where only marginal pump level could be achieved due to shortage of LO output power. Users can go back to HSpot to visualize in the frequency editor where their AORs were potentially affected by such effects. HIPE will normally flag the affected spectrometer channels.

• Beam efficiencies
  • The easiest way to convert the level2 data into the Tmb scale is to use the task "doMainBeamTemp". Beam efficiencies, as well as other beam-related parameters, can be found in the dedicated release notes.

• Side-band ratio and continuum
  • The data processed at level2 are corrected for the fact that HIFI observes in double-side band. In practice, a side-band ratio is assumed over most of the HIFI operational range, which simply means that single-side band intensities are twice those in double-side band. In the limited ranges where this side-band ratio deviates from unity, the intensity scaling will differ for respective USB and LSB spectra. It will also imply that the continuum cannot be trusted in these frequency range. A dedicated task to recover the continuum corresponding to a balanced side-band ratio was made available in HIPE 9.0 ( undoSidebandGain). General recipes on how to deal with line and continuum calibration in a Double-Sideband system are also given on the HIFI handbook.

• H/V polarisation pointing information
  • The observed differences in HIFI horizontal (H) and vertical (V) polarisation line profiles has made it desirable to have astrometry available for both H and V spectrometers. The possibility that small scale structural or velocity variations may be responsible can then be examined. In the past, the pointing information associated with both H- and V- spectra has been that of a synthesized aperture, which is a position midway between the H and V spectrometer beams. From HIPE 4.2 onwards, an option has been introduced in the doPointing task, allowing to compute separate attitude for the respective mixer polarizations. From HIPE 5 onwards, this option is the one used by default. In practice, note that this slight pointing mis-alignment can lead to differences between the H and V signals observed both in spectral lines and continuum, especially when the source structure features abrupt brightness temperature variation on small spatial scales.

• Platforming effect
  • There exists the possibility that residual platforming effect, i.e. imbalance between the average baseline level of individual WBS sub-bands, is still observed at level2. This is usually due to temperature drifts of the CCD during the observations. There is no dedicated tool in the HCSS to correct this effect and users should avoid stitching their data when such artifact is present in the data. Correction of the relative baseline level of each sub-band is left to the user's judgment.

• Quality flags in the quality product
  • Up to HIPE 13, all quality flags were reported irrespective of their level of severity of applicability. As a consequence, some of them could lead to false positive that the users should in fact not worry about. A re-organisation of the flags into public and private (mostly informative to instrument scientists) and their level of criticality has been performed in HIPE 14, allowing the end user to filter more straightforwardly the relevant information.

• Browse products
  • HIPE 9 introduced browse products for mapping and spectral scan observations. For maps of moving targets, note that the browse product images are displayed in the comoving frame, so a centre of RA/Dec = (0",0") is used. These include deconvolved spectra (but see following note) for spectral scans and spectra/line maps for a standardly gridded map for mapping observations.

• Deconvolution of Frequency Switching data
  • The deconvolution software will not work properly with frequency switching data. This is intrinsically linked to the fact that the deconvolution algorithm has troubles with constant steps in the frequency dimension, and that frequency switching observations uses uniform frequency throws. This creates some periodic ghost artifacts close to strong lines.

• Flux Conservation in Spectral Cubes from Mapping Observations
  • The doGridding task in the HIFI pipeline is responsible for convolving the spectral datasets acquired in an OTF or DBS Raster mapping observation into a spectral cube with a specified pixel scale, which by default should match how the scan lines and readout points within each line were spaced during the observation. The scheme of signal filtering and interpolation to put the data on the specified grid may affect the overall flux conservation, at a level which is low but you should be aware of. For example, the total signal summed from a spectral cube produced using a Gaussian filter over the datasets of a Nyquist-sampled OTF map is generally < 1% lower than the sum of the signal taken directly from the input datasets (the Level 2 HTP datasets) before convolution. A part or all of this slight mismatch may be on the assumed versus actual beam shape at the observed frequency. If you wish to put the map on a coarser grid, effectively reducing the spatial resolution to a wider beam in order to match another observation, then the flux losses become more noticeable. Doubling the pixels sizes from their default (native map point spacing) can reduce the total flux by as much as 10%, accompanied by an increase in baseline RMS noise. The effect is more prevalent in OTF maps than DBS Raster, and in addition to deviations from ideal beam shape characteristics becoming more important, the filtering and interpolation method, and parameter values can be influential. No changes to the doGridding algorithm are planned, and this issue applies to all HIPE versions.

• Retrieval of calibration files from the HSA
  • In HIPE versions from the track 9, there is small bug in the getHifiCal task. The task may erroneously report that the latest calibration tree available is HIFI_CAL_9_0, while more recent versions are now available. This issue is solved in HIPE 10. This is not a problem with the HIPE 11 build. One possible work-around is to clear the calibration pool in your .hcss/lstore, and call the task again or simply use the updated HIPE version. Then the latest calibration tree should be offered for download.

• Other calibration/pipeline problems/needs
  • For more information check the HIFI User's Manuals and/or contact the HSC helpdesk.

Pointing

Observations performed at the the so-called 'warm' attitude range (Star Tracker 2 temperature > -15 C), could present a degradation of the pointing performance due to thermoelastic effects. These would include a larger APE and a pointing drift.

In order to estimate such astrometry shifts, we have created a General Useful Scripts named Astrometry Thermoelastic Drift Correction, available in HIPE 15. The script calculates the average temperature of STR2 sensors during the mid-point of the observation and, if that is > -15 C, it uses an established STR2-Z-axis offset correlation function to get the Z-axis offset and derive RA and DEC offsets. Two possible outputs are provided: (1) A new pointing product that takes into account the updated rotation matrix/quaternion and corrects the filtered attitude. This product can be used by the user to reprocess the observation. (2) New Level 2 products (only applicable to Photometry) where the offsets in RA y DEC (derived from the Z-axis offset) have been used to adjust the WCS.

Additional Information

HIFI documents: see the HIFI Instrument and Calibration webpage

PACS documents: see the PACS Instrument and Calibration webpage

SPIRE documents: see the SPIRE Instrument and Calibration webpage