Data Products Known Issues

HCSS, SPG, and HIPE

The whole suite of Herschel software is known as the Herschel Common Science System (HCSS). This encompasses all the software aspects related to the mission: automatic pipelines, spacecraft calibrations, etc. As data are retrieved from the spacecraft they are ingested in the Herschel Science Archive (HSA) and processed with the current official version of the pipeline. That means that, at any given time, different data in the HSA may be reduced with different pipeline versions. The pipeline version is listed in the HSA GUI as SPG vX.X.X, where SPG stands for Standard Product Generator. The SPG version is available as the header keyword 'creator', in the data .fits files. Every HCSS even version, the whole HSA is bulk re-processed with the same up-to-date version of the pipelines.

From the user point of view, the most important piece of the HCSS system is the Herschel Interactive Processing Environment (HIPE). HIPE allows the astronomer the possibility of inspecting the data and re-process them, if the results from the automatic pipeline are not good enough for his/her purposes. Because HIPE is part of the HCSS, the latest version of HIPE will have the most up-to-date pipeline, calibrations and documentation available.

For example, some data in the HSA may be marked SPG v6.1.0, which means that they were processed with that HCSS version. If the current HCSS version is 8.2.0, HIPE is informally called HIPE 8.2.0. If you re-reduce the data with this HIPE version, they will get the result that would be produced with the official 8.2.0 pipelines.

The latest version of HIPE can be obtained here. Known HIPE issues/problems/bugs are detailed here.

In what follows, we provide a summary of the known issues that you may encounter when inspecting data processed with the automatic pipelines SPG versions 6.1.x to 8.x.x Most can be resolved by running the pipelines within HIPE and optimizing their parameters as explained below.

Note that some of this information can also be found in the quality report of the observation (QC Report) and as metadata with the FITS keyword "PCAVEATS".

TABLE OF CONTENTS

PACS Photometer (scan mapping)

• In order to obtain the best possible Level 2 PACS scan-map data, the observations should be reprocessed with the latest HIPE User Release.

• High Pass Filter artifacts in blue and red maps
  • These are due to a non-optimal estimate of background by the high-pass filter close to a bright source. The current masked high-pass filter pipeline is a trade-off, designed to get a good sensitivity on point-sources and preserve some extended emission up to a few arcmin scale. However due to the relatively large width in the second high-pass filtering, significant stripping from the 1/f noise is still present in the level 2 maps. On the other hand a large fraction of the extended emission is filtered out. The observer is advised to play with the threshold to define the mask and with the width of the high-pass filter to reduce these effects or move to MADmap scanamorphos to preserve extended emission at all scales. Level 2.5 MADmap maps, combining scans and crossed-scans on the other hands do preserve extended emission at all spatial scales.

• Calibration block transient in L2 maps
  • Several red (160um) scan maps are affected by a calibration block downwards transient when scheduled immediately before the science observation. This can be mitigated by running a high-pass filter with a smaller width, fitting the transient or just masking the affected frames at the start of the observation.

• Pointing/Astrometry issues
  • If your source is severely mis-pointed you should contact the HSC helpdesk. The measured Herschel APE (Absolute Pointing Error) on pointed observation is 2 arcsec (1 sigma) but was improved throughout the mission down to 1 arcsec (1-sigma). The astrometry of several PACS scan maps acquired between ODs 320 and 761 has been reported to be off by 4 arcsec (solid offset of the whole map) or even above for a few fields where the tracking stars are not homogeneously distributed in the star-tracker field-of-view. It is intended to improve the a posteriori reconstructed pointing for all observations in the future.

• No absolute noise map
  • Full noise propagation in the HSC pipelines will be available at later HCSS versions, so the current noise map is relatively but not absolutely correct. Furthermore, the default pipeline results in correlated noise in individual pixels. In order to estimate a proper background standard deviation, the observers are advised to make 10+ aperture photometry measurements in different sky patches around the source and to estimate the absolute standard deviation of the image as the sigma of those photometric points.

• Quality flags in the quality
  • Currently, the quality flags at the quality context inside the observation context are just meant for HSC/ICC internal evaluation of the quality of the products and not for the users. In case the data had some serious quality problem, the PI of the program has been contacted about it. Otherwise, only information in the quality summary, when available, should concern the observers.

• Other calibration/pipeline problems/needs
  • For more information check the PACS Instrument and Calibration webpage and/or contact the HSC helpdesk.

PACS Photometer Release Notes

PACS photometer AOT release notes see PACS Instrument and Calibration webpage

PACS Spectroscopy

• In order to get publication quality spectra, the observations should be reprocessed with the latest HIPE user release.

• Off-subtraction for unchopped long-range scan observations
  • Range Spectroscopy unchopped observation require off-position scans observed in a separate AOR. The off-source observation need to be subtracted from the on-scan after producing Level 2 rebinned spectra, this results a Level 2.5 product. The pipeline only generates Level 2 data products for unchopped spectroscopy in SPG 8.0. You need to combine interactively the on and off positions using the dedicated multi-observation unchopped pipeline script under the PACS pipeline menu.

• Unchopped grating scan flux calibration
  • The absolute flux calibration of the PACS spectrometer is based on observations of flux calibration standards using chopped spectroscopy modes. There are hints of systematic differences in the response scaling between chopped and unchopped mode due to response transients within the chopping pattern. In SPG 8.0 the flux calibration of unchopped data relies on the system response derived chopped scheme, therefore absolute flux values need to be carefully interpreted. Please contact Helpdesk for guidelines on the specific observation you have to deal with.

• Fringing
  • In order to remove the fringing-like bin-to-bin oscillations from the broad-range and SED mode observations (Nyquist sampled grating scans), the observers are advised to reprocess the observations with the user pipeline script in the HIPE version above, which includes flat-fielding. The optimal parameters for such task should be determined by inspection of the final results.

• Check for contaminating flux in chop-off positions
  • HIPE tools are provided in the latest user release to inspect the chop-off positions separately for nod A and B pointings. This way the observer can verify that the reference positions from which the sky and the telescope background are estimated are appropriate, i.e. no line- or continuum emission is detected at the wavelength of the observation. Note, the tool is reducing data separate for the on- and off-source positions what may not cancel out wriggles from the RSRF. As a consequence, faint line emission in the chop-off fields may not be revealed. For faint targets (line peak-to-continuum emission ~<5-10 Jy) we also recommend to inspect the differential signal of nod positions (A-B) what could reveal the difference of contaminating flux level between the two chop-off fields.

• PSF correction for extended sources
  • Current flux calibration provides Jy/spaxel in the IFU products (Rebinned Cubes and Projected Cubes). Small variations in the absolute solid angle of the 25 spaxels are currently not taken into account and you need to adopt 9.4"x9.4" size for all spaxels. Point source observations should rely on the PSF correction that the "point source correction" task provides for the central spaxel. Beam maps at selected wavelengths have been made available.

• Unstable/incorrect broad-band (dust) features
  • Broad spectral features (a few micrometer) and continuum shape difference can be introduced by transient effects and pointing offsets distorting the RSRF. Corrections for these effects are under study. In the mean time, such features should not be interpreted blindly, any feature profile broader than the 69 micron Forsterite feature cannot currently be trusted.

• Limitations on absolute spectrophotometric accuracy
  • The PACS spectrometer flux calibration accuracy is limited by detector response drifts and slight pointing offsets due to the standard 2" (1-sigma) pointing error . These limit both the absolute flux accuracy and relative accuracy within a band. Corrections for both effects are under study by the PACS ICC and will be provided to the user in forthcoming hipe software versions.

• Spectral line profiles
  • As for any slit-spectrographs, if the incoming light beam is neither homogeneously- nor symmetrically illuminating a spaxel then line profiles might be distorted from the ideal (fairly Gaussian) shape. In case of a point-source, the line shape is skewed as a function of photocentre offset measured along the instrument Z-axis (perpendicular the slit direction). Examples of skewed line shapes are given in the PACS Observers Manual. Correction for a Gaussian profile fit and/or estimation of skewed profile parameters are under study by the PACS ICC and will be provided to the user in forthcoming hipe software versions.

• Line flux correction due to strong wings in the instrumental profile (IP)
  • Pre-flight ground-based monochromatic measurements indicated the PACS spectrometer instrumental profile distributes measurable power in spectral line wings. The effect is below ~10% and only noticeable for wavelengths longer than ~150 micrometers. Future HIPE releases will provide a correction factor to apply on a Gaussian fit in order to compensate for line power lost in the IP wings.

• Spectral leakage
  • The order selection filters of the PACS spectrometer have a steep but not perfectly vertical transmission profile a the cut­off wavelengths of the spectral bands. The PACS spectra near the band borders of Bands R1, B3A and B2B are affected by higher or lower order wavelengths leaking into the spectra. Please consult the PACS Calibration Document for precise specification of the leakage regions and we advise to contact Helpdesk if you need to derive line fluxes within the affected wavelength intervals.

• Second-pass ghosts
  • A second pass in the optics of the PACS spectrometer can cause a ghost image on some spaxels (except the central spaxel). If a source in one of the originating spaxels shows a strong spectral line, typically an atomic fine structure line, a weak, broadened line can be seen at an offset wavelength in the corresponding spaxel affected by 2nd pass ghosts. The peak flux of this line is typically ~5% of the line peak of the originating line. The most prominent ghost is the 122 mu feature as its originates from the usually string CII+ 157.7 mu parent line, a list of strong ghosts as well as the direction of projected passes onto the 5x5 footprint is provided in the PACS Calibration Document.

• NaN's in the final cubes
  • Due to irregularities in data sampling and outlier masking, the rebinned Level 2 spectra could have no data in some bins (indicated as NaNs). Usually the amount of NaNs is less than ~15%, and can be corrected by interpolation (the pipeline does not interpolate by default).

• Quality flags in the quality product
  • Currently, the quality flags at the quality context inside the observation context are just meant for HSC/ICC internal evaluation of the quality of the products and not for the users. In case the data had some serious quality problem, the PI of the program has been contacted about it. Otherwise, only information in the quality summary, when available, should concern the observers.

• Other calibration/pipeline problems/needs
  • For more information check the PACS Instrument and Calibration webpage and/or contact the HSC helpdesk.

PACS Spectrometer Release Notes

PACS spectrometer AOT release notes see PACS Instrument and Calibration webpage

SPIRE Photometry

• In order to obtain the best possible Level 2 SPIRE photometry data, the observations might have to be reprocessed with the latest HIPE User Release.

• Stripes in PSW, PMW and/or PLW (Level 2) maps
  • Most of the stripes that are present in the final maps are due to a combination of thermal drifts (which in few cases are not efficiently removed) and median baseline subtraction. A similar effect is caused by very bright sources: in this case, the problem resides in the median baseline subtraction only. Suggested solutions:
    • switch to a baseline subtraction using a polynomial fitting using the optional task baselineRemovalPolynomial. If there are no jumps in the timelines, you may also try to run the baseline removal on the entire timeline;
    • in the case of bright sources, you may try to mask them before running the baseline removal (either median or polynomial): you can use this script as a templeate

• De-glitchter masks faint sources
  • The de-glitcher is a very delicate process. In particular, for data taken in Parallel Mode (sampling at 10Hz) and at high speed (60"/s) the de-glitcher with standard parameters may flag very faint sources as glitches. Bright sources are different from glitches in that they have a gaussian (i.e. beam/PSF) shape. For faint sources, the sampling rate could be not high enough and hence they have a "delta" shape, which is similar to a small glitch. The user might try to modify the correlation parameter to 0.95: this will decrease the number of detected glitches.

• Some sources have saturated the ADC and the corresponding data have been masked
  • There is nothing a user can do: the source was simply too bright. If the user has other sources still not observed and of the same intensity, it is suggested to change the AORs to use the bright source mode.

• Thermistor jumps
  • As of HIPE 6.0.3, a new module together called signalJumpDetector in place to identify the jump and to exclude the affected thermistor(s).

• NaNs pixels present in the PSW, PMW and/or PLW (Level 2) maps
  • This effect, related to data masked for various reasons and poor coverage (not enough redundancy), is more evident in single fast-scan Parallel Mode maps. To avoid NaNs, increase the pixel's dimension (i.e., decrease the map's resolution)

• Quality flags
  • Currently, the quality flags at the quality context inside the observation context are just meant for HSC/ICC internal evaluation of the quality of the products and not for the users. In case the data had some serious quality problem, the PI of the program has been contacted about it. Otherwise, only information in the quality summary, when available, should concern the observers.

• For more information check the SPIRE Wiki page, SPIRE User's Manuals and/or contact the HSC helpdesk.

SPIRE Photometer Release Notes

SPIRE Small Scan Map AOT release note: 17 Mar 2010

SPIRE Point Source Mode release note: 30 Apr 2010

SPIRE/FTS Spectroscopy

• In order to obtain the best possible Level 2 SPIRE FTS data, the observations should be reprocessed with the latest HIPE User Release.

• Bright source mode: observations in bright source mode processed with HIPE v7 result in spectra that have no scientific value. Bright source observations processed with HIPE v8 or above are fine. The observer should contact the HSC helpdesk or the FTS User Support Group for assistance for re-processing with HIPE v8 user script.

• Artefacts in Low resolution (LR) observations: currently all LR mode observations suffer from serious artefacts which are introduced in the data processing. The observer should contact the HSC helpdesk or the FTS User Support Group for assistance.

• Artefacts in the continuum for few repetitions
  • Very small repetition numbers (e.g. 2 or 4) make 2nd level deglitching, which is based on a statistical outlier criterion, more challenging. The deglitching module may either not identify a glitch at all or it may not remove it completely. In cases where the glitch is located within the double-sided portion of the interferogram, the additional energy from the glitch will translate into artefacts of the continuum level. This kind of problem can be identified by inspecting all detectors from all scans in the level-1 spectral products. For some detectors, one or several scans may appear to be outliers. As a work-around, it is recommended to reprocess the data with a lower thresholdFactor when calling deglitchIfgm. If the problem persists, the identified detector should be removed from the applicable scan in the SDI product.
    (NB: This affects HIPE 6 and higher)

• Line fitting
  • If you know that the line you wish to measure is unresolved then you may want to fix the line width to the instrumental line width. For the Sinc model, implemented in the SpectrumFitterGUI you should put the Sinc width equal to resolution/π, which, for HR is 1.2/π = 0.382 GHz if your spectrum is in frequency, and 0.04/π = 0.012 cm^-1 if your spectrum is in wavenumbers.

• Point source and extended source spectra
  • If your level-2 spectra show characteristic jumps at ~1250 GHz (42 cm^-1) and ~750 GHz (25 cm^-1) and the spectra from the two bands SSW and SLW do not match, then your target is extended in the SPIRE beam. You need to use level-1 data in this case. Check the SPIRE Data Reduction guide, Figure 6.59 and the text in Section 6.2.5

• Calibration/pipeline problems/needs
  • For more information check the SPIRE User's Manuals and/or contact the HSC helpdesk.

• Quality flags in the quality
  • Currently, the quality flags at the quality context inside the observation context are just meant for HSC/ICC internal evaluation of the quality of the products and not for the users. In case the data had some serious quality problem, the PI of the program has been contacted about it. Otherwise, only information in the quality summary, when available, should concern the observers.

SPIRE Spectroscopy Release Notes

SPIRE Spectrometer Point Source AOT release note: 21 May 2010

SPIRE Spectrometer Bright Source Mode AOT release note: 7 Sep 2010

SPIRE Spectrometer Mapping AOT release note: 30 Apr 2010

HIFI observations (point mode, spectral survey and mapping observations)

In order to obtain the best possible Level 2 HIFI data the observations should be reprocessed with the latest HIPE User Release.

• Strong/Weak Spurs
  • Spurs are still affecting the HIFI data at spot frequency points. They will be flagged automatically in the data using a SpurFinder^? task when present above a certain threshold in the internal load spectra. There are still some spur categories that are not yet properly detected but this is improving in each new versions of HIPE. Some spurs are so intense and broad that they can lead to the loss of complete WBS sub-bands. Note that the strong spur affecting the upper end of band 1a has now been completely cleaned - see the corresponding technical note: HIFI Information note on the removal of spurs in band 1a and the sampling in the mapping modes. The same is true for a weaker spur that did affect data in band 7b close to LO = 1834 GHz.

• Unpumped data
  • Due to gaps in the LO output power over the HIFI frequency range, some spectral scans will contain spectra that cannot be used due their very high noise temperature levels. These data should be discarded from further processing (e.g. deconvolution). For single frequency observations, there exist some places where the theoretical noise temperature (i.e. the one advertised by HSpot) should be nominal, but the sensitivity effectively achieved at the time of observations is noticeably worse. These are cases where the mixer tuning algorithm does not fully converge. Such cases have e.g. been reported around LO frequencies of ~ 1441.5 GHz. An overview of the expected unpumped frequency areas is given in the AOT release notes. Users observing such effects should contact the Helpdesk.

• Standing waves
  • There is a variety of standing waves known to affect the data at Level2 - see the AOT release notes for more details. At the present time, the FitHifiFringe tool usually does a decent job for most standing waves in SIS bands. In HEB bands, where the standing wave is not of optical nature and does have a strictly sinusoidal shape, FitHifiFringe can help when restricted to the frequency range where the line is present. Another alternative is currently under study but is not available in HIPE yet. You can contact Ian Avruch (i.avruch@sron.nl) at the HIFI ICC to get more details about this technique. In any case, it is advised not to use more than 3 components in FitHifiFringe. Since HIPE 5, a number of new features has been made available in FitHifiFringe. One of them ("sub_base" option) allows to fit a baseline at the same time. Since FitHifiFringe can identify automatically the lines in order to set masks, users should be careful when treating lines with large wings. In this case, the user should set a window by hand. The automatic line masking should then be switched off.

• Baseline distorsion
  • Residual drift from the detector response can translate into imperfect baseline structures. This usually manifests as wavy structures, slopes, or overall level offsets of the spectral baselines. Such artefacts are expected to be enhanced e.g. at the borders of spectra obtained in diplexer bands (sub-bands 1 and 4 for bands 3 and 4, sub-bands 2 and 4 for bands 6 and 7). Similar artefacts are also expected when using modes without reference, such as the FreqSwitchNoRef, or the LoadChopNoRef, and are enhanced with strong continuum sources. HIPE 8 introduces some updates to the FitBaseline task. Please refer to the Data Reduction Guide for details about those.

• Saturation
  • Saturation of the WBS CCDs may be observed at spot frequencies. These are usually due to either strong and broad spurs, or to cases where only marginal pump level could be achieved due to shortage of LO output power. Users can go back to HSpot to visualize in the frequency editor where their AORs were potentially affected by such effects. HIPE will normally flag the affected spectrometer channels.

• Beam efficiencies
  • This update contains a new set of values for the beam and aperture efficiencies. Note that until HIPE 5, a forward efficiency of 1 was used in the calibration files, implying in particular that Level 2 data were calibrated in a Ta' scale instead of the Ta* scale now obtained in HIPE 5. The only difference is that conversion into the Tmb scale should involve a simple division by the beam efficiency for data processed pre-HIPE 5, while they should also be multiplied by the forward efficiency for data processed with HIPE 5 and onward. The easiest way to obtain the scale correctly is to use the task "doMainBeamTemp".

• Side-band ratio and continuum
  • The data processed at level2 are corrected for the fact that HIFI observes in double-side band. In practice, a side-band ratio is assumed over most of the HIFI operational range, which simply means that single-side band intensities are twice those in double-side band. In the limited ranges where this side-band ratio deviates from unity, the intensity scaling will differ for respective USB and LSB spectra. It will also imply that the continuum cannot be trusted in these frequency range. A dedicated task to recover the continuum corresponding to a balanced side-band ratio will be made available in HIPE 9.0.

• H/V polarisation pointing information
  • The observed differences in HIFI horizontal (H) and vertical (V) polarisation line profiles has made it desirable to have astrometry available for both H and V spectrometers. The possibility that small scale structural or velocity variations may be responsible can then be examined. In the past, the pointing information associated with both H- and V- spectra has been that of a synthesized aperture, which is a position midway between the H and V spectrometer beams. From HIPE 4.2 onwards, an option has been introduced in the doPointing task, allowing to compute separate attitude for the respective mixer polarizations. From HIPE 5 onwards, this option is the one used by default. In practice, note that this slight pointing mis-alignment can lead to differences between the H and V signals observed both in spectral lines and continuum, especially when the source structure features abrupt brightness temperature variation on small spatial scales.

• CCD "scratches"
  • The WBS CCDs are known to be affected by so-called "scratches" present on their optics. These small damages can translate into small spikes showing up at particular channels of the WBS data, however they should be removed by the calibration scheme. This removal was improved in particular from HIPE 6.1 onwards. Users who think that they might be still observing such effects should contact the Helpdesk.

• Platforming effect
  • There exists the possibility that residual platforming effect, i.e. imbalance between the average baseline level of individual WBS sub-bands, is still observed at level2. This is usually due to temperature drifts of the CCD during the observations. There is no dedicated tool in the HCSS to correct this effect and users should avoid stitching their data when such artifact is present in the data. Correction of the relative baseline level of each sub-band is left to the user's judgment.

• RA/Dec information in headers
  • The astrometry information provided in the meta-data of the various products has not always been accurate for HIFI data. The headers of each product contains a pair of parameters "ra, dec" which is currently computed based on the average of pointings towards ON and OFF positions. For DBS observations e.g., this will usually give a position ~90" (i.e. half the chopper throw) away from the ON-source position. A more accurate header information is that given by the pair "raNominal, decNominal", which corresponds to the intended co-ordinates, i.e. the ones entered by the User in HSpot. Note that this discrepancy is currently propagated to the Archive products, which uses co-ordinates based on the "ra, dec" meta-data. You should be aware of this when running queries based on co-ordinates. This is fixed in the HIPE 8.0.1 release.

• Quality flags in the quality product
  • Currently, the quality flags at the quality context inside the observation context are just meant for HSC/ICC internal evaluation of the quality of the products and not for the users. In case the data had some serious quality problem, the PI of the program has been contacted about it. Otherwise, only information in the quality summary, when available, should concern the observers.

• Map visualisation and cubes
  • The visualisation software offered in HIPE 8 has had significant improvements since the previous version, especially to help analysing specifically the HIFI mapping data. Note that there are problems in the cubes generated in the case of single line (stripes) maps with HIPE 8.2. In those cases, the cube dimensions are not correct. Also, cubes for raster maps may not be optimal when generated with HIPE versions earlier than 8.2. This will be fixed in HIPE 9

• Deconvolution of Frequency Switching data
  • The deconvolution software will not work properly with frequency switching data. This is intrinsically linked to the fact that the deconvolution algorithm has troubles with constant steps in the frequency dimension, and that frequency switching observations uses uniform frequency throws. This creates some periodic ghost artifacts close to strong lines.

• Reprocessing from raw data
  • Some expert users may want to reprocess their data starting from raw telemetry (so-called level-1). In HIPE 8.2.1 (and expected in 9.0), the system will be expecting a new ACMS product, which does not yet exist in the data currently in the archive (this will be created when the bulk reprocessing with 8.2.1 is completed). In order to avoid problems with this missing products, the following shall be temporarily added in your local configuration property:

hcss.ia.spg.ops.AuxPlugin.products={..., herschel.ia.obs.auxiliary.acms.AcmsTelemetryProduct, ...}

• Other calibration/pipeline problems/needs
  • For more information check the HIFI User's Manuals and/or contact the HSC helpdesk.

HIFI Release Notes

HIFI Dual Beam Switching Observing Modes AOT release note: 9 Mar 2010

HIFI Oberving Modes AOT release note: 11 Apr 2010

HIFI AOT Observing Mode Release and Performance Notes v3.0: 24 September 2011

Other Technical Notes

PACS documents: see PACS Instrument and Calibration webpage

HIFI Information note on Mapping Modes: 30 Jun 2010

HIFI Beam Efficiencies from Mars Observations: 17 Nov 2010

HIFI Information note on the removal of spurs in band 1a and the sampling in the mapping modes: 19 Nov 2010

Calibration Documents

PACS photometer and spectrometer calibration documents: see PACS Instrument and Calibration webpage

SPIRE Photometer Beams (FTP repository)

HIFI System noise temperature IF spectra (FTP repository)

---

blog comments powered by Disqus

---