TWiki> Public Web>HPDPDefinitionPage (revision 38)EditAttach

Herschel Highly Processed Data Product Page


This page compiles the plans and needs to prepare and serve Highly Processed Data Products (HPDPs) to the Scientific Communities. There are two main ways considered to fulfil this: a web page, similar to that used for User Provided Data Products (UPDP, see also, and the HSA User Interface (HUI). The following focusses mostly on the latter.

Meetings and action items

  • 16-June-2015: Kick-off meeting with the ICS team and Eva Verdugo.
    • Minutes of Meeting
    • Viewgraphs used during the meeting
    • The main action item resulting from the meeting is that each instrument representative shall come up with a list of use cases and requirement wish list applying to the HPDP they are supervising or acting as contact point for. These requirements will be compiled in the corresponding section below. Deadline: second half of September.
  • 16-September-2015: Meeting #2
  • 12-January-2016: Meeting #3
  • 19-April-2016: Meeting #4

HPDP release notes

The following document is a template for release notes to be associated with HSC-curated HPDP deliveries.

HPDP plans per instruments

HPDPs have been identified by each instrument team, although it is likely that not all of them will be possible to prepared in certain cases. We indicate hereafter the HPDPs currently contemplated by each instruments, with some common delivery categories that will be considered or not.


HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
Expert Reduced Spectral Scans Baseline-cleaned deconvolved spectral scans Yes - level2.5 equivalent FITS files No Yes - postcards Yes  
Expert Reduced Spectral Maps Baseline-cleaned regridded cubes l scans Yes - level2.5 equivalent FITS files No Yes - postcards Yes  
Isolated OFF spectra OFF spectra as produced by the SPG, potentially cleaned further Yes - level2.5 equivalent FITS files No Yes - postcards Yes  
Non-averaged level2 for point mode Level2 data as produced by modified version of SPG Yes - level2.5 equivalent FITS files (multiple spectra) No Yes - postcards Yes  
Uncertainty tables Uncertainty tables as produced by the SPG No Yes - TableDataset as FITS files No Yes  
Line list catalogue Identified Spectral Line catalogue No Yes - one file per obsid, and one concatenated file Yes - static (PNGs) and interactive (html) images Yes  


HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
PACS Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note  
Improved level3 maps Further combination of fields with Unimap and JScanam Yes No ?? ??  
Additional level 2.5 maps Combination of fields currently unconsidered in the list for level2.5 pipeline Yes No ??
Exquisite Processing of GOODS fields with Unimap Use Unimap to combine hundreds of obsids from four GOODS cosmological fields (custodian: B. Altieri) Yes No Yes    


HPDP nameDescription, type of productPriorityRationalHowRequirementsEffortNumber of obsidsNotes
Point-source calibrated spectra: off-centred cases Provide the point-source corrected spectrum for pointed observations with point-like sources not located in the central spaxel; SpectrumTable (type of TableDataset) High All pointed observations have a SpectrumTable with point-source calibrated spectra in it, and if the astronomer knows the source is centred and a point, then this is a science product; if the source is off-centred, however, they would have to use HIPE to produce the PS spectrum Using the "POC" interactive pipeline script This script has been validated suficiently to be used for almost any flux level, but we would need a formal validation report and JB to tell us what parameters to set in the specific tasks Once we have the validation report and parameter values from JB (few days of work), just run the script and produce the standard result. Most of the effort is on identifying the off-centred point-like sources, which will require some investigation (a few weeks?) Need to check all pointed obsids (3500 unchoped and chopnod) but it is likely only a few dozen will produce this HPDP If producing this for off-centred point sources, why not for all point sources?
Red-leak spectra Produce the Level 2/2.5 products for spectra in the red leak region; entire L2/2.5 context High (but see notes) Currently the spectra are cut off at just after 190 microns, but we can now produce a reasonably-well line flux calibrated spectrum in this region Using the "calblock" SPG script Once validation of the red leak RSRF and the flatfielding has been done, this is ready to go Automatic, just running an SPG script There are 533 SED obsids covering this region; will be some line scan and range scan also However, this will probably become part of SPG 14.2

Unchopped obsids with continuum removed

Subtract the continuum from all unchopped obsids so as to provide a cleaner spectrum; Level 2/2.5 context Mid (low on ICC list but high on mine) The continuum is very uncertain and it had been agreed last year at an ICC meeting that removing this was a "good thing to do". It will avoid users being confused about funny continuum levels or shapes and, especially for the targets less than some-10sJy continuum level this will be the science-ready end product Range scans: use the RV methods to smooth and subtract the continuum; Line scans: use an optimised version of the line-finding task in the flatfielding task of the SPG. Fit/smooth and remove continuum and then produce all final results Need to test methods and thereafter process should be automatic Producing the scripts: a few weeks (?), checking the results: continuous (but semi-automatic) all ~1200 on-sources unchopped obsids: We also need to produce a report on the unchopped mode: based on DF paper but more examples and a proper measure of uncertainty based on e.g. comparing repeat observations, scatter in super-off, etc.
New transients script results for unchopped obsids Producing the output of the unchopped obsids reduced with the new transients script Low (because that script has not been validated sufficiently) This script is supposed to produce better results and users may want to use it (esp. if they read the DF paper). It can only be done in HIPE. Running the ipipe script; do the parameters need to be optimised to the target? (unknown) Need to know that this script produces reliable results when run in a semi-automatic way (would be too much work to do all obsids manually) The validation is most of the work: could take several weeks. If the results of the bulk that the NHSC are running can be used, then the validation is the only work to do here Are ~1200 on-source unchopped obsids; Priority on the red camera and the range scans
Unchopped observations with possible long-term transient If observed after a bright source, an unchopped observation will have a long-term transient and hence incorrect continuum level and shape; two possible outcomes (1) just a quality comment warning of this possibilty (2) a corrected Level 2/2.5 context High as QC, Low as HPDP The continuum shape and level will definitely be more wrong for these obsids than for the average unchopped obsid; warning the users of this case is necessary so they know what science they can get, providing corrected results can only be done in HIPE Identify sources taken after a bright observation. Running the new transients script and playing with the background subtraction task To find the candidate sources: look at the calibration block for a different response, look at the V/s of the different scans to look for a transient; look at the flux levels of the obsids preceeding the unchopped one; If the new transients script works well, then can use the NHSC bulk results; if need to play with the script to get the best results, then need to learn how to do this Finding the candidates could take a few weeks; correcting the results is unknown because no-one at HSC or still in the ICC has enough experience with the new transients script Are ~1200 on-source unchopped obsids;
Identifying background contamination; Removing the background contamination

For unchopped and chop-nod. QC: comment on obsids likely to be affected by contamination in the off ; HPDP remove the continamination before doing the off-subtraction and produce a Level 2/2.5 context, or provide the on-source and off-source cubes separately so the user can assess the damage

High as a QC, Low as an HPDP Results in SPG will be definitely not science-ready if the off position is contaminated Need to identifty contaminated sources by comparing spectra in different off positions, spectra again a super-off, looking for line peaks in the off-source spectrum Much effort in finding the sources, as while we have ideas how to do it they have not been tested; removing line contamination is probably fairly straightforward but will require intervention in the tasks of the SPG script, for continuum contamination we we will probably have to just remove a standard background Potentially some months, depends on whether our current idea work out-of-the-box or not All obsids need to be searched, but hoping some dozen of sources are only affected

Hav eto consider dases where there is emision in the off that is also in the on, whether at the same or a different level, or wavelength shifted or not, and also that is not in the on. For both, the SPG results will be wrong, but with the first you may not know (with the second you will probably see absorption lines where the contamination has been subtracted)

Line list catalogue Spectral Feature catalogue Nice to have but ICC have no interest here and consider it Low SPIRE and HIFI have one, so why not PACS. Catalogues always produce unexpected science and are a very good Legacy product Need scripts to find lines, fit lines and maybe identify lines Need to create the scripts: KE has a student at Leuven testing out fitting lines, and may have a student in 2016-17 to test out finding lines If the student projects go well, some time in 2017 the resulting scripts could be run on the PACS spectroscopy archive (or subsets of that)


HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
SPIRE Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note  
<!--| HiRes maps | Reprocessing of SPIRE-P maps with the HiRes algorithm | Yes - level 2 or level 2.5 equivalent FITS files | No | ?? | ?? | | -->


HPDP name Description DataSorted ascending Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
Line list catalogue Spectral Feature catalogue (see also this page ) No Yes - one file per obsid, and one concatenated file Yes Yes - paper  
Semi-Extended calibration correction SECT correction for semi-compact sources Yes - level2 equivalent FITS files No Yes - postcards Yes  
Extended background correction Extended background correction for point sources Yes - level2 equivalent FITS files No Yes - postcards Yes  
Anomalous data correction Isolated cases of failed processing due to some on-board anomaly Yes - level2 equivalent FITS files No Yes - postcards Yes  
Spectral scans Un-averaged level-2 spectra Yes - level2 equivalent products No No No

Ancillary Data Products

Ancillary Data Products are compiled in the following page

HUI requirements

We provide here the HUI requirements needed to support the storage, queries and access to the various HPDPs and Ancillary Data Products: HPDP_ADP_HUI_Requirement_v1.2.pdf

-- DavidTeyssier - 7 April 2016

Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf HPDP_ADP_HUI_Requirement_v1.2.pdf r1 manage 226.8 K 2016-04-07 - 15:47 DavidTeyssier  
PDFpdf HPDP_ALP_HUI_Requirement_v0.1.pdf r1 manage 226.4 K 2016-02-18 - 20:36 DavidTeyssier  
Microsoft Word filedocx HPDP_Release_Note_Template.docx r3 r2 r1 manage 95.2 K 2016-04-07 - 15:32 DavidTeyssier  
PDFpdf HPDP_meeting_ICS_12Jan2016_MoM.pdf r5 r4 r3 r2 r1 manage 100.0 K 2016-01-19 - 06:43 DavidTeyssier  
PDFpdf HPDP_meeting_ICS_16June2015.pdf r2 r1 manage 1420.1 K 2015-06-17 - 07:40 DavidTeyssier  
PDFpdf HPDP_meeting_ICS_16June2015_MoM.pdf r2 r1 manage 205.4 K 2015-06-19 - 14:08 DavidTeyssier  
PDFpdf HPDP_meeting_ICS_16Sep2015.pdf r1 manage 3045.6 K 2015-09-22 - 20:55 DavidTeyssier  
PDFpdf HPDP_meeting_ICS_16Sep2015_MoM.pdf r1 manage 70.7 K 2015-09-22 - 20:51 DavidTeyssier  
PDFpdf PACSPhpdp.pdf r5 r4 r3 r2 r1 manage 4757.8 K 2016-02-10 - 14:19 LucaCalzoletti  
Edit | Attach | Watch | Print version | History: r74 | r40 < r39 < r38 < r37 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r38 - 2016-04-07 - DavidTeyssier
This site is powered by the TWiki collaboration platform Powered by Perl