create new tag
view all tags

Herschel Highly Processed Data Product Page


This page compiles the plans and needs to prepare and serve Highly Processed Data Products (HPDPs) to the Scientific Communities. There are two main ways considered to fulfil this: a web page, similar to that used for User Provided Data Products (UPDP, see also http://www.cosmos.esa.int/web/herschel/user-provided-data-products), and the HSA User Interface (HUI). The following focusses mostly on the latter.

Meetings and action items

HPDP release notes

The following document is a template for release notes to be associated with HSC-curated HPDP deliveries.

HPDP plans per instruments

HPDPs have been identified by each instrument team, although it is likely that not all of them will be possible to prepared in certain cases. We indicate hereafter the HPDPs currently contemplated by each instruments, with some common delivery categories that will be considered or not.

Some prioritisation of the associated WPs is given in this page.


HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
Expert Reduced Spectral Scans Baseline-cleaned deconvolved spectral scans Yes - level2.5 equivalent FITS files No Yes - postcards Yes  
Expert Reduced Spectral Maps Baseline-cleaned regridded cubes l scans Yes - level2.5 equivalent FITS files No Yes - postcards Yes  
Isolated OFF spectra OFF spectra as produced by the SPG, potentially cleaned further Yes - level2.5 equivalent FITS files No Yes - postcards Yes  
Non-averaged level2 for point mode Level2 data as produced by modified version of SPG Yes - level2.5 equivalent FITS files (multiple spectra) No Yes - postcards Yes  
Uncertainty tables Uncertainty tables as produced by the SPG No Yes - TableDataset as FITS files No Yes  
Line list catalogue Identified Spectral Line catalogue No Yes - one file per obsid, and one concatenated file Yes - static (PNGs) and interactive (html) images Yes  
Anomalous observations Fix obsids that processed wrongly with 14.1 - see HIFI-6380 Yes - level2.5 equivalent FITS files No Yes - postcards Yes  


HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
PACS Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note  
JScanam HPDP Further combination of fields with JScanam Yes No Yes Yes Available on VOSpace
Additional level 2.5 maps with Unimap Combination of fields currently unconsidered in the list for level2.5 pipeline Yes No ??
Exquisite Processing of GOODS fields with Unimap Use Unimap to combine hundreds of obsids from four GOODS cosmological fields (custodian: B. Altieri) Yes No Yes    
Mini-Scan level 2.5 A series of observations taken as cross-scan maps, albeit with different phot. filters, resulting in the absence of level 2.5 products. The red observations are, however, available in both scanning directions hence level 2.5-type products can be generated. Yes No Yes    


HPDP name Description; product Priority Rational How Requirements Effort Number of obsids Notes
Red-leak spectra Produce the Level 2/2.5 products for spectra in the red leak region; Level 2/2.5 context High Currently the spectra are cut off at just above 190 microns, but we can now produce a reasonably-well line flux calibrated spectrum in this region Using a slightly modified version of the the "calblock" SPG script No extra requirements Automatic, just running an SPG script ~350  
PointingOffsetCorrection spectra for pointed observations Provide the point-source corrected spectrum for pointed observations; possibly focussing only on those with point-like sources not located in the central spaxel in them; a SpectrumTable High The results of the POC script can be better than those provided in the SPG, and this is especially so for off-centred point sources Using the "POC" interactive pipeline script This script has been validated sufficiently to be used for almost any flux level, but we would need to run sanity checks on the results to be sure this applies to the fainter sources. If we limit the provision to point-like sources, off-centred or not, then we need to identify those Most of the effort is on identifying the candidates (if not done for all pointed observations, willy-nilly). Some effort on sanity checking    
Blue spectra (<55mum) Provide the blue end of the spectral range, which is not part of SPG as it is uncalibratable. High A slightly modified version of the SPG script can be used to create these ranges, where we do not the the flux calibration Using a slightly modified ipipe/SPG script need to identify the obsids with this range not much effort, mostly on making sure that this "blue bulk" runs probably and creates correct products    
On and Off cubes provided separately The on-source and off-source Level 2 cubes provided for the unchopped line and all chop-nod observations Mid These provide the most simply way to identify off-source contamination using the SplitOnOff ipipe script for chop-nod, and using the SPG script for unchopped line but saving the products slightly earlier in the Level 2 part   little - scripts ready, just need to modify it to run on all observations and produce the desired FITS files with the desired naming convention. the postcard can be created using the same task as used currently by PACS all unchopped line and chop-nod  
Unchopped obsids with continuum removed Subtract the continuum from all unchopped obsids so as to provide a cleaner spectrum; Level 2/2.5 context Mid The continuum is very uncertain and it had been agreed last year at an ICC meeting that removing this was a "good thing to do” Range scans: use the RV methods to smooth and subtract the continuum; Line scans: use an optimised version of the line-finding task in the flatfielding task of the SPG. Fit/smooth and remove continuum and then produce all final results Need to test methods and thereafter processing should be automatic Producing the scripts: a few weeks (?), checking the results: continuous (but semi-automatic) all ~1200 on-sources unchopped obsid We also need to produce a report on the unchopped mode: based on DF paper but more examples and a proper measure of uncertainty based on e.g. comparing repeat observations, scatter in super-off, etc.
Ad-hoc HPDPs For obsids found to be badly handled by the SPG, an expert will make the necessary adjustments to the pipeline so it can run; Level 2/2.5 context Mid These obsids are either not provided, or the science products are not of the same standard as all other obsids These are obsids identified in an ad-hoc manner as needing help. Jira tickets, Kayako, unchopped with bad transients identified while the unchopped study is done, etc Will only be done if it is feasible in terms of time and manpower   so far <10 candidates suggested  
Line list catalogue Spectral Feature catalogue Nice to have but ICC have no interest here and consider it Low SPIRE and HIFI have one, so why not PACS? Catalogues always produce unexpected science and are a very good Legacy product Need scripts to find lines, fit lines and maybe identify lines Need to create the scripts: KE has a student at Leuven testing out fitting lines, and may have a student in 2016-17 to test out finding lines If the student projects go well, some time in 2017 the resulting scripts could be run on the PACS spectroscopy archive (or subsets of that)   Is being done by KME as a science project
New transients script results for unchopped obsids DE-SCOPED Producing the output of the unchopped obsids reduced with the new transients script Low This script is supposed to produce better results and users may want to use it (esp. if they read the DF paper) %RED; now realised that that this is not the case       ~1200  


HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
SPIRE Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note  
SPIRE HiRes Extract SPIRE-P HiRes products from SPG processing Yes No Yes Yes  
SPIRE destripped maps As names indicates - tentative HPDP Yes No Yes Yes  


HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
Semi-Extended calibration correction SECT correction for semi-compact sources Yes - level2 equivalent FITS files No Yes - postcards Yes  
Extended background correction Extended background correction for point sources Yes - level2 equivalent FITS files No Yes - postcards Yes  
Anomalous data correction Isolated cases of failed processing due to some on-board anomaly (see also DATA-75) Yes - level2 equivalent FITS files No Yes - postcards Yes  
Line list catalogue Spectral Feature catalogue (see also this page ) No Yes - one file per obsid, and one concatenated file Yes Yes - paper  
FTS high absoluteResolution cubes Remove isolated spectra with offending resolution and regenerate final cubes Yes - level2 equivalent FITS files No ? Yes  
Spectral scans Un-averaged level-2 spectra Yes - level2 equivalent products No YES No

Ancillary Data Products

Ancillary Data Products are compiled in the following page

HUI requirements

We provide here the HUI requirements needed to support the storage, queries and access to the various HPDPs and Ancillary Data Products: HPDP_ADP_HUI_Requirement_v1.3.pdf

Highly Processed Data Products Cosmos page

The Cosmos webpages hosting the HPDPs can be found at this link (currently orphan page)

Automatic README file generation

A Python script has been created by Ivan in order to automatically generate the xml README file required in order to ingest HPDPs into the HSA. The script is currently at version 3, with the following instructions.

-- DavidTeyssier - 18 September 2017

Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf HPDP_ADP_HUI_Requirement_v1.2.pdf r1 manage 226.8 K 2016-04-07 - 15:47 DavidTeyssier  
PDFpdf HPDP_ADP_HUI_Requirement_v1.3.pdf r1 manage 241.3 K 2016-04-15 - 12:38 DavidTeyssier  
Microsoft Word filedoc HPDP_Attributes.doc r2 r1 manage 73.5 K 2017-07-04 - 09:41 DavidTeyssier  
Microsoft Word filedocx HPDP_Release_Note_Template.docx r4 r3 r2 r1 manage 95.6 K 2016-12-15 - 13:29 DavidTeyssier  
Microsoft Word filedoc HPDP_meeting_ICS_04July2017_Agenda_v1.0.doc r2 r1 manage 59.5 K 2017-06-28 - 20:08 DavidTeyssier  
PDFpdf HPDP_meeting_ICS_04July2017_MoM.pdf r4 r3 r2 r1 manage 114.4 K 2017-07-07 - 09:51 DavidTeyssier  
PDFpdf HPDP_meeting_ICS_12Jan2016_MoM.pdf r5 r4 r3 r2 r1 manage 100.0 K 2016-01-19 - 06:43 DavidTeyssier  
PDFpdf HPDP_meeting_ICS_14Dec2016_MoM.pdf r2 r1 manage 101.6 K 2016-12-19 - 13:14 DavidTeyssier  
PDFpdf HPDP_meeting_ICS_16June2015.pdf r2 r1 manage 1420.1 K 2015-06-17 - 07:40 DavidTeyssier  
PDFpdf HPDP_meeting_ICS_16June2015_MoM.pdf r2 r1 manage 205.4 K 2015-06-19 - 14:08 DavidTeyssier  
PDFpdf HPDP_meeting_ICS_16Sep2015.pdf r1 manage 3045.6 K 2015-09-22 - 20:55 DavidTeyssier  
PDFpdf HPDP_meeting_ICS_16Sep2015_MoM.pdf r1 manage 70.7 K 2015-09-22 - 20:51 DavidTeyssier  
PDFpdf HPDP_meeting_ICS_19Apr2016_Agenda_v1.0.pdf r1 manage 59.9 K 2016-04-15 - 12:09 DavidTeyssier  
PDFpdf HPDP_meeting_ICS_19Apr2016_MoM.pdf r3 r2 r1 manage 204.8 K 2016-04-20 - 20:44 DavidTeyssier  
PDFpdf HPDP_meeting_ICS_19April2016.pdf r1 manage 2006.5 K 2016-04-20 - 11:26 DavidTeyssier  
PDFpdf HPDP_meeting_ICS_19Sep2017_MoM.pdf r1 manage 125.0 K 2017-09-20 - 21:33 DavidTeyssier  
PDFpdf HPDP_meeting_ICS_29June2016_Agenda_v1.0.pdf r1 manage 52.7 K 2016-06-27 - 21:34 DavidTeyssier  
PDFpdf HPDP_meeting_ICS_29June2016_MoM.pdf r4 r3 r2 r1 manage 89.5 K 2016-07-04 - 07:24 DavidTeyssier  
PDFpdf HPDP_meeting_ICS_29Mar2017_MoM.pdf r2 r1 manage 110.2 K 2017-03-31 - 11:26 DavidTeyssier  
PDFpdf PACSPhpdp.pdf r5 r4 r3 r2 r1 manage 4757.8 K 2016-02-10 - 14:19 LucaCalzoletti  
Unknown file formatmd README-make_xml.md r1 manage 4.4 K 2017-09-18 - 20:54 DavidTeyssier  
PDFpdf SPIRE_S-BKGS-HPDP_Release_Note_v1.pdf r1 manage 2636.4 K 2016-06-06 - 11:48 DavidTeyssier  
PDFpdf flowDiagramsHPDPs.pdf r1 manage 238.2 K 2016-07-04 - 07:04 DavidTeyssier  
Texttxt make_hpdp_xml_v2.py.txt r1 manage 11.0 K 2017-09-18 - 20:52 DavidTeyssier  
Texttxt make_hpdp_xml_v3.py.txt r1 manage 11.2 K 2017-09-20 - 11:50 DavidTeyssier  
Edit | Attach | Watch | Print version | History: r72 < r71 < r70 < r69 < r68 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r72 - 2017-09-20 - DavidTeyssier
This site is powered by the TWiki collaboration platform Powered by Perl