TWiki> Public Web>HPDPDefinitionPage (revision 12)EditAttach

Herschel Highly Processed Data Product Page

Introduction

This page compiles the plans and needs to prepare and serve Highly Processed Data Products (HPDPs) to the Scientific Communities. There are two main ways considered to fulfil this: a web page, similar to that used for User Provided Data Products (UPDP, see also http://www.cosmos.esa.int/web/herschel/user-provided-data-products), and the HSA User Interface (HUI). The following focusses mostly on the latter.

Meetings and action items

  • 17-June-2015: Kick-off meeting with the ICS team and Eva Verdugo.
    • Minutes of Meeting
    • Viewgraphs used during the meeting
    • The main action item resulting from the meeting is that each instrument representative shall come up with a list of use cases and requirement wish list applying to the HPDP they are supervising or acting as contact point for. These requirements will be compiled in the corresponding section below. Deadline: second half of September.
  • Mid-September 2015: Follow-up meeting to review requirements and agree on a consolidated request to SAT to be conveyed by Eva Verdugo

HPDP plans per instruments

HPDPs have been identified by each instrument team, although it is likely that not all of them will be possible to prepared in certain cases. We indicate hereafter the HPDPs currently contemplated by each instruments, with some common delivery categories that will be considered or not.

HIFI

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
Expert Reduced Spectral Scans Baseline-cleaned deconvolved spectral scans Yes - level2.5 equivalent FITS files No Yes - postcards Yes  
Expert Reduced Spectral Maps Baseline-cleaned regridded cubes l scans Yes - level2.5 equivalent FITS files No Yes - postcards Yes  
Isolated OFF spectra OFF spectra as produced by the SPG, potentially cleaned further Yes - level2.5 equivalent FITS files No Yes - postcards Yes  
Non-averaged level2 for point mode Level2 data as produced by modified version of SPG Yes - level2.5 equivalent FITS files (multiple spectra) No Yes - postcards Yes  
Uncertainty tables Uncertainty tables as produced by the SPG No Yes - TableDataset as FITS files No Yes  
Line list catalogue Identified Spectral Line catalogue No Yes - one file per obsid, and one concatenated file Yes - static (PNGs) and interactive (html) images Yes  

PACS-P

HPDP name Description DataSorted ascending Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
PACS Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note  
Improved level3 maps Further combination of fields with Unimap and JScanam Yes No ?? ??  
Additional level 2.5 maps Combination of fields currently unconsidered in the list for level2.5 pipeline Yes No ?? ??  

PACS-S

In order of priority: these first are HPDPs

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Notes Prototype delivery
High flux source data Very bright sources ought to be reduced with a difference pipeline script (to account for non-linearity/self-curing) Yes - level2/2.5 equivalent FITS files No Yes   Some level of manual reduction may be necessary, but are probably not many such observations  
Unchopped observations: response drifts - case 1 Unchopped range scans severely affected by response drift (e.g. after high glitch rate or after high-flux source observation) Yes - level2/2.5 equivalent FITS files No Yes   Highly desirable  
Unchopped observations: transients - case 1 Baseline-subtracted unchopped spectra Yes - level2/2.5 equivalent FITS files No Yes   Desirable, if case 2 is not possible this becomes Highly desirable  
Unchopped observations: transients - case 2 Improved transient correction and background subtraction Yes - level2/2.5 equivalent FITS files No Yes   Highly desirable, if it is possible to do this is preferred over case 1  
Point-source calibrated data - case 1 For off-centred sources that are likely to be point sources: SPG products processed through the standard pipeline script with non-standard extraction of final PS spectrum Yes - level2/2.5 equivalent FITS files No Yes   A useful possibility, feasibility will be more clear once QC work on off-centred observations and identification of likely centred point sources has been done  

In order of priority: these should be SPG or are of lower priority HPDPs

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Notes Prototype delivery
Point-source calibrated data - case 2 SPG products processed through the dedicated point-source correction script for sources consistent with being centred point source Yes - level2/2.5 equivalent FITS files No Yes   SPG: straightforward to do once the pipeline script has been fully validated and sources identified; validation is on-going  
Point-source calibrated data - case 3 SPG products processed through the standard pipeline script for sources consistent with being centred point sources Yes - level2/2.5 equivalent FITS files No Yes   SPG  
Long range spectro flat-fielding - main Including flatfielding for range scans in the SPG Yes - level2/2.5 equivalent FITS files No Yes   SPG  
Long range spectro flat-fielding - potential Correcting the few cases of bad flatfielding Yes - level2/2.5 equivalent FITS files No Yes   HPDP or SPG, depends on whether/how many of these will be present after the final SPG  
Red leak spectral range Run the specific pipeline that deals with red leak region (CalBlock+RSRF) and provide data stand-alone Yes - level2 equivalent FITS files No Yes   SPG  
Repair of failed observations Re-run pipeline on observations set to failed Yes - level2/2.5 equivalent FITS files No Yes   Observations that are classed as failed but which will now run in SPG are easy to do and could become standard SPG, for others that require manual reduction they could also become the standard SPG, probably not difficult  
Removing background contamination Line (and continuum?) Yes - level2/2.5 equivalent FITS files No Yes   HPDP, but depends on how possible it will be to identify these and how many we find, would be manual reduction and differently for chopNod and unchopped  
Line list catalogue Spectral Feature catalogue No Yes - one file per obsid, and one concatenated file No   HPDP: dependent on available human resources, would be completely HSC to decide we want to do this as is not on ICC plans  

SPIRE-P

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
SPIRE Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note  
HiRes maps Reprocessing of SPIRE-P maps with the HiRes algorithm Yes - level 2 or level 2.5 equivalent FITS files No ?? ??  

SPIRE-S

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
Semi-Extended calibration correction SECT correction for semi-compact sources Yes - level2 equivalent FITS files No Yes - postcards Yes  
Extended background correction Extended background correction for point sources Yes - level2 equivalent FITS files No Yes - postcards Yes  
Anomalous data correction Isolated cases of failed processing due to some on-board anomaly Yes - level2 equivalent FITS files No Yes - postcards Yes  
Line list catalogue Spectral Feature catalogue (see also this page ) No Yes - one file per obsid, and one concatenated file No ??  

HUI requirements

We provide here the HUI requirements needed to support the storage, queries and access to the various HPDPs.

TBD

-- DavidTeyssier - 17 Jun 2015

Edit | Attach | Watch | Print version | History: r74 | r14 < r13 < r12 < r11 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r12 - 2015-07-07 - KatrinaExter
 
This site is powered by the TWiki collaboration platform Powered by Perl