Herschel Highly Processed Data Product Page

Introduction

This page compiles the plans and needs to prepare and serve Highly Processed Data Products (HPDPs) to the Scientific Communities. There are two main ways considered to fulfil this: a web page, similar to that used for User Provided Data Products (UPDP, see also http://www.cosmos.esa.int/web/herschel/user-provided-data-products), and the HSA User Interface (HUI). The following focusses mostly on the latter.

Meetings and action items

  • 17-June-2015: Kick-off meeting with the ICS team and Eva Verdugo.
    • Minutes of Meeting
    • Viewgraphs used during the meeting
    • The main action item resulting from the meeting is that each instrument representative shall come up with a list of use cases and requirement wish list applying to the HPDP they are supervising or acting as contact point for. These requirements will be compiled in the corresponding section below. Deadline: second half of September.
  • Mid-September 2015: Follow-up meeting to review requirements and agree on a consolidated request to SAT to be conveyed by Eva Verdugo

HPDP plans per instruments

HPDPs have been identified by each instrument team, although it is likely that not all of them will be possible to prepared in certain cases. We indicate hereafter the HPDPs currently contemplated by each instruments, with some common delivery categories that will be considered or not.

HIFI

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
Expert Reduced Spectral Scans Baseline-cleaned deconvolved spectral scans Yes - level2.5 equivalent FITS files No Yes - postcards Yes
Expert Reduced Spectral Maps Baseline-cleaned regridded cubes l scans Yes - level2.5 equivalent FITS files No Yes - postcards Yes
Isolated OFF spectra OFF spectra as produced by the SPG, potentially cleaned further Yes - level2.5 equivalent FITS files No Yes - postcards Yes
Non-averaged level2 for point mode Level2 data as produced by modified version of SPG Yes - level2.5 equivalent FITS files (multiple spectra) No Yes - postcards Yes
Uncertainty tables Uncertainty tables as produced by the SPG No Yes - TableDataset as FITS files No Yes
Line list catalogue Identified Spectral Line catalogue No Yes - one file per obsid, and one concatenated file Yes - static (PNGs) and interactive (html) images Yes

PACS-P

HPDP nameSorted ascending Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
PACS Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note
Additional level 2.5 maps Combination of fields currently unconsidered in the list for level2.5 pipeline Yes No ?? ??
Improved level3 maps Further combination of fields with Unimap and JScanam Yes No ?? ??

PACS-S

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) notes
Point-source calibrated data - case 1 SPG products processed through the dedicated point-source correction script for cases of centred point source Yes - level2/2.5 equivalent FITS files No Yes   Straightforward to do once the pipeline script has been fully validated and sources identified; validation is on-going
Point-source calibrated data - case 2 SPG products processed through the standard pipeline script for cases of centred point sources Yes - level2/2.5 equivalent FITS files No Yes   Aiming for SPG 14 in fact, so should be part of SPG rather than HPDP
Point-source calibrated data - case 3 SPG products processed through the standard pipeline script for cases of off-centred point sources Yes - level2/2.5 equivalent FITS files No Yes   A possibility, difficult to identify, requires production of the uneven illumination correction task (which is on-going work)
Unchopped observations - case 1 Baseline-subtracted unchopped spectra Yes - level2/2.5 equivalent FITS files No Yes   Desirable, if case 2 is not possible this becomes Highly desirable
Unchopped observations - case 2 Improved transient correction and background subtraction Yes - level2/2.5 equivalent FITS files No Yes   Highly desirable, if it is possible to do this is preferred over case 1
Long range spectro flat-fielding Manual correction of the flat-fielding drift Yes - level2/2.5 equivalent FITS files No Yes   Must-have, however the aim is to have this as part of the SPG, and only those that still have problems would require manual correction
Red leak spectral range Run the specific pipeline that deals with red leak region (CalBlock+RSRF) and provide data stand-alone Yes - level2 equivalent FITS files No Yes   Must-have
High flux source data Very bright sources ought to be reduced with a difference pipeline script Yes - level2/2.5 equivalent FITS files No Yes   Some level of manual reduction may be necessary, but probably are not many such observations
Removing background contamination - case 1 chopNod mode Line and continuum Yes - level2/2.5 equivalent FITS files No Yes   Nice to have, depends on how possible it will be to identify these, would be manual reduction
Removing background contamination - case 2 unchopped mode Line and continuum Yes - level2/2.5 equivalent FITS files No Yes   Nice to have, easier to identify that for chopNod, but would here also be manual reduction
Repair of failed observations Re-run pipeline on observations set to failed Yes - level2/2.5 equivalent FITS files No Yes   Observations that are classed as failed but which will now run in SPG are easy to do and could become standard SPG, for others that require manual reduction they could also become the standard SPG, probably not difficult
Different wavelength regridding Re-run SPG with different wavelength grid parameter so the fluxes are no dependent and the noise characteristics not changed Yes - level2/2.5 equivalent FITS files No Yes   Has not been discussed by ICC yet, so whether this replaces SPG or is provided as an HPDP or done at all, has not yet been discussed
Line list catalogue Spectral Feature catalogue No Yes - one file per obsid, and one concatenated file No   dependent on available human resources

SPIRE-P

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
SPIRE Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note
HiRes maps Reprocessing of SPIRE-P maps with the HiRes algorithm Yes - level 2 or level 2.5 equivalent FITS files No ?? ??

SPIRE-S

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
Semi-Extended calibration correction SECT correction for semi-compact sources Yes - level2 equivalent FITS files No Yes - postcards Yes
Extended background correction Extended background correction for point sources Yes - level2 equivalent FITS files No Yes - postcards Yes
Anomalous data correction Isolated cases of failed processing due to some on-board anomaly Yes - level2 equivalent FITS files No Yes - postcards Yes
Line list catalogue Spectral Feature catalogue (see also this page ) No Yes - one file per obsid, and one concatenated file No ??

HUI requirements

We provide here the HUI requirements needed to support the storage, queries and access to the various HPDPs.

TBD

-- DavidTeyssier - 17 Jun 2015


This topic: Public > HSC > HPDPDefinitionPage
Topic revision: r8 - 2015-06-23 - KatrinaExter
 
This site is powered by the TWiki collaboration platform Powered by Perl