Herschel Highly Processed Data Product Page

Introduction

This page compiles the plans and needs to prepare and serve Highly Processed Data Products (HPDPs) to the Scientific Communities. There are two main ways considered to fulfil this: a web page, similar to that used for User Provided Data Products (UPDP, see also http://www.cosmos.esa.int/web/herschel/user-provided-data-products), and the HSA User Interface (HUI). The following focusses mostly on the latter.

Meetings and action items

• 16-June-2015: Kick-off meeting with the ICS team and Eva Verdugo.
  • Minutes of Meeting
  • Viewgraphs used during the meeting
  • The main action item resulting from the meeting is that each instrument representative shall come up with a list of use cases and requirement wish list applying to the HPDP they are supervising or acting as contact point for. These requirements will be compiled in the corresponding section below. Deadline: second half of September.
• 16-September-2015: Meeting #2
  • Minutes of Meeting
  • Viewgraphs used during the meeting
• 12-January-2016: Meeting #3
  • Minutes of Meeting
• Next meeting planned Mid-April 2016

HPDP plans per instruments

HPDPs have been identified by each instrument team, although it is likely that not all of them will be possible to prepared in certain cases. We indicate hereafter the HPDPs currently contemplated by each instruments, with some common delivery categories that will be considered or not.

HIFI

PACS-P

PACS-S

In order of priority+feasibility:

HPDP name	Description	Data	Catalogue (tables)	Images (PNGs, etc)	Documentation (Readme, QSummary, etc)	Notes	Prototype delivery
Unchopped observations: pipeline processing with new response drifts correction	Improved transient correction and background subtraction as provided by the new NHSC pipeline scripts	Yes - level2/2.5 equivalent FITS files, or entire obsContext if done via SPG	No	Yes	Could be proposed for 14.1 SPG rather than HPDP	Highly desirable, but requires a fuller validation of this script first (if the script does not provide improved results, neither SPG nor HPDP can be created here). Conditional to NHSC bulk to be run.
Pipeline processing bright sources	Very bright sources ought to be reduced with a different pipeline script (to account for non-linearity/self-curing)	Yes - level2/2.5 equivalent FITS files at a minimum, or entire obsContext if done via SPG	No	Yes		Some level of manual reduction may be necessary, but are probably not many such observations
Off-centred point-source calibrated data	For off-centred point-like sources: SPG products processed through the standard pipeline script with non-standard extraction of final PS spectrum	Yes - level2/2.5 equivalent FITS files	No	Yes		A useful possibility, feasibility will be more clear once QC work (http://herschel.esac.esa.int/twiki/bin/view/Pacs/QualityCommentsSpec) on off-centred observations and identification of likely centred point sources has been done
Long range spectrometer flat-fielding - potential	Correcting the few cases of bad flatfielding	Yes - level2/2.5 equivalent FITS files	No	Yes		May be unnecessary if the new range scan flatfielding works well. Pilar will check a subset of the 14.0 bulk results. Sample relevance analysis.
Unchopped observations: subtracting the baseline	If the continuum in these AOTs cannot be improved (wrt absolute level and shape, less wrt SNR) then instead subtract the baseline	Yes - level2/2.5 equivalent FITS files: baseline subtracted cubes/spectra and cubes of the baseline that was subtracted	No	Yes		Desirable, necessary if the new transients correction pipeline does not becomes SPG in 14.1. Sample relevance analysis.
Unchopped observations: manual processing to correct very bad transients	Improved transient correction and background subtraction provided by processing the data manually	Yes - level2/2.5 equivalent FITS files	No	Yes		Highly desirable, but depends on how many such observations there are and on progress on PACS-6264. Sample relevance analysis.
Point-source calibrated data using the "POC" script	products processed through the dedicated pointing offset correction script for bright point sources	Yes - level2/2.5 equivalent FITS files	No	Yes		Straightforward to do once the pipeline script has been fully validated and sources identified; validation is still on-going (postponed by the 14.0 AAT)
Line list catalogue	Spectral Feature catalogue	No	Yes - one file per obsid, and one concatenated file	No		HPDP: dependent on available human resources, would be completely HSC to do, exploratory studies are being done with help of MSc students at ESAC and Leuven and we should know better early in 2016
Identifying background contamination; removing it?	For ChopNod and Unchopped AOTs, looking at line (and continuum?) contamination in the off positions: primarily to make a quality comment, but if the number is manageable then also to correct for the contamination	Yes - level2/2.5 equivalent FITS files	No	Yes		HPDP, for the quality comment part (just identifying that there is contamination) the ICC had agreed to do this (although work is not yet on-going), but for the subtraction part it would have to be done at HSC, if deemed necessary

These have been done for 14.0 or will be SPG 14.1:

SPIRE-P

SPIRE-S

HUI requirements

We provide here the HUI requirements needed to support the storage, queries and access to the various HPDPs.

TBD

Ancillary Legacy Products

Ancillary Legacy Products are compiled in the following page

-- DavidTeyssier - 13 January 2016