(r29) HPDPDefinitionPage < Public

Public Web>HPDPDefinitionPage (revision 29) (raw view)~~EditAttach~~
---+ Herschel Highly Processed Data Product Page

<br />%TOC{depth="2"}%

---++ Introduction

This page compiles the plans and needs to prepare and serve Highly Processed Data Products (!HPDPs) to the Scientific Communities. There are two main ways considered to fulfil this: a web page, similar to that used for User Provided Data Products (!UPDP, see also [[http://www.cosmos.esa.int/web/herschel/user-provided-data-products]]), and the HSA User Interface (!HUI). The following focusses mostly on the latter.

---++ Meetings and action items

   * 16-June-2015: Kick-off meeting with the ICS team and Eva Verdugo.
      * [[%ATTACHURL%/HPDP_meeting_ICS_16June2015_MoM.pdf Minutes of Meeting]]
      * [[%ATTACHURL%/HPDP_meeting_ICS_16June2015.pdf Viewgraphs used during the meeting]]
      * *The main action item resulting from the meeting is that each instrument representative shall come up with a list of use cases and requirement wish list applying to the !HPDP they are supervising or acting as contact point for. These requirements will be compiled in the corresponding section below. Deadline: second half of September.*
   * 16-September-2015: Meeting #2
      * [[%ATTACHURL%/HPDP_meeting_ICS_16Sep2015_MoM.pdf Minutes of Meeting]]
      * [[%ATTACHURL%/HPDP_meeting_ICS_16Sep2015.pdf Viewgraphs used during the meeting]]
   * 12-January-2016: Meeting #3
      * [[%ATTACHURL%/HPDP_meeting_ICS_12Jan2016_MoM.pdf Minutes of Meeting]]
   * Next meeting planned Mid-April 2016

---++ !HPDP plans per instruments

!HPDPs have been identified by each instrument team, although it is likely that not all of them will be possible to prepared in certain cases. We indicate hereafter the !HPDPs currently contemplated by each instruments, with some common delivery categories that will be considered or not.

---+++ !HIFI

| *HPDP name*  | *Description*  | *Data*  | *Catalogue (tables)* | *Images (PNGs, etc)* | *Documentation (Readme, QSummary, etc)* | *Prototype delivery* |
| Expert Reduced Spectral Scans  | Baseline-cleaned deconvolved spectral scans  |  Yes - level2.5 equivalent FITS files | No  |  Yes - postcards |  Yes | |
| Expert Reduced Spectral Maps  | Baseline-cleaned regridded cubes l scans  |  Yes - level2.5 equivalent FITS files | No  |  Yes - postcards |  Yes | |
| Isolated OFF spectra  | OFF spectra as produced by the SPG, potentially cleaned further |  Yes - level2.5 equivalent FITS files | No  |  Yes - postcards |  Yes | |
| Non-averaged level2 for point mode  | Level2 data as produced by modified version of SPG |  Yes - level2.5 equivalent FITS files (multiple spectra) | No  |  Yes - postcards |  Yes | |
| Uncertainty tables  | Uncertainty tables as produced by the SPG  |  No |  Yes - TableDataset as FITS files  |  No |  Yes | |
| Line list catalogue  | Identified Spectral Line catalogue  |  No | Yes - one file per obsid, and one concatenated file  |  Yes - static (PNGs) and interactive (html) images | Yes | |

---+++ !PACS-P

| *HPDP name*  | *Description*  | *Data*  | *Catalogue (tables)* | *Images (PNGs, etc)* | *Documentation (Readme, QSummary, etc)* | *Prototype delivery* |
| !PACS Point Source Catalogue  | Point source catalogue as extracted from the photometer data  |  No | Yes - typically one large csv file per photometer band, and/or a unique concatenated one  |  No |  Yes - Global release note | |
| Improved level3 maps  | Further combination of fields with Unimap and JScanam |  Yes |  No  |  ?? |  ?? | |
| Additional level 2.5 maps  | Combination of fields currently unconsidered in the list for level2.5 pipeline |  Yes |  No  |  ?? |  ?? | |

---+++ !PACS-S

In order of priority+feasibility:
| *HPDP name*  | *Description*  | *Data*  | *Catalogue (tables)* | *Images (PNGs, etc)* | *Documentation (Readme, QSummary, etc)* | *Notes* | *Prototype delivery* |
| Pipeline processing bright sources  | Very bright sources ought to be reduced with a different pipeline script (to account for non-linearity/self-curing) | Yes - level2/2.5 equivalent FITS files at a minimum, or entire obsContext if done via SPG  | No  |  Yes | |  Some level of manual reduction may be necessary, but are probably not many such observations  | |
| Off-centred point-source calibrated data  | For off-centred point-like sources: SPG products processed through the standard pipeline script with non-standard extraction of final PS spectrum |  Yes - level2/2.5 equivalent FITS files | No  |  Yes | | A useful possibility, feasibility will be more clear once QC work (http://herschel.esac.esa.int/twiki/bin/view/Pacs/QualityCommentsSpec) on off-centred observations and identification of likely centred point sources has been done | |
| Long range spectrometer flat-fielding - potential  | Correcting the few cases of bad flatfielding |  Yes - level2/2.5 equivalent FITS files | No  |  Yes | | May be unnecessary if the new range scan flatfielding works well. Pilar will check a subset of the 14.0 bulk results. Sample relevance analysis. | |
| Unchopped observations: pipeline processing with new response drifts correction  | Improved transient correction and background subtraction as provided by the new NHSC pipeline scripts  |  Yes - level2/2.5 equivalent FITS files, or entire obsContext if done via SPG | No  |  Yes | Could be proposed for 14.1 SPG rather than HPDP | Highly desirable, but requires a fuller validation of this script first (if the script does not provide improved results, neither SPG nor HPDP can be created here). Conditional to NHSC bulk to be run. | |
| Unchopped observations: subtracting the baseline  | If the continuum in these AOTs cannot be improved (wrt absolute level and shape, less wrt SNR) then instead subtract the baseline  |  Yes - level2/2.5 equivalent FITS files: baseline subtracted cubes/spectra and cubes of the baseline that was subtracted | No  |  Yes | | Desirable, necessary if the new transients correction pipeline does not becomes SPG in 14.1. Sample relevance analysis. | |
| Unchopped observations: manual processing to correct very bad transients  | Improved transient correction and background subtraction provided by processing the data manually |  Yes - level2/2.5 equivalent FITS files | No  |  Yes | | Highly desirable, but depends on how many such observations there are and on progress on PACS-6264. Sample relevance analysis. | |
| Point-source calibrated data using the "POC" script  | products processed through the dedicated pointing offset correction script for bright point sources |  Yes - level2/2.5 equivalent FITS files | No  |  Yes | | Straightforward to do once the pipeline script has been fully validated and sources identified; validation is still on-going (postponed by the 14.0 AAT) | |
| Line list catalogue  | Spectral Feature catalogue |  No | Yes - one file per obsid, and one concatenated file  |  No | | HPDP: dependent on available human resources, would be completely HSC to do, exploratory studies are being done with help of MSc students at ESAC and Leuven and we should know better early in 2016 | |
| Identifying background contamination; removing it?  |  For ChopNod and Unchopped AOTs, looking at line (and continuum?) contamination in the off positions: primarily to make a quality comment, but if the number is manageable then also to correct for the contamination |  Yes - level2/2.5 equivalent FITS files | No | Yes | | HPDP, for the quality comment part (just identifying that there is contamination) the ICC had agreed to do this (although work is not yet on-going), but for the subtraction part it would have to be done at HSC, if deemed necessary  | |

These have been done for 14.0 or will be SPG 14.1:
| *HPDP name*  | *Description*  | *Data*  | *Catalogue (tables)* | *Images (PNGs, etc)* | *Documentation (Readme, QSummary, etc)* | *Notes* | *Prototype delivery* |
| Point-source calibrated data (pointed AOTs only)  | SPG products processed through the standard pipeline script and then creating point source spectra from them |  Yes, part of SPG obsContext | No  |  Yes | | Note: is done on all pointed AOTs, regardless of whether the source is know to be a point or not. Done in 14.0 | |
| Long range spectrometer flat-fielding - main  | Including flatfielding for range scans in the SPG  |  Yes, entire obsContext | No  |  Yes | | depends on validation in AAT 14. Done in 14. | |
| Red leak spectral range  | Run the specific pipeline that deals with red leak region (!CalBlock+RSRF) and provide data stand-alone |  Yes, part of entire obsContext | No  |  Yes | | For 14.1. Requires delivery of red de-leaked rsrf v5 and extra work on range flatfielding. | |
| Repair of failed observations  | Re-run pipeline on observations set to failed | Yes, entire obsContext  | No  |  Yes | | Observations that are classed as failed or delivered for reprocessing were processed in 14.0; of those that fail, are FAILED for good. Work is done by <br />the bulk. | |

---+++ !SPIRE-P

| *HPDP name*  | *Description*  | *Data*  | *Catalogue (tables)* | *Images (PNGs, etc)* | *Documentation (Readme, QSummary, etc)* | *Prototype delivery* |
| !SPIRE Point Source Catalogue  | Point source catalogue as extracted from the photometer data  |  No | Yes - typically one large csv file per photometer band, and/or a unique concatenated one  |  No |  Yes - Global release note | |
<!--| !HiRes maps  | Reprocessing of HSC.SPIRE-P maps with the !HiRes algorithm |  Yes - level 2 or level 2.5 equivalent FITS files | No  |  ?? |  ?? | | -->

---+++ !SPIRE-S

| *HPDP name*  | *Description*  | *Data*  | *Catalogue (tables)* | *Images (PNGs, etc)* | *Documentation (Readme, QSummary, etc)* | *Prototype delivery* |
| Semi-Extended calibration correction  | SECT correction for semi-compact sources |  Yes - level2 equivalent FITS files | No  |  Yes - postcards |  Yes | |
| Extended background correction  | Extended background correction for point sources |  Yes - level2 equivalent FITS files | No  |  Yes - postcards |  Yes | |
| Anomalous data correction  | Isolated cases of failed processing due to some on-board anomaly |  Yes - level2 equivalent FITS files | No  |  Yes - postcards |  Yes | |
| Line list catalogue  | Spectral Feature catalogue (see also [[Spire.FtsSpectralFeatureFinder][this page]] ) |  No | Yes - one file per obsid, and one concatenated file  | Yes | Yes - paper | |
| Spectral scans | Un-averaged level-2 spectra  | Yes - level2 equivalent products | No | No | No |

---++ !HUI requirements

We provide here the HUI requirements needed to support the storage, queries and access to the various !HPDPs.

TBD
---++ !Ancillary Legacy Products

Ancillary Legacy Products are compiled in [[Public.AncillaryLegacyProductsPage][the following page]]

-- Main.DavidTeyssier - 13 January 2016
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who
pdf	HPDP_meeting_ICS_12Jan2016_MoM.pdf	r5 r4 r3 r2 r1	manage	100.0 K	2016-01-19 - 06:43	DavidTeyssier
pdf	HPDP_meeting_ICS_16June2015.pdf	r2 r1	manage	1420.1 K	2015-06-17 - 07:40	DavidTeyssier
pdf	HPDP_meeting_ICS_16June2015_MoM.pdf	r2 r1	manage	205.4 K	2015-06-19 - 14:08	DavidTeyssier
pdf	HPDP_meeting_ICS_16Sep2015.pdf	r1	manage	3045.6 K	2015-09-22 - 20:55	DavidTeyssier
pdf	HPDP_meeting_ICS_16Sep2015_MoM.pdf	r1	manage	70.7 K	2015-09-22 - 20:51	DavidTeyssier
Topic revision: r29 - 2016-01-19 - DavidTeyssier
Public
Webs
Public
TWiki