(r14) HPDPDefinitionPage < Public

Public Web>HPDPDefinitionPage (revision 14) (raw view)~~EditAttach~~
---+ Herschel Highly Processed Data Product Page

%TOC{depth="2"}%

---++ Introduction

This page compiles the plans and needs to prepare and serve Highly Processed Data Products (!HPDPs) to the Scientific Communities. There are two main ways considered to fulfil this: a web page, similar to that used for User Provided Data Products (!UPDP, see also [[http://www.cosmos.esa.int/web/herschel/user-provided-data-products]]), and the HSA User Interface (!HUI). The following focusses mostly on the latter.

---++ Meetings and action items

   * 17-June-2015: Kick-off meeting with the ICS team and Eva Verdugo.
      * [[%ATTACHURL%/HPDP_meeting_ICS_16June2015_MoM.pdf Minutes of Meeting]]
      * [[%ATTACHURL%/HPDP_meeting_ICS_16June2015.pdf Viewgraphs used during the meeting]]
      * *The main action item resulting from the meeting is that each instrument representative shall come up with a list of use cases and requirement wish list applying to the !HPDP they are supervising or acting as contact point for. These requirements will be compiled in the corresponding section below. Deadline: second half of September.*
   * Mid-September 2015: Follow-up meeting to review requirements and agree on a consolidated request to !SAT to be conveyed by Eva Verdugo

---++ !HPDP plans per instruments

!HPDPs have been identified by each instrument team, although it is likely that not all of them will be possible to prepared in certain cases. 
We indicate hereafter the !HPDPs currently contemplated by each instruments, with some common delivery categories that will be considered or not.

---+++ !HIFI

| *HPDP name*                                    | *Description*  | *Data*  | *Catalogue (tables)* | *Images (PNGs, etc)* | *Documentation (Readme, QSummary, etc)* | *Prototype delivery* |
| Expert Reduced Spectral Scans          | Baseline-cleaned deconvolved spectral scans  |    Yes - level2.5 equivalent FITS files |   No    |   Yes - postcards |  Yes | |
| Expert Reduced Spectral Maps          | Baseline-cleaned regridded cubes l scans  |    Yes - level2.5 equivalent FITS files |   No    |   Yes - postcards |  Yes | |
| Isolated OFF spectra          | OFF spectra as produced by the SPG, potentially cleaned further |    Yes - level2.5 equivalent FITS files |   No    |   Yes - postcards |  Yes | |
| Non-averaged level2 for point mode          | Level2 data as produced by modified version of SPG |    Yes - level2.5 equivalent FITS files (multiple spectra) |   No    |   Yes - postcards |  Yes | |
| Uncertainty tables          | Uncertainty tables as produced by the SPG  |    No |   Yes - TableDataset as FITS files   |   No |  Yes | |
| Line list catalogue          | Identified Spectral Line catalogue  |    No|   Yes - one file per obsid, and one concatenated file    |   Yes - static (PNGs) and interactive (html) images | Yes | |

---+++ !PACS-P

| *HPDP name*                                    | *Description*  | *Data*  | *Catalogue (tables)* | *Images (PNGs, etc)* | *Documentation (Readme, QSummary, etc)* | *Prototype delivery* |
| !PACS Point Source Catalogue         | Point source catalogue as extracted from the photometer data  |   No |   Yes - typically one large csv file per photometer band,  and/or a unique concatenated one    |  No |  Yes - Global release note | |
| Improved level3 maps         | Further combination of fields with Unimap and JScanam |   Yes |   No   |  ?? |  ?? | |
| Additional level 2.5 maps         | Combination of fields currently unconsidered in the list for level2.5 pipeline |   Yes |   No   |  ?? |  ?? | |

---+++ !PACS-S

In order of priority: these first are HPDPs

| *HPDP name*                                    | *Description*  | *Data*  | *Catalogue (tables)* | *Images (PNGs, etc)* | *Documentation (Readme, QSummary, etc)* | *Notes* | *Prototype delivery* |
| Point-source calibrated data - case 1         | For off-centred sources that are likely to be point sources: SPG products processed through the standard pipeline script with non-standard extraction of final PS spectrum |    Yes - level2/2.5 equivalent FITS files |   No    |   Yes |  | A useful possibility, feasibility will be more clear once QC work on off-centred observations and identification of likely centred point sources has been done | |
| High flux source data        | Very bright sources ought to be reduced with a difference pipeline script (to account for non-linearity/self-curing) | Yes - level2/2.5 equivalent FITS files      |   No    |   Yes | To be proposed for 14.1 SPG |  Some level of manual reduction may be necessary, but are probably not many such observations  | |
| Unchopped observations: response drifts - case 1        | Unchopped range scans severely affected by response drift (e.g. after high glitch rate or after high-flux source observation)  |    Yes - level2/2.5 equivalent FITS files |   No    |   Yes | To be proposed for 14.1 SPG  | Highly desirable | |
| Unchopped observations: transients - case 1       | Baseline-subtracted unchopped spectra  |    Yes - level2/2.5 equivalent FITS files |   No    |   Yes |  | Desirable, if case 2 is not possible this becomes Highly desirable | |
| Unchopped observations: transients - case 2        | Improved transient correction and background subtraction  |    Yes - level2/2.5 equivalent FITS files |   No    |   Yes |  To be proposed for 14.1 SPG | Highly desirable, if it is possible to do this is preferred over case 1  | |
| Identifying background contamination  |  ChopNod and Unchopped, Line (and continuum?) |  Yes - level2/2.5 equivalent FITS files | No | Yes | |HPDP, but depends on how possible it will be to identify these and how many we find  | |
| Point-source calibrated data - case 2         | products processed through the dedicated pointing offset correction script for sources consistent with being centred point source |    Yes - level2/2.5 equivalent FITS files |   No    |   Yes|  |straightforward to do once the pipeline script has been fully validated and sources identified; validation is on-going but not guaranteed | |
| Line list catalogue                         | Spectral Feature catalogue |    No|   Yes - one file per obsid, and one concatenated file    |   No | | HPDP: dependent on available human resources, would be completely HSC to decide we want to do this as is not on ICC plans | |

These should be SPG:

| *HPDP name*                                    | *Description*  | *Data*  | *Catalogue (tables)* | *Images (PNGs, etc)* | *Documentation (Readme, QSummary, etc)* | *Notes* | *Prototype delivery* |
| Point-source calibrated data - case 3         | SPG products processed through the standard pipeline script for sources consistent with being centred point sources |    Yes - level2/2.5 equivalent FITS files |   No    |   Yes|  | SPG | |
| Long range spectro flat-fielding - main       | Including flatfielding for range scans in the SPG  |    Yes - level2/2.5 equivalent FITS files |   No    |  Yes |  | SPG| |
| Long range spectro flat-fielding - potential       | Correcting the few cases of bad flatfielding (probably unnecessary)  |    Yes - level2/2.5 equivalent FITS files |   No    |  Yes |  | HPDP or SPG, depends on whether/how many of these will be present after the final SPG| |
| Red leak spectral range       | Run the specific pipeline that deals with red leak region (!CalBlock+RSRF) and provide data stand-alone |    Yes - level2 equivalent FITS files |   No    |   Yes |  | SPG | |
| Repair of failed observations       | Re-run pipeline on observations set to failed |  Yes - level2/2.5 equivalent FITS files    |   No    |   Yes |  | Observations that are classed as failed but which will now run in SPG are easy to do and could become standard SPG, for others that require manual reduction they could also become the standard SPG, probably not difficult | |

---+++ !SPIRE-P

| *HPDP name*                                    | *Description*  | *Data*  | *Catalogue (tables)* | *Images (PNGs, etc)* | *Documentation (Readme, QSummary, etc)* | *Prototype delivery* |
| !SPIRE Point Source Catalogue         | Point source catalogue as extracted from the photometer data  |   No |   Yes - typically one large csv file per photometer band, and/or a unique concatenated one    |  No |  Yes - Global release note | |
| !HiRes maps  | Reprocessing of HSC.SPIRE-P maps with the !HiRes algorithm |    Yes - level 2 or level 2.5 equivalent FITS files |   No    |   ?? |  ?? | |

---+++ !SPIRE-S

| *HPDP name*                                    | *Description*  | *Data*  | *Catalogue (tables)* | *Images (PNGs, etc)* | *Documentation (Readme, QSummary, etc)* | *Prototype delivery* |
| Semi-Extended calibration correction          | SECT correction for semi-compact sources |    Yes - level2 equivalent FITS files |   No    |   Yes - postcards |  Yes | |
| Extended background correction          | Extended background correction for point sources |    Yes - level2 equivalent FITS files |   No    |   Yes - postcards |  Yes | |
| Anomalous data correction          | Isolated cases of failed processing due to some on-board anomaly |    Yes - level2 equivalent FITS files |   No    |   Yes - postcards |  Yes | |
| Line list catalogue                         | Spectral Feature catalogue (see also [[http://herschel.esac.esa.int/twiki/bin/view/Spire/FtsSpectralFeatureFinder this page]] ) |    No|   Yes - one file per obsid, and one concatenated file    |   No | ?? | |




---++ !HUI requirements

We provide here the HUI requirements needed to support the storage, queries and access to the various !HPDPs.

TBD



-- Main.DavidTeyssier - 17 Jun 2015
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
pdf	HPDP_meeting_ICS_16June2015.pdf	r2 r1	manage	1420.1 K	2015-06-17 - 07:40	DavidTeyssier
pdf	HPDP_meeting_ICS_16June2015_MoM.pdf	r2 r1	manage	205.4 K	2015-06-19 - 14:08	DavidTeyssier
Topic revision: r14 - 2015-09-16 - KatrinaExter
Public
Webs
Public
TWiki