(r60) HPDPDefinitionPage < Public

Public Web>HPDPDefinitionPage (revision 60) (raw view)~~EditAttach~~

---+ Herschel Highly Processed Data Product Page

<br />%TOC{depth="2"}%

---++ Introduction

This page compiles the plans and needs to prepare and serve Highly Processed Data Products (!HPDPs) to the Scientific Communities. There are two main ways considered to fulfil this: a web page, similar to that used for User Provided Data Products (!UPDP, see also [[http://www.cosmos.esa.int/web/herschel/user-provided-data-products]]), and the HSA User Interface (!HUI). The following focusses mostly on the latter.

---++ Meetings and action items

   * 16-June-2015: Kick-off meeting with the ICS team and Eva Verdugo.
      * [[%ATTACHURL%/HPDP_meeting_ICS_16June2015_MoM.pdf Minutes of Meeting]]
      * [[%ATTACHURL%/HPDP_meeting_ICS_16June2015.pdf Viewgraphs used during the meeting]]
      * *The main action item resulting from the meeting is that each instrument representative shall come up with a list of use cases and requirement wish list applying to the !HPDP they are supervising or acting as contact point for. These requirements will be compiled in the corresponding section below. Deadline: second half of September.*
   * 16-September-2015: Meeting #2
      * [[%ATTACHURL%/HPDP_meeting_ICS_16Sep2015_MoM.pdf Minutes of Meeting]]
      * [[%ATTACHURL%/HPDP_meeting_ICS_16Sep2015.pdf Viewgraphs used during the meeting]]
   * 12-January-2016: Meeting #3
      * [[%ATTACHURL%/HPDP_meeting_ICS_12Jan2016_MoM.pdf Minutes of Meeting]]
   * 19-April-2016: Meeting #4<br />
      * [[%ATTACHURL%/HPDP_meeting_ICS_19Apr2016_MoM.pdf Minutes of Meeting]]
      * [[%ATTACHURL%/HPDP_meeting_ICS_19April2016.pdf Viewgraphs used during the meeting]]
   * 29-June-2016: Meeting $5
      * [[%ATTACHURL%/HPDP_meeting_ICS_29June2016_MoM.pdf Minutes of Meeting]]
   * 14-Dec-2016: Meeting #6<br />
      * [[%ATTACHURL%/HPDP_meeting_ICS_14Dec2016_MoM.pdf][Minutes of Meeting]]
---++ !HPDP release notes

The following [[%ATTACHURL%/HPDP_Release_Note_Template.docx document]] is a template for release notes to be associated with HSC-curated HPDP deliveries.
---++ !HPDP plans per instruments

!HPDPs have been identified by each instrument team, although it is likely that not all of them will be possible to prepared in certain cases. We indicate hereafter the !HPDPs currently contemplated by each instruments, with some common delivery categories that will be considered or not.

Some prioritisation of the associated WPs is given in this [[HPDPPriority][page]].
---+++ !HIFI

| *HPDP name*  | *Description*  | *Data*  | *Catalogue (tables)* | *Images (PNGs, etc)* | *Documentation (Readme, QSummary, etc)* | *Prototype delivery* |
| Expert Reduced Spectral Scans  | Baseline-cleaned deconvolved spectral scans  |  Yes - level2.5 equivalent FITS files | No  |  Yes - postcards |  Yes | |
| Expert Reduced Spectral Maps  | Baseline-cleaned regridded cubes l scans  |  Yes - level2.5 equivalent FITS files | No  |  Yes - postcards |  Yes | |
| Isolated OFF spectra  | OFF spectra as produced by the SPG, potentially cleaned further |  Yes - level2.5 equivalent FITS files | No  |  Yes - postcards |  Yes | |
| [[%PUBURL%/Public/HifiDocsEditableTable/HIFI_nonAveragedLevel2_UPDP_ReleaseNote.pdf][Non-averaged level2 for point mode]]  | Level2 data as produced by modified version of SPG |  Yes - level2.5 equivalent FITS files (multiple spectra) | No  |  Yes - postcards |  Yes | |
| [[%PUBURL%/Public/HifiDocsEditableTable/HIFI_UncertaintyTables_HPDP_ReleaseNotes.pdf][Uncertainty tables]]  | Uncertainty tables as produced by the SPG  |  No |  Yes - TableDataset as FITS files  |  No |  Yes | |
| Line list catalogue  | Identified Spectral Line catalogue  |  No | Yes - one file per obsid, and one concatenated file  |  Yes - static (PNGs) and interactive (html) images | Yes | |
| Anomalous observations  | Fix obsids that processed wrongly with 14.1 - see HIFI-6380  |  Yes - level2.5 equivalent FITS files | No  |  Yes - postcards | Yes | |

---+++ !PACS-P

| *HPDP name*  | *Description*  | *Data*  | *Catalogue (tables)* | *Images (PNGs, etc)* | *Documentation (Readme, QSummary, etc)* | *Prototype delivery* |
| !PACS Point Source Catalogue  | Point source catalogue as extracted from the photometer data  |  No | Yes - typically one large csv file per photometer band, and/or a unique concatenated one  |  No |  Yes - Global release note | |
| JScanam HPDP  | Further combination of fields with JScanam |  Yes |  No  |  Yes |  Yes | Available on VOSpace |
| Additional level 2.5 maps with Unimap  | Combination of fields currently unconsidered in the list for level2.5 pipeline |  Yes |  No  |  ?? |  <br /> [[%ATTACHURLPATH%/PACSPhpdp.pdf][PACSPhpdp.pdf]] [[%ATTACHURLPATH%/PACSPhpdp.pdf][<br />]] | |
| Exquisite Processing of GOODS fields with Unimap  | Use Unimap to combine hundreds of obsids from four GOODS cosmological fields (custodian: B. Altieri) |  Yes |  No  |  Yes | | |
| Mini-Scan level 2.5  | A series of observations taken as cross-scan maps, albeit with different phot. filters, resulting in the absence of level 2.5 products. The red observations are, however, available in both scanning directions hence level 2.5-type products can be generated. |  Yes |  No  |  Yes | | |

---+++ !PACS-S

| *HPDP name*  | *Description; product*  | *Priority*  | *Rational* | *How* | *Requirements* | *Effort* | *Number of obsids* | *Notes* |
| Red-leak spectra | Produce the Level 2/2.5 products for spectra in the red leak region; Level 2/2.5 context | High | Currently the spectra are cut off at just above 190 microns, but we can now produce a reasonably-well line flux calibrated spectrum in this region | Using a slightly modified version of the the "calblock" SPG script | No extra requirements | Automatic, just running an SPG script | ~350 | |
| PointingOffsetCorrection spectra for pointed observations | Provide the point-source corrected spectrum for pointed observations; possibly focussing only on those with point-like sources not located in the central spaxel in them; a SpectrumTable  | High  | The results of the POC script can be better than those provided in the SPG, and this is especially so for off-centred point sources | Using the "POC" interactive pipeline script | This script has been validated sufficiently to be used for almost any flux level, but we would need to run sanity checks on the results to be sure this applies to the fainter sources. If we limit the provision to point-like sources, off-centred or not, then we need to identify those | Most of the effort is on identifying the candidates (if not done for all pointed observations, willy-nilly). Some effort on sanity checking  | | |
| Blue spectra (&lt;55mum) | Provide the blue end of the spectral range, which is not part of SPG as it is uncalibratable.  | High  | A slightly modified version of the SPG script can be used to create these ranges, where we do not the the flux calibration | Using a slightly modified ipipe/SPG script | need to identify the obsids with this range | not much effort, mostly on making sure that this "blue bulk" runs probably and creates correct products | | |
| On and Off cubes provided separately | The on-source and off-source Level 2 cubes provided for the unchopped line and all chop-nod observations  | Mid  | These provide the most simply way to identify off-source contamination  | using the SplitOnOff ipipe script for chop-nod, and using the SPG script for unchopped line but saving the products slightly earlier in the Level 2 part  | | little - scripts ready, just need to modify it to run on all observations and produce the desired FITS files with the desired naming convention. the postcard can be created using the same task as used currently by PACS  | all unchopped line and chop-nod  | |
| Unchopped obsids with continuum removed | Subtract the continuum from all unchopped obsids so as to provide a cleaner spectrum; Level 2/2.5 context | Mid | The continuum is very uncertain and it had been agreed last year at an ICC meeting that removing this was a "good thing to do” | Range scans: use the RV methods to smooth and subtract the continuum; Line scans: use an optimised version of the line-finding task in the flatfielding task of the SPG. Fit/smooth and remove continuum and then produce all final results | Need to test methods and thereafter processing should be automatic | Producing the scripts: a few weeks (?), checking the results: continuous (but semi-automatic) | all ~1200 on-sources unchopped obsid | We also need to produce a report on the unchopped mode: based on DF paper but more examples and a proper measure of uncertainty based on e.g. comparing repeat observations, scatter in super-off, etc. |
| Ad-hoc HPDPs | For obsids found to be badly handled by the SPG, an expert will make the necessary adjustments to the pipeline so it can run; Level 2/2.5 context | Mid | These obsids are either not provided, or the science products are not of the same standard as all other obsids | These are obsids identified in an ad-hoc manner as needing help. Jira tickets, Kayako, unchopped with bad transients identified while the unchopped study is done, etc | Will only be done if it is feasible in terms of time and manpower | | so far &lt;10 candidates suggested | |
| Line list catalogue | Spectral Feature catalogue | Nice to have but ICC have no interest here and consider it Low | SPIRE and HIFI have one, so why not PACS? Catalogues always produce unexpected science and are a very good Legacy product | Need scripts to find lines, fit lines and maybe identify lines | Need to create the scripts: KE has a student at Leuven testing out fitting lines, and may have a student in 2016-17 to test out finding lines | If the student projects go well, some time in 2017 the resulting scripts could be run on the PACS spectroscopy archive (or subsets of that) | | Is being done by KME as a science project |
| New transients script results for unchopped obsids %RED%DE-SCOPED%ENDCOLOR% | Producing the output of the unchopped obsids reduced with the new transients script | Low  | This script is supposed to produce better results and users may want to use it (esp. if they read the DF paper) %RED; now realised that that this is not the case%ENDCOLOR% | | | | ~1200  | |

---+++ !SPIRE-P

| *HPDP name*  | *Description*  | *Data*  | *Catalogue (tables)* | *Images (PNGs, etc)* | *Documentation (Readme, QSummary, etc)* | *Prototype delivery* |
| !SPIRE Point Source Catalogue  | Point source catalogue as extracted from the photometer data  |  No | Yes - typically one large csv file per photometer band, and/or a unique concatenated one  |  No |  Yes - Global release note | |
| SPIRE HiRes  | Extract SPIRE-P HiRes products from SPG processing  |  Yes | No  |  Yes |  Yes | |
| SPIRE destripped maps  | As names indicates - tentative HPDP  |  Yes | No  |  Yes |  Yes | |

---+++ !SPIRE-S

| *HPDP name*  | *Description*  | *Data*  | *Catalogue (tables)* | *Images (PNGs, etc)* | *Documentation (Readme, QSummary, etc)* | *Prototype delivery* |
| Semi-Extended calibration correction  | SECT correction for semi-compact sources |  Yes - level2 equivalent FITS files | No  |  Yes - postcards |  Yes | |
| [[%ATTACHURL%/SPIRE_S-BKGS-HPDP_Release_Note_v1.pdf][Extended background correction]]  | Extended background correction for point sources |  Yes - level2 equivalent FITS files | No  |  Yes - postcards |  Yes | |
| Anomalous data correction  | Isolated cases of failed processing due to some on-board anomaly (see also [[http://herschel.esac.esa.int/jira/browse/DATA-75][DATA-75]]) |  Yes - level2 equivalent FITS files | No  |  Yes - postcards |  Yes | |
| Line list catalogue  | Spectral Feature catalogue (see also [[Spire.FtsSpectralFeatureFinder][this page]] ) |  No | Yes - one file per obsid, and one concatenated file  | Yes | Yes - paper | |
| FTS high _absoluteResolution_ cubes  | Remove isolated spectra with offending resolution and regenerate final cubes |  Yes - level2 equivalent FITS files | No  | ? | Yes | |
| Spectral scans | Un-averaged level-2 spectra  | Yes - level2 equivalent products | No | YES | No |
---++ !Ancillary Data Products

Ancillary Data Products are compiled in [[AncillaryDataProductsPage][the following page]]
---++ !HUI requirements

We provide here the HUI requirements needed to support the storage, queries and access to the various !HPDPs and Ancillary Data Products: [[%ATTACHURL%/HPDP_ADP_HUI_Requirement_v1.3.pdf HPDP_ADP_HUI_Requirement_v1.3.pdf]]
---++ !Highly Processed Data Products Cosmos page

The Cosmos webpages hosting the HPDPs can be found at this [[http://www.cosmos.esa.int/web/herschel/highly-processed-data-products][link]] (currently orphan page)

-- Main.DavidTeyssier - 22 July 2016

Attachments

Topic attachments
I	Attachment	History	Action	Size	Date	Who
pdf	HPDP_ADP_HUI_Requirement_v1.2.pdf	r1	manage	226.8 K	2016-04-07 - 15:47	DavidTeyssier
pdf	HPDP_ADP_HUI_Requirement_v1.3.pdf	r1	manage	241.3 K	2016-04-15 - 12:38	DavidTeyssier
docx	HPDP_Release_Note_Template.docx	r4 r3 r2 r1	manage	95.6 K	2016-12-15 - 13:29	DavidTeyssier
pdf	HPDP_meeting_ICS_12Jan2016_MoM.pdf	r5 r4 r3 r2 r1	manage	100.0 K	2016-01-19 - 06:43	DavidTeyssier
pdf	HPDP_meeting_ICS_14Dec2016_MoM.pdf	r2 r1	manage	101.6 K	2016-12-19 - 13:14	DavidTeyssier
pdf	HPDP_meeting_ICS_16June2015.pdf	r2 r1	manage	1420.1 K	2015-06-17 - 07:40	DavidTeyssier
pdf	HPDP_meeting_ICS_16June2015_MoM.pdf	r2 r1	manage	205.4 K	2015-06-19 - 14:08	DavidTeyssier
pdf	HPDP_meeting_ICS_16Sep2015.pdf	r1	manage	3045.6 K	2015-09-22 - 20:55	DavidTeyssier
pdf	HPDP_meeting_ICS_16Sep2015_MoM.pdf	r1	manage	70.7 K	2015-09-22 - 20:51	DavidTeyssier
pdf	HPDP_meeting_ICS_19Apr2016_Agenda_v1.0.pdf	r1	manage	59.9 K	2016-04-15 - 12:09	DavidTeyssier
pdf	HPDP_meeting_ICS_19Apr2016_MoM.pdf	r3 r2 r1	manage	204.8 K	2016-04-20 - 20:44	DavidTeyssier
pdf	HPDP_meeting_ICS_19April2016.pdf	r1	manage	2006.5 K	2016-04-20 - 11:26	DavidTeyssier
pdf	HPDP_meeting_ICS_29June2016_Agenda_v1.0.pdf	r1	manage	52.7 K	2016-06-27 - 21:34	DavidTeyssier
pdf	HPDP_meeting_ICS_29June2016_MoM.pdf	r4 r3 r2 r1	manage	89.5 K	2016-07-04 - 07:24	DavidTeyssier
pdf	PACSPhpdp.pdf	r5 r4 r3 r2 r1	manage	4757.8 K	2016-02-10 - 14:19	LucaCalzoletti
pdf	SPIRE_S-BKGS-HPDP_Release_Note_v1.pdf	r1	manage	2636.4 K	2016-06-06 - 11:48	DavidTeyssier
pdf	flowDiagramsHPDPs.pdf	r1	manage	238.2 K	2016-07-04 - 07:04	DavidTeyssier

Topic revision: r60 - 2017-01-16 - DavidTeyssier

Public

Webs
Public
TWiki