(r44) HPDPDefinitionPage < Public

Public Web>HPDPDefinitionPage (revision 44) (raw view)~~EditAttach~~
---+ Herschel Highly Processed Data Product Page

<br />%TOC{depth="2"}%

---++ Introduction

This page compiles the plans and needs to prepare and serve Highly Processed Data Products (!HPDPs) to the Scientific Communities. There are two main ways considered to fulfil this: a web page, similar to that used for User Provided Data Products (!UPDP, see also [[http://www.cosmos.esa.int/web/herschel/user-provided-data-products]]), and the HSA User Interface (!HUI). The following focusses mostly on the latter.

---++ Meetings and action items

   * 16-June-2015: Kick-off meeting with the ICS team and Eva Verdugo.
      * [[%ATTACHURL%/HPDP_meeting_ICS_16June2015_MoM.pdf Minutes of Meeting]]
      * [[%ATTACHURL%/HPDP_meeting_ICS_16June2015.pdf Viewgraphs used during the meeting]]
      * *The main action item resulting from the meeting is that each instrument representative shall come up with a list of use cases and requirement wish list applying to the !HPDP they are supervising or acting as contact point for. These requirements will be compiled in the corresponding section below. Deadline: second half of September.*
   * 16-September-2015: Meeting #2
      * [[%ATTACHURL%/HPDP_meeting_ICS_16Sep2015_MoM.pdf Minutes of Meeting]]
      * [[%ATTACHURL%/HPDP_meeting_ICS_16Sep2015.pdf Viewgraphs used during the meeting]]
   * 12-January-2016: Meeting #3
      * [[%ATTACHURL%/HPDP_meeting_ICS_12Jan2016_MoM.pdf Minutes of Meeting]]
   * 19-April-2016: Meeting #4<br />
      * [[%ATTACHURL%/HPDP_meeting_ICS_19Apr2016_MoM.pdf Minutes of Meeting]]
      * [[%ATTACHURL%/HPDP_meeting_ICS_19April2016.pdf Viewgraphs used during the meeting]]
---++ !HPDP release notes

The following [[%ATTACHURL%/HPDP_Release_Note_Template.docx document]] is a template for release notes to be associated with HSC-curated HPDP deliveries.
---++ !HPDP plans per instruments

!HPDPs have been identified by each instrument team, although it is likely that not all of them will be possible to prepared in certain cases. We indicate hereafter the !HPDPs currently contemplated by each instruments, with some common delivery categories that will be considered or not.

---+++ !HIFI

| *HPDP name*  | *Description*  | *Data*  | *Catalogue (tables)* | *Images (PNGs, etc)* | *Documentation (Readme, QSummary, etc)* | *Prototype delivery* |
| Expert Reduced Spectral Scans  | Baseline-cleaned deconvolved spectral scans  |  Yes - level2.5 equivalent FITS files | No  |  Yes - postcards |  Yes | |
| Expert Reduced Spectral Maps  | Baseline-cleaned regridded cubes l scans  |  Yes - level2.5 equivalent FITS files | No  |  Yes - postcards |  Yes | |
| Isolated OFF spectra  | OFF spectra as produced by the SPG, potentially cleaned further |  Yes - level2.5 equivalent FITS files | No  |  Yes - postcards |  Yes | |
| Non-averaged level2 for point mode  | Level2 data as produced by modified version of SPG |  Yes - level2.5 equivalent FITS files (multiple spectra) | No  |  Yes - postcards |  Yes | |
| Uncertainty tables  | Uncertainty tables as produced by the SPG  |  No |  Yes - TableDataset as FITS files  |  No |  Yes | |
| Line list catalogue  | Identified Spectral Line catalogue  |  No | Yes - one file per obsid, and one concatenated file  |  Yes - static (PNGs) and interactive (html) images | Yes | |

---+++ !PACS-P

| *HPDP name*  | *Description*  | *Data*  | *Catalogue (tables)* | *Images (PNGs, etc)* | *Documentation (Readme, QSummary, etc)* | *Prototype delivery* |
| !PACS Point Source Catalogue  | Point source catalogue as extracted from the photometer data  |  No | Yes - typically one large csv file per photometer band, and/or a unique concatenated one  |  No |  Yes - Global release note | |
| Improved level3 maps  | Further combination of fields with Unimap and JScanam |  Yes |  No  |  ?? |  ?? | |
| Additional level 2.5 maps  | Combination of fields currently unconsidered in the list for level2.5 pipeline |  Yes |  No  |  ?? |  <br /> [[%ATTACHURLPATH%/PACSPhpdp.pdf][PACSPhpdp.pdf]] [[%ATTACHURLPATH%/PACSPhpdp.pdf][<br />]] | |
| Exquisite Processing of GOODS fields with Unimap  | Use Unimap to combine hundreds of obsids from four GOODS cosmological fields (custodian: B. Altieri) |  Yes |  No  |  Yes | | |

---+++ !PACS-S

<table align="left" border="1" cellpadding="0" cellspacing="1"> <tbody> <tr><th style="text-align: left;">HPDP name</th><th style="text-align: left;">Description, type of product</th><th style="text-align: left;">Priority</th><th>Rational</th><th>How</th><th>Requirements</th><th>Effort</th><th>Number of obsids</th><th>Notes</th></tr> <tr> <td style="text-align: left;">Point-source calibrated spectra: off-centred cases</td> <td>Provide the point-source corrected spectrum for pointed observations with point-like sources not located in the central spaxel; SpectrumTable (type of TableDataset)</td> <td style="text-align: left;">High</td> <td style="text-align: left;">All pointed observations have a SpectrumTable with point-source calibrated spectra in it, and if the astronomer knows the source is centred and a point, then this is a science product; if the source is off-centred, however, they would have to use HIPE to produce the PS spectrum</td> <td style="text-align: right;">Using the "POC" interactive pipeline script</td> <td>This script has been validated suficiently to be used for almost any flux level, but we would need a formal validation report and JB to tell us what parameters to set in the specific tasks</td> <td style="text-align: center;">Once we have the validation report and parameter values from JB (few days of work), just run the script and produce the standard result. Most of the effort is on identifying the off-centred point-like sources, which will require some investigation (a few weeks?)</td> <td>Need to check all pointed obsids (3500 unchoped and chopnod) but it is likely only a few dozen will produce this HPDP</td> <td>If producing this for off-centred point sources, why not for all point sources?</td> </tr> <tr> <td style="text-align: left;">Red-leak spectra</td> <td>Produce the Level 2/2.5 products for spectra in the red leak region; entire L2/2.5 context</td> <td style="text-align: right;">High (but see notes)</td> <td style="text-align: left;">Currently the spectra are cut off at just after 190 microns, but we can now produce a reasonably-well line flux calibrated spectrum in this region</td> <td style="text-align: right;">Using the "calblock" SPG script</td> <td>Once validation of the red leak RSRF and the flatfielding has been done, this is ready to go</td> <td>Automatic, just running an SPG script</td> <td>There are 533 SED obsids covering this region; will be some line scan and range scan also</td> <td>However, this will probably become part of SPG 14.2</td> </tr> <tr> <td style="text-align: left;">

Unchopped obsids with continuum removed</td> <td>Subtract the continuum from all unchopped obsids so as to provide a cleaner spectrum; Level 2/2.5 context</td> <td style="text-align: right;">Mid (low on ICC list but high on mine)</td> <td style="text-align: left;">The continuum is very uncertain and it had been agreed last year at an ICC meeting that removing this was a "good thing to do". It will avoid users being confused about funny continuum levels or shapes and, especially for the targets less than some-10sJy continuum level this will be the science-ready end product</td> <td style="text-align: right;">Range scans: use the RV methods to smooth and subtract the continuum; Line scans: use an optimised version of the line-finding task in the flatfielding task of the SPG. Fit/smooth and remove continuum and then produce all final results</td> <td>Need to test methods and thereafter process should be automatic</td> <td>Producing the scripts: a few weeks (?), checking the results: continuous (but semi-automatic)</td> <td>all ~1200 on-sources unchopped obsids:</td> <td>We also need to produce a report on the unchopped mode: based on DF paper but more examples and a proper measure of uncertainty based on e.g. comparing repeat observations, scatter in super-off, etc.</td> </tr> <tr> <td style="text-align: left;">New transients script results for unchopped obsids</td> <td style="text-align: left;">Producing the output of the unchopped obsids reduced with the new transients script</td> <td style="text-align: right;">Low (because that script has not been validated sufficiently)</td> <td style="text-align: left;">This script is supposed to produce better results and users may want to use it (esp. if they read the DF paper). It can only be done in HIPE.</td> <td style="text-align: right;">Running the ipipe script; do the parameters need to be optimised to the target? (unknown)</td> <td>Need to know that this script produces reliable results when run in a semi-automatic way (would be too much work to do all obsids manually)</td> <td>The validation is most of the work: could take several weeks. If the results of the bulk that the NHSC are running can be used, then the validation is the only work to do here</td> <td>Are ~1200 on-source unchopped obsids; Priority on the red camera and the range scans</td> <td> </td> </tr> <tr> <td style="text-align: left;">Unchopped observations with possible long-term transient</td> <td style="text-align: left;">If observed after a bright source, an unchopped observation will have a long-term transient and hence incorrect continuum level and shape; two possible outcomes (1) just a quality comment warning of this possibilty (2) a corrected Level 2/2.5 context</td> <td style="text-align: right;">High as QC, Low as HPDP</td> <td style="text-align: left;">The continuum shape and level will definitely be more wrong for these obsids than for the average unchopped obsid; warning the users of this case is necessary so they know what science they can get, providing corrected results can only be done in HIPE</td> <td style="text-align: right;">Identify sources taken after a bright observation. Running the new transients script and playing with the background subtraction task</td> <td>To find the candidate sources: look at the calibration block for a different response, look at the V/s of the different scans to look for a transient; look at the flux levels of the obsids preceeding the unchopped one; If the new transients script works well, then can use the NHSC bulk results; if need to play with the script to get the best results, then need to learn how to do this</td> <td>Finding the candidates could take a few weeks; correcting the results is unknown because no-one at HSC or still in the ICC has enough experience with the new transients script</td> <td>Are ~1200 on-source unchopped obsids;</td> <td> </td> </tr> <tr> <td style="text-align: left;">Identifying background contamination; Removing the background contamination</td> <td>

For unchopped and chop-nod. QC: comment on obsids likely to be affected by contamination in the off ; HPDP remove the continamination before doing the off-subtraction and produce a Level 2/2.5 context, or provide the on-source and off-source cubes separately so the user can assess the damage
</td> <td style="text-align: right;">High as a QC, Low as an HPDP</td> <td style="text-align: left;">Results in SPG will be definitely not science-ready if the off position is contaminated</td> <td style="text-align: right;">Need to identifty contaminated sources by comparing spectra in different off positions, spectra again a super-off, looking for line peaks in the off-source spectrum</td> <td>Much effort in finding the sources, as while we have ideas how to do it they have not been tested; removing line contamination is probably fairly straightforward but will require intervention in the tasks of the SPG script, for continuum contamination we we will probably have to just remove a standard background</td> <td>Potentially some months, depends on whether our current idea work out-of-the-box or not</td> <td>All obsids need to be searched, but hoping some dozen of sources are only affected</td> <td>

Hav eto consider dases where there is emision in the off that is also in the on, whether at the same or a different level, or wavelength shifted or not, and also that is not in the on. For both, the SPG results will be wrong, but with the first you may not know (with the second you will probably see absorption lines where the contamination has been subtracted)</td> </tr> <tr> <td style="text-align: left;">Line list catalogue</td> <td>Spectral Feature catalogue</td> <td style="text-align: right;">Nice to have but ICC have no interest here and consider it Low</td> <td style="text-align: left;">SPIRE and HIFI have one, so why not PACS. Catalogues always produce unexpected science and are a very good Legacy product</td> <td style="text-align: right;">Need scripts to find lines, fit lines and maybe identify lines</td> <td>Need to create the scripts: KE has a student at Leuven testing out fitting lines, and may have a student in 2016-17 to test out finding lines</td> <td>If the student projects go well, some time in 2017 the resulting scripts could be run on the PACS spectroscopy archive (or subsets of that)</td> <td> </td> <td> </td> </tr> </tbody> </table>

---+++ !SPIRE-P

| *HPDP name*  | *Description*  | *Data*  | *Catalogue (tables)* | *Images (PNGs, etc)* | *Documentation (Readme, QSummary, etc)* | *Prototype delivery* |
| !SPIRE Point Source Catalogue  | Point source catalogue as extracted from the photometer data  |  No | Yes - typically one large csv file per photometer band, and/or a unique concatenated one  |  No |  Yes - Global release note | |

---+++ !SPIRE-S

| *HPDP name*  | *Description*  | *Data*  | *Catalogue (tables)* | *Images (PNGs, etc)* | *Documentation (Readme, QSummary, etc)* | *Prototype delivery* |
| Semi-Extended calibration correction  | SECT correction for semi-compact sources |  Yes - level2 equivalent FITS files | No  |  Yes - postcards |  Yes | |
| Extended background correction  | Extended background correction for point sources |  Yes - level2 equivalent FITS files | No  |  Yes - postcards |  Yes | |
| Anomalous data correction  | Isolated cases of failed processing due to some on-board anomaly (see also [[http://herschel.esac.esa.int/jira/browse/DATA-75][DATA-75]]) |  Yes - level2 equivalent FITS files | No  |  Yes - postcards |  Yes | |
| Line list catalogue  | Spectral Feature catalogue (see also [[Spire.FtsSpectralFeatureFinder][this page]] ) |  No | Yes - one file per obsid, and one concatenated file  | Yes | Yes - paper | |
| Spectral scans | Un-averaged level-2 spectra  | Yes - level2 equivalent products | No | No | No |
---++ !Ancillary Data Products

Ancillary Data Products are compiled in [[AncillaryDataProductsPage][the following page]]
---++ !HUI requirements

We provide here the HUI requirements needed to support the storage, queries and access to the various !HPDPs and Ancillary Data Products: [[%ATTACHURL%/HPDP_ADP_HUI_Requirement_v1.3.pdf HPDP_ADP_HUI_Requirement_v1.3.pdf]]

-- Main.DavidTeyssier - 7 April 2016
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who
pdf	HPDP_ADP_HUI_Requirement_v1.2.pdf	r1	manage	226.8 K	2016-04-07 - 15:47	DavidTeyssier
pdf	HPDP_ADP_HUI_Requirement_v1.3.pdf	r1	manage	241.3 K	2016-04-15 - 12:38	DavidTeyssier
docx	HPDP_Release_Note_Template.docx	r3 r2 r1	manage	95.2 K	2016-04-07 - 15:32	DavidTeyssier
pdf	HPDP_meeting_ICS_12Jan2016_MoM.pdf	r5 r4 r3 r2 r1	manage	100.0 K	2016-01-19 - 06:43	DavidTeyssier
pdf	HPDP_meeting_ICS_16June2015.pdf	r2 r1	manage	1420.1 K	2015-06-17 - 07:40	DavidTeyssier
pdf	HPDP_meeting_ICS_16June2015_MoM.pdf	r2 r1	manage	205.4 K	2015-06-19 - 14:08	DavidTeyssier
pdf	HPDP_meeting_ICS_16Sep2015.pdf	r1	manage	3045.6 K	2015-09-22 - 20:55	DavidTeyssier
pdf	HPDP_meeting_ICS_16Sep2015_MoM.pdf	r1	manage	70.7 K	2015-09-22 - 20:51	DavidTeyssier
pdf	HPDP_meeting_ICS_19Apr2016_Agenda_v1.0.pdf	r1	manage	59.9 K	2016-04-15 - 12:09	DavidTeyssier
pdf	HPDP_meeting_ICS_19Apr2016_MoM.pdf	r3 r2 r1	manage	204.8 K	2016-04-20 - 20:44	DavidTeyssier
pdf	HPDP_meeting_ICS_19April2016.pdf	r1	manage	2006.5 K	2016-04-20 - 11:26	DavidTeyssier
pdf	PACSPhpdp.pdf	r5 r4 r3 r2 r1	manage	4757.8 K	2016-02-10 - 14:19	LucaCalzoletti
Topic revision: r44 - 2016-05-17 - DavidTeyssier
Public
Webs
Public
TWiki