TWiki> Public Web>HPDPDefinitionPage (revision 18)EditAttach

Herschel Highly Processed Data Product Page

Introduction

This page compiles the plans and needs to prepare and serve Highly Processed Data Products (HPDPs) to the Scientific Communities. There are two main ways considered to fulfil this: a web page, similar to that used for User Provided Data Products (UPDP, see also http://www.cosmos.esa.int/web/herschel/user-provided-data-products), and the HSA User Interface (HUI). The following focusses mostly on the latter.

Meetings and action items

  • 16-June-2015: Kick-off meeting with the ICS team and Eva Verdugo.
    • Minutes of Meeting
    • Viewgraphs used during the meeting
    • The main action item resulting from the meeting is that each instrument representative shall come up with a list of use cases and requirement wish list applying to the HPDP they are supervising or acting as contact point for. These requirements will be compiled in the corresponding section below. Deadline: second half of September.
  • 16-September-2015: Meeting #2
  • Next meeting planned Mid-December 2015

HPDP plans per instruments

HPDPs have been identified by each instrument team, although it is likely that not all of them will be possible to prepared in certain cases. We indicate hereafter the HPDPs currently contemplated by each instruments, with some common delivery categories that will be considered or not.

HIFI

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
Expert Reduced Spectral Scans Baseline-cleaned deconvolved spectral scans Yes - level2.5 equivalent FITS files No Yes - postcards Yes  
Expert Reduced Spectral Maps Baseline-cleaned regridded cubes l scans Yes - level2.5 equivalent FITS files No Yes - postcards Yes  
Isolated OFF spectra OFF spectra as produced by the SPG, potentially cleaned further Yes - level2.5 equivalent FITS files No Yes - postcards Yes  
Non-averaged level2 for point mode Level2 data as produced by modified version of SPG Yes - level2.5 equivalent FITS files (multiple spectra) No Yes - postcards Yes  
Uncertainty tables Uncertainty tables as produced by the SPG No Yes - TableDataset as FITS files No Yes  
Line list catalogue Identified Spectral Line catalogue No Yes - one file per obsid, and one concatenated file Yes - static (PNGs) and interactive (html) images Yes  

PACS-P

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
PACS Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note  
Improved level3 maps Further combination of fields with Unimap and JScanam Yes No ?? ??  
Additional level 2.5 maps Combination of fields currently unconsidered in the list for level2.5 pipeline Yes No ?? ??  

PACS-S

In order of priority/doability:

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Notes Prototype delivery
Pipeline processing bright sources Very bright sources ought to be reduced with a difference pipeline script (to account for non-linearity/self-curing) Yes - level2/2.5 equivalent FITS files at a minimum, or entire obsContext if done via SPG No Yes To be proposed for 14.1 SPG rather than provide as an HPDP Some level of manual reduction may be necessary, but are probably not many such observations  
Point-source calibrated data For off-centred point sources: SPG products processed through the standard pipeline script with non-standard extraction of final PS spectrum Yes - level2/2.5 equivalent FITS files No Yes   A useful possibility, feasibility will be more clear once QC work (http://herschel.esac.esa.int/twiki/bin/view/Pacs/QualityCommentsSpec) on off-centred observations and identification of likely centred point sources has been done  
Long range spectrometer flat-fielding - potential Correcting the few cases of bad flatfielding Yes - level2/2.5 equivalent FITS files No Yes   may be unnecessary if the new range scan flatfielding works well  
Unchopped observations: pipeline processing with new response drifts correction Improved transient correction and background subtraction as provided by the new NHSC pipeline scripts Yes - level2/2.5 equivalent FITS files, or entire obsContext if done via SPG No Yes To be proposed for 14.1 SPG rather than provide as HPSP Highly desirable, but requires a fuller validation of this script first (if the script does not provide improved results, neither SPG nor HPDP can be created here)  
Unchopped observations: subtracting the baseline If the continuum in these AOTs cannot be improved (wrt absolute level and shape, less wrt SNR) then instead subtract the baseline Yes - level2/2.5 equivalent FITS files: baseline subtracted cubes/spectra and cubes of the baseline that was subtracted No Yes   Desirable, necessary if the new transients correction pipeline does not becomes SPG in 14.1  
Unchopped observations: manual processing to correct very bad transients Improved transient correction and background subtraction provided by processing the data manually Yes - level2/2.5 equivalent FITS files No Yes   Highly desirable, but depends on how many such observations there are and on progress on PACS-6462  
Point-source calibrated data using the "POC" script products processed through the dedicated pointing offset correction script for bright point sources Yes - level2/2.5 equivalent FITS files No Yes   straightforward to do once the pipeline script has been fully validated and sources identified; validation is on-going (but not guaranteed to finish)  
Line list catalogue Spectral Feature catalogue No Yes - one file per obsid, and one concatenated file No   HPDP: dependent on available human resources, would be completely HSC to do, exploratory studies are being done with help of MSc students at ESAC and Leuven and we should know better early in 2016  
Identifying background contamination; removing it? For ChopNod and Unchopped AOTs, looking at line (and continuum?) contamination in the off positions: primarily to make a quality comment, but if the number is manageable then also to correct for the contamination Yes - level2/2.5 equivalent FITS files No Yes   HPDP, for the quality comment part the ICC have agreed to do this, but for the subtraction part it would have to be done at HSC (note: unlikely to ever actually be completed, however, due to it being a difficult job)  

These should actually be SPG (14 or 14.1):

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Notes Prototype delivery
Point-source calibrated data (pointed AOTs only) SPG products processed through the standard pipeline script and then creating point source spectra from them Yes, part of SPG obsContext No Yes   Note: is done on all pointed AOTs, regardless of whether the source is know to be a point or not  
Long range spectrometer flat-fielding - main Including flatfielding for range scans in the SPG Yes, entire obsContext No Yes   depends on validation in AAT 14 but is looking good  
Red leak spectral range Run the specific pipeline that deals with red leak region (CalBlock+RSRF) and provide data stand-alone Yes, part of entire obsContext No Yes   Hoping for 14, but more likely to be 14.1  
Repair of failed observations Re-run pipeline on observations set to failed Yes, entire obsContext No Yes   Observations that are classed as failed or delivered for reprocessing will be processed in 14; of those that fail, for those that can none-the-less be processed manually and are worth so-doing  

SPIRE-P

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
SPIRE Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note  
HiRes maps Reprocessing of SPIRE-P maps with the HiRes algorithm Yes - level 2 or level 2.5 equivalent FITS files No ?? ??  

SPIRE-S

HPDP name Description Data Catalogue (tables) Images (PNGs, etc)Sorted ascending Documentation (Readme, QSummary, etc) Prototype delivery
Line list catalogue Spectral Feature catalogue (see also this page ) No Yes - one file per obsid, and one concatenated file No ??  
Convolved spectral cubes Apply gridding algorithm to the spectral 2d pre-cubes Yes - level2 equivalent products No No No
Spectral scans Un-averaged level-2 spectra Yes - level2 equivalent products No No No
Semi-Extended calibration correction SECT correction for semi-compact sources Yes - level2 equivalent FITS files No Yes - postcards Yes  
Extended background correction Extended background correction for point sources Yes - level2 equivalent FITS files No Yes - postcards Yes  
Anomalous data correction Isolated cases of failed processing due to some on-board anomaly Yes - level2 equivalent FITS files No Yes - postcards Yes  

HUI requirements

We provide here the HUI requirements needed to support the storage, queries and access to the various HPDPs.

TBD

-- DavidTeyssier - 17 Jun 2015

Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf HPDP_meeting_ICS_16June2015.pdf r2 r1 manage 1420.1 K 2015-06-17 - 07:40 DavidTeyssier  
PDFpdf HPDP_meeting_ICS_16June2015_MoM.pdf r2 r1 manage 205.4 K 2015-06-19 - 14:08 DavidTeyssier  
PDFpdf HPDP_meeting_ICS_16Sep2015.pdf r1 manage 3045.6 K 2015-09-22 - 20:55 DavidTeyssier  
PDFpdf HPDP_meeting_ICS_16Sep2015_MoM.pdf r1 manage 70.7 K 2015-09-22 - 20:51 DavidTeyssier  
Edit | Attach | Watch | Print version | History: r74 | r20 < r19 < r18 < r17 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r18 - 2015-09-22 - DavidTeyssier
 
This site is powered by the TWiki collaboration platform Powered by Perl