Difference: HPDPDefinitionPage (33 vs. 34)

Revision 342016-03-01 - KatrinaExter

Line: 1 to 1

Herschel Highly Processed Data Product Page

Line: 44 to 44


In order of priority+feasibility:
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Notes Prototype delivery
Pipeline processing bright sources Very bright sources ought to be reduced with a different pipeline script (to account for non-linearity/self-curing) Yes - level2/2.5 equivalent FITS files at a minimum, or entire obsContext if done via SPG No Yes   Some level of manual reduction may be necessary, but are probably not many such observations  
Off-centred point-source calibrated data For off-centred point-like sources: SPG products processed through the standard pipeline script with non-standard extraction of final PS spectrum Yes - level2/2.5 equivalent FITS files No Yes   A useful possibility, feasibility will be more clear once QC work (http://herschel.esac.esa.int/twiki/bin/view/Pacs/QualityCommentsSpec) on off-centred observations and identification of likely centred point sources has been done  
Long range spectrometer flat-fielding - potential Correcting the few cases of bad flatfielding Yes - level2/2.5 equivalent FITS files No Yes   May be unnecessary if the new range scan flatfielding works well. Pilar will check a subset of the 14.0 bulk results. Sample relevance analysis.  
Unchopped observations: pipeline processing with new response drifts correction Improved transient correction and background subtraction as provided by the new NHSC pipeline scripts Yes - level2/2.5 equivalent FITS files, or entire obsContext if done via SPG No Yes Could be proposed for 14.1 SPG rather than HPDP Highly desirable, but requires a fuller validation of this script first (if the script does not provide improved results, neither SPG nor HPDP can be created here). Conditional to NHSC bulk to be run.  
Unchopped observations: subtracting the baseline If the continuum in these AOTs cannot be improved (wrt absolute level and shape, less wrt SNR) then instead subtract the baseline Yes - level2/2.5 equivalent FITS files: baseline subtracted cubes/spectra and cubes of the baseline that was subtracted No Yes   Desirable, necessary if the new transients correction pipeline does not becomes SPG in 14.1. Sample relevance analysis.  
Unchopped observations: manual processing to correct very bad transients Improved transient correction and background subtraction provided by processing the data manually Yes - level2/2.5 equivalent FITS files No Yes   Highly desirable, but depends on how many such observations there are and on progress on PACS-6264. Sample relevance analysis.  
Point-source calibrated data using the "POC" script products processed through the dedicated pointing offset correction script for bright point sources Yes - level2/2.5 equivalent FITS files No Yes   Straightforward to do once the pipeline script has been fully validated and sources identified; validation is still on-going (postponed by the 14.0 AAT)  
Line list catalogue Spectral Feature catalogue No Yes - one file per obsid, and one concatenated file No   HPDP: dependent on available human resources, would be completely HSC to do, exploratory studies are being done with help of MSc students at ESAC and Leuven and we should know better early in 2016  
Identifying background contamination; removing it? For ChopNod and Unchopped AOTs, looking at line (and continuum?) contamination in the off positions: primarily to make a quality comment, but if the number is manageable then also to correct for the contamination Yes - level2/2.5 equivalent FITS files No Yes   HPDP, for the quality comment part (just identifying that there is contamination) the ICC had agreed to do this (although work is not yet on-going), but for the subtraction part it would have to be done at HSC, if deemed necessary  

These have been done for 14.0 or will be SPG 14.1:

HPDP name Description Data
<-- -->
Sorted ascending
Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Notes Prototype delivery
Long range spectrometer flat-fielding - main Including flatfielding for range scans in the SPG Yes, entire obsContext No Yes   depends on validation in AAT 14. Done in 14.  
Repair of failed observations Re-run pipeline on observations set to failed Yes, entire obsContext No Yes   Observations that are classed as failed or delivered for reprocessing were processed in 14.0; of those that fail, are FAILED for good. Work is done by
the bulk.
Red leak spectral range Run the specific pipeline that deals with red leak region (CalBlock+RSRF) and provide data stand-alone Yes, part of entire obsContext No Yes   For 14.1. Requires delivery of red de-leaked rsrf v5 and extra work on range flatfielding.  
Point-source calibrated data (pointed AOTs only) SPG products processed through the standard pipeline script and then creating point source spectra from them Yes, part of SPG obsContext No Yes   Note: is done on all pointed AOTs, regardless of whether the source is know to be a point or not. Done in 14.0  
HPDP nameDescription, type of productPriorityRationalHowRequirementsEffortNumber of obsidsNotes
Point-source calibrated spectra: off-centred cases Provide the point-source corrected spectrum for pointed observations with point-like sources not located in the central spaxel; SpectrumTable (type of TableDataset) High All pointed observations have a SpectrumTable with point-source calibrated spectra in it, and if the astronomer knows the source is centred and a point, then this is a science product; if the source is off-centred, however, they would have to use HIPE to produce the PS spectrum Using the "POC" interactive pipeline script This script has been validated suficiently to be used for almost any flux level, but we would need a formal validation report and JB to tell us what parameters to set in the specific tasks Once we have the validation report and parameter values from JB (few days of work), just run the script and produce the standard result. Most of the effort is on identifying the off-centred point-like sources, which will require some investigation (a few weeks?) Need to check all pointed obsids (3500 unchoped and chopnod) but it is likely only a few dozen will produce this HPDP If producing this for off-centred point sources, why not for all point sources?
Red-leak spectra Produce the Level 2/2.5 products for spectra in the red leak region; entire L2/2.5 context High (but see notes) Currently the spectra are cut off at just after 190 microns, but we can now produce a reasonably-well line flux calibrated spectrum in this region Using the "calblock" SPG script Once validation of the red leak RSRF and the flatfielding has been done, this is ready to go Automatic, just running an SPG script There are 533 SED obsids covering this region; will be some line scan and range scan also However, this will probably become part of SPG 14.2

Unchopped obsids with continuum removed

Subtract the continuum from all unchopped obsids so as to provide a cleaner spectrum; Level 2/2.5 context Mid (low on ICC list but high on mine) The continuum is very uncertain and it had been agreed last year at an ICC meeting that removing this was a "good thing to do". It will avoid users being confused about funny continuum levels or shapes and, especially for the targets less than some-10sJy continuum level this will be the science-ready end product Range scans: use the RV methods to smooth and subtract the continuum; Line scans: use an optimised version of the line-finding task in the flatfielding task of the SPG. Fit/smooth and remove continuum and then produce all final results Need to test methods and thereafter process should be automatic Producing the scripts: a few weeks (?), checking the results: continuous (but semi-automatic) all ~1200 on-sources unchopped obsids: We also need to produce a report on the unchopped mode: based on DF paper but more examples and a proper measure of uncertainty based on e.g. comparing repeat observations, scatter in super-off, etc.
New transients script results for unchopped obsids Producing the output of the unchopped obsids reduced with the new transients script Low (because that script has not been validated sufficiently) This script is supposed to produce better results and users may want to use it (esp. if they read the DF paper). It can only be done in HIPE. Running the ipipe script; do the parameters need to be optimised to the target? (unknown) Need to know that this script produces reliable results when run in a semi-automatic way (would be too much work to do all obsids manually) The validation is most of the work: could take several weeks. If the results of the bulk that the NHSC are running can be used, then the validation is the only work to do here Are ~1200 on-source unchopped obsids; Priority on the red camera and the range scans
Unchopped observations with possible long-term transient If observed after a bright source, an unchopped observation will have a long-term transient and hence incorrect continuum level and shape; two possible outcomes (1) just a quality comment warning of this possibilty (2) a corrected Level 2/2.5 context High as QC, Low as HPDP The continuum shape and level will definitely be more wrong for these obsids than for the average unchopped obsid; warning the users of this case is necessary so they know what science they can get, providing corrected results can only be done in HIPE Identify sources taken after a bright observation. Running the new transients script and playing with the background subtraction task To find the candidate sources: look at the calibration block for a different response, look at the V/s of the different scans to look for a transient; look at the flux levels of the obsids preceeding the unchopped one; If the new transients script works well, then can use the NHSC bulk results; if need to play with the script to get the best results, then need to learn how to do this Finding the candidates could take a few weeks; correcting the results is unknown because no-one at HSC or still in the ICC has enough experience with the new transients script Are ~1200 on-source unchopped obsids;
Identifying background contamination; Removing the background contamination For unchopped and chop-nod. QC: comment on obsids likely to be affected by contamination in the off; HPDP remove the continamination before doing the off-subtraction and produce a Level 2/2.5 context High as a QC, Low as an HPDP Results in SPG will be definitely not science-ready if the off position is contaminated Need to identifty contaminated sources by comparing spectra in different off positions, spectra again a super-off, looking for line peaks in the off-source spectrum Much effort in finding the sources, as while we have ideas how to do it they have not been tested; removing line contamination is probably fairly straightforward but will require intervention in the tasks of the SPG script, for continuum contamination we we will probably have to just remove a standard background Potentially some months, depends on whether our current idea work out-of-the-box or not All obsids need to be searched, but hoping some dozen of sources are only affected
Line list catalogue Spectral Feature catalogue Nice to have but ICC have no interest here and consider it Low SPIRE and HIFI have one, so why not PACS. Catalogues always produce unexpected science and are a very good Legacy product Need scripts to find lines, fit lines and maybe identify lines Need to create the scripts: KE has a student at Leuven testing out fitting lines, and may have a student in 2016-17 to test out finding lines If the student projects go well, some time in 2017 the resulting scripts could be run on the PACS spectroscopy archive (or subsets of that)


This site is powered by the TWiki collaboration platform Powered by Perl