Difference: HPDPDefinitionPage (1 vs. 74)

Revision 742019-03-26 - MarkRKidger

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 95 to 95
 Ancillary Data Products are compiled in the following page

HUI requirements

Changed:
<
<		We provide here the HUI requirements needed to support the storage, queries and access to the various HPDPs and Ancillary Data Products: HPDP_ADP_HUI_Requirement_v1.3.pdf
>
>		We provide here the HUI requirements needed to support the storage, queries and access to the various HPDPs and Ancillary Data Products: HPDP_ADP_HUI_Requirement_v1.3.1.pdf
 

Highly Processed Data Products Cosmos page

The Cosmos webpages hosting the HPDPs can be found at this link (currently orphan page)

Line: 104 to 104
  A Python script has been created by Ivan in order to automatically generate the xml README file required in order to ingest HPDPs into the HSA. The script is currently at version 3, with the following instructions.
Changed:
<
<		-- DavidTeyssier - 18 September 2017
>
>		-- MarkKidger - 26 March 2018
 
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1442955081" name="HPDP_meeting_ICS_16Sep2015_MoM.pdf" path="HPDP_meeting_ICS_16Sep2015_MoM.pdf" size="72409" user="Main.DavidTeyssier" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1434526816" name="HPDP_meeting_ICS_16June2015.pdf" path="HPDP_meeting_ICS_16June2015.pdf" size="1454187" user="Main.DavidTeyssier" version="2"
Line: 113 to 113
 
META FILEATTACHMENT attachment="HPDP_meeting_ICS_12Jan2016_MoM.pdf" attr="" comment="" date="1453185828" name="HPDP_meeting_ICS_12Jan2016_MoM.pdf" path="HPDP_meeting_ICS_12Jan2016_MoM.pdf" size="102372" user="DavidTeyssier" version="5"
META FILEATTACHMENT attachment="PACSPhpdp.pdf" attr="" comment="" date="1455113973" name="PACSPhpdp.pdf" path="PACSPhpdp.pdf" size="4871999" user="LucaCalzoletti" version="5"
META FILEATTACHMENT attachment="HPDP_Release_Note_Template.docx" attr="" comment="" date="1481808598" name="HPDP_Release_Note_Template.docx" path="HPDP_Release_Note_Template.docx" size="97857" user="DavidTeyssier" version="4"
Deleted:
<
<
META FILEATTACHMENT attachment="HPDP_ADP_HUI_Requirement_v1.2.pdf" attr="" comment="" date="1460044077" name="HPDP_ADP_HUI_Requirement_v1.2.pdf" path="HPDP_ADP_HUI_Requirement_v1.2.pdf" size="232242" user="DavidTeyssier" version="1"
 
META FILEATTACHMENT attachment="HPDP_meeting_ICS_19Apr2016_Agenda_v1.0.pdf" attr="" comment="" date="1460722159" name="HPDP_meeting_ICS_19Apr2016_Agenda_v1.0.pdf" path="HPDP_meeting_ICS_19Apr2016_Agenda_v1.0.pdf" size="61323" user="DavidTeyssier" version="1"
Deleted:
<
<
META FILEATTACHMENT attachment="HPDP_ADP_HUI_Requirement_v1.3.pdf" attr="" comment="" date="1460723913" name="HPDP_ADP_HUI_Requirement_v1.3.pdf" path="HPDP_ADP_HUI_Requirement_v1.3.pdf" size="247095" user="DavidTeyssier" version="1"
 
META FILEATTACHMENT attachment="HPDP_meeting_ICS_19April2016.pdf" attr="" comment="" date="1461151563" name="HPDP_meeting_ICS_19April2016.pdf" path="HPDP_meeting_ICS_19April2016.pdf" size="2054639" user="DavidTeyssier" version="1"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_19Apr2016_MoM.pdf" attr="" comment="" date="1461185078" name="HPDP_meeting_ICS_19Apr2016_MoM.pdf" path="HPDP_meeting_ICS_19Apr2016_MoM.pdf" size="209672" user="DavidTeyssier" version="3"
META FILEATTACHMENT attachment="SPIRE_S-BKGS-HPDP_Release_Note_v1.pdf" attr="" comment="" date="1465213711" name="SPIRE_S-BKGS-HPDP_Release_Note_v1.pdf" path="SPIRE_S-BKGS-HPDP_Release_Note_v1.pdf" size="2699646" user="DavidTeyssier" version="1"
Line: 132 to 130
 
META FILEATTACHMENT attachment="make_hpdp_xml_v3.py.txt" attr="" comment="" date="1505908241" name="make_hpdp_xml_v3.py.txt" path="make_hpdp_xml_v3.py.txt" size="11445" user="DavidTeyssier" version="1"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_19Sep2017_MoM.pdf" attr="" comment="" date="1505943237" name="HPDP_meeting_ICS_19Sep2017_MoM.pdf" path="HPDP_meeting_ICS_19Sep2017_MoM.pdf" size="127962" user="DavidTeyssier" version="1"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_19Dec2017_MoM.pdf" attr="" comment="" date="1513863558" name="HPDP_meeting_ICS_19Dec2017_MoM.pdf" path="HPDP_meeting_ICS_19Dec2017_MoM.pdf" size="119103" user="DavidTeyssier" version="1"
Added:
>
>
META FILEATTACHMENT attachment="HPDP_ADP_HUI_Requirement_v1.3.1.pdf" attr="" comment="Corrected version with correct security tag" date="1553611358" name="HPDP_ADP_HUI_Requirement_v1.3.1.pdf" path="HPDP_ADP_HUI_Requirement_v1.3.1.pdf" size="479113" user="MarkRKidger" version="1"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 732017-12-21 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 32 to 32
 
Added:
>
>
 

HPDP release notes

The following document is a template for release notes to be associated with HSC-curated HPDP deliveries.

Line: 129 to 131
 
META FILEATTACHMENT attachment="README-make_xml.md" attr="" comment="" date="1505768052" name="README-make_xml.md" path="README-make_xml.md" size="4514" user="DavidTeyssier" version="1"
META FILEATTACHMENT attachment="make_hpdp_xml_v3.py.txt" attr="" comment="" date="1505908241" name="make_hpdp_xml_v3.py.txt" path="make_hpdp_xml_v3.py.txt" size="11445" user="DavidTeyssier" version="1"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_19Sep2017_MoM.pdf" attr="" comment="" date="1505943237" name="HPDP_meeting_ICS_19Sep2017_MoM.pdf" path="HPDP_meeting_ICS_19Sep2017_MoM.pdf" size="127962" user="DavidTeyssier" version="1"
Added:
>
>
META FILEATTACHMENT attachment="HPDP_meeting_ICS_19Dec2017_MoM.pdf" attr="" comment="" date="1513863558" name="HPDP_meeting_ICS_19Dec2017_MoM.pdf" path="HPDP_meeting_ICS_19Dec2017_MoM.pdf" size="119103" user="DavidTeyssier" version="1"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 722017-09-20 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 14 to 14
 
    • Minutes of Meeting
    • Viewgraphs used during the meeting
    • The main action item resulting from the meeting is that each instrument representative shall come up with a list of use cases and requirement wish list applying to the HPDP they are supervising or acting as contact point for. These requirements will be compiled in the corresponding section below. Deadline: second half of September.
Changed:
<
<
  • 16-September-2015: Meeting #2
>
>
  • 16-Sep-2015: Meeting #2
 
Changed:
<
<
  • 12-January-2016: Meeting #3
>
>
  • 12-Jan-2016: Meeting #3
 
Changed:
<
<
  • 19-April-2016: Meeting #4
>
>
  • 19-Apr-2016: Meeting #4
 
Changed:
<
<
  • 29-June-2016: Meeting $5
>
>
  • 29-Jun-2016: Meeting $5
 
Line: 30 to 30
 
Added:
>
>
 

HPDP release notes

The following document is a template for release notes to be associated with HSC-curated HPDP deliveries.

Line: 126 to 128
 
META FILEATTACHMENT attachment="make_hpdp_xml_v2.py.txt" attr="" comment="" date="1505767975" name="make_hpdp_xml_v2.py.txt" path="make_hpdp_xml_v2.py.txt" size="11213" user="DavidTeyssier" version="1"
META FILEATTACHMENT attachment="README-make_xml.md" attr="" comment="" date="1505768052" name="README-make_xml.md" path="README-make_xml.md" size="4514" user="DavidTeyssier" version="1"
META FILEATTACHMENT attachment="make_hpdp_xml_v3.py.txt" attr="" comment="" date="1505908241" name="make_hpdp_xml_v3.py.txt" path="make_hpdp_xml_v3.py.txt" size="11445" user="DavidTeyssier" version="1"
Added:
>
>
META FILEATTACHMENT attachment="HPDP_meeting_ICS_19Sep2017_MoM.pdf" attr="" comment="" date="1505943237" name="HPDP_meeting_ICS_19Sep2017_MoM.pdf" path="HPDP_meeting_ICS_19Sep2017_MoM.pdf" size="127962" user="DavidTeyssier" version="1"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 712017-09-20 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 98 to 98
 

Automatic README file generation

Changed:
<
<		A Python script has been created by Ivan in order to automatically generate the xml README file required in order to ingest HPDPs into the HSA. The script is currently at version 2, with the following instructions.
>
>		A Python script has been created by Ivan in order to automatically generate the xml README file required in order to ingest HPDPs into the HSA. The script is currently at version 3, with the following instructions.
  -- DavidTeyssier - 18 September 2017
Line: 125 to 125
 
META FILEATTACHMENT attachment="HPDP_meeting_ICS_04July2017_MoM.pdf" attr="" comment="" date="1499421081" name="HPDP_meeting_ICS_04July2017_MoM.pdf" path="HPDP_meeting_ICS_04July2017_MoM.pdf" size="117163" user="DavidTeyssier" version="4"
META FILEATTACHMENT attachment="make_hpdp_xml_v2.py.txt" attr="" comment="" date="1505767975" name="make_hpdp_xml_v2.py.txt" path="make_hpdp_xml_v2.py.txt" size="11213" user="DavidTeyssier" version="1"
META FILEATTACHMENT attachment="README-make_xml.md" attr="" comment="" date="1505768052" name="README-make_xml.md" path="README-make_xml.md" size="4514" user="DavidTeyssier" version="1"
Added:
>
>
META FILEATTACHMENT attachment="make_hpdp_xml_v3.py.txt" attr="" comment="" date="1505908241" name="make_hpdp_xml_v3.py.txt" path="make_hpdp_xml_v3.py.txt" size="11445" user="DavidTeyssier" version="1"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 702017-09-18 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 96 to 96
  The Cosmos webpages hosting the HPDPs can be found at this link (currently orphan page)
Changed:
<
<		-- DavidTeyssier - 22 July 2016
>
>

Automatic README file generation

A Python script has been created by Ivan in order to automatically generate the xml README file required in order to ingest HPDPs into the HSA. The script is currently at version 2, with the following instructions.

-- DavidTeyssier - 18 September 2017

 
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1442955081" name="HPDP_meeting_ICS_16Sep2015_MoM.pdf" path="HPDP_meeting_ICS_16Sep2015_MoM.pdf" size="72409" user="Main.DavidTeyssier" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1434526816" name="HPDP_meeting_ICS_16June2015.pdf" path="HPDP_meeting_ICS_16June2015.pdf" size="1454187" user="Main.DavidTeyssier" version="2"
Line: 119 to 123
 
META FILEATTACHMENT attachment="HPDP_meeting_ICS_04July2017_Agenda_v1.0.doc" attr="" comment="" date="1498680507" name="HPDP_meeting_ICS_04July2017_Agenda_v1.0.doc" path="HPDP_meeting_ICS_04July2017_Agenda_v1.0.doc" size="60928" user="DavidTeyssier" version="2"
META FILEATTACHMENT attachment="HPDP_Attributes.doc" attr="" comment="" date="1499161280" name="HPDP_Attributes.doc" path="HPDP_Attributes.doc" size="75264" user="DavidTeyssier" version="2"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_04July2017_MoM.pdf" attr="" comment="" date="1499421081" name="HPDP_meeting_ICS_04July2017_MoM.pdf" path="HPDP_meeting_ICS_04July2017_MoM.pdf" size="117163" user="DavidTeyssier" version="4"
Added:
>
>
META FILEATTACHMENT attachment="make_hpdp_xml_v2.py.txt" attr="" comment="" date="1505767975" name="make_hpdp_xml_v2.py.txt" path="make_hpdp_xml_v2.py.txt" size="11213" user="DavidTeyssier" version="1"
META FILEATTACHMENT attachment="README-make_xml.md" attr="" comment="" date="1505768052" name="README-make_xml.md" path="README-make_xml.md" size="4514" user="DavidTeyssier" version="1"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 692017-07-07 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 118 to 118
 
META FILEATTACHMENT attachment="HPDP_meeting_ICS_29Mar2017_MoM.pdf" attr="" comment="" date="1490959592" name="HPDP_meeting_ICS_29Mar2017_MoM.pdf" path="HPDP_meeting_ICS_29Mar2017_MoM.pdf" size="112895" user="DavidTeyssier" version="2"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_04July2017_Agenda_v1.0.doc" attr="" comment="" date="1498680507" name="HPDP_meeting_ICS_04July2017_Agenda_v1.0.doc" path="HPDP_meeting_ICS_04July2017_Agenda_v1.0.doc" size="60928" user="DavidTeyssier" version="2"
META FILEATTACHMENT attachment="HPDP_Attributes.doc" attr="" comment="" date="1499161280" name="HPDP_Attributes.doc" path="HPDP_Attributes.doc" size="75264" user="DavidTeyssier" version="2"
Changed:
<
<
META FILEATTACHMENT attachment="HPDP_meeting_ICS_04July2017_MoM.pdf" attr="" comment="" date="1499291488" name="HPDP_meeting_ICS_04July2017_MoM.pdf" path="HPDP_meeting_ICS_04July2017_MoM.pdf" size="116687" user="DavidTeyssier" version="3"
>
>
META FILEATTACHMENT attachment="HPDP_meeting_ICS_04July2017_MoM.pdf" attr="" comment="" date="1499421081" name="HPDP_meeting_ICS_04July2017_MoM.pdf" path="HPDP_meeting_ICS_04July2017_MoM.pdf" size="117163" user="DavidTeyssier" version="4"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 682017-07-05 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 118 to 118
 
META FILEATTACHMENT attachment="HPDP_meeting_ICS_29Mar2017_MoM.pdf" attr="" comment="" date="1490959592" name="HPDP_meeting_ICS_29Mar2017_MoM.pdf" path="HPDP_meeting_ICS_29Mar2017_MoM.pdf" size="112895" user="DavidTeyssier" version="2"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_04July2017_Agenda_v1.0.doc" attr="" comment="" date="1498680507" name="HPDP_meeting_ICS_04July2017_Agenda_v1.0.doc" path="HPDP_meeting_ICS_04July2017_Agenda_v1.0.doc" size="60928" user="DavidTeyssier" version="2"
META FILEATTACHMENT attachment="HPDP_Attributes.doc" attr="" comment="" date="1499161280" name="HPDP_Attributes.doc" path="HPDP_Attributes.doc" size="75264" user="DavidTeyssier" version="2"
Changed:
<
<
META FILEATTACHMENT attachment="HPDP_meeting_ICS_04July2017_MoM.pdf" attr="" comment="" date="1499203330" name="HPDP_meeting_ICS_04July2017_MoM.pdf" path="HPDP_meeting_ICS_04July2017_MoM.pdf" size="120473" user="DavidTeyssier" version="2"
>
>
META FILEATTACHMENT attachment="HPDP_meeting_ICS_04July2017_MoM.pdf" attr="" comment="" date="1499291488" name="HPDP_meeting_ICS_04July2017_MoM.pdf" path="HPDP_meeting_ICS_04July2017_MoM.pdf" size="116687" user="DavidTeyssier" version="3"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 672017-07-04 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 29 to 29
 
Changed:
<
<
>
>
 

HPDP release notes

The following document is a template for release notes to be associated with HSC-curated HPDP deliveries.

Line: 118 to 118
 
META FILEATTACHMENT attachment="HPDP_meeting_ICS_29Mar2017_MoM.pdf" attr="" comment="" date="1490959592" name="HPDP_meeting_ICS_29Mar2017_MoM.pdf" path="HPDP_meeting_ICS_29Mar2017_MoM.pdf" size="112895" user="DavidTeyssier" version="2"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_04July2017_Agenda_v1.0.doc" attr="" comment="" date="1498680507" name="HPDP_meeting_ICS_04July2017_Agenda_v1.0.doc" path="HPDP_meeting_ICS_04July2017_Agenda_v1.0.doc" size="60928" user="DavidTeyssier" version="2"
META FILEATTACHMENT attachment="HPDP_Attributes.doc" attr="" comment="" date="1499161280" name="HPDP_Attributes.doc" path="HPDP_Attributes.doc" size="75264" user="DavidTeyssier" version="2"
Added:
>
>
META FILEATTACHMENT attachment="HPDP_meeting_ICS_04July2017_MoM.pdf" attr="" comment="" date="1499203330" name="HPDP_meeting_ICS_04July2017_MoM.pdf" path="HPDP_meeting_ICS_04July2017_MoM.pdf" size="120473" user="DavidTeyssier" version="2"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 662017-07-04 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 117 to 117
 
META FILEATTACHMENT attachment="HPDP_meeting_ICS_14Dec2016_MoM.pdf" attr="" comment="" date="1482153262" name="HPDP_meeting_ICS_14Dec2016_MoM.pdf" path="HPDP_meeting_ICS_14Dec2016_MoM.pdf" size="103999" user="DavidTeyssier" version="2"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_29Mar2017_MoM.pdf" attr="" comment="" date="1490959592" name="HPDP_meeting_ICS_29Mar2017_MoM.pdf" path="HPDP_meeting_ICS_29Mar2017_MoM.pdf" size="112895" user="DavidTeyssier" version="2"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_04July2017_Agenda_v1.0.doc" attr="" comment="" date="1498680507" name="HPDP_meeting_ICS_04July2017_Agenda_v1.0.doc" path="HPDP_meeting_ICS_04July2017_Agenda_v1.0.doc" size="60928" user="DavidTeyssier" version="2"
Changed:
<
<
META FILEATTACHMENT attachment="HPDP_Attributes.doc" attr="" comment="" date="1498819848" name="HPDP_Attributes.doc" path="HPDP_Attributes.doc" size="75264" user="DavidTeyssier" version="1"
>
>
META FILEATTACHMENT attachment="HPDP_Attributes.doc" attr="" comment="" date="1499161280" name="HPDP_Attributes.doc" path="HPDP_Attributes.doc" size="75264" user="DavidTeyssier" version="2"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 652017-06-30 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 117 to 117
 
META FILEATTACHMENT attachment="HPDP_meeting_ICS_14Dec2016_MoM.pdf" attr="" comment="" date="1482153262" name="HPDP_meeting_ICS_14Dec2016_MoM.pdf" path="HPDP_meeting_ICS_14Dec2016_MoM.pdf" size="103999" user="DavidTeyssier" version="2"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_29Mar2017_MoM.pdf" attr="" comment="" date="1490959592" name="HPDP_meeting_ICS_29Mar2017_MoM.pdf" path="HPDP_meeting_ICS_29Mar2017_MoM.pdf" size="112895" user="DavidTeyssier" version="2"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_04July2017_Agenda_v1.0.doc" attr="" comment="" date="1498680507" name="HPDP_meeting_ICS_04July2017_Agenda_v1.0.doc" path="HPDP_meeting_ICS_04July2017_Agenda_v1.0.doc" size="60928" user="DavidTeyssier" version="2"
Added:
>
>
META FILEATTACHMENT attachment="HPDP_Attributes.doc" attr="" comment="" date="1498819848" name="HPDP_Attributes.doc" path="HPDP_Attributes.doc" size="75264" user="DavidTeyssier" version="1"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 642017-06-28 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 116 to 116
 
META FILEATTACHMENT attachment="HPDP_meeting_ICS_29June2016_MoM.pdf" attr="" comment="" date="1467617097" name="HPDP_meeting_ICS_29June2016_MoM.pdf" path="HPDP_meeting_ICS_29June2016_MoM.pdf" size="91609" user="DavidTeyssier" version="4"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_14Dec2016_MoM.pdf" attr="" comment="" date="1482153262" name="HPDP_meeting_ICS_14Dec2016_MoM.pdf" path="HPDP_meeting_ICS_14Dec2016_MoM.pdf" size="103999" user="DavidTeyssier" version="2"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_29Mar2017_MoM.pdf" attr="" comment="" date="1490959592" name="HPDP_meeting_ICS_29Mar2017_MoM.pdf" path="HPDP_meeting_ICS_29Mar2017_MoM.pdf" size="112895" user="DavidTeyssier" version="2"
Changed:
<
<
META FILEATTACHMENT attachment="HPDP_meeting_ICS_04July2017_Agenda_v1.0.doc" attr="" comment="" date="1498660654" name="HPDP_meeting_ICS_04July2017_Agenda_v1.0.doc" path="HPDP_meeting_ICS_04July2017_Agenda_v1.0.doc" size="60416" user="DavidTeyssier" version="1"
>
>
META FILEATTACHMENT attachment="HPDP_meeting_ICS_04July2017_Agenda_v1.0.doc" attr="" comment="" date="1498680507" name="HPDP_meeting_ICS_04July2017_Agenda_v1.0.doc" path="HPDP_meeting_ICS_04July2017_Agenda_v1.0.doc" size="60928" user="DavidTeyssier" version="2"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 632017-06-28 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 28 to 28
 
Added:
>
>
  • 04-July-2017: Meeting #8
 

HPDP release notes

The following document is a template for release notes to be associated with HSC-curated HPDP deliveries.

Line: 114 to 116
 
META FILEATTACHMENT attachment="HPDP_meeting_ICS_29June2016_MoM.pdf" attr="" comment="" date="1467617097" name="HPDP_meeting_ICS_29June2016_MoM.pdf" path="HPDP_meeting_ICS_29June2016_MoM.pdf" size="91609" user="DavidTeyssier" version="4"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_14Dec2016_MoM.pdf" attr="" comment="" date="1482153262" name="HPDP_meeting_ICS_14Dec2016_MoM.pdf" path="HPDP_meeting_ICS_14Dec2016_MoM.pdf" size="103999" user="DavidTeyssier" version="2"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_29Mar2017_MoM.pdf" attr="" comment="" date="1490959592" name="HPDP_meeting_ICS_29Mar2017_MoM.pdf" path="HPDP_meeting_ICS_29Mar2017_MoM.pdf" size="112895" user="DavidTeyssier" version="2"
Added:
>
>
META FILEATTACHMENT attachment="HPDP_meeting_ICS_04July2017_Agenda_v1.0.doc" attr="" comment="" date="1498660654" name="HPDP_meeting_ICS_04July2017_Agenda_v1.0.doc" path="HPDP_meeting_ICS_04July2017_Agenda_v1.0.doc" size="60416" user="DavidTeyssier" version="1"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 622017-03-31 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 113 to 113
 
META FILEATTACHMENT attachment="flowDiagramsHPDPs.pdf" attr="" comment="" date="1467615845" name="flowDiagramsHPDPs.pdf" path="flowDiagramsHPDPs.pdf" size="243948" user="DavidTeyssier" version="1"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_29June2016_MoM.pdf" attr="" comment="" date="1467617097" name="HPDP_meeting_ICS_29June2016_MoM.pdf" path="HPDP_meeting_ICS_29June2016_MoM.pdf" size="91609" user="DavidTeyssier" version="4"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_14Dec2016_MoM.pdf" attr="" comment="" date="1482153262" name="HPDP_meeting_ICS_14Dec2016_MoM.pdf" path="HPDP_meeting_ICS_14Dec2016_MoM.pdf" size="103999" user="DavidTeyssier" version="2"
Changed:
<
<
META FILEATTACHMENT attachment="HPDP_meeting_ICS_29Mar2017_MoM.pdf" attr="" comment="" date="1490873120" name="HPDP_meeting_ICS_29Mar2017_MoM.pdf" path="HPDP_meeting_ICS_29Mar2017_MoM.pdf" size="112725" user="DavidTeyssier" version="1"
>
>
META FILEATTACHMENT attachment="HPDP_meeting_ICS_29Mar2017_MoM.pdf" attr="" comment="" date="1490959592" name="HPDP_meeting_ICS_29Mar2017_MoM.pdf" path="HPDP_meeting_ICS_29Mar2017_MoM.pdf" size="112895" user="DavidTeyssier" version="2"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 612017-03-30 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 26 to 26
 
Added:
>
>
 

HPDP release notes

The following document is a template for release notes to be associated with HSC-curated HPDP deliveries.

Line: 111 to 113
 
META FILEATTACHMENT attachment="flowDiagramsHPDPs.pdf" attr="" comment="" date="1467615845" name="flowDiagramsHPDPs.pdf" path="flowDiagramsHPDPs.pdf" size="243948" user="DavidTeyssier" version="1"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_29June2016_MoM.pdf" attr="" comment="" date="1467617097" name="HPDP_meeting_ICS_29June2016_MoM.pdf" path="HPDP_meeting_ICS_29June2016_MoM.pdf" size="91609" user="DavidTeyssier" version="4"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_14Dec2016_MoM.pdf" attr="" comment="" date="1482153262" name="HPDP_meeting_ICS_14Dec2016_MoM.pdf" path="HPDP_meeting_ICS_14Dec2016_MoM.pdf" size="103999" user="DavidTeyssier" version="2"
Added:
>
>
META FILEATTACHMENT attachment="HPDP_meeting_ICS_29Mar2017_MoM.pdf" attr="" comment="" date="1490873120" name="HPDP_meeting_ICS_29Mar2017_MoM.pdf" path="HPDP_meeting_ICS_29Mar2017_MoM.pdf" size="112725" user="DavidTeyssier" version="1"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 602017-01-16 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 24 to 24
 
Changed:
<
<
  • 29-June-2016: Meeting #6
>
>
  • 14-Dec-2016: Meeting #6
 

HPDP release notes

Revision 592016-12-19 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 110 to 110
 
META FILEATTACHMENT attachment="HPDP_meeting_ICS_29June2016_Agenda_v1.0.pdf" attr="" comment="" date="1467063260" name="HPDP_meeting_ICS_29June2016_Agenda_v1.0.pdf" path="HPDP_meeting_ICS_29June2016_Agenda_v1.0.pdf" size="53984" user="DavidTeyssier" version="1"
META FILEATTACHMENT attachment="flowDiagramsHPDPs.pdf" attr="" comment="" date="1467615845" name="flowDiagramsHPDPs.pdf" path="flowDiagramsHPDPs.pdf" size="243948" user="DavidTeyssier" version="1"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_29June2016_MoM.pdf" attr="" comment="" date="1467617097" name="HPDP_meeting_ICS_29June2016_MoM.pdf" path="HPDP_meeting_ICS_29June2016_MoM.pdf" size="91609" user="DavidTeyssier" version="4"
Changed:
<
<
META FILEATTACHMENT attachment="HPDP_meeting_ICS_14Dec2016_MoM.pdf" attr="" comment="" date="1482149040" name="HPDP_meeting_ICS_14Dec2016_MoM.pdf" path="HPDP_meeting_ICS_14Dec2016_MoM.pdf" size="104174" user="DavidTeyssier" version="1"
>
>
META FILEATTACHMENT attachment="HPDP_meeting_ICS_14Dec2016_MoM.pdf" attr="" comment="" date="1482153262" name="HPDP_meeting_ICS_14Dec2016_MoM.pdf" path="HPDP_meeting_ICS_14Dec2016_MoM.pdf" size="103999" user="DavidTeyssier" version="2"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 582016-12-19 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 22 to 22
 
Changed:
<
<
  • 29-June-2016: Meeting #5
>
>
  • 29-June-2016: Meeting $5
 
Added:
>
>
 

HPDP release notes

The following document is a template for release notes to be associated with HSC-curated HPDP deliveries.

Line: 68 to 70
 
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
SPIRE Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note  
Added:
>
>
SPIRE HiRes Extract SPIRE-P HiRes products from SPG processing Yes No Yes Yes  
SPIRE destripped maps As names indicates - tentative HPDP Yes No Yes Yes  
 

SPIRE-S

Line: 106 to 110
 
META FILEATTACHMENT attachment="HPDP_meeting_ICS_29June2016_Agenda_v1.0.pdf" attr="" comment="" date="1467063260" name="HPDP_meeting_ICS_29June2016_Agenda_v1.0.pdf" path="HPDP_meeting_ICS_29June2016_Agenda_v1.0.pdf" size="53984" user="DavidTeyssier" version="1"
META FILEATTACHMENT attachment="flowDiagramsHPDPs.pdf" attr="" comment="" date="1467615845" name="flowDiagramsHPDPs.pdf" path="flowDiagramsHPDPs.pdf" size="243948" user="DavidTeyssier" version="1"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_29June2016_MoM.pdf" attr="" comment="" date="1467617097" name="HPDP_meeting_ICS_29June2016_MoM.pdf" path="HPDP_meeting_ICS_29June2016_MoM.pdf" size="91609" user="DavidTeyssier" version="4"
Added:
>
>
META FILEATTACHMENT attachment="HPDP_meeting_ICS_14Dec2016_MoM.pdf" attr="" comment="" date="1482149040" name="HPDP_meeting_ICS_14Dec2016_MoM.pdf" path="HPDP_meeting_ICS_14Dec2016_MoM.pdf" size="104174" user="DavidTeyssier" version="1"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 572016-12-15 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 96 to 96
 
META FILEATTACHMENT attachment="HPDP_meeting_ICS_16Sep2015.pdf" attr="" comment="" date="1442955314" name="HPDP_meeting_ICS_16Sep2015.pdf" path="HPDP_meeting_ICS_16Sep2015.pdf" size="3118644" stream="HPDP_meeting_ICS_16Sep2015.pdf" user="Main.DavidTeyssier" version="1"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_12Jan2016_MoM.pdf" attr="" comment="" date="1453185828" name="HPDP_meeting_ICS_12Jan2016_MoM.pdf" path="HPDP_meeting_ICS_12Jan2016_MoM.pdf" size="102372" user="DavidTeyssier" version="5"
META FILEATTACHMENT attachment="PACSPhpdp.pdf" attr="" comment="" date="1455113973" name="PACSPhpdp.pdf" path="PACSPhpdp.pdf" size="4871999" user="LucaCalzoletti" version="5"
Changed:
<
<
META FILEATTACHMENT attachment="HPDP_Release_Note_Template.docx" attr="" comment="" date="1460043169" name="HPDP_Release_Note_Template.docx" path="HPDP_Release_Note_Template.docx" size="97506" user="DavidTeyssier" version="3"
>
>
META FILEATTACHMENT attachment="HPDP_Release_Note_Template.docx" attr="" comment="" date="1481808598" name="HPDP_Release_Note_Template.docx" path="HPDP_Release_Note_Template.docx" size="97857" user="DavidTeyssier" version="4"
 
META FILEATTACHMENT attachment="HPDP_ADP_HUI_Requirement_v1.2.pdf" attr="" comment="" date="1460044077" name="HPDP_ADP_HUI_Requirement_v1.2.pdf" path="HPDP_ADP_HUI_Requirement_v1.2.pdf" size="232242" user="DavidTeyssier" version="1"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_19Apr2016_Agenda_v1.0.pdf" attr="" comment="" date="1460722159" name="HPDP_meeting_ICS_19Apr2016_Agenda_v1.0.pdf" path="HPDP_meeting_ICS_19Apr2016_Agenda_v1.0.pdf" size="61323" user="DavidTeyssier" version="1"
META FILEATTACHMENT attachment="HPDP_ADP_HUI_Requirement_v1.3.pdf" attr="" comment="" date="1460723913" name="HPDP_ADP_HUI_Requirement_v1.3.pdf" path="HPDP_ADP_HUI_Requirement_v1.3.pdf" size="247095" user="DavidTeyssier" version="1"

Revision 562016-12-15 - KatrinaExter

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 58 to 58
 
Red-leak spectra Produce the Level 2/2.5 products for spectra in the red leak region; Level 2/2.5 context High Currently the spectra are cut off at just above 190 microns, but we can now produce a reasonably-well line flux calibrated spectrum in this region Using a slightly modified version of the the "calblock" SPG script No extra requirements Automatic, just running an SPG script ~350  
PointingOffsetCorrection spectra for pointed observations Provide the point-source corrected spectrum for pointed observations; possibly focussing only on those with point-like sources not located in the central spaxel in them; a SpectrumTable High The results of the POC script can be better than those provided in the SPG, and this is especially so for off-centred point sources Using the "POC" interactive pipeline script This script has been validated sufficiently to be used for almost any flux level, but we would need to run sanity checks on the results to be sure this applies to the fainter sources. If we limit the provision to point-like sources, off-centred or not, then we need to identify those Most of the effort is on identifying the candidates (if not done for all pointed observations, willy-nilly). Some effort on sanity checking    
Blue spectra (<55mum) Provide the blue end of the spectral range, which is not part of SPG as it is uncalibratable. High A slightly modified version of the SPG script can be used to create these ranges, where we do not the the flux calibration Using a slightly modified ipipe/SPG script need to identify the obsids with this range not much effort, mostly on making sure that this "blue bulk" runs probably and creates correct products    
Added:
>
>
On and Off cubes provided separately The on-source and off-source Level 2 cubes provided for the unchopped line and all chop-nod observations Mid These provide the most simply way to identify off-source contamination using the SplitOnOff ipipe script for chop-nod, and using the SPG script for unchopped line but saving the products slightly earlier in the Level 2 part   little - scripts ready, just need to modify it to run on all observations and produce the desired FITS files with the desired naming convention. the postcard can be created using the same task as used currently by PACS all unchopped line and chop-nod  
 
Unchopped obsids with continuum removed Subtract the continuum from all unchopped obsids so as to provide a cleaner spectrum; Level 2/2.5 context Mid The continuum is very uncertain and it had been agreed last year at an ICC meeting that removing this was a "good thing to do” Range scans: use the RV methods to smooth and subtract the continuum; Line scans: use an optimised version of the line-finding task in the flatfielding task of the SPG. Fit/smooth and remove continuum and then produce all final results Need to test methods and thereafter processing should be automatic Producing the scripts: a few weeks (?), checking the results: continuous (but semi-automatic) all ~1200 on-sources unchopped obsid We also need to produce a report on the unchopped mode: based on DF paper but more examples and a proper measure of uncertainty based on e.g. comparing repeat observations, scatter in super-off, etc.
Ad-hoc HPDPs For obsids found to be badly handled by the SPG, an expert will make the necessary adjustments to the pipeline so it can run; Level 2/2.5 context Mid These obsids are either not provided, or the science products are not of the same standard as all other obsids These are obsids identified in an ad-hoc manner as needing help. Jira tickets, Kayako, unchopped with bad transients identified while the unchopped study is done, etc Will only be done if it is feasible in terms of time and manpower   so far <10 candidates suggested  
Line list catalogue Spectral Feature catalogue Nice to have but ICC have no interest here and consider it Low SPIRE and HIFI have one, so why not PACS? Catalogues always produce unexpected science and are a very good Legacy product Need scripts to find lines, fit lines and maybe identify lines Need to create the scripts: KE has a student at Leuven testing out fitting lines, and may have a student in 2016-17 to test out finding lines If the student projects go well, some time in 2017 the resulting scripts could be run on the PACS spectroscopy archive (or subsets of that)   Is being done by KME as a science project
Deleted:
<
<
Identifying background contamination; Removing the background contamination DE-SCOPED as an HPDP, but identifying them for QC is still on the table   Low Results in SPG will be definitely not science-ready if the off position is contaminated Need to identifty contaminated sources by comparing spectra in different off positions, spectra again a super-off, looking for line peaks in the off-source spectrum        
 
New transients script results for unchopped obsids DE-SCOPED Producing the output of the unchopped obsids reduced with the new transients script Low This script is supposed to produce better results and users may want to use it (esp. if they read the DF paper) %RED; now realised that that this is not the case       ~1200  

SPIRE-P

Revision 552016-10-10 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 76 to 76
 
Extended background correction Extended background correction for point sources Yes - level2 equivalent FITS files No Yes - postcards Yes  
Anomalous data correction Isolated cases of failed processing due to some on-board anomaly (see also DATA-75) Yes - level2 equivalent FITS files No Yes - postcards Yes  
Line list catalogue Spectral Feature catalogue (see also this page ) No Yes - one file per obsid, and one concatenated file Yes Yes - paper  
Added:
>
>
FTS high absoluteResolution cubes Remove isolated spectra with offending resolution and regenerate final cubes Yes - level2 equivalent FITS files No ? Yes  
 
Spectral scans Un-averaged level-2 spectra Yes - level2 equivalent products No YES No

Ancillary Data Products

Revision 542016-09-20 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 47 to 47
 
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
PACS Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note  
Changed:
<
<
Improved level3 maps Further combination of fields with Unimap and JScanam Yes No ?? ??  
Additional level 2.5 maps Combination of fields currently unconsidered in the list for level2.5 pipeline Yes No ??
PACSPhpdp.pdf
 
>
>
JScanam HPDP Further combination of fields with JScanam Yes No Yes Yes Available on VOSpace
Additional level 2.5 maps with Unimap Combination of fields currently unconsidered in the list for level2.5 pipeline Yes No ??
PACSPhpdp.pdf
 
 
Exquisite Processing of GOODS fields with Unimap Use Unimap to combine hundreds of obsids from four GOODS cosmological fields (custodian: B. Altieri) Yes No Yes    
Added:
>
>
Mini-Scan level 2.5 A series of observations taken as cross-scan maps, albeit with different phot. filters, resulting in the absence of level 2.5 products. The red observations are, however, available in both scanning directions hence level 2.5-type products can be generated. Yes No Yes    
 

PACS-S

HPDP name Description; product Priority Rational How Requirements Effort Number of obsids Notes
Red-leak spectra Produce the Level 2/2.5 products for spectra in the red leak region; Level 2/2.5 context High Currently the spectra are cut off at just above 190 microns, but we can now produce a reasonably-well line flux calibrated spectrum in this region Using a slightly modified version of the the "calblock" SPG script No extra requirements Automatic, just running an SPG script ~350  
PointingOffsetCorrection spectra for pointed observations Provide the point-source corrected spectrum for pointed observations; possibly focussing only on those with point-like sources not located in the central spaxel in them; a SpectrumTable High The results of the POC script can be better than those provided in the SPG, and this is especially so for off-centred point sources Using the "POC" interactive pipeline script This script has been validated sufficiently to be used for almost any flux level, but we would need to run sanity checks on the results to be sure this applies to the fainter sources. If we limit the provision to point-like sources, off-centred or not, then we need to identify those Most of the effort is on identifying the candidates (if not done for all pointed observations, willy-nilly). Some effort on sanity checking    
Changed:
<
<
Blue spectra (<55mum) Provide the blue end of the spectral range, which is not part of SPG as it is uncalibratable. High A slightly modified version of the SPG script can be used to create these ranges, where we do not the the flux calibration Using a slightly modified ipipe/SPG script need to identify the obsids with this range not much effort, mostly on making sure that this "blue bulk" runs probably and creates correct products    
>
>
Blue spectra (<55mum) Provide the blue end of the spectral range, which is not part of SPG as it is uncalibratable. High A slightly modified version of the SPG script can be used to create these ranges, where we do not the the flux calibration Using a slightly modified ipipe/SPG script need to identify the obsids with this range not much effort, mostly on making sure that this "blue bulk" runs probably and creates correct products    
 
Unchopped obsids with continuum removed Subtract the continuum from all unchopped obsids so as to provide a cleaner spectrum; Level 2/2.5 context Mid The continuum is very uncertain and it had been agreed last year at an ICC meeting that removing this was a "good thing to do” Range scans: use the RV methods to smooth and subtract the continuum; Line scans: use an optimised version of the line-finding task in the flatfielding task of the SPG. Fit/smooth and remove continuum and then produce all final results Need to test methods and thereafter processing should be automatic Producing the scripts: a few weeks (?), checking the results: continuous (but semi-automatic) all ~1200 on-sources unchopped obsid We also need to produce a report on the unchopped mode: based on DF paper but more examples and a proper measure of uncertainty based on e.g. comparing repeat observations, scatter in super-off, etc.
Ad-hoc HPDPs For obsids found to be badly handled by the SPG, an expert will make the necessary adjustments to the pipeline so it can run; Level 2/2.5 context Mid These obsids are either not provided, or the science products are not of the same standard as all other obsids These are obsids identified in an ad-hoc manner as needing help. Jira tickets, Kayako, unchopped with bad transients identified while the unchopped study is done, etc Will only be done if it is feasible in terms of time and manpower   so far <10 candidates suggested  
Line list catalogue Spectral Feature catalogue Nice to have but ICC have no interest here and consider it Low SPIRE and HIFI have one, so why not PACS? Catalogues always produce unexpected science and are a very good Legacy product Need scripts to find lines, fit lines and maybe identify lines Need to create the scripts: KE has a student at Leuven testing out fitting lines, and may have a student in 2016-17 to test out finding lines If the student projects go well, some time in 2017 the resulting scripts could be run on the PACS spectroscopy archive (or subsets of that)   Is being done by KME as a science project

Revision 532016-09-16 - KatrinaExter

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 55 to 55
 
HPDP name Description; product Priority Rational How Requirements Effort Number of obsids Notes
Red-leak spectra Produce the Level 2/2.5 products for spectra in the red leak region; Level 2/2.5 context High Currently the spectra are cut off at just above 190 microns, but we can now produce a reasonably-well line flux calibrated spectrum in this region Using a slightly modified version of the the "calblock" SPG script No extra requirements Automatic, just running an SPG script ~350  
Changed:
<
<
PointingOffsetCorrection spectra for pointed observations Provide the point-source corrected spectrum for pointed observations; possibly focussing only on those with point-like sources not located in the central spaxel in them; a SpectrumTable High The results of the POC script can be better than those provided in the SPG, and this is especially so for off-centred point sources Using the "POC" interactive pipeline script This script has been validated sufficiently to be used for almost any flux level, but we would need to run sanity checks on the results to be sure this applies to the fainter sources. If we limit the provision to point-like sources, off-centred or not, then we need to identify those Most of the effort is on identifying the candidates (if not done for all pointed observations, willy-nilly). Some effort on sanity checking Initial pool: all 3500 pointed obsids  
>
>
PointingOffsetCorrection spectra for pointed observations Provide the point-source corrected spectrum for pointed observations; possibly focussing only on those with point-like sources not located in the central spaxel in them; a SpectrumTable High The results of the POC script can be better than those provided in the SPG, and this is especially so for off-centred point sources Using the "POC" interactive pipeline script This script has been validated sufficiently to be used for almost any flux level, but we would need to run sanity checks on the results to be sure this applies to the fainter sources. If we limit the provision to point-like sources, off-centred or not, then we need to identify those Most of the effort is on identifying the candidates (if not done for all pointed observations, willy-nilly). Some effort on sanity checking    
Blue spectra (<55mum) Provide the blue end of the spectral range, which is not part of SPG as it is uncalibratable. High A slightly modified version of the SPG script can be used to create these ranges, where we do not the the flux calibration Using a slightly modified ipipe/SPG script need to identify the obsids with this range not much effort, mostly on making sure that this "blue bulk" runs probably and creates correct products    
 
Unchopped obsids with continuum removed Subtract the continuum from all unchopped obsids so as to provide a cleaner spectrum; Level 2/2.5 context Mid The continuum is very uncertain and it had been agreed last year at an ICC meeting that removing this was a "good thing to do” Range scans: use the RV methods to smooth and subtract the continuum; Line scans: use an optimised version of the line-finding task in the flatfielding task of the SPG. Fit/smooth and remove continuum and then produce all final results Need to test methods and thereafter processing should be automatic Producing the scripts: a few weeks (?), checking the results: continuous (but semi-automatic) all ~1200 on-sources unchopped obsid We also need to produce a report on the unchopped mode: based on DF paper but more examples and a proper measure of uncertainty based on e.g. comparing repeat observations, scatter in super-off, etc.
Ad-hoc HPDPs For obsids found to be badly handled by the SPG, an expert will make the necessary adjustments to the pipeline so it can run; Level 2/2.5 context Mid These obsids are either not provided, or the science products are not of the same standard as all other obsids These are obsids identified in an ad-hoc manner as needing help. Jira tickets, Kayako, unchopped with bad transients identified while the unchopped study is done, etc Will only be done if it is feasible in terms of time and manpower   so far <10 candidates suggested  
Line list catalogue Spectral Feature catalogue Nice to have but ICC have no interest here and consider it Low SPIRE and HIFI have one, so why not PACS? Catalogues always produce unexpected science and are a very good Legacy product Need scripts to find lines, fit lines and maybe identify lines Need to create the scripts: KE has a student at Leuven testing out fitting lines, and may have a student in 2016-17 to test out finding lines If the student projects go well, some time in 2017 the resulting scripts could be run on the PACS spectroscopy archive (or subsets of that)   Is being done by KME as a science project

Revision 522016-07-22 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 81 to 81
 

HUI requirements

We provide here the HUI requirements needed to support the storage, queries and access to the various HPDPs and Ancillary Data Products: HPDP_ADP_HUI_Requirement_v1.3.pdf

Added:
>
>

Highly Processed Data Products Cosmos page

 
Changed:
<
<		-- DavidTeyssier - 7 April 2016
>
>		The Cosmos webpages hosting the HPDPs can be found at this link (currently orphan page)

-- DavidTeyssier - 22 July 2016

 
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1442955081" name="HPDP_meeting_ICS_16Sep2015_MoM.pdf" path="HPDP_meeting_ICS_16Sep2015_MoM.pdf" size="72409" user="Main.DavidTeyssier" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1434526816" name="HPDP_meeting_ICS_16June2015.pdf" path="HPDP_meeting_ICS_16June2015.pdf" size="1454187" user="Main.DavidTeyssier" version="2"

Revision 512016-07-11 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 31 to 31
  HPDPs have been identified by each instrument team, although it is likely that not all of them will be possible to prepared in certain cases. We indicate hereafter the HPDPs currently contemplated by each instruments, with some common delivery categories that will be considered or not.
Added:
>
>		Some prioritisation of the associated WPs is given in this page.
 

HIFI

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery

Revision 502016-07-04 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 23 to 23
 
Changed:
<
<
>
>
 

HPDP release notes

The following document is a template for release notes to be associated with HSC-curated HPDP deliveries.

Line: 40 to 40
 
Non-averaged level2 for point mode Level2 data as produced by modified version of SPG Yes - level2.5 equivalent FITS files (multiple spectra) No Yes - postcards Yes  
Uncertainty tables Uncertainty tables as produced by the SPG No Yes - TableDataset as FITS files No Yes  
Line list catalogue Identified Spectral Line catalogue No Yes - one file per obsid, and one concatenated file Yes - static (PNGs) and interactive (html) images Yes  
Added:
>
>
Anomalous observations Fix obsids that processed wrongly with 14.1 - see HIFI-6380 Yes - level2.5 equivalent FITS files No Yes - postcards Yes  
 

PACS-P

Line: 97 to 97
 
META FILEATTACHMENT attachment="HPDP_meeting_ICS_19Apr2016_MoM.pdf" attr="" comment="" date="1461185078" name="HPDP_meeting_ICS_19Apr2016_MoM.pdf" path="HPDP_meeting_ICS_19Apr2016_MoM.pdf" size="209672" user="DavidTeyssier" version="3"
META FILEATTACHMENT attachment="SPIRE_S-BKGS-HPDP_Release_Note_v1.pdf" attr="" comment="" date="1465213711" name="SPIRE_S-BKGS-HPDP_Release_Note_v1.pdf" path="SPIRE_S-BKGS-HPDP_Release_Note_v1.pdf" size="2699646" user="DavidTeyssier" version="1"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_29June2016_Agenda_v1.0.pdf" attr="" comment="" date="1467063260" name="HPDP_meeting_ICS_29June2016_Agenda_v1.0.pdf" path="HPDP_meeting_ICS_29June2016_Agenda_v1.0.pdf" size="53984" user="DavidTeyssier" version="1"
Added:
>
>
META FILEATTACHMENT attachment="flowDiagramsHPDPs.pdf" attr="" comment="" date="1467615845" name="flowDiagramsHPDPs.pdf" path="flowDiagramsHPDPs.pdf" size="243948" user="DavidTeyssier" version="1"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_29June2016_MoM.pdf" attr="" comment="" date="1467617097" name="HPDP_meeting_ICS_29June2016_MoM.pdf" path="HPDP_meeting_ICS_29June2016_MoM.pdf" size="91609" user="DavidTeyssier" version="4"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 492016-07-02 - KatrinaExter

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 56 to 56
 
PointingOffsetCorrection spectra for pointed observations Provide the point-source corrected spectrum for pointed observations; possibly focussing only on those with point-like sources not located in the central spaxel in them; a SpectrumTable High The results of the POC script can be better than those provided in the SPG, and this is especially so for off-centred point sources Using the "POC" interactive pipeline script This script has been validated sufficiently to be used for almost any flux level, but we would need to run sanity checks on the results to be sure this applies to the fainter sources. If we limit the provision to point-like sources, off-centred or not, then we need to identify those Most of the effort is on identifying the candidates (if not done for all pointed observations, willy-nilly). Some effort on sanity checking Initial pool: all 3500 pointed obsids  
Unchopped obsids with continuum removed Subtract the continuum from all unchopped obsids so as to provide a cleaner spectrum; Level 2/2.5 context Mid The continuum is very uncertain and it had been agreed last year at an ICC meeting that removing this was a "good thing to do” Range scans: use the RV methods to smooth and subtract the continuum; Line scans: use an optimised version of the line-finding task in the flatfielding task of the SPG. Fit/smooth and remove continuum and then produce all final results Need to test methods and thereafter processing should be automatic Producing the scripts: a few weeks (?), checking the results: continuous (but semi-automatic) all ~1200 on-sources unchopped obsid We also need to produce a report on the unchopped mode: based on DF paper but more examples and a proper measure of uncertainty based on e.g. comparing repeat observations, scatter in super-off, etc.
Ad-hoc HPDPs For obsids found to be badly handled by the SPG, an expert will make the necessary adjustments to the pipeline so it can run; Level 2/2.5 context Mid These obsids are either not provided, or the science products are not of the same standard as all other obsids These are obsids identified in an ad-hoc manner as needing help. Jira tickets, Kayako, unchopped with bad transients identified while the unchopped study is done, etc Will only be done if it is feasible in terms of time and manpower   so far <10 candidates suggested  
Changed:
<
<
Line list catalogue Spectral Feature catalogue Nice to have but ICC have no interest here and consider it Low SPIRE and HIFI have one, so why not PACS? Catalogues always produce unexpected science and are a very good Legacy product Need scripts to find lines, fit lines and maybe identify lines Need to create the scripts: KE has a student at Leuven testing out fitting lines, and may have a student in 2016-17 to test out finding lines If the student projects go well, some time in 2017 the resulting scripts could be run on the PACS spectroscopy archive (or subsets of that)    
>
>
Line list catalogue Spectral Feature catalogue Nice to have but ICC have no interest here and consider it Low SPIRE and HIFI have one, so why not PACS? Catalogues always produce unexpected science and are a very good Legacy product Need scripts to find lines, fit lines and maybe identify lines Need to create the scripts: KE has a student at Leuven testing out fitting lines, and may have a student in 2016-17 to test out finding lines If the student projects go well, some time in 2017 the resulting scripts could be run on the PACS spectroscopy archive (or subsets of that)   Is being done by KME as a science project
 
Identifying background contamination; Removing the background contamination DE-SCOPED as an HPDP, but identifying them for QC is still on the table   Low Results in SPG will be definitely not science-ready if the off position is contaminated Need to identifty contaminated sources by comparing spectra in different off positions, spectra again a super-off, looking for line peaks in the off-source spectrum        
New transients script results for unchopped obsids DE-SCOPED Producing the output of the unchopped obsids reduced with the new transients script Low This script is supposed to produce better results and users may want to use it (esp. if they read the DF paper) %RED; now realised that that this is not the case       ~1200  

Revision 482016-06-29 - KatrinaExter

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 51 to 51
 

PACS-S

Changed:
<
<
HPDP nameDescription, type of productPriorityRationalHowRequirementsEffortNumber of obsidsNotes
Point-source calibrated spectra: off-centred cases Provide the point-source corrected spectrum for pointed observations with point-like sources not located in the central spaxel; SpectrumTable (type of TableDataset) High All pointed observations have a SpectrumTable with point-source calibrated spectra in it, and if the astronomer knows the source is centred and a point, then this is a science product; if the source is off-centred, however, they would have to use HIPE to produce the PS spectrum Using the "POC" interactive pipeline script This script has been validated suficiently to be used for almost any flux level, but we would need a formal validation report and JB to tell us what parameters to set in the specific tasks Once we have the validation report and parameter values from JB (few days of work), just run the script and produce the standard result. Most of the effort is on identifying the off-centred point-like sources, which will require some investigation (a few weeks?) Need to check all pointed obsids (3500 unchoped and chopnod) but it is likely only a few dozen will produce this HPDP If producing this for off-centred point sources, why not for all point sources?
Red-leak spectra Produce the Level 2/2.5 products for spectra in the red leak region; entire L2/2.5 context High (but see notes) Currently the spectra are cut off at just after 190 microns, but we can now produce a reasonably-well line flux calibrated spectrum in this region Using the "calblock" SPG script Once validation of the red leak RSRF and the flatfielding has been done, this is ready to go Automatic, just running an SPG script There are 533 SED obsids covering this region; will be some line scan and range scan also However, this will probably become part of SPG 14.2
>
>
HPDP name Description; product Priority Rational How Requirements Effort Number of obsids Notes
Red-leak spectra Produce the Level 2/2.5 products for spectra in the red leak region; Level 2/2.5 context High Currently the spectra are cut off at just above 190 microns, but we can now produce a reasonably-well line flux calibrated spectrum in this region Using a slightly modified version of the the "calblock" SPG script No extra requirements Automatic, just running an SPG script ~350  
PointingOffsetCorrection spectra for pointed observations Provide the point-source corrected spectrum for pointed observations; possibly focussing only on those with point-like sources not located in the central spaxel in them; a SpectrumTable High The results of the POC script can be better than those provided in the SPG, and this is especially so for off-centred point sources Using the "POC" interactive pipeline script This script has been validated sufficiently to be used for almost any flux level, but we would need to run sanity checks on the results to be sure this applies to the fainter sources. If we limit the provision to point-like sources, off-centred or not, then we need to identify those Most of the effort is on identifying the candidates (if not done for all pointed observations, willy-nilly). Some effort on sanity checking Initial pool: all 3500 pointed obsids  
Unchopped obsids with continuum removed Subtract the continuum from all unchopped obsids so as to provide a cleaner spectrum; Level 2/2.5 context Mid The continuum is very uncertain and it had been agreed last year at an ICC meeting that removing this was a "good thing to do” Range scans: use the RV methods to smooth and subtract the continuum; Line scans: use an optimised version of the line-finding task in the flatfielding task of the SPG. Fit/smooth and remove continuum and then produce all final results Need to test methods and thereafter processing should be automatic Producing the scripts: a few weeks (?), checking the results: continuous (but semi-automatic) all ~1200 on-sources unchopped obsid We also need to produce a report on the unchopped mode: based on DF paper but more examples and a proper measure of uncertainty based on e.g. comparing repeat observations, scatter in super-off, etc.
Ad-hoc HPDPs For obsids found to be badly handled by the SPG, an expert will make the necessary adjustments to the pipeline so it can run; Level 2/2.5 context Mid These obsids are either not provided, or the science products are not of the same standard as all other obsids These are obsids identified in an ad-hoc manner as needing help. Jira tickets, Kayako, unchopped with bad transients identified while the unchopped study is done, etc Will only be done if it is feasible in terms of time and manpower   so far <10 candidates suggested  
Line list catalogue Spectral Feature catalogue Nice to have but ICC have no interest here and consider it Low SPIRE and HIFI have one, so why not PACS? Catalogues always produce unexpected science and are a very good Legacy product Need scripts to find lines, fit lines and maybe identify lines Need to create the scripts: KE has a student at Leuven testing out fitting lines, and may have a student in 2016-17 to test out finding lines If the student projects go well, some time in 2017 the resulting scripts could be run on the PACS spectroscopy archive (or subsets of that)    
Identifying background contamination; Removing the background contamination DE-SCOPED as an HPDP, but identifying them for QC is still on the table   Low Results in SPG will be definitely not science-ready if the off position is contaminated Need to identifty contaminated sources by comparing spectra in different off positions, spectra again a super-off, looking for line peaks in the off-source spectrum        
New transients script results for unchopped obsids DE-SCOPED Producing the output of the unchopped obsids reduced with the new transients script Low This script is supposed to produce better results and users may want to use it (esp. if they read the DF paper) %RED; now realised that that this is not the case       ~1200  
 
Deleted:
<
<		Unchopped obsids with continuum removed
Subtract the continuum from all unchopped obsids so as to provide a cleaner spectrum; Level 2/2.5 context Mid (low on ICC list but high on mine) The continuum is very uncertain and it had been agreed last year at an ICC meeting that removing this was a "good thing to do". It will avoid users being confused about funny continuum levels or shapes and, especially for the targets less than some-10sJy continuum level this will be the science-ready end product Range scans: use the RV methods to smooth and subtract the continuum; Line scans: use an optimised version of the line-finding task in the flatfielding task of the SPG. Fit/smooth and remove continuum and then produce all final results Need to test methods and thereafter process should be automatic Producing the scripts: a few weeks (?), checking the results: continuous (but semi-automatic) all ~1200 on-sources unchopped obsids: We also need to produce a report on the unchopped mode: based on DF paper but more examples and a proper measure of uncertainty based on e.g. comparing repeat observations, scatter in super-off, etc.
New transients script results for unchopped obsids Producing the output of the unchopped obsids reduced with the new transients script Low (because that script has not been validated sufficiently) This script is supposed to produce better results and users may want to use it (esp. if they read the DF paper). It can only be done in HIPE. Running the ipipe script; do the parameters need to be optimised to the target? (unknown) Need to know that this script produces reliable results when run in a semi-automatic way (would be too much work to do all obsids manually) The validation is most of the work: could take several weeks. If the results of the bulk that the NHSC are running can be used, then the validation is the only work to do here Are ~1200 on-source unchopped obsids; Priority on the red camera and the range scans
Unchopped observations with possible long-term transient If observed after a bright source, an unchopped observation will have a long-term transient and hence incorrect continuum level and shape; two possible outcomes (1) just a quality comment warning of this possibilty (2) a corrected Level 2/2.5 context High as QC, Low as HPDP The continuum shape and level will definitely be more wrong for these obsids than for the average unchopped obsid; warning the users of this case is necessary so they know what science they can get, providing corrected results can only be done in HIPE Identify sources taken after a bright observation. Running the new transients script and playing with the background subtraction task To find the candidate sources: look at the calibration block for a different response, look at the V/s of the different scans to look for a transient; look at the flux levels of the obsids preceeding the unchopped one; If the new transients script works well, then can use the NHSC bulk results; if need to play with the script to get the best results, then need to learn how to do this Finding the candidates could take a few weeks; correcting the results is unknown because no-one at HSC or still in the ICC has enough experience with the new transients script Are ~1200 on-source unchopped obsids;
Identifying background contamination; Removing the background contamination

For unchopped and chop-nod. QC: comment on obsids likely to be affected by contamination in the off ; HPDP remove the continamination before doing the off-subtraction and produce a Level 2/2.5 context, or provide the on-source and off-source cubes separately so the user can assess the damage

High as a QC, Low as an HPDP Results in SPG will be definitely not science-ready if the off position is contaminated Need to identifty contaminated sources by comparing spectra in different off positions, spectra again a super-off, looking for line peaks in the off-source spectrum Much effort in finding the sources, as while we have ideas how to do it they have not been tested; removing line contamination is probably fairly straightforward but will require intervention in the tasks of the SPG script, for continuum contamination we we will probably have to just remove a standard background Potentially some months, depends on whether our current idea work out-of-the-box or not All obsids need to be searched, but hoping some dozen of sources are only affected

Hav eto consider dases where there is emision in the off that is also in the on, whether at the same or a different level, or wavelength shifted or not, and also that is not in the on. For both, the SPG results will be wrong, but with the first you may not know (with the second you will probably see absorption lines where the contamination has been subtracted)
Line list catalogue Spectral Feature catalogue Nice to have but ICC have no interest here and consider it Low SPIRE and HIFI have one, so why not PACS. Catalogues always produce unexpected science and are a very good Legacy product Need scripts to find lines, fit lines and maybe identify lines Need to create the scripts: KE has a student at Leuven testing out fitting lines, and may have a student in 2016-17 to test out finding lines If the student projects go well, some time in 2017 the resulting scripts could be run on the PACS spectroscopy archive (or subsets of that)
 

SPIRE-P

Revision 472016-06-29 - IvanV

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 72 to 72
 
Extended background correction Extended background correction for point sources Yes - level2 equivalent FITS files No Yes - postcards Yes  
Anomalous data correction Isolated cases of failed processing due to some on-board anomaly (see also DATA-75) Yes - level2 equivalent FITS files No Yes - postcards Yes  
Line list catalogue Spectral Feature catalogue (see also this page ) No Yes - one file per obsid, and one concatenated file Yes Yes - paper  
Changed:
<
<
Spectral scans Un-averaged level-2 spectra Yes - level2 equivalent products No No No
>
>
Spectral scans Un-averaged level-2 spectra Yes - level2 equivalent products No YES No
 

Ancillary Data Products

Ancillary Data Products are compiled in the following page

Revision 462016-06-27 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 22 to 22
 
Added:
>
>
  • 29-June-2016: Meeting #5
 

HPDP release notes

The following document is a template for release notes to be associated with HSC-curated HPDP deliveries.

Line: 93 to 95
 
META FILEATTACHMENT attachment="HPDP_meeting_ICS_19April2016.pdf" attr="" comment="" date="1461151563" name="HPDP_meeting_ICS_19April2016.pdf" path="HPDP_meeting_ICS_19April2016.pdf" size="2054639" user="DavidTeyssier" version="1"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_19Apr2016_MoM.pdf" attr="" comment="" date="1461185078" name="HPDP_meeting_ICS_19Apr2016_MoM.pdf" path="HPDP_meeting_ICS_19Apr2016_MoM.pdf" size="209672" user="DavidTeyssier" version="3"
META FILEATTACHMENT attachment="SPIRE_S-BKGS-HPDP_Release_Note_v1.pdf" attr="" comment="" date="1465213711" name="SPIRE_S-BKGS-HPDP_Release_Note_v1.pdf" path="SPIRE_S-BKGS-HPDP_Release_Note_v1.pdf" size="2699646" user="DavidTeyssier" version="1"
Added:
>
>
META FILEATTACHMENT attachment="HPDP_meeting_ICS_29June2016_Agenda_v1.0.pdf" attr="" comment="" date="1467063260" name="HPDP_meeting_ICS_29June2016_Agenda_v1.0.pdf" path="HPDP_meeting_ICS_29June2016_Agenda_v1.0.pdf" size="53984" user="DavidTeyssier" version="1"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 452016-06-06 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 35 to 35
 
Expert Reduced Spectral Scans Baseline-cleaned deconvolved spectral scans Yes - level2.5 equivalent FITS files No Yes - postcards Yes  
Expert Reduced Spectral Maps Baseline-cleaned regridded cubes l scans Yes - level2.5 equivalent FITS files No Yes - postcards Yes  
Isolated OFF spectra OFF spectra as produced by the SPG, potentially cleaned further Yes - level2.5 equivalent FITS files No Yes - postcards Yes  
Changed:
<
<
Non-averaged level2 for point mode Level2 data as produced by modified version of SPG Yes - level2.5 equivalent FITS files (multiple spectra) No Yes - postcards Yes  
Uncertainty tables Uncertainty tables as produced by the SPG No Yes - TableDataset as FITS files No Yes  
>
>
Non-averaged level2 for point mode Level2 data as produced by modified version of SPG Yes - level2.5 equivalent FITS files (multiple spectra) No Yes - postcards Yes  
Uncertainty tables Uncertainty tables as produced by the SPG No Yes - TableDataset as FITS files No Yes  
 
Line list catalogue Identified Spectral Line catalogue No Yes - one file per obsid, and one concatenated file Yes - static (PNGs) and interactive (html) images Yes  

PACS-P

Line: 67 to 67
 
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
Semi-Extended calibration correction SECT correction for semi-compact sources Yes - level2 equivalent FITS files No Yes - postcards Yes  
Changed:
<
<
Extended background correction Extended background correction for point sources Yes - level2 equivalent FITS files No Yes - postcards Yes  
>
>
Extended background correction Extended background correction for point sources Yes - level2 equivalent FITS files No Yes - postcards Yes  
 
Anomalous data correction Isolated cases of failed processing due to some on-board anomaly (see also DATA-75) Yes - level2 equivalent FITS files No Yes - postcards Yes  
Line list catalogue Spectral Feature catalogue (see also this page ) No Yes - one file per obsid, and one concatenated file Yes Yes - paper  
Spectral scans Un-averaged level-2 spectra Yes - level2 equivalent products No No No
Line: 92 to 92
 
META FILEATTACHMENT attachment="HPDP_ADP_HUI_Requirement_v1.3.pdf" attr="" comment="" date="1460723913" name="HPDP_ADP_HUI_Requirement_v1.3.pdf" path="HPDP_ADP_HUI_Requirement_v1.3.pdf" size="247095" user="DavidTeyssier" version="1"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_19April2016.pdf" attr="" comment="" date="1461151563" name="HPDP_meeting_ICS_19April2016.pdf" path="HPDP_meeting_ICS_19April2016.pdf" size="2054639" user="DavidTeyssier" version="1"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_19Apr2016_MoM.pdf" attr="" comment="" date="1461185078" name="HPDP_meeting_ICS_19Apr2016_MoM.pdf" path="HPDP_meeting_ICS_19Apr2016_MoM.pdf" size="209672" user="DavidTeyssier" version="3"
Added:
>
>
META FILEATTACHMENT attachment="SPIRE_S-BKGS-HPDP_Release_Note_v1.pdf" attr="" comment="" date="1465213711" name="SPIRE_S-BKGS-HPDP_Release_Note_v1.pdf" path="SPIRE_S-BKGS-HPDP_Release_Note_v1.pdf" size="2699646" user="DavidTeyssier" version="1"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 442016-05-17 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 68 to 68
 
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
Semi-Extended calibration correction SECT correction for semi-compact sources Yes - level2 equivalent FITS files No Yes - postcards Yes  
Extended background correction Extended background correction for point sources Yes - level2 equivalent FITS files No Yes - postcards Yes  
Changed:
<
<
Anomalous data correction Isolated cases of failed processing due to some on-board anomaly Yes - level2 equivalent FITS files No Yes - postcards Yes  
>
>
Anomalous data correction Isolated cases of failed processing due to some on-board anomaly (see also DATA-75) Yes - level2 equivalent FITS files No Yes - postcards Yes  
 
Line list catalogue Spectral Feature catalogue (see also this page ) No Yes - one file per obsid, and one concatenated file Yes Yes - paper  
Spectral scans Un-averaged level-2 spectra Yes - level2 equivalent products No No No

Ancillary Data Products

Revision 432016-05-04 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 62 to 62
 
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
SPIRE Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note  
Deleted:
<
<		<!--| HiRes maps | Reprocessing of SPIRE-P maps with the HiRes algorithm | Yes - level 2 or level 2.5 equivalent FITS files | No | ?? | ?? | | -->
 

SPIRE-S

Revision 422016-04-20 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 92 to 92
 
META FILEATTACHMENT attachment="HPDP_meeting_ICS_19Apr2016_Agenda_v1.0.pdf" attr="" comment="" date="1460722159" name="HPDP_meeting_ICS_19Apr2016_Agenda_v1.0.pdf" path="HPDP_meeting_ICS_19Apr2016_Agenda_v1.0.pdf" size="61323" user="DavidTeyssier" version="1"
META FILEATTACHMENT attachment="HPDP_ADP_HUI_Requirement_v1.3.pdf" attr="" comment="" date="1460723913" name="HPDP_ADP_HUI_Requirement_v1.3.pdf" path="HPDP_ADP_HUI_Requirement_v1.3.pdf" size="247095" user="DavidTeyssier" version="1"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_19April2016.pdf" attr="" comment="" date="1461151563" name="HPDP_meeting_ICS_19April2016.pdf" path="HPDP_meeting_ICS_19April2016.pdf" size="2054639" user="DavidTeyssier" version="1"
Added:
>
>
META FILEATTACHMENT attachment="HPDP_meeting_ICS_19Apr2016_MoM.pdf" attr="" comment="" date="1461185078" name="HPDP_meeting_ICS_19Apr2016_MoM.pdf" path="HPDP_meeting_ICS_19Apr2016_MoM.pdf" size="209672" user="DavidTeyssier" version="3"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 412016-04-20 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 19 to 19
 
Changed:
<
<
>
>
 

HPDP release notes

The following document is a template for release notes to be associated with HSC-curated HPDP deliveries.

Line: 91 to 91
 
META FILEATTACHMENT attachment="HPDP_ADP_HUI_Requirement_v1.2.pdf" attr="" comment="" date="1460044077" name="HPDP_ADP_HUI_Requirement_v1.2.pdf" path="HPDP_ADP_HUI_Requirement_v1.2.pdf" size="232242" user="DavidTeyssier" version="1"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_19Apr2016_Agenda_v1.0.pdf" attr="" comment="" date="1460722159" name="HPDP_meeting_ICS_19Apr2016_Agenda_v1.0.pdf" path="HPDP_meeting_ICS_19Apr2016_Agenda_v1.0.pdf" size="61323" user="DavidTeyssier" version="1"
META FILEATTACHMENT attachment="HPDP_ADP_HUI_Requirement_v1.3.pdf" attr="" comment="" date="1460723913" name="HPDP_ADP_HUI_Requirement_v1.3.pdf" path="HPDP_ADP_HUI_Requirement_v1.3.pdf" size="247095" user="DavidTeyssier" version="1"
Added:
>
>
META FILEATTACHMENT attachment="HPDP_meeting_ICS_19April2016.pdf" attr="" comment="" date="1461151563" name="HPDP_meeting_ICS_19April2016.pdf" path="HPDP_meeting_ICS_19April2016.pdf" size="2054639" user="DavidTeyssier" version="1"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 402016-04-15 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 20 to 20
 
Added:
>
>
 

HPDP release notes

Line: 76 to 77
 Ancillary Data Products are compiled in the following page

HUI requirements

Changed:
<
<		We provide here the HUI requirements needed to support the storage, queries and access to the various HPDPs and Ancillary Data Products: HPDP_ADP_HUI_Requirement_v1.2.pdf
>
>		We provide here the HUI requirements needed to support the storage, queries and access to the various HPDPs and Ancillary Data Products: HPDP_ADP_HUI_Requirement_v1.3.pdf
  -- DavidTeyssier - 7 April 2016
Line: 88 to 89
 
META FILEATTACHMENT attachment="PACSPhpdp.pdf" attr="" comment="" date="1455113973" name="PACSPhpdp.pdf" path="PACSPhpdp.pdf" size="4871999" user="LucaCalzoletti" version="5"
META FILEATTACHMENT attachment="HPDP_Release_Note_Template.docx" attr="" comment="" date="1460043169" name="HPDP_Release_Note_Template.docx" path="HPDP_Release_Note_Template.docx" size="97506" user="DavidTeyssier" version="3"
META FILEATTACHMENT attachment="HPDP_ADP_HUI_Requirement_v1.2.pdf" attr="" comment="" date="1460044077" name="HPDP_ADP_HUI_Requirement_v1.2.pdf" path="HPDP_ADP_HUI_Requirement_v1.2.pdf" size="232242" user="DavidTeyssier" version="1"
Added:
>
>
META FILEATTACHMENT attachment="HPDP_meeting_ICS_19Apr2016_Agenda_v1.0.pdf" attr="" comment="" date="1460722159" name="HPDP_meeting_ICS_19Apr2016_Agenda_v1.0.pdf" path="HPDP_meeting_ICS_19Apr2016_Agenda_v1.0.pdf" size="61323" user="DavidTeyssier" version="1"
META FILEATTACHMENT attachment="HPDP_ADP_HUI_Requirement_v1.3.pdf" attr="" comment="" date="1460723913" name="HPDP_ADP_HUI_Requirement_v1.3.pdf" path="HPDP_ADP_HUI_Requirement_v1.3.pdf" size="247095" user="DavidTeyssier" version="1"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 392016-04-12 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 86 to 86
 
META FILEATTACHMENT attachment="HPDP_meeting_ICS_16Sep2015.pdf" attr="" comment="" date="1442955314" name="HPDP_meeting_ICS_16Sep2015.pdf" path="HPDP_meeting_ICS_16Sep2015.pdf" size="3118644" stream="HPDP_meeting_ICS_16Sep2015.pdf" user="Main.DavidTeyssier" version="1"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_12Jan2016_MoM.pdf" attr="" comment="" date="1453185828" name="HPDP_meeting_ICS_12Jan2016_MoM.pdf" path="HPDP_meeting_ICS_12Jan2016_MoM.pdf" size="102372" user="DavidTeyssier" version="5"
META FILEATTACHMENT attachment="PACSPhpdp.pdf" attr="" comment="" date="1455113973" name="PACSPhpdp.pdf" path="PACSPhpdp.pdf" size="4871999" user="LucaCalzoletti" version="5"
Deleted:
<
<
META FILEATTACHMENT attachment="HPDP_ALP_HUI_Requirement_v0.1.pdf" attr="" comment="" date="1455827776" name="HPDP_ALP_HUI_Requirement_v0.1.pdf" path="HPDP_ALP_HUI_Requirement_v0.1.pdf" size="231798" user="DavidTeyssier" version="1"
 
META FILEATTACHMENT attachment="HPDP_Release_Note_Template.docx" attr="" comment="" date="1460043169" name="HPDP_Release_Note_Template.docx" path="HPDP_Release_Note_Template.docx" size="97506" user="DavidTeyssier" version="3"
META FILEATTACHMENT attachment="HPDP_ADP_HUI_Requirement_v1.2.pdf" attr="" comment="" date="1460044077" name="HPDP_ADP_HUI_Requirement_v1.2.pdf" path="HPDP_ADP_HUI_Requirement_v1.2.pdf" size="232242" user="DavidTeyssier" version="1"
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 382016-04-07 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 71 to 71
 
Anomalous data correction Isolated cases of failed processing due to some on-board anomaly Yes - level2 equivalent FITS files No Yes - postcards Yes  
Line list catalogue Spectral Feature catalogue (see also this page ) No Yes - one file per obsid, and one concatenated file Yes Yes - paper  
Spectral scans Un-averaged level-2 spectra Yes - level2 equivalent products No No No
Changed:
<
<

Ancillary Legacy Products

>
>

Ancillary Data Products

 
Changed:
<
<		Ancillary Legacy Products are compiled in the following page
>
>		Ancillary Data Products are compiled in the following page
 

HUI requirements

Changed:
<
<		We provide here the HUI requirements needed to support the storage, queries and access to the various HPDPs and Ancillary Legacy Products: HPDP_ALP_HUI_Requirement_v0.1.pdf
>
>		We provide here the HUI requirements needed to support the storage, queries and access to the various HPDPs and Ancillary Data Products: HPDP_ADP_HUI_Requirement_v1.2.pdf
 
Changed:
<
<		-- DavidTeyssier - 18 February 2016
>
>		-- DavidTeyssier - 7 April 2016
 
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1442955081" name="HPDP_meeting_ICS_16Sep2015_MoM.pdf" path="HPDP_meeting_ICS_16Sep2015_MoM.pdf" size="72409" user="Main.DavidTeyssier" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1434526816" name="HPDP_meeting_ICS_16June2015.pdf" path="HPDP_meeting_ICS_16June2015.pdf" size="1454187" user="Main.DavidTeyssier" version="2"
Line: 87 to 87
 
META FILEATTACHMENT attachment="HPDP_meeting_ICS_12Jan2016_MoM.pdf" attr="" comment="" date="1453185828" name="HPDP_meeting_ICS_12Jan2016_MoM.pdf" path="HPDP_meeting_ICS_12Jan2016_MoM.pdf" size="102372" user="DavidTeyssier" version="5"
META FILEATTACHMENT attachment="PACSPhpdp.pdf" attr="" comment="" date="1455113973" name="PACSPhpdp.pdf" path="PACSPhpdp.pdf" size="4871999" user="LucaCalzoletti" version="5"
META FILEATTACHMENT attachment="HPDP_ALP_HUI_Requirement_v0.1.pdf" attr="" comment="" date="1455827776" name="HPDP_ALP_HUI_Requirement_v0.1.pdf" path="HPDP_ALP_HUI_Requirement_v0.1.pdf" size="231798" user="DavidTeyssier" version="1"
Changed:
<
<
META FILEATTACHMENT attachment="HPDP_Release_Note_Template.docx" attr="" comment="" date="1456938180" name="HPDP_Release_Note_Template.docx" path="HPDP_Release_Note_Template.docx" size="98093" user="DavidTeyssier" version="1"
>
>
META FILEATTACHMENT attachment="HPDP_Release_Note_Template.docx" attr="" comment="" date="1460043169" name="HPDP_Release_Note_Template.docx" path="HPDP_Release_Note_Template.docx" size="97506" user="DavidTeyssier" version="3"
META FILEATTACHMENT attachment="HPDP_ADP_HUI_Requirement_v1.2.pdf" attr="" comment="" date="1460044077" name="HPDP_ADP_HUI_Requirement_v1.2.pdf" path="HPDP_ADP_HUI_Requirement_v1.2.pdf" size="232242" user="DavidTeyssier" version="1"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 372016-04-07 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 19 to 19
 
Changed:
<
<
  • Next meeting planned Mid-April 2016
>
>
  • 19-April-2016: Meeting #4
 

HPDP release notes

Line: 44 to 44
 
PACS Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note  
Improved level3 maps Further combination of fields with Unimap and JScanam Yes No ?? ??  
Additional level 2.5 maps Combination of fields currently unconsidered in the list for level2.5 pipeline Yes No ??
PACSPhpdp.pdf
 
Added:
>
>
Exquisite Processing of GOODS fields with Unimap Use Unimap to combine hundreds of obsids from four GOODS cosmological fields (custodian: B. Altieri) Yes No Yes    
 

PACS-S

Revision 362016-03-02 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 21 to 21
 
Added:
>
>

HPDP release notes

The following document is a template for release notes to be associated with HSC-curated HPDP deliveries.

 

HPDP plans per instruments

HPDPs have been identified by each instrument team, although it is likely that not all of them will be possible to prepared in certain cases. We indicate hereafter the HPDPs currently contemplated by each instruments, with some common delivery categories that will be considered or not.

Line: 51 to 54
 For unchopped and chop-nod. QC: comment on obsids likely to be affected by contamination in the off ; HPDP remove the continamination before doing the off-subtraction and produce a Level 2/2.5 context, or provide the on-source and off-source cubes separately so the user can assess the damage
High as a QC, Low as an HPDP Results in SPG will be definitely not science-ready if the off position is contaminated Need to identifty contaminated sources by comparing spectra in different off positions, spectra again a super-off, looking for line peaks in the off-source spectrum Much effort in finding the sources, as while we have ideas how to do it they have not been tested; removing line contamination is probably fairly straightforward but will require intervention in the tasks of the SPG script, for continuum contamination we we will probably have to just remove a standard background Potentially some months, depends on whether our current idea work out-of-the-box or not All obsids need to be searched, but hoping some dozen of sources are only affected
Changed:
<
<		Hav eto consider dases where there is emision in the off that is also in the on, whether at the same or a different level, or wavelength shifted or not, and also that is not in the on. For both, the SPG results will be wrong, but with the first you may not know (with the second you will probably see absorption lines where the contamination has been subtracted)
Line list catalogue Spectral Feature catalogue Nice to have but ICC have no interest here and consider it Low SPIRE and HIFI have one, so why not PACS. Catalogues always produce unexpected science and are a very good Legacy product Need scripts to find lines, fit lines and maybe identify lines Need to create the scripts: KE has a student at Leuven testing out fitting lines, and may have a student in 2016-17 to test out finding lines If the student projects go well, some time in 2017 the resulting scripts could be run on the PACS spectroscopy archive (or subsets of that)
>
>		Hav eto consider dases where there is emision in the off that is also in the on, whether at the same or a different level, or wavelength shifted or not, and also that is not in the on. For both, the SPG results will be wrong, but with the first you may not know (with the second you will probably see absorption lines where the contamination has been subtracted)
Line list catalogue Spectral Feature catalogue Nice to have but ICC have no interest here and consider it Low SPIRE and HIFI have one, so why not PACS. Catalogues always produce unexpected science and are a very good Legacy product Need scripts to find lines, fit lines and maybe identify lines Need to create the scripts: KE has a student at Leuven testing out fitting lines, and may have a student in 2016-17 to test out finding lines If the student projects go well, some time in 2017 the resulting scripts could be run on the PACS spectroscopy archive (or subsets of that)
 

SPIRE-P

Line: 84 to 86
 
META FILEATTACHMENT attachment="HPDP_meeting_ICS_12Jan2016_MoM.pdf" attr="" comment="" date="1453185828" name="HPDP_meeting_ICS_12Jan2016_MoM.pdf" path="HPDP_meeting_ICS_12Jan2016_MoM.pdf" size="102372" user="DavidTeyssier" version="5"
META FILEATTACHMENT attachment="PACSPhpdp.pdf" attr="" comment="" date="1455113973" name="PACSPhpdp.pdf" path="PACSPhpdp.pdf" size="4871999" user="LucaCalzoletti" version="5"
META FILEATTACHMENT attachment="HPDP_ALP_HUI_Requirement_v0.1.pdf" attr="" comment="" date="1455827776" name="HPDP_ALP_HUI_Requirement_v0.1.pdf" path="HPDP_ALP_HUI_Requirement_v0.1.pdf" size="231798" user="DavidTeyssier" version="1"
Added:
>
>
META FILEATTACHMENT attachment="HPDP_Release_Note_Template.docx" attr="" comment="" date="1456938180" name="HPDP_Release_Note_Template.docx" path="HPDP_Release_Note_Template.docx" size="98093" user="DavidTeyssier" version="1"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 352016-03-02 - KatrinaExter

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 46 to 46
 
HPDP nameDescription, type of productPriorityRationalHowRequirementsEffortNumber of obsidsNotes
Point-source calibrated spectra: off-centred cases Provide the point-source corrected spectrum for pointed observations with point-like sources not located in the central spaxel; SpectrumTable (type of TableDataset) High All pointed observations have a SpectrumTable with point-source calibrated spectra in it, and if the astronomer knows the source is centred and a point, then this is a science product; if the source is off-centred, however, they would have to use HIPE to produce the PS spectrum Using the "POC" interactive pipeline script This script has been validated suficiently to be used for almost any flux level, but we would need a formal validation report and JB to tell us what parameters to set in the specific tasks Once we have the validation report and parameter values from JB (few days of work), just run the script and produce the standard result. Most of the effort is on identifying the off-centred point-like sources, which will require some investigation (a few weeks?) Need to check all pointed obsids (3500 unchoped and chopnod) but it is likely only a few dozen will produce this HPDP If producing this for off-centred point sources, why not for all point sources?
Red-leak spectra Produce the Level 2/2.5 products for spectra in the red leak region; entire L2/2.5 context High (but see notes) Currently the spectra are cut off at just after 190 microns, but we can now produce a reasonably-well line flux calibrated spectrum in this region Using the "calblock" SPG script Once validation of the red leak RSRF and the flatfielding has been done, this is ready to go Automatic, just running an SPG script There are 533 SED obsids covering this region; will be some line scan and range scan also However, this will probably become part of SPG 14.2
Changed:
<
<		 Unchopped obsids with continuum removed
Subtract the continuum from all unchopped obsids so as to provide a cleaner spectrum; Level 2/2.5 context Mid (low on ICC list but high on mine) The continuum is very uncertain and it had been agreed last year at an ICC meeting that removing this was a "good thing to do". It will avoid users being confused about funny continuum levels or shapes and, especially for the targets less than some-10sJy continuum level this will be the science-ready end product Range scans: use the RV methods to smooth and subtract the continuum; Line scans: use an optimised version of the line-finding task in the flatfielding task of the SPG. Fit/smooth and remove continuum and then produce all final results Need to test methods and thereafter process should be automatic Producing the scripts: a few weeks (?), checking the results: continuous (but semi-automatic) all ~1200 on-sources unchopped obsids: We also need to produce a report on the unchopped mode: based on DF paper but more examples and a proper measure of uncertainty based on e.g. comparing repeat observations, scatter in super-off, etc.
New transients script results for unchopped obsids Producing the output of the unchopped obsids reduced with the new transients script Low (because that script has not been validated sufficiently) This script is supposed to produce better results and users may want to use it (esp. if they read the DF paper). It can only be done in HIPE. Running the ipipe script; do the parameters need to be optimised to the target? (unknown) Need to know that this script produces reliable results when run in a semi-automatic way (would be too much work to do all obsids manually) The validation is most of the work: could take several weeks. If the results of the bulk that the NHSC are running can be used, then the validation is the only work to do here Are ~1200 on-source unchopped obsids; Priority on the red camera and the range scans
Unchopped observations with possible long-term transient If observed after a bright source, an unchopped observation will have a long-term transient and hence incorrect continuum level and shape; two possible outcomes (1) just a quality comment warning of this possibilty (2) a corrected Level 2/2.5 context High as QC, Low as HPDP The continuum shape and level will definitely be more wrong for these obsids than for the average unchopped obsid; warning the users of this case is necessary so they know what science they can get, providing corrected results can only be done in HIPE Identify sources taken after a bright observation. Running the new transients script and playing with the background subtraction task To find the candidate sources: look at the calibration block for a different response, look at the V/s of the different scans to look for a transient; look at the flux levels of the obsids preceeding the unchopped one; If the new transients script works well, then can use the NHSC bulk results; if need to play with the script to get the best results, then need to learn how to do this Finding the candidates could take a few weeks; correcting the results is unknown because no-one at HSC or still in the ICC has enough experience with the new transients script Are ~1200 on-source unchopped obsids;
Identifying background contamination; Removing the background contamination For unchopped and chop-nod. QC: comment on obsids likely to be affected by contamination in the off; HPDP remove the continamination before doing the off-subtraction and produce a Level 2/2.5 context High as a QC, Low as an HPDP Results in SPG will be definitely not science-ready if the off position is contaminated Need to identifty contaminated sources by comparing spectra in different off positions, spectra again a super-off, looking for line peaks in the off-source spectrum Much effort in finding the sources, as while we have ideas how to do it they have not been tested; removing line contamination is probably fairly straightforward but will require intervention in the tasks of the SPG script, for continuum contamination we we will probably have to just remove a standard background Potentially some months, depends on whether our current idea work out-of-the-box or not All obsids need to be searched, but hoping some dozen of sources are only affected
Line list catalogue Spectral Feature catalogue Nice to have but ICC have no interest here and consider it Low SPIRE and HIFI have one, so why not PACS. Catalogues always produce unexpected science and are a very good Legacy product Need scripts to find lines, fit lines and maybe identify lines Need to create the scripts: KE has a student at Leuven testing out fitting lines, and may have a student in 2016-17 to test out finding lines If the student projects go well, some time in 2017 the resulting scripts could be run on the PACS spectroscopy archive (or subsets of that)
>
>		Unchopped obsids with continuum removed
Subtract the continuum from all unchopped obsids so as to provide a cleaner spectrum; Level 2/2.5 context Mid (low on ICC list but high on mine) The continuum is very uncertain and it had been agreed last year at an ICC meeting that removing this was a "good thing to do". It will avoid users being confused about funny continuum levels or shapes and, especially for the targets less than some-10sJy continuum level this will be the science-ready end product Range scans: use the RV methods to smooth and subtract the continuum; Line scans: use an optimised version of the line-finding task in the flatfielding task of the SPG. Fit/smooth and remove continuum and then produce all final results Need to test methods and thereafter process should be automatic Producing the scripts: a few weeks (?), checking the results: continuous (but semi-automatic) all ~1200 on-sources unchopped obsids: We also need to produce a report on the unchopped mode: based on DF paper but more examples and a proper measure of uncertainty based on e.g. comparing repeat observations, scatter in super-off, etc.
New transients script results for unchopped obsids Producing the output of the unchopped obsids reduced with the new transients script Low (because that script has not been validated sufficiently) This script is supposed to produce better results and users may want to use it (esp. if they read the DF paper). It can only be done in HIPE. Running the ipipe script; do the parameters need to be optimised to the target? (unknown) Need to know that this script produces reliable results when run in a semi-automatic way (would be too much work to do all obsids manually) The validation is most of the work: could take several weeks. If the results of the bulk that the NHSC are running can be used, then the validation is the only work to do here Are ~1200 on-source unchopped obsids; Priority on the red camera and the range scans
Unchopped observations with possible long-term transient If observed after a bright source, an unchopped observation will have a long-term transient and hence incorrect continuum level and shape; two possible outcomes (1) just a quality comment warning of this possibilty (2) a corrected Level 2/2.5 context High as QC, Low as HPDP The continuum shape and level will definitely be more wrong for these obsids than for the average unchopped obsid; warning the users of this case is necessary so they know what science they can get, providing corrected results can only be done in HIPE Identify sources taken after a bright observation. Running the new transients script and playing with the background subtraction task To find the candidate sources: look at the calibration block for a different response, look at the V/s of the different scans to look for a transient; look at the flux levels of the obsids preceeding the unchopped one; If the new transients script works well, then can use the NHSC bulk results; if need to play with the script to get the best results, then need to learn how to do this Finding the candidates could take a few weeks; correcting the results is unknown because no-one at HSC or still in the ICC has enough experience with the new transients script Are ~1200 on-source unchopped obsids;
Identifying background contamination; Removing the background contamination

For unchopped and chop-nod. QC: comment on obsids likely to be affected by contamination in the off ; HPDP remove the continamination before doing the off-subtraction and produce a Level 2/2.5 context, or provide the on-source and off-source cubes separately so the user can assess the damage

High as a QC, Low as an HPDP Results in SPG will be definitely not science-ready if the off position is contaminated Need to identifty contaminated sources by comparing spectra in different off positions, spectra again a super-off, looking for line peaks in the off-source spectrum Much effort in finding the sources, as while we have ideas how to do it they have not been tested; removing line contamination is probably fairly straightforward but will require intervention in the tasks of the SPG script, for continuum contamination we we will probably have to just remove a standard background Potentially some months, depends on whether our current idea work out-of-the-box or not All obsids need to be searched, but hoping some dozen of sources are only affected

Hav eto consider dases where there is emision in the off that is also in the on, whether at the same or a different level, or wavelength shifted or not, and also that is not in the on. For both, the SPG results will be wrong, but with the first you may not know (with the second you will probably see absorption lines where the contamination has been subtracted)

Line list catalogue Spectral Feature catalogue Nice to have but ICC have no interest here and consider it Low SPIRE and HIFI have one, so why not PACS. Catalogues always produce unexpected science and are a very good Legacy product Need scripts to find lines, fit lines and maybe identify lines Need to create the scripts: KE has a student at Leuven testing out fitting lines, and may have a student in 2016-17 to test out finding lines If the student projects go well, some time in 2017 the resulting scripts could be run on the PACS spectroscopy archive (or subsets of that)
 

SPIRE-P

Revision 342016-03-01 - KatrinaExter

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 44 to 44
 

PACS-S

Changed:
<
<		In order of priority+feasibility:
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Notes Prototype delivery
Pipeline processing bright sources Very bright sources ought to be reduced with a different pipeline script (to account for non-linearity/self-curing) Yes - level2/2.5 equivalent FITS files at a minimum, or entire obsContext if done via SPG No Yes   Some level of manual reduction may be necessary, but are probably not many such observations  
Off-centred point-source calibrated data For off-centred point-like sources: SPG products processed through the standard pipeline script with non-standard extraction of final PS spectrum Yes - level2/2.5 equivalent FITS files No Yes   A useful possibility, feasibility will be more clear once QC work (http://herschel.esac.esa.int/twiki/bin/view/Pacs/QualityCommentsSpec) on off-centred observations and identification of likely centred point sources has been done  
Long range spectrometer flat-fielding - potential Correcting the few cases of bad flatfielding Yes - level2/2.5 equivalent FITS files No Yes   May be unnecessary if the new range scan flatfielding works well. Pilar will check a subset of the 14.0 bulk results. Sample relevance analysis.  
Unchopped observations: pipeline processing with new response drifts correction Improved transient correction and background subtraction as provided by the new NHSC pipeline scripts Yes - level2/2.5 equivalent FITS files, or entire obsContext if done via SPG No Yes Could be proposed for 14.1 SPG rather than HPDP Highly desirable, but requires a fuller validation of this script first (if the script does not provide improved results, neither SPG nor HPDP can be created here). Conditional to NHSC bulk to be run.  
Unchopped observations: subtracting the baseline If the continuum in these AOTs cannot be improved (wrt absolute level and shape, less wrt SNR) then instead subtract the baseline Yes - level2/2.5 equivalent FITS files: baseline subtracted cubes/spectra and cubes of the baseline that was subtracted No Yes   Desirable, necessary if the new transients correction pipeline does not becomes SPG in 14.1. Sample relevance analysis.  
Unchopped observations: manual processing to correct very bad transients Improved transient correction and background subtraction provided by processing the data manually Yes - level2/2.5 equivalent FITS files No Yes   Highly desirable, but depends on how many such observations there are and on progress on PACS-6264. Sample relevance analysis.  
Point-source calibrated data using the "POC" script products processed through the dedicated pointing offset correction script for bright point sources Yes - level2/2.5 equivalent FITS files No Yes   Straightforward to do once the pipeline script has been fully validated and sources identified; validation is still on-going (postponed by the 14.0 AAT)  
Line list catalogue Spectral Feature catalogue No Yes - one file per obsid, and one concatenated file No   HPDP: dependent on available human resources, would be completely HSC to do, exploratory studies are being done with help of MSc students at ESAC and Leuven and we should know better early in 2016  
Identifying background contamination; removing it? For ChopNod and Unchopped AOTs, looking at line (and continuum?) contamination in the off positions: primarily to make a quality comment, but if the number is manageable then also to correct for the contamination Yes - level2/2.5 equivalent FITS files No Yes   HPDP, for the quality comment part (just identifying that there is contamination) the ICC had agreed to do this (although work is not yet on-going), but for the subtraction part it would have to be done at HSC, if deemed necessary  

These have been done for 14.0 or will be SPG 14.1:

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Notes Prototype delivery
Point-source calibrated data (pointed AOTs only) SPG products processed through the standard pipeline script and then creating point source spectra from them Yes, part of SPG obsContext No Yes   Note: is done on all pointed AOTs, regardless of whether the source is know to be a point or not. Done in 14.0  
Long range spectrometer flat-fielding - main Including flatfielding for range scans in the SPG Yes, entire obsContext No Yes   depends on validation in AAT 14. Done in 14.  
Red leak spectral range Run the specific pipeline that deals with red leak region (CalBlock+RSRF) and provide data stand-alone Yes, part of entire obsContext No Yes   For 14.1. Requires delivery of red de-leaked rsrf v5 and extra work on range flatfielding.  
Repair of failed observations Re-run pipeline on observations set to failed Yes, entire obsContext No Yes   Observations that are classed as failed or delivered for reprocessing were processed in 14.0; of those that fail, are FAILED for good. Work is done by
the bulk. 		 
>
>
HPDP nameDescription, type of productPriorityRationalHowRequirementsEffortNumber of obsidsNotes
Point-source calibrated spectra: off-centred cases Provide the point-source corrected spectrum for pointed observations with point-like sources not located in the central spaxel; SpectrumTable (type of TableDataset) High All pointed observations have a SpectrumTable with point-source calibrated spectra in it, and if the astronomer knows the source is centred and a point, then this is a science product; if the source is off-centred, however, they would have to use HIPE to produce the PS spectrum Using the "POC" interactive pipeline script This script has been validated suficiently to be used for almost any flux level, but we would need a formal validation report and JB to tell us what parameters to set in the specific tasks Once we have the validation report and parameter values from JB (few days of work), just run the script and produce the standard result. Most of the effort is on identifying the off-centred point-like sources, which will require some investigation (a few weeks?) Need to check all pointed obsids (3500 unchoped and chopnod) but it is likely only a few dozen will produce this HPDP If producing this for off-centred point sources, why not for all point sources?
Red-leak spectra Produce the Level 2/2.5 products for spectra in the red leak region; entire L2/2.5 context High (but see notes) Currently the spectra are cut off at just after 190 microns, but we can now produce a reasonably-well line flux calibrated spectrum in this region Using the "calblock" SPG script Once validation of the red leak RSRF and the flatfielding has been done, this is ready to go Automatic, just running an SPG script There are 533 SED obsids covering this region; will be some line scan and range scan also However, this will probably become part of SPG 14.2

Unchopped obsids with continuum removed

 Subtract the continuum from all unchopped obsids so as to provide a cleaner spectrum; Level 2/2.5 context Mid (low on ICC list but high on mine) The continuum is very uncertain and it had been agreed last year at an ICC meeting that removing this was a "good thing to do". It will avoid users being confused about funny continuum levels or shapes and, especially for the targets less than some-10sJy continuum level this will be the science-ready end product Range scans: use the RV methods to smooth and subtract the continuum; Line scans: use an optimised version of the line-finding task in the flatfielding task of the SPG. Fit/smooth and remove continuum and then produce all final results Need to test methods and thereafter process should be automatic Producing the scripts: a few weeks (?), checking the results: continuous (but semi-automatic) all ~1200 on-sources unchopped obsids: We also need to produce a report on the unchopped mode: based on DF paper but more examples and a proper measure of uncertainty based on e.g. comparing repeat observations, scatter in super-off, etc.
New transients script results for unchopped obsids Producing the output of the unchopped obsids reduced with the new transients script Low (because that script has not been validated sufficiently) This script is supposed to produce better results and users may want to use it (esp. if they read the DF paper). It can only be done in HIPE. Running the ipipe script; do the parameters need to be optimised to the target? (unknown) Need to know that this script produces reliable results when run in a semi-automatic way (would be too much work to do all obsids manually) The validation is most of the work: could take several weeks. If the results of the bulk that the NHSC are running can be used, then the validation is the only work to do here Are ~1200 on-source unchopped obsids; Priority on the red camera and the range scans
Unchopped observations with possible long-term transient If observed after a bright source, an unchopped observation will have a long-term transient and hence incorrect continuum level and shape; two possible outcomes (1) just a quality comment warning of this possibilty (2) a corrected Level 2/2.5 context High as QC, Low as HPDP The continuum shape and level will definitely be more wrong for these obsids than for the average unchopped obsid; warning the users of this case is necessary so they know what science they can get, providing corrected results can only be done in HIPE Identify sources taken after a bright observation. Running the new transients script and playing with the background subtraction task To find the candidate sources: look at the calibration block for a different response, look at the V/s of the different scans to look for a transient; look at the flux levels of the obsids preceeding the unchopped one; If the new transients script works well, then can use the NHSC bulk results; if need to play with the script to get the best results, then need to learn how to do this Finding the candidates could take a few weeks; correcting the results is unknown because no-one at HSC or still in the ICC has enough experience with the new transients script Are ~1200 on-source unchopped obsids;
Identifying background contamination; Removing the background contamination For unchopped and chop-nod. QC: comment on obsids likely to be affected by contamination in the off; HPDP remove the continamination before doing the off-subtraction and produce a Level 2/2.5 context High as a QC, Low as an HPDP Results in SPG will be definitely not science-ready if the off position is contaminated Need to identifty contaminated sources by comparing spectra in different off positions, spectra again a super-off, looking for line peaks in the off-source spectrum Much effort in finding the sources, as while we have ideas how to do it they have not been tested; removing line contamination is probably fairly straightforward but will require intervention in the tasks of the SPG script, for continuum contamination we we will probably have to just remove a standard background Potentially some months, depends on whether our current idea work out-of-the-box or not All obsids need to be searched, but hoping some dozen of sources are only affected
Line list catalogue Spectral Feature catalogue Nice to have but ICC have no interest here and consider it Low SPIRE and HIFI have one, so why not PACS. Catalogues always produce unexpected science and are a very good Legacy product Need scripts to find lines, fit lines and maybe identify lines Need to create the scripts: KE has a student at Leuven testing out fitting lines, and may have a student in 2016-17 to test out finding lines If the student projects go well, some time in 2017 the resulting scripts could be run on the PACS spectroscopy archive (or subsets of that)
 

SPIRE-P

Revision 332016-02-18 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 77 to 77
 
Anomalous data correction Isolated cases of failed processing due to some on-board anomaly Yes - level2 equivalent FITS files No Yes - postcards Yes  
Line list catalogue Spectral Feature catalogue (see also this page ) No Yes - one file per obsid, and one concatenated file Yes Yes - paper  
Spectral scans Un-averaged level-2 spectra Yes - level2 equivalent products No No No
Deleted:
<
<

HUI requirements

We provide here the HUI requirements needed to support the storage, queries and access to the various HPDPs.

TBD

 

Ancillary Legacy Products

Ancillary Legacy Products are compiled in the following page

Added:
>
>

HUI requirements

 
Changed:
<
<		-- DavidTeyssier - 13 January 2016
>
>		We provide here the HUI requirements needed to support the storage, queries and access to the various HPDPs and Ancillary Legacy Products: HPDP_ALP_HUI_Requirement_v0.1.pdf
 
Changed:
<
<
>
>		-- DavidTeyssier - 18 February 2016
 
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1442955081" name="HPDP_meeting_ICS_16Sep2015_MoM.pdf" path="HPDP_meeting_ICS_16Sep2015_MoM.pdf" size="72409" user="Main.DavidTeyssier" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1434526816" name="HPDP_meeting_ICS_16June2015.pdf" path="HPDP_meeting_ICS_16June2015.pdf" size="1454187" user="Main.DavidTeyssier" version="2"
Line: 97 to 92
 
META FILEATTACHMENT attachment="HPDP_meeting_ICS_16Sep2015.pdf" attr="" comment="" date="1442955314" name="HPDP_meeting_ICS_16Sep2015.pdf" path="HPDP_meeting_ICS_16Sep2015.pdf" size="3118644" stream="HPDP_meeting_ICS_16Sep2015.pdf" user="Main.DavidTeyssier" version="1"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_12Jan2016_MoM.pdf" attr="" comment="" date="1453185828" name="HPDP_meeting_ICS_12Jan2016_MoM.pdf" path="HPDP_meeting_ICS_12Jan2016_MoM.pdf" size="102372" user="DavidTeyssier" version="5"
META FILEATTACHMENT attachment="PACSPhpdp.pdf" attr="" comment="" date="1455113973" name="PACSPhpdp.pdf" path="PACSPhpdp.pdf" size="4871999" user="LucaCalzoletti" version="5"
Added:
>
>
META FILEATTACHMENT attachment="HPDP_ALP_HUI_Requirement_v0.1.pdf" attr="" comment="" date="1455827776" name="HPDP_ALP_HUI_Requirement_v0.1.pdf" path="HPDP_ALP_HUI_Requirement_v0.1.pdf" size="231798" user="DavidTeyssier" version="1"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 322016-02-10 - LucaCalzoletti

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 40 to 40
 
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
PACS Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note  
Improved level3 maps Further combination of fields with Unimap and JScanam Yes No ?? ??  
Changed:
<
<
Additional level 2.5 maps Combination of fields currently unconsidered in the list for level2.5 pipeline Yes No ?? PACSPhpdp.pdf

 
>
>
Additional level 2.5 maps Combination of fields currently unconsidered in the list for level2.5 pipeline Yes No ??
PACSPhpdp.pdf
 
 

PACS-S

Line: 89 to 89
  -- DavidTeyssier - 13 January 2016
Added:
>
>
 
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1442955081" name="HPDP_meeting_ICS_16Sep2015_MoM.pdf" path="HPDP_meeting_ICS_16Sep2015_MoM.pdf" size="72409" user="Main.DavidTeyssier" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1434526816" name="HPDP_meeting_ICS_16June2015.pdf" path="HPDP_meeting_ICS_16June2015.pdf" size="1454187" user="Main.DavidTeyssier" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1434722888" name="HPDP_meeting_ICS_16June2015_MoM.pdf" path="HPDP_meeting_ICS_16June2015_MoM.pdf" size="210371" user="Main.DavidTeyssier" version="2"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_16Sep2015.pdf" attr="" comment="" date="1442955314" name="HPDP_meeting_ICS_16Sep2015.pdf" path="HPDP_meeting_ICS_16Sep2015.pdf" size="3118644" stream="HPDP_meeting_ICS_16Sep2015.pdf" user="Main.DavidTeyssier" version="1"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_12Jan2016_MoM.pdf" attr="" comment="" date="1453185828" name="HPDP_meeting_ICS_12Jan2016_MoM.pdf" path="HPDP_meeting_ICS_12Jan2016_MoM.pdf" size="102372" user="DavidTeyssier" version="5"
Changed:
<
<
META FILEATTACHMENT attachment="PACSPhpdp.pdf" attr="" comment="" date="1454509256" name="PACSPhpdp.pdf" path="PACSPhpdp.pdf" size="5394547" user="LucaCalzoletti" version="3"
>
>
META FILEATTACHMENT attachment="PACSPhpdp.pdf" attr="" comment="" date="1455113973" name="PACSPhpdp.pdf" path="PACSPhpdp.pdf" size="4871999" user="LucaCalzoletti" version="5"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 312016-02-03 - LucaCalzoletti

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 40 to 40
 
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
PACS Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note  
Improved level3 maps Further combination of fields with Unimap and JScanam Yes No ?? ??  
Changed:
<
<
Additional level 2.5 maps Combination of fields currently unconsidered in the list for level2.5 pipeline Yes No ?? PACSPhpdp.pdf  
>
>
Additional level 2.5 maps Combination of fields currently unconsidered in the list for level2.5 pipeline Yes No ?? PACSPhpdp.pdf

 
 

PACS-S

Line: 94 to 94
 
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1434722888" name="HPDP_meeting_ICS_16June2015_MoM.pdf" path="HPDP_meeting_ICS_16June2015_MoM.pdf" size="210371" user="Main.DavidTeyssier" version="2"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_16Sep2015.pdf" attr="" comment="" date="1442955314" name="HPDP_meeting_ICS_16Sep2015.pdf" path="HPDP_meeting_ICS_16Sep2015.pdf" size="3118644" stream="HPDP_meeting_ICS_16Sep2015.pdf" user="Main.DavidTeyssier" version="1"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_12Jan2016_MoM.pdf" attr="" comment="" date="1453185828" name="HPDP_meeting_ICS_12Jan2016_MoM.pdf" path="HPDP_meeting_ICS_12Jan2016_MoM.pdf" size="102372" user="DavidTeyssier" version="5"
Changed:
<
<
META FILEATTACHMENT attachment="PACSPhpdp.pdf" attr="" comment="" date="1454409937" name="PACSPhpdp.pdf" path="PACSPhpdp.pdf" size="5373149" user="LucaCalzoletti" version="1"
>
>
META FILEATTACHMENT attachment="PACSPhpdp.pdf" attr="" comment="" date="1454509256" name="PACSPhpdp.pdf" path="PACSPhpdp.pdf" size="5394547" user="LucaCalzoletti" version="3"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 302016-02-02 - LucaCalzoletti

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 40 to 40
 
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
PACS Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note  
Improved level3 maps Further combination of fields with Unimap and JScanam Yes No ?? ??  
Changed:
<
<
Additional level 2.5 maps Combination of fields currently unconsidered in the list for level2.5 pipeline Yes No ?? ??  
>
>
Additional level 2.5 maps Combination of fields currently unconsidered in the list for level2.5 pipeline Yes No ?? PACSPhpdp.pdf  
 

PACS-S

Line: 67 to 67
 
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
SPIRE Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note  
Changed:
<
<
<--| HiRes maps  | Reprocessing of SPIRE-P maps with the HiRes algorithm |  Yes - level 2 or level 2.5 equivalent FITS files | No  |  ?? |  ?? | | -->
>
>		<!--| HiRes maps | Reprocessing of SPIRE-P maps with the HiRes algorithm | Yes - level 2 or level 2.5 equivalent FITS files | No | ?? | ?? | | -->
 

SPIRE-S

Line: 94 to 94
 
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1434722888" name="HPDP_meeting_ICS_16June2015_MoM.pdf" path="HPDP_meeting_ICS_16June2015_MoM.pdf" size="210371" user="Main.DavidTeyssier" version="2"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_16Sep2015.pdf" attr="" comment="" date="1442955314" name="HPDP_meeting_ICS_16Sep2015.pdf" path="HPDP_meeting_ICS_16Sep2015.pdf" size="3118644" stream="HPDP_meeting_ICS_16Sep2015.pdf" user="Main.DavidTeyssier" version="1"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_12Jan2016_MoM.pdf" attr="" comment="" date="1453185828" name="HPDP_meeting_ICS_12Jan2016_MoM.pdf" path="HPDP_meeting_ICS_12Jan2016_MoM.pdf" size="102372" user="DavidTeyssier" version="5"
Added:
>
>
META FILEATTACHMENT attachment="PACSPhpdp.pdf" attr="" comment="" date="1454409937" name="PACSPhpdp.pdf" path="PACSPhpdp.pdf" size="5373149" user="LucaCalzoletti" version="1"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 292016-01-19 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 93 to 93
 
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1434526816" name="HPDP_meeting_ICS_16June2015.pdf" path="HPDP_meeting_ICS_16June2015.pdf" size="1454187" user="Main.DavidTeyssier" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1434722888" name="HPDP_meeting_ICS_16June2015_MoM.pdf" path="HPDP_meeting_ICS_16June2015_MoM.pdf" size="210371" user="Main.DavidTeyssier" version="2"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_16Sep2015.pdf" attr="" comment="" date="1442955314" name="HPDP_meeting_ICS_16Sep2015.pdf" path="HPDP_meeting_ICS_16Sep2015.pdf" size="3118644" stream="HPDP_meeting_ICS_16Sep2015.pdf" user="Main.DavidTeyssier" version="1"
Changed:
<
<
META FILEATTACHMENT attachment="HPDP_meeting_ICS_12Jan2016_MoM.pdf" attr="" comment="" date="1452688017" name="HPDP_meeting_ICS_12Jan2016_MoM.pdf" path="HPDP_meeting_ICS_12Jan2016_MoM.pdf" size="102325" user="DavidTeyssier" version="4"
>
>
META FILEATTACHMENT attachment="HPDP_meeting_ICS_12Jan2016_MoM.pdf" attr="" comment="" date="1453185828" name="HPDP_meeting_ICS_12Jan2016_MoM.pdf" path="HPDP_meeting_ICS_12Jan2016_MoM.pdf" size="102372" user="DavidTeyssier" version="5"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 282016-01-13 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 93 to 93
 
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1434526816" name="HPDP_meeting_ICS_16June2015.pdf" path="HPDP_meeting_ICS_16June2015.pdf" size="1454187" user="Main.DavidTeyssier" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1434722888" name="HPDP_meeting_ICS_16June2015_MoM.pdf" path="HPDP_meeting_ICS_16June2015_MoM.pdf" size="210371" user="Main.DavidTeyssier" version="2"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_16Sep2015.pdf" attr="" comment="" date="1442955314" name="HPDP_meeting_ICS_16Sep2015.pdf" path="HPDP_meeting_ICS_16Sep2015.pdf" size="3118644" stream="HPDP_meeting_ICS_16Sep2015.pdf" user="Main.DavidTeyssier" version="1"
Changed:
<
<
META FILEATTACHMENT attachment="HPDP_meeting_ICS_12Jan2016_MoM.pdf" attr="" comment="" date="1452681238" name="HPDP_meeting_ICS_12Jan2016_MoM.pdf" path="HPDP_meeting_ICS_12Jan2016_MoM.pdf" size="101972" user="DavidTeyssier" version="3"
>
>
META FILEATTACHMENT attachment="HPDP_meeting_ICS_12Jan2016_MoM.pdf" attr="" comment="" date="1452688017" name="HPDP_meeting_ICS_12Jan2016_MoM.pdf" path="HPDP_meeting_ICS_12Jan2016_MoM.pdf" size="102325" user="DavidTeyssier" version="4"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 272016-01-13 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 18 to 18
 
Added:
>
>
 
  • Next meeting planned Mid-April 2016

HPDP plans per instruments

Line: 86 to 87
  Ancillary Legacy Products are compiled in the following page
Changed:
<
<		-- DavidTeyssier - 11 January 2016
>
>		-- DavidTeyssier - 13 January 2016
 
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1442955081" name="HPDP_meeting_ICS_16Sep2015_MoM.pdf" path="HPDP_meeting_ICS_16Sep2015_MoM.pdf" size="72409" user="Main.DavidTeyssier" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1434526816" name="HPDP_meeting_ICS_16June2015.pdf" path="HPDP_meeting_ICS_16June2015.pdf" size="1454187" user="Main.DavidTeyssier" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1434722888" name="HPDP_meeting_ICS_16June2015_MoM.pdf" path="HPDP_meeting_ICS_16June2015_MoM.pdf" size="210371" user="Main.DavidTeyssier" version="2"
META FILEATTACHMENT attachment="HPDP_meeting_ICS_16Sep2015.pdf" attr="" comment="" date="1442955314" name="HPDP_meeting_ICS_16Sep2015.pdf" path="HPDP_meeting_ICS_16Sep2015.pdf" size="3118644" stream="HPDP_meeting_ICS_16Sep2015.pdf" user="Main.DavidTeyssier" version="1"
Added:
>
>
META FILEATTACHMENT attachment="HPDP_meeting_ICS_12Jan2016_MoM.pdf" attr="" comment="" date="1452681238" name="HPDP_meeting_ICS_12Jan2016_MoM.pdf" path="HPDP_meeting_ICS_12Jan2016_MoM.pdf" size="101972" user="DavidTeyssier" version="3"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 262016-01-12 - KatrinaExter

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 45 to 45
  In order of priority+feasibility:
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Notes Prototype delivery
Changed:
<
<
Pipeline processing bright sources Very bright sources ought to be reduced with a difference pipeline script (to account for non-linearity/self-curing) Yes - level2/2.5 equivalent FITS files at a minimum, or entire obsContext if done via SPG No Yes   Some level of manual reduction may be necessary, but are probably not many such observations  
>
>
Pipeline processing bright sources Very bright sources ought to be reduced with a different pipeline script (to account for non-linearity/self-curing) Yes - level2/2.5 equivalent FITS files at a minimum, or entire obsContext if done via SPG No Yes   Some level of manual reduction may be necessary, but are probably not many such observations  
 
Off-centred point-source calibrated data For off-centred point-like sources: SPG products processed through the standard pipeline script with non-standard extraction of final PS spectrum Yes - level2/2.5 equivalent FITS files No Yes   A useful possibility, feasibility will be more clear once QC work (http://herschel.esac.esa.int/twiki/bin/view/Pacs/QualityCommentsSpec) on off-centred observations and identification of likely centred point sources has been done  
Long range spectrometer flat-fielding - potential Correcting the few cases of bad flatfielding Yes - level2/2.5 equivalent FITS files No Yes   May be unnecessary if the new range scan flatfielding works well. Pilar will check a subset of the 14.0 bulk results. Sample relevance analysis.  
Unchopped observations: pipeline processing with new response drifts correction Improved transient correction and background subtraction as provided by the new NHSC pipeline scripts Yes - level2/2.5 equivalent FITS files, or entire obsContext if done via SPG No Yes Could be proposed for 14.1 SPG rather than HPDP Highly desirable, but requires a fuller validation of this script first (if the script does not provide improved results, neither SPG nor HPDP can be created here). Conditional to NHSC bulk to be run.  

Revision 252016-01-12 - IvanV

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 66 to 66
 
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
SPIRE Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note  
Changed:
<
<		<!--| HiRes maps | Reprocessing of SPIRE-P maps with the HiRes algorithm | Yes - level 2 or level 2.5 equivalent FITS files | No | ?? | ?? | | -->
>
>
<--| HiRes maps  | Reprocessing of SPIRE-P maps with the HiRes algorithm |  Yes - level 2 or level 2.5 equivalent FITS files | No  |  ?? |  ?? | | -->
 

SPIRE-S

Line: 74 to 74
 
Semi-Extended calibration correction SECT correction for semi-compact sources Yes - level2 equivalent FITS files No Yes - postcards Yes  
Extended background correction Extended background correction for point sources Yes - level2 equivalent FITS files No Yes - postcards Yes  
Anomalous data correction Isolated cases of failed processing due to some on-board anomaly Yes - level2 equivalent FITS files No Yes - postcards Yes  
Changed:
<
<
Line list catalogue Spectral Feature catalogue (see also this page ) No Yes - one file per obsid, and one concatenated file No ??  
>
>
Line list catalogue Spectral Feature catalogue (see also this page ) No Yes - one file per obsid, and one concatenated file Yes Yes - paper  
 
Spectral scans Un-averaged level-2 spectra Yes - level2 equivalent products No No No

HUI requirements

Revision 242016-01-12 - KatrinaExter

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Changed:
<
<
>
>
 

Introduction

Line: 22 to 22
 

HPDP plans per instruments

Changed:
<
<		HPDPs have been identified by each instrument team, although it is likely that not all of them will be possible to prepared in certain cases. We indicate hereafter the HPDPs currently contemplated by each instruments, with some common delivery categories that will be considered or not.
>
>		HPDPs have been identified by each instrument team, although it is likely that not all of them will be possible to prepared in certain cases. We indicate hereafter the HPDPs currently contemplated by each instruments, with some common delivery categories that will be considered or not.
 

HIFI

Line: 44 to 43
 

PACS-S

Changed:
<
<		In order of priority/doability:
>
>		In order of priority+feasibility:
 
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Notes Prototype delivery
Pipeline processing bright sources Very bright sources ought to be reduced with a difference pipeline script (to account for non-linearity/self-curing) Yes - level2/2.5 equivalent FITS files at a minimum, or entire obsContext if done via SPG No Yes   Some level of manual reduction may be necessary, but are probably not many such observations  
Changed:
<
<
Off-centred point-source calibrated data For off-centred point sources: SPG products processed through the standard pipeline script with non-standard extraction of final PS spectrum Yes - level2/2.5 equivalent FITS files No Yes   A useful possibility, feasibility will be more clear once QC work (http://herschel.esac.esa.int/twiki/bin/view/Pacs/QualityCommentsSpec) on off-centred observations and identification of likely centred point sources has been done  
Long range spectrometer flat-fielding - potential Correcting the few cases of bad flatfielding Yes - level2/2.5 equivalent FITS files No Yes   may be unnecessary if the new range scan flatfielding works well. Pilar will check some of the 14.0 bulk results. Sample relevance analysis.  
>
>
Off-centred point-source calibrated data For off-centred point-like sources: SPG products processed through the standard pipeline script with non-standard extraction of final PS spectrum Yes - level2/2.5 equivalent FITS files No Yes   A useful possibility, feasibility will be more clear once QC work (http://herschel.esac.esa.int/twiki/bin/view/Pacs/QualityCommentsSpec) on off-centred observations and identification of likely centred point sources has been done  
Long range spectrometer flat-fielding - potential Correcting the few cases of bad flatfielding Yes - level2/2.5 equivalent FITS files No Yes   May be unnecessary if the new range scan flatfielding works well. Pilar will check a subset of the 14.0 bulk results. Sample relevance analysis.  
 
Unchopped observations: pipeline processing with new response drifts correction Improved transient correction and background subtraction as provided by the new NHSC pipeline scripts Yes - level2/2.5 equivalent FITS files, or entire obsContext if done via SPG No Yes Could be proposed for 14.1 SPG rather than HPDP Highly desirable, but requires a fuller validation of this script first (if the script does not provide improved results, neither SPG nor HPDP can be created here). Conditional to NHSC bulk to be run.  
Unchopped observations: subtracting the baseline If the continuum in these AOTs cannot be improved (wrt absolute level and shape, less wrt SNR) then instead subtract the baseline Yes - level2/2.5 equivalent FITS files: baseline subtracted cubes/spectra and cubes of the baseline that was subtracted No Yes   Desirable, necessary if the new transients correction pipeline does not becomes SPG in 14.1. Sample relevance analysis.  
Changed:
<
<
Unchopped observations: manual processing to correct very bad transients Improved transient correction and background subtraction provided by processing the data manually Yes - level2/2.5 equivalent FITS files No Yes   Highly desirable, but depends on how many such observations there are and on progress on PACS-6462. Sample relevance analysis.  
Point-source calibrated data using the "POC" script products processed through the dedicated pointing offset correction script for bright point sources Yes - level2/2.5 equivalent FITS files No Yes   straightforward to do once the pipeline script has been fully validated and sources identified; validation is on-going (but not guaranteed to finish)  
>
>
Unchopped observations: manual processing to correct very bad transients Improved transient correction and background subtraction provided by processing the data manually Yes - level2/2.5 equivalent FITS files No Yes   Highly desirable, but depends on how many such observations there are and on progress on PACS-6264. Sample relevance analysis.  
Point-source calibrated data using the "POC" script products processed through the dedicated pointing offset correction script for bright point sources Yes - level2/2.5 equivalent FITS files No Yes   Straightforward to do once the pipeline script has been fully validated and sources identified; validation is still on-going (postponed by the 14.0 AAT)  
 
Line list catalogue Spectral Feature catalogue No Yes - one file per obsid, and one concatenated file No   HPDP: dependent on available human resources, would be completely HSC to do, exploratory studies are being done with help of MSc students at ESAC and Leuven and we should know better early in 2016  
Changed:
<
<
Identifying background contamination; removing it? For ChopNod and Unchopped AOTs, looking at line (and continuum?) contamination in the off positions: primarily to make a quality comment, but if the number is manageable then also to correct for the contamination Yes - level2/2.5 equivalent FITS files No Yes   HPDP, for the quality comment part the ICC had agreed to do this, but for the subtraction part it would have to be done at HSC (note: unlikely to ever actually be completed, however, due to it being a difficult job)  
These should actually be SPG (14.1):
>
>
Identifying background contamination; removing it? For ChopNod and Unchopped AOTs, looking at line (and continuum?) contamination in the off positions: primarily to make a quality comment, but if the number is manageable then also to correct for the contamination Yes - level2/2.5 equivalent FITS files No Yes   HPDP, for the quality comment part (just identifying that there is contamination) the ICC had agreed to do this (although work is not yet on-going), but for the subtraction part it would have to be done at HSC, if deemed necessary  
 
Added:
>
>		These have been done for 14.0 or will be SPG 14.1:
 
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Notes Prototype delivery
Point-source calibrated data (pointed AOTs only) SPG products processed through the standard pipeline script and then creating point source spectra from them Yes, part of SPG obsContext No Yes   Note: is done on all pointed AOTs, regardless of whether the source is know to be a point or not. Done in 14.0  
Long range spectrometer flat-fielding - main Including flatfielding for range scans in the SPG Yes, entire obsContext No Yes   depends on validation in AAT 14. Done in 14.  
Line: 68 to 66
 
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
SPIRE Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note  
Changed:
<
<
<--| HiRes maps  | Reprocessing of SPIRE-P maps with the HiRes algorithm |  Yes - level 2 or level 2.5 equivalent FITS files | No  |  ?? |  ?? | | -->
>
>		<!--| HiRes maps | Reprocessing of SPIRE-P maps with the HiRes algorithm | Yes - level 2 or level 2.5 equivalent FITS files | No | ?? | ?? | | -->
 

SPIRE-S

Revision 232016-01-11 - ElenaPuga

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 47 to 47
 In order of priority/doability:

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Notes Prototype delivery
Changed:
<
<
Pipeline processing bright sources Very bright sources ought to be reduced with a difference pipeline script (to account for non-linearity/self-curing) Yes - level2/2.5 equivalent FITS files at a minimum, or entire obsContext if done via SPG No Yes To be proposed for 14.1 SPG rather than provide as an HPDP Some level of manual reduction may be necessary, but are probably not many such observations  
>
>
Pipeline processing bright sources Very bright sources ought to be reduced with a difference pipeline script (to account for non-linearity/self-curing) Yes - level2/2.5 equivalent FITS files at a minimum, or entire obsContext if done via SPG No Yes   Some level of manual reduction may be necessary, but are probably not many such observations  
 
Off-centred point-source calibrated data For off-centred point sources: SPG products processed through the standard pipeline script with non-standard extraction of final PS spectrum Yes - level2/2.5 equivalent FITS files No Yes   A useful possibility, feasibility will be more clear once QC work (http://herschel.esac.esa.int/twiki/bin/view/Pacs/QualityCommentsSpec) on off-centred observations and identification of likely centred point sources has been done  
Changed:
<
<
Long range spectrometer flat-fielding - potential Correcting the few cases of bad flatfielding Yes - level2/2.5 equivalent FITS files No Yes   may be unnecessary if the new range scan flatfielding works well  
Unchopped observations: pipeline processing with new response drifts correction Improved transient correction and background subtraction as provided by the new NHSC pipeline scripts Yes - level2/2.5 equivalent FITS files, or entire obsContext if done via SPG No Yes To be proposed for 14.1 SPG rather than provide as HPSP Highly desirable, but requires a fuller validation of this script first (if the script does not provide improved results, neither SPG nor HPDP can be created here)  
Unchopped observations: subtracting the baseline If the continuum in these AOTs cannot be improved (wrt absolute level and shape, less wrt SNR) then instead subtract the baseline Yes - level2/2.5 equivalent FITS files: baseline subtracted cubes/spectra and cubes of the baseline that was subtracted No Yes   Desirable, necessary if the new transients correction pipeline does not becomes SPG in 14.1  
Unchopped observations: manual processing to correct very bad transients Improved transient correction and background subtraction provided by processing the data manually Yes - level2/2.5 equivalent FITS files No Yes   Highly desirable, but depends on how many such observations there are and on progress on PACS-6462  
>
>
Long range spectrometer flat-fielding - potential Correcting the few cases of bad flatfielding Yes - level2/2.5 equivalent FITS files No Yes   may be unnecessary if the new range scan flatfielding works well. Pilar will check some of the 14.0 bulk results. Sample relevance analysis.  
Unchopped observations: pipeline processing with new response drifts correction Improved transient correction and background subtraction as provided by the new NHSC pipeline scripts Yes - level2/2.5 equivalent FITS files, or entire obsContext if done via SPG No Yes Could be proposed for 14.1 SPG rather than HPDP Highly desirable, but requires a fuller validation of this script first (if the script does not provide improved results, neither SPG nor HPDP can be created here). Conditional to NHSC bulk to be run.  
Unchopped observations: subtracting the baseline If the continuum in these AOTs cannot be improved (wrt absolute level and shape, less wrt SNR) then instead subtract the baseline Yes - level2/2.5 equivalent FITS files: baseline subtracted cubes/spectra and cubes of the baseline that was subtracted No Yes   Desirable, necessary if the new transients correction pipeline does not becomes SPG in 14.1. Sample relevance analysis.  
Unchopped observations: manual processing to correct very bad transients Improved transient correction and background subtraction provided by processing the data manually Yes - level2/2.5 equivalent FITS files No Yes   Highly desirable, but depends on how many such observations there are and on progress on PACS-6462. Sample relevance analysis.  
 
Point-source calibrated data using the "POC" script products processed through the dedicated pointing offset correction script for bright point sources Yes - level2/2.5 equivalent FITS files No Yes   straightforward to do once the pipeline script has been fully validated and sources identified; validation is on-going (but not guaranteed to finish)  
Line list catalogue Spectral Feature catalogue No Yes - one file per obsid, and one concatenated file No   HPDP: dependent on available human resources, would be completely HSC to do, exploratory studies are being done with help of MSc students at ESAC and Leuven and we should know better early in 2016  
Changed:
<
<
Identifying background contamination; removing it? For ChopNod and Unchopped AOTs, looking at line (and continuum?) contamination in the off positions: primarily to make a quality comment, but if the number is manageable then also to correct for the contamination Yes - level2/2.5 equivalent FITS files No Yes   HPDP, for the quality comment part the ICC have agreed to do this, but for the subtraction part it would have to be done at HSC (note: unlikely to ever actually be completed, however, due to it being a difficult job)  
These should actually be SPG (14 or 14.1):
>
>
Identifying background contamination; removing it? For ChopNod and Unchopped AOTs, looking at line (and continuum?) contamination in the off positions: primarily to make a quality comment, but if the number is manageable then also to correct for the contamination Yes - level2/2.5 equivalent FITS files No Yes   HPDP, for the quality comment part the ICC had agreed to do this, but for the subtraction part it would have to be done at HSC (note: unlikely to ever actually be completed, however, due to it being a difficult job)  
These should actually be SPG (14.1):
 
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Notes Prototype delivery
Changed:
<
<
Point-source calibrated data (pointed AOTs only) SPG products processed through the standard pipeline script and then creating point source spectra from them Yes, part of SPG obsContext No Yes   Note: is done on all pointed AOTs, regardless of whether the source is know to be a point or not  
Long range spectrometer flat-fielding - main Including flatfielding for range scans in the SPG Yes, entire obsContext No Yes   depends on validation in AAT 14 but is looking good  
Red leak spectral range Run the specific pipeline that deals with red leak region (CalBlock+RSRF) and provide data stand-alone Yes, part of entire obsContext No Yes   Hoping for 14, but more likely to be 14.1  
Repair of failed observations Re-run pipeline on observations set to failed Yes, entire obsContext No Yes   Observations that are classed as failed or delivered for reprocessing will be processed in 14; of those that fail, for those that can none-the-less be processed manually and are worth so-doing  
>
>
Point-source calibrated data (pointed AOTs only) SPG products processed through the standard pipeline script and then creating point source spectra from them Yes, part of SPG obsContext No Yes   Note: is done on all pointed AOTs, regardless of whether the source is know to be a point or not. Done in 14.0  
Long range spectrometer flat-fielding - main Including flatfielding for range scans in the SPG Yes, entire obsContext No Yes   depends on validation in AAT 14. Done in 14.  
Red leak spectral range Run the specific pipeline that deals with red leak region (CalBlock+RSRF) and provide data stand-alone Yes, part of entire obsContext No Yes   For 14.1. Requires delivery of red de-leaked rsrf v5 and extra work on range flatfielding.  
Repair of failed observations Re-run pipeline on observations set to failed Yes, entire obsContext No Yes   Observations that are classed as failed or delivered for reprocessing were processed in 14.0; of those that fail, are FAILED for good. Work is done by
the bulk. 		 
 

SPIRE-P

Revision 222016-01-11 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 84 to 84
 We provide here the HUI requirements needed to support the storage, queries and access to the various HPDPs.

TBD

Added:
>
>

Ancillary Legacy Products

 
Changed:
<
<		-- DavidTeyssier - 7 January 2016
>
>		Ancillary Legacy Products are compiled in the following page

-- DavidTeyssier - 11 January 2016

 
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1442955081" name="HPDP_meeting_ICS_16Sep2015_MoM.pdf" path="HPDP_meeting_ICS_16Sep2015_MoM.pdf" size="72409" user="Main.DavidTeyssier" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1434526816" name="HPDP_meeting_ICS_16June2015.pdf" path="HPDP_meeting_ICS_16June2015.pdf" size="1454187" user="Main.DavidTeyssier" version="2"

Revision 212016-01-07 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 17 to 17
 
Changed:
<
<
  • Next meeting planned Mid-December 2015
>
>
  • 12-January-2016: Meeting #3
  • Next meeting planned Mid-April 2016
 

HPDP plans per instruments

Line: 84 to 85
  TBD
Changed:
<
<		-- DavidTeyssier - 17 Jun 2015
>
>		-- DavidTeyssier - 7 January 2016
 
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1442955081" name="HPDP_meeting_ICS_16Sep2015_MoM.pdf" path="HPDP_meeting_ICS_16Sep2015_MoM.pdf" size="72409" user="Main.DavidTeyssier" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1434526816" name="HPDP_meeting_ICS_16June2015.pdf" path="HPDP_meeting_ICS_16June2015.pdf" size="1454187" user="Main.DavidTeyssier" version="2"

Revision 202015-11-15 - IvanV

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 67 to 67
 
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
SPIRE Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note  
Changed:
<
<
HiRes maps Reprocessing of SPIRE-P maps with the HiRes algorithm Yes - level 2 or level 2.5 equivalent FITS files No ?? ??  
>
>
<--| HiRes maps  | Reprocessing of SPIRE-P maps with the HiRes algorithm |  Yes - level 2 or level 2.5 equivalent FITS files | No  |  ?? |  ?? | | -->
 

SPIRE-S

Line: 76 to 76
 
Extended background correction Extended background correction for point sources Yes - level2 equivalent FITS files No Yes - postcards Yes  
Anomalous data correction Isolated cases of failed processing due to some on-board anomaly Yes - level2 equivalent FITS files No Yes - postcards Yes  
Line list catalogue Spectral Feature catalogue (see also this page ) No Yes - one file per obsid, and one concatenated file No ??  
Deleted:
<
<
Convolved spectral cubes Apply gridding algorithm to the spectral 2d pre-cubes Yes - level2 equivalent products No No No
 
Spectral scans Un-averaged level-2 spectra Yes - level2 equivalent products No No No

HUI requirements

Revision 192015-10-20 - ElenaPuga

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 48 to 47
 
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Notes Prototype delivery
Pipeline processing bright sources Very bright sources ought to be reduced with a difference pipeline script (to account for non-linearity/self-curing) Yes - level2/2.5 equivalent FITS files at a minimum, or entire obsContext if done via SPG No Yes To be proposed for 14.1 SPG rather than provide as an HPDP Some level of manual reduction may be necessary, but are probably not many such observations  
Changed:
<
<
Point-source calibrated data For off-centred point sources: SPG products processed through the standard pipeline script with non-standard extraction of final PS spectrum Yes - level2/2.5 equivalent FITS files No Yes   A useful possibility, feasibility will be more clear once QC work (http://herschel.esac.esa.int/twiki/bin/view/Pacs/QualityCommentsSpec) on off-centred observations and identification of likely centred point sources has been done  
>
>
Off-centred point-source calibrated data For off-centred point sources: SPG products processed through the standard pipeline script with non-standard extraction of final PS spectrum Yes - level2/2.5 equivalent FITS files No Yes   A useful possibility, feasibility will be more clear once QC work (http://herschel.esac.esa.int/twiki/bin/view/Pacs/QualityCommentsSpec) on off-centred observations and identification of likely centred point sources has been done  
 
Long range spectrometer flat-fielding - potential Correcting the few cases of bad flatfielding Yes - level2/2.5 equivalent FITS files No Yes   may be unnecessary if the new range scan flatfielding works well  
Unchopped observations: pipeline processing with new response drifts correction Improved transient correction and background subtraction as provided by the new NHSC pipeline scripts Yes - level2/2.5 equivalent FITS files, or entire obsContext if done via SPG No Yes To be proposed for 14.1 SPG rather than provide as HPSP Highly desirable, but requires a fuller validation of this script first (if the script does not provide improved results, neither SPG nor HPDP can be created here)  
Unchopped observations: subtracting the baseline If the continuum in these AOTs cannot be improved (wrt absolute level and shape, less wrt SNR) then instead subtract the baseline Yes - level2/2.5 equivalent FITS files: baseline subtracted cubes/spectra and cubes of the baseline that was subtracted No Yes   Desirable, necessary if the new transients correction pipeline does not becomes SPG in 14.1  
Line: 77 to 75
 
Semi-Extended calibration correction SECT correction for semi-compact sources Yes - level2 equivalent FITS files No Yes - postcards Yes  
Extended background correction Extended background correction for point sources Yes - level2 equivalent FITS files No Yes - postcards Yes  
Anomalous data correction Isolated cases of failed processing due to some on-board anomaly Yes - level2 equivalent FITS files No Yes - postcards Yes  
Changed:
<
<
Line list catalogue Spectral Feature catalogue (see also this page ) No Yes - one file per obsid, and one concatenated file No ??  
>
>
Line list catalogue Spectral Feature catalogue (see also this page ) No Yes - one file per obsid, and one concatenated file No ??  
 
Convolved spectral cubes Apply gridding algorithm to the spectral 2d pre-cubes Yes - level2 equivalent products No No No
Spectral scans Un-averaged level-2 spectra Yes - level2 equivalent products No No No

Revision 182015-09-22 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 10 to 10
 

Meetings and action items

Changed:
<
<
  • 17-June-2015: Kick-off meeting with the ICS team and Eva Verdugo.
>
>
  • 16-June-2015: Kick-off meeting with the ICS team and Eva Verdugo.
 
    • Minutes of Meeting
    • Viewgraphs used during the meeting
    • The main action item resulting from the meeting is that each instrument representative shall come up with a list of use cases and requirement wish list applying to the HPDP they are supervising or acting as contact point for. These requirements will be compiled in the corresponding section below. Deadline: second half of September.
Changed:
<
<
  • Mid-September 2015: Follow-up meeting to review requirements and agree on a consolidated request to SAT to be conveyed by Eva Verdugo
>
>
 

HPDP plans per instruments

Line: 89 to 93
  -- DavidTeyssier - 17 Jun 2015
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1442955081" name="HPDP_meeting_ICS_16Sep2015_MoM.pdf" path="HPDP_meeting_ICS_16Sep2015_MoM.pdf" size="72409" user="Main.DavidTeyssier" version="1"
 
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1434526816" name="HPDP_meeting_ICS_16June2015.pdf" path="HPDP_meeting_ICS_16June2015.pdf" size="1454187" user="Main.DavidTeyssier" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1434722888" name="HPDP_meeting_ICS_16June2015_MoM.pdf" path="HPDP_meeting_ICS_16June2015_MoM.pdf" size="210371" user="Main.DavidTeyssier" version="2"
Added:
>
>
META FILEATTACHMENT attachment="HPDP_meeting_ICS_16Sep2015.pdf" attr="" comment="" date="1442955314" name="HPDP_meeting_ICS_16Sep2015.pdf" path="HPDP_meeting_ICS_16Sep2015.pdf" size="3118644" stream="HPDP_meeting_ICS_16Sep2015.pdf" user="Main.DavidTeyssier" version="1"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 172015-09-16 - KatrinaExter

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 40 to 40
 

PACS-S

Changed:
<
<		In order of priority: these first are HPDPs
>
>		In order of priority/doability:
 
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Notes Prototype delivery
Changed:
<
<
Point-source calibrated data - case 1 For off-centred sources that are likely to be point sources: SPG products processed through the standard pipeline script with non-standard extraction of final PS spectrum Yes - level2/2.5 equivalent FITS files No Yes   A useful possibility, feasibility will be more clear once QC work on off-centred observations and identification of likely centred point sources has been done  
High flux source data Very bright sources ought to be reduced with a difference pipeline script (to account for non-linearity/self-curing) Yes - level2/2.5 equivalent FITS files No Yes To be proposed for 14.1 SPG Some level of manual reduction may be necessary, but are probably not many such observations  
Unchopped observations: response drifts - case 1 Unchopped range scans severely affected by response drift (e.g. after high glitch rate or after high-flux source observation) Yes - level2/2.5 equivalent FITS files No Yes To be proposed for 14.1 SPG Highly desirable  
Unchopped observations: transients - case 1 Baseline-subtracted unchopped spectra Yes - level2/2.5 equivalent FITS files No Yes   Desirable, if case 2 is not possible this becomes Highly desirable  
Unchopped observations: transients - case 2 Improved transient correction and background subtraction Yes - level2/2.5 equivalent FITS files No Yes To be proposed for 14.1 SPG Highly desirable, if it is possible to do this is preferred over case 1  
Identifying background contamination ChopNod and Unchopped, Line (and continuum?) Yes - level2/2.5 equivalent FITS files No Yes   HPDP, but depends on how possible it will be to identify these and how many we find  
Point-source calibrated data - case 2 products processed through the dedicated pointing offset correction script for sources consistent with being centred point source Yes - level2/2.5 equivalent FITS files No Yes   straightforward to do once the pipeline script has been fully validated and sources identified; validation is on-going but not guaranteed  
Line list catalogue Spectral Feature catalogue No Yes - one file per obsid, and one concatenated file No   HPDP: dependent on available human resources, would be completely HSC to decide we want to do this as is not on ICC plans  
>
>
Pipeline processing bright sources Very bright sources ought to be reduced with a difference pipeline script (to account for non-linearity/self-curing) Yes - level2/2.5 equivalent FITS files at a minimum, or entire obsContext if done via SPG No Yes To be proposed for 14.1 SPG rather than provide as an HPDP Some level of manual reduction may be necessary, but are probably not many such observations  
Point-source calibrated data For off-centred point sources: SPG products processed through the standard pipeline script with non-standard extraction of final PS spectrum Yes - level2/2.5 equivalent FITS files No Yes   A useful possibility, feasibility will be more clear once QC work (http://herschel.esac.esa.int/twiki/bin/view/Pacs/QualityCommentsSpec) on off-centred observations and identification of likely centred point sources has been done  
Long range spectrometer flat-fielding - potential Correcting the few cases of bad flatfielding Yes - level2/2.5 equivalent FITS files No Yes   may be unnecessary if the new range scan flatfielding works well  
Unchopped observations: pipeline processing with new response drifts correction Improved transient correction and background subtraction as provided by the new NHSC pipeline scripts Yes - level2/2.5 equivalent FITS files, or entire obsContext if done via SPG No Yes To be proposed for 14.1 SPG rather than provide as HPSP Highly desirable, but requires a fuller validation of this script first (if the script does not provide improved results, neither SPG nor HPDP can be created here)  
Unchopped observations: subtracting the baseline If the continuum in these AOTs cannot be improved (wrt absolute level and shape, less wrt SNR) then instead subtract the baseline Yes - level2/2.5 equivalent FITS files: baseline subtracted cubes/spectra and cubes of the baseline that was subtracted No Yes   Desirable, necessary if the new transients correction pipeline does not becomes SPG in 14.1  
Unchopped observations: manual processing to correct very bad transients Improved transient correction and background subtraction provided by processing the data manually Yes - level2/2.5 equivalent FITS files No Yes   Highly desirable, but depends on how many such observations there are and on progress on PACS-6462  
Point-source calibrated data using the "POC" script products processed through the dedicated pointing offset correction script for bright point sources Yes - level2/2.5 equivalent FITS files No Yes   straightforward to do once the pipeline script has been fully validated and sources identified; validation is on-going (but not guaranteed to finish)  
Line list catalogue Spectral Feature catalogue No Yes - one file per obsid, and one concatenated file No   HPDP: dependent on available human resources, would be completely HSC to do, exploratory studies are being done with help of MSc students at ESAC and Leuven and we should know better early in 2016  
Identifying background contamination; removing it? For ChopNod and Unchopped AOTs, looking at line (and continuum?) contamination in the off positions: primarily to make a quality comment, but if the number is manageable then also to correct for the contamination Yes - level2/2.5 equivalent FITS files No Yes   HPDP, for the quality comment part the ICC have agreed to do this, but for the subtraction part it would have to be done at HSC (note: unlikely to ever actually be completed, however, due to it being a difficult job)  
 
Changed:
<
<		These should be SPG:
>
>		These should actually be SPG (14 or 14.1):
 
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Notes Prototype delivery
Changed:
<
<
Point-source calibrated data - case 3 SPG products processed through the standard pipeline script for sources consistent with being centred point sources Yes - level2/2.5 equivalent FITS files No Yes   SPG  
Long range spectro flat-fielding - main Including flatfielding for range scans in the SPG Yes - level2/2.5 equivalent FITS files No Yes   SPG  
Long range spectro flat-fielding - potential Correcting the few cases of bad flatfielding (probably unnecessary) Yes - level2/2.5 equivalent FITS files No Yes   HPDP or SPG, depends on whether/how many of these will be present after the final SPG  
Red leak spectral range Run the specific pipeline that deals with red leak region (CalBlock+RSRF) and provide data stand-alone Yes - level2 equivalent FITS files No Yes   SPG  
Repair of failed observations Re-run pipeline on observations set to failed Yes - level2/2.5 equivalent FITS files No Yes   Observations that are classed as failed but which will now run in SPG are easy to do and could become standard SPG, for others that require manual reduction they could also become the standard SPG, probably not difficult  
>
>
Point-source calibrated data (pointed AOTs only) SPG products processed through the standard pipeline script and then creating point source spectra from them Yes, part of SPG obsContext No Yes   Note: is done on all pointed AOTs, regardless of whether the source is know to be a point or not  
Long range spectrometer flat-fielding - main Including flatfielding for range scans in the SPG Yes, entire obsContext No Yes   depends on validation in AAT 14 but is looking good  
Red leak spectral range Run the specific pipeline that deals with red leak region (CalBlock+RSRF) and provide data stand-alone Yes, part of entire obsContext No Yes   Hoping for 14, but more likely to be 14.1  
Repair of failed observations Re-run pipeline on observations set to failed Yes, entire obsContext No Yes   Observations that are classed as failed or delivered for reprocessing will be processed in 14; of those that fail, for those that can none-the-less be processed manually and are worth so-doing  
 

SPIRE-P

Revision 162015-09-16 - IvanV

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 75 to 75
 
Anomalous data correction Isolated cases of failed processing due to some on-board anomaly Yes - level2 equivalent FITS files No Yes - postcards Yes  
Line list catalogue Spectral Feature catalogue (see also this page ) No Yes - one file per obsid, and one concatenated file No ??  
Convolved spectral cubes Apply gridding algorithm to the spectral 2d pre-cubes Yes - level2 equivalent products No No No
Added:
>
>
Spectral scans Un-averaged level-2 spectra Yes - level2 equivalent products No No No
 

Revision 152015-09-16 - IvanV

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 74 to 74
 
Extended background correction Extended background correction for point sources Yes - level2 equivalent FITS files No Yes - postcards Yes  
Anomalous data correction Isolated cases of failed processing due to some on-board anomaly Yes - level2 equivalent FITS files No Yes - postcards Yes  
Line list catalogue Spectral Feature catalogue (see also this page ) No Yes - one file per obsid, and one concatenated file No ??  
Changed:
<
<
>
>
Convolved spectral cubes Apply gridding algorithm to the spectral 2d pre-cubes Yes - level2 equivalent products No No No
 

Revision 142015-09-16 - KatrinaExter

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 49 to 49
 
Unchopped observations: transients - case 1 Baseline-subtracted unchopped spectra Yes - level2/2.5 equivalent FITS files No Yes   Desirable, if case 2 is not possible this becomes Highly desirable  
Unchopped observations: transients - case 2 Improved transient correction and background subtraction Yes - level2/2.5 equivalent FITS files No Yes To be proposed for 14.1 SPG Highly desirable, if it is possible to do this is preferred over case 1  
Identifying background contamination ChopNod and Unchopped, Line (and continuum?) Yes - level2/2.5 equivalent FITS files No Yes   HPDP, but depends on how possible it will be to identify these and how many we find  
Added:
>
>
Point-source calibrated data - case 2 products processed through the dedicated pointing offset correction script for sources consistent with being centred point source Yes - level2/2.5 equivalent FITS files No Yes   straightforward to do once the pipeline script has been fully validated and sources identified; validation is on-going but not guaranteed  
 
Line list catalogue Spectral Feature catalogue No Yes - one file per obsid, and one concatenated file No   HPDP: dependent on available human resources, would be completely HSC to decide we want to do this as is not on ICC plans  

These should be SPG:

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Notes Prototype delivery
Deleted:
<
<
Point-source calibrated data - case 2 SPG products processed through the dedicated pointing offset correction script for sources consistent with being centred point source Yes - level2/2.5 equivalent FITS files No Yes   SPG: straightforward to do once the pipeline script has been fully validated and sources identified; validation is on-going  
 
Point-source calibrated data - case 3 SPG products processed through the standard pipeline script for sources consistent with being centred point sources Yes - level2/2.5 equivalent FITS files No Yes   SPG  
Long range spectro flat-fielding - main Including flatfielding for range scans in the SPG Yes - level2/2.5 equivalent FITS files No Yes   SPG  
Long range spectro flat-fielding - potential Correcting the few cases of bad flatfielding (probably unnecessary) Yes - level2/2.5 equivalent FITS files No Yes   HPDP or SPG, depends on whether/how many of these will be present after the final SPG  

Revision 132015-09-15 - ElenaPuga

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 43 to 43
 In order of priority: these first are HPDPs

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Notes Prototype delivery
Deleted:
<
<
High flux source data Very bright sources ought to be reduced with a difference pipeline script (to account for non-linearity/self-curing) Yes - level2/2.5 equivalent FITS files No Yes   Some level of manual reduction may be necessary, but are probably not many such observations  
Unchopped observations: response drifts - case 1 Unchopped range scans severely affected by response drift (e.g. after high glitch rate or after high-flux source observation) Yes - level2/2.5 equivalent FITS files No Yes   Highly desirable  
Unchopped observations: transients - case 1 Baseline-subtracted unchopped spectra Yes - level2/2.5 equivalent FITS files No Yes   Desirable, if case 2 is not possible this becomes Highly desirable  
Unchopped observations: transients - case 2 Improved transient correction and background subtraction Yes - level2/2.5 equivalent FITS files No Yes   Highly desirable, if it is possible to do this is preferred over case 1  
 
Point-source calibrated data - case 1 For off-centred sources that are likely to be point sources: SPG products processed through the standard pipeline script with non-standard extraction of final PS spectrum Yes - level2/2.5 equivalent FITS files No Yes   A useful possibility, feasibility will be more clear once QC work on off-centred observations and identification of likely centred point sources has been done  
Added:
>
>
High flux source data Very bright sources ought to be reduced with a difference pipeline script (to account for non-linearity/self-curing) Yes - level2/2.5 equivalent FITS files No Yes To be proposed for 14.1 SPG Some level of manual reduction may be necessary, but are probably not many such observations  
Unchopped observations: response drifts - case 1 Unchopped range scans severely affected by response drift (e.g. after high glitch rate or after high-flux source observation) Yes - level2/2.5 equivalent FITS files No Yes To be proposed for 14.1 SPG Highly desirable  
Unchopped observations: transients - case 1 Baseline-subtracted unchopped spectra Yes - level2/2.5 equivalent FITS files No Yes   Desirable, if case 2 is not possible this becomes Highly desirable  
Unchopped observations: transients - case 2 Improved transient correction and background subtraction Yes - level2/2.5 equivalent FITS files No Yes To be proposed for 14.1 SPG Highly desirable, if it is possible to do this is preferred over case 1  
Identifying background contamination ChopNod and Unchopped, Line (and continuum?) Yes - level2/2.5 equivalent FITS files No Yes   HPDP, but depends on how possible it will be to identify these and how many we find  
Line list catalogue Spectral Feature catalogue No Yes - one file per obsid, and one concatenated file No   HPDP: dependent on available human resources, would be completely HSC to decide we want to do this as is not on ICC plans  
 
Changed:
<
<		In order of priority: these should be SPG or are of lower priority HPDPs
>
>		These should be SPG:
 
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Notes Prototype delivery
Changed:
<
<
Point-source calibrated data - case 2 SPG products processed through the dedicated point-source correction script for sources consistent with being centred point source Yes - level2/2.5 equivalent FITS files No Yes   SPG: straightforward to do once the pipeline script has been fully validated and sources identified; validation is on-going  
>
>
Point-source calibrated data - case 2 SPG products processed through the dedicated pointing offset correction script for sources consistent with being centred point source Yes - level2/2.5 equivalent FITS files No Yes   SPG: straightforward to do once the pipeline script has been fully validated and sources identified; validation is on-going  
 
Point-source calibrated data - case 3 SPG products processed through the standard pipeline script for sources consistent with being centred point sources Yes - level2/2.5 equivalent FITS files No Yes   SPG  
Long range spectro flat-fielding - main Including flatfielding for range scans in the SPG Yes - level2/2.5 equivalent FITS files No Yes   SPG  
Changed:
<
<
Long range spectro flat-fielding - potential Correcting the few cases of bad flatfielding Yes - level2/2.5 equivalent FITS files No Yes   HPDP or SPG, depends on whether/how many of these will be present after the final SPG  
>
>
Long range spectro flat-fielding - potential Correcting the few cases of bad flatfielding (probably unnecessary) Yes - level2/2.5 equivalent FITS files No Yes   HPDP or SPG, depends on whether/how many of these will be present after the final SPG  
 
Red leak spectral range Run the specific pipeline that deals with red leak region (CalBlock+RSRF) and provide data stand-alone Yes - level2 equivalent FITS files No Yes   SPG  
Repair of failed observations Re-run pipeline on observations set to failed Yes - level2/2.5 equivalent FITS files No Yes   Observations that are classed as failed but which will now run in SPG are easy to do and could become standard SPG, for others that require manual reduction they could also become the standard SPG, probably not difficult  
Deleted:
<
<
Removing background contamination Line (and continuum?) Yes - level2/2.5 equivalent FITS files No Yes   HPDP, but depends on how possible it will be to identify these and how many we find, would be manual reduction and differently for chopNod and unchopped  
Line list catalogue Spectral Feature catalogue No Yes - one file per obsid, and one concatenated file No   HPDP: dependent on available human resources, would be completely HSC to decide we want to do this as is not on ICC plans  
 

SPIRE-P

Revision 122015-07-07 - KatrinaExter

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 47 to 47
 
Unchopped observations: response drifts - case 1 Unchopped range scans severely affected by response drift (e.g. after high glitch rate or after high-flux source observation) Yes - level2/2.5 equivalent FITS files No Yes   Highly desirable  
Unchopped observations: transients - case 1 Baseline-subtracted unchopped spectra Yes - level2/2.5 equivalent FITS files No Yes   Desirable, if case 2 is not possible this becomes Highly desirable  
Unchopped observations: transients - case 2 Improved transient correction and background subtraction Yes - level2/2.5 equivalent FITS files No Yes   Highly desirable, if it is possible to do this is preferred over case 1  
Changed:
<
<
Point-source calibrated data - case 1 SPG products processed through the standard pipeline script for cases of off-centred point sources Yes - level2/2.5 equivalent FITS files No Yes   A possibility, difficult to identify, requires production of the uneven illumination correction task (which is on-going work)  
>
>
Point-source calibrated data - case 1 For off-centred sources that are likely to be point sources: SPG products processed through the standard pipeline script with non-standard extraction of final PS spectrum Yes - level2/2.5 equivalent FITS files No Yes   A useful possibility, feasibility will be more clear once QC work on off-centred observations and identification of likely centred point sources has been done  
  In order of priority: these should be SPG or are of lower priority HPDPs

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Notes Prototype delivery
Changed:
<
<
Point-source calibrated data - case 2 SPG products processed through the dedicated point-source correction script for cases of centred point source Yes - level2/2.5 equivalent FITS files No Yes   SPG: straightforward to do once the pipeline script has been fully validated and sources identified; validation is on-going  
Point-source calibrated data - case 3 SPG products processed through the standard pipeline script for cases of centred point sources Yes - level2/2.5 equivalent FITS files No Yes   SPG  
>
>
Point-source calibrated data - case 2 SPG products processed through the dedicated point-source correction script for sources consistent with being centred point source Yes - level2/2.5 equivalent FITS files No Yes   SPG: straightforward to do once the pipeline script has been fully validated and sources identified; validation is on-going  
Point-source calibrated data - case 3 SPG products processed through the standard pipeline script for sources consistent with being centred point sources Yes - level2/2.5 equivalent FITS files No Yes   SPG  
 
Long range spectro flat-fielding - main Including flatfielding for range scans in the SPG Yes - level2/2.5 equivalent FITS files No Yes   SPG  
Long range spectro flat-fielding - potential Correcting the few cases of bad flatfielding Yes - level2/2.5 equivalent FITS files No Yes   HPDP or SPG, depends on whether/how many of these will be present after the final SPG  
Red leak spectral range Run the specific pipeline that deals with red leak region (CalBlock+RSRF) and provide data stand-alone Yes - level2 equivalent FITS files No Yes   SPG  

Revision 112015-06-24 - KatrinaExter

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 40 to 40
 

PACS-S

Added:
>
>		In order of priority: these first are HPDPs

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Notes Prototype delivery
High flux source data Very bright sources ought to be reduced with a difference pipeline script (to account for non-linearity/self-curing) Yes - level2/2.5 equivalent FITS files No Yes   Some level of manual reduction may be necessary, but are probably not many such observations  
Unchopped observations: response drifts - case 1 Unchopped range scans severely affected by response drift (e.g. after high glitch rate or after high-flux source observation) Yes - level2/2.5 equivalent FITS files No Yes   Highly desirable  
Unchopped observations: transients - case 1 Baseline-subtracted unchopped spectra Yes - level2/2.5 equivalent FITS files No Yes   Desirable, if case 2 is not possible this becomes Highly desirable  
Unchopped observations: transients - case 2 Improved transient correction and background subtraction Yes - level2/2.5 equivalent FITS files No Yes   Highly desirable, if it is possible to do this is preferred over case 1  
Point-source calibrated data - case 1 SPG products processed through the standard pipeline script for cases of off-centred point sources Yes - level2/2.5 equivalent FITS files No Yes   A possibility, difficult to identify, requires production of the uneven illumination correction task (which is on-going work)  

In order of priority: these should be SPG or are of lower priority HPDPs

 
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Notes Prototype delivery
Changed:
<
<
Point-source calibrated data - case 1 SPG products processed through the dedicated point-source correction script for cases of centred point source Yes - level2/2.5 equivalent FITS files No Yes   Straightforward to do once the pipeline script has been fully validated and sources identified; validation is on-going  
Point-source calibrated data - case 2 SPG products processed through the standard pipeline script for cases of centred point sources Yes - level2/2.5 equivalent FITS files No Yes   Aiming for SPG 14 in fact, so should be part of SPG rather than HPDP  
Point-source calibrated data - case 3 SPG products processed through the standard pipeline script for cases of off-centred point sources Yes - level2/2.5 equivalent FITS files No Yes   A possibility, difficult to identify, requires production of the uneven illumination correction task (which is on-going work)  
Unchopped observations - case 1 Baseline-subtracted unchopped spectra Yes - level2/2.5 equivalent FITS files No Yes   Desirable, if case 2 is not possible this becomes Highly desirable  
Unchopped observations - case 2 Improved transient correction and background subtraction Yes - level2/2.5 equivalent FITS files No Yes   Highly desirable, if it is possible to do this is preferred over case 1  
Long range spectro flat-fielding Manual correction of the flat-fielding drift Yes - level2/2.5 equivalent FITS files No Yes   Must-have, however the aim is to have this as part of the SPG, and only those that still have problems would require manual correction  
Red leak spectral range Run the specific pipeline that deals with red leak region (CalBlock+RSRF) and provide data stand-alone Yes - level2 equivalent FITS files No Yes   Must-have  
High flux source data Very bright sources ought to be reduced with a difference pipeline script Yes - level2/2.5 equivalent FITS files No Yes   Some level of manual reduction may be necessary, but probably are not many such observations  
Removing background contamination - case 1 chopNod mode Line (and continuum?) Yes - level2/2.5 equivalent FITS files No Yes   Nice to have, depends on how possible it will be to identify these, would be manual reduction  
Removing background contamination - case 2 unchopped mode Line (and continuum?) Yes - level2/2.5 equivalent FITS files No Yes   Nice to have, easier to identify that for chopNod, but would here also be manual reduction  
>
>
Point-source calibrated data - case 2 SPG products processed through the dedicated point-source correction script for cases of centred point source Yes - level2/2.5 equivalent FITS files No Yes   SPG: straightforward to do once the pipeline script has been fully validated and sources identified; validation is on-going  
Point-source calibrated data - case 3 SPG products processed through the standard pipeline script for cases of centred point sources Yes - level2/2.5 equivalent FITS files No Yes   SPG  
Long range spectro flat-fielding - main Including flatfielding for range scans in the SPG Yes - level2/2.5 equivalent FITS files No Yes   SPG  
Long range spectro flat-fielding - potential Correcting the few cases of bad flatfielding Yes - level2/2.5 equivalent FITS files No Yes   HPDP or SPG, depends on whether/how many of these will be present after the final SPG  
Red leak spectral range Run the specific pipeline that deals with red leak region (CalBlock+RSRF) and provide data stand-alone Yes - level2 equivalent FITS files No Yes   SPG  
 
Repair of failed observations Re-run pipeline on observations set to failed Yes - level2/2.5 equivalent FITS files No Yes   Observations that are classed as failed but which will now run in SPG are easy to do and could become standard SPG, for others that require manual reduction they could also become the standard SPG, probably not difficult  
Changed:
<
<
Different wavelength regridding Re-run SPG with different wavelength grid parameter so the fluxes are no dependent and the noise characteristics not changed Yes - level2/2.5 equivalent FITS files No Yes   Has not been discussed by ICC yet, so whether this replaces SPG or is provided as an HPDP or done at all, has not yet been discussed  
Line list catalogue Spectral Feature catalogue No Yes - one file per obsid, and one concatenated file No   dependent on available human resources  
>
>
Removing background contamination Line (and continuum?) Yes - level2/2.5 equivalent FITS files No Yes   HPDP, but depends on how possible it will be to identify these and how many we find, would be manual reduction and differently for chopNod and unchopped  
Line list catalogue Spectral Feature catalogue No Yes - one file per obsid, and one concatenated file No   HPDP: dependent on available human resources, would be completely HSC to decide we want to do this as is not on ICC plans  
 

SPIRE-P

Revision 102015-06-24 - KatrinaExter

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 49 to 49
 
Long range spectro flat-fielding Manual correction of the flat-fielding drift Yes - level2/2.5 equivalent FITS files No Yes   Must-have, however the aim is to have this as part of the SPG, and only those that still have problems would require manual correction  
Red leak spectral range Run the specific pipeline that deals with red leak region (CalBlock+RSRF) and provide data stand-alone Yes - level2 equivalent FITS files No Yes   Must-have  
High flux source data Very bright sources ought to be reduced with a difference pipeline script Yes - level2/2.5 equivalent FITS files No Yes   Some level of manual reduction may be necessary, but probably are not many such observations  
Changed:
<
<
Removing background contamination - case 1 chopNod mode Line and continuum Yes - level2/2.5 equivalent FITS files No Yes   Nice to have, depends on how possible it will be to identify these, would be manual reduction  
Removing background contamination - case 2 unchopped mode Line and continuum Yes - level2/2.5 equivalent FITS files No Yes   Nice to have, easier to identify that for chopNod, but would here also be manual reduction  
>
>
Removing background contamination - case 1 chopNod mode Line (and continuum?) Yes - level2/2.5 equivalent FITS files No Yes   Nice to have, depends on how possible it will be to identify these, would be manual reduction  
Removing background contamination - case 2 unchopped mode Line (and continuum?) Yes - level2/2.5 equivalent FITS files No Yes   Nice to have, easier to identify that for chopNod, but would here also be manual reduction  
 
Repair of failed observations Re-run pipeline on observations set to failed Yes - level2/2.5 equivalent FITS files No Yes   Observations that are classed as failed but which will now run in SPG are easy to do and could become standard SPG, for others that require manual reduction they could also become the standard SPG, probably not difficult  
Different wavelength regridding Re-run SPG with different wavelength grid parameter so the fluxes are no dependent and the noise characteristics not changed Yes - level2/2.5 equivalent FITS files No Yes   Has not been discussed by ICC yet, so whether this replaces SPG or is provided as an HPDP or done at all, has not yet been discussed  
Line list catalogue Spectral Feature catalogue No Yes - one file per obsid, and one concatenated file No   dependent on available human resources  

Revision 92015-06-23 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 23 to 23
 

HIFI

Changed:
<
<
HPDP name Description Data Catalogue (tables)
<-- -->
Sorted descending 		Images (PNGs, etc) Documentation (Readme, QSummary, etc)
Uncertainty tables Uncertainty tables as produced by the SPG No Yes - TableDataset as FITS files No Yes
Line list catalogue Identified Spectral Line catalogue No Yes - one file per obsid, and one concatenated file Yes - static (PNGs) and interactive (html) images Yes
Expert Reduced Spectral Scans Baseline-cleaned deconvolved spectral scans Yes - level2.5 equivalent FITS files No Yes - postcards Yes
Expert Reduced Spectral Maps Baseline-cleaned regridded cubes l scans Yes - level2.5 equivalent FITS files No Yes - postcards Yes
Isolated OFF spectra OFF spectra as produced by the SPG, potentially cleaned further Yes - level2.5 equivalent FITS files No Yes - postcards Yes
Non-averaged level2 for point mode Level2 data as produced by modified version of SPG Yes - level2.5 equivalent FITS files (multiple spectra) No Yes - postcards Yes
>
>
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
Expert Reduced Spectral Scans Baseline-cleaned deconvolved spectral scans Yes - level2.5 equivalent FITS files No Yes - postcards Yes  
Expert Reduced Spectral Maps Baseline-cleaned regridded cubes l scans Yes - level2.5 equivalent FITS files No Yes - postcards Yes  
Isolated OFF spectra OFF spectra as produced by the SPG, potentially cleaned further Yes - level2.5 equivalent FITS files No Yes - postcards Yes  
Non-averaged level2 for point mode Level2 data as produced by modified version of SPG Yes - level2.5 equivalent FITS files (multiple spectra) No Yes - postcards Yes  
Uncertainty tables Uncertainty tables as produced by the SPG No Yes - TableDataset as FITS files No Yes  
Line list catalogue Identified Spectral Line catalogue No Yes - one file per obsid, and one concatenated file Yes - static (PNGs) and interactive (html) images Yes  
 

PACS-P

Changed:
<
<
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
PACS Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note
Improved level3 maps Further combination of fields with Unimap and JScanam Yes No ?? ??
Additional level 2.5 maps Combination of fields currently unconsidered in the list for level2.5 pipeline Yes No ?? ??
>
>
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
PACS Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note  
Improved level3 maps Further combination of fields with Unimap and JScanam Yes No ?? ??  
Additional level 2.5 maps Combination of fields currently unconsidered in the list for level2.5 pipeline Yes No ?? ??  
 

PACS-S

Changed:
<
<
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) notes
Point-source calibrated data - case 1 SPG products processed through the dedicated point-source correction script for cases of centred point source Yes - level2/2.5 equivalent FITS files No Yes   Straightforward to do once the pipeline script has been fully validated and sources identified; validation is on-going
Point-source calibrated data - case 2 SPG products processed through the standard pipeline script for cases of centred point sources Yes - level2/2.5 equivalent FITS files No Yes   Aiming for SPG 14 in fact, so should be part of SPG rather than HPDP
Point-source calibrated data - case 3 SPG products processed through the standard pipeline script for cases of off-centred point sources Yes - level2/2.5 equivalent FITS files No Yes   A possibility, difficult to identify, requires production of the uneven illumination correction task (which is on-going work)
Unchopped observations - case 1 Baseline-subtracted unchopped spectra Yes - level2/2.5 equivalent FITS files No Yes   Desirable, if case 2 is not possible this becomes Highly desirable
Unchopped observations - case 2 Improved transient correction and background subtraction Yes - level2/2.5 equivalent FITS files No Yes   Highly desirable, if it is possible to do this is preferred over case 1
Long range spectro flat-fielding Manual correction of the flat-fielding drift Yes - level2/2.5 equivalent FITS files No Yes   Must-have, however the aim is to have this as part of the SPG, and only those that still have problems would require manual correction
Red leak spectral range Run the specific pipeline that deals with red leak region (CalBlock+RSRF) and provide data stand-alone Yes - level2 equivalent FITS files No Yes   Must-have
High flux source data Very bright sources ought to be reduced with a difference pipeline script Yes - level2/2.5 equivalent FITS files No Yes   Some level of manual reduction may be necessary, but probably are not many such observations
Removing background contamination - case 1 chopNod mode Line and continuum Yes - level2/2.5 equivalent FITS files No Yes   Nice to have, depends on how possible it will be to identify these, would be manual reduction
Removing background contamination - case 2 unchopped mode Line and continuum Yes - level2/2.5 equivalent FITS files No Yes   Nice to have, easier to identify that for chopNod, but would here also be manual reduction
Repair of failed observations Re-run pipeline on observations set to failed Yes - level2/2.5 equivalent FITS files No Yes   Observations that are classed as failed but which will now run in SPG are easy to do and could become standard SPG, for others that require manual reduction they could also become the standard SPG, probably not difficult
Different wavelength regridding Re-run SPG with different wavelength grid parameter so the fluxes are no dependent and the noise characteristics not changed Yes - level2/2.5 equivalent FITS files No Yes   Has not been discussed by ICC yet, so whether this replaces SPG or is provided as an HPDP or done at all, has not yet been discussed
Line list catalogue Spectral Feature catalogue No Yes - one file per obsid, and one concatenated file No   dependent on available human resources
>
>
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Notes Prototype delivery
Point-source calibrated data - case 1 SPG products processed through the dedicated point-source correction script for cases of centred point source Yes - level2/2.5 equivalent FITS files No Yes   Straightforward to do once the pipeline script has been fully validated and sources identified; validation is on-going  
Point-source calibrated data - case 2 SPG products processed through the standard pipeline script for cases of centred point sources Yes - level2/2.5 equivalent FITS files No Yes   Aiming for SPG 14 in fact, so should be part of SPG rather than HPDP  
Point-source calibrated data - case 3 SPG products processed through the standard pipeline script for cases of off-centred point sources Yes - level2/2.5 equivalent FITS files No Yes   A possibility, difficult to identify, requires production of the uneven illumination correction task (which is on-going work)  
Unchopped observations - case 1 Baseline-subtracted unchopped spectra Yes - level2/2.5 equivalent FITS files No Yes   Desirable, if case 2 is not possible this becomes Highly desirable  
Unchopped observations - case 2 Improved transient correction and background subtraction Yes - level2/2.5 equivalent FITS files No Yes   Highly desirable, if it is possible to do this is preferred over case 1  
Long range spectro flat-fielding Manual correction of the flat-fielding drift Yes - level2/2.5 equivalent FITS files No Yes   Must-have, however the aim is to have this as part of the SPG, and only those that still have problems would require manual correction  
Red leak spectral range Run the specific pipeline that deals with red leak region (CalBlock+RSRF) and provide data stand-alone Yes - level2 equivalent FITS files No Yes   Must-have  
High flux source data Very bright sources ought to be reduced with a difference pipeline script Yes - level2/2.5 equivalent FITS files No Yes   Some level of manual reduction may be necessary, but probably are not many such observations  
Removing background contamination - case 1 chopNod mode Line and continuum Yes - level2/2.5 equivalent FITS files No Yes   Nice to have, depends on how possible it will be to identify these, would be manual reduction  
Removing background contamination - case 2 unchopped mode Line and continuum Yes - level2/2.5 equivalent FITS files No Yes   Nice to have, easier to identify that for chopNod, but would here also be manual reduction  
Repair of failed observations Re-run pipeline on observations set to failed Yes - level2/2.5 equivalent FITS files No Yes   Observations that are classed as failed but which will now run in SPG are easy to do and could become standard SPG, for others that require manual reduction they could also become the standard SPG, probably not difficult  
Different wavelength regridding Re-run SPG with different wavelength grid parameter so the fluxes are no dependent and the noise characteristics not changed Yes - level2/2.5 equivalent FITS files No Yes   Has not been discussed by ICC yet, so whether this replaces SPG or is provided as an HPDP or done at all, has not yet been discussed  
Line list catalogue Spectral Feature catalogue No Yes - one file per obsid, and one concatenated file No   dependent on available human resources  
 

SPIRE-P

Changed:
<
<
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
SPIRE Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note
HiRes maps Reprocessing of SPIRE-P maps with the HiRes algorithm Yes - level 2 or level 2.5 equivalent FITS files No ?? ??
>
>
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
SPIRE Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note  
HiRes maps Reprocessing of SPIRE-P maps with the HiRes algorithm Yes - level 2 or level 2.5 equivalent FITS files No ?? ??  
 

SPIRE-S

Changed:
<
<
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
Semi-Extended calibration correction SECT correction for semi-compact sources Yes - level2 equivalent FITS files No Yes - postcards Yes
Extended background correction Extended background correction for point sources Yes - level2 equivalent FITS files No Yes - postcards Yes
Anomalous data correction Isolated cases of failed processing due to some on-board anomaly Yes - level2 equivalent FITS files No Yes - postcards Yes
Line list catalogue Spectral Feature catalogue (see also this page ) No Yes - one file per obsid, and one concatenated file No ??
>
>
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) Prototype delivery
Semi-Extended calibration correction SECT correction for semi-compact sources Yes - level2 equivalent FITS files No Yes - postcards Yes  
Extended background correction Extended background correction for point sources Yes - level2 equivalent FITS files No Yes - postcards Yes  
Anomalous data correction Isolated cases of failed processing due to some on-board anomaly Yes - level2 equivalent FITS files No Yes - postcards Yes  
Line list catalogue Spectral Feature catalogue (see also this page ) No Yes - one file per obsid, and one concatenated file No ??  
 

Revision 82015-06-23 - KatrinaExter

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 40 to 40
 

PACS-S

Changed:
<
<
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
Point-source calibrated data - case 1 SPG products processed through the dedicated point-source correction script for cases of centred point source Yes - level2.5 equivalent FITS files No ?? ??
Point-source calibrated data - case 2 SPG products processed through the dedicated point-source correction script for cases of non-centred point source Yes - level2.5 equivalent FITS files No ?? ??
Unchopped observations Baseline-subracted unchopped spectra Yes - level2.5 equivalent FITS files No ?? Must-have
Long range spectro flat-fielding Manual correction of the flat-fielding drift Yes - level2.5 equivalent FITS files No ?? Must-have
Red leak spectral range Run parallel pipeline that deals with red leak region (CalBlock+RSRF) and provide data stand-alone Yes - level2.5 equivalent FITS files No ?? Must-have
High flux source data ?? ?? No ?? ??
Repair of failed observations Re-run pipeline on observations set to failed Yes - level2.5 equivalent FITS files No ?? ??
Different wavelength regridding re-run SPG with different wavelength grid parameter for more rigourous noise computation Yes - level2.5 equivalent FITS files No ?? ??
Line list catalogue Spectral Feature catalogue No Yes - one file per obsid, and one concatenated file No ??
>
>
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc) notes
Point-source calibrated data - case 1 SPG products processed through the dedicated point-source correction script for cases of centred point source Yes - level2/2.5 equivalent FITS files No Yes   Straightforward to do once the pipeline script has been fully validated and sources identified; validation is on-going
Point-source calibrated data - case 2 SPG products processed through the standard pipeline script for cases of centred point sources Yes - level2/2.5 equivalent FITS files No Yes   Aiming for SPG 14 in fact, so should be part of SPG rather than HPDP
Point-source calibrated data - case 3 SPG products processed through the standard pipeline script for cases of off-centred point sources Yes - level2/2.5 equivalent FITS files No Yes   A possibility, difficult to identify, requires production of the uneven illumination correction task (which is on-going work)
Unchopped observations - case 1 Baseline-subtracted unchopped spectra Yes - level2/2.5 equivalent FITS files No Yes   Desirable, if case 2 is not possible this becomes Highly desirable
Unchopped observations - case 2 Improved transient correction and background subtraction Yes - level2/2.5 equivalent FITS files No Yes   Highly desirable, if it is possible to do this is preferred over case 1
Long range spectro flat-fielding Manual correction of the flat-fielding drift Yes - level2/2.5 equivalent FITS files No Yes   Must-have, however the aim is to have this as part of the SPG, and only those that still have problems would require manual correction
Red leak spectral range Run the specific pipeline that deals with red leak region (CalBlock+RSRF) and provide data stand-alone Yes - level2 equivalent FITS files No Yes   Must-have
High flux source data Very bright sources ought to be reduced with a difference pipeline script Yes - level2/2.5 equivalent FITS files No Yes   Some level of manual reduction may be necessary, but probably are not many such observations
Removing background contamination - case 1 chopNod mode Line and continuum Yes - level2/2.5 equivalent FITS files No Yes   Nice to have, depends on how possible it will be to identify these, would be manual reduction
Removing background contamination - case 2 unchopped mode Line and continuum Yes - level2/2.5 equivalent FITS files No Yes   Nice to have, easier to identify that for chopNod, but would here also be manual reduction
Repair of failed observations Re-run pipeline on observations set to failed Yes - level2/2.5 equivalent FITS files No Yes   Observations that are classed as failed but which will now run in SPG are easy to do and could become standard SPG, for others that require manual reduction they could also become the standard SPG, probably not difficult
Different wavelength regridding Re-run SPG with different wavelength grid parameter so the fluxes are no dependent and the noise characteristics not changed Yes - level2/2.5 equivalent FITS files No Yes   Has not been discussed by ICC yet, so whether this replaces SPG or is provided as an HPDP or done at all, has not yet been discussed
Line list catalogue Spectral Feature catalogue No Yes - one file per obsid, and one concatenated file No   dependent on available human resources
 

SPIRE-P

Revision 72015-06-22 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 14 to 14
 
    • Minutes of Meeting
    • Viewgraphs used during the meeting
    • The main action item resulting from the meeting is that each instrument representative shall come up with a list of use cases and requirement wish list applying to the HPDP they are supervising or acting as contact point for. These requirements will be compiled in the corresponding section below. Deadline: second half of September.
Changed:
<
<
  • Sept-2015: Follow-up meeting to review requirements and agree on a consolidated request to SAT to be conveyed by Eva Verdugo
>
>
  • Mid-September 2015: Follow-up meeting to review requirements and agree on a consolidated request to SAT to be conveyed by Eva Verdugo
 

HPDP plans per instruments

Revision 62015-06-19 - JeanMatagne

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 10 to 10
 

Meetings and action items

Changed:
<
<
  • 17-June-201HSC.5: Kick-off meeting with the ICS team and Eva Verdugo.
>
>
  • 17-June-2015: Kick-off meeting with the ICS team and Eva Verdugo.
 
Changed:
<
<
    • Viewgraphs used during the meeting
    • The main action item resulting from the meeting is that each instrument representative shall come up with a list of use cases and requirement wish list applying to the HSC.HPDP they are supervising or acting as contact point for. These requirements will be compiled in the corresponding section below. Deadline: second half of September.
  • Sept-201HSC.5: Follow-up meeting to review requirements and agree on a consolidated request to SAT to be conveyed by Eva Verdugo
>
>
    • Viewgraphs used during the meeting
    • The main action item resulting from the meeting is that each instrument representative shall come up with a list of use cases and requirement wish list applying to the HPDP they are supervising or acting as contact point for. These requirements will be compiled in the corresponding section below. Deadline: second half of September.
  • Sept-2015: Follow-up meeting to review requirements and agree on a consolidated request to SAT to be conveyed by Eva Verdugo
 
Changed:
<
<

HSC.HPDP plans per instruments

>
>

HPDP plans per instruments

  HPDPs have been identified by each instrument team, although it is likely that not all of them will be possible to prepared in certain cases. We indicate hereafter the HPDPs currently contemplated by each instruments, with some common delivery categories that will be considered or not.
Changed:
<
<

HSC.HIFI

>
>

HIFI

 
Changed:
<
<
HSC.HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
>
>
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
 
Expert Reduced Spectral Scans Baseline-cleaned deconvolved spectral scans Yes - level2.5 equivalent FITS files No Yes - postcards Yes
Expert Reduced Spectral Maps Baseline-cleaned regridded cubes l scans Yes - level2.5 equivalent FITS files No Yes - postcards Yes
Isolated OFF spectra OFF spectra as produced by the SPG, potentially cleaned further Yes - level2.5 equivalent FITS files No Yes - postcards Yes
Line: 31 to 31
 
Uncertainty tables Uncertainty tables as produced by the SPG No Yes - TableDataset as FITS files No Yes
Line list catalogue Identified Spectral Line catalogue No Yes - one file per obsid, and one concatenated file Yes - static (PNGs) and interactive (html) images Yes
Changed:
<
<

HSC.PACS-P

>
>

PACS-P

 
Changed:
<
<
HSC.HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
HSC.PACS Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note
>
>
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
PACS Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note
 
Improved level3 maps Further combination of fields with Unimap and JScanam Yes No ?? ??
Additional level 2.5 maps Combination of fields currently unconsidered in the list for level2.5 pipeline Yes No ?? ??
Changed:
<
<

HSC.PACS-S

>
>

PACS-S

 
Changed:
<
<
HSC.HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
>
>
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
 
Point-source calibrated data - case 1 SPG products processed through the dedicated point-source correction script for cases of centred point source Yes - level2.5 equivalent FITS files No ?? ??
Point-source calibrated data - case 2 SPG products processed through the dedicated point-source correction script for cases of non-centred point source Yes - level2.5 equivalent FITS files No ?? ??
Unchopped observations Baseline-subracted unchopped spectra Yes - level2.5 equivalent FITS files No ?? Must-have
Line: 51 to 51
 
Different wavelength regridding re-run SPG with different wavelength grid parameter for more rigourous noise computation Yes - level2.5 equivalent FITS files No ?? ??
Line list catalogue Spectral Feature catalogue No Yes - one file per obsid, and one concatenated file No ??
Changed:
<
<

HSC.SPIRE-P

>
>

SPIRE-P

 
Changed:
<
<
HSC.HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
HSC.SPIRE Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note
>
>
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
SPIRE Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note
 
HiRes maps Reprocessing of SPIRE-P maps with the HiRes algorithm Yes - level 2 or level 2.5 equivalent FITS files No ?? ??
Changed:
<
<

HSC.SPIRE-S

>
>

SPIRE-S

 
Changed:
<
<
HSC.HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
>
>
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
 
Semi-Extended calibration correction SECT correction for semi-compact sources Yes - level2 equivalent FITS files No Yes - postcards Yes
Extended background correction Extended background correction for point sources Yes - level2 equivalent FITS files No Yes - postcards Yes
Anomalous data correction Isolated cases of failed processing due to some on-board anomaly Yes - level2 equivalent FITS files No Yes - postcards Yes
Line: 76 to 76
 
Changed:
<
<		-- DavidTeyssier - 17 Jun 201HSC.5
>
>		-- DavidTeyssier - 17 Jun 2015
 
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1434526816" name="HPDP_meeting_ICS_16June2015.pdf" path="HPDP_meeting_ICS_16June2015.pdf" size="1454187" user="Main.DavidTeyssier" version="2"
Changed:
<
<
META FILEATTACHMENT attachment="HPDP_meeting_ICS_16June2015_MoM.pdf" attr="" comment="" date="1434722888" name="HPDP_meeting_ICS_16June2015_MoM.pdf" path="HPDP_meeting_ICS_16June2015_MoM.pdf" size="210371" stream="HPDP_meeting_ICS_16June2015_MoM.pdf" user="Main.DavidTeyssier" version="2"
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1434722888" name="HPDP_meeting_ICS_16June2015_MoM.pdf" path="HPDP_meeting_ICS_16June2015_MoM.pdf" size="210371" user="Main.DavidTeyssier" version="2"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 52015-06-19 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="HSC.WebHome"

Herschel Highly Processed Data Product Page

Line: 79 to 79
 -- DavidTeyssier - 17 Jun 201HSC.5

META FILEATTACHMENT attr="" autoattached="1" comment="" date="1434526816" name="HPDP_meeting_ICS_16June2015.pdf" path="HPDP_meeting_ICS_16June2015.pdf" size="1454187" user="Main.DavidTeyssier" version="2"
Changed:
<
<
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1434719201" name="HPDP_meeting_ICS_16June2015_MoM.pdf" path="HPDP_meeting_ICS_16June2015_MoM.pdf" size="210478" user="Main.DavidTeyssier" version="1"
>
>
META FILEATTACHMENT attachment="HPDP_meeting_ICS_16June2015_MoM.pdf" attr="" comment="" date="1434722888" name="HPDP_meeting_ICS_16June2015_MoM.pdf" path="HPDP_meeting_ICS_16June2015_MoM.pdf" size="210371" stream="HPDP_meeting_ICS_16June2015_MoM.pdf" user="Main.DavidTeyssier" version="2"
 
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 42015-06-19 - JeanMatagne

Line: 1 to 1
Changed:
<
<
META TOPICPARENT name="WebHome"
>
>
META TOPICPARENT name="HSC.WebHome"
 

Herschel Highly Processed Data Product Page

Line: 10 to 10
 

Meetings and action items

Changed:
<
<
  • 17-June-2015: Kick-off meeting with the ICS team and Eva Verdugo.
>
>
  • 17-June-201HSC.5: Kick-off meeting with the ICS team and Eva Verdugo.
 
Changed:
<
<
    • Viewgraphs used during the meeting
    • The main action item resulting from the meeting is that each instrument representative shall come up with a list of use cases and requirement wish list applying to the HPDP they are supervising or acting as contact point for. These requirements will be compiled in the corresponding section below. Deadline: second half of September.
  • Sept-2015: Follow-up meeting to review requirements and agree on a consolidated request to SAT to be conveyed by Eva Verdugo
>
>
    • Viewgraphs used during the meeting
    • The main action item resulting from the meeting is that each instrument representative shall come up with a list of use cases and requirement wish list applying to the HSC.HPDP they are supervising or acting as contact point for. These requirements will be compiled in the corresponding section below. Deadline: second half of September.
  • Sept-201HSC.5: Follow-up meeting to review requirements and agree on a consolidated request to SAT to be conveyed by Eva Verdugo
 
Changed:
<
<

HPDP plans per instruments

>
>

HSC.HPDP plans per instruments

  HPDPs have been identified by each instrument team, although it is likely that not all of them will be possible to prepared in certain cases. We indicate hereafter the HPDPs currently contemplated by each instruments, with some common delivery categories that will be considered or not.
Changed:
<
<

HIFI

>
>

HSC.HIFI

 
Changed:
<
<
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
>
>
HSC.HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
 
Expert Reduced Spectral Scans Baseline-cleaned deconvolved spectral scans Yes - level2.5 equivalent FITS files No Yes - postcards Yes
Expert Reduced Spectral Maps Baseline-cleaned regridded cubes l scans Yes - level2.5 equivalent FITS files No Yes - postcards Yes
Isolated OFF spectra OFF spectra as produced by the SPG, potentially cleaned further Yes - level2.5 equivalent FITS files No Yes - postcards Yes
Line: 31 to 31
 
Uncertainty tables Uncertainty tables as produced by the SPG No Yes - TableDataset as FITS files No Yes
Line list catalogue Identified Spectral Line catalogue No Yes - one file per obsid, and one concatenated file Yes - static (PNGs) and interactive (html) images Yes
Changed:
<
<

PACS-P

>
>

HSC.PACS-P

 
Changed:
<
<
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
PACS Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note
>
>
HSC.HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
HSC.PACS Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note
 
Improved level3 maps Further combination of fields with Unimap and JScanam Yes No ?? ??
Additional level 2.5 maps Combination of fields currently unconsidered in the list for level2.5 pipeline Yes No ?? ??
Changed:
<
<

PACS-S

>
>

HSC.PACS-S

 
Changed:
<
<
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
>
>
HSC.HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
 
Point-source calibrated data - case 1 SPG products processed through the dedicated point-source correction script for cases of centred point source Yes - level2.5 equivalent FITS files No ?? ??
Point-source calibrated data - case 2 SPG products processed through the dedicated point-source correction script for cases of non-centred point source Yes - level2.5 equivalent FITS files No ?? ??
Unchopped observations Baseline-subracted unchopped spectra Yes - level2.5 equivalent FITS files No ?? Must-have
Line: 51 to 51
 
Different wavelength regridding re-run SPG with different wavelength grid parameter for more rigourous noise computation Yes - level2.5 equivalent FITS files No ?? ??
Line list catalogue Spectral Feature catalogue No Yes - one file per obsid, and one concatenated file No ??
Changed:
<
<

SPIRE-P

>
>

HSC.SPIRE-P

 
Changed:
<
<
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
SPIRE Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note
HiRes maps Reprocessing of SPIRE-P maps with the HiRes algorithm Yes - level 2 or level 2.5 equivalent FITS files No ?? ??
>
>
HSC.HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
HSC.SPIRE Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note
HiRes maps Reprocessing of SPIRE-P maps with the HiRes algorithm Yes - level 2 or level 2.5 equivalent FITS files No ?? ??
 
Changed:
<
<

SPIRE-S

>
>

HSC.SPIRE-S

 
Changed:
<
<
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
>
>
HSC.HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
 
Semi-Extended calibration correction SECT correction for semi-compact sources Yes - level2 equivalent FITS files No Yes - postcards Yes
Extended background correction Extended background correction for point sources Yes - level2 equivalent FITS files No Yes - postcards Yes
Anomalous data correction Isolated cases of failed processing due to some on-board anomaly Yes - level2 equivalent FITS files No Yes - postcards Yes
Line: 76 to 76
 
Changed:
<
<		-- DavidTeyssier - 17 Jun 2015
>
>		-- DavidTeyssier - 17 Jun 201HSC.5
 
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1434526816" name="HPDP_meeting_ICS_16June2015.pdf" path="HPDP_meeting_ICS_16June2015.pdf" size="1454187" user="Main.DavidTeyssier" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1434719201" name="HPDP_meeting_ICS_16June2015_MoM.pdf" path="HPDP_meeting_ICS_16June2015_MoM.pdf" size="210478" user="Main.DavidTeyssier" version="1"
Added:
>
>
META TOPICMOVED by="JeanMatagne" date="1434722312" from="HSC.HPDPDefinitionPage" to="Public.HPDPDefinitionPage"

Revision 32015-06-19 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Herschel Highly Processed Data Product Page

Line: 13 to 13
 
Changed:
<
<
    • The main action item resulting from the meeting is that each instrument representative shall come up with a list of use cases and requirement wish list applying to the HPDP they are supervising or acting as contact point for. These requirements will be compiled in the corresponding section below
  • TBD-TBD-2015: Follow-up meeting to review requirements and agree on a consolidated request to SAT to be conveyed by EvaVerdugo
>
>
    • The main action item resulting from the meeting is that each instrument representative shall come up with a list of use cases and requirement wish list applying to the HPDP they are supervising or acting as contact point for. These requirements will be compiled in the corresponding section below. Deadline: second half of September.
  • Sept-2015: Follow-up meeting to review requirements and agree on a consolidated request to SAT to be conveyed by Eva Verdugo
 

HPDP plans per instruments

Line: 36 to 36
 
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
PACS Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note
Improved level3 maps Further combination of fields with Unimap and JScanam Yes No ?? ??
Added:
>
>
Additional level 2.5 maps Combination of fields currently unconsidered in the list for level2.5 pipeline Yes No ?? ??
 

PACS-S

Line: 77 to 78
  -- DavidTeyssier - 17 Jun 2015
Changed:
<
<
META FILEATTACHMENT attachment="HPDP_meeting_ICS_16June2015.pdf" attr="" comment="" date="1434526816" name="HPDP_meeting_ICS_16June2015.pdf" path="HPDP_meeting_ICS_16June2015.pdf" size="1454187" stream="HPDP_meeting_ICS_16June2015.pdf" user="Main.DavidTeyssier" version="2"
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1434526816" name="HPDP_meeting_ICS_16June2015.pdf" path="HPDP_meeting_ICS_16June2015.pdf" size="1454187" user="Main.DavidTeyssier" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="" date="1434719201" name="HPDP_meeting_ICS_16June2015_MoM.pdf" path="HPDP_meeting_ICS_16June2015_MoM.pdf" size="210478" user="Main.DavidTeyssier" version="1"

Revision 22015-06-17 - DavidTeyssier

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Herschel Highly Processed Data Product Page

Line: 25 to 25
 
HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
Expert Reduced Spectral Scans Baseline-cleaned deconvolved spectral scans Yes - level2.5 equivalent FITS files No Yes - postcards Yes
Added:
>
>
Expert Reduced Spectral Maps Baseline-cleaned regridded cubes l scans Yes - level2.5 equivalent FITS files No Yes - postcards Yes
Isolated OFF spectra OFF spectra as produced by the SPG, potentially cleaned further Yes - level2.5 equivalent FITS files No Yes - postcards Yes
Non-averaged level2 for point mode Level2 data as produced by modified version of SPG Yes - level2.5 equivalent FITS files (multiple spectra) No Yes - postcards Yes
Uncertainty tables Uncertainty tables as produced by the SPG No Yes - TableDataset as FITS files No Yes
 
Line list catalogue Identified Spectral Line catalogue No Yes - one file per obsid, and one concatenated file Yes - static (PNGs) and interactive (html) images Yes

PACS-P

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
Changed:
<
<
PACS Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large file per photometer band, and/or a unique concatenated one No Yes - Global release note
>
>
PACS Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note
Improved level3 maps Further combination of fields with Unimap and JScanam Yes No ?? ??
 

PACS-S

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
Changed:
<
<
Point-source calibrated data SPG products processed through the dedicated point-source correction script Yes - level2.5 equivalent FITS files No ?? ??
>
>
Point-source calibrated data - case 1 SPG products processed through the dedicated point-source correction script for cases of centred point source Yes - level2.5 equivalent FITS files No ?? ??
Point-source calibrated data - case 2 SPG products processed through the dedicated point-source correction script for cases of non-centred point source Yes - level2.5 equivalent FITS files No ?? ??
Unchopped observations Baseline-subracted unchopped spectra Yes - level2.5 equivalent FITS files No ?? Must-have
Long range spectro flat-fielding Manual correction of the flat-fielding drift Yes - level2.5 equivalent FITS files No ?? Must-have
Red leak spectral range Run parallel pipeline that deals with red leak region (CalBlock+RSRF) and provide data stand-alone Yes - level2.5 equivalent FITS files No ?? Must-have
High flux source data ?? ?? No ?? ??
Repair of failed observations Re-run pipeline on observations set to failed Yes - level2.5 equivalent FITS files No ?? ??
Different wavelength regridding re-run SPG with different wavelength grid parameter for more rigourous noise computation Yes - level2.5 equivalent FITS files No ?? ??
Line list catalogue Spectral Feature catalogue No Yes - one file per obsid, and one concatenated file No ??
 

SPIRE-P

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
Changed:
<
<
SPIRE Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large file per photometer band, and/or a unique concatenated one No Yes - Global release note
>
>
SPIRE Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large csv file per photometer band, and/or a unique concatenated one No Yes - Global release note
HiRes maps Reprocessing of SPIRE-P maps with the HiRes algorithm Yes - level 2 or level 2.5 equivalent FITS files No ?? ??
 

SPIRE-S

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
Added:
>
>
Semi-Extended calibration correction SECT correction for semi-compact sources Yes - level2 equivalent FITS files No Yes - postcards Yes
Extended background correction Extended background correction for point sources Yes - level2 equivalent FITS files No Yes - postcards Yes
Anomalous data correction Isolated cases of failed processing due to some on-board anomaly Yes - level2 equivalent FITS files No Yes - postcards Yes
 
Line list catalogue Spectral Feature catalogue (see also this page ) No Yes - one file per obsid, and one concatenated file No ??

Revision 12015-06-17 - DavidTeyssier

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="WebHome"

Herschel Highly Processed Data Product Page

Introduction

This page compiles the plans and needs to prepare and serve Highly Processed Data Products (HPDPs) to the Scientific Communities. There are two main ways considered to fulfil this: a web page, similar to that used for User Provided Data Products (UPDP, see also http://www.cosmos.esa.int/web/herschel/user-provided-data-products), and the HSA User Interface (HUI). The following focusses mostly on the latter.

Meetings and action items

  • 17-June-2015: Kick-off meeting with the ICS team and EvaVerdugo.
    • Minutes of Meeting
    • Viewgraphs used during the meeting
    • The main action item resulting from the meeting is that each instrument representative shall come up with a list of use cases and requirement wish list applying to the HPDP they are supervising or acting as contact point for. These requirements will be compiled in the corresponding section below
  • TBD-TBD-2015: Follow-up meeting to review requirements and agree on a consolidated request to SAT to be conveyed by EvaVerdugo

HPDP plans per instruments

HPDPs have been identified by each instrument team, although it is likely that not all of them will be possible to prepared in certain cases. We indicate hereafter the HPDPs currently contemplated by each instruments, with some common delivery categories that will be considered or not.

HIFI

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
Expert Reduced Spectral Scans Baseline-cleaned deconvolved spectral scans Yes - level2.5 equivalent FITS files No Yes - postcards Yes
Line list catalogue Identified Spectral Line catalogue No Yes - one file per obsid, and one concatenated file Yes - static (PNGs) and interactive (html) images Yes

PACS-P

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
PACS Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large file per photometer band, and/or a unique concatenated one No Yes - Global release note

PACS-S

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
Point-source calibrated data SPG products processed through the dedicated point-source correction script Yes - level2.5 equivalent FITS files No ?? ??

SPIRE-P

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
SPIRE Point Source Catalogue Point source catalogue as extracted from the photometer data No Yes - typically one large file per photometer band, and/or a unique concatenated one No Yes - Global release note

SPIRE-S

HPDP name Description Data Catalogue (tables) Images (PNGs, etc) Documentation (Readme, QSummary, etc)
Line list catalogue Spectral Feature catalogue (see also this page ) No Yes - one file per obsid, and one concatenated file No ??

HUI requirements

We provide here the HUI requirements needed to support the storage, queries and access to the various HPDPs.

TBD

-- DavidTeyssier - 17 Jun 2015

META FILEATTACHMENT attachment="HPDP_meeting_ICS_16June2015.pdf" attr="" comment="" date="1434526816" name="HPDP_meeting_ICS_16June2015.pdf" path="HPDP_meeting_ICS_16June2015.pdf" size="1454187" stream="HPDP_meeting_ICS_16June2015.pdf" user="Main.DavidTeyssier" version="2"
View topic | History: r74 < r73 < r72 < r71 | More topic actions...
 
This site is powered by the TWiki collaboration platform Powered by Perl