Difference: ContributorsGuide (1 vs. 6)

Revision 62012-05-23 - DavideRizzo

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Herschel Tools Contributor's Guide

Line: 49 to 49
 The responsibility to maintain and update the contributed tools lays on the contributors. The Herschel Science Centre will provide no support for problems with these tools, or for problems arising in HIPE because of these tools. Users will download and install the tools at their own risk. This disclaimer will be clearly displayed on any download page provided by the Herschel Science Centre.

The Herschel Science Centre may decide to integrate tools of outstanding quality into the official HIPE distribution. In this case, support to users will be provided.

Deleted:
<
<

<-- COMMENT BOX CODE - DO NOT EDIT -->
blog comments powered by Disqus
<-- END OF COMMENT BOX CODE -->

 \ No newline at end of file

Revision 52012-05-14 - DavideRizzo

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Herschel Tools Contributor's Guide

Line: 49 to 49
 The responsibility to maintain and update the contributed tools lays on the contributors. The Herschel Science Centre will provide no support for problems with these tools, or for problems arising in HIPE because of these tools. Users will download and install the tools at their own risk. This disclaimer will be clearly displayed on any download page provided by the Herschel Science Centre.

The Herschel Science Centre may decide to integrate tools of outstanding quality into the official HIPE distribution. In this case, support to users will be provided.

Added:
>
>

<-- COMMENT BOX CODE - DO NOT EDIT -->
blog comments powered by Disqus
<-- END OF COMMENT BOX CODE -->

 \ No newline at end of file

Revision 42012-03-14 - DavideRizzo

Line: 1 to 1
 
META TOPICPARENT name="WebHome"
Changed:
<
<

Herschel Data Products and Tools Contributor's Guide

>
>

Herschel Tools Contributor's Guide

 
Changed:
<
<
Issue 1.5, 10 February 2011
>
>
Issue 2.0, 14 March 2012
 
PDF Version Click here

Changed:
<
<
This document describes how you can provide user reduced data products, catalogues and data reduction tools to the Herschel Science Centre, to make them available to the astronomical community.
>
>
This document describes how you can provide data reduction tools to the Herschel Science Centre, to make them available to the astronomical community.
 
Changed:
<
<

Quick steps

>
>
You are welcome to share your scripts and tools, implemented in Jython or Java, with other observatory users. Submitted tools must be accompanied by documentation that describes their purpose and algorithms, and how to install and use them (see Tools documentation in this document).
 
Changed:
<
<
Read this section if you just want to know what you have to do to contribute your data.

  1. Create a directory in an FTP site. The name of the directory is not important, but it must contain three subdirectories: data, scripts and docs.
  2. In the data directory, put the reduced data products in FITS format. For more information about the format of the products, see Format of submitted products.
  3. In the scripts directory, put any custom scripts you have used to reduce your data. If you have developed more complex software tools to extend HIPE, you may also include them in this directory. See Contributing tools for more information on contributing scripts and other software.
  4. In the docs directory, put any instructions and documentation needed to understand the scripts you provide and reproduce your data. See Product documentation for the documentation to include with data products. See Tools documentation for the documentation to include with scripts and other software.
    Note that you do not need to include documentation in this directory if it is already available elsewhere. For example, if a script is already documented by code comments or in a refereed paper, just mention this without duplicating content.
  5. Contact the Herschel Science Centre Helpdesk to notify of the availability of the data. In response, the HSC Helpdesk will provide detailed instructions on how to proceed.

Contributors

Key Programmes Observers

If you have been awarded observation time through a Key Programme Announcement of Opportunity, you have followed in your proposals the guidelines specified in the Policies and Procedures document for the KP AO. In particular, this document states the following:

"Consortia proposing for Herschel observing time on the occasion of this AO need to demonstrate ability and commitment to perform data reduction and make products (at a 'publishable' level of quality) and related tools publicly available through the HSC at the end of the proprietary period.

Besides fulfilling their scientific goals, Key Programme consortia must provide data products which will:

  1. allow for early science exploitation by the community, based on the data products themselves;
  2. provide the community with information usable as the basis of follow-up proposals for Herschel observing time during the mission.

The Key Programme consortia are also required to share the methods employed to generate the data products provided to the HSC. It is possible that these methods could be incorporated into or adapted for use in the standard HSC data processing software for public release."

Consequently, in order to enable follow up observations to be proposed by the astronomical community, each KP consortium included in their proposal a description of the data products and methods that they planned to provide at the end of the proprietary period. The approval of the proposal by HOTAC and the allocation of observing time implicitly assume the fulfilment by the KP consortia of the committed deliveries.

Regular Programme Observers

If you have been awarded observation time through a Regular Programme Announcement of Opportunity, you are encouraged to contribute, on voluntary basis, with highly processed data products and software tools to the Observatory, and to make use of the HSC facilities to widespread their results to the astronomical community. Contributed products shall be supported by the corresponding paper in a refereed publication. Deliveries should follow the guidelines and procedures specified in this document.

Contributing data products

Herschel data products are classified based on their processing level (from raw data to highly processed) and their contents. For an overview see the HSC Data Products pages and for a more detailed description please refer to the Products Definition Document. Generally, it is expected that contributed products will be advanced processed data of the following type:

  • Highly processed level-2 products
  • Level-3 products, per AOR or combination of AORs (e.g., large maps, spectral surveys)
  • Catalogues (e.g., astronomical sources catalogues, spectral line lists)
  • Ancillary data (e.g., model SEDs, reference spectral line lists)

Highly processed data products can be made available to the HSC by providing the address of the FTP site where the products are located. The address will be included in the Herschel Science Centre public Web pages, and will be associated with the corresponding proposal and observations in the HSA.

You should also provide any other web addresses from which data products are accessible (for instance, CDS for catalogues).

In the future you will be able to submit your data directly for ingestion in the Herschel Science Archive as "Highly Processed Data Products", so that they are integrated in the HSA facilities and become part of the Virtual Observatory.

Product documentation

Contributed products must be accompanied by the following information in a separate file (text, PDF or other standard format):

  1. Proposal Identification (as provided by HSpot and in the product metadata keyword "proposal")
  2. List of products contained in the delivery. For each product please include the following:
    1. Observation identifiers ("obsid" values) of the observations that have been used to create the product.
    2. Product type (image, cube, spectrum, line list, catalogue and so on)
    3. Product format (HCSS FITS, CLASS FITS, standard FITS).
  3. The address of the FTP site where the products are located.
  4. References to refereed papers.
  5. For KP associated products, a brief description of the algorithms, methods and processing steps involved in the creation of the product (see below).

Methods used in the generation of the scientific products shall be described indicating:

  1. If SPG products have been used, the SPG version and the maximum SPG level from which the interactive/user analysis has started (see HSC Data Products pages) Example: SPG v1.0, level-1.
  2. A detailed description of the data reduction steps carried out on top of the standard product to generate the final product.
    • For data reduced in the Herschel Interactive Processing Environment (HIPE), the HIPE version number must be provided, and a description of the user input for each step.
    • For steps that do not involve official HIPE routines, you should provide a detailed description of the algorithms used. As stated in the KP Policies and Procedures document, the HSC may decide to include or implement these methods in the standard HSC data processing software for public release.

Format of submitted products

The format of products that are submitted to the Herschel Science Centre should conform to the following specifications:

  1. Products shall be provided as FITS files. The structure and keyword definition shall conform to the FITS standard as defined by version 3.0 (2008).
  2. The product metadata must contain, at least, the compulsory keywords listed in Table 1, Contributed products compulsory metadata.

Table 1. Contributed products compulsory metadata

FITS keyword Type Description Herschel DP keyword name Notes
DATE String Date of product creation creationDate  
CREATOR String The name of the software that created the product creator  
DESC String Full name of product description  
OBSERVER String Name of observer observer As given in SPG products
PROPOSAL String Proposal name proposal As given in SPG products
TYPE String Product type identification type It should be equal to "User Contribution"
OBS_ID Long integer Observation identification obsid Mandatory metadata only for products associated with an individual observation

To ensure consistency with HIPE and to maximise commonality, please deliver your products following the standard Herschel product definitions. In particular, you should implement the following:

Redelivery of products

As time advances in the mission, our knowledge of the satellite and instrument behaviour will increase, and this will be reflected in improvements of the data processing software and calibration. For the benefit of the mission's legacy, you should consider re-processing your observations with upgraded versions of data reduction software and calibration, and re-delivering them to the HSC.

Contributing tools

You are welcome to share your scripts and tools, implemented in Jython or Java, with other observatory users. Submitted tools must be accompanied by documentation that describes their purpose and algorithms, and how to install and use them (see Tools documentation).

Contributing scripts

>
>

Contributing scripts

  Contributing single Jython scripts, or collections of related scripts, is the simplest way to share code with the community. Such scripts are intended to be run from the Editor view of HIPE, much like routines from the IDL Astronomy User's Library are run from the IDL environment.

You can also contribute scripts for other environments, such as IDL or CLASS, if these played a role in the reduction of your data.

Changed:
<
<

Contributing Jython or Java plugins

>
>

Contributing Jython or Java plug-ins

With plug-in we denote any external component that can be integrated into HIPE. This can be a task (a special type of function, usually for data processing, such as fitFringe), a view (a window within HIPE, such as the Editor view) or a perspective (a set of windows within HIPE, such as the Workbench perspective).

 
Changed:
<
<
With plugin we denote any external component that can be integrated into HIPE. This can be a task (a special type of function, usually for data processing, such as fitFringe), a view (a window within HIPE, such as the Editor view) or a perspective (a set of windows within HIPE, such as the Work Bench perspective).
>
>
The Herschel Science Centre has developed a framework to install, update and uninstall user-contributed plug-ins within HIPE:
 
Changed:
<
<
The Herschel Science Centre has developed a framework to install, update and uninstall user-contributed plugins within HIPE:
>
>
  • To check the list of available plug-ins, see this page.
  • To learn how to install and uninstall plug-ins within HIPE, see the user's manual.
  • To learn how to contribute plug-ins for HIPE, see the author's manual.
 
Changed:
<
<
>
>
For an introduction to all the ways in which you can contribute to HIPE, with an emphasis on developing and modifying plug-ins, see the Contributing to HIPE tutorials on the HIPE Community website.
 
Changed:
<
<
Note that you can also contribute tasks as Jython scripts: a description of how to develop Jython-based tasks is given in Chapter 4 of the Scripting and Data Mining guide, shipped with HIPE.
>
>
For a more in-depth description of HIPE internals and more advanced development topics, see the Contributing to HIPE pages on the official Herschel Science Centre wiki.
 
Changed:
<
<

Tools documentation

>
>

Tools documentation

 
Changed:
<
<
The delivery should include documentation containing:
>
>
The delivery should include documentation containing the following:
 
  • Name of author or authors, affiliation, email.
  • Associated Herschel proposal identifier.
  • List of HIPE versions with which the module is compatible.
  • Purpose of the software.
  • Description of the algorithms.
Changed:
<
<
  • Short user's manual. If you provided a HIPE component as a JAR file, please include instructions for users to integrate it into their HIPE installation.
>
>
  • Short user's manual.
 
  • Description of the Highly Processed Data Products derived with the modules, if applicable.
Changed:
<
<

Maintenance

>
>

Maintenance

  The responsibility to maintain and update the contributed tools lays on the contributors. The Herschel Science Centre will provide no support for problems with these tools, or for problems arising in HIPE because of these tools. Users will download and install the tools at their own risk. This disclaimer will be clearly displayed on any download page provided by the Herschel Science Centre.

The Herschel Science Centre may decide to integrate tools of outstanding quality into the official HIPE distribution. In this case, support to users will be provided.

Deleted:
<
<

Acronyms

AO Announcement of Opportunity
AOR Astronomical Observation Request
DP Data Processing
HIPE Herschel Interactive Processing Environment
HOTAC Herschel Observing Time Allocation Committee
HSA Herschel Science Archive
HSC Herschel Science Centre
KP Key Programmes
SED Spectral Energy Distribution
SPG Systematic Product Generation

Revision 32011-02-10 - DavideRizzo

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Herschel Data Products and Tools Contributor's Guide

Changed:
<
<
Issue 1.4, 26 November 2010
>
>
Issue 1.5, 10 February 2011
 
PDF Version Click here
Line: 62 to 62
 Herschel data products are classified based on their processing level (from raw data to highly processed) and their contents. For an overview see the HSC Data Products pages and for a more detailed
Changed:
<
<
description please refer to the Products Definition Document. Generally,
>
>
description please refer to the Products Definition Document. Generally,
  it is expected that contributed products will be advanced processed data of the following type:
Deleted:
<
<

 
  • Highly processed level-2 products
  • Level-3 products, per AOR or combination of AORs (e.g., large maps, spectral surveys)
  • Catalogues (e.g., astronomical sources catalogues, spectral line lists)
  • Ancillary data (e.g., model SEDs, reference spectral line lists)
Deleted:
<
<

 Highly processed data products can be made available to the HSC by providing the address of the FTP site where the products are located. The address will be included in
Line: 108 to 99
 
  1. References to refereed papers.
  2. For KP associated products, a brief description of the algorithms, methods and processing steps involved in the creation of the product (see below).
Deleted:
<
<

 Methods used in the generation of the scientific products shall be described indicating:
Deleted:
<
<

 
  1. If SPG products have been used, the SPG version and the maximum SPG level from which the interactive/user analysis has started (see HSC Data Products pages) Example: SPG v1.0, level-1.
  2. A detailed description of the data reduction steps carried out on top of the standard product to generate the final product.
    • For data reduced in the Herschel Interactive Processing
Line: 147 to 128
  To ensure consistency with HIPE and to maximise commonality, please deliver your products following the standard Herschel product definitions. In particular, you should implement the following:
Changed:
<
<
  • Images following the structure of the SimpleImage Product (see section 2.4.4 of the Herschel Products Definitions Document)
  • Image cubes following the structure of the SimpleCube Product (see section 2.4.4 of the Herschel Products Definitions Document)
  • Metadata keywords as listed in Appendix A of the Herschel Products Definitions Document.
  • Spectral cubes following the structure of the SpectralSimpleCube Product (see section 2.4.4 of the Herschel Products Definitions Document)
  • Spectra following the structure of the Spectrum1d or Spectrum2d datasets as appropriate (see section 2.4.3 of the Herschel Products Definitions Document)
  • Spectral line list catalogues following the definition as given in section 6.3.1 of the Herschel Products Definitions Document
  • Source list catalogues following the definition as given in section 6.3.2 of the Herschel Products Definitions Document
>
>
 

Redelivery of products

Revision 22010-11-26 - DavideRizzo

Line: 1 to 1
 
META TOPICPARENT name="WebHome"

Herschel Data Products and Tools Contributor's Guide

Line: 14 to 14
  Read this section if you just want to know what you have to do to contribute your data.
Changed:
<
<
  1. Create a directory in an FTP site. The name of the directory is not important, but it must contain three subdirectories: data, scripts and docs.
  2. In the data directory, put the reduced data products in FITS format. For more information about the format of the products, see Section .
  3. In the scripts directory, put any custom scripts you have used to reduce your data. If you have developed more complex software tools to extend HIPE, you may also include them in this directory. See Section for more information on contributing scripts and other software.
  4. In the docs directory, put any instructions and documentation needed to understand the scripts you provide and reproduce your data. See Section for the documentation to include with data products. See Section for the documentation to include with scripts and other software.
  5. Note that you do not need to include documentation in this directory if it is already available elsewhere. For example, if a script is already documented by code comments or in a refereed paper, just mention this without duplicating content.
  6. Contact the Herschel Science Centre Helpdesk (http://herschel.esac.esa.int/esupport/) to notify of the availability of the data. In response, the HSC Helpdesk will provide detailed instructions on how to proceed.
>
>
  1. Create a directory in an FTP site. The name of the directory is not important, but it must contain three subdirectories: data, scripts and docs.
  2. In the data directory, put the reduced data products in FITS format. For more information about the format of the products, see Format of submitted products.
  3. In the scripts directory, put any custom scripts you have used to reduce your data. If you have developed more complex software tools to extend HIPE, you may also include them in this directory. See Contributing tools for more information on contributing scripts and other software.
  4. In the docs directory, put any instructions and documentation needed to understand the scripts you provide and reproduce your data. See Product documentation for the documentation to include with data products. See Tools documentation for the documentation to include with scripts and other software.
    Note that you do not need to include documentation in this directory if it is already available elsewhere. For example, if a script is already documented by code comments or in a refereed paper, just mention this without duplicating content.
  5. Contact the Herschel Science Centre Helpdesk to notify of the availability of the data. In response, the HSC Helpdesk will provide detailed instructions on how to proceed.
 

Contributors

Key Programmes Observers

Changed:
<
<
If you have been awarded observation time through a Key Programme Announcement of Opportunity, you have followed in your proposals the guidelines specified in the Policies and Procedures document for the KP AO. In particular, this document states:
>
>
If you have been awarded observation time through a Key Programme Announcement of Opportunity, you have followed in your proposals the guidelines specified in the Policies and Procedures document for the KP AO. In particular, this document states the following:
 
Changed:
<
<
"Consortia proposing for Herschel observing time on the occasion of this AO need to demonstrate ability and commitment to perform data reduction and make products (at a 'publishable' level of quality) and related tools publicly available through the HSC at the end of the proprietary period.
>
>
"Consortia proposing for Herschel observing time on the occasion of this AO need to demonstrate ability and commitment to perform data reduction and make products (at a 'publishable' level of quality) and related tools publicly available through the HSC at the end of the proprietary period.
 
Changed:
<
<
Besides fulfilling their scientific goals, Key Programme consortia must provide data products which will:
>
>
Besides fulfilling their scientific goals, Key Programme consortia must provide data products which will:
 
Changed:
<
<
  1. allow for early science exploitation by the community, based on the data products themselves;
  2. provide the community with information usable as the basis of follow-up proposals for Herschel observing time during the mission.
>
>
  1. allow for early science exploitation by the community, based on the data products themselves;
  2. provide the community with information usable as the basis of follow-up proposals for Herschel observing time during the mission.
 
Changed:
<
<
The Key Programme consortia are also required to share the methods employed to generate the data products provided to the HSC. It is possible that these methods could be incorporated into or adapted for use in the standard HSC data processing software for public release."
>
>
The Key Programme consortia are also required to share the methods employed to generate the data products provided to the HSC. It is possible that these methods could be incorporated into or adapted for use in the standard HSC data processing software for public release."
 

Line: 47 to 46
  implicitly assume the fulfilment by the KP consortia of the committed deliveries.
Deleted:
<
<

 

Regular Programme Observers

Deleted:
<
<

 If you have been awarded observation time through a Regular Programme Announcement of Opportunity, you are encouraged to contribute, on voluntary basis, with highly processed data products
Line: 67 to 57
  a refereed publication. Deliveries should follow the guidelines and procedures specified in this document.
Deleted:
<
<

 

Contributing data products

Deleted:
<
<

 Herschel data products are classified based on their processing level (from raw data to highly processed) and their contents. For an
Changed:
<
<
overview see the HSC Data Products pages and for a more detailed
>
>
overview see the HSC Data Products pages and for a more detailed
  description please refer to the Products Definition Document. Generally, it is expected that contributed products will be advanced processed data of the following type:
Line: 108 to 87
  associated with the corresponding proposal and observations in the HSA.
Deleted:
<
<

 You should also provide any other web addresses from which data products are accessible (for instance, CDS for catalogues).
Deleted:
<
<
 In the future you will be able to submit your data directly for ingestion in the Herschel Science Archive as "Highly Processed Data Products", so that they are integrated in the HSA facilities and become part of the
Line: 119 to 94
  are integrated in the HSA facilities and become part of the Virtual Observatory.
Deleted:
<
<

 

Product documentation

Deleted:
<
<

 Contributed products must be accompanied by the following information in a separate file (text, PDF or other standard format):
Deleted:
<
<

 
  1. Proposal Identification (as provided by HSpot and in the product metadata keyword "proposal")
  2. List of products contained in the delivery. For each product please include the following:
Changed:
<
<
    1. Observation identifiers ("obsid" values) of the observations that have been used to create the product.
    2. Product type (image, cube, spectrum, line list, catalogue and so on)
    3. Product format (HCSS FITS, CLASS FITS, standard FITS).
>
>
    1. Observation identifiers ("obsid" values) of the observations that have been used to create the product.
    2. Product type (image, cube, spectrum, line list, catalogue and so on)
    3. Product format (HCSS FITS, CLASS FITS, standard FITS).
 
  1. The address of the FTP site where the products are located.
  2. References to refereed papers.
  3. For KP associated products, a brief description of the algorithms, methods and processing steps involved in the creation of the product (see below).
Line: 155 to 120
 

Changed:
<
<
  1. If SPG products have been used, the SPG version and the maximum SPG level from which the interactive/user analysis has started (see HSC Data Products pages) Example: SPG v1.0, level-1.
>
>
  1. If SPG products have been used, the SPG version and the maximum SPG level from which the interactive/user analysis has started (see HSC Data Products pages) Example: SPG v1.0, level-1.
 
  1. A detailed description of the data reduction steps carried out on top of the standard product to generate the final product.
    • For data reduced in the Herschel Interactive Processing Environment (HIPE), the HIPE version number must be provided, and a description of the user input for each step.
Changed:
<
<
    • For steps that do not involve official HIPE routines, you should provide a detailed description of the algorithms used. As stated in the KP Policies and Procedures document, the HSC may decide to include or implement these methods in the standard HSC data processing software for public release.

>
>
    • For steps that do not involve official HIPE routines, you should provide a detailed description of the algorithms used. As stated in the KP Policies and Procedures document, the HSC may decide to include or implement these methods in the standard HSC data processing software for public release.
 

Format of submitted products

Added:
>
>
The format of products that are submitted to the Herschel Science Centre should conform to the following specifications:
 
Changed:
<
<

The format of products that are submitted to the Herschel Science Centre should conform to the following specifications:

  1. Products shall be provided as FITS files. The structure and keyword definition shall conform to the FITS standard as defined by version 3.0 (2008).
>
>
  1. Products shall be provided as FITS files. The structure and keyword definition shall conform to the FITS standard as defined by version 3.0 (2008).
 
  1. The product metadata must contain, at least, the compulsory keywords listed in Table 1, Contributed products compulsory metadata.
Changed:
<
<
Table 1. Contributed products compulsory metadata

>
>
Table 1. Contributed products compulsory metadata
 
FITS keyword Type Description Herschel DP keyword name Notes
DATE String Date of product creation creationDate  
Line: 222 to 161
 

Contributing tools

Changed:
<
<
You are welcome to share your scripts and tools, implemented in Jython or Java, with other observatory users. Submitted tools must be accompanied by documentation that describes their purpose and algorithms, and how to install and use them (see the section called “Documentation”).
>
>
You are welcome to share your scripts and tools, implemented in Jython or Java, with other observatory users. Submitted tools must be accompanied by documentation that describes their purpose and algorithms, and how to install and use them (see Tools documentation).
 

Contributing scripts

Changed:
<
<
Contributing single Jython scripts, or collections of related scripts, is the simplest way to share code with the community. Such scripts are intended to be run from the Editor view of HIPE, much like routines from the IDL Astronomy User's Library are run from the IDL environment.
>
>
Contributing single Jython scripts, or collections of related scripts, is the simplest way to share code with the community. Such scripts are intended to be run from the Editor view of HIPE, much like routines from the IDL Astronomy User's Library are run from the IDL environment.
  You can also contribute scripts for other environments, such as IDL or CLASS, if these played a role in the reduction of your data.

Contributing Jython or Java plugins

Changed:
<
<
With plugin we denote any external component that can be integrated into HIPE. This can be a task (a special type of function, usually for data processing, such as fitFringe), a view (a window within HIPE, such as the Editor view) or a perspective (a set of windows within HIPE, such as the Work Bench perspective).
>
>
With plugin we denote any external component that can be integrated into HIPE. This can be a task (a special type of function, usually for data processing, such as fitFringe), a view (a window within HIPE, such as the Editor view) or a perspective (a set of windows within HIPE, such as the Work Bench perspective).
  The Herschel Science Centre has developed a framework to install, update and uninstall user-contributed plugins within HIPE:
Line: 241 to 180
 
Changed:
<
<
Note that you can also contribute tasks as Jython scripts: a description of how to develop Jython-based tasks is given in Chapter 4 of the Scripting and Data Mining guide, shipped with HIPE.
>
>
Note that you can also contribute tasks as Jython scripts: a description of how to develop Jython-based tasks is given in Chapter 4 of the Scripting and Data Mining guide, shipped with HIPE.
 
Changed:
<
<

Documentation

>
>

Tools documentation

  The delivery should include documentation containing:

Revision 12010-11-26 - DavideRizzo

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="WebHome"

Herschel Data Products and Tools Contributor's Guide

Issue 1.4, 26 November 2010

PDF Version Click here

This document describes how you can provide user reduced data products, catalogues and data reduction tools to the Herschel Science Centre, to make them available to the astronomical community.

Quick steps

Read this section if you just want to know what you have to do to contribute your data.

  1. Create a directory in an FTP site. The name of the directory is not important, but it must contain three subdirectories: data, scripts and docs.
  2. In the data directory, put the reduced data products in FITS format. For more information about the format of the products, see Section .
  3. In the scripts directory, put any custom scripts you have used to reduce your data. If you have developed more complex software tools to extend HIPE, you may also include them in this directory. See Section for more information on contributing scripts and other software.
  4. In the docs directory, put any instructions and documentation needed to understand the scripts you provide and reproduce your data. See Section for the documentation to include with data products. See Section for the documentation to include with scripts and other software.
  5. Note that you do not need to include documentation in this directory if it is already available elsewhere. For example, if a script is already documented by code comments or in a refereed paper, just mention this without duplicating content.
  6. Contact the Herschel Science Centre Helpdesk (http://herschel.esac.esa.int/esupport/) to notify of the availability of the data. In response, the HSC Helpdesk will provide detailed instructions on how to proceed.

Contributors

Key Programmes Observers

If you have been awarded observation time through a Key Programme Announcement of Opportunity, you have followed in your proposals the guidelines specified in the Policies and Procedures document for the KP AO. In particular, this document states:

"Consortia proposing for Herschel observing time on the occasion of this AO need to demonstrate ability and commitment to perform data reduction and make products (at a 'publishable' level of quality) and related tools publicly available through the HSC at the end of the proprietary period.

Besides fulfilling their scientific goals, Key Programme consortia must provide data products which will:

  1. allow for early science exploitation by the community, based on the data products themselves;
  2. provide the community with information usable as the basis of follow-up proposals for Herschel observing time during the mission.

The Key Programme consortia are also required to share the methods employed to generate the data products provided to the HSC. It is possible that these methods could be incorporated into or adapted for use in the standard HSC data processing software for public release."

Consequently, in order to enable follow up observations to be proposed by the astronomical community, each KP consortium included in their proposal a description of the data products and methods that they planned to provide at the end of the proprietary period. The approval of the proposal by HOTAC and the allocation of observing time implicitly assume the fulfilment by the KP consortia of the committed deliveries.

Regular Programme Observers

If you have been awarded observation time through a Regular Programme Announcement of Opportunity, you are encouraged to contribute, on voluntary basis, with highly processed data products and software tools to the Observatory, and to make use of the HSC facilities to widespread their results to the astronomical community. Contributed products shall be supported by the corresponding paper in a refereed publication. Deliveries should follow the guidelines and procedures specified in this document.

Contributing data products

Herschel data products are classified based on their processing level (from raw data to highly processed) and their contents. For an overview see the HSC Data Products pages and for a more detailed description please refer to the Products Definition Document. Generally, it is expected that contributed products will be advanced processed data of the following type:

  • Highly processed level-2 products
  • Level-3 products, per AOR or combination of AORs (e.g., large maps, spectral surveys)
  • Catalogues (e.g., astronomical sources catalogues, spectral line lists)
  • Ancillary data (e.g., model SEDs, reference spectral line lists)

Highly processed data products can be made available to the HSC by providing the address of the FTP site where the products are located. The address will be included in the Herschel Science Centre public Web pages, and will be associated with the corresponding proposal and observations in the HSA.

You should also provide any other web addresses from which data products are accessible (for instance, CDS for catalogues).

In the future you will be able to submit your data directly for ingestion in the Herschel Science Archive as "Highly Processed Data Products", so that they are integrated in the HSA facilities and become part of the Virtual Observatory.

Product documentation

Contributed products must be accompanied by the following information in a separate file (text, PDF or other standard format):

  1. Proposal Identification (as provided by HSpot and in the product metadata keyword "proposal")
  2. List of products contained in the delivery. For each product please include the following:
    1. Observation identifiers ("obsid" values) of the observations that have been used to create the product.
    2. Product type (image, cube, spectrum, line list, catalogue and so on)
    3. Product format (HCSS FITS, CLASS FITS, standard FITS).
  3. The address of the FTP site where the products are located.
  4. References to refereed papers.
  5. For KP associated products, a brief description of the algorithms, methods and processing steps involved in the creation of the product (see below).

Methods used in the generation of the scientific products shall be described indicating:

  1. If SPG products have been used, the SPG version and the maximum SPG level from which the interactive/user analysis has started (see HSC Data Products pages) Example: SPG v1.0, level-1.
  2. A detailed description of the data reduction steps carried out on top of the standard product to generate the final product.
    • For data reduced in the Herschel Interactive Processing Environment (HIPE), the HIPE version number must be provided, and a description of the user input for each step.
    • For steps that do not involve official HIPE routines, you should provide a detailed description of the algorithms used. As stated in the KP Policies and Procedures document, the HSC may decide to include or implement these methods in the standard HSC data processing software for public release.

Format of submitted products

The format of products that are submitted to the Herschel Science Centre should conform to the following specifications:

  1. Products shall be provided as FITS files. The structure and keyword definition shall conform to the FITS standard as defined by version 3.0 (2008).
  2. The product metadata must contain, at least, the compulsory keywords listed in Table 1, Contributed products compulsory metadata.

Table 1. Contributed products compulsory metadata

FITS keyword Type Description Herschel DP keyword name Notes
DATE String Date of product creation creationDate  
CREATOR String The name of the software that created the product creator  
DESC String Full name of product description  
OBSERVER String Name of observer observer As given in SPG products
PROPOSAL String Proposal name proposal As given in SPG products
TYPE String Product type identification type It should be equal to "User Contribution"
OBS_ID Long integer Observation identification obsid Mandatory metadata only for products associated with an individual observation

To ensure consistency with HIPE and to maximise commonality, please deliver your products following the standard Herschel product definitions. In particular, you should implement the following:

  • Images following the structure of the SimpleImage Product (see section 2.4.4 of the Herschel Products Definitions Document)
  • Image cubes following the structure of the SimpleCube Product (see section 2.4.4 of the Herschel Products Definitions Document)
  • Metadata keywords as listed in Appendix A of the Herschel Products Definitions Document.
  • Spectral cubes following the structure of the SpectralSimpleCube Product (see section 2.4.4 of the Herschel Products Definitions Document)
  • Spectra following the structure of the Spectrum1d or Spectrum2d datasets as appropriate (see section 2.4.3 of the Herschel Products Definitions Document)
  • Spectral line list catalogues following the definition as given in section 6.3.1 of the Herschel Products Definitions Document
  • Source list catalogues following the definition as given in section 6.3.2 of the Herschel Products Definitions Document

Redelivery of products

As time advances in the mission, our knowledge of the satellite and instrument behaviour will increase, and this will be reflected in improvements of the data processing software and calibration. For the benefit of the mission's legacy, you should consider re-processing your observations with upgraded versions of data reduction software and calibration, and re-delivering them to the HSC.

Contributing tools

You are welcome to share your scripts and tools, implemented in Jython or Java, with other observatory users. Submitted tools must be accompanied by documentation that describes their purpose and algorithms, and how to install and use them (see the section called “Documentation”).

Contributing scripts

Contributing single Jython scripts, or collections of related scripts, is the simplest way to share code with the community. Such scripts are intended to be run from the Editor view of HIPE, much like routines from the IDL Astronomy User's Library are run from the IDL environment.

You can also contribute scripts for other environments, such as IDL or CLASS, if these played a role in the reduction of your data.

Contributing Jython or Java plugins

With plugin we denote any external component that can be integrated into HIPE. This can be a task (a special type of function, usually for data processing, such as fitFringe), a view (a window within HIPE, such as the Editor view) or a perspective (a set of windows within HIPE, such as the Work Bench perspective).

The Herschel Science Centre has developed a framework to install, update and uninstall user-contributed plugins within HIPE:

Note that you can also contribute tasks as Jython scripts: a description of how to develop Jython-based tasks is given in Chapter 4 of the Scripting and Data Mining guide, shipped with HIPE.

Documentation

The delivery should include documentation containing:

  • Name of author or authors, affiliation, email.
  • Associated Herschel proposal identifier.
  • List of HIPE versions with which the module is compatible.
  • Purpose of the software.
  • Description of the algorithms.
  • Short user's manual. If you provided a HIPE component as a JAR file, please include instructions for users to integrate it into their HIPE installation.
  • Description of the Highly Processed Data Products derived with the modules, if applicable.

Maintenance

The responsibility to maintain and update the contributed tools lays on the contributors. The Herschel Science Centre will provide no support for problems with these tools, or for problems arising in HIPE because of these tools. Users will download and install the tools at their own risk. This disclaimer will be clearly displayed on any download page provided by the Herschel Science Centre.

The Herschel Science Centre may decide to integrate tools of outstanding quality into the official HIPE distribution. In this case, support to users will be provided.

Acronyms

AO Announcement of Opportunity
AOR Astronomical Observation Request
DP Data Processing
HIPE Herschel Interactive Processing Environment
HOTAC Herschel Observing Time Allocation Committee
HSA Herschel Science Archive
HSC Herschel Science Centre
KP Key Programmes
SED Spectral Energy Distribution
SPG Systematic Product Generation

 
This site is powered by the TWiki collaboration platform Powered by Perl