<-- COMMENT BOX CODE - DO NOT EDIT -->

<-- END OF COMMENT BOX CODE -->

Herschel Tools Contributor's Guide

Contributing scripts

Contributing Jython or Java plug-ins

With plug-in we denote any external component that can be integrated into HIPE. This can be a task (a special type of function, usually for data processing, such as fitFringe), a view (a window within HIPE, such as the Editor view) or a perspective (a set of windows within HIPE, such as the Workbench perspective).

• To check the list of available plug-ins, see this page.
• To learn how to install and uninstall plug-ins within HIPE, see the user's manual.
• To learn how to contribute plug-ins for HIPE, see the author's manual.

Tools documentation

• Short user's manual.

Maintenance

• Images following the structure of the SimpleImage Product (see section 2.4.4 of the Herschel Products Definitions Document)
• Image cubes following the structure of the SimpleCube Product (see section 2.4.4 of the Herschel Products Definitions Document)
• Metadata keywords as listed in Appendix A of the Herschel Products Definitions Document.
• Spectral cubes following the structure of the SpectralSimpleCube Product (see section 2.4.4 of the Herschel Products Definitions Document)
• Spectra following the structure of the Spectrum1d or Spectrum2d datasets as appropriate (see section 2.4.3 of the Herschel Products Definitions Document)
• Spectral line list catalogues following the definition as given in section 6.3.1 of the Herschel Products Definitions Document.
• Source list catalogues following the definition as given in section 6.3.2 of the Herschel Products Definitions Document.

1. Create a directory in an FTP site. The name of the directory is not important, but it must contain three subdirectories: data, scripts and docs.
2. In the data directory, put the reduced data products in FITS format. For more information about the format of the products, see Format of submitted products.
3. In the scripts directory, put any custom scripts you have used to reduce your data. If you have developed more complex software tools to extend HIPE, you may also include them in this directory. See Contributing tools for more information on contributing scripts and other software.
4. In the docs directory, put any instructions and documentation needed to understand the scripts you provide and reproduce your data. See Product documentation for the documentation to include with data products. See Tools documentation for the documentation to include with scripts and other software.
   Note that you do not need to include documentation in this directory if it is already available elsewhere. For example, if a script is already documented by code comments or in a refereed paper, just mention this without duplicating content.
5. Contact the Herschel Science Centre Helpdesk to notify of the availability of the data. In response, the HSC Helpdesk will provide detailed instructions on how to proceed.

1. allow for early science exploitation by the community, based on the data products themselves;
2. provide the community with information usable as the basis of follow-up proposals for Herschel observing time during the mission.

1. 1. Observation identifiers ("obsid" values) of the observations that have been used to create the product.
   2. Product type (image, cube, spectrum, line list, catalogue and so on)
   3. Product format (HCSS FITS, CLASS FITS, standard FITS).

1. If SPG products have been used, the SPG version and the maximum SPG level from which the interactive/user analysis has started (see HSC Data Products pages) Example: SPG v1.0, level-1.

• • For steps that do not involve official HIPE routines, you should provide a detailed description of the algorithms used. As stated in the KP Policies and Procedures document, the HSC may decide to include or implement these methods in the standard HSC data processing software for public release.

1. Products shall be provided as FITS files. The structure and keyword definition shall conform to the FITS standard as defined by version 3.0 (2008).

Tools documentation

Herschel Data Products and Tools Contributor's Guide

Issue 1.4, 26 November 2010

This document describes how you can provide user reduced data products, catalogues and data reduction tools to the Herschel Science Centre, to make them available to the astronomical community.

Quick steps

Read this section if you just want to know what you have to do to contribute your data.

1. Create a directory in an FTP site. The name of the directory is not important, but it must contain three subdirectories: data, scripts and docs.
2. In the data directory, put the reduced data products in FITS format. For more information about the format of the products, see Section .
3. In the scripts directory, put any custom scripts you have used to reduce your data. If you have developed more complex software tools to extend HIPE, you may also include them in this directory. See Section for more information on contributing scripts and other software.
4. In the docs directory, put any instructions and documentation needed to understand the scripts you provide and reproduce your data. See Section for the documentation to include with data products. See Section for the documentation to include with scripts and other software.
5. Note that you do not need to include documentation in this directory if it is already available elsewhere. For example, if a script is already documented by code comments or in a refereed paper, just mention this without duplicating content.
6. Contact the Herschel Science Centre Helpdesk (http://herschel.esac.esa.int/esupport/) to notify of the availability of the data. In response, the HSC Helpdesk will provide detailed instructions on how to proceed.

Contributors

Key Programmes Observers

If you have been awarded observation time through a Key Programme Announcement of Opportunity, you have followed in your proposals the guidelines specified in the Policies and Procedures document for the KP AO. In particular, this document states:

"Consortia proposing for Herschel observing time on the occasion of this AO need to demonstrate ability and commitment to perform data reduction and make products (at a 'publishable' level of quality) and related tools publicly available through the HSC at the end of the proprietary period.

Besides fulfilling their scientific goals, Key Programme consortia must provide data products which will:

1. allow for early science exploitation by the community, based on the data products themselves;
2. provide the community with information usable as the basis of follow-up proposals for Herschel observing time during the mission.

The Key Programme consortia are also required to share the methods employed to generate the data products provided to the HSC. It is possible that these methods could be incorporated into or adapted for use in the standard HSC data processing software for public release."

Consequently, in order to enable follow up observations to be proposed by the astronomical community, each KP consortium included in their proposal a description of the data products and methods that they planned to provide at the end of the proprietary period. The approval of the proposal by HOTAC and the allocation of observing time implicitly assume the fulfilment by the KP consortia of the committed deliveries.

Regular Programme Observers

If you have been awarded observation time through a Regular Programme Announcement of Opportunity, you are encouraged to contribute, on voluntary basis, with highly processed data products and software tools to the Observatory, and to make use of the HSC facilities to widespread their results to the astronomical community. Contributed products shall be supported by the corresponding paper in a refereed publication. Deliveries should follow the guidelines and procedures specified in this document.

Contributing data products

Herschel data products are classified based on their processing level (from raw data to highly processed) and their contents. For an overview see the HSC Data Products pages and for a more detailed description please refer to the Products Definition Document. Generally, it is expected that contributed products will be advanced processed data of the following type:

• Highly processed level-2 products
• Level-3 products, per AOR or combination of AORs (e.g., large maps, spectral surveys)
• Catalogues (e.g., astronomical sources catalogues, spectral line lists)
• Ancillary data (e.g., model SEDs, reference spectral line lists)

Highly processed data products can be made available to the HSC by providing the address of the FTP site where the products are located. The address will be included in the Herschel Science Centre public Web pages, and will be associated with the corresponding proposal and observations in the HSA.

You should also provide any other web addresses from which data products are accessible (for instance, CDS for catalogues).

In the future you will be able to submit your data directly for ingestion in the Herschel Science Archive as "Highly Processed Data Products", so that they are integrated in the HSA facilities and become part of the Virtual Observatory.

Product documentation

Contributed products must be accompanied by the following information in a separate file (text, PDF or other standard format):

1. Proposal Identification (as provided by HSpot and in the product metadata keyword "proposal")
2. List of products contained in the delivery. For each product please include the following:
   1. Observation identifiers ("obsid" values) of the observations that have been used to create the product.
   2. Product type (image, cube, spectrum, line list, catalogue and so on)
   3. Product format (HCSS FITS, CLASS FITS, standard FITS).
3. The address of the FTP site where the products are located.
4. References to refereed papers.
5. For KP associated products, a brief description of the algorithms, methods and processing steps involved in the creation of the product (see below).

Methods used in the generation of the scientific products shall be described indicating:

1. If SPG products have been used, the SPG version and the maximum SPG level from which the interactive/user analysis has started (see HSC Data Products pages) Example: SPG v1.0, level-1.
2. A detailed description of the data reduction steps carried out on top of the standard product to generate the final product.
   • For data reduced in the Herschel Interactive Processing Environment (HIPE), the HIPE version number must be provided, and a description of the user input for each step.
   • For steps that do not involve official HIPE routines, you should provide a detailed description of the algorithms used. As stated in the KP Policies and Procedures document, the HSC may decide to include or implement these methods in the standard HSC data processing software for public release.

Format of submitted products

The format of products that are submitted to the Herschel Science Centre should conform to the following specifications:

1. Products shall be provided as FITS files. The structure and keyword definition shall conform to the FITS standard as defined by version 3.0 (2008).
2. The product metadata must contain, at least, the compulsory keywords listed in Table 1, Contributed products compulsory metadata.

Table 1. Contributed products compulsory metadata

To ensure consistency with HIPE and to maximise commonality, please deliver your products following the standard Herschel product definitions. In particular, you should implement the following:

• Images following the structure of the SimpleImage Product (see section 2.4.4 of the Herschel Products Definitions Document)
• Image cubes following the structure of the SimpleCube Product (see section 2.4.4 of the Herschel Products Definitions Document)
• Metadata keywords as listed in Appendix A of the Herschel Products Definitions Document.
• Spectral cubes following the structure of the SpectralSimpleCube Product (see section 2.4.4 of the Herschel Products Definitions Document)
• Spectra following the structure of the Spectrum1d or Spectrum2d datasets as appropriate (see section 2.4.3 of the Herschel Products Definitions Document)
• Spectral line list catalogues following the definition as given in section 6.3.1 of the Herschel Products Definitions Document
• Source list catalogues following the definition as given in section 6.3.2 of the Herschel Products Definitions Document

Redelivery of products

As time advances in the mission, our knowledge of the satellite and instrument behaviour will increase, and this will be reflected in improvements of the data processing software and calibration. For the benefit of the mission's legacy, you should consider re-processing your observations with upgraded versions of data reduction software and calibration, and re-delivering them to the HSC.

Contributing tools

You are welcome to share your scripts and tools, implemented in Jython or Java, with other observatory users. Submitted tools must be accompanied by documentation that describes their purpose and algorithms, and how to install and use them (see the section called “Documentation”).

Contributing scripts

Contributing single Jython scripts, or collections of related scripts, is the simplest way to share code with the community. Such scripts are intended to be run from the Editor view of HIPE, much like routines from the IDL Astronomy User's Library are run from the IDL environment.

You can also contribute scripts for other environments, such as IDL or CLASS, if these played a role in the reduction of your data.

Contributing Jython or Java plugins

With plugin we denote any external component that can be integrated into HIPE. This can be a task (a special type of function, usually for data processing, such as fitFringe), a view (a window within HIPE, such as the Editor view) or a perspective (a set of windows within HIPE, such as the Work Bench perspective).

The Herschel Science Centre has developed a framework to install, update and uninstall user-contributed plugins within HIPE:

• To check the list of available plugins, see this page.
• To learn how to install and uninstall plugins within HIPE, see the user's manual.
• To learn how to contribute plugins for HIPE, see the developer's manual.
• To learn about the internals of HIPE, see the contributing to HIPE pages.

Note that you can also contribute tasks as Jython scripts: a description of how to develop Jython-based tasks is given in Chapter 4 of the Scripting and Data Mining guide, shipped with HIPE.

Documentation

The delivery should include documentation containing:

• Name of author or authors, affiliation, email.
• Associated Herschel proposal identifier.
• List of HIPE versions with which the module is compatible.
• Purpose of the software.
• Description of the algorithms.
• Short user's manual. If you provided a HIPE component as a JAR file, please include instructions for users to integrate it into their HIPE installation.
• Description of the Highly Processed Data Products derived with the modules, if applicable.

Maintenance

The responsibility to maintain and update the contributed tools lays on the contributors. The Herschel Science Centre will provide no support for problems with these tools, or for problems arising in HIPE because of these tools. Users will download and install the tools at their own risk. This disclaimer will be clearly displayed on any download page provided by the Herschel Science Centre.

The Herschel Science Centre may decide to integrate tools of outstanding quality into the official HIPE distribution. In this case, support to users will be provided.

Acronyms