2.8. Products

A product is an object containing a set of metadata entries and one or more datasets.

With respect to composite datasets, products offer more features, such as a history recording all the changes (see Section 2.8.7) and a set of metadata parameters added automatically whenever a product is created.

The automatic metadata values are type, creator, creationDate, description, instrument, modelName, startDate, endDate and formatVersion.

2.8.1. Creating a product

The following example shows how to create a product:

# Create an empty product
myProduct = Product()
# Create a product with a given description
myProduct = Product("This is my product")
# Create a product by specifying some of the compulsory metadata values
myProduct = Product(creator="Myself", instrument="SPIRE", \
description="An empty product", modelName="PFM", type="ABC")

Example 2.66. Creating an empty product with some metadata.


Adding datasets or other products to a product follows the same syntax as adding datasets to a composite dataset:

# Assume that t is a table dataset and p is a product
p["measurements"] = t  # Adding t to p with the name measurements

2.8.2. Modifying a product

The following example shows how to modify a product:

# Substituting an array/product (same as adding an array/product)
p["Measurements"] = myOtherDataset
# Removing an array/product
p.remove("Measurements")

Example 2.67. Overwriting an array within a dataset.


To change a dataset name, remove it and add it again with the new name.

There is a simplified syntax to set the compulsory metadata values:

myProduct.type = "ABC"
myProduct.creator = "Myself"
myProduct.description = "An empty product"
myProduct.instrument = "SPIRE"
myProduct.modelName = "PFM"  # And so on

Example 2.68. The most common metadata have attributes defined.


2.8.3. Setting date and time in product metadata

The startDate, endDate and creationDate are mandatory metadata parameters and are set to the current date and time when the product is created. To set any of these parameters to the system date and time, use these commands:

myProduct.creationDate = FineTime(java.util.Date())
myProduct.startDate = FineTime(java.util.Date())
myProduct.endDate = FineTime(java.util.Date())

Example 2.69. Some of the time attributes are instances of FineTime.


To set these parameters to an arbitrary date and time, expressed as UTC or TAI, use the following commands:

formatter = SimpleTimeFormat(TimeScale.UTC)
timeUtc = formatter.parse("2008-01-31T12:35:00.0Z")    # Z at the end is mandatory for UTC

formatter = SimpleTimeFormat(TimeScale.TAI)            # or just SimpleTimeFormat()
timeTai = formatter.parse("2008-01-31T12:35:00.0TAI")  # TAI at the end is mandatory for TAI

myProduct.creationDate = timeUtc # or 
myProduct.creationDate = timeTai

Example 2.70. Creating TAI or UTC time strings to set time metadata.


Note that the two previous dates, represented as FineTime, are different:

HIPE> print timeUtc
2008-01-31T12:35:33.000000 TAI (1580474133000000)
HIPE> print timeTai
2008-01-31T12:35:00.000000 TAI (1580474100000000)

See Section 9.1 for more information on representing time in the Herschel software.

2.8.4. Inspecting a product

The following example shows how to inspect a product:

print myProduct.description  # Print the description of a product
# You can use the above syntax for any of the compulsory metadata. For example:
print myProduct.type
print myProduct.creationDate  # And so on
print myProduct.isEmpty()  # Check whether a product is empty. Returns True or False.
print myProduct.sets.size()  # Print the number of dataset/products in the product.
print myProduct.keySet() # Print a vector with the names of all the datasets and products in this product
print myProduct.default # Print the first inserted dataset.
print myProduct["anotherDataset"]  # Print the dataset called anotherDataset.

Example 2.71. Inspecting an object from a subclass of Product.


Instead of just printing out the datasets you get, you can assign them to variables and execute other operations on them. To see how to explore the contents of datasets please refer to the previous sections of this chapter.

See the HIPE Owner's Guide for ways of inspecting a product via a graphical interface.

2.8.5. Product contexts

Contexts are special types of products that contain references to other products. Contexts are used to organise related products into a coherent structure. For example, the observation context contains all the data related to an observation.

There are two standard types of context products provided: ListContext, for grouping products into sequences or lists, and MapContext, for grouping products into containers with access to each by key. The observation context is a map context.

Most likely you will not have to create your own contexts, but just access those you retrieve from the Herschel Science Archive.

2.8.6. Observation contexts

Every observation you download from the Herschel Science Archive is an observation context. An observation context contains other contexts and many data products. While the high-level structure of observation contexts is broadly the same, the detailed structure and product types vary according to the observation type. See the following resources for more information on observation contexts:

2.8.7. Product history

The Product history is generated or updated whenever a task is run on a product. It contains the tasks which have been run to generate the product (including used parameters), as well as the used calibration files and the track and build number of the used build.

You can retrieve the history of a product in Jython as follows:

history = product.history

A simple print shows which tasks, build numbers and calibration files have been used:

HIPE> print history

Additional interesting functionalities of the history list are the following:

  • Get the history as a Jython script:

    script = product.history.script
    product.history.saveScript("script.py")

    Example 2.72. Saving the product history to a script file.


  • Find out if a certain task has been run (useful in a task which depends on another task):

    if not product.history.isTaskPerformed("someTask"):
          print "You have to run someTask first!"
          exit

    Example 2.73. Checking if a task has been executed on a product (either locally or as part of standard processing).


Calibration file appear in the history with identifiers constructed from meta keywords. How this is done depends on the instrument:

  • For PACS the identifier is constructed from the meta keywords calFileId, modelName and calFileVersion as follows:

    Common|Photometer|Spectrometer_calFileId_modelName_calFileVersion

    Also, if you want to introduce your own handcrafted PACS calibration file you should change especially the calFileId keyword to make sure that this is visible in the history.

  • For SPIRE the fileName meta keyword is used to identify calibration files.

  • For HIFI no standard has been implemented yet.