A product is an object containing a set of metadata entries and one or more datasets.
With respect to composite datasets, products offer more features, such as a history recording all the changes (see Section 2.8.7) and a set of metadata parameters added automatically whenever a product is created.
The automatic metadata values are
The following example shows how to create a product:
# Create an empty product myProduct = Product() # Create a product with a given description myProduct = Product("This is my product") # Create a product by specifying some of the compulsory metadata values myProduct = Product(creator="Myself", instrument="SPIRE", \ description="An empty product", modelName="PFM", type="ABC")
Example 2.66. Creating an empty product with some metadata.
Adding datasets or other products to a product follows the same syntax as adding datasets to a composite dataset:
# Assume that t is a table dataset and p is a product p["measurements"] = t # Adding t to p with the name
The following example shows how to modify a product:
# Substituting an array/product (same as adding an array/product) p["Measurements"] =
myOtherDataset# Removing an array/product p.remove("Measurements")
Example 2.67. Overwriting an array within a dataset.
To change a dataset name, remove it and add it again with the new name.
There is a simplified syntax to set the compulsory metadata values:
myProduct.type = "ABC"
myProduct.creator = "Myself"
myProduct.description = "An empty product"
myProduct.instrument = "SPIRE"
myProduct.modelName = "PFM" # And so on
Example 2.68. The most common metadata have attributes defined.
creationDate are mandatory metadata parameters and are set to the current date and time when the product is created. To set any of these
parameters to the system date and time, use these commands:
myProduct.creationDate = FineTime(java.util.Date())
myProduct.startDate = FineTime(java.util.Date())
myProduct.endDate = FineTime(java.util.Date())
Example 2.69. Some of the time attributes are instances of FineTime.
To set these parameters to an arbitrary date and time, expressed as UTC or TAI, use the following commands:
formatter = SimpleTimeFormat(TimeScale.UTC) timeUtc = formatter.parse("2008-01-31T12:35:00.0Z") # Z at the end is mandatory for UTC formatter = SimpleTimeFormat(TimeScale.TAI) # or just SimpleTimeFormat() timeTai = formatter.parse("2008-01-31T12:35:00.0TAI") # TAI at the end is mandatory for TAI
myProduct.creationDate = timeUtc # or
myProduct.creationDate = timeTai
Example 2.70. Creating TAI or UTC time strings to set time metadata.
Note that the two previous dates, represented as
HIPE>print timeUtc 2008-01-31T12:35:33.000000 TAI (1580474133000000)
HIPE>print timeTai 2008-01-31T12:35:00.000000 TAI (1580474100000000)
See Section 9.1 for more information on representing time in the Herschel software.
The following example shows how to inspect a product:
myProduct.description # Print the description of a product # You can use the above syntax for any of the compulsory metadata. For example: print
myProduct.creationDate # And so on print
myProduct.isEmpty() # Check whether a product is empty. Returns True or False. print
myProduct.sets.size() # Print the number of dataset/products in the product. print
myProduct.keySet() # Print a vector with the names of all the datasets and products in this product print
myProduct.default # Print the first inserted dataset. print
anotherDataset"] # Print the dataset called
Example 2.71. Inspecting an object from a subclass of Product.
Instead of just printing out the datasets you get, you can assign them to variables and execute other operations on them. To see how to explore the contents of datasets please refer to the previous sections of this chapter.
See the HIPE Owner's Guide for ways of inspecting a product via a graphical interface.
Contexts are special types of products that contain references to other products. Contexts are used to organise related products into a coherent structure. For example, the observation context contains all the data related to an observation.
There are two standard types of context products provided: ListContext, for grouping products into sequences or lists, and MapContext, for grouping products into containers with access to each by key. The observation context is a map context.
Most likely you will not have to create your own contexts, but just access those you retrieve from the Herschel Science Archive.
Every observation you download from the Herschel Science Archive is an observation context. An observation context contains other contexts and many data products. While the high-level structure of observation contexts is broadly the same, the detailed structure and product types vary according to the observation type. See the following resources for more information on observation contexts:
For a brief introduction to observation contexts see the Products Definitions Document: Section 2.9, “The Herschel Observation Context”.
For more information on HIFI observation contexts, see the HIFI Data Reduction Guide.
For more information on PACS observation contexts, see the PACS Products Explained.
For more information on SPIRE observation contexts, see the SPIRE Data Reduction Guide.
The Product history is generated or updated whenever a task is run on a product. It contains the tasks which have been run to generate the product (including used parameters), as well as the used calibration files and the track and build number of the used build.
You can retrieve the history of a product in Jython as follows:
history = product.history
Additional interesting functionalities of the history list are the following:
Get the history as a Jython script:
script = product.history.script product.history.saveScript("script.py")
Example 2.72. Saving the product history to a script file.
Find out if a certain task has been run (useful in a task which depends on another task):
if not product.history.isTaskPerformed("someTask"): print "You have to run someTask first!" exit
Example 2.73. Checking if a task has been executed on a product (either locally or as part of standard processing).
Calibration file appear in the history with identifiers constructed from meta keywords. How this is done depends on the instrument:
For PACS the identifier is constructed from the meta keywords
calFileVersion as follows:
Also, if you want to introduce your own handcrafted PACS calibration file you should change especially the
calFileId keyword to make sure that this is visible in the history.
For SPIRE the
fileName meta keyword is used to identify calibration files.
For HIFI no standard has been implemented yet.