8.2. Standard Product Generation

The HSC received the raw telemetry downlinked from the Herschel spacecraft, via MOC, on a daily basis after each DTCP (Daily Telecommunications Period). This raw telemetry set, corresponded to the observations performed in the previous 24 hours approximately, the so-called Observational Day, or OD, were ingested in the local database at the HSC and simultaneously propagated also to the main ICCs for quick-look analysis activities.

A watchdog, setup in the data processing system at HSC, monitored the system status after each DTCP ended and automatically decided on the readiness of the system to process the data from that OD, based on specific criteria being met. When these criteria were met (completion of telemetry ingestion for that OD in the local database and pointing Product availability in the system for that OD), the watchdog system launched the Standard Product Generation (SPG) pipelines, one for each observation on that OD, in the distributed computing system (GRID) available at the HSC. This distributed system uses several worker nodes containing multiple processors that allow parallel data processing, and is an essential and very powerful infrastructure for the reduction of the large volumes of data generated by Herschel observations daily.

During the automatic data processing of one observation, level 0, 1, 2 and Quality Control products are generated. These data products, together with the inputs used in the processing (auxiliary and calibration data) are 'bundled' together in a so called observation context, a top level container which is the main output of each individual pipeline processing. When the pipeline processing finishes and this top level container is created, it is immediately ingested in the Herschel Science Archive, making it available for retrieval by observers. The Quality Control cycle then starts then on the generated pipeline products, following established Quality Control procedures specific to each observing mode.

The whole process, from observation of an astronomical object to its automatic data reduction using pipelines and ingestion into the science archive, typically took less than 48 hrs. This is one of the unique features of the Herschel mission in which data products were processed and made available to observers in an immediately usable condition, shortly after the data has been taken.