1.4. Getting observations from the Herschel Science Archive

This section outlines the steps for getting your data out of the Herschel Science Archive, and serves as a map for the rest of the subsections. Four common workflows are outlined in the following flowchart. See the text after the flowchart for links to the relevant documentation.

Workflows for retrieving observation data from the Herschel Science Archive

Figure 1.4. Workflows for retrieving observation data from the Herschel Science Archive


1.4.1. Logging into the HSA

To easily access data in the archive, you must first ensure that you are logged into the HSA inside HIPE. Typically this must be done only once.

To check your login status, you can inspect the bottom status bar of HIPE. If you are not logged in, the text "HSA Log-in" will be visible as shown in Figure 1.5 . Click on this part of the sidebar and enter your Herschel username and password. Tick the Remember me checkbox if you want HIPE to store your credentials securely—then you won't have to re-enter your login and password in future HIPE sessions.

Logging in to the Herschel Science Archive

Figure 1.5. Logging in to the Herschel Science Archive


Once you have logged into the archive in HIPE, you can move to the Welcome page (see below).

Accessing the Herschel Science Archive

Figure 1.6. Accessing the Herschel Science Archive


Starting on the Welcome page (the default view on first opening HIPE), click on Access Data. In the Access to Herschel Data view that appears, click Data Access (see Figure 1.6). The Data Access perspective appears. From it you can use the Herschel Science Archive view to launch the archive interface.

[Tip] Tip

If you cannot see the Welcome page, click the icon at the top right corner of the HIPE main window. Also, in the same toolbar to the left, there is an always-visible icon that instantly launches the HSA interface.

The HSA view

Figure 1.7. The HSA view


In the Herschel Science Archive view, click Open HSA User Interface to launch the HSA application. You may need to accept the security certificate (see Figure 1.7 ).

[Tip] Tip

If you cannot find the Open HSA User Interface button in the HIPE window, go to the top-level menu item WindowShow ViewData AccessHerschel Science Archive .

Only authorised users can access data covered by proprietary rights. The same rule applies to the viewable quick-look products of observations, as well as to proposal-related files. They can only be viewed by the logged-in observation owner.

1.4.2. Finding observations in the HSA

Using the Web interface for the Herschel Science Archive: The Web interface for the Herschel Science Archive opens on the Search screen: see below.

The Web Herschel Science Archive UI main interface.

Figure 1.8. The Web Herschel Science Archive UI main interface.


To define your query, set all the relevant fields in the Search screen. You can search for observations by name, coordinates, date, position, instrument mode, proprietary status, observer, program identifier and many other criteria. Click Search in the bottom of the screen to run your search. A new screen is displayed (see below) with the observations matching the query.

The list of results returned by a query in the Web Herschel Science Archive interface.

Figure 1.9. Results screen in the Web Herschel Science Archive interface.


To get additional details, click on the magnifying glass in the results list. This will open a sidebar to the right, with a bigger thumbnail image and several tabs containing information about the observation.

The list of results returned by a query in the Web Herschel Science Archive interface, with additional details in a sidebar.

Figure 1.10. Results screen in the Web Herschel Science Archive interface with detailed information.


See Section 1.4.3 for an explanation of the summary information.

Sending query results to HIPE. You can send all or some of the query results to HIPE as a table dataset.

  • To send all results:

    • Click the top-left checkbox in the results list (just to the left of the column headers).

    • Then, click on this icon in the top-right icon group of the results screen. This allows you to send all the content of all the observations to HIPE. You can send the Standalone Browse Products (SBP), if the observations have them, to other SAMP-enabled applications (including HIPE).

      The share dialogue allows to send observations to HIPE or just the SBP to any other SAMP-enabled application (including HIPE).

      Figure 1.11. Share observations dialogue via SAMP.


  • To send some results:

    • Select the results you want to send by ticking the checkbox next to each entry.

    • Click the same share icon as before.

  • To only send one result, just use the icon in the appropriate result row. There is no need to check the box at the left of the row.

1.4.3. Inspecting the query results of an observation

A query made with the Web interface for the HSA returns a table with one row per observation, as shown in Figure 1.9. Click the magnifying lens icon to display additional information, including an enlarged version of the observation thumbnail image, if available.

Information about the quality of science data in an observation is held in a special product called Quality Control Report. It includes the assessment of the execution of the observation by the spacecraft and the instruments, the evaluation of the success of the data processing, the outcome of the systematic inspection of the final product and, if required, the instrument specialist and community support astronomer analysis.

In the query results screen of the Web interface, for all observations, you can click on the QCR button to access the Quality Control Report summary.

Information summary for an observation returned as a query result.

Figure 1.12. Information summary for an observation returned as a query result.


The Quality Control Report summary lists all the quality information relevant to you. This summary is only generated once the quality control cycle of an observation has been completed, which can take some time.

[Note] Note

If the observation does not have a Quality Control Report summary, clicking the button will display the following warning message:

QCR summary missing for this observation.

Figure 1.13. QCR summary missing for this observation.


Once the quality control assessment of an observation is complete, the Quality Control Report summary Status field displays its outcome as one of PASSED, FAILED or PENDING. Further explanations and known caveats about the outcome are included in the Comments section of the Quality Control Report summary. The PENDING flag is used when the quality control cycle has finished but there are still actions to be taken. Typically this involves the reprocessing of an observation.

1.4.4. Finding observation IDs outside the HUI

The Observing Log.  The most common alternative to the HUI for finding observation IDs, or program/proposal IDs, is to consult the Herschel Observing Log. By default this webpage shows the most recent downlinked observations (the last one was Obs. Id 1342271266 on the 29th of April of 2013), as shown in Figure 1.14. You can also use the form at the top of the page to query by any of the column headings: operational day (OD), target name, proposal ID, observing mode, observation ID, AOR label, AOR duration, start time, processing (SPG) version, or quality control state. You can click the column headings themselves to sort the values for that column, sorting on all items found and not just those displayed on the current page.

The Observing Log webpage.

Figure 1.14. The Observing Log webpage.


The Observing Log form does not allow querying by coordinates or proprietary status.

Using IRSA. In your web browser, navigate to the IRSA homepage at irsa.ipac.caltech.edu. Type a target name into the Search box and click Search . Scroll to the bottom of the search results page, and find the table "Archive Availability". Look for lines containing Mission = Herschel, such as Herschel PACS Data. Click on the Go button in the last column ( Explore Data ). This takes you to a results page similar to the results tab in the HUI, where you can browse postcards and locate the observation id of interest.

Using Vizier. In your web browser, navigate to the Vizier catalog of Herschel observations . In the Simple Target tab, enter the source name. In the Constraints Table be sure to tick the ObsId entry under the Show column. Then click on Submit . As for the HUI and IRSA, browse the postcards to find the observation id of interest.

1.4.5. Downloading one entire observation

This section explains how to download all the data for a single observation from the Herschel Science Archive to your local disk.

GUI Method: Using the HUI

Prerequisites.  You must have logged into the HSA, as described in Section 1.4.1, and searched for one or more observations, as described in Section 1.4.2.

In the query results tab of the HUI, click the download icon to access the download screen, from which you can select what parts of the observation to download. Choose All to download the entire observation. Once you have made your choice, you are prompted to save the tar file with the observation.

Downloading an observation from the Herschel Science Archive.

Figure 1.15. Downloading an observation from the Herschel Science Archive.


Using this method is only recommended for individual observations. To download many observations at once, see Section 1.4.7 .

Where are my data? Your data are contained in a tar file on your disk. HIPE is not yet aware of the data.

Where do I go from here?  Once you have downloaded your observation from the Archive, you must load it into HIPE, as explained in Section 1.5 .

Command-line method: Using getObservation

Prerequisites.  You must be logged into the HSA as described in Section 1.4.1 .

Use the getObservation command to download an observation from the Herschel Science Archive and save it to disk. You need to know the obsid of your observation:

myObs = getObservation(obsid=1342231345, useHsa=True, save=True)

Example 1.5. Saving an observation to disk from the HSA given the observation ID.


[Warning] Warning

When retrieving SPIRE/PACS parallel mode observations, you must specify the instrument parameter with a value of either PACS or SPIRE, as shown by the following examples:

myObs = getObservation(obsid=1342183046, instrument='PACS', useHsa=True, save=True)

myObs = getObservation(obsid=1342183046, instrument='SPIRE', useHsa=True, save=True)

Where are my data?  Your observation has been saved to the My HSA repository on your disk. Do not access the files directly. You can access your observation in HIPE at any time with the Product Browser, as explained in Section 1.7 , or with this command, which will look for all copies of an observation with a given obsid:

myObs = getObservation(obsid=1342183046)

Example 1.6. Getting an observation from the HSA given an observation ID.


For more information on managing your My HSA repository, see Section 1.6 .

[Note] Note

In some cases you may want to configure getObservation so that it doesn't look for and download observations from the HSA. There is a setting within Preferences in Data Access > My HSA called Save data on-demand that controls this behaviour. In a script, you can set it on and off using the following commands:

# Turn it on 
MyHSAPool.getInstance().setConnection(herschel.ia.pal.pool.hsa.MyHSAConnection.ON)

Example 1.7. Setting on the connected status of the MyHSA pool.


# Turn it off 
MyHSAPool.getInstance().setConnection(herschel.ia.pal.pool.hsa.MyHSAConnection.OFF)

Example 1.8. Setting off the connected status of the MyHSA pool.


Where do I go from here?  Your observation is now referenced by the myObs variable. You can now work on your data and then save your modified data to disk as explained in Section 1.10 .

In future HIPE sessions, you can retrieve the original observation or the modified one as explained in Section 1.7 .

1.4.6. Using HIPE to inspect an observation in the HSA with known OBSID

With HIPE you can inspect the contents of observations stored in the HSA without having to save them first. What is sent to HIPE are just references to data products, not the products themselves.

GUI Method: Using the HUI

Prerequisites.  You must have logged into the HSA, as described in Section 1.4.1 , and searched for one or more observations, as described in Section 1.4.2 .

In the query result tab, click the share icon to access a Send to HIPE menu. With this you can choose whether to inspect the whole observation or just a portion of it.

Selecting which part of an observation to inspect.

Figure 1.16. Selecting which part of an observation to inspect.


Select an option and, if the connection between the two applications (WHUI and HIPE, through the SAMP Hub) is well established, a marquee in the top of the screen displays the message SAMP message(s) successfully sent. Data starts to load into HIPE automatically.

If the option All was selected, a variable called obsid_xxxxxxxxxx is created in HIPE, with an actual observation number. Other options are also stored in different variables as illustrated in Figure 1.17.

Product loaded into HIPE from the HSA.

Figure 1.17. Product loaded into HIPE from the HSA.


Where are my data?  Your data are not stored on your machine, but read on-demand from the HSA. Note also that for this, the Internet connection must be kept open throughout the session.

Where do I go from here?  Once you have finished browsing the data, you can save them to disk as described in Section 1.10.

Command-line method: Using getObservation

Prerequisites.  You must be logged into the HSA as described in Section 1.4.1 .

Use the getObservation command to browse an observation from the Herschel Science Archive. You must know the OBSID of the observation:

myObs = getObservation(obsid = 1342231345, useHsa = True)

Example 1.9. Retrieving an observation from the HSA given the observation ID.


[Warning] Warning

When retrieving SPIRE/PACS parallel mode observations, you must specify the instrument parameter with a value of either PACS or SPIRE, as shown by the following examples:

myObs = getObservation(obsid=1342183046, instrument='PACS', useHsa=True, save=True)

myObs = getObservation(obsid=1342183046, instrument='SPIRE', useHsa=True, save=True)

Double click on the myObs variable in the Variables view to open the Observation Viewer. You can also print out portions of the observation context via commands like print myObs.level2 , but in general the Observation Viewer will be much more convenient for browsing.

Where are my data?  Your data are not being accessed from storage on your machine, but are read on-demand from the HSA. Note also that for this to work, the Internet connection must be kept open throughout the session. If either the save=True option is passed to getObservation, or the "Save data on-demand" option is turned on as detailed in Section 1.6.1 , data are saved into your MyHSA location.

Where do I go from here?  Once you have finished browsing the data, you can save them to disk as described in Section 1.10 .

1.4.6.1. Multiple versions of the same observation

There are two additional cases when, given an observation ID, there are multiple instances of the observation in the HSA or in your computer. They are identified by two metadata fields, one of which changes its meaning between these two different locations.

  • spgVersion is always the version of HCSS that was used to generate the product as part of the Standard Product Generation (SPG) pipeline. Its format is M.m.p where M is major version, m is minor version and p is patch version like in 11.1.0.

  • version can be two things, identified by an integer number, depending on the location (even if, conceptually, they are the same thing):

    • In the HSA: it is the number of bulk reprocessings performed on the data.

    • In your computer: it is the number of changes followed by a save to the same pool that you have made to the data.

Using the command line, you can browse all versions in a very powerful way:

obs = getObservation(obsid = 1342197792, useHsa = True, allVersions = True)

Example 1.10. Browse all the versions of an observation (part 1).


If you haven't downloaded this observation before, all the versions displayed are the ones available in the HSA, as in this example. The output of the command can be seen below:

More than one observation found. Please, refine your query
    
The following parameters:
urn = null
obsid = 1342197792
od = -1
instrument = null
tag = null
spgVersion = null
creationDate = null
version = -1

Produces more than one result:
Pool id | Obsid | Tag | Version | Track id | Urn | Creator
hsa | 1342197792 |  | 8 | OBS_12508@wn25.n1grid.lan_1_16441247557132381 | urn:hsa:herschel.ia.obs.ObservationContext:429666 | SPG v11.1.0 | 
hsa | 1342197792 |  | 9 | OBS_12508@wn25.n1grid.lan_1_16441247557132381 | urn:hsa:herschel.ia.obs.ObservationContext:526876 | SPG v12.1.0 | 
hsa | 1342197792 | OBS:P:0050005020 | 10 | OBS_12508@wn25.n1grid.lan_1_16441247557132381 | urn:hsa:herschel.ia.obs.ObservationContext:620498 | SPG v13.0.0 | 

To retrieve the observation please provide a version, urn, or a SPG version number like '11.1.0'

This output text advises you that spgVersion is a substring of the metadata field creator present in the archive. This way, SPG v11.1.0 becomes 11.1.0 for the purposes of retrieving the observation. If you now type:

obs = getObservation(obsid = 1342197792, useHsa = True, spgVersion="11.1.0")

Example 1.11. Browse all the versions of an observation (part 2a).


Or:

obs = getObservation(obsid = 1342197792, useHsa = True, version = 8)

Example 1.12. Browse all the versions of an observation (part 2b).


Which in this case is the same, you will download the desired observation version to disk.

[Note] Note

When downloading observations to your computer and, as stated above, the version number is reset to zero (0) to represent, from that moment on, user changes, so if you want to retrieve a specific SPG version from a local pool, you are required to use spgVersion. To load from disk the previously downloaded observation, you should type:

obs = getObservation(obsid = 1342197792, useHsa = False, spgVersion="11.1.0")

Example 1.13. Browse all the versions of an observation (part 3).


1.4.7. Downloading multiple observations

GUI Method: Using the HUI

Prerequisites.  You must have logged into the HSA, as described in Section 1.4.1, and searched for one or more observations, as described in Section 1.4.2 .

You can download multiple observations thanks to the multi-selection capabilities of the results list. This works in the same way as described in the Sending query results to HIPE paragraph.

To select data for retrieval, tick the checkbox to the left of each record of the observations list, and then click the inbox icon at the top-right of the results list. The download screen that appears has additional options to save the selected observations as a CSV file or a VOTable. It also allows you to choose between the selection you just made or all the rows in the list.

Dialogue to download multiple observations from the HSA.

Figure 1.18. Dialogue to download multiple observations from the HSA.


Alternatively, download one observation at a time by clicking the download icon in the observation's table row. The standard download screen appears. See Figure 1.15.

Where are my data? Your data are contained in a tar file on your disk. HIPE is not yet aware of the data.

Where do I go from here?  Once you have downloaded your observations from the archive, you must load them into HIPE, as explained in Section 1.5 .

Command-line method: Using getObservation

Prerequisites.  You must be logged into the HSA as described in Section 1.4.1 .

You can use the getObservation command within a loop to download multiple observations from the Herschel Science Archive and save them to disk. You need to know the obsid of all the observations:

myObsids = [1342242595, 1342239349]

for i in myObsids:
  print "Downloading obsid " + `i`
  myObs = getObservation(obsid=i, useHsa=True, save=True)

Example 1.14. Downloading multiple observations from an array of obs ids.


It is also possible to do this with tarred observations retrieved using the HAIO

from java.nio.file import Files
import urllib

# Create temporary directory for TAR.GZ download
tempdir = Files.createTempDirectory("hipe")
observationIds = [1342231052, 1342231345]

# Output dictionary of observations (empty for now)
observations = {}

# Do the loop
for obs in observationIds:
	# Download a TAR.GZ observation using the HAIO and builtin urllib
	strObsPath = str(tempdir)+"/obs.tar.gz"
	
	urllib.urlretrieve("http://archives.esac.esa.int/hsa/aio/jsp/product.jsp?"+\
			"OBSID="+str(obs)+"&PRODUCT_LEVEL=Auxiliary&COMPRESSION=TARGZ&"+\
			"PROTOCOL=HTTP", strObsPath)
	# Decompress the file
	outputDir = str(tempdir)+"/obs"+str(obs)
	decompress(strObsPath, outputDir)
	
	# One more step as there is another directory with a random name within outputDir 
	randomDir = os.listdir(outputDir)[0]
	finalPath = outputDir+"/"+randomDir
	
	# Open the tarred observation from disk
	observations[obs] = getObservation(path = finalPath) 

Example 1.15. Retrieving several observations from the HSA as tar.gz and opening them in HIPE.


[Warning] Warning

When retrieving SPIRE/PACS parallel mode observations, you must specify the instrument parameter with a value of either PACS or SPIRE, as shown by the following examples:

myObs = getObservation(obsid=1342183046, instrument='PACS', useHsa=True, save=True)

myObs = getObservation(obsid=1342183046, instrument='SPIRE', useHsa=True, save=True)

Where are my data?  Your observations have been saved to the My HSA repository on your disk. Do not access the files directly. You can access your observations in HIPE at any time with the Product Browser, as explained in Section 1.7 , or with this command, which will look for all copies of an observation with a given obsid:

myObs = getObservation(obsid=1342183046)

Example 1.16. Retrieve an observation from the HSA given the observation ID.


For more information on managing your My HSA repository, see Section 1.6 .

Where do I go from here?  The last observation downloaded is now referenced by the myObs variable. You can find and load the other observations via the Product Browser as explained in Section 1.7 . Then you can work on your data and save the modified data to disk as explained in Section 1.10 .

In future HIPE sessions, you can retrieve the original observations or the modified ones as explained in Section 1.7 .