Chapter 4. SPIRE Observation Context Data Structure

Table of Contents

4.1. Accessing SPIRE Data
4.1.1. Understanding how HIPE accesses data
4.1.2. Accessing data directly from the Herschel Science Archive
4.1.3. Querying the SPIRE Observation Log
4.2. SPIRE Observation Context Data Structure
4.2.1. Anatomy of a SPIRE Observation: Products, Pools, Storage, and Building Blocks
4.2.2. Linking it altogether: Introducing the Context
4.2.3. The Level 2.5 Data in the Herschel Science Archive
4.2.4. The Level 3. Data in the Herschel Science Archive
4.2.5. Looking at your Observation Context in HIPE
4.2.6. What the Observation Context looks like on your hard disk
4.3. Pipelines, versions and the archive


4.1. Accessing SPIRE Data

4.1.1. Understanding how HIPE accesses data

This section gives a brief description of how to get your data into a HIPE session, from the HSA directly. This section is not an exhaustive guide and users are referred to the HIPE Help documentation on this subject contained within the Quick Start Guide and the Herschel Data Analysis Guide for a fully comprehensive explanation.

After your observation has been made, the data is archived within the Herschel Science Archive (HSA) at ESAC. In order to examine and /or reprocess your observation data, the data itself has to be retrieved from the archive. The architecture and machinery by which HIPE accesses data can seem a little bewildering at first and involves the learning of a new set of vocabulary to refer to data and its storage. Figure 4.1 schematically shows how HIPE accesses data. All data are grouped into convenient containers (like a directory on your computer) known as Pools. However, there are various types of pool depending on their location and accessibility. In Figure 4.1, four important types of Pool are described, although there are other types and a comprehensive description can be found in the Scripting Guide within the HIPE Help Documentation. Very briefly;

  • HSA Pools : These are data pools stored on the Herschel Science Archive.

  • Local Pools : These are data pools stored on your own local disk.

  • HTTP Pools : These are data pools accessible via a URL.

  • MyHSA : A special read-only pool of HSA data downloads stored on your own local disk.

In order to access any type of pool, HIPE must first initialize a gateway to the pool. This is known as a Storage and the code included in this gateway includes everything you need to access and fetch/write data in pools. Using such a gateway to access a pool is referred to as registering a pool with a storage. This registration is generally taken care of automatically by HIPE.

How HIPE performs data access

Figure 4.1. How HIPE performs data access

4.1.2. Accessing data directly from the Herschel Science Archive

HIPE provides a relatively straightforward method to access your data directly from the Herschel Science Archive. Within HIPE, select the menu item Window -- Show View -- Data Access -- Herschel Science Archive as in Figure 4.2. This will open a new window in HIPE allowing you to connect to the HSA. You may need to enter your username and password for the HSA in the boxes provided, then press the login button as shown in Figure 4.3. Once logged in, the connection to the HSA is started by pressing the Open HSA User Interface button. This will open the HSA browser in a separate window. Navigating the HSA is beyond the scope of this chapter and the user is referred to the Quick Start Guide and the Data Analysis Guide for details on querying the HSA. In summary, selecting Send to External Application from the HSA window will result in a pointer to your data appearing in the Variable window within HIPE as shown in Figure 4.4. This is the Observation Context and is named following the unique Observation ID (1342183475) for this particular observation. The Observation Context itself, is basically a catalogue of pointers describing the structure and associations within your data. The Observation Context is described in detail in Section 4.2. Note that this is still effectively a pointer to your data on the HSA and no data is stored on your hard disk yet! In order to actually save your data on your own hard disk, i.e. in a Local Pool, you can use the following line of code;

 saveProduct(product=obsid_1342183475, pool="MyGalaxyMap", tag="My data")

The first parameter is the Observation Context, in this case, as it appeared in the Variables panel in Figure 4.4. The second parameter is the name of the Pool (i.e. the name you have chosen for the directory on disk where the data will be saved). The third parameter is a tag to differentiate reprocessed versions from one another when browsing data stored on your hard disk.

Accessing the Herschel Science Archive through the HIPE menus

Figure 4.2. Accessing the Herschel Science Archive through the HIPE menus

Accessing the Herschel Science Archive through the HIPE menus

Figure 4.3. Accessing the Herschel Science Archive through the HIPE menus

The Observation Context available via the HIPE variable window

Figure 4.4. The Observation Context available via the HIPE variable window

If you do not want to use the GUI access to the HSA and you already know the observation ID, then the same process can optionally be performed from the command line using the command:

myobs=getObservation(myObsID, useHsa=True)

Note that for this command to work, HIPE expects you to have your HSA username and password written in your user.props file (usually located in your home directory under .hcss/user.props. If these lines are there, they should look like;

hcss.ia.pal.pool.hsa.haio.login_usr = your_username 
hcss.ia.pal.pool.hsa.haio.login_pwd = your_password 

Note that you may also have a hipe.props in the same location which will supercede any settings in user.props so care should be taken (See the "Properties and Preferences" section in the HIPE Owners Guide). If these are not in your user.props file then they can be initiated in your current session by;

loginUserProperty = "hcss.ia.pal.pool.hsa.haio.login_usr"
loginPasswordProperty = "hcss.ia.pal.pool.hsa.haio.login_pwd"

Finally you must of course save your observation to disk using the saveProduct(myObsContext, pool=myPool, tag='My tag') command described earlier in this section.

4.1.3. Querying the SPIRE Observation Log

The SPIRE Observation Log is a file containing details all SPIRE observations made during the Herschel mission lifetime including obsid, target, mode, position and all instrument and AOR parameters. The SPIRE Observation Log is included within the HIPE build and can be queried and interrogated via a specialised script; Query SPIRE Observation Log, acessed from the Useful Scripts menu in HIPE as shown in Figure 3.1. The SPIRE Observation Log is queried by creating an instance of the log, e.g. obsLog = SpireObsLogProduct().

The log can be queried on many different parameters but perhaps the simplest is the obsid as shown in the example taken from the script below. Using the obsid 1342201139, the log returns the entry for this obsid in the obsidProps variable that can either be accessed from the variables pane in the HIPE window, printed as in the script below, or interrogated for individual parameters such as RA and Dec.

#Load Full SPIRE Observation Log (obsLog)
obsLog = SpireObsLogProduct()
# ------------------------------------------------------------
#Get Properties of a single Obsid
obsid = 1342201139
obsidProps =  obsLog.getObsidInfo(obsid)
# Print source position
print "RA, Dec: ",obsidProps['ra'],['dec']
print "Target: ",obsidProps['source']
# Print full contents of log for this obsid
print "SPIRE Observation log entry for obsid %i:"%(obsid)
for key in obsidProps.keySet():
   print "%10s: %s"%(key, obsidProps[key])
# ------------------------------------------------------------

In addition to a single obsid, the observation log can also be queried on other parameters such as a source name as shown in the code example below. The log is queried on the source ngc1275 and the results stored in the variable subObsLog1. Right-clicking on subObsLog1 in the variables pane in the HIPE window shows all the observations retrieved using this search parameter as shown in Figure 4.5. Alternatively, specific results for items, e.g. obsid or observation mode can be retrieved by selecting individual columns in the information retrieved from the log as shown in the code snippet below.

#Load Full SPIRE Observation Log (obsLog)
obsLog = SpireObsLogProduct()
# ------------------------------------------------------------
subObsLog1 = obsLog.filterBySourceName('ngc1275')
# List all obsids recovered from obsLog
obsidList =  subObsLog1["obsLog"]["obsId"].data
print "List of obsids from query:"
print obsidList
# List all observation modes recovered from obsLog
obsModeList = subObsLog1["obsLog"]["obsMode"].data
print "List of observing modes from query:"
HIPE> [1342203614,1342249054,1342249055]
HIPE> ["SpirePhotoSmallScan","SpireSpectroPoint","SpireSpectroPoint"]
# ------------------------------------------------------------
print obsModeList

Query the SPIRE Onsevation Log on source name.

Figure 4.5. Query the SPIRE Onsevation Log on source name.

The log can also be searched on position (RA, Dec) using a search cone as shown in the code example below. The log is searched at position RA=187.2 degrees and Dec=12.0 degerees within a search radius of 2 degrees. The log returns all observations whose centres are within a 2 degree radius of the search position.

#Load Full SPIRE Observation Log (obsLog)
obsLog = SpireObsLogProduct()
# ------------------------------------------------------------
# Filter by radius and print sources found
ra = 187.2      # position RA (deg)
dec = 12.0      # position dec (deg)
radius = 2.0    # search radius (deg)
subObsLog2 = obsLog.filterByRadius(ra,dec,radius)
sourceList =  subObsLog2["obsLog"]["source"].data
print "Targets at RA=%.3f deg, dec=%.3f deg in search radius %.3f deg:"%(ra,dec,radius)
print sourceList
# ------------------------------------------------------------
HIPE> Targets at RA=187.200 deg, dec=12.000 deg in search radius 2.000 deg:
HIPE> ["V2","V2","M86","NGC4435/4438-1","V2","V2","NGC 4459","V2","V2","NGC4388","V2","V2","NGC4325","IC 3467","NGC4459","NGC4388"]

There are also methods available to query the log on specific observation modes. For example, to select all sparse sampled spectrometer observations, the code example below can be used.

#Load Full SPIRE Observation Log (obsLog)
obsLog = SpireObsLogProduct()
# ------------------------------------------------------------
# Filter by observation mode
# Select sparse sampled spectrometer observations
subObsLog3 = obsLog.filterSpecSparse()
# Print first obsid in the list
print  "First obsid in list = ",subObsLog3["obsLog"]["obsId"].data[0]
# ------------------------------------------------------------
HIPE>First obsid in list =  1342183470L