Chapter 1. HIFI Launch Pad

Table of Contents

1.1. Introduction
1.1.1. How to get your data
1.1.2. Looking at your data
1.1.3. Re-pipelining Observations
1.1.4. Common data reduction steps
1.1.5. Dealing with observing mode specific issues
1.1.6. Exporting data

Last updated 3 February, 2014.

1.1. Introduction

Welcome to the HIFI Launch Pad!

The Launch Pad is intended to get you off the ground and into HIFI data reduction quickly. We summarise how you get your data, quickly inspect it, and re-pipeline it (if needed) and point you toward information in the rest of the HIFI Data Reduction Guide and the Herschel Data Analysis Guide about common aspects of HIFI data reduction and analysis. More detailed information regarding specific observing modes can be found in the cookbooks in the following chapters.

If you are new to HIPE it is recommended that you also look at the Quick Start Guide and the HIPE Owners Guide.

1.1.1. How to get your data

Herschel data are stored in ESA’s Herschel Science Archive (HSA):

  • You will need to log into the HSA using the "HSA Log-in" button in the bottom status bar of HIPE (see Figure 1.1) (for more details, see Logging into the HSA)

  • Herschel data are identified with a unique number known as the Observation ID (ObsID) (e.g. 1342212115)

  • You can query the HSA from within HIPE using the Data Access perspective, from the WindowShow Perspectives menu (see Figure 1.1).

  • You can also access the Herschel Science Archive by selecting (left-click) the HSA User Interface button (see Figure 1.2).

  • HIPE expects the data in the form of a pool. A pool is like a database, with observations organised in an Observation Context, containing links to all data, calibration, and supplementary files.

HSA Log-in and Herschel Science Archive User Interface from menu

Figure 1.1. HSA Log-in and Herschel Science Archive User Interface from menu


Herschel Science Archive User Interface from button

Figure 1.2. Herschel Science Archive User Interface from button


...from the HSA

There are several ways to get your data from the HSA, they are described in detail in the first chapter of the Herschel Data Analysis Guide (DAG) and summarised here.

  1. To get multiple observations, download a tar file (not a pool) using the shopping basket (see details in the DAG first chapter)

  2. To get one observation you can directly access the HSA:

    • via the command line

    obs=getObservation(ObsID, useHsa=True)
    
    # example
    
    obs=getObservation(1342212115, useHsa=True)
    

    Note that if you are not logged into HSA, a login window will pop-up, allowing you to log in.

    If you find that you have trouble obtaining the latest version of your observation it may be that you have it cached on your disk. In that case, try:

     obs=getObservation(ObsID, useHsa=True, useCache=False)
    
    • or you can use the “Send to external application” in the HSA User Interface.

If you directly access the HSA the data is not automatically saved to your disk and you should immediately save your observation with:

saveObservation(obs, poolName="myPool", saveCalTree=True)

...from disk

Once the data are saved on your hard disk, the Observation Context can be read into HIPE using:

myobs=getObservation(ObsID, poolName=”myPool”)

If you try to access this pool in the same HIPE session in which you saved it, you must update your list of local pools before HIPE will be able to find it. To do this, run the following command before running getObservation:

PoolManager.getInstance().propertyChange(java.beans.PropertyChangeEvent \
(PoolCreatorFactory.getCreators().get("lstore"), \
LocalStoreFactory.LOCAL_POOL_DEFINITIONS_PROP, None, None))

Alternatively, you can press the Refesh button in Storages and Pools in the HIPE Preferences.

...from a tar file

If your data came from a tar file (multiple observations download), uncompress and untar the file, then, from HIPE view Navigator (see HIPE Owners Guide for more details), locate the folder on your disk, and open the content of the folder by a left-click on the plus sign. You will see folders associated to all your observations, an auxiliary folder, a calibration folder, and all your obsIDs with a small image on the left-side of the obsID's name. To import an obsID into HIPE, just double-left-click on the file with an image (see Figure 1.3). The ObservationContext of that obsID will then load in the HIPE view Variables, under Observations.

Note that it is also possible to uncompress and untar your file directly from the HIPE view Navigator - once you locate your tar file, just double-left-click on the .tar.gz file. A task called decompress will open and allow you to unpack you tar file.

Similarly to data accessed from HSA, you can save your observation in a localPool once it is uppacked:

# obsID located in the HIPE view Variables, under Observations
saveObservation(obsID, poolName="myPool")
Importing data into HIPE from a tar file

Figure 1.3. Importing data into HIPE from a tar file


1.1.2. Looking at your data

...A quick look

The best way to quickly inspect your data is to look at the Browse Product in the Observation Summary, which you can click on to enlarge. The browse products show:

  • Point Mode observations: Two plots of Level 2 spectra with the H-polarisation to the left and the V-polarisation to the right. The upper and lower axes of the plots show the LSB and USB frequency scale, respectively. The AOR label and observation number are used to title the plot. The observing mode, source name, requested RA and dec are below the plot title.

  • Mapping observations: The map average spectra for the WBS-H (top) and WBS-V (bottom) is plotted per subband beside the integrated map of that subband. The upper and lower axes of the spectral plots show the LSB and USB frequency scale, respectively. The plots are arranged in order of increasing frequency. The integrated maps created with no correction done for any baseline issues. The x- and y- axes show the RA and declination, respectively, while the auxiliary axes show pixel coordinates. The colour scale used for the image is 'heat' and the intensity scale used is 'ramp' so the strongest emission in the map appears white.

    The AOR label and observation number are used to title the browse product. The observing mode, source name, requested RA and dec are below the plot title.

  • Spectral Scans: The single sideband solution after deconvolution of the Level 2 WBS spectra. No baseline correction has been done prior to deconvolution. The H-polarisation is shown to the left and the V-polarisation to the right. The AOR label and observation number are used to title the plot. The observing mode, source name, requested RA and dec are below the plot title.

...A deeper look

You can navigate through the Observation Context using the ObservationViewer, see Chapter 3 for more information about the contents of the Observation Context.

SpectrumExplorer is the main tool to plot and inspect spectra.

  • It can be started by right mouse clicking on a SpectrumDataset or product (e.g. HTP ) and selecting Open WithSpectrum Explorer. When viewing the contents of an HTP you must click in the box next to the variable name to plot the spectra.

  • The selector panel appears at the bottom, with a row for each scan and a box (initially grey) for each subband. Clicking on each square will plot the spectrum for that subband, double-clicking will remove the plot.

  • Tooltips describe the actions possible from the button bar above the plot.

1.1.3. Re-pipelining Observations

When should I reprocess my data?

The HIFI ICC recommends that your data should ideally be processed using the HIPE version that you are using for data analysis to ensure that you do not run into data-software compatibility problems. However, it is not anticipated that you will have problems if you keep within one HIPE version, e.g. perform data analysis in HIPE 13 on data that was pipelined with HIPE 12. In some cases we may recommend that certain observing modes are processed with a particular HIPE version, see the HIFI Instrument and Calibration page .

How do I reprocess my data?

To reprocess (re-run the pipeline) data you can find hifiPipeline in the Applicable Tasks menu when you click on an Observation Context in the Variables View. Select the spectrometers and the levels between which you want to reprocess for and hit accept. Or, in the command line:

# Create a new Observation Context called newobs by reprocessing all levels of the Observation Context “obs”
newobs = hifiPipeline(obs = obs, save=False)
#
# To just reprocess from Level 1 to Level 2, and only for the HRS
newobs = hifiPipeline(obs = obs, fromLevel=1, upToLevel=2, apids = ['HRS-H', 'HRS-V'], save=False)

See Chapter 5 for more details.

How do I use newer calibration data?

To re-pipeline data using the latest calibration version available from the HSA, rather than the calibration data that was shipped with your data, you must first configure the pipeline to go to the HSA for the calibration data. To do so, first run the configureHifiPipeline task:

calconfig = configureHifiPipeline(useHsa=True)

You can tell the pipeline to use the newer calibration by checking the cal box in the GUI and by passing the variable calconfig to the palStore bullet, before running the pipeline.

In the command line this is all done as follows:

calconfig = configureHifiPipeline(useHsa=True)
obs_1 = hifiPipeline(obs=obs, cal=1, palStore=calconfig)

See Section 5.3 for more details.

1.1.4. Common data reduction steps

1.1.4.1. Correcting for Standing Waves

  • Standing waves can be fitted with sine waves and subtracted using FitHifiFringe, which appears as an applicable task when one clicks on the Observation Context variable.

  • On the command line, it is typically done as follows:

    fitHifiFringe(data=obs,nfringes=2,product='WBS-V-LSB')
    
  • See Chapter 12 for more details.

1.1.4.2. Correcting for Baseline Drifts

  • Baselines can be easily removed using the fitBaseline task. To create a new observation context with baseline corrected Level 2 products, use:

    obs_fit = fitBaseline(data=obs)
    

    and follow the instructions that appear in the console. For more information, see Section 13.3

1.1.4.3. Spectrum Arithmetics and Manipulation

When a spectrum is selected in the Variables View, HIPE becomes the spectrum toolbox; a range of tools for performing spectral arithmetics and manipulation become automatically available.

  • View a spectrum in Spectrum Explorer and click on the crossed hammer and spanner icon in the toolbar above, the toolbox will appear to the right of the plot. Select the task you wish to perform from the drop-down menu and complete the GUI form.

  • The spectrum tasks available in the toolbox can also be used in scripts or in the command line. The simplest way to find the syntax is to copy the command from the console after using the GUI. Here, we give a couple of examples.

    #
    # First extract a spectrum from the Observation Context, for example the Level 2 WBS-H-USB
    spectrum=obs.refs["level2"].product.refs["WBS-H-USB"].product.refs["box_001"].product["0001"]
    #
    #Stitch subbands together:
    StitchedSpectrum = stitch(ds=spectrum, variant="crossoverPoints", edgeTolerance=0.01,stepsize=0.0)
    #
    #extract a range in frequency (the ranges are given in the units in the spectrum)
    ExtractedSpectrum = extract(ds=spectrum,ranges=[(555.0, 558.7)])
    

    The use of all the tasks is described generally in Spectrum Toolbox of the Herschel Data Analysis Guide where you will find more information. Some HIFI-specific examples can also be found in Chapter 17.

1.1.4.4. Fitting spectra

Spectral features can be fitted and removed using the SpectrumFitterGUI , which is available from the Spectrum Explorer. The tool can be used to interactively fit a single spectrum, or in automatic mode on multiple spectra using a previously defined model.

  • To interactively fit a spectrum, extract the region of interest or stitch all the subbands of the data together so that you have only one segment of data (this is one box in the Spectrum Explorer selector panel).

1.1.5. Dealing with observing mode specific issues

Interactive tasks and data issues that are particular to given observing modes are described in a series of cookbooks in Chapter 2:

1.1.6. Exporting data

1.1.6.1. Exporting Data to CLASS

  • HIFI data can be exported to CLASS readable FITS files using the HiClass task. You may wish to stitch subbands together before exporting to CLASS.

    #  Export one dataset to a FITS file:
       HiClassTask()(data = myspectra, fileName = 'myspectra.fits')
    #
    #  Export one HIFI timeline product to a FITS file:
          HiClassTask()(data = myhtp, fileName = 'myhtp.fits')
    #  Export the Level 2 spectra to a FITS file by supplying the ObservationContxt:
          HiClassTask()(data = obs, fileName = 'myl2spectra.fits')
    

    The FITS file is written in the directory you started HIPE from unless you specify a path.

  • To read the FITS file into CLASS:

        file out MyHIFISpectra.hifi mul
        fits read MyHIFISpectra.fits
    #
    # Now you have a CLASS file named MyHIFISpectra.hifi (you can use whatever you want as an
    # extension) that you can access like you always do in CLASS:
    #
        file in MyHIFISpectra.hifi
        find
        get first
        set unit f i
        device image white
        plot
      
  • Note that you need to use a recent version of CLASS, i.e. the version from April 2010 onwards.

  • See Chapter 23 for detailed information and examples on how to use the task.

1.1.6.2. Exporting Data to FITS

Use the simpleFitsWriter to save spectra to FITS file.

simpleFitsWriter(product=spectrum, file='spectrum.fits')

1.1.6.3. Exporting Data to ASCII

Use the exportSpectrumToAscii task available from the Spectrum Toolbox to save spectra to text file.

exportSpectrumToAscii(ds=spectrum, file='spectrum.txt')