1.5. Loading observations downloaded from the HSA into HIPE

Prerequisites.  You have already downloaded one or more observations as tar files from the Herschel Science Archive as described in Section 1.4.5 and Section 1.4.7 . Alternatively, you have received on-demand reprocessed data as explained in Section 1.15 .

You need to load the observations into HIPE before you can view and reprocess the data. You need to do it only once for each observation you download from the Herschel Science Archive.

[Note] Note

If you downloaded your observation via the command line using the getObservation command with the save=True parameter, you do not need to follow these steps. Your observation has already been loaded into HIPE. See Section 1.7 for information on how to find your observation in HIPE.

GUI method: using the Navigator view

  1. Uncompress the tar file you downloaded from the Herschel Science Archive. One directory is created for each observation contained in the tar file.

    [Warning] Warning

    Some compression applications (especially in non-UNIX operating systems, i.e. Windows) need to convert the line endings of the text files inside the TAR archive from the UNIX standard (using only LF characters). For a common utility that has problems with this, please see this Known Issue describing the issue (and a workaround). You can also use other software such as 7-Zip, or the decompress task in HIPE: Section 1.104, “decompress”

  2. In the Navigator view of HIPE, open the directory that was created when you uncompressed the tar file. You will see an item with a Saturn icon, as shown in the following image.

    A Herschel observation ready to be loaded into HIPE.

    Figure 1.19. A Herschel observation ready to be loaded into HIPE.


  3. Double click on the item with the Saturn icon. HIPE loads and opens the observation.

Where are my data?  Your data are still in the directory created from uncompressing the tar file. The data has been indexed in the My HSA repository, but HIPE has not copied the data elsewhere, which is why you must not delete this directory. You must also not modify the data files directly. For more information on managing your downloads from the Herschel Science Archive, including how to delete unwanted data, see Section 1.6 .

Where do I go from here?  You can now start working on your observation. Once you are finished, to save your modified data see Section 1.10 . To load the original or modified observation in future HIPE sessions, see Section 1.7 .

Command-line method: using getObservation

  1. Uncompress the tar file you downloaded from the Herschel Science Archive. One directory is created for each observation contained in the tar file.

  2. Issue the following command in the Console view of HIPE, where /path/to/dir is the path to the directory containing the observation (for example, /home/joe/joe141729940 ):

    myObs = getObservation(path="/path/to/dir")

    Example 1.17. Load an observation from disk into a new variable.


    For a complete example downloading the observation in tar.gz format from the Herschel Science Archive using Herschel Archive InterOperability subsystem (HAIO), see below:

    from java.nio.file import Files
    
    # Create temporary directory for TAR.GZ download
    tempdir = Files.createTempDirectory("hipe")
    obs = 1342231052
    # Download a TAR.GZ observation using the HAIO and builtin urllib
    strObsPath = str(tempdir)+"/obs.tar.gz"
    import urllib
    urllib.urlretrieve("http://archives.esac.esa.int/hsa/aio/jsp/product.jsp?"+\
    		"OBSID=1342231052&PRODUCT_LEVEL=Auxiliary&COMPRESSION=TARGZ&"+\
    		"PROTOCOL=HTTP", strObsPath)
    # Decompress the file
    outputDir = str(tempdir)+"/obs"
    decompress(strObsPath, outputDir)
    
    # One more step as there is another directory with a random name within outputDir 
    randomDir = os.listdir(outputDir)[0]
    finalPath = outputDir+"/"+randomDir
    
    # Open the tarred observation from disk
    obs = getObservation(path = finalPath)

    Example 1.18. Retrieving an observation from the HSA as a tar.gz and opening it in HIPE.


    HIPE loads the observation, assigning it to variable myObs .

    If the directory contains multiple observations, you must specify the obsid of the observation you want to load:

    myObs = getObservation(path="/path/to/dir", obsid=1342183046)

    Example 1.19. Load an observation from disk, specifying both path and observation ID.


Where are my data?  Your data are still in the directory created from uncompressing the tar file. The data has been indexed in the My HSA repository, but HIPE has not copied the data elsewhere, which is why you must not delete this directory. You must also not modify the data files directly. For more information on managing your downloads from the Herschel Science Archive, including how to delete unwanted data, see Section 1.6 .

Where do I go from here?  You can now start working on your observation. Once you are finished, to save your modified data see Section 1.10 . To load the original or modified observation in future HIPE sessions, see Section 1.7 .