1.3. How data are stored on your disk

This section describes where HIPE keeps Herschel data on your disk. Note that you do not need to know where a certain data product or observation is stored, because you can use HIPE to search through all the Herschel data stored on your computer. See Section 1.7 for more information.

Data from the Herschel Science Archive. Observations you download from the Herschel Science Archive are stored in tar files. Once you unpack an observation and load it into HIPE, the contents are indexed and referenced by a special data pool called MyHSA .

If you get your data from the Herschel Science Archive via the getObservation command, with the useHsa=True and save=True options, the data are stored directly in the MyHSA pool. See Section 1.4.5 for more details.

You can set the directory where the MyHSA pool is stored by choosing Edit → Preferences and clicking MyHSA in the left-hand side list. The repository is kept by default in a MyHSA directory inside the .hcss directory.

Note that MyHSA contains only unmodified data downloaded from the Archive. It is like your local copy of the Herschel Science Archive with the observations of interest to you. Reprocessed data are stored elsewhere.

	Warning
	Do not change the contents of the `MyHSA` directory manually. Always handle your data through HIPE, as described in this chapter.

Reprocessed observations. Observations and data products you save after reprocessing are stored into pools , which are grouped into storages . A pool is a repository you can use to save, load and search observations and data products. A storage groups one or more pools. Every pool must be registered to a storage. A common situation is a storage containing a single pool. If a storage contains more than one pool, only the first registered pool is accessible for writing.

Pools and storages. All pools must be registered to a storage. A pool can be registered to more than one storage. A common situation is a storage containing a single pool.

Figure 1.2. Pools and storages. All pools must be registered to a storage. A pool can be registered to more than one storage. A common situation is a storage containing a single pool.

There are many types of pools, for handling local and remote data. The local pool , or local store , is probably the one you will use most often. As the name suggests, this pool is held locally on your system, usually in a .hcss/lstore directory under your home directory. Although products are stored as FITS files, you should use the graphical tools provided by HIPE and described in this chapter (see for instance Section 1.7 ) rather than manipulating the files directly.

	Note
	Local pools are also called local stores for historical reasons, but they are pools , not storages .

See Section 1.3.1 for more information on how to manage pools and storages on your system.

This chapter contains all you need to know to use pools and storages in most situations. If you want to delve deeper and learn how to manage pools and storages from the command line, see the Scripting Guide : Chapter 7.

1.3.1. Managing storages and pools

Storages and pools are the two tools with which you can store and retrieve data on your computer (see Section 1.3 for more details). With the Storages & Pools panel in the Preferences dialogue window (see Figure 1.3 ) you can create, delete and associate storages and pools.

To open this window, choose Edit → Preferences or press Alt+Enter , then go to Data Access > Storages & Pools .

Figure 1.3. Preferences for data access

Within this window you can accomplish the following tasks:

Creating and deleting pools. In the Pools area, click Add , enter the pool name and click OK . You can choose what kind of pool to create from the drop-down list. If you are unsure, or if you just want to store data on your local disk, leave the Local Pool default.

For pools fetching remote data (Versant Database Pool and HTTP Pool) you can cache data locally by ticking the Use cache checkbox.

You can set other properties of the new pool (for instance, the directory of a Local Pool) in the parameters table.

Click Apply when you are done.

To delete a pool, select it from the list and click Remove .

To refresh the list of pools, click Refresh . This is useful if, for instance, you copy a pool into your local pool directory while HIPE is running.
Creating and deleting storages. Use the Add and Remove buttons in the Storages area, in the same way as with pools.
Registering pools to storages. Select a pool in the Pools area and drag it to a storage in the Storages area.

By ticking the Add pools/storages from variables automatically checkboxes, any new pools or storages you create elsewhere in HIPE (for instance, from the command line) will appear automatically in this dialogue window.

Note that this does not work the other way around: pools and storages created in this dialogue window will not appear as variables in the Variables view.

	Warning
	Do not remove pools from disk while you have variables in HIPE referring to data in those pools. Since HIPE does not always keep all the contents of a variable in memory, you may not be able to save the variable contents to disk again.