Many of the tools provided in HIPE to help you reduce and analyse your data are accessible from the Spectrum Explorer. Spectral arithmetic and manipulation tools are gathered in the Spectrum Toolbox and are described in this section. Line fitting can be done using the Spectrum Fitter package, which is described in Chapter 7.
When a spectrum is viewed in Spectrum Explorer the Spectrum Toolbox can be opened by clicking on the crossed hammer and screwdriver icon in the toolbar. A task GUI will appear to the right of the spectrum and the spectrum tasks available in the toolbox can be selected from a drop-down menu. Alternatively, you can open the tasks described in this section by clicking on a spectrum in the Variables view and opening the Applicable folder in the Tasks view.
The tasks work on the spectrum (or spectra) that are displayed or
on a selection that you make in the Spectrum Panel. In all cases the
output of the task is of the same
input, for example, if you pass a
task, the output will also be a
In HIPE 13 or later, the execution of tasks using the data displayed in the Spectrum Explorer is asynchronous, that is, the task runs in the background and the GUI remains responsive during all the process. A waiting cursor is displayed when you hover the mouse cursor over GUI elements yet to be refreshed (the Spectrum Explorer) for the duration of the task.
|About tasks that create new data|
For the spectrum and cube toolbox tasks that create new cubes or spectra and at the same time add them to the data selection pane, the focus will pass to the new product. This should be taken into account for additional runs of the same or other tasks, as the new product will be the new input for the task and some of the parameter entries in the task pane may change.
It is worth noting that if you open the tasks using the Spectrum Toolbox then you can open the User Reference Manual (URM) entry in the help for each task by clicking on the small question mark icon that is situated at the bottom right of the task GUI. If you prefer to open the tasks from the Task view then you can open the URM entry by right-clicking on the task name and selecting Help in URM. The URM entries for the spectral tasks, which are linked to the task name in the sections below, provide detailed descriptions and good code examples, therefore in this section we deal only with the GUI usage of the tasks.
It is better to create a variable for the spectrum you wish to
work on than to open up a spectrum from the Context or Observation
Viewer and work on that. Some of the spectrum tasks will not work
you operate from the Context (Observation) Viewer because they
change the data they work on and HIPE will not allow you to change
contents of an
All of the tasks available in the toolbox take advantage of the possibility to make selections on data while applying the task and some tasks can include or exclude ranges of data. In addition many of the tasks provide the possibility to account for flags in data. Therefore, the information in this section is relevant for all of the tasks in the spectrum toolbox.
task can be used to select a subset of spectra
from a dataset, the output from the task is the
same as the input and can be passed to other
spectrum tasks, this can be helpful when dealing
with large datasets. However, the
task can be used by other spectrum tasks whilst
running them. Here we describe how to make
selections on spectra in HIPE in a general sense
and then go on to discuss the usage of the
Selection of spectra is done with the spectrum selection mode of the Spectrum Explorer. Click on the arrow icon in the button bar and then click on a spectrum in the plot area to select it. Continue clicking on other spectra to select more. A selected spectrum is indicated by dot symbols and will automatically be used by the task in the toolbox panel.
All of the spectra in a dataset can be selected by right-clicking in the plot and pressing→ and you can deselect them all with → .
The selected spectrum can be dragged to the Variables view where it is stored as a new variable that can be plotted using PlotXY or splot, see Chapter 3.
The Select task GUI has fields for "selection", "selection_lookup" and "segments". These fields are seen on the GUI forms of all the spectral tasks that allow you to make a selection on your data. Enter indices of datasets to be selected in the "selection" field and segment numbers in the "segment" field, recalling that both start counting from 0. The "selection_lookup" field allows you to make selections on your data without needing to know the indices and make selections based on attributes (such as observation mode) or observation start time in the data is also possible in the command line (see examples in the URM entries). However, so far only HIFI has taken advantage of the ability to include attributes in data.
extract : Extracts data from a minimum to a maximum frequency/wavelength range for the complete set of spectra in a dataset. The spectral ranges to be extracted can be written into the boxes in the GUI but it is easier to draw a range on the spectrum using the select range mode of Spectrum Explorer.
To do so, click on the select one or more ranges icon in the button bar. Click and drag to select ranges in the plot window (the middle mouse button can be used anytime for this as well). This will create a vertical grey bar and automatically enter the start (minimum) and end (maximum) values in the GUI. Drawing a second range adds more rows in the GUI.
Ranges can be resized by clicking near the edge of the marker and dragging the edge of the marker to the desired position. Ranges can be removed from the GUI and the plot by right-clicking on the drawn range and selecting or from the marker menu. The right-click menu also gives you the option to change the colour of the marker and of the line.
When using the Extract GUI, clicking accept after drawing ranges will produce a new spectrum consisting only of the data in the ranges drawn on the original spectrum.
flagPixels : Flags pixels in a spectrum. A prerequisite for this task to work is that the data should contain flag values, which may not be the case for some PACS and SPIRE data. If you try to run the task and you see the error message:
java.lang.RuntimeException: No flag arrays included in data. Prepare the data properly.
that means you have no flag array and this task will not work on your data.
Another pre-requisite is that the task should be able to
modify the data (as flag values are overwritten); this means that
you cannot use this task on a dataset that is still embedded in an
tree, instead you should create a variable of
the dataset and work on that.
If no selection is made on the displayed spectrum, the task
will apply new flag values to the entire spectrum. To select points
(or pixels) to flag use the
mode: a single click will select a point, a
box drawn around a range of points will select the points in that
range. Several selections can be made this way, simply continue to
draw boxes around data you wish to flag. Right clicking on one set
of selected points and choosing the
will remove all
the selections from the plot. You can remove one selection by
drawing another box around it, drawing a box while holding
down the control key will extend the point selection. The
task will pick up any point selections in the Spectrum Explorer
with no need to enter anything into the task GUI.
Instead of selecting a range on the spectrum you also have the option of defining a "mask" parameter, which allows you to define regions by subband and channel number (or index number) for flagging. This is more usefully used in a script than when working with the GUI and the interested reader is referred to the URM entry for further information.
You can select the flag to use from the "flag" field. How this field is populated is instrument-dependent. The task tries to identify what instrument the data comes from and offer only the flags for that instrument. If the instrument cannot be identified then all of the flags in the build you are using will be offered. This means that if you are using a version of HIPE with only PACS installed you would only see the PACS flags but if you were using HIPE with all three instruments installed and the task could not identify what instrument your data belongs to then you will see the flags for HIFI, PACS and SPIRE.
Some flags that are set by the flagPixel task may be recognised by pipelines for some instruments, but not in all cases. Flag values that are recognised by instrument pipelines or instrument-specific data reduction tools are documented in the instrument Data Reduction Guides. There is no general 'ignore' or 'bad data' flag.
Whilst flagging data, you can optionally set the flux value of the flagged pixels to NaN by checking the "setFluxToNaN" box.
Note that it is possible to flag data from the Spectrum Panel
plot without requiring recourse to the task GUI. After selecting
points to be flagged using the select points mode you can
right-click on the selected points and choose
flag and remove
menu. This approach provides lists of HIFI and
SPIRE flags and a manual option with which one
may assign an integer value for a flag. the
flag and remove
option is equivalent to the
option in the
add, subtract, multiply, divide : Adds/subtracts/multiplies/divides a scalar value that should be entered in the param field to/from all spectra in the selected dataset. If your data contains multiple spectra and/or segments you can select a subset to work on by following the instructions in Section 5.4.2.
By selecting the pair-wise option from the drop-menu, adds/subtracts/multiplies/divides groups of spectra or single spectra together. With the Pair-wise mode, if your datasets contains multiple spectra, the first spectrum of the datasets will be combined, then the second, etc. As for the scalar mode, you can select a subset of spectra to work on and specify segments.
One note about command line usage: the tasks can also be performed using the common +, -, *, / symbols so:
spectrum_add_2 = add(ds = spectrum, param = 2.0) # # Is equivalent to spectrum_add_2 = spectrum + 2
Example 5.5. Adding spectra using the overloaded method add()
avg : Averages a selection of spectra from a dataset.
Flags and weights for individual channels/pixels can be taken into account using the variant parameter if they available in the spectrum (this will be the case for HIFI data, perhaps not for PACS or SPIRE data). Flagged data can be ignored by the task, note that flags are specific to each instrument so you should refer to the Data Reduction Guide of your instrument to check flag values. See the URM entry for a detailed explanation of the effects of the different options for the variant parameter.
Selections can be applied before running the task, see Section 5.4.2.
In some cases the data, or your selection from the data, may fall into groups. For example, spectra taken at the same sky position, or with the same instrumental tuning. By checking the per_group box you can calculate the average for each of these groups separately.
pairAvg : Averages spectra pairwise from two input spectrum containers. The first spectrum in the first and second container are averaged, and so on. In case the size of the two containers is different, the result will contain a number of point spectra equal to the lowest size. A subset of spectra and segments, if available, within the dataset can be worked on.
accumulate : The accumulate task averages spectra to a common wave scale grid and will automatically resample data if needed. In addition, the accumulate task does not require that the spectra have the same length and will resample overlapping regions of spectra as required. The accumulate task will not average spectra that are not on a common flux scale or spectra that are not taken at the same position, within a given tolerance that you can specify.
The task works on the all spectra that are displayed in the Spectrum Panel or on the selected spectra in the display. In addition you can use the selection task to make a selection and pass that to the accumulate task by dragging it to the selection bullet from the Variables view. In addition, you may specify a range to average over by drawing that range on the plot after enabling the "Select one or more ranges" mode from the Spectrum Explorer button bar.
You may specify how the average is done with the variant parameter in the same way as is done for the avg task. You may also choose which flag to ignore, rather than ignore all flagged data. To identify the flag you must enter its integer value, rather than a flag name, check instrument guides for these.
The task will resample the data if required but you can also specify the frequency grid to be used (the unit parameter refers to the units of the frequency grid) and the resampling method to be used. You are directed to the URM entry for details.
Finally, you can specify the tolerance in wavelength/frequency and position to be used by the task. These are specified in the units of these values in the data.
statistics : Performs statistical operations on the datasets, always calculating the mean, rms, median and percentiles (quantiles). The quantities are automatically calculated for each segment is the data, which is typically one for PACS and SPIRE and may be more for HIFI.
There are two modes of operation available from the drop-down menu:
perChannel: this mode operates per channel. Taking the
mean as an example, if you had, say, ten spectra in your
dataset, this mode would calculate the mean in the first data
from all ten spectra, and then for the
second bin, then the third, and so on.
This mode produces a Product, e.g.
"stats", containing a
for each statistic, which may be plotted
in Spectrum Explorer.
acrossChannel: this mode operates across the range of each segment in
the data for each spectrum in the data. Taking the same example as
above and assuming one segment, the result would be ten mean values.
This mode produces a
that can be inspected in HIPE with the Dataset Viewer, allowing one to
read off the values, and also export to text file, see
Chapter 2. When
using this mode you can also use sigma clipping, by supplying
the clip value and a flag value to be assigned to the clipped
data. Data will be clipped when it is above
clip value *
You can select ranges to include or exclude by drawing a range (see Section 5.4.2) on the plot and choose whether the range is to be included or excluded from the drop-down menu found by clicking on include ranges.
resample : Resamples data using a Trapezoidal or Euler box. When using a box filter you should either supply a frequency grid (grid) that the data should be resampled to as an array of Double1ds, or supply a fixed width (resolution). Note that the resolution you supply is actually twice the width that the data is resampled to. Another option is to use a Gaussian filter, which requires that you set a smoothing width in the kernel field. All resampling modes conserve flux.
By default the density option is set to True (the box is checked) and this means that the flux data is treated as a flux density (per wavescale unit), if set to false the flux is treated as a per wavescale bin quantity (i.e. the integrated flux per bin).
smooth : Transforms the displayed (or selected) spectra via a box or gaussian (of user-selected width) smooth of the spectra in a dataset. You must supply a value for the width parameter. See the URM entry for details of this and the other task parameters.
replace : Replaces all or part of a spectrum with another. This is potentially useful if (part of) one integration among several is "bad" and could safely be replaced with the average of the others. In theory, bad parts of spectra could be replaced with NaNs in the case that analysis tools do not honour flags in the data but this is not tested.
In order to replace one spectrum with another you should plot both and then use the selection mode (see Section 5.4.2) to select the spectrum to be replaced, ds, and the one to replace it with, by.
To replace part of one spectrum by another you should extract
Section 5.4.2) the part of the spectrum you wish to
insert and use that as the dataset in the
Gridding : Used to grid data in a SpectrumContainer into spectral cubes. This is originally a HIFI task and still requires that you install the HIFI build to have access to it but is currently being adapted by SPIRE to produce spectral cubes. For descriptions on usage see the SPIRE and HIFI Data Reduction Guides.
stitch : Stitches together spectra or spectral segments. This is mostly used for HIFI data to stitch subbands. In theory, one could combine overlapping spectra from the same or different instruments into one dataset and stitch them together with this task but this has not yet been systematically tested.
fold : A HIFI-specific tool that folds frequency-switched spectra. The frequency throw is found in the meta data and is picked up automatically by the task.
convertWavescale : Transforms the wavescale between frequency, velocity, wavelength and wavenumber. When converting to velocity you must supply a reference frequency and the units of the reference frequency.
You can use the Spectrum Fitter, which produces the integral under the model of a fitted line as one of its outputs, see Chapter 7.
A command line tool is available with which to calculate the integral under a line. It allows to integrate over user-defined regions of a spectrum and to optionally remove a Polynomial background.
The data is expected to be in the form of a
in the examples below) and the value reurned is a
Double, which cannot be saved to disk unless you
wrap it in a Product but can be used in scripts.
relate to spectra is
described in the
Scripting Guide. A
quick example is given here showing how to extract a
contains only one spectrum, in
the Spectrum Explorer Data Selection Panel you would see only one row
with one box, to get to
associated with that data:
# Note that PointSpectra are counted from 0 and SpectralSegments from 1! segment = MySpectrum.getPointSpectrum(0).getSegment(1)
Example 5.6. Selecting point spectra and segments.
If MySpectrum contained, say, five spectra and you wanted the fourth you would use:
# Note that PointSpectra are counted from 0 and SpectralSegments from 1! segment = MySpectrum.getPointSpectrum(3).getSegment(1)
Example 5.7. Selecting point spectra and segments (second variation).
Some PointSpectra can contain multiple
SpectralSegments, for example, HIFI data contains
one segment per subband, specify the segment number you want.
Before using the integrator you must import it:
from herschel.ia.toolbox.spectrum.integrator import Integrator
Example 5.8. Importing the Integrator class
Integration over a range or ranges:
The ranges (
[windows] in the example) over which to integrate are formatted as [start,
stop, start, stop, ...]. So for one range from a to b: [a, b], and
for n ranges: [a1, b1, a2, b2, ..., an, bn]. The ranges should be
given in the same units as the abscissa data.
Integration over a range or ranges with 1st order background removal:
The background is removed by fitting a 1st order poly in the [masks] areas. The format of [masks] is similar to the [windows] format and, similarly, masks have the same units as the data.
i = Integrator.doIntegration(seg, [windows], [masks])
Example 5.10. Integration over a set of ranges with masking.
Integration over a range or ranges with nth order background removal:
i = Integrator.doIntegration(seg, [windows], [masks], n)
Example 5.11. Integration over a set of ranges with masking and removing up to n levels of background
Weights and errors in datasets are set by the instrument pipelines and are propagated by the spectral arithmetics tasks described above. Datsets of HIFI spectra contain a weights column, while SPIRE datasets contain an error column (and may also contain a weight column but the values contained in it are calculated from the error). PACS does not currently assign any errors or weights in its datasets so PACS users should ensure they exclude weights when running any of the spectral tasks that can consider weight as a variant.
In practice, the spectral arithmetics tasks only propagate weights (w) and uses the standard weight-sigma relation to propagate errors, w = σ-2
Until HIPE 9.0, weight propagation was carried out using a simplistic scheme. From HIPE 9.0 on, the propagation is done such that errors are also correctly propagated for scalar and pair-wise addition, subtraction, multiplication, division and pair-average. A weight propagation scheme that also correctly propagates errors for the remaining tasks was set in place in HIPE 10.0. The table below shows the weight propagation scheme used from HIPE 9.0 on, throughout the subscript 1 refers to the first dataset passed to the task and the subscript 2 to the second.
The terms "flags" and "masks" are used interchangeably in the Herschel project, with HIFI typically favouring the term "flag" and PACS and SPIRE preferring "mask". The table below also shows how the flags/masks are propagated by the spectral tasks.
|Task||Weight propagation scheme||Comment||Flag/mask propagation scheme|
|Pair-wise add/subtract||w = (w 1 * w 2 ) / (w 1 + w 2 )||If the denominator is zero then a zero weight is defined|
|Pair-wise multiply/divide||w = (u 1 * u 2 ) / (u 1 + u 2 ) * f -2||where u k = w k * f k 2 and f is flux. If the denomimator is zero then a zero weight is defined.|
|Scalar multiply/divide||w = w / k 2||where k is the scalar||unchanged|
|Pair-average||w arithmetic mean = 4 * (w 1 * w 2 ) / (w 1 + w 2 ) , w weighted = (w 1 + w 2 )||If the denomimator is zero then a zero weight is defined. The arithmetic mean is calculated with variant="flux", the weighted mean with variant = "flux-weight".|
|Smooth||same smoothing as chosen for the fluxes||bitwise-OR logic|
|Resample||same scheme as used for the fluxes||bitwise-OR logic|