5.4. Working on Spectra

Many of the tools provided in HIPE to help you reduce and analyse your data are accessible from the Spectrum Explorer. Spectral arithmetic and manipulation tools are gathered in the Spectrum Toolbox and are described in this section. Line fitting can be done using the Spectrum Fitter package, which is described in Chapter 7.

5.4.1. Using the Spectrum Toolbox

When a spectrum is viewed in Spectrum Explorer the Spectrum Toolbox can be opened by clicking on the crossed hammer and screwdriver icon in the toolbar. A task GUI will appear to the right of the spectrum and the spectrum tasks available in the toolbox can be selected from a drop-down menu. Alternatively, you can open the tasks described in this section by clicking on a spectrum in the Variables view and opening the Applicable folder in the Tasks view.

The tasks work on the spectrum (or spectra) that are displayed or on a selection that you make in the Spectrum Panel. In all cases the output of the task is of the same class as the input, for example, if you pass a Spectrum1d to a task, the output will also be a Spectrum1d.

[Note] Task execution

In HIPE 13 or later, the execution of tasks using the data displayed in the Spectrum Explorer is asynchronous, that is, the task runs in the background and the GUI remains responsive during all the process. A waiting cursor is displayed when you hover the mouse cursor over GUI elements yet to be refreshed (the Spectrum Explorer) for the duration of the task.

[Note] About tasks that create new data

For the spectrum and cube toolbox tasks that create new cubes or spectra and at the same time add them to the data selection pane, the focus will pass to the new product. This should be taken into account for additional runs of the same or other tasks, as the new product will be the new input for the task and some of the parameter entries in the task pane may change.

It is worth noting that if you open the tasks using the Spectrum Toolbox then you can open the User Reference Manual (URM) entry in the help for each task by clicking on the small question mark icon that is situated at the bottom right of the task GUI. If you prefer to open the tasks from the Task view then you can open the URM entry by right-clicking on the task name and selecting Help in URM. The URM entries for the spectral tasks, which are linked to the task name in the sections below, provide detailed descriptions and good code examples, therefore in this section we deal only with the GUI usage of the tasks.

[Note] Warning

It is better to create a variable for the spectrum you wish to work on than to open up a spectrum from the Context or Observation Viewer and work on that. Some of the spectrum tasks will not work if you operate from the Context (Observation) Viewer because they change the data they work on and HIPE will not allow you to change the contents of an ObservationContext in this way.

5.4.2. Spectral Selection: extraction, and flagging

All of the tasks available in the toolbox take advantage of the possibility to make selections on data while applying the task and some tasks can include or exclude ranges of data. In addition many of the tasks provide the possibility to account for flags in data. Therefore, the information in this section is relevant for all of the tasks in the spectrum toolbox.

  • Select : The select task can be used to select a subset of spectra from a dataset, the output from the task is the same as the input and can be passed to other spectrum tasks, this can be helpful when dealing with large datasets. However, the select task can be used by other spectrum tasks whilst running them. Here we describe how to make selections on spectra in HIPE in a general sense and then go on to discuss the usage of the select task.

    Selection of spectra is done with the spectrum selection mode of the Spectrum Explorer. Click on the arrow icon in the button bar and then click on a spectrum in the plot area to select it. Continue clicking on other spectra to select more. A selected spectrum is indicated by dot symbols and will automatically be used by the task in the toolbox panel.

    All of the spectra in a dataset can be selected by right-clicking in the plot and pressing SelectSelect all and you can deselect them all with SelectSelect none.

    The selected spectrum can be dragged to the Variables view where it is stored as a new variable that can be plotted using PlotXY or splot, see Chapter 3.

    The Select task GUI has fields for "selection", "selection_lookup" and "segments". These fields are seen on the GUI forms of all the spectral tasks that allow you to make a selection on your data. Enter indices of datasets to be selected in the "selection" field and segment numbers in the "segment" field, recalling that both start counting from 0. The "selection_lookup" field allows you to make selections on your data without needing to know the indices and make selections based on attributes (such as observation mode) or observation start time in the data is also possible in the command line (see examples in the URM entries). However, so far only HIFI has taken advantage of the ability to include attributes in data.

  • extract : Extracts data from a minimum to a maximum frequency/wavelength range for the complete set of spectra in a dataset. The spectral ranges to be extracted can be written into the boxes in the GUI but it is easier to draw a range on the spectrum using the select range mode of Spectrum Explorer.

    To do so, click on the select one or more ranges icon in the button bar. Click and drag to select ranges in the plot window (the middle mouse button can be used anytime for this as well). This will create a vertical grey bar and automatically enter the start (minimum) and end (maximum) values in the GUI. Drawing a second range adds more rows in the GUI.

    Ranges can be resized by clicking near the edge of the marker and dragging the edge of the marker to the desired position. Ranges can be removed from the GUI and the plot by right-clicking on the drawn range and selecting Remove or Remove all from the marker menu. The right-click menu also gives you the option to change the colour of the marker and of the line.

    When using the Extract GUI, clicking accept after drawing ranges will produce a new spectrum consisting only of the data in the ranges drawn on the original spectrum.

  • flagPixels : Flags pixels in a spectrum. A prerequisite for this task to work is that the data should contain flag values, which may not be the case for some PACS and SPIRE data. If you try to run the task and you see the error message:

    java.lang.RuntimeException: No flag arrays included in data. Prepare the
    data properly.
    

    that means you have no flag array and this task will not work on your data.

    Another pre-requisite is that the task should be able to modify the data (as flag values are overwritten); this means that you cannot use this task on a dataset that is still embedded in an ObservationContext tree, instead you should create a variable of the dataset and work on that.

    If no selection is made on the displayed spectrum, the task will apply new flag values to the entire spectrum. To select points (or pixels) to flag use the select points mode: a single click will select a point, a box drawn around a range of points will select the points in that range. Several selections can be made this way, simply continue to draw boxes around data you wish to flag. Right clicking on one set of selected points and choosing the deselect option under Point selection will remove all the selections from the plot. You can remove one selection by drawing another box around it, drawing a box while holding down the control key will extend the point selection. The flagPixel task will pick up any point selections in the Spectrum Explorer GUI with no need to enter anything into the task GUI.

    Instead of selecting a range on the spectrum you also have the option of defining a "mask" parameter, which allows you to define regions by subband and channel number (or index number) for flagging. This is more usefully used in a script than when working with the GUI and the interested reader is referred to the URM entry for further information.

    You can select the flag to use from the "flag" field. How this field is populated is instrument-dependent. The task tries to identify what instrument the data comes from and offer only the flags for that instrument. If the instrument cannot be identified then all of the flags in the build you are using will be offered. This means that if you are using a version of HIPE with only PACS installed you would only see the PACS flags but if you were using HIPE with all three instruments installed and the task could not identify what instrument your data belongs to then you will see the flags for HIFI, PACS and SPIRE.

    Some flags that are set by the flagPixel task may be recognised by pipelines for some instruments, but not in all cases. Flag values that are recognised by instrument pipelines or instrument-specific data reduction tools are documented in the instrument Data Reduction Guides. There is no general 'ignore' or 'bad data' flag.

    Whilst flagging data, you can optionally set the flux value of the flagged pixels to NaN by checking the "setFluxToNaN" box.

    Note that it is possible to flag data from the Spectrum Panel plot without requiring recourse to the task GUI. After selecting points to be flagged using the select points mode you can right-click on the selected points and choose flag or flag and remove from the PointSelection menu. This approach provides lists of HIFI and SPIRE flags and a manual option with which one may assign an integer value for a flag. the flag and remove option is equivalent to the setFluxToNaN option in the flagPixel task.

5.4.3. Spectrum Arithmetics

  • add, subtract, multiply, divide : Adds/subtracts/multiplies/divides a scalar value that should be entered in the param field to/from all spectra in the selected dataset. If your data contains multiple spectra and/or segments you can select a subset to work on by following the instructions in Section 5.4.2.

    By selecting the pair-wise option from the drop-menu, adds/subtracts/multiplies/divides groups of spectra or single spectra together. With the Pair-wise mode, if your datasets contains multiple spectra, the first spectrum of the datasets will be combined, then the second, etc. As for the scalar mode, you can select a subset of spectra to work on and specify segments.

    One note about command line usage: the tasks can also be performed using the common +, -, *, / symbols so:

    spectrum_add_2 = add(ds = spectrum, param = 2.0)
    #
    # Is equivalent to
    spectrum_add_2 = spectrum + 2
        

    Example 5.5. Adding spectra using the overloaded method add()


5.4.4. Spectral Averaging and Statistics

  • avg : Averages a selection of spectra from a dataset.

    Flags and weights for individual channels/pixels can be taken into account using the variant parameter if they available in the spectrum (this will be the case for HIFI data, perhaps not for PACS or SPIRE data). Flagged data can be ignored by the task, note that flags are specific to each instrument so you should refer to the Data Reduction Guide of your instrument to check flag values. See the URM entry for a detailed explanation of the effects of the different options for the variant parameter.

    Selections can be applied before running the task, see Section 5.4.2.

    In some cases the data, or your selection from the data, may fall into groups. For example, spectra taken at the same sky position, or with the same instrumental tuning. By checking the per_group box you can calculate the average for each of these groups separately.

  • pairAvg : Averages spectra pairwise from two input spectrum containers. The first spectrum in the first and second container are averaged, and so on. In case the size of the two containers is different, the result will contain a number of point spectra equal to the lowest size. A subset of spectra and segments, if available, within the dataset can be worked on.

  • accumulate : The accumulate task averages spectra to a common wave scale grid and will automatically resample data if needed. In addition, the accumulate task does not require that the spectra have the same length and will resample overlapping regions of spectra as required. The accumulate task will not average spectra that are not on a common flux scale or spectra that are not taken at the same position, within a given tolerance that you can specify.

    The task works on the all spectra that are displayed in the Spectrum Panel or on the selected spectra in the display. In addition you can use the selection task to make a selection and pass that to the accumulate task by dragging it to the selection bullet from the Variables view. In addition, you may specify a range to average over by drawing that range on the plot after enabling the "Select one or more ranges" mode from the Spectrum Explorer button bar.

    You may specify how the average is done with the variant parameter in the same way as is done for the avg task. You may also choose which flag to ignore, rather than ignore all flagged data. To identify the flag you must enter its integer value, rather than a flag name, check instrument guides for these.

    The task will resample the data if required but you can also specify the frequency grid to be used (the unit parameter refers to the units of the frequency grid) and the resampling method to be used. You are directed to the URM entry for details.

    Finally, you can specify the tolerance in wavelength/frequency and position to be used by the task. These are specified in the units of these values in the data.

  • statistics : Performs statistical operations on the datasets, always calculating the mean, rms, median and percentiles (quantiles). The quantities are automatically calculated for each segment is the data, which is typically one for PACS and SPIRE and may be more for HIFI.

    There are two modes of operation available from the drop-down menu:

    • perChannel: this mode operates per channel. Taking the mean as an example, if you had, say, ten spectra in your dataset, this mode would calculate the mean in the first data bin (or channel) from all ten spectra, and then for the second bin, then the third, and so on. This mode produces a Product, e.g. "stats", containing a Spectrum1d for each statistic, which may be plotted in Spectrum Explorer.

    • acrossChannel: this mode operates across the range of each segment in the data for each spectrum in the data. Taking the same example as above and assuming one segment, the result would be ten mean values. This mode produces a TableDataset that can be inspected in HIPE with the Dataset Viewer, allowing one to read off the values, and also export to text file, see Chapter 2. When using this mode you can also use sigma clipping, by supplying the clip value and a flag value to be assigned to the clipped data. Data will be clipped when it is above clip value * sigma (standard deviation).

    You can select ranges to include or exclude by drawing a range (see Section 5.4.2) on the plot and choose whether the range is to be included or excluded from the drop-down menu found by clicking on include ranges.

5.4.5. Spectral Manipulation: resampling, smoothing, replacing, gridding, stitching, and folding

  • resample : Resamples data using a Trapezoidal or Euler box. When using a box filter you should either supply a frequency grid (grid) that the data should be resampled to as an array of Double1ds, or supply a fixed width (resolution). Note that the resolution you supply is actually twice the width that the data is resampled to. Another option is to use a Gaussian filter, which requires that you set a smoothing width in the kernel field. All resampling modes conserve flux.

    By default the density option is set to True (the box is checked) and this means that the flux data is treated as a flux density (per wavescale unit), if set to false the flux is treated as a per wavescale bin quantity (i.e. the integrated flux per bin).

  • smooth : Transforms the displayed (or selected) spectra via a box or gaussian (of user-selected width) smooth of the spectra in a dataset. You must supply a value for the width parameter. See the URM entry for details of this and the other task parameters.

  • replace : Replaces all or part of a spectrum with another. This is potentially useful if (part of) one integration among several is "bad" and could safely be replaced with the average of the others. In theory, bad parts of spectra could be replaced with NaNs in the case that analysis tools do not honour flags in the data but this is not tested.

    In order to replace one spectrum with another you should plot both and then use the selection mode (see Section 5.4.2) to select the spectrum to be replaced, ds, and the one to replace it with, by.

    To replace part of one spectrum by another you should extract (see Section 5.4.2) the part of the spectrum you wish to insert and use that as the dataset in the by bullet.

  • Gridding : Used to grid data in a SpectrumContainer into spectral cubes. This is originally a HIFI task and still requires that you install the HIFI build to have access to it but is currently being adapted by SPIRE to produce spectral cubes. For descriptions on usage see the SPIRE and HIFI Data Reduction Guides.

  • stitch : Stitches together spectra or spectral segments. This is mostly used for HIFI data to stitch subbands. In theory, one could combine overlapping spectra from the same or different instruments into one dataset and stitch them together with this task but this has not yet been systematically tested.

  • fold : A HIFI-specific tool that folds frequency-switched spectra. The frequency throw is found in the meta data and is picked up automatically by the task.

5.4.6. Spectral Unit Conversion

  • convertWavescale : Transforms the wavescale between frequency, velocity, wavelength and wavenumber. When converting to velocity you must supply a reference frequency and the units of the reference frequency.

5.4.7. Finding the integral under a line

You can use the Spectrum Fitter, which produces the integral under the model of a fitted line as one of its outputs, see Chapter 7.

A command line tool is available with which to calculate the integral under a line. It allows to integrate over user-defined regions of a spectrum and to optionally remove a Polynomial background.

The data is expected to be in the form of a SpectralSegment (segment in the examples below) and the value reurned is a Double, which cannot be saved to disk unless you wrap it in a Product but can be used in scripts.

How SpectralSegments relate to spectra is described in the Scripting Guide. A quick example is given here showing how to extract a SpectralSegment from a SpectrumDataset, MySpectrum. Assume MySpectrum contains only one spectrum, in the Spectrum Explorer Data Selection Panel you would see only one row with one box, to get to SpectralSegment associated with that data:

# Note that PointSpectra are counted from 0 and SpectralSegments from 1!
    segment = MySpectrum.getPointSpectrum(0).getSegment(1)
    

Example 5.6. Selecting point spectra and segments.


If MySpectrum contained, say, five spectra and you wanted the fourth you would use:

# Note that PointSpectra are counted from 0 and SpectralSegments from 1!
    segment = MySpectrum.getPointSpectrum(3).getSegment(1)
    

Example 5.7. Selecting point spectra and segments (second variation).


Some PointSpectra can contain multiple SpectralSegments, for example, HIFI data contains one segment per subband, specify the segment number you want.

Before using the integrator you must import it:

from herschel.ia.toolbox.spectrum.integrator import Integrator  
  

Example 5.8. Importing the Integrator class


  • Integration over a range or ranges:

    The ranges ([windows] in the example) over which to integrate are formatted as [start, stop, start, stop, ...]. So for one range from a to b: [a, b], and for n ranges: [a1, b1, a2, b2, ..., an, bn]. The ranges should be given in the same units as the abscissa data.

    i = Integrator.doIntegration(segment, [windows]) 
        

    Example 5.9. Integrating over a set of ranges


  • Integration over a range or ranges with 1st order background removal:

    The background is removed by fitting a 1st order poly in the [masks] areas. The format of [masks] is similar to the [windows] format and, similarly, masks have the same units as the data.

    i = Integrator.doIntegration(seg, [windows], [masks]) 
        

    Example 5.10. Integration over a set of ranges with masking.


  • Integration over a range or ranges with nth order background removal:

    i = Integrator.doIntegration(seg, [windows], [masks], n)
        

    Example 5.11. Integration over a set of ranges with masking and removing up to n levels of background


5.4.8. Weight/error and flag/mask propagation

Weights and errors in datasets are set by the instrument pipelines and are propagated by the spectral arithmetics tasks described above. Datsets of HIFI spectra contain a weights column, while SPIRE datasets contain an error column (and may also contain a weight column but the values contained in it are calculated from the error). PACS does not currently assign any errors or weights in its datasets so PACS users should ensure they exclude weights when running any of the spectral tasks that can consider weight as a variant.

In practice, the spectral arithmetics tasks only propagate weights (w) and uses the standard weight-sigma relation to propagate errors, w = σ-2

Until HIPE 9.0, weight propagation was carried out using a simplistic scheme. From HIPE 9.0 on, the propagation is done such that errors are also correctly propagated for scalar and pair-wise addition, subtraction, multiplication, division and pair-average. A weight propagation scheme that also correctly propagates errors for the remaining tasks was set in place in HIPE 10.0. The table below shows the weight propagation scheme used from HIPE 9.0 on, throughout the subscript 1 refers to the first dataset passed to the task and the subscript 2 to the second.

The terms "flags" and "masks" are used interchangeably in the Herschel project, with HIFI typically favouring the term "flag" and PACS and SPIRE preferring "mask". The table below also shows how the flags/masks are propagated by the spectral tasks.

Task Weight propagation scheme Comment Flag/mask propagation scheme
Pair-wise add/subtract w = (w 1 * w 2 ) / (w 1 + w 2 ) If the denominator is zero then a zero weight is defined  
Scalar add/subtract unchanged   unchanged
Pair-wise multiply/divide w = (u 1 * u 2 ) / (u 1 + u 2 ) * f -2 where u k = w k * f k 2 and f is flux. If the denomimator is zero then a zero weight is defined.  
Scalar multiply/divide w = w / k 2 where k is the scalar unchanged
Pair-average w arithmetic mean = 4 * (w 1 * w 2 ) / (w 1 + w 2 ) , w weighted = (w 1 + w 2 ) If the denomimator is zero then a zero weight is defined. The arithmetic mean is calculated with variant="flux", the weighted mean with variant = "flux-weight".  
Stitch      
Accumulate      
Smooth same smoothing as chosen for the fluxes   bitwise-OR logic
Resample same scheme as used for the fluxes   bitwise-OR logic