1.59. doOffSubtract

Full Name: herschel.hifi.pipeline.generic.DoOffSubtractTask
Alias: doOffSubtract
Type: Java Task - Java Task
Import: from herschel.hifi.pipeline.generic import DoOffSubtractTask

HIFI/Pipeline/Level 1 Pipeline


Computes the differences between ON and OFF positions in a way depending on the observing mode.

  • Either, the values found in a baseline calibration product are suitably interpolated in time (obsTime) and subtracted from the source measurements (FSwitch-, LoadChop-, MappingOTF-modes.)

  • Or, the off position also contains signal but with reversed optical paths (and standing waves with opposite sign) so that the ON and the OFF datasets are basically averaged on a scan-by-scan basis (DBS- and FastDBS-modes).

  • Or, the the off scans are subtracted on a scan-by-scan basis (PositionSwitch-mode)

A validation mechanism checks the spectra to be subtracted from each other and, if needed, sets in the result data a row flag. The default mechanism inspects the mixer currents (columns 'MJC_hor' or 'MJC_Ver', respectively) and tests their relative deviation (of the ON and the OFF spectra) to be less than a given tolerance. Once the tolerance is exceeded, for the result spectrum a row flag (bit 15) is set. The tolerance can be set by setting the 'validatorTolerance'-parameter. By default it is set to 0.025. This default checking mechanism can be overruled by passing as 'validator'-parameter an instance of the {@link DataValidator} interface. When setting this parameter to None, no checking is done.

A filter options allows to ignore flagged spectra found in the baselines. Two parameters are typically set here:

  • Boolean parameter ignoreFlagged: if set to true flagged scans (rowflag) contained in the baseline are ignored.

  • bits : Specify the bits the rowflag should be tested for. Once one of the given bits are set in the flag value the corresponding scan is ignored. Either you can specify an integer value or a (Py-)list of such.

Note that if the result of this filtering would be that no baseline would be left no filtering is done.


Example 1: In HIPE:
# the mode typically used for DBS modes
htp = doOffSubtract(htp=spectra, mode='addOff')
Example 2: In HIPE:
# baseline: a CalOffBaseline-product typically generated by the MkOffSmooth step
htp = doOffSubtract(htp=spectra, mode='subtractBaseline', cal=baseline)
Example 3: In HIPE:
# baseline: a CalOffBaseline-product typically generated by the MkOffSmooth step
htp = doOffSubtract(htp=spectra, mode='subtractBaseline', interpolator='NEXT', cal=baseline)
Example 4: In HIPE:
# the validator tolerance is re-set.
htp = doOffSubtract(htp=spectra, cal=baseline, validatorTolerance=0.1)
Example 5: In HIPE:
# validation is switched off.
htp = doOffSubtract(htp=spectra, cal=baseline, validator=None)
Example 6: In HIPE:
# filter out scans in the OFF datasets with non-zero row flags.
htp = doOffSubtract(htp=spectra, cal=baseline, ignoreFlagged=True)
Example 7: In HIPE:
# filter out scans in the OFF datasets with row flags that at least one of the specified bits set.
htp = doOffSubtract(htp=spectra, cal=baseline, ignoreFlagged=True, bits=[8,15])
Example 8: In HIPE:
# an alternative validation mechanism is set (here based, as dummy, on the LoFrequency)
from herschel.hifi.pipeline.generic.dorefsubtract import SubtractionDataValidatorImpl
validator = SubtractionDataValidatorImpl()
validator.configure('LoFrequency', 0.001, True)
htp = doOffSubtract(htp=spectra, cal=baseline, validator=validator)

API details


Object validatorTolerance [INPUT, OPTIONAL, default=0.025]

The tolerance to be used for the default validator that checks the difference in the mixer currents and sets a row flag once the mixer currents of the ON and OFF deviate by more than the tolerance from each other. A relative distance measure is used to quantify the deviation (2abs(x1-x2)/(abs(x1)+abs(x2))). Alternatively, a calibration object of type GenericPipelineCalProductcan be passed from which the band-specific tolerances will be extracted. Yet another possibility is to pass the whole calibration context - the task retrieves the appropriate product with the tolerances. Furthermore, you can use the parameter 'calibration' for that purpose. Once a product of type GenericPipelineCalProduct is obtained, the appropriate tolerance is looked up from a table (associated with the given date) from a column named 'doOffSubtract'.

DataValidator validator [INPUT, OPTIONAL, default=no default value]

Custom validator that can be passed to the task to check source and ref that is subtracted from each other and flag data once the tolerance exceeds a given tolerance. Note that the validatorTolerance cannot be used to specify this threshold. Furthermore, the default validator is overwritten once a validator or None is passed here.

MapContext calibration [INPUT, OPTIONAL, default=no default value]

Parameter to pass a calibration context from which all the calibration input, here just mixer current tolerances, can be retrieved.

Boolean useCalTree [INPUT, OPTIONAL, default=no default value]

Parameter to specify whether the calibration tree should be used as primary source of information. It plays a role for task parameters that can explicitly be set as task parameter value (here, for the validatorTolerance-parameter) or, alternatively, retrieved from the calibration tree (see parameter calibration). If useCalTree=True but no reference to the calibration tree is provided the validatorTolerance parameter is used.

Boolean ignoreFlagged [INPUT, OPTIONAL, default=False]

If true it ignores the scans in the baseline data with row flag that have one of the bits set (see the bits-parameter). If the bits is not set all the non-zero row flags are ignored.

Object bits [INPUT, OPTIONAL, default=no default value]

Once one of these bits is set in the row flags and the ignoreFlagged is set to true the associated scan in the baseline data is ignored. Note that this option only applies in combination with the mode=subtractBaseline. The bits are specified as an integer or a list of integers.

See also


  • 2007-06-28 - Melchior: New version.
  • 2007-11-29 - Melchior: Javadoc updated.
  • 2008-04-02 - Melchior: Handle different LoFrequency groups.
  • 2009-08-10 - Melchior: Add interpolation parameter.
  • 2009-09-23 - Melchior: Validation mechanism.
  • 2009-11-24 - Melchior: Filter for row flags with specific bits set.
  • 2011-08-14 - Melchior: Renamed to DoOffSubtractTask
  • 2012-01-16 - Melchior: Mixer current deviation row flag bit set to RowMask.MCD_OFF.getIndex() --> URM adjusted