1.63. doRefSubtract

Full Name: herschel.hifi.pipeline.generic.DoRefSubtractTask
Alias: doRefSubtract
Type: Java Task - Java Task
Import: from herschel.hifi.pipeline.generic import DoRefSubtractTask
Category

HIFI/Pipeline/Level 1 Pipeline

Description

Subtracts suitable reference spectra that are, depending on the observing mode,

obtained from a reference sky position, cold load or with a switched LO frequency. The way how to identify source and reference measurement scans depends on the particular AOT / observing mode applied.

In all cases, only the science data is involved - but typically both, the ON and the OFF measurements.

  • Position switch reference modes:

    Here the reference position is obtained by moving the telescope to a different blank sky position. In the pipeline processing, this is treated rather with the DoOffSmooth module. So the application of the DoRefSubtract here has no impact.

  • Slow chop DBS modes:

    In ON and OFF datasets, we expect a pattern of the form ref-source-source-ref-... for ON and source-ref-ref-source-source-... for the OFF spectra. The subtraction is carried through pairwise for each subsequent source/ref-pair.

  • Fast chop DBS modes:

    In ON and OFF datasets, we expect a pattern of the form ref-source-ref-source-ref-... for ON and source-ref-source-ref-source-... for the OFF spectra. The subtraction is carried through pairwise for each subsequent source/ref-pair.

  • Load chop modes:

    The reference position is the cold load. Again, a pattern of the form ref-source-source-ref-ref-... is assumed that should be reflected by the Chopper value. Again the subtraction is carried through pairwise for each sub-sequent source/ref-pair.

  • Frequency switch mode: The reference is not a different position but rather a different setting for the LO frequency. Here, a pattern of the form source-ref-ref-source-... is assumed (i.e. starting with 'source', the prime LO frequency) and the subtraction is carried through pairwise for each sub-sequent source/ref-pair.

Different configuration possibilities are available:

  • Assume a pattern of the form A-B-B-A or A-B-A-B: Specify indicator="pattern" and set the isABBA-parameter to true or false, respectively. Furthermore, specify the startsWithRef and the offStartWithOpp-flags.

  • Use the "buffer"-values and map its values to 'source' or 'ref': Specify indicator="buffer" and either

    • set the startsWithRef: in this case, the first value found (within a dataset) is associated with the reference or the source depending on whether it has been set to true or false.

    • specify a dictionary of the form {"source":2,"ref":1} to map the source and ref to specific buffer values.

  • Use the "LoFrequency"- or the "Chopper"-values and map its values to 'source' or 'ref': Specify indicator="LoFrequency" or "Chopper", respectively and either

    • set the startWithRef: in this case, the first value found (within a dataset) is associated with the reference or the source depending on whether it has been set to true or false.

    • specify a dictionary of the form {"source":-4.1,"ref":4.8} to map the source and ref to specific values found in the "LoFrequency" or "Chopper"-column. Note that for these double values, internally set measurement errors are used for the LoFrequency and the ChopperPositions.

A validation mechanism checks the spectra to be subtracted from each other and, if needed, sets in the result data a row flag. The default mechanism inspects the mixer currents (columns 'MJC_hor' or 'MJC_Ver', respectively) and tests their relative deviation (of the source and ref spectra) to be less than a given tolerance. Once the tolerance is exceeded, for the result spectrum a row flag (bit 14) is set. The tolerance can be set by setting the 'validatorTolerance'-parameter. By default it is set to 0.025. This default checking mechanism can be overruled by passing as 'validator'-parameter an instance of the {@link DataValidator} interface. When setting this parameter to None, no checking is done.

Examples

Example 1: In HIPE:
htp = doRefSubtract(htp=spectra, isABBA=True, startsWithRef=True, offStartWithOpp=True)
Example 2: In HIPE:
htp = doRefSubtract(htp=spectra, indicator ="buffer", phaseValues={"source":2,"ref":1}, offStartWithOpp=True)
Example 3: In HIPE:
from herschel.hifi.pipeline.generic.utils import ChopperPosition
htp = doRefSubtract(htp=spectra, indicator ="Chopper",
 phaseValues={"source":ChopperPosition.CENTER,"ref":ChopperPosition.COLD}, offStartWithOpp=True)
Example 4: In HIPE:
# the validator tolerance is re-set.
htp = doRefSubtract(htp=spectra, isABBA=True, startsWithRef=True, offStartWithOpp=True, validatorTolerance=0.1)
Example 5: In HIPE:
# validation is switched off.
htp = doRefSubtract(htp=spectra, isABBA=True, startsWithRef=True, offStartWithOpp=True, validator=None)
Example 6: In HIPE:
# an alternative validation mechanism is set (here based, as dummy, on the LoFrequency)
from herschel.hifi.pipeline.generic.dorefsubtract import SubtractionDataValidatorImpl
validator = SubtractionDataValidatorImpl()
validator.configure('LoFrequency', 0.001, True)
htp = doRefSubtract(htp=spectra, isABBA=True, startsWithRef=True, offStartWithOpp=True, validator=validator)

API details

Properties

HifiTimelineProduct htp [INOUT, MANDATORY, default=no default value]

The input htp to be processed.

PipelineConfiguration params [INPUT, OPTIONAL, default=no default value]

Pipeline configuration parameters that can be passed to the task.

String indicator [INPUT, OPTIONAL, default=no default value]

The indicator from which source and ref can be identified. Either a column name ("Chopper", "LoFrequency", "buffer") or "pattern".

Boolean isABBA [INPUT, OPTIONAL, default=no default value]

If set to true an A-B-B-A pattern is assumed - otherwise, a A-B-A-B pattern is taken.

PyDictionary phaseValues [INPUT, OPTIONAL, default=no default value]

Specify how to identify 'source' and 'ref' from the the values found in the indicator column.

Boolean startWithRef [INPUT, OPTIONAL, default=no default value.]

Assumes that the sequence starts with a reference measurement if set to true (for DBS modes: ON datasets start with ref, OFF with signal).

Boolean offStartWithOpp [INPUT, OPTIONAL, default=no default value]

If set to true the sequence of scans in ON and OFF datasets starts with their counterparts (source / ref) scans (for DBS and FastDBS modes).

Object validatorTolerance [INPUT, OPTIONAL, default=0.025]

The tolerance to be used for the default validator that checks the difference in the mixer currents and sets a row flag once the mixer currents of the source and ref deviate by more than the tolerance from each other. A relative distance measure is used to quantify the deviation (2abs(x1-x2)/(abs(x1)+abs(x2))). Alternatively, a calibration object of type GenericPipelineCalProduct can be passed from which the band-specific tolerances will be extracted. Yet another possibility is to pass the whole calibration context - the task retrieves the appropriate product with the tolerances. Furthermore, you can use the parameter 'calibration' for that purpose. Once a product of type GenericPipelineCalProduct is obtained, the appropriate tolerance is looked up from a table (associated with the given date) from a column named 'doRefSubtract'.

DataValidator validator [INPUT, OPTIONAL, default=no default value]

Custom validator that can be passed to the task to check source and ref that is subtracted from each other and flag data once the tolerance exceeds a given tolerance. Note that the validatorTolerance cannot be used to specify this threshold. Furthermore, the default validator is overwritten once a validator or None is passed here.

MapContext calibration [INPUT, OPTIONAL, default=no default value]

Parameter to pass a calibration context from which all the calibration input, here mixer current tolerances and possibly chopper positions, can be retrieved.

Boolean useCalTree [INPUT, OPTIONAL, default=no default value]

Parameter to specify whether the calibration tree should be used as primary source of information. It plays a role for task parameters that can explicitly be set as task parameter value (here, for the validatorTolerance-parameter) or, alternatively, retrieved from the calibration tree (see parameter calibration). If useCalTree=True but no reference to the calibration tree is provided the validatorTolerance parameter is used.

Boolean ignore [INPUT, OPTIONAL, default=no default value.]

Ignore the execution of this module (as e.g. for position switch modes)

See also

History

  • 2007-05-16 - Melchior: New version.
  • 2007-11-14 - Melchior: Javadoc updated.
  • 2008-04-08 - Melchior: Some minor changes.
  • 2009-09-23 - Melchior: Validation mechanism.
  • 2011-08-14 - Melchior: Renamed to DoRefSubtractTask
  • 2012-01-16 - Melchior: Mixer current deviation row flag bit set to RowMask.MCD_REF.getIndex() --> URM adjusted