1.60. doRefSubtract

Full Name:	herschel.hifi.pipeline.generic.DoRefSubtractTask
Alias:	doRefSubtract
Type:	Java Task -
Import:	from herschel.hifi.pipeline.generic import DoRefSubtractTask
Category:	HIFI/Pipeline/Level 1 Pipeline

Description

Subtracts suitable reference spectra that are, depending on the observing mode, obtained from a reference sky position, cold load or with a switched LO frequency.

The way how to identify source and reference measurement scans depends on the particular AOT / observing mode applied.

In all cases, only the science data is involved - but typically both, the ON and the OFF measurements.

Position switch reference modes:

Here the reference position is obtained by moving the telescope to a different blank sky position. In the pipeline processing, this is treated rather with the DoOffSmooth module. So the application of the DoRefSubtract here has no impact.
Slow chop DBS modes:

In ON and OFF datasets, we expect a pattern of the form ref-source-source-ref-... for ON and source-ref-ref-source-source-... for the OFF spectra. The subtraction is carried through pairwise for each subsequent source/ref-pair.
Fast chop DBS modes:

In ON and OFF datasets, we expect a pattern of the form ref-source-ref-source-ref-... for ON and source-ref-source-ref-source-... for the OFF spectra. The subtraction is carried through pairwise for each subsequent source/ref-pair.
Load chop modes:

The reference position is the cold load. Again, a pattern of the form ref-source-source-ref-ref-... is assumed that should be reflected by the Chopper value. Again the subtraction is carried through pairwise for each sub-sequent source/ref-pair.
Frequency switch mode: The reference is not a different position but rather a different setting for the LO frequency. Here, a pattern of the form source-ref-ref-source-... is assumed (i.e. starting with 'source', the prime LO frequency) and the subtraction is carried through pairwise for each sub-sequent source/ref-pair.

Different configuration possibilities are available:

Assume a pattern of the form A-B-B-A or A-B-A-B: Specify indicator="pattern" and set the isABBA-parameter to true or false, respectively. Furthermore, specify the startsWithRef and the offStartsWithOpposite-flags.
Use the "buffer"-values and map its values to 'source' or 'ref': Specify indicator="buffer" and either
- set the startsWithRef: in this case, the first value found (within a dataset) is associated with the reference or the source depending on whether it has been set to true or false.
- specify a dictionary of the form {"source":2,"ref":1} to map the source and ref to specific buffer values.
Use the "LoFrequency"- or the "Chopper"-values and map its values to 'source' or 'ref': Specify indicator="LoFrequency" or "Chopper", respectively and either
- set the startWithRef: in this case, the first value found (within a dataset) is associated with the reference or the source depending on whether it has been set to true or false.
- specify a dictionary of the form {"source":-4.1,"ref":4.8} to map the source and ref to specific values found in the "LoFrequency" or "Chopper"-column. Note that for these double values, internally set measurement errors are used for the LoFrequency and the ChopperPositions.

A validation mechanism checks the spectra to be subtracted from each other and, if needed, sets in the result data a row flag. The default mechanism inspects the mixer currents (columns 'MJC_hor' or 'MJC_Ver', respectively) and tests their relative deviation (of the source and ref spectra) to be less than a given tolerance. Once the tolerance is exceeded, for the result spectrum a row flag (bit 14) is set. The tolerance can be set by setting the 'validatorTolerance'-parameter. By default it is set to 0.025. This default checking mechanism can be overruled by passing as 'validator'-parameter an instance of the {@link DataValidator} interface. When setting this parameter to None, no checking is done.

Examples

Example 1: In HIPE:

htp = doRefSubtract(htp=spectra, isABBA=True, startsWithRef=True, offStartsWithOpposite=True)

Example 2: In HIPE:

htp = doRefSubtract(htp=spectra, indicator ="buffer", phaseValues={"source":2,"ref":1}, offStartsWithOpposite=True)

Example 3: In HIPE:

from herschel.hifi.pipeline.generic.utils import ChopperPosition
htp = doRefSubtract(htp=spectra, indicator ="Chopper", phaseValues={"source":ChopperPosition.CENTER,"ref":ChopperPosition.COLD}, offStartsWithOpposite=True)

Example 4: In HIPE:

# the validator tolerance is re-set.
htp = doRefSubtract(htp=spectra, isABBA=True, startsWithRef=True, offStartsWithOpposite=True, validatorTolerance=0.1)

Example 5: In HIPE:

# validation is switched off.
htp = doRefSubtract(htp=spectra, isABBA=True, startsWithRef=True, offStartsWithOpposite=True, validator=None)

Example 6: In HIPE:

# an alternative validation mechanism is set (here based, as dummy, on the LoFrequency)
from herschel.hifi.pipeline.generic.dorefsubtract import SubtractionDataValidatorImpl
validator = SubtractionDataValidatorImpl()
validator.configure('LoFrequency', 0.001, True)
htp = doRefSubtract(htp=spectra, isABBA=True, startsWithRef=True, offStartsWithOpposite=True, validator=validator)

API Summary

Properties
`HifiTimelineProduct htp [INOUT, MANDATORY, default=no default value]`
`PipelineConfiguration params [INPUT, OPTIONAL, default=no default value]`
`String indicator [INPUT, OPTIONAL, default=no default value]`
`Boolean isABBA [INPUT, OPTIONAL, default=no default value]`
`PyDictionary phaseValues [INPUT, OPTIONAL, default=no default value]`
`Boolean startWithRef [INPUT, OPTIONAL, default=no default value.]`
`Boolean offStartsWithOpposite [INPUT, OPTIONAL, default=no default value]`
`Object validatorTolerance [INPUT, OPTIONAL, default=0.025]`
`DataValidator validator [INPUT, OPTIONAL, default=no default value]`
`MapContext calibration [INPUT, OPTIONAL, default=no default value]`
`Boolean useCalTree [INPUT, OPTIONAL, default=no default value]`
`Boolean ignore [INPUT, OPTIONAL, default=no default value.]`

API details

Properties

`HifiTimelineProduct htp [INOUT, MANDATORY, default=no default value]`
	The input htp to be processed.

`PipelineConfiguration params [INPUT, OPTIONAL, default=no default value]`
	Pipeline configuration parameters that can be passed to the task.

`String indicator [INPUT, OPTIONAL, default=no default value]`
	The indicator from which source and ref can be identified. Either a column name ("Chopper", "LoFrequency", "buffer") or "pattern".

`Boolean isABBA [INPUT, OPTIONAL, default=no default value]`
	If set to true an A-B-B-A pattern is assumed - otherwise, a A-B-A-B pattern is taken.

`PyDictionary phaseValues [INPUT, OPTIONAL, default=no default value]`
	Specify how to identify 'source' and 'ref' from the the values found in the indicator column.

`Boolean startWithRef [INPUT, OPTIONAL, default=no default value.]`
	Assumes that the sequence starts with a reference measurement if set to true (for DBS modes: ON datasets start with ref, OFF with signal).

`Boolean offStartsWithOpposite [INPUT, OPTIONAL, default=no default value]`
	If set to true the sequence of scans in ON and OFF datasets starts with their counterparts (source / ref) scans (for DBS and FastDBS modes).

`Object validatorTolerance [INPUT, OPTIONAL, default=0.025]`
	The tolerance to be used for the default validator that checks the difference in the mixer currents and sets a row flag once the mixer currents of the source and ref deviate by more than the tolerance from each other. A relative distance measure is used to quantify the deviation (`2abs(x1-x2)/(abs(x1)+abs(x2))`). Alternatively, a calibration object of type `GenericPipelineCalProduct` can be passed from which the band-specific tolerances will be extracted. Yet another possibility is to pass the whole calibration context - the task retrieves the appropriate product with the tolerances. Furthermore, you can use the parameter 'calibration' for that purpose. Once a product of type `GenericPipelineCalProduct` is obtained, the appropriate tolerance is looked up from a table (associated with the given date) from a column named 'doRefSubtract'.

Object validatorTolerance [INPUT, OPTIONAL, default=0.025]

The tolerance to be used for the default validator that checks the difference in the mixer currents and sets a row flag once the mixer currents of the source and ref deviate by more than the tolerance from each other. A relative distance measure is used to quantify the deviation (2abs(x1-x2)/(abs(x1)+abs(x2))). Alternatively, a calibration object of type GenericPipelineCalProduct can be passed from which the band-specific tolerances will be extracted. Yet another possibility is to pass the whole calibration context - the task retrieves the appropriate product with the tolerances. Furthermore, you can use the parameter 'calibration' for that purpose. Once a product of type GenericPipelineCalProduct is obtained, the appropriate tolerance is looked up from a table (associated with the given date) from a column named 'doRefSubtract'.

`DataValidator validator [INPUT, OPTIONAL, default=no default value]`
	Custom validator that can be passed to the task to check source and ref that is subtracted from each other and flag data once the tolerance exceeds a given tolerance. Note that the validatorTolerance cannot be used to specify this threshold. Furthermore, the default validator is overwritten once a validator or None is passed here.

`MapContext calibration [INPUT, OPTIONAL, default=no default value]`
	Parameter to pass a calibration context from which all the calibration input, here mixer current tolerances and possibly chopper positions, can be retrieved.

`Boolean useCalTree [INPUT, OPTIONAL, default=no default value]`
	Parameter to specify whether the calibration tree should be used as primary source of information. It plays a role for task parameters that can explicitly be set as task parameter value (here, for the `validatorTolerance`-parameter) or, alternatively, retrieved from the calibration tree (see parameter `calibration`). If `useCalTree=True` but no reference to the calibration tree is provided the `validatorTolerance` parameter is used.

Boolean useCalTree [INPUT, OPTIONAL, default=no default value]

Parameter to specify whether the calibration tree should be used as primary source of information. It plays a role for task parameters that can explicitly be set as task parameter value (here, for the validatorTolerance-parameter) or, alternatively, retrieved from the calibration tree (see parameter calibration). If useCalTree=True but no reference to the calibration tree is provided the validatorTolerance parameter is used.

`Boolean ignore [INPUT, OPTIONAL, default=no default value.]`
	Ignore the execution of this module (as e.g. for position switch modes)

History

2007-05-16 - Melchior: New version.
2007-11-14 - Melchior: Javadoc updated.
2008-04-08 - Melchior: Some minor changes.
2009-09-23 - Melchior: Validation mechanism.
2011-08-14 - Melchior: Renamed to DoRefSubtractTask
2012-01-23 - Melchior: Mixer current deviation row flag bit set to RowMask.MCD_REF.getIndex() --> URM adjusted