Difference: FeatureFinderObservationsToNote (1 vs. 3)

Revision 32017-03-15 - RosalindHopwood

Line: 1 to 1
 
META TOPICPARENT name="Public"
Changed:
<
<

Observations where non-default data or settings were used to obtain improved Feature Finder (FF) products

>
>

HR observations for which non-standard data or bespoke settings were used to obtain improved Feature Finder (FF) products

 
Changed:
<
<

Highly Processed Data Products

  • For the repeated observations of SPIRE Spectrometer calibration sources that are presented in Hopwood et al. (2015) (arXiv:1502.05717) there are highly processed data product, which are corrected for pointing offset and, where necessary, for source extent or for high background emission.
  • The "Flag" column (in the FF HR Sparse point-source calibrated product pages: FeatureFinderProducts) indicates whether a HPDP was used, with the HPDP flag.
  • Where a HPDP was used, this is reported under the metadata entry "hpdp" for the individual FF significant feature catalogues, and in the "hpdp" column for the combined significant feature catalogue.
  • The HPDPs themselves can be accessed from the Spectrometer calibrators legacy data page.

Background Subtracted Spectra

  • A number of observations were visually assessed as suffering from the same "double bump" systematic noise that is corrected for LR data (where it was more extreme and prevalent throughout LR observations).
  • This issues is only seen in a small number of observations of faint point-like targets (86), but can be corrected for by subtracting a mean sum of smoothed off-axis detectors, as described in Hopwood et al. (2015).
  • The "Flag" column (in the FF HR Sparse point-source calibrated product pages: FeatureFinderProducts) indicates whether background subtracted (BGS) data was used, with the BGS flag.
  • All BGS data was corrected using the HIPE Background Subtraction useful script in interactive mode.
  • If a BGS spectral data were used, this is reported under the metadata entry "bgs" for the individual FF significant feature catalogues, and in the "bgs" column for the combined significant features catalogue.

Fewer negative SNR thresholds

  • The FF looks for peaks by iterating over a number of SNR thresholds: positive thresholds and then negative thresholds
  • For one particularly spectral rich observation (1342210847), if the default set of negative SNR thresholds are used, so many features are found there are not enough unmasked data points are available for the final SNR estimates and many features are thus discarded
  • The lost features are recovered by omitting the -30 and -10 SNR thresholds
>
>

1. Non-standard data

 
Changed:
<
<

Final SNR estimate

  • To optimise the final SNR estimate, the number of unmasked data points in the local region used to calculate the residual standard deviation is stipulated to be a minimum of 17 (5 GHz)
>
>
Highly Processed Data Products
  • Highly Processed Data Products (HPDPs) are available for the repeated observations of SPIRE Spectrometer calibration sources presented in Hopwood et al. (2015) (arXiv:1502.05717). These HPDPs are SPIRE spectra that have been corrected for pointing offset and, where necessary, for source extent or high background emission.
  • If a HPDP was used instead of the standard Herschel Science Archive product, this is reported under the metadata entry "hpdp" for the individual FF feature catalogues, and in the "HPDP" column of SAFECAT.
  • The "Flag" column (in the FF HR Sparse point-source calibrated product pages) indicates whether a HPDP was used, with a "HPDP" flag.
  • The HPDPs themselves can be accessed from the SPIRE-S calibration targets legacy data page.

Background Subtracted Spectra

  • A number of high resolution (HR) observations were visually assessed as suffering from the same "double bump" systematic noise that is corrected in LR data (where it was more extreme and prevalent in a large fraction of LR observations).
  • This issues is only seen in a small number of HR observations of faint point-like targets (84)
  • The observations were corrected by subtracting a mean sum of smoothed off-axis detectors, as described in Hopwood et al. (2015).
  • All BGS data were corrected using the HIPE Background Subtraction useful script in interactive mode.
  • If BGS data were used instead of the standard Herschel Science Archive product, this is reported under the metadata entry "bgs" for the individual FF feature catalogues, and in the "BGS" column of SAFECAT.
  • The "Flag" column (in the FF HR Sparse point-source calibrated product pages) indicates whether BGS data was used, with a "BGS" flag.
  • The BGS spectra are available here

2. Bespoke FF settings

If, for a particular observations, non-default FF parameters were used or bespoke treatment was applied, this is indicated with a "1" in the "bespokeTreatment" column of hrSparseObservations_FF_1stRelease.csv, which lists all HR observations for which there is a set of FF products.

Fewer negative SNR thresholds

  • By default, the FF iterates over a number of SNR thresholds when looking for peaks: +[100, 50, 30, 10, 5, 3] followed by -[100, 50, 30]
  • For one particularly spectral rich observation (1342210847), if the default set of negative SNR thresholds are used, so many features are found there are not enough unmasked data points available for the final SNR estimates and many features are thus discarded
  • This lost of features is prevented by omitting the -30 and -10 SNR thresholds

Final SNR estimate

  • To optimise the final SNR estimate, the FF stipulates that the number of unmasked data points in the local region used to calculate the residual standard deviation must be at least 17 (5 GHz)
 
  • If this condition is not met, then the local region around the fitted peak is widened.
Changed:
<
<
  • For two spectral rich sources this condition is never matched for the majority of the features found, and so in these cases no minimum is set for the number of data points available
  • This applies to the observations 1342192834,1342197466
>
>
  • For the observations of two spectral rich sources (1342192834,1342197466) this condition is never matched for the majority of the features found, and so for the default settings, the majority of features are discarded. So in these two cases no minimum is set for the number of data points required for the the final SNR estimate to go ahead.
 
Changed:
<
<

Features added by hand

>
>
Features added by hand
 
  • During the FF SNR threshold iterations, the SNR is taken using the spectral dataset "error" column.
Changed:
<
<
  • For a handful of observations this can lead to no significant SNR peak at the position of significant spectral peaks and the corresponding features are never found, regardless of how low the SNR threshold drops
  • The clear missing features were added at the appropriate SNR threshold during the FF process
  • This fix was applied for the observations: 1342197466, 1342248242, 1342216879, 1342197466, 1342193670, 1342210847

-- Rosalind Hopwood - 2017-02-27

>
>
  • For a handful of observations this can lead to no significant SNR peak at the position of significant spectral peaks and the corresponding features are therefore never found by the FF, regardless of how low the SNR threshold drops
  • A handful of missing significant features were added at the appropriate SNR threshold during the FF process for six observations: 1342197466, 1342248242, 1342216879, 1342197466, 1342193670, 1342210847

Revision 22017-03-08 - RosalindHopwood

Line: 1 to 1
Changed:
<
<
META TOPICPARENT name="TheSpireFtsSpectralFeatureFinder"

Observations where non-default data or settings were used to obtain improved FF products

>
>
META TOPICPARENT name="Public"

Observations where non-default data or settings were used to obtain improved Feature Finder (FF) products

 

Highly Processed Data Products

Changed:
<
<

  • If a HPDP was used, this is reported under the metadata entry "hpdp"
>
>
  • For the repeated observations of SPIRE Spectrometer calibration sources that are presented in Hopwood et al. (2015) (arXiv:1502.05717) there are highly processed data product, which are corrected for pointing offset and, where necessary, for source extent or for high background emission.
  • The "Flag" column (in the FF HR Sparse point-source calibrated product pages: FeatureFinderProducts) indicates whether a HPDP was used, with the HPDP flag.
  • Where a HPDP was used, this is reported under the metadata entry "hpdp" for the individual FF significant feature catalogues, and in the "hpdp" column for the combined significant feature catalogue.
  • The HPDPs themselves can be accessed from the Spectrometer calibrators legacy data page.
 

Background Subtracted Spectra

Changed:
<
<
  • If a BGS spectral data were used, this is reported under the metadata entry "bugs"
>
>
  • A number of observations were visually assessed as suffering from the same "double bump" systematic noise that is corrected for LR data (where it was more extreme and prevalent throughout LR observations).
  • This issues is only seen in a small number of observations of faint point-like targets (86), but can be corrected for by subtracting a mean sum of smoothed off-axis detectors, as described in Hopwood et al. (2015).
  • The "Flag" column (in the FF HR Sparse point-source calibrated product pages: FeatureFinderProducts) indicates whether background subtracted (BGS) data was used, with the BGS flag.
  • All BGS data was corrected using the HIPE Background Subtraction useful script in interactive mode.
  • If a BGS spectral data were used, this is reported under the metadata entry "bgs" for the individual FF significant feature catalogues, and in the "bgs" column for the combined significant features catalogue.
 

Fewer negative SNR thresholds

Changed:
<
<

  • This applies to the observations:
>
>
  • The FF looks for peaks by iterating over a number of SNR thresholds: positive thresholds and then negative thresholds
  • For one particularly spectral rich observation (1342210847), if the default set of negative SNR thresholds are used, so many features are found there are not enough unmasked data points are available for the final SNR estimates and many features are thus discarded
  • The lost features are recovered by omitting the -30 and -10 SNR thresholds
 

Final SNR estimate

Changed:
<
<

  • This applies to the observations:
>
>
  • To optimise the final SNR estimate, the number of unmasked data points in the local region used to calculate the residual standard deviation is stipulated to be a minimum of 17 (5 GHz)
  • If this condition is not met, then the local region around the fitted peak is widened.
  • For two spectral rich sources this condition is never matched for the majority of the features found, and so in these cases no minimum is set for the number of data points available
  • This applies to the observations 1342192834,1342197466
 

Features added by hand

Changed:
<
<
  • This applies to the observations: 1342197466, 1342248242, 1342216879, 1342197466, 1342193670, 1342210847
>
>
  • During the FF SNR threshold iterations, the SNR is taken using the spectral dataset "error" column.
  • For a handful of observations this can lead to no significant SNR peak at the position of significant spectral peaks and the corresponding features are never found, regardless of how low the SNR threshold drops
  • The clear missing features were added at the appropriate SNR threshold during the FF process
  • This fix was applied for the observations: 1342197466, 1342248242, 1342216879, 1342197466, 1342193670, 1342210847
  -- Rosalind Hopwood - 2017-02-27

Revision 12017-02-27 - RosalindHopwood

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="TheSpireFtsSpectralFeatureFinder"

Observations where non-default data or settings were used to obtain improved FF products

Highly Processed Data Products

  • If a HPDP was used, this is reported under the metadata entry "hpdp"

Background Subtracted Spectra

  • If a BGS spectral data were used, this is reported under the metadata entry "bugs"

Fewer negative SNR thresholds

  • This applies to the observations:

Final SNR estimate

  • This applies to the observations:

Features added by hand

  • This applies to the observations: 1342197466, 1342248242, 1342216879, 1342197466, 1342193670, 1342210847

-- Rosalind Hopwood - 2017-02-27

 
This site is powered by the TWiki collaboration platform Powered by Perl