In HIPE 10, the option to choose a different CAL version via the hifiPipelineGUI always gives the latest even if a different version is chosen. This has been fixed in HIPE 11.
The SPG version is now updated in the meta data when you run the pipeline
The pipeline will only try to run for spectrometers for which there is data at level 0 (previously the pipeline always tried to run for all spectrometers)
The option to customise the pipeline from the GUI is disabled by default when starting processing from Level 2
Update of the list of corrupted data-frames for flagging at Level 0, updates to quality flag meta-data and uplink information for mapping modes
Changed:
< <
* The NN interpolation of DoWbsFreqTask (default) seem not well applied to the dataest.
> >
Updates to the HIFI Data Reduction Guide:
Updates to "HIFI Baseline Removal" chapter, "FitBaseline Options" section
The NN interpolation of DoWbsFreqTask (default) seem not well applied to the datasets.
HIFI Data Processing Tools: Baseline Removal
New fitBaseline parameter 'useWbs' allows subtraction of WBS polynomials from HRS spectra.
fitBaseline reacts much better to HIPE 'stop' button.
New fitBaseline parameter 'plot' allows switching plotting in non-interactive mode off or on. The default mode is False, which speeds up the task and greatly reduces memory consumption.
fitBaseline domask=2 option (automated masking) now works in non-interactive mode as well, and can be interrupted using the HIPE 'stop' button.
The following minor changes have been made:
HCSS
* Known issue: Can't open plug-ins panel under KDE or Gnome 3 (added to known issues page, patch planned for HIPE 10.2)
* The new pools with the new lucene: instructions to deal with them need to be better
* Bulk GUI does not launch pipelines unless COMMON preprocessing has run
HIFI
* Update to fitBaseline chapter - documentation
* The NN interpolation of DoWbsFreqTask (default) seem not well applied to the dataest.
PACS
o Masked data propagated as NaNs in the rebinned/projected data when that range is removed by OutofBandSPIRE
o Problem with bright mode for new tempDriftCorr cal files
o The engineering conversion fails for 1342182603 - OD=100
o CalibrationPlugin fails to set the aperture
fitHifiFringe can be directly applied to spectra in HIFI mapping mode cubes through new input option 'cube'.
Added:
> >
A script for treating electric standing waves (so-called matching technique) is available in the source directory of the 'expert' install of HIPE: scripts/hifi/scripts/users/engineering/HEBStWvCatalogCorrection.py It is extensively commented. The script can make use of OFF positions from other obsids, and currently only works for observations taken in DBS mode. Other modes can be treated with a non-released version - please contact the helpdesk for more details.
Baseline Removal
Changed default for 'movemasks' option from 2 to 0 in fitBaseline, i.e., masks are not propagated between spectra.
User releases for Windows and Linux already include the correct Java version.
Developer releases and user releases for Mac do not include Java. You must ensure that you have a compatible version of Java installed on your system.
If the minor version of your installed Java is different from the recommended one (for instance, 1.6.0_35 instead of 1.6.0_37) HIPE prints a warning in the command line window from which it was started. Minor version differences are unlikely to cause problems, but this cannot be guaranteed.
Changed:
< <
If the major version of your installed Java is different (for instance, 1.5.0 instead of 1.6.0) HIPE shows a warning in a dialogue window at startup. Major version differences are very likely to cause problems. We do not recommend that you try to run HIPE 10.0 with a Java version other than 1.6.0. In particular, HIPE is known to give incorrect results if used under Java 7 with a locale different from en_US.
> >
If the major version of your installed Java is different (for instance, 1.5.0 instead of 1.6.0) HIPE shows a warning in a dialogue window at startup. Major version differences are very likely to cause problems. We do not recommend that you try to run HIPE 10.0 with a Java version other than 1.6.0.
There is a new task pacsExtractSpectralRange to spectrally crop a (Sliced)PacsCube or (Sliced)PacsRebinnedCube.
The footprint of pointed observations is shown by projecting the rebinned cubes on a finer grid (0.5 arcsec pixelsize).
Changed:
< <
There is a new mask OUTOFBAND which masks the data at wavelengths where the signal is completely blocked by the used filter.
> >
There is a new mask OUTOFBAND which masks the data at wavelengths where the signal is completely blocked by the filter (i.e. for when your wavelength range was set in the red, leading to an out-of-band wavelength range in the blue).
Slices which are completely masked by this OUTOFBAND mask are discarded during the level-0.5 processing.
Spectra are now by default rebinned to a standard wavelength grid, so that we avoid incompatible wavelength grid when subtracting on and off spectra from different observations.
Changed:
< <
New useful script for point source correction.
> >
New useful script for point source flux loss correction.
Spectral flatfielding: several issues had been reported for the specific case of zero or weak continuum in specFlatFieldLine and specFlatFieldRange (e.g. non-central spaxels). These were fixed.
One can now slice the level-0 data with pacsSpecSliceLevel0 to reduce the memory consumption of the processing till level-0.5.
Most of the ipipe scripts have been slightly simplified. Mainly, selectSlices has been removed: all tasks can cope with sliced products transparently, so selectSlices is optional.
Line: 271 to 271
Fixed some wrong values of the 'onOffSource' meta keyword.
Fixed some inconsistencies in the units and labels of the pipeline products.
Added the missing keyword 'faintLines' to the observation context.
Added:
> >
For the background normalisation pipeline there is a new correction for telescope temperature and ageing, with a new calibration based on the analytical telescope model.
Task extractCentralSpectrum to extract the spectrum of a point source in the central spaxel, corrected for the PSF and pointing offsets. The corrections for the PSF is necessary, the others are optional. For line+short range scan AOTs the recommended correction is a median over wavelength, for range and SED AOTs we recommend the wavelength dependent correction.
Browse products (and the Level 2 SpectralSimpleCubes [projected cubes] they come from) are created by the pipeline task specProject with very small spaxel sizes (1") for non-mapping observations: with this you can still make out the original spaxels, but with a regular sky WCS.
Calibration
Photometry
Line: 282 to 286
As a result of the new calibration scheme, the absolute fluxes found at level 2 are slightly reduced.
For extended sources, the new calibration is more accurate
The new calibration should not impact point sources, thanks to the so-called "central to 3x3" correction performed in extractCentralSpectrum. That correction was not necessary in hipe 9.0, as it was implicitly contained in the flux calibration files. In hipe 10, by default no average mis-pointing is included in the calibration files anymore, hence this extra correction, making use of the flux distribution between the spaxels to estimate the mispointing and correct for it.
Added:
> >
New versions of the calfiles ObservedResponse and CalSourceFlux released (cal set 44)
There is a calibration set update for the Telescope Background product (45) used in the Background Normalisation ipipe script
There is no change in the PACS photometer calibration since Hipe 9 and version 41 of the calibration tree.
Spectroscopy
Changed:
< <
In order to be less sensitive to (on average) mispointed flux calibration observations, the flux calibration scheme has changed: the construction of the calibration tables is now based on the standard calibrators fluxes coming from the central 9 spaxels instead of the central one only. This implies that two new calibration tables were introduced in the calibration tree, namely observedResponse3x3 and calSourceFlux3x3. Those tables are used by default in specDiffCs and should lead to more reliable fluxes, notably in case of mispointed observations. An optional parameter, use3x3 (True by default), was added to specDiffCs. You can still use the former calibration scheme by setting use3x3 to False although it is not recommended.
> >
In order to be less sensitive to (on average) mispointed flux calibration observations, the flux calibration scheme has changed: the construction of the calibration tables is now based on the standard calibrators fluxes coming from the central 9 spaxels instead of the central one only. This implies that two new calibration tables were introduced in the calibration tree (version 45 or higher required), namely observedResponse3x3 and calSourceFlux3x3. Those tables are used by default in specDiffCs and should lead to more reliable fluxes, notably in case of mispointed observations. An optional parameter, use3x3 (True by default), was added to specDiffCs. You can still use the former calibration scheme by setting use3x3 to False although it is not recommended.
As a result of the new calibration scheme, the absolute fluxes found at level 2 are slightly reduced.
For extended sources, the new calibration is more accurate
The new calibration should not impact point sources, thanks to the so-called "central to 3x3" correction performed in extractCentralSpectrum. That correction was not necessary in hipe 9.0, as it was implicitly contained in the flux calibration files. In hipe 10, by default no average mis-pointing is included in the calibration files anymore, hence this extra correction, making use of the flux distribution between the spaxels to estimate the mispointing and correct for it.
The maximum memory value shown in the memory bar reflects more accurately the actual memory available to HIPE. This value can vary slightly during a HIPE session.
The Locale is established as English so that the decimal separator will be a dot even in computers with language settings that use a comma for it. This ensures proper processing of tasks relying on it being a dot.
Deleted:
< <
Plotting
TablePlotter
OverPlotter
Astronomical utilities
New package herschel.ia.toolbox.astro.healpix (based on JPL Healpix Java library).
Healpix interpolation functionality
Line: 85 to 81
Numeric routines
Data fitting
Increased robustness in the case of fitting in the presence of heavy outliers.
Deleted:
< <
Data interpolation
Random numbers
Statistics
Trend analysis
Mapping
Other functions
Deleted:
< <
Source extraction
Images
Display
Image Display uses less memory and is a lot faster. It is now possible to display large image without the need for an enormous amount of memory.
It is possible to set the standard cut levels which should be used in the preference panel.
Deleted:
< <
Analysis
Spectra
Display
Analysis
Data cubes
Line: 122 to 104
It is now possible to send the results of a query in the HSA User Interface directly to HIPE. The results will appear as TableDataset, allowing one to use the geometric search in the HSA UI (for instance), and directly use the result in a script, or save it to disk.
Deleted:
< <
Calibration sources
Products and datasets
Observation context
Product Access
Line: 148 to 126
For Jython task developers: the following wrappers (decorators) have been added to support working with the EDT: execLater, execAndWait (similar to the SwingUtils functions). An example is provided in the source of ia_task tests (TestTaskForViewsEdt.py)
Deleted:
< <
Quality control
Systematic product generation
Data input-output
poolDataReader , poolDataWriter, loadObs, downObs and localStoreWriter removed : saveProduct and MyHSA (getObservation) contain a superset of their features.
Tables
Line: 171 to 145
Fix issues with WebStart applications (such as Aladin, HSA User Interface): Problems starting the applications on Windows XP, problems with automatic log-in.
Deleted:
< <
Installer
Documentation
Updates to the Glossary, also available in a tab in the HIPE Help System sidebar.
Line: 196 to 168
HIFI
Deleted:
< <
HIFI Software Configuration and Properties
HIFI Calibration
Updates to table of unusable data frames
Quality flags are now raised for monitoring SEUs (Sudden Event Upsets)
Line: 313 to 283
For extended sources, the new calibration is more accurate
The new calibration should not impact point sources, thanks to the so-called "central to 3x3" correction performed in extractCentralSpectrum. That correction was not necessary in hipe 9.0, as it was implicitly contained in the flux calibration files. In hipe 10, by default no average mis-pointing is included in the calibration files anymore, hence this extra correction, making use of the flux distribution between the spaxels to estimate the mispointing and correct for it.
Deleted:
< <
Analysis tools
Photometry
Spectroscopy
Product changes
Photometry
Line: 324 to 290
Error maps in SPG10 level2 scan maps are produced by the task photCoverage2Noise, where error maps are derived from the coverage map and include cross-correlation noise.
The mask used for the masked-highpass filtering of the pixel timelines is now available as a map in level 2, in the extensions 'HPFmask'. It can be used to assess if the source of interest has been properly mask, in this case the photometry in the L2 point-source is reliable.
Deleted:
< <
Spectroscopy
Documentation
The PACS Data Reduction Guide (PDRG) now contains a chapter that describes the processing of data obtained with the chopped PS AOT.
You can now send the results of a query in the HSA User Interface directly to HIPE as a table dataset.
HIFI
Changed:
< <
Introduction of a pipeline task to allow comparison of emission in DBS chop positions, !mkDbsReference task. It differences the chop positions in all DBS observations and after applying the band-pass correction stores them in a product in calibration->pipeline-out called ReferenceSpectra.
> >
Introduction of a pipeline task to allow comparison of emission in DBS chop positions, mkDbsReference task. It differences the chop positions in all DBS observations and after applying the band-pass correction stores them in a product in calibration->pipeline-out called ReferenceSpectra.
Major updates to flagTool
Rotated cubes are now produced in the Level 2.5 product by default for maps carried out with a non-zero position angle, a non-rotated cube is also produced.
A new quality flag is introduced to warn of a single COMB failure
Improved checks and threshold values for the ZERO spectra.
Changed:
< <
HRS Pipeline
> >
Level 1 Pipeline
Addition of a task to allow comparison of emission in DBS chop positions, mkDbsReference task. Differences the chop positions in all DBS observations and after applying the band-pass correction stores them in a product in calibration->pipeline-out called ReferenceSpectra. You may configure the task to average the differenced spectra together (a resampling is automatically done) and to resample the spectra yourself.
Additional parameters available to DoFilterLoads
Line: 234 to 234
HifiUplinkProduct format changed to a table dataset (easier to read)
HIFI Spectrum Toolbox
Changed:
< <
Spectrum Explorer (HIFI specifics)
> >
Spectrum tasks (HIFI specifics)
A new task, doOffset, to calculate the offset positions from the requested position is introduced. You can use it to switch between offset and absolute coordinates.
Line: 262 to 262
All subbands can be displayed together (without gluing).
Flag type can be selected from the options above the plot instead of from a context menu
Deleted:
< <
HiClass export tool
Documentation
Updates to the HIFI Data Reduction Guide:
Added:
> >
Updates to DBS Cookbook
Addition of a chapter describing how to convert positions in data to offsets
Improvements and updates to "How to add and remove flags" chapter, including documentation of new flagTool in the "Setting flags interactively for many spectra" section
Improvements and updates to "Running the HIFI pipeline" chapter
Added:
> >
Updates to "Sideband Deconvolution" chapter, including a description of how to export result to CLASS
Update to mapping browse product description in "Tour of a HIFI observation context" chapter
Removal of references to the old CubeSpectrumAnalysisToolbox and replacement with references to the new Cube Toolbox
Various typos, improvements to style consistency and bug fixes.
The parameter types for detectGlitches, detectLargeGl and repairGlitches parameters are changed to Boolean.
Photometer Pipeline
Added:
> >
New definition of Leve2 products:
For versions of the HCSS prior to HIPE 10.0, a single point source calibrated (Janskys/beam) map was provided in the Level 2 product for each of the PSW, PMW, PLW bands. However, for observations processed with HIPE 10.0 or later, more than one map calibration is made available within the Level 2 product. Maps are provided for the following scenarios for post HIPE 10.0 processing;
Description
New Name (HIPE10+)
Original (pre-HIPE10) Name
Point Source (standard) Maps
psrcPSW
PSW
Extended Emission Maps
extdPSW
-
Solar System Object Maps
ssoPSW
-
Point Source Destriper Diagnostic
psrcPSWdiag
pddPSW
Extended Emission Destriper Diagnostic
extdPSWdiag
-
New zeroPointCorrection task for cross-calibration with Planck HFI maps. This can only be run at the HSC until the HFI maps are made public. The output maps are included in the Level-2 context.
New ssoMotionCorrection task to produce modified Level-1 timelines adjusted to the reference frame of the moving object. Additional maps are created from this timeline and included in the Level-2 context.
There is also a similar task photCoverage2NoisePointSource that does the same for data obtained with the chopped PS AOT. It is independent of the version for scan maps. To date, it does not provide an estimate of the correlated noise.
The task MaskFromCatalogueTask allows to build a mask from a source list; a very useful task for deep imaging, to properly mask even faint point-sources for highpass filtering.
A new wrapper task photMaskedHpfDeglitch allows to deglitch L1 frames, with highpass filtering of the timelines before the second level deglitching creating a mask, that is returned attached to the non-filtered L1 frames. This allows for instance a finer deglitching of the L1 frames for MADmap, as the default deglitching in SPG is rather conservative with nsigma=30.
Added:
> >
The noise cube that was previously injected in the pipeline to level 1 has be removed as it was not used for proper error propagation. As a result the size of level 1 frames is halved. As a consequence the memory required in Hipe to reduce a given photometer scan map is reduced by a factor 2.
There is a a new task photCoverage2Noise, run in the SPG pipeline to derive an error map from the coverage map of scan maps, including the correction for correlated noise. It is also available for interactive processing.
There is also a similar task photCoverage2NoisePointSource that does the same for data obtained with the chopped PS AOT. It is independent of the version for scan maps. To date, it does not provide an estimate of the correlated noise.
The task MaskFromCatalogueTask allows to build a mask from a source list; a very useful task for deep imaging, to properly mask even faint point-sources for highpass filtering.
Added:
> >
A new wrapper task photMaskedHpfDeglitch allows to deglitch L1 frames, with highpass filtering of the timelines before the second level deglitching creating a mask, that is returned attached to the non-filtered L1 frames. This allows for instance a finer deglitching of the L1 frames for MADmap, as the default deglitching in SPG is rather conservative with nsigma=30.
Can be opened and used in the Herschel Spectrum Explorer
More tasks available
Arithmetic operations (+, -, *, /) work in scripts, when performing calculations between an SDS product and either another SDS product, a spectrum, or a number
Deleted:
< <
Observation Viewer:
Information is displayed in the viewer to show the building block names for Level 0 and Level 0.5 products
SDS Explorer
N-E arrows added to show orientation of the spectrometer arrays
Changed:
< <
* A prototype for a new tool to correct semi-extended sources (due to coupling with the beam) was added - note that this is a prototype and is not explained in the documentation yet - please contact the helpdesk/FTS support group for details.
* convolutionMapper
* New task to create a kernel-filtered map directly from photometer timeline data
> >
A new useful script has been added to carry out spectral line fitting for spectral cubes
The useful script to combine PACS and SPIRE spectra is now available from the SPIRE scripts menu (as well as the PACS scripts menu)
Documentation
Added:
> >
Spectrometer
Included more links to main Herschel document in line fitting section
Added section about combining 2 observations into a map
Updated SDS/SDI Explorer section
Move apodization to end of pipeline and explain spectral apodization - and update pipeline flowchart
Simplify phase correction section
Updated figures showing Level-0.5 data
Updated memory use section
Add cube line fitting description and more basic description of cube toolbox
Updates to background subtraction and noise script sections
There is a a new task photCoverage2Noise, run in the SPG pipeline to derive an error map from the coverage map of scan maps, including the correction for correlated noise. It is also available for interactive processing.
There is also a similar task photCoverage2NoisePointSource that does the same for data obtained with the chopped PS AOT. It is independent of the version for scan maps. To date, it does not provide an estimate of the correlated noise.
The task MaskFromCatalogueTask allows to build a mask from a source list; a very useful task for deep imaging, to properly mask even faint point-sources for highpass filtering.
Deleted:
< <
There is a new Ipipe script available for processing data obtained with the chopped PS AOT.
Spectroscopy
Line: 318 to 317
Photometry
From SPG10 onwards, level 2 maps are processed with a very small high-pass filter width to filter as much the 1/f noise and get optimal sensitivity to point-sources. Half-widths of 15 and 25 readouts are used in the blue and red channel respectively. As a consequence extended emission at virtually all scales is filtered out. The level 2.5 maps processed with MADmap should be used to get extended emission, which is preserved at all scales in the MADmap maps. Note that SPG10 bulk pipeline processing will only start in early 2013.
Changed:
< <
Error maps in SPG10 level2 maps are produced by the task photCoverage2Noise, where error maps are derived from the coverage map and include cross-correlation noise.
> >
Error maps in SPG10 level2 scan maps are produced by the task photCoverage2Noise, where error maps are derived from the coverage map and include cross-correlation noise.
The mask used for the masked-highpass filtering of the pixel timelines is now available as a map in level 2, in the extensions 'HPFmask'. It can be used to assess if the source of interest has been properly mask, in this case the photometry in the L2 point-source is reliable.
Spectroscopy
Line: 408 to 405
Can be opened and used in the Herschel Spectrum Explorer
More tasks available
Arithmetic operations (+, -, *, /) work in scripts, when performing calculations between an SDS product and either another SDS product, a spectrum, or a number
Added:
> >
Observation Viewer:
Information is displayed in the viewer to show the building block names for Level 0 and Level 0.5 products
SDS Explorer
N-E arrows added to show orientation of the spectrometer arrays
Changed:
< <
A prototype for a new tool to correct semi-extended sources (due to coupling with the beam) was added - note that this is a prototype and is not explained in the documentation yet - please contact the helpdesk/FTS support group for details.
> >
* A prototype for a new tool to correct semi-extended sources (due to coupling with the beam) was added - note that this is a prototype and is not explained in the documentation yet - please contact the helpdesk/FTS support group for details.
* convolutionMapper
* New task to create a kernel-filtered map directly from photometer timeline data
Telescope model coefficients updated, and separate coefficients added for off-axis detectors - this reduces the additive offset due to the telescope model subtraction, both in the centre detectors, and in off-axis detectors.
Changed:
< <
The split beamParam, teleRsrf and instRsrf for the two BSM positions (before and after OD1011) lead to significantly better overall calibration for observations after OD1011. For faint sources this helps with the background subtraction using off-axis detectors.
The LR calibration is improved by adding additional observations into the average used for the teleRsrf and instRsrf
The noise level in mapping observations is improved for longer observations as the teleRsrf for off-centre jiggle positions is now deeper.
The updated beam profile and new spatial correction coefficient allow the Semi-extended correction tool to work (prototype is included in v10).
> >
The beamParam, teleRsrf and instRsrf products have been updated to provide a separate calibration for ODs after 1011. At this point the position of the BSM was corrected to be on-axis. This split in the calibration (before and after OD1011) leads to significantly better overall calibration for observations after OD1011. For faint sources this helps when background subtraction is carried out using off-axis detectors.
The LR calibration has been improved by adding additional observations into the average used for the teleRsrf and instRsrf
The noise level in mapping observations has been improved for longer observations by making the teleRsrf for off-centre jiggle positions deeper.
The beam profile product has been updated with the fitted 2D frequency-dependent beam shape
A new Spatial Correction Coefficient calibration product has been defined for use when correcting semi-extended sources. This is used in a tool being developed for this correction.
The MR and apodized calibration products have been cleaned up from the calibration tree as these are not used in the v10 pipeline - this makes the calibration tree smaller.
Calibration Framework
Preprocessing
Changed:
< <
The Pcal products now have correctly filled in metadata.
> >
The Pcal products attatched to the calibration context now have correctly filled in metadata.
Engineering Conversion
Line: 377 to 382
Spectrometer Pipeline
Changed:
< <
History is now propagated from level-1 products to level-2 products when spirePreprocessCube and spireProjection are used in the spectrometer pipeline.
> >
Apodization is now carried out at the end of the pipeline after calibration of the final spectra - this is done using an inverse Fourier transform, applying the apodization function and re-applying the forward Fourier transform. This means that separate calibration products for apodized data are no longer needed, and in principle any apodization function can be applied (although the aNB_15 function is still the recommended one).
The wavelet deglitcher task now works with multiple processing threads.
History is now propagated from Level-1 products to Level-2 products when spirePreprocessCube and spireProjection are used in the spectrometer pipeline.
Source extraction
Line: 387 to 394
Interactive Analysis and Tools
Added:
> >
Observation Viewer:
Information is displayed in the viewer to show the building block names for Level 0 and Level 0.5 products
Photometer
convolutionMapper
New task to create a kernel-filtered map directly from photometer timeline data
Spectrometer
SpectrometerDetectorSpectrum (SDS) products have more functionality
Changed:
< <
Can be opened and used in the Spectrum Explorer
> >
Can be opened and used in the Herschel Spectrum Explorer
More tasks available
Arithmetic operations (+, -, *, /) work in scripts, when performing calculations between an SDS product and either another SDS product, a spectrum, or a number
Deleted:
< <
Observation Viewer:
More information displayed in the viewer for level 0 and level 0.5 products
SDS Explorer
Changed:
< <
N-E arrows added to show orientation of the arrays
convolutionMapper
New task to create a kernel-filtered map directly from timeline data
> >
N-E arrows added to show orientation of the spectrometer arrays
A prototype for a new tool to correct semi-extended sources (due to coupling with the beam) was added - note that this is a prototype and is not explained in the documentation yet - please contact the helpdesk/FTS support group for details.
The cube tools part of the Spectrum Explorer has now a new tool to compare N cubes: One can link them now by world or pixel coordinates, so that selecting a spaxel in one cube, selects simultaneously the same spaxel in the other linked cubes.
Analysis
Line: 107 to 106
Analysis
Data cubes
Added:
> >
Display
Added:
> >
The cube tools part of the Spectrum Explorer has now a new tool to compare N cubes: One can link them now by world or pixel coordinates, so that selecting a spaxel in one cube, selects simultaneously the same spaxel in the other linked cubes.
Analysis
Added:
> >
The following improvements have been made to the Spectrum Explorer Cube Tools:
The subtractBaselineFromCube task now also has an option to divide by the baseline instead of subtracting it.
The computPVMap task automatically scales the PV map in the spatial direction in case the aspect ratio is too big (> 2 by default).
Herschel Science Archive
It is now possible to send the results of a query in the HSA User Interface directly to HIPE. The results will appear as TableDataset, allowing one to use the geometric search in the HSA UI (for instance), and directly use the result in a script, or save it to disk.
Telescope model coefficients updated, and separate coefficients added for off-axis detectors - this reduces the additive offset due to the telescope model subtraction, both in the centre detectors, and in off-axis detectors.
The split beamParam, teleRsrf and instRsrf for the two BSM positions (before and after OD1011) lead to significantly better overall calibration for observations after OD1011. For faint sources this helps with the background subtraction using off-axis detectors.
The LR calibration is improved by adding additional observations into the average used for the teleRsrf and instRsrf
The noise level in mapping observations is improved for longer observations as the teleRsrf for off-centre jiggle positions is now deeper.
The updated beam profile and new spatial correction coefficient allow the Semi-extended correction tool to work (prototype is included in v10).
Calibration Framework
Preprocessing
Line: 335 to 342
Engineering Conversion
Common Pipeline
Added:
> >
Metadata in products is sorted into a more helpful order. Further improvements to the ordering are likely in HCSS 11.
SigmaKappaDeglitcherTask
Added a diagnosticDetector task parameter for diagnostic purpose
All the long task parameters are shortened
Line: 350 to 357
Added useOnlyMasks and ignoreDefaultMasks task parameters in synch with map mapper.
Added jumpThresh and jumpIter task parameters for jump detections.
Two second level deglitch algorithms are added, sigma-kappa and two sigmas
There is a a new task photCoverage2Noise, run in the SPG pipeline to derive an error map from the coverage map of scan maps, including the correction for correlated noise. It is also available for interactive processing.
There is also a similar task photCoverage2NoisePointSource that does the same for data obtained with the chopped PS AOT. It is independent of the version for scan maps. To date, it does not provide an estimate of the correlated noise.
Changed:
< <
The task MaskFromCatalogueTask allows to build a mask from a source list; a very useful task for deep imaging, to properly mask even faint point-sources for highpass filtering.
> >
The task MaskFromCatalogueTask allows to build a mask from a source list; a very useful task for deep imaging, to properly mask even faint point-sources for highpass filtering.
There is a new Ipipe script available for processing data obtained with the chopped PS AOT.
Spectroscopy
Line: 312 to 312
Error maps in SPG10 level2 maps are produced by the task photCoverage2Noise, where error maps are derived from the coverage map and include cross-correlation noise.
Added:
> >
The mask used for the masked-highpass filtering of the pixel timelines is now available as a map in level 2, in the extensions 'HPFmask'. It can be used to assess if the source of interest has been properly mask, in this case the photometry in the L2 point-source is reliable.
Rotated cubes are now produced in the Level 2.5 product by default for maps carried out with a non-zero position angle, a non-rotated cube is also produced.
PACS
Changed:
< <
new task available to derive error maps for maps produced with photProject task on highpass filtered timelines.
> >
New task available to derive error maps for maps produced with photProject task on highpass filtered timelines.
SPIRE
New zeroPointCorrection task for cross-calibration with Planck HFI maps producing extended emission maps.
Line: 269 to 269
Pipeline
Photometry
Changed:
< <
There is a a new task photCoverage2Noise, run in the SPG pipeline to derive an error map from the coverage map, including the correction for correlated noise.
> >
There is a a new task photCoverage2Noise, run in the SPG pipeline to derive an error map from the coverage map of scan maps, including the correction for correlated noise. It is also available for interactive processing.
There is also a similar task photCoverage2NoisePointSource that does the same for data obtained with the chopped PS AOT. It is independent of the version for scan maps. To date, it does not provide an estimate of the correlated noise.
The task MaskFromCatalogueTask allows to build a mask from a source list; a very useful task for deep imaging, to properly mask even faint point-sources for highpass filtering.
Changed:
< <
> >
There is a new Ipipe script available for processing data obtained with the chopped PS AOT.
Spectroscopy
Line: 314 to 315
Spectroscopy
Documentation
Added:
> >
The PACS Data Reduction Guide (PDRG) now contains a chapter that describes the processing of data obtained with the chopped PS AOT.
Rotated cubes are now produced in the Level 2.5 product by default for maps carried out with a non-zero position angle, a non-rotated cube is also produced.
PACS
Added:
> >
new task available to derive error maps for maps produced with photProject task on highpass filtered timelines.
SPIRE
New zeroPointCorrection task for cross-calibration with Planck HFI maps producing extended emission maps.
There is a a new task photCoverage2Noise, run in the SPG pipeline to derive an error map from the coverage map, including the correction for correlated noise.
Added:
> >
The task MaskFromCatalogueTask allows to build a mask from a source list; a very useful task for deep imaging, to properly mask even faint point-sources for highpass filtering.
Spectroscopy
Line: 290 to 291
Calibration
Photometry
Changed:
< <
There is no change in the PACS photometer calibration since Hipe 9 and version 9 of the calibration tree.
> >
There is no change in the PACS photometer calibration since Hipe 9 and version 41 of the calibration tree.
Spectroscopy
In order to be less sensitive to (on average) mispointed flux calibration observations, the flux calibration scheme has changed: the construction of the calibration tables is now based on the standard calibrators fluxes coming from the central 9 spaxels instead of the central one only. This implies that two new calibration tables were introduced in the calibration tree, namely observedResponse3x3 and calSourceFlux3x3. Those tables are used by default in specDiffCs and should lead to more reliable fluxes, notably in case of mispointed observations. An optional parameter, use3x3 (True by default), was added to specDiffCs. You can still use the former calibration scheme by setting use3x3 to False although it is not recommended.
There is no change in the PACS photometer calibration since Hipe 9 and version 9 of the calibration tree.
Spectroscopy
In order to be less sensitive to (on average) mispointed flux calibration observations, the flux calibration scheme has changed: the construction of the calibration tables is now based on the standard calibrators fluxes coming from the central 9 spaxels instead of the central one only. This implies that two new calibration tables were introduced in the calibration tree, namely observedResponse3x3 and calSourceFlux3x3. Those tables are used by default in specDiffCs and should lead to more reliable fluxes, notably in case of mispointed observations. An optional parameter, use3x3 (True by default), was added to specDiffCs. You can still use the former calibration scheme by setting use3x3 to False although it is not recommended.
As a result of the new calibration scheme, the absolute fluxes found at level 2 are slightly reduced.
Line: 301 to 304
Product changes
Photometry
Added:
> >
From SPG10 onwards, level 2 maps are processed with a very small high-pass filter width to filter as much the 1/f noise and get optimal sensitivity to point-sources. Half-widths of 15 and 25 readouts are used in the blue and red channel respectively. As a consequence extended emission at virtually all scales is filtered out. The level 2.5 maps processed with MADmap should be used to get extended emission, which is preserved at all scales in the MADmap maps. Note that SPG10 bulk pipeline processing will only start in early 2013.
Error maps in SPG10 level2 maps are produced by the task photCoverage2Noise, where error maps are derived from the coverage map and include cross-correlation noise.
Slices which are completely masked by this OUTOFBAND mask are discarded during the level-0.5 processing.
Spectra are now by default rebinned to a standard wavelength grid, so that we avoid incompatible wavelength grid when subtracting on and off spectra from different observations.
New useful script for point source correction.
Changed:
< <
Improvements in the flatfielding:
Fixed flux conservation problem in some cases with background normalization in specFlatFieldRange.
Switched to linear fitter in specFlatFieldLine.
Improved line flux conservation in cases of low continuum.
> >
Spectral flatfielding: several issues had been reported for the specific case of zero or weak continuum in specFlatFieldLine and specFlatFieldRange (e.g. non-central spaxels). These were fixed.
One can now slice the level-0 data with pacsSpecSliceLevel0 to reduce the memory consumption of the processing till level-0.5.
Changed:
< <
Some simplifications in the ipipe scripts.
Drizzle works now also with sliced products.
> >
Most of the ipipe scripts have been slightly simplified. Mainly, selectSlices has been removed: all tasks can cope with sliced products transparently, so selectSlices is optional.
Spectral drizzling is now offered for the line-scan observations (ChopNodLineScan, UnchoppedLineScan). Narrow wavelength ranges can be extracted from range-scans and fed in the drizzling as well. For this, use the new crop tasks described above
Fixed some wrong values of the 'onOffSource' meta keyword.
Fixed some inconsistencies in the units and labels of the pipeline products.
Added the missing keyword 'faintLines' to the observation context.
Line: 294 to 291
Photometry
Spectroscopy
In order to be less sensitive to (on average) mispointed flux calibration observations, the flux calibration scheme has changed: the construction of the calibration tables is now based on the standard calibrators fluxes coming from the central 9 spaxels instead of the central one only. This implies that two new calibration tables were introduced in the calibration tree, namely observedResponse3x3 and calSourceFlux3x3. Those tables are used by default in specDiffCs and should lead to more reliable fluxes, notably in case of mispointed observations. An optional parameter, use3x3 (True by default), was added to specDiffCs. You can still use the former calibration scheme by setting use3x3 to False although it is not recommended.
Added:
> >
As a result of the new calibration scheme, the absolute fluxes found at level 2 are slightly reduced.
For extended sources, the new calibration is more accurate
The new calibration should not impact point sources, thanks to the so-called "central to 3x3" correction performed in extractCentralSpectrum. That correction was not necessary in hipe 9.0, as it was implicitly contained in the flux calibration files. In hipe 10, by default no average mis-pointing is included in the calibration files anymore, hence this extra correction, making use of the flux distribution between the spaxels to estimate the mispointing and correct for it.
There is a a new task photCoverage2Noise, run in the SPG pipeline to derive an error map from the coverage map, including the correction for correlated noise.
Spectroscopy
There is a new task pacsExtractSpectralRange to spectrally crop a (Sliced)PacsCube or (Sliced)PacsRebinnedCube.
The cube tools part of the Spectrum Explorer has now a new tool to compare N cubes: One can link them now by world or pixel coordinates, so that selecting a spaxel in one cube, selects simultaneously the same spaxel in the other linked cubes.
Analysis
Data cubes
Line: 263 to 266
Photometry
Spectroscopy
Added:
> >
There is a new task pacsExtractSpectralRange to spectrally crop a (Sliced)PacsCube or (Sliced)PacsRebinnedCube.
The footprint of pointed observations is shown by projecting the rebinned cubes on a finer grid (0.5 arcsec pixelsize).
There is a new mask OUTOFBAND which masks the data at wavelengths where the signal is completely blocked by the used filter.
Slices which are completely masked by this OUTOFBAND mask are discarded during the level-0.5 processing.
Spectra are now by default rebinned to a standard wavelength grid, so that we avoid incompatible wavelength grid when subtracting on and off spectra from different observations.
New useful script for point source correction.
Improvements in the flatfielding:
Fixed flux conservation problem in some cases with background normalization in specFlatFieldRange.
Switched to linear fitter in specFlatFieldLine.
Improved line flux conservation in cases of low continuum.
One can now slice the level-0 data with pacsSpecSliceLevel0 to reduce the memory consumption of the processing till level-0.5.
Some simplifications in the ipipe scripts.
Drizzle works now also with sliced products.
Fixed some wrong values of the 'onOffSource' meta keyword.
Fixed some inconsistencies in the units and labels of the pipeline products.
Added the missing keyword 'faintLines' to the observation context.
Optimised the registration and initialisation of tasks. This can reduce HIPE start-up time by up to 20%.
Displaying images uses less memory and is much faster.
Data fitting is now more robust in the presence of heavy outliers.
You can now edit table metadata directly in the GUI. Use the Add parameter, Edit parameter and Remove parameter entries from the table's popup menu.
You can now send the results of a query in the HSA User Interface directly to HIPE as a table dataset.
HIFI
Introduction of a pipeline task to allow comparison of emission in DBS chop positions, !mkDbsReference task. It differences the chop positions in all DBS observations and after applying the band-pass correction stores them in a product in calibration->pipeline-out called ReferenceSpectra.
Major updates to flagTool
Line: 40 to 46
Tables in viewers
Changed:
< <
Metadata can be edited directly in the GUI. It can be done by means of the options Add parameter, Edit parameter and Remove parameter in the table's popup menu.
Cells in table dataset viewers can be edited by selecting Allow editing in the popup menu, then double clicking on the target cells and writing the new value. A corresponding Jython command will be echoed.
Integral numbers in tables are left justified, as any other value. To easily compare them, an option to align them at the right is available in the table's popup mnu.
> >
You can edit metadata directly in the GUI. It can be done by means of the options Add parameter, Edit parameter and Remove parameter in the table's popup menu.
You can edit cells in table dataset viewers by selecting Allow editing in the popup menu, then double clicking on the target cells and writing the new value. A corresponding Jython command will be echoed.
Integer numbers in tables are left justified, as any other value. To easily compare them, an option to align them at the right is available in the table's popup menu.
Variables view
Changed:
< <
The variables panel is now split in two: observations at the top, other variables at the bottom. The old single panel can still be shown by pressing the button with tip Show single panel within the view's toolbar. The user's choice is remembered among HIPE sessions.
> >
The variables panel is now split in two: observations at the top, other variables at the bottom. The old single panel can still be shown by pressing the button with tip Show single panel within the view's toolbar. Your choice is remembered across HIPE sessions.
File and directory selection
Line: 64 to 70
OverPlotter
Astronomical utilities
Changed:
< <
New package herschel.ia.toolbox.astro.healpix (based on JPL Healpix Java library)
> >
New package herschel.ia.toolbox.astro.healpix (based on JPL Healpix Java library).
Healpix interpolation functionality
radialV task: removed corrections parameter (enum), replaced with correct (String).
Line: 113 to 119
Usability improvements. The selection of which pools to query is now more user-friendly, and right-click menus for certain actions on pools have been implemented. New pools can be created directly from the Product Browser by right-clicking, and local pools can be moved, renamed, deleted, etc.
Tasks
Changed:
< <
Speed-up of the registration and initialization of tasks. Tasks no longer create a default GUI (in Views). If needed, this GUI will be created on demand. This can speed up start-up time of hcss.dp.core by 20%.
> >
Speed-up of the registration and initialization of tasks. Tasks no longer create a default GUI (in Views). If needed, this GUI will be created on demand. This can reduce HIPE start-up time by up to 20%.
Task Preferences
Preference to detect data retention in task parameters: when you reclaim memory via the "dustbin" button, tasks are checked to see if their parameters have data. You can then select which tasks to reset.
The system will now issue a warning if a task has apparently completed to 100% and more than 3 seconds pass before it has really completed execution (it detects tasks that "lie" about their completion status).
Task GUIs Help button will fall-back to on-line documentation if local documentation is not available.
Improved (updated) the information provided from tasks in the console (deprecated features and errors in tasks)
Changed:
< <
pause & resume are no longer tasks but just plain functions. They appear documented in the UM (they were in the URM before).
> >
pause & resume are no longer tasks but just plain functions. They are documented in the Scripting Guide (they were in the User's Reference Manual before).
open* (Task, View, File) tasks open a confirmation dialogue if they fail to execute.
You can ask for a list with all outputs when you call a task passing __list__=True as an argument (in beta, may be removed, experimental): out, out2, out3 = taskName(arg1, ..., argN, __list__=True)
For Jython task developers: the following wrappers (decorators) have been added to support working with the EDT: execLater, execAndWait (similar to the SwingUtils functions). An example is provided in the source of ia_task tests (TestTaskForViewsEdt.py)
Line: 133 to 139
poolDataReader , poolDataWriter, loadObs, downObs and localStoreWriter removed : saveProduct and MyHSA (getObservation) contain a superset of their features.
Tables
asciiTableReader task improvements:
Changed:
< <
Documentation: More examples, better tooltips and manual page (URM)
added more values to tableType : IPAC, SExtractor. SPACES now also reads TopCat ASCII Format
Better support for table header reading
New task sortTable will return a new table where rows have been sorted according to user criteria
> >
Documentation: More examples, better tooltips and manual page (URM).
added more values to tableType : IPAC, SExtractor. SPACES now also reads TopCat ASCII Format.
Better support for table header reading.
New task sortTable will return a new table where rows have been sorted according to user criteria.
FITS
fitsReader and simpleFitsReader 's type parameters can now accept strings (not just Java enum values).
Changed:
< <
Import View will check and ask for confirmation if the expected memory needs exceed the available memory
> >
Import View will check and ask for confirmation if the expected memory needs exceed the available memory.
Virtual Observatory
Line: 260 to 266
Calibration
Photometry
Spectroscopy
Changed:
< <
In order to be less sensitive to (on average) mispointed flux calibration observations, the flux calibration scheme has changed: the construction of the calibration tables is now based on the standard calibrators fluxes coming from the central 9 spaxels instead of the central one only. This implies that two new calibration tables were introduced in the calibration tree, namely observedResponse3x3 and calSourceFlux3x3. Those tables are used by default in specDiffCs and should lead to more reliable fluxes, notably in case of mispointed observations. An optional parameter, use3x3 (True by default), was added to specDiffCs. The user still can use the former calibration scheme by setting use3x3 to False although it is not recommended.
> >
In order to be less sensitive to (on average) mispointed flux calibration observations, the flux calibration scheme has changed: the construction of the calibration tables is now based on the standard calibrators fluxes coming from the central 9 spaxels instead of the central one only. This implies that two new calibration tables were introduced in the calibration tree, namely observedResponse3x3 and calSourceFlux3x3. Those tables are used by default in specDiffCs and should lead to more reliable fluxes, notably in case of mispointed observations. An optional parameter, use3x3 (True by default), was added to specDiffCs. You can still use the former calibration scheme by setting use3x3 to False although it is not recommended.
In order to be less sensitive to (on average) mispointed flux calibration observations, the flux calibration scheme has changed and is now based on the central 9 spaxels instead of the central one only. This implies that two new calibration tables were introduced, namely observedResponse3x3 and calSourceFlux3x3. Those tables are used by default in specDiffCs and should lead to more reliable fluxes, notably in case of mispointed observations. An optional parameter, use3x3 (True by default), was added to specDiffCs. The user still can use the former calibration scheme by setting use3x3 to _False although it is not recommended.
> >
In order to be less sensitive to (on average) mispointed flux calibration observations, the flux calibration scheme has changed: the construction of the calibration tables is now based on the standard calibrators fluxes coming from the central 9 spaxels instead of the central one only. This implies that two new calibration tables were introduced in the calibration tree, namely observedResponse3x3 and calSourceFlux3x3. Those tables are used by default in specDiffCs and should lead to more reliable fluxes, notably in case of mispointed observations. An optional parameter, use3x3 (True by default), was added to specDiffCs. The user still can use the former calibration scheme by setting use3x3 to False although it is not recommended.
In order to be less sensitive to (on average) mispointed flux calibration observations, the flux calibration scheme has changed and is now based on the central 9 spaxels instead of the central one only. This implies that two new calibration tables were introduced, namely observedResponse3x3 and calSourceFlux3x3. Those tables are used by default in specDiffCs and should lead to more reliable fluxes, notably in case of mispointed observations. An optional parameter, use3x3 (True by default), was added to specDiffCs. The user still can use the former calibration scheme by setting use3x3 to _False although it is not recommended.
User releases for Windows and Linux already include the correct Java version.
Developer releases and user releases for Mac do not include Java. You must ensure that you have a compatible version of Java installed on your system.
If the minor version of your installed Java is different from the recommended one (for instance, 1.6.0_35 instead of 1.6.0_37) HIPE prints a warning in the command line window from which it was started. Minor version differences are unlikely to cause problems, but this cannot be guaranteed.
Changed:
< <
If the major version of your installed Java is different (for instance, 1.5.0 instead of 1.6.0) HIPE shows a warning in a dialogue window at startup. Major version differences are very likely to cause problems. We do not recommend that you try to run HIPE 10.0 with a Java version other than 1.6.0. In particular, HIPE 10 is known to give incorrect results if used under Java 7.
> >
If the major version of your installed Java is different (for instance, 1.5.0 instead of 1.6.0) HIPE shows a warning in a dialogue window at startup. Major version differences are very likely to cause problems. We do not recommend that you try to run HIPE 10.0 with a Java version other than 1.6.0. In particular, HIPE is known to give incorrect results if used under Java 7 with a locale different from en_US.
<-- ---+++HIFI Software Configuration and Properties -->
> >
HIFI Software Configuration and Properties
HIFI Calibration
Updates to table of unusable data frames
Line: 193 to 193
A new quality flag is introduced to warn of a single COMB failure
Improved checks and threshold values for the ZERO spectra.
Changed:
< <
<-- ---++++ HRS Pipeline -->
> >
HRS Pipeline
Level 1 Pipeline
Changed:
< <
Addition of a task to allow comparison of emission in DBS chop positions, !mkDbsReference task. Differences the chop positions in all DBS observations and after applying the band-pass correction stores them in a product in calibration->pipeline-out called ReferenceSpectra. You may configure the task to average the differenced spectra together (a resampling is automatically done) and to resample the spectra yourself.
> >
Addition of a task to allow comparison of emission in DBS chop positions, mkDbsReference task. Differences the chop positions in all DBS observations and after applying the band-pass correction stores them in a product in calibration->pipeline-out called ReferenceSpectra. You may configure the task to average the differenced spectra together (a resampling is automatically done) and to resample the spectra yourself.
Additional parameters available to DoFilterLoads
Level 2 Pipeline
Line: 211 to 211
HifiUplinkProduct format changed to a table dataset (easier to read)
HIFI Spectrum Toolbox
Changed:
< <
<-- ---++++ Spectrum Explorer (HIFI specifics) -->
> >
Spectrum Explorer (HIFI specifics)
Spectrum tasks (HIFI specifics)
Changed:
< <
A new task, !doOffset, to calculate the offset positions from the requested position is introduced. You can use it to switch between offset and absolute coordinates.
> >
A new task, doOffset, to calculate the offset positions from the requested position is introduced. You can use it to switch between offset and absolute coordinates.
DoFold task now doubles in the reported integration times in the metadata after folding.
HIFI Data Processing Tools
Standing Wave Removal
Added:
> >
fitHifiFringe can be directly applied to spectra in HIFI mapping mode cubes through new input option 'cube'.
Baseline Removal
Added:
> >
Changed default for 'movemasks' option from 2 to 0 in fitBaseline, i.e., masks are not propagated between spectra.
Improved performance stop button in non-interactive mode in fitBaseline
Deconvolution
Added a 'BRIGHT_LINE' toggle. If enabled, doDeconvolution will ignore data set with this channel flag. Useful for preventing `ghosts' of bright lines caused by the iterative nature of the deconvolution algorithm. You need to manually mask these lines using, for example, flagTool.
Plotting output has been cleaned up and simplified.
Line: 233 to 239
All subbands can be displayed together (without gluing).
Flag type can be selected from the options above the plot instead of from a context menu
It is possible to undo-redo in the console and in text fields for task parameters, by means of Ctrl-Z / Ctrl-Y (Linux & Windows) or Cmd-Z / Cmd-Y (OS X).
The maximum memory value shown in the memory bar reflects more accurately the actual memory available to HIPE. This value can vary slightly during a HIPE session.
The Locale is established as English so that the decimal separator will be a dot even in computers with language settings that use a comma for it. This ensures proper processing of tasks relying on it being a dot.
Deleted:
< <
The Logging preferences panel is now available for installations without the PACS instrument too.
Metadata can be edited directly in the GUI. It can be done by means of the options Add parameter, Edit parameter and Remove parameter in the table's popup menu.
Cells in table dataset viewers can be edited by selecting Allow editing in the popup menu, then double clicking on the target cells and writing the new value. A corresponding Jython command will be echoed.
Added:
> >
Integral numbers in tables are left justified, as any other value. To easily compare them, an option to align them at the right is available in the table's popup mnu.
Variables view
Line: 54 to 55
Miscellanea
Added:
> >
It is possible to undo-redo in the console and in text fields for task parameters, by means of Ctrl-Z / Ctrl-Y (Linux & Windows) or Cmd-Z / Cmd-Y (OS X).
The maximum memory value shown in the memory bar reflects more accurately the actual memory available to HIPE. This value can vary slightly during a HIPE session.
Added:
> >
The Locale is established as English so that the decimal separator will be a dot even in computers with language settings that use a comma for it. This ensures proper processing of tasks relying on it being a dot.
The Logging preferences panel is now available for installations without the PACS instrument too.
Introduction of a pipeline task to allow comparison of emission in DBS chop positions, !mkDbsReference task. It differences the chop positions in all DBS observations and after applying the band-pass correction stores them in a product in calibration->pipeline-out called ReferenceSpectra.
Major updates to flagTool
Rotated cubes are now produced in the Level 2.5 product by default for maps carried out with a non-zero position angle, a non-rotated cube is also produced.
PACS
Line: 189 to 192
<-- ---++++ HRS Pipeline -->
Level 1 Pipeline
Changed:
< <
Addition of a task to allow comparision of emission in DBS chop positions, !mkDbsReference task. Differences the chop positions in all DBS observations and after applying the band-pass correction stores them in a product in calibration->pipeline-out called ReferenceSpectra. You may configure the task to average the differenced spectra together (a resampling is automatically done) and to resample the spectra yourself.
> >
Addition of a task to allow comparison of emission in DBS chop positions, !mkDbsReference task. Differences the chop positions in all DBS observations and after applying the band-pass correction stores them in a product in calibration->pipeline-out called ReferenceSpectra. You may configure the task to average the differenced spectra together (a resampling is automatically done) and to resample the spectra yourself.
<-- ---+++HIFI Software Configuration and Properties -->
HIFI Calibration
Added:
> >
Updates to table of unusable data frames
Quality flags are now raised for monitoring SEUs (Sudden Event Upsets)
HIFI Pipeline
hifiPipeline task
Changed:
< <
Added a quality flag that is raised when doDeconvolution cannot complete in the Level 2.5 pipeline
> >
The SPG version is now updated in the meta data when you run the pipeline
The pipeline will only try to run for spectrometers for which there is data at level 0 (previously the pipeline always tried to run for all spectrometers)
The option to customise the pipeline from the GUI is disabled by default when starting processing from Level 2
Mapping browse products changed so that the average spectrum and integrated cubes are shown for each subband separately
Level 0 Pipeline
Added:
> >
The DoBadLo step (a step to flag out datasets known to be bad or unnecessary repeats) is reprocessed when a new calibration is passed to the pipeline.
WBS Pipeline
Changed:
< <
HRS Pipeline
> >
A new quality flag is introduced to warn of a single COMB failure
Improved checks and threshold values for the ZERO spectra.
<-- ---++++ HRS Pipeline -->
Level 1 Pipeline
Addition of a task to allow comparision of emission in DBS chop positions, !mkDbsReference task. Differences the chop positions in all DBS observations and after applying the band-pass correction stores them in a product in calibration->pipeline-out called ReferenceSpectra. You may configure the task to average the differenced spectra together (a resampling is automatically done) and to resample the spectra yourself.
Added:
> >
Additional parameters available to DoFilterLoads
Level 2 Pipeline
Added:
> >
DoAvgTask now propagates ra_center and dec_center to metadata
A new task, MkFLagSummaryTable, creates a table containing all the flags raised by the pipeline for quick inspection
Level 2.5 Pipeline
Added:
> >
Rotated cubes are now produced by default for maps carried out with a non-zero position angle, a non-rotated cube is also produced.
Flagged data are omitted from spectral cubes
Added a quality flag that is raised when doDeconvolution cannot complete in the Level 2.5 pipeline
HIFI Products
HifiUplinkProduct format changed to a table dataset (easier to read)
HIFI Spectrum Toolbox
Changed:
< <
Spectrum Explorer (HIFI specifics)
> >
<-- ---++++ Spectrum Explorer (HIFI specifics) -->
Spectrum tasks (HIFI specifics)
Added:
> >
A new task, !doOffset, to calculate the offset positions from the requested position is introduced. You can use it to switch between offset and absolute coordinates.
DoFold task now doubles in the reported integration times in the metadata after folding.
HIFI Data Processing Tools
Standing Wave Removal
Line: 196 to 217
Added a 'BRIGHT_LINE' toggle. If enabled, doDeconvolution will ignore data set with this channel flag. Useful for preventing `ghosts' of bright lines caused by the iterative nature of the deconvolution algorithm. You need to manually mask these lines using, for example, flagTool.
Plotting output has been cleaned up and simplified.
doDeconvolution now gracefully exits whenever it fails to process an observation (occurs only for the most troublesome of data and is rare).
Added:
> >
Spectral Cube creation
Added:
> >
The GUI will by default pick up the rotation angle (flyAngle in the GUI) used when the map was observed. When the pipeline is run with the interactive level 2.5 pipeline selected or modified in any way then only this angle will be used, when the pipeline is run with all the default settings then non-rotated and rotated cubes will be produced. In the case that the flyAngle is zero only one set of cubes will be created.
flagTool
Added:
> >
Major updates to make the tool faster, more intuitive and easier to use:
All data are displayed in a table to the left of the plot, enabling you to quickly select the datasets you want to flag without needing to cycle through all the data if not required.
All subbands can be displayed together (without gluing).
Flag type can be selected from the options above the plot instead of from a context menu
Changed:
< <
HiClass export tool
> >
<-- ---+++ HiClass export tool -->
Documentation
Added:
> >
Updates to the HIFI Data Reduction Guide:
Addition of a chapter describing how to convert positions in data to offsets
Improvements and updates to "How to add and remove flags" chapter, including documentation of new flagTool in the "Setting flags interactively for many spectra" section
Improvements and updates to "Running the HIFI pipeline" chapter
Update to mapping browse product description in "Tour of a HIFI observation context" chapter
Removal of references to the old CubeSpectrumAnalysisToolbox and replacement with references to the new Cube Toolbox
Various typos, improvements to style consistency and bug fixes.
Level1SpireMaskEditor is added to display/edit mask flags for Level1 data
Level0_5SpireMaskEditor is added to display/edit mask flags for Level0_5 data
Added:
> >
baselineRemovalPolynomial
"outlierRejectionThreshold" parameter renamed to "outlierThreshold"
Spectrometer Pipeline
Added:
> >
History is now propagated from level-1 products to level-2 products when spirePreprocessCube and spireProjection are used in the spectrometer pipeline.
Source extraction
sourceExtractorTimeline
Significant speed improvements
Parameters renamed: "fitEllipticalGaussian" renamed to "fitEllipticalGauss2d" and "modelGauss2DFixedWidth" renamed to "modelGauss2dSigma".
Interactive Analysis and Tools
Added:
> >
SpectrometerDetectorSpectrum (SDS) products have more functionality
Can be opened and used in the Spectrum Explorer
More tasks available
Arithmetic operations (+, -, *, /) work in scripts, when performing calculations between an SDS product and either another SDS product, a spectrum, or a number
Observation Viewer:
More information displayed in the viewer for level 0 and level 0.5 products
SDS Explorer
N-E arrows added to show orientation of the arrays
convolutionMapper
New task to create a kernel-filtered map directly from timeline data
Metadata can be edited directly in the GUI. It can be done by means of the options Add parameter, Edit parameter and Remove parameter in the table's popup menu.
Added:
> >
Cells in table dataset viewers can be edited by selecting Allow editing in the popup menu, then double clicking on the target cells and writing the new value. A corresponding Jython command will be echoed.
Variables view
The variables panel is now split in two: observations at the top, other variables at the bottom. The old single panel can still be shown by pressing the button with tip Show single panel within the view's toolbar. The user's choice is remembered among HIPE sessions.
Added:
> >
File and directory selection
The open and save dialogs include user defined areas in the drop-down list labeled Look In or Save In.
When selecting a directory with a file chooser, files are also shown, although greyed out to indicate that they cannot be selected.
Miscellanea
Changed:
< <
The maximum memory value shown in the memory bar now reflects more accurately the actual memory available to HIPE. This value can vary slightly during a HIPE session.
When selecting a directory with a file chooser, files are also shown, though greyed out to indicate that they cannot be selected.
> >
The maximum memory value shown in the memory bar reflects more accurately the actual memory available to HIPE. This value can vary slightly during a HIPE session.
Metadata can be edited directly in the GUI. It can be done by means of the options Add parameter, Edit parameter and Remove parameter in the table's popup menu.
Variables view
The variables panel is now split in two: observations at the top, other variables at the bottom. The old single panel can still be shown by pressing the button with tip Show single panel within the view's toolbar. The user's choice is remembered among HIPE sessions.
Miscellanea
The maximum memory value shown in the memory bar now reflects more accurately the actual memory available to HIPE. This value can vary slightly during a HIPE session.
Added:
> >
When selecting a directory with a file chooser, files are also shown, though greyed out to indicate that they cannot be selected.
Plotting
TablePlotter
OverPlotter
Deleted:
< <
Dataset viewer
Astronomical utilities
New package herschel.ia.toolbox.astro.healpix (based on JPL Healpix Java library)
User releases for Windows and Linux already include the correct Java version.
Developer releases and user releases for Mac do not include Java. You must ensure that you have a compatible version of Java installed on your system.
If the minor version of your installed Java is different from the recommended one (for instance, 1.6.0_35 instead of 1.6.0_37) HIPE prints a warning in the command line window from which it was started. Minor version differences are unlikely to cause problems, but this cannot be guaranteed.
Changed:
< <
If the major version of your installed Java is different (for instance, 1.5.0 instead of 1.6.0) HIPE shows a warning in a dialogue window at startup. Major version differences are very likely to cause problems. We do not recommend that you try to run HIPE 10.0 with a Java version other than 1.6.0.
> >
If the major version of your installed Java is different (for instance, 1.5.0 instead of 1.6.0) HIPE shows a warning in a dialogue window at startup. Major version differences are very likely to cause problems. We do not recommend that you try to run HIPE 10.0 with a Java version other than 1.6.0. In particular, HIPE 10 is known to give incorrect results if used under Java 7.
Increased robustness in the case of fitting in the presence of heavy outliers.
Data interpolation
Random numbers
Statistics
Line: 155 to 156
HIFI Pipeline
hifiPipeline task
Added:
> >
Added a quality flag that is raised when doDeconvolution cannot complete in the Level 2.5 pipeline
Level 0 Pipeline
WBS Pipeline
HRS Pipeline
Level 1 Pipeline
Added:
> >
Addition of a task to allow comparision of emission in DBS chop positions, !mkDbsReference task. Differences the chop positions in all DBS observations and after applying the band-pass correction stores them in a product in calibration->pipeline-out called ReferenceSpectra. You may configure the task to average the differenced spectra together (a resampling is automatically done) and to resample the spectra yourself.
Level 2 Pipeline
Level 2.5 Pipeline
HIFI Products
Added:
> >
HifiUplinkProduct format changed to a table dataset (easier to read)
HIFI Spectrum Toolbox
Spectrum Explorer (HIFI specifics)
Line: 172 to 176
Standing Wave Removal
Baseline Removal
Deconvolution
Added:
> >
Added a 'BRIGHT_LINE' toggle. If enabled, doDeconvolution will ignore data set with this channel flag. Useful for preventing `ghosts' of bright lines caused by the iterative nature of the deconvolution algorithm. You need to manually mask these lines using, for example, flagTool.
Plotting output has been cleaned up and simplified.
doDeconvolution now gracefully exits whenever it fails to process an observation (occurs only for the most troublesome of data and is rare).
Added a diagnosticDetector task parameter for diagnostic purpose
All the long task parameters are shortened
The parameter types for detectGlitches, detectLargeGl and repairGlitches parameters are changed to Boolean.
Photometer Pipeline
New zeroPointCorrection task for cross-calibration with Planck HFI maps. This can only be run at the HSC until the HFI maps are made public. The output maps are included in the Level-2 context.
New ssoMotionCorrection task to produce modified Level-1 timelines adjusted to the reference frame of the moving object. Additional maps are created from this timeline and included in the Level-2 context.
Added:
> >
DestriperTask
Added the minVel, maxVel, chanNoise and method task parameters to be in synch with naive mapper.
Added useOnlyMasks and ignoreDefaultMasks task parameters in synch with map mapper.
Added jumpThresh and jumpIter task parameters for jump detections.
Two second level deglitch algorithms are added, sigma-kappa and two sigmas
The "Save Product to Pool" view used to require that Enter was pressed after entering a tag to use when saving the product. This is now longer necessary. User friendliness of the view is also improved by providing direct user feedback when a tag is entered that is not valid, by coloring the cell red as you type.
Product Browser
Usability improvements. The selection of which pools to query is now more user-friendly, and right-click menus for certain actions on pools have been implemented. New pools can be created directly from the Product Browser by right-clicking, and local pools can be moved, renamed, deleted, etc.
Tasks
Speed-up of the registration and initialization of tasks. Tasks no longer create a default GUI (in Views). If needed, this GUI will be created on demand. This can speed up start-up time of hcss.dp.core by 20%.
It is now possible to send the results of a query in the HSA User Interface directly to HIPE. The results will appear as TableDataset, allowing one to use the geometric search in the HSA UI (for instance), and directly use the result in a script, or save it to disk.
Calibration sources
Products and datasets
Line: 107 to 109
Virtual Observatory
Added:
> >
Improved stability when sending data (on some platforms, sending data from within observations could cause problems, due to quotes being included in the name of an intermediate file).
Fix issues with WebStart applications (such as Aladin, HSA User Interface): Problems starting the applications on Windows XP, problems with automatic log-in.
HIPE 10.0 was built and tested against the Oracle Java Runtime Environment (JRE) version 1.6.0_37.
User releases for Windows and Linux already include the correct Java version.
Developer releases and user releases for Mac do not include Java. You must ensure that you have a compatible version of Java installed on your system.
If the minor version of your installed Java is different from the recommended one (for instance, 1.6.0_35 instead of 1.6.0_37) HIPE prints a warning in the command line window from which it was started. Minor version differences are unlikely to cause problems, but this cannot be guaranteed.
If the major version of your installed Java is different (for instance, 1.5.0 instead of 1.6.0) HIPE shows a warning in a dialogue window at startup. Major version differences are very likely to cause problems. We do not recommend that you try to run HIPE 10.0 with a Java version other than 1.6.0.
New package herschel.ia.toolbox.astro.healpix (based on JPL Healpix Java library)
Healpix interpolation functionality
radialV task: removed corrections parameter (enum), replaced with correct (String).
Numeric routines
Data fitting
Line: 67 to 71
Speed-up of the registration and initialization of tasks. Tasks no longer create a default GUI (in Views). If needed, this GUI will be created on demand. This can speed up start-up time of hcss.dp.core by 20%.
Task Preferences
Preference to detect data retention in task parameters: when you reclaim memory via the "dustbin" button, tasks are checked to see if their parameters have data. You can then select which tasks to reset.
Changed:
< <
Pause & Resume are no longer tasks but just plain functions. They appear documented in the UM.
> >
The system will now issue a warning if a task has apparently completed to 100% and more than 3 seconds pass before it has really completed execution (it detects tasks that "lie" about their completion status).
Task GUIs Help button will fall-back to on-line documentation if local documentation is not available.
Improved (updated) the information provided from tasks in the console (deprecated features and errors in tasks)
pause & resume are no longer tasks but just plain functions. They appear documented in the UM (they were in the URM before).
open* (Task, View, File) tasks open a confirmation dialogue if they fail to execute.
You can ask for a list with all outputs when you call a task passing __list__=True as an argument (in beta, may be removed, experimental): out, out2, out3 = taskName(arg1, ..., argN, __list__=True)
For Jython task developers: the following wrappers (decorators) have been added to support working with the EDT: execLater, execAndWait (similar to the SwingUtils functions). An example is provided in the source of ia_task tests (TestTaskForViewsEdt.py)
Quality control
Systematic product generation
Data input-output
Added:
> >
poolDataReader , poolDataWriter, loadObs, downObs and localStoreWriter removed : saveProduct and MyHSA (getObservation) contain a superset of their features.
Tables
asciiTableReader task improvements:
Documentation: More examples, better tooltips and manual page (URM)
added more values to tableType : IPAC, SExtractor. SPACES now also reads TopCat ASCII Format
Better support for table header reading
New task sortTable will return a new table where rows have been sorted according to user criteria
FITS
fitsReader and simpleFitsReader 's type parameters can now accept strings (not just Java enum values).
Import View will check and ask for confirmation if the expected memory needs exceed the available memory
The Pcal products now have correctly filled in metadata.
Engineering Conversion
Common Pipeline
Photometer Pipeline
Added:
> >
New zeroPointCorrection task for cross-calibration with Planck HFI maps. This can only be run at the HSC until the HFI maps are made public. The output maps are included in the Level-2 context.
New ssoMotionCorrection task to produce modified Level-1 timelines adjusted to the reference frame of the moving object. Additional maps are created from this timeline and included in the Level-2 context.
The maximum memory value shown in the memory bar now reflects more accurately the actual memory available to HIPE. This value can vary slightly during a HIPE session.
Speed-up of the registration and initialization of tasks. Tasks no longer create a default GUI (in Views). If needed, this GUI will be created on demand. This can speed up start-up time of hcss.dp.core by 20%.
Task Preferences
Preference to detect data retention in task parameters: when you reclaim memory via the "dustbin" button, tasks are checked to see if their parameters have data. You can then select which tasks to reset.
Added:
> >
Pause & Resume are no longer tasks but just plain functions. They appear documented in the UM.
Speed-up of the registration and initialization of tasks. Tasks no longer create a default GUI (in Views). If needed, this GUI will be created on demand. This can speed up start-up time of hcss.dp.core by 20%.
Task Preferences
Preference to detect data retention in task parameters: when you reclaim memory via the "dustbin" button, tasks are checked to see if their parameters have data. You can then select which tasks to reset.