Interested in what's new for other versions of HIPE? See this page for links to all the What's New documents from HIPE 4.0 onwards.
This document lists the changes in HIPE version 9.0 with respect to the 8.x series. Additional pages list changes between minor versions of the 9.x series.
New cube toolbox. The nex toolbox replaces the Cube Spectrum Analysis Toolbox (CSAT). Common look and feel with the Spectrum Explorer and improved algorithms.
Easier data access. Handling Herschel data is now done via the new MyHSA tool and the improved getObservation task.
Human-made index. Added to three core manuals: HIPE Owner's Guide, Data Analysis Guide and Scripting and Data Mining Guide.
Comments on documentation. You can now leave comments at the bottom of any page in the HIPE Help System.
Support to OSX Lion. HIPE is now officially supported on version 10.7 of Mac OS X.
HIFI
HIFI level 2.5 product introduced. For Spectral Scans it holds the deconvolved single sideband solutions; for Maps it holds the cubes; for all cases it holds a copy of the level 2 HTP.
New interactive pipeline between level 2 and 2.5. This allows the insertion of tasks like fitHifiFringe and fitBaseline into the pipeline between level 2 and 2.5.
Browse products added. For spectral scans and maps.
PACS
Photometer. Addressing additional specific cases in new interactive pipeline scripts.
Spectrometer pipeline processing. Improved pipeline processing for better reliability of broadband features.
Flux losses. Correction for flux losses due to pointing jitter and pointing offset
Flux extraction for point-sources. Use of central 3x3 spaxel fluxes for a more reliable beam correction
Convolution kernels. For photometer and spectrometer maps, aiding the interpretation of extended and point sources.
SPIRE
Destriper. Now this is the default task for baseline removal in both SPG and User pipelines.
Faster processing. Improved processing speed using multi-threading.
Cooler burps. Improved task for correction of cooler burps.
Photometer pipeline processing. Improved error maps and baseline correction.
Spectrometer calibration. Improved point source and bright mode calibration. Better calibration for low resolution observations.
Spectrometer scripts. New and updated useful scripts for background subtraction and spectral noise.
Java version
HIPE 9.0 was built and tested against the Oracle Java Runtime Environment (JRE) version 1.6.0_33.
User releases for Windows and Linux already include the correct Java version.
Developer releases and user releases for Mac do not include Java. You must ensure that you have a compatible version of Java installed on your system.
If the minor version of your installed Java is different from the recommended one (for instance, 1.6.0_31 instead of 1.6.0_33) HIPE prints a warning in the command line window from which it was started. Minor version differences are unlikely to cause problems, but this cannot be guaranteed.
If the major version of your installed Java is different (for instance, 1.5.0 instead of 1.6.0) HIPE shows a warning in a dialogue window at startup. Major version differences are very likely to cause problems. We do not recommend that you try to run HIPE 9.0 with a Java version other than 1.6.0.
Core system
HIPE
Status bar
Execution of for loops is reflected in the progress bar by being increased along with the percentage of executed iterations.
Startup & shutdown
It is possible to set a number of py files to be executed at startup, in the new Imports preferences panel.
The main window's state (size, location, maximized state) is remembered among HIPE sessions, so it is not necessary to set it after every startup.
Check of local pool locks (probably left from a previous session that was abnormally terminated) is executed at startup. This check is configurable in preferences.
The running Java version is compared with the supported one; if they are different, a warning is popped up. This check can be enabled or disabled at will, in preferences.
Editors & Viewers
The tabs within the Editor area show the close button even if not selected, so they can be closed with 1 click instead of 2.
The viewers shown in the Open With menu are the most specific ones for the selected variable.
The viewer for lists shows data values instead of just their type.
Autosave of text editors and Jython script editors is turned on by default.
The Jython Editor preferences now appear under the Text Editor preferences in the preferences window.
Boolean values in tables are shown as true / false instead of the previous less clear 1 / 0.
Tooltips in table headers include more information on the columns.
The load of a node in the product tree, when selected, can be cancelled with the Stop button.
Console & Log
Obsolete names (renamed or removed tasks, functions, etc.) can be registered by developers so meaningful messages, explaining the reason of the name removal, can be shown instead of a plain NameError.
Incremental search (with Ctrl+I) is now available in the Log view.
Navigator
The refresh period of the Navigator view can be configured in preferences.
User areas are included in the Open File dialogue.
Help
Video Tutorials entry has been added to the Help menu.
Context sensitive help on the focused view has been added. It can be accessed through Help > Help on <name> view or with F2.
If debug is enabled (see preferences, General > Debug), a threads dump can be obtained with the menu option Tools > Debug > Dump thread stacks. It can be helpful when informing the Helpdesk on software problems.
A new tab with a glossary of terms has been added to the Help System.
You can leave comments at the bottom of any help page. You must be connected to the Internet to read and leave comments.
You can get the full address of any help page, for sharing or adding to your browser's bookmarks. Click on the icon shaped like a chain in the Help System toolbar.
Many improvements to the search interface of the Help System:
You can now filter searches on any manual or group of manuals.
After each search, a feedback form asks you whether you found what you were looking for. Your feedback will be used to further improve the search engine.
Glossary terms are always at the top of search results.
Plotting
TablePlotter
TablePlotter now supports the axis.Date functionality.
OverPlotter
OverPlotter now supports the axis.Date functionality.
Source extraction
SUSSEXtractor algorithm implemented in a fully consistent manner, with detailed mathematical documentation provided
Defaults for sourceExtractorSussextractor now: useSignalToNoise=False, fitBackground=True, fluxPriorsLambda=0.0
"beamArea" parameter for sourceExtractorSussextractor deprecated
"inputSourceList" parameter now accepts [ra, dec] and other lists of values as input, as well as a SourceListProduct.
Many internal changes to the tasks, improving speed and consistency in applying the algorithms
Multi-threading: increasing speed of tasks significantly on multi-core machines
Images
Display
Reworked the save methods on Display. You can now save to jpg, png, eps or pdf files. The methods to use are :
save(filename), saveAsEPS(filename), saveAsPNG(filename), saveAsJPG(filename) and saveAsPDF(filename) : These methods will save the full image, including annotations.
saveCurrentView(filename), saveCurrentViewAsEPS(filename), saveCurrentViewAsPNG(filename), saveCurrentViewAsJPG(filename) and saveCurrentViewAsPDF(filename) : These methods will save the current view of the image, including annotations and axes.
The coordinate axes on the image panner are always visible now
There is a preference for image display to turn of the echoing of commands to the command line. It is possible to turn all these commands off, or only the zoom commands.
Flagged out pixels and pixels with a value of NaN will be shown in the color of the background.
Image can be matched with the Wcs. This means that the images can be oriented so that north is shown at the top and east to the left.
There is a preference for image display to always show the image with the 'match wcs' orientation.
A lot of bugfixes.
Spectra
Display
Major updates to Spectrum Explorer (SE) to improve performance and provide more direct control over plot appearance.
The Cube Spectrum Analysis Toolbox (CSAT) has been integrated into the Spectrum Explorer and is now called the Cube Toolbox:
The cube toolbox tasks can now be found under a icon in the SE toolbox.
All cube toolbox tasks use the standard HIPE Task GUI interface, with some additions to receive spectral and spatial selections from the SE. You can e.g. either fill in a spectral range, or select the range in the spectrum plot.
Each task has it own specialized selection tab, so that the original spectra selection do not disappear when one opens a task.
Analysis
The Smooth task is corrected for a bug in the ZEROS option.
As a consequence of this fix, which is more mathematically correct, it is found that smoothed spectra for SPIRE are quite changed. It is recommended instead to use the REPEAT option.
Introduction of a new "Accumulate" task allowing to automatically combine data taken on different frequency grids, with or without frequency overlap.
Improvements to the weights/error handling of the tasks in the Spectral Toolbox so that errors are now propagated correctly for the simple arithmetics tasks.
Herschel data access and download
Access to data in the Herschel Science Archive is now handled via the MyHSA utility, which dramatically improves performance and allows working with partial observations.
The old Import HSA data to HIPE task is no longer needed. MyHSA is mainly used through the improved getObservation task.
For more details see Chapter 1 of the Data Analysis Guide.
Results of testing performance improvements with 5 HIFI, 9 PACS and 10 SPIRE observations of all major observing modes. Tests were made with the old getObservation/saveObservation and with the new getObservation/MyHSA, with the same network connection on two different machines. Results show that downloads are 20 times faster and that 66% less hard disk space is needed to store the same data:
On-demand data reprocessing
All data levels are now regenerated (from level0 to level2).
All observation files are delivered, including auxiliary, calibration and level0 products.
The zip file with the results contains now an HSA hierarchical directory structure instead of a local pool structure.
For more details see Section 1.15 of the Data Analysis Guide.
Product Access Layer
When storing pools on shared NFS-mounted disks, HIPE now uses a more robust mechanism to prevent corruption. The default locking mechanism for Local Pools was changed to a mechanism that works more reliably on remote disks.
Usability improvements for the Product Browser and for working with tags. One can now update existing tags (for example to fix a typo).
Tasks
Task Preferences
Preference to generate full task call from the task GUIs
Preference to check input and inout parameter modification/replacement by tasks
Deprecated and removed tasks (names and parameters) documented in HIPE.
Task Viewer:
Drag and Drop of the (full) signature of tasks from Task Viewer to Editor & Console: whether to drop only mandatory parameters or the full signature is governed by the preference to generate full task calls
Reordering of nodes and reduced depth
Tooltips include the Categories and deprecated tasks appear crossed out
Task Outline:
Deprecated tasks & parameters marked
Data input-output
Deprecation of duplicated functionalities: poolDataWriter, localStoreWriter, poolDataReader. Use getObservation and saveProduct.
Deprecation of MyHSA prototype duplicated functionalities: loadObs and downObs. Use getObservation
getObservation is now a proper task.
simpleAsciiTableReader deprecated, use asciiTableReader.
Documentation
Quick Start Guide
Updated to reflect the latest changes in data access (MyHSA and eHSA).
HIPE Owner's Guide
Updated throughout to reflect the latest changes in HIPE.
In particular, the latest improvements in the help search engine are now described.
New calibration database HIFI_CAL_10_0, containing the following updates
Introduction of a new HifiUplink? product to back-fill most the observational parameters as of HSpot optimisation back into the Uplink product, and used at particular stages of the pipeline (e.g. cube making)
Update of a number of engineering threshold for more accurate quality flagging related to hardware housekeeping
Update of the list of a priori known corrupted data-frames for dedicated flagging in level0 data
HIFI Pipeline
hifiPipeline task
Addition of an interactive pipeline between level 2 and 2.5, which allows the insertion of tasks like fitHifiFringe and fitBaseline into the pipeline between level 2 and 2.5.
Inclusion in the pipeline GUI of a drop-down box to select calibration version
Spectrometer checkboxes in the pipeline GUI are now only checked if data for that spectrometer are available at level 0.
The DoBadLo step of the level 0 pipeline is now run when reprocessing WBS data from level 0 with new calibration. This is to allow you to take advantage of flags indicating bad LO settings that are raised by this pipeline step without having to wait for a new reduction from raw data to be available in the HSA.
Level 0 Pipeline
flagging of rare corrupted data-frames according to a list of pre-defined data-frames indices. The difference will only be noticed after the bulk reprocessing has taken place
revision of quality checks on diplexer out of limits, the effect is to reduce the number of benign quality flags.
WBS Pipeline
Introduction of quality flag to warn about a bad zero.
Introduction of a flag for a wrong scan count
Improvements to flagging of saturated channels
Improvements to algorithm to fit to COMB lines
HRS Pipeline
Level 1 Pipeline
Addition of parameters in the doFilterLoads to allow configuration of the filtering.
Addition of a new quality products "flagsSummary" compiling the list of all rowflags set to the data at this level
Level 2 Pipeline
doAvg now honours user-defined flags, RowMask.IGNORE_DATA and HifiMask.IGNORE_DATA
Level 2.5 Pipeline
Introduction of a level 2.5 pipeline
cube creation is moved from the level 2 pipeline to the level 2.5 pipeline for mapping observations (see Cube Creation section for updates to gridding done in the pipeline)
deconvolution is done for spectral scan observations (see Data Processing Tools -> Deconvolution section for updates to deconvolution done in the pipeline)
the browse product for all observing modes is done (see HIFI Products section for updates to browse products)
HIFI Products
Level 2.5 product introduced: for Spectral Scans it holds the deconvolved single sideband soluions; for Maps it holds the cubes; for all cases it holds a copy of the level 2 HTP.
Addition of browse products for spectral scans and maps. The spectral scan browse product is the deconvolved single sideband solution for H and V. The mapping browse product shows the map averaged spectra and the integrated map over the range of the WBS for the H- and V-polarisations.
Point mode browse product axis labels corrected.
Cubes for Solar System Objects are now created in the comoving frame of the target
Addition of a flag array to cubes, allowing pixels to be flagged in the same way as points in spectra can be.
HTP default view is changed from SpectrumExplorer to the ContextViewer
Observing band is added to HTP metadata
HIFI Spectrum Toolbox
Spectrum Explorer (HIFI specifics)
Introduction of a HIFI task panel in the Spectrum Explorer. Still in development
Spectrum tasks (HIFI specifics)
ConvertK2Jy now uses the same scheme as doGridding to select frequency when interpolating the efficiency values from the calibration tree: the task searches first for the LO Frequency in the metadata, if it is not found the task looks in the metadata for the value inserted in HSpot. If neither are found then the middle frequency of the observing band (e.g. band 1a) is used.
Bug fix in ConvertK2Jy: non-zero source size data on frequency scale can now be converted to Jy
ConvertK2Jy will accept data with any wavescale
HIFI Data Processing Tools
Standing Wave Removal
fitHifiFringe now accept SimpleSpectrum data so that, for example, spectra extracted from cubes can be processed.
All flag values are represented by 0.5 in the fitHifiFringe plots (masked lines by 1.0, as before). This avoids huge y-axis ranges.
Baseline Removal
fitBaseline has a new input parameter replace_weights. If set, the pipeline weight columns in the datasets are replaced by 1/(measured RMS noise)^2. This empirical weight is picked up by doDeconvolution to improve the weighting of the different channels.
The backends input parameter of fitBaseline allows for individual sideband products to be processed (USB,LSB) in the GUI (it was already possible on the command line).
The polynomial order used in a previous run of fitBaseline and stored in the fitResult tables of an obsContext is automatically applied if an observation is reprocessed with the obsContext input parameter. Thus the order=-1 option is now obsolete.
Deconvolution
When multiple observations are passed to doDeconvolution the metadata are now concatenated and put into the decon_result
doDeconvolution made more robust against failure when there are many bad datasets in the HTP.
A median subtraction is possible prior to running doDeconvolution.
Spectral Cube creation
Default output of doGridding changed from a cubes array to a MapContext, this will affect scripts in which a cube is extracted from the output of doGridding. See the HIFI Data Reduction Guide "How to make a Spectral Cube" chapter.
The comoving option is now automatically turned on for mapping observations of Solar System Objects
Fix to a bug in the way the pixel size was calculated for strip maps
Data flagged as bad_DATA or IGNORE_DATA are ignored during the cube creation.
flagTool
The flagTool task introduced in HIPE 8 is now fully functional and can process both spectrometers, both polarizations, and both subbands in one go.
Capabilities are:
choice of 4 flag types (LINE, BRIGHT_LINE, SPUR_CANDIDATE, IGNORE_DATA)
automatic flag propagation
natural zooming enabled in the plot window
easy flag switching
automatic merging and splitting of the flags
copying of flag from one polarization to the other for a same sideband
subband gluing (keyword doglue)
embedded use of fitHifiFringe, and embedded use of fitBaseline
HiClass export tool
Sideband gain information in the metadata are now exported to CLASS.
Documentation
Updates to DBS Cookbook
Updates to HIFI Data Reduction Guide for changes to:
fitBaseline
fitHifiFringe
PACS
Pipeline
Photometry
updated ipipe scripts with:
optimized second order deglitching with the MapDeglitch task,
optimized mask generation for the highpass filtering
now using the photMaskCrosstalk and filterSlew tasks
improved MADmap scripts with a new pre-processing task photBaselineDriftCorrection and a post-processing task to remove bright point-source artifacts photCorrMadmapArtifacts
the filterOnScanSpeed task now only masks the turnover frames instead of deleting them.
Spectroscopy
estimate of continuum RMS is included in plotting function of ipipe scripts
new task that extracts the central spaxel spectrum with a 3x3/central correction
the task specFlagGlitchFramesQTest can optionally be threaded
background normalization ipipe script introduced also for long range SED
improved flatfielding task, for line-scan mode
improved drizzling routines
available for chopped and unchopped mode
uses observed beam size for the calculation of the pixel scaling of the spatial grid
test versions of pointing correction tasks
improved error propagation
correction of SSO pointing task correctRaDec4Sso introduced in spg pipeline
level 2.5 product generation for unchopped range spectroscopy multi-observation mode (off-subtracted spectrum)
Calibration
Photometry
new calibration files for :
flat-field (flatField, version 4)
responsivity (version 7): the change in photometry for extended sources is about +3% in the red channel (160 microns), + 1.5% at 100 microns and none at 70 microns. For point-sources the change of photometry is minimal, about 2% in at 160 microns and less than 1% for usual photometry apertures at 70 and 100 microns.
aperture correction (apertureCorrection v3)
non-linearity correction ( nonLinearCoef v2)
Spectroscopy
new calibration product offRatio for all bands and chopper throws
new calibration product beams (raster beams of neptune) for all bands
update of the pointSourceLoss calibration product
new calibration product beamSize
new viewer for the calibration set release notes
Analysis tools
Spectroscopy
refactoring of the spectrum explorer
improved GUI for depiction of the spectrometer footprint
Product changes
Spectroscopy
complete browse product generation
quality flag "SLOWSAMP" added in case of low sampled spectra (level 2 products)
Documentation
improvements of URM
split of the PDRG into two: Spectroscopy and Photometry
SPIRE
Calibration Products
These changes are in the SPIRE_CAL_9_1 calibration tree.
Photometer:
A column containing frequency in GHz was added to the Photometer filter profile product, SCalPhotRsrf. This now matches Spectrometer data which are in frequency and not wavenumber.
Electrical crosstalk coefficients introduced for bolometer-thermistor crosstalk in the SCalPhotElecCross product. This means the electrical crosstalk module can be included in the pipeline - it affects the thermistor timelines but not the bolometer timelines.
A new product was added:
SCalSpecPcalModel - this contains a model PCAL flash profile, used in the new PCAL processing algorithm
Spectrometer:
Point source calibration updated in the SCalSpecBeamParam product to:
use a new Uranus model (ESA4) derived by Glenn Orton. This is much more rigorous and traceable than the previous model used. It only makes a small difference to the overall calibration: SSW +0.5%, SLW up to -5% at the low frequency end
to separate the Uranus observation used to produce the LR and HR versions of the file - this improves the LR calibration
update the beam FWHM to more recent numbers - affects SLW more than SSW
point source conversion factors included for most off-axis detectors (although not used in the standard pipeline)
forwardRsrf and reverseRsrf columns removed as these were not used
SCalSpecTeleRsrf and SCalSpecInstRsrf updated:
to be consistent with other pipeline and calibration changes
to use a separate processing to fill LR and HR products separately - this leads to better LR calibration
Added errors to the following calibration products:
SCalSpecBeamParam
SCalSpecSmecStepFactor
SCalSpecBolPhase
SCalSpecNomPcal product updated to be consistent with the new PCAL processing algorithm
Units were corrected in the SCalSpecBrightGain product
The interferogram limits were adjusted in the SCalSpecOpdLimits product to be consistent with the new phase correction algorithm and to allow correct processing of CR observations as HR in the pipeline
The SCalBeamProf product is not used in the pipeline and contained model data from before launch. In order to avoid confusion it was filled with NaNs until we have enough information to fill it properly.
Two new products were added:
SCalSpecTeleModel - this contains an empirical correction to the emissivity used in the telescope model - it leads to lower residuals in very faint observations
SCalSpecPcalModel - this contains a model PCAL flash profile, used in the new PCAL processing algorithm
Photometer Pipeline
Electrical crosstalk correction has been reinstated in the photometer pipeline.
Implemented crosstalk correction for bolometer-thermistor crosstalk only.
Bright Source Calibration will follow in HCSS 10
DestriperTask
Now the default task for baseline removal in both SPG and User pipelines.
Added the "useSink" task parameter. The default value is False. It if it is set to True, the ProductSink on disk is used for the Level 1 output. This allows larger observations to be processed with a limited amount of RAM, but makes the task slower.
Added the "nThreads" task parameter. The default value is 1. The parameter specifies the number of threads to use, increasing speed significantly on multi-core architectures.
Implemented Level2 deglitching. The new "l2DeglichAlgorithm" task parameter specifies the algorithm to be used and can be set to "sigmaKappa" (default) or "twoSigmas".
Added the "l2IterMax" task parameter. It limits the number of iterations of the Level2 deglitcher within a destriper iteration. The default value is 5.
Added the "startParameters" task parameter. This input parameter accepts a diagnostic product that was provided at the end of a prior destriper run. It contains a table dataset with the last fit parameters and a flag column that determines which scans were excluded. Manual changes to these flags will be honored by a second destriper run in an interactive analysis session. If provided, the contents of the table are used as start parameters for a new destriper run, enabling faster convergence, and improvements to the map by deselecting "bad" scans. The default value is null.
Added the "storeTod" task parameter. The default value is False. In this case no TOD files will be stored and the returned TOD parameter is null. Setting the parameter to True, returns a TOD product and leaves the associated files (pixel, signal and weight) stored in the temporary directory until the end of the HIPE session.
Cooler Burp
A new task called RecalcTempDriftTask to deal with the cooler burp problem was introduced in the build. During the cooler burp the drifts of bolometers is unusual and for this reason the temperature drift correction leaves residuals. The proposed solution is a empirical solution that assume that the problem is due a change in temperature drift coefficients A used in TempDriftCorrectionTask. These parameters are normally read from calibration table and are necessary to link bolometers' signal and voltage thermistors. The task recalculates the A parameters via linear fitting of uncorrected signal vs thermistor voltages and the new parameters are then used in TempDriftCorrectionTask.
The user photometer pipeline scripts for parallel and large map observations was modified introducing the possibility to check for cooler burp and to use this new task to correct it. To do this, a new boolean parameter was introduced at the start of the user scripts: coolerBurpDetection. The default value is false, the user that wants check for the cooler burp he must set this parameter value to true.
Spectrometer Pipeline
Improved calibration accuracy: updated telescope model
The telescope model has been updated to include an OD dependent multiplicative correction factor to the emissivity of the primary mirror. This leads to a more accurate model, and better calibration, particularly noticable for faint sources. The residual additive uncertainty in the pipeline products was previously 1-3 Jy, and this has now been reduced to 0.5 Jy.
Low resolution processing
Low resolution observations are now processed using calibration products based only on Low resolution input data. This leads to a much improved spectral shape for Low resolution observations.
Faster and slimmer pipeline processing
Multi-threading has been implemented in the most time consuming modules - speeds up processing time.
Un-used columns have been removed from interferogram product - this can reduce the size of the interferogram products by 44%. This leads to a small reduction in the total size of the final Observation Context because some other products increased in size slightly.
Average Spectra
The Average Spectra task has been updated to correctly propagate existing errors - now saves the number of scans averaged in "numScans" column
PCAL processing
The PCAL flash processing has been improved to take better account of the PCAL flash baseline. This is important for Bright mode observations of very bright point sources. In these cases there can be a large baseline caused by pointing jitter. This is correctly fitted and removed by the new PCAL algorithm and leads to improvements in spectral shape for very bright targets.
Interactive Analysis and Tools
Photometer:
End of scan line turnaround data can now be used in processing and mapmaking.
The naive mapper can now create an alternative error map using bolometer weights.
SpireMaskEditor
Level0_5 SpireMaskEditor was added to the right click menu on an observation context. It is used for displaying and editing the mask data of a Level0_5 Context.
Level1 SpireMaskEditor was added to the right click menu on an observation context. It is used for displaying and editing the mask data of a Level1Context.
These two additions effectively replace the SpireMaskEditorComponent which was deprecated.
Spectrometer:
Useful scripts added for estimating spectral noise and convolving a high resolution spectrum to the SPIRE resolution
Useful script for background subtraction updated to include smoothing of off-axis detectors, and interactive selection
Useful script for line fitting improved
Implemented drag and drop from SDS Explorer detector layout to variables tab
Interested in what's new for other versions of HIPE? See this page for links to all the What's New documents from HIPE 4.0 onwards. 
png 
Public Web
Create New Topic
Index
Search
Changes
Notifications
Statistics
Preferences
Public
