TWiki> Public Web>HipeWhatsNew>HipeWhatsNew9x (revision 37)EditAttach

What's new in HIPE 9

PDF Version Click here

See also the HIPE known issues and the Data products known issues pages for last-minute information on known problems.



  • Common look and feel for the spectrum explorer and cube analysis toolbox
  • Easing the data access via MyHSA
  • Improvement of handling of ASCII tables
  • Mechanism to notify users on removed names (tasks, functions...), why they were removed, and their replacement
  • Smaller memory footprint for HIPE editor
  • Speed-up of HIPE start-up
  • Progress bar informs on progress of for-loops
  • Human-made indexes added to two core manuals (HIPE Owner's Guide and Scripting and Data Mining Guide)
  • You can leave comments at the bottom of any page in the HIPE Help System
  • Support to OSX Lion
  • Easy access to calibration data, including version and updates (TBS)
  • Improvements of search capability of documentation system
  • Publication ready plots (TBS)


  • HIFI level 2.5 product introduced: for Spectral Scans it holds the deconvolved single sideband soluions; for Maps it holds the cubes; for all cases it holds a copy of the level 2 HTP.
  • Addition of an interactive pipeline between level 2 and 2.5, which allows the insertion of tasks like fitHifiFringe and fitBaseline into the pipeline between level 2 and 2.5.
  • Browse products for spectral scans and maps added


  • Photometer: Addressing further specific cases in IPipe scripts
  • Spectrometer
    • Broadband reliability
    • Improved spectral pixel flatfield broad ranges (SED)
    • Correction Flux loss due to pointing
  • Convolution kernels for photometer and spectrometer maps aiding the interpretation of extended and point sources
  • Reduction in memory consumption (TBS)


  • common (both Photometer and Spectrometer)
    • improved processing speed using multi-threading
    • improved task for correction of "cooler burps"
  • Photometer
    • Pipeline Processing
      • improved error maps (possibly as an option)
      • improved baseline correction using "destriper“
    • Interactive processing
      • source timeline fitter task
      • improvements to bolometer finder tool
      • selection of turnaround data in map creation
  • Spectrometer
    • Pipeline Processing: smaller size for level 1 products
    • Improved calibration and data processing
      • updated point source calibration (improved Uranus model)
      • updated bright mode calibration (better PCAL module)
      • improved calibration for Low Resolution observations (hopefully improved calibration, but at minimum, updated advice)
      • more straightforward calling sequence for phase-correction task
      • corrected spectral averaging - more flexible usage
    • Interactive processing
      • drag and drop from SDS Explorer detector layout to variables tab
      • new and updated useful scripts for background subtraction and spectral noise

Core system


Status bar

Execution of for loops is reflected in the progress bar by being increased along with the percentage of executed iterations.

Startup & shutdown

  • It is possible to set a number of py files to be executed at startup, in the new Imports preferences panel.
  • The main window's state (size, location, maximized state) is remembered among HIPE sessions, so it is not necessary to set it after every startup.
  • Check of local pool locks (probably left from a previous session that was abnormally terminated) is executed at startup. This check is configurable in preferences.
  • The running Java version is compared with the supported one; if they are different, a warning is popped up. This check can be enabled or disabled at will, in preferences.

Editors & Viewers

  • The tabs within the Editor area show the close button even if not selected, so they can be closed with 1 click instead of 2.
  • The viewers shown in the Open With menu are the most specific ones for the selected variable.
  • The viewer for lists shows data values instead of just their type.
  • Autosave of text editors and Jython script editors is turned on by default.
  • The Jython Editor preferences now appear under the Text Editor preferences in the preferences window.
  • Boolean values in tables are shown as true / false instead of the previous less clear 1 / 0.
  • Tooltips in table headers include more information on the columns.
  • The load of a node in the product tree, when selected, can be cancelled with the Stop button.

Console & Log

  • Obsolete names (renamed or removed tasks, functions, etc.) can be registered by developers so meaningful messages, explaining the reason of the name removal, can be shown instead of a plain NameError.
  • Incremental search (with Ctrl+I) is now available in the Log view.


  • The refresh period of the Navigator view can be configured in preferences.
  • User areas are included in the Open File dialogue.


  • Video Tutorials entry has been added to the Help menu.
  • Context sensitive help on the focused view has been added. It can be accessed through Help > Help on <name> view or with F2.
  • If debug is enabled (see preferences, General > Debug), a threads dump can be obtained with the menu option Tools > Debug > Dump thread stacks. It can be helpful when informing the Helpdesk on software problems.
  • A new tab with a glossary of terms has been added to the Help System.
  • You can leave comments at the bottom of any help page. You must be connected to the Internet to read and leave comments.
  • You can get the full address of any help page, for sharing or adding to your browser's bookmarks. Click on the icon shaped like a chain in the Help System toolbar.
  • Many improvements to the search interface of the Help System:
    • You can now filter searches on any manual or group of manuals.
    • After each search, a feedback form asks you whether you found what you were looking for. Your feedback will be used to further improve the search engine.
    • Glossary terms are always at the top of search results.



  • TablePlotter now supports the axis.Date functionality.


  • OverPlotter now supports the axis.Date functionality.

Dataset viewer

Astronomical utilities

Numeric routines

Data fitting

Data interpolation

Random numbers


Trend analysis


Other functions

Source extraction

  • SUSSEXtractor algorithm implemented in a fully consistent manner, with detailed mathematical documentation provided
  • Defaults for sourceExtractorSussextractor now: useSignalToNoise=False, fitBackground=True, fluxPriorsLambda=0.0
  • "beamArea" parameter for sourceExtractorSussextractor deprecated
  • "inputSourceList" parameter now accepts [ra, dec] and other lists of values as input, as well as a SourceListProduct.
  • Many internal changes to the tasks, improving speed and consistency in applying the algorithms
  • Multi-threading: increasing speed of tasks significantly on multi-core machines



  • Reworked the save methods on Display. You can now save to jpg, png, eps or pdf files. The methods to use are :
    • save(filename), saveAsEPS(filename), saveAsPNG(filename), saveAsJPG(filename) and saveAsPDF(filename) : These methods will save the full image, including annotations.
    • saveCurrentView(filename), saveCurrentViewAsEPS(filename), saveCurrentViewAsPNG(filename), saveCurrentViewAsJPG(filename) and saveCurrentViewAsPDF(filename) : These methods will save the current view of the image, including annotations and axes.
  • The coordinate axes on the image panner are always visible now
  • There is a preference for image display to turn of the echoing of commands to the command line. It is possible to turn all these commands off, or only the zoom commands.
  • Flagged out pixels and pixels with a value of NaN will be shown in the color of the background.
  • Image can be matched with the Wcs. This means that the images can be oriented so that north is shown at the top and east to the left.
  • There is a preference for image display to always show the image with the 'match wcs' orientation.
  • A lot of bugfixes.




  • Major updates to Spectrum Explorer to improve performance and provide more direct control over plot appearance


  • The Smooth task is corrected for a bug in the ZEROS option.
    • As a consequence of this fix, which is more mathematically correct, it is found that smoothed spectra for SPIRE are quite changed. It is recommended instead to use the REPEAT option.
  • Introduction of a new "Accumulate" task allowing to automatically combine data taken on different frequency grids, with or without frequency overlap.

Data cubes



Herschel Science Archive

  • Improvements in download speed and memory requirements for data downloaded from HSA using a behind-the-scenes utility called MyHSA
    • Testing with 5 HIFI, 9 PACS and 10 SPIRE representative observations of all major observing modes simultaneously with getObservation/saveObservation and with getObs/MyHSA with the same network connection on two different machines show 20 times faster download speed and that 66% less hard disk space is needed to hold the same data:
      MyHSA_downloadTimes.png MyHSA_diskUsage.png

Calibration sources

Products and datasets

Observation context

Product Access Layer

When storing pools on shared NFS-mounted disks, HIPE now uses a more robust mechanism to prevent corruption. The default locking mechanism for Local Pools was changed to a mechanism that works more reliably on remote disks.

Usability improvements for the Product Browser and for working with tags. One can now update existing tags (for example to fix a typo).


  • Task Preferences
    • Preference to generate full task call from the task GUIs
    • Preference to check input and inout parameter modification/replacement by tasks
  • Deprecated and removed tasks (names and parameters) documented in HIPE.
  • Task Viewer:
    • Drag and Drop of the (full) signature of tasks from Task Viewer to Editor & Console
    • Reordering of nodes and reduced depth
    • Tooltips include the Categories and deprecated tasks appear crossed out
  • Task Outline:
    • Deprecated tasks & parameters marked

Quality control

Systematic product generation

Data input-output

  • Deprecation of duplicated functionalities: poolDataWriter, localStoreWriter, poolDataReader. Use getObservation and saveProduct.
  • Deprecation of MyHSA prototype duplicated functionalities: loadObs and downObs. Use getObservation
  • getObservation promoted to a Task (currently getObs).
  • simpleAsciiTableReader deprecated, use asciiTableReader.

Virtual Observatory



  • Quick Start Guide
    • Updated to reflect the latest changes in data access (MyHSA and eHSA).

  • HIPE Owner's Guide
    • Updated throughout to reflect the latest changes in HIPE.
    • In particular, the latest improvements in the help search engine are now described.
    • Added man-made index.

  • Scripting and Data Mining


HIFI Software Configuration and Properties

HIFI Calibration

  • New calibration database HIFI_CAL_9_0, containing the following updates
    • Side-band ratio correction over the whole of bands 5a and 5b
    • Update of a number of engineering threshold for more accurate quality flagging related to hardware housekeeping
    • introduction of a list of a priori known corrupted data-frames for dedicated flagging in level0 data

HIFI Pipeline

hifiPipeline task

  • Addition of an interactive pipeline between level 2 and 2.5, which allows the insertion of tasks like fitHifiFringe and fitBaseline into the pipeline between level 2 and 2.5.
  • Inclusion in the pipeline GUI of a drop-down box to select calibration version
  • Spectrometer checkboxes in the pipeline GUI are now only checked if data for that spectrometer are available at level 0.
  • The DoBadLo step of the level 0 pipeline is now run when reprocessing WBS data from level 0 with new calibration. This is to allow you to take advantage of flags indicating bad LO settings that are raised by this pipeline step without having to wait for a new reduction from raw data to be available in the HSA.

Level 0 Pipeline

  • flagging of rare corrupted data-frames according to a list of pre-defined data-frames indices. The difference will only be noticed after the bulk reprocessing has taken place
  • revision of quality checks on diplexer out of limits, the effect is to reduce the number of benign quality flags.

WBS Pipeline

  • Introduction of quality flag to warn about a bad zero.
  • Introduction of a flag for a wrong scan count
  • Improvements to flagging of saturated channels
  • Improvements to algorithm to fit to COMB lines

HRS Pipeline

Level 1 Pipeline

  • Addition of parameters in the doFilterLoads to allow configuration of the filtering.
  • Addition of a new quality products "flagsSummary" compiling the list of all rowflags set to the data at this level

Level 2 Pipeline

  • doAvg now honours user-defined flags, RowMask.IGNORE_DATA and HifiMask.IGNORE_DATA

Level 2.5 Pipeline

  • Introduction of a level 2.5 pipeline
    • cube creation is moved from the level 2 pipeline to the level 2.5 pipeline for mapping observations (see Cube Creation section for updates to gridding done in the pipeline)
    • deconvolution is done for spectral scan observations (see Data Processing Tools -> Deconvolution section for updates to deconvolution done in the pipeline)
    • the browse product for all observing modes is done (see HIFI Products section for updates to browse products)

HIFI Products

  • Level 2.5 product introduced: for Spectral Scans it holds the deconvolved single sideband soluions; for Maps it holds the cubes; for all cases it holds a copy of the level 2 HTP.
  • Addition of browse products for spectral scans and maps. The spectral scan browse product is the deconvolved single sideband solution for H and V. The mapping browse product shows the map averaged spectra and the integrated map over the range of the WBS for the H- and V-polarisations.
  • Point mode browse product axis labels corrected.
  • Addition of a flag array to cubes, allowing pixels to be flagged in the same way as points in spectra can be.
  • HTP default view is changed from SpectrumExplorer to the ContextViewer
  • Observing band is added to HTP metadata

HIFI Spectrum Toolbox

Spectrum Explorer (HIFI specifics)

  • Introduction of a HIFI task panel in the Spectrum Explorer. Still in development

Spectrum tasks (HIFI specifics)

  • ConvertK2Jy now uses the same scheme as doGridding to select frequency when interpolating the efficiency values from the calibration tree: the task searches first for the LO Frequency in the metadata, if it is not found the task looks in the metadata for the value inserted in HSpot. If neither are found then the middle frequency of the observing band (e.g. band 1a) is used.
  • Bug fix in ConvertK2Jy: non-zero source size data on frequency scale can now be converted to Jy
  • ConvertK2Jy will accept data with any wavescale

HIFI Data Processing Tools

Standing Wave Removal

  • Improved handling of pipeline flags in fitHifiFringe
  • fitHifiFringe now accept SimpleSpectrum data

Baseline Removal


  • When multiple observations are passed to doDeconvolution the metadata are now concatenated and put into the decon_result
  • doDeconvolution made more robust against failure when there are many bad datasets in the HTP.
  • A median subtraction is possible prior to running doDeconvolution.

Spectral Cube creation

  • Default output of doGridding changed from a cubes array to a MapContext, this will affect scripts in which a cube is extracted from the output of doGridding. See the HIFI Data Reduction Guide "How to make a Spectral Cube" chapter.
  • The comoving option is now automatically turned on for mapping observations of Solar System Objects
  • Fix to a bug in the way the pixel size was calculated for strip maps
  • Data flagged as bad_DATA or IGNORE_DATA are ignored during the cube creation.


  • The flagTool task introduced in HIPE 8 is now fully functional and can process both spectrometers, both polarizations, and both subbands in one go.
  • Capabilities are:
    • choice of 4 flag types (LINE, BRIGHT_LINE, SPUR_CANDIDATE, IGNORE_DATA)
    • automatic flag propagation
    • natural zooming enabled in the plot window
    • easy flag switching
    • automatic merging and splitting of the flags
    • copying of flag from one polarization to the other for a same sideband
    • subband gluing (keyword doglue)
    • embedded use of fitHifiFringe, and embedded use of fitBaseline

HiClass export tool

  • Sideband gain information in the metadata are now exported to CLASS.


  • Updates to DBS Cookbook
  • Updates to HIFI Data Reduction Guide for changes to:
    • fitBaseline
    • fitHifiFringe




  • updated ipipe scripts with:
    • optimized second order deglitching with the MapDeglitch task,
    • optimized mask generation for the highpass filtering
    • now using the photMaskCrosstalk and filterSlew tasks
    • improved MADmap scripts with a new pre-processing task photBaselineDriftCorrection and a post-processing task to remove bright point-source artifacts photCorrMadmapArtifacts
  • the filterOnScanSpeed task now only masks the turnover frames instead of deleting them.


  • estimate of continuum RMS is included in plotting function of ipipe scripts
  • new task that extracts the central spaxel spectrum with a 3x3/central correction
  • the task specFlagGlitchFramesQTest can optionally be threaded
  • background normalization ipipe script introduced also for long range SED
  • improved flatfielding task, for line-scan mode
  • improved drizzling routines
    • available for chopped and unchopped mode
    • uses observed beam size for the calculation of the pixel scaling of the spatial grid
  • test versions of pointing correction tasks
  • improved error propagation
  • correction of SSO pointing task correctRaDec4Sso introduced in spg pipeline



  • new calibration products for :
    • the flat-field (flatField, v4)
    • responsivity change of ~2% only at 100 and 160 microns (responsivity v7)
    • aperture correction (apertureCorrection v3)
    • *non-linearity correction ( nonLinearCoef v2)


  • new calibration product offRatio for all bands and chopper throws
  • new calibration product beams (raster beams of neptune) for all bands
  • update of the pointSourceLoss calibration product
  • new calibration product beamSize
  • new viewer for the calibration set release notes

Analysis tools



  • refactoring of the spectrum explorer
  • improved GUI for depiction of the spectrometer footprint

Product changes



  • complete browse product generation
  • quality flag "SLOWSAMP" added in case of low sampled spectra (level 2 products)


  • improvements of URM
  • improvements of the PDRG


Calibration Products

Calibration Framework


Engineering Conversion

Common Pipeline

Photometer Pipeline

  • DestriperTask
    • Added the "useSink" task parameter. The default value is False. It if it is set to True, the ProductSink on disk is used for the Level 1 output. This allows larger observations to be processed with a limited amount of RAM, but makes the task slower.
    • Added the "nThreads" task parameter. The default value is 1. The parameter specifies the number of threads to use, increasing speed significantly on multi-core architectures.
    • Implemented Level2 deglitching. The new "l2DeglichAlgorithm" task parameter specifies the algorithm to be used and can be set to "sigmaKappa" (default) or "twoSigmas".
    • Added the "l2IterMax" task parameter. It limits the number of iterations of the Level2 deglitcher within a destriper iteration. The default value is 5.
    • Added the "startParameters" task parameter. This input parameter accepts a diagnostic product that was provided at the end of a prior destriper run. It contains a table dataset with the last fit parameters and a flag column that determines which scans were excluded. Manual changes to these flags will be honored by a second destriper run in an interactive analysis session. If provided, the contents of the table are used as start parameters for a new destriper run, enabling faster convergence, and improvements to the map by deselecting "bad" scans. The default value is null.
    • Added the "storeTod" task parameter. The default value is False. In this case no TOD files will be stored and the returned TOD parameter is null. Setting the parameter to True, returns a TOD product and leaves the associated files (pixel, signal and weight) stored in the temporary directory until the end of the HIPE session.

  • Cooler Burp
    • A new task called RecalcTempDriftTask to deal with the cooler burp problem was introduced in the build. During the cooler burp the drifts of bolometers is unusual and for this reason the temperature drift correction leaves residuals. The proposed solution is a empirical solution that assume that the problem is due a change in temperature drift coefficients A used in TempDriftCorrectionTask. These parameters are normally read from calibration table and are necessary to link bolometers' signal and voltage thermistors. The task recalculates the A parameters via linear fitting of uncorrected signal vs thermistor voltages and the new parameters are then used in TempDriftCorrectionTask.
    • The user photometer pipeline scripts for parallel and large map observations was modified introducing the possibility to check for cooler burp and to use this new task to correct it. To do this, a new boolean parameter was introduced at the start of the user scripts: coolerBurpDetection. The default value is false, the user that wants check for the cooler burp he must set this parameter value to true.

Spectrometer Pipeline

Interactive Analysis and Tools

  • SpireMaskEditor
    • Level0_5 SpireMaskEditor was added to the right click menu on an observation context. It is used for displaying and editing the mask data of a Level0_5 Context.
    • Level1 SpireMaskEditor was added to the right click menu on an observation context. It is used for displaying and editing the mask data of a Level1Context.
    • These two additions effectively replace the SpireMaskEditorComponent which was deprecated.


Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng MyHSA_diskUsage.png r1 manage 83.8 K 2012-04-04 - 21:49 CarolynMcCoey HSA disk usage
PNGpng MyHSA_downloadTimes.png r1 manage 50.2 K 2012-04-04 - 21:50 CarolynMcCoey HSA download times
Edit | Attach | Watch | Print version | History: r75 | r39 < r38 < r37 < r36 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r37 - 2012-05-03 - CarolynMcCoey
This site is powered by the TWiki collaboration platform Powered by Perl