TWiki> Public Web>HipeWhatsNew>HipeWhatsNew13x (revision 22)EditAttach

What's coming up in HIPE 13

PDF Version Click here
Help Interested in what's new for other versions of HIPE? See this page for links to all the What's New documents from HIPE 4.0 onwards.
See also the HIPE known issues and the Data products known issues pages for last-minute information on known problems.

Planned work


   * TBD


   * TBD


 * TBD


   * TBD



   * TBD


  • A new level 2.5 products has been generated for point mode observations. It is a stitched version of the level 2 spectra for each available spectrometer, in the form of a Spectrum1d.
  • All observations context will now contain a (Ta*) calibrated OFF position spectrum, in the same sky frequency scale as the ON position data.   * The new calibration tree contains updated beam models and efficiencies, as released initially as stand-alone products in September 2014. See for more details
  • In bands 6 and 7, the Electrical Standing Wave correction is now part of the standard pipeline, based on a data-base of optimised solutions stored in the calibration tree for each observation
  • For spectral scans, a significant fraction of the deconvolved level 2.5 products will benefit from flag tables optimised manually by the ICC, resulting in spur-free deconvolved spectra. This is not yet available for all spectral scan obsids though, but will be by HIPE 14. The list of optimised flags can be found in the calibration tree and further refined by the user using the flagTool if the User wishes so.


   * TBD


   * TBD

Core system


Batch mode

In batch mode, you can use sys.argv to get the input arguments. For example:

import sys
print str(sys.argv)

Then the output of this script would be:

> hipe arg1 arg2 arg3
['', 'arg1', 'arg2', 'arg3']


In order to give better user feedback on GUI operations, screen transitions have been introduced in the following cases:

  • Switching perspective is performed by fading to the new perspective.
  • Navigation between pages in the welcome perspective is also done by fading to the new page.
  • Maximizing or restoring the data viewer within the observation context viewer is shown with an animation.
  • Maximizing or restoring the browse image of the observation context viewer is animated too.

Editors & Viewers

Status bar

From the Preferences > Data Access > MyHSA > Advanced panel, the user can:

  • Show the MyHSA on-line/off-line checkbox in the status bar.
  • Hide the MyHSA on-line/off-line checkbox in the status bar.

This label provides a more prominent location reflecting the current connection status of MyHSA, irrespective of what view the user is currently using. It also offers the ability to update this status, the status is synced with that shown in the product browser and vice versa.

File handling

When saving a file of a known type with the file dialogue, the extension is automatically added to the file name if not explicitly done.

Rescue mechanism

If HIPE gets in a non responding state for a period of time, it is automatically detected so that a popup window asks to restart it in an ordered way. This means all unsaved editors will be saved, and a new fresh HIPE session with the state of the frozen one will be opened.

The time intervals to run this check and to consider the application as not responding can be configured in Preferences > General > Rescue, where this mechanism can also be disabled altogether if you prefer so.




Astronomical utilities


Numeric routines

 Data fitting

 Data interpolation

Random numbers


 Trend analysis


Other functions

Source extraction



The following changes are made to the Display class:

  • Replaced setCutLevels() by calcCutLevels()
  • Removed setCutLevels(double,double)
  • Renamed setCutLevels(double) to setCutLevelsPercentage(double)

  • The images can not only be flipped around the y-axis, but also around the x-, xy- and yx-axes. It is also possible to combine these flip-options : for example : y- and xy.
  • The annotation in Display are totally reworked. All old annotation methods are deprectated, the new one end with Annotation. For example : addEllipse becomes addEllipseAnnotation.

A lot of fixes in Display :

  • Display now displays images with some extremely high or low values correctly.
  • It is now possible to extract the current layer from a SimpleImage or from a SimpleCube with only one layer.
  • Display can now work with more than one extension.
  • The zoom in / out was by a factor of 4 instead of 2.
  • The Wcs viewer only shows the important wcs keywords now.


Bugfixes :

  • Mosaics close to the north or south pole missed a part of the sky.
  • Aperture photometry tasks have clear description on how the background subtracted flux is calculated.
  • Bugfixes in aperture photometry tasks.
  • You can choose to add an input parameter to annularSkyAperturePhotometry for choosing the step size of curve-of-growth output



* The mosaic view of the Spectrum Explorer shows axes coordinates in a more convenient way, with less digits.


Data cubes



Herschel Science Archive

Calibration sources

Products and datasets

 Observation context

Product Access Layer 


  • Warning, important __list__ argument removed: if your task call still has it, remove it.
  • convertUnits : It can convert units from double parameters, array datasets and columns, plus units from table datasets or products, by allowing to select the target dataset or column.

The GetObs Task now includes the possibility to retrieve observation using the spg version that created the Observation. See up to date Task definition with examples: (HCSS-19063)

The GetObs Task path and poolLocation parameters have been consolidated. See up to date Task definition with examples: (HCSS-19241)

The FitsReaderTask and ImportSpectralCubeTask now support reading and loading of ALMA/CASA cubes directly into HIPE and the Spectrum Explorer - HCSS-18110

Quality control

Systematic product generation

Data input-output

Virtual Observatory

The link to VOPlot in the External Tools sub-menu will be removed and substituted by Topcat.




HIFI Software Configuration and Properties

HIFI Calibration

The calibration tree released together with HIPE 13 contains the following updates:

  • it will contain a list of optimised channel flags for spectral scans obsids, that will be honoured by the deconvolution task and result in spur-free level 2.5 data in this mode. This is not yet available for all spectral scan obsids though, but will be by HIPE 14.
  • it will contain solution tables for Electrical Standing Wave correction in the bands 6 and 7. This allows for a correction of this affect in the pipeline itself. Note that this is not a guarantee that the baseline artefact will be perfectly removed but it will provide the best correction available to data. It is then possible to further optimise the correction - see the dedicated sections in the DRG.
  • it will contain the updated beam efficiencies and 2D beam models

HIFI Pipeline

hifiPipeline task

Level 0 Pipeline

WBS Pipeline

HRS Pipeline

Level 1 Pipeline

Level 2 Pipeline

  • the sideband ratio correction is now taking into account the possible IF-dependency of the sideband ratio over the IF. It uses the IF-dependence information as given in the calibration tree. As of now, only the lower end of band 2a makes such an assumption. The presence of an IF-dependency will result in additional meta-data usbGain_n or lsbGain_n, which are the coefficients used to described the polynomial approximation of the gain dependence. Coefficients with index _0 simply correspond to the value of the constant, scalar, sideband ratio we used so far
  • a new task will automatically check for platforming in the WBS data and report any detection with a dedicated quality flag
  • a new task will create spectra for the OFF position data calibrated into the Ta* scale, on both USB and LSB frequency scales. For spectral scans, these can then be fed into deconvolution to create a deconvolved spectrum of the OFF position (option use_reference)

Level 2.5 Pipeline

  • Mapping mode: The cubes produced automatically by the pipeline (Level 2.5) (doGridding task), and consequently by hifiPipeline if you choose to run it again, will stitch the WBS subbands and create one final cube (with the index '_1'). You will not see individual cubes per subbands except for the HRS.
  • Point mode: Level 2 HTPs , stitched, are forming the Level 2.5 products for this mode.
  • A large number of statistics (primarily noise and mean) is performed both on the level 2 and level 2.5 products, as well as on an equivalent products converted to the Tmb scale with H and V merging (this latter is then removed from the observation context and only the statistics is stored). This computation is controlled by the option computeRms. It is only set to True when run for the bulk reprocessing, but ignored by default when the level 2.5 pipeline is called interactively (this computation can be extremely time consumming for maps and spectral scans)

HIFI Products

HIFI Spectrum Toolbox

Spectrum Explorer (HIFI specifics)

Spectrum tasks (HIFI specifics)

  • The PolarPair class has been removed. Instead the polarPair task should be used. This should only affect scripts written prior to 2013.

HIFI Data Processing Tools

Standing Wave Removal

  • Updates to default band-dependent period characteristics in task fitHifiFringe :
    • Band 1 change nfringe 2->3, endPeriod 120->200
    • Band 3 change typicalPeriod 620->95
    • Band 4 change typicalPeriod 620->95
    • Band 6 change startPeriod 100->80, typicalPeriod 320->600
    • Band 7 change startPeriod 100->80, typicalPeriod 320->625
    • Band 2,5 no change
  • swTable had wrong unit for amplitude (kHz instead of K). This is now fixed.
  • New output variable swTable was created to enable machine-readable output of fitted frequencies and amplitudes.
  • Output for fitHifiFringe now contains corrected data and summary table.
  • New input parameters "cal" and "saveInput" was created in fitHifiFringe to extract user input parameters into the calTree for later use on demand.
  • When a large portion of channels is flagged as invalid, i.e. when not enough channels can be fitted, fitHifiFringe will raise a comment that it is skipping the spectrum and then it will continue to the next one.

Baseline Removal

  • In order to prevent fitBaseline from crashing when 'useWbs' is set to 'True', especially when doglue is also set to 'True', the task will automatically override doglue to 'False'.

Electrical Standing Wave correction in HEB bands

  • the task doHebCorrection is now part of the default pipeline, with several new options compared to its predecessor hebCorrection.


  • by default, the option chosen for spur_rejection will be based on the availability of not of optimised flag tables for a given obsid (see also the calibration section). When no table exists, the option is enfored to REJECT_SCANS_WITH_SPURS. When tables exist, the option is enfored to DO_NOT_REJECT_SCANS_WITH_SPURS
  • A new option called enforce_spur_rejection is however made available to help control the existing spur_rejection option.
  • A new option use_reference indicates that the deconvolution should be made on the OFF position spectra computed with HIPE 13
  • doDeconvolution has also been updated so it can take into account the IF-dependence of sideband ratio and use this a priori knowledge as preset to build the single sideband solution


  • with the availability of the new beam efficiencies and beam model, the task has been upgraded in order to make best use of the detailed beam knowledge. A new option hifiBeam, set to True by default, will use the fully-detailled HIFI beam to compute the coupling factor involved in the temperature to flux conversion for sources of non-zero size.
  • for non-point-like sources, the task now offers disk or Gaussian source models (option shape)

Spectral Cube creation


  • Interactive (suitable for batch processing) and Expert modes now available.
  • maskTables: a new variable to manage Linemasks and Rowmasks files.


  • IdentifyLines is a new task that allows you to identify lines in your spectrum and then run a comparison of known lines with a Linelist.

HiClass export tool


  • Updates to the HIFI Data Reduction Guide:
    • A new chapter on The HIFI line identification tool (identifyLines task) is now available.
    • Updates to the chapter How to add and remove flags, section Setting flags interactively for many spectra (flagTool task).
    • Updates to the chapter Sideband deconvolution, section Basic strategy for running the deconvolution tool (doDeconvolution task).
    • Updates to the chapter Standing Wave Removal (fitHifiFringe task).
    • Updates to the chapter How to make a spectral cube to reflect changes in the level 2.5.






Analysis tools



Product changes





Calibration Products

Note that the size of the calibration tree has increased by 120 MB, mainly due to the addition of the finely sampled (1") photometer beam maps.


  • The beam profile maps are improved using the shadow observations of Neptune (i.e. better characterising and removing the background).
  • The corresponding beam areas given in the metadata of the beam products are updated accordingly.
  • The radial beam calibration product has been updated to match the new beams.
  • All the colour correction products have been updated following the beam updates.
  • In order to improve the operation of the source extractors, the beam profiles are adjusted to be relevant for a 1 Jy source - previously the peak of the profile was fixed to 1 Jy, but the new profiles integrate to 1 Jy. In addition, the map consists of an odd number of pixels so that the beam centre is precisely at the centre of the central pixel.
  • A new "fine" beam map is provided for each band with 1" pixels. The "nominal" maps are also available with pixel sizes of 6", 10" and 14" for PSW, PMW and PLW.
  • The K4 parameters in the Flux Conversion product metadata are updated (very small change to K4E)
  • The Planck HFI colour correction table has been updated following the beam updates (very small change).
  • The parameters in the temperature drift calibration product that are used to perform cooler burp correction have been updated to improve the correction.


  • The non-linearity product has been updated to better fit observations made with low subktemp at the beginning of each pair of FTS observing days (i.e. the observations following the cooler recycle). This results in a much improved SLWC3 spectral shape.
  • The other RSRF, telescope model and flux conversion products were updated following the non-linearity update.
  • The PhaseCorrLim product was updated to correct the metadata - no effect on the data.
  • The TempDriftCorr product has been removed as it is not used by the pipeline.

Calibration Framework


Engineering Conversion

Common Pipeline

Photometer Pipeline

Spectrometer Pipeline

  • Update to the naming within the Observation Context to make it clearer which products are spectra/cubes and pointsource/extended calibrated.
  • Moved all of the final spectral products into the Level-2 context. Previously the extended calibrated spectra were in the Level-1 context, but now both point source and extended calibrated data are in the Level-2 context.
  • Introduced a new spectrum2d product for mapping observations to contain all of the spectra before they are gridded into a regular spectral cube. Previously these spectra were saved in individual products for each jiggle/raster position. The new product is added to the Level-2 context.
  • A new task is added to perform spectral apodization. This step was previously carried out with several tasks to take the inverse Fourier transform and apply the apodization on the interferogram. The new task works directly on the spectra.
  • Quality control parameters related to the unaveraged FTS scans are now written back into the interferogram products in the Level-1 context. Previously these metadata items were lost after averaging the spectra over all of the scans.
  • New metadata parameters to contain known pointing offsets from the commanded position have been added to the Level-2 spectra.
  • A limit has been placed on the spectral resolution allowed as input when a spectral mapping observation is gridded into a cube. This eliminates spectra that were clipped, and improves the quality of the maps.

Interactive Analysis and Tools






  • Minor updates, bug fixes, and improved documentation for the Semi-extended correction task
  • Updates to the task to take synthetic photometry of FTS spectra for comparison with the SPIRE photometer
  • New GUI interface for the script that determines the noise level in a spectrum
  • Small corrections to the useful scripts



Release Notes

   * Full list of tickets implemented: here.

blog comments powered by Disqus

-- AlvarGarcia - 15 Jan 2014

Edit | Attach | Watch | Print version | History: r68 | r24 < r23 < r22 < r21 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r22 - 2014-12-09 - JaimeSaiz
This site is powered by the TWiki collaboration platform Powered by Perl