This document lists the changes in HIPE version 4.2 with respect to the 3.x series. Additional pages list changes between minor versions of the 4.x series.
The Navigator view allows adding customized user areas.
The Navigator view is able to remove an entire directory along with its contents. As usually done with removals, this operation requires user confirmation.
Selecting a FITS image in the Navigator view shows a preview of it in the Outline view. Previews are cached so re-visiting them is faster.
Editors & Viewers
Text and Jython editors will save backup files so that, if HIPE exits abnormally, you can recover (at least part of) your unsaved changes.
When an error is produced the error highlight is now set to the closest (deepest in the call stack) line in the currently open script. Before, it was set to the top line (highest in the call stack); now it "goes into functions".
Dragging a variable from the Variables view and dropping it into a Jython editor will write its name in the drop location.
Variables can be created directly from nodes in the Product Viewer's tree, with the option Create Variable of its popup menu.
If a Product Viewer or Dataset Viewer shows data that is changed with a Jython command, it informs that the contents are outdated, and offers the possibility of a refresh.
Data tables can be sorted by clicking on the corresponding column header. Successive clicks will either add more columns to the sorting, revert the order (if re-click in the same column), or remove it by returning to the original state.
Console & History
Console capabilities have been improved: to show the list of available key bindings, press Ctrl-Shift-H with the Console as the current view.
The History now ignores empty lines. This means that, if you go back to previous commands in the Console with the up arrow, empty lines are skipped automatically, making the command navigation easier.
Startup & Shutdown
When variables have been saved in a previous session, HIPE asks whether to load them. This dialog can be optionally omitted.
If there are running jobs when closing the application, HIPE asks for confirmation before exiting.
HIPE checks whether there is a new user release at startup; if so, it informs the user with a link to the web page from where to download it.
Error handling
If HIPE crashes, it will inform about a file dump at next start, if possible. If you raise a ticket, please attach this dump for helping developers to debug the error.
HIPE has been made more robust against Out of Memory errors: instead of hanging, it offers the possibility of exiting in an ordered manner (for example, by saving unsaved editors), or try to continue working to manually free resources.
Perspectives
Perspectives have been simplified: now there is a unique Work Bench, while the proof-of-concept Classic (JIDE) perspective has been suppressed.
The PAL Browser perspective has now completely replaced the Data Access view.
Reset perspective option has been added at the right side of the perspectives toolbar, so it is now more visible and accessible.
Miscellanea
When opening a file, the window's title shows the complete path.
Tooltip in the spinning throbber (at bottom right corner) informs about all running jobs.
Tips of the Day have been added and improved.
This version contains some bugfixes with respect to the previous user release.
Plotting
New auxiliary axis class ReciprocalAuxAxis, providing reciprocal tick allocation.
TablePlotter
New APIs are introduced which allow to set initial column mode and to specify if data extraction is allowed or not.
OverPlotter
A legend check box is added to display the plot legend.
An Appearance menu item is introduced in Layer Props which allows to change layer's name in single layer mode and change both layer's name and color in all layer mode.
Astronomical utilities
Radial velocity's LSR definition is now consistent with MPS definition.
Radial Velocity can now also use data from downlink.
Reorganised aberration correction methods in the Aberration class of the ia.obs.auxiliary.pointing package.
Updated documentation.
Pointing
SSO correction now maintains the overall shape of a raster observation.
SSO can now also use data from downlink.
Updated documentation.
Numeric routines
New functions
NAN_FILTER: Returns the NaN elements of an array (in a one dimensional array) leaving the original array untouched. If no NaN element is found, the original array is returned.
IS_ANY_NAN: Checks if the given array contains a NaN value.
Median function (MEDIAN)
Median function returns NaN if any array value is NaN.
Source extraction
Every source in the input source list will now be included in the output source list, including sources that lie outside of the image.
Extraction now fails if the image has non-square pixels.
Images
Display
Display now has the SLS and the HSV color tables as known from ds9. It is also possible to change the color table according to the square and a power algorithm (Linear Scaling, Sqrt, Log and Histogram equalisation were already available)
Display can use Regions of interest.
added method getROI([row1, column1, row2, column2, ...]) to Display which returns a Selection.
added getFlagFromROI([row1, column1, row2, column2, ...]) to Display which returns a flag where all pixels are masked out, expect the pixels in the Selection.
added getFlagFromROI(selection) to Display which returns a flag where all pixels are masked out, expect the pixels in the Selection.
A preview is given when a fits file with a SimpleImage is selected in the Navigator.
In the Image Viewer section of the HIPE Preferences window you can now set the default algorithm and intensity of the colour table, as already possible by right-clicking on an image and selecting Edit colours.
Analysis
A lot of Tasks were incorrectly available twice. The tasknames ending in -Task are not longer available. For example: cropTask is not longer available, crop should be used instead.
Spectra
Display
Improved information displayed in the status bar when hovering mouse.
In the single spaxel selection mode the color of the rectangle correspond the color of the spectrum in the plotter
Analysis
Region specrum extraction
Added Sum and Median in addition to Average in the graphical interface.
Line intensity map
Added the Voigt profile, in automatic mode.
Creation
Cubes created using NearestNeighbourProjectionTask now contain unit and metadata information.
NearestNeighbourProjectionTask is now capable of regridding a list of spectral cubes.
NearestNeighbourProjectionTask is now capable of creating a default target grid (world coordinate system) from a list of target grids.
Interoperation with the Herschel Science Archive
Support for the new HSA data format (top directory named after HSA request ID and visible XML file at top level).
Support for importing data in HSA format without unpacking.
Calibration sources
Products and datasets
TableDataset
Column name duplicity is no longer allowed.
Product Access Layer
Improved the GUI editor for local stores.
The result table of the PAL browser is now customisable.
Added task localStoreCopier to create a compressed copy of a Local Store.
Local Store
You can rename local stores newly created with HCSS 4.0 by renaming their pool directory. For older local stores you need to run PoolSchemaEvoler.evolve() to update them to the new format before using this feature.
It is now possible to turn off the backup when re-indexing. Use the property hcss.ia.pal.pool.lstore.index.backup to control this feature.
Cached Pool
Caches now have a default size limit of 10 GB. Instead of continuing to grow, if more than 10 GB of data is retrieved into a cache, products that have been least recently used will be deleted. If this happens and the product is accessed again, it will be retrieved from the remote pool again. The 10 GB size limit is approximate and applies to the downloaded products only. Metadata and other small items are cached as well, but these are not taken into account for this limit.
Several issues on Windows have been fixed, mostly related to removing items from the cache.
HsaPool
Changes related to PAL interfaces modifications (added new methods, removed deprecated methods).
Support for importing observations using XML files located under .exported directory and root directory.
View: Save Products to Pool
With the HIPE view Save Products to Pool you can save products to a PAL product pool. New product pools (local stores) can now be created directly from this view.
HSA Access: HIPE Login
The HIPE login panel on the HSA view now encrypts the username and password, before sending them over the network (using HTTPS).
Tasks
All tasks can now be started and cleared with the HIPE toolbar buttons.
Improved default layout of task GUIs and behaviour of parameter editors.
Background tasks appear in the active list of tasks and send messages to the console when starting and finishing.
Quality control
New option in the File menu to get all the observations for an operational day. Optionally you can filter these observations by instrument.
New option in the Tools menu to create the summary for a set of observations in one single step.
Date added to the time column into the logs table.
Systematic product generation
New observations will contain only the pointing samples covered by the observation time range. This means that, for each observation, a new PointingProduct will be created containing only a copy of the datasets identified with the same obsid.
Attached the Horizons, Orbit Events and Satellite Housekeeping new products to the auxiliary context.
Quality tickets include the instrument mode for every observation.
Data input-output
simpleFitsWriter no longer stores the path of the FITS file when zipping.
Improved product recognition in fitsReader.
Improved GUI of fitsReader and simpleAsciiTableReader: they now automatically fill the output variable name based on the input file name.
getObservation has a new optional instrument argument which you can use in parallel mode to specify the instrument for which to retrieve the data.
Extended exportPalToUfDir task to allow renaming/moving
Import View improvements: redesigned user interface, supports importing zipped pools (and renaming, if necessary) and tarred (and gzipped) HSA data, importing observations to new pools, and importing each observation to a new pool.
Export View improvements: redesigned user interface, supports exporting zipped pools. It currently supports exporting HSA data as loose files.
Installer
The Versant setup screen in the advanced installation mode has been added to the Mac OS X platform.
Removed the option to cancel an installation once the download phase starts (when the installer connects to the repository to download the missing components).
Documentation
What's New
The What's New document is now web-based.
Quick Start Guide
Updated most screenshots.
Scripting and Data Mining
Expanded section on Cached Pool.
Added more links to numeric functions in the URM.
Removed section on JIDE view in HIPE.
Data Analysis Guide
Fully updated data input/output chapter.
Fully updated plot chapter.
Updated section on Spectrum Explorer.
Added links to spectrum arithmetics tasks in URM.
HIPE Owner's Guide
Fully updated text and screenshots of views and perspectives.
Updated location of dump files.
Expanded section on the HIPE Help System.
Added table on mouse events in HIPE.
Added section on HIPE toolbar and status bar.
HIFI
Configuration
Property Configuration
Using the build installer for all instruments will mean that the HIFI properties are correct and the HIFI pipeline will run out-of-the-box.
No need to set the HCSS_PROPS environment variable or use the hifi-new.props or hifi-installer-new.props files.
Properties files replaced by hifi.dp.properties, which is automatically picked up by the system.
hifiPipelineTask
HRS Pipeline
WBS Pipeline
Level 1 Pipeline
DoVelocityCorrection:
Use of Horizons ephemeris data for transforming to the rest frame of the source for SSO's introduced, using naifid to identify SSO's
Improvements to transform to the rest frame of the source for fixed target when using the task interactively
'freqFrame' meta data parameter, which informs what frame the frequencies are expressed in, is introduced
DoRadialVelocity:
No longer a part of the automatic pipeline but supplied for interactive use, it now uses the auxiliary product rather than rely on the velocity column in the datasets.
Compliance with DoVelocityCorrection in the sense that the LO Frequency is also adjusted and the same meta data parameters are set.
Fix of the problem associated with the interpolations schemes used in the DoOffSubtract and the DoFluxHotCold. Use of the linear interpolation scheme had lead to a "noise explosion" in the pipeline. A truncated linear interpolation scheme is introduced which uses nearest neighbour in case measurements are not bracketed and linear in case they are. The pipeline is now configured to use:
nearest neighbour interpolation for FSwitch and LoadChop in DoOffSubtract
truncated linear interpolation for all the other observing modes in DoOffSubtract (actually only OTF).
truncated linear interpolation for all the observing modes in DoFluxHotCold.
Level 2 Pipeline
Fixes for cross maps: the points in a cross map are no longer averaged into one point.
Names of map contexts at level 2 changed to fit HCSS convention.
Products
Hot and cold load (hc) observations are boxed in the HTP in the same way that datasets are
DoDeconvolution
Introduction of an 'expert' panel in the GUI
The following parameters are found in the 'expert' panel, which is on the right hand side:
use_entropy
lambda1_channels
lamda2_gains
cont_offset
Variable name for interim_output
Plotting of chi^2 against iteration number is possible from the expert panel. On the same plot the channel rms'es, the channel and gain inverse entropy are plotted as well.
It is also possible to plot SSB solution against frequency for each iteration of the deconvolution. There are four plots to a page.
Bin size is automatically set by DoDeconvolution to be the same as the sampling of the DSB data, the information is taken from the metadata "frequencyWidth".
Only the USB is used as input data to avoid the smearing of narrow lines that can arise after averaging LSB and USB data.
DoGridding
FitHifiFringe
A number of bug fixes have made FitHifiFringe more stable in HIPE 4. In case a problem still occurs, a warning message is given, but the task will not crash.
Also, the sine wave fitting parameters applied to the data are now stored in a table in the ObservationContext (click on 'calibration" and then "pipeline-out')
HiClass export tool
Flags
SpectrumExplorer (HIFI specifics)
Faster loading of HifiTimelineProduct with ability to select which datasets to load by displaying summary table information.
HIFI DP scripts (User tool scripts)
Introduction of PolarPairs, a class to average and difference H and V polarisations
Documentation
HIFI User's Manual
Introduction of HIFI Cookbook, as first chapter of HIFI User's Manual
New chapter for PolarPairs
New chapter describing relative (rms) performance of HRS and WBS
Updates to HiClass (export to CLASS) chapter
Updates to HIFI pipeline chapter
Bug fixes and improvement of figures in pdf versions of document
HIFI Pipeline Specification Document
"Generic Pipeline" (verrrrry old name for what is now known as Level 1 and Level 2 pipelines) chapter split into sections for Level 1 and Level 2 pipelines
Addition of DoVelocityCorrection section, rearrangement of names and orders of sections to reflect current status of pipeline
New flow diagrams for levels 1- and 2- pipeline
Introduction of Level 0 pipeline description (incomplete)
Bug fixes and improvement of figures in pdf versions of document
Other
PACS
Pipeline
Photometry
Photometer Flux calibration Calibration products involved in the photometry adjustment are stable. Responsivity determination comes from astronomical sources and gives with the flat an acceptable flux. Estimated accuracy on point sources ± 5%, estimated accuracy on extended sources ± 10%
Spectroscopy
specAddNod is not part of the level 1 chopNod pipeline anymore. Instead, the level1 product is sliced according to nod positions (SlicingRule "NoddingPosition") and the nod slices are then recombined in specProject speccAddNod now works on cubes.
specCorrectHerschelVelocity can now optionally just compute the S/C velocity without applying it to the wavelengths.
specAddNodCubes has been improved and now works on contexts with multiple raster positions as well. Also, you no longer have to split the data to raster position before running this task.
Calibration
Proper treatment of time dependent SIAM calibration product. Now we have a first time dependency with the SIAM. It is important to pass the observation context to getCalTree:
calTree = getCalTree(obs=obs)
or
calTree = getCalTree(time="20-Apr-2010 12:00:00")
If this is not done, the latest SIAM will be applied, which will give a 5" offset for observations before OD320
getCalTree now accepts a FineTime in addition to a String time representation.
Two new bad pixels identified for the red photometer Analysis based on the pixel noise then individual/neighbour pixel behavior comparison Pixels flagged [4,5] and [3,6]
Photometry
Blue and green separated in ArrayInstrument calfile and in the code using it.
Analysis tools
Ipipe scripts are now part of the HIPE top Menu (pipeline -> PACS)
Photometry
New Task: MapIndex2signalCubeTask allows to recreate a signal cube from a MapIndex.
PhotProjectPointSource has been extended to accept the same options as PhotProject
Photometer 2nd level deglitching more memory efficient
A more memory efficient deglitching is available. The map can be devided into a chess-board like pattern and deglitiching may be done tile by tile. This way, less memory is necessary, because the MapIndex will be calculated only for the currently processed tile. Quicker processing becomes possible by using the FullMapindex instead of the SlimMapindex.
Madmap improvements
Flux correction for scale < 1. A factor of 1/(1/scale)^2 is missing in previous releases when scale < 1.
Huge improvement on memory usage.
Use Mattools to calculate WCS, which got rid of usage of Basic.MEDIAN that causes JVM crashes for large datasets.
The Madmap output now contains coverage map instead of exposure map.
Spectroscopy
The Spectrum Explorer now also works with the PACS Frames and PacsCube products. You can open the Spectrum Explorer in HIPE by right clicking on a variable of such a product and selecting Spectrum Explorer. We have extended this tool with additional panels to perform the following tasks:
(De)activate masks.
(De)select complete spaxels in a Frames object (all 16 pixels).
(De)select blocks from the blocktable. Most standard functionalities are available, but selecting data points and creating a user mask does not yet work properly.
Product changes
PACS Products now contain all available meta data.
Spectrometer pipeline scripts now contain addSliceMetaData step which adds the meta keywords aotMode, lineDescription and rasterId to the slices and adds three columns to the MasterBlockTable: MinWave, MaxWave and LineDescription
Spectroscopy
Reordered the PacsCube data in the wavelength domain and added PacsStatus dataset
Documentation
Updated the PACS Data Reduction Guide, mainly in the spectroscopy chapter. The new version has updated explanations of the spectroscopy pipeline and of the so-called slicing pipeline.
SPIRE
Data Products
SpirePreprocessedCube now contains raster/jiggle position data.
Level-1 Spectral Products are now all calibrated using the extended-source flux conversion.
New Level-2 Product introduced for Point-source sparse sampling observations. This Level-2 data product contains the point-source calibrated detectors for the two target (typically the two central) detectors of each Spectrometer array.
Calibration Products
New calibration product introduced to convert from Level-1 extended-source calibration to Level-2 point source calibration for Point-source sparse sampling observations.
Reference interferograms and Flux Calibration products have been updated to allow for the creation of proper Level-2 Spectral Maps.
Reference interferograms now created such that the interferogram baseline has been removed.
Preprocessing
Some uplink parameters now copied into the metadata of the ObservationContext.
Common Pipeline
Metadata propagated into various level contexts.
Photometer Pipeline
Added new deglitcher task to remove concurrent glitches (concurrentGlitchDeglitcher). Not yet included in the standard pipeline but available to users.
Added new task to detect jumps in bolometer timelines (signalJumpDetector). Not yet included in the standard pipeline but available to users.
Spectrometer Pipeline
All Pipelines
Order of pipeline processing has changed to improve SLW signal to noise ratio. Telescope/SCAL correction now follows 2nd-Level deglitching.
Level-1 and Level-2 spectra data by default now only contain spectral information within a range defined by the Spectrometer Band Edge calibration product (see SPIRE Observer's Manual for spectral ranges). NB To access out-of-band spectral samples, you must use the pipeline in an interactive manner.
Pixel Sizes have been updated for mapping observations based on in-flight calibration data:
Single-pointing Level-2 now contains a point-source calibrated spectra for the two target (typically the two central) detectors of each Spectrometer array.
Raster Level-2 spectral maps are now produced for SOF1 Raster observations.
Spectral Maps (Raster/Jiggle observations)
SpirePreprocessCubeTask has added three filtering parameters for specifying which spectra are passed to output: UNVIGNETTED, detectors, and arrayType. These parameters may be used to help with construction of spectral cubes from customized lists of detectors.
Interested in what's new for other versions of HIPE? See this page for links to all the What's New documents from HIPE 4.0 onwards. 
Public Web
Create New Topic
Index
Search
Changes
Notifications
Statistics
Preferences
Public
