TWiki> Public Web>HipeWhatsNew>HipeWhatsNew8x (revision 34)EditAttach

What's coming up in HIPE 8

PDF Version Click here

See also the HIPE known issues and the Data products known issues pages for last-minute information on known problems.

Core system

Jython major upgrade. The version of Jython on which HIPE is based has been upgraded from 2.1 to 2.5.2. You may have to adapt your scripts for them to work with the new version. See this page for the details.

HIPE

Status bar

The maximum memory level reached is indicated with a vertical line in the memory bar. The tooltip shows the actual reached value, and allows to reset it so that further computations can record its own maximum.

Editors & Views

Variables View
  • Added contextual menu action Show contents to show the contents of a variable in the console.
  • Added contextual menu action Send To -> Text File to save the contents of TableDatasets in files.
  • Names of variables created from the Product Viewer are more consistent and meaningful.

Product Viewer
  • The title (product's description) can be selected and copied.
  • The expanded sections can be resized, for example to show more metadata rows.
  • The summary of the Observation Viewer has been enriched with useful metadata of the observation being shown.
  • Products can be saved directly from the tree with the Sent To action of the popup menu, without needing to create an intermediate variable.

Jython & Text Editors
  • The supported Jython language has been upgraded from version 2.1 to 2.5.2.
  • Scripts and text files coming with the installation or with plug-ins are safe to edit without losing the original version.

Help & Debug

  • The Tip of the day contains information on the Console meta commands (those starting with !)
  • If you get temporary (or permanent) freezes in the HIPE GUI, or you have created a custom task dialogue or view in HIPE, you can debug it by enabling the corresponding preference in Edit > Preferences > General > Debug. Any problem found with this debugging can be reported like previously, with the HSC Helpdesk or Give Feedback options of the Help menu.

Plug-ins

  • More protection against problematic plug-ins. You can now disable all plug-ins without having to start HIPE. This can be convenient when you know or suspect that a plug-in causes HIPE not to start. To disable all plug-ins, run the disable_plugins utility located in the bin directory of the installation. You can use the Plug-ins panel (from the Tools menu) to re-enable your plug-ins.
  • More flexible update mechanism. If multiple updates for a plug-in are found, they are now all shown. Previously only the most recent update was shown.
  • Improved user feedback. You now get popup messages when no updates are found for installed plug-ins, and when you install a plug-in incompatible with your version of HIPE.
  • Many improvements to plug-in installation:
    • The installation process is now much faster for large plug-ins, such as those containing data pools.
    • The HIPE interface is no longer unresponsive while installing a plug-in.
    • You can follow the progress of the installation in the progress bar at the bottom of the HIPE window.
  • Fixed a problem with the scanamorphos plug-in, which would not read correctly some scanamorphos output. The plug-in assumed that the PrimaryImage HDU contained a 3-dim array. Output containing a 2-dim array in the PrimaryImage HDU was not read correctly.

Upgrades

If there is a new user release available, HIPE would notice it and offer to install the new version.

Plotting

PlotXY is now able to display images (i.e. 2D data). Look at an advanced example here.

TablePlotter

TablePlotter can display plot with Date/Time format.

OverPlotter

OverPlotter can display plot with Date/Time format.

Dataset viewer

Astronomical utilities

  • Planck class moved from spire_ia_pipeline_spec_util to ia_toolbox_astro. If you are doing an explicit import (it is not needed) you have to change it to (preferred) import herschel.ia.toolbox.astro.Planck or from herschel.ia.toolbox.astro import Planck
  • New task astrometryFix: adjusts the absolute pointing of map data to give the best possible agreement in overall astrometry with some ancillary data. This is done either by cross correlation of the map(s) with ancillary map data, or by stacking the map(s) at a list of known source positions (e.g., from radio observations).

Numeric routines

Data fitting

Data interpolation

Random numbers

Statistics

Trend analysis

Mapping

Wavelet transform

  • Continuous Wavelet Transform synthesis operation now adds low frequencies for a reliable reconstruction
  • Continuous Wavelet Transform is able to deal with Complex1d signal

Other functions

Source extraction

  • Improvements to handling of pixel coordinates and region of interest in sourceExtractor* tasks:
    • Output source list now contains pixel coordinates of source (x, y) = (column, row)
    • Added "getWcs" and "setWcs" methods to the SourceListProduct/Dataset
    • Added "roi" parameter (region of interest) to tasks
    • Removed "corner*" and "returnPixelCoordinates" parameters from tasks
  • Non-standard columns are now accepted in a SourceListDataset, e.g., when converting from a TableDataset (an exception will be thrown if there are no standard columns at all)
  • Removed "subtractMedianBackground" parameter from sourceExtractorSussextractor (it didn't do anything)
  • Minor bug fixes, performance improvements and various internal changes

Images

Display

  • The methods addCircle, addEllipse and addRectangle is not longer dependent on the zoomfactor of the image.

Analysis

Spectra

Display

Analysis

The SpectrumFitter now recognizes a CUBE such that the user can give the x,y of the spectrum. The script as written by the SpectrumFitterGUI also takes the correct spectrum.

Data cubes

Display

Analysis

Herschel Science Archive

Calibration sources

Products and datasets

All products

  • Removed the legacy diff methods from Product and Dataset interface. All functionality moved into ProductDiff class which is available as part of the utility toolbox (herschel.ia.toolbox.util)

Observation context

Product Access Layer

Tasks

  • General clean up of tasks:
    • All mandatory parameters must be filled in by users: removed default values from mandatory parameters, warnings issued (detection of programming mistakes)
    • Improved parameter checking and messages on bad data: Validators in primary inputs, they return false on error and log a warning (detection of programming mistakes)
    • All task GUIs and nodes should now show tooltips
  • Added some new user-friendly parameter editors (see For Task developers only)
  • Default task layout:
    • Parameter name tooltips now show the parameter name (for long names)
    • big modifiers can occupy a full row
    • groups (tabs) can be navigated with the mouse wheel
    • Parameters can be reset individually via contextual menu on parameter names (vs resetting all parameters with Clear button)
    • Parameters can list the variables they can accept via contextual menu on parameter names (to know before hand if we have the proper data to use a parameter)
  • For Task developers only:
    • Task registration will issue warnings if either the task was already registered or the taskname was already used or both.
    • Command generation: added converters so that arbitrary types can be represented with Jython strings (part of better visibility for defaults)
    • New shared Modifiers (parameter editors):
      • Added Editor for selecting a subset of a list of values (JListModifier, N values of M possible, M >= N)
      • Added a variant of JOptionModifier (Combo box) that returns some values but shows others (example: parameter uses integers but we show strings to users)

Misc Tasks

  • OpenTask parameter names have changed capitalization to comply with lower camel case

Quality control

Systematic product generation

Data input-output

  • URM entries (documentation) improved
  • Task importUFDirToPal renamed to importObservation
  • Task exportPalToUFDir renamed to exportObservation
  • Task simpleAsciiTableReader extended to support all options of asciiTableReader via Advanced tab. (eventually, asciiTableReader will subsume simpleAsciiTableReader)
  • Improved type guessing for table reading (affects (simple)AsciiTableReader): for floating-point data it will detect precision loss and maintain representation. If a floating-point number will lose precision as a float (32-bits) a double (64-bits) will be used instead: up to 17 significant digits but in these case it will use twice the memory.
  • Improved documentation and information provided by getObservation
  • Added contextual menu option to Variables View to save tables with asciiTableWriter

Virtual Observatory

Installer

Sometimes, especially when a user has a shaky internet connection, an installation might fail in such a way that the local (in the user's machine) repository of hcss libraries might get corrupted. The workaround that existed so far can be found in the known issued page in: http://herschel.esac.esa.int/twiki/bin/view/Hcss/KnownIssuesHipeInstaller#Installer_fails_with_a_zip_error. The solution in short was to remove the local repository and all existing HIPE installation and start over. However a new solution that hopefully solves this issue has been implemented in 8.0. We know that bad proxy setup, hardware issues in routers can cause packet loss even in TCP. Such loss should always result in a file which is smaller than actually intended. Normally such a situation should result in an exception, but with packet loss in e.g. a router this will go unnoticed and a normal EOF completes the transmission. However, the resulting ZIP file will then always fail to unzip. The installer has been modified to extract the content-length from the HTTP header and validate the number of bytes actually received and raise an exception if such thing takes place. Moreover, a better exception reporting has been implemented to give more information on the basic cause of the exception. Finally the user installer reports the estimated time of completion while installing to give a rough feeling to the user.

Documentation

HIFI

HIFI Software Configuration and Properties

HIFI Calibration

HIFI Pipeline

hifiPipeline task

Level 0 Pipeline

WBS Pipeline

HRS Pipeline

Level 1 Pipeline

Level 2 Pipeline

HIFI Products

HIFI Spectrum Toolbox

Spectrum Explorer (HIFI specifics)

Spectrum tasks (HIFI specifics)

HIFI Data Processing Tools

Standing Wave Removal

Baseline Removal

Deconvolution

Spectral Cube creation

HiClass export tool

Documentation

PACS

Pipeline

Photometry

Spectroscopy

Calibration

The calibration sets that are installed on your local system can now be inspected with a 'Calibration Sets Viewer'. The viewer is located in the menu Window -> Show View -> Workbench -> Calibration Sets. Select a version from the available buttons, the you can inspect the release note for that version or the product list with the versions of the individual calibration products.

Photometry

Spectroscopy

Flux calibration now based on observations of astronomical standards and the calibration block attached to each observation. The pipeline task specDiffCs computes the detector response of the current observation.

Analysis tools

The Preferences now contain a Logging panel under the general tab which allows the user/developer to increase or decrease the number of logging messages send to the terminal.

Photometry

Spectroscopy

Product changes

Photometry

Spectroscopy

Documentation

SPIRE

Calibration Products

Calibration Framework

Preprocessing

Engineering Conversion

Common Pipeline

The default value for reconstruction in the sigmaKappa deglitcher is changed to "DIRECT".

Photometer Pipeline

Spectrometer Pipeline

Averaging Spectra

The Average Spectra module no longer supports the bandEdge parameter. This parameter was used only for the module's defunct outlier rejection functionality, and so it is no longer needed. Please ensure that your scripts do not use this parameter.

Interactive Analysis and Tools

Destriper

The following new features are introduced in Hipe 8:
  • Level 2 deglitching is available through setting l2DeglitchRepeat parameter.
  • The 5 individual output parameters are combined to one ArrayList parameter where each of 5 parameter can be retrieved base on its position.
  • The new diagnostic product is introduced which provides with useful diagnostic information.
  • Both Tod and TodBuffer are acceptable input for Tod parameter.
  • The input parameter "array" now is MANDATORY.
  • Pixel size can be changed through setting pixelSize parameter.
  • brightSource parameter is changed to boolean.
  • A boolean parameter, "fitWithWeight" is introduced.
  • Interruption is added so that the destriper can be terminated at the user's request.
  • Tod files stored in temporary directory are available until Hipe is terminated.
  • The destriper speed is increased 30%.
  • The input parameter name "Level1" changed to "level1".
  • The input parameter name "Tod" changed to "tod".
  • The input parameter name "L2DeglitchRepeat" changed to "l2DeglitchRepeat".

Baseline Removal

  • New "roi" (region of interest) parameter for baselineRemoval* tasks, specifying the region of the sky to be used in estimating the baseline
  • Removed "doInterpolation" task parameter
  • Minor changes and bug fixes (for example, the tasks now do ignore masked data samples when a TodBuffer is used internally)

Documentation

Edit | Attach | Watch | Print version | History: r96 | r36 < r35 < r34 < r33 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r34 - 2011-10-25 - ThanosTsounis
 
This site is powered by the TWiki collaboration platform Powered by Perl