TWiki> Public Web>HipeWhatsNew>HipeWhatsNew5x (revision 40)EditAttach

What's New in HIPE 5.0

PDF Version Click here

Core system

HIPE

Editors & Views

  • Files ending with extension ".csv" are recognized as tables, and opened as such with double click in the Navigator view or with File > Open.
  • The tooltip shown when the mouse cursor is over a task in the Task view is now more informative.
  • Autocompletion in the Console has been improved.
  • A variable value can be copied by right clicking on the variable in the Variables view and selecting Copy value; it can be pasted in the Console or Jython editor afterwards. This works only for strings and numbers.
  • Drag & drop a variable from the Variables view to the Console writes the variable name. The same behaviour can be achieved when dropping it to a Jython editor.
  • Drag & drop selected text from a Jython editor to the Console or vice-versa copy & paste that text.
  • The small level-2 preview shown in the top-right corner of the Observation Context viewer can be enlarged by just clicking on it.

Startup & Shutdown

If the running Java platform is not supported, a warning is popped up.

Look & Feel

Aqua look & feel has been replaced in Mac OS X by the HIPE look & feel that is used among all platforms.

Plug-ins

HIPE now supports plug-ins, with are small packages that anyone can easily create and distribute to share code (Java or Jython) and data (LocalStores). Plug-ins can be downloaded and installed with a single-click.

Plotting

  • Title and label can accept LaTeX commands \textrm \textit \textbf \mathrm \mathit \mathbf. for example, V$_{\textrm{LSR}}$

TablePlotter

OverPlotter

Dataset viewer

Astronomical utilities

  • RadialVelocity now uses TimeCorrProduct to select pointing data
  • SSo now uses TimeCorrProduct to select pointing data
  • Improved documentation

Numeric routines

Data fitting

  • A complete new GUI for the SpectrumFitter is added. Start it from the SpectrumExplorer right mouse menu, select SpectrumFitterGUI. Besides the functionality that was available in the SpectrumFitterTool, the GUI has an export TAB that enables the user to export the residual, total model, and individual models as an object with the type of the input object and as ASCII.
  • The SpectrumFitter (commandline version) accepts Container with PointSpectrum and SpectralSegment index.
  • The SpectrumFitter (commandline version) has methods to export the data as the input type and as ASCII.
  • The SpectrumFitter (commandline version) has method that returns a Jython script

Data interpolation

Random Numbers

  • New random numbers generators from different distribution: Uniform, Gaussian, Poisson, Gamma, Cauchy (Lorentz), Exponential and Jeffreys.

Statistics functions

  • RMS has been renamed to QRMS
  • RMS implements a new functionality (see URM)
  • Improved StdDev and Variance algorithms.
  • Improved CorrelateMatrix algorithm.

New functions
  • GeoMean which calculates the geometric mean of a array of numeric data types with 1 to 5 dimensions.
  • Mode which returns the mode(s), or the most common element(s), of an array of numeric data with 1 to 5 dimensions.
  • Covariance which yields the covariance between two random variables/vectors x and y with finite second moments. If one vector is longer than the other, only the values up to the length of the shorter vector will be taken into account.
  • CovarianceMatrix which returns the covariance matrix of the input M x N matrix. The result is a N x N matrix with each i, j value equal to the covariance of the ith and jth columns of the original matrix.

Trend analysis

Mapping

Other functions

  • SigClip
    • Median mode uses Median Absolute Deviation instead of Standard deviation.
    • Iterative use is allowed.

Source extraction

  • No significant changes to functionality.

Images

Display

  • Images can be opened as RgbImage directly from the Navigator in HIPE.
  • Regular image files (jpg, gif, png, ...) are shown as preview in the outline when being clicked on in the navigator view of hipe.
  • 2 images can be compared by setting the opacity of the image. This can be done using the setOpacity(float) method or using a slider in the Image Display.

Analysis

  • Added the methods containsNorthCelestialPole and containsSouthCelestialPole on the Wcs.
  • Regridding an image on the grid of another image using RegridTask
  • Speed improvement of createRgbImageTask
  • Update of URM documentation
  • Cropping an image by drawing a rectangle on it
  • Correction of the calculation of the dimensions of a mosaic

Spectra

Display

Analysis

Data cubes

Display

Analysis

Herschel Science Archive

  • Support for exporting data in packed HSA format.
  • Import and Export Views improvements:
    • better error reporting
    • file dialogs can show hidden files
    • versions of observations inside pools shown

Pipeline processing

  • Crated a daemon dispatcher mechanism.
  • Aliases set tool has been modified for performing the required verifications before a tag is ingested.
  • HsaPoolDaemon
    • releases memory calling Java garbage collector if it is needed.
    • can write compressed files.
    • can save a binary representation of a product if the FITS save operation fails.
  • HsaStagePool can export products into a LocalStore.
  • HSA ingestion procedure can generate an XML ingestion request file even if a previous version does not contain its own XML.
  • Improved test-harnesses to include an HAIO simulated server framework.

Calibration sources

A new view has been added to HIPE for easy access to the Calibration Sources Database (Calsdb).

Products and datasets

  • Product comparison API is available in ia_toolbox_util module.

Observation context

  • Include new metadata "slewTime" returning the scheduled start time of the slew as a FineTime
    • Specific accessors, getSlewStartTime() and setSlewStartTime(), where added to the ObservationContext class
    • For old observations in the hsa without this value, the method will return a null value indicating that this field was never set before

History

A new history dataset viewer is available: When you click on the History dataset in your product then two tables appear above another:

  • An overview table which shows all the tasks used to generate the product. For each task, the name, execution time, build number and used calibration files are shown.
  • When you click on a row in this overview table then a second table is shown with details for the selected task. This table contains the details of this task: For each parameter its simple value and some product information is shown (when available).

Product Access Layer

Cache

Several bugfixes to improve the effectiveness of caching and the robustness has been introduced. One of these is a check to verify that the default LocalStore directory is not modified at any time during the cache's existence, which is a common source of corruption of the cache.

HsaPool

  • HsaReadPool/HsaXmlPool can work with compressed data.
  • HsaReadPool can retrieve product metadata directly.

Preferences

Product storages and pools can be managed in the new Data Access > Storages & Pools preferences panel.

Tasks

  • Updated task sample code
  • Updated documentation:
    • wiki chapters
    • Javadoc of general tasks
    • Standarized _ doc _ of general tasks (signature, description, parameter descriptions)
  • Tasks View provides tooltips with the signature of the tasks
  • Examples of general tasks in the URM are now tested automatically
  • Compress and Decompress tasks: they support zip, tar, gzip, tgz in a portable way and decompress can guess the compression of a file.
  • FitsReader supports compressed FITS files.

Quality control

  • New metadata into the QualityContext product. The field is fixed and the main interest is to include this information into the FITs file associated to the product.

FITS keyword Value
PCAVEATS Please refer to http://herschel.esac.esa.int/DpKnownIssues.shtml for known problems in products

  • Perform bulk actions on several qc reports simultaneously
  • Flags color codes depending on their importance

Systematic product generation

  • New framework to process/combine data from several observations in the same pipeline
    • This will allow, for instances, better results using MADmap algorithm with cross-scan data from two or more observations
  • Automatic processing:
    • New start-time algorithm based on the DTCP time of the OD being processed instead of fixed timestamps
    • New check on TM data based on the gap checker reports. Only those observations with complete tm data will be processed at this point
  • Uplink plugin is executed without problems in different conditions, even for manual commanding and no uplink data at all.
  • Obsids shown also in decimal format in all the application tables
  • Include proper motion data into the ObservationContext
  • SPG stand-alone application can now run on MAC O.S. with no versant libraries available.
  • On-demand request are now recorded into a report file

Data input-output

  • Imporved error reporting in AsciiTableTool
  • FitsArchive
    • FITS Metadata direct access.
    • ESO Hierarch keyword implemented when reading FITS files.
    • FITS translation dictionary updated:
      Herschel keyword FITS keyword
      pmRA PMRA
      pmDEC PMDEC
      state STATUS
      action ACTION
      slewTime SLEWTIME
      proCaveats PCAVEATS
    • FitsArchive allow flag for compression when saving FITS files.
    • FitsArchive support reading gzipped FITS files and compressed stream headers.
    • HeaderWriter exceptions have been modified to include more information.
    • Documentation updates.
    • JExamples automatic testing support.

Virtual Observatory

Installer

There was a major restructuring of the pre-installation steps
  • The Basic/Advance selection page has been removed.
  • The memory, proxy and versant settings were restructured. There are no separate panels asking if the user wants to set a proxy or a versant. There is a dedicated page for the proxy setting and one for the versant settings. In the top of the panel there is the question with a check box. Then there are the text fields not editable by default which will become editable upon user action on the above check box.
  • Instead of the old instrument selection page, the user will now be offered the following list with check boxes for selection:
    Option Description
    PACS (preselected by default) PACS pipelines and analysis tools for astronomers.
    PACS expert Extra instrument-related applications for Calibration Scientists.
    SPIRE (preselected by default) SPIRE pipelines and analysis tools for astronomers.
    SPIRE expert Extra instrument-related applications for Calibration Scientists.
    HIFI (preselected by default) HIFI pipelines and analysis tools for astronomers.
    HIFI expert Extra instrument-related applications for Calibration Scientists.

  • The instrument selection window should allow to click in any combination of items to install those components.
  • It should also allow the deselection of all instruments to get HIPE framework only for FITS i/o and numerics analysis.
  • If possible, the installer should inform the user of the amount of data to download for the selected combination of modules.
  • The Proxy selection page should contain the Yes/No button and the entry fields, which are kept in grey until "Yes" is selected. The default is kept to "No".
  • The Versant selection page should contain the Yes/No button and the entry fields, which are kept in grey until "Yes" is selected. The default is kept to "No".
  • The rest of the pages should stay as they are.

Documentation

To improve the quality of the examples, we have introduced automatic testing of (many of) the source code examples occurring in manuals such as the Data Analysis Guide and the HIPE Owner's Guide. We have also introduced automatic testing of the examples occurring in the User's Reference Manual.

HIFI

hifiPipelineTask

HRS Pipeline

WBS Pipeline

Level 1 Pipeline

Level 2 Pipeline

Products

DoDeconvolution

DoGridding

FitHifiFringe

HiClass export tool

Flags

SpectrumExplorer (HIFI specifics)

HIFI DP scripts (User tool scripts)

Documentation

Other

PACS

Pipeline

  • PACS "New Style Sliced" pipeline implementation is now activated
    • PacsSpr:2647 : phot and spec ipipe script names are changed to be more logic and intuitive
    • Pipeline Script access
      • HIPE -> Menu -> pipeline -> PACS
  • Calibration framework Calibration products are no longer delivered with the build in a local store.
    Calibration products are now delivered as a directory of FITS files.
    This directory with calibration products can be found in the data/pcal sub-directory of your installation directory.
    HIPE> fm = getCalTree(verbose=1)
    CalTree read from: /Software/Herschel/hcss.dp.pacs-5.0.500/data/pcal
    The calibration framework is adapted and will read these FITS files by default.
    The getCalTree() command and all access to calibration data has not changed, under the hood, the local store has been replaced by the directory of FITS files. Other small changes:
    • getCalTree(), getCalProduct() now understand the 'verbose=True' keyword and will print information on the location of the calibration products.
    • getCalProduct() the third argument, 'version', is no longer mandatory, if not given, the last version of the calibration product will be returned.
  • a new Pacs Documentation page has been opened on the public wiki.

Photometry

  • MadMap
    • Extended Madmap (using scan and cross scan) for interactive usage
      • HIPE -> Menu -> pipeline -> PACS -> L2_scanMap.py
        Based on HCSS-9361, a prototype for the extended Madmap pipeline is now available in HIPE for interactive usage. The extended Madmap pipeline is designed to combine scans and cross-scans in order to produce optional map. Unlike DP pipelines (which normally starts with one ObservationContext), this pipeline takes an obsList (an ArrayList) which is a list of ObservationContext. The pipeline will run even there is only one obs in the obsList. Below is an example creating an obsList:
        HIPE> obsList=ArrayList()
        HIPE> obsList.add(obs1)
        HIPE> obsList.add(obs2)

  • New script for scan Map pipeline
    • PacsSpr:2801 Proposal for a two-stage photometer pipeline with masked highpass filtering
      • Now THE standard processing script for scan maps
  • Photometer Mapping
    • PacsSpr:2853: Photometer mapping accepts a variable input pixelsize (parameter pixfrac in the PhotProjectTask and the MapIndexTask).
    • PacsSpr:2789: The MapIndexViewer has a Sigclip gui to display the effect of different Sigclip parameters on the second level deglitching. Now it is not necessary any more to guess Sigclip parameters, run the deglitching and check the quality afterwards. The new option allows to find the best parameter quickly BEFORE deglitching has been applied.
      The MapIndexViewer now also displays the signal vectors in the "timeordered" mode.
    • PacsSpr:2921: the MapIndexViewer shows mean/median +/- error permanently in the signal plot
    • PacsSpr:2807: the Full MapIndex needs less memory than before
    • PacsSpr:2859: Wcs4mapTask got a crota option to manually set the requested rotation angle of a map.
    • PacsSpr:2925: CorrectRaDec4SsoTask is now also accepting the new Horizons class definition. An example how to use it can be found in the attached
    • PacsSpr:2751 : photProject error map has been checked for NaN
    • PacsSpr:2543 : exposure map has been removed in favour of the coverage map
  • Noise treatmement
    • PacsSpr:2752: A new modul herschel.pacs.spg.phot.PhotAddNoisePerPixelTask provides 4 ways to calculate the initial photometer noise. *Noise propagation is also implemented in the HighpassFilter, Mean- and MedianFilters.
  • Pointing
  • Others
    • PacsSpr:2761 PACS EDP: phot pipeline adaptation to cope with variable scan velocity
    • A new module photMaskFrames offer masking :
    • framesOut = photMaskFrames(framesIn [,beforeFirstScanLeg] [,noScanData] [,afterCalblock=0] )
      • beforeFirstScanLeg : Mask data befor the first scan leg
      • noScanData : mask data where the ScanBBID is not set (includes turn arounds)
      • afterCalblock : Mask n data after calibration block
    • HcssSpr:6681: Sigclip can be applied n-times in one go.

Spectrometry

  • new AOR mode "unchopped" including improved spatial calibration:
    • First draft of the unchopped mode pipeline added to the build
      • PacsScr:2779: a "newstyle" unchopped mode pipeline is in place which can be run automatically or interactively
    • PacsScr:1048 New task specSubtractOffPosition implemented for unchopped pipeline
  • new interactive tool SpectrumExplorer:
  • improved wavelength calibration:
    • PacsScr:2964: new wavelength calibration applied in waveCalc task using the wavePolynomes cal file
  • spectrometer trend analysis product for the calibration blocks:
    • PacsScr:2777: calibration block trendAnalysis product added to the observation context
  • Extra options to specDiffChop (note these are EXPERT options for the analysis of special observations!)
    • PacsScr:2717: extra option 'extend' added which allows for other pairwise differencing schemes. These schemes will subtract more datapoints from each other than the standard scheme.
    • PacsScr:2427: extra option 'chopNodScheme' which provides different scheme for determining which label is on-source and which is off-source for a given nodding position. Needed for some special test AORs.
  • Different bands in a range scan are now stored in their own blocks in the block table. This means that it is now possible to slice SEDs per band (PacsScr:2963).
  • Several fixes to specWaveRebin, etc. to reduce the amount of NaNs considerably in the rebinned data.
  • SpecProject sets the flux to Jy.

Calibration

Photometry

Spectrometry

  • PacsSpr:2488 : New Wavelength Calibration Product
  • PacsSpr:2855 v8 of the Pacs Spectrometer Array to Intrument and v4 of the Module2Array cal file is inserted in the system. This generates correct values also for center chopper positions (unchopped and waveswitch mode).
  • PacsSpr:2715 : updated spectrometer spatial cal file --> PCalSpectrometer_ArrayInstrument_FM_v6.fits
  • PacsSpr:2607 The RSRF calibration tables contain "NaN" which are then introduced into the signal by rsrfCal

Analysis tools

Photometry

Spectrometry

  • Compatibility of PACS products with HCSS tools improved: One can now properly select data points and use the fitter tool on PACS frames and cubes.

Product changes

Photometry

Spectrometry

Documentation

Other

SPIRE

Calibration Products

Photometer

Spectrometer

New
  1. Non-linear phase
    • This calibration product is used by the phase correction module to make the data symmetric prior to application of the Fourier transform.
  2. Spectrometer Instrument RSRF
    • This calibration product is used, along with the instrument temperatures from the housekeeping telemetry, to compute a model for the contribution to the measured spectra from the SPIRE instrument.
  3. Telescope RSRF
    • This calibration product is used, along with the Telescope temperatures from the housekeeping telemetry, to compute a model for the contribution to the measured spectra from the Herschel Telescope.
    • This calibration product is also used for extended source flux calibration.

Updated

  1. Point-source RSRF
    • This calibration product has been updated to use a higher signal-to-noise observation of Uranus.

Calibration Framework

Preprocessing

Engineering Conversion

Common Pipeline

  • Only minor changes in logs and documentation.

Photometer Pipeline

  • Pipeline scripts:
    • Added time correlation task and concurrent glitch deglitcher task in scan map pipelines
    • Comment out optical and electrical crosstalk corrections
    • Removed plotting blocks
  • Pipeline tasks:

Spectrometer Pipeline

The spectrometer pipelines have undergone major changes for version 5. It is HIGHLY recommended that users with data processed with older versions update to version 5 to reprocess their data.

Level-0 -> Level 0.5

No change

Level-0.5 -> Level 1 (IFGM)

  • Electrical crosstalk correction has been commented out of the SPG pipelines.

Level 1 (IFGM) -> Single-sided Fourier transform

  • Subtraction of the Instrument/Telescope contribution has been moved to the spectral domain.
  • Phase correction now uses a Jiggle dependent empirically-derived Non-linear phase cal product.

Single-sided Fourier transform -> Level 1 (SPEC)

  • Correction for the contribution of the SPIRE instrument. A new processing step has been added to remove from the measured spectrum the contribution from the SPIRE instrument.
  • Correction for the contribution of the Herschel Telescope. A new processing step has been added to remove from the measured spectrum the contribution from the Herschel Telescope.
  • Optical crosstalk correction has been commented out of the SPG pipelines.

Level 1 (SPEC) -> Level 2

  1. SOF1 Sparse spatial sampling mode
    • A minor change in that the point source calibration product is now tailored to spectral resolution and apodization.
  2. Mapping Modes
    • Added NaiveProjectionTask, a new task for creating spectral simple cubes from preprocessed spectral data. This task creates spectral cubes by averaging the flux of samples within each pixel. Pixels without samples are given a value of NaN.

Interactive Analysis and Tools

  • Added the new DTMosaic class that allows the user to plot data recorded by many detectors during one observation on a single page to compare data more effectively (mosaic plot). This features is available from HIPE command line or within DetectorTimelineExplorer
  • DetectorTimelineExplorer and DetectorTimelineExplorerComponent :
    • Added a new features that allows the creation of mosaic plot of the detectors via right-click on the desired array
    • Added a new Color&Mask Preferences panel
    • Improved the color scale (using the Color&Mask Preferences it is now possible to select the Min/Max value for the color scale)
    • Improved the mask visualization (using the Color&Mask Preferences it's now possible to visualize any specific mask value directly on the array display)
  • Made available the Mosaic class which creates a mosaic of plots in the shape of the SLW, SSW or coaligned detectors of the SPIRE FTS. The Mosaic class is intended to provide a means for viewing (at a glance) the spectra or interferograms associated with each detector.
  • Changed SpecExplorer's detector display so that it automatically re-sizes to fit the enclosing frame. Also, SpecExplorer's detectors are now enclosed in a splitter pane. This enables you to maximize or hide the detector display with a single click.

Documentation

Other

Edit | Attach | Watch | Print version | History: r66 | r42 < r41 < r40 < r39 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r40 - 2010-09-30 - TrevorFulton
 
This site is powered by the TWiki collaboration platform Powered by Perl