TWiki> Public Web>HipeWhatsNew>HipeWhatsNew9x (revision 21)EditAttach

What's new in HIPE 9

PDF Version Click here

See also the HIPE known issues and the Data products known issues pages for last-minute information on known problems.

Highlights

HIPE

  • Common look and feel for the spectrum explorer and cube analysis toolbox
  • Easing the data access via MyHSA
  • Improvement of handling of ASCII tables
  • Mechanism to notify users on removed names (tasks, functions...), why they were removed, and their replacement
  • Smaller memory footprint for HIPE editor
  • Speed-up of HIPE start-up
  • Progress bar informs on progress of for-loops
  • Human-made indexes added to the core manual (HIPE Owner's Guide, Scripting Guide, Data Analysis Guide)
  • Possibility to leave comments at the bottom of any page in the HIPE Help System
  • Support to OSX Lion
  • Easy access to calibration data, including version and updates (TBS)
  • Improvements of search capability of documentation system (TBS)
  • Publication ready plots (TBS)

HIFI

  • HIFI level 2.5 product introduced: for Spectral Scans it holds the deconvolved single sideband soluions; for Maps it holds the cubes; for all cases it holds a copy of the level 2 HTP.
  • Addition of an interactive pipeline between level 2 and 2.5, which allows the insertion of tasks like fitHifiFringe and fitBaseline into the pipeline between level 2 and 2.5.
  • Browse products for spectral scans and maps added

PACS

  • Photometer: Addressing further specific cases in IPipe scripts
  • Spectrometer
    • Broadband reliability
    • Improved spectral pixel flatfield broad ranges (SED)
    • Correction Flux loss due to pointing
  • Convolution kernels for photometer and spectrometer maps aiding the interpretation of extended and point sources
  • Reduction in memory consumption (TBS)

SPIRE

  • common (both Photometer and Spectrometer)
    • improved processing speed using multi-threading
    • improved task for correction of "cooler burps"
  • Photometer
    • Pipeline Processing
      • kernel-smoothed maps (aka "drizzling")
      • improved error maps (possibly as an option)
      • improved baseline correction using "destriper“
    • Interactive processing
      • source timeline fitter task
      • improvements to bolometer finder tool
      • selection of turnaround data in map creation
  • Spectrometer
    • Pipeline Processing: smaller size for level 1 products
    • Improved calibration and data processing
      • updated point source calibration (improved Uranus model)
      • updated bright mode calibration (better PCAL module)
      • improved calibration for Low Resolution observations (hopefully improved calibration, but at minimum, updated advice)
      • more straightforward calling sequence for phase-correction task
      • corrected spectral averaging - more flexible usage
    • Interactive processing
      • drag and drop from SDS Explorer detector layout to variables tab
      • new and updated useful scripts for background subtraction and spectral noise

Core system

HIPE

Plotting

TablePlotter

  • TablePlotter now supports the axis.Date functionality.

OverPlotter

  • OverPlotter now supports the axis.Date functionality.

Dataset viewer

Astronomical utilities

Numeric routines

Data fitting

Data interpolation

Random numbers

Statistics

Trend analysis

Mapping

Other functions

Source extraction

Images

Display

  • Reworked the save methods on Display. You can now save to jpg, png, eps or pdf files. The methods to use are :
    • save(filename), saveAsEPS(filename), saveAsPNG(filename), saveAsJPG(filename) and saveAsPDF(filename) : These methods will save the full image, including annotations.
    • saveCurrentView(filename), saveCurrentViewAsEPS(filename), saveCurrentViewAsPNG(filename), saveCurrentViewAsJPG(filename) and saveCurrentViewAsPDF(filename) : These methods will save the current view of the image, including annotations and axes.
  • The coordinate axes on the image panner are always visible now
  • There is a preference for image display to turn of the echoing of commands to the command line. It is possible to turn all these commands off, or only the zoom commands.
  • Flagged out pixels and pixels with a value of NaN will be shown in the color of the background.
  • Image can be matched with the Wcs. This means that the images can be oriented so that north is shown at the top and east to the left.
  • There is a preference for image display to always show the image with the 'match wcs' orientation.
  • A lot of bugfixes.

Analysis

Spectra

Display

Analysis

  • The Smooth task is corrected for a bug in the ZEROS option.
    • As a consequence of this fix, which is more mathematically correct, it is found that smoothed spectra for SPIRE are quite changed. It is recommended instead to use the REPEAT option.

Data cubes

Display

Analysis

Herschel Science Archive

  • Improvements in download speed and memory requirements for data downloaded from HSA using a behind-the-scenes utility called MyHSA
    • Testing with 5 HIFI, 9 PACS and 10 SPIRE representative observations of all major observing modes simultaneously with getObservation/saveObservation and with getObs/MyHSA with the same network connection on two different machines show 20 times faster download speed and that 66% less hard disk space is needed to hold the same data:
      MyHSA_downloadTimes.png MyHSA_diskUsage.png

Calibration sources

Products and datasets

Observation context

Product Access Layer

When storing pools on shared NFS-mounted disks, HIPE now uses a more robust mechanism to prevent corruption. The default locking mechanism for Local Pools was changed to a mechanism that works more reliably on remote disks.

Usability improvements for the Product Browser and for working with tags. One can now update existing tags (for example to fix a typo).

Tasks

  • Task Preferences
  • Preference to generate full task call from the task GUIs
  • Preference to check input and inout parameter modification/replacement by tasks
  • Deprecated and removed tasks (names and parameters) documented in HIPE.
  • Task Viewer:
    • Drag and Drop of the (full) signature of tasks from Task Viewer to Editor & Console
    • Reordering of nodes and reduced depth
    • Tooltips include the Categories and deprecated tasks appear crossed out
  • Task Outline:
    • Deprecated tasks & parameters marked

Quality control

Systematic product generation

Data input-output

  • Deprecation of duplicated functionalities: poolDataWriter, localStoreWriter, poolDataReader. Use getObservation and saveProduct.
  • Deprecation of MyHSA prototype duplicated functionalities: loadObs and downObs. Use getObservation
  • getObservation promoted to a Task (currently getObs).
  • simpleAsciiTableReader deprecated, use asciiTableReader.

Virtual Observatory

Installer

Documentation

HIFI

HIFI Software Configuration and Properties

HIFI Calibration

HIFI Pipeline

hifiPipeline task

  • Addition of an interactive pipeline between level 2 and 2.5, which allows the insertion of tasks like fitHifiFringe and fitBaseline into the pipeline between level 2 and 2.5.
  • Inclusion in the pipeline GUI of a drop-down box to select calibration version

Level 0 Pipeline

WBS Pipeline

  • Introduction of quality flag to warn about a bad zero.

HRS Pipeline

Level 1 Pipeline

  • Addition of parameters in the doFilterLoads to allow configuration of the filtering.

Level 2 Pipeline

  • doAvg now honours user-defined flags, RowMask.IGNORE_DATA and HifiMask.IGNORE_DATA

HIFI Products

  • Level 2.5 product introduced: for Spectral Scans it holds the deconvolved single sideband soluions; for Maps it holds the cubes; for all cases it holds a copy of the level 2 HTP.
  • Addition of browse products for spectral scans and maps. The spectral scan browse product is the deconvolved single sideband solution for H and V. The mapping browse product shows the map averaged spectra and the integrated map over the range of the WBS for the H- and V-polarisations.
  • Point mode browse product axis labels corrected.
  • Addition of a flag array to cubes, allowing pixels to be flagged in the same way as points in spectra can be.
  • HTP default view is changed from SpectrumExplorer to the ContextViewer
  • Observing band is added to HTP metadata

HIFI Spectrum Toolbox

Spectrum Explorer (HIFI specifics)

Spectrum tasks (HIFI specifics)

  • ConvertK2Jy now uses the same scheme as doGridding to select frequency when interpolating the efficiency values from the calibration tree: the task searches first for the LO Frequency in the metadata, if it is not found the task looks in the metadata for the value inserted in HSpot. If neither are found then the middle frequency of the observing band (e.g. band 1a) is used.
  • Bug fix in ConvertK2Jy: non-zero source size data on frequency scale can now be converted to Jy

HIFI Data Processing Tools

Standing Wave Removal

  • Improved handling of pipeline flags in fitHifiFringe
  • fitHifiFringe now accept SimpleSpectrum data

Baseline Removal

Deconvolution

  • When multiple observations are passed to doDeconvolution the metadata are now concatenated and put into the decon_result
  • doDeconvolution made more robust against failure when there are many bad datasets in the HTP.

Spectral Cube creation

HiClass export tool

  • Sideband gain information in the metadata are now exported to CLASS.

Documentation

  • Updates to DBS Cookbook
  • Updates to HIFI Data Reduction Guide for changes to:
    • fitBaseline
    • fitHifiFringe

PACS

Pipeline

Photometry

Spectroscopy

  • estimate of continuum RMS is included in plotting function of ipipe scripts
  • new task that extracts the central spaxel spectrum with a 3x3/central correction
  • the task specFlagGlitchFramesQTest can optionally be threaded
  • background normalization ipipe script introduced also for long range SED
  • improved flatfielding task, for line-scan mode
  • improved drizzling routines
    • available for chopped and unchopped mode
    • uses observed beam size for the calculation of the pixel scaling of the spatial grid
  • test versions of pointing correction tasks
  • improved error propagation
  • correction of SSO pointing task correctRaDec4Sso introduced in spg pipeline

Calibration

Photometry

Spectroscopy

  • new calibration product offRatio for all bands and chopper throws
  • new calibration product beams (raster beams of neptune) for all bands
  • update of the pointSourceLoss calibration product
  • new calibration product beamSize
  • new viewer for the calibration set release notes

Analysis tools

Photometry

Spectroscopy

  • refactoring of the spectrum explorer
  • improved GUI for depiction of the spectrometer footprint

Product changes

Photometry

Spectroscopy

  • complete browse product generation
  • quality flag "SLOWSAMP" added in case of low sampled spectra (level 2 products)

Documentation

  • improvements of URM
  • improvements of the PDRG

SPIRE

Calibration Products

Calibration Framework

Preprocessing

Engineering Conversion

Common Pipeline

Photometer Pipeline

  • DestriperTask
    • The "useSink" task parameter is added. The default value is False. It if it True, the ProductSink is used to save the output.
    • The "nThreads" task parameter is added. The default value is 1. Multi-threads will be used when nThreads>1.
    • The Level2 deglitching is implemented. The "l2DeglichAlgorithm" task parameter was added. The default value is sigmaKappa.
    • The "l2IterMax" task parameter is added. The default value is 5. It specifies how many iterations the Level2 deglitcher will run.
    • The "starParameters" task parameter is added. The default value is null. It can be used to de-selected some detectors or scans with known issues. It can also be used as a "check state" to restart from the previous run.
    • The "storeTod" task parameter is added. The default value is False and no tod files will be stored and returned tod is null. When it is True, the tod associated files (pixel, signal and weight) will be stored in the temporary directory.

Spectrometer Pipeline

Interactive Analysis and Tools

  • SpireMaskEditor
    • Level0_5 SpireMaskEditor is introduced. It can only be used for editing and displaying Level0_5 Context.
    • Level1 SpireMaskEditor is introduced. It can only be used for editing and displaying Level1Context.
    • SpireMaskEditorComponent is deprecated.

Documentation

Topic attachments
I Attachment History Action Size Date Who Comment
PNGpng MyHSA_diskUsage.png r1 manage 83.8 K 2012-04-04 - 21:49 CarolynMcCoey HSA disk usage
PNGpng MyHSA_downloadTimes.png r1 manage 50.2 K 2012-04-04 - 21:50 CarolynMcCoey HSA download times
Edit | Attach | Watch | Print version | History: r75 | r23 < r22 < r21 < r20 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r21 - 2012-04-17 - PierreRoyer
 
This site is powered by the TWiki collaboration platform Powered by Perl