TWiki> Public Web>HipeWhatsNew>HipeWhatsNew10x (revision 34)EditAttach

What's new in HIPE 10

PDF Version Click here

Help Interested in what's new for other versions of HIPE? See this page for links to all the What's New documents from HIPE 4.0 onwards.

See also the HIPE known issues and the Data products known issues pages for last-minute information on known problems.

Highlights

HIPE

  • Optimised the registration and initialisation of tasks. This can reduce HIPE start-up time by up to 20%.
  • Displaying images uses less memory and is much faster.
  • Data fitting is now more robust in the presence of heavy outliers.
  • You can now edit table metadata directly in the GUI. Use the Add parameter, Edit parameter and Remove parameter entries from the table's popup menu.
  • You can now send the results of a query in the HSA User Interface directly to HIPE as a table dataset.

HIFI

  • Introduction of a pipeline task to allow comparison of emission in DBS chop positions, !mkDbsReference task. It differences the chop positions in all DBS observations and after applying the band-pass correction stores them in a product in calibration->pipeline-out called ReferenceSpectra.
  • Major updates to flagTool
  • Rotated cubes are now produced in the Level 2.5 product by default for maps carried out with a non-zero position angle, a non-rotated cube is also produced.

PACS

SPIRE

  • New zeroPointCorrection task for cross-calibration with Planck HFI maps producing extended emission maps.
  • Automatic correction to SSO frame for moving objects
  • Improved photometry consistency across all Source Extraction and Photometry tasks

Java version

  • HIPE 10.0 was built and tested against the Oracle Java Runtime Environment (JRE) version 1.6.0_37.
  • User releases for Windows and Linux already include the correct Java version.
  • Developer releases and user releases for Mac do not include Java. You must ensure that you have a compatible version of Java installed on your system.
  • If the minor version of your installed Java is different from the recommended one (for instance, 1.6.0_35 instead of 1.6.0_37) HIPE prints a warning in the command line window from which it was started. Minor version differences are unlikely to cause problems, but this cannot be guaranteed.
  • If the major version of your installed Java is different (for instance, 1.5.0 instead of 1.6.0) HIPE shows a warning in a dialogue window at startup. Major version differences are very likely to cause problems. We do not recommend that you try to run HIPE 10.0 with a Java version other than 1.6.0. In particular, HIPE is known to give incorrect results if used under Java 7 with a locale different from en_US.

Core system

HIPE

Tables in viewers

  • You can edit metadata directly in the GUI. It can be done by means of the options Add parameter, Edit parameter and Remove parameter in the table's popup menu.
  • You can edit cells in table dataset viewers by selecting Allow editing in the popup menu, then double clicking on the target cells and writing the new value. A corresponding Jython command will be echoed.
  • Integer numbers in tables are left justified, as any other value. To easily compare them, an option to align them at the right is available in the table's popup menu.

Variables view

  • The variables panel is now split in two: observations at the top, other variables at the bottom. The old single panel can still be shown by pressing the button with tip Show single panel within the view's toolbar. Your choice is remembered across HIPE sessions.

File and directory selection

  • The open and save dialogs include user defined areas in the drop-down list labeled Look In or Save In.
  • When selecting a directory with a file chooser, files are also shown, although greyed out to indicate that they cannot be selected.

Miscellanea

  • It is possible to undo-redo in the console and in text fields for task parameters, by means of Ctrl-Z / Ctrl-Y (Linux & Windows) or Cmd-Z / Cmd-Y (OS X).
  • The maximum memory value shown in the memory bar reflects more accurately the actual memory available to HIPE. This value can vary slightly during a HIPE session.
  • The Locale is established as English so that the decimal separator will be a dot even in computers with language settings that use a comma for it. This ensures proper processing of tasks relying on it being a dot.

Plotting

TablePlotter

OverPlotter

Astronomical utilities

  • New package herschel.ia.toolbox.astro.healpix (based on JPL Healpix Java library).
  • Healpix interpolation functionality
  • radialV task: removed corrections parameter (enum), replaced with correct (String).

Numeric routines

Data fitting

  • Increased robustness in the case of fitting in the presence of heavy outliers.

Data interpolation

Random numbers

Statistics

Trend analysis

Mapping

Other functions

Source extraction

Images

Display

  • Image Display uses less memory and is a lot faster. It is now possible to display large image without the need for an enormous amount of memory.
  • It is possible to set the standard cut levels which should be used in the preference panel.

Analysis

Spectra

Display

  • The cube tools part of the Spectrum Explorer has now a new tool to compare N cubes: One can link them now by world or pixel coordinates, so that selecting a spaxel in one cube, selects simultaneously the same spaxel in the other linked cubes.

Analysis

Data cubes

Display

Analysis

Herschel Science Archive

It is now possible to send the results of a query in the HSA User Interface directly to HIPE. The results will appear as TableDataset, allowing one to use the geometric search in the HSA UI (for instance), and directly use the result in a script, or save it to disk.

Calibration sources

Products and datasets

Observation context

Product Access

The "Save Product to Pool" view used to require that Enter was pressed after entering a tag to use when saving the product. This is now longer necessary. User friendliness of the view is also improved by providing direct user feedback when a tag is entered that is not valid, by coloring the cell red as you type.

Product Browser

Usability improvements. The selection of which pools to query is now more user-friendly, and right-click menus for certain actions on pools have been implemented. New pools can be created directly from the Product Browser by right-clicking, and local pools can be moved, renamed, deleted, etc.

Tasks

  • Speed-up of the registration and initialization of tasks. Tasks no longer create a default GUI (in Views). If needed, this GUI will be created on demand. This can reduce HIPE start-up time by up to 20%.
  • Task Preferences
    • Preference to detect data retention in task parameters: when you reclaim memory via the "dustbin" button, tasks are checked to see if their parameters have data. You can then select which tasks to reset.
  • The system will now issue a warning if a task has apparently completed to 100% and more than 3 seconds pass before it has really completed execution (it detects tasks that "lie" about their completion status).
  • Task GUIs Help button will fall-back to on-line documentation if local documentation is not available.
  • Improved (updated) the information provided from tasks in the console (deprecated features and errors in tasks)
  • pause & resume are no longer tasks but just plain functions. They are documented in the Scripting Guide (they were in the User's Reference Manual before).
  • open* (Task, View, File) tasks open a confirmation dialogue if they fail to execute.
  • You can ask for a list with all outputs when you call a task passing __list__=True as an argument (in beta, may be removed, experimental): out, out2, out3 = taskName(arg1, ..., argN, __list__=True)
  • For Jython task developers: the following wrappers (decorators) have been added to support working with the EDT: execLater, execAndWait (similar to the SwingUtils functions). An example is provided in the source of ia_task tests (TestTaskForViewsEdt.py)

Quality control

Systematic product generation

Data input-output

  • poolDataReader , poolDataWriter, loadObs, downObs and localStoreWriter removed : saveProduct and MyHSA (getObservation) contain a superset of their features.
  • Tables
    • asciiTableReader task improvements:
      • Documentation: More examples, better tooltips and manual page (URM).
      • added more values to tableType : IPAC, SExtractor. SPACES now also reads TopCat ASCII Format.
      • Better support for table header reading.
    • New task sortTable will return a new table where rows have been sorted according to user criteria.
  • FITS
    • fitsReader and simpleFitsReader 's type parameters can now accept strings (not just Java enum values).
    • Import View will check and ask for confirmation if the expected memory needs exceed the available memory.

Virtual Observatory

  • Improved stability when sending data (on some platforms, sending data from within observations could cause problems, due to quotes being included in the name of an intermediate file).
  • Fix issues with WebStart applications (such as Aladin, HSA User Interface): Problems starting the applications on Windows XP, problems with automatic log-in.

Installer

Documentation

HIFI

HIFI Software Configuration and Properties

HIFI Calibration

  • Updates to table of unusable data frames
  • Quality flags are now raised for monitoring SEUs (Sudden Event Upsets)

HIFI Pipeline

hifiPipeline task

  • The SPG version is now updated in the meta data when you run the pipeline
  • The pipeline will only try to run for spectrometers for which there is data at level 0 (previously the pipeline always tried to run for all spectrometers)
  • The option to customise the pipeline from the GUI is disabled by default when starting processing from Level 2
  • Mapping browse products changed so that the average spectrum and integrated cubes are shown for each subband separately

Level 0 Pipeline

  • The DoBadLo step (a step to flag out datasets known to be bad or unnecessary repeats) is reprocessed when a new calibration is passed to the pipeline.

WBS Pipeline

  • A new quality flag is introduced to warn of a single COMB failure
  • Improved checks and threshold values for the ZERO spectra.

HRS Pipeline

Level 1 Pipeline

  • Addition of a task to allow comparison of emission in DBS chop positions, mkDbsReference task. Differences the chop positions in all DBS observations and after applying the band-pass correction stores them in a product in calibration->pipeline-out called ReferenceSpectra. You may configure the task to average the differenced spectra together (a resampling is automatically done) and to resample the spectra yourself.
  • Additional parameters available to DoFilterLoads

Level 2 Pipeline

  • DoAvgTask now propagates ra_center and dec_center to metadata
  • A new task, MkFLagSummaryTable, creates a table containing all the flags raised by the pipeline for quick inspection

Level 2.5 Pipeline

  • Rotated cubes are now produced by default for maps carried out with a non-zero position angle, a non-rotated cube is also produced.
  • Flagged data are omitted from spectral cubes
  • Added a quality flag that is raised when doDeconvolution cannot complete in the Level 2.5 pipeline

HIFI Products

  • HifiUplinkProduct format changed to a table dataset (easier to read)

HIFI Spectrum Toolbox

Spectrum Explorer (HIFI specifics)

Spectrum tasks (HIFI specifics)

  • A new task, doOffset, to calculate the offset positions from the requested position is introduced. You can use it to switch between offset and absolute coordinates.
  • DoFold task now doubles in the reported integration times in the metadata after folding.

HIFI Data Processing Tools

Standing Wave Removal

  • fitHifiFringe can be directly applied to spectra in HIFI mapping mode cubes through new input option 'cube'.

Baseline Removal

  • Changed default for 'movemasks' option from 2 to 0 in fitBaseline, i.e., masks are not propagated between spectra.
  • Improved performance stop button in non-interactive mode in fitBaseline

Deconvolution

  • Added a 'BRIGHT_LINE' toggle. If enabled, doDeconvolution will ignore data set with this channel flag. Useful for preventing `ghosts' of bright lines caused by the iterative nature of the deconvolution algorithm. You need to manually mask these lines using, for example, flagTool.
  • Plotting output has been cleaned up and simplified.
  • doDeconvolution now gracefully exits whenever it fails to process an observation (occurs only for the most troublesome of data and is rare).

Spectral Cube creation

  • The GUI will by default pick up the rotation angle (flyAngle in the GUI) used when the map was observed. When the pipeline is run with the interactive level 2.5 pipeline selected or modified in any way then only this angle will be used, when the pipeline is run with all the default settings then non-rotated and rotated cubes will be produced. In the case that the flyAngle is zero only one set of cubes will be created.

flagTool

  • Major updates to make the tool faster, more intuitive and easier to use:
    • All data are displayed in a table to the left of the plot, enabling you to quickly select the datasets you want to flag without needing to cycle through all the data if not required.
    • All subbands can be displayed together (without gluing).
    • Flag type can be selected from the options above the plot instead of from a context menu

HiClass export tool

Documentation

  • Updates to the HIFI Data Reduction Guide:
    • Addition of a chapter describing how to convert positions in data to offsets
    • Improvements and updates to "How to add and remove flags" chapter, including documentation of new flagTool in the "Setting flags interactively for many spectra" section
    • Improvements and updates to "Running the HIFI pipeline" chapter
    • Update to mapping browse product description in "Tour of a HIFI observation context" chapter
    • Removal of references to the old CubeSpectrumAnalysisToolbox and replacement with references to the new Cube Toolbox
    • Various typos, improvements to style consistency and bug fixes.

PACS

Pipeline

Photometry

  • There is a a new task photCoverage2Noise, run in the SPG pipeline to derive an error map from the coverage map, including the correction for correlated noise.

Spectroscopy

  • There is a new task pacsExtractSpectralRange to spectrally crop a (Sliced)PacsCube or (Sliced)PacsRebinnedCube.
  • The footprint of pointed observations is shown by projecting the rebinned cubes on a finer grid (0.5 arcsec pixelsize).
  • There is a new mask OUTOFBAND which masks the data at wavelengths where the signal is completely blocked by the used filter.
  • Slices which are completely masked by this OUTOFBAND mask are discarded during the level-0.5 processing.
  • Spectra are now by default rebinned to a standard wavelength grid, so that we avoid incompatible wavelength grid when subtracting on and off spectra from different observations.
  • New useful script for point source correction.
  • Spectral flatfielding: several issues had been reported for the specific case of zero or weak continuum in specFlatFieldLine and specFlatFieldRange (e.g. non-central spaxels). These were fixed.
  • One can now slice the level-0 data with pacsSpecSliceLevel0 to reduce the memory consumption of the processing till level-0.5.
  • Most of the ipipe scripts have been slightly simplified. Mainly, selectSlices has been removed: all tasks can cope with sliced products transparently, so selectSlices is optional.
  • Spectral drizzling is now offered for the line-scan observations (ChopNodLineScan, UnchoppedLineScan). Narrow wavelength ranges can be extracted from range-scans and fed in the drizzling as well. For this, use the new crop tasks described above
  • Fixed some wrong values of the 'onOffSource' meta keyword.
  • Fixed some inconsistencies in the units and labels of the pipeline products.
  • Added the missing keyword 'faintLines' to the observation context.

Calibration

Photometry

  • There is no change in the PACS photometer calibration since Hipe 9 and version 9 of the calibration tree.

Spectroscopy

  • In order to be less sensitive to (on average) mispointed flux calibration observations, the flux calibration scheme has changed: the construction of the calibration tables is now based on the standard calibrators fluxes coming from the central 9 spaxels instead of the central one only. This implies that two new calibration tables were introduced in the calibration tree, namely observedResponse3x3 and calSourceFlux3x3. Those tables are used by default in specDiffCs and should lead to more reliable fluxes, notably in case of mispointed observations. An optional parameter, use3x3 (True by default), was added to specDiffCs. You can still use the former calibration scheme by setting use3x3 to False although it is not recommended.
  • As a result of the new calibration scheme, the absolute fluxes found at level 2 are slightly reduced.
    • For extended sources, the new calibration is more accurate
    • The new calibration should not impact point sources, thanks to the so-called "central to 3x3" correction performed in extractCentralSpectrum. That correction was not necessary in hipe 9.0, as it was implicitly contained in the flux calibration files. In hipe 10, by default no average mis-pointing is included in the calibration files anymore, hence this extra correction, making use of the flux distribution between the spaxels to estimate the mispointing and correct for it.

Analysis tools

Photometry

Spectroscopy

Product changes

Photometry

  • From SPG10 onwards, level 2 maps are processed with a very small high-pass filter width to filter as much the 1/f noise and get optimal sensitivity to point-sources. Half-widths of 15 and 25 readouts are used in the blue and red channel respectively. As a consequence extended emission at virtually all scales is filtered out. The level 2.5 maps processed with MADmap should be used to get extended emission, which is preserved at all scales in the MADmap maps. Note that SPG10 bulk pipeline processing will only start in early 2013.

  • Error maps in SPG10 level2 maps are produced by the task photCoverage2Noise, where error maps are derived from the coverage map and include cross-correlation noise.

Spectroscopy

Documentation

SPIRE

Spire Dataset

  • New =DestriperDiagnosticProduct is added.

Calibration Products

Calibration Framework

Preprocessing

  • The Pcal products now have correctly filled in metadata.

Engineering Conversion

Common Pipeline

  • SigmaKappaDeglitcherTask
    • Added a diagnosticDetector task parameter for diagnostic purpose
    • All the long task parameters are shortened
    • The parameter types for detectGlitches, detectLargeGl and repairGlitches parameters are changed to Boolean.

Photometer Pipeline

  • New zeroPointCorrection task for cross-calibration with Planck HFI maps. This can only be run at the HSC until the HFI maps are made public. The output maps are included in the Level-2 context.
  • New ssoMotionCorrection task to produce modified Level-1 timelines adjusted to the reference frame of the moving object. Additional maps are created from this timeline and included in the Level-2 context.

  • DestriperTask
    • Added the minVel, maxVel, chanNoise and method task parameters to be in synch with naive mapper.
    • Added useOnlyMasks and ignoreDefaultMasks task parameters in synch with map mapper.
    • Added jumpThresh and jumpIter task parameters for jump detections.
    • Two second level deglitch algorithms are added, sigma-kappa and two sigmas
    • The startParameters type changed to DestriperDiagnosticProduct.
    • The meta data in startParameters has been changed to include all requirement meta keywords.

  • SpireMaskEditor
    • SpireMaskEditor is deleted
    • Level1SpireMaskEditor is added to display/edit mask flags for Level1 data
    • Level0_5SpireMaskEditor is added to display/edit mask flags for Level0_5 data

  • baselineRemovalPolynomial
    • "outlierRejectionThreshold" parameter renamed to "outlierThreshold"

Spectrometer Pipeline

  • History is now propagated from level-1 products to level-2 products when spirePreprocessCube and spireProjection are used in the spectrometer pipeline.

Source extraction

  • sourceExtractorTimeline
    • Significant speed improvements
    • Parameters renamed: "fitEllipticalGaussian" renamed to "fitEllipticalGauss2d" and "modelGauss2DFixedWidth" renamed to "modelGauss2dSigma".

Interactive Analysis and Tools

  • SpectrometerDetectorSpectrum (SDS) products have more functionality
    • Can be opened and used in the Spectrum Explorer
    • More tasks available
    • Arithmetic operations (+, -, *, /) work in scripts, when performing calculations between an SDS product and either another SDS product, a spectrum, or a number
  • Observation Viewer:
    • More information displayed in the viewer for level 0 and level 0.5 products
  • SDS Explorer
    • N-E arrows added to show orientation of the arrays
  • convolutionMapper
    • New task to create a kernel-filtered map directly from timeline data

Documentation


blog comments powered by Disqus

Edit | Attach | Watch | Print version | History: r70 | r36 < r35 < r34 < r33 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r34 - 2012-12-10 - BrunoAltieri
 
This site is powered by the TWiki collaboration platform Powered by Perl