Tags:
create new tag
view all tags

What's new in HIPE 11

PDF Version Click here

Help Interested in what's new for other versions of HIPE? See this page for links to all the What's New documents from HIPE 4.0 onwards.

See also the HIPE known issues and the Data products known issues pages for last-minute information on known problems.

Highlights

HIPE

  • New indexing for pools. This results in faster search for local products but requires rebuilding the index when going back to previous versions. More information can be found here.
  • Star tracker focal-length induced offsets of less than 5 arcsec have been corrected for data from ODs < 320.
  • Combined pipeline and aperture photometry script for PACS scanmaps of point sources.
HIFI
  • flagTool enhanced to allow better and simpler flagging of data for reprocessing.
  • Updates to CleanDF and spur tables and ghost identifications.
  • Improved performance of standing waves and baseline removal tasks.
  • doPointing now uses the proper identifier in the orbit ephemeris product for a given SSO.
PACS
  • Photometer:
    • Level 2.5 MADmap maps now contain error maps.
    • Level 3 products generated in the standard product generation (SPG) pipeline.
  • Spectrometer:
    • Flatfielding has been added for line scans only to the standard product generation (SPG) pipeline.
    • Spectrometer RSRF band R1 provides correct line fluxes in red leak region (beyond 190 microns). This RSRF is not applied by default since it increases the noise in the resulting spectrum, but can be used interactively within Hipe.
    • Update of the spectrometer telescope background model with new parameter set representing the aging model of the optical surface.
SPIRE
  • Photometer:
    • Update to flux calibration to use new Neptune model (the ESA4 model instead of the ESA2 model).
  • Spectrometer:
    • The telescope and instrument RSRFs have been updated to produce 20% lower instrument noise.
    • The beam FWHM inside the SCalSpecBeamParam product has been updated.

Core system

HIPE

Batch mode

The hipe executable can be used as a replacement of jylaunch to run scripts in batch mode.

In HIPE 11, jylaunch will still be available, although it will print a warning informing that it shall be removed in HIPE 12.

Plug-ins

  • If the latest version of a plug-in is not compatible with an older version of HIPE, when running the older version of HIPE, it will not pick up an older version of the plug-in that is compatible (if available).
  • Tasks defined in plug-ins now have their source code available from the Task GUI.

Navigator

  • Compressed files can be opened with either double click or right click and Open With.
  • The default folder that is offered in file dialogues when opening or saving files is now customizable. The behaviour can be set in the General > Directories preferences panel.

Data Access & HSA

  • There is a new Data Access Perspective, bundling all functionality needed to get data in and out of HIPE.
  • The HIPE status bar now indicates the log-in status with the HSA (logged in as...). If the connection to the HSA is lost, for example because a WiFi connection is dropped, it will be shown as well.
  • HIPE now shows the currently logged in user on the status bar, next to a lock icon. If no user is logged in, one can double-click the icon to log in. Also, if no user is logged in and log-in information is required, rather than failing, HIPE will now pop up a log-in window, allowing the user to specify username and password.

Editors & Viewers

  • Edition of product's metadata through the GUI does not allow to remove mandatory parameters (product's attributes).
  • The search dialogue for text editors is now more user-fiendly:
    • It focuses the text pane once opened, so the user doesn't need to click on it.
    • It shows an info label saying how to trigger the search with the keyboard: Ctrl+Enter.
    • It does not lose the focus, so that multiple entries can be searched with the keyboard one after the other.
    • Undo/redo is available in the search text pane, by means of the usual key combinations.

Console

Commands in the history are indexed so that the first executed command in the current session has index 0. Commands in previous sessions have negative indices.

As previously, the complete command history can be obtained by running !history, and a specific command can be recovered by its index through !<index>.

Plotting

  • A new axis type TAI
  • Bug fixes

TablePlotter

OverPlotter

Astronomical utilities

Numeric routines

Data fitting

Data interpolation

Random numbers

Statistics

Trend analysis

Mapping

Other functions

Source extraction

  • sourceExtractorSimultaneous now has a region of interest (roi) parameter

Images

Display

  • Speed improvements.
  • A lot of bug fixes.
  • Getting the distance between two points can be done using left mouse click, combined with SHIFT (and CONTROL).
  • Contour drag and drop also shows the color settings on the console.
  • photApertureCorrectionPointSource can be drawn on an Image.
  • Coordinates can be shown as decimal degrees.
  • It is possible to show extra datasets added to the image or the cube in Display.
  • It is possible to select which flags should be overlayed on the display.
  • Match Wcs has improved.

Analysis

  • Bug fixes in Wcs handling of SimpleImage and SimpleCube.
  • Support of extensions of Wcs.
  • Exporting and importing a session does no longer fail when there are ImageContours.
  • herschel.ia.dataset.image is thoroughly tested.
  • Mosaic task can convert to equatorial, ecliptic or galactic coordinates (when the input wcs is different than the wcs of the input images).
  • Improvements in reading fits files.
  • It is possible to open the the image analysis gui's without selecting an image.
  • Corrections in the SkyAperturePhotometry tasks.
  • Mosaic task can work with all sort of Lists.
  • Centroiding in aperture photometry tools should work more robust.

Spectra

Display

  • When selecting plots/axes during an Align or Lock operation (by holding the Control key pressed, and clicking with the mouse), selected plots/axes are marked with blue lines.
  • New options Save and Print are available in the popup menu of the mosaic panel.
  • Icons of the toolboxes buttons pretend now to be more obvious.
  • Number of different colours to plot spectra increased from 7 to 10. Consecutive colours are now more distinguishable.
  • When a spectrum task is going to be executed on a product belonging to a context tree, user confirmation is required.

Analysis

  • SpectrumFitterGUI has undergone some changes such as a new layout which includes a revamp of the "export" panel.
  • A new Sinc-Gauss profile is available for use with the fitter. This is needed for extragalactic sources which may have partially resolved spectral lines.

Data cubes

Display

Analysis

  • Added rectangular and line selection to extractRegionSpectrum
  • Several improvements in computePVmap

Calibration sources

Products and datasets

Observation context

Product Access Layer

  • The format of a PAL browser result table is now automatically persisted over multiple HIPE sessions.
  • The underlying Apache Lucene library is upgraded from 3.0 to 4.1. Local stores generated in earlier version need to run rebuild index to be accessed.

Tasks

Users

  • Warning, important Jython code with calls to tasks with multiple outputs must be updated: see User manual.
  • JTask deprecated, use Task instead (just remove the J).
  • Warning message when a task shows 100% in the progress bar and stays there for a long time (bad progress reporting of the task).

Developers

  • You can link to external help by providing your own URL String with documentation implementing herschel.ia.gui.kernel.parts.ExternalHelp.
  • You can now mark parameters as not shown in the GUI (see TaskParameter.setHasGui).
  • Framework now provides also a layout with just one column (one parameter per row) (see JTaskSignatureComponentNarrow and Wiki ).
  • Added more methods and classes to Task GUIs to ease alternative implementations and customisations.
  • Tasks can veto applicability (ex. a task whose prime input is a String but has a validator should not be applicable ever), see Task.canBeApplicable.
  • Interrupted tasks now throw CancellationException (was plain RuntimeException).
  • Task parameters can now be of type Class.
  • Renamed Task.getHistory() and Task.setHistory() (to Task.isHistoryRecordingEnabled() and Task.setHistoryRecordingEnabled() respectively).
  • Added methods to NameSpaceTest to test deprecation and removal of task parameters.

Quality control

Systematic product generation

Pointing Product

  • Solar Aspect Angle was added to the Pointing Product
  • A focal length correction script is applied to the pointing product for all ODs up to 866 (excluding OD 858)

Data input-output

  • Import View and importObservation are deprecated : use MyHSA.
  • save and restore : the file extension is now ".ser", they can be opened by double clicking in the Navigator view.
  • localStoreCopier : parameters have been renamed, "warn" default value is now True.
  • simpleFitsWriter : The default extension when one selects compression with GZIP, is now ".gz" (it was ".gzip").
  • fitsReader & import tasks: Truncated (missing padding) FITS files can now be read by the system.
  • asciiTableReader : Warning, important Table I/O is being simplified: you will have to change your calls just once
    • Parameters renamed, descriptions improved, info log messages with configuration.
    • columnType (old name: parserGuess) now supports (and prints) also String values (besides integers).
    • tableType: improved support for the different types of tables.
    • skipRows (old name:parserSkip): is now a "basic" parameter (can be used also with tableType <> ADVANCED).
  • asciiTableWriter : Warning, important Table I/O is being simplified: you will have to change your calls just once
    • Parameters renamed, descriptions improved
  • convertUnits: Converts numeric data expressed in one unit, in terms of another unit.
  • convertAngles : Converts angle array data between string and numeric representations.
  • decompress : When double clicking compressed files in the Navigator the task will be opened.

Virtual Observatory

HIPE now supports sending Spectrum2d objects. When sending a 2-dimensional spectrum, HIPE will send all the plain spectra in sequence.

Installer

Documentation

  • Documentation viewer is loaded significantly faster.
  • Search indexes are created at build time (faster start up of local help system)
  • Search provides more accurate results, new auto completion algorithm

HIFI

HIFI Software Configuration and Properties

HIFI Calibration

  • Updates to CleanDF tables
  • Update of spur table after 3b purification at higher end
  • Ghost frames, where previously identified in specific observations, are now removed automatically in the HIFI pipeline

HIFI Pipeline

hifiPipeline task

  • The option to choose a different CAL version via the hifiPipelineGUI is now fixed.

Level 0 Pipeline

  • Ephemeris positions of SSOs (ra_centre/dec_centre) were off by up to ~10" due to a bug; this is fixed in HIPE 11.

WBS Pipeline

HRS Pipeline

Level 1 Pipeline

  • doFold task now allow for overwrite=True when handling frequency switch observations

Level 2 Pipeline

Level 2.5 Pipeline

HIFI Products

HIFI Spectrum Toolbox

Spectrum Explorer (HIFI specifics)

Spectrum tasks (HIFI specifics)

HIFI Data Processing Tools

Standing Wave Removal

  • New parameter 'resolution' used internally to speed up the fitHifiFringe task. The default is 10 MHz and should be reduced for processing of spectra with standing wave periods less than about 30 MHz (non-HIFI spectra).
  • Defaults for input parameters fitHifiFringe (nfringes, typical_period, start_period, and end_period) are automatically set to optimal values, matching standing waves in each HIFI band.

Baseline Removal

  • New fitBaseline parameter 'useWbs' allows subtraction of WBS polynomials from HRS spectra.
  • fitBaseline reacts much better to HIPE 'stop' button.
  • New fitBaseline parameter 'plot' allows switching plotting in non-interactive mode off or on. The default mode is False, which speeds up the task and greatly reduces memory consumption.
  • fitBaseline domask=2 option (automated masking) now works in non-interactive mode as well, and can be interrupted using the HIPE 'stop' button.
  • On the command line, the output is retrieved using out=fitBaseline(..) for all input types.

Deconvolution

  • Improvements to the display of SScan browse products.
  • Improvements to the treatment of weights within deconvolution.
  • The deconvolution task can now be interrupted using the red 'stop' button in HIPE.

Spectral Cube creation

flagTool

  • Major change

  • Improvements of the global behaviour:
    • Each subband has a given colour
    • Annotations are on the left-hand side of the plot
    • Better handle of compound flag names
    • Scan the data to find flags and create the corresponding masks if necessary while launching flagTool
    • Improve display of the information when scanning the htp and the maskTable
    • The selection of a subband can be done with left-click in the interactive table

  • Improvements of the GUI:
    • Use parameter level in flagTool
    • Set dropdown menus for selecting values of some parameters
    • Rowflag is selected by choosing "category=rowflag"
    • Both rowflag and channel flags can be selected by choosing "category=both"

  • Bug fixes:
    • Crashes when deleting existing masks
    • Doesn't flag in response to mouse clicks
    • Adding a marker removes the previous one
    • Crashes because it uses a deleted method for converting the colours of the flags
    • Fix annotation issues
    • Linemasks file not showing all masks after the first flagTool run
    • FlagTool should apply FitBaseline and FitHifiFringe only on the selected dataset row

HiClass export tool

  • Improvements of the global behaviour
    • A default filename is provided if the user forgets it
    • Append ".fits" to filename if the user forgets it

  • Improvements of the GUI
    • Add "verbose" checkbox

  • Bug fixes
    • HiClassTask accepts input and then complains
    • Inverted velocity axis when exporting from HIPE to CLASS

Documentation

  • Updates to the HIFI Data Reduction Guide:
    • Addition of the Mapping Mode Cookbook
    • Improvements and updates to "How to add and remove flags" chapter: flagTool in the "Setting flags interactively for many spectra" section
    • Addition of the pipeline level 1 mkDbsRefTask definition
    • Improvements and updates to pipeline level 0 doPointing section and examples
    • Updates to HIFI Tips of the Day
    • Regular updates to the Quality Flags Table in the "Flags in HIFI data" chapter
    • Updates to "HIFI Baseline Removal" chapter, "FitBaseline Options" section
    • Updates to "Standing Wave Removal" chapter
    • Updates to "Exporting HIFI data to CLASS" chapter
    • Corrected typos and bad/missing links

PACS

Pipeline

  • The PACS spectrometer pipeline has now been fully threaded.
    • This gives an approximately 20% improvement in processing time over the previous version.

  • Improvements in the multi-threaded running of tasks on slices.
    • This can now be turned on/off without needing to restart HIPE.
    • Stability improved: all tasks have been checked for thread safety.
    • Improved memory handling.

Photometry

  • Error estimate for Level 2.5 MADmap maps. MADmap maps now include an error estimate per map pixel. The error is computed as for the photProject pipeline, but is then modified to reflect the same relative sensitivity between the High-pass filter branch and extended emission branch as determined for HSpot. For the naivemap error is the formal error of the mean. This is unchanged. The MADmap error is formal and likely optimistic than reality in that user diligence in preprocessing PACS pipelines and subsequent execution of MADmap pipeline with proper parameters is not accounted for. These could introduce systematical errors in the maps that are significantly larger than formal errors. We will try to account for such systematics in subsequent versions of the error analysis code for HIPE 12.
  • Masked out matrix 9 after OD 1375.
  • Level 3 processing.
  • Deglitching improvements.
  • Many improvements to scanamorphos.
  • Better treatment of removal of calibration blocks.
  • Improved noise computations.
  • Improved memory handling.

Spectroscopy

  • Simplified fitting cubes and support projecting fitted parameters by using the new (Pacs)ParameterCube.
    • Introduced a new script for fitting and then projecting.
  • Tasks specRespCalToTelescope and specRespCalToBackground are now merged.
  • Improved extracting spectra from a cube.
  • Improved the range scan flatfielding, including better diagnostic plots and avoiding noise with new RSRF.
  • Flatfielding has been added for line scans only to the standard product generation (SPG) pipeline.
    • The level-2 product will now contain flatfielded PACS cubes.
  • Improvements to the script to combine PACS and SPIRE spectra.
  • Improved the pseudo WCS coordinates of PACS (rebinned) cubes, which contain just the row/column coordinates of their RA and Dec cubes.
  • Improved the background subtraction schemes in the unchopped pipelines.
  • Error propagation added to specProject.

Calibration

Photometry

  • Improved cooler recycling times calibration file.

Spectroscopy

  • Improved telescope background calibration file.
  • New calibration files with high resolution beams per spaxel.
  • Improved telescope emissivity model calibration file.
  • Relative spectral response updates to cover the edges of the spectral bands
  • Relative spectral response function for the red leak region (for use in hipe)

Analysis tools

  • Extension to tree viewers now available for all PACS sliced products.
    This means that also the outline viewer shows information about the slice content.

Photometry

Spectroscopy

  • Display shows now properly the special pseudo WCS of PACS rebinned (5x5) cubes.
  • The PACS extensions of the Spectrum Explorer have been improved:
    • Orientation of spaxel selector in Frames panel now represents the real orientation of the cube spaxels.
    • Excluding NaNs when creating the average image for the cube selection panel.
    • Fixed some problems when multiple products are loaded in the SE.

Product changes

  • An observation summary has been added to the browse product of an observation context (currently only available for the spectrometer).
  • Amount of details in the product history reduced: E.g., it no longer goes back to where a calibration file has been read by its creator.
  • Meta data sliceNumber added, which is used to add a slice number to the FITS filenames.

Photometry

Spectroscopy

  • A special product for parameter cubes: PacsParameterCube extends ParameterCube.
  • New sliced products have been added all SimpleCube derived products.
    The sliced framework utilities are now for all spectral cubes available.
  • Fixed some unit labels in various products.
  • Fixed orientation of browse image (was upside down).
  • Added lineId checking and updating when adding a slice to a spectrometer sliced product, so that the lineId will never conflict with existing ranges.

Documentation

  • Improved MadMap documentation.
  • Improved use cases and options description for interactive pipeline scripts.
  • Improved explanation of wavelength grid parameters. * Flatfielding has been added for line scans only to the standard product generation (SPG) pipeline.

  • New section on spectrometer drizzling.
  • Better documentation on how to fit cubes and then project them.
  • Many improvements to the data reduction guides.

SPIRE

Calibration Products

Photometer

  • Update to flux calibration to use new Neptune model (the ESA4 model instead of the ESA2 model).
  • The temperature drift correction and extended gains calibration products are also updated to be consistent with the new flux conversion.

Spectrometer

  • The telescope and instrument RSRFs have been updated to have much higher signal-to-noise. This is due to a new method that makes it possible to increase the number of observations used to derive them. The consequence of this is that long observations have up to 20% lower noise level.
  • The beam FWHM inside the SCalSpecBeamParam product has been updated to be consistent with the 2D beam profiles contained in the SCalSpecBeamProf product, and to contain numbers taken from the beam profile paper (Makiwa et al., Applied Optics, in press 2013).
  • The telescope model correction was improved by including recent days in the derivation of the calibration product - this improves the continuum shape of recent observations of faint targets.
  • Calibration products were updated to allow bright mode observations to be processed in the same pipeline as nominal mode observations:
    • The non-linearity correction product for bright mode was updated
    • The bright mode gains calibration product was updated
    • The SCalSpecNomPcal product was removed (no longer needed as functionality incorporated into the bright gains calibration product)
    • All bright mode specific editions of the RSRFs and beam parameter file were removed - both bright and nominal pipelines now use the same products.
    • The channel time constant product was updated to allow bright mode specific time constants to be stored - this functionality is not yet used, but is ready for a future HIPE version.

Calibration Framework

  • The way that jiggled RSRF files are selected for spectrometer mapping observations was improved. This may affect (improve) maps made before OD1011.

Preprocessing

Engineering Conversion

Common Pipeline

Photometer Pipeline

  • New definition of Level 2.5 and Level 3 products. Level 2.5 is now much more homogenised with PACS and contains single pairs of nominal and orthogonal SPIRE destriped parallel maps. The new Level 3 products are similar to the old Level 2.5 maps, but with different units (MJy/ster rather than Jy/beam). They contain mosaics of overlapping regions from a given program using the absolute calibrated maps from the Planck zero point.

Spectrometer Pipeline

  • Level-1 spectra are now averaged over scans by the pipeline (although level-1 interferograms remain unaveraged). This reduces the size of the observation context. The mapping algorithm has been updated to correctly propagate the errors when averaged scans are used. The individual scans can still be examined by running the pipeline script (either from the beginning or starting with the interferograms).
  • Point source calibrated data are now included in the Level-2 product for unvignetted detectors - previously only the central detectors were included.
  • Bright mode observations are now processed using the same pipeline as nominal mode. This leads to higher accuracy in bright mode observations (although the overall change from HIPE v10 is <10%).
  • The way that the WCS is calculated for mapping observations was optimised (in SpireWcsCreator) - this means that the grid used for mapping observations is different to previous versions (although pixel sizes remain unchanged). The exact pixel position of NaN values in the final Level-2 cube does change due to the shift in the grid compared to the observed sky positions.
  • The temperature drift correction task was removed from the pipeline - this task made no difference to the output spectra.
  • Spectral cubes now have the correct final resolution in their metadata
  • Second level deglitching parameters were optimised for observations with greater than 30 repetitions

Interactive Analysis and Tools

Spectrometer

  • Spectrometer Level1 Mask Editor is available.
  • A new semiExtendedCorrector task is available for correcting point source spectra for slightly-extended sources
  • A new "convolutionProjection" mapper is available for spectrometer data. The map maker used by the standard pipeline is still the Naive Map maker. The new convolution mapper is very similar to the HIFI gridding projection (distributing the signal from each detector across multiple pixels using a kernel), but allows a frequency dependent kernel.
  • Update to the SDS/SDI Explorer - now a plot is launched with a single mouse click on a detector in the footprint. A second click removes the spectrum from the plot.
  • Minor updates/corrections to useful scripts
  • spireProduct2SimpleSpectrum is now more flexible, and can combine two spectra into one SimpleSpectrum
  • A new task to extract point sources from mapping observations has been added (specPointSourceExtractor).

Photometer

  • convolutionMapper has a new weightKernel parameter, so that the signal from the bolometers can be distributed across multiple pixels when making a map

Documentation

Release Notes

  • Full list of tickets implemented: here.


blog comments powered by Disqus

Edit | Attach | Watch | Print version | History: r70 < r69 < r68 < r67 < r66 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r70 - 2013-12-02 - ElenaPuga
 
This site is powered by the TWiki collaboration platform Powered by Perl