Numeric routines

HIFI Products

HIFI Spectrum Toolbox

Plug-ins

We have also added a few pre-set applications (Topcat, Aladin and VOSpec) to the Send To menu. This menu is accessible by right-clicking on a variable. The pre-set applications are visible, even if the application in question is not running and/or HIPE is not connected to the VO. Using these pre-set is transparent to the user: When sending a variable, the application will be automatically started first and the data will be sent.

Plug-ins

It is now possible for plug-in authors to publish new versions of plug-ins, which will be automatically picked up by existing users of the plug-in. If a new version of a plug-in is found by HIPE, and the author has indicated that the update is compatible with the user's version of HIPE, then HIPE will ask the user if the plug-in should be updated. The update mechanism is automatic, though HIPE must be restarted for the changes to be completed.

• Detailed error information can be opened with a double click in the Trace column of the History view.

• New bin-shaped button to the right of the memory bar, at the bottom of the HIPE window, to run the garbage collector and free memory manually.

• Multiple cells can be selected and copied in the TableDataset viewer.

• Find/Replace dialog enhanced. It can now show now whitespace characters.

• Added method addArcSecs to add a line of a given length in arcseconds to an image.

• You can now load all monochrome or indexed gif files using importRgbImageTask.

• The SpectrumFitterGUI has, in the export TAB, also the possibility to export fit parameters to file.
• Addition to SpectrumFitterGUI: user preference for initial models. If switched on, you can select from:

• • User specific initial models - you can specify the models.
• Addition to SpectrumFitterGUI, MultiTAB:

• Parameter error detection (red X) also checks the validator of the parameter.
• A new Help button gives easy direct access to the URM description of the task.
• White space in layout moved to the lower part of the panel.
• Editors based on text fields now require half the width.

• Support for grouping inputs in multiple tabs. For tasks that have inputs related to different aspects or simply a lot of them, they can be presented in several tabs.

• simpleFitsReader and simpleFitsWriter now also support reading/writting datasets (not just products).
• The Export view now shows the tags associated with each observation version.
• The localStoreWriter task now allows to add a tag to the saved product.
• localStoreWriter and localStoreCopier now use a better integrated GUI to select and create a pool.

• The clear task now has a simplified syntax to delete all variables (clear(), without any parameter, will delete all variables).

• Replaced the Read Me First document with the Welcome documentation page, and by a linked page listing the available manuals.
• In the welcome page, added links to a YouTube channel of tutorials and a Twitter channel of tips.
• Completely revised text and screenshots of the Quick Start Guide.
• Data Analysis Guide
  • Added section on the benefits of pools.
  • Added information on inspecting the quality of an observation.
  • Updated section on working with the VO.
  • Added section on importing and exporting images.
  • Improved tip on changing the colour and size of source symbols on images.
  • Updated the Spectrum Fitter section.
  • Added draft section on combining PACS and SPIRE data.
• Scripting and Data Mining guide
  • Many updates to the chapter on storing and accessing data products.
  • In particular, updated section on pools for remote data.
• HIPE Owner's Guide
  • Revised the whole document.
  • Improved instruction on updating Java on a Mac.
  • Added warning on HSA password and unencrypted property files.
  • Mentioned the new hipe_help command.
  • Documented the new Console view commands.
  • Added a partial list of keyboard shortcuts.
• Frequently Asked Questions
  • Revised the whole document, removed obsolete entries.
  • Added entry 9.4 on problems with monospaced fonts.
  • Added entry 10.1 on temporary pools not deleted.
• Products Definitions Document
  • Replaced some hardcoded references with hyperlinks.
  • Updates to Chapter 5 and Chapter 11 for Level 2.5 products.

• New internal commands !clear, !help and !history.

Status bar

• The garbage collector can be run with the button located at the right side of the memory bar.

Generic viewers

Dataset viewer

• Multiple cells can be selected and copied in the TableDataset viewer.

Jython editor

• Find/Replace dialog enhanced. It can show now whitespace characters.
• Autoindentation of Jython code improved.
• Indent and unindent buttons added to the toolbar.
• If the Monospaced font is not really monospaced, HIPE proposes an alternate font.

• new default doplot=2 option opens no more than 2 fitHifiFringe plot windows at the time and allows for inspection before processing next spectrum
• averscan option fitHifiFringe has been removed as standing wave characteristics vary too much between scans
• fitHifiFringe now shows up as applicable task to both HTPs and ObsContexts

• New non-stop option in interactive mode of fitBaseline allows automatic processing of remaining spectra
• The htpcopy option of fitBaseline has been removed as modified HTPs are now properly displayed in SpectrumExplorer

MadMap

• All level-2 tasks now do an internal loop over line ID. They behave now the same as the level-0/1 tasks: If you do not select a particular line/range before then the level-2 tasks just process all lines/ranges. As a consequence the level-2 products have changed a bit:
• We no longer have a list context for each line ID within another list context: All slices of a level-2 product are now stored in a single list context. This is much more practical when selecting slices based on arbitrary criteria.
• The pacsSliceContext task now contains all the rules in the ipipe scripts as default rules: Also, it slices by line ID, raster column, raster line, nodding position, nod cycle number, is out of field and band. The rules in the scripts are also now optional and merely for information.
• The interactive pipeline scripts (ipipe) are now tested in the nightly tester
• The interactive pipeline scripts (ipipe) are moved in the build from the directory scripts/pacs/spg/pipeline/ipipe/spec to scripts/pacs/scripts/ipipe/spec since they had to be moved to the package pacs_scripts

• Wrong coordinates in Spectrometer results: update of spec spatial cal file
• Improvements in wavelength calibration scatter for spectral pixels in band B2B -> wavelength calibration file updated

• Status item "GratingCycle" and "IndexInCycle" is now created for unchopped modes

Improvements for SSO scan maps

Old behavior :

frames = correctRaDec4Sso(frames, horizonsProduct=horizonsProduct, timeOffset=timeOffset, orbitEphem=orbitEphem)
frames = correctRaDec4Sso(frames, horizonsProduct=horizonsProduct, timeOffset=timeOffset, linear=1, orbitEphem=orbitEphem)

frames = correctRaDec4Sso(frames, horizonsProduct=horizonsProduct, timeOffset=timeOffset, linear=0, orbitEphem=orbitEphem) 

Aberration correction

Flux calibration

HIPE> c=getCalTree()
HIPE> print c.photometer

MadMap

HIPE> c=getCalTree()
HIPE> print c.photometer

Slicing

• Saving intermediate data on machines with low memory is now done more clever: The slices are only saved to the temporary pool when the memory usage is more than 75% and the sliced product contains more than one slice.
• The temporary pools of HIPE are now written under a directory 'null value' which is created where you start HIPE ( HcssSpr:12911 ).
  Workaround: Set the following property in your user.props file:

hcss.ia.pg.temporalpoolparent = ${user.home}/.hcss/lstore

These temporary pools are called tmp_product_sink_NNNNN

• A new Mask Array Viewer displays Masks with their correct size (also the timeline).
• Add start time of last cooler recycling to meta data of products A new calibration product was introduced : CoolerRecyclingTimes
  • A convenience tool times = extractCoolerRecTimes(obs, calTree) returns the last cooler recycling of an observation
  • This file will be updated on a regular base

• Map2signalCubeTask gets an additional shortcut: m2sc (besides map2signalCube)
• Spectrum Explorer : New extension panel to edit masks is now fully working
• Spectrum Explorer : Performance improvements

• calTree version is now in meta data of products

• Frames and PacsCube meta data is made consistent for every product change (like select or join)
• Additional meta keywords added: onOffSource, minWave, maxWave, nodCycle, lineId
• All masks which are not excluded in rebinning are propagated from PacsCube to PacsRebinnedCube. They can be obtained with the getFlag() method which returns a Flag dataset, which is similar to a PACS mask.
• We have now two more sliced products: SlicedPacsCube? and SlicedPacsRebinnedCube?. SlicedPacsCube? offers the same functionality as SlicedFrames?, but SlicedPacsRebinnedCube? has limited functionality, since it does not have a blocktable. Add, remove and replace work, but selection should be done with the pacsSelectSlices task.

Each slice is described by the following slice info meta keywords (recognizable by description prefix "SLICE_INFO:"):

Also, since these keywords are now always consistent with the BlockTable and Status contents they can and should be used for slice selection. The new task pacsSelectSlices does this meta-keyword based selection. It has the same interface as the selectSlices Python function.

• The wavelength grid is now also stored in a list context
  • All level-2 tasks now do an internal loop over line ID. They behave now the same as the level-0/1 tasks: If you do not select a particular line/range before then the level-2 tasks just process all lines/ranges. As a consequence the level-2 products have changed a bit:
  • The wavelength grid is now also stored in a list context, since one needs to pass several ranges from task to task.
  • We no longer have a list context for each line ID within another list context: All slices of a level-2 product are now stored in a single list context. This is much more practical when selecting slices based on arbitrary criteria.

• New documentation "How to work with masks" : http://herschel.esac.esa.int/twiki/bin/view/Pacs/HowToWorkWithMasks

Jython editor

• Find/Replace dialog enhanced. It now can show whitespace characters.
• Autoindentation of Jython improved.
• Indent and unindent buttons added to the toolbar.

Console

• Incremental search (Ctrl+I) available.
• Console help text enhanced.
• New internal commands !clear, !help and !history added.

History

• Detailed error can be opened by double-click in the history table.

Jobs control

• Progress bar shows the current line number of the executing script.

• SpecFluxConversionTask now precedes SpecTelescopeCorrectionTask in the Standard pipelines.
• SpecTelescopeCorrectionTask now, by default, subtracts only a model of the Herschel telescope and not the model multiplied by the RSRF for Telescope (Sky) port.

• SpecFluxConversionTask no longer automatically removes non-seeing channels. One must now use FilterChannelsTask to remove these channels from SpectrometerDetectorSpectrum products.

• AverageSpectraTask no longer provides functionality to remove out-of-band spectral samples. One must use RemoveOutOfBandTask instead.
• AverageSpectraTask no longer provides functionality to reject outlier from the average computation.

Level 0 to Level 0.5

Level 0.5 to Level 1

Changes to existing Tasks

• WaveletDeglitcherTask now offers the option to make the number of corrected samples dependent on the glitch size by setting the optionReconstruction parameter to "adaptative" or "adaptativeLinear".
• SpecFluxConversionTask no longer automatically removes non-seeing channels. One must now use FilterChannelsTask to remove these channels from SpectrometerDetectorSpectrum products.

Level 1 to Level 2

New Tasks

• SpireProduct2SimpleSpectrumTask to convert SPIRE Level-1 and Level-2 data to SimpleSpectrum products
• GriddingProjectionTask for spectral cube creation. Given SpirePreprocessedCube input, GriddingProjectionTask produces a spectral cube output. This task uses HIFI's GriddingTask.

Changes to existing Tasks

• NaiveProjectionTask now creates a spectral cube containing error instead of weight.
• AverageSpectraTask no longer provides functionality to remove out-of-band spectral samples. One must use RemoveOutOfBandTask instead.
• AverageSpectraTask no longer provides functionality to reject outlier from the average computation.

• Various internal changes have been made to the Source Extractor tasks, but no changes to the functionality.

• New tasks BaselineRemovalMedianTask and BaselineRemovalPolynomialTask operate on SPIRE Level-1 data to remove a baseline from the data. This is necessary due to the residual baseline that remains in the Level-1 data, which can produce stripes in the map if not removed. For each bolometer, the signal timeline is taken, either for each scan line separately, or for the whole observation as one long timeline. Then either the median value is subtracted (BaselineRemovalMedianTask), or a polynomial is fit to the data and subtracted. These tasks can be thought of as a "lite" version of the DestriperTask: they are simpler and faster, but for some observations containing a lot of structure (such as Galactic cirrus), there will be significant improvement if the DestriperTask is used.

• New task DestriperTask to iteratively remove the relative offsets between Level-1 detector timelines of the same detector array, and thus removing the stripes in the Level-2 map. Signal timelines are re-sampled from a preliminary naive map and subtracted from the original timelines. The resulting offset functions are modeled as polynomials with default degree (n=0) and subtracted. A new naive map is generated from the new difference of original signal and polynomial for the next iteration. The bright source threshold is used to suppress the influence of strong spikes in the data. The primary output is a SpireListContext containing updated PointedPhotTimelines. The additional outputs are a SimpleImage map, the convergence ratio, Tod etc.

• New task DestriperTask to remove the offset from Level-1 data and thus remove the stripes in the Level-2 map. The offset is modeled as a polynomial with default degree (n=0). The bright source threshold is used to eliminate the spikes in the data. The primary output is a SpireListContext containing updated PointedPhotTimelines. The additional outputs are a SimpleImage map, the convergence ratio, Tod etc.

• The HTTP pool now supports digest as well as basic authentication.

• An updated TempDriftCorr product allows a new algorithm in the Signal Jump Detector module to be used (nominal mode only).

• The spireCal task has new features and modified behaviour:

• • The trees available in the archive can be listed.
• The API has new methods for showing differences between two calibration trees and for verifying that expected metadata is present and correct.

• The mosaicing of maps from overlapping observations in a proposal is now fully functional (this is actually available from 6.1).

• Parameter error detection (red X) also checks the validator of the parameter
• A new Help button gives easy direct access to the URM description of the task
• White space in layout moved to the lower part of the panel
• Editors based on text fields now require half the width
• Labels of parameters have now a maximum size to reserve space for the editors.
• Support for grouping inputs in multiple Tabs. For tasks that have inputs related to different aspects or simply a lot of them, they can be presented in several tabs.

There is now a detailed outline shown in the Outline View when a task is selected in the Task View. This allows inspection of the structure of a task without the need to either use the Console or open its GUI in the Editor. Also, the system is undergoing a review of task compliancy (naming, documentation, tooltips ...) that will make all tasks self-explanatory (ongoing for HIPE 8).

• simpleFitsReader and simpleFitsWriter now also support reading/writting Datasets (not just Products).
• The Export View now shows the tags associated with each Observation version.
• The localStoreWriter task now allows to add a tag to the saved product.
• Now, when you use the contextual menu to save a product variable you will use the Save to Pool tool instead of the localStoreWriter task.
• localStoreWriter and localStoreCopier now use a better integrated GUI to select-create a pool.
• File filters for file dialogs are now self-explanatory (accepted extensions ...).
• The clear task now has a simplified syntax to delete all variables (clear() without any parameter, will delete all).

• Spectrometer calibration products for nominal detector settings have been updated to make use of in-flight bolometer model parameters.

• The LpfResponseCorrectionTask now accepts a new optional, boolean input parameter "doMirrorSignal". Setting this parameter to true, the task will perform the correction, by mirroring the signal time-lines in the y-axis. The effect of this is to avoid any edge-ringing, resulting from the default correction method when applied to timelines with a large slope. The draw-back of the new method is that it can double the processing time of the Task, however, not to a detrimental effect to single observation processing (It will definitely be noticeable when processing a large number of observations).

• WaveletDeglitcherTask offers the option to make the number of corrected samples dependent on the glitch size by setting the "optionReconstruction" parameter to 'adaptative' or 'adaptativeLinear'.

• Pipeline Specification Manual: Entries on spectroscopy have been reviewed, examples have been validated.