Changes between HIPE 5.0 and 5.1

Core system

• New loadObs task to open in HIPE an observation downloaded from the Herschel Archive.
• New Level 2.5 added to Observation Context for PACS/SPIRE data. Mentioned in the Products Definitions Document.
• Pipelines can now process multiple observations at once (for instance, scan and cross-scan data). The results of this so-called extended processing are stored in the new Level 2.5 data.
• You can now specify the local store directory in the Preferences window (Data Access --> Local Store).
• Fixed a problem in the Save Products to Pools view. Entering the name of a new pool would not create the pool.
• Fixed a problem with the handling of serialised pools in the Storages & Pools section of the HIPE preferences window. Serialised pool definitions would not be removed when pressing Restore Defaults, and would not be restores when pressing Cancel after Delete.
• The LocalStoreWriter task now creates new local pools in the default $HOME/.hcss/lstore directory.
• Fixed a problem that would prevent HIPE from starting when the list of local pool names, separated by commas, exceeded 8096 characters.
• Pools created with localStoreWriter when saving a product now appear in the Storages & Pools configuration panel.
• Improved the Product Access Layer performance by a more efficient loading of descriptors.
• Fixed a problem that would cause errors with multi-user access to local pools, due to lock files not being properly removed. In the unlikely case that lock files for a local pool are still not cleaned (for example after a software crash) you can remove them manually with the command herschel.ia.pal.pool.lstore.LStoreMaintenanceUtil.clearLocks(localPool), where poolName is a variable repreenting the local pool.
• Improved error message when trying to import partial observation from the Herschel Archive.
• Fixed a problem with the saveObservation, which would fail to save an Observation Context to a pool with default settings.
• Fixed a problem with imageSubtract, which would give wrong results when used with WCS.
• Zooming on an image no longer causes HIPE to hang. Zooming is also much faster.
• Fixed a problem with the NearestNeighbourProjectionTask task, which creates a spectral cube from three-dimensional data. The task would fail to create a target grid out of a set of target grids. A command such as targetGrid = nearestNeighbourProjection.targetGrid([tg1,tg2,tg3]) would fail with an IllegalArgumentException.
• The Wcs.getCrota2() method now returns CROTA2 in degrees instead of radians.
• Fixed a problem with FITS generation that would cut AOR labels containing a single quote character.
• Fixed a problem with loading FITS files. Datasets with long class names (requiring a CONTINUE card) would cause an error in some cases.
• Many stability improvements to the plugin framework.
• Improved HIPE performance when creating, changing or deleting many variables.
• Fixed a problem that caused deleted variables not to be removed from the Variables view in some cases.
• Improved the Startup & Shutdown section of the HIPE preferences window.
• Product history now records the value of small arrays.
• The default value of the var.hcss.workdir configuration property has been changed from ${java.io.tmpdir}/hcss/${user.name} to ${java.io.tmpdir}/${user.name}/hcss.
• The Help --> About dialogue window now shows the user release version in addition to the build number.
• The HIPE Help menu now include entries for instrument data reduction manuals.
• The HIPE Help menu now contains a link to the Herschel Science Centre Helpdesk.
• HIPE can now send crash reports to the Herschel Science Centre for analysis.
• The HIPE Owner's Guide and Data Analysis Guide have been updated with all the new HIPE features.
• The Quick Start Guide now clarifies that the described data access method is for quick inspection rather than full data retrieval.
• Many improvements in the categorised view of commands in the core User's Reference Manual. The remaining glitches are being fixed for HCSS 6.0.
• New step-by-step guide for getting data out of the Herschel Science Archive and into HIPE. <-- HCSS-11635 -->
• Updated SPIRE tables in the Products Definitions Document.
• Appendix A of the Scripting and Data Mining guide, about advanced Product Access Layer usage, has been completely revised.
• Many references to the old JIDE application have been removed from the documentation. A warning has been added in the User's Reference Manual, where JIDE is still referred to, especially in example titles. Any remaining references to JIDE can be safely applied to HIPE.
• Instrument manual names have been changed to be more consistent.
• Fixed hyperlinks among manuals in core and instrument documentation. Note that links between core and instrument manuals are not yet implemented. <-- HCSS-10907 -->

SPIRE

• The signalJumpDetector task is now initialised at HIPE startup
• Fixed a problem with AbstractMapperTask, which might treat incorrectly a Wcs provided as input. When the input Wcs was defined using the CDELTi keywords, the result was correct, but when the pixels were defined with the CDi_j keywords, these were translated into CDELTi and CROTA2 keywords.
• Fixed a problem with AbstractMapperTask. There was a bug in the generation of tod files which could result in an ArrayIndexOutOfBoundsException at the map generation.
• Fixed an example of spectrometer map making using Naive Projection in the Data Reduction Guide. <-- SPIRE-2985-->
• The concurrentGlitchDeglitcher entries in all scan map pipeline scripts now explicitly mention all inputs.
• All the parameters of the concurrentGlitchDeglitcher task are now explained in the Pipeline Specification Manual.
• Fixed a problem with the spectrum explorer for SpectrometerDetectorSpectrum objects. Units would always be displayed as Flux [V.cm] even after being changed to W/(m².Hz.sr).
• The spireCal task now raises an error if the input pool does not exist, instead of creating an empty calibration context.
• Updated spectrometer user scripts to clarify that option G has only effect for observations in "High+Low" mode.
• More helpful error message in case a pipeline stops because the latest calibration tree is not available.
• Fixed a problem with the TelescopeCorrection task, which would always use the forward scan in the TeleRsrf calibration product. <-- SPIRE-3043 -->

HIFI

• New task convertKtoJy to convert HIFI Data from K to Jy.
• New task convertSingleHifiSpectrum to convert final HIFI Spectrum1d datasets to SimpleSpectrum products.
• Fixed a problem with fitBaseline, which could crash on fully spur-flagged subbands.
• The fitBaseline task now saves intermediate results when quitting.
• Ra/Dec coordinates in data cube meta-data are no longer incorrect.
• Fixed a problem with PolarPair, which would return the first dataset supplied to it because the tolerance level was too low by default (five channels of difference in frequency between the two polarisations). The default tolerance was raised to 100 channels of difference.
• The HiClass task has been updated to export the source velocity as provided in the AOR into new FITS header keywords, closely resemble the usual keywords for transporting the velocity information. The PVELTYPE keyword tells the type of velocity.

• Fixes to the User Manual:
  • Updates to pipeline chapters:
    • Documented the functionality to pass a list of obsids to the pipeline.
    • Corrected the pipeline options upToLevel and fromLevel, which were wrongly written as UpToLevel and FromLevel.
    • Documented how to get H and V pointing.
    • Clarified how to process with new calibration.
  • Updates to doGridding chapter:
    • Corrected references to half-beam and Nyquist sampling are correct.
  • Updates to plotting Chapter:
    • Noted how to create a variable for a HifiSpectrumDataset so that example can be followed.

PACS

• New maskInfo(slicedProduct) function to print an overview of the mask status in a sliced product.
• Obsolete scripts were removed from the HIPE Pipeline menu.
• Several improvements to the single-obsid photProject pipeline scripts, which now produces much better results.
• Several improvements to the documentation of the pipeline scripts found in the HIPE Pipeline menu.
• Fixed a problem in L1_pointSource.py script where sources could be interpreted as glitches.
• Fixed a problem with HighPassFilter that could cause the pipeline to crash.
• Fixed a problem with the unchopped mode raster observation pipeline, which would fail at SpecWaveRebinTask.
• Fixed a problem in the findBlocks task, which would not propagate metadata.
• Updated map pipeline scripts: scanmap_BrightPointsource.py and scanmap_Deep_Survey_miniscan_Pointsource.py.
• Fixed a problem with FindBlocksTask, which would give an error when creating a ListContext.
• In the photProjectPointSource task, the calibration = false mode has been deprecated because it is not flux conserving.
• Included MadMap documentation in the Data Reduction Guide.
• Several improvements to the spectrum extraction documentation at http://www.herschel.be/twiki/pub/Pacs/SpectrumExtractionTask/prototypeSpectrumExtraction.pdf.