---+ What's New in HIPE 6 | *PDF Version* | [[%SCRIPTURLPATH%/genpdf%SCRIPTSUFFIX%/%WEB%/%TOPIC%?pdforientation=portrait&pdftoclevels=2][Click here]] | This document lists the changes in HIPE version 6.0 with respect to the 5.x series. Additional pages list changes between minor versions of the 6.x series. * [[HipeWhatsNew61][Changes between HIPE 6.0.3 and 6.1]] * [[HipeWhatsNew603][Changes between HIPE 6.0.2 and 6.0.3]] * [[HipeWhatsNew602][Changes between HIPE 6.0.1 and 6.0.2]] * [[HipeWhatsNew601][Changes between HIPE 6.0.0 and 6.0.1]] *See also the [[HipeKnownIssues][HIPE known issues]] and the [[DpKnownIssues][Data products known issues]] pages for last-minute information on known problems.* %TOC% ---++ Core system The Java Virtual Machine included in Windows and Linux downloads has been updated to version 1.6.0_22. ---+++ HIPE ---++++ Help & Support * _Help > About_ shows the HIPE release number as well as the build number. * Data reduction guides for the different instruments are available directly from the Help menu. * HSC Helpdesk can be accessed from the Help menu. * Direct access to on-line help from Help menu. * Crash reports can be sent for analysis to the HSC upon user confirmation. * Tip of the Day additions. * Enhanced help for =jylaunch= tool in the terminal. ---++++ Startup & Shutdown * You are informed about variables that could not be saved. _Note that saving variables on exit is optional; it can be set in preferences._ * If HIPE is closed while having opened a script from the installation, and a different HIPE version is started afterwards, a warning is shown informing that the opened script belongs to other version. * LOCK files from previous connections to the HSA not properly closed are asked to be removed at HIPE startup. * Panel for Startup & Shutdown preferences has been improved. ---++++ Editors & Views <blockquote> ---+++++ Outline When a task is selected, a summary information of it is shown. ---+++++ Console The horizontal scroll bar has been removed. ---+++++ History Copying an entire exception trace can be done with the history entry collapsed. ---+++++ Variables _Delete_ (selection) and _Delete all_ buttons distinguished. ---+++++ Navigator * Compressed FITS files are recognized and opened as such. * Enhanced mechanism for file identification; it supports XML file descriptors of observations. ---+++++ Jython editor * Autoindent and unident improved. _Note that the corresponding preference must be marked to make use of this functionality._ * The line continuation character \ is highlighted at the end of a line. ---+++++ Task dialogue Checkboxes for boolean parameters accept variables holding an integer with value 0 (false) or 1 (true). </blockquote> ---++++ Menus * All dialogs and windows opened from HIPE, including plots, can be accessed through _Window > Select Window_. * Available views have been categorized under _Window > Show View_. * Entries of the _File > New..._ menu have been simplified. ---++++ Preferences * Configuration properties can be edited directly from the preferences dialogue, by means of the new _Advanced..._ button. * Saving preferences (with _OK_ or _Apply_) is now faster and robust against failures. * Panel for Startup & Shutdown preferences has been improved. ---++++ Jobs control * Running jobs can be consulted and cancelled individually with the new _Running Jobs_ button at the left side of the progress bar. * If more than one job is running, clicking on the _Stop_ button will show a similar dialogue to choose the job(s) to stop. ---++++ Miscellanea * The criterion for determining the valid FITS extensions has been unified among different views and dialogues of HIPE. * The error dialogue that includes an exception trace has it collapsed by default. ---++++ Plug-ins A number of different small issues related to plug-ins have been resolved, including * issues with the installation, * path names containing spaces (important for Windows XP), * proper clean-up in case of failed installation, * reducing the number of required restarts. Plug-ins can now also be used when scripts are run using =jylaunch=. ---+++ Plotting * A small subplot can be put inside a bigger plot. <!-- ---++++ !TablePlotter ---++++ !OverPlotter ---++++ Dataset viewer ---+++ Astronomical utilities ---+++ Numeric routines ---++++ Data fitting ---++++ Data interpolation ---++++ Random numbers ---++++ Statistics ---++++ Trend analysis ---++++ Mapping --> ---++++ Wavelet transform * Delivered components: | *Algorithm* || *One dimensional signal* | *two dimensional signal* | *Gaussian noise estimator tool* | *Thresholding tool* | *Universal thresholding tool* | | CWT | Continuous Wavelet Transform | supported | - | not available | not available | not available | | DWT | Discrete Wavelet Transform | supported | supported | available | available | available | | SWT | Stationary Wavelet Transform | supported | supported | available | available | available | | *Wavelet* | *Daubechie* | *Symlet* | *Coiflet* | *MexicanHat* | *Morlet* | | CWT | 1 | - | - | yes | no | | DWT | 1-10 | 1-10 | 1-5 | - | - | | SWT | 1-10 | 1-10 | 1-5 | - | - | <!-- ---++++ Other functions --> ---+++ Source extraction * SUSSEXtractor no longer fails to find sources close to !NaN pixels in the image or in the error map * SUSSEXtractor produced strange results when then error map had very small values. This was a problem because the naive mapper error is the standard error on the mean of the bolometer samples in the pixel, which can be extremely small if the pixel contains a small number of samples that happen to have very similar values. SUSSEXtractor now estimates its own value for the error when the naive mapper uncertainty is implausibly small. ---+++ Images ---++++ Display * Major speedup when zooming in on images. Less memory is used when zooming in. * Fixed some memory issues. The display does no longer keep a lot of memory in use. * It is possible to hide or show the panel with the panner and the zoom. <!-- ---++++ Analysis --> ---+++ Spectra <!-- ---++++ Display --> ---++++ Analysis * A Spectrum Integrator is available as prototype. Currently only the basic functionality is available from the command line and it allows to integrate part of a (background subtracted) spectrum. The data is expected to be in the form of a !SpectralSegment (_seg_ in the examples below). * *Integration:* The return value is a double. * i = Integrator.doIntegration(seg) * *Integration over ranges:* The return value is an array of n doubles. The windows are formatted as [start, stop, start, stop, ...]. So for one window from a to b: [a, b], and for n windows: [a1, b1, a2, b2, ..., an, bn] * i = Integrator.doIntegration(seg, [windows]) * *Integration over ranges with 1st order background removal:* The background is removed by fitting a 1st order poly in the [masks] areas. The format of [masks] is similar to the [windows] format. * i = Integrator.doIntegration(seg, [windows], [masks]) * *Integration over ranges with nth order background removal:* * from herschel.ia.toolbox.spectrum.integrator import Integrator * i = Integrator.doIntegration(seg, [windows], [masks], n) *How to get a !SpectralSegment* (HIFI example) To get a !SpectralSegment from the HIFI observation context 'obs', e.g. the first dataset of the level-2 HRS_L_LSB: d = obs.refs["level2"].product.refs["HRS-H-LSB"].product.refs["1"].product["dataset"] This is a !HrsSpectrumDataset (which is a !SpectrumContainer). Visualising this in the !SpectrumExplorer shows that is contains one spectrum with 8 subbands. Each of the subbands is a !SpectralSegment. Get any as: seg = d.getPointSpectrum(0).getSegment('any') The variable 'seg' can now be used in the Integrator. <!-- ---+++ Data cubes ---++++ Display ---++++ Analysis ---+++ Herschel Science Archive ---+++ Calibration sources ---+++ Products and datasets ---++++ Observation context ---+++ Product Access Layer --> ---+++ Tasks * Improved documentation. * Added descriptions to tasks. All tasks can provide a high level description (shown in tooltips in Task View). * Simplified localStoreWriter task. <!-- ---+++ Data input-output ---+++ Virtual Observatory ---+++ Installer --> ---+++ Documentation Many fixes and updates to align the documentation with changes in HIPE. Some highlights are given in the following sections. Links are to the online version of the documentation ---++++ Quick start guide * Added level 2.5 to the [[http://herschel.esac.esa.int/hcss-doc-6.0/index.jsp#quickstart:qs_inspect_data][list of data processing levels]]. ---++++ HIPE Owner's Guide * Improved section on [[http://herschel.esac.esa.int/hcss-doc-6.0/index.jsp#hipeowner:site.hipe.props][properties and preferences]]. * Improved section on [[http://herschel.esac.esa.int/hcss-doc-6.0/index.jsp#hipeowner:site.perspective.change][customising views and perspectives]]. ---++++ Data Analysis Guide * Added section on [[http://herschel.esac.esa.int/hcss-doc-6.0/index.jsp#howtos:sec-getobservation][getObservation and saveObservation]]. * Added section on [[http://herschel.esac.esa.int/hcss-doc-6.0/index.jsp#howtos:sec-ondemand][on-demand reprocessing]]. * Added [[http://herschel.esac.esa.int/hcss-doc-6.0/index.jsp#howtos:Dag.DataIO.HowTo.AccessHsa][warning on storing sensitive data]] in property files. * Added section on [[http://herschel.esac.esa.int/hcss-doc-6.0/index.jsp#howtos:Dag.Plot.Subplots][subplots]]. * Updated section on [[http://herschel.esac.esa.int/hcss-doc-6.0/index.jsp#howtos:Dag.Display.HowTo.ViewingACube][viewing spectra and cubes]]. * Added section on [[http://herschel.esac.esa.int/hcss-doc-6.0/index.jsp#howtos:sec-running-image-analysis-tasks][running image analysis tasks]]. * Removed duplicated content from [[http://herschel.esac.esa.int/hcss-doc-6.0/index.jsp#howtos:Dag.ImageAnalysis.InDepth.ImageArithmetics][image arithmetics]] section, added links to User's Reference Manual. * Added tip on [[http://herschel.esac.esa.int/hcss-doc-6.0/index.jsp#howtos:Dag.Display.InDepth.SimpleImageEditing][drawing filled shapes]] on an image display. ---++++ Scripting and Data Mining * Revised section on [[http://herschel.esac.esa.int/hcss-doc-6.0/index.jsp#um:sec-seldatfilmask][array filtering]]. * Added instructions on [[http://herschel.esac.esa.int/hcss-doc-6.0/index.jsp#um:d0e3844][converting PACS data cubes]]. * Added section on a couple of [[http://herschel.esac.esa.int/hcss-doc-6.0/index.jsp#um:sec-lstore-troubleshooting][common problems]] with local pools. * Added information on [[http://herschel.esac.esa.int/hcss-doc-6.0/index.jsp#um:sec-products-equal][comparing products held in pools]]. ---++++ Products Definitions Document * More updates to [[http://herschel.esac.esa.int/hcss-doc-6.0/index.jsp#pdd:spireobsprodp][SPIRE product tables]]. * Added level 2.5 to the [[http://herschel.esac.esa.int/hcss-doc-6.0/index.jsp#pdd:observationprods][list of data processing levels]]. ---+++ Help viewer * Completely redesigned Welcome page. * Advanced search functionality. * Added information about currently viewed document and current version of HIPE. * New functionality to jump to the index page of the currently open manual. ---++ HIFI ---+++HIFI Software Configuration and Properties ---+++HIFI Calibration * Addition of getCalTree enables download of new calibration from HSA into a local store. It is used in the following way: =cal = getHifiCal(useHsa=True, overWrite=False, calName="hifi-cal")= * Update to spur table in the calibration tree * Calibration products no longer registered as proprietary in the HSA. In consequence, it is no longer necessary to have HSA authorisation to run the pipeline and update calibration. ---+++HIFI Pipeline * The little-used Hifi Pipeline view is removed from HIPE * Data processed at the HSA On Demand will now write the version of the pipeline used (SPG version) in reprocessing in the creator field of the observation context meta data * It is now necessary to configure the HIFI pipeline to access the HSA before running the pipeline using the last released calibration tree =obs = getObservation("1342205520", useHsa=True)= =cal = configureHifiPipeline()= =obs_1 = hifiPipeline(obs=obs,fromLevel=0.0,upToLevel=2.0, cal=1, aux=1, palStore=cal, save=False)= * Various changed to HIFI pipeline GUI * "instrument" replaced with "spectrometer" * palStore moved to the Expert Window * hifical moved to the Basic Window * hifical tooltip should indicate that setting hifical uses the latest calibration available to Astronomers ---++++ !hifiPipeline task * Bug fix: the auxilliary product was not repopulated when running the hifiPipeline task with aux=True. This is now fixed. * First steps towards making the HIFI pipeline fully configurable are made with the introduction of doPipelineConfiguration. At present changes to the level 2 pipeline algorithm are not passed to the hifiPipeline task. This feature will be much more robust and user-friendly in HIPE 7.0 ---++++ Level 0 Pipeline * Now only the subbands affected by a major spur are flagged, previously it was all four subbands of the WBS regardless of whether they were all affected or not. ---++++ WBS Pipeline * Threshold behaviour is changed in !MkWbsFreq: the threshold value is not divided by the scancount unless it is > 1023 * Structure put in place to cross-calibrate WBS data with that from HRS <!-- ---++++ HRS Pipeline --> ---++++ Level 1 Pipeline * Rearranged pipeline algorithm for Frequency Switch and Load Chop data. The bandpass correction is now done before the smoothing and subtraction of the OFF position. This has the consequence of removing small spurs seen in the WBS V and also improving baseline quality. * Introduction of truncated linear interpolation scheme in the MkOffSmooth step of the level 1 pipeline. This is to avoid numerical problems when two Tsys spectra (used to calculate the weights used in averaging) are very far apart. * Smoothing widths in MkOffSmooth optimised in calibration tree. Further work and improvements expected ---++++ Level 2 Pipeline * The WBS metadata produced by the !doFreqGrid step of the pipeline has been clarified: the !frequencyWidth metadata description is changed from 'The unique (!) channel width for the output grids of all subband' to 'The spacing of the frequency grids after resampling'. * The items 'forwardEff' and 'beamEff' and are added to the metadata after the completion of the !doAntennaTemp or !doMainBeamTemp steps of the level 2 pipeline, as applicable. * !MkFreqGrid task is adjusted so that all output spectra have the same length of subbands. This allows all types of output to continue to be handled by !HifiSpectrumDataset * Improvements to messages from !doMainBeamTemp, the following messages are now given: * When no beam efficiency is passed, or no beam efficiency is present in the calibration product passed to the task: "Product with name beamEfficiency-H not found in context. No information on the beam efficiency has been passed to the task !doMainBeamTemp or the calibration data is outdated. Correction in task !doMainBeamTemp is not applied." * When no input has been found in the calibration product: "No information on the beam efficiency was found in the calibration product. Please provide a value by hand". ---+++HIFI Products * !HifiSpectrumDataset is converted such that it contains upConvert_H or upConvert_V metadata when the band is 6 or 7. This allows to read off the value in order to convert to sky frequency. * Additions to Quality Products to indicate if there were LO tuning problems. * New statistics trend product. Statistics (rms etc) are calculated at the end of the Levels 1 and 2 pipeline and results stored in the new statistics product. Meant for trending analysis for the ICC but may be of interest to astronomers. Note that statistics are calculated 'blind' with no line masking! * Clash between cached products are references resolved so consistent behaviour seen when modifying products. ---+++ HIFI Spectrum Toolbox <!-- ---++++ Spectrum Explorer (HIFI specifics) --> ---++++ Spectrum tasks (HIFI specifics) * Problem with statistics task whereby the preview of the task result did not update when several spectra selected is fixed ---+++ HIFI Data Processing Tools * Bug fix: tasks under hifi_dp_tools did not load into HIPE * Improved error messages from !ConvertK2Jy ---++++ Standing Wave Removal * New 'automask' option that allows automated line masking to be switched off. * New 'sds_index' option allowing the processing of specific datasets in !ObservationContexts. * Problem whereby sine waves added to datasets containing multiple scans (e.g. OTF maps) fixed * Problem where masks appeared to be displaced from bright lines is fixed ---++++ Baseline Removal * New options !wingChannels and !minChannels to improve automated line masking (domask=2). ---++++ Deconvolution * !DoDeconvolution now accepts an array of obsids, enabling a combination of data for which deconvolution can be performed. * !DoDeconvolution now keeps a log of problems encountered with its inputs, outputs, and difficulties. The log is accessible with an ASCII reader. * Improved error message handling from output of !DoDeconvolution ---++++ Spectral Cube creation * Major updates to !doGridding GUI: improved parameter entry, tooltips and design. Task GUI now opens on the expert view. * !doGridding makes a work around to calculate an appropriate pixel size regardless of whether the map was made with half beam spacing or was Nyquist sampled. Previously only a value appropriate for half-beam spacing was used. In the future the values used for the observation will be taken from the downlink product. <!-- ---+++ !HiClass export tool --> ---+++ Documentation * Updates to the following sections of the HIFI User Manual: Data Primer, Running the HIFI Pipeline, Quality Flags, HIFI Standing Wave Removal Tool, HIFI Baseline Removal, Exporting HIFI Data to CLASS, HIFI Cookbook * Addition of two sections to the HIFI User Manual: Reference Frames in HIFI Data, Making Publication Quality Plots * Many outdated references to JIDE removed from URM * addition of !getHifiCal to URM * Addition of HIFI cookbook link directly from HIPE Help -> Data Reduction menu ---++ PACS ---+++ Pipeline * Fully functional slicing framework. * Used for spectrometer processing, only partly used for photometer processing. * Significant cleanup and re organisation of interactive processing scripts. * New structure of Menu -> Pipeline -> PACS. * A number of instrument-mode-specific user-provided IA scripts are available now. ---++++ Photometry * Implemented proposal for a two-stage photometer pipeline with masked highpass filtering. * MMT Deglitching may mask a glitch, although the signal would be unmodified. * Improvements on data filtering (usage of BBID in removeCalBlocks, usage of scan legs etc.). * Significant improvements in the SSO astrometry (correctRaDec4Sso). * Fully functional !MadMap processing implementation inclusive of cross scans. ---++++ Spectroscopy * Linearity correction removed from pipeline scripts. * Usage of elements of the Spectrum Explorer. * The fitter tool can now be used for all PACS products. ---+++ Calibration <!-- ---++++ Photometry --> ---++++ Spectroscopy * New wavelength calibration applied by using the cal file wavePolynomes. * The !NominalResponse cal file is active, including the 1.3 and 1.1 ILT to flight conversion factors. * This means that if you process data you shall not apply these correction factors manually. * Also the pixel-pixel flatfield improvement described in PICC-KL-TN-045 (available in the central file) is implemented in this !NominalResponse calfile. ---+++ Analysis tools * Calibration framework: As of build 6.0.1452 the calibration framework will read the calibration products no longer from the distribution, but from a directory on your local machine. * You will be presented with a popup dialog asking to install the calibration. The dialog looks like this. * Screen Shot Calibration Framework: <br /> <img src="%ATTACHURLPATH%/Screen_shot_2010-12-14_at_10.34.04.png" alt="Screen_shot_2010-12-14_at_10.34.04.png" width='692' height='324' /> * Please go through the process of installing the calibration products on your local machine. * During the HIPE startup process, the calibration framework will check for new updates in the background and notify you of any updates. * If you want to start the update manually, go to the _Tools > pacs-cal_ menu and select _run Updater_. * You can read more about the PACS Calibration Framework in the comprehensive document available in the public part of the Wiki. * Data Access: A new keyword is introduced to getObservation which allows the user to disable the Cache when using the HSA. * obs = getObservation(123456789, useHsa=True, useCache=False) * This should solve many problems when you just want to download and save an observation from the HSA on your local machine, as the data will not be saved twice (once in the cache and once in the local store pool) anymore. <!-- ---++++ Photometry ---++++ Spectroscopy ---+++ Product changes ---++++ Photometry ---++++ Spectroscopy --> ---+++ Documentation * Updated the PACS Data Reduction Guide to follow the new spectroscopy pipeline scripts (Chapter 3), and added a chapter on !MadMap. Minor updates to the other chapters to reflect other changes in the system. ---++ SPIRE ---+++ Calibration Products ---++++ Photometer * Improvement to flux conversion. ---++++ Spectrometer * Improved phase correction module in the pipeline no longer requires a calibration file. The final correction should be better using the new method. * Rsrfs and beamParam updated to follow the new v6 pipeline scheme and use improved phase correction - should be improvement in shape of final spectra. * !BeamParam file modified to improve information for IA - updated beamAreas and added point source RSRF - this should have no effect on the SPG pipeline. * !OpdLimits added to clean up the pipeline script - should be no effect on data. * Fix to non-linearity correction. * Added PCAL calibration products for the spectrometer. ---+++ Calibration Framework ---+++ Preprocessing * Added processing of PCAL flashes executed in spectrometer observations. <!-- ---+++ Engineering Conversion ---+++ Common Pipeline --> ---+++ Photometer Pipeline * Jump detection in signal timelines, allowing better temperature drift correction. * Added flag to include or exclude turnaround data in scan map observations. ---+++ Spectrometer Pipeline ---++++ Level 0 to Level 0.5 See [[http://herschel.be/twiki/bin/view/Public/HipeWhatsNew6x#Engineering_Conversion][Engineering Conversion]]. ---++++ Level 0.5 to Level 1 ---+++++ Processing Steps ---++++++ Phase Correction A new algorithm has been implemented that corrects for asymmetries using the low-resolution portion of the measured interferograms. ---+++++ Data Products Level-1 spectral products now contain all measured extended flux-calibrated spectra rather than the average spectra per detector. ---++++ Level 1 to Level 2 ---+++++ Processing Steps ---++++++ Spectral Averaging 1. The spectral averaging step is now only apply to Sparse, Point-source observations. 1. The error column in the output data product is now populated with the standard error of the mean (Standard Deviation / SQRT(Number of Samples)) of the input spectral samples. ---++++++ Spectral Maps Level2 spectral cubes are now created by averaging flux values within each pixel in the output map. In addition, the execution speed of Spectral Mapping processing step has been greatly improved - a set of high resolution spectra is now processed in approximately 10 seconds. ---+++++ Data Products Level-1 spectral products now contain all measured extended flux-calibrated spectra rather than the average spectra per detector. ---++++ New Utitlities A new set of utility functions and tasks have been implemented for HIPE version 6. 1. ==filterChannels==. A task that allows one to remove a selection of channels from any type of SPRIE FTS data product. 1. ==removeOutOfBand== A utility task that allows one to remove a from a SPIRE FTS Spectral data product spectral samples outside a specified band. 1. ==makeSameOpds== A utility task that allows one to remove a from a SPIRE FTS Interferogram data product samples outside a specified OPD range. 1. ==mergeFtsBuildingBlocks== A utility task that merges Level-1 SPIRE FTS Interferogram and Nominal Housekeeping products. 1. ==specMergeHighLowSds== A utility task that is used for =HR=+=LR= SPIRE FTS observations. This task appends the =LR= portion of the interferograms from the =HR= buildling block to the interferograms from the =LR= building block at each Point and Jiggle position of the observation. 1. ==observationLayout== A utility task that inspects the Level-0.5 context of a SPIRE FTS observation and returns a table linking the Building Block ID numbers with Point numbers, Jiggle numbers, and the commanded resolution. ---+++ Interactive Analysis and Tools ---++++ ==Mosaic== plotting utility A new plotting utility, ==Mosaic==, has been developed to allow one to plot the contents of a SPIRE FTS Interferogram or Spectral Data Product in a grid that corresponds to the physical layout of the SPIRE SLW and SSW detector arrays. <!-- * Set ALLOWTOPICCHANGE = Main.RegisteredUsersGroup, Main.TWikiAdminGroup -->
Attachments
Attachments
Topic attachments
I
Attachment
History
Action
Size
Date
Who
Comment
png
Screen_shot_2010-12-14_at_10.34.04.png
r1
manage
53.5 K
2011-01-17 - 14:29
EkkehardWieprecht
Screen Shot Calibration Framework
This topic: Public
>
WebHome
>
HipeWhatsNew
>
HipeWhatsNew6x
Topic revision: r46 - 2011-07-04 - BertrandMorin