TWiki
>
Public Web
>
HipeWhatsNew
>
HipeWhatsNew4x
(2012-10-30,
DavideRizzo
)
(raw view)
E
dit
A
ttach
Tags:
create new tag
view all tags
<!-- ANALYTICS CODE - DO NOT EDIT --> %INCLUDE{"%ATTACHURL%/GoogleAnalytics.txt" raw="on"}% <!-- END OF ANALYTICS CODE --> ---+ What's New in HIPE 4 | *PDF Version* | [[%SCRIPTURLPATH%/genpdf%SCRIPTSUFFIX%/%WEB%/%TOPIC%?pdforientation=portrait&pdftoclevels=2][Click here]] | | %ICON{"help"}% *Interested in what's new for other versions of HIPE? See [[HipeWhatsNew][this page]] for links to all the What's New documents from HIPE 4.0 onwards.* | %TOC% This document lists the changes in HIPE version 4.2 with respect to the 3.x series. Additional pages list changes between minor versions of the 4.x series. * [[HipeWhatsNew43][Changes between HIPE 4.2 and 4.3]] * [[HipeWhatsNew44][Changes between HIPE 4.3 and 4.4]] * [[HipeWhatsNew45][Changes between HIPE 4.4 and 4.5]] * [[HipeWhatsNew46][Changes between HIPE 4.5 and 4.6]] ---++ Core system ---+++ HIPE ---++++ Navigator view * The _Navigator view_ allows adding customized user areas. * The _Navigator view_ is able to remove an entire directory along with its contents. As usually done with removals, this operation requires user confirmation. * Selecting a FITS image in the _Navigator view_ shows a preview of it in the _Outline view_. Previews are cached so re-visiting them is faster. ---++++ Editors & Viewers * Text and Jython editors will save backup files so that, if HIPE exits abnormally, you can recover (at least part of) your unsaved changes. * When an error is produced the error highlight is now set to the closest (deepest in the call stack) line in the currently open script. Before, it was set to the top line (highest in the call stack); now it "goes into functions". * Dragging a variable from the _Variables_ view and dropping it into a Jython editor will write its name in the drop location. * Variables can be created directly from nodes in the _Product Viewer's_ tree, with the option _Create Variable_ of its popup menu. * If a _Product Viewer_ or _Dataset Viewer_ shows data that is changed with a Jython command, it informs that the contents are outdated, and offers the possibility of a refresh. * Data tables can be sorted by clicking on the corresponding column header. Successive clicks will either add more columns to the sorting, revert the order (if re-click in the same column), or remove it by returning to the original state. ---++++ Console & History * _Console_ capabilities have been improved: to show the list of available key bindings, press Ctrl-Shift-H with the _Console_ as the current view. * The _History_ now ignores empty lines. This means that, if you go back to previous commands in the _Console_ with the up arrow, empty lines are skipped automatically, making the command navigation easier. ---++++ Startup & Shutdown * When variables have been saved in a previous session, HIPE asks whether to load them. This dialog can be optionally omitted. * If there are running jobs when closing the application, HIPE asks for confirmation before exiting. * HIPE checks whether there is a new user release at startup; if so, it informs the user with a link to the web page from where to download it. ---++++ Error handling * If HIPE crashes, it will inform about a file dump at next start, if possible. If you raise a ticket, please attach this dump for helping developers to debug the error. * HIPE has been made more robust against _Out of Memory_ errors: instead of hanging, it offers the possibility of exiting in an ordered manner (for example, by saving unsaved editors), or try to continue working to manually free resources. ---++++ Perspectives * Perspectives have been simplified: now there is a unique _Work Bench_, while the proof-of-concept _Classic (JIDE)_ perspective has been suppressed. * The _PAL Browser_ perspective has now completely replaced the _Data Access_ view. * _Reset perspective_ option has been added at the right side of the perspectives toolbar, so it is now more visible and accessible. ---++++ Miscellanea * When opening a file, the window's title shows the complete path. * Tooltip in the spinning throbber (at bottom right corner) informs about all running jobs. * Tips of the Day have been added and improved. * This version contains some bugfixes with respect to the previous user release. ---+++ Plotting * New auxiliary axis class ==ReciprocalAuxAxis==, providing reciprocal tick allocation. ---++++ !TablePlotter * New APIs are introduced which allow to set initial column mode and to specify if data extraction is allowed or not. ---++++ !OverPlotter * A legend check box is added to display the plot legend. * An _Appearance_ menu item is introduced in _Layer Props_ which allows to change layer's name in single layer mode and change both layer's name and color in all layer mode. ---+++ Astronomical utilities * Radial velocity's LSR definition is now consistent with MPS definition. * Radial Velocity can now also use data from downlink. * Reorganised aberration correction methods in the ==Aberration== class of the ==ia.obs.auxiliary.pointing== package. * Updated documentation. ---++++ Pointing * SSO correction now maintains the overall shape of a raster observation. * SSO can now also use data from downlink. * Updated documentation. ---+++ Numeric routines ---++++ New functions * NAN_FILTER: Returns the !NaN elements of an array (in a one dimensional array) leaving the original array untouched. If no !NaN element is found, the original array is returned. * IS_ANY_NAN: Checks if the given array contains a !NaN value. ---++++ Median function (MEDIAN) Median function returns !NaN if any array value is !NaN. ---+++ Source extraction * Every source in the input source list will now be included in the output source list, including sources that lie outside of the image. * Extraction now fails if the image has non-square pixels. ---+++ Images ---++++ Display * Display now has the SLS and the HSV color tables as known from ds9. It is also possible to change the color table according to the square and a power algorithm (Linear Scaling, Sqrt, Log and Histogram equalisation were already available) * Display can use Regions of interest. * added method ==getROI([row1, column1, row2, column2, ...])== to Display which returns a Selection. * added ==getFlagFromROI([row1, column1, row2, column2, ...])== to Display which returns a flag where all pixels are masked out, expect the pixels in the Selection. * added ==getFlagFromROI(selection)== to Display which returns a flag where all pixels are masked out, expect the pixels in the Selection. * Example: ==roi = d.getROI([276, 269, 387, 312, 326, 436, 186, 428, 189, 318])== * A preview is given when a fits file with a !SimpleImage is selected in the Navigator. * In the _Image Viewer_ section of the HIPE Preferences window you can now set the default algorithm and intensity of the colour table, as already possible by right-clicking on an image and selecting _Edit colours_. ---++++ Analysis * A lot of Tasks were incorrectly available twice. The tasknames ending in _-Task_ are not longer available. For example: ==cropTask== is not longer available, ==crop== should be used instead. ---+++ Spectra ---++++ Display * Improved information displayed in the status bar when hovering mouse. * Channel / flag information to the display. * Tooltips on FilterPanel, Table headers and cells. * Middle-click copies content of table cells. * Relativistic frequency-to-velocity transform on auxiliary axis. ---++++ Analysis * New task, ExportSpectrumToASCII, that allows to export spectra to ASCII. * TruncatedLinearInterpolator for spectrum interpolation added to the toolbox. * SpectrumStatistics: ignoreNaN's by default now set to true. * Fold task can now deal with missing segments when used with overwrite=False. * Checks in Extract task, which used to throw an exception when dimensionality >1, relaxed for cubes. * ResampleFrequency: Problem that gave different results for resampling a Spectrum1d and resampling a SimpleSpectralCube is fixed. ---+++ Data cubes ---++++ Display * Cube Spectrum Analysis Toolbox (CSAT) * Main interface * You can define a "reference spaxel" for the real time spectrum viewer. * WCS viewer section corrected for the WCS with imageindex section. * Single spectrum extraction * You can extract many single spaxel spectra and save them to HIPE or to separates FITS files. * You can clear selections. * Region spectrum extraction * Added polygon and ellipse selection modes, with the ability to move and resize * You can see the spaxel effectively selected. * Comparison cube * With this tool you can compare directly the spectra of two different cubes. * Spectrum Explorer * Added scroll bar in the panel for SpectralCubes. * In the single spaxel selection mode the color of the rectangle correspond the color of the spectrum in the plotter ---++++ Analysis * Region specrum extraction * Added Sum and Median in addition to Average in the graphical interface. * Line intensity map * Added the Voigt profile, in automatic mode. ---++++ Creation * Cubes created using !NearestNeighbourProjectionTask now contain unit and metadata information. * !NearestNeighbourProjectionTask is now capable of regridding a list of spectral cubes. * !NearestNeighbourProjectionTask is now capable of creating a default target grid (world coordinate system) from a list of target grids. ---+++ Interoperation with the Herschel Science Archive * Support for the new HSA data format (top directory named after HSA request ID and visible XML file at top level). * Support for importing data in HSA format without unpacking. ---+++ Calibration sources ---+++ Products and datasets ---++++ !TableDataset Column name duplicity is no longer allowed. ---+++ Product Access Layer * Improved the GUI editor for local stores. * The result table of the PAL browser is now customisable. * Added task ==localStoreCopier== to create a compressed copy of a Local Store. ---++++ Local Store * You can rename local stores newly created with HCSS 4.0 by renaming their pool directory. For older local stores you need to run ==PoolSchemaEvoler.evolve()== to update them to the new format before using this feature. * It is now possible to turn off the backup when re-indexing. Use the property ==hcss.ia.pal.pool.lstore.index.backup== to control this feature. ---++++ Cached Pool * Caches now have a default size limit of 10 GB. Instead of continuing to grow, if more than 10 GB of data is retrieved into a cache, products that have been least recently used will be deleted. If this happens and the product is accessed again, it will be retrieved from the remote pool again. The 10 GB size limit is approximate and applies to the downloaded products only. Metadata and other small items are cached as well, but these are not taken into account for this limit. * Several issues on Windows have been fixed, mostly related to removing items from the cache. ---++++ !HsaPool * Changes related to PAL interfaces modifications (added new methods, removed deprecated methods). * Query performance improvements: Changed subclasses retrieval mechanism. * Support for importing observations using XML files located under <tt>.exported</tt> directory and root directory. ---++++ View: Save Products to Pool With the HIPE view _Save Products to Pool_ you can save products to a PAL product pool. New product pools (local stores) can now be created directly from this view. ---++++ HSA Access: HIPE Login The HIPE login panel on the HSA view now encrypts the username and password, before sending them over the network (using HTTPS). ---+++ Tasks * All tasks can now be started and cleared with the HIPE toolbar buttons. * Improved default layout of task GUIs and behaviour of parameter editors. * Background tasks appear in the active list of tasks and send messages to the console when starting and finishing. ---+++ Quality control * New option in the _File_ menu to get all the observations for an operational day. Optionally you can filter these observations by instrument. * New option in the _Tools_ menu to create the summary for a set of observations in one single step. * Date added to the time column into the logs table. ---+++ Systematic product generation * New observations will contain only the pointing samples covered by the observation time range. This means that, for each observation, a new !PointingProduct will be created containing only a copy of the datasets identified with the same obsid. * Attached the Horizons, Orbit Events and Satellite Housekeeping new products to the auxiliary context. * Quality tickets include the instrument mode for every observation. ---+++ Data input-output * simpleFitsWriter no longer stores the path of the FITS file when zipping. * Improved product recognition in fitsReader. * Improved GUI of fitsReader and simpleAsciiTableReader: they now automatically fill the output variable name based on the input file name. * getObservation has a new optional _instrument_ argument which you can use in parallel mode to specify the instrument for which to retrieve the data. * Extended exportPalToUfDir task to allow renaming/moving * Import View improvements: redesigned user interface, supports importing zipped pools (and renaming, if necessary) and tarred (and gzipped) HSA data, importing observations to new pools, and importing each observation to a new pool. * Export View improvements: redesigned user interface, supports exporting zipped pools. It currently supports exporting HSA data as loose files. ---+++ Installer * The Versant setup screen in the advanced installation mode has been added to the Mac OS X platform. * Removed the option to cancel an installation once the download phase starts (when the installer connects to the repository to download the missing components). ---+++ Documentation * What's New * The _What's New_ document is now web-based. * Quick Start Guide * Updated most screenshots. * Scripting and Data Mining * Expanded section on Cached Pool. * Added more links to numeric functions in the URM. * Removed section on JIDE view in HIPE. * Data Analysis Guide * Fully updated data input/output chapter. * Fully updated plot chapter. * Updated section on Spectrum Explorer. * Added links to spectrum arithmetics tasks in URM. * HIPE Owner's Guide * Fully updated text and screenshots of views and perspectives. * Updated location of dump files. * Expanded section on the HIPE Help System. * Added table on mouse events in HIPE. * Added section on HIPE toolbar and status bar. ---++ HIFI ---+++Configuration * Property Configuration * Using the build installer for all instruments will mean that the HIFI properties are correct and the HIFI pipeline will run out-of-the-box. * No need to set the HCSS_PROPS environment variable or use the ==hifi-new.props== or ==hifi-installer-new.props== files. * Properties files replaced by ==hifi.dp.properties==, which is automatically picked up by the system. ---+++ hifiPipelineTask ---+++ HRS Pipeline ---+++ WBS Pipeline ---+++ Level 1 Pipeline * !DoVelocityCorrection: * Use of Horizons ephemeris data for transforming to the rest frame of the source for SSO's introduced, using naifid to identify SSO's * Improvements to transform to the rest frame of the source for fixed target when using the task interactively * 'freqFrame' meta data parameter, which informs what frame the frequencies are expressed in, is introduced * !DoRadialVelocity: * No longer a part of the automatic pipeline but supplied for interactive use, it now uses the auxiliary product rather than rely on the velocity column in the datasets. * Compliance with !DoVelocityCorrection in the sense that the LO Frequency is also adjusted and the same meta data parameters are set. * Fix of the problem associated with the interpolations schemes used in the !DoOffSubtract and the !DoFluxHotCold. Use of the linear interpolation scheme had lead to a "noise explosion" in the pipeline. A truncated linear interpolation scheme is introduced which uses nearest neighbour in case measurements are not bracketed and linear in case they are. The pipeline is now configured to use: * nearest neighbour interpolation for !FSwitch and !LoadChop in !DoOffSubtract * truncated linear interpolation for all the other observing modes in !DoOffSubtract (actually only OTF). * truncated linear interpolation for all the observing modes in !DoFluxHotCold. ---+++ Level 2 Pipeline * Fixes for cross maps: the points in a cross map are no longer averaged into one point. * Names of map contexts at level 2 changed to fit HCSS convention. ---+++ Products * Hot and cold load (hc) observations are boxed in the HTP in the same way that datasets are ---+++ !DoDeconvolution * Introduction of an 'expert' panel in the GUI * The following parameters are found in the 'expert' panel, which is on the right hand side: * use_entropy * lambda1_channels * lamda2_gains * cont_offset * Variable name for interim_output * Plotting of chi^2 against iteration number is possible from the expert panel. On the same plot the channel rms'es, the channel and gain inverse entropy are plotted as well. * It is also possible to plot SSB solution against frequency for each iteration of the deconvolution. There are four plots to a page. * Bin size is automatically set by !DoDeconvolution to be the same as the sampling of the DSB data, the information is taken from the metadata "frequencyWidth". * Only the USB is used as input data to avoid the smearing of narrow lines that can arise after averaging LSB and USB data. ---+++ !DoGridding ---+++ !FitHifiFringe * A number of bug fixes have made !FitHifiFringe more stable in HIPE 4. In case a problem still occurs, a warning message is given, but the task will not crash. * Also, the sine wave fitting parameters applied to the data are now stored in a table in the !ObservationContext (click on 'calibration" and then "pipeline-out') ---+++ !HiClass export tool ---+++ Flags ---+++ !SpectrumExplorer (HIFI specifics) * Faster loading of HifiTimelineProduct with ability to select which datasets to load by displaying summary table information. ---+++ HIFI DP scripts (User tool scripts) * Introduction of !PolarPairs, a class to average and difference H and V polarisations ---+++ Documentation * HIFI User's Manual * Introduction of HIFI Cookbook, as first chapter of HIFI User's Manual * New chapter for !PolarPairs * New chapter describing relative (rms) performance of HRS and WBS * Updates to !HiClass (export to CLASS) chapter * Updates to HIFI pipeline chapter * Bug fixes and improvement of figures in pdf versions of document * HIFI Pipeline Specification Document * "Generic Pipeline" (verrrrry old name for what is now known as Level 1 and Level 2 pipelines) chapter split into sections for Level 1 and Level 2 pipelines * Addition of !DoVelocityCorrection section, rearrangement of names and orders of sections to reflect current status of pipeline * New flow diagrams for levels 1- and 2- pipeline * Introduction of Level 0 pipeline description (incomplete) * Bug fixes and improvement of figures in pdf versions of document ---+++ Other ---++ PACS ---+++ Pipeline ---++++ Photometry * Photometer Flux calibration <br> Calibration products involved in the photometry adjustment are stable.<br> Responsivity determination comes from astronomical sources and gives with the flat an acceptable flux. <br> *Estimated accuracy on point sources ± 5%, estimated accuracy on extended sources ± 10%* <br> ---++++ Spectroscopy * specAddNod is not part of the level 1 chopNod pipeline anymore. <br> Instead, the level1 product is sliced according to nod positions (!SlicingRule "NoddingPosition") and the nod slices are then recombined in !specProject <br> !speccAddNod now works on cubes. * !specCorrectHerschelVelocity can now optionally just compute the S/C velocity without applying it to the wavelengths. * !specAddNodCubes has been improved and now works on contexts with multiple raster positions as well.<br> Also, you no longer have to split the data to raster position before running this task. ---+++ Calibration * Proper treatment of time dependent SIAM calibration product.<br> Now we have a first time dependency with the SIAM. <br> It is important to pass the observation context to getCalTree: <br> <pre> calTree = getCalTree(obs=obs) or calTree = getCalTree(time="20-Apr-2010 12:00:00") </pre> *If this is not done, the latest SIAM will be applied, which will give a 5" offset for observations before OD320* <br> * getCalTree now accepts a !FineTime in addition to a String time representation. * Two new bad pixels identified for the red photometer <br> Analysis based on the pixel noise then individual/neighbour pixel behavior comparison <br> Pixels flagged [4,5] and [3,6] <br> ---++++ Photometry * Blue and green separated in !ArrayInstrument calfile and in the code using it. ---+++ Analysis tools * Ipipe scripts are now part of the HIPE top Menu (pipeline -> PACS) <br> ---++++ Photometry * New Task: !MapIndex2signalCubeTask allows to recreate a signal cube from a !MapIndex. <br> * !PhotProjectPointSource has been extended to accept the same options as !PhotProject <br> * Photometer 2nd level deglitching more memory efficient <br> * A more memory efficient deglitching is available. <br> The map can be devided into a chess-board like pattern and deglitiching may be done tile by tile. <br> This way, less memory is necessary, because the !MapIndex will be calculated only for the <br> currently processed tile. Quicker processing becomes possible by using the !FullMapindex <br> instead of the !SlimMapindex. * !Madmap improvements <br> * Flux correction for scale < 1. A factor of 1/(1/scale)^2 is missing in previous releases when scale < 1. * Huge improvement on memory usage. * Use Mattools to calculate WCS, which got rid of usage of Basic.MEDIAN that causes JVM crashes for large datasets. * The !Madmap output now contains coverage map instead of exposure map. ---++++ Spectroscopy * The Spectrum Explorer now also works with the PACS Frames and !PacsCube products. <br> You can open the Spectrum Explorer in HIPE by right clicking on a variable of such a product <br> and selecting _Spectrum Explorer_. We have extended this tool with additional panels to perform the following tasks: * (De)activate masks. * (De)select complete spaxels in a Frames object (all 16 pixels). * (De)select blocks from the blocktable. Most standard functionalities are available, but selecting data points and creating a user mask <br> does not yet work properly. ---+++ Product changes * PACS Products now contain all available meta data. * Spectrometer pipeline scripts now contain !addSliceMetaData step which adds the meta keywords !aotMode, !lineDescription and !rasterId to the slices and adds three columns to the !MasterBlockTable: !MinWave, !MaxWave and !LineDescription ---++++ Spectroscopy * Reordered the !PacsCube data in the wavelength domain and added !PacsStatus dataset ---+++ Documentation * Updated the PACS Data Reduction Guide, mainly in the spectroscopy chapter. The new version has updated explanations of the spectroscopy pipeline and of the so-called slicing pipeline. ---++ SPIRE ---+++ Data Products * !SpirePreprocessedCube now contains raster/jiggle position data. * Level-1 Spectral Products are now all calibrated using the extended-source flux conversion. * New Level-2 Product introduced for Point-source sparse sampling observations. This Level-2 data product contains the point-source calibrated detectors for the two target (typically the two central) detectors of each Spectrometer array. ---+++ Calibration Products * New calibration product introduced to convert from Level-1 extended-source calibration to Level-2 point source calibration for Point-source sparse sampling observations. * Reference interferograms and Flux Calibration products have been updated to allow for the creation of proper Level-2 Spectral Maps. * Reference interferograms now created such that the interferogram baseline has been removed. ---+++ Preprocessing * Some uplink parameters now copied into the metadata of the !ObservationContext. ---+++ Common Pipeline * Metadata propagated into various level contexts. ---+++ Photometer Pipeline * Added new deglitcher task to remove concurrent glitches (==concurrentGlitchDeglitcher==). Not yet included in the standard pipeline but available to users. * Added new task to detect jumps in bolometer timelines (==signalJumpDetector==). Not yet included in the standard pipeline but available to users. ---+++ Spectrometer Pipeline ---++++ All Pipelines * Order of pipeline processing has changed to improve SLW signal to noise ratio. Telescope/SCAL correction now follows 2<sup>nd</sup>-Level deglitching. * Level-1 and Level-2 spectra data by default now *only* contain spectral information within a range defined by the Spectrometer Band Edge calibration product (see [[http://herschel.esac.esa.int/Docs/SPIRE/html/spire_om.html][SPIRE Observer's Manual]] for spectral ranges).<br/> *NB* To access out-of-band spectral samples, you must use the pipeline in an interactive manner. * Pixel Sizes have been updated for mapping observations based on in-flight calibration data: | *AOT* | *Mode* | *Spatial Sampling* | *Pixel Size (SLW)* | *Pixel Size (SSW)* | | SOF1 | Raster | Sparse | 2 x 35.0" | 2 x 19.0" | | SOF2 | Single/Raster | Intermediate | 1 x 35.0" | 1 x 19.0" | | SOF2 | Single/Raster | Full | 0.5 x 35.0" | 0.5 x 19.0" | ---++++ Sparse Sampling (SOF1) Pipeline. * *Single-pointing* Level-2 now contains a point-source calibrated spectra for the two target (typically the two central) detectors of each Spectrometer array. * *Raster* Level-2 spectral maps are now produced for SOF1 Raster observations. ---++++ Spectral Maps (Raster/Jiggle observations) * !SpirePreprocessCubeTask has added three filtering parameters for specifying which spectra are passed to output: UNVIGNETTED, detectors, and arrayType. These parameters may be used to help with construction of spectral cubes from customized lists of detectors. <hr/> <!-- COMMENT BOX CODE - DO NOT EDIT --> <div id="disqus_thread"></div> <script type="text/javascript"> var disqus_shortname = 'herscheltwiki'; // required: replace example with your forum shortname (function() { var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true; dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js'; (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq); })(); </script> <noscript>Please enable JavaScript to view the <a href="http://disqus.com/?ref_noscript">comments powered by Disqus.</a></noscript> <a href="http://disqus.com" class="dsq-brlink">blog comments powered by <span class="logo-disqus">Disqus</span></a> <!-- END OF COMMENT BOX CODE --> <hr/> <!-- * Set ALLOWTOPICCHANGE = Main.RegisteredUsersGroup, Main.TWikiAdminGroup -->
E
dit
|
A
ttach
|
Watch
|
P
rint version
|
H
istory
: r54
<
r53
<
r52
<
r51
<
r50
|
B
acklinks
|
V
iew topic
|
Ra
w
edit
|
M
ore topic actions
Topic revision: r54 - 2012-10-30
-
DavideRizzo
Public
Log In
Public Web
Create New Topic
Index
Search
Changes
Notifications
Statistics
Preferences
Webs
Public
TWiki