TWiki> Public Web>HipeWhatsNew>HipeWhatsNew13x (revision 50)EditAttach

What's coming up in HIPE 13

PDF Version Click here
Help Interested in what's new for other versions of HIPE? See this page for links to all the What's New documents from HIPE 4.0 onwards.
See also the HIPE known issues and the Data products known issues pages for last-minute information on known problems.

Highlights

  • HIFI
    • Bands 6&7 corrected for electric standing waves.
    • Level 2.5 for point spectra WBS subbands and convert to simple spectrum (the polarisations will still not be averaged).
    • Level 2.5 cubes for maps are now stitched
    • New HIFI beams.

  • PACS
    • Photometer: New maps valid for point source and extended emission analysis: JScanam, UniMap (new!).
    • Spectrometer: Cubes with evenly sampled spectral info.

  • SPIRE
    • Photometer: Better deglitched maps (multi-pass pipeline).
    • Spectrometer: Improved product structure inside observation contexts.

Core system

HIPE

Batch mode

In batch mode, you can use sys.argv to get the input arguments. For example:

import sys
print sys.argv

Then the output of this script would be:

> hipe test.py arg1 arg2 arg3
['/home/user/test.py', 'arg1', 'arg2', 'arg3']

Transitions

In order to give better user feedback on GUI operations, screen transitions have been introduced in the following cases:

  • Switching perspective is performed by fading to the new perspective.
  • Navigation between pages in the welcome perspective is also done by fading to the new page.
  • Maximizing or restoring the data viewer within the observation context viewer is shown with an animation.
  • Maximizing or restoring the browse image of the observation context viewer is animated too.

Outine

  • The Outline will show a small image for applicable variables like spectra, simple images, etc. when selected in the Variables view.
  • When selecting String variables, both the string contents and a literal to use it in a script will be shown in the Outline.
  • When selecting Unit variables, specific information about the unit is shown.

Editors & Viewers

  • Backups for new files are automatically created, and reopened in a subsequent session upon crashes like a power cut. If the session terminates gracefully, the backup files are deleted.
  • Editing metadata and dataset cells in the Product or Observation viewers keeps the result in memory, and the table is refreshed accordingly with the new value.
  • Animations when maximizing or restoring parts of the viewer will be shown (see Transitions).
  • The browse image of the Observation viewer can be opened with the Image viewer by means of Right click > Open in image viewer.
  • The ArrayDataset viewer shows the unit when specified.
  • Column objects from a TableDataset have now a custom viewer.

Status bar

From the Preferences > Data Access > MyHSA > Advanced panel, the user can:

  • Show the MyHSA on-line/off-line checkbox in the status bar.
  • Hide the MyHSA on-line/off-line checkbox in the status bar.

This label provides a more prominent location reflecting the current connection status of MyHSA, irrespective of what view the user is currently using. It also offers the ability to update this status, the status is synced with that shown in the product browser and vice versa.

File handling

  • When saving a file of a known type with the file dialogue, the extension is automatically added to the file name if not explicitly done.
  • The filter for file types in the Navigator view is now more visible. In addition, it includes all kinds of defined file types.

Rescue mechanism

If HIPE gets in a non responding state for a period of time, it is automatically detected so that a popup window asks to restart it in an ordered way. This means all unsaved editors will be saved, and a new fresh HIPE session with the state of the frozen one will be opened.

The time intervals to run this check and to consider the application as not responding can be configured in Preferences > General > Rescue, where this mechanism can also be disabled altogether if you prefer so.

Miscellany

  • The Help > Check Java platform option shows the running Java version.
  • The Login panel is shown back at the top left corner of the Data Access perspective, and it has been simplified.

Plotting

  • PlotXY no longer has an explicit way of forcing a batch of plotting statements. This is now done automatically and the script developer must not do anything special to take advantage of it.

TablePlotter

OverPlotter

Astronomical utilities

  • Gyro-based attitude reconstruction, see pdf.

Numeric routines

Data fitting

Data interpolation

Random numbers

Statistics

Trend analysis

Mapping

  • Mapping uses subdir per HIPE session for temp files

Other functions

Source extraction

Images

Display

The following changes are made to the Display class:

  • Replaced setCutLevels() by calcCutLevels()
  • Removed setCutLevels(double,double)
  • Renamed setCutLevels(double) to setCutLevelsPercentage(double)

  • The images can not only be flipped around the y-axis, but also around the x-, xy- and yx-axes. It is also possible to combine these flip-options : for example : y- and xy.
  • The annotation in Display are totally reworked. All old annotation methods are deprecated, the new one end with Annotation. For example : addEllipse becomes addEllipseAnnotation.

A lot of fixes in Display:

  • Display now displays images with some extremely high or low values correctly.
  • It is now possible to extract the current layer from a SimpleImage or from a SimpleCube with only one layer.
  • Display can now work with more than one extension.
  • The zoom in / out was by a factor of 4 instead of 2.
  • The Wcs viewer only shows the important wcs keywords now.

Analysis

Bugfixes :

  • Mosaics close to the north or south pole missed a part of the sky.
  • Aperture photometry tasks have clear description on how the background subtracted flux is calculated.
  • Bugfixes in aperture photometry tasks.
  • You can choose to add an input parameter to annularSkyAperturePhotometry for choosing the step size of curve-of-growth output.

Spectra

Display

* The mosaic view of the Spectrum Explorer shows axes coordinates in a more convenient way, with less digits.

Analysis

  • multiFit task now accepts limits for the model parameters.

Data cubes

Display

  • Disable the selection tools when some other data cube than the flux is shown.
  • Fixed bug in which the upper and right rows and columns were not included in the selection for the cropCube task.

Analysis

  • Moment algorithm in computeVelocityMap improved with 'line strength' weighting. Especially dispersion is more accurate.
  • Show proper model cube when absorption line fitting is done in computeVelocityMap.
  • More metadata in extractRegionSpectrum.

Herschel Science Archive

=HSAWritePool= now supports the removal of a list of tags.

Calibration sources

Products and datasets

A new Auxiliary Operational Day Summary Product which can be retrieved using the tag LATEST_OPERATIONAL_DAY_SUMMARY_PRODUCT. This product contains one dataset with the operational day number, start and end times. Useful methods exist for querying the data as documented here. A new Auxiliary Operational Day Summary Product which can be retrieved using the tag LATEST_OPERATIONAL_DAY_SUMMARY_PRODUCT. This product contains one dataset with the operational day number, start and end times. Useful methods exist for querying the data as documented here: OperationalDaySummaryProduct

A new utility method in the Product class for querying a products duration.

ParameterFactory added to easily create meta data parameters.

Observation context

Pointing Product

Product Access Layer

Tasks

  • Conversion tool to adapt multi-output tasks to new style in off-line User scripts.
  • Warning, important __list__ argument removed: if your task call still has it, remove it.
  • Preference "Generate task calls that returns full set of outputs" removed: tasks GUIs now always generate a call with the expanded list of outputs assigned to.
  • Task status messages appear in the status bar
  • convertUnits : It can convert units from double parameters, array datasets and columns, plus units from table datasets or products, by allowing to select the target dataset or column.

Quality control

  • It is possible to create Quality Control summaries for several instruments in one go.
  • The QCR Browser is more interactive, as it is no longer blocked when performing an operation. This way, tasks can be performed in parallel on different windows.
  • Panels in the QCR Display are ordered in a more sensible way.
  • The OD is shown in the QCR Display when available.
  • The operator can remove comments from several observations in a single operation.
  • The report showing the result of an ingestion or removal operation can be saved to a text file.
  • Specifying a ticket number for several observations is now possible.

Systematic product generation

Data input-output

  • getObservation (getObs task):
    • Now includes the possibility to retrieve observation using the SPG version that created the Observation (parameter spgVersion).
    • The task path and poolLocation parameters have been consolidated. See up to date Task definition with examples.
  • saveObservation:
    • Improved documentation
  • Reading of ALMA/CASA cubes directly into HIPE and the Spectrum Explorer: fitsReader and importSpectralCube
  • asciiTableReader:
    • Improved GUI
    • Improved error reporting for IPAC tables
  • fitsReader:
    • Identify velocity cubes (CTYPE3 = VELO-LSR) as spectralCubes
    • Identify ALMA/CASA cubes as spectralCubes
  • "Export Herschel data from HIPE" view:
    • automatically add the proper extension to output files
    • Improved GUI: changed default tab

Virtual Observatory

  • The link to VOPlot in the External Tools sub-menu will be removed and substituted by Topcat.
  • Improved messages when failing to launch external applications
  • When you send a table to Topcat it returns a table (before it returned a product)

Documentation

HIFI

HIFI Software Configuration and Properties

HIFI Calibration

The calibration tree released together with HIPE 13 contains the following updates:

  • A list of optimised channel flags for spectral scans obsids, that will be honoured by the deconvolution task and result in spur-free level 2.5 data in this mode. This is not yet available for all spectral scan obsids though, but will be by HIPE 14.
  • Solution tables for Electrical Standing Wave correction in the bands 6 and 7. This allows for a correction of this affect in the pipeline itself. Note that this is not a guarantee that the baseline artefact will be perfectly removed but it will provide the best correction available to data. It is then possible to further optimise the correction - see the dedicated sections in the DRG.
  • Updated beam efficiencies and 2D beam models

HIFI Pipeline

hifiPipeline task

Level 0 Pipeline

WBS Pipeline

HRS Pipeline

Level 1 Pipeline

  • The Electrical Standing Wave correction is now automatically performed on the level 1 data, based on optimum correction solutions computed by the HIFI team and stored in the calibration tree.

Level 2 Pipeline

  • The sideband ratio correction is now taking into account the possible IF-dependency of the sideband ratio over the IF. It uses the IF-dependence information as given in the calibration tree. As of now, only the lower end of band 2a makes such an assumption. The presence of an IF-dependency will result in additional meta-data usbGain_n or lsbGain_n, which are the coefficients used to described the polynomial approximation of the gain dependence. Coefficients with index _0 simply correspond to the value of the constant, scalar, sideband ratio we used so far
  • A new task will automatically check for platforming in the WBS data and report any detection with a dedicated quality flag
  • A new task will create spectra for the OFF position data calibrated into the Ta* scale, on both USB and LSB frequency scales. For spectral scans, these can then be fed into deconvolution to create a deconvolved spectrum of the OFF position (option use_reference)
  • The spur detection is now handed over to the direct application of an a priori spur table stored in the calibration tree. Obsids for which no entry exist will not have any spurs flags. This calibration tree is currently being populated and already covers hundreds of observations (mostly spectral scans) in HIPE 13. For those spectral scans, the users can expected improved quality of the deconvolved spectra at level 2.5

Level 2.5 Pipeline

  • Mapping mode: The cubes produced automatically by the pipeline (Level 2.5) (doGridding task), and consequently by hifiPipeline if you choose to run it again, will stitch the WBS subbands and create one final cube (with the index '_1'). You will not see individual cubes per subbands except for the HRS.
  • Point mode: Level 2 HTPs , stitched, are forming the Level 2.5 products for this mode.
  • A large number of statistics (primarily noise and mean) is performed both on the level 2 and level 2.5 products, as well as on an equivalent products converted to the Tmb scale with H and V merging (this latter is then removed from the observation context and only the statistics is stored). This computation is controlled by the option computeRms. It is only set to True when run for the bulk reprocessing, but ignored by default when the level 2.5 pipeline is called interactively (this computation can be extremely time consumming for maps and spectral scans)

HIFI Products

HIFI Spectrum Toolbox

Spectrum Explorer (HIFI specifics)

Spectrum tasks (HIFI specifics)

  • The !PolarPair class has been removed. Instead the !polarPair task should be used. This should only affect scripts written prior to 2013.

HIFI Data Processing Tools

Standing Wave Removal

  • Updates to default band-dependent period characteristics in task fitHifiFringe :
    • Band 1 change nfringe 2->3, endPeriod 120->200
    • Band 3 change typicalPeriod 620->95
    • Band 4 change typicalPeriod 620->95
    • Band 6 change startPeriod 100->80, typicalPeriod 320->600
    • Band 7 change startPeriod 100->80, typicalPeriod 320->625
    • Band 2,5 no change
  • swTable had wrong unit for amplitude (kHz instead of K). This is now fixed.
  • New output variable swTable was created to enable machine-readable output of fitted frequencies and amplitudes.
  • Output for fitHifiFringe now contains corrected data and summary table.
  • New input parameters "cal" and "saveInput" was created in fitHifiFringe to extract user input parameters into the calTree for later use on demand.
  • When a large portion of channels is flagged as invalid, i.e. when not enough channels can be fitted, fitHifiFringe will raise a comment that it is skipping the spectrum and then it will continue to the next one.

Baseline Removal

  • In order to prevent fitBaseline from crashing when 'useWbs' is set to 'True', especially when doglue is also set to 'True', the task will automatically override doglue to 'False'.

Electrical Standing Wave correction in HEB bands

  • The task doHebCorrection is now part of the default pipeline, with several new options compared to its predecessor hebCorrection.

Deconvolution

  • By default, the option chosen for spur_rejection will be based on the availability of not of optimised flag tables for a given obsid (see also the calibration section). When no table exists, the option is enfored to REJECT_SCANS_WITH_SPURS. When tables exist, the option is enfored to DO_NOT_REJECT_SCANS_WITH_SPURS
  • A new option called enforce_spur_rejection is however made available to help control the existing spur_rejection option.
  • A new option use_reference indicates that the deconvolution should be made on the OFF position spectra computed with HIPE 13
  • doDeconvolution has also been updated so it can take into account the IF-dependence of sideband ratio and use this a priori knowledge as preset to build the single sideband solution

convertK2Jy

  • with the availability of the new beam efficiencies and beam model, the task has been upgraded in order to make best use of the detailed beam knowledge. A new option hifiBeam, set to True by default, will use the fully-detailled HIFI beam to compute the coupling factor involved in the temperature to flux conversion for sources of non-zero size.
  • for non-point-like sources, the task now offers disk or Gaussian source models (option shape)

Spectral Cube creation

flagTool

  • Interactive (suitable for batch processing) and Expert modes now available.
  • maskTables: a new variable to manage Linemasks and Rowmasks files.

identifyLines

  • IdentifyLines is a new task that allows you to identify lines in your spectrum and then run a comparison of known lines with a Linelist.

HiClass export tool

Documentation

  • Updates to the HIFI Data Reduction Guide:
    • A new chapter on The HIFI line identification tool (identifyLines task) is now available.
    • Updates to the chapter How to add and remove flags, section Setting flags interactively for many spectra (flagTool task).
    • Updates to the chapter Sideband deconvolution, section Basic strategy for running the deconvolution tool (doDeconvolution task).
    • Updates to the chapter Standing Wave Removal (fitHifiFringe task) - note also: new syntax for startPeriod, endPeriod, typicalPeriod, and subBase.
    • Updates to the chapter How to make a spectral cube to reflect changes in the level 2.5.

PACS

Pipeline

Photometry

  • Unimap: a new Generalized Least Square (GLS) mapmaker is introduced into the Standard Product Generator. Unimap is a MATLAB algorithm that is spawned from HCSS. Level2.5 and Level3 products generated by Unimap replace MadMap maps for PG13.
  • Level2 products: the quality of the High-pass Filtered Level2 products (HPPPMAP) is improved by adding the scanamorphosMaskLongTermGlitches() and the scanamorphosBaselinePreprocessing() tasks, which are capable to detect jumps and to correct drift on the detector timelines.
  • Level2.5 products: Unimap Level2.5 contexts contain the same types of maps generated by Madmap (nave, GLS and Post-GLS), nevertheless they are differently arranged within the context (see Product changes).
  • Level3 products: High-pass Filtered maps (HPPHPFMAP) are removed from Level3 products, having finally JScanam and Unimap maps. Level3 mosaics are created by using scanamorphosAlignImages(), a new task that links the flux by exploiting the overlapping regions between images.
  • Ipipe scripts: MadMap pipeline is still available as interactive script in the Pipelines menu. The interactive script scanmpa_Unimap allows the users to run Unimap by tuning some parameters. It uses the new task runUnimap(), that calls the Unimap release available within the user’s computer.
  • Harmonisation of map pixel sizes: at any Level, maps have the following pixel sizes:
    • 3.2 arcsec for the red channel, Prime and Parallel Modes;
    • 1.6 arcsec for the blue channel, Prime Mode;
    • 3.2 arcsec for the blue channel, Parallel Mode.
  • Correction of timing shifts: frame pixels are not simultaneously read, but in SPG12 the frames are associated to a single readout time. That results in a time shift that can correspond up to 2 arcsec delay for fast scanning. This effect is corrected by setting the parameter correctReadoutTimes=True by default in every task that is affected by time offsets

Spectroscopy

  • Chop/nod SPG pipeline now uses the background normalisation method instead of the calibration blocks for the flux calibration.
  • Drizzle now also done for the unchopped line observations.
  • Interpolated cubes are computed for pointed observations with the new task specInterpolate.
  • Drizzled, projected or interpolated cubes are converted to an equidistant wavelength grid cube with complete standard WCS using the new task specRegridWavelength. These equidistant products are stored next to the non-regular wavelength grid products in the level-2 context.
  • Tables containing all the rebinned cubes per camera are produced with the new task pacsSpecCubeToTable.

Analysis tools

Photometry

Spectroscopy

  • New long/short term Transient Correction algorithm and script.
  • Extra options to wavelengthGrid task to make regular grids with either the mean -, a fraction of the minimum -, or a specified bin size.
  • specSubtractOffPosition can now also combine off-positions using several algorithms.
  • An image can be used as model for extended to point correction task.
  • Improvements to Pointing Offset Correction.
  • specDiffChop now also works on calibration blocks.
  • slicedSummary works now also on rebinned/projected cubes.
  • The footprint viewer is now a real viewer, which can be opened with the "Open with" menu on a product.
  • Interactive script improvements:
    • Made more user friendly, clear and with less verbose info.
    • Improvements of diagnostic plots.
    • New scripts for fitting mapping observations (from rebinned and projected cubes).
    • A new script to fit single pointings on interpolated cubes.
    • A new script to convolve 2 spectral images to the longer wavelength resolution.
    • Added a capability to overlay the footprint on an image.

Product changes

  • The meta keywords of all products have been improved. Especially every meta keyword has now a 8 character FITS equivalent, so that the META_XX keywords will not appear anymore in the FITS files for any of the important keywords.
  • The ra/dec of the level-1/2 products now reflect the actual average center of the particular cube/image containing these keywords. Only raNominal/decNominal contain the requested pointing coordinates.
  • Simplified the structure of the level 0-2 data in the observation context (fewer layers).
  • Moved the observation summary to the top level of the observation context and improved its contents.
  • Improved browse images, replaced browse product by links to the level-2 products.

Photometry

  • The naming convention for Unimap Level2.5/3 is HPPUNIMAP[B][R], for the blue and red channel, respectively.
  • A single context is released for Unimap Level2.5. The Unimap context contains the following products:
    • image is the Weighted-GLS map. Generally, it is the most reliable product;
    • pgls is the Post-GLS map;
    • gls is the GLS map;
    • naive is the Nave map;
    • error is that is the standard deviation map. It can be properly assumed as the error map only if associated to the Nave map.

Spectroscopy

  • Drizzled cubes are also provided for unchopped observations.
  • We provide interpolated cubes for pointed observations (with 4.7" spaxel size) in the context under HPS3DIR and HPS3DIB.
  • New level-2 products for easier handling outside of HIPE:
    • Single table with all the rebinned cubes (PacsSpecTable product under HPSTBRR and HPSTBRB).
    • Many cubes are also regridded to an equidistant wavelength grid (store in the context under HPS3DEQ*), which are easier to view in viewers like DS9, since now also the wavelength is part of the standard WCS.

Documentation

  • Improved the table of contents of the PACS User Reference Manual. Created a new content for the runUnimap() task.
  • Detailed documentation of Unimap within the PACS Data Reduction.
  • Created PACS Products Explained document to describe and explain PACS products, from ObservationContext to the final maps and cubes
  • Cookbooks to guide users through certain common tasks done with PACS data in HIPE will be released, beginning in the spring of 2015

SPIRE

Calibration Products

Note that the size of the calibration tree has increased by 120 MB, mainly due to the addition of the finely sampled (1") photometer beam maps.

Photometer

  • The beam profile maps are improved using the shadow observations of Neptune (i.e. better characterising and removing the background).
  • The corresponding beam areas given in the metadata of the beam products are updated accordingly.
  • The radial beam calibration product has been updated to match the new beams.
  • All the colour correction products have been updated following the beam updates.
  • In order to improve the operation of the source extractors, the beam profiles are adjusted to be relevant for a 1 Jy source - previously the peak of the profile was fixed to 1 Jy, but the new profiles integrate to 1 Jy. In addition, the map consists of an odd number of pixels so that the beam centre is precisely at the centre of the central pixel.
  • A new "fine" beam map is provided for each band with 1" pixels. The "nominal" maps are also available with pixel sizes of 6", 10" and 14" for PSW, PMW and PLW.
  • The K4 parameters in the Flux Conversion product metadata are updated (very small change to K4E)
  • The Planck HFI colour correction table has been updated following the beam updates (very small change).
  • The parameters in the temperature drift calibration product that are used to perform cooler burp correction have been updated to improve the correction.

Spectrometer

  • The non-linearity product has been updated to better fit observations made with low subktemp at the beginning of each pair of FTS observing days (i.e. the observations following the cooler recycle). This results in a much improved SLWC3 spectral shape.
  • The other RSRF, telescope model and flux conversion products were updated following the non-linearity update.
  • The PhaseCorrLim product was updated to correct the metadata - no effect on the data.
  • The TempDriftCorr product has been removed as it is not used by the pipeline.

Calibration Framework

Preprocessing

Engineering Conversion

Common Pipeline

Photometer Pipeline

  • The new 2-Pass Pipeline has replaced the standard pipelines for Large, Small and Parallel modes. The 2-Pass Pipeline runs iteratively on the data to produce optimally cleaned timelines. The first pass is run with minimal tasks (e.g. no FT tasks). Glitches are detected and flagged using 2nd level deglitching outlier detection. The full pipeline is then run using all the standard tasks.
  • New useful scripts to allow users to reconstruct the beam model and all calibration parameters (Photometer Calibration scripts) for both point, extended and semi-extended sources.
  • New SPG map merging script utilising the 2-pass pipeline to produce Level 2.5 products. All parallel mode observations are reprocessed together as pairs at the Level 2.5 stage. After the level 2.5 stage all the Level 0.5, Level 1,Level 2, Level 2.5 data are updated with the improved quality data.
  • Optimized parameters for Timeline Fitter in photometry script. Background annulus changed for faster run time.

Spectrometer Pipeline

  • Update to the naming within the Observation Context to make it clearer which products are spectra/cubes and pointsource/extended calibrated.
  • Moved all of the final spectral products into the Level-2 context. Previously the extended calibrated spectra were in the Level-1 context, but now both point source and extended calibrated data are in the Level-2 context.
  • Introduced a new spectrum2d product for mapping observations to contain all of the spectra before they are gridded into a regular spectral cube. Previously these spectra were saved in individual products for each jiggle/raster position. The new product is added to the Level-2 context.
  • A new task is added to perform spectral apodization. This step was previously carried out with several tasks to take the inverse Fourier transform and apply the apodization on the interferogram. The new task works directly on the spectra.
  • Quality control parameters related to the unaveraged FTS scans are now written back into the interferogram products in the Level-1 context. Previously these metadata items were lost after averaging the spectra over all of the scans.
  • New metadata parameters to contain known pointing offsets from the commanded position have been added to the Level-2 spectra.
  • A limit has been placed on the spectral resolution allowed as input when a spectral mapping observation is gridded into a cube. This eliminates spectra that were clipped, and improves the quality of the maps.

Interactive Analysis and Tools

General

Photometer

  • Significant improvements to the bolometer finder tool have been made.

Spectrometer

  • Minor updates, bug fixes, and improved documentation for the Semi-extended correction task.
  • Updates to the task to take synthetic photometry of FTS spectra for comparison with the SPIRE photometer.
  • New GUI interface for the script that determines the noise level in a spectrum.
  • Small corrections to the useful scripts.

Documentation

Release Notes

* Full list of tickets implemented: here.


blog comments powered by Disqus

-- AlvarGarcia - 15 Jan 2014

Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf Guide_gyro_recon_issue1.0.pdf r1 manage 837.1 K 2014-12-31 - 10:09 JavierDiaz pointing refinements
Edit | Attach | Watch | Print version | History: r68 | r52 < r51 < r50 < r49 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r50 - 2015-01-21 - SylvieFBeaulieu
 
This site is powered by the TWiki collaboration platform Powered by Perl