Public Web>HipeWhatsNew>HipeWhatsNew14x (revision 25)~~EditAttach~~

What's coming up in HIPE 14

PDF Version	Click here

Interested in what's new for other versions of HIPE? See this page for links to all the What's New documents from HIPE 4.0 onwards.

Interested in what's new for other versions of HIPE? See this page for links to all the What's New documents from HIPE 4.0 onwards.

See also the HIPE known issues and the Data products known issues pages for last-minute information on known problems.

What's coming up in HIPE 14

Planned work

HIPE

* TBD

HIFI

* TBD

PACS

* TBD

SPIRE

* TBD

Highlights

HIPE

HIFI

PACS

SPIRE

FTS
- Two new tasks and corresponding calibration products have been introduced for SPIRE FTS data. One task corrects for artefacts in low resolution point source calibrated spectra. The other corrects the level of all extended source calibrated data in or to get better agreement between the SPIRE Spectrometer and Photometer.

Core system

HIPE

Editors & Viewers

Default editor

The default editor for every type of data, which is the one used when double clicking on a variable, is preserved regardless the editors opened through the Open With menu.

The previous behaviour, which is to use the last opened editor as the default, can still be set, by means of a new preference in the Editors & Viewers category.

Recent open files

The size of the recently opened files list, which can be accessed through the File menu, is now configurable through preferences.

Metadata tables

A new option in the popup menu of metadata tables allows copying cell values in other way than with the usual key combinations (Ctrl+C or Cmd+C).

Array data viewer

The array data viewer shows tooltips over the spinners (above the data table) for informing about the array dimensions.

Quick Analysis

There is a new perspective in HIPE, which is accessible by clicking on the magnifying glass at the top right toolbar, for quickly inspecting scientific data from the Herschel Science Archive.

Observations can be searched by target name, observation id, operational day or Ra/Dec.

When clicking on an observation, its standalone browse products are opened in the editor area for quick analysis, by means of the image, spectra or cube viewers.

Opening the observation context instead is also possible, either with the popup menu or by configuring so as the default behaviour, in the Quick Analysis preferences category.

Command line

It is now possible to pass arguments to HIPE from the command line, which will be tried to be interpreted as files to be opened with the proper viewers. For example:

bash> hipe myImage.jpg cube.fits resultTable.csv

would open HIPE with three viewers: one for the image, a spectrum explorer for the cube, and a dataset viewer for the CSV table.

Note:

Passing a Jython script as the first argument will still behave as running HIPE in batch mode:

bash> hipe myScript.py arg1 arg2

would not open the HIPE window, but execute the script with the given arguments passed to it.

Console

There are interesting enhancements in the Console view with HIPE 14.0:

Text edit options (cut/copy/paste) are available not only by the usual key combinations, but also with the popup menu.
Autocompletion through the tab key has been tremendously improved in this release, making Jython coder's life much easier.
The console built-in help, shown with help, is now more practical and understandable.

Plotting

TablePlotter

OverPlotter

Dataset viewer

Astronomical utilities

Numeric routines

Data fitting

Data interpolation

Random numbers

Statistics

Trend analysis

Mapping

Other functions

Source extraction

Improved error reporting for bad inputs.

Images

When requesting the unit, the FITS compatible unit is returned.
The naxis keywords are not longer duplicated in the fits header.
The Wcs ia added to all datasets of the images (like the stddev, ...).

Display

Added shortcuts to some of the task directly in the image viewer.

Analysis

Spectra

Display

Analysis

Data cubes

When requesting the unit, the FITS compatible unit is returned.
The naxis keywords are not longer duplicated in the fits header.
The Wcs ia added to all datasets of the images (like the stddev, ...).

Display

Added shortcuts to some of the task directly in the image viewer.

Analysis

Herschel Science Archive

Calibration sources

Products and datasets

A new Auxiliary Observation Summary Product which can be retrieved using the tag LATEST_OBSERVATION_SUMMARY_PRODUCT. This product contains one dataset with a summary of information for each observation. Useful methods exist for querying the data as documented here: ObservationSummaryProduct

Observation context

A new Calibration Table Product exists under the Auxiliary Context in each Observation Context. This product contains calibration tables data for the given instrument including exactly how the hardware was configured, hard limits, soft limits etc. This provides valuable information for the long term completeness of an observation. Useful methods exist for querying the table data as documented here: SimpleCalTableProduct

Product Access Layer

Tasks

Improved history recording so that it can be added to the sub-products of a context.

Quality control

Reorganization of quality flags as private or public for HCSS, HIFI, PACS and SPIRE Quality Flags. This ensures that only the flags useful for the assessment of the science by the user (the public ones) make its way to the quality summary.All the information from the quality flags remain at the quality product, for internal assessment or history.

In the quality report, a new "Ïmpact" column in the "Quality flags" table, after "Concept" and "Value" columns has been added. This new column is an automatically filled column with a brief text describing the possible consequences for science data from the situations that caused that quality flag to be raised.

Improvement of readability of quality flags in the meta data of products in FITS files. Now one meta data item per quality flag making proper use of parameter values and descriptions.

eg.) qflag_PARXXXXX_p <value> ⁄ <description>

Systematic product generation

Data input-output

Virtual Observatory

Installer

The user installer has been reduced by 400-800MB by removing the pointing distortion maps previously included with the data a part of user installation.

Documentation

HIFI

HIFI Software Configuration and Properties

HIFI Calibration

HIFI Pipeline

hifiPipeline task

Level 0 Pipeline

WBS Pipeline

HRS Pipeline

Level 1 Pipeline

Level 2 Pipeline

Level 2.5 Pipeline

HIFI Products

HIFI Spectrum Toolbox

Spectrum Explorer (HIFI specifics)

Spectrum tasks (HIFI specifics)

HIFI Data Processing Tools

Standing Wave Removal

Baseline Removal

Deconvolution

Spectral Cube creation

flagTool

identifyLines

Updates to the task include:
- Implementation of threading to speed up the task
- Improvements of the rejection of false detected lines
- Changes on the Herschel Spectral Line List columns names
- Implementation of the image band line identification for pointed and mapping observations
- Fix to the rms computation
- Implementation of the exportLines task

HiClass export tool

Documentation

For HCSS 14, many figures were refreshed in both manuals (HIFI Data Reduction Guide and HIFI Pipeline Specification).

Updates to the HIFI Data Reduction Guide:
- Updates to the chapters Standing wave Removal and HIFI Baseline Removal : adding definition of the new parameter addMedianContinuum
- Updates to the chapter Flags in HIFI data, section Quality Flags (this chapter is being revamped, please check regularly)
- Updates to the chapter How to make a spectral cube: added a definition for parameters ignoreOffs, outputNames and extrapolate
- Updates to the chapter Sideband Deconvolution: the output deconvolution product ssb is now called dataset (please note that scripts prior to this change will break)
- Updates to the chapter Standing wave Removal, ESW correction in HEB bands section: update to the scripts in the demonstration subsection
- Updates to the chapter How to add and remove flags, section Setting and Clearing Flags with command line tools: improved the definition of the parameter selection for Level 2.5 cubes
- A new chapter called Understanding the uncertainty table information in your data has been added
- Updates to the chapter The HIFI line identification tool, adding a section called The exportLines Task
Updates to the HIFI Pipeline Specification manual
- This manual has been restructured
- Removal of the task mkSpur
- Corrections to broken links pointing to the HIFI Data Reduction Guide
- Correction due to a missing figure Level 0 Pipeline flow diagram
- Added text to tasks which did not have text, or were poorly documented, including mkRms
- Updates to several tasks on flags information

PACS

Pipeline

Photometry

Level2.5 products:
- Unimap and Jscanam maps are now generated for Solar System Objects, from combined scan and cross-scan observations.
- A block detection and masking method has been implemented, this avoids strong signal gradients being generated by the calibration blocks. The quality of Level2.5 maps is enhanced in particular for those observations acquired at the early stage of the mission (e.g. Science Demonstration Phase), when the calibration blocks were acquired in the middle of the an observation.
Interactive pipeline scripts:
- The JScanam ipipe script scanmap_JScanam_multiplePairs is designed to process multiple pairs of obsIDs and it has been upgraded by introducing the scanamorphosAlignFrames Task. This task aligns the Frames signal levels, belonging to every observation, to a common offset level, by exploiting the overlapping sky regions. This new capability is particulary useful for recovering a common background level over observations that are partially or slightly overlapping.
- The script scanmap_Unimap_memory_optimized has been added to the Pipelines menu. This allows one to run the Unimap release located on the user’s computer while using the minimum amount of memory. It is particularly useful for processing large observations or/and if several obsIDs should be processed all together in a single run.
- The script photextended_pipeline_L3 is now available in the Pipelines menu. This can be used to generate mosaics of Unimap and Jscanam Level2.5 maps.
Unimap : Unimap 6.2.0 is spawned by the Standard Product Generator (SPG) to create Level2.5 products. See the Photometer Data Reduction Guide and the offical Unimap webpages (http://infocom.uniroma1.it/unimap/) to know What's New about the 6.2.0 Unimap release.

Spectroscopy

Level 2/2.5 extracted spectra tables:
- A new product is provided at Level 2 for all (line, range, chopNod, unchopped) pointed observations, and also at Level 2.5 for the subset of unchopped, rangeScan observations. This table, HPSSPEC[R|B], contains the extracted spectrum of the central spaxel, and the point source calibrated spectra "c1" and "c129" from the pipeline task extractCentralSpectrum. Since the point-source calibrated spectra are taken from the centre of the cube, it is necessary that the source is located within spaxel 2,2 (the central one). However, we make no determination that the target of the observation actually is a point source - that is up to the end-user to establish. These tables are also provided as standalone browse products.
Level 3 extracted spectra table:
- The new spectrum tables from Level 2 for the pointed, chopNod, full SED observations, are concatenated into a single table which is placed in Level 3. This table, HPSSPEC, contains the extracted spectra taken from all the observations that were taken to cover the full spectral range (the full SED) of PACS for that target (this will be usually two, sometimes three, separate observations). Moreoever, the red and blue camera, and all slices, are included this single table. This table is also provided as a standalone browse product.
Interactive pipeline scripts:
- The pipeline menu has been reworded, so that the names reflect better what the individual scripts do. The first script offered in each menu is also the preferred one, and that on which the SPG scripts are based.
- A new pipeline script has been written to create drizzled cubes, based on (and almost hardly different to) the 'Telescope normalisation' pipeline script for chopNod line scans. With this script it is now possible to create drizzle cubes using this preferred pipeline method (in Track 13 it was not possible).
- There have been small, but potentially significant changes to some of the tasks in the 'Pointing offset correction' scripts (chopNod) and the '...with transients" scripts (unchopped). These are, respectively, to the tasks specDetermineChiSquare and specLongTermTransient. The improvements are "under the hood" so there is no change in the user scripts themselves.
- The range scan flatfielding task now incorporates a spline fitting to the continuum of the spectra. This produces the same results for shorter ranges but improved results for longer ranges (especially SEDs) and where there are strong curvatures in the spectra.
SPG scripts:
- The range scan observations are now flatfielded: using the spline fitting method for ranges longer than 5 microns, and using the line scan flatfielding for the shorter ranges
- The interpolated cubes are no longer produced for oversample range scan mapping observations, and they also have a smaller spaxel size of 3" (rather than 4.7" before).

Calibration

Photometry

Spectroscopy

There is a new Extended Source Correction calibration file (ExtendedSourceLoss, used in the pipeline tasks specRespCal and specRespCalToTelescope) which produces a better spatial flatfield for the PACS spectrometer. The fluxes of the spaxels of all cubes will be increased by ~30% in the blue to a decrease of ~5% in the red. The calibration file can be inspected to extract the curve:
w = calTree.spectrometer.extendedSourceLoss["wavelength"].data
f = calTree.spectrometer.extendedSourceLoss["fraction"].data
For point sources calibrated with extractCentralSpectrum, this extended source correction is taken out before the point source corrections are computed. However, if working with off-centred point sources and using the tasks extractSpaxelSpectrum and pointSourceLossCorrection, this correction is not taken out, and hence the results will be incorrect. An extra task/function is necessary: to get this, raise a helpdesk ticket.
New Beam calibration files have been delivered, differing from the previous files only in the way they are normalised. These will be used in a new task being developed in HIPE 15, and do not impact on HIPE 14 or SPG 14 in any way.
A slightly updated telescope background model for use in the Pointing Offset Correction pipeline script for point sources has been developed. This update is to allow better consistency between the "POC" script and the default pipeline script "Telescope normaliation", for chopNod AOTs. The changes are at the percent level and largest in band B3A.

Analysis tools

Photometry

The PhotometryConvolveResolution Kernel script was added within the PACS Useful Script section. This script convolves and regrids a map at a certain wavelength to the PSF of another map at longer wavelength, by using an appropriate kernel.

Spectroscopy

A new script has been provided to allow users to plot the telescope background spectrum model. The code and necessary files are provided on http://herschel.esac.esa.int/twiki/bin/view/Public/PacsCalibrationWeb?template=viewprint (PACS Spectrometer Calibration section).

Product changes

Photometry

The error layer within Unimap Level 2.5 products is renamed as stdDev, being the standard deviation computed at every sky pixel with respect to the samples that are falling on it. It can be properly assumed as the error map only if associated to the Naïve map.

Spectroscopy

Interpolated cubes: Cubes produced by the task specInterpolate now have a spaxel size of 3".
Interpolated cubes: Are no longer provided for oversampled mapping observations.
Interpolated cubes: The way the task specInterpolate handles NaNs as it creates these cubes has changed. This will primarily affect cubes with saturated data. Saturated data are masked out and replaced with NaNs when rebinned cubes are built. Interpolated cubes are then created from the rebinned cubes. NaNs are now filtered out as the interpolation task does its work on the input cubes, and all affected and immediately surrounding spaxels of the output interpolated cube will also be NaN. Hence, at the affected wavelengths, the NaN area will be larger than before...but this is also more correct than before.
Standalone browse products: we now provide the new extracted spectra tables, HPSSPEC[R|B] and HPSSPEC (see above), as SBPs.
Drizzle cubes: in SPG 13 the drizzle cube fluxes were incorrect. In SPG 14 they are now correct.
PacsCubes and PacsRebinnedCubes created by the pipeline (HPS3D [R|B] and HPS3DR [R|B]) will now display in cube viewers with a flip over the diagonal, as compared to previously. The new orientation makes it easier to compare the cubes in cube viewers (the Standard Cube Viewer or the Spectrum Explorer) to the display of the PacsSpectralFootprint viewer.

Documentation

SPIRE

Calibration Products

Spectrometer
- A new calibration product has been added that contains correction curves for point-source calibrated LR data.
- A new calibration product has been added that contains polynomial coefficients that are used to generate correction curves for extended source calibrated data.

Calibration Framework

Preprocessing

Engineering Conversion

Common Pipeline

Photometer Pipeline

Spectrometer Pipeline

A new task has been added to the point source pipeline that applies the LR correction.
A new task has been added to all pipelines that applies the extended calibration correction in order to get better agreement between FTS and Phototmeter photometry.

Interactive Analysis and Tools

There is a new useful script, Spectrometer Pointing Offset Corrector, for estimating and correcting pointing offset in sparse point observations. The scripts assesses the difference between the SLW and SSW spectra of the centre detectors, over the overlapping frequency range, uses this to estimate any pointing offset and corrects the data using SECT.