Tags:
create new tag
view all tags

What's new in HIPE 12

PDF Version Click here

Help Interested in what's new for other versions of HIPE? See this page for links to all the What's New documents from HIPE 4.0 onwards.

See also the HIPE known issues and the Data products known issues pages for last-minute information on known problems.

Highlights

HIPE

  • Drastic improvement in memory consumption for certain pipeline and analysis tasks (due to the changes in how output parameters are handled, see below).
    • Tasks with multiple outputs now return a list (__list__ argument deprecated) to save memory.
    • Warning, important Errors mentioning ArrayList suggest that your code is using tasks that return multiple outputs and you need to update it.
    • An automatic tool in HIPE will detect scripts with multiple outputs tasks: In case you open such a script then a dialog box pops up showing which lines need to be changed and asking your permission to convert it. Its current limitations are described in the Known Issues for HIPE 12 page. See Task parameters if your script contains any of the syntax features that are not currently implemented or if after running the conversion there are still errors that could be related to output parameters.
  • Improved contents and documentation in the headers of FITS files generated with HCSS or HIPE.

HIFI

  • Special task to mitigate electronic standing waves in HIFI HEB bands.
  • There is no strong reason to reprocess HIFI data with this release. However for data taken in bands 6 and 7 (HEB bands) it is recommended to try and remove the electrical standing waves with the new dedicated task.

PACS

  • Systematic pipeline produces Level 2.5 and Level 3 mosaics.
  • All observations should be reprocessed with this HIPE version.

SPIRE

  • Super-resolution map-making algorithm for SPIRE-PHOT data.
  • There is no strong reason to reprocess SPIRE-PHOT data with this version and for the SPIRE-SPEC data it depends on the science case.

Core system

HIPE

Batch mode

The jylaunch application has been removed.

To execute scripts in batch mode, run hipe with file path arguments.

Warning, important Note: the script file shall have .py extension

Status bar

Any code can echo messages in the status bar. For example, from a Jython task:

from herschel.ia.gui.kernel.util import SiteUtil
SiteUtil.getSite().getStatusManager().submit(message)

Directories

From the General > Directories preferences panel, the user can:

  • Select the default directory where to open file dialogues.
  • Select the working directory, which will be used when reading or saving files with no absolute path specified.

Drag & Drop

  • Multiple result rows can be selected at once in the Product Browser and create variables for them in a single drag & drop operation to the Variables View.
  • Observation variables created by drag & drop from the Product Browser onto the Variables View have more sensible names.
  • If a drop in the Variables View is not valid for certain variable (for example, trying to drop a non-observation in the Observations section), a forbidden icon will be shown to let the user know.
  • Files can be dragged from the Navigator and dropped in different places:
    • In the Console, they echo their absolute path.
    • Within task dialogs, in parameters expecting a file path.

Editors & Viewers

  • The reduced browse image at the top-right corner of the Observation viewer has much better quality, so details can be distinguished even without having to enlarge it. The same applies for observations shown in the Outline.
  • Map and List viewers have been improved. It is possible to open their elements in separate viewers, or to create variables for them.
  • If the Data section of the Observation, Product or Dataset viewers is collapsed (by clicking on its header), the available space is given to the Meta Data section.
  • A preference to restart Jython scripts when finished has been added. It is disabled by default, but can be enabled in the Editors & Viewers > Text Editor > Jython Editor preference panel.
  • When autosave of text editors is on, a backup file is generated only when the original file is modified.

Miscellany

  • Minimized views are remembered as part of the persisted layout of perspectives.
  • HIPE now asks to restart upon changes that require a new session in order to be applied.

Astronomical utilities

  • Sso: Added a variant that uses internal Horizons.
  • radialV and RadialV.getProjection(...) : if Ephem is null the system will use its own if needed.

Images

  • SimpleCube.copy() and SimpleImage.copy() create accurate copies now.
  • WCS 3rd axis does follow FITS convention for reference pixel now.
  • The WCS metadata is copied to all planes from images (spectra).

Display

  • Display ignore flags when the description is prefixed with 'INFO:'.
  • Colour table drawn in the axes is corrected.
  • Display does also work when image contains a TableDataset.
  • Zoom does work also on RGB images.

Analysis

  • SkyAperturePhotometry tasks can now be opened and filled from the command line. An example :
openTask("annularSkyAperturePhotometry",inputName="image,radiusArcsec,centerX,centerY,innerArcsec,outerArcsec",variableName="myImage,myRadiusArcsec,xCenter,yCenter,innerRadius,outerRadius")
  • ImageCeil, imageFloor and imageRound tasks allow the user to decide which decimal is to be rounded.
  • ImageDivide also works with images without a unit.
  • The output of the MosaicTask now has the metadata of the first image of the mosaic.
  • Fixes in the GUI of SkyAperturePhotometry tasks.

Spectra

Display

  • The raster panel allows to show coordinates in either sexagesimal or decimal format.
  • Text of some popup menu options improved.

Analysis

SpectrumFitterGUI
  • Update to export panel to make more intuitive to use
  • Generated scripts have been updated
  • A ParameterCube can now be exported
  • All implemented models now have their own GUI (previously an "abstract GUI" was used for all but the most common models)
  • Can set lower and upper bounds limits in model GUIs

Spectral Cubes

Display

  • Improved the display of cubes in the Spectrum Explorer and added the panner with compass to the display panel.
  • Improved loading large cubes.

Analysis

  • The cropCube task now also accepts ranges in wavelength units.
  • The moment algorithm of computeVelocityMap has been significantly improved. Especially the error on the dispersion is now much smaller.
  • Fixed a bug in integrateSpectralMap when using a cube with regular wavelength grid (in WCS).
  • All cube tasks now propagate the description of the cubes, the instrument name and the units to the result.

Tasks

  • Task GUIs: You can now drag and drop from the Navigator into 'file' arguments.

Data input-output

  • Table reading:
    • metadata and table descriptions are now read.
    • nulls are converted to valid values (can be user supplied).
  • Table writing:
    • FixedWidthFormatter: can easily generate an appropriate formatter passing a table.
  • FITS:
    • BUNIT is now written
    • quality flags: (ongoing) now generate keywords.

  • asciiTableWriter :
    • Warning, important default value for warn has changed to false. By default the system will not ask for confirmation if the file to be written already exists.
    • By default, it will save meta-data.
    • Added tableType parameter that can generate output in fixed-width format (3 spaces between columns).
    • Simplified GUI and improved documentation.
  • asciiTableReader :
    • Added null conversion parameters.
    • Simplified GUI and improved documentation.
  • convertAngles : (no longer a prototype) It converts from numeric to string formats and vice versa.
  • convertUnits : several improvements.
  • Export View : simplified GUI.
  • getObservation:
    • improved GUI.
    • If an observation is not found locally (and the main MyHSA is connected to the archive) the Archive will be tried.
  • help : Also provides access to Python help.
  • openTask : you can now open a task attaching variables to any parameters.

Installer

  • The installer now bundles one specific version of the Java Runtime Environment (JRE) for Mac OS as it is already done with Linux and Windows. This will help isolate HIPE from the JRE installed at operating system level, but has the inevitable consequence of adding about 50 MB to the installer size.

Documentation

  • The internal name for the Data Analysis Guide was renamed from howtos to dag. Now when searching the documentation the results within this manual appear below the dag name.
  • The internal name for the Scripting Guide was renamed from um to sg. Now when searching the documentation the results within this manual appear below the sg name.

HIFI

HIFI Calibration

  • Updates to CleanDF tables (Level 0). These tables allow the pipeline to omit corrupted data frames.
  • Nonsense LO settings in found spectral scan data are corrected at Level 0 using a calibration table storing the correct values. This impacts only one observation: 342194735 (PSP2_HEXOS-oribar-hifiss-b6a-main)
  • Calibration tables for Electrical Standing Wave task hebCorrection: WBS-Splines (6aH, 6aV, 6bH, 6bV, 7aH, 7aV, 7bH, 7bV)

HIFI Pipeline

Level 2.5 Pipeline

  • Level 2.5 pipeline can be run in parallel (command line only)
  • More tasks, for example doDeconvolution and PolarPair, are better integrated into the pipeline.

HIFI Products

  • In the case of mapping observations, the level 2.5 products containing up until now only the cubes now also contain a copy of the level 2 individual spectra complemented with the necessary information in order to optimally compute the cube grid dimension.
  • Information available in metadata has been generally improved with the intention that they can (eventually) be used when querying the HSA

HIFI Data Processing Tools

Standing Wave Removal

  • fitHifiFringe
    • Input parameter data replaces the multiple inputs such as "obs_or_htp", "simple_spectrum", "sds", and "cube".
    • For the last (or single) spectrum, the move to next option has been removed from the pop-up dialogue window. The task exits more elegantly and leaves the two plot windows open (for the default plot=2 option).

Baseline Removal

  • fitBaseline
    • Input parameter data replaces the multiple inputs such as "obs", "htp", "sds", and "simpleSpectrum".
    • now works from within SpectrumExplorer to create and manipulate line mask windows and to allow plot scaling with the right mouse button.
    • works on a simpleSpectrum .
    • skips a spectrum if "empty".
  • Bug Fixes:
    • The backends option now works correctly - only the ones selected are processed. This is true for obs, htp, and sds.
    • If data cannot be fitted fitBaseline will move onto the next dataframe. Previously it hung.

Electrical Standing Wave correction in HEB bands

  • hebCorrection
    • A new task is now available to help correct the Electronic Standing Waves in bands 6 and 7. See instructions for use in the HIFI Data Reduction Guide

Deconvolution

  • Channel weights are now calculated for the SSB output (calculated as 1/rms^2)
  • Minor bug fixes
  • Major bug in gain fitting discovered. Feature should not be used.

Spectral Cube creation

  • DoGridding uses a new command line syntax. This is a result of the general change in behaviour of Tasks with multiple outputs.
  • A new task MergeHtpsTask is available in order to merge (not average) HTPs of H and V polarisation. This allows spectral cubes to be created using all of the pointing information in the observation
  • DoGridding is adapted to use a 2d model

flagTool

  • New GUI figure to reflect the addition of an input parameter for htp

HiClass export tool

  • It is now possible to open the hiClass task (to export in FITS) by using a right-click on a variable (Send to / CLASS FITS file)

Documentation

  • Updates to the HIFI Data Reduction Guide:
    • Added 3 figures to Introduction, How to get your data. Updated text.
    • Added four new cookbooks to chapter 2: "HIFI Cookbooks": Point Mode Frequency Switch, Point Mode Position Switch, Point Mode Load Chop, and Spectral Scan.
    • Update to section 4.2 "What was done to my data" / "Level 0".
    • Update to section 5.4 "Re-pipelining with a new Auxiliary product".
    • Regular updates to chapter 8: "Flags in HIFI data", specifically to the Quality Flags Table.
    • Update to chapter 9, flagTool GUI figure (possible input parameters are now "obs" and "htp"). Adjusted text accordingly.
    • Update to chapter 10: "Standing Wave Removal, fitHifiFringe GUI figure to reflect change in input parameter named data. Adjusted text accordingly.
    • Update to chapter 10: "Standing Wave Removal, specifically to the section "Electrical Standing Wave correction in HEB bands".
    • Update to chapter 11, fitBaseline GUI figure to reflect change in input parameter named data. Adjusted text accordingly.
    • Added a new subsection called "Caveats" in chapter 11, "HIFI Baseline Removal" chapter, section "The FitBaseline Task".
    • Updates to chapter 12: Sideband Deconvolution.
    • Update to chapter 13, "How to make a spectral cube", section "...to change beam, pixel, map size". Introducing new syntax (for tasks with multiple results) to create cubes.
    • Update to chapter 20, Added a note for HIPE 12 on right-click on variable to open hiClass GUI. Improved method to export results of deconvolution to Class.
  • Updates to the HIFI Pipeline Specification manual
    • Added tasks doMainBeamTemp, polarPair, doFold, and mkRms in General Pipeline, Level 2.5 Pipeline section

PACS

Interactive Analysis and Tools

Photometer

  • A new task was introduced convertToFixedPixelSize() that performs the optical distortion correction. It is now applied in MADmap ipipe and will be used in the SPG12 processing, correcting for a 6% flux scale overestimate in the red channel and 2% underestimate in the blue channel.

  • JScanam: We have introduced significant improvements in the JScanam pipeline. Most of JScanam task run now considerably faster and require less memory than in their previous HIPE 11 version, specially when the debug option is deactivated. The quality of the final products has also improved. In particular, the task scanamorphosIndividualDrifts is reducing the correlated noise in parallel observations in a better way.

  • A new task photTevCorrection() is available and used for SPG, see section on calibration.

  • There is a new task for chopped observations photCalPsToScan() that accounts for the difference in responsivity as compared to the scan map flux calibration scheme. When applying this task, the photometry of pint sources observed with the chopped photometry mode are in line with the results from scan map observations. For details of the method see the publication in Experimental Astronomy.

Spectroscopy

  • The pointing offset correction ipipe scripts for line (ChopNodPointingCorrectionLineScan.py) and range scans (ChopNodPointingCorrection.py) of point sources are made available in the hipe pipeline menu inclusive detailed documentation on the pointing offset correction algorithms.
  • For the point source pointing offset correction pipeline 2 new tasks were developed to determine the pointing offset correction in case the accurate pointing product is available for an observation.
    • specDeterminePreCalculatedPointing: it calculates the pointing offset using the ra/dec/rollAngle information of the Frames.
    • specDeterminePointingOffsetFromPreCalculatedPointing: it determines the flux correction factors by using the pointing offset determined by specDeterminePreCalculatedPointing.
  • We provide a new task (specExtendedToPointCorrection) for computing the correction factor for slightly extended sources relative to the point source correction, based on a model of the source flux distribution. This tool is described extensively in section 7.8 of the PACS Spectrometer Data Reduction Guide.
  • The extractCentralSpectrum task has been improved to return both the central and central 3x3 spectra, and to handle the masks better. Also the syntax has been changed, since it returns now a list of spectra (central, SUM(central_3x3) and a scaled version of the central spaxel computed so that its continuum matches that of the Sum(central_3x3)). See the URM and section 7.6 of the PACS Spectrometer Data Reduction Guide for details.
  • Improved the long-term transient correction for the unchopped line spectroscopy pipeline.
  • One of the spectrometer's interactive reduction pipeline scripts is now offered in a simplified version (ChopNodLineScan.py). The processing itself is not affected, but the number of helper-plots ("verbose plots") has been reduced and the comments have been abridged. Saving intermediate or final results is also made easier (without necessarily editing the script). In absence of negative feedback, the same approach will be followed for all other scripts from hipe 13.0 on.
  • plotCubesStddev: a new parameter wranges allows the user to select the continuum windows of his/her choice.

SPG Pipeline

Photometry

  • A correction is applied on level 1 frames to correct for influences of the evaporator temperature.
  • A correction is applied on level 1 frames of the chopped photometry data to correct for differences in the responsivity as compared to scan maps.
  • JScanam maps will be available in level 2.5 and 3.
  • MADmap maps will be corrected for the flux scale due to the fact that optical distortion was not applied so far.
  • All maps now have now the same pixel size in the SPG, so that they can be compared more easily.

Calibration

Photometry

  • A new task photTevCorrection() and related calibration file is available to correct for the dependence of the responsitivity to the evaporate temperature. The correction is typically below 1%. Only 1.2% of the observation will have a larger correction.

  • There is a new task for chopped observations photCalPsToScan() that accounts for the difference in responsivity as compared to the scan map flux calibration scheme. When applying this task, the photometry of pint sources observed with the chopped photometry mode are in line with the results from scan map observations. For details of the method see the publication in Experimental Astronomy.

Spectroscopy

  • For point source pointing offset flux calibration a time dependent correction to the current telescope background model is implemented. The calfiles for each spectral band are called TelBackCorR1, TelBackCorB2A, TelBackCorB2B, TelBackCorB3A and used by the corresponding task called specCorrectTelescopeBackground.

Product changes

  • Improved the browse product structure (numbers changed to meaningful names).
  • Browse images have been improved significantly (postcards in the HSA).
  • Added a new observation summary dataset to the browse products. The obsSummary task shows the contents of this dataset and it can be viewed also in the Observation Context Viewer. Note that in order to update this summary to reflect changes in the observation context you have to run this task with forceUpdate=True.
  • Improved the names of many FITS keywords, which makes the headers of PACS products easier to read outside HIPE.
  • Added new meta keyword sliceNumber to slices. This number also appears in the FITS filenames in the pools, which makes them more readable.
  • Fixed some problems with copying products.
  • The structure of the level 2.5/3 contexts in the photometer observation contexts have been improved. All the slots in these contexts have now consistent names for each type of map:
    • HPPJSMAP[R|B|G] for red, blue or green JScanamorphous maps.
    • HPPNAVMAP[R|B|G] for red, blue or green naive maps.
    • HPPCORMAP[R|B|G] for red, blue or green corrected maps.
    • HPPMADMAP[R|B|G] for red, blue or green MadMap maps.
    • HPPHPFMAP[R|B|G] for red, blue or green High Pass Filter maps.
  • Fixed duplication of FITS file names when observations are exported from the HSA into a tar file.
  • Made sure that the ra and dec in level 2 photometer maps correspond to the center of the map.

Spectroscopy

  • Improved the meta keywords which describe the slices. The following keywords were added:
    • meanWave: the mean wavelength of the wavelength range of a slice.
    • prime: True if a slice belongs to the prime channel, false otherwise.
    • inLeak: Can be 'yes', 'no' or 'partial', depending on whether a slice is fully, not or partially in the leak region.
    • repetition: the repetition number of a slice.
    • gratingStep: the grating step size within a slice.
  • Added more flags to the rebinned cubes produced by the specWaveRebin task: We now propagate besides the saturation masks also all other inactive masks into the Flag dataset of the rebinned cubes.

Documentation

  • Improved the table of contents of the PACS User Reference Manual.
  • Detailed documentation of the point source pointing offset correction algorithm is delivered, which is a general description of the processing and a detailed description of the single tasks used including a flow chart.

SPIRE

Calibration Products

Spectrometer

spire_cal_12_2

  • Point source calibration for off-axis detectors corrected to account for the pointing of Uranus in the calibration measurements. Only affects the off-axis detectors. Centre detector point source calibration changes by less than 0.15%.
  • Unused Spatial Correction Coefficient product removed (was designed for use with the semi-extended correction tool, but never actually used).

spire_cal_12_3

  • The spectrometer bands are increased slightly in the overlap region and at the high frequency end of SSW:
    • Old bands: SLW: 446.7-989.4 GHz; SSW: 959.3-1544.0 GHz
    • New bands: SLW: 446.7-1018.0 GHz; SSW: 944.0-1568.0 GHz
  • The extended calibration is not changed from spire_cal_12_2 (apart from wider bands).
  • The Low resolution point-source calibration is changed by <5% (before OD1000) and <2.7% (after OD1000) to bring it into line with the High resolution calibration. This was achieved by correcting for the pointing offset of Uranus during the calibration observations. The effect is to reduce the flux density of the final spectra. Changes are greatest for SSWD4 at high frequency, and smallest for SLWC3 (~1% for SLW).
  • The High resolution point-source calibration has also been adjusted using improved corrections for the pointing offset of Uranus in the calibration observations. The effect for SSWD4 is ~2.5% before OD1000, and 0.8% after OD 1000. For SLWC3, the effect is <0.8% before OD1000 and <0.1% after OD1000. The effect is to increase the flux density in the final spectra.
  • A small correction has been made to the point-source calibration for off-axis detectors for observations made before OD1000.
  • The frequency grid for the beam profile shape been updated to match exactly that used in Makiwa et al. (2013, Applied Optics, 42, 3864) - this gives slightly better coverage at the edges of the new bands. The shapes themselves do not change.

Calibration Framework

  • New SPIRE Calibration Automatic Updater introduced: it runs during HIPE startup and notifies the user if a new calibration tree is available. The tool allows also the installation of the new calibration tree.
  • New SPIRE Calibration View introduced: this is a new HIPE view callable from HIPE view menu. The view shows all the available SPIRE calibration trees with their release notes, and allows installation/removal of each tree. Different colours are used to distinguish default/installed/not installed trees.
  • New SPIRE Calibration Preferences panel added to the HIPE preferences: the preferences panel allows some configurations for the SPIRE Calibration View and the SPIRE Calibration Automatic Updater.

Photometer Pipeline

  • TemperatureDriftCorrectionTask now reacts to the "coolerBurpDetect" parameter defined in the timeline.
  • The quality of some Level-3 mosaic maps has been improved by constructing them from Level-2.5 maps where available.

Spectrometer Pipeline

  • All spectra and spectral cubes are now corrected to the Local Standard of Rest. The frequency grid is interpolated back to the original scale so that all observations still have a consistent frequency grid. A metadata keyword, freqFrame (or SPECSYS in FITS), tells you whether the LSR correction has been applied or not.
  • Vignetted detectors are now included in spectral maps by default. The calibration of the vignetted outer ring of detectors was found to be reasonably consistent with the other detectors. This increases the area covered by the maps.
  • The frequency axis for spectral cubes is now contained in the 3rd axis of the WCS (to match HIFI cubes), rather than in a separate dataset called ImageIndex.
  • SpireWcsCreator has been renamed SpecWcsCreator to distinguish it from the photometer WCS task.
  • Pipeline tasks with multiple outputs must now be specified explicitly in the pipeline - this affects 5 tasks: waveletDeglitcher, sigmaKappaDeglitcher, baselineSubtraction, phaseCorrection and spireProjection. Pipeline scripts have been updated with the new syntax, which is explained in the SPIRE Pipeline Specification Manual.
  • The functionality of the preCube product was extended to make it easier to add or remove spectra.

Interactive Analysis and Tools

Photometer

  • The Maximum Correlation Method (MCM) super-resolution algorithm developed for IRAS data can be applied to SPIRE photometer maps and is now available as the "hiresMapper" task. A useful script is also provided to help with using it.
  • The new "Bolometer Finder Tool" provides a friendlier front-end to the "boloFinder" task, and functions as an interactive deglitching tool. In principle it makes the corresponding "useful script" obsolete. Its features include:
    • Identify a suspect pixel by image visualisation.
    • Plot the timeline data contributing to a selected pixel.
    • Identify glitches by clicking on the plots.
    • Remake the image with the identified glitches masked out.
    • Undockable tabs to allow side-by-side comparisons.
    • Save the modified masks back to the Level-1 and Level-0.5 timelines.
  • SPIA (SPIRE Photometric Interactive Analysis), previously available as a plugin, is now integrated into HIPE.

Spectrometer

  • A new SPIA task has been added to carry out background subtraction, called "spiaFtsBackgroundRemoval".
  • The Semi-extended Correction Tool has been updated:
    • Sersic profile added.
    • The ability to select a Gaussian reference beam has been added.
    • A threshold parameter to exclude off-axis detectors from the output when they do not overlap with the source.
  • A new tool is available to calculate FTS synthetic photometry called "specMatchPhot".
  • Small additions/corrections to the FTS useful scripts.
  • SpecWcsCreator allows a WCS to be calculated in instrument coordinates.

Documentation

Release Notes

  • Full list of tickets implemented: here.

blog comments powered by Disqus

-- StephanOtt - 01 Mar 2013

Edit | Attach | Watch | Print version | History: r56 < r55 < r54 < r53 < r52 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r56 - 2015-01-07 - TrevorFulton
 
This site is powered by the TWiki collaboration platform Powered by Perl