• • New auxiliary product OperationalDaySummaryProduct that helps filtering and handling data per Operational Day (OD) and new gyro-based pointing reconstruction algorithm.

• The new calibration tree contains updated beam models and efficiencies, as released initially as stand-alone products in September 2014. See for more details http://herschel.esac.esa.int/twiki/pub/Public/HifiCalibrationWeb/HifiBeamReleaseNote_Sep2014.pdf

• TBD

• TBD

• The browse image of the Observation viewer can be opened with the Image viewer by means of Right click > Open in image viewer.

• The Help > Check Java platform option shows the running Java version.

• HIFI is now providing a handful of useful scripts - see the menu Scripts > HIFI Useful Scripts.

• Gyro-based attitude reconstruction, see the dedicated section in the Spacecraft and Observatory pages.

Mapping

Display

Analysis

Display

Analysis

Display

Analysis

Product Access Layer

• __list__ argument removed: if your task call still has it, remove it.

• • Band 1 change nfringe 2->3, endPeriod 120->200
  • Band 3 change typicalPeriod 620->95
  • Band 4 change typicalPeriod 620->95
  • Band 6 change startPeriod 100->80, typicalPeriod 320->600
  • Band 7 change startPeriod 100->80, typicalPeriod 320->625

• Ipipe scripts: MADMap pipeline is still available as an interactive script in the Pipelines menu. The interactive script scanmap_Unimap allows users to run Unimap while tuning some parameters. It uses the new task runUnimap() that calls the Unimap release located on the user’s computer.

• Significant improvements to the bolometer finder tool have been made.

• Gyro-based attitude reconstruction, see the dedicated section in the Spacecraft and Observatory pages.
• Special corrections in pointing products to handle S/C operational events have been included in the auxiliary processors. Namely, the corrections are:
  • STR switch-overs, i.e. changes from the usually prime star-tracker (STR1) to the backup unit (STR2)
  • Reset of the Spacecraft Velocity Vector (SVV) that is used by the STR to compute the aberration correction of the coordinates of guide stars.

For more  information please refer to the the dedicated section in the Spacecraft and Observatory pages.

• • New auxiliary product OperationalDaySummaryProduct that helps filtering and handling data per Operational Day (OD) and new gyro-based pointing reconstruction algorithm.

• • New HIFI useful scripts

• HIFI is now providing a handful of useful scripts - see the menu Scripts > HIFI Useful Scripts.

• • Spectrometer: Cubes with evenly-sampled spectral grid. Rebinned cubes provided as tables for easier viewing outside HIPE. New cube interpolation task for undersampled mapping observations.

• Unimap: a new Generalized Least Square (GLS) mapmaker is introduced into the Standard Product Generator. Unimap is a MATLAB algorithm that is spawned from HCSS. Level2.5 and Level3 products generated by Unimap replace MADMap maps for SPG13.
• Level2 products: the quality of the High-pass Filtered Level2 products (HPPPMAP) is improved by adding the scanamorphosMaskLongTermGlitches() and the scanamorphosBaselinePreprocessing() tasks, which can detect jumps and correct for drifts on the detector timelines.
• Level2.5 products: Unimap Level2.5 contexts contain the same types of maps generated by MADmap (naïve, GLS and Post-GLS), but they are arranged differently within the context (see Product changes).
• Level3 products: High-pass Filtered maps (HPPHPFMAP) have been removed from Level3 products, leaving JScanam and Unimap maps. Level3 mosaics are created by using scanamorphosAlignImages(), a new task that links the flux by exploiting overlapping regions between images.
• Ipipe scripts: MADMap pipeline is still available as an interactive script in the Pipelines menu. The interactive script scanmap_Unimap allows users to run Unimap while tuning some parameters. It uses the new task runUnimap() that calls the Unimap release located on the user’s computer.

• Correction of timing shifts: frame pixels are not read simultaneously, so in SPG12 the frames are associated with a single readout time. That results in a time shift that can correspond to up to a 2 arcsec delay for fast scanning observations. This effect is corrected by setting the parameter correctReadoutTimes=True by default in every task that is affected by time offsets.

• The Chop/nod SPG pipeline now uses the background normalisation method instead of the calibration blocks for the flux calibration.
• At Level 2/2.5 we now also produce drizzle cubes (HPS3DDR and HPS3DDB), and "interpolated" cubes (HSP3DIR and HPS3DIB) created with the new task specInterpolate.
• Hence at Level 2/2.5 we now produce three cubes which have an equidistant spatial grid and which for mapping observations are also a mosaic: drizzled, projected, and interpolated cubes. Each observation will have two of these three cubes, chosen independently for each camera. Which two cubes you get depends on the AOT and the type of mapping raster: see the PACS Products Explained document (in HIPE) to learn more.
• One of these three cubes is also provided with an equidistant wavelength grid, created with the new task specRegridWavelength.
• Tables containing the data of the rebinned cubes are created produced with the new task pacsSpecCubeToTable (HPSTBRR and HPSTBRB). The spectral data of each spaxel of each raster position is included in a single table, and each wavelength range and camera is a separate table.

• New long/short term Transient Correction algorithms are included in a new pipeline script for the unchopped line and range scan modes.
• Extra options have been added to the wavelengthGrid task to create a spectral grid with bins of the same size (equidistant grids). The bin size chosen can be the mean, minumum, a fraction of the minimum, or a specified value.
• specSubtractOffPosition can now combine off-positions using different algorithms, and it works with non-rebinned PacsCube products as well. The task specSubtractOffPositionNotRebinned is now superfluous and has been deprecated.
• An image can now be used as model in the extended to point correction task.
• There are several improvements to the Pointing Offset Correction pipeline scripts (range and line scan, chopNod), including more useful plots.
• specDiffChop now also works on a product containing only calibration block data.
• slicedSummary works now also on all cubes.
• The footprint viewer is now a real viewer, and can be opened with the "Open with" menu when clicking on an appropriate product.
• General interactive pipeline script improvements:
  • Made more user-friendly, clear, and concise
  • Improvements to the diagnostic plots
  • Added a capability to overlay the IFU footprint on an image of the source in the pipeline helper task plotCubesRaDec

• New/improved scripts have been added to the PACS Useful scripts menu:
  • Fitting mapping observations (from drizzled, interpolated, and projected cubes and secondly from rebinned cubes).
  • Fitting single pointings starting from interpolated cubes
  • A new script to convolve 2 spectral images to the longer wavelength resolution
  • A new script to re-create the standalone browse products
  • A script for point sources located in the central spaxel and a script for point sources located in another spaxel
  • A script summarising the post-processing tasks for extended sources

• The meta keywords of all products have been improved. Most meta keywords now have an 8 character FITS equivalent, so fewer META_XX keywords will appear in the FITS files of the images and cubes.
• The ra/dec of the Level 1/2 products now reflect the actual average centre of the particular cube/image containing these keywords. Only raNominal/decNominal contain the observer-requested pointing coordinates.
• Simplified the structure of the Level 0--2 data in the observation context (fewer layers).

• Improved the browse images, and replaced the browse products (the images and cubes) with links to the appropriate Level 2 products. Note that for spectroscopy the images are not necessarily created from the cubes that are the "browse products".
• Removed redundant quality flags (for all SPG 13 products).
• Improved the quality comments (will be visible for products of SPG 13 but only towards the end of 2015)

• Drizzled cubes are also provided in the context under HPS3DDR and HPS3DDB
• We provide interpolated cubes with a 4.7" spaxel size in the context under HPS3DIR and HPS3DIB.

• • A single table (all spaxels, all raster position, but only one wavelength range per table) with all the spectral data from the rebinned cubes (PacsSpecTable product under HPSTBRR and HPSTBRB).
  • Cubes regridded on to an equidistant wavelength grid (stores in the context under HPS3DEQ*), which are easier to inspect in viewers such as DS9, since the wavelength grid is now part of the WCS.
• We now provide Standalone Browse Products via the HSA, and these also can be found in the "browse product" part of any observation context. These products are: the new cubes we create with an equidistant spectral grid (HPS3DEQ[I|R|D][R|B], and the rebinned cubes tables (HPSTBR[R|B]).

• Detailed documentation of Unimap has been added to the PACS Data Reduction Guide for photometry.
• Created PACS Products Explained document to describe and explain PACS products: from the observation context to the final maps and cubes
• Improved the first two chapters of the PACS Data Reduction Guide for spectroscopy to help archive users decide what to do with their observations downloaded from the HSA: re-process or additionally process in HIPE or not? The description of the crucial pipeline tasks has also been improved.
• Cookbooks to guide users through certain common tasks done with PACS data in HIPE will be released, beginning in the summer of 2015

• • HIFI is using a particular approach in the usage of the new gyro-based pointing reconstruction. All details and impact on the HIFI data are explained in this memo.

• HIFI has a particular approach to the new gyro-based pointing reconstruction. The peculiarity of the approach and recommendation to the archive user are summarised in this memo.

• The doPointing task is checking a quality figure in the pointing product in order to decide between the gyro-based or simple pointing reconstruction

• • in case the gyro-based reconstructed pointing is considered to be under-performing, and the pointing product in used is reverted to the simple Pointing reconstruction from HIPE 12

• Unimap: a new Generalized Least Square (GLS) mapmaker is introduced into the Standard Product Generator. Unimap is a MATLAB algorithm that is spawned from HCSS. Level2.5 and Level3 products generated by Unimap replace MadMap maps for SPG13.

• Ipipe scripts: MadMap pipeline is still available as interactive script in the Pipelines menu. The interactive script scanmap_Unimap allows the users to run Unimap by tuning some parameters. It uses the new task runUnimap(), that calls the Unimap release available within the user’s computer.

• Temporal files for mapping are now created in a subdirectory: each HIPE session running will use a different subdirectory.

What's new in HIPE 13

• Core
  • New auxiliary product OperationalDaySummaryProduct that helps filtering and handling data per Operational Day (OD) and new gyro-based pointing reconstruction algorithm.
  • HIPE now opens ALMA files and identifies ALMA cubes as spectral cubes, displayed in the Spectrum Explorer.
  • A rescue mode has been added to HIPE for cases where intensive scripts or memory thrashing issues make the GUI unresponsive.

• New chapter on converting units in the Data Analysis Guide.
• Many more scripts in the manuals are tested periodically to ensure that they work correctly.

• • Level 2.5 for Point mode observations contains spectra that are stitched (HRS spectra will contain NaNs between subbands, unless they overlap in frequency) and converted to simple spectrum. Frequency Switch observations are folded.

• Point mode: Level 2 HTPs are stitched. HRS spectra will contain NaNs between subbands unless the subbands overlap in frequency. If the data is frequency switched it is then folded. All spectra are converted to simpleSpectrum format.

• Mapping mode: Level 2 HTPs are stitched and stored in Level 2.5. Cubes are then generated from the stitched HTP. HRS data are only stitched if the subbands are overlapping in frequency. Therefore you will find one cube for each WBS polarisation and backend and the number of cubes generated for the HRS depends on the spectral setup for the observation. In the case of frequency switch observations, spectra are folded after stitching and prior to gridding.

• • A new chapter describing tools to use the new HIFI beam model, Understanding and using HIFI beam information in your data is now available.
  • Updates to the chapter Running the HIFI pipeline for changes to the Interactive Level 2.5 pipeline, particularly the use of mkRms.

• • Updates to the chapter Unit conversions for changes to convertK2Jy

• • Updates to all cookbooks to describe the OFF position spectra provided in the ObservationContext Calibration tree (ReferenceSpectra).

• • Updates to the chapter Exporting HIFI data to CLASS.
  • Updates to all cookbooks to introduce the new OFF position spectra now provided in the ObservationContext Calibration products (ReferenceSpectra).

• A list of optimised channel flags for spectral scans obsids, that will be honoured by the deconvolution task and result in spur-free level 2.5 data in this mode. This is not yet available for all spectral scan observations but work will be completed by HIPE 14.

• Updated beam efficiencies and 2D beam models. For more details see the HIFI Instrument and Calibration Webpage.

• The Electrical Standing Wave correction is now automatically performed on the level 1 data, by the doHebCorrection task. It is based on optimum correction solutions computed by the HIFI team and stored in the calibration tree.
• A new task, mkUncertaintyTable creates a table from the uncertainty values present in the calibration tree and puts them into the calibration out part of the tree

• Mapping mode: Level 2 HTPs are stitched and stored in Level 2.5. Cubes are then generated from the stitched HTP. HRS data are only stitched if the subbands are overlapping in frequency. Therefore you will find one cube for each WBS polarisation and backend and the number of cubes generated for the HRS depends on the spectral setup for the observation. In the case of frequency switch observations, spectra a folded after stitching and prior to gridding.
• Point mode: Level 2 HTPs are stitched. HRS subbands are stitched only if they overlap in frequency. If the data is frequency switched it is then folded. All spectra are converted to simpleSpectrum format.
• A large number of statistics (primarily noise and mean) is performed both on the level 2 and level 2.5 products, as well as on an equivalent products converted to the Tmb scale with H and V merging (this latter is then removed from the observation context and only the statistics is stored). This computation is done by the =mkRms task and is controlled by the option computeRms. The calculation can be extremely time consuming for maps and spectral scans and is, by default, is disabled in HIPE (it is always done during bulk reprocessing at the HSA). You can turn it on in a HIPE session by setting computeRms=True

• New quality flags informing:
  • if it was not possible to apply the HEB electric standing wave correction
  • if the intended and computed pointing differ by more than three times the Absolute Pointing Error (APE)
  • if the APE check could not be performed

• Introduction of HifiTool. This is an interface that allows you to conveniently display HIFI spectra with commonly used axes (velocity, USB, LSB, IF, Ta*, Jy). Click on the HIFI ICC icon to activate the tool.

• • Updated beam models and efficiencies. See http://herschel.esac.esa.int/twiki/pub/Public/HifiCalibrationWeb/HifiBeamReleaseNote_Sep2014.pdf for more details

• • Level 2.5 for Point mode observations contains spectra that are stitched (HRS subbands are only stitched if they overlap in frequency) and convert to simple spectrum.
  • Level 2.5 for Maps contain HTPs of stitched spectra in addition to cubes created from the stitched spectra (HRS subbands are only stitched if they overlap in frequency).
  • Many Spectral Scan observations benefit from flag tables optimised manually by the ICC, resulting in spur-free deconvolved spectra.
  • All observations now contain a (Ta*) calibrated OFF position spectrum, in the same sky frequency scale as the ON position data.

• A new level 2.5 product has been generated for point mode observations. It is a stitched version of the level 2 spectra for each available spectrometer, in the form of a Spectrum1d.

• • Spectrometer: Cubes with evenly sampled spectral info. Improved mapping with additional 2D interpolation task.
  • Spectrometer and photometer: many improvements to the structure of the observation context and names and descriptions of products.

• Flags are now also propagated by specProject and specInterpolate.

• specSubtractOffPosition can now also combine off-positions using several algorithms and works with non-rebinned PacsCube products as well. The task specSubtractOffPositionNotRebinned is now superfluous and has been deprecated.

• Fixed many problems with the mask editing functionality of the Spectrum Explorer on PACS frames and cubes.

• By default, the option chosen for spur_rejection will be based on the availability of not of optimised flag tables for a given obsid (see also the calibration section). When no table exists, the option is enforced to REJECT_SCANS_WITH_SPURS. When tables exist, the option is enfored to DO_NOT_REJECT_SCANS_WITH_SPURS

• doDeconvolution has also been updated so it can take into account the IF-dependence of sideband ratio and use this a priori knowledge as preset to build the single sideband solution

• It is now possible to use the mergeHtps task to combine HTPs prior to gridding. This can be used for combining any number of HTPs but is particularly useful for combining the H and V polarisation data of a given observation. Previously this could be done by following a script provided in the 'How to make a Spectral Cube' chapter of the HIFI Data Reduction Guide.

• maskTables: a new variable to manage Linemasks and Rowmasks files.

• identifyLines is a new task that allows you to identify lines in your spectrum and then run a comparison of known lines with a Linelist.

• • Updates to the chapter Standing Wave Removal (fitHifiFringe task) - note also: new syntax for startPeriod, endPeriod, typicalPeriod, and subBase.

• PlotXY no longer has an explicit way of forcing a batch of plotting statements. This is now done automatically and the script developer must not do anything special to take advantage of it.

• Chop/nod SPG pipeline now uses the background normalisation method instead of the calibration blocks for the flux calibration.

• Interpolated cubes are computed for pointed observations with the new task specInterpolate.

• Created PACS Products Explained document to describe and explain PACS products, from ObservationContext to the final maps and cubes
• Cookbooks to guide users through certain common tasks done with PACS data in HIPE will be released, beginning in the spring of 2015

Pointing Product

• Gyro-based attitude reconstruction, see pdf.

• Mapping uses subdir per HIPE session for temp files

• Preference "Generate task calls that returns full set of outputs" removed: tasks GUIs now always generate a call with the expanded list of outputs assigned to.
• Task status messages appear in the status bar

• saveObservation:
  • Improved documentation

• fitsReader:
  • Identify velocity cubes (CTYPE3 = VELO-LSR) as spectralCubes
  • Identify ALMA/CASA cubes as spectralCubes

• The link to VOPlot in the External Tools sub-menu will be removed and substituted by Topcat.
• Improved messages when failing to launch external applications
• When you send a table to Topcat it returns a table (before it returned a product)

• Editing metadata and dataset cells in the Product or Observation viewers keeps the result in memory, and the table is refreshed accordingly with the new value.

• Conversion tool to adapt multi-output tasks to new style in off-line User scripts.

• • Now includes the possibility to retrieve observation using the SPG version that created the Observation (parameter spgVersion).

• asciiTableReader:
  • Improved GUI
  • Improved error reporting for IPAC tables
• "Export Herschel data from HIPE" view:
  • automatically add the proper extension to output files
  • Improved GUI: changed default tab

• • Level 2.5 cubes for maps are now stitched

• The Electrical Standing Wave correction is now automatically performed on the level 1 data, based on optimum correction solutions computed by the HIFI team and stored in the calibration tree.

• The spur detection is now handed over to the direct application of an a priori spur table stored in the calibration tree. Obsids for which no entry exist will not have any spurs flags. This calibration tree is currently being populated and already covers hundreds of observations (mostly spectral scans) in HIPE 13. For those spectral scans, the users can expected improved quality of the deconvolved spectra at level 2.5

• The annotation in Display are totally reworked. All old annotation methods are deprecated, the new one end with Annotation. For example : addEllipse becomes addEllipseAnnotation.

• Display now displays images with some extremely high or low values correctly.
• It is now possible to extract the current layer from a SimpleImage or from a SimpleCube with only one layer.
• Display can now work with more than one extension.

• You can choose to add an input parameter to annularSkyAperturePhotometry for choosing the step size of curve-of-growth output.

• multiFit task now accepts limits for the model parameters.

• Disable the selection tools when some other data cube than the flux is shown.
• Fixed bug in which the upper and right rows and columns were not included in the selection for the cropCube task.

• Moment algorithm in computeVelocityMap improved with 'line strength' weighting. Especially dispersion is more accurate.
• Show proper model cube when absorption line fitting is done in computeVelocityMap.
• More metadata in extractRegionSpectrum.

• A list of optimised channel flags for spectral scans obsids, that will be honoured by the deconvolution task and result in spur-free level 2.5 data in this mode. This is not yet available for all spectral scan obsids though, but will be by HIPE 14.
• Solution tables for Electrical Standing Wave correction in the bands 6 and 7. This allows for a correction of this affect in the pipeline itself. Note that this is not a guarantee that the baseline artefact will be perfectly removed but it will provide the best correction available to data. It is then possible to further optimise the correction - see the dedicated sections in the DRG.
• Updated beam efficiencies and 2D beam models

• The sideband ratio correction is now taking into account the possible IF-dependency of the sideband ratio over the IF. It uses the IF-dependence information as given in the calibration tree. As of now, only the lower end of band 2a makes such an assumption. The presence of an IF-dependency will result in additional meta-data usbGain_n or lsbGain_n, which are the coefficients used to described the polynomial approximation of the gain dependence. Coefficients with index _0 simply correspond to the value of the constant, scalar, sideband ratio we used so far
• A new task will automatically check for platforming in the WBS data and report any detection with a dedicated quality flag
• A new task will create spectra for the OFF position data calibrated into the Ta* scale, on both USB and LSB frequency scales. For spectral scans, these can then be fed into deconvolution to create a deconvolved spectrum of the OFF position (option use_reference)

• The !PolarPair class has been removed. Instead the !polarPair task should be used. This should only affect scripts written prior to 2013.

• The task doHebCorrection is now part of the default pipeline, with several new options compared to its predecessor hebCorrection.

• By default, the option chosen for spur_rejection will be based on the availability of not of optimised flag tables for a given obsid (see also the calibration section). When no table exists, the option is enfored to REJECT_SCANS_WITH_SPURS. When tables exist, the option is enfored to DO_NOT_REJECT_SCANS_WITH_SPURS

convertK2Jy

flagTool

identifyLines

• IdentifyLines is a new task that allows you to identify lines in your spectrum and then run a comparison of known lines with a Linelist.

• specSubtractOffPosition can now also combine off-positions using several algorithms.

• specDiffChop now also works on calibration blocks.
• slicedSummary works now also on rebinned/projected cubes.

• New SPG map merging script utilising the 2-pass pipeline to produce Level 2.5 products. All parallel mode observations are reprocessed together as pairs at the Level 2.5 stage. After the level 2.5 stage all the Level 0.5, Level 1,Level 2, Level 2.5 data are updated with the improved quality data.

• Minor updates, bug fixes, and improved documentation for the Semi-extended correction task.
• Updates to the task to take synthetic photometry of FTS spectra for comparison with the SPIRE photometer.
• New GUI interface for the script that determines the noise level in a spectrum.
• Small corrections to the useful scripts.

• getObservation (getObs task):
  • Now includes the possibility to retrieve observation using the SPG version that created the Observation.
  • The task path and poolLocation parameters have been consolidated. See up to date Task definition with examples.

• Reading of ALMA/CASA cubes directly into HIPE and the Spectrum Explorer: fitsReader and importSpectralCube

• Disable the selection tools when some other data cube then the flux is shown.
• Fixed bug in which the upper and right rows and column were not included in the selection for the cropCube task.

 Analysis

• Moment algorithm in computeVelocityMap improved with 'line strength' weighting. Especially dispersion more accurate.
• Show proper model cube when absorption line fitting is done in computeVelocityMap.
• More metadata in extractRegionSpectrum.

• New long/short term Transient Correction algorithm and script.
• Extra options to wavelengthGrid task to make regular grids with either the mean -, a fraction of the minimum -, or a specified bin size.
• specSubtractOffPosition can now also combine off-positions using several algorithms.
• An image can be used as model for extended to point correction task.
• Improvements to Pointing Offset Correction.
• specDiffChop now also works on calibration blocks.
• slicedSummary works now also on rebinned/projected cubes.
• The footprint viewer is now a real viewer, which can be opened with the "Open with" menu on a product.
• Interactive script improvements:
  • Made more user friendly, clear and with less verbose info.
  • Improvements of diagnostic plots.
  • New scripts for fitting mapping observations (from rebinned and projected cubes).
  • A new script to fit single pointings on interpolated cubes.
  • A new script to convolve 2 spectral images to the longer wavelength resolution.
  • Added a capability to overlay the footprint on an image.

• The meta keywords of all products have been improved. Especially every meta keyword has now a 8 character FITS equivalent, so that the META_XX keywords will not appear anymore in the FITS files for any of the important keywords.
• The ra/dec of the level-1/2 products now reflect the actual average center of the particular cube/image containing these keywords. Only raNominal/decNominal contain the requested pointing coordinates.
• Simplified the structure of the level 0-2 data in the observation context (fewer layers).
• Moved the observation summary to the top level of the observation context and improved its contents.
• Improved browse images, replaced browse product by links to the level-2 products.

• Chop/nod SPG pipeline now uses the background normalization method instead of the calibration blocks for the flux calibration.
• Drizzle now also done for the unchopped line observations.
• Interpolated cubes are computed for pointed observations with the new task specIterpolate.
• Drizzled, projected or interpolated cubes are converted to an equidistant wavelength grid cube with complete standard WCS using the new task specRegridWavelength. These equidistant products are stored next to the non-regular wavelength grid products in the level-2 context.
• Tables containing all the rebinned cubes per camera are produced with the new task pacsSpecCubeToTable.

Analysis tools

Spectroscopy

• Drizzled cubes are also provided for unchopped observations.
• We provide interpolated cubes for pointed observations (with 4.7" spaxel size) in the context under HPS3DIR and HPS3DIB.
• New level-2 products for easier handling outside of HIPE:
  • Single table with all the rebinned cubes (PacsSpecTable product under HPSTBRR and HPSTBRB).
  • Many cubes are also regridded to an equidistant wavelength grid (store in the context under HPS3DEQ*), which are easier to view in viewers like DS9, since now also the wavelength is part of the standard WCS.

ParameterFactory added to easily create meta data parameters.

• Now includes the possibility to retrieve observation using the SPG version that created the Observation.
• The task path and poolLocation parameters have been consolidated. See up to date Task definition with examples.

• It is possible to create Quality Control summaries for several instruments in one go.
• The QCR Browser is more interactive, as it is no longer blocked when performing an operation. This way, tasks can be performed in parallel on different windows.
• Panels in the QCR Display are ordered in a more sensible way.
• The OD is shown in the QCR Display when available.
• The operator can remove comments from several observations in a single operation.
• The report showing the result of an ingestion or removal operation can be saved to a text file.
• Specifying a ticket number for several observations is now possible.

A new utility method in the Product class for querying a products duration.

• Unimap: a new Generalized Least Square (GLS) mapmaker is introduced into the Standard Product Generator. Unimap is a MATLAB algorithm that is spawned from HCSS. Level2.5 and Level3 products generated by Unimap replace MadMap maps for PG13.
• Level2 products: the quality of the High-pass Filtered Level2 products (HPPPMAP) is improved by adding the scanamorphosMaskLongTermGlitches() and the scanamorphosBaselinePreprocessing() tasks, which are capable to detect jumps and to correct drift on the detector timelines.
• Level2.5 products: Unimap Level2.5 contexts contain the same types of maps generated by Madmap (naïve, GLS and Post-GLS), nevertheless they are differently arranged within the context (see Product changes).
• Level3 products: High-pass Filtered maps (HPPHPFMAP) are removed from Level3 products, having finally JScanam and Unimap maps. Level3 mosaics are created by using scanamorphosAlignImages(), a new task that links the flux by exploiting the overlapping regions between images.
• Ipipe scripts: MadMap pipeline is still available as interactive script in the Pipelines menu. The interactive script scanmpa_Unimap allows the users to run Unimap by tuning some parameters. It uses the new task runUnimap(), that calls the Unimap release available within the user’s computer.
• Harmonisation of map pixel sizes: at any Level, maps have the following pixel sizes:
  • 3.2 arcsec for the red channel, Prime and Parallel Modes;
  • 1.6 arcsec for the blue channel, Prime Mode;
  • 3.2 arcsec for the blue channel, Parallel Mode.
• Correction of timing shifts: frame pixels are not simultaneously read, but in SPG12 the frames are associated to a single readout time. That results in a time shift that can correspond up to 2 arcsec delay for fast scanning. This effect is corrected by setting the parameter correctReadoutTimes=True by default in every task that is affected by time offsets

What about gyro attitude?? What about qflags?

• The naming convention for Unimap Level2.5/3 is HPPUNIMAP[B][R], for the blue and red channel, respectively.
• A single context is released for Unimap Level2.5. The Unimap context contains the following products:
  • image is the Weighted-GLS map. Generally, it is the most reliable product;
  • pgls is the Post-GLS map;
  • gls is the GLS map;
  • naive is the Naïve map;
  • error is that is the standard deviation map. It can be properly assumed as the error map only if associated to the Naïve map.

• Improved the table of contents of the PACS User Reference Manual. Created a new content for the runUnimap() task.
• Detailed documentation of Unimap within the PACS Data Reduction.

• HIFI
  • Bands 6&7 corrected for electric standing waves.
  • Level 2.5 for point spectra WBS subbands and convert to simple spectrum (the polarisations will still not be averaged).
  • New HIFI beams.

• PACS
  • Photometer: New maps valid for point source and extended emission analysis: JScanam, UniMap (new!).
  • Spectrometer: Cubes with evenly sampled spectral info.

• SPIRE
  • Photometer: Better deglitched maps (multi-pass pipeline).
  • Spectrometer: Improved product structure inside observation contexts.

• The browse image of the Observation viewer can be opened with the Image viewer by means of Right click > Open in image viewer.

• The Help > Check Java platform option shows the running Java version.
• The Login panel is shown back at the top left corner of the Data Access perspective, and it has been simplified.

• New SPG map merging script utilising the 2-pass pipeline to produce Level 2.5 products. All parallel mode observations are reprocessed together as pairs at the Level 2.5 stage. After the level 2.5 stage all the Level 0.5, Level 1,Level 2, Level 2.5 data are updated with the impoved quality data.

• Animations when maximizing or restoring parts of the viewer will be shown (see Transitions).

• Column objects from a TableDataset have now a custom viewer.

• The new 2-Pass Pipeline has replaced the standard pipelines for Large, Small and Parallel modes. The 2-Pass Pipeline runs iteratively on the data to produce optimally cleaned timelines. The first pass is run with minimal tasks (e.g. no FT tasks). Glitches are detected and flagged using 2nd level deglitching outlier detection. The full pipeline is then run using all the standard tasks.
• New useful scripts to allow users to reconstruct the beam model and all calibration parameters (Photometer Calibration scripts) for both point, extended and semi-extended sources.
• New SPG map merging script utilising the 2-pass pipeline to produce Level 2.5 products.
• Optimized parameters for Timeline Fitter in photometry script. Background annulus changed for faster run time.

• Backups for new files are automatically created, and reopened in a subsequent session upon crashes like a power cut. If the session terminates gracefully, the backup files are deleted.
• Animations when maximizing or restoring parts of the viewer will be shown; see Transitions.
• The ArrayDataset viewer shows the unit when specified.

• Significant improvements to the bolometer finder tool have been made. 

Outine

• The Outline will show a small image for applicable variables like spectra, simple images, etc. when selected in the Variables view.
• When selecting String variables, both the string contents and a literal to use it in a script will be shown in the Outline.
• When selecting Unit variables, specific information about the unit is shown.

• When saving a file of a known type with the file dialogue, the extension is automatically added to the file name if not explicitly done.
• The filter for file types in the Navigator view is now more visible. In addition, it includes all kinds of defined file types.

Miscellany

The Help > Check Java platform option shows the running Java version.

import sys
print str(sys.argv)

Then the output of this script would be:

> hipe test.py arg1 arg2 arg3
['test.py', 'arg1', 'arg2', 'arg3']

Transitions

In order to give better user feedback on GUI operations, screen transitions have been introduced in the following cases:

• Switching perspective is performed by fading to the new perspective.
• Navigation between pages in the welcome perspective is also done by fading to the new page.
• Maximizing or restoring the data viewer within the observation context viewer is shown with an animation.
• Maximizing or restoring the browse image of the observation context viewer is animated too.

Editors & Viewers

Status bar

• Show the MyHSA on-line/off-line checkbox in the status bar.
• Hide the MyHSA on-line/off-line checkbox in the status bar.

File handling

Rescue mechanism

 TablePlotter

 OverPlotter

• convertUnits : It can convert units from double parameters, array datasets and columns, plus units from table datasets or products, by allowing to select the target dataset or column.

• In order to prevent fitBaseline from crashing when 'useWbs' is set to 'True', especially when doglue is also set to 'True', the task will automatically override doglue to 'False'.

Photometer

• The beam profile maps are improved using the shadow observations of Neptune (i.e. better characterising and removing the background).
• The corresponding beam areas given in the metadata of the beam products are updated accordingly.
• The radial beam calibration product has been updated to match the new beams.
• All the colour correction products have been updated following the beam updates.
• In order to improve the operation of the source extractors, the beam profiles are adjusted to be relevant for a 1 Jy source - previously the peak of the profile was fixed to 1 Jy, but the new profiles integrate to 1 Jy. In addition, the map consists of an odd number of pixels so that the beam centre is precisely at the centre of the central pixel.
• A new "fine" beam map is provided for each band with 1" pixels. The "nominal" maps are also available with pixel sizes of 6", 10" and 14" for PSW, PMW and PLW.
• The K4 parameters in the Flux Conversion product metadata are updated (very small change to K4E)
• The Planck HFI colour correction table has been updated following the beam updates (very small change).
• The parameters in the temperature drift calibration product that are used to perform cooler burp correction have been updated to improve the correction.

• The non-linearity product has been updated to better fit observations made with low subktemp at the beginning of each pair of FTS observing days (i.e. the observations following the cooler recycle). This results in a much improved SLWC3 spectral shape.
• The other RSRF, telescope model and flux conversion products were updated following the non-linearity update.
• The PhaseCorrLim product was updated to correct the metadata - no effect on the data.
• The TempDriftCorr product has been removed as it is not used by the pipeline.

• When a large portion of channels is flagged as invalid, i.e. when not enough channels can be fitted, fitHifiFringe will raise a comment that it is skipping the spectrum and then it will continue to the next one.

• In order to correct fitBaseline from crashing when 'useWbs' is set to 'True', especially when doglue is also set to 'True', the task will automatically override doglue to 'False'.

• A new level 2.5 products has been generated for point mode observations. It is a stitched version of the level 2 spectra for each available spectrometer, in the form of a Spectrum1d.
• All observations context will now contain a (Ta*) calibrated OFF position spectrum, in the same sky frequency scale as the ON position data.   * The new calibration tree contains updated beam models and efficiencies, as released initially as stand-alone products in September 2014. See for more details http://herschel.esac.esa.int/twiki/pub/Public/HifiCalibrationWeb/HifiBeamReleaseNote_Sep2014.pdf
• In bands 6 and 7, the Electrical Standing Wave correction is now part of the standard pipeline, based on a data-base of optimised solutions stored in the calibration tree for each observation
• For spectral scans, a significant fraction of the deconvolved level 2.5 products will benefit from flag tables optimised manually by the ICC, resulting in spur-free deconvolved spectra. This is not yet available for all spectral scan obsids though, but will be by HIPE 14. The list of optimised flags can be found in the calibration tree and further refined by the user using the flagTool if the User wishes so.

• it will contain a list of optimised channel flags for spectral scans obsids, that will be honoured by the deconvolution task and result in spur-free level 2.5 data in this mode. This is not yet available for all spectral scan obsids though, but will be by HIPE 14.
• it will contain solution tables for Electrical Standing Wave correction in the bands 6 and 7. This allows for a correction of this affect in the pipeline itself. Note that this is not a guarantee that the baseline artefact will be perfectly removed but it will provide the best correction available to data. It is then possible to further optimise the correction - see the dedicated sections in the DRG.
• it will contain the updated beam efficiencies and 2D beam models

• the sideband ratio correction is now taking into account the possible IF-dependency of the sideband ratio over the IF. It uses the IF-dependence information as given in the calibration tree. As of now, only the lower end of band 2a makes such an assumption. The presence of an IF-dependency will result in additional meta-data usbGain_n or lsbGain_n, which are the coefficients used to described the polynomial approximation of the gain dependence. Coefficients with index _0 simply correspond to the value of the constant, scalar, sideband ratio we used so far
• a new task will automatically check for platforming in the WBS data and report any detection with a dedicated quality flag
• a new task will create spectra for the OFF position data calibrated into the Ta* scale, on both USB and LSB frequency scales. For spectral scans, these can then be fed into deconvolution to create a deconvolved spectrum of the OFF position (option use_reference)

• Point mode: Level 2 HTPs , stitched, are forming the Level 2.5 products for this mode.
• A large number of statistics (primarily noise and mean) is performed both on the level 2 and level 2.5 products, as well as on an equivalent products converted to the Tmb scale with H and V merging (this latter is then removed from the observation context and only the statistics is stored). This computation is controlled by the option computeRms. It is only set to True when run for the bulk reprocessing, but ignored by default when the level 2.5 pipeline is called interactively (this computation can be extremely time consumming for maps and spectral scans)

• the task doHebCorrection is now part of the default pipeline, with several new options compared to its predecessor hebCorrection.

• by default, the option chosen for spur_rejection will be based on the availability of not of optimised flag tables for a given obsid (see also the calibration section). When no table exists, the option is enfored to REJECT_SCANS_WITH_SPURS. When tables exist, the option is enfored to DO_NOT_REJECT_SCANS_WITH_SPURS
• A new option called enforce_spur_rejection is however made available to help control the existing spur_rejection option.
• A new option use_reference indicates that the deconvolution should be made on the OFF position spectra computed with HIPE 13
• doDeconvolution has also been updated so it can take into account the IF-dependence of sideband ratio and use this a priori knowledge as preset to build the single sideband solution

convertK2Jy

• with the availability of the new beam efficiencies and beam model, the task has been upgraded in order to make best use of the detailed beam knowledge. A new option hifiBeam, set to True by default, will use the fully-detailled HIFI beam to compute the coupling factor involved in the temperature to flux conversion for sources of non-zero size.
• for non-point-like sources, the task now offers disk or Gaussian source models (option shape)

• • Updates to the chapter How to make a spectral cube to reflect changes in the level 2.5.

• New input parameters "cal" and "saveInput" was created in fitHifiFringe to extract user input parameters into the calTree for later use on demand.

• • Updates to the chapter Standing Wave Removal (fitHifiFringe task).

• Replaced setCutLevels() by calcCutLevels()
• Removed setCutLevels(double,double)
• Renamed setCutLevels(double) to setCutLevelsPercentage(double)

• The images can not only be flipped around the y-axis, but also around the x-, xy- and yx-axes. It is also possible to combine these flip-options : for example : y- and xy.
• The annotation in Display are totally reworked. All old annotation methods are deprectated, the new one end with Annotation. For example : addEllipse becomes addEllipseAnnotation.

A lot of fixes in Display :

• Display now displays images with some extremely high or low values correctly.
• It is now possible to extract the current layer from a SimpleImage or from a SimpleCube with only one layer.
• Display can now work with more than one extension.
• The zoom in / out was by a factor of 4 instead of 2.
• The Wcs viewer only shows the important wcs keywords now.

• Mosaics close to the north or south pole missed a part of the sky.
• Aperture photometry tasks have clear description on how the background subtracted flux is calculated.
• Bugfixes in aperture photometry tasks.
• You can choose to add an input parameter to annularSkyAperturePhotometry for choosing the step size of curve-of-growth output

• Updates to default band-dependent period characteristics in task fitHifiFringe :
  • Band 1 change nfringe 2->3, endPeriod 120->200
  • Band 3 change typicalPeriod 620->95
  • Band 4 change typicalPeriod 620->95
  • Band 6 change startPeriod 100->80, typicalPeriod 320->600
  • Band 7 change startPeriod 100->80, typicalPeriod 320->625
  • Band 2,5 no change
• Will now handle the combined flags NOT_CALIBRATED and SATURATED.
• swTable had wrong unit for amplitude (kHz instead of K). This is now fixed.
• New output variable swTable was created to enable machine-readable output of fitted frequencies and amplitudes.
• Output for fitHifiFringe now contains corrected data and summary table.
• New input parameters "cal" and "saveInput" was created fitHifiFringe to extract user input parameters into the calTree for later use on demand.

Baseline Removal

• In order to correct fitBaseline from crashing when 'useWbs' is set to 'True', especially when doGlue is also set to 'True', the task will automatically override doGlue to 'False'.

• Mapping mode: The cubes produced automatically by the pipeline (Level 2.5) (doGridding task), and consequently by hifiPipeline if you choose to run it again, will stitch the WBS subbands and create one final cube (with the index '_1'). You will not see individual cubes per subbands except for the HRS.
• Mapping mode: Level 2 HTPs , stitched, are now part of the Level 2.5.

• New option called enforce_spur_rejection to help control the existing spur_rejection option.