HipeWhatsNew13x < Public

Public Web>HipeWhatsNew>HipeWhatsNew13x (2015-09-29, LucaCalzoletti) (raw view)
<!-- ANALYTICS CODE - DO NOT EDIT -->
%INCLUDE{"%ATTACHURL%/GoogleAnalytics.txt" raw="on"}%
<!-- END OF ANALYTICS CODE -->
---+ What's new in HIPE 13

| *PDF Version* | [[%SCRIPTURLPATH%/genpdf/Public/HipeWhatsNew13x?pdforientation=portrait&pdftoclevels=2][Click here]] |
| %ICON{"help"}% *Interested in what's new for other versions of HIPE? See [[HipeWhatsNew][this page]] for links to all the What's New documents from HIPE 4.0 onwards.* |
*See also the [[HipeKnownIssues][HIPE known issues]] and the [[DpKnownIssues][Data products known issues]] pages for last-minute information on known problems.*

%TOC{depth="4"}%

---++ Highlights

   * *Core*
      * New auxiliary product [[http://herschel.esac.esa.int/hcss-doc-13.0/index.jsp#hcss_drm:herschel.ia.obs.auxiliary.od.OperationalDaySummaryProduct][OperationalDaySummaryProduct]] that helps filtering and handling data per Operational Day (OD) and [[ImprovedPointingGyro][new gyro-based pointing reconstruction algorithm]].
      * HIPE now opens ALMA files and identifies ALMA cubes as spectral cubes, displayed in the Spectrum Explorer.
      * A rescue mode has been added to HIPE for cases where intensive scripts or memory thrashing issues make the GUI unresponsive.

   * *HIFI*
      * Updated beam models and efficiencies. See http://herschel.esac.esa.int/twiki/pub/Public/HifiCalibrationWeb/HifiBeamReleaseNote_Sep2014.pdf for more details
      * Bands 6&7 corrected for electric standing waves.
      * Level 2.5 for Point mode observations contains spectra that are stitched (HRS spectra will contain NaNs between subbands, unless they overlap in frequency) and converted to simple spectrum. Frequency Switch observations are folded.
      * Level 2.5 for Maps contain HTPs of stitched spectra in addition to cubes created from the stitched spectra (HRS subbands are only stitched if they overlap in frequency).
      * Many Spectral Scan observations benefit from flag tables optimised manually by the ICC, resulting in spur-free deconvolved spectra.
      * All observations now contain a (Ta*) calibrated OFF position spectrum, in the same sky frequency scale as the ON position data.
      * HIFI is using a particular approach in the usage of the new gyro-based pointing reconstruction. All details and impact on the HIFI data are explained in [[%ATTACHURL%/HIFI_PointingAccuracyNote.pdf this memo]].
      * New HIFI useful scripts

   * *PACS*
      * Photometer: New maps valid for point source and extended emission analysis: JScanam, !UniMap (new!).
      * Spectrometer: Cubes with evenly-sampled spectral grid. Rebinned cubes provided as tables for easier viewing outside HIPE. New cube interpolation task for undersampled mapping observations.
      * Spectrometer and photometer: many improvements to the structure of the observation context and names and descriptions of products.

   * *SPIRE*
      * Photometer: Better deglitched maps (multi-pass pipeline).
      * Spectrometer: Improved product structure inside observation contexts.

<!--

*HIFI*

   * A new level 2.5 product has been generated for point mode observations. It is a stitched version of the level 2 spectra for each available spectrometer, in the form of a Spectrum1d.
   * All observations context will now contain a (Ta*) calibrated OFF position spectrum, in the same sky frequency scale as the ON position data.
   * The new calibration tree contains updated beam models and efficiencies, as released initially as stand-alone products in September 2014. See for more details http://herschel.esac.esa.int/twiki/pub/Public/HifiCalibrationWeb/HifiBeamReleaseNote_Sep2014.pdf
   * In bands 6 and 7, the Electrical Standing Wave correction is now part of the standard pipeline, based on a data-base of optimised solutions stored in the calibration tree for each observation
   * For spectral scans, a significant fraction of the deconvolved level 2.5 products will benefit from flag tables optimised manually by the ICC, resulting in spur-free deconvolved spectra. This is not yet available for all spectral scan obsids though, but will be by HIPE 14. The list of optimised flags can be found in the calibration tree and further refined by the user using the flagTool if the User wishes so.
   * HIFI has a particular approach to the new gyro-based pointing reconstruction. The peculiarity of the approach and recommendation to the archive user are summarised in [[%ATTACHURL%/HIFI_PointingAccuracyNote.pdf this memo]].



*PACS*

   * TBD

*SPIRE*

   * TBD

-->

---++ Core system

---+++ HIPE

---++++ Batch mode

In batch mode, you can use =sys.argv= to get the input arguments. For example:

<blockquote>
<verbatim>
import sys
print sys.argv</verbatim>

</blockquote>

Then the output of this script would be:

<blockquote>
<verbatim>
> hipe test.py arg1 arg2 arg3
['/home/user/test.py', 'arg1', 'arg2', 'arg3']</verbatim>

</blockquote>

---++++ Transitions

In order to give better user feedback on GUI operations, screen transitions have been introduced in the following cases:

   * Switching perspective is performed by fading to the new perspective.
   * Navigation between pages in the welcome perspective is also done by fading to the new page.
   * Maximizing or restoring the data viewer within the observation context viewer is shown with an animation.
   * Maximizing or restoring the browse image of the observation context viewer is animated too.

---++++ Outine

   * The _Outline_ will show a small image for applicable variables like spectra, simple images, etc. when selected in the _Variables_ view.
   * When selecting =String= variables, both the string contents and a literal to use it in a script will be shown in the _Outline_.
   * When selecting =Unit= variables, specific information about the unit is shown.

---++++ Editors & Viewers

   * Backups for new files are automatically created, and reopened in a subsequent session upon crashes like a power cut. If the session terminates gracefully, the backup files are deleted.
   * Editing metadata and dataset cells in the Product or Observation viewers keeps the result in memory, and the table is refreshed accordingly with the new value.
   * Animations when maximizing or restoring parts of the viewer will be shown (see _Transitions_).
   * The browse image of the Observation viewer can be opened with the Image viewer by means of _Right click &gt; Open in image viewer_.
   * The =ArrayDataset= viewer shows the unit when specified.
   * =Column= objects from a =TableDataset= have now a custom viewer.

---++++ Status bar

From the _Preferences &gt; Data Access &gt; !MyHSA &gt; Advanced_ panel, the user can:

   * Show the !MyHSA on-line/off-line checkbox in the status bar.
   * Hide the !MyHSA on-line/off-line checkbox in the status bar.

This label provides a more prominent location reflecting the current connection status of !MyHSA, irrespective of what view the user is currently using.
It also offers the ability to update this status, the status is synced with that shown in the product browser and vice versa.

---++++ File handling

   * When saving a file of a known type with the file dialogue, the extension is automatically added to the file name if not explicitly done.
   * The filter for file types in the _Navigator_ view is now more visible. In addition, it includes all kinds of defined file types.

---++++ Rescue mechanism

If HIPE gets in a non responding state for a period of time, it is automatically detected so that a popup window asks to restart it in an ordered way. This means all unsaved editors will be saved, and a new fresh HIPE session with the state of the frozen one will be opened.

The time intervals to run this check and to consider the application as not responding can be configured in _Preferences &gt; General &gt; Rescue_, where this mechanism can also be disabled altogether if you prefer so.

---++++ Miscellany

   * The _Help &gt; Check Java platform_ option shows the running Java version.
   * The _Login_ panel is shown back at the top left corner of the _Data Access_ perspective, and it has been simplified.
   * HIFI is now providing a handful of useful scripts - see the menu Scripts &gt; HIFI Useful Scripts.

---+++ Plotting
   * =PlotXY= no longer has an [[http://herschel.esac.esa.int/hcss-doc-13.0/load/hcss_drm/api/herschel/ia/gui/plot/PlotXY.html#setBatch(boolean)][explicit way]] of forcing a batch of plotting statements. This is now done automatically and the script developer must not do anything special to take advantage of it.

---+++ Astronomical utilities
   * Gyro-based attitude reconstruction, see [[ImprovedPointingGyro][the dedicated section ]] in the Spacecraft and Observatory pages.
   * Special corrections in pointing products to handle S/C operational events have been included in the auxiliary processors. Namely, the corrections are:
      * STR switch-overs, i.e. changes from the usually prime star-tracker (STR1) to the backup unit (STR2)
      * Reset of the Spacecraft Velocity Vector (SVV) that is used by the STR to compute the aberration correction of the coordinates of guide stars.

   For more information please refer to the  [[StrSwitchOversAndSvvResets][the dedicated section ]] in the Spacecraft and Observatory pages.

---+++ Numeric routines
---++++ Mapping
   * Temporal files for mapping are now created in a subdirectory: each HIPE session running will use a different subdirectory.

---+++ Images

---++++ Display

The following changes are made to the ==Display== class:
   * Replaced =setCutLevels()= by =calcCutLevels()=
   * Removed =setCutLevels(double,double)=
   * Renamed =setCutLevels(double)= to =setCutLevelsPercentage(double)=

   * The images can not only be flipped around the y-axis, but also around the x-, xy- and yx-axes. It is also possible to combine these flip-options : for example : y- and xy.
   * The annotation in ==Display== are totally reworked. All old annotation methods are deprecated, the new one end with ==Annotation==. For example : =addEllipse= becomes =addEllipseAnnotation=.

A lot of fixes in ==Display==:
   * ==Display== now displays images with some extremely high or low values correctly.
   * It is now possible to extract the current layer from a ==SimpleImage== or from a ==SimpleCube== with only one layer.
   * ==Display== can now work with more than one extension.
   * The zoom in / out was by a factor of 4 instead of 2.
   * The Wcs viewer only shows the important wcs keywords now.

---++++ Analysis

Bugfixes :
   * Mosaics close to the north or south pole missed a part of the sky.
   * Aperture photometry tasks have clear description on how the background subtracted flux is calculated.
   * Bugfixes in aperture photometry tasks.
   * You can choose to add an input parameter to annularSkyAperturePhotometry for choosing the step size of curve-of-growth output.

---+++ Spectra
---++++ Display

    * The mosaic view of the Spectrum Explorer shows axes coordinates in a more convenient way, with less digits.

---++++ Analysis

   * ==multiFit== task now accepts limits for the model parameters.

---+++ Data cubes

---++++ Display

   * Disable the selection tools when some other data cube than the flux is shown.
   * Fixed bug in which the upper and right rows and columns were not included in the selection for the =cropCube= task.

---++++ Analysis

   * Moment algorithm in =computeVelocityMap= improved with 'line strength' weighting. Especially dispersion is more accurate.
   * Show proper model cube when absorption line fitting is done in =computeVelocityMap=.
   * More metadata in =extractRegionSpectrum=.

---+++ Products and datasets

A new Auxiliary Operational Day Summary Product which can be retrieved using the tag LATEST_OPERATIONAL_DAY_SUMMARY_PRODUCT. This product contains one dataset with the operational day number, start and end times. Useful methods exist for querying the data as documented here:
 [[http://herschel.esac.esa.int/hcss-doc-13.0/index.jsp#hcss_drm:herschel.ia.obs.auxiliary.od.OperationalDaySummaryProduct][OperationalDaySummaryProduct]]

A new utility method in the =Product= class for querying a products duration.

=ParameterFactory= added to easily create meta data parameters.

Several columns and meta-data keywords related to the gyro-based attitude reconstruction process have been added to the pointing product. See [[ImprovedPointingGyro#Pointing_information_provided][a brief description of the changes]] in the Spacecraft and Observatory pages.
---+++ Product Access Layer 

---+++ Tasks
   * Conversion tool to adapt multi-output tasks to new style in off-line User scripts.
   * %ICON{"warning"}% <b><span class="WYSIWYG_TT">__list__</span></b> argument removed: if your task call still has it, remove it.
   * Preference "Generate task calls that returns full set of outputs" removed: tasks GUIs now always generate a call with the expanded list of outputs assigned to.
   * Task status messages appear in the status bar
   * ==convertUnits== : It can convert units from double parameters, array datasets and columns, plus units from table datasets or products, by allowing to select the target dataset or column.

---+++ Quality control

   * It is possible to create Quality Control summaries for several instruments in one go.
   * The QCR Browser is more interactive, as it is no longer blocked when performing an operation. This way, tasks can be performed in parallel on different windows.
   * Panels in the QCR Display are ordered in a more sensible way.
   * The OD is shown in the QCR Display when available.
   * The operator can remove comments from several observations in a single operation.
   * The report showing the result of an ingestion or removal operation can be saved to a text file.
   * Specifying a ticket number for several observations is now possible.

---+++ Data input-output
   * ==getObservation== ( [[http://herschel.esac.esa.int/hcss-doc-13.0/load/hcss_urm/html/herschel.ia.toolbox.util.GetObsTask.html][getObs]] task):
      * Now includes the possibility to retrieve observation using the SPG version that created the Observation (parameter ==spgVersion==).
      * The task path and poolLocation parameters have been consolidated. See up to date Task definition with examples.
   * ==saveObservation==:
      * Improved documentation
   * Reading of ALMA/CASA cubes directly into HIPE and the Spectrum Explorer: ==fitsReader== and ==importSpectralCube==
   * ==asciiTableReader==:
      * Improved GUI
      * Improved error reporting for IPAC tables
   * ==fitsReader==:
      * Identify velocity cubes (CTYPE3 = VELO-LSR) as spectralCubes
      * Identify ALMA/CASA cubes as spectralCubes
   * "Export Herschel data from HIPE" view:
      * automatically add the proper extension to output files
      * Improved GUI: changed default tab

---+++ Virtual Observatory

   * The link to VOPlot in the External Tools sub-menu will be removed and substituted by Topcat.
   * Improved messages when failing to launch external applications
   * When you send a table to Topcat it returns a table (before it returned a product)

---+++ Documentation
   * New chapter on converting units in the [[http://herschel.esac.esa.int/hcss-doc-13.0/index.jsp#dag:Dag.Chp.UnitConversion][Data Analysis Guide]].
   * Many more scripts in the manuals are tested periodically to ensure that they work correctly.

---++ HIFI

---+++ HIFI Calibration

The calibration tree released together with HIPE 13 contains the following updates:
   * A list of optimised channel flags for spectral scans obsids, that will be honoured by the deconvolution task and result in spur-free level 2.5 data in this mode. This is not yet available for all spectral scan observations but work will be completed by HIPE 14.
   * Solution tables for Electrical Standing Wave correction in the bands 6 and 7. This allows for a correction of this affect in the pipeline itself. Note that this is not a guarantee that the baseline artefact will be perfectly removed but it will provide the best correction available to data. It is then possible to further optimise the correction - see the dedicated sections in the DRG.
   * Updated beam efficiencies and 2D beam models. For more details see the [[HifiCalibrationWeb][HIFI Instrument and Calibration Webpage]].

---+++ HIFI Pipeline
---++++ !hifiPipeline task
---++++ Level 0 Pipeline
   * The doPointing task is checking a quality figure in the pointing product in order to decide between the gyro-based or simple pointing reconstruction
---++++ WBS Pipeline
---++++ HRS Pipeline
---++++ Level 1 Pipeline
   * The Electrical Standing Wave correction is now automatically performed on the level 1 data, by the ==doHebCorrection== task. It is based on optimum correction solutions computed by the HIFI team and stored in the calibration tree.
   * A new task, ==mkUncertaintyTable== creates a table from the uncertainty values present in the calibration tree and puts them into the calibration out part of the tree
---++++ Level 2 Pipeline
   * The sideband ratio correction is now taking into account the possible IF-dependency of the sideband ratio over the IF. It uses the IF-dependence information as given in the calibration tree. As of now, only the lower end of band 2a makes such an assumption. The presence of an IF-dependency will result in additional meta-data usbGain_n or lsbGain_n, which are the coefficients used to described the polynomial approximation of the gain dependence. Coefficients with index _0 simply correspond to the value of the constant, scalar, sideband ratio we used so far
   * A new task will automatically check for platforming in the WBS data and report any detection with a dedicated quality flag
   * A new task will create spectra for the OFF position data calibrated into the Ta* scale, on both USB and LSB frequency scales. For spectral scans, these can then be fed into deconvolution to create a deconvolved spectrum of the OFF position (option _use_reference_)
   * The spur detection is now handed over to the direct application of an a priori spur table stored in the calibration tree. Obsids for which no entry exist will not have any spurs flags. This calibration tree is currently being populated and already covers hundreds of observations (mostly spectral scans) in HIPE 13. For those spectral scans, the users can expected improved quality of the deconvolved spectra at level 2.5
---++++ Level 2.5 Pipeline
   * Mapping mode: Level 2 HTPs are stitched and stored in Level 2.5. Cubes are then generated from the stitched HTP. HRS data are only stitched if the subbands are overlapping in frequency. Therefore you will find one cube for each WBS polarisation and backend and the number of cubes generated for the HRS depends on the spectral setup for the observation. In the case of frequency switch observations, spectra are folded after stitching and prior to gridding.
   * Point mode: Level 2 HTPs are stitched. HRS spectra will contain NaNs between subbands unless the subbands overlap in frequency. If the data is frequency switched it is then folded. All spectra are converted to !simpleSpectrum format.
   * A large number of statistics (primarily noise and mean) is performed both on the level 2 and level 2.5 products, as well as on an equivalent products converted to the Tmb scale with H and V merging (this latter is then removed from the observation context and only the statistics is stored). This computation is done by the ==mkRms= task and is controlled by the option _computeRms_. The calculation can be extremely time consuming for maps and spectral scans and is, by default, is disabled in HIPE (it is always done during bulk reprocessing at the HSA). You can turn it on in a HIPE session by setting _computeRms=True_

---+++ HIFI Products

   * New quality flags informing:
      * if it was not possible to apply the HEB electric standing wave correction
      * if the intended and computed pointing differ by more than three times the Absolute Pointing Error (APE)
      * if the APE check could not be performed
      * in case the gyro-based reconstructed pointing is considered to be under-performing, and the pointing product in used is reverted to the simple Pointing reconstruction from HIPE 12

---+++ HIFI Spectrum Toolbox
---++++ Spectrum Explorer (HIFI specifics)

   * Introduction of !HifiTool. This is an interface that allows you to conveniently display HIFI spectra with commonly used axes (velocity, USB, LSB, IF, Ta*, Jy). Click on the HIFI ICC icon to activate the tool.

---++++ Spectrum tasks (HIFI specifics)

   * The ==!PolarPair== class has been removed. Instead the ==!polarPair== task should be used. This should only affect scripts written prior to 2013.

---+++ HIFI Data Processing Tools
---++++ Standing Wave Removal
   * Updates to default band-dependent period characteristics in task _fitHifiFringe_ :
      * Band 1 change nfringe 2-&gt;3, endPeriod 120-&gt;200
      * Band 3 change typicalPeriod 620-&gt;95
      * Band 4 change typicalPeriod 620-&gt;95
      * Band 6 change startPeriod 100-&gt;80, typicalPeriod 320-&gt;600
      * Band 7 change startPeriod 100-&gt;80, typicalPeriod 320-&gt;625
      * Band 2,5 no change
   * *swTable* had wrong unit for amplitude (kHz instead of K). This is now fixed.
   * New output variable *swTable* was created to enable machine-readable output of fitted frequencies and amplitudes.
   * Output for _fitHifiFringe_ now contains corrected data and summary table.
   * New input parameters "cal" and "saveInput" was created in _fitHifiFringe_ to extract user input parameters into the calTree for later use on demand.
   * When a large portion of channels is flagged as invalid, i.e. when not enough channels can be fitted, *fitHifiFringe* will raise a comment that it is skipping the spectrum and then it will continue to the next one.

---++++ Baseline Removal
   * In order to prevent _fitBaseline_ from crashing when 'useWbs' is set to 'True', especially when doglue is also set to 'True', the task will automatically override doglue to 'False'.

---++++ Electrical Standing Wave correction in HEB bands
   * The task =doHebCorrection= is now part of the default pipeline, with several new options compared to its predecessor =hebCorrection=.

---++++ Deconvolution
   * By default, the option chosen for _spur_rejection_ will be based on the availability of not of optimised flag tables for a given obsid (see also the calibration section). When no table exists, the option is enforced to _REJECT_SCANS_WITH_SPURS_. When tables exist, the option is enfored to _DO_NOT_REJECT_SCANS_WITH_SPURS_
   * A new option called _enforce_spur_rejection_ is however made available to help control the existing _spur_rejection_ option.
   * A new option _use_reference_ indicates that the deconvolution should be made on the OFF position spectra computed with HIPE 13
   * !doDeconvolution has also been updated so it can take into account the IF-dependence of sideband ratio and use this a priori knowledge as preset to build the single sideband solution

---++++ =convertK2Jy=
   * with the availability of the new beam efficiencies and beam model, the task has been upgraded in order to make best use of the detailed beam knowledge. A new option _hifiBeam_, set to True by default, will use the fully-detailled HIFI beam to compute the coupling factor involved in the temperature to flux conversion for sources of non-zero size.
   * for non-point-like sources, the task now offers disk or Gaussian source models (option _shape_)

---++++ Spectral Cube creation
   * It is now possible to use the =mergeHtps= task to combine HTPs prior to gridding. This can be used for combining any number of HTPs but is particularly useful for combining the H and V polarisation data of a given observation. Previously this could be done by following a script provided in the 'How to make a Spectral Cube' chapter of the HIFI Data Reduction Guide.

---++++ =flagTool=
   * Interactive (suitable for batch processing) and Expert modes now available.
   * !maskTables: a new variable to manage Linemasks and Rowmasks files.

---++++ =identifyLines=
   * !identifyLines is a *new* task that allows you to identify lines in your spectrum and then run a comparison of known lines with a ==Linelist==.

---+++ !HiClass export tool

---+++ Documentation
   * Updates to the HIFI Data Reduction Guide:
      * A new chapter on *The HIFI line identification tool* (_identifyLines_ task) is now available.
      * A new chapter describing tools to use the new HIFI beam model, *Understanding and using HIFI beam information in your data* is now available.
      * Updates to the chapter *Running the HIFI pipeline* for changes to the Interactive Level 2.5 pipeline, particularly the use of _mkRms_.
      * Updates to the chapter *How to add and remove flags*, section _Setting flags interactively for many spectra_ (_flagTool_ task).
      * Updates to the chapter *Sideband deconvolution*, section _Basic strategy for running the deconvolution tool_ (_doDeconvolution_ task).
      * Updates to the chapter *Standing Wave Removal* (_fitHifiFringe_ task) - note also: new syntax for startPeriod, endPeriod, typicalPeriod, and subBase.
      * Updates to the chapter *How to make a spectral cube* to reflect changes in the level 2.5.
      * Updates to the chapter *Unit conversions* for changes to _convertK2Jy_
      * Updates to the chapter *Exporting HIFI data to CLASS*.
      * Updates to all cookbooks to describe the OFF position spectra provided in the !ObservationContext Calibration tree (!ReferenceSpectra).

---++ PACS

---+++ Pipeline

---++++ Photometry

   * *Unimap*: a new Generalized Least Square (GLS) mapmaker is introduced into the Standard Product Generator. Unimap is a MATLAB algorithm that is spawned from HCSS. Level2.5 and Level3 products generated by Unimap replace MADMap maps for SPG13.
   * *Level2 products*: the quality of the High-pass Filtered Level2 products (HPPPMAP) is improved by adding the _scanamorphosMaskLongTermGlitches()_ and the _scanamorphosBaselinePreprocessing(_) tasks, which can detect jumps and correct for drifts on the detector timelines.
   * *Level2.5 products*: Unimap Level2.5 contexts contain the same types of maps generated by MADmap (naïve, GLS and Post-GLS), but they are arranged differently within the context (see Product changes).
   * *Level3 products*: High-pass Filtered maps (HPPHPFMAP) have been removed from Level3 products, leaving JScanam and Unimap maps. Level3 mosaics are created by using _scanamorphosAlignImages()_, a new task that links the flux by exploiting overlapping regions between images.
   * *Ipipe scripts*: MADMap pipeline is still available as an interactive script in the Pipelines menu. The interactive script _scanmap_Unimap_ allows users to run Unimap while tuning some parameters. It uses the new task runUnimap() that calls the Unimap release located on the user’s computer.
   * *Harmonisation of map pixel sizes*: at any Level, maps have the following pixel sizes:
      * 3.2 arcsec for the red channel, Prime and Parallel Modes;
      * 1.6 arcsec for the blue channel, Prime Mode;
      * 3.2 arcsec for the blue channel, Parallel Mode.
   * *Correction of timing shifts*: frame pixels are not read simultaneously, so in SPG12 the frames are associated with a single readout time. That results in a time shift that can correspond to up to a 2 arcsec delay for fast scanning observations. This effect is corrected by setting the parameter _correctReadoutTimes=True_ by default in every task that is affected by time offsets.

---++++ Spectroscopy

   * The Chop/nod SPG pipeline now uses the background normalisation method instead of the calibration blocks for the flux calibration.
   * At Level 2/2.5 we now also produce drizzle cubes (HPS3DDR and HPS3DDB), and "interpolated" cubes (HSP3DIR and HPS3DIB) created with the new task *specInterpolate*.
   * Hence at Level 2/2.5 we now produce three cubes which have an equidistant spatial grid and which for mapping observations are also a mosaic: drizzled, projected, and interpolated cubes. Each observation will have two of these three cubes, chosen independently for each camera. Which two cubes you get depends on the AOT and the type of mapping raster: see the PACS Products Explained document (in HIPE) to learn more.
   * One of these three cubes is also provided with an equidistant wavelength grid, created with the new task *specRegridWavelength*.
   * Tables containing the data of the rebinned cubes are created produced with the new task *pacsSpecCubeToTable* (HPSTBRR and HPSTBRB). The spectral data of each spaxel of each raster position is included in a single table, and each wavelength range and camera is a separate table.
   * Flags are now also propagated by specProject and specInterpolate.

---+++ Analysis tools

---++++ Spectroscopy

   * New long/short term Transient Correction algorithms are included in a new pipeline script for the unchopped line and range scan modes.
   * Extra options have been added to the wavelengthGrid task to create a spectral grid with bins of the same size (equidistant grids). The bin size chosen can be the mean, minumum, a fraction of the minimum, or a specified value.
   * =specSubtractOffPosition= can now combine off-positions using different algorithms, and it works with non-rebinned !PacsCube products as well. The task =specSubtractOffPositionNotRebinned= is now superfluous and has been deprecated.
   * An image can now be used as model in the extended to point correction task.
   * There are several improvements to the Pointing Offset Correction pipeline scripts (range and line scan, chopNod), including more useful plots.
   * =specDiffChop= now also works on a product containing only calibration block data.
   * =slicedSummary= works now also on all cubes.
   * The footprint viewer is now a real viewer, and can be opened with the "Open with" menu when clicking on an appropriate product.
   * General interactive pipeline script improvements:
      * Made more user-friendly, clear, and concise
      * Improvements to the diagnostic plots
      * Added a capability to overlay the IFU footprint on an image of the source in the pipeline helper task =plotCubesRaDec=
   * Fixed many problems with the mask editing functionality of the Spectrum Explorer on PACS frames and cubes.
   * New/improved scripts have been added to the PACS Useful scripts menu:
      * Fitting mapping observations (from drizzled, interpolated, and projected cubes and secondly from rebinned cubes).
      * Fitting single pointings starting from interpolated cubes
      * A new script to convolve 2 spectral images to the longer wavelength resolution
      * A new script to re-create the standalone browse products
      * A script for point sources located in the central spaxel and a script for point sources located in another spaxel
      * A script summarising the post-processing tasks for extended sources

---+++ Product changes

   * The meta keywords of all products have been improved. Most meta keywords now have an 8 character FITS equivalent, so fewer META_XX keywords will appear in the FITS files of the images and cubes.
   * The ra/dec of the Level 1/2 products now reflect the actual average centre of the particular cube/image containing these keywords. Only raNominal/decNominal contain the observer-requested pointing coordinates.
   * Simplified the structure of the Level 0--2 data in the observation context (fewer layers).
   * Moved the observation summary to the top level of the observation context and improved its contents.
   * Improved the browse images, and replaced the browse products (the images and cubes) with links to the appropriate Level 2 products. Note that for spectroscopy the images are not necessarily created from the cubes that are the "browse products".
   * Removed redundant quality flags (for all SPG 13 products).
   * Improved the quality comments (will be visible for products of SPG 13 but only towards the end of 2015)

---++++ Photometry

   * The naming convention for Unimap Level2.5/3 is HPPUNIMAP[B][R], for the blue and red channel, respectively.
   * A single context is released for Unimap Level2.5. The Unimap context contains the following products:
      * _image_ is the Weighted-GLS map. Generally, it is the most reliable product;
      * _pgls_ is the Post-GLS map;
      * _gls_ is the GLS map;
      * _naive_ is the Naïve map;
      * _error_ is that is the standard deviation map. It can be properly assumed as the error map only if associated to the Naïve map.

---++++ Spectroscopy

   * Drizzled cubes are also provided in the context under HPS3DDR and HPS3DDB
   * We provide interpolated cubes with a 4.7" spaxel size in the context under HPS3DIR and HPS3DIB.
   * New level-2 products for easier handling outside of HIPE:
      * A single table (all spaxels, all raster position, but only one wavelength range per table) with all the spectral data from the rebinned cubes (_PacsSpecTable_ product under HPSTBRR and HPSTBRB).
      * Cubes regridded on to an equidistant wavelength grid (stores in the context under HPS3DEQ*), which are easier to inspect in viewers such as DS9, since the wavelength grid is now part of the WCS.
   * We now provide Standalone Browse Products via the HSA, and these also can be found in the "browse product" part of any observation context. These products are: the new cubes we create with an equidistant spectral grid (HPS3DEQ [I|R|D][R|B], and the rebinned cubes tables (HPSTBR[R|B]).
---+++ Documentation

   * Improved the table of contents of the PACS User Reference Manual. Created a new content for the _runUnimap()_ task.
   * Detailed documentation of Unimap has been added to the PACS Data Reduction Guide for photometry.
   * Created PACS Products Explained document to describe and explain PACS products: from the observation context to the final maps and cubes
   * Improved the first two chapters of the PACS Data Reduction Guide for spectroscopy to help archive users decide what to do with their observations downloaded from the HSA: re-process or additionally process in HIPE or not? The description of the crucial pipeline tasks has also been improved.
   * Cookbooks to guide users through certain common tasks done with PACS data in HIPE will be released, beginning in the summer of 2015

---++ SPIRE

---+++ Calibration Products

Note that the size of the calibration tree has increased by 120 MB, mainly due to the addition of the finely sampled (1") photometer beam maps.

---++++ Photometer

   * The beam profile maps are improved using the shadow observations of Neptune (i.e. better characterising and removing the background).
   * The corresponding beam areas given in the metadata of the beam products are updated accordingly.
   * The radial beam calibration product has been updated to match the new beams.
   * All the colour correction products have been updated following the beam updates.
   * In order to improve the operation of the source extractors, the beam profiles are adjusted to be relevant for a 1 Jy source - previously the peak of the profile was fixed to 1 Jy, but the new profiles integrate to 1 Jy. In addition, the map consists of an odd number of pixels so that the beam centre is precisely at the centre of the central pixel.
   * A new "fine" beam map is provided for each band with 1" pixels. The "nominal" maps are also available with pixel sizes of 6", 10" and 14" for PSW, PMW and PLW.
   * The K4 parameters in the Flux Conversion product metadata are updated (very small change to K4E)
   * The Planck HFI colour correction table has been updated following the beam updates (very small change).
   * The parameters in the temperature drift calibration product that are used to perform cooler burp correction have been updated to improve the correction.

---++++ Spectrometer

   * The non-linearity product has been updated to better fit observations made with low subktemp at the beginning of each pair of FTS observing days (i.e. the observations following the cooler recycle). This results in a much improved SLWC3 spectral shape.
   * The other RSRF, telescope model and flux conversion products were updated following the non-linearity update.
   * The PhaseCorrLim product was updated to correct the metadata - no effect on the data.
   * The TempDriftCorr product has been removed as it is not used by the pipeline.

---+++ Photometer Pipeline
   * The new 2-Pass Pipeline has replaced the standard pipelines for Large, Small and Parallel modes. The 2-Pass Pipeline runs iteratively on the data to produce optimally cleaned timelines. The first pass is run with minimal tasks (e.g. no FT tasks). Glitches are detected and flagged using 2nd level deglitching outlier detection. The full pipeline is then run using all the standard tasks.
   * New useful scripts to allow users to reconstruct the beam model and all calibration parameters (Photometer Calibration scripts) for both point, extended and semi-extended sources.
   * New SPG map merging script utilising the 2-pass pipeline to produce Level 2.5 products. All parallel mode observations are reprocessed together as pairs at the Level 2.5 stage. After the level 2.5 stage all the Level 0.5, Level 1,Level 2, Level 2.5 data are updated with the improved quality data.
   * Optimized parameters for Timeline Fitter in photometry script. Background annulus changed for faster run time.

---+++ Spectrometer Pipeline

   * Update to the naming within the Observation Context to make it clearer which products are spectra/cubes and pointsource/extended calibrated.
   * Moved all of the final spectral products into the Level-2 context. Previously the extended calibrated spectra were in the Level-1 context, but now both point source and extended calibrated data are in the Level-2 context.
   * Introduced a new spectrum2d product for mapping observations to contain all of the spectra before they are gridded into a regular spectral cube. Previously these spectra were saved in individual products for each jiggle/raster position. The new product is added to the Level-2 context.
   * A new task is added to perform spectral apodization. This step was previously carried out with several tasks to take the inverse Fourier transform and apply the apodization on the interferogram. The new task works directly on the spectra.
   * Quality control parameters related to the unaveraged FTS scans are now written back into the interferogram products in the Level-1 context. Previously these metadata items were lost after averaging the spectra over all of the scans.
   * New metadata parameters to contain known pointing offsets from the commanded position have been added to the Level-2 spectra.
   * A limit has been placed on the spectral resolution allowed as input when a spectral mapping observation is gridded into a cube. This eliminates spectra that were clipped, and improves the quality of the maps.

---+++ Interactive Analysis and Tools

---++++ Photometer
   * Significant improvements to the bolometer finder tool have been made. 

---++++ Spectrometer

   * Minor updates, bug fixes, and improved documentation for the Semi-extended correction task.
   * Updates to the task to take synthetic photometry of FTS spectra for comparison with the SPIRE photometer.
   * New GUI interface for the script that determines the noise level in a spectrum.
   * Small corrections to the useful scripts.

---++ Release Notes

  * Full list of tickets implemented: [[HcssReleaseNotes13][here]].

---

<!-- COMMENT BOX CODE - DO NOT EDIT -->
<div id="disqus_thread"> </div>

<script type="text/javascript">
    var disqus_shortname = 'herscheltwiki'; // required: replace example with your forum shortname




    (function() {
        var dsq = document.createElement('script'); dsq.type = 'text/javascript'; dsq.async = true;
        dsq.src = 'http://' + disqus_shortname + '.disqus.com/embed.js';
        (document.getElementsByTagName('head')[0] || document.getElementsByTagName('body')[0]).appendChild(dsq);
    })();
</script>
<noscript>Please enable JavaScript to view the [[http://disqus.com/?ref_noscript][comments powered by Disqus.]]</noscript>
[[http://disqus.com][blog comments powered by <span class="logo-disqus">Disqus</span>]]
<!-- END OF COMMENT BOX CODE -->

---

<!--
   * Set ALLOWTOPICCHANGE = Main.RegisteredUsersGroup, Main.TWikiAdminGroup, Main.MarkRKidger
-->

-- Main.AlvarGarcia - 15 Jan 2014
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
pdf	Guide_gyro_recon_issue1.0.pdf	r1	manage	837.1 K	2014-12-31 - 10:09	JavierDiaz	pointing refinements
pdf	HIFI_PointingAccuracyNote.pdf	r1	manage	372.2 K	2015-04-08 - 10:27	DavidTeyssier
Topic revision: r68 - 2015-09-29 - LucaCalzoletti
Public
Webs
Public
TWiki