(r62) HipeWhatsNew5x < Public

Public Web>HipeWhatsNew>HipeWhatsNew5x (revision 62) (raw view)~~EditAttach~~
---+ What's New in HIPE 5

| *PDF Version* | [[%SCRIPTURLPATH%/genpdf%SCRIPTSUFFIX%/%WEB%/%TOPIC%?pdforientation=portrait&pdftoclevels=3][Click here]] |

This document lists the changes in HIPE version 5.0 with respect to the 4.x series. Additional pages list changes between minor versions of the 5.x series.

   * [[HipeWhatsNew51][Changes between HIPE 5.0 and 5.1]]
   * [[HipeWhatsNew52][Changes between HIPE 5.1 and 5.2]]

*See also the [[HipeKnownIssues][HIPE known issues]] and the [[DpKnownIssues][Data products known issues]] pages for last-minute information on known problems.*

%TOC%

---++ Core system

---+++ HIPE

---++++ Editors & Views

   * Files ending with extension ".csv" are recognized as tables, and opened as such with double click in the _Navigator view_ or with _File > Open_.
   * The tooltip shown when the mouse cursor is over a task in the _Task view_ is now more informative.
   * Autocompletion in the _Console_ has been improved.
   * A variable value can be copied by right clicking on the variable in the _Variables view_ and selecting _Copy value_; it can be pasted in the _Console_ or _Jython editor_ afterwards. This works only for strings and numbers.
   * Drag & drop a variable from the _Variables view_ to the _Console_ to write the variable name. The same behaviour can be achieved when dropping it to a _Jython editor_.
   * Drag & drop selected text from a _Jython editor_ to the _Console_ or vice-versa to copy & paste that text.
   * The small level-2 preview shown in the top-right corner of the _Observation Context viewer_ can be enlarged by clicking on it.

---++++ Startup & Shutdown

If the running Java platform is not supported, a warning is displayed.

---++++ Feedback

A feedback button has been added for documentation and HIPE functionality.

---++++ Look & Feel

Aqua look & feel has been replaced in Mac OS X by the HIPE look & feel used on the other platforms.

---++++ Plug-ins

HIPE now supports [[DpHipePlugins][plug-ins]], small packages that anyone can easily create and distribute to share code (Java or Jython) and data (!LocalStores). Plug-ins can be downloaded _and_ installed with a single-click.

---+++ Plotting

   * Title and label can now accept the !LaTeX commands ==\textrm \textit \textbf \mathrm \mathit \mathbf==. For example: ==V$_{\textrm{LSR}}$==

---+++ Astronomical utilities
   * !RadialVelocity now uses !TimeCorrProduct to select pointing data.
   * !SSO now uses !TimeCorrProduct to select pointing data.
   * Improved documentation.

---+++ Numeric routines
---++++  Data fitting
   * A complete new GUI for the !SpectrumFitter was added. Start it from the spectrum toolbox menu, select !SpectrumFitterGUI. Besides the functionality that was available in the !SpectrumFitterTool, the GUI has an export tab that enables you to export the residual, total model, and individual models as an object with the type of the input object and as text.
   * The !SpectrumFitter (commandline version) accepts Container with !PointSpectrum and !SpectralSegment index.
   * The !SpectrumFitter (commandline version) has methods to export the data as the input type and as text.
   * The !SpectrumFitter (commandline version) has method that returns a Jython script.
---++++  Data interpolation
---++++  Random Numbers
   * New random numbers generators from different distribution: Uniform, Gaussian, Poisson, Gamma, Cauchy (Lorentz), Exponential and Jeffreys.
---++++  Statistics functions
   * RMS has been renamed to QRMS.
   * RMS implements a new functionality (see the _User Reference Manual_).
   * Improved !StdDev and !Variance algorithms.
   * Improved !CorrelateMatrix algorithm.

---+++++ New functions
   * *GeoMean*, which calculates the geometric mean of a array of numeric data types with 1 to 5 dimensions. 
   * *Mode*, which returns the mode(s), or the most common element(s), of an array of numeric data with 1 to 5 dimensions.
   * *Covariance*, which returns the covariance between two random variables/vectors x and y with finite second moments.  If one vector is longer than the other, only the values up to the length of the shorter vector will be taken into account.
   * *CovarianceMatrix*, which returns the covariance matrix of the input M x N matrix. The result is a N x N matrix with each i, j value equal to the covariance of the ith and jth columns of the original matrix.

---++++  Other functions
   * !SigClip
      * Median mode uses median absolute deviation instead of standard deviation.
      * Iterative use is allowed.

---+++ Images
---++++  Display
   * Images can be opened as ==RgbImage== directly from the Navigator in HIPE.
   * Regular image files (jpg, gif, png, ...) are shown as preview in the outline when being clicked on in the navigator view of HIPE.
   * Two images can be compared by setting the opacity of the image. This can be done using the setOpacity(float) method or using a slider in the Image Display.
---++++  Analysis
   * Added the methods ==containsNorthCelestialPole== and ==containsSouthCelestialPole== to the Wcs.
   * Regridding an image on the grid of another image using ==RegridTask==.
   * Speed improvement of ==createRgbImageTask==.
   * Update of documentation in the _User Reference Manual_.
   * You can now crop an image by drawing a rectangle on it
   * Correction of the calculation of the dimensions of a mosaic.
---+++ Spectra
---++++  Display
   * Task & Tools Toolbox: All task and tool panels are displayed in the same toolbox panel and selectable through a drop-down list. The task GUI components have become clearer as well.   
   * The new spectrum toolbox panel is opened by clicking on <img src="%ATTACHURLPATH%/toolbox.gif" alt="toolbox.gif" width='16' height='16' /> in the toolbar
   * Instead of dragging a selection of spectra to the dataset parameter, the task is applied on either the _displayed_ or the _selected_ spectra within the plot.
   * Load selectable datasets from a product: When opening the spectrum explorer on a product, a selection table is displayed showing all datasets that are stored in this product along with certain characteristics. 
      Just as within the original spectrum selection panel, these datasets can be sorted by clicking on the header columns. To load a set of datasets instead of just one or all, select each row and right click to load the _selected datasets_.
   * Realtime statistics information: Selecting the Statistics task within the Toolbox displays realtime statistics while dragging ranges in the plot.
   * Added relativistic frequency to velocity transform: When changing the unit of the wave axis to velocity the transform is now relativistic.
---++++  Analysis
   * Gaussian resampling introduced.
   * 'Varient' and 'overwrite' options removed from ExtractSpectrum task

---+++ Herschel Science Archive
   * Support for exporting data in packed HSA format.
   * Import and Export Views improvements: 
      * Better error reporting
      * File dialogue windows can show hidden files
      * Versions of observations inside pools are now shown

---++++  Pipeline processing
   * Crated a daemon dispatcher mechanism.
   * Aliases set tool has been modified for performing the required verifications before a tag is ingested.
   * !HsaPoolDaemon
      * Releases memory calling Java garbage collector if it is needed.
      * Can write compressed files.
      * Can save a binary representation of a product if the FITS save operation fails.
   * !HsaStagePool can export products into a !LocalStore.
   * HSA ingestion procedure can generate an XML ingestion request file even if a previous version does not contain its own XML.

---+++ Calibration sources

A new view has been added to HIPE for easy access to the Calibration Sources Database (Calsdb).

---+++ Products and datasets

   * Product comparison API is available in the ==ia_toolbox_util== module.

---++++  Observation context

   * Included new metadata ==slewTime== returning the scheduled start time of the slew as a !FineTime.
      * Specific accessors, ==getSlewStartTime()== and ==setSlewStartTime()==, where added to the !ObservationContext class.
      * For old observations in the HSA without this value, the method will return a null value indicating that this field was never set before.

---++++ History

A new history dataset viewer is available: When you click on the History dataset in your product two tables appear:
   * An overview table which shows all the tasks used to generate the product. For each task, the name, execution time, build number and used calibration files are shown.
   * When you click on a row in this overview table then a second table is shown with details for the selected task. This table contains the details of this task: For each parameter its simple value and some product information is shown (when available).

---+++ Product Access Layer 

---++++ Cache

Several bugfixes to improve the effectiveness and robustness of caching have been introduced. One of these is a check to verify that the default !LocalStore directory is not modified at any time during the cache's existence, which is a common source of corruption of the cache.

---++++ !HsaPool
   * !HsaReadPool/HsaXmlPool can work with compressed data.
   * !HsaReadPool can retrieve product metadata directly.

---++++ Preferences

Product storages and pools can be managed in the new _Data Access > Storages & Pools_ preferences panel.

---+++ Tasks
   * Updated task sample code.
   * Updated documentation: 
      * Wiki chapters
      * Javadoc of general tasks
      * Standardised ==__ doc __== of general tasks (signature, description, parameter descriptions)
   * Tasks View provides tooltips with the signature of the tasks
   * Examples of general tasks in the URM are now tested automatically
   * Compress and Decompress tasks: they support zip, tar, gzip, tgz in a portable way. Decompress can guess the compression of a file.
   * !FitsReader supports compressed FITS files.

---+++ Quality control

   * New metadata into the !QualityContext product. The field is fixed and the main interest is to include this information into the FITS file associated to the product.

   | *FITS keyword* | *Value* | 
   | PCAVEATS | Please refer to http://herschel.esac.esa.int/DpKnownIssues.shtml for known problems in products |

   * Perform bulk actions on several quality control reports simultaneously.
   * Flags colour codes depending on their importance.

---+++ Systematic product generation

   * New framework to process/combine data from several observations in the same pipeline.
      * This will allow, for instances, better results using MADmap algorithm with cross-scan data from two or more observations.
   * Automatic processing:
      * New start-time algorithm based on the DTCP time of the OD being processed instead of fixed timestamps.
      * New check on TM data based on the gap checker reports. Only those observations with complete tm data will be processed at this point.
   * Uplink plugin is executed without problems in different conditions, even for manual commanding and no uplink data at all.
   * Obsids shown also in decimal format in all the application tables.
   * Include proper motion data into the !ObservationContext.
   * SPG stand-alone application can now run on Mac OS with no Versant libraries available.
   * On-demand request are now recorded into a report file.

---+++ Data input-output
   * Improved error reporting in !AsciiTableTool.
   * !FitsArchive:
      * FITS Metadata direct access.
      * ESO Hierarch keyword implemented when reading FITS files.
      * FITS translation dictionary updated:
           | Herschel keyword | FITS keyword |
           | pmRA | PMRA |
           | pmDEC | PMDEC |
           | state | STATUS |
           | action | ACTION |
           | slewTime | SLEWTIME |
           | proCaveats | PCAVEATS |
      * !FitsArchive allow flag for compression when saving FITS files.
      * !FitsArchive support reading gzipped FITS files and compressed stream headers.
      * !HeaderWriter exceptions have been modified to include more information.
      * Documentation updates.

---+++ Installer
There was a major restructuring of the pre-installation steps:
   * The Basic/Advanced selection page has been removed.
   * The memory, proxy and Versant settings were restructured. There are no separate panels asking if you want to set a proxy or a Versant database. There is a dedicated page for the proxy setting and one for the Versant settings.
   * Instead of the old instrument selection page, you are now offered the following list with check boxes for selection:
         | Option | Description |
         | PACS (preselected by default) | PACS pipelines and analysis tools for astronomers. |
         | PACS expert | Extra instrument-related applications for Calibration Scientists. |
         | SPIRE (preselected by default) | SPIRE pipelines and analysis tools for astronomers. |
         | SPIRE expert | Extra instrument-related applications for Calibration Scientists. |
         | HIFI (preselected by default) | HIFI pipelines and analysis tools for astronomers. |
         | HIFI expert | Extra instrument-related applications for Calibration Scientists. |

   * The instrument selection window allows you to click in any combination of items to install those components.
   * You can also deselect all the instruments to get a barebone HIPE framework only for FITS I/O and numeric analysis.
   * If possible, the installer informs you of the amount of data to download for the selected combination of modules.

---+++ Documentation

To improve the quality of the examples, we have introduced automatic testing of (many of) the source code examples occurring in manuals such as the _Data Analysis Guide_. We have also introduced automatic testing of the examples occurring in the _User's Reference Manual_.

---++ HIFI

---+++ HIFI Software Configuration and Properties
   * Generally most HIFI properties have moved from the hcss.dp.hifi build to the hcss.icc.hifi (expert) area and it is recommended that all internal ICC users always install the expert build in order to be able to run the pipeline from level -1.0. 

   * Those using the expert build and wishing to run the pipeline from level zero will need to set the property  var.hcss.instrument = hifi ' this so the system knows to use the HIFI binstruct settings for reading the packets. Those using the legacy hifi-new.props file will already have this property set.

   * The hcss.dp.hifi (non-expert) build has pre-configured pools to access the HSA and the local pool via the Product Browser and PAL Storage Manager

   * Unnecessary and deprecated properties files have been removed from the system 

---+++ HIFI Calibration
   * The Spur table has been updated to reflect the improvements in band 1a.  For data taken after OD 516 the spur table now only shows three entries:
      * 1a 537500.00 3000.0 LO_settling 1,2,3,4
      * 1a 540929.25 1000.0 Strong_spur 1
      * 1a 542306.25 1000.0 Weak_spur 1

   * The minimum and maximum chopper threshold values have been changed.
      * !HkThresholdsContext_H !fpuChopperMin -> -0.04
      * !HkThresholdsContext_V !fpuChopperMin -> -0.04
      * !HkThresholdsContext_H !fpuChopperMax -> 0.06
      * !HkThresholdsContext_H !fpuChopperMax -> 0.06
Previous values used were those appropriate for the Hot and Cold black bodies, which are not representative of the full, planned range of the chopper.   As a result, the quality control flag  "FPU Check: Chopper measured values differ from the commanded" had been frequently raised more than necessary. 

   * The upper limit of the mixer current quality control check for HEBs (bands 6 and 7) have been decreased from 0.08 mA to 0.055 mA, where tests have shown the mixers start to be under-pumped.

   * Update to calibration tree for results of Mars beam measurements
The following items have been added to the HIFI Calibration tree for the aperture efficiencies (H and V), main beam efficiencies (H and V), and the half power beam widths (H and V) :
      * !apertureEfficiency-H
      * !apertureEfficiency-V
      * !beamEfficiency-H
      * !beamEfficiency-V
      * !beamWidths-H
      * !beamWidths-V
   
The approach so far is to strictly use *observed numbers* (see tables below *with initial results*), without involving fitting results or empirical formulas. In particular this means:
      * For beam widths the fitted widths of circular Gaussians, corrected for the convolution with the Mars disk. Open question: is the fit of a circular Gaussian correct? Need to check by how much Mars moved during an integration.
      * For main beam efficiencies the values derived using the observed beam widths for the beam coupling. It is not yet decided if an average value or a linear interpolation between (and beyond) frequencies produces the best results
      * For aperture efficiencies the values are derived, again, using observed beam widths for the coupling of the beam to the source.

In addition, the already available entries for the forward efficiencies (H and V) are updated from 1.0 to 0.96 (in all bands, all frequencies) to be compatible with above results. This requires the change of constant 1.0 to 0.96 .

|Aperture Efficiency (H) || |
|Band | Frequency (GHz) | Eta_A |
| 1a | 489.0 | 0.676 |
| 1a | 512.0 | 0.675 |
| 1a | 548.0 | 0.675 |
| 1b | 564.0 | 0.675 |
| 1b | 595.0 | 0.674 |
| 1b | 627.0 | 0.673 |
| 2a | 641.0 | 0.673 |
| 2a | 677.0 | 0.672 |
| 2a | 710.0 | 0.671 |
| 2b | 724.0 | 0.671 |
| 2b | 756.0 | 0.670 |
| 2b | 792.0 | 0.669 |
| 3a | 809.0 | 0.669 |
| 3a | 830.0 | 0.668 |
| 3a | 850.0 | 0.668 |
| 3b | 868.0 | 0.667 |
| 3b | 910.0 | 0.666 |
| 3b | 945.0 | 0.665 |
| 4a | 959.0 | 0.664 |
| 4a | 1006.0 | 0.663 |
| 4a | 1052.0 | 0.661 |
| 4b | 1064.0 | 0.661 |
| 4b | 1086.0 | 0.660 |
| 4b | 1112.0 | 0.659 |
| 5a | 1118.0 | 0.562 |
| 5a | 1179.0 | 0.560 |
| 5a | 1240.0 | 0.558 |
| 5b | 1148.0 | 0.561 |
| 5b | 1207.0 | 0.559 |
| 5b | 1268.0 | 0.557 |
| 6a | 1431.0 | 0.646 |
| 6a | 1503.0 | 0.642 |
| 6a | 1574.0 | 0.639 |
| 6b | 1579.0 | 0.638 |
| 6b | 1637.0 | 0.635 |
| 6b | 1694.0 | 0.632 |
| 7a | 1708.0 | 0.631 |
| 7a | 1768.0 | 0.628 |
| 7a | 1813.0 | 0.626 |
| 7b | 1720.0 | 0.631 |
| 7b | 1815.0 | 0.625 |
| 7b | 1897.0 | 0.621 |



|Aperture Efficiency (V) |||
|Band | Frequency (GHz) | Eta_A |
| 1a | 489.0 | 0.676 |
| 1a | 512.0 | 0.675 |
| 1a | 548.0 | 0.675 |
| 1b | 564.0 | 0.675 |
| 1b | 595.0 | 0.674 |
| 1b | 627.0 | 0.673 |
| 2a | 641.0 | 0.673 |
| 2a | 677.0 | 0.672 |
| 2a | 710.0 | 0.671 |
| 2b | 724.0 | 0.671 |
| 2b | 756.0 | 0.670 |
| 2b | 792.0 | 0.669 |
| 3a | 809.0 | 0.669 |
| 3a | 830.0 | 0.668 |
| 3a | 850.0 | 0.668 |
| 3b | 868.0 | 0.667 |
| 3b | 910.0 | 0.666 |
| 3b | 945.0 | 0.665 |
| 4a | 959.0 | 0.664 |
| 4a | 1006.0 | 0.663 |
| 4a | 1052.0 | 0.661 |
| 4b | 1064.0 | 0.661 |
| 4b | 1086.0 | 0.660 |
| 4b | 1112.0 | 0.659 |
| 5a | 1118.0 | 0.562 |
| 5a | 1179.0 | 0.560 |
| 5a | 1240.0 | 0.558 |
| 5b | 1148.0 | 0.561 |
| 5b | 1207.0 | 0.559 |
| 5b | 1268.0 | 0.557 |
| 6a | 1431.0 | 0.646 |
| 6a | 1503.0 | 0.642 |
| 6a | 1574.0 | 0.639 |
| 6b | 1579.0 | 0.638 |
| 6b | 1637.0 | 0.635 |
| 6b | 1694.0 | 0.632 |
| 7a | 1708.0 | 0.631 |
| 7a | 1768.0 | 0.628 |
| 7a | 1813.0 | 0.626 |
| 7b | 1720.0 | 0.631 |
| 7b | 1815.0 | 0.625 |
| 7b | 1897.0 | 0.621 |


| Beam Efficiency (H) |||
| Band | frequency (GHz) | eta_mb |
| 1a | 489.0 | 0.755 |
| 1a | 512.0 | 0.755 |
| 1a | 548.0 | 0.754 |
| 1b | 564.0 | 0.754 |
| 1b | 595.0 | 0.753 |
| 1b | 627.0 | 0.752 |
| 2a | 641.0 | 0.752 |
| 2a | 677.0 | 0.751 |
| 2a | 710.0 | 0.750 |
| 2b | 724.0 | 0.750 |
| 2b | 756.0 | 0.749 |
| 2b | 792.0 | 0.748 |
| 3a | 809.0 | 0.747 |
| 3a | 830.0 | 0.747 |
| 3a | 850.0 | 0.746 |
| 3b | 868.0 | 0.746 |
| 3b | 910.0 | 0.744 |
| 3b | 945.0 | 0.743 |
| 4a | 959.0 | 0.742 |
| 4a | 1006.0 | 0.741 |
| 4a | 1052.0 | 0.739 |
| 4b | 1064.0 | 0.738 |
| 4b | 1086.0 | 0.738 |
| 4b | 1112.0 | 0.737 |
| 5a | 1118.0 | 0.639 |
| 5a | 1179.0 | 0.637 |
| 5a | 1240.0 | 0.635 |
| 5b | 1148.0 | 0.638 |
| 5b | 1207.0 | 0.636 |
| 5b | 1268.0 | 0.634 |
| 6a | 1431.0 | 0.722 |
| 6a | 1503.0 | 0.718 |
| 6a | 1574.0 | 0.714 |
| 6b | 1579.0 | 0.713 |
| 6b | 1637.0 | 0.710 |
| 6b | 1694.0 | 0.707 |
| 7a | 1708.0 | 0.706 |
| 7a | 1768.0 | 0.702 |
| 7a | 1813.0 | 0.699 |
| 7b | 1720.0 | 0.705 |
| 7b | 1815.0 | 0.699 |
| 7b | 1897.0 | 0.694 |


| Beam Efficiency (V) |||
| Band | Frequency (GHz) | eta_mb |
| 1a | 489.0 | 0.755 |
| 1a | 512.0 | 0.755 |
| 1a | 548.0 | 0.754 |
| 1b | 564.0 | 0.754 |
| 1b | 595.0 | 0.753 |
| 1b | 627.0 | 0.752 |
| 2a | 641.0 | 0.752 |
| 2a | 677.0 | 0.751 |
| 2a | 710.0 | 0.750 |
| 2b | 724.0 | 0.750 |
| 2b | 756.0 | 0.749 |
| 2b | 792.0 | 0.748 |
| 3a | 809.0 | 0.747 |
| 3a | 830.0 | 0.747 |
| 3a | 850.0 | 0.746 |
| 3b | 868.0 | 0.746 |
| 3b | 910.0 | 0.744 |
| 3b | 945.0 | 0.743 |
| 4a | 959.0 | 0.742 |
| 4a | 1006.0 | 0.741 |
| 4a | 1052.0 | 0.739 |
| 4b | 1064.0 | 0.738 |
| 4b | 1086.0 | 0.738 |
| 4b | 1112.0 | 0.737 |
| 5a | 1118.0 | 0.639 |
| 5a | 1179.0 | 0.637 |
| 5a | 1240.0 | 0.635 |
| 5b | 1148.0 | 0.638 |
| 5b | 1207.0 | 0.636 |
| 5b | 1268.0 | 0.634 |
| 6a | 1431.0 | 0.722 |
| 6a | 1503.0 | 0.718 |
| 6a | 1574.0 | 0.714 |
| 6b | 1579.0 | 0.713 |
| 6b | 1637.0 | 0.710 |
| 6b | 1694.0 | 0.707 |
| 7a | 1708.0 | 0.706 |
| 7a | 1768.0 | 0.702 |
| 7a | 1813.0 | 0.699 |
| 7b | 1720.0 | 0.705 |
| 7b | 1815.0 | 0.699 |
| 7b | 1897.0 | 0.694 |

| Beam Widths (H) |||
| Band | Frequency (GHz) | HPBW |
| 1a | 489.0 | 43.362 |
| 1a | 512.0 | 41.414 |
| 1a | 548.0 | 38.694 |
| 1b | 564.0 | 37.596 |
| 1b | 595.0 | 35.637 |
| 1b | 627.0 | 33.818 |
| 2a | 641.0 | 33.080 |
| 2a | 677.0 | 31.321 |
| 2a | 710.0 | 29.865 |
| 2b | 724.0 | 29.287 |
| 2b | 756.0 | 28.048 |
| 2b | 792.0 | 26.773 |
| 3a | 809.0 | 26.210 |
| 3a | 830.0 | 25.547 |
| 3a | 850.0 | 24.946 |
| 3b | 868.0 | 24.429 |
| 3b | 910.0 | 23.301 |
| 3b | 945.0 | 22.438 |
| 4a | 959.0 | 22.111 |
| 4a | 1006.0 | 21.078 |
| 4a | 1052.0 | 20.156 |
| 4b | 1064.0 | 19.929 |
| 4b | 1086.0 | 19.525 |
| 4b | 1112.0 | 19.068 |
| 5a | 1118.0 | 18.966 |
| 5a | 1179.0 | 17.985 |
| 5a | 1240.0 | 17.100 |
| 5b | 1148.0 | 18.470 |
| 5b | 1207.0 | 17.568 |
| 5b | 1268.0 | 16.722 |
| 6a | 1431.0 | 14.818 |
| 6a | 1503.0 | 14.108 |
| 6a | 1574.0 | 13.471 |
| 6b | 1579.0 | 13.429 |
| 6b | 1637.0 | 12.953 |
| 6b | 1694.0 | 12.517 |
| 7a | 1708.0 | 12.415 |
| 7a | 1768.0 | 11.993 |
| 7a | 1813.0 | 11.696 |
| 7b | 1720.0 | 12.328 |
| 7b | 1815.0 | 11.683 |
| 7b | 1897.0 | 11.178 |

| Beam Widths (V) |||
| Band | Frequency (GHz) | HPBW |
| 1a | 489.0 | 43.362 |
| 1a | 512.0 | 41.414 |
| 1a | 548.0 | 38.694 |
| 1b | 564.0 | 37.596 |
| 1b | 595.0 | 35.637 |
| 1b | 627.0 | 33.818 |
| 2a | 641.0 | 33.080 |
| 2a | 677.0 | 31.321 |
| 2a | 710.0 | 29.865 |
| 2b | 724.0 | 29.287 |
| 2b | 756.0 | 28.048 |
| 2b | 792.0 | 26.773 |
| 3a | 809.0 | 26.210 |
| 3a | 830.0 | 25.547 |
| 3a | 850.0 | 24.946 |
| 3b | 868.0 | 24.429 |
| 3b | 910.0 | 23.301 |
| 3b | 945.0 | 22.438 |
| 4a | 959.0 | 22.111 |
| 4a | 1006.0 | 21.078 |
| 4a | 1052.0 | 20.156 |
| 4b | 1064.0 | 19.929 |
| 4b | 1086.0 | 19.525 |
| 4b | 1112.0 | 19.068 |
| 5a | 1118.0 | 18.966 |
| 5a | 1179.0 | 17.985 |
| 5a | 1240.0 | 17.100 |
| 5b | 1148.0 | 18.470 |
| 5b | 1207.0 | 17.568 |
| 5b | 1268.0 | 16.722 |
| 6a | 1431.0 | 14.818 |
| 6a | 1503.0 | 14.108 |
| 6a | 1574.0 | 13.471 |
| 6b | 1579.0 | 13.429 |
| 6b | 1637.0 | 12.953 |
| 6b | 1694.0 | 12.517 |
| 7a | 1708.0 | 12.415 |
| 7a | 1768.0 | 11.993 |
| 7a | 1813.0 | 11.696 |
| 7b | 1720.0 | 12.328 |
| 7b | 1815.0 | 11.683 |
| 7b | 1897.0 | 11.178 |


---+++ HIFI Pipeline
---++++ hifiPipelineTask
   
   *  The path/filename for a user defined/edited pipeline algorithm can now be passed to the !hifiPipeline task via the command line, previously this was only possible in the GUI.


*obs =hifiPipeline (obs=obs,  level2AlgoPath="/home/Me/myLevel2Algo.py")*


The older method of defining a function with the *def* keyword in Jython and passing that function to the !hifiPipeline task still exists

*obs =hifiPipeline (obs=obs,  level2Algo=level2PipelineAlgo)*


   * !hifiPipeline will make an !ObservationContext even if creation of trend products fails
   * !hifiPipeline GUI layout improved to make more efficient use of space
   * When reprocessing with the !hifiPipeline task it is not recommend to set aux=True when using the HSA to retrieve your data as not all Auxiliary data products will be updated correctly. The default option of aux=False is valid for all observations created with SPG v4.0+ (which should be all HIFI observations in the HSA).

---++++ Level 0 Pipeline

   * H and V spectrometer astrometry
!SpectrumDatasets contain columns, in decimal degrees, for the commanded pointing (a position between the H and V beams known as the synthesized aperture), labelled *longitude_cmd* and *latitude_cmd*, while the actual coordinates for the spectrometer beam are found in the *longitude* and *latitude* columns.   Note that HIPE 5 RC 9 and later versions only should be used for this.
 

---++++ WBS  Pipeline

   * Fixes for frequency calibration in the case that line centres fall precisely in the middle of two pixels

---++++ Level 2 Pipeline
   * DoMainBeamTemp now multiplies by the forward efficiency.  In earlier versions of HIPE this was not an issue because a value of Feff=1 was assumed.  This update was required as a consequence of the adoption of Feff=0.96 in the calibration tree.
   * The ra and dec in cube meta data is now set to be the centre of the cube.   Previously the ra and dec in the first dataset of the HTP had been used.

---+++HIFI Products

   * Modifications to !HifiSpectrumDatasets are now reflected in !SpectrumExplorer when the HTP is viewed there.
   * It is now possible to create a !SimpleSpectrum from a HIFI product or dataset containing only one spectrum.   A !SimpleSpectrum is easily written to FITS and maintains metadata.   The conversion is done as:
simpleSpectrum = convertSingleHifiSpectrum(spectra=data)




---+++ HIFI Data Processing Tools
   * PolarPairs now has a tolerance of 100.  Previously it as 5 and this sometimes caused the task to fail resulting in the output being identical to the first input product. 
   * A task to allow conversion of Kelvin to Jansky, ConverK2Jy, is introduced

---++++ Standing Wave Removal
 
   * It is now possible to remove the smoothed baseline calculated by !FitHifiFringe from the spectrum, in addition to the fitted sine waves. This option (sub_base) should be used with care as real spectral features can be removed this way. For interactive fitting of polynomial baselines, the new task !FitBaseline is recommended.
   * Tooltips implemented in the !FitHifiFringe GUI
   * Improved error messaging from !FitHifiFringe
   * Bug that appeared in early versions of HIPE 5 where usermask max/min boxes in !FitHifiFringe GUI disappeared fixed.
  
   * A Deep Fourier Transform (DFT) module is available (not a part of !FitHifiFringe)

---++++ Baseline removal
   * Introduction of a new task, !FitBaseline, to interactively fit and subtract or divide out baselines in HIFI spectra.
   * A bug that appeared in early versions of HIPE 5 where domask=2 in did not work in !FitBaseline (preventing automatic line masking) now fixed.
   * A bug that appeared in early versions of HIPE 5 in which fitBaseline would not save on an intermediate quit is resolved.
   * The fitting routine in !fitBaseline  has a condition set such that no polynomials of order n are fitted if less than n+1 channels are available.   This prevents crashes of the task in the case of heavily flagged data.

---++++ Deconvolution

   * More understandable error message when deconvolution fails (usual due to poor data input).
   * Plotting of DSB solution as deconvolutions runs now possible when diag_mode_on selected.


---+++ !HiClass export tool

   * !HiClass will now appear an an applicable task for HTP as well as !ObservationContexts
   * !HiClass now exports the correct frequencies for bands 6 and 7
   *  A bug that appeared in early versions of HIPE 5 preventing export of Spectral Scans now fixed
   *  !HiClass now exports the source velocity as provided by the astronomer in the AOR into a new FITS header keyword. At this time, the CLASS software will not do anything with this information, but the requirement to provide ALL the information is satisfied by having the velocity information present in the FITS header somewhere.  The keywords used, which are slightly different than the usual ones used to reflect the fact that only the velocity in the _proposal_ is exported, are tabulated below.

| reference frame | usual keyword | proposed keyword |
| topocentric | VELO-OBS 	| PVEL-OBS |
| geocentric | VELO-GEO | PVEL-GEO |
| heliocentric | VELO-HEL | PVEL-HEL |
| LSRk | VELO-LSR | PVEL-LSR |
| LSRd | VELO-LSR | PVEL-LSR |
| LSR | VELO-LSR | PVEL-LSR |
| galactocentric | VELO-GAL | PVEL-GAL |
| source | VELOREST 	| PVELREST |
|unknown | VELOCITY | PVELCITY |

These keyword takes care of the value, the unit and the reference frame.  The PVELTYPE keyword will identify the type of 'velocity' (velocity, redshift, radio or optical convention).


---+++ HIFI Documentation

   * New chapters in the HIFI User's Manual:  _What was done to my data?_ and _HIFI Baseline Removal_.
   * Updates to the HIFI Users Manual: _HIFI cookbook_, _Data Primer_, _Running the HIFI Pipeline_, _HIFI Standing Wave Removal Tool_ , _How to make Spectral Cubes_ and _Exporting HIFI data to CLASS_.
   * Updates to HIFI Pipeline Specification document: improved pipeline flow diagrams,  improved information about Euler resampling used in !DoFreq grid task of pipeline.
   * Creation of !FitHifiFringe URM entry.
   * The HIFI cookbook can now be selected from the Help menu in the HIPE toolbar under Help -> Data Reduction -> HIFI 

---++ PACS

---+++ Pipeline
   * PACS "New Style Sliced" pipeline implementation is now activated.
      * Phot and spec ipipe script names were changed to be more logic and intuitive.
      * Pipeline Script access:
         * HIPE -> Menu -> pipeline -> PACS 
   * Calibration framework:<br/>
   Calibration products are no longer delivered with the build in a local store. <br>
   Calibration products are now delivered as a directory of FITS files. <br>
   This directory with calibration products can be found in the data/pcal sub-directory of your installation directory.
   <blockquote>
   HIPE> fm = getCalTree(verbose=1)<br>
   !CalTree read from: /Software/Herschel/hcss.dp.pacs-5.0.500/data/pcal
   </blockquote>
   The calibration framework is adapted and will read these FITS files by default. <br>
   The getCalTree() command and all access to calibration data has not changed, under the hood, the local store has been replaced by the directory of FITS files.<br/>
   Other small changes:
      * getCalTree(), getCalProduct() now understand the 'verbose=True' keyword and will print information on the location of the calibration products.
      * getCalProduct() the third argument, 'version',  is no longer mandatory, if not given, the last version of the calibration product will be returned.
   * A new Pacs Documentation page has been opened on the [[http://www.herschel.be/twiki/bin/view/Public/PacsDocumentation public wiki.]] 

---++++ Photometry
   * !MadMap 
      * Extended Madmap (using scan and cross scan) for interactive usage 
         * HIPE -> Menu -> pipeline -> PACS  -> L2_scanMap.py
         <br>
         A prototype for the extended Madmap pipeline is now available in HIPE for interactive usage. The extended Madmap pipeline is designed to combine scans and cross-scans in order to produce optional map.  Unlike DP pipelines (which normally starts with one !ObservationContext), this pipeline takes an obsList (an !ArrayList) which is a list of !ObservationContext.  The pipeline will run even there is only one obs in the obsList.  Below is an example creating an obsList:
         <blockquote>
         HIPE> obsList=ArrayList() <br>
         HIPE> obsList.add(obs1) <br>
         HIPE> obsList.add(obs2) <br>
        </blockquote>

   * New script for scan Map pipeline
      * Two-stage photometer pipeline with masked highpass filtering
         * Now _the_ standard processing script for scan maps
   * Photometer Mapping
      * Photometer mapping accepts a variable input pixelsize (parameter pixfrac in the !PhotProjectTask and the !MapIndexTask). 
      * The !MapIndexViewer has a Sigclip gui to display the effect of different Sigclip parameters on the second level deglitching. Now it is not necessary any 
                             more to guess Sigclip parameters, run the deglitching and check the quality afterwards. The new option allows to find the best parameter quickly 
                             _before_ deglitching has been applied. <br>
                            The !MapIndexViewer now also displays the signal vectors in the "timeordered" mode.
      * The !MapIndexViewer shows mean/median +/- error permanently in the signal plot
      * The Full !MapIndex needs less memory than before
      * !Wcs4mapTask got a crota option to manually set the requested rotation angle of a map.
      * !CorrectRaDec4SsoTask is now also accepting the new Horizons class definition. An example how to use it can be found in the attached.
      * photProject error map has been checked for !NaN.
      * Exposure map has been removed in favour of the coverage map.
   * Noise treatment
      * A new module !herschel.pacs.spg.phot.PhotAddNoisePerPixelTask provides four ways to calculate the initial photometer noise.
      *Noise propagation is also implemented in the !HighpassFilter, Mean- and !MedianFilters. <br>
   * Pointing 
      * New Horizons Product used in PACS pipeline (SSO observations).
      * Residual spatial offset PACS photometer blue/green vs red.
      * Corners shall follow changes in ==PhotAssignRaDec==.
      * Separate blue and green in ==ArrayInstrument== calfile and the code using it.
      * Astrometric offset between two scan maps in parallel mode on the same field. 
   * Others 
      * PACS EDP: phot pipeline adaptation to cope with variable scan velocity.
      * A new module photMaskFrames offer masking:
      * framesOut = photMaskFrames(framesIn [,beforeFirstScanLeg] [,noScanData] [,afterCalblock=0] )
         * beforeFirstScanLeg: Mask data befor the first scan leg
         * noScanData: mask data where the !ScanBBID is not set (includes turn arounds)
         * afterCalblock: Mask n data after calibration block
      * Sigclip can now be applied n-times in one go. 

---++++ Spectrometry

   * new AOR mode "unchopped" including improved spatial calibration:
      * First draft of the unchopped mode pipeline added to the build.
         * A "newstyle" unchopped mode pipeline is in place which can be run automatically or interactively.
      * New task specSubtractOffPosition implemented for unchopped pipeline .
   * New interactive tool !SpectrumExplorer:
      * PACS Product interface to !SpectrumExplorer
   * Improved wavelength calibration:
      * New wavelength calibration applied in waveCalc task using the wavePolynomes cal file.
   * spectrometer trend analysis product for the calibration blocks:
      * Calibration block trendAnalysis product added to the observation context.
   * Extra options to specDiffChop (note these are _expert_ options for the analysis of special observations!).
      * Extra option 'extend' added which allows for other pairwise differencing schemes. These schemes will subtract more datapoints from each other than the standard scheme. 
      * Extra option 'chopNodScheme' which provides different scheme for determining which label is on-source and which is off-source for a given nodding position. Needed for some special test AORs.
   * Different bands in a range scan are now stored in their own blocks in the block table. This means that it is now possible to slice SEDs per band. 
   * Several fixes to specWaveRebin, etc. to reduce the amount of !NaNs considerably in the rebinned data.
   * ==SpecProject== sets the flux to Jy.

---+++ Calibration


---++++ Photometry

   * Residual spatial offset PACS photometer blue/green vs red --> !PCalPhotometer_ArrayInstrument_FM_v6.fits.
---++++ Spectrometry
   * New Wavelength Calibration Product.
   * Version 8 of the Pacs Spectrometer Array to Instrument  and version 4 of the Module2Array cal file are inserted in the system. This generates correct values also for center chopper positions (unchopped and waveswitch mode).
   * Updated spectrometer spatial cal file --> !PCalSpectrometer_ArrayInstrument_FM_v6.fits.
   * The RSRF calibration tables contain "NaN" which are then introduced into the signal by rsrfCal.

---+++ Analysis tools

---++++ Spectrometry

   * Compatibility of PACS products with HCSS tools improved: you can now properly select data points and use the fitter tool on PACS frames and cubes.

---++ SPIRE

---+++ Calibration Products
---++++ Photometer 

*Updated*
   1. *Flux Calibration* updated for the nominal voltages.
   1. *Temperature Drift Correction* to match the new flux conversion.

---++++ Spectrometer
*New*
   1. *Non-linear phase*
      * This calibration product is used by the phase correction module to make the data symmetric prior to application of the Fourier transform.
   1. *Spectrometer Instrument RSRF*
      * This calibration product is used, along with the instrument temperatures from the housekeeping telemetry, to compute a model for the contribution to the measured spectra from the SPIRE instrument.
   1. *Telescope RSRF*
      * This calibration product is used, along with the Telescope temperatures from the housekeeping telemetry, to compute a model for the contribution to the measured spectra from the Herschel Telescope.
      * This calibration product is also used for extended source flux calibration.

*Updated*
   1. *Point-source RSRF*
      * This calibration product has been updated to use a higher signal-to-noise observation of Uranus.

---+++ Calibration Framework

   * Updated to match the new products and dependencies.
   * The =spireCal= task now has an option to read from a local pool.

---+++ Common Pipeline
   * Minor changes in logs and documentation.

---+++ Photometer Pipeline
   * Pipeline scripts:
      * Added time correlation task and concurrent glitch deglitcher task in scan map pipelines.
      * Comment out optical and electrical crosstalk corrections.
      * Removed plotting blocks.
   * Pipeline tasks:

---+++ Spectrometer Pipeline
The spectrometer pipelines have undergone major changes for version 5.  *It is HIGHLY recommended that users with data processed with older versions update to version 5 to reprocess their data.*

---++++ Level-0 -> Level 0.5
No change.

---++++ Level-0.5 -> Level 1 (IFGM)
   * Electrical crosstalk correction has been commented out of the SPG pipelines.

---++++ Level 1 (IFGM) -> Single-sided Fourier transform
   * Subtraction of the Instrument/Telescope contribution has been moved to the spectral domain.
   * Phase correction now uses a Jiggle dependent empirically-derived Non-linear phase cal product.

---++++ Single-sided Fourier transform -> Level 1 (SPEC)
   * Correction for the contribution of the SPIRE instrument.  A new processing step has been added to remove from the measured spectrum the contribution from the SPIRE instrument.
   * Correction for the contribution of the Herschel Telescope.  A new processing step has been added to remove from the measured spectrum the contribution from the Herschel Telescope.
   * Optical crosstalk correction has been commented out of the SPG pipelines.

---++++ Level 1 (SPEC) -> Level 2
   1. *SOF1 Sparse spatial sampling mode*
      * A minor change in that the point source calibration product is now tailored to spectral resolution and apodization.
   2. *Mapping Modes*
      * Added !NaiveProjectionTask, a new task for creating spectral simple cubes from preprocessed spectral data.  This task creates spectral cubes by averaging the flux of samples within each pixel.  Pixels without samples are given a value of !NaN.
 
---+++ Interactive Analysis and Tools
   * Added the new DTMosaic class that allows the user to plot data recorded by many detectors during one observation on a single page to compare data more effectively (mosaic plot). This features is available from HIPE command line or within ==DetectorTimelineExplorer==.
   * ==DetectorTimelineExplorer== and ==DetectorTimelineExplorerComponent==:
      * Added a new features that allows the creation of mosaic plot of the detectors via right-click on the desired array.
      * Added a new Color&Mask Preferences panel.
      * Improved the color scale (using the Color&Mask Preferences it is now possible to select the Min/Max value for the color scale).
      * Improved the mask visualization (using the Color&Mask Preferences it's now possible to visualize any specific mask value directly on the array display).
   * Made available the !Mosaic class which creates a mosaic of plots in the shape of the SLW, SSW or coaligned detectors of the SPIRE FTS.  The !Mosaic class is intended to provide a means for viewing (at a glance) the spectra or interferograms associated with each detector.
   * Changed !SpecExplorer's detector display so that it automatically re-sizes to fit the enclosing frame.  Also, !SpecExplorer's detectors are now enclosed in a splitter pane.  This enables you to maximize or hide the detector display with a single click. 
   * A new =SpireMultiObs= task to process several observations through the pipeline at once.

<!--
   * Set ALLOWTOPICCHANGE = Main.RegisteredUsersGroup, Main.TWikiAdminGroup
-->
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
gif	toolbox.gif	r1	manage	1.0 K	2010-12-16 - 22:27	CarolynMcCoey	New spectrum toolbox icon
Topic revision: r62 - 2011-02-07 - DavideRizzo
Public
Webs
Public
TWiki