Difference: HipeWhatsNew6x (1 vs. 49)

Revision 492012-10-30 - DavideRizzo

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"
<-- ANALYTICS CODE - DO NOT EDIT -->
<-- Google Analytics script BEGIN -->
<-- Google Analytics script END -->
Line: 7 to 7
 
PDF Version Click here
Added:
>
>
Help Interested in what's new for other versions of HIPE? See this page for links to all the What's New documents from HIPE 4.0 onwards.
 This document lists the changes in HIPE version 6.0 with respect to the 5.x series. Additional pages list changes between minor versions of the 6.x series.

Revision 482012-10-19 - DavideRizzo

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"
Added:
>
>
<-- ANALYTICS CODE - DO NOT EDIT -->
<-- Google Analytics script BEGIN -->
<-- Google Analytics script END -->

<-- END OF ANALYTICS CODE -->
 

What's New in HIPE 6

PDF Version Click here
Line: 486 to 489
  -->
Changed:
<
<
META FILEATTACHMENT attachment="Screen_shot_2010-12-14_at_10.34.04.png" attr="" comment="Screen Shot Calibration Framework" date="1295274545" name="Screen_shot_2010-12-14_at_10.34.04.png" path="Screen_shot_2010-12-14_at_10.34.04.png" size="54742" stream="Screen_shot_2010-12-14_at_10.34.04.png" user="Main.EkkehardWieprecht" version="1"
>
>
META FILEATTACHMENT attr="h" autoattached="1" comment="Script for Google Analytics" date="1350656122" name="GoogleAnalytics.txt" path="GoogleAnalytics.txt" size="549" user="Main.DavideRizzo" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Screen Shot Calibration Framework" date="1295274546" name="Screen_shot_2010-12-14_at_10.34.04.png" path="Screen_shot_2010-12-14_at_10.34.04.png" size="54742" user="Main.EkkehardWieprecht" version="1"

Revision 472012-05-14 - DavideRizzo

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 465 to 465
 

Mosaic plotting utility

A new plotting utility, Mosaic, has been developed to allow one to plot the contents of a SPIRE FTS Interferogram or Spectral Data Product in a grid that corresponds to the physical layout of the SPIRE SLW and SSW detector arrays.
Added:
>
>

<-- COMMENT BOX CODE - DO NOT EDIT -->
blog comments powered by Disqus
<-- END OF COMMENT BOX CODE -->

 

Revision 462011-07-04 - BertrandMorin

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 108 to 108
 

Statistics

Trend analysis

Mapping

Added:
>
>
-->

Wavelet transform

  • Delivered components:
    Algorithm One dimensional signal two dimensional signal Gaussian noise estimator tool Thresholding tool Universal thresholding tool
    CWT Continuous Wavelet Transform supported - not available not available not available
    DWT Discrete Wavelet Transform supported supported available available available
    SWT Stationary Wavelet Transform supported supported available available available
    Wavelet Daubechie Symlet Coiflet MexicanHat Morlet
    CWT 1 - - yes no
    DWT 1-10 1-10 1-5 - -
    SWT 1-10 1-10 1-5 - -

Revision 452011-04-12 - DavideRizzo

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 6 to 6
  This document lists the changes in HIPE version 6.0 with respect to the 5.x series. Additional pages list changes between minor versions of the 6.x series.
Added:
>
>
 

Revision 442011-03-25 - CarolynMcCoey

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 225 to 225
 

HIFI Software Configuration and Properties

HIFI Calibration

Changed:
<
<
  • Addition of getCalTree enables download of new calibration from HSA into a local store
    • task is called as cal = getHifiCal(useHsa=True, overWrite=False, calName="hifi-cal")
>
>
  • Addition of getCalTree enables download of new calibration from HSA into a local store. It is used in the following way:
cal = getHifiCal(useHsa=True, overWrite=False, calName="hifi-cal")
  • Update to spur table in the calibration tree
  • Calibration products no longer registered as proprietary in the HSA. In consequence, it is no longer necessary to have HSA authorisation to run the pipeline and update calibration.
 

HIFI Pipeline

  • The little-used Hifi Pipeline view is removed from HIPE
Added:
>
>
  • Data processed at the HSA On Demand will now write the version of the pipeline used (SPG version) in reprocessing in the creator field of the observation context meta data
  • It is now necessary to configure the HIFI pipeline to access the HSA before running the pipeline using the last released calibration tree
obs = getObservation("1342205520", useHsa=True)

cal = configureHifiPipeline()

obs_1 = hifiPipeline(obs=obs,fromLevel=0.0,upToLevel=2.0, cal=1, aux=1, palStore=cal, save=False)

  • Various changed to HIFI pipeline GUI
    • "instrument" replaced with "spectrometer"
    • palStore moved to the Expert Window
    • hifical moved to the Basic Window
    • hifical tooltip should indicate that setting hifical uses the latest calibration available to Astronomers

 

hifiPipeline task

  • Bug fix: the auxilliary product was not repopulated when running the hifiPipeline task with aux=True. This is now fixed.
  • First steps towards making the HIFI pipeline fully configurable are made with the introduction of doPipelineConfiguration. At present changes to the level 2 pipeline algorithm are not passed to the hifiPipeline task. This feature will be much more robust and user-friendly in HIPE 7.0
Line: 237 to 254
 
  • Now only the subbands affected by a major spur are flagged, previously it was all four subbands of the WBS regardless of whether they were all affected or not.

WBS Pipeline

  • Threshold behaviour is changed in MkWbsFreq: the threshold value is not divided by the scancount unless it is > 1023
Added:
>
>
  • Structure put in place to cross-calibrate WBS data with that from HRS
 
Added:
>
>

Level 1 Pipeline

  • Rearranged pipeline algorithm for Frequency Switch and Load Chop data. The bandpass correction is now done before the smoothing and subtraction of the OFF position. This has the consequence of removing small spurs seen in the WBS V and also improving baseline quality.
  • Introduction of truncated linear interpolation scheme in the MkOffSmooth step of the level 1 pipeline. This is to avoid numerical problems when two Tsys spectra (used to calculate the weights used in averaging) are very far apart.
  • Smoothing widths in MkOffSmooth optimised in calibration tree. Further work and improvements expected
 

Level 2 Pipeline

  • The WBS metadata produced by the doFreqGrid step of the pipeline has been clarified: the frequencyWidth metadata description is changed from 'The unique (!) channel width for the output grids of all subband' to 'The spacing of the frequency grids after resampling'.
  • The items 'forwardEff' and 'beamEff' and are added to the metadata after the completion of the doAntennaTemp or doMainBeamTemp steps of the level 2 pipeline, as applicable.
Line: 252 to 275
 

HIFI Products

  • HifiSpectrumDataset is converted such that it contains upConvert_H or upConvert_V metadata when the band is 6 or 7. This allows to read off the value in order to convert to sky frequency.
  • Additions to Quality Products to indicate if there were LO tuning problems.
Added:
>
>
  • New statistics trend product. Statistics (rms etc) are calculated at the end of the Levels 1 and 2 pipeline and results stored in the new statistics product. Meant for trending analysis for the ICC but may be of interest to astronomers. Note that statistics are calculated 'blind' with no line masking!
  • Clash between cached products are references resolved so consistent behaviour seen when modifying products.
 
Deleted:
<
<
Added:
>
>

Spectrum tasks (HIFI specifics)

  • Problem with statistics task whereby the preview of the task result did not update when several spectra selected is fixed
 

HIFI Data Processing Tools

  • Bug fix: tasks under hifi_dp_tools did not load into HIPE
Line: 266 to 293
 

Standing Wave Removal

  • New 'automask' option that allows automated line masking to be switched off.
  • New 'sds_index' option allowing the processing of specific datasets in ObservationContexts.
Added:
>
>
  • Problem whereby sine waves added to datasets containing multiple scans (e.g. OTF maps) fixed
  • Problem where masks appeared to be displaced from bright lines is fixed
 

Baseline Removal

  • New options wingChannels and minChannels to improve automated line masking (domask=2).
Line: 276 to 305
 
  • Improved error message handling from output of DoDeconvolution

Spectral Cube creation

Changed:
<
<
  • Major updates to doGridding GUI: improved variable entry, tooltips and design. Task GUI now opens on the expert view.
>
>
  • Major updates to doGridding GUI: improved parameter entry, tooltips and design. Task GUI now opens on the expert view.
  • doGridding makes a work around to calculate an appropriate pixel size regardless of whether the map was made with half beam spacing or was Nyquist sampled. Previously only a value appropriate for half-beam spacing was used. In the future the values used for the observation will be taken from the downlink product.
 
<-- 

HiClass export tool

-->

Documentation

Changed:
<
<
  • Updates to the following sections of the HIFI User Manual: Data Primer, Running the HIFI Pipeline, Quality Flags, HIFI Standing Wave Removal Tool, HIFI Baseline Removal, Exporting HIFI Data to CLASS
>
>
  • Updates to the following sections of the HIFI User Manual: Data Primer, Running the HIFI Pipeline, Quality Flags, HIFI Standing Wave Removal Tool, HIFI Baseline Removal, Exporting HIFI Data to CLASS, HIFI Cookbook
 
  • Addition of two sections to the HIFI User Manual: Reference Frames in HIFI Data, Making Publication Quality Plots
  • Many outdated references to JIDE removed from URM
Added:
>
>
  • addition of getHifiCal to URM
 
  • Addition of HIFI cookbook link directly from HIPE Help -> Data Reduction menu

Revision 432011-03-10 - DavideRizzo

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 6 to 6
  This document lists the changes in HIPE version 6.0 with respect to the 5.x series. Additional pages list changes between minor versions of the 6.x series.
Added:
>
>
 

Revision 422011-03-09 - DavideRizzo

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 6 to 6
  This document lists the changes in HIPE version 6.0 with respect to the 5.x series. Additional pages list changes between minor versions of the 6.x series.
Added:
>
>
 

See also the HIPE known issues and the Data products known issues pages for last-minute information on known problems.

Revision 412011-03-08 - DavideRizzo

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

PDF Version Click here
Added:
>
>
This document lists the changes in HIPE version 6.0 with respect to the 5.x series. Additional pages list changes between minor versions of the 6.x series.

See also the HIPE known issues and the Data products known issues pages for last-minute information on known problems.

 

Core system

Revision 402011-03-04 - DavideRizzo

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 162 to 162
 
  • Added descriptions to tasks. All tasks can provide a high level description (shown in tooltips in Task View).
  • Simplified localStoreWriter task.
Deleted:
<
<

Quality control

  • QCR Browser allows you to perform bulk actions on several quality control reports simultaneously.
    • This includes: Generate QC Summary directly as a menu option
  • Show metadata proCaveats in the browser.
  • Operator can assign different colors to the quality flags.

Systematic product generation

  • SPG GUI can now launch OD preprocessing (Missing TM , Events Log , HK products).
  • Connect to database only when needed, releasing this resource otherwise.
  • Extended process:
    • Re-process feature for SYSTEMATIC and ON_DEMAND when the original data is obsolete.
    • Cancel the OC ingestion if the process throws an Error or Exception.
    • Limit the number of observations being processed.
    • Memory management in operation's environment for subordinate jobs.
  • On-demand pipelines must be able to process any observation with the latest auxiliary and calibration data.

Bulk specific tool:

  • Operator can now re-execute individual jobs.
  • Ability to launch one of two parallel mode obsids.
 

Revision 392011-03-04 - BrunoMerin

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 16 to 16
 
  • Help > About shows the HIPE release number as well as the build number.
  • Data reduction guides for the different instruments are available directly from the Help menu.
  • HSC Helpdesk can be accessed from the Help menu.
Added:
>
>
  • Direct access to on-line help from Help menu.
 
  • Crash reports can be sent for analysis to the HSC upon user confirmation.
  • Tip of the Day additions.
  • Enhanced help for jylaunch tool in the terminal.
Line: 33 to 34
 When a task is selected, a summary information of it is shown.

Console
Changed:
<
<
The awkward horizontal scroll bar has been removed.
>
>
The horizontal scroll bar has been removed.
 
History
Copying an entire exception trace can be done with the history entry collapsed.

Revision 382011-03-03 - CarolynMcCoey

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 250 to 250
 

Level 0 Pipeline

  • Now only the subbands affected by a major spur are flagged, previously it was all four subbands of the WBS regardless of whether they were all affected or not.

WBS Pipeline

Changed:
<
<
  • Threshold behaviour is changed in MkWbsFreq: the threshold value is not divided by the scancount unless it is > 1023
>
>
  • Threshold behaviour is changed in MkWbsFreq: the threshold value is not divided by the scancount unless it is > 1023
 

Revision 372011-03-02 - CarolynMcCoey

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 244 to 244
 

HIFI Pipeline

  • The little-used Hifi Pipeline view is removed from HIPE
Changed:
<
<
<-- ---++++ hifiPipeline task -->
>
>

hifiPipeline task

  • Bug fix: the auxilliary product was not repopulated when running the hifiPipeline task with aux=True. This is now fixed.
  • First steps towards making the HIFI pipeline fully configurable are made with the introduction of doPipelineConfiguration. At present changes to the level 2 pipeline algorithm are not passed to the hifiPipeline task. This feature will be much more robust and user-friendly in HIPE 7.0
 

Level 0 Pipeline

  • Now only the subbands affected by a major spur are flagged, previously it was all four subbands of the WBS regardless of whether they were all affected or not.
Changed:
<
<
>
>

Level 1 Pipeline

-->
 

Level 2 Pipeline

  • The WBS metadata produced by the doFreqGrid step of the pipeline has been clarified: the frequencyWidth metadata description is changed from 'The unique (!) channel width for the output grids of all subband' to 'The spacing of the frequency grids after resampling'.
  • The items 'forwardEff' and 'beamEff' and are added to the metadata after the completion of the doAntennaTemp or doMainBeamTemp steps of the level 2 pipeline, as applicable.
Line: 281 to 286
 

Deconvolution

  • DoDeconvolution now accepts an array of obsids, enabling a combination of data for which deconvolution can be performed.
  • DoDeconvolution now keeps a log of problems encountered with its inputs, outputs, and difficulties. The log is accessible with an ASCII reader.
Added:
>
>
  • Improved error message handling from output of DoDeconvolution
 

Spectral Cube creation

  • Major updates to doGridding GUI: improved variable entry, tooltips and design. Task GUI now opens on the expert view.

Revision 362011-03-02 - DavideRizzo

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 83 to 83
 

Plotting

  • A small subplot can be put inside a bigger plot.
Added:
>
>
 

Source extraction

  • SUSSEXtractor no longer fails to find sources close to NaN pixels in the image or in the error map
  • SUSSEXtractor produced strange results when then error map had very small values. This was a problem because the naive mapper error is the standard error on the mean of the bolometer samples in the pixel, which can be extremely small if the pixel contains a small number of samples that happen to have very similar values. SUSSEXtractor now estimates its own value for the error when the naive mapper uncertainty is implausibly small.
Line: 108 to 112
 
  • Fixed some memory issues. The display does no longer keep a lot of memory in use.
  • It is possible to hide or show the panel with the panner and the zoom.
Changed:
<
<

Analysis

>
>
<-- ---++++  Analysis -->
 

Spectra

Changed:
<
<

Display

>
>
<-- ---++++  Display -->
 

Analysis

  • A Spectrum Integrator is available as prototype. Currently only the basic functionality is available from the command line and it allows to integrate part of a (background subtracted) spectrum. The data is expected to be in the form of a SpectralSegment (seg in the examples below).
    • Integration: The return value is a double.
Line: 137 to 141
  The variable 'seg' can now be used in the Integrator.
Changed:
<
<
>
>
 

Tasks

  • Improved documentation.
Line: 179 to 184
 
  • Operator can now re-execute individual jobs.
  • Ability to launch one of two parallel mode obsids.
Added:
>
>
 

Documentation

Line: 237 to 244
 

HIFI Pipeline

  • The little-used Hifi Pipeline view is removed from HIPE
Changed:
<
<

hifiPipeline task

>
>
<-- ---++++ hifiPipeline task -->
 

Level 0 Pipeline

  • Now only the subbands affected by a major spur are flagged, previously it was all four subbands of the WBS regardless of whether they were all affected or not.
Changed:
<
<

WBS Pipeline

>
>
 

Level 2 Pipeline

  • The WBS metadata produced by the doFreqGrid step of the pipeline has been clarified: the frequencyWidth metadata description is changed from 'The unique (!) channel width for the output grids of all subband' to 'The spacing of the frequency grids after resampling'.
  • The items 'forwardEff' and 'beamEff' and are added to the metadata after the completion of the doAntennaTemp or doMainBeamTemp steps of the level 2 pipeline, as applicable.
Line: 254 to 261
 

HIFI Products

  • HifiSpectrumDataset is converted such that it contains upConvert_H or upConvert_V metadata when the band is 6 or 7. This allows to read off the value in order to convert to sky frequency,
Added:
>
>
 

HIFI Data Processing Tools

  • Bug fix: tasks under hifi_dp_tools did not load into HIPE
Line: 276 to 285
 

Spectral Cube creation

  • Major updates to doGridding GUI: improved variable entry, tooltips and design. Task GUI now opens on the expert view.
Changed:
<
<

HiClass export tool

>
>
<-- ---+++ HiClass export tool -->
 

Documentation

  • Updates to the following sections of the HIFI User Manual: Data Primer, Running the HIFI Pipeline, Quality Flags, HIFI Standing Wave Removal Tool, HIFI Baseline Removal, Exporting HIFI Data to CLASS
Line: 305 to 314
 
  • The fitter tool can now be used for all PACS products.

Calibration

Changed:
<
<

Photometry

>
>
<-- ---++++ Photometry -->
 

Spectroscopy

  • New wavelength calibration applied by using the cal file wavePolynomes.
  • The NominalResponse cal file is active, including the 1.3 and 1.1 ILT to flight conversion factors.
Line: 328 to 337
 
    • obs = getObservation(123456789, useHsa=True, useCache=False)
    • This should solve many problems when you just want to download and save an observation from the HSA on your local machine, as the data will not be saved twice (once in the cache and once in the local store pool) anymore.
Added:
>
>
 

Documentation

  • Updated the PACS Data Reduction Guide to follow the new spectroscopy pipeline scripts (Chapter 3), and added a chapter on MadMap. Minor updates to the other chapters to reflect other changes in the system.
Line: 358 to 369
 

Preprocessing

  • Added processing of PCAL flashes executed in spectrometer observations.
Changed:
<
<

Engineering Conversion

>
>
 

Photometer Pipeline

  • Jump detection in signal timelines, allowing better temperature drift correction.
Line: 407 to 418
 

Interactive Analysis and Tools

Changed:
<
<
Mosaic plotting utility
>
>

Mosaic plotting utility

 A new plotting utility, Mosaic, has been developed to allow one to plot the contents of a SPIRE FTS Interferogram or Spectral Data Product in a grid that corresponds to the physical layout of the SPIRE SLW and SSW detector arrays.
Deleted:
<
<

Documentation

 
<--  
-->

Revision 352011-02-25 - AnthonySmith

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 99 to 99
 

Other functions

Source extraction

Added:
>
>
  • SUSSEXtractor no longer fails to find sources close to NaN pixels in the image or in the error map
  • SUSSEXtractor produced strange results when then error map had very small values. This was a problem because the naive mapper error is the standard error on the mean of the bolometer samples in the pixel, which can be extremely small if the pixel contains a small number of samples that happen to have very similar values. SUSSEXtractor now estimates its own value for the error when the naive mapper uncertainty is implausibly small.
 

Images

Display

Revision 342011-02-24 - DavideRizzo

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 199 to 199
 

Data Analysis Guide

Added:
>
>
 

Revision 332011-02-23 - SteveGuest

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 338 to 338
 

SPIRE

Calibration Products

Added:
>
>

Photometer

  • Improvement to flux conversion.

Spectrometer

  • Improved phase correction module in the pipeline no longer requires a calibration file. The final correction should be better using the new method.
  • Rsrfs and beamParam updated to follow the new v6 pipeline scheme and use improved phase correction - should be improvement in shape of final spectra.
  • BeamParam file modified to improve information for IA - updated beamAreas and added point source RSRF - this should have no effect on the SPG pipeline.
  • OpdLimits added to clean up the pipeline script - should be no effect on data.
  • Fix to non-linearity correction.
 
  • Added PCAL calibration products for the spectrometer.
Added:
>
>
 

Calibration Framework

Preprocessing

Line: 349 to 360
 

Common Pipeline

Photometer Pipeline

Added:
>
>
  • Jump detection in signal timelines, allowing better temperature drift correction.
 
  • Added flag to include or exclude turnaround data inscan map observations.

Spectrometer Pipeline

Revision 322011-02-23 - CarolynMcCoey

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 229 to 229
 

HIFI Software Configuration and Properties

HIFI Calibration

Added:
>
>
  • Addition of getCalTree enables download of new calibration from HSA into a local store
    • task is called as cal = getHifiCal(useHsa=True, overWrite=False, calName="hifi-cal")
 

HIFI Pipeline

Added:
>
>
  • The little-used Hifi Pipeline view is removed from HIPE
 

hifiPipeline task

Level 0 Pipeline

Added:
>
>
  • Now only the subbands affected by a major spur are flagged, previously it was all four subbands of the WBS regardless of whether they were all affected or not.
 

WBS Pipeline

HRS Pipeline

Level 1 Pipeline

Level 2 Pipeline

Added:
>
>
  • The WBS metadata produced by the doFreqGrid step of the pipeline has been clarified: the frequencyWidth metadata description is changed from 'The unique (!) channel width for the output grids of all subband' to 'The spacing of the frequency grids after resampling'.
  • The items 'forwardEff' and 'beamEff' and are added to the metadata after the completion of the doAntennaTemp or doMainBeamTemp steps of the level 2 pipeline, as applicable.
  • MkFreqGrid task is adjusted so that all output spectra have the same length of subbands. This allows all types of output to continue to be handled by HifiSpectrumDataset
  • Improvements to messages from doMainBeamTemp, the following messages are now given:
    • When no beam efficiency is passed, or no beam efficiency is present in the calibration product passed to the task: "Product with name beamEfficiency-H not found in context. No information on the beam efficiency has been passed to the task doMainBeamTemp or the calibration data is outdated. Correction in task doMainBeamTemp is not applied."
    • When no input has been found in the calibration product: "No information on the beam efficiency was found in the calibration product. Please provide a value by hand".
 

HIFI Products

Added:
>
>
  • HifiSpectrumDataset is converted such that it contains upConvert_H or upConvert_V metadata when the band is 6 or 7. This allows to read off the value in order to convert to sky frequency,
 

HIFI Spectrum Toolbox

Spectrum Explorer (HIFI specifics)

Spectrum tasks (HIFI specifics)

HIFI Data Processing Tools

Added:
>
>
  • Bug fix: tasks under hifi_dp_tools did not load into HIPE
  • Improved error messages from ConvertK2Jy
 

Standing Wave Removal

  • New 'automask' option that allows automated line masking to be switched off.
Changed:
<
<
>
>
  • New 'sds_index' option allowing the processing of specific datasets in ObservationContexts.
 

Baseline Removal

Changed:
<
<
  • New options wingChannels and minChannels to improve automated line masking (domask=2).
>
>
  • New options wingChannels and minChannels to improve automated line masking (domask=2).
 

Deconvolution

Added:
>
>
  • DoDeconvolution now accepts an array of obsids, enabling a combination of data for which deconvolution can be performed.
  • DoDeconvolution now keeps a log of problems encountered with its inputs, outputs, and difficulties. The log is accessible with an ASCII reader.
 

Spectral Cube creation

Added:
>
>
  • Major updates to doGridding GUI: improved variable entry, tooltips and design. Task GUI now opens on the expert view.
 

HiClass export tool

Documentation

Added:
>
>
  • Updates to the following sections of the HIFI User Manual: Data Primer, Running the HIFI Pipeline, Quality Flags, HIFI Standing Wave Removal Tool, HIFI Baseline Removal, Exporting HIFI Data to CLASS
  • Addition of two sections to the HIFI User Manual: Reference Frames in HIFI Data, Making Publication Quality Plots
  • Many outdated references to JIDE removed from URM
 

PACS

Revision 312011-02-22 - AdwinBoogert

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 246 to 246
 

HIFI Data Processing Tools

Standing Wave Removal

Added:
>
>
  • New 'automask' option that allows automated line masking to be switched off.
  • New 'sds_index' option allowing the processing of specific datasets in ObservationContexts.
 

Baseline Removal

Added:
>
>
  • New options wingChannels and minChannels to improve automated line masking (domask=2).
 

Deconvolution

Spectral Cube creation

Revision 302011-02-22 - PaulBalm

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 79 to 79
 
  • proper clean-up in case of failed installation,
  • reducing the number of required restarts.
Added:
>
>
Plug-ins can now also be used when scripts are run using jylaunch.
 

Plotting

  • A small subplot can be put inside a bigger plot.

TablePlotter

Revision 292011-02-18 - DavideRizzo

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 183 to 183
 

Documentation

Added:
>
>
Many fixes and updates to align the documentation with changes in HIPE. Some highlights are given in the following sections. Links are to the online version of the documentation

Quick start guide

HIPE Owner's Guide

Data Analysis Guide

Scripting and Data Mining

Products Definitions Document

 

Help viewer

  • Completely redesigned Welcome page.
  • Advanced search functionality.

Revision 282011-02-18 - DavideRizzo

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 222 to 222
 

PACS

Pipeline

Changed:
<
<
  • Full functional slicing framework
    • used for spectrometer processing, only partly used for photometer processing
>
>
  • Fully functional slicing framework.
    • Used for spectrometer processing, only partly used for photometer processing.
 
  • Significant cleanup and re organisation of interactive processing scripts.
Changed:
<
<
  • New structure of Menu -> Pipeline -> PACS
  • A number of instrument mode specific user provided IA scripts are available now
>
>
  • New structure of Menu -> Pipeline -> PACS.
  • A number of instrument-mode-specific user-provided IA scripts are available now.
 

Photometry

Changed:
<
<
  • Proposal for a two-stage photometer pipeline with masked highpass filtering implemented
  • MMT Deglitching may mask a glitch, although the signal would be unmodified
  • Improvements on data filtering (usage of BBID in removeCalBlocks, usage of scan legs etc.)
  • Significant improvements in the SSO astrometry (correctRaDec4Sso)
  • Full functional MadMap processing implementation inclusive of cross scans
>
>
  • Implemented proposal for a two-stage photometer pipeline with masked highpass filtering.
  • MMT Deglitching may mask a glitch, although the signal would be unmodified.
  • Improvements on data filtering (usage of BBID in removeCalBlocks, usage of scan legs etc.).
  • Significant improvements in the SSO astrometry (correctRaDec4Sso).
  • Fully functional MadMap processing implementation inclusive of cross scans.
 

Spectroscopy

Changed:
<
<
  • Linearity correction removed from pipeline scripts
  • Usage of elements of the Spectrum Explorer
>
>
  • Linearity correction removed from pipeline scripts.
  • Usage of elements of the Spectrum Explorer.
 
  • The fitter tool can now be used for all PACS products.

Calibration

Photometry

Spectroscopy

Changed:
<
<
  • New wavelength calibration applied by using the cal file wavePolynomes
  • The NominalResponse cal file is active, including the 1.3 and 1.1 ILT to flight conversion factors.
    • This means, if you process data you shall not apply these correction factors manually.
>
>
  • New wavelength calibration applied by using the cal file wavePolynomes.
  • The NominalResponse cal file is active, including the 1.3 and 1.1 ILT to flight conversion factors.
    • This means that if you process data you shall not apply these correction factors manually.
 
  • Also the pixel-pixel flatfield improvement described in PICC-KL-TN-045 (available in the central file) is implemented
Changed:
<
<
in this NominalResponse calfile
>
>
in this NominalResponse calfile.
 

Analysis tools

  • Calibration framework : As of build 6.0.1452 the calibration framework will read the calibration
Line: 258 to 257
  Screen_shot_2010-12-14_at_10.34.04.png
    • Please go through the process of installing the calibration products on your local machine.
    • During the HIPE startup process, the calibration framework will check for new updates in the background and notify you of any updates.
Changed:
<
<
    • If you want to start the update manually, go to the Tools>pacs-cal menu and select Run Updater...
>
>
    • If you want to start the update manually, go to the Tools > pacs-cal menu and select run Updater.
 
    • You can read more about the PACS Calibration Framework in the comprehensive document available in the public part of the Wiki.

  • Data Access : A new keyword is introduced to getObservation which allows the user to disable the Cache when using the HSA.
    • obs = getObservation(123456789, useHsa=True, useCache=False)
Changed:
<
<
    • This should solve many problems when the user just wants to download and save an observation from the HSA on his/her local machine, as the data will not be saved twice (once in the cache and once in the local store pool) anymore.
>
>
    • This should solve many problems when you just want to download and save an observation from the HSA on your local machine, as the data will not be saved twice (once in the cache and once in the local store pool) anymore.
 

Photometry

Spectroscopy

Line: 273 to 272
 

Spectroscopy

Documentation

Changed:
<
<
  • Updated the PACS Data Reduction Guide to follow the new spectroscopy pipeline scripts (chp3), and added a chapter on MadMap (this only got in a bit later in 6Track, however). Minor updates to the other chapters to reflect other changes in the system.
>
>
  • Updated the PACS Data Reduction Guide to follow the new spectroscopy pipeline scripts (Chapter 3), and added a chapter on MadMap. Minor updates to the other chapters to reflect other changes in the system.
 

SPIRE

Calibration Products

Changed:
<
<
  • Added PCAL calibration products for the spectrometer
>
>
  • Added PCAL calibration products for the spectrometer.
 

Calibration Framework

Preprocessing

Changed:
<
<
  • Added processing of PCAL flashes executed in spectrometer observations
>
>
  • Added processing of PCAL flashes executed in spectrometer observations.
 

Engineering Conversion

Line: 294 to 293
 

Spectrometer Pipeline

Level 0 to Level 0.5

Changed:
<
<
See Engineering Conversion
>
>
See Engineering Conversion.
 

Level 0.5 to Level 1

Revision 272011-02-17 - DavideRizzo

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 21 to 21
 
  • Enhanced help for jylaunch tool in the terminal.

Startup & Shutdown

Changed:
<
<
  • The user is informed about variables that couldn't be saved. Note that saving variables on exit is optional; it can be set in preferences.
>
>
  • You are informed about variables that could not be saved. Note that saving variables on exit is optional; it can be set in preferences.
 
  • If HIPE is closed while having opened a script from the installation, and a different HIPE version is started afterwards, a warning is shown informing that the opened script belongs to other version.
  • LOCK files from previous connections to the HSA not properly closed are asked to be removed at HIPE startup.
  • Panel for Startup & Shutdown preferences has been improved.
Line: 43 to 43
 
Navigator
  • Compressed FITS files are recognized and opened as such.
Changed:
<
<
  • Enhanced mechanism for file identification; it supports XML file descriptors of observations -see MyHSA-.
>
>
  • Enhanced mechanism for file identification; it supports XML file descriptors of observations.
 
Jython editor
  • Autoindent and unident improved. Note that the corresponding preference must be marked to make use of this functionality.
Line: 103 to 101
 

Images

Display

  • Major speedup when zooming in on images. Less memory is used when zooming in.
Changed:
<
<
  • Fixed some memory issues. The display does not longer keep a lot of memory in use.
  • It is possible to hide or show the panel with the panner and the zoom
>
>
  • Fixed some memory issues. The display does no longer keep a lot of memory in use.
  • It is possible to hide or show the panel with the panner and the zoom.
 

Analysis

Line: 150 to 148
 

Product Access Layer

Tasks

Changed:
<
<
  • Improved documentation
>
>
  • Improved documentation.
 
  • Added descriptions to tasks. All tasks can provide a high level description (shown in tooltips in Task View).
Changed:
<
<
  • Simplified localStoreWriter task
>
>
  • Simplified localStoreWriter task.
 

Quality control

Changed:
<
<
  • QCR Browser allow to perform bulk actions on several qc reports simultaneously
>
>
  • QCR Browser allows you to perform bulk actions on several quality control reports simultaneously.
 
    • This includes: Generate QC Summary directly as a menu option
Changed:
<
<
  • Show metadata proCaveats in the browser
  • Operator can assign different colors to the quality flags
>
>
  • Show metadata proCaveats in the browser.
  • Operator can assign different colors to the quality flags.
 

Systematic product generation

Changed:
<
<
  • SPG GUI can now launch OD preprocessing ( Missing TM , Events Log , HK products )
  • Connect to database only when needed, releasing this resource otherwise
>
>
  • SPG GUI can now launch OD preprocessing (Missing TM , Events Log , HK products).
  • Connect to database only when needed, releasing this resource otherwise.
 
  • Extended process:
Changed:
<
<
    • Re-process feature for SYSTEMATIC and ON_DEMAND when the original data is obsolete
>
>
    • Re-process feature for SYSTEMATIC and ON_DEMAND when the original data is obsolete.
 
    • Cancel the OC ingestion if the process throws an Error or Exception.
Changed:
<
<
    • Limit the number of obs. being processed
    • Memory management in operation's environment for subordinate jobs
  • On-demand pipelines must be able to process any observation with the latest auxiliary and calibration data
>
>
    • Limit the number of observations being processed.
    • Memory management in operation's environment for subordinate jobs.
  • On-demand pipelines must be able to process any observation with the latest auxiliary and calibration data.
  Bulk specific tool:
Changed:
<
<
  • Operator can now re-execute individual jobs
  • Ability to launch one of 2 parallel mode obsids
>
>
  • Operator can now re-execute individual jobs.
  • Ability to launch one of two parallel mode obsids.
 

Data input-output

Line: 186 to 184
 

Documentation

Help viewer

Changed:
<
<
  • Advanced search functionality
  • Added information about currently viewed document and current version of HIPE
  • New functionality to jump to the index page of the currently open manual
>
>
  • Completely redesigned Welcome page.
  • Advanced search functionality.
  • Added information about currently viewed document and current version of HIPE.
  • New functionality to jump to the index page of the currently open manual.
 

HIFI

Revision 262011-02-01 - CarolynMcCoey

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 111 to 111
 

Spectra

Display

Analysis

Added:
>
>
  • A Spectrum Integrator is available as prototype. Currently only the basic functionality is available from the commandline and it allows to integrate part of a (background subtracted) spectrum. The data is expected to be in the form of a SpectralSegment (seg in the examples below).
    • Integration: The return value is a double.
      • i = Integrator.doIntegration(seg)
    • Integration over ranges: The return value is an array of n doubles. The windows are formatted as [start, stop, start, stop, ...]. So for one window from a to b: [a, b], and for n windows: [a1, b1, a2, b2, ..., an, bn]
      • i = Integrator.doIntegration(seg, [windows])
    • Integration over ranges with 1st order background removal: The background is removed by fitting a 1st order poly in the [masks] areas. The format of [masks] is similar to the [windows] format.
      • i = Integrator.doIntegration(seg, [windows], [masks])
    • Integration over ranges with nth order background removal:
      • from herschel.ia.toolbox.spectrum.integrator import Integrator
      • i = Integrator.doIntegration(seg, [windows], [masks], n)

How to get a SpectralSegment (HIFI example)

To get a SpectralSegment from the HIFI observation context 'obs', e.g. the first dataset of the level-2 HRS_L_LSB:

d = obs.refs["level2"].product.refs["HRS-H-LSB"].product.refs["1"].product["dataset"]

This is a HrsSpectrumDataset (which is a SpectrumContainer). Visualising this in the SpectrumExplorer shows that is contains one spectrum with 8 subbands. Each of the subbands is a SpectralSegment. Get any as:

seg = d.getPointSpectrum(0).getSegment('any')

The variable 'seg' can now be used in the Integrator.

 

Data cubes

Display

Revision 252011-01-25 - PasqualePanuzzo

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 254 to 255
 

SPIRE

Calibration Products

Changed:
<
<
>
>
  • Added PCAL calibration products for the spectrometer
 

Calibration Framework

Preprocessing

Added:
>
>
  • Added processing of PCAL flashes executed in spectrometer observations
 

Engineering Conversion

Common Pipeline

Photometer Pipeline

Added:
>
>
  • Added flag to include or exclude turnaround data inscan map observations.
 

Spectrometer Pipeline

Revision 242011-01-24 - PaulBalm

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 71 to 71
 
  • The criterion for determining the valid FITS extensions has been unified among different views and dialogues of HIPE.
  • The error dialogue that includes an exception trace has it collapsed by default.
Added:
>
>

Plug-ins

A number of different small issues related to plug-ins have been resolved, including

  • issues with the installation,
  • path names containing spaces (important for Windows XP),
  • proper clean-up in case of failed installation,
  • reducing the number of required restarts.

 

Plotting

  • A small subplot can be put inside a bigger plot.

TablePlotter

Revision 232011-01-20 - KatrinaExter

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 240 to 240
 

Spectroscopy

Documentation

Changed:
<
<
>
>
  • Updated the PACS Data Reduction Guide to follow the new spectroscopy pipeline scripts (chp3), and added a chapter on MadMap (this only got in a bit later in 6Track, however). Minor updates to the other chapters to reflect other changes in the system.
 

SPIRE

Calibration Products

Revision 222011-01-18 - JaimeSaiz

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 12 to 12
 

HIPE

Added:
>
>

Help & Support

  • Help > About shows the HIPE release number as well as the build number.
  • Data reduction guides for the different instruments are available directly from the Help menu.
  • HSC Helpdesk can be accessed from the Help menu.
  • Crash reports can be sent for analysis to the HSC upon user confirmation.
  • Tip of the Day additions.
  • Enhanced help for jylaunch tool in the terminal.

Startup & Shutdown

  • The user is informed about variables that couldn't be saved. Note that saving variables on exit is optional; it can be set in preferences.
  • If HIPE is closed while having opened a script from the installation, and a different HIPE version is started afterwards, a warning is shown informing that the opened script belongs to other version.
  • LOCK files from previous connections to the HSA not properly closed are asked to be removed at HIPE startup.
  • Panel for Startup & Shutdown preferences has been improved.

Editors & Views

Outline
When a task is selected, a summary information of it is shown.

Console
The awkward horizontal scroll bar has been removed.

History
Copying an entire exception trace can be done with the history entry collapsed.

Variables
Delete (selection) and Delete all buttons distinguished.

Navigator
  • Compressed FITS files are recognized and opened as such.
  • Enhanced mechanism for file identification; it supports XML file descriptors of observations -see MyHSA-.

Jython editor
  • Autoindent and unident improved. Note that the corresponding preference must be marked to make use of this functionality.
  • The line continuation character \ is highlighted at the end of a line.

Task dialogue
Checkboxes for boolean parameters accept variables holding an integer with value 0 (false) or 1 (true).

Menus

  • All dialogs and windows opened from HIPE, including plots, can be accessed through Window > Select Window.
  • Available views have been categorized under Window > Show View.
  • Entries of the File > New... menu have been simplified.

Preferences

  • Configuration properties can be edited directly from the preferences dialogue, by means of the new Advanced... button.
  • Saving preferences (with OK or Apply) is now faster and robust against failures.
  • Panel for Startup & Shutdown preferences has been improved.

Jobs control

  • Running jobs can be consulted and cancelled individually with the new Running Jobs button at the left side of the progress bar.
  • If more than one job is running, clicking on the Stop button will show a similar dialogue to choose the job(s) to stop.

Miscellanea

  • The criterion for determining the valid FITS extensions has been unified among different views and dialogues of HIPE.
  • The error dialogue that includes an exception trace has it collapsed by default.
 

Plotting

  • A small subplot can be put inside a bigger plot.

TablePlotter

Revision 212011-01-17 - EkkehardWieprecht

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 162 to 162
 
  • Calibration framework : As of build 6.0.1452 the calibration framework will read the calibration products no longer from the distribution, but from a directory on your local machine.
    • You will be presented with a popup dialog asking to install the calibration. The dialog looks like this.
Changed:
<
<
      • Screen_shot_2011-01-13_at_12.28.15.png
>
>
    • Screen Shot Calibration Framework:
      Screen_shot_2010-12-14_at_10.34.04.png
  *Please go through the process of installing the calibration products on your local machine.
    • During the HIPE startup process, the calibration framework will check for new updates in the background and notify you of any updates.
    • If you want to start the update manually, go to the Tools>pacs-cal menu and select Run Updater...
Line: 245 to 246
 
<--  
-->
\ No newline at end of file
Added:
>
>
META FILEATTACHMENT attachment="Screen_shot_2010-12-14_at_10.34.04.png" attr="" comment="Screen Shot Calibration Framework" date="1295274545" name="Screen_shot_2010-12-14_at_10.34.04.png" path="Screen_shot_2010-12-14_at_10.34.04.png" size="54742" stream="Screen_shot_2010-12-14_at_10.34.04.png" user="Main.EkkehardWieprecht" version="1"

Revision 202011-01-17 - EkkehardWieprecht

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 130 to 130
 

PACS

Pipeline

Added:
>
>
  • Full functional slicing framework
    • used for spectrometer processing, only partly used for photometer processing
  • Significant cleanup and re organisation of interactive processing scripts.
  • New structure of Menu -> Pipeline -> PACS
  • A number of instrument mode specific user provided IA scripts are available now
 

Photometry

Added:
>
>
  • Proposal for a two-stage photometer pipeline with masked highpass filtering implemented
  • MMT Deglitching may mask a glitch, although the signal would be unmodified
  • Improvements on data filtering (usage of BBID in removeCalBlocks, usage of scan legs etc.)
  • Significant improvements in the SSO astrometry (correctRaDec4Sso)
  • Full functional MadMap processing implementation inclusive of cross scans
 

Spectroscopy

Added:
>
>
  • Linearity correction removed from pipeline scripts
  • Usage of elements of the Spectrum Explorer
  • The fitter tool can now be used for all PACS products.
 

Calibration

Photometry

Spectroscopy

Added:
>
>
  • New wavelength calibration applied by using the cal file wavePolynomes
  • The NominalResponse cal file is active, including the 1.3 and 1.1 ILT to flight conversion factors.
    • This means, if you process data you shall not apply these correction factors manually.
  • Also the pixel-pixel flatfield improvement described in PICC-KL-TN-045 (available in the central file) is implemented in this NominalResponse calfile
 

Analysis tools

Added:
>
>
  • Calibration framework : As of build 6.0.1452 the calibration framework will read the calibration products no longer from the distribution, but from a directory on your local machine.
    • You will be presented with a popup dialog asking to install the calibration. The dialog looks like this.
      • Screen_shot_2011-01-13_at_12.28.15.png *Please go through the process of installing the calibration products on your local machine.
    • During the HIPE startup process, the calibration framework will check for new updates in the background and notify you of any updates.
    • If you want to start the update manually, go to the Tools>pacs-cal menu and select Run Updater...
    • You can read more about the PACS Calibration Framework in the comprehensive document available in the public part of the Wiki.

  • Data Access : A new keyword is introduced to getObservation which allows the user to disable the Cache when using the HSA.
    • obs = getObservation(123456789, useHsa=True, useCache=False)
    • This should solve many problems when the user just wants to download and save an observation from the HSA on his/her local machine, as the data will not be saved twice (once in the cache and once in the local store pool) anymore.
 

Photometry

Spectroscopy

Revision 192011-01-17 - DavideRizzo

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Revision 182011-01-14 - JoseOsinde

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 63 to 63
 

Quality control

Changed:
<
<
  • QCR Browser should allow to perform bulk actions on several qc reports simultaneously
>
>
  • QCR Browser allow to perform bulk actions on several qc reports simultaneously
 
    • This includes: Generate QC Summary directly as a menu option
  • Show metadata proCaveats in the browser
  • Operator can assign different colors to the quality flags

Revision 172011-01-14 - TrevorFulton

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 164 to 164
 

Spectrometer Pipeline

Level 0 to Level 0.5

Changed:
<
<
>
>
See Engineering Conversion
 

Level 0.5 to Level 1

Processing Steps

Phase Correction
Changed:
<
<
  • A new algorithm has been implemented that corrects for asymmetries using the low-resolution portion of the measured interferograms.
>
>
A new algorithm has been implemented that corrects for asymmetries using the low-resolution portion of the measured interferograms.
 
Data Products
Changed:
<
<
  • Level-1 spectral products now contain all measured extended flux-calibrated spectra rather than the average spectra per detector.
>
>
Level-1 spectral products now contain all measured extended flux-calibrated spectra rather than the average spectra per detector.
 

Level 1 to Level 2

Processing Steps
Spectral Averaging
Changed:
<
<
  • The spectral averaging step is now only apply to Sparse, Point-source observations.
  • The error column in the output data product is now populated with the standard error of the mean (Standard Deviation / SQRT(Number of Samples)) of the input spectral samples.
>
>
  1. The spectral averaging step is now only apply to Sparse, Point-source observations.
  2. The error column in the output data product is now populated with the standard error of the mean (Standard Deviation / SQRT(Number of Samples)) of the input spectral samples.
 
Spectral Maps
Changed:
<
<
  • NaiveProjectionTask replaces NearestNeighbourProjectionTask for spectral cube creation
Level2 spectral cubes are now generated by NaiveProjectionTask. NaiveProjectionTask creates spectral cubes by averaging flux values within each pixel.
  • All measured spectra per detector per Point and Jiggle position are now supplied as input to the Spectral mapping step.
  • The execution speed of NaiveProjectionTask has been greatly improved - a set of high resolution spectra is now processed in approximately 10 seconds.
>
>
Level2 spectral cubes are now created by averaging flux values within each pixel in the output map. In addition, the execution speed of Spectral Mapping processing step has been greatly improved - a set of high resolution spectra is now processed in approximately 10 seconds.
 
Data Products
Changed:
<
<
  • Level-1 spectral products now contain all measured extended flux-calibrated spectra rather than the average spectra per detector.
>
>
Level-1 spectral products now contain all measured extended flux-calibrated spectra rather than the average spectra per detector.
 

New Utitlities

A new set of utility functions and tasks have been implemented for HIPE version 6.
Changed:
<
<
filterChannels
* A task that allows one to remove a selection of channels from any type of SPRIE FTS data product.

removeOutOfBand
* A utility task that allows one to remove a from a SPIRE FTS Spectral data product spectral samples outside a specified band.

makeSameOpds
* A utility task that allows one to remove a from a SPIRE FTS Interferogram data product samples outside a specified OPD range.

mergeFtsBuildingBlocks
* A utility task that merges Level-1 SPIRE FTS Interferogram and Nominal Housekeeping products.

specMergeHighLowSds
* A utility task that is used for HR=+=LR SPIRE FTS observations. This task appends the LR portion of the interferograms from the HR buildling block to the interferograms from the LR building block at each Point and Jiggle position of the observation.

observationLayout
* A utility task that inspects the Level-0.5 context of a SPIRE FTS observation and returns a table linking the Building Block ID numbers with Point numbers, Jiggle numbers, and the commanded resolution.
>
>
  1. filterChannels. A task that allows one to remove a selection of channels from any type of SPRIE FTS data product.
  2. removeOutOfBand A utility task that allows one to remove a from a SPIRE FTS Spectral data product spectral samples outside a specified band.
  3. makeSameOpds A utility task that allows one to remove a from a SPIRE FTS Interferogram data product samples outside a specified OPD range.
  4. mergeFtsBuildingBlocks A utility task that merges Level-1 SPIRE FTS Interferogram and Nominal Housekeeping products.
  5. specMergeHighLowSds A utility task that is used for HR=+=LR SPIRE FTS observations. This task appends the LR portion of the interferograms from the HR buildling block to the interferograms from the LR building block at each Point and Jiggle position of the observation.
  6. observationLayout A utility task that inspects the Level-0.5 context of a SPIRE FTS observation and returns a table linking the Building Block ID numbers with Point numbers, Jiggle numbers, and the commanded resolution.
 

Interactive Analysis and Tools

Line: 215 to 201
 

Interactive Analysis and Tools

Added:
>
>
 
Mosaic plotting utility
Changed:
<
<
* A new plotting utility, Mosaic, has been developed to allow one to plot the contents of a SPIRE FTS Interferogram or Spectral Data Product in a grid that corresponds to the physical layout of the SPIRE SLW and SSW detector arrays.
>
>
A new plotting utility, Mosaic, has been developed to allow one to plot the contents of a SPIRE FTS Interferogram or Spectral Data Product in a grid that corresponds to the physical layout of the SPIRE SLW and SSW detector arrays.
 

Documentation

Revision 162011-01-14 - TrevorFulton

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 162 to 162
 

Photometer Pipeline

Spectrometer Pipeline

Deleted:
<
<

NaiveProjectionTask replaces NearestNeighbourProjectionTask for spectral cube creation

Level2 spectral cubes are now generated by NaiveProjectionTask. NaiveProjectionTask creates spectral cubes by averaging flux values within each pixel. The execution speed of NaiveProjectionTask has been greatly improved - a set of high resolution spectra is now processed in approximately 10 seconds.

Browse product image for observations

Within Hipe, observations with spectral cubes (observations with multiple raster positions or jiggle positions) now display an image summarizing the cubes. The image consists of the median flux values for each pixel of the SLW and SSW unapodized cubes. For observations without spectral cubes (sparse observations, no jiggle), an image showing a plot spectra of the central detectors is displayed.
 
Changed:
<
<

New Module: spire_ia_pipeline_spec_util

>
>

Level 0 to Level 0.5

 
Changed:
<
<
This module contains helper classes used in the SPIRE spectrometer pipelines.
>
>

Level 0.5 to Level 1

 
Changed:
<
<
FilterChannelsTask
>
>
Processing Steps
 
Changed:
<
<
This task removes or retains a list of channels from an SDT, SDI, or SDS.
>
>
Phase Correction
  • A new algorithm has been implemented that corrects for asymmetries using the low-resolution portion of the measured interferograms.
 
Changed:
<
<
RemoveOutOfBandTask
>
>
Data Products
  • Level-1 spectral products now contain all measured extended flux-calibrated spectra rather than the average spectra per detector.
 
Changed:
<
<
This task removes out-of-band spectral components from an input SDS.
>
>

Level 1 to Level 2

 
Changed:
<
<
ObservationLayoutTask
>
>
Processing Steps
Spectral Averaging
  • The spectral averaging step is now only apply to Sparse, Point-source observations.
  • The error column in the output data product is now populated with the standard error of the mean (Standard Deviation / SQRT(Number of Samples)) of the input spectral samples.
 
Changed:
<
<
This task determines the characteristics of an observation, namely point numbers, jiggle numbers, resolution, and building blocks IDs, and places them into a SpecObsContents dataset.
>
>
Spectral Maps
  • NaiveProjectionTask replaces NearestNeighbourProjectionTask for spectral cube creation
Level2 spectral cubes are now generated by NaiveProjectionTask. NaiveProjectionTask creates spectral cubes by averaging flux values within each pixel.
  • All measured spectra per detector per Point and Jiggle position are now supplied as input to the Spectral mapping step.
  • The execution speed of NaiveProjectionTask has been greatly improved - a set of high resolution spectra is now processed in approximately 10 seconds.
 
Changed:
<
<
MakeSameOpdsTask
>
>
Data Products
  • Level-1 spectral products now contain all measured extended flux-calibrated spectra rather than the average spectra per detector.
 
Changed:
<
<
This task makes all interferograms in an SDI have an OPD range consistent with the commanded resolution, truncating the data if necessary.
>
>

New Utitlities

A new set of utility functions and tasks have been implemented for HIPE version 6.
 
Changed:
<
<
MergeFtsBuildingBlocksTask
>
>
filterChannels
* A task that allows one to remove a selection of channels from any type of SPRIE FTS data product.

removeOutOfBand
* A utility task that allows one to remove a from a SPIRE FTS Spectral data product spectral samples outside a specified band.

makeSameOpds
* A utility task that allows one to remove a from a SPIRE FTS Interferogram data product samples outside a specified OPD range.

mergeFtsBuildingBlocks
* A utility task that merges Level-1 SPIRE FTS Interferogram and Nominal Housekeeping products.

specMergeHighLowSds
* A utility task that is used for HR=+=LR SPIRE FTS observations. This task appends the LR portion of the interferograms from the HR buildling block to the interferograms from the LR building block at each Point and Jiggle position of the observation.

observationLayout
* A utility task that inspects the Level-0.5 context of a SPIRE FTS observation and returns a table linking the Building Block ID numbers with Point numbers, Jiggle numbers, and the commanded resolution.
 
Deleted:
<
<
This task merges multiple SDIs and NHKTs into a single SDI and NHKT.
 

Interactive Analysis and Tools

Added:
>
>
Mosaic plotting utility
* A new plotting utility, Mosaic, has been developed to allow one to plot the contents of a SPIRE FTS Interferogram or Spectral Data Product in a grid that corresponds to the physical layout of the SPIRE SLW and SSW detector arrays.
 

Documentation

Revision 152011-01-13 - JoseOsinde

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 63 to 63
 

Quality control

Added:
>
>
  • QCR Browser should allow to perform bulk actions on several qc reports simultaneously
    • This includes: Generate QC Summary directly as a menu option
  • Show metadata proCaveats in the browser
  • Operator can assign different colors to the quality flags
 

Systematic product generation

Added:
>
>
  • SPG GUI can now launch OD preprocessing ( Missing TM , Events Log , HK products )
  • Connect to database only when needed, releasing this resource otherwise
  • Extended process:
    • Re-process feature for SYSTEMATIC and ON_DEMAND when the original data is obsolete
    • Cancel the OC ingestion if the process throws an Error or Exception.
    • Limit the number of obs. being processed
    • Memory management in operation's environment for subordinate jobs
  • On-demand pipelines must be able to process any observation with the latest auxiliary and calibration data

Bulk specific tool:

  • Operator can now re-execute individual jobs
  • Ability to launch one of 2 parallel mode obsids
 

Data input-output

Virtual Observatory

Revision 142011-01-13 - MarcoSoldati

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 73 to 73
 

Documentation

Added:
>
>

Help viewer

* Advanced search functionality * Added information about currently viewed document and current version of HIPE * New functionality to jump to the index page of the currently open manual
 

HIFI

HIFI Software Configuration and Properties

Revision 132011-01-13 - JavierDiaz

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 58 to 58
 

Tasks

  • Improved documentation
Changed:
<
<
  • Added descriptions to tasks. All tasks should provide a high level description (shown in tooltips in Task View).
>
>
  • Added descriptions to tasks. All tasks can provide a high level description (shown in tooltips in Task View).
 
  • Simplified localStoreWriter task

Quality control

Revision 122011-01-12 - PeterKennedy

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 138 to 138
 

Photometer Pipeline

Spectrometer Pipeline

Added:
>
>

NaiveProjectionTask replaces NearestNeighbourProjectionTask for spectral cube creation

Level2 spectral cubes are now generated by NaiveProjectionTask. NaiveProjectionTask creates spectral cubes by averaging flux values within each pixel. The execution speed of NaiveProjectionTask has been greatly improved - a set of high resolution spectra is now processed in approximately 10 seconds.

Browse product image for observations

Within Hipe, observations with spectral cubes (observations with multiple raster positions or jiggle positions) now display an image summarizing the cubes. The image consists of the median flux values for each pixel of the SLW and SSW unapodized cubes. For observations without spectral cubes (sparse observations, no jiggle), an image showing a plot spectra of the central detectors is displayed.
 

New Module: spire_ia_pipeline_spec_util

Revision 112011-01-12 - JavierDiaz

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 59 to 59
 

Tasks

  • Improved documentation
  • Added descriptions to tasks. All tasks should provide a high level description (shown in tooltips in Task View).
Changed:
<
<
>
>
  • Simplified localStoreWriter task
 

Quality control

Revision 102011-01-12 - DavidSharpe

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 149 to 149
 
RemoveOutOfBandTask
Changed:
<
<
This task removes out-of-band spectral components from an input SDS. The input SDS will be modified and returned as output.
>
>
This task removes out-of-band spectral components from an input SDS.
 
ObservationLayoutTask

Revision 92011-01-12 - JavierDiaz

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 57 to 57
 

Product Access Layer

Tasks

Added:
>
>
  • Improved documentation
  • Added descriptions to tasks. All tasks should provide a high level description (shown in tooltips in Task View).
 

Quality control

Revision 82011-01-12 - JingjingLi

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 13 to 13
 

HIPE

Plotting

Added:
>
>
  • A small subplot can be put inside a bigger plot.
 

TablePlotter

OverPlotter

Dataset viewer

Revision 72011-01-11 - DavidSharpe

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 135 to 135
 

Spectrometer Pipeline

Added:
>
>

New Module: spire_ia_pipeline_spec_util

This module contains helper classes used in the SPIRE spectrometer pipelines.

FilterChannelsTask

This task removes or retains a list of channels from an SDT, SDI, or SDS.

RemoveOutOfBandTask

This task removes out-of-band spectral components from an input SDS. The input SDS will be modified and returned as output.

ObservationLayoutTask

This task determines the characteristics of an observation, namely point numbers, jiggle numbers, resolution, and building blocks IDs, and places them into a SpecObsContents dataset.

MakeSameOpdsTask

This task makes all interferograms in an SDI have an OPD range consistent with the commanded resolution, truncating the data if necessary.

MergeFtsBuildingBlocksTask

This task merges multiple SDIs and NHKTs into a single SDI and NHKT.

 

Interactive Analysis and Tools

Documentation

Revision 62011-01-10 - DavideRizzo

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 97 to 97
 

HiClass export tool

Changed:
<
<

HIFI Documentation

>
>

Documentation

 

PACS

Pipeline

Photometry

Changed:
<
<

Spectrometry

>
>

Spectroscopy

 

Calibration

Photometry

Changed:
<
<

Spectrometry

>
>

Spectroscopy

 

Analysis tools

Photometry

Changed:
<
<

Spectrometry

>
>

Spectroscopy

 

Product changes

Photometry

Changed:
<
<

Spectrometry

>
>

Spectroscopy

 

Documentation

Deleted:
<
<

Other

 

SPIRE

Calibration Products

Line: 142 to 139
 

Documentation

Deleted:
<
<

Other

 
<--  
-->
\ No newline at end of file

Revision 52010-12-14 - CarolynMcCoey

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 71 to 71
 

HIFI

Changed:
<
<

hifiPipelineTask

>
>

HIFI Software Configuration and Properties

 
Changed:
<
<

HRS Pipeline

>
>

HIFI Calibration

 
Changed:
<
<

WBS Pipeline

Level 1 Pipeline

Level 2 Pipeline

Products

DoDeconvolution

DoGridding

FitHifiFringe

>
>

HIFI Pipeline

hifiPipeline task

Level 0 Pipeline

WBS Pipeline

HRS Pipeline

Level 1 Pipeline

Level 2 Pipeline

HIFI Products

HIFI Spectrum Toolbox

Spectrum Explorer (HIFI specifics)

Spectrum tasks (HIFI specifics)

HIFI Data Processing Tools

Standing Wave Removal

Baseline Removal

Deconvolution

Spectral Cube creation

 

HiClass export tool

Changed:
<
<

Flags

>
>

HIFI Documentation

 
Deleted:
<
<

SpectrumExplorer (HIFI specifics)

HIFI DP scripts (User tool scripts)

Documentation

Other

 

PACS

Revision 42010-12-02 - DavideRizzo

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 144 to 144
 

Documentation

Other

\ No newline at end of file
Added:
>
>
<--  
-->
 \ No newline at end of file

Revision 32010-11-02 - DavideRizzo

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 8 to 8
 

Core system

Added:
>
>
The Java Virtual Machine included in Windows and Linux downloads has been updated to version 1.6.0_22.
 

HIPE

Plotting

Revision 22010-10-05 - WimDeMeester

Line: 1 to 1
 
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

Line: 30 to 30
 

Images

Display

Added:
>
>
  • Major speedup when zooming in on images. Less memory is used when zooming in.
  • Fixed some memory issues. The display does not longer keep a lot of memory in use.
  • It is possible to hide or show the panel with the panner and the zoom
 

Analysis

Spectra

Revision 12010-10-01 - DavideRizzo

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="HipeWhatsNew"

What's New in HIPE 6

PDF Version Click here

Core system

HIPE

Plotting

TablePlotter

OverPlotter

Dataset viewer

Astronomical utilities

Numeric routines

Data fitting

Data interpolation

Random numbers

Statistics

Trend analysis

Mapping

Other functions

Source extraction

Images

Display

Analysis

Spectra

Display

Analysis

Data cubes

Display

Analysis

Herschel Science Archive

Calibration sources

Products and datasets

Observation context

Product Access Layer

Tasks

Quality control

Systematic product generation

Data input-output

Virtual Observatory

Installer

Documentation

HIFI

hifiPipelineTask

HRS Pipeline

WBS Pipeline

Level 1 Pipeline

Level 2 Pipeline

Products

DoDeconvolution

DoGridding

FitHifiFringe

HiClass export tool

Flags

SpectrumExplorer (HIFI specifics)

HIFI DP scripts (User tool scripts)

Documentation

Other

PACS

Pipeline

Photometry

Spectrometry

Calibration

Photometry

Spectrometry

Analysis tools

Photometry

Spectrometry

Product changes

Photometry

Spectrometry

Documentation

Other

SPIRE

Calibration Products

Calibration Framework

Preprocessing

Engineering Conversion

Common Pipeline

Photometer Pipeline

Spectrometer Pipeline

Interactive Analysis and Tools

Documentation

Other

 
This site is powered by the TWiki collaboration platform Powered by Perl