Herschel Data Processing Workshop for Newcomers, 24-27 June 2013



Download the latest HIPE User Release Candidate (HIPE 11.0.0 RC2)

What is HIPE?

HIPE stands for 'Herschel Interactive Processing Environment'. This is the software for browsing, analyzing and reprocessing Herschel data.

Which version of HIPE do I need?

The latest version of HIPE 11 (Release Candidate #2) will be the version of HIPE that will be used during the workshop. This version of HIPE (11.0.0 RC2) can be downloaded from the links below. Please install HIPE on your personal laptop (minimum requirements: RAM 4GB) in advance to the start of the workshop following the installation instructions given below for the different operating systems. Report any problems during the installation process through the HSC Helpdesk.

Important note: You have to keep your internet connection on during the process as you download all the necessary components during the installation.

Note that this is an advanced version of HIPE 11 which is still undergoing the final acceptance tests before it becomes an official user release, something that we anticipate will happen in July.

Release candidates (RCs) are made available for internal use only. Do not distribute this software!.

The available installers for Windows and Linux come with a 32 or a 64 bit virtual machine (VM). The available installer for Mac OS X does not come with a virtual machine but will use the one available at your system.

Download HIPE 11.0.0 RC2

Platform With Java 32 bit VM With Java 64 bit VM Without Java VM
Windows Download HIPE 11.0.0 RC2 Download HIPE 11.0.0 RC2
MacOSX     Download HIPE 11.0.0 RC2
Linux Download HIPE 11.0.0 RC2 Download HIPE 11.0.0 RC2

HIPE Installation Instructions

  • Windows Instructions:
    • After downloading, double-click HIPE_11_0_0_RC2_INSTALLER.exe

  • Linux Instructions:
    • To download the linux installer you should do a right click and then "save link as"
    • After downloading open a terminal and, cd to the directory where you downloaded the installer.
    • At the prompt type: sh ./HIPE_11_0_0_RC2_INSTALLER.bin

  • Mac OS X Instructions:
    • After downloading, double-click HIPE_11_0_0_RC2_INSTALLER

  • In the eventual case of problems with these GUI-based installers, you can always use the command line installer following the instructions below:

    • 2.- unzip the file

    • 3.- and from the command line type >./hcss.installer hcss 11.0 2825, this installs the exact same software as the graphical installer.

    • 4.- Once it is installed, you can run the application by calling hcss-11.0.2825/bin/hipe

Calibration contexts applicable for HIPE v11

In order to process data with HIPE v11 you may also need to download the corresponding calibration tree.

  • SPIRE calibration tree, spire_cal_11_0 is available from here. More details on how to get and install the latest calibration are provided here.

HIPE Users' Documentation

Please have a look at the on-line HIPE 11 documentation clicking on the link below.

Final Agenda

PDF Version Click here

Herschel Data Processing Workshop for Newcomers, 24-27 June 2013

Final Agenda

Monday 24 June 2013


09:15-09:30 CEST: Welcome, local set-up and start of webex session
09:30-09:45 CEST: Welcome to Herschel Data Processing S.Ott
09:45-10:00 CEST: Welcome to the Herschel Science Archive P.García-Lario
10:00-10:30 CEST: An introduction to the HSA User Interface E.Verdugo
10:30-11:00 CEST: An introduction to HIPE B.Merín
11:00-11:30 CEST: Coffee break
11:30-11:45 CEST: Q&A on HSA/HIPE general issues
11:45-12:15 CEST: Data Handling in HIPE E.Verdugo
12:15-13:00 CEST: Quick inspection of Herschel data products B.Merín
13:00-13:30 CEST: Q&A on HSA/HIPE data handling and quick inspection of products

13:30-14:30 CEST: Lunch break

14:30-16:00 CEST: Hands-on session - Part I (HSA/data access/data handling)
16:00-16:30 CEST: Coffee break
16:30-18:00 CEST: Hands-on session - Part II (HIPE general issues/quick inspection of products)

Tuesday 25 June 2013

PACS (@ Room B3)

Spectrometer Session I: Overview of PACS Spectroscopy data and their outlook in HIPE

09:00 Introduction to PACS Spectrometer and observing modes R. Vavrek
09:30 Demo 1: Accessing and inspecting PACS Products in HIPE K. Exter

Spectrometer Session II: Processing data for scientific use

10:30 Calibration and data processing pipeline overview R. Vavrek
11:00 - 11:30 Coffee break
11:30 Demo 2: Interactive reduction of a Line Spectroscopy observation R. Vavrek
12:00 Demo 3: Line fitting and flux extraction for mapping observations K. Exter
12:30 Hands-on session

13:30 - 14:30 Lunch break

14:30 Quick introduction to PACS photometer products and data processing B.Altieri
15:00 PACS photometry pipelines B.Altieri
15:15 PACS photometer calibration (photometric calibration/PSF) Z.Balog
15:30 PACS point source photometry (aperture and PSF photometry, correlated noise and errors) Z.Balog

16:00 - 16:30 Coffee break

16:30 PACS photometry:mapping algorithms for extended emission V.Doublier-Pritchard/B.Altieri
17:00 PACS photometry: ipipes tutorials / Hands-on session B.Altieri

Wednesday 26 June 2013

SPIRE (@ Room B3)

09:00-09:30: Quick introduction to SPIRE photometer products and data processing Luca Conversi
09:30-11:00: SPIRE Photometry Tutorials Luca Conversi
11:00-11:30: Coffee break  
11:30-12:00: Q&A on SPIRE Photometry Luca Conversi
12:00-12:30: Quick introduction to SPIRE FTS products and data processing Ivan Valtchanov
12:30-13:10: SPIRE Spectroscopy Tutorial Ivan Valtchanov
13:10-13:30: Q&A on SPIRE Spectroscopy Ivan Valtchanov
13:30-14:30: Lunch break  
14:30-15:00 Quick introduction to photometry with SPIRE Luca Conversi
15:00-16:00 Hands-on session - Part I (SPIRE Photometry) Luca Conversi
16:00-16:30 Coffee break  
16:30-18:00 Hands-on session - Part II (SPIRE Spectroscopy) Ivan Valtchanov

Thursday 27 June 2013

HIFI (@ Room B3)

09:00-10:00: HIFI observing modes and associated data processing products Anthony Marston
10:00-10:30: Running HIFI pipelines - general and alternatives plus data export Tutorial 1 Anthony Marston
10:30-11:00: Q&A on HIFI Tutorial 1
11:00-11:30: Coffee break
11:30-12:15 Remaining issues: Handling HIFI electrical and optical standing waves, tutorial 2 Elena Puga
12:15-13:00 Basic HIFI data analysis, Tutorial 3 Anthony Marston
13:00-13:30: Q&A on HIFI Tutorials 2 and 3

13:30-14:30: Lunch break

14:30-16:00 Hands-on session - Part I (HIFI), point source modes
16:00-16:30 Coffee break
16:30-18:00 Hands-on session - Part II (HIFI), maps and spectral scans

Sample Datasets

These are some OBSIDs that you can use for your tests with the sysyem. Enjoy Herschel data! The PACS Range Spectroscopy ones will be used during the demos tomorrow.

PACS Phot: 1342215720 and 1342215721 GX 339-4 Point-like source
PACS Phot: 1342185575 and 1342185576 Sh104 Extended source
PACS Phot: 1342185553 and 1342185554 RCW120 Extended source - but requires more memory!
SPIRE Phot: 1342183678 RCW 120  
HIFI Single Point: 1342242856 W28A  
HIFI Single Point: 1342205520 Sgr A 50" offset used in demo
HIFI Map: 1342190850 Orion Bar small map used in demo
HIFI Spectral Scan: 1342191601 Orion KL band 4a used in demo
PACS Line Spec: 1342182002 NGC 5315  
PACS Range Spec: 1342231295 NGC 2440 will be used in demo
PACS Range Spec: 1342197795 R Dor will be used in demo
PACS Line Spec: 1342191294 NGC1365 will be used in demo
SPIRE Spec: 1342256105 IRC+10216 Nice spectrum
  1342214831 NGC6240 will be used in demo
  1342214832 dark sky for NGC6240 will be used in demo
  1342203586 SPIRE Photo map of NGC6240 will be used in demo
  1342265845 SPIRE Spectral map of Orion Bar (HD37041) will be used in hands-on
SPIRE PACS Parallel: 1342186121, 1342186122 Rosette Nebula Nice image

Tutorials & additional workshop material

PACS Spectroscopy tutorials

PACS Photometry tutorials

SPIRE Photometry tutorials

(NOTE: you may need to remove the .txt from the filenames associated to the python scripts above so that these files can be recognized as python scripts by HIPE - the .txt suffix is added automatically by the twiki for security reasons)

HIFI tutorials


PACS Spectroscopy

Q1: I have a rather specific question. I am reducing unchopped line scans. I use oversample=2 and I get a warning for the specwaveRebinTask (not enough good values etc, try oversample=1). However when I look at the final result (end interactive L2) there doesn't seem to be a problem, for instance in the plotCubesStdev display there are only low exposure pixels at the edges. Is this something to worry about? I hope I managed to explain this clearly enough....

A1: Hi xxx, in many pipeline tasks you get warning messages (actually, many are just information, but the developers seem to like to scare people with WARNING!! comments ;-). so at first glance your message seems to be just a kind warning. if the end result looks ok then you don't need to worry. there is a task at the end of the pipeline that plots little yellow? squares on the bins that are "low exposure". If these are all at the end of the spectral range, you don't need to worry. In fact, yes, the task plotStdDev is the one that tells you which bins are low exposure. so it looks like you do not need to worry. but you can also change the wavelength grid up/over sample values and overplot the different end results to compare. More about up and over sampling is in the PDRG chapter…5 or 6 or 7, I forgot, and more about comparing spectra at the end of the pipeline is also in the PDRG (chap 5 or 6 or 7 also - a section that talks about flatfielding), or use the spectrum explorer.

Q2: can I use data cubes that we get from the archive for spectroscopic AND photometric studies?

A2: well, if you have enough continuum in your cube then you can of course use it for photometry. the question is "what is enough". The thing to remember is that the continuum you get at the end of the pipeline is subject to ± error. This error depends on the strength of your source with respect to the teelscope. The telescope spectrum is some 100 Jy or so, and so even 1% error is a ± error of 1 Jy in the continuum. In the calibration document that is on the pacs calibration and instrument wiki the errors for the end cubes are given. ± a few Jy you should expect will be the error in the continuum level. http://herschel.esac.esa.int/twiki/pub/Public/PacsCalibrationWeb/PacsSpectroscopyPerformanceAndCalibration_v2_4.pdf For full science quality you need to re-pipeline the data - take advantage of the latest calibrations as the HSA products don't run all the pipeline tasks. This is documented in chp 1 of the PDRG. You can get fluxes (Jy/spaxel) and wavelengths (microns). Though the fits files do have a rather awkward format. They can be hard to read directly into other software - we are working on this and if you have problems it is fastest to raise a helpdesk ticket. One last thing: for unchopped mode, the flux level of the continuum is not guaranteed to be correct. That mode was never intended for continuum studies.

Q3: where can I find the sky background contamination?

A3: If the question means: where can I find the spectra of the sky (+telescope) then for chop-nod observations these are the spectra taken in the off-chop positions. The easiest way to see these data is to run the script Split On-Off, which you find in the HIPE Pipelines menu (Spectroscopy->both chopped menus -> Split On-Off. This is also explained in the pipeline chapters (chp 3) of the PACS Data Reduction Guide. This script will produce separate cubes of the on-source spectra and the off-source spectra.

Q4: can you subtract a baseline to spectra? Could it be included in the script?

A4: The currently best documented way is to fit the continuum with a poly and then get the residual (data - fit). This is explained in chapter 7 of the "Data Analysis Guide" (about the SpectrumFitter and SpectrumFitterGUI), which you can get via the HIPE Help->Help contents menu. The baseline fitter is a PACS specific task that you can also use but it is not well documented. This takes the data and identifies where the baseline is, i.e. it ignores lines, and then that baseline can be subtracted from the spectrum. If someone wants to use this task, or know more details, then please raise a helpdesk ticket to ask specifically for that.

Q5: is the split on-off applicable to unchopped observations?

A5: For the unchopped line scan mode, the off-source spectra are separated out from the on-source spectra within the pipeline (it will be very clear when this happens as you follow the interactive pipeline script Spectroscopy->unchopped line scan->"lineScan") and you can then compare them by plotting them together (this is also explained in the PDRG pipeline chap 4 and some additional information in chp 8). For the unchopped range scan, on and off (ie source and sky) are separate observations and once you have reduced each through to Level 2, you can compare them e.g. as cubes in the Spectrum Explorer.

PACS Photometry

Q1: Although I missed some of talks, I have a simple question. For a point source like a protostar on a bright background, how do you subtract the background using aperture photometry?

A1: challenging !! wink I would say it's challenging at any wavelength (photometry on bright extended background) You could try getSources : http://www.herschel.fr/cea/gouldbelt/en/getsources/

Q2: Another question, I don't know what the color correction should be. In principle, do we need to apply a color correction for the aperture photometry of a protostar?

A2: color correction factors for various type of SED are given in : "PACS Photometer Passbands and Colour Correction Factors for various source SEDs (2.5 Mb), PICC-ME-TN-038, version 1.0, 12 April 2011"

Q3: The PACS photometry tutorials have been extremely helpful for me. However, so far they have dealt with point sources only and my targets of interest are not point sources. Can an expert comment briefly on extended source photometry? For example, how someone would create 24-500 um SED within circular apertures (say, 20" diameter) for a spiral arm?

A3: Indeed photometry is primarily focussed toward point-sources. For slightly extended sources the aperture correction factors are not easy to compute, actually one would need to compute it depending on the shape and the extend of the source. For very large sources - hence very large apertures, say, several arcmin) -, the aperture correction will be very close to 1, hence almost negligible. But there is a regime in between where it's more complex. However you mention a circular aperture of 20" which is rather small, so your source must be rather compact I would say that the point-source EEF could be used, though slightly underestimating the correction.

Q4: Hello, I do have a question on making a map of a region with madmap, if posssible. the map I produce looks not optimal to me, but I would like to know if this is true and if so, how can i optimize it. the obsid is 1342216409(10,11,12) and I run the madmap pipeline on all and the final output looks very similar to what is in the archive. I am trying to run scanamorphous but running out of memory. any hint will be appreciated. thanks.

A4: you could try photExtractRaDecFrames() for that purpose. it allows to extract frames around a position (ra, dec fieldSize). However madmap requires continuous timelines as much as possible, so it might lead to suboptimal maps because the drift correction will get confused by the signal jumps. Best would be to run madmap on the 4 obsids, I am surprised scanamorphous went out of memory, did you try JyPScanam in Hipe 11? it works rather nicely.

Q5: Why can't one simply deconvolve PACS images the way radio astronomers do using algorithms like CLEAN? If the PSF is well known, stable, and of high-S/N then one should be able to "squeeze" the flux from the wings back into the PSF core, thus making need for aperture corrections moot, and making ordinary photometry easier. Since there are no CLEAN tasks in HIPE, there must be some reason why this can't be done. Just wondering...

A5: there are deconvolution attempts on PACS (and SPIRE) that work relatively nicely. But bringing back the PSF wings - that extend up to several arcmin - into the PSF core seems utopic to me. But I must confess I am not a CLEAN expert. One additional limitation is that there is not a single unique PSF for PACS, the PSF depends on the observation parameters (scan direction and speed) and data processing (map-making) used. Hope it helps.

SPIRE photometry

Q1: How much memory is needed for the reduction of a large SPIRE field (e.g. like a 2x2 degree Hi-GAL tile)?

A1: 8-10 GB is usually more than enough to reduce a large map like this. Small scan maps can be reduced without any problem with a 4GB laptop.

Q2: Are there any limitations in the reliability of the scripts used for determining the Planck offsets to be applied to SPIRE maps? In particular, I am interested in whether they can be applied to very small maps (since the corrections are based on the average absolute fluxes covered by the much bigger Planck maps)?

A2: In principle not, but it could be convenient to look at the special characteristics of the emission observed in your small field. If you have a particular sky field dominated by galactic cirrus emission it could be an issue. Raise a helpdesk ticket indicating the OBSID of the map you want to analyse and we will tell you whether we see any problem with applying the offset derived from Planck data.

SPIRE spectroscopy

Q1: Should the semi-extended correction be applied after all other steps (for instance improve background etc)?

A1: semi-extended correction should be applied as a last step after background subtraction because the galactic emission could make the source look as semi-extended

Q2: How do I find the appropriate dark observation to be used in the processing of a SPIRE FTS observation?

A2: You should look at the dark observations taken in the same operational day (OD) when your science observation was taken, or the nearest in time if there is none available on that particular OD. The OD number you can get it from the Observing Log or from the Archive (HSA). Once you know the OD, you query in HSA for all SPIRE FTS observations taken on that OD or a small range of ODs around the date when the observation was taken. There is also a list with all available dark observations taken (in the standard observing mode) throughout the mission at : http://herschel.esac.esa.int/twiki/bin/view/Public/SpireDailyDarkObservations


Q1: Do you get automatic notifications when there is any new version of the HIFI calibration files/tree?

A1: No, it's not like for PACS that you get a notification whenever a new calibration tree is available. Usually a new version of HIFI calibration comes always (and only) with every new major version of HIPE, but not more frequently.

Q2: Is there a list of known spurs available?

A2: Yes, there is a catalogue that you can inspect coming as part of the calibration files delivered through the HSA where a detailed list of known spurs is provided, with indication of the frequency at which they are seen and their main characteristics. These frequencies are flagged/masked in the pipeline so that you do not need to care about them.

Topic attachments
I Attachment History Action Size Date Who Comment
PDFpdf DP_workshop_PACS_Products.pdf r1 manage 15086.8 K 2013-06-25 - 14:22 PedroGarciaLario  
PDFpdf EP_StdWvRemoval_DPWS2013.pdf r1 manage 1495.0 K 2013-06-27 - 12:57 ElenaPuga Presentation and demos description, proposal for hands-on session
PDFpdf FTS_demo.pdf r1 manage 43.8 K 2013-06-26 - 07:16 IvanV  
PDFpdf FTS_for_newcomers.pdf r1 manage 8308.1 K 2013-06-26 - 07:11 IvanV FTS for newcomers
PowerPointppt HIFI_DP_tutorial2.ppt r1 manage 647.5 K 2013-06-26 - 19:45 AnthonyMarston HIFI tutorial 3 - APM
PowerPointppt HIFI_handson_exercises.ppt r1 manage 810.0 K 2013-06-26 - 19:36 AnthonyMarston HIFI hands on - APM
PowerPointppt HIFIpipeline-tutorial_1.ppt r1 manage 219.0 K 2013-06-26 - 19:34 AnthonyMarston HIFI tutorial 1 - APM
PDFpdf Herschel_DP_Welcome_Ott.pdf r1 manage 2635.0 K 2013-06-24 - 22:33 PedroGarciaLario  
PDFpdf Herschel_PGL_Newcomers.pdf r1 manage 12289.2 K 2013-06-24 - 23:34 PedroGarciaLario  
Texttxt MapMerged_TAR.py.txt r1 manage 5.6 K 2013-06-25 - 15:29 LucaConversi  
PDFpdf PACSspec_demo_kexter_v2.pdf r1 manage 1670.3 K 2013-06-25 - 11:10 PedroGarciaLario  
Texttxt PACSspec_demo_kexter_v2.py.txt r1 manage 11.6 K 2013-06-25 - 11:27 PedroGarciaLario  
Texttxt PSF_comp.py.txt r1 manage 8.6 K 2013-06-25 - 21:59 PedroGarciaLario  
PowerPointppt Paris_School_HIFI_Instrument_and_Data.ppt r1 manage 6127.0 K 2013-06-26 - 19:34 AnthonyMarston HIFI introduction - APM
Texttxt Remove_CoolerBurp.py.txt r1 manage 14.8 K 2013-06-25 - 15:29 LucaConversi  
Texttxt Remove_Glitches.py.txt r1 manage 11.8 K 2013-06-25 - 15:29 LucaConversi  
PDFpdf SPIRE_Photometry.pdf r1 manage 3380.8 K 2013-06-25 - 15:58 LucaConversi  
Texttxt SPIRE_Photometry.py.txt r1 manage 5.3 K 2013-06-26 - 11:27 LucaConversi  
PDFpdf SPIRE_Pipeline.pdf r1 manage 9296.6 K 2013-06-25 - 15:58 LucaConversi  
Unknown file formatfits baseline_avg_table_WBS-V_1342190806_2012-09-30T13_21_30.fits r1 manage 5914.7 K 2013-06-27 - 08:27 PedroGarciaLario  
PDFpdf pacsSpectroInstrument.pdf r1 manage 9612.2 K 2013-06-25 - 13:15 PedroGarciaLario  
PDFpdf pacsSpectroPipelines.pdf r1 manage 1004.7 K 2013-06-25 - 13:15 PedroGarciaLario  
Edit | Attach | Watch | Print version | History: r46 < r45 < r44 < r43 < r42 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r46 - 2014-06-22 - PedroGarciaLario
This site is powered by the TWiki collaboration platform Powered by Perl