Herschel On-line Data Processing Workshop for Newcomers 2014, 24-27 June 2014



   * TBD  

Download the latest HIPE User Release (HIPE 12.0)

What is HIPE?

HIPE stands for 'Herschel Interactive Processing Environment'. This is the software for browsing, analyzing and reprocessing Herschel data.

Which version of HIPE do I need?

The latest version of HIPE 12 will be the version of HIPE that will be used during the workshop. This version of HIPE 12 can be downloaded from the links below.  Please install HIPE on your personal laptop (minimum requirements: RAM 4GB) in advance to the start of the workshop  following the installation instructions given below for the different operating systems. Report any problems during the installation process through the HSC Helpdesk

Important note: You have to keep your internet connection on during the process as you download all the necessary components during the installation. 

Download HIPE 12.0

HIPE Users' Documentation

You can have a look at the on-line HIPE 12 documentation clicking on the link below.

Final Agenda

| PDF Version | Click here


Herschel On-line Data Processing Workshop for Newcomers 2014, 24-27 June 2014

Final Agenda

Tuesday 24 June 2014 -- General issues

Add to calendar
10:00-10:30: Welcome to Herschel P.García-Lario
10:30-11:00: An introduction to the Herschel Science Archive and its products E.Verdugo
11:00-11:30: An introduction to HIPE B.Merín
11:30-12:00: Coffee/Tea break
12:00-12:30: Analysis of Herschel data in HIPE B.Merín
12:30-13:00: Questions & Answers on the Herschel Science Archive, products and HIPE All

Wednesday 25 June 2014 -- PACS

Add to calendar
10:00-10:30: Introduction to PACS instrument and data K. Exter
10:30-11:00: PACS photometer products and data processing L. Calzoletti
11:00-11:30: PACS photometry data analysis L. Calzoletti
12:30-12:00: Coffee/Tea break
12:00-12:30: PACS spectrometer products and data processing K. Exter
12:30-13:00: PACS spectrometer data analysis K. Exter

Thursday 26 June 2014 -- HIFI

Add to calendar
10:00-10:45: Introduction to HIFI instrument and data D. Teyssier
10:45-11:30: Specifics and post-processing of HIFI pointed observations D. Teyssier
12:30-12:00: Coffee break
12:00-12:30: Specifics and post-processing of HIFI spectral scan observations D. Teyssier
12:30-13:00: Specifics and post-processing of HIFI mapping observations I. Avruch

Friday 27 June 2014 -- SPIRE

Add to calendar
10:00-10:30: Introduction to SPIRE instrument and data T. Lim
10:30-11:00: SPIRE photometer products and data processing T. Lim
11:00-11:30: SPIRE photometry data analysis T. Lim
12:30-12:00: Coffee/Teabreak
12:00-12:30: SPIRE spectrometer products and data processing E. Polehampton
12:30-13:00: SPIRE spectrometer data analysis E. Polehampton


Sample Datasets (from previous workshop)

  • Do we need to keep this??

These are some OBSIDs that you can use for your tests with the sysyem. Enjoy Herschel data! The PACS Range Spectroscopy ones will be used during the demos tomorrow.

PACS Phot: 1342215720 and 1342215721 GX 339-4 Point-like source
PACS Phot: 1342185575 and 1342185576 Sh104 Extended source
PACS Phot: 1342185553 and 1342185554 RCW120 Extended source - but requires more memory!
SPIRE Phot: 1342183678 RCW 120  
HIFI Single Point: 1342242856 W28A  
HIFI Single Point: 1342205520 Sgr A 50" offset used in demo
HIFI Map: 1342190850 Orion Bar small map used in demo
HIFI Spectral Scan: 1342191601 Orion KL band 4a used in demo
PACS Line Spec: 1342182002 NGC 5315  
PACS Range Spec: 1342231295 NGC 2440 will be used in demo
PACS Range Spec: 1342197795 R Dor will be used in demo
PACS Line Spec: 1342191294 NGC1365 will be used in demo
SPIRE Spec: 1342256105 IRC+10216 Nice spectrum
  1342214831 NGC6240 will be used in demo
  1342214832 dark sky for NGC6240 will be used in demo
  1342203586 SPIRE Photo map of NGC6240 will be used in demo
  1342265845 SPIRE Spectral map of Orion Bar (HD37041) will be used in hands-on
SPIRE PACS Parallel: 1342186121, 1342186122 Rosette Nebula Nice image

Tutorials & additional workshop  material (from previous workshop)

PACS Spectroscopy tutorials

   * PACSspec_demo_kexter_v2.pdf: PACSspec_demo_kexter_v2.pdf

   * PACSspec_demo_kexter_v2.py.txt: PACSspec_demo_kexter_v2.py.txt 

PACS Photometry tutorials

   * PSF_comp.py.txt: PSF subtraction script by Zoltan Balog 

SPIRE Photometry tutorials

   * Remove_CoolerBurp.py.txt: tutorial to remove booler burp from affected observations

   * MapMerged_TAR.py.txt: tutorial to merge 2 (or more) observations and calculate the absolute offset from cross-calibration with Planck.        * The script assumes you have the 2 parallel mode observations of Rosette Nebula (1342186121, 1342186122) copied on your hard drive.       * You also need to copy the following files in the same directory: DX9_map_545_smooth_8arcmin.fits, DX9_map_857_smooth_8arcmin.fits and SpireHfiColourCorrTab_v1.1.fits

   * Remove_Glitches.py.txt: tutorial to find and remove undetected glitched from maps

   * SPIRE_Photometry.py.txt: tutorial running different photometry algorithms on the calibration star Gamma Dra

    (NOTE: you  may need to remove the .txt from the filenames associated to the python scripts above so that these files can be recognized as python scripts by HIPE - the .txt suffix is added automatically by the twiki for security reasons)

HIFI tutorials

   * demoDPWS2013_fitHifiFringe.py.txt: HIFI Optical Standing Wave demo

   * demoDPWS20_HEBStWvCatalogCorrection.py.txt: HIFI HEB Electrical Standing Wave demo including the HEB IF Matching Technique script v0.53

   * baseline_avg_table_WBS-V_1342190806_2012-09-30T13_21_30.fits: Template Catalog band 7 WBS-V for HEB IF Matching Technique

Q&A (from previous workshop)

PACS Spectroscopy

Q1: I have a rather specific question. I am reducing unchopped line scans. I use oversample=2 and I get a warning for the specwaveRebinTask (not enough good values etc, try oversample=1). However when I look at the final result (end interactive L2) there doesn't seem to be a problem, for instance in the plotCubesStdev display there are only low exposure pixels at the edges. Is this something to worry about? I hope I managed to explain this clearly enough.... 

A1: Hi xxx, in many pipeline tasks you get warning messages (actually, many are just information, but the developers seem to like to scare people with WARNING!! comments  ;-). so at first glance your message seems to be just a kind warning. if the end result looks ok then you don't need to worry. there is a task at the end of the pipeline that plots little yellow? squares on the bins that are "low exposure". If these are all at the end of the spectral range, you don't need to worry. In fact, yes, the task plotStdDev is the one that tells you which bins are low exposure. so it looks like you do not need to worry. but you can also change the wavelength grid up/over sample values and overplot the different end results to compare. More about up and over sampling is in the PDRG chapter…5 or 6 or 7, I forgot, and more about comparing spectra at the end of the pipeline is also in the PDRG (chap 5 or 6 or 7 also - a section that talks about flatfielding), or use the spectrum explorer. 

Q2: can I use data cubes that we get from the archive for spectroscopic AND photometric studies? 

A2: well, if you have enough continuum in your cube then you can of course use it for photometry. the question is "what is enough". The thing to remember is that the continuum you get at the end of the pipeline is subject to ± error. This error depends on the strength of your source with respect to the teelscope. The telescope spectrum is some 100 Jy or so, and so even 1% error is a ± error of 1 Jy in the continuum. In the calibration document that is on the pacs calibration and instrument wiki the errors for the end cubes are given. ± a few Jy you should expect will be the error in the continuum level. http://herschel.esac.esa.int/twiki/pub/Public/PacsCalibrationWeb/PacsSpectroscopyPerformanceAndCalibration_v2_4.pdf  For full science quality you need to re-pipeline the data - take advantage of the latest calibrations as the HSA products don't run all the pipeline tasks. This is documented in chp 1 of the PDRG.  You can get fluxes (Jy/spaxel) and wavelengths (microns). Though the fits files do have a rather awkward format. They can be hard to read directly into other software - we are working on this and if you have problems it is fastest to raise a helpdesk ticket. One last thing: for unchopped mode, the flux level of the continuum is not guaranteed to be correct. That mode was never intended for continuum studies.

Q3: where can I find the sky background contamination?                       

A3: If the question means: where can I find the spectra of the sky (+telescope) then for chop-nod observations these are the spectra taken in the off-chop positions. The easiest way to see these data is to run the script Split On-Off, which you find in the HIPE Pipelines menu (Spectroscopy->both chopped menus -> Split On-Off. This is also explained in the pipeline chapters (chp 3) of the PACS Data Reduction Guide. This script will produce separate cubes of the on-source spectra  and the off-source spectra.

Q4: can you subtract a baseline to spectra? Could it be included in the script?        

A4: The currently best documented way is to fit the continuum with a poly and then get the residual (data - fit). This is explained in chapter 7 of the "Data Analysis Guide" (about the SpectrumFitter and SpectrumFitterGUI), which you can get via the HIPE Help->Help contents menu. The baseline fitter is a PACS specific task that you can also use but it is not well documented. This takes the data and identifies where the baseline is, i.e. it ignores lines, and then that baseline can be subtracted from the spectrum. If someone wants to use this task, or know more details, then please raise a helpdesk ticket to ask specifically for that.

Q5: is the split on-off applicable to unchopped observations?   

A5: For the unchopped line scan mode, the off-source spectra are separated out from the on-source spectra within the pipeline (it will be very clear when this happens as you follow the interactive pipeline script Spectroscopy->unchopped line scan->"lineScan") and you can then compare them by plotting them together (this is also explained in the PDRG pipeline chap 4 and some additional information in chp 8). For the unchopped range scan, on and off (ie source and sky) are separate observations and once you have reduced each through to Level 2, you can compare them e.g. as cubes in the Spectrum Explorer.

PACS Photometry

Q1: Although I missed some of talks, I have a simple question. For a point source like a protostar on a bright background, how do you subtract the background using aperture photometry?

A1: challenging !! wink I would say it's challenging at any wavelength (photometry on bright extended background) You could try getSources : http://www.herschel.fr/cea/gouldbelt/en/getsources/

Q2: Another question, I don't know what the color correction should be. In principle, do we need to apply a color correction for the aperture photometry of a protostar?

A2: color correction factors for various type of SED are given in :     "PACS Photometer Passbands and Colour Correction Factors for various source SEDs (2.5 Mb), PICC-ME-TN-038, version 1.0, 12 April 2011

Q3: The PACS photometry tutorials have been extremely helpful for me. However, so far they have dealt with point sources only and my targets of interest are not point sources. Can an expert comment briefly on extended source photometry? For example, how someone would create 24-500 um SED within circular apertures (say, 20" diameter) for a spiral arm?

A3: Indeed photometry is primarily focussed toward point-sources. For slightly extended sources the aperture correction factors are not easy to compute, actually one would need to compute it depending on the shape and the extend of the source. For very large sources - hence very large apertures, say, several arcmin) -,  the aperture correction will be very close to 1, hence almost negligible. But there is a regime in between where it's more complex. However you mention a circular aperture of 20" which is rather small, so your source must be rather compact I would say that the point-source EEF could be used, though slightly underestimating the correction.

Q4: Hello, I do have a question on making a map of a region with madmap, if posssible. the map I produce looks not optimal to me, but I would like to know if this is true and if so, how can i optimize it. the obsid is 1342216409(10,11,12) and I run the madmap pipeline on all and the final output looks very similar to what is in the archive. I am trying to run scanamorphous but running out of memory. any hint will be appreciated. thanks.

A4: you could try photExtractRaDecFrames() for that purpose. it allows to extract frames around a position (ra, dec fieldSize). However madmap requires continuous timelines as much as possible, so it might lead to suboptimal maps because the drift correction will get confused by the signal jumps. Best would be to run madmap on the 4 obsids, I am surprised scanamorphous went out of memory, did you try JyPScanam in Hipe 11? it works rather nicely.

Q5: Why can't one simply deconvolve PACS images the way radio astronomers do using algorithms like CLEAN? If the PSF is well known, stable, and of high-S/N then one should be able to "squeeze" the flux from the wings back into the PSF core, thus making need for aperture corrections moot, and making ordinary photometry easier. Since there are no CLEAN tasks in HIPE, there must be some reason why this can't be done. Just wondering...

A5: there are deconvolution attempts on PACS (and SPIRE) that work relatively nicely. But bringing back the PSF wings - that extend up to several arcmin - into the PSF core seems utopic to me. But I must confess I am not a CLEAN expert. One additional limitation is that there is not a single unique PSF for PACS, the PSF depends on the observation parameters (scan direction and speed) and data processing (map-making) used. Hope it helps.

SPIRE photometry

Q1: How much memory is needed for the reduction of a large SPIRE field (e.g.  like a 2x2 degree Hi-GAL tile)?

A1: 8-10 GB is usually more than enough to reduce a large map like this. Small scan maps can be reduced without any problem with a 4GB  laptop.

Q2: Are there any limitations in the reliability of the scripts used for determining the Planck offsets to be applied to SPIRE maps? In particular, I am interested in whether they can be applied to  very small maps (since the corrections are based on the average absolute fluxes covered by the much bigger Planck maps)? 

A2: In principle not, but it could be convenient to look at the special characteristics of the emission observed in your small field. If you have a particular sky field dominated by galactic cirrus emission it could be an issue. Raise a helpdesk ticket indicating the OBSID of the map you want to analyse and we will tell you whether we see any problem with applying the offset derived from Planck data.

SPIRE spectroscopy

Q1: Should the semi-extended correction be applied after all other steps (for instance improve background etc)?

A1: semi-extended correction should be applied as a last step after background subtraction because the galactic emission could make the source look as semi-extended

Q2: How do I find the appropriate dark observation to be used in the processing of a SPIRE FTS observation?

A2: You should look at the dark observations taken in the same operational day (OD) when your science observation was taken, or the nearest in time if there is none available on that particular OD. The OD number you can get it from the Observing Log or from the Archive (HSA). Once you know the OD, you query in HSA for all SPIRE FTS observations taken on that OD or a small range of ODs around the date when the observation was taken. There is also a list with all available dark observations taken (in the standard observing mode) throughout the mission at : http://herschel.esac.esa.int/twiki/bin/view/Public/SpireDailyDarkObservations


Q1: Do you get automatic notifications when there is any new version of the HIFI calibration files/tree?

A1: No, it's not like for PACS that you get a notification whenever a new calibration tree is available. Usually a new version of HIFI calibration comes always (and only) with every new major version of HIPE, but not more frequently.

Q2:  Is there a list of known spurs available?

A2: Yes, there is a catalogue that you can inspect coming as part of the calibration files delivered through the HSA where a detailed list of known spurs is provided, with indication of the frequency at which they are seen and their main characteristics. These frequencies are flagged/masked in the pipeline so that you do not need to care about them.

-- BrunoMerin - 29 May 2014

Topic attachments
I Attachment History Action Size Date Who Comment
Unknown file formatics 249567877508062647001.ics r1 manage 3.4 K 2014-06-02 - 15:43 AlvarGarcia WebEx Meeting 24th of June
Unknown file formatics 249568332508062647001.ics r1 manage 3.4 K 2014-06-02 - 15:44 AlvarGarcia WebEx Meeting 25th of June
Unknown file formatics 249568507508062647001.ics r1 manage 3.4 K 2014-06-02 - 15:44 AlvarGarcia WebEx Meeting 27th of June
Unknown file formatics 249568527508062647001.ics r1 manage 3.4 K 2014-06-02 - 15:44 AlvarGarcia WebEx Meeting 26th of June
Edit | Attach | Watch | Print version | History: r37 | r8 < r7 < r6 < r5 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r6 - 2014-06-04 - DavidTeyssier
This site is powered by the TWiki collaboration platform Powered by Perl