Herschel Data Processing Workshop for Newcomers, 24-27 June 2013
TABLE OF CONTENTS
Participants
Download the latest HIPE User Release Candidate (HIPE 11.0.0 RC2)
What is HIPE?
HIPE stands for 'Herschel Interactive Processing Environment'. This is the software for browsing, analyzing and reprocessing Herschel data.
Which version of HIPE do I need?
The latest version of HIPE 11 (Release Candidate #2) will be the version of HIPE that will be used during the workshop. This version of HIPE (11.0.0 RC2) can be downloaded from the links below. Please install HIPE on your personal laptop (minimum requirements: RAM 4GB) in advance to the start of the workshop following the installation instructions given below for the different operating systems. Report any problems during the installation process through the
HSC Helpdesk.
Important note: You have to keep your internet connection on during the process as you download all the necessary components during the installation.
Note that this is an advanced version of HIPE 11 which is still undergoing the final acceptance tests before it becomes an official user release, something that we anticipate will happen in July.
Release candidates (RCs) are made available for internal use only. Do not distribute this software!.
The available installers for Windows and Linux come with a 32 or a 64 bit virtual machine (VM). The available installer for Mac OS X does not come with a virtual machine but will use the one available at your system.
Download HIPE 11.0.0 RC2
HIPE Installation Instructions
- Windows Instructions:
- After downloading, double-click HIPE_11_0_0_RC2_INSTALLER.exe
- Linux Instructions:
- To download the linux installer you should do a right click and then "save link as"
- After downloading open a terminal and, cd to the directory where you downloaded the installer.
- At the prompt type: sh ./HIPE_11_0_0_RC2_INSTALLER.bin
- Mac OS X Instructions:
- After downloading, double-click HIPE_11_0_0_RC2_INSTALLER
- In the eventual case of problems with these GUI-based installers, you can always use the command line installer following the instructions below:
-
- 3.- and from the command line type >./hcss.installer hcss 11.0 2825, this installs the exact same software as the graphical installer.
-
- 4.- Once it is installed, you can run the application by calling hcss-11.0.2825/bin/hipe
Calibration contexts applicable for HIPE v11
In order to process data with HIPE v11 you may also need to download the corresponding calibration tree.
- SPIRE calibration tree,
spire_cal_11_0 is available from here
. More details on how to get and install the latest calibration are provided here.
HIPE Users' Documentation
Please have a look at the on-line HIPE 11 documentation clicking on the link below.
Final Agenda
Herschel Data Processing Workshop for Newcomers, 24-27 June 2013
Final Agenda
Monday 24 June 2013
GENERAL ISSUES (@ Room B3)
| 09:15-09:30 CEST: |
Welcome, local set-up and start of webex session |
| 09:30-09:45 CEST: |
Welcome to Herschel Data Processing |
S.Ott |
| 09:45-10:00 CEST: |
Welcome to the Herschel Science Archive |
P.García-Lario |
| 10:00-10:30 CEST: |
An introduction to the HSA User Interface |
E.Verdugo |
| 10:30-11:00 CEST: |
An introduction to HIPE |
B.Merín |
| 11:00-11:30 CEST: |
Coffee break |
| 11:30-11:45 CEST: |
Q&A on HSA/HIPE general issues |
| 11:45-12:15 CEST: |
Data Handling in HIPE |
E.Verdugo |
| 12:15-13:00 CEST: |
Quick inspection of Herschel data products |
B.Merín |
| 13:00-13:30 CEST: |
Q&A on HSA/HIPE data handling and quick inspection of products |
| 13:30-14:30 CEST: |
Lunch break |
| 14:30-16:00 CEST: |
Hands-on session - Part I (HSA/data access/data handling) |
| 16:00-16:30 CEST: |
Coffee break |
| 16:30-18:00 CEST: |
Hands-on session - Part II (HIPE general issues/quick inspection of products) |
Tuesday 25 June 2013
PACS (@ Room B3)
Spectrometer Session I: Overview of PACS Spectroscopy data and their outlook in HIPE
Spectrometer Session II: Processing data for scientific use
| 10:30 |
Calibration and data processing pipeline overview |
R. Vavrek |
| 11:00 - 11:30 |
Coffee break |
| 11:30 |
Demo 2: Interactive reduction of a Line Spectroscopy observation |
R. Vavrek |
| 12:00 |
Demo 3: Line fitting and flux extraction for mapping observations |
K. Exter |
| 12:30 |
Hands-on session |
| 13:30 - 14:30 |
Lunch break |
| 16:00 - 16:30 |
Coffee break |
| 16:30 |
PACS photometry:mapping algorithms for extended emission |
V.Doublier-Pritchard/B.Altieri |
| 17:00 |
PACS photometry: ipipes tutorials / Hands-on session |
B.Altieri |
Wednesday 26 June 2013
SPIRE (@ Room B3)
Thursday 27 June 2013
HIFI (@ Room B3)
Sample Datasets
These are some OBSIDs that you can use for your tests with the sysyem. Enjoy Herschel data! The PACS Range Spectroscopy ones will be used during the demos tomorrow.
| PACS Phot: |
1342215720 and 1342215721 |
GX 339-4 |
Point-like source |
| PACS Phot: |
1342185575 and 1342185576 |
Sh104 |
Extended source |
| PACS Phot: |
1342185553 and 1342185554 |
RCW120 |
Extended source - but requires more memory! |
| SPIRE Phot: |
1342183678 |
RCW 120 |
|
| HIFI Single Point: |
1342242856 |
W28A |
|
| HIFI Single Point: |
1342205520 |
Sgr A 50" offset |
used in demo |
| HIFI Map: |
1342190850 |
Orion Bar small map |
used in demo |
| HIFI Spectral Scan: |
1342191601 |
Orion KL band 4a |
used in demo |
| PACS Line Spec: |
1342182002 |
NGC 5315 |
|
| PACS Range Spec: |
1342231295 |
NGC 2440 |
will be used in demo |
| PACS Range Spec: |
1342197795 |
R Dor |
will be used in demo |
| PACS Line Spec: |
1342191294 |
NGC1365 |
will be used in demo |
| SPIRE Spec: |
1342256105 |
IRC+10216 |
Nice spectrum |
| |
1342214831 |
NGC6240 |
will be used in demo |
| |
1342214832 |
dark sky for NGC6240 |
will be used in demo |
| |
1342203586 |
SPIRE Photo map of NGC6240 |
will be used in demo |
| |
1342265845 |
SPIRE Spectral map of Orion Bar (HD37041) |
will be used in hands-on |
| SPIRE PACS Parallel: |
1342186121, 1342186122 |
Rosette Nebula |
Nice image |
Tutorials & additional workshop material
PACS Spectroscopy tutorials
PACS Photometry tutorials
SPIRE Photometry tutorials
- MapMerged_TAR.py.txt: tutorial to merge 2 (or more) observations and calculate the absolute offset from cross-calibration with Planck.
(NOTE: you may need to remove the .txt from the filenames associated to the python scripts above so that these files can be recognized as python scripts by HIPE - the .txt suffix is added automatically by the twiki for security reasons)
HIFI tutorials
Q&A
PACS Spectroscopy
Q1: I have a rather specific question. I am reducing unchopped line scans. I use oversample=2 and I get a warning for the specwaveRebinTask (not enough good values etc, try oversample=1). However when I look at the final result (end interactive L2) there doesn't seem to be a problem, for instance in the plotCubesStdev display there are only low exposure pixels at the edges. Is this something to worry about? I hope I managed to explain this clearly enough....
A1: Hi xxx, in many pipeline tasks you get warning messages (actually, many are just information, but the developers seem to like to scare people with WARNING!! comments ;-). so at first glance your message seems to be just a kind warning. if the end result looks ok then you don't need to worry. there is a task at the end of the pipeline that plots little yellow? squares on the bins that are "low exposure". If these are all at the end of the spectral range, you don't need to worry. In fact, yes, the task plotStdDev is the one that tells you which bins are low exposure. so it looks like you do not need to worry. but you can also change the wavelength grid up/over sample values and overplot the different end results to compare. More about up and over sampling is in the PDRG chapter…5 or 6 or 7, I forgot, and more about comparing spectra at the end of the pipeline is also in the PDRG (chap 5 or 6 or 7 also - a section that talks about flatfielding), or use the spectrum explorer.
Q2: can I use data cubes that we get from the archive for spectroscopic AND photometric studies?
A2: well, if you have enough continuum in your cube then you can of course use it for photometry. the question is "what is enough". The thing to remember is that the continuum you get at the end of the pipeline is subject to ± error. This error depends on the strength of your source with respect to the teelscope. The telescope spectrum is some 100 Jy or so, and so even 1% error is a ± error of 1 Jy in the continuum. In the calibration document that is on the pacs calibration and instrument wiki the errors for the end cubes are given. ± a few Jy you should expect will be the error in the continuum level.
http://herschel.esac.esa.int/twiki/pub/Public/PacsCalibrationWeb/PacsSpectroscopyPerformanceAndCalibration_v2_4.pdf
For full science quality you need to re-pipeline the data - take advantage of the latest calibrations as the HSA products don't run all the pipeline tasks. This is documented in chp 1 of the PDRG.
You can get fluxes (Jy/spaxel) and wavelengths (microns). Though the fits files do have a rather awkward format. They can be hard to read directly into other software - we are working on this and if you have problems it is fastest to raise a helpdesk ticket. One last thing: for unchopped mode, the flux level of the continuum is
not guaranteed to be correct. That mode was never intended for continuum studies.
Q3: where can I find the sky background contamination?
A3: If the question means: where can I find the spectra of the sky (+telescope) then for chop-nod observations these are the spectra taken in the off-chop positions. The easiest way to see these data is to run the script Split On-Off, which you find in the HIPE Pipelines menu (Spectroscopy->both chopped menus -> Split On-Off. This is also explained in the pipeline chapters (chp 3) of the PACS Data Reduction Guide. This script will produce separate cubes of the on-source spectra and the off-source spectra.
Q4: can you subtract a baseline to spectra? Could it be included in the script?
A4: The currently best documented way is to fit the continuum with a poly and then get the residual (data - fit). This is explained in chapter 7 of the "Data Analysis Guide" (about the
SpectrumFitter and
SpectrumFitterGUI), which you can get via the HIPE Help->Help contents menu. The baseline fitter is a PACS specific task that you can also use but it is not well documented. This takes the data and identifies where the baseline is, i.e. it ignores lines, and then that baseline can be subtracted from the spectrum. If someone wants to use this task, or know more details, then please raise a helpdesk ticket to ask specifically for that.
Q5: is the split on-off applicable to unchopped observations?
A5: For the unchopped line scan mode, the off-source spectra are separated out from the on-source spectra within the pipeline (it will be very clear when this happens as you follow the interactive pipeline script Spectroscopy->unchopped line scan->"lineScan") and you can then compare them by plotting them together (this is also explained in the PDRG pipeline chap 4 and some additional information in chp 8). For the unchopped range scan, on and off (ie source and sky) are separate observations and once you have reduced each through to Level 2, you can compare them e.g. as cubes in the Spectrum Explorer.
PACS Photometry
Q1: Although I missed some of talks, I have a simple question. For a point source like a protostar on a bright background, how do you subtract the background using aperture photometry?
A1: challenging !!
I would say it's challenging at any wavelength (photometry on bright extended background)
You could try getSources :
http://www.herschel.fr/cea/gouldbelt/en/getsources/
Q2: Another question, I don't know what the color correction should be. In principle, do we need to apply a color correction for the aperture photometry of a protostar?
A2: color correction factors for various type of SED are given in :
"PACS Photometer Passbands and Colour Correction Factors for various source SEDs (2.5 Mb), PICC-ME-TN-038, version 1.0, 12 April 2011"
Q3: The PACS photometry tutorials have been extremely helpful for me. However, so far they have dealt with point sources only and my targets of interest are not point sources. Can an expert comment briefly on extended source photometry? For example, how someone would create 24-500 um SED within circular apertures (say, 20" diameter) for a spiral arm?
A3: Indeed photometry is primarily focussed toward point-sources. For slightly extended sources the aperture correction factors are not easy to compute,
actually one would need to compute it depending on the shape and the extend of the source. For very large sources - hence very large apertures, say, several arcmin) -,
the aperture correction will be very close to 1, hence almost negligible. But there is a regime in between where it's more complex.
However you mention a circular aperture of 20" which is rather small, so your source must be rather compact I would say that the point-source EEF
could be used, though slightly underestimating the correction.
Q4: Hello, I do have a question on making a map of a region with madmap, if posssible. the map I produce looks not optimal to me, but I would like to know if this is true and if so, how can i optimize it. the obsid is 1342216409(10,11,12) and I run the madmap pipeline on all and the final output looks very similar to what is in the archive. I am trying to run scanamorphous but running out of memory. any hint will be appreciated. thanks.
A4: you could try photExtractRaDecFrames() for that purpose. it allows to extract frames around a position (ra, dec fieldSize).
However madmap requires continuous timelines as much as possible, so it might lead to suboptimal maps because the drift correction
will get confused by the signal jumps. Best would be to run madmap on the 4 obsids, I am surprised scanamorphous went out of memory, did you try
JyPScanam in Hipe 11?
it works rather nicely.
Q5: Why can't one simply deconvolve PACS images the way radio astronomers do using algorithms like CLEAN? If the PSF is well known, stable, and of high-S/N then one should be able to "squeeze" the flux from the wings back into the PSF core, thus making need for aperture corrections moot, and making ordinary photometry easier. Since there are no CLEAN tasks in HIPE, there must be some reason why this can't be done. Just wondering...
A5: there are deconvolution attempts on PACS (and SPIRE) that work relatively nicely. But bringing back the PSF wings - that extend up to several arcmin - into the PSF core seems utopic to me.
But I must confess I am not a CLEAN expert. One additional limitation is that there is not a single unique PSF for PACS,
the PSF depends on the observation parameters (scan direction and speed) and data processing (map-making) used. Hope it helps.
SPIRE photometry
Q1: How much memory is needed for the reduction of a large SPIRE field (e.g. like a 2x2 degree Hi-GAL tile)?
A1: 8-10 GB is usually more than enough to reduce a large map like this. Small scan maps can be reduced without any problem with a 4GB laptop.
Q2: Are there any limitations in the reliability of the scripts used for determining the Planck offsets to be applied to SPIRE maps? In particular, I am interested in whether they can be applied to very small maps (since the corrections are based on the average absolute fluxes covered by the much bigger Planck maps)?
A2: In principle not, but it could be convenient to look at the special characteristics of the emission observed in your small field. If you have a particular sky field dominated by galactic cirrus emission it could be an issue. Raise a helpdesk ticket indicating the OBSID of the map you want to analyse and we will tell you whether we see any problem with applying the offset derived from Planck data.
SPIRE spectroscopy
Q1: Should the semi-extended correction be applied after all other steps (for instance improve background etc)?
A1: semi-extended correction should be applied as a last step after background subtraction because the galactic emission could make the source look as semi-extended
Q2: How do I find the appropriate dark observation to be used in the processing of a SPIRE FTS observation?
A2: You should look at the dark observations taken in the same operational day (OD) when your science observation was taken, or the nearest in time if there is none available on that particular OD. The OD number you can get it from the Observing Log or from the Archive (HSA). Once you know the OD, you query in HSA for all SPIRE FTS observations taken on that OD or a small range of ODs around the date when the observation was taken. There is also a list with all available dark observations taken (in the standard observing mode) throughout the mission at :
http://herschel.esac.esa.int/twiki/bin/view/Public/SpireDailyDarkObservations
HIFI
Q1: Do you get automatic notifications when there is any new version of the HIFI calibration files/tree?
A1: No, it's not like for PACS that you get a notification whenever a new calibration tree is available. Usually a new version of HIFI calibration comes always (and only) with every new major version of HIPE, but not more frequently.
Q2: Is there a list of known spurs available?
A2: Yes, there is a catalogue that you can inspect coming as part of the calibration files delivered through the HSA where a detailed list of known spurs is provided, with indication of the frequency at which they are seen and their main characteristics. These frequencies are flagged/masked in the pipeline so that you do not need to care about them.
Public Web
Create New Topic
Index
Search
Changes
Notifications
Statistics
Preferences
Public
