Difference: DatasetReaderPlugin (1 vs. 4)

Revision 42016-07-26 - JaimeSaiz

Line: 1 to 1
 
META TOPICPARENT name="DpHipePlugins"

Dataset Reader Plug-in

Line: 16 to 14
 
obs = getObservation(...)
framesRef = obs.refs["level1"].product.refs["HPPAVGR"].product.refs[0]
Changed:
<
<
urn = framesRef.urn
>
>
urn = framesRef.urn
  To read a single dataset from a given product:
Deleted:
<
<
 
urn = 'urn:hsa:herschel.pacs.signal.Frames:895939'
Changed:
<
<
dataset = getSingleDatasetFromHSA(urn, 'Signal')
>
>
dataset = getSingleDatasetFromHSA(urn, 'Signal')
  To read several datasets from a given product, the following is faster than repeating the above several times:
Deleted:
<
<
 
map = getDatasetsFromHSA(urn, ['Signal', 'Status'])
signal = map['Signal']
Line: 31 to 25
 
map = getDatasetsFromHSA(urn, ['Signal', 'Status'])
signal = map['Signal']
Changed:
<
<
status = map['Status']
>
>
status = map['Status']
  Finally, the plug-in provides a dedicated function to reconstruct partial PACS Frames products:
Deleted:
<
<
 
Changed:
<
<
framesPartial = buildSignalOnlyFrames(framesRef)
>
>
framesPartial = buildSignalOnlyFrames(framesRef)
  This product, framesPartial, contains the Signal and Status dataset only. This compares to the "standard way":
Deleted:
<
<
 
Changed:
<
<
framesFull = obs.refs["level1"].product.refs["HPPAVGR"].product.refs[0].product
>
>
framesFull = obs.refs["level1"].product.refs["HPPAVGR"].product.refs[0].product
  To read the framesFull product, for this particular observation of about 5 hours, one needs around 8 GB of memory. To create the framesPartial product, 4 GB is enough, and one can use this partial product to make maps, to run the high-pass filter task, and so on.
Line: 52 to 41
  To install the plug-in, choose Plug-ins from the Tools menu. Click Install new... and enter the following URL:
Changed:
<
<
http://herschel.esac.esa.int/twiki/pub/Public/DatasetReaderPlugin/datasetReader.xml
>
>
http://herschel.esac.esa.int/twiki/pub/Public/DatasetReaderPlugin/datasetReader_1.0.jar
  Warning, important HIPE 9 users, use the following URL: http://herschel.esac.esa.int/twiki/pub/Public/DatasetReaderPlugin/datasetReader_0.2.jar

Revision 32013-02-05 - PaulBalm

Line: 1 to 1
 
META TOPICPARENT name="DpHipePlugins"

Dataset Reader Plug-in

Added:
>
>
 This plug-in provides functions that allow reading of individual datasets from Products from the HSA, without the need to read the entire product into memory.
Added:
>
>
The concrete example for which it was written, are PACS Level 1 Frames products. By not loading the noise cube, for a given amount of memory, one can load products twice as large. This plug-in can be used for any kind of product, but it provides a special function to the PACS Frames, which produces a Frames product that can be reprocessed, but that does not contain the noise cube.

How to use

 Products are identified using their URN, which can be retrieved for example as follows, for a PACS Level 1 Frames product:
Line: 58 to 65
  -- PaulBalm - 05 Nov 2012
Changed:
<
<
META FILEATTACHMENT attr="" autoattached="1" comment="Version registry for the plugin - add version 0.2" date="1354118185" name="datasetReader.xml" path="datasetReader.xml" size="1355" user="Main.PaulBalm" version="3"
META FILEATTACHMENT attr="" autoattached="1" comment="Version 0.2 -- fixes import problem in buildSignalOnlyFrames(...)" date="1354118170" name="datasetReader_0.2.jar" path="datasetReader_0.2.jar" size="2610" user="Main.PaulBalm" version="1"
>
>
META FILEATTACHMENT attachment="datasetReader_1.0.jar" attr="" comment="Version 1.0 -- fixes variables left behind" date="1360069995" name="datasetReader_1.0.jar" path="datasetReader_1.0.jar" size="3004" stream="datasetReader_1.0.jar" user="Main.PaulBalm" version="2"
META FILEATTACHMENT attr="" autoattached="1" comment="Version registry for the plugin - add version 1.0" date="1360069612" name="datasetReader.xml" path="datasetReader.xml" size="1888" user="Main.PaulBalm" version="4"
 
META FILEATTACHMENT attr="" autoattached="1" comment="Version 0.1" date="1352126470" name="datasetReader_0.1.jar" path="datasetReader_0.1.jar" size="2580" user="Main.PaulBalm" version="2"
Added:
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Version 0.2 -- fixes import problem in buildSignalOnlyFrames(...)" date="1354118170" name="datasetReader_0.2.jar" path="datasetReader_0.2.jar" size="2610" user="Main.PaulBalm" version="1"

Revision 22012-11-28 - PaulBalm

Line: 1 to 1
 
META TOPICPARENT name="DpHipePlugins"

Dataset Reader Plug-in

Line: 16 to 16
 
urn = 'urn:hsa:herschel.pacs.signal.Frames:895939'
Changed:
<
<
dataset = getDatasetFromHSA(urn, 'Signal')
>
>
dataset = getSingleDatasetFromHSA(urn, 'Signal')
 

To read several datasets from a given product, the following is faster than repeating the above several times:

Line: 47 to 47
  http://herschel.esac.esa.int/twiki/pub/Public/DatasetReaderPlugin/datasetReader.xml
Changed:
<
<
Warning, important HIPE 9 users, use the following URL: http://herschel.esac.esa.int/twiki/pub/Public/DatasetReaderPlugin/datasetReader_0.1.jar
>
>
Warning, important HIPE 9 users, use the following URL: http://herschel.esac.esa.int/twiki/pub/Public/DatasetReaderPlugin/datasetReader_0.2.jar
 

Limitations

Line: 58 to 58
  -- PaulBalm - 05 Nov 2012
Changed:
<
<
META FILEATTACHMENT attr="" autoattached="1" comment="Version registry for the plugin" date="1352125964" name="datasetReader.xml" path="datasetReader.xml" size="830" user="Main.PaulBalm" version="2"
META FILEATTACHMENT attachment="datasetReader_0.1.jar" attr="" comment="Version 0.1" date="1352126470" name="datasetReader_0.1.jar" path="datasetReader_0.1.jar" size="2580" stream="datasetReader_0.1.jar" user="Main.PaulBalm" version="2"
>
>
META FILEATTACHMENT attr="" autoattached="1" comment="Version registry for the plugin - add version 0.2" date="1354118185" name="datasetReader.xml" path="datasetReader.xml" size="1355" user="Main.PaulBalm" version="3"
META FILEATTACHMENT attr="" autoattached="1" comment="Version 0.2 -- fixes import problem in buildSignalOnlyFrames(...)" date="1354118170" name="datasetReader_0.2.jar" path="datasetReader_0.2.jar" size="2610" user="Main.PaulBalm" version="1"
META FILEATTACHMENT attr="" autoattached="1" comment="Version 0.1" date="1352126470" name="datasetReader_0.1.jar" path="datasetReader_0.1.jar" size="2580" user="Main.PaulBalm" version="2"

Revision 12012-11-05 - PaulBalm

Line: 1 to 1
Added:
>
>
META TOPICPARENT name="DpHipePlugins"

Dataset Reader Plug-in

This plug-in provides functions that allow reading of individual datasets from Products from the HSA, without the need to read the entire product into memory.

Products are identified using their URN, which can be retrieved for example as follows, for a PACS Level 1 Frames product:

obs = getObservation(...)
framesRef = obs.refs["level1"].product.refs["HPPAVGR"].product.refs[0]
urn = framesRef.urn

To read a single dataset from a given product:

urn = 'urn:hsa:herschel.pacs.signal.Frames:895939'
dataset = getDatasetFromHSA(urn, 'Signal')

To read several datasets from a given product, the following is faster than repeating the above several times:

map = getDatasetsFromHSA(urn, ['Signal', 'Status'])
signal = map['Signal']
status = map['Status']

Finally, the plug-in provides a dedicated function to reconstruct partial PACS Frames products:

framesPartial = buildSignalOnlyFrames(framesRef)

This product, framesPartial, contains the Signal and Status dataset only. This compares to the "standard way":

framesFull = obs.refs["level1"].product.refs["HPPAVGR"].product.refs[0].product

To read the framesFull product, for this particular observation of about 5 hours, one needs around 8 GB of memory. To create the framesPartial product, 4 GB is enough, and one can use this partial product to make maps, to run the high-pass filter task, and so on.

Installation

To install the plug-in, choose Plug-ins from the Tools menu. Click Install new... and enter the following URL:

http://herschel.esac.esa.int/twiki/pub/Public/DatasetReaderPlugin/datasetReader.xml

Warning, important HIPE 9 users, use the following URL: http://herschel.esac.esa.int/twiki/pub/Public/DatasetReaderPlugin/datasetReader_0.1.jar

Limitations

  • This plug-in currently only works with products from the HSA.
  • Until the resolution of HcssScr:17096, the downloaded FITS file has to be uncompressed as a separate step, which is a little time-consuming.

-- PaulBalm - 05 Nov 2012

META FILEATTACHMENT attr="" autoattached="1" comment="Version registry for the plugin" date="1352125964" name="datasetReader.xml" path="datasetReader.xml" size="830" user="Main.PaulBalm" version="2"
META FILEATTACHMENT attachment="datasetReader_0.1.jar" attr="" comment="Version 0.1" date="1352126470" name="datasetReader_0.1.jar" path="datasetReader_0.1.jar" size="2580" stream="datasetReader_0.1.jar" user="Main.PaulBalm" version="2"
 
This site is powered by the TWiki collaboration platform Powered by Perl