Public Web>DpHipe>DpHipePlugins>DpHipePluginsDeveloperManual (2018-03-08, PedroGarciaLario)

Plug-in Author Manual

This manual describes first how to create plug-ins for HIPE. In most cases, you will only have to read the relevant sections from the Quick Guide chapter. The final chapter describes the mechanism that HIPE component developers can use to become aware of plug-ins. For more information on the plug-in framework, see the Design document. User documentation on plugins is available on this page.

Plug-ins are available in HIPE starting version 5.0.

Quick Guide: Creating Simple Plug-ins
Slightly more advanced topics
Advanced topics
Automatic Updates
Examples
For HIPE component developers: Working with plug-ins in HIPE
- PluginRegistry and Plugin class
- PluginProcessor
Further information

Quick Guide: Creating Simple Plug-ins

Plug-ins can be created to do lots of fancy things, but most of the time people will want to use them to share some pools, some Jython scripts, and/or possibly some Java code. These standard cases are described in this chapter. One case (pools, Jython or Java) does not exclude the other; combining the steps from the different cases described here will work.

General procedure for creating a plug-in

In general, a plug-in is just a JAR or a ZIP-file with a name that follows the pattern: [name]_[version].[extension]. The name can be any string (it is allowed to contain underscores). The name is followed by an underscore. The version is a series of integers separated by dots: "1", "2.0", "2.3.4.5", etc, are allowed as the version. The extension should be "jar" or "zip" (lowercase). An example of a plug-in filename is myplugin_0.1.jar.

On Unix you can use the jar facility from the bin directory of the JDK to create jar-files. I recommend to cd to the directory where you have prepared your plug-in and create the jar-file in a different place, as follows:

> jar cvf $HOME/myplugin_0.1.jar *

The options to jar we used here are cvf: c for "create", v for "verbose" and f for filename.

If you use Windows, you can use any compression program, as long as it a can write "zip" or "jar" file format. For example, I like PeaZip.

All components of a plug-in are optional: Even an empty JAR-file is a valid plug-in that you can install perfectly well. Only, it doesn't do anything of course. For that, see the next sections.

How to create a plug-in with some pools

Plug-ins can contain LocalStores. To create a plug-in with one or more LocalStores, create a directory called pools, copy the LocalStores in there and create a jar or zip with the pools directory.

Unix shell example: Assuming your LocalStores are in the default directory, $HOME/.hcss/lstore and you want to create a plug-in with poolA and poolB:

> mkdir pools
> cp -r $HOME/.hcss/lstore/poolA pools
> cp -r $HOME/.hcss/lstore/poolB pools
> jar cf $HOME/myplugin_0.1.jar pools

You probably will want to name your plug-in differently than "myplugin" though -- give it a name that says something about its contents. To change the name of the plug-in, you can just rename the file.

When the plug-in is installed, the pools will appear in the Data Access section of the preferences (see Edit > Preferences).

You can also share LocalStores using the Export View in HIPE. This will give you a compressed file with the pool, in the format of your choosing. Another user can import this file using the Import View, to recreate the LocalStore in his HIPE session (and in future HIPE sessions). This is simple and straightforward and there's nothing wrong with it. Plug-ins have more features, such as versions, title and description, and a plug-in can contain accompanying scripts, for example. Also, pools delivered as part of a plug-in are not installed to your LocalStore directory, they are kept in the plug-in directory, so they do not mix with your personal LocalStores. If a plug-in is published on a web-site (HTTP or FTP), then installation of a plug-in is slightly easier, since the download and unpacking are done automatically in a single step instead of two steps.

How to create a plug-in with some Jython code

We make a distinction between two types of Jython code, and how you create the plug-in depends which type you have. First, there is what we call "user scripts". These are meant to be scripts that a user should execute, such as a script that does some kind of processing. The other kind is a script that defines some functionality, for example a Task. In this second case, the user should not execute the script to use it (which would define the Task), the user should run the Task. This kind of script should be run by HIPE when it starts, so that the user has the Task available for use. We therefore call that kind "initialization scripts".

To create a plug-in with one or more user scripts, copy them to a directory called scripts and create a zip or jar containing that directory.

Unix shell example:

> mkdir scripts
> cp ~/work/scriptsForPlugin/*.py scripts
> jar cf $HOME/myplugin_0.1.jar scripts

When this plug-in is installed, a menu item with the name of the plug-in will appear in the Tools menu. Pointing to the menu item opens a submenu with all scripts. Clicking on one of the scripts will open it in the editor.

Plug-ins support only one initialization script, but you can define multiple Tasks from a single script. The initialization script must be called plugin.py and it must be at the top or root-level of the jar or zip.

Unix shell example:

> mkdir scripts
> cp ~/work/myTasks.py plugin.py
> jar cf $HOME/myplugin_0.1.jar plugin.py

This plugin.py script is similar in function to the __init__.py script that is contained in many of the Java packages in the HCSS. However, the __init__.py must have some obligatory lines of code, which are not required for the plugin.py script.

How to create a plug-in with some Java code

To add (compiled) Java code to a plug-in, the Java code has to be in a JAR itself. Add the JAR to a directory called "jars" and create a plug-in with this directory:

> mkdir jars
> cp ~/work/out/library.jar jars
> jar cf $HOME/myplugin_0.1.jar jars

Note that the JARs are included integrally in the plug-in JAR; you should not unpack your JARs with Java code and include the loose class files in the plug-in JAR. To clarify further, if you get a listing from your plug-in JAR using jar tf plugin-1.0.jar, you might get a listing like this:

META-INF/
META-INF/MANIFEST.MF
jars/
jars/filtering.jar
plugin.py

This plug-in contains a JAR called filtering.jar, which in turn might contain any number of Java classes.

For information on how to create a JAR file with your Java code, see the JAR Documentation.

How to create a plug-in with a Task

To easiest way to avoid the hassle that is involved in "making a new task work in HIPE", is to create a plug-in with the task inside. If your Task is in Java, please refer to MyFirstJavaTask and if it's in Jython, see MyFirstJythonTask on the HIPE Community Wiki. Both these pages start by explaining how to write a task in their respective language, and both pages contain a section "Turning the task into a HIPE plug-in".

Combinations of the above

If you want to create a plug-in that has an initialization script, some user scripts and a handful of pools, the easiest thing is to create a separate directory. Copy everything in there (the scripts directory, pools directory, etc) and create a zip or jar with everything:

> jar cf $HOME/myplugin_0.1.jar *

What you must avoid / Invalid plug-ins

When sharing things with plug-ins, you are not limited the items described in the previous section. You can add anything to a plug-in, but there are a few things that would make the plug-in invalid. An invalid plug-in would fail to install.

The following would make the plug-in invalid:

a file called scripts, pools, bundle.xml or jars at the top-level of the file

Slightly more advanced topics

Organizing your `plugin.py` into several shorter scripts

If your plugin.py grows large, it becomes difficult to maintain. You can organize the plugin.py into several files and call these from the plugin.py script as follows. First, you have to find the directory in which your plug-in is installed. The next few lines assign this directory to the variable basedir:

from herschel.ia.gui.apps.plugin import PluginRegistry
reg = PluginRegistry.getInstance()

name = 'myPluginName' # fill in the name of the plug-in here
plugin = reg.get(name)
basedir = plugin.pluginDir.absolutePath

Using this base directory, you can execute scripts in your plug-in using the execfile command:

execfile(basedir + '/a.py')
execfile(basedir + '/b.py')
execfile(basedir + '/c.py')

Note that you can store these scripts in subdirectories within your plug-in as well.

Compatibility information

In general, each version of the plug-in will be compatible with a range of HIPE versions. Maybe version 1.0 to 1.7 of the plug-in is compatible with HIPE 7 to 9, but then for version 2.0 of the plug-in, the plug-in was redesigned, and it's only compatible with HIPE version 10 and up. Or an incompatible change was introduced in HIPE that made that the plug-in versions 1.7 and below stopped working. It's possible to specify this information precisely, either in the plug-in itself, or in an XML file that is accessible from the web, called the plug-in version registry. The advantage of putting the information into the plug-in version registry is that you as plug-in author can update it whenever you want. HIPE will retrieve this information automatically upon start-up and it will take priority over information that was included in the plug-in.

If a plug-in is said not to be compatible with the HIPE version that you're starting, then HIPE will disable the plug-in, to prevent problems. If you exit HIPE and start a (older?) version of HIPE that is compatible, then the plug-in will be enabled normally. Or, if you're running two different versions of HIPE at the same time, then your plug-in might be enabled in one version and disabled in another. Also, following the above example, suppose you run HIPE 9 and version 1.7 of the plug-in, then upgrade to HIPE 10 and version 2.0 of the plug-in. Then, as long as you don't explicitly remove plug-in versions, starting HIPE 9 will start with version 1.7 of the plug-in enabled (the highest compatible version), whereas HIPE 10 will start normally with version 2.0 of the plug-in.

Here we will describe how to include compatibility information inside the plug-in itself, but if you can, it's advisable to use a plug-in version registry. This is described in the section Automatic Updates below.

Compatible HIPE versions are specified giving a minimum version with which HIPE is compatible, and a maximum version. Experience shows that unless you already know that the latest HIPE version is incompatible, it's best to specify * as the maximum compatible version. Otherwise you will have to regularly update the maximum version to allow for new HIPE versions. It's less maintenance to act when an incompatibility is found, and specify a maximum version at that time.

Specifying a minimum version of 1.0.* and maximum version of * means that your plug-in is compatible with all builds from the 1.0 track, and all future versions. The "version" is specified using build track plus build number, not with the release number. So, for example, release 4.6 was built on the 4.0 track and it's build 1467. In the version compatibility information in plugin.xml you would specify release 4.6 as 4.0.1467. If an incompatibility is introduced in HIPE in build 1000 on the 4.0 track, then your maximum version should be set to 4.0.999.

Finally, it should be noted that the asterisk is not a wildcard. It's interpreted as a very high number, so it is only intended to be used to specify the upper limit of compatible versions (maxVersion). Using it in the lower limit, minVersion, is allowed, but this is normally not what you want. See the example in the next section.

The above information is set in a file called plugin.xml which allows for much more customization. An example plugin.xml, which includes compatibility information, is given in the next section.

Customizing your plug-in

HIPE allows for the plug-in to specify additional information about itself using a file called plugin.xml at the root-level of the plug-in jar or zip.

Below is an example of plugin.xml file with some of the different elements it can have:

<?xml version="1.0" encoding="UTF-8"?>
<plugin xmlns:em="http://www.mozilla.org/2004/em-rdf#">

   <description>This plug-in provides tasks for analysis of data.</description>
   <contributors_page>http://www.esa.int</contributors_page>
   <icon>herschel/ia/gui/apps/plugin/Extension_32x32.png</icon>
   <properties>custom.properties</properties>

   <!-- Compatible starting user release 5.0 (build 1760), not with earlier builds -->
   <em:minVersion>5.0.1760</em:minVersion>
   <!-- Compatible with all future HIPE versions -->
   <em:maxVersion>*</em:maxVersion>

   <em:updateURL>https://www.mysite.com/registry.xml</em:updateURL>

</plugin>

All of the elements of this XML file are optional, except <plugin>. If you're not interested in any of the elements, just don't add this file.

The description and the contributors page are shown on the plug-in details panel. In HIPE, select Plug-ins from the Tools menu. Select a plug-in (if you have one installed!) and click "Details".

The icon is used on the Plug-ins panel (the one that opens when you select Plug-ins from the Tools menu in HIPE). The value is the location of a "resource" on the classpath. The value given here gives the default plug-in icon, which is a green puzzle piece. To add your own icon, add it to a jar, add the jar to the jars directory and specify the filename of the icon in the plugin.xml. Icons not having 32x32 pixel resolution will be scaled to fit this size when shown in the plug-ins panel.

The properties elements specifies a properties file in the plug-in, containing properties that are loaded when the plug-in is started. The plug-in is started at HIPE start-up, unless it is disabled from the GUI. If the custom properties file contains properties that are already defined in the system, they are ignored and a warning is printed.

The compatibility information, minVersion, maxVersion, and also the updateURL will be explained later.

Adding a license and/or release notes

It is possible to add a license text to a plug-in. All you have to do is add a file called LICENSE.txt to the root-level of the plug-in. This license will be displayed on the License tab of the Plug-in details panel.

For release notes, it works the same way: Add a file called RELEASE_NOTES.txt to the root-level of the plug-in. The contents of this file will be displayed on the Release Notes tab of the details panel of the plug-in.

Adding a menu item to the Help menu

You may want to add an entry in HIPE's Help menu, that the user can use to get help about your plug-in specifically. One way to do this is to add a menu item that opens the web-browser on the user's system and directs it to an on-line resource (the Help section of your plug-in's homepage for instance). An example plug-in is attached. See the plugin.py and the helpMenu.src.jar in this plug-in for the source code of this example.

Linking to external help pages

To provide more integrated help, you can link an external web page to your tasks (for example) by implementing herschel.ia.gui.kernel.parts.ExternalHelp.

class ImportScanAmorphosTask(Task,ExternalHelp):
# ...
  def getHelpURL(self):
        return String("http://herschel.esac.esa.int/twiki/bin/view/Public/ScanAmorphosPlugin")

Thus, by typing in console

help(importScanAmorphos)

pressing the task's GUI help button, or selecting the task in the Tasks view and selecting the Help -> Help (External) menu option, your browser will open the web page you have provided.

Using preferences in your plug-in

A plug-in can use preferences as any other code integrated in HIPE.

It can also register custom preferences panels, which will be included in the preferences tree that is shown when executing Edit > Preferences.

Warning:

Preferences cannot be read during the Jython initialization. For plugins, it means that trying to consult a preference while running the plugin.py would throw an IllegalStateException, for example:

from herschel.ia.gui.kernel.prefs import UserPreferences
print UserPreferences.get("MyPluginCategory", "myPreference") # Exception that prevents the plugin from correctly finalizing its initialization

Now, preferences can be accessed once HIPE start-up has finished. For that, schedule it like in this example:

from herschel.ia.gui.kernel import SiteApplication
from java.lang import Runnable

class PreferenceReader(Runnable):
    def run(self):
        from herschel.ia.gui.kernel.prefs import UserPreferences
        print UserPreferences.get("MyPluginCategory", "myPreference")

SiteApplication.getSite().runWhenStarted(PreferenceReader(), False)

Using your plug-in in HIPE batch mode (was: jylaunch)

The jylaunch command was replaced in HIPE 11 by "HIPE batch mode", which is invoked simply by calling the hipe command, followed by the filename of the script to execute.

Plug-ins can be used in the same way in batch mode as from the GUI, with the same caveats as processing scripts. HIPE in batch mode does not load all Herschel classes on start-up, like the GUI does, to save time. The HIPE GUI executes the following command at start-up:

from herschel.all import *

The fact that in batch mode this command is not executed, can lead to different problems. A typical error caused by not having imported all Herschel code is:

NameError: TableDataset

TableDataset is a class that is readily imported after start-up, in the GUI, but not in batch mode.

To make a plug-in work in batch mode, the recommended approach is to import the classes that are needed at the start of the plugin.py file. You can also use the above general Herschel import statement, but note that it takes significant time to execute.

Advanced topics

The plug-in installation process

The standard plug-in installation process just decompresses the plug-in into the plug-in directory and "starts" the plug-in. The plug-in directory is $HOME/.hcss/apps/hipe/plugins/[name]/[version]. You can view this directory by opening the plug-ins panel in HIPE (select Plug-ins from the Tools menu). Click the plug-in and select "Browse plug-in" from the View menu in the plug-ins panel.

Starting the plug-in means:

add all JARs to the Java and Jython class-loader(s)
load the custom properties
execute the initialization script, load
add the menu items for the user scripts
execute any registered PluginProcessors (see below)

In this order.

In general, the installation process of a plug-in consists of two steps or phases. For simple plug-ins, such as have been described until now, the second phase is skipped. Only the first phase, which is called "bootstrap", is executed. During this bootstrap installation, the plug-in is downloaded from the URL that indicates its location, and the jar or zip is decompressed in the plug-in directory. Note that the directory, that the plug-in is unpacked into, contains the version of the plug-in. This means that multiple versions of a plug-in can exist on a user's disk, but when HIPE starts, only the latest version of each plug-in will be started.

The second phase is called "custom installation". This allows the plug-in developer to execute an external installation program and dynamically configure the plug-in. For example, one can run a complete, separate installation program during this step, which installs an external Java program somewhere on the user's disk. Then for this plug-in to contribute something to HIPE, probably the JARs of the external program have to be added to the HIPE classpath. This can be done during this phase as well.

Using a custom installer

To run a custom installer during phase 2, you can add a class called Installer to the plug-in top-level dir. This class must implement the interface herschel.ia.gui.apps.plugin.Installer and it must be in the default package (i.e. no package).

After completion of phase 1, the install(PluginEnv) method of this interface is called. This method could call the main method of the installation program, included in the jars directory. When this completes, the Installer could add references to external JARs that were just installed. The method addJar of the PluginEnv class can be used to do that.

The custom installation is executed in a background thread. Note that it implies that any code related with GUIs would require an explicit call to be run in the EDT (Event Dispatching Thread), for example by means of SwingUtilities.invokeLater, EDT.run or EDT.call.

Uninstallation

The Plug-ins panel in HIPE, from the Edit menu, allows to "remove a plug-in completely" or to "roll back to a previous version". This means to remove either all versions or just the latest one, undoing in effect the bootstrap installation. There is no way to run a "custom uninstaller" explicitly.

Enabling and disabling plug-ins

The Plug-ins panel in HIPE also allows to "disable" or to "enable" a plug-in (depending on whether it is currently enabled or disabled). A disabled plug-in is simply not started. It is loaded into the Plug-in Registry (see below), but it is not started.

The disabled or enabled status is persisted to disk, so that if a plug-in is disabled or enabled, it will still be in that state when HIPE is restarted.

Loading classes from included JARs

Normally, when including one or more JARs in your plug-in, you don't have to worry about class paths, class loaders, etcetera. If you import and instantiate a class from such a JAR using Jython or in Java, this will work as normal.

The exception is when you explicitly use a class loader to load a class. Suppose you include a JAR with a JDBC driver to connect to a relational database. Typically you will have to load the driver by its class name, to be able to use it. To be able to load such a class, which is in a JAR that is not on the standard Java class path but inside a plug-in, you must use a class loader that knows about this JAR. The recommended way to do this is to use the ObjectUtil class:

// to get the Class object:
Class<?> c = ObjectUtil.getClass("my.package.MyClass");
// or to create an object of this class
my.package.MyClass object = ObjectUtil<my.package.MyClass>.newInstance("my.package.MyClass");

The newInstance method uses the default constructor. If you need to use a different constructor, use the Class object.

Automatic Updates

The plug-in author has two ways to provide the users with updates: It is possible to update the compatibility of plug-ins that users have installed, and it is possible to offer users new versions of plug-ins.

If a plug-in was released before HIPE v6 was released, its plugin.xml should state that it is compatible up to and including HIPE "5.0.*". When the first version (release candidate?) of HIPE v6 comes out, the author should check if the existing version of the plug-in is still compatible with HIPE v6. Normally, this will be the case. If so, the author will want to update the compatibility information in the plug-in on the user's disk, to include compatibility with HIPE v6.

The point of offering new versions of plug-ins is self-evident.

The mechanism to provide this information relies on the plug-in registry, which is an XML file on a web-server. This XML file should contain entries for the available versions of the plug-in, plus the compatibility information of each version. Such a registry file could look as follows:

<?xml version="1.0" encoding="UTF-8"?>

<RDF:RDF xmlns:RDF="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
         xmlns:em="http://www.mozilla.org/2004/em-rdf#">

  <!-- This Description includes all the update and compatibility information for a
       single plug-in called "example1". You can list information for multiple plug-ins in
       the same file. -->
  <!-- NOTE The value of the "about" attribute is not a description but the name of the plug-in! -->
  <RDF:Description about="example1">
    <em:updates>
      <RDF:Seq>

        <!-- Each "li" entry is a different version of the same plug-in -->
        <RDF:li>
          <RDF:Description>
            <em:version>2.2</em:version> <!-- This is the version number of the plug-in -->

            <em:minVersion>5.0.1760</em:minVersion>
            <em:maxVersion>6.0.*</em:maxVersion>

           <!-- This is where this version of the plug-in will be downloaded from -->
           <em:updateLink>https://www.mysite.com/example1_2.2.jar</em:updateLink>

          </RDF:Description>
        </RDF:li>

        <RDF:li>
          <RDF:Description>
            <em:version>2.3</em:version>

            <em:minVersion>6.0.1883</em:minVersion>
            <em:maxVersion>*</em:maxVersion>

           <!-- This is where this version of the plug-in will be downloaded from -->
           <em:updateLink>https://www.mysite.com/example1_2.3.jar</em:updateLink>

          </RDF:Description>
        </RDF:li>

      </RDF:Seq>
    </em:updates>

  </RDF:Description>
</RDF:RDF>

The above example registry file lists two versions of a plug-in called "example1": Version 2.2 and 2.3. Version 2.2 is stated to be compatible with HIPE v5.0 (the user release, not earlier builds) and also all builds of the 6.0 track. Version 2.3 is compatible version HIPE release 6.0 starting RC1 and all future versions of HIPE.

Contrary to earlier recommendations, I now recommend to specify that the latest (current) version of your plug-in is compatible with all future versions, by specifying an asterisk: *. If it it turns out that at some version of HIPE the plug-in does not work anymore, users can disable it manually and/or you can update the registry file to specify the new maximum. Experience shows this works better than specifying the current HIPE version as the maximum version, and updating it when new HIPE versions come out. Plug-ins tend to get blocked inadvertently.

If version 2.4 of the plug-in is released, it can also simply be added to the registry. HIPE instances having the plug-in installed will suggest the update to the user, if the version of HIPE is compatible.

The only pending question is how HIPE instances that have the plug-in "example1" installed find the registry. The answer is: Via an "updateURL" reference in the plugin.xml.

Examples

For examples, see DpHipePlugins. Any of the plug-ins can be downloaded and inspected using the jar tool or any other zip program. Among examples offered are:

SPIA: This plug-in contains:
- A HIPE SPIA perspective
- An initialization script defining Tasks. This initialization scripts also has the registration for the perspective at the bottom.
The "demonstration plug-in" contains:
- A more fully-featured plug-in descriptor file, plugin.xml
- An initialization script
- A few user scripts (the scripts themselves are trivial)
- Custom properties that are added to the HIPE configuration
- A custom installer for phase 2 ("custom installer" -- see advanced topics). The source code for this installer is also included, this is for your convenience only.
- A license

For HIPE component developers: Working with plug-ins in HIPE

Some HIPE components may have to perform some actions when a plug-in is installed. For example, the Preferences panel is the one that picks up the pools from the plug-ins when they are installed. Such things are done by registering a PluginProcessor.

Once plug-ins have been installed and are started, HIPE components can find them in the Plug-in Registry (for example, the Plug-ins Panel that can be opened from the Tools menu).

`PluginRegistry` and `Plugin` class

The Plug-in Registry, implemented by the PluginRegistry class, is a singleton that allows access to all plug-ins currently installed (latest versions only). The elements that in the registry are of the Plugin class. They are keyed by the plug-in name and sorted alphabetically.

The Plugin class reflects the installed plug-in in its plug-in directory: It has accessors to get the information from the plugin.xml file, you can ask it which JARs and scripts the user has installed, etc. And finally, you can use this class to start, stop and enable/disable the plug-in. Starting and stopping is normally managed by HIPE though, so it is unlikely that you will need to call those methods.

`PluginProcessor`

You can implement the PluginProcessor interface (herschel.ia.gui.apps.plugin package) and register it on the PluginRegistry. This processor will be called when any plug-in is started (the onStart method) and also when any plug-in is stopped (onStop method).

In principle, all plug-ins are started when HIPE is started, and presumably you will also register your PluginProcessor when HIPE is started. This gives rise to a race condition: Depending on whether you manage to register the processor a little bit earlier or later, it may or may not get called for some of the plug-ins. This is solved by making sure that all PluginProcessors that are registered are also called for all plug-ins that have been started already. Effectively this simply means you do not have to worry about this; if you register your processor, it will be called for all plug-ins that have been started and those that will still be started during HIPE start-up.