Plug-in Framework Design Document

This document describes the design choices underpinning the Plug-in Framework in HIPE. The SCR that originally requested the Plug-in Framework was HcssScr:9471.

What a Plug-in is and can do

Many people write software either in Java or in Jython and then want to share this software with other people. Sometimes this software is a script that has to be run manually, sometimes it has to be run every time HIPE is started (for example if it defines some Tasks). Or people may generate pools of data that they want to share.

This software and this data can be zipped and put on a web-server. If the zip-file has a version in it, this version is lost when the zip-file is unpacked: After importing the data pool in HIPE, HIPE doesn't "remember" which version this was or where it came from. It will be like any other data pool.

The Plug-in Framework makes this kind of sharing easier. The bundles that are shared are called plug-ins. Plug-ins can contain user scripts (such as pipeline scripts), initialization scripts (scripts that have to be run on every start-up of HIPE), pools and even software written in Java.

A plug-in, then, should be able to handle three kinds of "objects":

• one or more LocalStores
• JARs and Jython initialization scripts
• user scripts

The LocalStores are "made available" in HIPE by adding to the pools in the Data Access Preferences panel (see Edit --> Preferences).

The user scripts are scripts that a user should execute, such as a pipeline scripts. This contrasts with scripts that should be executed automatically when HIPE starts, such as scripts defining tasks. The latter is called an "initialization script".

User scripts that are contributed by plug-ins are added to a sub-menu of the Tools menu in HIPE, similar to the instrument menus under the Pipeline menu in HIPE.

The Jython initialization scripts should be executed on HIPE start-up. From Jython (the HIPE console) it is possible to import Java classes from the JARs from the plug-in. For this, the Java classes have to be available to Jython somehow. We also have to make sure that if a class is loaded, whether from Jython or from Java, that it is using the same classloader always. If a class is loaded twice with different classloaders, these are different classes, and objects of the two Class objects are not considered to be of the same class, they are not assignable to each other, etc. This functionality is implemented using the "context classloader" of the current thread (see Thread.setContextClassLoader(...)). This context classloader is used always, whether the thread is executing Jython commands, or not. Each plug-in has one and only one classloader (herschel.ia.gui.apps.PluginClassLoader), that is added to a herschel.ia.gui.apps.plugin.ChainClassLoader, which implements a chain of responsibility checking the available classloaders when a class needs to be loaded.

Any JARs can be contributed as a plug-in as an easier way of providing Java code to the system for people external to the HSC. Until now, there were three ways to add a JAR to the system: Either it has to be either accepted as a third-party library and added to the reference platform. Or the software could become part of the HCSS in the form of a module, in which case the software will be allowed to make use of the HCSS itself as well (libraries on the reference platform can only be referenced by the HCSS and cannot reference the HCSS themselves). Or the user of the library has to add the JAR to his/her classpath manually. This means that, in case of using the reference platform or an HCSS module, one will have to follow HSC procedures and most importantly, updates are only possible when the entire HCSS is updated, or manual actions from the user are required. In the case of plug-ins, a new version can be released and installed in HIPE as often as one pleases and installation is fully automatic.

A plug-in will also be able to provide configuration information about itself, all of it optional. This information goes into a configuration file plugin.xml, in the root directory of the plug-in. (The configuration file itself is completely optional, it is ok if it doesn't exist or if it's empty.)

Packaging and publishing plug-ins

A plug-in installable bundle, as a plug-in is published, is a JAR or a ZIP file. This file may contain the following items:

• XML file plugin.xml. This is the "deployment descriptor" containing all sorts of (optional) information about the plug-in, from simple things such as the plug-in description and homepage, to more advanced items like the location of a custom installer. Compatibility information should be included here as well (e.g. "compatible with HIPE v4.6 and later until and including 6.0").
• A class file for a class called Installer to execute the configuration phase of the plug-in. See the section on plug-in installation for details. This is thought to be most useful in combination with a custom installer.
• Jython script: plugin.py. This script is executed to "start" the plug-in (directly after installation and at HIPE start-up). It can be compared to the init.py for Java packages in the HCSS.
• Directory called jars. This directory should contain all jar files that should be added to the classpath.
• Directory called scripts. This directory should contain all Jython scripts that the plug-in contributes. These should be scripts that are intended for the user to execute, such as pipeline scripts. This is contrary to scripts like plugin.py which are executed by HIPE automatically.
• Directory called pools. This directory should contain all LocalStore pools that the plug-in contributes.
• License file LICENSE.txt
• Release notes in RELEASE_NOTES.txt

All the above elements are optional: It is perfectly ok to omit any one.

When the author has the plug-in ready, it can be sent to the HSC and we will add it to the public plug-ins Wiki DpHipePlugins. Also, inside HIPE's Plug-ins dialogue (Tools -> Plug-ins) there is a button "Find more...", which will open a browser window pointing to the public list Wiki page. This list page/web-site can be extended in the future, but for now we will use a manually maintained Wiki. If the maintenance gets out of hand, we can migrate to a more automated system, that allows authors to upload directly (for example).

Installing plug-ins

After the user specifies a URL where a plug-in is located, the installation will be performed completely automatically. The URL may point to the plug-in bundle (potentially on the local file system), or to the plug-in version registry. The plug-in version registry is an XML file on a web-server, maintained and updated by the plug-in author. It's explained in the section Updating Plug-ins. The installation action is implemented in herschel.ia.gui.apps.plugin.gui.actions.InstallAction.

The plug-in directory is by default under $HOME/.hcss/apps/hipe/plugins. This means that the installed plug-ins are preserved when, for example, a new version of HIPE is installed. Plug-ins are shared between all installed HIPE versions. Plug-ins are not shared between users: The framework currently provides no mechanism for a directory with plug-ins that is shared by multiple users.

The installation has two phases: bootstrap and custom. The bootstrap installation is implemented by HIPE and is executed always when a plug-in is installed, the custom phase is optional.

Installation phase 1: Bootstrap

During the bootstrap phase, the plug-in installable package is unjarred or unzipped into its proper directory. This directory exists under the default project directory .hcss. For a plug-in called xyz, version 1.2.3, the default directory would be .hcss/apps/hipe/plugins/xyz/1.2.3.

Installation phase 2: Custom installation

During the custom installation phase, code is executed that is contained in the plug-in. This should be a class called Installer in the default package (i.e. not in any package) implementing an interface called herschel.ia.gui.apps.plugin.Installer. This Installer must have a public zero-argument constructor. This step can be used to inform HIPE about any external JAR archives that may have been installed during the custom install phase.

If JARs are added to the classpath this way, they are recorded in a file called bundle.xml (created and maintained by HIPE). This file is used when the plug-in is loaded or activated (I use loading and activation as synonyms).

The custom installation is executed on the Swing Event Dispatch Thread (the "EDT"). Unless the custom installation specifically avoids this, HIPE will be blocked and not redrawn while the custom installation is in progress. If long operations are included in the custom installation process (meaning taking more than one or a few seconds), it is advisable to off-load this work to a background thread, for example using javax.swing.SwingWorker.

Once installed in the plug-ins directory, the plug-in will have a sub-directory jars (optionally). The plug-in directory will also, optionally, contain a bundle.xml listing jar files.

Loading/activating plug-ins

After installation, and on every HIPE start, the highest compatible version of all installed plug-ins is determined, and this version of the plug-in is activated. This is implemented using the PluginProcessor interface.

There are some default actions that need to be done for every plug-in:

• The plugin.py needs to be executed,
• any JARs need to be added to the classloader,
• pools need to added to the PoolManager,
• user scripts need to be added to the HIPE Tools menu (similarly to the instrument items under the Pipeline menu, pointing to that item will open the list of contributed scripts).

These actions are implemented by the DefaultProcessor. Other HIPE components may have to react on new plug-ins in different ways, and they can implement their own PluginProcessor. I don't believe that there are any implementations (yet), so this mechanism may seem overly complicated if there is just one implementation. But it helps to move the starting and stopping of the plug-ins out of the Plugin class, which has a tendency to become a God object.

Plug-ins panel

HIPE provides a general "plug-ins panel" to work with plug-ins: Install, remove, disable, etc. This is implemented in the package herschel.ia.gui.apps.plugin.gui. The panel is PluginPanel and the different activities are in the actions subpackage.

Selecting "Plug-ins" from the Tools menu, opens something similar to the Add-ons panel in Firefox, where a user can see the plug-ins he has installed, including:

• plug-in name
• description
• version
• plug-in homepage
• status (enabled/disabled)
• license
• custom properties

From this GUI one can enable/disable the plug-ins (requires HIPE restart). It is also possible to uninstall a plug-in (see next section).

Uninstalling plug-ins

We provide a default uninstaller in herschel.ia.gui.apps.plugin.DefaultInstaller. No mechanism to provide a custom uninstaller is currently implemented. The default uninstaller will simply delete the plug-in directory. A HIPE restart is required to remove any classes already loaded, so that the system will be clean.

Updating plug-ins

The user can "manually" upgrade a plug-in, simply by installing a later version. If so desired, earlier versions can be uninstalled first, but this is not necessary. On HIPE start-up, the highest compatible version of each plug-in will be started (unless it has been manually disabled). Therefore, installing a lower version of a plug-in than one that is installed already, will typically not have any effect.

The plug-in author also has the possibility to publish updates. An update can be a new version of a plug-in, or updated compatibility information. The compatibility information is initially provided in the plugin.xml and indicates the application versions that the plug-in is compatible with. The update mechanism allows to update this information, so that when a new version of HIPE is released, and it is found to be compatible, this version can be included. If a new version of the plug-in is published, HIPE will offer the user an automatic update. The mechanism is described below.

A plug-in installable package, or a bundle, contains a plugin.xml file. This XML file should indicate compatibility with versions of the software. The compatibility is based on HIPE version track and build number. For example, release 4.6 has version track 4.0 and build 1467. The application will only know that it is release 4.6 if it was installed with the user installer. This is not always the case (mainly for developers), so for the compatibility checks to work, they are made on the track and build number. When a plug-in is installed, the compatibility information is copied to the bundle.xml, which is a file with additional configuration information that is not delivered as-is with the plug-in, but managed by HIPE.

Aside from the plug-in packages, the plug-in author can publish an XML file, typically by putting it on a web-server, containing a listing of version items for one or more plug-ins. This file we call the plug-in registry and the class representing it is herschel.ia.gui.apps.plugin.update.Manifest. The ManifestParser parses the XML file and creates a Manifest object containing the information. Each version item (an Entry) has a link to the plug-in installable package and compatibility information (see the class Compatibility). For this mechanism to work, the plugin.xml of every plug-in installed in HIPE should contain a link to it. At every start-up, HIPE checks this registry for every plug-in that is installed. If multiple plug-ins use the same registry, this registry is checked only once.

If a plug-in that is installed is found in the registry, its compatibility information is copied to the bundle.xml. The information in the registry has precedence when determining the compatibility of a plug-in. If a registry can't be contacted or a plug-in (in the installed version) doesn't occur in the registry, the local information is used. A plug-in is started at HIPE start-up if it is compatible with the HIPE version that is being started. If the compatibility information is modified, several things can happen:

• The plug-in was incompatible before the update and is still incompatible. The plug-in was not started and will not be started.
• The plug-in was compatible before the update and is still compatible. The plug-in was started and will be left running.
• The plug-in was incompatible before the update, but after the update it becomes compatible (according to the compatibility information). The plug-in was not started, but will be started after the update.
• The plug-in was compatible before the update, but after the update it becomes incompatible. The plug-in was started, though this may have failed because it's not compatible. The plug-in will be stopped.

In the last two cases, a HIPE restart is recommended to guarantee proper functioning of the application. Even though the checks are performed at HIPE start-up, we do not want the application start-up to wait until the checks are completed. Therefore, we have to assume that the application is completely up and running by the time the compatibility checks complete. As plug-ins can only be guaranteed to function correctly if they are started as part of the HIPE start-up process, a restart is required to be sure the plug-ins work.

If a higher version of a plug-in is found in a registry, that is still compatible with the current version of the application (HIPE), then an automatic upgrade will be offered to the user. HIPE will have to be restarted after the upgrade.

Incompatible plug-ins

Plug-ins can be incompatible based on the application version and the compatibility information either in the plugin.xml or as published in the plug-in registry (see previous section). If it happens that a plug-in is compatible with HIPE up to and including version 5.0, and the user is currently running 6.0, then the plug-in will be tagged in memory as "not compatible". Contrary to the "disabled" state, this state is not persisted to disk and has to be determined each time, because different versions of HIPE may be running at the same time, looking at the same files on disk.

Working with Plug-ins in HIPE

See the relevant section of the Developer Manual.

Open issues

Plug-in dependencies: Can a plug-in require for example that the PACS classes are installed? If we are talking about a PACS calibration data plug-in, should this disable itself if it finds we are running HIPE with no PACS classes?

Do we need the possibility to specify a custom uninstaller?

Read-only access to plug-in files: Currently there is no protection against the user modifying the scripts and pools that are delivered with a plug-in. Should this be protected?

Installer failures: If the custom installer (for example WebStart installer) fails somehow, or is cancelled, currently there is no way to pick this up.

Further information

For any questions regarding plug-in development, please send an email to Alvar.Garcia@sciops.esa.int.

Original author: Paul Balm - 16 Aug 2010