Plug-in Framework Design Document

This document describes the design choices underpinning the Plug-in Framework in HIPE. The SCR that originally requested the Plug-in Framework was HcssScr:9471.

What a Plug-in is and can do

Many people write software either in Java or in Jython and then want to share this software with other people. Sometimes this software is a script that has to be run manually, sometimes it has to be run every time HIPE is started (for example if it defines some Tasks). Or people may generate pools of data that they want to share.

This software and this data can be zipped and put on a web-server. If the zip-file has a version in it, this version is lost when the zip-file is unpacked: After importing the data pool in HIPE, HIPE doesn't "remember" which version this was or where it came from. It will be like any other data pool.

The Plug-in Framework makes this kind of sharing easier. The bundles that are shared are called plug-ins. Plug-ins can contain user scripts (such as pipeline scripts), initialization scripts (scripts that have to be run on every start-up of HIPE), pools and even software written in Java.

A plug-in, then, should be able to handle three kinds of "objects":

  • one or more LocalStores
  • JARs and Jython initialization scripts
  • user scripts

The LocalStores are "made available" in HIPE by adding to the pools in the Data Access Preferences panel (see Edit --> Preferences).

The user scripts are scripts that a user should execute, such as a pipeline scripts. This contrasts with scripts that should be executed automatically when HIPE starts, such as scripts defining tasks. The latter is called an "initialization script".

User scripts that are contributed by plug-ins are added to a sub-menu of the Tools menu in HIPE, similar to the instrument menus under the Pipeline menu in HIPE.

The Jython initialization scripts should be executed on HIPE start-up. From Jython (the HIPE console) it is possible to import Java classes from the JARs from the plug-in. For this, the Java classes have to be on the Jython system path somehow. To be able to add the jars to the Jython path (such that they will be picked up), the jars have to be unpacked. Jython 2.1 does not appear to be capable of dealing with JARs that are added to the system path dynamically (i.e. using Jython commands and not by setting the CLASSPATH before start-up). The jars are unpacked in a directory called "work" in the plug-in directory. Therefore, the plug-in is not allowed to have a directory called "work" already from itself, for this will interfere.

Any JARs can be contributed as a plug-in as an easier way of providing Java code to the system for people external to the HSC. Until now, there were three ways to add a JAR to the system: Either it has to be either accepted as a third-party library and added to the reference platform. Or the software could become part of the HCSS in the form of a module, in which case the software will be allowed to make use of the HCSS itself as well (libraries on the reference platform can only be referenced by the HCSS and cannot reference the HCSS themselves). Or the user of the library has to add the JAR to his/her classpath manually. This means that, in case of using the reference platform or an HCSS module, one will have to follow HSC procedures and most importantly, updates are only possible when the entire HCSS is update, or manual actions from the user are required. In the case of plug-ins, a new version can be released and installed in HIPE as often as one pleases and installation is fully automatic.

A plug-in will also be able to provide configuration information about itself, all of it optional. This information goes into a configuration file plugin.xml, in the root directory of the plug-in. (The configuration file itself is completely optional, it is ok if it doesn't exist or if it's empty.)

Packaging and publishing plug-ins

A plug-in installable bundle, as a plug-in is published, is a JAR or a ZIP file. This file has the following contents:

  • XML file plugin.xml. This is the "deployment descriptor" containing all sorts of (optional) information about the plug-in, from simple things such as the plug-in description and homepage, to more advanced items like the location of a custom installer. Compatibility information should be included here as well (e.g. "compatible with HIPE v4.6 and later until and including 6.0").
  • A class file for a class called Installer to execute the configuration phase of the plug-in. See the section on plug-in installation for details. This is thought to be most useful in combination with a custom installer.
  • Jython script: plugin.py. This script is executed to "start" the plug-in (directly after installation and at HIPE start-up). It can be compared to the init.py for Java packages in the HCSS.
  • Directory called jars. This directory should contain all jar files that should be added to the classpath.
  • Directory called scripts. This directory should contain all Jython scripts that the plug-in contributes. These should be scripts that are intended for the user to execute, such as pipeline scripts. This is contrary to scripts like plugin.py which are executed by HIPE automatically.
  • Directory called pools. This directory should contain all LocalStore pools that the plug-in contributes.
  • License file LICENSE.txt

All the above elements are optional: It is perfectly ok to omit any one.

When the author has the plug-in ready, it can be sent to the HSC and we will add it to the public plug-ins Wiki. Also, from HIPE there will be a button "Find plug-ins", which will open a browser window pointing to this Wiki page. This page/web-site can be extended in the future, but for now we will use a manually maintained Wiki: DpHipePlugins. If the maintenance gets out of hand, we can migrate to a more automated system, that allows authors to upload directly (for example).

Installing plug-ins

After the user specifies a URL where a plug-in is located, the installation will be performed completely automatically. It will be possible as well to browse for the plug-in bundle on the local file system. The installation has three phases: bootstrap, custom, and configuration. The bootstrap installation is implemented by HIPE and is executed always when a plug-in is installed, the other two phases are optional.

The plug-in directory is by default under $HOME/.hcss/apps/hipe/plugins. This means that the installed plug-ins are preserved when, for example, a new version of HIPE is installed. Plug-ins are shared between all installed HIPE versions. Plug-ins are not shared between users: The framework currently provides no mechanism for a directory with plug-ins that is shared by multiple users.

Installation phase 1: Bootstrap

The first step during the bootstrap phase to perform the first basic installation (basically, unjarring or unzipping in the plug-in directory). After this, the various contributions from the plug-in are added to HIPE: For contributed JARs, nothing special needs to be done, since they will be added to the classpath dynamically on every HIPE start-up (from the next restart on). LocalStores are added in-place to the HIPE Preferences under Data Access. This means that an entry for the LocalStore is added to the pool in the Preferences, which "points" to the LocalStore inside the plug-in. The LocalStore is not copied. The same for contributed scripts.

Installation phase 2: Custom installation

The custom install phase allows for the plug-in author to specify a completely custom installation program. The system offers two ways of doing this: By specifying the URL of an installer that can be started using Java WebStart, or by including the installer as a Java program in the plug-in.

The WebStart URL is specified in the plug-in deployment descriptor, plugin.xml. For example, CASSIS is a program that has its own installer. This installer will be executed during this phase.

The other possibility is to specify the class name of the main class in the plugin.xml configuration file. If this main class implements the interface herschel.ia.gui.apps.plugin.Installer, then the install method of this class will be called. The advantage of this is that the install method is passed a PluginEnv object. This PluginEnv object can be used to add JAR files to the classpath. Normally this will not be necessary: Any JARs that exist in the jars directory in the plug-in are added to the classpath automatically. But if the custom installer installs JARs in a different location, you will want to access these JARs from HIPE. To do so, they must be added to the classpath. This work can also be done during the configuration phase (phase 3 -- see next paragraph).

Another advantage of having access to the PluginEnv object can be that it gives access to several plug-in attributes (custom properties, license text), though most of this will already be known and not particularly useful at this stage.

If the custom installer main class does not implement herschel.ia.gui.apps.plugin.Installer, then its main method will be called. If it doesn't have a main method either, then an error is generated.

Installation phase 3: Configuration

Finally, during the configuration phase, code is executed that is contained in the plug-in. This should be a class called Installer in the default package (i.e. not in any package) implementing an interface called herschel.ia.gui.apps.plugin.Installer. This Installer must have a public zero-argument constructor. This step can be used to inform HIPE about any external JAR archives that may have been installed during the custom install phase.

If JARs are added to the classpath this way, they are recorded in a file called bundle.xml and used when the plug-in is loaded or activated (I use loading and activation as synonyms).

Loading/activating plug-ins

Once installed in the plug-ins directory, the plug-in will have a sub-directory jars (optionally). The plug-in directory will also, optionally, contain a bundle.xml listing jar files.

HIPE will construct a custom classloader that includes the jars in the jars directory in the plug-in directory, as well as the jars listed in the bundle.xml. All these jars will then also be dynamically added to the Python system path, for which they have to be unpacked (in Jython 2.1 – probably no need to unpack with Jython 2.5). The JARs will be unpacked into the work directory in the plug-in directory. Note that this means you have two copies of the class-files: Once in the JAR and once unpacked in the work-directory. The JARs are used from Java and the unpacked class files are used from Jython. This is not optimal, but it's a current limitation of Jython (2.1).

If any scripts are contained in the plug-in, a menu item with the name of the plug-in will be added to the Tools menu. Similarly to the instrument items under the Pipeline menu, pointing to that item will open the list of contributed scripts.

Finally, HIPE will run the Jython initialization script in the plug-in directory (hard-coded to be plugin.py).

Plug-in status panel

Selecting "Plug-ins" from the Tools menu, opens something similar to the Add-ons panel in Firefox, where a user can see the plug-ins he has installed, including:

  • plug-in name
  • description
  • version
  • plug-in homepage
  • status (enabled/disabled)
  • license
  • custom properties

From this GUI one can enable/disable the plug-ins (requires HIPE restart). It is also possible to uninstall a plug-in (see next section).

Uninstalling plug-ins

We provide a default uninstaller. No mechanism to provide a custom uninstaller is currently implemented. The default uninstaller will simply delete the plug-in directory. A HIPE restart is required to remove any classes already loaded, so that the system will be clean.

Updating plug-ins

The user can "manually" upgrade a plug-in, simply by installing a later version. If so desired, earlier versions can be uninstalled first, but this is not necessary. If a user installs an earlier version,

The plug-in author also has the possibility to publish updates. An update can be a new version of a plug-in, or updated compatibility information. The compatibility information is initially provided in the plugin.xml and indicates the application and application versions that the plug-in is compatible with. The update mechanism allows to update this information, so that when a new version of HIPE is released, and it is found to be compatible, this version can be included. If a new version of the plug-in is published, HIPE will offer the user an automatic update. The mechanism is described below.

A plug-in installable package, or a bundle, contains a plugin.xml file. This XML file should indicate compatibility with versions of the software. The compatibility is based on HIPE version track and build number. For example, release 4.6 has version track 4.0 and build 1467. The application will only know that it is release 4.6 if it was installed with the user installer. This is not always the case (mainly for developers), so for the compatibility checks to work, they are made on the track and build number. When a plug-in is installed, the compatibility information is copied to the bundle.xml, which is a file with additional configuration information that is not delivered as-is with the plug-in, but managed by the application.

Aside from the plug-in packages, the plug-in author can publish an XML file, containing a listing of version items for one or more plug-ins. Each version item has a link to the plug-in installable package and compatibility information. We call this file the "plug-in registry". At every start-up of HIPE, HIPE checks this registry for every plug-in that is installed. If multiple plug-ins use the same registry, this registry is checked only once.

If a plug-in that is installed is found in the registry, its compatibility information is copied to the bundle.xml. The information in the registry is leading when determining the compatibility of a plug-in. If a registry can't be contacted or a plug-in (in the installed version) doesn't occur in any registry, the local information is used. A plug-in is started at HIPE start-up if it is compatible with the HIPE version that is being started. If the compatibility information is modified, several things can happen:

  • The plug-in was incompatible before the update and is still incompatible. The plug-in was not started and will not be started.
  • The plug-in was compatible before the update and is still compatible. The plug-in was started and will be left running.
  • The plug-in was incompatible before the update, but after the update it becomes compatible (according to the compatibility information). The plug-in was not started, but will be started after the upgrade.
  • The plug-in was compatible before the update, but after the update it becomes incompatible. The plug-in was started, though this may have failed because it's not compatible. The plug-in will be stopped.

In the last two cases, a HIPE restart is recommended to guarantee proper functioning of the application. Even though the checks are performed at HIPE start-up, we do not want the application start-up to wait until the checks are completed. Therefore, we have to assume that the application is completely up and running by the time the compatibility checks complete. As plug-ins can only be guaranteed to function correctly if they are started as part of the HIPE start-up process, a restart is required to be sure the plug-ins work.

If a higher version of a plug-in is found in a registry, that is still compatible with the current version of the application (HIPE), then an automatic upgrade will be offered to the user. HIPE will have to be restarted after the upgrade.

Incompatible plug-ins

Plug-ins can be incompatible based on the application version and the compatibility information either in the plugin.xml or as published in the plug-in registry (see previous section). If it happens that a plug-in is compatible with HIPE up to and including version 5.0, and the user is currently running 6.0, then the plug-in will be tagged in memory as "not compatible". Contrary to the "disabled" state, this state is not persisted to disk and has to be determined each time, because different versions of HIPE may be running at the same time, looking at the same files on disk.

Working with Plug-ins in HIPE

See the relevant section of the Developer Manual.

Open issues

Plug-in dependencies: Can a plug-in require for example that the PACS classes are installed? If we are talking about a PACS calibration data plug-in, should this disable itself if it finds we are running HIPE with no PACS classes?

If plug-ins disable themselves if they find that they are not compatible with the HIPE version or project (hcss, hcss.dp.core or hcss.dp.pacs), should they go through the list of the plug-ins again when we run a different HIPE version? I believe Firefox does this: It checks your plug-ins when you start a new Firefox that is a different version from the one you ran yesterday. This requires that you somehow store the version of HIPE that you are running in the plugins directory, or it requires that you check all on every start-up. But the latter defeats the purpose of disabling plug-ins and writing this to disk.

Do we need the possibility to specify a custom uninstaller?

Read-only access to plug-in files: Currently there is no protection against the user modifying the scripts and pools that are delivered with a plug-in. Should this be protected?

Auto-update: When a new version of a plug-in comes on-line, does HIPE discover this automatically?

Installer failures: If the custom installer (for example WebStart installer) fails somehow, or is cancelled, currently there is no way to pick this up.

-- PaulBalm - 16 Aug 2010

Edit | Attach | Watch | Print version | History: r8 | r6 < r5 < r4 < r3 | Backlinks | Raw View | Raw edit | More topic actions...
Topic revision: r4 - 2011-01-12 - PaulBalm
 
This site is powered by the TWiki collaboration platform Powered by Perl