Http Pool

This page serves as the main documentation for running an HTTP server pool. The corresponding client pool is contained in the build and not covered here other than in brief. Most of this page is about configuring Tomcat to run servlets using the HCSS in general, rather than specifically HttpPool.

These instructions have been updated for HCSS 9.0.

Installation

The server pool requires a web server that supports Java servlets. This page assumes that Tomcat is used for this purpose. It should work with the latest version (10.0.1443 at the time of writing). The first step then is to install Tomcat, although adding the pool servlet to an existing installation is also fine. Note that this does not detail all possible configurations that experts may wish to set up, but a single simple one.

Note that only one HTTP pool server is needed, regardless of the number of pools it accesses.

There is no need to install Tomcat in a privileged account, and probably good security reasons for not doing so. Set the environment variable CATALINA_HOME to point to the directory where it is installed. You can also use the CATALINA_OPTS variable to set JVM options and HCSS configuration as explained later. This should be used to increase the available memory. My settings look like this:

CATALINA_HOME=/home/spire/hcssbld/tomcat
CATALINA_OPTS="-server -Xmx4096m -Dhcss.init.logging=false -Dvar.hcss.dir=$CATALINA_HOME/webapps/hcss/WEB-INF"

Note: Because the startup script of tomcat will call another script to launch the tomcat jvm, you must export the environment variable with the export command (or other shell equivalent).

The global configuration files are in Tomcat's conf directory. You should only need to make minor changes to server.xml. The default port is 8080 and defined in the server.xml file. You can change this if you like, but running on a privileged port makes things a bit more complicated, see below.

User Access Control

I recommend enabling this for two reasons:

1. If you do not your pools will be open to the world...
2. It makes troubleshooting easier as you can see who is doing what in the logs.

There are several ways to set this up with Tomcat. I use the simplest, which is what is described here. Its main disadvantage is that it does not scale well to large numbers of users and major enterprise systems. See the Tomcat documentation for other possibilities.

Edit the server.xml file in the conf directory. Look for the Realm entry and specify digest as SHA. It should now look like this:

      <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
             resourceName="UserDatabase" digest="SHA"/>

Edit the tomcat-users.xml file in the conf directory. Add some roles, like this:

  <role rolename="spire_admin"/>
  <role rolename="spire_user"/>

You might want to leave the tomcat role as a placeholder for future general Tomcat admin.

This is how to add a user.

1. Get a user name and password. Passwords should not be the same as login passwords, as while they are encrypted, the means used is not particularly secure. There are no restrictions on passwords. The main purpose is to offer some protection and log who is doing what and when.
2. Encrypt the password. The command is java -cp $CATALINA_HOME/lib/catalina.jar:$CATALINA_HOME/bin/tomcat-juli.jar org.apache.catalina.realm.RealmBase -a SHA password. You may wish to define an alias for this...
3. Edit the file tomcat-users.xml and add a new user line. password should be set to the encrypted password and roles should normally be set to the user role above (give yourself admin as well).

The server can now be configured for either "Basic" or "Digest" authentication and the clients will automatically pick up the right one. If configured for "Digest" then encrypting the passwords with SHA does not work. In this case you should specify "MD5" instead of "SHA" and encrypt the string "user:realm:password" instead of just "password". Here is my script for doing the encryption.

Adding a servlet

Servlets live inside the webapps directory. Applications have their own area in this directory and run in a sandbox isolated from the others. Note that a single application area can contain any number of servlets.

Now we go through how to set up an application called hcss. In the webapps directory, create a directory structure like this:

hcss - WEB-INF - lib
               - classes

The general idea is to put jar files in the lib directory and class files in classes. Everything you need should be in there. Don't assume that it will pick up your CLASSPATH environment variable - it won't.

These libraries should be updated whenever a new version of the HCSS is deployed, although in practice it usually continues working anyway (except for schema evolutions which kill database accessing servlets).

It is also necessary to create the application configuration file web.xml in WEB-INF. A version with a servlet defined is in the next section. You can also look at the versions of this file in the other applications that come preinstalled with Tomcat.

The HTTP pool servlet

Make sure that the hcss.services project of the CIB. is installed. This project contains these modules:

• ia_server_util (utility classes)
• ia_pal_pool_http_server (the PAL HTTP server)
• access_server (telemetry and data frame server)

While the static dependencies for the servlets can be identified - and this will work for the TM server - the runtime dependencies for the PAL HTTP pool cannot. It is therefore necessary to copy most of the jar files into the servlet directory. Fortunately a program is provided for this purpose in the bin directory of your installed build. This program comes with the hcss.services project, which is included in the hcss project. The program is called copyBuildJars. It is used as follows:

rm $CATALINA_HOME/webapps/hcss/WEB-INF/lib/*.jar
copyBuildJars $CATALINA_HOME/webapps/hcss/WEB-INF/lib

The servlet itself is in the ia_pal_pool_http_server jar file.

Now copy the project.xml file from the build directory into the WEB-INF directory. The property var.hcss.dir needs to be set to the same directory. It can be set in CATALINA_OPTS, see example above. This is needed for the correct project info to be passed from server to client. (It doesn't work in v3.0, see HcssSpr:10188, fixed in v4.0).

Application configuration

Now create the web.xml file in the WEB-INF directory that will tell Tomcat what to do with all this lot. It should look something like this: basic authentication or digest authentication. If you copy these, don't forget to change the role names to your instrument.

This example does two things in addition to defining the servlet:

1. It sets up user authentication for this area (note that different authentication can be applied to different application areas).
2. It defines a compression filter. This is for performance reasons; the effect is TBC.

Logging configuration

Tomcat will normally by default write log messages to catalina.out, but it is generally better for applications to write to their own logs. However the reinitialisation of Java logging performed by the Configuration class can interfere with Tomcat's logging and prevent it from working properly, see HcssScr:6672.

In the classes directory under hcss create a file called logging.properties like this:

handlers = org.apache.juli.FileHandler, java.util.logging.ConsoleHandler

############################################################
# Handler specific properties.
# Describes specific configuration info for Handlers.
############################################################

org.apache.juli.FileHandler.level = FINE
org.apache.juli.FileHandler.directory = ${catalina.base}/logs
org.apache.juli.FileHandler.prefix = hcss.
org.apache.juli.FileHandler.formatter = herschel.share.log.api.StandardFormatter

java.util.logging.ConsoleHandler.level = FINE
java.util.logging.ConsoleHandler.formatter = herschel.share.log.api.StandardFormatter

The fix for HcssScr:6672 is only implemented as of HCSS 4.0.635. If an earlier version than this is being used, then for this logging configuration to work properly it is necessary to update the Configuration.class file. How to do this is explained in the Trouble Shooting section below.

The system property hcss.init.logging should be set to false. It will not work to add it user.props or any other HCSS property file! You can add it to CATALINA_OPTS, as in the example setting of this variable above.

The hcss application area should now log to a file called hcss.date.log.

Starting and stopping

The scripts can be found in Tomcat's bin directory. Use startup to start it, and shutdown to stop it. You may wish to check the catalina.out log to see if it started correctly.

Tomcat should automatically notice if a new servlet is defined and load it. You can force a servlet to reload by starting and stopping Tomcat. Wait a few seconds between stopping and starting or it might get confused (see next section). There is also a Tomcat "Manager" application that can do this from a web interface without stopping Tomcat - see the Tomcat documentation for details.

Occasionally it can get in a mess, which can be caused by things such as running out of memory, and refuse to restart. In this case you should see a message in the log along the lines of:

LifecycleException:  Protocol handler initialization failed: java.net.BindException: Address already in use:8080

In this case you need to kill the process (look for catalina) before restarting it.

HCSS properties configuration

Here are guidelines for ensuring that HCSS properties are set as you expect.

1. You need only one property set, and that is var.hcss.dir within CATALINA_OPTS. It is recommended that this be set to the WEB-INF directory of the application.
2. Create a directory config/properties in that directory.
3. Place any desired property files in that directory. They must have the extenstion .properties.

Note that the user.props of the user owning the Tomcat process will also be read if it exists and with higher precedence. You might not want this. This is one reason for running the server from a dedicated account rather than a standard user account.

Trying it out

Following the instructions above and starting Tomcat, we should now have a running PAL server with url http://whatever:8080/hcss/pal. Check the log for any initialisation errors. Here is an example Jython script for trying it out on the client side:

from herschel.ia.pal.pool.http import HttpClientPool
pool = HttpClientPool ("http://wakefield.bnsc.rl.ac.uk/hcss/pal", "sims3")
st = ProductStorage (pool)
st.authenticate()
r = st.select (Query (ObservationContext, "1"))

Updating an existing installation

This is all that is necessary to update to a new HCSS version:

cd $CATALINA_HOME/webapps/hcss/WEB-INF/
rm lib/*.jar
copyBuildJars lib/
cp <hcss-dir>/project.xml .

Then restart the server.

If you get an error because sun.grid/8.0.1/drmaa_8.0.1 is the wrong number of bytes then edit <hcss-dir>/installed.userlibraries, search for drmaa and delete that line. This was a bug and is fixed in 10.0.1423.

Troubleshooting

It is important to understand that the Tomcat scripts do not use the Java CLASSPATH environment variable. All required resources must be present in the classes or lib directories. For full details read the section on "Classloading" in the Tomcat documentation.

The first rule of troubleshooting is to check the log file. The principle Tomcat log file is $CATALINA_HOME/logs/catalina.out. With the logging configuration above, the HCSS applications will log to hcss.date.log. Don't forget that what is written to the log can be configured as desired.

Sometimes there is no substitute for putting debugging statements in the code. You can do this by making modifications in a developer environment as normal, and then copying the class file(s) into the classes directory of the servlet. For example:

cvs co ia_pal_pool_http_server
edit HttpPoolServlet.java
jake

Then copy the output HttpPoolServlet.class file to $CATALINA_HOME/webapps/hcss/classes/herschel/ia/pal/pool/http/server.

If a needed library is somehow missing, it is possible to be plagued by the dreaded NoClassDefFoundError. The class reported to be missing is not necessarily the one that is. If you know it's there, follow the dependency trail from the reported class to look for something missing. Pay particular attention to what is used in static initialisation statements.

Running on a privileged port

This is normally the standard HTTP port 80. The port is set by editing the server.xml file in the conf directory. I was not able to get the default Tomcat startup script to work when installed on this port. Here is my script.

This script must be started with root privilege. Note that the daemon still runs as a less privileged user.

Further complications arise if there is a need to access the Versant library, as this accesses native shared libraries, e.g. for DbPool or the TM/data frame server. Whilst the startup script uses the java.library.path argument to pass the location of these libraries to the daemon process, this does not entirely work since the first called library then calls other libraries, which does not work since the process does not inherit the value of the LD_LIBRARY_PATH environment variable (it defines its own). A workaround solution is to link to the required libraries from the Java installation, which is accessible, e.g.

cd  $JAVA_HOME/jre/lib/amd64
ln -s $VERSANT_ROOT/lib/liboscfe.so liboscfe.so
ln -s $VERSANT_ROOT/lib/libxa.so libxa.so

Note that this is only necessary since the server is running as a daemon process in order to access the privileged port 80. There may well be a better solution than this.

Using the telemetry and data frame server

Put the classes for the access_server module in the hcss applications area, as for the PAL pool.

Then add this servlet definition to the web.xml file in WEB-APPS.

    <servlet>
        <servlet-name>tm</servlet-name>
        <servlet-class>herschel.access.server.HcssTmHttpServer</servlet-class>
    </servlet>
    <servlet-mapping>
        <servlet-name>tm</servlet-name>
        <url-pattern>/tm</url-pattern>
    </servlet-mapping>

This corresponds to the url http://whatever:8080/hcss/tm, assuming default settings.

This servlet has a feature to allow redirection of a database request to a specific server. This is done by means of HCSS properties. (It might be cleaner to redefine them as parameters specified in the web.xml file). The easiest way to set them is to put them in a HcssTmHttpServer.defaults file in the same directory as the corresponding class file. These are:

hcss.access.server.default

If a server is not specified, use this one. It itself defaults to localhost.

hcss.access.server.redirect

Redirect all database server requests to this one. The purpose of this is to ensure that operationally critical servers cannot be accessed by this process; the data of course has to be present on the server redirected to.

This property should also be set on the server side. This will be made the default for this application in the next version of the module. Note that this value is specific to the server and inappropriate for interactive use.

hcss.access.store = herschel.access.db.SimpleStoreHandler

This servlet is accessed by using the normal access API. These client-side properties need to be set:

hcss.access.connection=herschel.access.net.NetworkConnection
hcss.access.authentication=true
hcss.access.url=http://whatever:8080/hcss/tm

Convenience methods to set these properties dynamically are provided in the herschel.access.Access class.

Using Tomcat's Manager application

This is a very useful tool for monitoring, stopping and restarting applications. To use it you have to give yourself privilege. Either:

• Add the manager role to tomcat-users.xml and add the role to your username.
• Edit the web.xml file for the manager application and change the manager role to spire_admin or whatever.

Then go to the "manager/html" web page on your server. I found that with my configuration I was getting HTTP 401 errors when I tried this. I got it to work by commenting out the error-page entry in the manager web.xml file.

-- SteveGuest - 03 Aug 2012