HttpPool < Public

Public Web>HttpPool (2012-10-19, DavideRizzo) (raw view)
<!-- ANALYTICS CODE - DO NOT EDIT -->
%INCLUDE{"%ATTACHURL%/GoogleAnalytics.txt" raw="on"}%
<!-- END OF ANALYTICS CODE -->
---++ Http Pool

This page serves as the main documentation for running an HTTP _server_ pool. The corresponding client pool is contained in the build and not covered here other than in brief. Most of this page is about configuring Tomcat to run servlets using the HCSS in general, rather than specifically !HttpPool.

These instructions have been updated for HCSS 9.0.

---+++ Installation

The server pool requires a web server that supports Java servlets. This page assumes that [[http://tomcat.apache.org/][Tomcat]] is used for this purpose. It should work with the latest version (10.0.1443 at the time of writing). The first step then is to install Tomcat, although adding the pool servlet to an existing installation is also fine. Note that this does not detail all possible configurations that experts may wish to set up, but a single simple one.

Note that only one HTTP pool server is needed, regardless of the number of pools it accesses.

There is no need to install Tomcat in a privileged account, and probably good security reasons for not doing so. Set the environment variable =CATALINA_HOME= to point to the directory where it is installed. You can also use the =CATALINA_OPTS= variable to set JVM options and HCSS configuration as explained later. This should be used to increase the available memory. My settings look like this:
<verbatim>
CATALINA_HOME=/home/spire/hcssbld/tomcat
CATALINA_OPTS="-server -Xmx4096m -Dhcss.init.logging=false -Dvar.hcss.dir=$CATALINA_HOME/webapps/hcss/WEB-INF"
</verbatim>

Note: Because the startup script of tomcat will call another script to launch the tomcat jvm, you must *export* the environment variable with the _export_ command (or other shell equivalent).

The global configuration files are in Tomcat's =conf= directory. You should only need to make minor changes to =server.xml=. The default port is 8080 and defined in the =server.xml= file. You can change this if you like, but running on a privileged port makes things a bit more complicated, see below.

---++++ User Access Control

I recommend enabling this for two reasons:
   1. If you do not your pools will be open to the world...
   1. It makes troubleshooting easier as you can see who is doing what in the logs.

There are several ways to set this up with Tomcat. I use the simplest, which is what is described here. Its main disadvantage is that it does not scale well to large numbers of users and major enterprise systems. See the Tomcat documentation for other possibilities.

Edit the =server.xml= file in the =conf= directory. Look for the =Realm= entry and specify =digest= as =SHA=. It should now look like this:
<verbatim>
      <Realm className="org.apache.catalina.realm.UserDatabaseRealm"
             resourceName="UserDatabase" digest="SHA"/>
</verbatim>

Edit the =tomcat-users.xml= file in the =conf= directory. Add some roles, like this:
<verbatim>
  <role rolename="spire_admin"/>
  <role rolename="spire_user"/>
</verbatim>
You might want to leave the =tomcat= role as a placeholder for future general Tomcat admin.

This is how to add a user.
   1. Get a user name and password. Passwords should _not_ be the same as login passwords, as while they are encrypted, the means used is not particularly secure. There are no restrictions on passwords. The main purpose is to offer some protection and  log who is doing what and when.
   1. Encrypt the password. The command is =java -cp $CATALINA_HOME/lib/catalina.jar:$CATALINA_HOME/bin/tomcat-juli.jar org.apache.catalina.realm.RealmBase -a SHA password=. You may wish to define an alias for this...
   1. Edit the file =tomcat-users.xml= and add a new user line. =password= should be set to the encrypted password and =roles= should normally be set to the user role above (give yourself admin as well).

%N% The server can now be configured for either "Basic" or "Digest" authentication and the clients will automatically pick up the right one. If configured for "Digest" then encrypting the passwords with SHA  _does not work_. In this case you should specify "MD5" instead of "SHA" and encrypt the string "user:realm:password" instead of just "password". Here is my [[%ATTACHURL%/genpwd.sh][script]] for doing the encryption.

---+++ Adding a servlet
Servlets live inside the =webapps= directory. Applications have their own area in this directory and run in a sandbox isolated from the others. Note that a single application area can contain any number of servlets.

Now we go through how to set up an application called =hcss=. In the =webapps= directory, create a directory structure like this:
<verbatim>
hcss - WEB-INF - lib
               - classes
</verbatim>

The general idea is to put =jar= files in the =lib= directory and =class= files in classes. Everything you need should be in there. Don't assume that it will pick up your =CLASSPATH= environment variable - it won't. 

These libraries should be updated whenever a new version of the HCSS is deployed, although in practice it usually continues working anyway (except for schema evolutions which kill database accessing servlets). 

It is also necessary to create the application configuration file =web.xml= in =WEB-INF=. A version with a servlet defined is in the next section. You can also look at the versions of this file in the other applications that come preinstalled with Tomcat.

---++++ The HTTP pool servlet
Make sure that the =hcss.services= project of the CIB. is installed. This project contains these modules:
   * =ia_server_util= (utility classes)
   * =ia_pal_pool_http_server= (the PAL HTTP server)
   * =access_server= (telemetry and data frame server)

While the static dependencies for the servlets can be identified - and this will work for the TM server - the runtime dependencies for the PAL HTTP pool cannot. It is therefore necessary to copy most of the =jar= files into the servlet directory. Fortunately a program is provided for this purpose in the bin directory of your installed build. This program comes with the =hcss.services= project, which is included in the =hcss= project. The program is called =copyBuildJars=. It is used as follows:

<verbatim>
rm $CATALINA_HOME/webapps/hcss/WEB-INF/lib/*.jar
copyBuildJars $CATALINA_HOME/webapps/hcss/WEB-INF/lib
</verbatim>
The servlet itself is in the =ia_pal_pool_http_server= jar file.

Now copy the =project.xml= file from the build directory into the =WEB-INF= directory. The property =var.hcss.dir= needs to be set to the same directory. It can be set in =CATALINA_OPTS=, see example above. This is needed for the correct project info to be passed from server to client. (It doesn't work in v3.0, see HcssSpr:10188, fixed in v4.0).

---+++ Application configuration

%U% Now create the =web.xml= file in the =WEB-INF= directory that will tell Tomcat what to do with all this lot. It should look something like this: [[%ATTACHURL%/basic.xml][basic authentication]] or  [[%ATTACHURL%/digest.xml][digest authentication]]. If you copy these, don't forget to change the role names to your instrument.

This example does two things in addition to defining the servlet:
   1. It sets up user authentication for this area (note that different authentication can be applied to different application areas).
   2. It defines a compression filter. This is for performance reasons; the effect is TBC.

---+++ Logging configuration

Tomcat will normally by default write log messages to =catalina.out=, but it is generally better for applications to write to their own logs. _However the reinitialisation of Java logging performed by the Configuration class can interfere with Tomcat's logging and prevent it from working properly_, see HcssScr:6672.

In the =classes= directory under =hcss= create a file called =logging.properties= like this:
<verbatim>
handlers = org.apache.juli.FileHandler, java.util.logging.ConsoleHandler

############################################################
# Handler specific properties.
# Describes specific configuration info for Handlers.
############################################################

org.apache.juli.FileHandler.level = FINE
org.apache.juli.FileHandler.directory = ${catalina.base}/logs
org.apache.juli.FileHandler.prefix = hcss.
org.apache.juli.FileHandler.formatter = herschel.share.log.api.StandardFormatter

java.util.logging.ConsoleHandler.level = FINE
java.util.logging.ConsoleHandler.formatter = herschel.share.log.api.StandardFormatter
</verbatim>

%T% The fix for HcssScr:6672 is only implemented as of HCSS 4.0.635. If an earlier version than this is being used, then for this logging configuration to work properly it is necessary to update the =Configuration.class= file. How to do this is explained in the Trouble Shooting section below.

The _system_ property =hcss.init.logging= should be set to =false=. It will not work to add it =user.props= or any other HCSS property file! You can add it to =CATALINA_OPTS=, as in the example setting of this variable above.

The =hcss= application area should now log to a file called =hcss.date.log=.

---+++ Starting and stopping
The scripts can be found in Tomcat's =bin= directory. Use =startup= to start it, and =shutdown= to stop it. You may wish to check the =catalina.out= log to see if it started correctly.

Tomcat should automatically notice if a new servlet is defined and load it. You can force a servlet to reload by starting and stopping Tomcat. Wait a few seconds between stopping and starting or it might get confused (see next section). There is also a Tomcat "Manager" application that can do this from a web interface without stopping Tomcat - see the Tomcat documentation for details.

Occasionally it can get in a mess, which can be caused by things such as running out of memory, and refuse to restart. In this case you should see a message in the log along the lines of:
<verbatim>
LifecycleException:  Protocol handler initialization failed: java.net.BindException: Address already in use:8080
</verbatim>
In this case you need to kill the process (look for =catalina=) before restarting it.

---+++ HCSS properties configuration

Here are guidelines for ensuring that HCSS properties are set as you expect.

   1.  You need only one property set, and that is =var.hcss.dir= within CATALINA_OPTS. It is recommended that this be set to the =WEB-INF= directory of the application.
   1.  Create a directory config/properties in that directory.
   1.  Place any desired property files in that directory. They _must_ have the extenstion =.properties=.

Note that the =user.props= of the user owning the Tomcat process will also be read if it exists _and with higher precedence_. You might not want this. This is one reason for running the server from a dedicated account rather than a standard user account.

---+++ Trying it out

Following the instructions above and starting Tomcat, we should now have a running PAL server with url =http://whatever:8080/hcss/pal=. Check the log for any initialisation errors. Here is an example Jython script for trying it out on the client side:

<verbatim>
from herschel.ia.pal.pool.http import HttpClientPool
pool = HttpClientPool ("http://wakefield.bnsc.rl.ac.uk/hcss/pal", "sims3")
st = ProductStorage (pool)
st.authenticate()
r = st.select (Query (ObservationContext, "1"))
</verbatim>

---+++ %N% Updating an existing installation

This is all that is necessary to update to a new HCSS version:

<verbatim>
cd $CATALINA_HOME/webapps/hcss/WEB-INF/
rm lib/*.jar
copyBuildJars lib/
cp <hcss-dir>/project.xml .
</verbatim>

Then restart the server.

If you get an error because sun.grid/8.0.1/drmaa_8.0.1 is the wrong number of bytes then edit <hcss-dir>/installed.userlibraries, search for drmaa and delete that line. _This was a bug and is fixed in 10.0.1423_.

---+++ Troubleshooting

It is important to understand that the Tomcat scripts do not use the Java CLASSPATH environment variable. All required resources must be present in the classes or lib directories. For full details read the section on "Classloading" in the Tomcat documentation. 

The first rule of troubleshooting is to check the log file. The principle Tomcat log file is =$CATALINA_HOME/logs/catalina.out=. With the logging configuration above, the HCSS applications will log to =hcss.date.log=. Don't forget that what is written to the log can be configured as desired.

Sometimes there is no substitute for putting debugging statements in the code. You can do this by making modifications in a developer environment as normal, and then copying the =class= file(s) into the =classes= directory of the servlet. For example:

<verbatim>
cvs co ia_pal_pool_http_server
edit HttpPoolServlet.java
jake
</verbatim>
Then copy the output =HttpPoolServlet.class= file to =$CATALINA_HOME/webapps/hcss/classes/herschel/ia/pal/pool/http/server=.

%T% If a needed library is somehow missing, it is possible to be plagued by the dreaded =NoClassDefFoundError=. _The class reported to be missing is not necessarily the one that is_. If you know it's there, follow the dependency trail from the reported class to look for something missing. Pay particular attention to what is used in static initialisation statements.

---+++ Running on a privileged port

This is normally the standard HTTP port 80. The port is set by editing the =server.xml= file in the =conf= directory. I was not able to get the default Tomcat startup script to work when installed on this port. Here is my [[%ATTACHURL%/tomcat80][script]].

This script must be started with =root= privilege. Note that the daemon still runs as a less privileged user.

Further complications arise if there is a need to access the Versant library, as this accesses native shared libraries, e.g. for !DbPool or the TM/data frame server. Whilst the startup script uses the java.library.path argument to pass the location of these libraries to the daemon process, this does not entirely work since the first called library then calls other libraries, which does not work since the process does not inherit the value of the LD_LIBRARY_PATH environment variable (it defines its own). A workaround solution is to link to the required libraries from the Java installation, which is accessible, e.g.

<verbatim>
cd  $JAVA_HOME/jre/lib/amd64
ln -s $VERSANT_ROOT/lib/liboscfe.so liboscfe.so
ln -s $VERSANT_ROOT/lib/libxa.so libxa.so
</verbatim>

Note that this is only necessary since the server is running as a daemon process in order to access the privileged port 80. There may well be a better solution than this.

---+++ Using the telemetry and data frame server

Put the classes for the =access_server= module in the =hcss= applications area, as for the PAL pool.

Then add this servlet definition to the =web.xml= file in =WEB-APPS=.
<verbatim>
    <servlet>
        <servlet-name>tm</servlet-name>
        <servlet-class>herschel.access.server.HcssTmHttpServer</servlet-class>
    </servlet>
    <servlet-mapping>
        <servlet-name>tm</servlet-name>
        <url-pattern>/tm</url-pattern>
    </servlet-mapping>
</verbatim>

This corresponds to the url =http://whatever:8080/hcss/tm=, assuming default settings.

This servlet has a feature to allow redirection of a database request to a specific server. This is done by means of HCSS properties. (It might be cleaner to redefine them as parameters specified in the =web.xml= file). The easiest way to set them is to put them in a =HcssTmHttpServer.defaults= file in the same directory as the corresponding class file. These are:
   $ =hcss.access.server.default=: If a server is not specified, use this one. It itself defaults to =localhost=.
   $ =hcss.access.server.redirect=: Redirect all database server requests to this one. The purpose of this is to ensure that operationally critical servers cannot be accessed by this process; the data of course has to be present on the server redirected to.

This property should also be set on the server side. This will be made the default for this application in the next version of the module. _Note that this value is specific to the server and inappropriate for interactive use_.
<verbatim>
hcss.access.store = herschel.access.db.SimpleStoreHandler
</verbatim>

This servlet is accessed by using the normal =access= API. These client-side properties need to be set:

<verbatim>
hcss.access.connection=herschel.access.net.NetworkConnection
hcss.access.authentication=true
hcss.access.url=http://whatever:8080/hcss/tm
</verbatim>

Convenience methods to set these properties dynamically are provided in the =herschel.access.Access= class.

---+++ Using Tomcat's Manager application
This is a very useful tool for monitoring, stopping and restarting applications. To use it you have to give yourself privilege. Either:
   * Add the =manager= role to =tomcat-users.xml= and add the role to your username.
   * Edit the =web.xml= file for the =manager= application and change the =manager= role to =spire_admin= or whatever.

Then go to the "manager/html" web page on your server. I found that with my configuration I was getting HTTP 401 errors when I tried this. I got it to work by commenting out the =error-page= entry in the manager =web.xml= file.


-- Main.SteveGuest - 03 Aug 2012
Topic revision: r34 - 2012-10-19 - DavideRizzo
Public
Webs
Public
TWiki