Did you spot something wrong or missing on this page? If you have an account on this TWiki, you can fix it yourself by editing the page. If you don't have an account, you can leave a message at the bottom of the page to tell us about it. Thank you in advance!

Writing tasks

This page includes information on developing HIPE tasks.

If you are just interested in running HIPE tasks, rather than developing them, read the Running tasks chapter of the Scripting Guide.

This page assumes some familiarity with task development. If you are just starting out, please read these tutorials on the HIPE community website:

• For an introduction to writing Jython tasks, see the Creating your first Jython task tutorial.
• For an introduction to writing Java tasks, see the Creating your first Java task tutorial.

If you are an internal developer, additional documents you can access are the task checklist and the task FAQ.

What is a HIPE task?

A HIPE task is a standardised data processor. By wrapping your algorithm with a HIPE task, you can take advantage of the additional features and services of HIPE, among which are the following:

• Command line help: exposing available algorithms and how they should be called.
• HIPE integration: you can register a task to be listed in HIPE: dedicated registry, detection of applicable tasks for a given variable.
• Additional checking: checking of input parameters, specific error messages, interruptible.
• Product support: Recording of history.
• Automatic GUI generation: a common look-and-feel for the graphical interface.
• Re-usability: tasks are the building blocks of pipelines, and complex tasks can be composed of other tasks.
• Modifiability: a task defines how to execute the internal algorithm, how to provide inputs and to get outputs. You can modify the internal algorithm or even change it altogether, as long as input and output parameters remain the same.
• Evolution: preamble and postamble allow you to change the internals while keeping support for the old implementations. Support for deprecations is available too.
• Modes: Support for different environments: interactive, on demand ... and preferences.
• Additional Java support: tasks provide the support for calling your code from Jython as if it were written in native Jython: named positional parameters, optional parameters with just one method definition (execute) ...

This has some similarities to the Interactive Reduction and Analysis Facility (IRAF) where tasks/programs have a set of their own parameters and help. In IRAF, the history is placed in the header information of the files worked on.

A task is ideal if you want to create routines for other non-advanced users. You get the consistency checking, help and history recording in a consistent fashion for a little bit of extra packaging. This does not mean that you should embed all your algorithms inside tasks. The most versatile usage pattern is to initially develop some algorithm as functions in Java or Jython and then create additional task wrappers for those functions where you need the extra features listed above. For the rest of your code, plain old scripting should be enough.

You can execute tasks via the HIPE graphical interface, call them from the Console view or from your Jython scripts.

Naming a task

A task class name must end with the Task suffix for easy identification.

The instance task name and the (registered) variable name should match and follow these rules:

• Start with a lower case letter.
• Contain only letters and numbers.
• Do not end with Task (the only exception is openTask).
• Be shorter than 20 characters if possible. Names longer than 20 characters may appear truncated in GUI elements, such as task dialogue windows. Names longer than 60 characters may appear truncated in other text strings, such as tooltips.
• Be specific. For example, a task adding spectra should be called addSpectrum rather than add. A more specific name reduces the risk of name conflicts and better communicates the purpose of the task.
• Do not try to reuse reserved words (for , if , print ...), otherwise Jython will not be able to understand your task call.

The task name and the class name should be related (for example, SimpleFitsReaderTask and simpleFitsReader, or OpenVariableTask and openVariable). This is recommended, although exceptions are possible. For example, the BackgroundTask class corresponds to the bg task name.

You can create a second instance of a task class, rename it and register it, but only when the two instances have different configurations, like different primary inputs. Do not do this just to offer two names for your task. Users can create new task names by creating aliases or new instances:

mvln = myVeryLongName # Alias
mvln = MyVeryLongNameTask() # New instance

The TaskUtil class offers the isValidName method to check a task name and getDefaultName to generate a default name.

Naming style: given a new functionality for a task, there are 2 options:

• Role naming: This is how most old task are named, it is object-oriented but is ill suited for functions. (Ex fitsReader, asciiTableReader)
• Action-target naming: This is how we recommend to name new tasks as the action comes first and then (optionally) the target. (Ex readFits, readTable, openFile, sortTable, save, restore, pause ...). This is better suited to the functional style that tasks support.

Task parameters

Every task has parameters, defined by the TaskParameter class. Task parameters have the following features:

• They can be of three kinds: input (IN), output (OUT) or input-output (IO). Use an input-output parameter if your task needs to modify (or replace) the passed object. To set a parameter type:

JYTHON: myParameter.type = OUT
JAVA: myParameter.setType(TaskParameter.OUT);

• They accept values of a given type, such as TableDataset or Double1d. You can put more stringent limits on what values a parameter accepts by implementing a validator. To set a parameter value type:

JYTHON: myParameter.valueType = TableDataset
JAVA: myParameter.setValueType(TableDataset.class);

• They can be mandatory or optional. To set a parameter as mandatory:

JYTHON: myParameter.mandatory = True
JAVA: myParameter.setMandatory(true);

• They can have a default value. To set a parameter default value:

JYTHON: myParameter.defaultValue = 42
JAVA: myParameter.setDefaultValue(42);

• They can have a description, which will appear in a tooltip when you hover the mouse pointer on the parameter name in the task dialogue window. You must provide a description for each task parameter. To set the description of a parameter:

JYTHON: myParameter.description = "This is my parameter"
JAVA: myParameter.setDescription("This is my parameter");

Of course, most of the above properties should be set when creating a task parameter:

JYTHON: myParameter = TaskParameter("data", valueType = TableDataset, defaultValue=null, ...)
JAVA: TaskParameter myParameter = new TaskParameter("data", TableDataset.class);

Do not try to reuse reserved words for parameter names (for , if , print ...), otherwise Jython will not be able to understand your task call.

Using arrays as input parameters

The following example shows a task using an array as input parameter. The task transforms the array into a table dataset.

from java.lang import Integer

class TransformerTask(Task):
    
    def __init__(self, name = 'transformer'):
        Task.__init__(self, name)
        p = TaskParameter(name = 'input', valueType = array(Integer), mandatory = 1)
        self.addTaskParameter(p)
        p = TaskParameter( name = "result", valueType = TableDataset, type = OUT)
        self.addTaskParameter(p)
    
    def execute(self):
        self.result = TableDataset(description = 'Integrated vector as column zero')
        r = Double1d(len(self.input))
        index = 0
        for data in (self.input):
            r[index] = data
            index = index + 1
        self.result['0'] = Column(r)

You can use the following code to try the task:

# Test data
sample = [10, 20, 30, 40]
# Create the task:
transform = TransformerTask()
# Execute the task:
table = transform(sample)
# Print:
print table
print table[0]

Tasks must have at least one input parameter

You should not define a task without input parameters, because it will have very limited support in HIPE:

• The task will not be registered.
• The task will not even appear in the All folder of the Tasks view.

This means that users can only invoke the task via the command line, either in a script or at the prompt in the Console view. (The root cause is that the Task View, and ToolRegistry, require a primary input).

There were two such tasks in the core HIPE software: pause and resume, both in the ia.toolbox.util package, but they are now defined as plain functions.

Multiple data types for a parameter

You cannot set multiple data types for a parameter, but you can achieve the same effect by using one of the following workarounds:

• Use a more general data type, then define a validator doing additional checks.

For example, this is a task whose primary input accepts String, Integer and Product:

public class CombinedTask extends Task {
        private static final String TASKNAME = "combined";
        private static final String PRIME = "input";
        /** Jython doc string */
        public static PyString __doc__;

        /**Constructor*/
        public CombinedTask() {
            super(TASKNAME);
            addTaskParameter(TaskUtil.buildParameter(PRIME, Object.class, INPUT, MANDATORY, null, NULL_NOT_ALLOWED,
                    "String, Integer or Product"));

            getParameter(PRIME).setParameterValidator(new ParameterValidatorAdapter() {
                @Override
                public void validate(Object value) throws ParameterValidationException {
                    if (value instanceof String || value instanceof Integer || value instanceof Product) {
                        return;
                    }
                    throw new ParameterValidationException("Not String, Integer or Product but " + value.getClass());
                }
            });
            setDescription("task to test object parameters");
            __doc__ = TaskUtil.makeDoc(this);
        }


        @Override
        public void execute() {
            Object prime = getValue(PRIME);
            if (prime instanceof String) {
                System.out.println("A String " + prime);
            } else if (prime instanceof Integer) {
                System.out.println("An Integer " + prime);
            } else {
                System.out.println("A Product " + prime);
            }
        }
}

In Jython:

        # ...
        p = TaskParameter('input', mandatory = 1, valueType = java.lang.Object)
        p.description = " A desc 4"
        p.parameterValidator= InstanceOfAnyValidator(String, Integer, Product)
        self.addTaskParameter(p)
        # ...

• Use multiple optional parameters (not recommended, too many parameters)

Cleaning of parameters after execution

The execution of a task should not depend on any previous executions. Thus, your task class should not have member or class variables (everything should be done through parameters). The framework cleans the data from parameters after executing a task.

CHANGES FOR JYTHON CALLS TO TASKS THAT HAVE MULTIPLE OUTPUTS
The way client code retrieves the outputs for tasks that have multiple outputs is changing. The current way was not Python-like and could generate memory leaks (and even dependencies from previous executions in the case of IO (INOUT) parameters). This will only affect tasks with multiple outputs. When the transition is complete, in the case of multiple outputs, all outputs will be returned as a list (and thus the signature will be cleaned just after execution in all cases). var1, var2 = myTask(). The migration to this new style has been staged as follows:

• HIPE 11
  • To enable the new syntax you have to add __list__=True to the task call: var1, var2 = myTask(arg=1, ..., __list__=True). If you do not use this form the system will issue a warning every time the task is called. Note that this means that if you just put a variable in the left side of the assignment (var= myTask(arg=1, ..., __list__=True)), var will be a list with all outputs if there is more than one defined (remember that INOUTs are outputs too!).
• HIPE 12
  • The default syntax will be the new one and you no longer need to add __list__=True. var1, var2 = myTask() will work as is (given that myTask has two outputs defined, of course).
• HIPE 13 or later
  • __list__ is no longer a valid optional parameter in HIPE 13 or later.

Note that tasks with multiple outputs will have more restrictions regarding changes in their outputs (order is fixed, adding or removing outputs may result in syntax errors ...). If you want to keep the flexibility to evolve your task it is recommended that you evaluate moving to a task with just one compound output. The best time to do it was during HIPE 11 development but it is never too late. Also note that it is impossible for the task framework to know how the task has been called with respect to the left side of an assignment (the framework just 'sees' the right hand side, i.e. see INFO Task perform: [Full Call] log messages).

This is why most parameters are cleared (reset to their default values) after a task is executed in Jython:

• The first output parameter is cleared after execution of the task. You should store the value of this parameter into a variable when you execute the task: var = myTask().
• IN DEPRECATION: The second and additional output (or input-output) parameters are kept in memory until they are accessed, then they are cleared: var2 = myTask.output2.
• Input parameters are cleared after the task is executed.

Try to use a single output parameter per task, so that there are no leftover values after a task execution. If you need to return multiple values you can use collections or define an appropriate type that contains all outputs (check herschel.ia.dataset.Metadata, a general Dictionary-like class with map and hash implementations).

If you execute tasks from Java code, there is no automatic cleanup. You need to reset the outputs manually like this:

myTask.perform();
myTask. ... // get myTask results
myTask.reset();

Of course, if myTask is going to be garbage collected there is no need to clear it up. Note that the task instances already available in HIPE are not going to be garbage collected.

Running a task in the HIPE Console view

The Task framework offers different ways to execute the task as well: Calling the execute method using "positional parameters" and "named parameters". Suppose that a task expects three input parameters: One image parameter called 'img', one float indicating a rotation angle 'angle', and one boolean (true or false) called 'takeNeg' indicating whether the negative of the image shall be produced.

If you call the execute methods using positional parameters, then the simple order of the parameters tells the Task which is the image, which is the angle and which is the boolean. If you use named parameters, then you are allowed to use any order for the parameters, but every parameter you add after the first named one must be a named parameter too (If you were allowed to put positional parameters after a named one their positions would not be evident).

Including Tasks in HIPE

Tasks should be included in HIPE such that they can be accessed directly from its user interface, in particular the Tasks View.

In short, including a task in HIPE comprises:

• Adding the task to HIPE registry. It is also possible to specify a specific category for your task, such that the task will be available for a specific type of variable.
• Optionally setting validation requirements for a specific parameter, using a ParameterValidator. See Validating prime input.
• Optionally provide a custom task dialog. HIPE automatically provides a default Task dialog panel for the registered tasks.

Please refer to the detailed instructions for a full explanation.

Handling exceptions in tasks

If an error occurs during execution of the task, and the task cannot recover from it, an exception should be raised. This exception should be derived from TaskException (a RuntimeException ). The Task API does not declare any exception (Task.perform), so we can't use checked exceptions. If you must raise a checked exception, use the initCause() method of a TaskException (or subclass thereof).

Interrupting the execution of a task

The Stop button in the HIPE user interface generates an interruption event and sends it to the separate thread where Jython code is running.

This signal is read between consecutive commands of the Jython script. For example, pressing the Stop button interrupts immediately this infinite loop:

while 1:
   print 1

However, this may not be so when the interpreter releases control to a time-consuming Task. If the Task does not periodically check for the interruption signal, pressing the Stop button will have no effect.

You can check for the interruption signal within your code by using the checkInterrupted method of the Task class. If an interruption request is detected, the method will throw an InterruptedException that would be caught by the interpreter, thus cancelling the task. You may pass a String representing the message to issue when the Task is interrupted:

myTask.checkInterrupted(); # Uses default message
myTask.checkInterrupted("Interruption signal received");

Your code should check for the interruption signal with sufficient frequency to allow a timely interruption. For example:

// Within the execute method of your task
for (int i = 0; i < nSteps; i++) {
    Task.checkInterrupted(getName() + " interrupted in step " + i);
    doStep(i);
}

Tasks and tools involving products and datasets

For tasks that deal with products and datasets, the users (astronomers and calibration scientists) should not be required to perform type conversions between products and datasets. If such conversions are required, they should be dealt with by the software. This in part stems from the fact that manipulation of datasets loses any associated history. It is recommended that all tasks (and tools) should work with, and save outputs as, products.

Managing user level feedback

Suppose a task can not create the expected outputs because it found an error in the input data. This can not be considered a program failure and has more to do with the notifications an interactive system should give to its operators/final users. Even if the task terminates normally, it should be desirable to provide a way to know if a task was already executed or it is still pending or was interrupted during the execution. Another common situation is that where a time consumer task needs to give some feedback to the user during it execution.

For these situations, all tasks include the following fields in the Signature class: Status, StatusMessage and the Progress.

The Status parameter stores a numeric code without restrictions but with a few, standardized, values:

    Status    Status       Meaning/   
    Value     "Name"       Status Message
    ------    -----------  ---------------------------------------------------------
    100       UNKNOWN      Signature is created but not assigned to a given task yet
    200       READY        Task is ready to be executed
    400       RUNNING      During the execution of the perform method of the task
    000       SUCCESS      Task has finished without problems
    500       INTERRUPTED  Task has been interrupted during its execution
    900       FAILED       Error found during the task execution

Once a task is created the READY value is assigned to it. UNKNOWN value is only used for those cases where a Signature is created before its assignment to a Task or even a pipeline.

Now, when the task perform method is called, its Status is automatically changed to the RUNNING value and it will keep this value until its successful conclusion ( SUCCESS ), an error is found ( FAILED ) or it happens to be interrupted before finish (INTERRUPTED).

These transitions between states are automatically registered into the Status field by the default implementation. The developer can use this information as a quick feedback to the user about the final result of the task. Also, the developer can update the StatusMessage field during the execution of the task as a way to notify the user of current stage into a time consuming task.

Showing task execution progress

The Progress field marks the advance in the execution of a given task as a integer value between 0 and 100 . This provide a way to show the user how a single task is performing and how much time is expected this task to last until its conclusion. The developer can update this value at any time during the execution process and the Task package includes a bean to easily display this value on the screen as the typical progress bar.

The Progress parameter is accessible via the name Signature.PROGRESS and can be used as in the following example.

      public void execute(){
        // assuming to iterate for a number of _iterations
        for (int j=0; j<=_iterations ; j++) {
            // DO SOMETHING DURING THE ITERATION
            //xxxxx
            // LOG PROGRESS INFORMATION
            setValue(Signature.PROGRESS, new Integer(((j*100)/_iterations)));
        // Also, this shortcut is valid for any subclass of Task:
        // setProgress((j*100)/_iterations)
        // Additionally, the following code updates both progress and status message (preferred)
        advance((j*100)/_iterations, "iteration " + j+1 + " of " + _iterations);  
        }
      }

      def execute(self):
        #assuming to iterate for a number of _iterations
        for j in range(_iterations):
            ## DO SOMETHING DURING THE ITERATION
            ##xxxxx
            ## LOG PROGRESS INFORMATION
            self.progress = j*100/_iterations

The FAILED status

The developer must clearly separate the situation where the Task is suppose to return a failed result from any other problem derived from an implementation error or an unexpected situation not well managed by the code. In the latter cases, a RuntimeException shall be raised and the calling method (maybe the graphical environment where the task is running) take care of it.

By definition, a task should be marked as FAILED when, even assuming an error free algorithm and giving the valid inputs, it fails to produce the expected output. A simple example is a task that implements an algorithm to identify the starts into a image. This task should fail when the provided image doesn't contains any start or the algorithm can not distinguish them due to a noisy background.

Optional parameters

Parameters that are facultative must have been given a default value before the task is executed, failure to do so will provoke an exception when parameters values are checked in the preamble. Usually, the default value for a parameter is provided by the task implementer, not the end user. Parameters requiring a value from the user should be tagged as mandatory . Contrariwise, mandatory parameters cannot have a default value.

Task compliance checklist

Developing a task that looks native involves quite a few steps. We have provided a list with the qualities a task should hold: ChecklistTask (this is a TWiki page only for internal developers).