Difference: DpHipeTools (55 vs. 56)

Revision 562010-06-22 - JavierDiaz

Line: 1 to 1

Adding Tools to HIPE

Line: 86 to 86
  Name of the variable in Jython : reduce
Warning, important For naming tasks we follow the Java conventions, we use camel-case , not underscores : "reduceLight" is valid, "reduce_light" is not. Info log messages are produced for invalid task names.
Warning, important For naming tasks we follow the Java conventions, we use camel-case , not underscores : "reduceLight" is valid, "reduce_light" is not. Info log messages are produced for invalid task names. Note that the task names are used as variable names (of type task) automatically created when starting HIPE.

Prime input validation

Line: 572 to 572

Triggering Events

For a full detailed section about triggering events have a look at DpHipeCommonUtilities.

Task Compliance Checklist

Developing a task that looks "native" takes time and a quite a few steps. To help you comply with all requirements, we offer a (high level) list of common deficiencies and mistakes to be taken into account. This is not a MUST comply list (yet).

  • GUI
  1. Use a two column layout for the Inputs
  2. Use foldable sections for the task
  3. Use decorated labels (bold, asterisk ...)
  4. Use tooltips on the labels describing the parameters
  5. Use specialized modifiers
  6. Prefer enumerations (Combo boxes of String lists) to open ended texts
  7. Prefer no scrolling, then vertical to horizontal.

Points 1 to 4 are given for free if you just use the default implementation

  • Usage
  1. Update the progress while the task is executing (if it takes more than a second)
  2. Allow the task to be started and cleaned from the toolbar of HIPE
  3. Allow your task to be interrupted
  4. Provide informative error messages (in the constructor of your exceptions)
  5. Log important information for users to check
  6. Follow the naming conventions
  7. Register your task properly
  8. Prefer strings to enums (harder to write without autocomplete) for textual invocation
  9. Use validators to restrict applicability

  • Documentation
  1. Put descriptions in your task parameters
  2. Use _ doc _ to show the signature of your task
  3. Check the generated information in the URM
  4. Provide reproductible (even if not complete) examples
  5. Try to provide alternative documentation to jtags in Javadoc

All of the tasks in ia_toolbox_util try to follow this recomendations. A sample task illustrating some of the previous points:

 * The Decompress task.
 * <p>Decompresses a (relative path) archive file in a  user-chosen directory. 
 * Supports arbitrary nesting (but not per entry) of the following algorithms: TAR, ZIP, GZIP.
 * So tar,gz or zip.gz will be completely expanded but zip entries in tar archive 
 * will only be expanded up to the zip entries.
 *<li><b>archive</b> (INPUT, mandatory) = String with the archive file path
 *<li><b>dirout</b> (INPUT, mandatory) = String with the output directory path (may not exist).
 *<li><b>compression</b> (INPUT) = String with the type of compression in the file.
 *  <p>Valid options:
 *  <ul>
 *  <li> <b>"Guess"<b>: (default) automatically detect the type of compression used </li>
 *  <li> <b> "ZIP" </b>: use unzip (or equivalent) to decompress</li>
 *  <li> <b>"TAR"</b>: use untar (or equivalent) to decompress</li>
 *  <li> <b> "GZ"</b>: use gunzip (or equivalent) to decompress</li>
 *  <li> <b> "TGZ"</b>: use gunzip then untar (or equivalent) to decompress</li>
 *  </ul>
 * #Decompress a tar archive in a temporal directory
 * decompress("./mytar.tar", "/tmp") 
 * </pre>

Above, Documentation point 5 (Parts: title, description, parameters (with options), sample usage. Check HTML usage)

 * @author jadiaz
 * @jhelp Decompresses a (relative path) compressed file in a  user-chosen directory
 * Two first arguments are mandatory. Supports arbitrary nesting (but not per entry) of the following algorithms: TAR, ZIP, GZIP.
 * So tar,gz or zip.gz will be completely expanded but zip entries in tar archive will only be expanded up to the zip entries.
 * @jalias decompress
 * @jcategory task
 * @jsynopsis
 * decompress(<archive>, <dirout> [, <compression>])
 * @jexample-preamble
 *   from java.io import File, FileOutputStream
 *   from com.ice.tar import TarOutputStream, TarEntry
 *   file = File("./mytar.tar");
 *   testFile = File("deleteMe");
 *   if testFile.exists(): testFile.createNewFile()
 *   stream = TarOutputStream(FileOutputStream(file))
 *   entry = TarEntry(testFile)
 *   stream.putNextEntry(entry)
 * @jexample Untarring in temp
 *   decompress("./mytar.tar", "/tmp")  
 * @jexample-postamble
 *   file.delete()
 *   testFile.delete()
 *   del(file, testFile, stream, entry) 
 *   del(File, FileOutputStream, TarOutputStream, TarEntry)
 * @jparameter tar, INPUT, String, MANDATORY, null
 *  Path of the file to be untared
 * @jparameter dirout, INPUT, String, MANDATORY, null
 *  Directory (that may not exist yet) where we want to untar the tar file
 * @jparameter compression, INPUT, String, OPTIONAL, "Guess"
 * Type of compression of the input file, values: GUESS, ZIP, TAR, GZ, TGZ
 * @jlimitation
 * Gzipped files will expand to the filename minus the ".gz" or ".gzip" if present, else they will add ".new" to 
 * avoid overwritting the original (if you are expanding in the same directory where the archive is). 
 * @jhistory
 * 2010-05-25 JDS first release
 * 2010-06-09 JDS expanded, renamed to decompress

Above, Documentation point 3 and 4 (Check jtags definitions)

public class DecompressTask extends Task {
    private static final long serialVersionUID = 1L;
    private static final Logger _Log = Logger.getLogger(DecompressTask.class.getName());

Above, Usage point 5 (provide a logger)

    private static final String TASKNAME = "decompress", 
        ARCHIVE = "archive", DIROUT = "dirout", COMPRESS= "compression";

Above, Usage point 6 (we minimize string literals and follwo naming conventions)

    public static PyString __doc__ = new PyString(
            "decompress(<archive>, <dirout> [, <compression>])\n" +
            "\nWhere all parameters are strings, archive is the source file and dirout a (possibly not existing) directory"     

Above, Documentation point 2 (Line 1: syntax (@jsynopsis) , rest of lines: description (@jhelp))

    public enum CompressType {
        TGZ ("TGZ");
        public String toString() {
            return _text;
        public static CompressType get(String text) {
            for (CompressType e: CompressType.values()) { //vs valueOf(arg0) -> CONSTANT name
                if (text.equals(e._text)) return e;
            return null;
        public static String [] toStringArray() {
            CompressType [] data = CompressType.values();
            String [] out = new String [data.length];
            for (int i = 0; i < data.length; i++) {
                out[i] = data[i].toString();
            return out;

        private CompressType(String text) {
            _text = text;

        private String _text;

    public DecompressTask() {

      "The tar file to expand");

Above, Documentation point 1 (each task parameter has a string describing it , note syntax similar to @jparameter)


                DIROUT, String.class, INPUT, MANDATORY, null, NULL_NOT_ALLOWED, 
                "The directory where the tar will be expanded");

Above, Usage point 8 (this is an enum , but we define it as a string, and we show it like a Combobox)

                COMPRESS, String.class, INPUT, OPTIONAL, CompressType.UNKNOWN.toString(), NULL_NOT_ALLOWED,  
                "The directory where the tar will be expanded");


    public Map<String, Modifier> getCustomModifiers() {
        Map<String, Modifier> m = new LinkedHashMap<String, Modifier>();
        //TODO: also for gzipped files
        m.put(ARCHIVE, new JFilePathModifier(FileSelectionMode.OPEN, 
                new FileNameExtensionFilter("tar files", "tar"),
                new FileNameExtensionFilter("gzip files", "gz", "tgz", "gzip"),
                new FileNameExtensionFilter("zip files", "zip")                
        m.put(DIROUT, new JFilePathModifier(FileSelectionMode.DIRECTORY));
        m.put(COMPRESS, new JOptionModifier((Object []) CompressType.toStringArray()));

Above, GUI point 6 (we use a JCombobox), GUI point 5 (we provide FIlePaths instead of just text fields)

        return m;       

    public void execute() throws InterruptedException {         
   String tarpath =  (String) getValue(ARCHIVE);
   String dirpath = (String) getValue(DIROUT);
   String compress = (String) getValue(COMPRESS);

   File tarfile = new File(tarpath);
   File dirfile = new File(dirpath);

   advance(5,"Validating parameters ...");      
   if (isEmpty(tarpath) || ! tarfile.exists() || tarfile.isDirectory()) {
       throw new IllegalArgumentException(ARCHIVE + " must point to an archive file.");
        if (isEmpty(dirpath) || (dirfile.exists() && ! dirfile.isDirectory())) {
            throw new IllegalArgumentException(DIROUT + " must point to a directory.");
        CompressType type = null;
        if (isEmpty(compress)) {
            compress = null;
        } else {
            type = CompressType.get(compress);
            if (type == null) { //not found
                throw new IllegalArgumentException(COMPRESS + " invalid value, allowed values are " + Arrays.toString(CompressType.values()));

Above, Usage point 4 (we check validity of inputs, with taylored error messages)

        advance(10,"Prepare environment ..."); 
        if (! dirfile.exists() &&  ! dirfile.mkdirs()) {
            throw new RuntimeException(getExceptionMessage("Could not create " + DIROUT + "."));               
        advance(20,"Analyzing archive ...");
        InputStream i = null;
        try {
            if (type == null || CompressType.UNKNOWN.equals(type)) {
                i = getStream(tarfile);
            } else {    
                i = new BufferedInputStream(new FileInputStream(tarpath));
                switch (type) {
                    case TAR : i = new TarInputStream(i); break;
                    case GZIP : i = new GZIPInputStream(i); break;
                    case ZIP : i = new ZipInputStream(i); break;
                    case TGZ : i = new TarInputStream(new GZIPInputStream(i)); break;
                //default :
            advance(30,"Decompressing archive ...");  

Above, Usage point 1 (every time we do something significant to the user (or time consuming), we update the progress)

            if (i instanceof TarInputStream) {
                process((TarInputStream) i, dirfile);
            } else if (i instanceof GZIPInputStream) { //need a file name
                process((GZIPInputStream) i, new File(dirfile, getGZIPName(tarfile.getName())));   
            } else if (i instanceof ZipInputStream) {
                process((ZipInputStream)i, dirfile);
            } else { //unknown compression
                throw new IllegalArgumentException("Unknown compression algorithm in file " + tarpath);

        } catch (IOException e) {
            throw new RuntimeException(getExceptionMessage("Could not expand " + ARCHIVE, e),e);

Above, Usage point 4 (even if we do not check each error, we provide a default "personalized" error message : while expanding something went wrong)


        } finally {
            if (i != null) { 
                try {
                } catch (IOException e) { } 



<-- Author: JorgoBakker - 21 Dec 2007 -->
This site is powered by the TWiki collaboration platform Powered by Perl