2.2. Numeric arrays

Numeric arrays available in HIPE are shown in the following table. A numeric array can have from one to five dimensions.

For completeness the following table also shows the String1d array type, which is not a numeric array. String arrays can only be one-dimensional.

Table 2.1. Types of numeric array (N = 1...5)





















Differences with Jython native arrays. Numeric arrays are optimised for holding Herschel data and for working with HIPE tasks and tools. Many tasks and functions accept Numeric arrays as their input, but not native Jython arrays. You are advised to use Numeric arrays when manipulating Herschel data.

2.2.1. Creating an array

See the following example for various ways in which you can create a one-dimensional array:

y = Double1d() # Create an empty array
y = Double1d([3.0, 5.5, 2.1, 6.0]) # Create from a Jython array
y = Double1d(4) # [0.0, 0.0, 0.0, 0.0]
y = Double1d(4, 42.0) # [42.0, 42.0, 42.0, 42.0]
y = Double1d.range(4) # [0.0, 1.0, 2.0, 3.0]

Example 2.1. Declaring an array of doubles.

You can create a complex array, with the same commands. In addition, you can specify real and imaginary parts separately:

y = Complex1d() # Create an empty array
y = Complex1d([1+4j, 2+3j, 5+8j]) # Create from a Jython array
y = Complex1d([1, 2, 5], [4, 3, 8]) # Same result as previous command
y = Complex1d(4) # [0j, 0j, 0j, 0j]
y = Complex1d(3, 1+4j) # [(1+4j),(1+4j),(1+4j)]
y = Complex1d.range(4) # [0j,(1+0j),(2+0j),(3+0j)

The following example shows how to create a two-dimensional array. You can create arrays of more dimensions in the same way:

y = Double2d() # Create an empty array
y = Double2d([[3.0, 5.5], [2.1, 6.0]]) # Create from a Jython array
y = Double2d(4, 4) # Creates a 4x4 array whose values are all zero.
y = Double2d(4, 4, 42.0) # Creates a 4x4 array whose values are all 42.

Example 2.2. Declaring a two-dimensional array of doubles.

Rectangular and jagged arrays. Rectangular arrays are multidimensional arrays which always have the same number of elements in each row or column. Jython and Java allow you to create jagged arrays. Jagged arrays are multidimensional arrays where each row can have a different number of elements. The following example creates a two-dimensional jagged array, with two rows of two and three elements, respectively:

x = [[1,2], [3,4,5]]

Example 2.3. Creating Jython jagged arrays.

However, you cannot create Numeric jagged arrays:

x = Double2d([[1,2], [3,4,5]]) # Gives an error

Example 2.4. It is impossible to create Numeric jagged arrays.

2.2.2. Inspecting an array

Array elements along each dimension are defined by an index running from zero to the array length, minus one:

x = Double1d([3.0, 5.0, 9.0, 4.3])  # Four elements
print x[0]  # Prints the first element, 3.0
print x[3]  # Prints the fourth element, 4.3

Example 2.5. Accessing array elements using the indices.

To access ranges of elements, or slices, use a notation like [i:j]:

print x[1:3]  # From index 1, included, to 3, excluded
# [5.0,9.0]
print x[2:]  # From index 2, included, to end of array
# [9.0,4.3]
print x[:3]  # From start of array to index 3, excluded
# [3.0,5.0,9.0]

Example 2.6. Using array slices to access ranges from Jython lists.

To access elements in multi-dimensional arrays, separate indices or ranges along each dimension with commas:

x = Int2d([[1,2,3],[4,5,6]])
#   1 2 3
#   4 5 6
print x[1]                # 2 (second element of the first row)
print x[0,:]              # Row 0: [1,2,3]
print x[1,1]              # Individual element: 5
print x[:,:]              # Print entire array. Same as print x
print x[:,1]              # Column 1: [2,5]

Example 2.7. Accessing ranges of indices using slices.

Multi-dimensional arrays are conceptually arrays of lower-dimensional arrays. For a two-dimensional array, the first subscript selects a row and the second subscript selects an element within that row (the column).

[Note] Note

This is the opposite order to some other computer languages, but it is the same behaviour as in the Java programming language.

Note the difference in syntax when inspecting native Jython arrays and numeric arrays:

# Jython array:
x = [[1,2,3,4],[5,6,7,8]]
print x[1][2]       # 7
print x[1][1:3]     # 6, 7
# Numeric array:
y = Int2d([[1,2,3,4],[5,6,7,8]])
print y[1,2]        # 7
print y[1,1:3]      # 6, 7

Example 2.8. Checking the differences between Jython arrays and numeric arrays.

2.2.3. Inspecting a complex array

Complex arrays offer the following additional commands:

z = Complex1d([1,2,3,4],[4,3,2,1]) # Set up complex array
print z.real # [1.0,2.0,3.0,4.0]
print z.imag # [4.0,3.0,2.0,1.0]
print z.conjugate() # [(1.0-4.0j),(2.0-3.0j),(3.0-2.0j),(4.0-1.0j)]

Example 2.9. Inspecting and manipulating a Complex Numeric array.

2.2.4. Modifying an array

You can append single values or entire arrays to an existing array:

y = Double1d()
y.append(2.0)  # Append a single value
y.append(Double1d([3.0, 7.5, 2.8]))  # Append a whole array

Example 2.10. Appending values to an array.

Individual elements or slices can be set as follows:

x[1,2] = 22 # Set an element in place
x[0,1:3] = 42
print x # [
        # [2.0,42.0,42.0], 
        # [1.0,3.0,22.0]
        # ]

Example 2.11. Assigning values with the use of indices and slice notation.

It is possible to set a row to a copy of a 1d array of the same length:

x[0,:] = [5,6,7,8]        # Set a row to (a copy of) a Jython array
y[1,:] = Int1d([9,7,6,5]) # Set a row to a Double1d array

Example 2.12. Assign arrays to arrays using slice notation.

2.2.5. Ordering of array elements

The following line of code creates an array of two rows and three columns:

x = Double2d([[2,4,6],[1,3,5]])

Example 2.13. Declaring multidimensional Numeric array.

You can access the element corresponding to the i-th row and j-th column like this:

x[i, j]

Example 2.14. Accessing elements in multidimensional arrays.

The values are stored sequentially in memory as follows:

[2 4 6 1 3 5]

This means that, if you go through the array elements as they are stored in memory, their indices would vary as follows:

x[0,0] x[0,1] x[0,2] x[1,0] x[1,1] x[1,2]

That is, index j varies more rapidly than index i. This can be generalised to more than two dimensions by saying that the rightmost index varies most rapidly. This is called row-major ordering, and is the convention followed by languages such as Java and C, but not Fortran.

This has an implication on performance. When looping through a multidimensional array, it is more efficient to read its elements in the order they are stored in memory.

Confusion may arise when dealing with images, which are stored as two-dimensional arrays. If you visualise the array with horizontal rows and vertical columns, then the number of rows and columns represents the size of the vertical (y) and horizontal (x) side of the image, respectively. When accessing a particular pixel (array element), you have to specify the y coordinate before the x coordinate:

myImage(y, x)

2.2.6. Numeric array arithmetic

HIPE numeric arrays support arithmetic operations that are applied element-by-element. For example:

y = Double1d.range(5)                  # [0.0,1.0,2.0,3.0,4.0]
print y * y * 2 + 1                    # [1.0,3.0,9.0,19.0,33.0]

Example 2.15. Applying multiplication and addition to all elements of an array.

This is much simpler (and runs much faster) than writing an explicit loop in Jython. The '+' operator does not concatenate arrays, as it does with Jython arrays. For example:

# Adding Jython arrays
print [0,1,2,3] + [4,5,6,7]                  # [0, 1, 2, 3, 4, 5, 6, 7]

# Adding numeric arrays
print Double1d([0,1,2,3]) + Double1d([4,5,6,7])       # [4.0,6.0,8.0,10.0]

# Concatenate two numeric arrays
print Double1d([0,1,2,3]).append(Double1d([4,5,6,7])) 
# [0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0]

# Adding Jython arrays to numeric arrays
print [0,1,2,3] + Double1d([4,5,6,7])        # [4.0,6.0,8.0,10.0]
print Double1d([0,1,2,3]) + [4,5,6,7]        # [4.0,6.0,8.0,10.0]

Example 2.16. Concatenating numeric arrays.

All arrays support the following arithmetic operators:

+, -, *, /, %, **

Note that the 'modulo' operator '%' provides the normal Jython semantics for this operation, which is not the same as that of the Java '%' operator. The Jython definition is more consistent with the mathematical notion of congruence for negative values.

The following relational operators return a Bool1d array:

<, >, <=, >=, ==, !=

For example:

y = Double1d([0,1,2,3,4])
print y > 2                         # [false,false,false,true,true]

Example 2.17. Applying relational operators to a Numeric array.

2.2.7. Selecting and filtering array values

Use the where function to select the elements of an array that satisfy a given condition:

y = Double1d([2,6,3,8,1,9])
print y.where(y > 4) # [1,3,5] Indices of elements greater than four

Example 2.18. Filtering array elements with the where method.

Note that the result contains the indices of the elements that satisfy the condition, not the elements themselves. You can combine more conditions with the & (and), | (or) and ~ (not) operators:

print y.where((y > 4) & (y < 9))  # [1,3]
print y.where((y > 8) | (y < 2))  # [4,5]
print y.where(~(y > 4))  # [0,2,4]

Example 2.19. More complex filtering using the where method.

You can obtain the actual array elements instead of the indices as follows:

print y[y.where(y > 4)]  # [6.0,8.0,9.0]
print y.get(y > 4)  # [6.0,8.0,9.0]

Example 2.20. Accessing the array values with a filter.

Obtaining the indices rather than the actual values can be useful, for instance, when you want to select the same elements from different arrays:

x = Double1d([5,6,7,8,9,10])
s = y.where(y > 4)
print x[s] + y[s]  # [12.0,16.0,19.0]

Example 2.21. Adding two arrays with the same set of filtered array indices.

You can also use the where function to set values:

s = y.where(y > 4)
y[s] = 0 # Set all matching elements to 0
print y  # [2.0,0.0,3.0,0.0,1.0,0.0]
y[s] = [9,8,7] # Set matching elements using an array
print y  # [2.0,9.0,3.0,8.0,1.0,7.0]

Example 2.22. Assigning values with the results of the where method.

You cannot use the where function like this:

a = Double1d.range(10)
b = a.where(a < 3)
print b[0]  # AttributeError: __getitem__
print b[0:2]  # AttributeError: __getitem__
print a[b[0]]  # AttributeError: __getitem__

Example 2.23. The output list of where is not accessible by index.

The commands fail because b is a Selection object rather than a Jython or Numeric array. For the above to work you need to convert it to Int1d:

c = b.toInt1d()
print c[0]  # 0
print c[0:2]  # [0,1]
print a[c[0]]  # 0.0

Example 2.24. Converting the output of where to a normal array that you can manipulate.

By converting to Int1d you can also loop over all the selected elements:

for i in c:
  print a[i]

Example 2.25. Converting the output of where makes the resulting object iterable.

Another useful function is get, with which you can extract individual elements or a subset of element values from an array. There are four ways to use it:

  • Get a single value:

    HIPE> print y.get(0)

    This is the same as print y[0].

  • Retrieve elements based on a Bool1d array (in other words, a mask):

    HIPE> mask = Bool1d([0,0,1,0,1,1])
    HIPE> print y.get(mask)

    The mask array can be shorter than the target array (in which case the remaining elements in the target array will be ignored), but not longer.

  • Retrieve elements based on a Selection object:

    HIPE> indices = Selection([1,2,4])
    HIPE> print y.get(indices)

    An out of bounds index will cause an error.

  • Retrieve elements based on a Range object:

    HIPE> range = Range(2,4)
    print y.get(range)

    Note how the lower boundary (second element) is included, while the upper boundary (fourth element) is excluded. An out of bounds value will cause an error.

You can combine get calls to perform the same operation as a compound IDL WHERE execution such as the one shown in this example:

IDL> a = [1, 2, 3, 4, 5, 6]
IDL> b = [2, 3, 4, 5, 6, 7]
IDL> c = [3, 4, 5, 6, 7, 8]
IDL> q =  WHERE(a ge 2 and b lt 6 and c gt 5)
IDL> x = [a[q],b[q],c[q]]
IDL> print, x
       4       5       6

This is the equivalent in HIPE:

q = (a >= 2) & (b < 6) & (c > 5)
x = a.get(q),b.get(q),c.get(q) # x == ([4.0], [5.0], [6.0])
HIPE> a = Int1d([1, 2, 3, 4, 5, 6])
HIPE> b = Int1d([2, 3, 4, 5, 6, 7])
HIPE> c = Int1d([3, 4, 5, 6, 7, 8])
HIPE> q = (a >= 2) & (b < 6) & (c > 5)
HIPE> x = a.get(q),b.get(q),c.get(q)
HIPE> print x
    ([4], [5], [6])

2.2.8. Using logical operators with arrays

The Jython logical operators and, or and not work like normal Boolean operators (see Appendix A for more details), but using them with arrays (both the native Jython arrays and HIPE numeric arrays) can give unexpected results. The reason is that these operators do not work on an element-by-element basis when applied to arrays, but they evaluate the entire array at once.

The Jython bitwise operators, represented by the symbols &, | and ^ (see Appendix A for more details) can be used with numeric arrays (Int1d, Bool3d and so on), but what you get is not a bitwise comparison. Instead, these operators perform the usual boolean comparisons, but this time working element by element. Precisely what and, or and not do not do.

Finally, Numeric array classes have the and, or and xor methods acting like boolean operators working element by element. An example will clarify the differences among all these operators:

jythonOne = [1, 0, 0, 1]
jythonTwo = [0, 0, 1, 1]
numericOne = Bool1d(jythonOne)
numericTwo = Bool1d(jythonTwo)
print jythonOne and jythonTwo
## [0, 0, 1, 1] # jythonOne is not empty so it is treated as true, which means that
                # jythonTwo is evaluated and returned
print numericOne and numericTwo
## [false,false,true,true] # Same thing as with the Jython native arrays
print jythonOne & jythonTwo
## Here an error is returned
print numericOne & numericTwo
## [false,false,false,true] # Here the operator works element by element
print numericOne.and(numericTwo)
## [false,false,false,true] # Same thing as the & operator

Example 2.26. Differences between Jython and Numeric arrays.

2.2.9. Removing infinite and NaN values from arrays

To remove infinite and NaN values from an array, use the IS_FINITE, IS_INFINITE and IS_NAN functions as shown in the following examples.

First example:

# Set up an array with non-finite elements
x = Double1d.range(5)+1
x[0] = Double.NaN
print x     # [NaN,2.0,3.0,Infinity,5.0]

# Keeping finite values
q = x.where(IS_FINITE)
print q     # [1,2,4]
print x[q]  # [2.0,3.0,5.0]

Example 2.27. Removing infinite and NaN values from an array.

Second example:

# Creating a mask
mask = IS_FINITE(x)
print mask  # [false,true,true,false,true]

q = x.where(mask)
print q     # [1,2,4]        (as before)
print x[q]  # [2.0,3.0,5.0]  (as before)

# Asking to filter the non-finite numbers:
q = x.where(~mask)
print q     # [0,3]
print x[q]  # [NaN,Infinity]

# Replacing selected values
x[q] = 0
print x     # [0.0,2.0,3.0,0.0,5.0] 

Example 2.28. Creating a filter (mask) with a function ro remove NaNs.

2.2.10. Advanced tips for improved performance

The underlying array operations and functions are very fast, as they are implemented in Java. The overhead of invoking them from Jython is relatively small for large arrays. However, the advanced user may find the following tips useful to improve performance in cases where it becomes a problem.

The arithmetic operations, such as '+', have versions that allow in-place modification of an array without copying. For example:

y = Double1d.range(10000)
y = y + 1 # The array is copied
y += 1 # The array is modified in place

Example 2.29. Avoiding unnecessary array allocation for the addition operation.

Copying an array is slow as it involves allocating memory (and subsequently garbage collecting it). For simple operations, such as addition, the copying can take longer than the actual addition.

Function application also involves copying the array. This can be avoided by using the Java API instead of the simple prefix function notation. For example:

x = Double1d.range(10000)
x = SIN(x) * COS(x) # This operation involves three copies
x = x.apply(SIN).multiply(x.apply(COS)) # Only one copy

Example 2.30. Using Java array utility methods to avoid wasteful array allocation.

When writing array expressions, it is better to group scalar operations together to avoid unnecessary array operations. For example:

y = Double1d([1,2,3,4])
print y * 2 * 3 # 2 array multiplications
print y * (2 * 3) # 1 array multiplication
print 2 * 3 * y # 1 array multiplication

Example 2.31. Grouping scalar multiplication avoids costly array multiplication.

It is better to avoid explicit loops over the elements of an array. It is often possible to achieve the same effect using existing array operations and functions. For example:

sum = 0.0
for i in y:
  sum = sum + i * i # Explicit iteration

sum = SUM(y * y) # Array operations

Example 2.32. Using arithmetic operations on arrays to avoid loops.

2.2.11. Type conversions

Since the numeric library supports different types it would be very convenient to be able to convert an array from one type to another. The numeric library supports both implicit conversion from within Jython for all supported dimensions and explicit conversion from one data type to another. Explicit conversion

Explicit conversion is supported for all data types by constructing a numeric array from another numeric array of the same or a different type. Note however that some explicit conversions may result in rounding and/or truncation of the values e.g. an explicit conversion from Long1d to Double1d will reduce the number of significant digits.

i = Int1d([1,2,3])                     # [1,2,3]
r = Double1d(i)                        # [1.0,2.0,3.0]
c = Complex1d(r)                       # [(1.0+0.0j),(2.0+0.0j),(3.0+0.0j)]
b = Byte1d(r)                          # [1,2,3]

Example 2.33. Converting types explicitly requires the creation of a numeric array of a specific type. Implicit conversion

Implicit conversions are conversions that can be done automatically, provided that such a conversion is a widening operation, for example. from Int1d to Double1d. Implicit narrowing conversions are not allowed and result in an error message.

[Note] Note

A widening operation is when the result is stored in more bits than the source, thus not losing any accuracy. A narrowing operation is the opposite and data loss is possible as the result is stored using fewer bits than the source.

The library supports implicit conversions in the following cases:

  • access: [...]

  • operators: +, -, *, /, ^ and %

  • in-line operators: +, -, *, /, ^ and %

The examples below show allowed implicit conversions.

d = Double1d(5)                        # [0.0,0.0,0.0,0.0,0.0]
d[1] = 3                               # [0.0,3.0,0.0,0.0,0.0]
d[1:4] = [-5, 0, 5]                    # [0.0,-5.0,0.0,5.0,0.0]

Example 2.34. Converting types implicitly in Jython.

HIPE considers the conversion from int to float and from long to float/double as an automatic widening operation, but some of the least significant digits of the value may be lost during the conversion. You will not be notified of this loss of significant digits.

Another thing to notice is that floating point operations will never throw an exception or error. As shown in the following example, a division by zero results in NaN or Infinity.

d = Double1d.range(5)
l = Long1d.range(5)
print d/l                         # [NaN,1.0,1.0,1.0,1.0]
print d/SHIFT(l, 1)               # [0.0,Infinity,2.0,1.5,1.3333333333333333]

Example 2.35. Dividing by zero will generate NaN or Infinity as appropriate.