Numeric arrays available in HIPE are shown in the following table. A numeric array can have from one to five dimensions.
For completeness the following table also shows the
String1d array type, which is not a numeric array. String arrays can only be one-dimensional.
Table 2.1. Types of numeric array (N = 1...5)
Differences with Jython native arrays. Numeric arrays are optimised for holding Herschel data and for working with HIPE tasks and tools. Many tasks and functions accept Numeric arrays as their input, but not native Jython arrays. You are advised to use Numeric arrays when manipulating Herschel data.
See the following example for various ways in which you can create a one-dimensional array:
y = Double1d() # Create an empty array y = Double1d([3.0, 5.5, 2.1, 6.0]) # Create from a Jython array y = Double1d(4) # [0.0, 0.0, 0.0, 0.0] y = Double1d(4, 42.0) # [42.0, 42.0, 42.0, 42.0] y = Double1d.range(4) # [0.0, 1.0, 2.0, 3.0]
Example 2.1. Declaring an array of doubles.
You can create a complex array, with the same commands. In addition, you can specify real and imaginary parts separately:
y = Complex1d() # Create an empty array y = Complex1d([1+4j, 2+3j, 5+8j]) # Create from a Jython array y = Complex1d([1, 2, 5], [4, 3, 8]) # Same result as previous command y = Complex1d(4) # [0j, 0j, 0j, 0j] y = Complex1d(3, 1+4j) # [(1+4j),(1+4j),(1+4j)] y = Complex1d.range(4) # [0j,(1+0j),(2+0j),(3+0j)
The following example shows how to create a two-dimensional array. You can create arrays of more dimensions in the same way:
y = Double2d() # Create an empty array y = Double2d([[3.0, 5.5], [2.1, 6.0]]) # Create from a Jython array y = Double2d(4, 4) # Creates a 4x4 array whose values are all zero. y = Double2d(4, 4, 42.0) # Creates a 4x4 array whose values are all 42.
Example 2.2. Declaring a two-dimensional array of doubles.
Rectangular and jagged arrays. Rectangular arrays are multidimensional arrays which always have the same number of elements in each row or column. Jython and Java allow you to create jagged arrays. Jagged arrays are multidimensional arrays where each row can have a different number of elements. The following example creates a two-dimensional jagged array, with two rows of two and three elements, respectively:
However, you cannot create Numeric jagged arrays:
x = Double2d([[1,2], [3,4,5]]) # Gives an error
Example 2.4. It is impossible to create Numeric jagged arrays.
Array elements along each dimension are defined by an index running from zero to the array length, minus one:
x = Double1d([3.0, 5.0, 9.0, 4.3]) # Four elements print x # Prints the first element, 3.0 print x # Prints the fourth element, 4.3
Example 2.5. Accessing array elements using the indices.
To access ranges of elements, or slices, use a notation like
print x[1:3] # From index 1, included, to 3, excluded # [5.0,9.0] print x[2:] # From index 2, included, to end of array # [9.0,4.3] print x[:3] # From start of array to index 3, excluded # [3.0,5.0,9.0]
Example 2.6. Using array slices to access ranges from Jython lists.
To access elements in multi-dimensional arrays, separate indices or ranges along each dimension with commas:
x = Int2d([[1,2,3],[4,5,6]]) # 1 2 3 # 4 5 6 print x # 2 (second element of the first row) print x[0,:] # Row 0: [1,2,3] print x[1,1] # Individual element: 5 print x[:,:] # Print entire array. Same as print x print x[:,1] # Column 1: [2,5]
Example 2.7. Accessing ranges of indices using slices.
Multi-dimensional arrays are conceptually arrays of lower-dimensional arrays. For a two-dimensional array, the first subscript selects a row and the second subscript selects an element within that row (the column).
This is the opposite order to some other computer languages, but it is the same behaviour as in the Java programming language.
Note the difference in syntax when inspecting native Jython arrays and numeric arrays:
# Jython array: x = [[1,2,3,4],[5,6,7,8]] print x # 7 print x[1:3] # 6, 7 # Numeric array: y = Int2d([[1,2,3,4],[5,6,7,8]]) print y[1,2] # 7 print y[1,1:3] # 6, 7
Example 2.8. Checking the differences between Jython arrays and numeric arrays.
Complex arrays offer the following additional commands:
z = Complex1d([1,2,3,4],[4,3,2,1]) # Set up complex array print z.real # [1.0,2.0,3.0,4.0] print z.imag # [4.0,3.0,2.0,1.0] print z.conjugate() # [(1.0-4.0j),(2.0-3.0j),(3.0-2.0j),(4.0-1.0j)]
Example 2.9. Inspecting and manipulating a Complex Numeric array.
You can append single values or entire arrays to an existing array:
y = Double1d() y.append(2.0) # Append a single value y.append(Double1d([3.0, 7.5, 2.8])) # Append a whole array
Example 2.10. Appending values to an array.
Individual elements or slices can be set as follows:
x[1,2] = 22 # Set an element in place x[0,1:3] = 42 print x # [ # [2.0,42.0,42.0], # [1.0,3.0,22.0] # ]
Example 2.11. Assigning values with the use of indices and slice notation.
It is possible to set a row to a copy of a 1d array of the same length:
x[0,:] = [5,6,7,8] # Set a row to (a copy of) a Jython array y[1,:] = Int1d([9,7,6,5]) # Set a row to a Double1d array
Example 2.12. Assign arrays to arrays using slice notation.
The following line of code creates an array of two rows and three columns:
You can access the element corresponding to the i-th row and j-th column like this:
The values are stored sequentially in memory as follows:
[2 4 6 1 3 5]
This means that, if you go through the array elements as they are stored in memory, their indices would vary as follows:
x[0,0] x[0,1] x[0,2] x[1,0] x[1,1] x[1,2]
That is, index j varies more rapidly than index i. This can be generalised to more than two dimensions by saying that the rightmost index varies most rapidly. This is called row-major ordering, and is the convention followed by languages such as Java and C, but not Fortran.
This has an implication on performance. When looping through a multidimensional array, it is more efficient to read its elements in the order they are stored in memory.
Confusion may arise when dealing with images, which are stored as two-dimensional arrays. If you visualise the array with horizontal rows and vertical columns, then the number of rows and columns represents the size of the vertical (y) and horizontal (x) side of the image, respectively. When accessing a particular pixel (array element), you have to specify the y coordinate before the x coordinate:
HIPE numeric arrays support arithmetic operations that are applied element-by-element. For example:
y = Double1d.range(5) # [0.0,1.0,2.0,3.0,4.0] print y * y * 2 + 1 # [1.0,3.0,9.0,19.0,33.0]
Example 2.15. Applying multiplication and addition to all elements of an array.
This is much simpler (and runs much faster) than writing an explicit loop in Jython. The '+' operator does not concatenate arrays, as it does with Jython arrays. For example:
# Adding Jython arrays print [0,1,2,3] + [4,5,6,7] # [0, 1, 2, 3, 4, 5, 6, 7] # Adding numeric arrays print Double1d([0,1,2,3]) + Double1d([4,5,6,7]) # [4.0,6.0,8.0,10.0] # Concatenate two numeric arrays print Double1d([0,1,2,3]).append(Double1d([4,5,6,7])) # [0.0,1.0,2.0,3.0,4.0,5.0,6.0,7.0] # Adding Jython arrays to numeric arrays print [0,1,2,3] + Double1d([4,5,6,7]) # [4.0,6.0,8.0,10.0] print Double1d([0,1,2,3]) + [4,5,6,7] # [4.0,6.0,8.0,10.0]
Example 2.16. Concatenating numeric arrays.
All arrays support the following arithmetic operators:
+, -, *, /, %, **
Note that the 'modulo' operator '%' provides the normal Jython semantics for this operation, which is not the same as that of the Java '%' operator. The Jython definition is more consistent with the mathematical notion of congruence for negative values.
The following relational operators return a
<, >, <=, >=, ==, !=
y = Double1d([0,1,2,3,4]) print y > 2 # [false,false,false,true,true]
Example 2.17. Applying relational operators to a Numeric array.
where function to select the elements of an array that satisfy a given condition:
y = Double1d([2,6,3,8,1,9]) print y.where(y > 4) # [1,3,5] Indices of elements greater than four
Example 2.18. Filtering array elements with the where method.
Note that the result contains the indices of the elements that satisfy the condition, not the elements themselves. You can combine more conditions with the
| (or) and
~ (not) operators:
print y.where((y > 4) & (y < 9)) # [1,3] print y.where((y > 8) | (y < 2)) # [4,5] print y.where(~(y > 4)) # [0,2,4]
Example 2.19. More complex filtering using the where method.
You can obtain the actual array elements instead of the indices as follows:
print y[y.where(y > 4)] # [6.0,8.0,9.0] print y.get(y > 4) # [6.0,8.0,9.0]
Example 2.20. Accessing the array values with a filter.
Obtaining the indices rather than the actual values can be useful, for instance, when you want to select the same elements from different arrays:
x = Double1d([5,6,7,8,9,10]) s = y.where(y > 4) print x[s] + y[s] # [12.0,16.0,19.0]
Example 2.21. Adding two arrays with the same set of filtered array indices.
You can also use the
where function to set values:
s = y.where(y > 4) y[s] = 0 # Set all matching elements to 0 print y # [2.0,0.0,3.0,0.0,1.0,0.0] y[s] = [9,8,7] # Set matching elements using an array print y # [2.0,9.0,3.0,8.0,1.0,7.0]
Example 2.22. Assigning values with the results of the where method.
You cannot use the
where function like this:
a = Double1d.range(10) b = a.where(a < 3) print b # AttributeError: __getitem__ print b[0:2] # AttributeError: __getitem__ print a[b] # AttributeError: __getitem__
Example 2.23. The output list of where is not accessible by index.
The commands fail because
b is a
Selection object rather than a Jython or Numeric array. For the above to work you need to convert it to
c = b.toInt1d() print c # 0 print c[0:2] # [0,1] print a[c] # 0.0
Example 2.24. Converting the output of where to a normal array that you can manipulate.
By converting to
Int1d you can also loop over all the selected elements:
for i in c: print a[i]
Example 2.25. Converting the output of where makes the resulting object iterable.
Another useful function is
get, with which you can extract individual elements or a subset of element values from an array. There are four ways to use it:
Get a single value:
HIPE>print y.get(0) 2.0
This is the same as
Retrieve elements based on a
Bool1d array (in other words, a mask):
HIPE>mask = Bool1d([0,0,1,0,1,1])
HIPE>print y.get(mask) [3.0,1.0,9.0]
The mask array can be shorter than the target array (in which case the remaining elements in the target array will be ignored), but not longer.
Retrieve elements based on a
HIPE>indices = Selection([1,2,4])
HIPE>print y.get(indices) [6.0,3.0,1.0]
An out of bounds index will cause an error.
Retrieve elements based on a
HIPE>range = Range(2,4) print y.get(range) [3.0,8.0]
Note how the lower boundary (second element) is included, while the upper boundary (fourth element) is excluded. An out of bounds value will cause an error.
You can combine
get calls to perform the same operation as a compound IDL
WHERE execution such as the one shown in this example:
IDL>a = [1, 2, 3, 4, 5, 6]
IDL>b = [2, 3, 4, 5, 6, 7]
IDL>c = [3, 4, 5, 6, 7, 8]
IDL>q = WHERE(a ge 2 and b lt 6 and c gt 5)
IDL>x = [a[q],b[q],c[q]]
IDL>print, x 4 5 6
This is the equivalent in HIPE:
q = (a >= 2) & (b < 6) & (c > 5) x = a.get(q),b.get(q),c.get(q) # x == ([4.0], [5.0], [6.0])
HIPE>a = Int1d([1, 2, 3, 4, 5, 6])
HIPE>b = Int1d([2, 3, 4, 5, 6, 7])
HIPE>c = Int1d([3, 4, 5, 6, 7, 8])
HIPE>q = (a >= 2) & (b < 6) & (c > 5)
HIPE>x = a.get(q),b.get(q),c.get(q)
HIPE>print x (, , )
The Jython logical
not work like normal Boolean operators
(see Appendix A for more details), but using them with arrays
(both the native Jython arrays and HIPE numeric arrays) can
give unexpected results. The reason is that these operators do not work
on an element-by-element basis when applied to arrays, but they evaluate
the entire array at once.
The Jython bitwise operators, represented by the
Appendix A for more details) can be used with numeric arrays (
Bool3d and so on), but what you get is
not a bitwise comparison. Instead, these operators
perform the usual boolean comparisons, but this time working element by
element. Precisely what
not do not do.
Finally, Numeric array classes have the
xor methods acting
like boolean operators working element by element. An
example will clarify the differences among all these operators:
jythonOne = [1, 0, 0, 1] jythonTwo = [0, 0, 1, 1] numericOne = Bool1d(jythonOne) numericTwo = Bool1d(jythonTwo) print jythonOne and jythonTwo ## [0, 0, 1, 1] # jythonOne is not empty so it is treated as true, which means that # jythonTwo is evaluated and returned print numericOne and numericTwo ## [false,false,true,true] # Same thing as with the Jython native arrays print jythonOne & jythonTwo ## Here an error is returned print numericOne & numericTwo ## [false,false,false,true] # Here the operator works element by element print numericOne.and(numericTwo) ## [false,false,false,true] # Same thing as the & operator
Example 2.26. Differences between Jython and Numeric arrays.
To remove infinite and NaN values from an array, use the
IS_NAN functions as shown in the following examples.
# Set up an array with non-finite elements x = Double1d.range(5)+1 x = Double.NaN x = Double.POSITIVE_INFINITY print x # [NaN,2.0,3.0,Infinity,5.0] # Keeping finite values q = x.where(IS_FINITE) print q # [1,2,4] print x[q] # [2.0,3.0,5.0]
Example 2.27. Removing infinite and NaN values from an array.
# Creating a mask mask = IS_FINITE(x) print mask # [false,true,true,false,true] q = x.where(mask) print q # [1,2,4] (as before) print x[q] # [2.0,3.0,5.0] (as before) # Asking to filter the non-finite numbers: q = x.where(~mask) print q # [0,3] print x[q] # [NaN,Infinity] # Replacing selected values x[q] = 0 print x # [0.0,2.0,3.0,0.0,5.0]
Example 2.28. Creating a filter (mask) with a function ro remove NaNs.
The underlying array operations and functions are very fast, as they are implemented in Java. The overhead of invoking them from Jython is relatively small for large arrays. However, the advanced user may find the following tips useful to improve performance in cases where it becomes a problem.
The arithmetic operations, such as '
+', have versions
that allow in-place modification of an array without copying. For
y = Double1d.range(10000) y = y + 1 # The array is copied y += 1 # The array is modified in place
Example 2.29. Avoiding unnecessary array allocation for the addition operation.
Copying an array is slow as it involves allocating memory (and subsequently garbage collecting it). For simple operations, such as addition, the copying can take longer than the actual addition.
Function application also involves copying the array. This can be avoided by using the Java API instead of the simple prefix function notation. For example:
x = Double1d.range(10000) x = SIN(x) * COS(x) # This operation involves three copies x = x.apply(SIN).multiply(x.apply(COS)) # Only one copy
Example 2.30. Using Java array utility methods to avoid wasteful array allocation.
When writing array expressions, it is better to group scalar operations together to avoid unnecessary array operations. For example:
y = Double1d([1,2,3,4]) print y * 2 * 3 # 2 array multiplications print y * (2 * 3) # 1 array multiplication print 2 * 3 * y # 1 array multiplication
Example 2.31. Grouping scalar multiplication avoids costly array multiplication.
It is better to avoid explicit loops over the elements of an array. It is often possible to achieve the same effect using existing array operations and functions. For example:
sum = 0.0 for i in y: sum = sum + i * i # Explicit iteration sum = SUM(y * y) # Array operations
Example 2.32. Using arithmetic operations on arrays to avoid loops.
Since the numeric library supports different types it would be very convenient to be able to convert an array from one type to another. The numeric library supports both implicit conversion from within Jython for all supported dimensions and explicit conversion from one data type to another.
Explicit conversion is supported for all data types by constructing a numeric array from another numeric array of the same or a different type. Note however that some explicit conversions may result in rounding and/or truncation of the values e.g. an explicit conversion from Long1d to Double1d will reduce the number of significant digits.
i = Int1d([1,2,3]) # [1,2,3] r = Double1d(i) # [1.0,2.0,3.0] c = Complex1d(r) # [(1.0+0.0j),(2.0+0.0j),(3.0+0.0j)] b = Byte1d(r) # [1,2,3]
Example 2.33. Converting types explicitly requires the creation of a numeric array of a specific type.
Implicit conversions are conversions that can be done automatically, provided that such a conversion is a widening operation, for example. from Int1d to Double1d. Implicit narrowing conversions are not allowed and result in an error message.
A widening operation is when the result is stored in more bits than the source, thus not losing any accuracy. A narrowing operation is the opposite and data loss is possible as the result is stored using fewer bits than the source.
The library supports implicit conversions in the following cases:
operators: +, -, *, /, ^ and %
in-line operators: +, -, *, /, ^ and %
The examples below show allowed implicit conversions.
d = Double1d(5) # [0.0,0.0,0.0,0.0,0.0] d = 3 # [0.0,3.0,0.0,0.0,0.0] d[1:4] = [-5, 0, 5] # [0.0,-5.0,0.0,5.0,0.0]
Example 2.34. Converting types implicitly in Jython.
HIPE considers the conversion from int to float and from long to float/double as an automatic widening operation, but some of the least significant digits of the value may be lost during the conversion. You will not be notified of this loss of significant digits.
Another thing to notice is that floating point operations will never throw an exception or error. As shown in the following example, a division by zero results in NaN or Infinity.
d = Double1d.range(5) l = Long1d.range(5) print d/l # [NaN,1.0,1.0,1.0,1.0] print d/SHIFT(l, 1) # [0.0,Infinity,2.0,1.5,1.3333333333333333]
Example 2.35. Dividing by zero will generate NaN or Infinity as appropriate.