Added a lot of functionality today. SciEx now support arrays of floats of dimension up to 6 (SciEx.Float64.Array1, …, SciEx.Float64.Array6). Most of the rust code (and some of the elixir code) in the bindings is now programmatically generated from the Elixir code as part of a separate build step.

I have added support for the most common floating point functions, including some special functions. With the new programmatic code generation tools, adding support for more functions is trivial. The functions are currently only vectorized in the first argument, but I plan on adding vectorization on the other arguments.

I have added support for drawing random variables according to the given distribution. For example, one can draw 15 values from a Cauchy distribution:

iex> SciEx.Random.draw_from_cauchy(0.0, 1.0, 15)
#SciEx.F64.Array1<[0.5075428433765116, 6.5273500168873, -0.09238828295381869,
-0.5017462511724131, 0.31446532306974234, -0.12084509091920821, -0.35170244950655,
-7.796726646392933, 0.4049382192392038, -1.5996980781784342, -1.0434302700931213, 
0.8270864602500656, -1.7333229127273764, 0.541653889771595, 129.4432667174581],
shape=[15], strides=[1], layout=CFcf (0xf), const ndim=1>

The PDF and CDF for the supported distributions is also implemented. However, they are only vectorized on the first argument.

I’m not sure on whether I’m taking the right approach regarding the rust code, though. I am writing (and compiling) separate rust functions for each supported type combination because that makes the interop with Elixir easier, although I could probably get away with generating less rust code…

I have also added support for arithmetic operators on arrays and scalars. Array types and dimensions must match, otherwise SciEx will raise a mistake. Everything is very explicit (you can use SciEx.Operators).

iex(4)> a = SciEx.Random.draw_from_normal(0.0, 1.0, 10)
#SciEx.F64.Array1<[0.980546308591242, -0.4124865578379766, -0.2678229458353892, 0.13895119159920113, -0.05895938590440134, 0.5463383504051798, -0.47139019152800626, 1.1432875805108502, -1.053279376911445, -0.4890897953025381], shape=[10], strides=[1], layout=CFcf (0xf), const ndim=1>
iex(5)> b = SciEx.Random.draw_from_normal(0.0, 1.0, 10)
#SciEx.F64.Array1<[-0.6112979257481231, 0.04187616378275234, 0.024658043411414837, -0.7423086166282216, 2.5287898587121993, -1.5968438074887625, 0.3028680311098233, 1.1657682958696634, -1.3245816353940532, 1.1693983808728805], shape=[10], strides=[1], layout=CFcf (0xf), const ndim=1>
iex(6)> 0.5 * a + 0.7 * b # This will fail beause it will try to use the Kernel operators
** (ArithmeticError) bad argument in arithmetic expression: 0.5 * #SciEx.F64.Array1<[0.980546308591242, -0.4124865578379766, -0.2678229458353892, 0.13895119159920113, -0.05895938590440134, 0.5463383504051798, -0.47139019152800626, 1.1432875805108502, -1.053279376911445, -0.4890897953025381], shape=[10], strides=[1], layout=CFcf (0xf), const ndim=1>
    :erlang.*(0.5, #SciEx.F64.Array1<[0.980546308591242, -0.4124865578379766, -0.2678229458353892, 0.13895119159920113, -0.05895938590440134, 0.5463383504051798, -0.47139019152800626, 1.1432875805108502, -1.053279376911445, -0.4890897953025381], shape=[10], strides=[1], layout=CFcf (0xf), const ndim=1>)
iex(6)> use SciEx.Operators # this imports the operators in SciEx and hides the ones in the Kernel
SciEx
iex(7)> 0.5 * a + 0.7 * b  
#SciEx.F64.Array1<[0.06236460627193485, -0.17692996427106167, -0.1166508425297042, -0.45014043584015456, 1.7406732081463385, -0.8446214900395438, -0.02368747398712684, 1.3876815973641894, -1.4538468332315597, 0.5740339689597472], shape=[10], strides=[1], layout=CFcf (0xf), const ndim=1>

Due to the fact that Explorer (the elixir library) is based on Polars (the rust library), which is based on a rust implementation of Arrow (a standard to store data in memory), I have tried to re-implement SciEx to be based on Arrow instead of ndarray, in order to make it comaptible with Explorer. It would also make it much easier to store and move data around.

However, I haven’t been able to make it work in a satisfying way… Arrow doesn’t even provide a multidimensional array type you can operate on as a unit. It does provide tensors, but it does not provide vectoried operations on tensors and I can’t think of a good way to implement something like a Fast Fourier Transform algorithm on Arrow datatypes.

If anyone can think of a way to “port” SciEx to Arrrow I’d be very interested.

I have added support for loading 2D-arrays from grayscale PNG images and support for contouring “heightmaps” (represented as 2D arrays) using the marching-squares algorithm. @kip I’m sorry, but I haven’t been able to work on integrating SciEx with your image processing tools yet.

I plan on wrapping my own kernel density estimation (KDE) functions and expose them to Elixir as part of SciEx. Once this is done, I plan on using SciEx for the most computationally heavy parts of my Quartz plotting library.

Request for help

I’d like to request help (or maybe even tips) regarding the following topics:

Code quality and compile times

Currently, I implement arrays of complex numbers (32 or 64 bits) for 1 to 6 dimensions as separate rust types (2x2x6=24 different types of arrays). I think this makes it easy to improve communication between Elixir and rust because it makes almost everything “monomorphic” on the rust side and allows me to deal with non-unique types only on the Elixir side. However, I wonder if this doesn’t make it a bit harder to implement functions with multile arguments on the rust side. It also requires creating a huge number of rust functions (usually at least one per array type). Is there a way to leverage traits or something like that to decrease compile-times and improve the code size?

Array dimensions

Currently, I have hardcoded the array dimensions as being between 1 and 6. Is this a problem? Has anyone ever used dense arrays of dim > 6 in practice? A dense array of dimension > 6 is usully so huge I can’t think of a use for it…

RustlerPrecompiled

Is anyone interested in integrating rustler precompiled in this package?

Thanks in advance to all interested people