A high-powered, low resource toolkit for your data

Unleash the full power of your hardware to snappily process your big multidimensional arrays either in memory or on disk.

 

Why ironArray?

We do believe that Big Data handling can play better with our environment by making a more intelligent use of energy. ironArray performs computations with large datasets using modern, cost-effective, multi-core CPUs and fast, local storage in an energy efficient way.

Highlights​

  • High performance multidimensional array computations
  • Automatic data compression in-memory or on disk
  • Dynamically tuned performance optimizations using artificial intelligence that leverage your specific CPUs, caches, memory and disks
  • User Defined Functions

Open-Source pedigree

The data container is based on the Blosc2 library and format and leverages the Caterva library for fast manipulation of multidimensional data. Get snappy slices of your datasets, no matter how big they are.

Open data format

ironArray uses the simple, expandable, well documented and open-source format from the Blosc2 library. Use contiguous or sparse storage, both in-memory or on-disk, with no compromises in speed.

How it works

ironArray organizes your data into chunks that fit into the last-level cache of your CPU and blocks that fit in first-levels caches. Then uses standard map, reduce, filter, and collect algorithms to perform calculations on large arrays directly in the high-speed CPU cache.

As a result, computations can make a much better use of scarce memory bandwidth in modern computers.  Furthermore, if performance computations are limited by memory bandwidth, this limit can even be surmounted!

Just express your performance and storage needs

And let ironArray do the rest.  Based on your preferences, ironArray will tune your configuration using state-of-the-art artificial intelligence algorithms, then create new arrays and perform calculations using the best optimizations.

For example, you can set a preference for high compression ratios, or for high-speed computation, or for some balance between the two.  Based on that, ironArray will compute suitable chunk and block shapes, number of threads and other parameters to better adapt to your requirements.

What ironArray can do for you

Reduce memory consumption

Tackle larger problems using less memory

Reduce cluster size​

Minimize infrastructure cost

Reduce complexity​

Minimize development and maintenance efforts

Improve slicing performance

Be smarter and fetch the data that you are interested in, not more.

Leverage compression automatically

Do more with less

Compute faster

Unleash the full capabilities of your hardware