A high-powered, low resource toolkit for your data
Unleash the full power of your hardware to snappily process your big multidimensional arrays either in memory or on disk.
We do believe that Big Data handling can play better with our environment by making a more intelligent use of the energy, so we are making ironArray in such a way that it allows computations with large datasets to make a more effective use of modern, cost-effective, multi-core CPUs and fast, local storage.
- High performance multidimensional array computations
- Automatic data compression in-memory or on disk
- Dynamically tuned performance optimizations using artificial intelligence that leverage your specific CPUs, caches, memory and disks
- User Defined Functions
The data container is based on the Blosc2 library and format and leverages the Caterva library for fast manipulation of multidimensional data. Get snappy slices of your datasets, no matter how big they are.
Open data format
ironArray uses the simple, expandable, well documented and open-source format from the Blosc2 library. Use contiguous or sparse storage, both in-memory or on-disk, with no compromises in speed.
How it works
ironArray organizes your data into chunks that fit into the last-level cache of your CPU and blocks that fit in first-levels caches. Then uses standard map, reduce, filter, and collect algorithms to perform calculations on large arrays directly in the high-speed CPU cache.
As a result, computations can make a much better use of scarce memory bandwidth in modern computers. Furthermore, if performance computations are limited by memory bandwidth, this limit can even be surmounted!
Just express your performance and storage needs
And let ironArray do the rest. Based on your preferences, ironArray will tune your configuration using state-of-the-art artificial intelligence algorithms, then create new arrays and perform calculations using the best optimizations.
For example, you can set a preference for high compression ratios, or for high-speed computation, or for some balance between the two. Based on that, ironArray will compute suitable chunk and block shapes, number of threads and other parameters to better adapt to your requirements.