# 2 posts tagged with "big data"

View All Tags

## Evaluating Expressions in Blosc2

Intern at ironArray SLU

## What expressions are?​

The forthcoming version of Blosc2 will bring a powerful tool for performing mathematical operations on pre-compressed arrays, that is, on arrays whose data has been reduced in size using compression techniques. This functionality provides a flexible and efficient way to perform a wide range of operations, such as addition, subtraction, multiplication and other mathematical functions, directly on compressed arrays. This approach saves time and resources, especially when working with large data sets.

An example of expression evaluation in Blosc2 might be:

``dtype = np.float64shape = [30_000, 4_000]size = shape[0] * shape[1]a = np.linspace(0, 10, num=size, dtype=dtype).reshape(shape)b = np.linspace(0, 10, num=size, dtype=dtype).reshape(shape)c = np.linspace(0, 10, num=size, dtype=dtype).reshape(shape)#  Convert numpy arrays to Blosc2 arraysa1 = blosc2.asarray(a, cparams=cparams)b1 = blosc2.asarray(b, cparams=cparams)c1 = blosc2.asarray(c, cparams=cparams)# Perform the mathematical operationexpr = a1 + b1 * c1  # LazyExpr expressionexpr += 2  # expressions can be modifiedoutput = expr.eval(cparams=cparams)  # evaluate! (output is compressed too)``

Compressed arrays ( a1, b1, c1) are created from existing numpy arrays ( a, b, c) using Blosc2, then mathematical operations are performed on these compressed arrays using general algebraic expressions. The evaluation of these expressions is lazy, in that they are not evaluated immediately, but are meant to be evaluated later. Finally, the resulting expression is actually evaluated (via `.eval()`) and the desired output (compressed as well) is obtained.