What is ICE?

The ICE (In-Memory Compute Engine) powers interactive analytics in Tellius. It keeps frequently used data hot in a vectorized columnar format so queries, search, and visualizations run fast.

Purpose: Low-latency, in-memory analytics. Storage type: Hot cache (memory/disk), not permanent storage. Data flow: Source → ICE (optimize/compress/partition) → Cache → Interactive compute → (optional) Publish to FQE.

What it does

• Loads data in parallel from databases, cloud stores, and files.

• Stores and compresses data in a vectorized columnar format with fast in-memory compression for low-latency access.

• Shards and partitions data to maximize parallelism and data locality across the cluster.

• Caches data in memory, on local disk, or in a hybrid mode for predictable performance.

• Serves interactive workloads such as search, visualizations, transformations (ETL), Insights, and AutoML.

• Publishes denormalized Business Views to the query engine (FQE) for sub-second queries.

How caching works

• Memory cache: Keeps optimized, compressed columns in RAM for the fastest response.

• Disk cache: Stores the same optimized format on local disks to extend capacity with strong performance.

• Hybrid: Uses a configurable portion of RAM and spills the rest to disk for balanced speed and scale.

• Eviction policy: Least-recently used datasets are evicted automatically to make space for new workloads.

• Copy to System (optional): Keeps a Parquet copy in distributed storage to accelerate cache rebuilds without re-hitting source systems.

When to use ICE

• Interactive exploration with dashboards and search.

• Fast iteration on transformations and feature engineering.

• Running Insights and AutoML on hot data.

• Preparing denormalized Business Views for FQE.