Installation steps for Tellius

How to install Tellius

Tellius requires deployment on a Kubernetes cluster. This can be hosted on a managed service like AWS EKS or Azure AKS, or on a single VM (single node) cluster for initial pilot use cases.

There are two deployment options:

• Customer Hosted

• Tellius Hosted

Customer Hosted

If the infrastructure is hosted on the customer’s cloud or on-prem setup, the customer has complete control over the infrastructure (instances, subnets, networking etc.). In this case, Tellius team will only be responsible for deploying the application and will need proper access to the cluster to do the installation.

Tellius Hosted

If Tellius handles the ownership of the infrastructure, usually on AWS EKS or Azure AKS, the Tellius team is responsible for both deploying the application and managing the infrastructure.

Deployment Options

There are two components in Tellius that can be deployed in standalone or distributed mode:

• ClickHouse

• Spark

ClickHouse Standalone

In ClickHouse Standalone mode, the pods clickhouse-set-0 are deployed in the Kubernetes cluster.

ClickHouse Distributed

In ClickHouse Distributed mode, a ClickHouse operator is installed and shards are used to replicate data across ClickHouse nodes. In this type of installation, pods like chi-repl-05-replicated-0-0-0 are deployed, depending on the number of ClickHouse replicas that have been decided for the Tellius deployment.

Spark Standalone

In the standalone version, only the spark-restserver-0 pod is present, which runs all the Spark-related jobs. This is a good fit for low-intensity data operations.

Spark Distributed

When using Spark in distributed mode, there will be additional components deployed along with the spark-restserver-0 pod. These include the spark-master deployment and one or more spark-worker pods. The workload of the spark jobs and tasks will be distributed across these pods instead of running in a single location, making it suitable for processing large amounts of data.

Deployment Steps

Prerequisites

• Working Kubernetes Cluster

• Access to the Kubernetes Cluster

• Access to the internet

• Access to Tellius Private Elastic Container Registry (ECR) for Docker Images

• Download the Tellius-deploy zip file provided by the Tellius team.

The Tellius-deploy zip file will contain the manifests needed to setup the Kubernetes resources on the cluster.

Steps to install

1. Unzip the zip file and navigate to the folder from the command line.

2. Install the following resources by running the kubectl commands. Based on your type of installation, choose CHOICE A or CHOICE B.

# install the docker cronjob to create the ECR secret
kubectl create -f cron-jobs
kubectl create job -n docker-registry --from=cronjob/aws-registry-credential-cron aws-registry-credential-cron-001

# setup telemetry services
kubectl create ns observability
kubectl create -f telemetry/jaeger-development.yml -n observability

# install tellius services
kubectl create -f rabbitmq
kubectl create -f secrets
kuebctl create -f telemetry/app-jaeger-config.yml
kubectl create -f sqe
kubectl create -f spark-restserver
kubectl create -f frontend
kubectl create -f azkaban 
kubectl create -f opencpu
kubectl create -f bot-server
kubectl create -f hdfs
kubectl create -f cdn
kubectl create -f python-evaluator
kubectl create -f viz-engine
kubectl create -f resource_tracking
kubectl create -f data-loader
kubectl create -f clickhouse-autoscaler

# [CHOICE A] for standalone mode
kubectl create -f sqe/clickhouse.yaml

# [CHOICE B] for distributed mode
# install spark-cluster and clickhouse operator
kubectl create -f spark-cluster
kubectl create -f clickhouse-operator

# setup monitoring
kubectl create namespace monitoring
kubectl create -f monitoring

Validation

After completing the setup, you can verify the status of the pods by running the command kubectl get pods. Ensure that all the pods are running without any errors. If everything has been configured correctly, the setup should be up and running within 30 minutes. You can access the Tellius UI by using the load balancer IP or DNS domain name that has been set up.

Additionally, after creating a new instance, Tellius QA conducts a comprehensive checklist on the environment to ensure that there are no issues with the configuration or networking.

Troubleshooting and Assistance

If you encounter any issues during the setup process, you can contact the Tellius DevOps team for assistance in resolving the issues and getting the application up and running.

Please note that the above-mentioned steps may be subject to change based on the version of Tellius or the configuration of your infrastructure. Additional functionality, such as connecting to data sources outside of the cluster or through other gateways, will need to be handled on a case-by-case basis by the Tellius DevOps Team.

Please use your individual username to access Tellius. Sharing usernames can lead to complications and unintended consequences on the platform. For a secure and seamless experience, each user must log in with a unique username.