Refresh pipeline

The Refresh Pipeline module in Tellius provides a smart orchestration layer for synchronizing Business Views (BVs) and their underlying datasets. It ensures streamlined data refreshes, reduces redundancy, and delivers end-to-end visibility into scheduled and on-demand data jobs.

Manage refresh pipeline

1. We've added a new tab Refresh Pipeline under Data module. Under Data → Refresh Pipeline, you can find the above page.

2. The search bar allows you to quickly find a pipeline by name.

3. Click on Add New Pipeline to open the configuration panel for creating a new pipeline.

4. Table Columns:

• Name: Name of the pipeline

• Business Views: Lists all Business Views that are part of the pipeline

• Frequency: Indicates how often the pipeline is scheduled to run

• Created By: Displays the creator of the pipeline

• Last Updated: Shows when the pipeline was last modified.

1. To edit a pipeline, click on the three-dot menu of that pipeline and select Edit.

2. To delete a pipeline, click on the three-dot menu of that pipeline and select Delete. Deleting a pipeline permanently removes the pipeline and an irreversible action.

How to create a new pipeline

This is your starting point for defining custom refresh workflows based on your organization’s data dependencies.

1. Click on the Add New Pipeline button and the following window will be displayed.

New pipeline creation

1. In the Pipeline Name, provide a unique name for your pipeline.

2. Select one or more Business Views to include in the pipeline.

3. Click on Add Pipeline button to create the pipeline. Or, click on Cancel to dismiss.

Pipeline Canvas

Once you've added a pipeline, it opens the following visual canvas editor. The visual editor is where you view the flow of your refresh pipeline. how the selected datasets and Business Views are arranged, execution order, and configure refresh rules. It gives you full visibility into how your data refreshes are orchestrated.

Pipeline canvas

1. In the left pane, you can find the pipeline scheduler:

• Start Date: Choose the date the pipeline should begin executing.

• Frequency: Set how often the pipeline should run (e.g., daily, weekly).

• Start Time: Set the exact time of day the pipeline should trigger.

1. In the central canvas area, you can find the visual flow builder.

2. Each node block represents a Dataset or a Business View. Shows which datasets are a part of each Business View in the pipeline.

3. Flat file nodes show the warning “Flat files can’t be refreshed” indicating non-refreshable sources.

4. For each node block, you can enable/disable refresh using the toggles. This helps you skip static or unnecessary datasets, reduce load time, and debug specific parts without deleting nodes.

Sometimes, different datasets within a pipeline often have different refresh cadences. That's why Tellius provides node-level toggle control. This allows reuse of the same pipeline for varied refresh schedules without duplication.

1. Click on Add Business View if you want to add Business View(s) to the current pipeline.

2. Click on Publish to save the pipeline and activate the configured schedule.

3. Click on Execute Now to trigger an on-demand refresh instantly.

4. If a node fails, execution halts, and errors are logged. You can retry from the point of failure.

5. Real-time updates on success/failure are pushed to the Notifications page.