# Connecting to Spark SQL

Spark SQL allows structured queries against data in an Apache Spark environment. By providing the proper host, port, and credentials, Tellius can communicate with your Spark cluster, retrieve schemas, and load data for downstream analytics.

Once you select Spark SQL, you are presented with a form to specify your connection parameters.

<figure><img src="https://content.gitbook.com/content/s16h5onryWtbaHwBa10b/blobs/qJ4NSXONjCGIoEY9kSZa/image.png" alt="" width="563"><figcaption><p>Spark SQL</p></figcaption></figure>

1. **Hostname:** This is the network location of your Spark SQL system. The hostname or IP address of the Oracle database server (e.g., 52.71.252.45). Without a correct hostname, Tellius cannot establish a connection.
2. **Port:** Databases listen on specific ports. Specify the port number through which your Spark Thrift server listens. Providing the correct port ensures your requests reach the right service.
3. **Database Name:** The **schema** or **database** (logical grouping of tables) you want Tellius to explore.
4. **User:** Provide the username with appropriate permissions (atleast read-access) to read data.
5. **Password:** Provide the corresponding password for the User provided.
6. **Datasource Name:** A user-friendly name for this connection in Tellius.
7. **Save and Browse Host:** Saves the connection details and attempts to browse the host for available schemas and tables. This initiates the handshake with the Spark server.

{% hint style="info" %}
If your database is behind a firewall, we display a Tellius IP address in this page that you may need to whitelist.
{% endhint %}

### Using validated datasource connection details

If you’ve previously validated and saved a Spark SQL connection, you can reuse its details:

<figure><img src="https://content.gitbook.com/content/s16h5onryWtbaHwBa10b/blobs/cx7Au2p2DiiOiXjVtjjN/image.png" alt="" width="563"><figcaption><p>Using already established connections</p></figcaption></figure>

1. **Use validated datasource connection details:** When enabled, it reveals a dropdown to choose from existing, previously configured Spark connections.
2. **Select datasource:** Lists all pre-validated Spark connections. Select the one you want to reuse and all the fields will be filled automatically as configured.
3. **Browse Host:** Similar to **“Save and Browse Host”**, but now it just navigates forward using the chosen existing connection’s parameters.

### Advanced Options

In the **Advanced option** view, you can override the default host/port fields by directly specifying a JDBC URL. This is especially useful if your Spark setup requires custom parameters or if your environment differs from typical host/port authentication.

<figure><img src="https://content.gitbook.com/content/s16h5onryWtbaHwBa10b/blobs/oP1B2mviiRjnEOuRcEBG/image.png" alt="" width="563"><figcaption><p>Advanced Option</p></figcaption></figure>

1. **JDBC URL:** A full JDBC connection string, such as:

   ```arduino
   jdbc:postgresql://52.71.252.45:1521/databaseName
   ```

   or the equivalent Spark SQL JDBC URL (e.g., `jdbc:hive2://host:10000/default;transportMode=http;httpPath=cliservice` for Spark Thrift).
2. **User:** The same **username** as in the main form, but here it pairs with your custom JDBC string if you’re not relying on the standard host/port fields.
3. **Password:** Provide the relevant password for the specified user in the advanced JDBC context.
4. **Datasource Name:** A user-friendly name for this connection in Telliu
5. **Save and Browse Host:** Saves the connection details and attempts to browse the host for available schemas and tables. This initiates the handshake with the Spark server.

### Loading tables

After establishing a connection, you will see options to load data from Spark SQL tables.

<figure><img src="https://content.gitbook.com/content/s16h5onryWtbaHwBa10b/blobs/z4jVK40obYbPkYONKFUq/image.png" alt="" width="563"><figcaption><p>Loading tables</p></figcaption></figure>

1. **Select a table**: Displays all available tables under the chosen Spark SQL schema. Pick the tables you need for analysis. If there are many tables, you can narrow down your selection.
2. **Search for table:** Filters the displayed tables based on your search term.
3. **Import:** Imports the selected table(s) into Tellius.

### Using Custom SQL

If you prefer more granular control or want to write your own SQL queries to load precisely the data you need, switch to "**Custom SQL**" tab.

<figure><img src="https://content.gitbook.com/content/s16h5onryWtbaHwBa10b/blobs/jKeEJfNHUFjGveUOam4t/image.png" alt="" width="563"><figcaption><p>Custom SQL window</p></figcaption></figure>

1. **Table filter**: Helps locate a particular table by name before writing your SQL.
2. **Select a table**: Choose a table name to use in your custom query.
3. **Query:** A field for your custom SQL statement (e.g., `SELECT * FROM SYS.WRI$_DBU_CPU_USAGE`).
4. **Preview:** Executes the SQL query and displays a few sample rows of the data you’re about to import in the **“Dataset Preview”** area. Allows you to validate that the query returns the correct data before fully importing it. This helps catch syntax errors or incorrect filters early.
5. **Import:** Once satisfied with the preview, click **Import** to load the data returned by the SQL query into Tellius.

### Advanced Settings

Once you import, you’ll have the option to refine how the dataset is handled:

1. **Dataset name:** Assign a valid name to your new dataset (e.g., `XYZ_THRESHOLD`). Names should follow the allowed naming conventions (letters, numbers, underscores, no leading underscores/numbers, no special chars/spaces).
2. **Connection Mode**\
   When the **Live** checkbox is selected, the queries will be fetched from the database each time, and the data will not be copied to Tellius. Live mode ensures the most up-to-date data at the cost of potential query latency.

<figure><img src="https://content.gitbook.com/content/s16h5onryWtbaHwBa10b/blobs/PcMv31hgSpE3gjliU2if/image.png" alt="" width="563"><figcaption><p>Advanced Settings</p></figcaption></figure>

{% hint style="warning" %}
When **Live** mode is enabled, then only **Create Business View** option will be displayed.
{% endhint %}

<figure><img src="https://content.gitbook.com/content/s16h5onryWtbaHwBa10b/blobs/x62Cff2qWORJgPoZHBrg/image.png" alt="" width="563"><figcaption><p>For non-live datasets</p></figcaption></figure>

3. **Copy to system**: If enabled, copies the imported data onto Tellius’s internal storage for faster performance. Reduces dependency on the source database’s speed and network latency. Good for frequently queried datasets.
4. **Cache dataset in memory:** If enabled, keeps a cached copy of the dataset in memory (RAM) for even faster query responses. Memory caching dramatically reduces query time, beneficial for dashboards and frequently accessed data.

When only one table is imported, the following options will also be displayed:

5. **Partitioning:** If enabled, it splits a large dataset into smaller logical chunks (partitions). Improves performance on large tables, enabling parallel processing and faster load times. For more details, check out [this](https://app.gitbook.com/o/S3VKMrzMgXbC36NqGRj8/s/JHwf1QFuv1BRPzfSnL2Z/~/changes/137/data/create-new-datasource/understanding-partitioning-your-data) dedicated page on **Partitioning**.

* **Partition column**: The column used as a basis for partitioning.
* **Number of partitions**: How many segments to break the data into. (e.g., 5 partitions)
* **Lower bound/Upper bound**: Approximate value range in the partition column to evenly distribute data.

6. **Create Business View**: If enabled, after loading data, you will be guided into the Business View creation stage.

Click on **Load** to finalize the process. After clicking **Load**, your dataset appears under **Data → Dataset**, ready for exploration, preparation, or business view configuration. Else, click on **Cancel** to discard the current importing process without creating the dataset.

After the dataset is created, you can navigate to **"Dataset",** where you can **r**eview and further refine your newly created dataset. Apply transformations, joins, or filters in the **Prepare** module.
