> ## Documentation Index
> Fetch the complete documentation index at: https://cubed3-feat-druid-driver-streaming.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Databricks

> Databricks is a unified data intelligence platform.

[Databricks](https://www.databricks.com) is a unified data intelligence platform.

## Prerequisites

* [A JDK installation][gh-cubejs-jdbc-install]
* The [JDBC URL][databricks-docs-jdbc-url] for the [Databricks][databricks]
  cluster

## Setup

### Environment Variables

Add the following to a `.env` file in your Cube project:

```dotenv theme={null}
CUBEJS_DB_TYPE=databricks-jdbc
# CUBEJS_DB_NAME is optional
CUBEJS_DB_NAME=default
# You can find this inside the cluster's configuration
CUBEJS_DB_DATABRICKS_URL=jdbc:databricks://dbc-XXXXXXX-XXXX.cloud.databricks.com:443/default;transportMode=http;ssl=1;httpPath=sql/protocolv1/o/XXXXX/XXXXX;AuthMech=3;UID=token
# You can specify the personal access token separately from [`CUBEJS_DB_DATABRICKS_URL`](/reference/configuration/environment-variables#cubejs_db_databricks_url) by doing this:
CUBEJS_DB_DATABRICKS_TOKEN=XXXXX
# This accepts the Databricks usage policy and must be set to `true` to use the Databricks JDBC driver
CUBEJS_DB_DATABRICKS_ACCEPT_POLICY=true
```

### Docker

Create a `.env` file [as above](#environment-variables), then extend the
`cubejs/cube:jdk` Docker image tag to build a Cube image with the JDBC driver:

```dockerfile theme={null}
FROM cubejs/cube:jdk

COPY . .
RUN npm install
```

You can then build and run the image using the following commands:

```bash theme={null}
docker build -t cube-jdk .
docker run -it -p 4000:4000 --env-file=.env cube-jdk
```

## Environment Variables

| Environment Variable                                                                                                                  | Description                                                                                                   | Possible Values       | Required |
| ------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------- | --------------------- | :------: |
| [`CUBEJS_DB_NAME`](/reference/configuration/environment-variables#cubejs_db_name)                                                     | The name of the database to connect to                                                                        | A valid database name |     ✅    |
| [`CUBEJS_DB_DATABRICKS_URL`](/reference/configuration/environment-variables#cubejs_db_databricks_url)                                 | The URL for a JDBC connection                                                                                 | A valid JDBC URL      |     ✅    |
| [`CUBEJS_DB_DATABRICKS_ACCEPT_POLICY`](/reference/configuration/environment-variables#cubejs_db_databricks_accept_policy)             | Whether or not to accept the license terms for the Databricks JDBC driver                                     | `true`, `false`       |     ✅    |
| [`CUBEJS_DB_DATABRICKS_OAUTH_CLIENT_ID`](/reference/configuration/environment-variables#cubejs_db_databricks_oauth_client_id)         | The OAuth client ID for [service principal][ref-databricks-m2m-oauth] authentication                          | A valid client ID     |     ❌    |
| [`CUBEJS_DB_DATABRICKS_OAUTH_CLIENT_SECRET`](/reference/configuration/environment-variables#cubejs_db_databricks_oauth_client_secret) | The OAuth client secret for [service principal][ref-databricks-m2m-oauth] authentication                      | A valid client secret |     ❌    |
| [`CUBEJS_DB_DATABRICKS_TOKEN`](/reference/configuration/environment-variables#cubejs_db_databricks_token)                             | The [personal access token][databricks-docs-pat] used to authenticate the Databricks connection               | A valid token         |     ❌    |
| [`CUBEJS_DB_DATABRICKS_CATALOG`](/reference/configuration/environment-variables#cubejs_db_databricks_catalog)                         | The name of the [Databricks catalog][databricks-catalog] to connect to                                        | A valid catalog name  |     ❌    |
| [`CUBEJS_DB_EXPORT_BUCKET_MOUNT_DIR`](/reference/configuration/environment-variables#cubejs_db_export_bucket_mount_dir)               | The path for the [Databricks DBFS mount][databricks-docs-dbfs] (Not needed if using Unity Catalog connection) | A valid mount path    |     ❌    |
| [`CUBEJS_DB_MAX_POOL`](/reference/configuration/environment-variables#cubejs_db_max_pool)                                             | The maximum number of concurrent database connections to pool. Default is `8`                                 | A valid number        |     ❌    |
| [`CUBEJS_CONCURRENCY`](/reference/configuration/environment-variables#cubejs_concurrency)                                             | The number of [concurrent queries][ref-data-source-concurrency] to the data source                            | A valid number        |     ❌    |

[ref-data-source-concurrency]: /admin/connect-to-data/concurrency#data-source-concurrency

## Pre-Aggregation Feature Support

### count\_distinct\_approx

Measures of type
[`count_distinct_approx`][ref-schema-ref-types-formats-countdistinctapprox] can
be used in pre-aggregations when using Databricks as a source database. To learn
more about Databricks's support for approximate aggregate functions, [click
here][databricks-docs-approx-agg-fns].

## Pre-Aggregation Build Strategies

<Info>
  To learn more about pre-aggregation build strategies, [head
  here][ref-caching-using-preaggs-build-strats].
</Info>

| Feature       | Works with read-only mode? | Is default? |
| ------------- | :------------------------: | :---------: |
| Simple        |              ✅             |      ✅      |
| Export Bucket |              ✅             |      ❌      |

By default, Databricks JDBC uses a [simple][self-preaggs-simple] strategy to
build pre-aggregations.

### Simple

No extra configuration is required to configure simple pre-aggregation builds
for Databricks.

### Export Bucket

Databricks supports using both [AWS S3][aws-s3] and [Azure Blob
Storage][azure-bs] for export bucket functionality.

#### AWS S3

To use AWS S3 as an export bucket, first complete [the Databricks guide on
connecting to cloud object storage using Unity Catalog][databricks-docs-uc-s3].

<Info>
  Ensure the AWS credentials are correctly configured in IAM to allow reads and
  writes to the export bucket in S3.
</Info>

```dotenv theme={null}
CUBEJS_DB_EXPORT_BUCKET_TYPE=s3
CUBEJS_DB_EXPORT_BUCKET=s3://my.bucket.on.s3
CUBEJS_DB_EXPORT_BUCKET_AWS_KEY=<AWS_KEY>
CUBEJS_DB_EXPORT_BUCKET_AWS_SECRET=<AWS_SECRET>
CUBEJS_DB_EXPORT_BUCKET_AWS_REGION=<AWS_REGION>
```

#### Google Cloud Storage

<Info>
  When using an export bucket, remember to assign the **Storage Object Admin**
  role to your Google Cloud credentials ([`CUBEJS_DB_EXPORT_GCS_CREDENTIALS`](/reference/configuration/environment-variables#cubejs_db_export_gcs_credentials)).
</Info>

To use Google Cloud Storage as an export bucket, first complete [the Databricks guide on
connecting to cloud object storage using Unity Catalog][databricks-docs-uc-gcs].

```dotenv theme={null}
CUBEJS_DB_EXPORT_BUCKET=gs://databricks-export-bucket
CUBEJS_DB_EXPORT_BUCKET_TYPE=gcs
CUBEJS_DB_EXPORT_GCS_CREDENTIALS=<BASE64_ENCODED_SERVICE_CREDENTIALS_JSON>
```

#### Azure Blob Storage

To use Azure Blob Storage as an export bucket, follow [the Databricks guide on
connecting to Azure Data Lake Storage Gen2 and Blob Storage][databricks-docs-azure].

[Retrieve the storage account access key][azure-bs-docs-get-key] from your Azure
account and use as follows:

```dotenv theme={null}
CUBEJS_DB_EXPORT_BUCKET_TYPE=azure
CUBEJS_DB_EXPORT_BUCKET=wasbs://my-container@my-storage-account.blob.core.windows.net
CUBEJS_DB_EXPORT_BUCKET_AZURE_KEY=<AZURE_STORAGE_ACCOUNT_ACCESS_KEY>
```

Access key provides full access to the configuration and data,
to use a fine-grained control over access to storage resources, follow [the Databricks guide on authorize with Azure Active Directory][authorize-with-azure-active-directory].

[Create the service principal][azure-authentication-with-service-principal] and replace the access key as follows:

```dotenv theme={null}
CUBEJS_DB_EXPORT_BUCKET_AZURE_TENANT_ID=<AZURE_TENANT_ID>
CUBEJS_DB_EXPORT_BUCKET_AZURE_CLIENT_ID=<AZURE_CLIENT_ID>
CUBEJS_DB_EXPORT_BUCKET_AZURE_CLIENT_SECRET=<AZURE_CLIENT_SECRET>
```

## SSL/TLS

Cube does not require any additional configuration to enable SSL/TLS for
Databricks JDBC connections.

## Additional Configuration

### Cube Cloud

To accurately show partition sizes in the Cube Cloud APM, [an export
bucket][self-preaggs-export-bucket] **must be** configured.

[aws-s3]: https://aws.amazon.com/s3/

[azure-bs]: https://azure.microsoft.com/en-gb/services/storage/blobs/

[azure-bs-docs-get-key]: https://docs.microsoft.com/en-us/azure/storage/common/storage-account-keys-manage?toc=%2Fazure%2Fstorage%2Fblobs%2Ftoc.json&tabs=azure-portal#view-account-access-keys

[authorize-with-azure-active-directory]: https://learn.microsoft.com/en-us/rest/api/storageservices/authorize-with-azure-active-directory

[azure-authentication-with-service-principal]: https://learn.microsoft.com/en-us/azure/developer/java/sdk/identity-service-principal-auth

[databricks]: https://databricks.com/

[databricks-docs-dbfs]: https://docs.databricks.com/en/dbfs/mounts.html

[databricks-docs-azure]: https://docs.databricks.com/data/data-sources/azure/azure-storage.html

[databricks-docs-uc-s3]: https://docs.databricks.com/en/connect/unity-catalog/index.html

[databricks-docs-uc-gcs]: https://docs.databricks.com/gcp/en/connect/unity-catalog/cloud-storage.html

[databricks-docs-jdbc-url]: https://docs.databricks.com/integrations/bi/jdbc-odbc-bi.html#get-server-hostname-port-http-path-and-jdbc-url

[databricks-docs-pat]: https://docs.databricks.com/dev-tools/api/latest/authentication.html#token-management

[databricks-catalog]: https://docs.databricks.com/en/data-governance/unity-catalog/create-catalogs.html

[gh-cubejs-jdbc-install]: https://github.com/cube-js/cube/blob/master/packages/cubejs-jdbc-driver/README.md#java-installation

[ref-caching-using-preaggs-build-strats]: /docs/pre-aggregations/using-pre-aggregations#pre-aggregation-build-strategies

[ref-schema-ref-types-formats-countdistinctapprox]: /reference/data-modeling/measures#type

[databricks-docs-approx-agg-fns]: https://docs.databricks.com/en/sql/language-manual/functions/approx_count_distinct.html

[self-preaggs-simple]: #simple

[self-preaggs-export-bucket]: #export-bucket

[ref-databricks-m2m-oauth]: https://docs.databricks.com/aws/en/dev-tools/auth/oauth-m2m