Kafka exporter
Kafka exporter
The Kafka exporter: a separate SQLLoad mode that runs continuously and streams Kafka topics into Snowflake, Databricks (beta) or BigQuery (beta) tables.
The Kafka exporter is a separate mode of SQLLoad. It runs continuously, reads Kafka topics and writes each topic into a table. Its destinations are Snowflake, Databricks (beta) and BigQuery (beta), each through a streaming route or staged files.
How it differs from a load project
Section titled “How it differs from a load project”| Load project | Kafka exporter | |
|---|---|---|
sqlload.toml |
[connections], [targets] and [sources] |
[source] with kind = "kafka", [destination] and [staging], plus the optional sections in Configuration |
| Runs | Once per sqlload load |
Continuously: sqlload run exports until stopped, or for --max-duration |
| Destinations | Every target | Snowflake: Snowpipe Streaming, or files staged on a table stage or in S3. Databricks (beta): Zerobus Ingest, or Parquet staged in a Unity Catalog volume or in S3, then COPY INTO. BigQuery (beta): the Storage Write API, or Parquet load jobs, with the file sent along or staged in Cloud Storage |
| Commands | debug, plan, apply, load, copy, contract, logged-state |
debug, tables, plan, apply, run, status |
load, copy, contract and logged-state refuse an exporter configuration. The repository’s
sqlload.example.toml
is a complete exporter configuration.
[source]kind = "kafka"brokers = "${ENV:KAFKA_BROKERS}"include = ["shop.*"]
[destination]kind = "snowflake"account = "myorg-myaccount"user = "SQLLOAD"auth = { method = "pat", token = "${ENV:SNOWFLAKE_TOKEN}" }database = "RAW"schema = "KAFKA"
[staging]kind = "none"The same file for Databricks changes only [destination]:
[destination]kind = "databricks"host = "dbc-1234.cloud.databricks.com"client_id = "${ENV:DATABRICKS_CLIENT_ID}"client_secret = "${ENV:DATABRICKS_CLIENT_SECRET}"warehouse_id = "${ENV:DATABRICKS_WAREHOUSE_ID}"catalog = "main"schema = "kafka"zerobus_endpoint = "https://1234567890.zerobus.us-west-2.cloud.databricks.com"And for BigQuery:
[destination]kind = "bigquery"project = "my-project"dataset = "kafka"location = "EU"service_account_key = "${ENV:BIGQUERY_KEY}"| Page | Covers |
|---|---|
| Run it | The commands in order, how rows reach each destination, and running several instances. |
| Configuration | Every section and key of the exporter’s sqlload.toml, with defaults. |
| Topic naming | How topics map to tables, stages, pipes and consumer groups. |