Skip to content

Kafka exporter

Kafka exporter

The Kafka exporter: a separate SQLLoad mode that runs continuously and streams Kafka topics into Snowflake, Databricks (beta) or BigQuery (beta) tables.

The Kafka exporter is a separate mode of SQLLoad. It runs continuously, reads Kafka topics and writes each topic into a table. Its destinations are Snowflake, Databricks (beta) and BigQuery (beta), each through a streaming route or staged files.

Load project Kafka exporter
sqlload.toml [connections], [targets] and [sources] [source] with kind = "kafka", [destination] and [staging], plus the optional sections in Configuration
Runs Once per sqlload load Continuously: sqlload run exports until stopped, or for --max-duration
Destinations Every target Snowflake: Snowpipe Streaming, or files staged on a table stage or in S3. Databricks (beta): Zerobus Ingest, or Parquet staged in a Unity Catalog volume or in S3, then COPY INTO. BigQuery (beta): the Storage Write API, or Parquet load jobs, with the file sent along or staged in Cloud Storage
Commands debug, plan, apply, load, copy, contract, logged-state debug, tables, plan, apply, run, status

load, copy, contract and logged-state refuse an exporter configuration. The repository’s sqlload.example.toml is a complete exporter configuration.

[source]
kind = "kafka"
brokers = "${ENV:KAFKA_BROKERS}"
include = ["shop.*"]
[destination]
kind = "snowflake"
account = "myorg-myaccount"
user = "SQLLOAD"
auth = { method = "pat", token = "${ENV:SNOWFLAKE_TOKEN}" }
database = "RAW"
schema = "KAFKA"
[staging]
kind = "none"

The same file for Databricks changes only [destination]:

[destination]
kind = "databricks"
host = "dbc-1234.cloud.databricks.com"
client_id = "${ENV:DATABRICKS_CLIENT_ID}"
client_secret = "${ENV:DATABRICKS_CLIENT_SECRET}"
warehouse_id = "${ENV:DATABRICKS_WAREHOUSE_ID}"
catalog = "main"
schema = "kafka"
zerobus_endpoint = "https://1234567890.zerobus.us-west-2.cloud.databricks.com"

And for BigQuery:

[destination]
kind = "bigquery"
project = "my-project"
dataset = "kafka"
location = "EU"
service_account_key = "${ENV:BIGQUERY_KEY}"
Page Covers
Run it The commands in order, how rows reach each destination, and running several instances.
Configuration Every section and key of the exporter’s sqlload.toml, with defaults.
Topic naming How topics map to tables, stages, pipes and consumer groups.