Reference
sqlload.toml
Every key of a loader project's sqlload.toml: connections, targets, sources and table settings.
sqlload.toml is a loader project’s configuration. sqlload reads it from the working directory,
or from -c/--config or SQLLOAD_CONFIG. Unknown keys are errors. See
Projects for how the parts fit together.
${ENV:NAME} anywhere in the file is replaced with the environment variable NAME; an unset
variable is an error. $${ is a literal ${.
The Kafka exporter uses a different file format.
Top level
Section titled “Top level”| Key | Default | Meaning |
|---|---|---|
python_paths |
none | Folders, relative to the file, scanned for Python sources. |
default_target |
none | The target for sources that name no destination. Not needed with one target. |
concurrency |
4 | The most tables loading at once. --concurrency overrides it. |
memory_limit |
80% of a container’s or Windows job object’s limit, else 50% of the machine’s memory | The memory a run stays under, such as "4GiB", or "off" for no budget. --memory-limit overrides it. See Memory budget. |
microbatch_concurrency |
false |
Lets tables read several batch_size windows at once with batch_concurrency; plan refuses batch_concurrency above 1 without it. |
[connections.NAME] |
How to reach a system. | |
[targets.NAME] |
Where loaded tables go. | |
[sources.NAME] |
Where data comes from. |
Connections
Section titled “Connections”Each connection has a kind.
postgres
Section titled “postgres”For Postgres sources, and for Postgres targets too.
| Key | Default | Meaning |
|---|---|---|
host |
required | |
port |
5432 | |
user |
required | |
password |
none | Use ${ENV:NAME}. |
database |
required | |
sslmode |
prefer |
disable, prefer, require, verify-ca or verify-full, as in libpq. See TLS. |
sslrootcert |
none | CA certificates (PEM) to trust, instead of the public roots built into SQLLoad. |
sslcert, sslkey |
none | A client certificate and its private key (PEM). Set both or neither. |
ssh |
none | A bastion to reach host through. See SSH tunnels. |
The ssh table:
| Key | Default | Meaning |
|---|---|---|
host |
required | The bastion. |
port |
22 | |
user |
required | |
private_key_file |
ssh-agent | The private key to sign in with. ~/ is the home folder. Without it (and without password), SQLLoad tries each key ssh-agent holds. |
private_key_passphrase |
none | Use ${ENV:NAME}. |
password |
none | Signs in when no private_key_file is set. Use ${ENV:NAME}. |
known_hosts |
~/.ssh/known_hosts |
The file the bastion’s host key must be listed in. |
snowflake
Section titled “snowflake”| Key | Default | Meaning |
|---|---|---|
account |
required | The account identifier, such as myorg-myaccount. |
user |
required | |
auth |
required | { method = "key_pair", private_key_file = "..." }, { method = "pat", token = "${ENV:...}" }, or { method = "oauth_authorization_code" } to sign in in the browser. See Signing in. |
role |
none | |
warehouse |
none | |
database |
required | |
load_method |
required for targets | How data gets into Snowflake. A connection only a source reads needs none. See Load methods. |
databricks
Section titled “databricks”In beta.
| Key | Meaning |
|---|---|
host, client_id, client_secret, warehouse_id, catalog, volume |
See Databricks. All required, except volume on a connection only a source uses. |
bigquery
Section titled “bigquery”In beta.
| Key | Default | Meaning |
|---|---|---|
location |
required | Where query jobs run and datasets are created, such as EU. |
service_account_key |
required | The service account key file’s JSON text. Use ${ENV:NAME}. |
project |
the key’s project | The project tables load into and jobs run in. |
See BigQuery.
clickhouse
Section titled “clickhouse”| Key | Default | Meaning |
|---|---|---|
host |
required | |
port |
8123, or 8443 with secure |
The HTTP port. |
user |
default |
|
password |
none | Use ${ENV:NAME}. |
secure |
false |
true for HTTPS. |
See ClickHouse.
duckdb
Section titled “duckdb”| Key | Meaning |
|---|---|
database |
The database file, or :memory:. |
motherduck
Section titled “motherduck”In beta, as a target and as a source. It connects through a separate
DuckDB 1.5 process: install sqlload[motherduck]; see MotherDuck.
| Key | Meaning |
|---|---|
database |
The MotherDuck database, with or without md:. Required, here or in the target. It can’t hold the token. |
token |
A MotherDuck access token. Use ${ENV:NAME}. Default: the motherduck_token or MOTHERDUCK_TOKEN environment variable. |
load_method |
{ type = "local" } (default): zstd Parquet chunks sent from this machine. { type = "external_stage", url = "s3://…" | "gs://…", connection = "…", keep_staged_files = false }: chunks written to a bucket, which MotherDuck’s servers read. See Staging in a bucket. |
| Key | Meaning |
|---|---|
bucket |
The bucket. |
region |
The bucket’s region. |
endpoint |
A custom S3-compatible endpoint. |
profile |
The profile of the AWS shared credentials file to take keys from. Default: AWS_PROFILE, else default. |
SQLLoad takes the first credentials it finds, in this order:
AWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEY(withAWS_SESSION_TOKEN) in the environment. As in the AWS SDKs and DuckDB,AWS_TOKENisn’t read as the session token.- Keys in the shared credentials file (
AWS_SHARED_CREDENTIALS_FILE, default~/.aws/credentials), under the connection’sprofile, elseAWS_PROFILE, elsedefault. - Web identity.
- The instance or container role.
A profile you name, with profile or AWS_PROFILE, must have
aws_access_key_id and aws_secret_access_key in that file; otherwise the connection fails rather
than trying the next source. SSO, assumed roles and credential_process in ~/.aws/config aren’t
read. sqlload debug shows where each S3 connection’s keys come from, never the
keys.
SQLLoad removes files in bulk, up to 1,000 per request. For an S3-compatible store without bulk
delete, set AWS_DISABLE_BULK_DELETE=true: SQLLoad then deletes each file with its own request.
A Google Cloud Storage bucket, for files sources and files targets.
| Key | Meaning |
|---|---|
bucket |
The bucket. |
service_account_key |
A service account’s JSON key, such as "${ENV:GCS_KEY}". |
hmac_key_id, hmac_secret |
An HMAC key for the bucket, which DuckDB and MotherDuck need to read its Parquet files themselves. Set both or neither; write the secret as "${ENV:NAME}". |
endpoint |
A custom endpoint, such as a GCS emulator’s. |
SQLLoad reads and writes the bucket with the first credentials it finds, in this order:
- The connection’s
service_account_key. - The key file
GOOGLE_APPLICATION_CREDENTIALSnames, which must exist. - gcloud’s application default credentials (
gcloud auth application-default login) (Beta). - The service account of the machine SQLLoad runs on, from its metadata server (Beta).
They need roles/storage.objectViewer on the bucket to read it, and roles/storage.objectUser
to write files or remove them after a load. sqlload debug shows where each GCS connection’s
credentials come from, never the credentials.
DuckDB and MotherDuck read GCS only with an HMAC key, not a service-account key. To have them read
a bucket’s Parquet files themselves, create an HMAC key for a service account that can read it
(Cloud Storage → Settings → Interoperability) and set hmac_key_id and hmac_secret. Without one,
SQLLoad reads the files and sends their rows, and sqlload plan -v <table> says why.
BigQuery needs no HMAC key: it reads a bucket’s Parquet files with its own connection’s service
account, on a table that excludes _sqlload_source_row; see Parquet in Google Cloud
Storage.
filesystem
Section titled “filesystem”| Key | Meaning |
|---|---|
root |
A folder on the machine SQLLoad runs on. On Windows also a drive path ('C:\data\lake') or a network share ('\\server\share\lake'); write it as a TOML literal string, in single quotes, so the backslashes stay. |
An SFTP server that files sources read from.
| Key | Default | Meaning |
|---|---|---|
host |
required | |
port |
22 | |
user |
required | |
private_key_file |
none | The private key to sign in with. ~/ is the home folder. |
private_key_passphrase |
none | Use ${ENV:NAME}. |
password |
none | Signs in when no private_key_file is set. Use ${ENV:NAME}. Without either, SQLLoad tries each key ssh-agent holds. |
known_hosts |
required | The file the server’s host key must be listed in. SQLLoad never trusts a server on first use, and host key checks can’t be turned off. |
See ODBC.
| Key | Meaning |
|---|---|
connection_string |
The driver’s connection string. Use ${ENV:NAME}. |
dsn |
A data source name, instead of connection_string. |
user, password |
With dsn, when the DSN doesn’t hold them. Set connection_string or dsn, not both. |
For Python HTTP sources; its settings reach table functions as
ctx.connection. The connection holds everything about the API that every request shares: how
requests authenticate, the limits they stay under together, and how often a failed one is retried.
| Key | Default | Meaning |
|---|---|---|
base_url |
required | The API’s base URL. |
auth |
none | How requests authenticate, a table such as { kind = "bearer" }; see the table below. Without it, requests carry no credential. |
token |
none | For auth bearer and api_key. Use ${ENV:NAME}. |
username, password |
none | For auth basic and oauth2_password. |
client_id, client_secret |
none | For auth oauth2_client_credentials, and for oauth2_refresh_token or oauth2_password when the API wants the client too. |
refresh_token |
none | For auth oauth2_refresh_token. |
headers |
none | Headers sent with every request. |
rate_limit |
none | Requests started per second, minute or hour: "10/s", "600/m" or "1000/h". |
max_in_flight |
none | The most requests open at once, from 1 to 10,000. |
retries |
5 | Retries of a request that failed with a transport error, 429 or a 5xx, at most 100. |
rate_limit and max_in_flight hold for every request to the connection in one sqlload load:
every table, page, read_ahead request, fan-out key
and retry. A retried request takes a turn and a place again. Separate processes don’t share them,
so load one API’s tables in one sqlload load; in an orchestrator, that means one task per API, not
one per endpoint.
auth |
Sends |
|---|---|
{ kind = "bearer" } |
Authorization: Bearer <token>. |
{ kind = "api_key", header = "X-Api-Key" } |
token in that header. |
{ kind = "api_key", query = "key" } |
token in that query parameter. |
{ kind = "basic" } |
Basic auth with username and password. |
{ kind = "oauth2_client_credentials", token_url = "...", scope = "...", client_auth = "basic" } |
Authorization: Bearer <access token>, the token token_url issues for client_id and client_secret. scope is optional; client_auth is basic (default) or post. See OAuth2. |
{ kind = "oauth2_refresh_token", token_url = "..." } |
The same, with the token issued for refresh_token. Takes scope and client_auth too. |
{ kind = "oauth2_password", token_url = "..." } |
The same, with the token issued for username and password (the resource-owner password grant). Takes scope and client_auth too. |
[connections.shop]kind = "http"base_url = "https://api.example.com/v1"token = "${ENV:SHOP_TOKEN}"auth = { kind = "bearer" }rate_limit = "10/s"max_in_flight = 4plan shows each http connection’s settings, never its credentials:
http shop rate_limit 10/s, max_in_flight 4, retries 5, auth bearerTargets
Section titled “Targets”| Key | Applies to | Meaning |
|---|---|---|
connection |
all | A snowflake, databricks, bigquery, clickhouse, postgres, duckdb, s3 or filesystem connection, or a motherduck connection. |
loader_schema |
warehouses, required | The schema for tables whose source sets no destination_schema; a BigQuery target’s dataset, a ClickHouse target’s database. |
database |
warehouses except ClickHouse | Overrides the connection’s database. |
path |
files | Each table’s folder template, with {source}, {schema}, {table}, {cursor} and {cursor:date}. Default {source}/{table}/. |
format |
files | parquet (default), or csv. |
compression |
files, csv | none (default), gzip or zstd. |
csv |
files, csv | { delimiter, header, quote, null_value }; default ,, true, " and an empty string. |
single_file |
files | One file per table or partition, named after the table folder. Default false. |
max_footer_reads |
files | The most Parquet footers plan reads for one table’s schema. Default 100. |
naming |
warehouses | snake_case (default) or source: the case of table and column names. A source or table may override it. See Naming. |
compat |
warehouses | sqlload (default) or dlt: the names of the columns a load adds and of the child tables unnest makes. A source or table may override it. See compat. |
send |
warehouses | at_commit or per_batch: whether each commit’s rows wait locally and reach the target in one burst at the commit, or each batch goes as it’s read. Default at_commit on Snowflake, Databricks and MotherDuck, per_batch elsewhere. A source or table may override it. See Sending rows at commit. |
staging_dir |
warehouses | The folder send = "at_commit" holds rows in until their commit. Default: the system’s temporary folder. |
json_type |
DuckDB, MotherDuck | json (default) or variant: whether json columns are DuckDB JSON or VARIANT. See VARIANT columns. |
Sources
Section titled “Sources”| Key | Applies to | Meaning |
|---|---|---|
kind |
all | postgres, postgres_cdc, snowflake, odbc, bigquery, databricks or files. Python sources are declared in Python. |
connection |
all | The connection the source reads. |
description |
all | The source’s description; a files source’s table description. |
destination |
all | The target; default_target when omitted. |
destination_schema |
warehouse targets | The schema the source’s tables land in; the target’s loader_schema when omitted. The schema they are read from is chosen by include. |
table_name |
all | The destination name template, with {source}, {schema} (the source schema) and {table}. Default {source}__{table}, or {source} for files. |
destination_path |
files targets | The folder template for this source’s tables, overriding the target’s path. |
include, exclude |
postgres, postgres_cdc, snowflake, odbc, bigquery, databricks | schema.table patterns to sync and to leave out (dataset.table for bigquery). |
new_tables |
postgres | sync (default), skip or fail. See Sources. |
cdc_mode, slot, publication, batch_interval |
postgres_cdc | See Postgres CDC settings. |
exclude_columns, allow_columns |
all | Column patterns to leave out, and to keep anyway. |
skip_tables_with_columns |
postgres | Leave out tables with a column matching one of these patterns. |
mask_key |
all | The key for hmac masks, normally ${ENV:MASK_KEY}. |
warehouse |
snowflake | The warehouse unloads run on; the connection’s when omitted. |
stage |
snowflake | Where unloads go: { url, storage_integration, region } or { name }; the user stage when omitted. See Snowflake. |
max_text_size, max_binary_size |
odbc | The longest text and binary value a read holds, in bytes; caps every text and binary column. See ODBC. |
max_batch_memory |
odbc | The fetch buffer of each table: the most the rows of one fetch hold, such as "2GB"; 512 MiB, or a quarter of each table’s share of the memory budget when less, when omitted. It doesn’t bound the load’s whole memory. See ODBC. |
long_column_threshold |
odbc | The buffer per value over which a text or binary column is read value by value; "4KiB" when omitted. See ODBC. |
project |
bigquery | The project the tables live in; the connection’s when omitted. |
path |
files | The folder holding the files, relative to the connection. |
format |
files | parquet (default), ndjson, json or csv. |
compression |
files | auto (default), none, gzip or zstd. |
csv |
files | { delimiter, header, quote, null_values }; defaults ",", true, '"', [""]. Also escape (none by default), quote = false and encoding_check (true). See Quotes and escapes. |
pattern |
files | A glob each file’s path below path must match, such as "file_*.txt", in place of the format’s name suffix. See Choosing files. |
shape |
files | columns (default) or record: each JSON object or CSV line whole in one json column. See Whole records. |
record_column |
files | The column shape = "record" loads into; record when omitted. |
partition_columns |
files | Hive key=value folders as typed columns, such as { dt = "date" }. |
path_columns |
files | A path template whose captures load as typed columns, such as "{region}/{day:date}/*". See Columns from paths. |
unmatched_paths |
files | fail (default) or skip: what a file whose path doesn’t match path_columns does. |
order |
files | modified (default) or path: the order a merge or delete_insert by replace_key applies files in, the last file winning a key several hold. See Merging files. |
on_empty |
files | fail (default) or skip: what a load does when the folder exists and holds no files. See Empty folders. |
copy_to_internal_stage |
files into Snowflake | Upload files no external stage covers to @~. Much slower. |
[sources.NAME.tables."schema.table"] |
all | Per-table settings. A files source’s table is named after the source. |
A source can also set any table setting, as the default for its tables.
Postgres CDC settings
Section titled “Postgres CDC settings”A postgres_cdc source takes these:
| Key | Default | Meaning |
|---|---|---|
kind |
postgres_cdc. |
|
connection |
A postgres connection. |
|
include, exclude |
schema.table patterns, as for a Postgres source. |
|
cdc_mode |
merge |
merge keeps one row per key; changelog appends every change. A files target takes changelog only. |
deletes |
soft |
cdc_mode = "merge" only: soft marks a deleted row with _sqlload_deleted_at; hard removes it. A table setting, so a table may override it. |
slot |
sqlload_<source> |
The replication slot. |
publication |
sqlload_<source> |
The publication. SQLLoad creates sqlload_<source> when it’s missing. |
batch_interval |
10s, 1m into files |
With load --follow: how often it writes what it has read. At least 1s. |
Column settings (columns, exclude_columns, mask and the rest) work as for other sources. A
captured table takes no write_strategy, cursor_column, batch_size or state. merge
matches rows on the primary key, and a unique_key must lie within it.
Table settings
Section titled “Table settings”Set on a source for all its tables, or in a [sources.NAME.tables."schema.table"] entry for one. The schema in an entry’s name is the source schema; where tables land is the source’s destination_schema.
| Key | Default | Meaning |
|---|---|---|
write_strategy |
required | append, merge, delete_insert or replace. Every table needs one, from its own entry or its source; plan refuses a table without it. See Write strategies. |
unique_key |
required for merge |
A column or list of columns, or "{primary_key}" for the primary key the source reports (not for Python, files or Google Sheets sources). A merge table needs one. See merge. |
on_duplicate_keys |
dedupe |
What merge does with rows of one commit that repeat a key: dedupe keeps one row per key, fail refuses the commit. See Repeated keys. |
null_keys |
match |
What merge does with a NULL in a unique_key column, and delete_insert in a replace_key column: match matches it with a NULL, so the row is updated rather than inserted again; fail refuses a commit holding one. See NULLs in keys. |
replace_key |
none | A column or list of columns whose values delete_insert replaces whole: each commit deletes the rows holding a value its rows hold, then inserts them. Not unique. Only for delete_insert, and not with a cursor_column: each load reads the whole source, or a files source’s new and changed files. See Replacing rows by a key. |
deletes |
none | soft or hard: what load --sync-deletes does with the rows of a merge table whose key the source no longer holds (Postgres, ODBC, Snowflake and BigQuery sources, and Databricks sources, into DuckDB and Postgres targets, Snowflake, MotherDuck, and BigQuery and Databricks). soft sets _sqlload_deleted_at; hard removes the row. A postgres_cdc source applies captured deletes this way, soft by default. See Rows deleted at the source. |
deletes_from |
none | Tables only: { table = "audit.deleted_items", cursor_column = "deleted_at" }, a source table recording deleted keys. Every load loads its new rows into <table>__sqlload_deletes and marks the rows they name, as deletes says. lookback = "1h" reads that far back again on every load, for tombstones written by long transactions. See Deletes from a tombstone table. |
on_overlapping_files |
fail |
What a files source’s delete_insert by replace_key does when two files of its folder hold one key value: fail refuses the commit holding the later file, naming both; last_wins lets the file last in order replace the value. See Replacing rows by a key. |
newest_by |
none | A column, or a list compared in order, whose higher value is a row’s newer version: merge keeps the newest of a commit’s rows sharing a key, and never overwrites the table’s row with an older one. Without it, the row read last wins. See Repeated keys. |
cursor_column |
none | The column runs continue from. A list such as ["updated_at", "created_at"] reads the first non-NULL column of each row. See Cursors. |
cursor_start |
none | Where the first run starts. |
lookback |
none | Read this far before the maximum: a duration on a date or timestamp cursor, such as "2h"; a number of cursor units on an integer cursor, such as 1. See Cursors. |
lookback_from |
destination |
destination (the destination’s maximum) or now: what lookback counts back from. See Reload a window. |
lookahead |
none | Read only cursor values before now plus this; a date cursor reads up to and including today plus that many days. |
append_cursor_inclusive |
false |
Whether append reads from the maximum itself (inserting its rows again) rather than after it. |
where |
none | Postgres, ODBC, Snowflake and BigQuery tables, including those include matches; not query tables (put the condition in their sql). A condition in the source database’s SQL, such as "status <> 'X' AND site = 'BE01'", ANDed into every SELECT SQLLoad reads the table with, next to the cursor filter, so the table keeps its primary key and the database can use its indexes. A table’s own replaces its source’s. plan checks it on the source and shows it on a filter line. A row that later stops matching isn’t removed from a merge, append or delete_insert table; replace and deletes remove it. See Filtering rows. |
cursor_bounds |
parameters |
ODBC sources only. parameters binds the cursor’s bounds as typed parameters; literals writes them into the query as ODBC escapes ({ts '…'}, {d '…'}), for a driver that refuses a bound value. See Cursor bounds. |
on_cursor_out_of_range |
fail |
Python tables with a cursor only. What a load does with rows the function returns outside the window it was asked for: fail fails the table, naming them and their likely cause; skip drops them; warn loads them. The load report counts them under skip and warn. See Rows outside the window. |
null_cursor |
fail |
What an incremental read does with source rows whose cursor is NULL: fail fails the load when there are any, include reads them on every run (merge only), ignore skips them. See NULL cursor values. |
batch_size |
none | Commit one cursor window at a time: a duration ("1d") or an integer step, or "none". |
batch_concurrency |
1 | Postgres tables with batch_size windows; needs microbatch_concurrency = true. The most windows read at once, up to 16, each over its own connection in one shared snapshot, still committed one at a time in window order. See Reading windows at once. |
contract |
none |
none or enforced. See Contracts. |
on_schema_change |
append_new_columns |
append_new_columns, ignore or fail. |
auto_widen |
false |
Widen inferred columns along bool → int → decimal, timestamp_ntz → timestamp_tz. |
infer_types |
true |
false loads every column SQLLoad would infer (files but Parquet, Google Sheets, Python dict rows) as text; pinned columns keep their types. See Loading every column as text. |
on_nonfinite |
keep |
Python tables and tables with a transform only. null loads NaN and infinite floats in dict rows as NULL; keep keeps them where the column holds them and fails the table, naming the column and row, where it can’t. See NaN and infinities. |
allow_empty |
false |
Python tables only. true lets replace (or --reload) empty the table when the function yields no rows. See Python. |
state |
stateless |
Python tables only. logged keeps the table’s position in the run log (ctx.state) instead of deriving it from the data. See Logged state. |
type_change_rebuild |
require_confirmation |
allow, require_confirmation or deny. |
checks |
none | { unique, not_null, min_rows, max_row_drop, warn }, run on the staged rows before they’re published. See Checks before publish. |
constant_columns |
none | The same value in every row, such as { tenant_id = "acme" }, typed by its TOML value or { value, type }. On every source but postgres_cdc, and per table, where a table’s map replaces its source’s. A files source replaces a file column of the same name, and plan notes it; every other source fails at plan when it already has a column of the name, ignoring case, before anything is written. See Several tenants in one table. |
unnest |
none | In a table entry only: the dotted paths of the list fields whose items become rows of child tables, such as ["items", "items.parts"], or "all". See Nested data. |
flatten |
none | In a table entry only: the dotted paths of the objects whose keys load as <object>__<key> columns, such as ["customer", "customer.address"], or "all". JSON and NDJSON files default to "all" under compat = "dlt". See Flattening objects. |
max_depth |
1000 |
In a table entry with unnest or flatten: how many levels of child tables it makes and objects it flattens at most. |
parse |
none | In a table entry only: the text columns read as times, by column, such as { updated = { timestamp = ["iso", "%m/%d/%Y %H:%M"], tz = "Europe/Paris" } } or { created = "epoch_ms" }. See Parsing text into times. |
timezone |
none | The IANA zone, such as "Europe/Paris", of times without an offset: those a parse without its own tz reads, and a cursor_start, --start-cursor or --end-cursor on a cursor of times. A Python table’s ctx.cursor_value and ctx.cursor_end arrive in it. UTC when unset. See Times without an offset. |
naming |
the target’s | snake_case or source. See naming. |
rename |
none | A map of source names to loaded names, such as { "Order ID" = "order_ref" }, or the name a Python function was registered under with @sl.rename. See rename. |
on_duplicate_names |
number |
number or fail: what happens when columns load under one name, ignoring case. See on_duplicate_names. |
compat |
the target’s | sqlload or dlt. See compat. |
send |
the target’s | at_commit or per_batch. See Sending rows at commit. |
row_id |
false |
Give every row a unique id, in _sqlload_row_id. Always on with unnest and under compat = "dlt", where it is _dlt_id. |
transform |
none | Files and google_sheets sources. The name a Python function was registered under with @sl.transform (its own name, or the one given as @sl.transform("kpis")) in a module under python_paths, run on each file’s or range’s rows before they load. plan fails on a name no function has, or two do. Needs python_paths and the sqlload Python package. An sl.http job takes transform= instead. See Transforming each file and Transforming each range. |
column_names |
none | The columns’ names, in order, for data without a header row: CSV and Excel files with header = false, and google_sheets ranges with header = false. The count must match the data’s columns. See Naming headerless columns. |
ClickHouse targets also take these, for the tables SQLLoad creates. Other targets refuse them, and Python tables don’t take them. See Table layout.
| Key | Default | Meaning |
|---|---|---|
engine |
MergeTree |
The table engine with its arguments, of the MergeTree family. |
order_by |
the unique_key, else unsorted |
The sorting key: an expression or a list of them. |
partition_by |
none | The partitioning expression, such as "toYYYYMM(updated_at)". |
primary_key |
none | A prefix of order_by. |
table_settings |
none | Table settings by name, such as { index_granularity = 4096 }. |
Only in a table entry:
| Key | Meaning |
|---|---|
description |
The table’s description, which becomes its comment. |
name |
The destination table name, overriding table_name. |
sql |
A query whose rows are the table (snowflake sources; odbc sources; bigquery sources; databricks sources); the entry’s key is its name. |
columns |
Pinned column types, such as { total = "decimal(18,2)" }. See Types. |
mask |
Masked columns: { email = "hmac" } or "hash". |
exclude_columns, allow_columns, destination_path |
As on the source, for this table. |
Python tables take the same settings as decorator arguments, except lookahead and
lookback_from, which they refuse, plus allow_empty and state. See
Python sources.