Run Types

Values

Hash inputs

Two hashes are compared against the model’s last DuckLake commit, and either one changing forces a backfill.

Model hash — the SQL body plus @kind, @unique_key, @group_key, @partitioned_by, @incremental, @incremental_initial, @fetch and @push. A mismatch reports run_reason: sql changed.

Config hash — the config/ SQL that applies to this model. A mismatch reports run_reason: config changed.

Upgrading to per-model config hashing

Before this change, config content was folded into sql_hash. Splitting it out changes sql_hash for every model in any project that has a config/ directory, so the first run after upgrading rebuilds every model once. That rebuild reports run_reason: hash format changed (upgrade) rather than sql changed, so it is distinguishable in --json output and in commit_extra_info. Subsequent runs are stable.

Plan for it before upgrading a production lake:

• append and merge models are rebuilt with TRUNCATE + INSERT. If a model’s source no longer holds the full history — a rolling API window, a truncated staging table — that history is not recoverable from the source.
• scd2 models keep their history (see SCD2 rebuilds). Since the model’s output is unchanged, the rebuild writes no new versions.
• @fetch models re-fetch from the beginning: the incremental cursor resets to @incremental_initial.

Projects with no config/ directory are unaffected: their sql_hash is unchanged and nothing rebuilds.

Config hash scope

Every statement in config/ lands in one of three buckets, and a model is rebuilt only for what actually reaches it.

Classification is per statement, not per file: one unrecognized statement in a macro file sends only itself to the global bucket instead of dragging the file’s macros with it. Global is the default — a statement is scoped or dropped only when it is positively recognized.

Sources and the search path

Scoping a source by its alias assumes a model that reads it names it. A search path breaks that assumption: with SET search_path = 'warehouse' in config, a model reading FROM orders resolves to warehouse.orders without the word warehouse appearing anywhere in it, and repointing that ATTACH would skip exactly the models it affects.

So if any statement anywhere in config/ is a USE, a SET search_path, or a SET schema, source scoping is switched off and sources.sql is treated as session-wide. Only projects that actually use that pattern pay for it. Macros and variables stay scoped either way — they are always called by name, so a search path cannot hide the dependency.

Which settings are inert

79 DuckDB settings cannot change what a query returns and are dropped from the hash: thread and memory limits, temp_directory, optimizer and join thresholds, caching and prefetching, checkpoint/WAL/write buffering, and the profiling, logging and progress-bar families. Bumping threads or memory_limit rebuilds nothing.

The same holds for the I/O knobs you reach for when object storage is slow or flaky: async I/O scheduling (async_threads, read_ahead_depth) and HTTP transport (http_client_pool_capacity, http_timeout, http_retries, http_retry_wait_ms, http_retry_backoff, http_keep_alive, httpfs_connection_caching). They decide whether and how fast a read succeeds, not what it returns, so tuning them in config/settings.sql rebuilds nothing. All but the first three come from the httpfs extension.

Everything else is treated as semantic, including settings not on the list at all — a knob added by a future DuckDB release keeps forcing a rebuild until it is classified. Notable settings that stay semantic despite sounding harmless:

Which files stay global

secrets.sql — DuckDB secrets bind to a query by SCOPE, a path prefix, never by name. A model never mentions the secret it uses, so there is nothing to scope against. Note that credentials do not live here: .env is loaded into the process environment and every config file is passed through os.ExpandEnv, so secret values are ${VAR} references. What the file carries is topology — ENDPOINT, REGION, SCOPE, HOST — and repointing that changes which bytes a model reads.

catalog.sql — its ATTACH accepts SNAPSHOT_TIME and SNAPSHOT_VERSION, i.e. time travel for the whole lake, plus DATA_PATH and METADATA_PATH. Models address targets as schema.table under USE, so scoping by the catalog alias would miss them.

extensions.sql — INSTALL/LOAD decides which functions exist and which implementation answers.

Two further properties

• Comments and formatting are ignored. Config content is normalized before hashing — -- line comments, /* … */ block comments and whitespace all come out — so reformatting or re-commenting a config file rebuilds nothing. Case is significant, because config carries endpoints, paths and environment names; and text inside 'literals' and "quoted identifiers" is kept verbatim, so a change there always counts.
• Macro-to-macro references are followed. If a model calls outer() and outer() calls inner(), editing inner() rebuilds the model.

Name matching is lexical and deliberately over-inclusive: a macro named total is treated as a dependency of any model containing the word total. An extra dependency costs a needless rebuild; a missing one would serve stale data. Two known limits: a Starlark model that assembles a macro name at runtime through string concatenation is not seen, and .env is not hashed — repointing an endpoint there changes results with no hashed file changing, so follow it with a full refresh.

What executes per run type

Rebuild from an empty fetch

A backfill replaces the target with the result (an existing scd2 target is compared with it instead; see SCD2 rebuilds). If every lib call in the model returned 0 rows with empty_result: "no_change" (the default) and the result is empty, there is nothing to rebuild from, and replacing the target would empty it. The run keeps the existing rows instead and warns: all libs reported no change and the result is empty, keeping existing rows instead of rebuilding. This applies to every kind, table included, whatever made the run a backfill — a changed model or config hash, or an escalation during the run.

It does not apply to the ordinary full run of a @kind: table @fetch model, which re-fetches on every run: an empty result there replaces the target as usual.

No rows are written, so there is nothing to validate either: audits do not execute, and a constraint violation in the empty result is not reported. A constraint or audit that does not parse, or a constraint that cannot execute (it names a column the model does not produce), still fails the run.

An added column is still applied. Any other schema change — a type change, a rename, a dropped column — would rewrite columns of the rows being kept (a type change is applied as drop and re-add, which empties the column), so it is deferred with the rebuild and the run warns schema change would rewrite the kept rows and the result is empty, deferring it with the rebuild. That run commits nothing, so the next run still sees the model as changed.

The rebuild is not dropped. A run that keeps the rows commits with rebuild_pending: true in commit_extra_info, and the next run is a backfill with run_reason: rebuild pending. Each run that fetches nothing again keeps the rows and defers the rebuild once more, until a run has data to build from.

On a first run there is no target to keep, so the empty result creates an empty target as usual.

SCD2 rebuilds

A backfill of an existing scd2 target keeps its history. The new result is compared with the current versions the way an incremental run compares them: changed rows are closed and get a new version, rows missing from the result are closed, and rows whose values did not change keep their version. A logic change therefore shows up as new versions at the snapshot of the change, in a commit with run_reason: sql changed, and rows_affected counts the versions written rather than the whole result.

The comparison joins on @unique_key against the current versions, so it only holds while the stored versions carry the same identity as the new result. In these cases the target is rebuilt from scratch instead — every row becomes current with a new valid_from_snapshot and valid_from_at — and the run warns scd2 history reset: <reason>:

• the @unique_key column changed type (the type change empties the stored key)
• @unique_key names a column the target does not have
• @unique_key names another column than the previous run’s (recorded as unique_key in commit_extra_info)
• the target was built under another kind
• the current versions are not unique and non-null on the key

A target whose last commit was written before unique_key was recorded has nothing to compare the key with. There a change to another column is caught only when the current versions are not unique on it; a change to a column that is also unique keeps the history and compares on the new key. Every scd2 commit from this version on records the key, so only a rebuild before the first such commit is affected.

Schema changes apply to the stored versions too. An added column is NULL in older versions, and each current row gets a new version the first time. A dropped column disappears from the whole history. A type change is applied as drop and re-add, so the column is NULL in every closed version.

To start the history over deliberately, drop the target (DROP TABLE schema.table); the next run is a first run.

Sink on skip

When a model is skipped but has @push, the runner still drains the pending state-store backlog from previous failed pushes. No new delta is generated.

Incremental kwargs

On backfill: is_backfill=True, last_value reset to @incremental_initial. On incremental: is_backfill=False, last_value is MAX(cursor_column) from target.