Datasets
Source: docs/guide/01-datasets.md at origin/main (3688d236), synced 2026-11-05. Do not edit this page here; change the source and run yarn sync:semantic.
A dataset is one source in your warehouse, either a table or a read-only query, and it owns that source's fields and its outgoing relationships. A dataset lives on exactly one connection, which says which warehouse runs it and in which SQL dialect.
name: orders
display_name: Orders
source:
table: {database: acme, schema: sales, name: orders}
primary_key: [order_id]
time:
default_field: completed_at
partition:
column: completed_date
datatype: Date
fields:
- {name: order_id, expression: order_id, datatype: String}
- {name: completed_at, expression: completed_at, datatype: DateTimeTz}
- {name: region, expression: region, datatype: String}
- {name: amount, expression: amount, datatype: Decimal}
relationships:
- {name: orders_to_customers, to: customers, from_columns: [customer_id], to_columns: [customer_id]}
Source
A source is one of two kinds.
- A table, named by its parts:
{table: {database, schema, name}}. Each part is spelled exactly as the warehouse spells it, capitals and dots included (Brand,Brand.v2 copy). The layer quotes every part, so the warehouse never foldsreno.Brandintoreno.brand.schemamay be left out, anddatabasewith it, in which case the warehouse fills them from its own defaults. When the connection has tables selected into its mirror, the source must name one of them exactly (UNBOUND_SOURCEotherwise). - A query: a read-only
SELECT(orWITH … SELECT) in the connection's dialect, used as a subquery.
A dotted string such as source: acme.sales.orders is refused, because it cannot say where one
part ends and the next begins. Writing a dataset proves on the warehouse that the source and every
field exist.
Keys
primary_key lists the columns that identify a row, and unique_keys lists other column sets that
do. They matter for one thing: a relationship into this dataset must join
on its primary key or one of its unique keys. The write checks them against the warehouse's own
declared keys when it has any, and warns on a mismatch rather than refusing.
The time axis
time.default_field names the dataset's time axis: the moment each row happened. A query's
window and its DATE_TRUNC buckets read it (Time), and so does every metric
anchored on the dataset, unless the metric names its own time_field.
An axis is one of two kinds, set by the field's datatype:
datatype | Kind | Buckets it offers |
|---|---|---|
DateTimeTz | a moment, one instant everywhere | minute and coarser, or the field's resolution and coarser |
Date | a day, a calendar date | day and coarser |
A DateTime (a clock reading with no time zone), a Time or a non-temporal field cannot be an
axis, and the write is refused (AXIS_KIND_INVALID) with the conversion to declare. If the column
holds text or epoch numbers, declare a field whose expression converts it, then name that field.
For example, TRY_STRPTIME(last_verified, '%Y-%m-%d')::DATE declared as Date.
A dataset with no time axis can still be queried, but no window can bound a metric on it.
Partition
partition:
column: event_date # the raw column the warehouse prunes on
datatype: Date # Date | DateTime | DateTimeTz | Integer
encodes: event # event (default) | ingestion
partition names the raw column your warehouse partitions or clusters the table on. Every scan of
the dataset then also filters that column, so the warehouse reads only the partitions that can hold
the window, even when the time axis is a converted expression the warehouse cannot prune through.
The filter is always a little wider than the window, a day on each side, so it can never change an
answer. It only cuts what is read.
encodes: ingestion is for a column that holds when the row landed (BigQuery's _PARTITIONTIME)
rather than when it happened. Only the window's start is applied to it, because a row lands at or
after its event.
Sync writes partition from the table's own metadata, and the sync report says what it took the
column to mean. A write warns when a partitioned table declares no partition, and refuses one
whose warehouse requires a partition filter on a column the dataset does not declare.
What else a dataset carries
display_name,descriptionandai_contextdescribe the dataset to people and to agents. They change no answer.fieldsandrelationshipsshare one namespace: a field and a relationship of one dataset cannot have the same name.- A dataset's
namenever changes. To rename one, create the new dataset and delete the old one.
Datasets on the tenant's lake
Every tenant also has a lake, a connection named bicycle whose tables the layer itself writes.
It is for results you produce and want to query like any other data, such as a forecast. Declaring
a dataset there creates its table, adding a field adds its column, and deleting the dataset drops
the table. Rows are written with POST /datasets/{id}/rows in one of the modes append,
overwrite, upsert, update or delete, and each write is one lake snapshot. An answer that
read the lake says which snapshot in semantics.data_versions.