Skip to main content

Datasets

Synced from semantic-layer

Source: docs/guide/01-datasets.md at origin/main (3688d236), synced 2026-11-05. Do not edit this page here; change the source and run yarn sync:semantic.

A dataset is one source in your warehouse, either a table or a read-only query, and it owns that source's fields and its outgoing relationships. A dataset lives on exactly one connection, which says which warehouse runs it and in which SQL dialect.

name: orders
display_name: Orders
source:
table: {database: acme, schema: sales, name: orders}
primary_key: [order_id]
time:
default_field: completed_at
partition:
column: completed_date
datatype: Date
fields:
- {name: order_id, expression: order_id, datatype: String}
- {name: completed_at, expression: completed_at, datatype: DateTimeTz}
- {name: region, expression: region, datatype: String}
- {name: amount, expression: amount, datatype: Decimal}
relationships:
- {name: orders_to_customers, to: customers, from_columns: [customer_id], to_columns: [customer_id]}

Source​

A source is one of two kinds.

  • A table, named by its parts: {table: {database, schema, name}}. Each part is spelled exactly as the warehouse spells it, capitals and dots included (Brand, Brand.v2 copy). The layer quotes every part, so the warehouse never folds reno.Brand into reno.brand. schema may be left out, and database with it, in which case the warehouse fills them from its own defaults. When the connection has tables selected into its mirror, the source must name one of them exactly (UNBOUND_SOURCE otherwise).
  • A query: a read-only SELECT (or WITH … SELECT) in the connection's dialect, used as a subquery.

A dotted string such as source: acme.sales.orders is refused, because it cannot say where one part ends and the next begins. Writing a dataset proves on the warehouse that the source and every field exist.

Keys​

primary_key lists the columns that identify a row, and unique_keys lists other column sets that do. They matter for one thing: a relationship into this dataset must join on its primary key or one of its unique keys. The write checks them against the warehouse's own declared keys when it has any, and warns on a mismatch rather than refusing.

The time axis​

time.default_field names the dataset's time axis: the moment each row happened. A query's window and its DATE_TRUNC buckets read it (Time), and so does every metric anchored on the dataset, unless the metric names its own time_field.

An axis is one of two kinds, set by the field's datatype:

datatypeKindBuckets it offers
DateTimeTza moment, one instant everywhereminute and coarser, or the field's resolution and coarser
Datea day, a calendar dateday and coarser

A DateTime (a clock reading with no time zone), a Time or a non-temporal field cannot be an axis, and the write is refused (AXIS_KIND_INVALID) with the conversion to declare. If the column holds text or epoch numbers, declare a field whose expression converts it, then name that field. For example, TRY_STRPTIME(last_verified, '%Y-%m-%d')::DATE declared as Date.

A dataset with no time axis can still be queried, but no window can bound a metric on it.

Partition​

partition:
column: event_date # the raw column the warehouse prunes on
datatype: Date # Date | DateTime | DateTimeTz | Integer
encodes: event # event (default) | ingestion

partition names the raw column your warehouse partitions or clusters the table on. Every scan of the dataset then also filters that column, so the warehouse reads only the partitions that can hold the window, even when the time axis is a converted expression the warehouse cannot prune through. The filter is always a little wider than the window, a day on each side, so it can never change an answer. It only cuts what is read.

encodes: ingestion is for a column that holds when the row landed (BigQuery's _PARTITIONTIME) rather than when it happened. Only the window's start is applied to it, because a row lands at or after its event.

Sync writes partition from the table's own metadata, and the sync report says what it took the column to mean. A write warns when a partitioned table declares no partition, and refuses one whose warehouse requires a partition filter on a column the dataset does not declare.

What else a dataset carries​

  • display_name, description and ai_context describe the dataset to people and to agents. They change no answer.
  • fields and relationships share one namespace: a field and a relationship of one dataset cannot have the same name.
  • A dataset's name never changes. To rename one, create the new dataset and delete the old one.

Datasets on the tenant's lake​

Every tenant also has a lake, a connection named bicycle whose tables the layer itself writes. It is for results you produce and want to query like any other data, such as a forecast. Declaring a dataset there creates its table, adding a field adds its column, and deleting the dataset drops the table. Rows are written with POST /datasets/{id}/rows in one of the modes append, overwrite, upsert, update or delete, and each write is one lake snapshot. An answer that read the lake says which snapshot in semantics.data_versions.