Example: a backtest mapped over origins

Synced from bicycle-studio-api

Source: studio_guide("example-backtest") (GET /api/studio/v1/guide/example-backtest) rendered at origin/platform (1afd83f), synced 2026-09-28. Do not edit this page here; change the source and run yarn sync:studio.

Build this when you want to know how well a forecast would have done: run it from several past dates (origins), one function call per origin, then score all of them in one table.

Person-only stops: a person publishes demo_backtest in Studio; you re-pin the published version; a person publishes the workflow.

Files

demo_backtest/function.json

{
 "schema": "bicycle.function/v1",
 "name": "demo_backtest",
 "kind": "code",
 "title": "Backtest one origin",
 "entrypoint": "main:handler",
 "image": "bda-python:3",
 "mode": "async",
 "timeout_ms": 30000,
 "resources": {
  "class": "fn-xs"
 },
 "deterministic": true,
 "input_schema": {
  "type": "object",
  "properties": {
   "params": {
    "type": "object",
    "required": [
     "origin"
    ],
    "properties": {
     "origin": {
      "type": "string"
     },
     "horizon_days": {
      "type": "integer"
     }
    }
   }
  }
 },
 "output_schema": {
  "type": "object",
  "required": [
   "errors"
  ],
  "properties": {
   "errors": {
    "type": "array"
   },
   "origin": {
    "type": "string"
   },
   "n": {
    "type": "integer"
   }
  }
 },
 "capabilities": [],
 "visibility": {
  "audience": "tenant",
  "expose": {
   "apps": false,
   "workflows": true,
   "agents": false,
   "mcp": false
  }
 },
 "docs": {
  "summary": "Backtests one forecast origin and returns the absolute error per horizon step."
 },
 "tests": [
  {
   "name": "one-origin",
   "input": {
    "params": {
     "origin": "2026-08-03",
     "horizon_days": 7
    }
   },
   "expect": {
    "/n": 7,
    "/errors/0/abs_err": 9,
    "/errors/0/origin": "2026-08-03"
   }
  }
 ]
}

demo_backtest/main.py

import hashlib


def handler(input, ctx):
    """One backtest origin: a deterministic stand-in for 'forecast from this origin, compare with actuals'."""
    p = input.get("params") or {}
    origin, h = str(p["origin"]), int(p.get("horizon_days", 7))
    seed = int(hashlib.sha256(origin.encode()).hexdigest()[:12], 16)
    rows = []
    for step in range(1, h + 1):
        actual = 100 + (seed >> step) % 20
        forecast = 100 + (seed >> (step + 5)) % 20
        rows.append({"origin": origin, "step": step, "actual": actual, "forecast": forecast,
                     "abs_err": abs(actual - forecast)})
    return {"errors": rows, "origin": origin, "n": len(rows)}

In your own backtest the handler forecasts from origin and compares with actuals. This one returns a small, fixed error per origin so you can check the plumbing.

workflow.json

{
  "schema": "bicycle.workflow/v1",
  "title": "Backtest over six origins",
  "partitions": {"origin": {"type": "static", "values_file": "data/origins.json"}},
  "artifacts": {"errors": {"type": "table", "partitioned_by": "origin"}, "scorecard": {"type": "table"}},
  "nodes": {
    "backtest": {"kind": "function", "config": {"ref": "fn:<tenant>/demo_backtest@1", "params": {"origin": "${partition.origin}", "horizon_days": 7}}, "outputs": {"errors": "errors"}, "map": {"over": {"partition": "origin"}, "max_width": 16, "concurrency": 4, "on_failure": {"quorum": 0.9}}},
    "score": {"kind": "sql", "config": {"file": "sql/scorecard.sql"}, "inputs": {"errors": {"artifact": "errors", "partitions": {"origin": "run"}, "collect": true}}, "outputs": {"scorecard": "scorecard"}}
  },
  "triggers": {"manual": {"type": "manual"}}
}

data/origins.json

["2026-08-03", "2026-08-10", "2026-08-17", "2026-08-24", "2026-08-31", "2026-09-07"]

sql/scorecard.sql

SELECT origin,
       count(*)                                         AS steps,
       round(avg(abs_err), 2)                           AS mae,
       round(100 * sum(abs_err) / sum(actual), 2)       AS wape_pct
FROM errors
GROUP BY origin
ORDER BY origin

• partitions.origin is a static list read from data/origins.json; it is frozen into the revision on save.
• map.over.partition runs backtest once per origin, at most 4 at a time. ${partition.origin} is that origin.
• errors is partitioned_by: origin: one version per origin.
• "partitions": {"origin": "run"}, "collect": true reads every origin's rows as one errors table.

Calls, in order

1. function_create("demo_backtest", title), then function_put_files("demo_backtest", {"function.json": ..., "main.py": ...}): version 1, validated.
2. function_test("demo_backtest", 1): passed, state tested.
3. workflow_create(title, document, files): valid, a function_draft warning.
4. workflow_validate(workflow_id): a try run would start.
5. workflow_plan(workflow_id): backtest × 6, score, and score.errors reads origin=2026-08-03 … 6 keys.
6. workflow_run(workflow_id, revision=1, logical_date="2026-09-22"): a try run.
7. workflow_run_describe(workflow_id, run_id): six backtest rows, one per origin=..., each 7 rows with its own invocation; score succeeded with 6 rows.
8. workflow_artifact(workflow_id, "scorecard", version_id): one row per origin: steps 7, mae, wape_pct (for example 2026-08-03: 4.86 and 4.33).

Switch it off

workflow_disable(workflow_id, reason="...")
function_disable("demo_backtest", reason="...")

Verified on preview 2026-09-28: function_create, function_put_files, function_test, workflow_create, workflow_validate, workflow_plan, workflow_run (try), workflow_run_describe, workflow_artifact, workflow_disable, function_disable ran; invocation ids inv_01M3J729891RTSQK58TF2MYKZB (run), inv_01M3J72BMH2QRNCY5KX8EAWZ58, inv_01M3J72AJWF9CYG6PGXNXJK9FQ, inv_01M3J72AJPCDJ889JYY4W1RQ5P, inv_01M3J72AHRAPDPJ5DKPPRMN4NJ, inv_01M3J72E41WQ638PZ30BY5MPC5, inv_01M3J72FAGFR3VGG3D6FV4YT0F (one per origin).

guide_version bb2461328de4