YAML Model with Shared Library

Reusable source function with YAML configuration — one connector, many models

Overview

This example shows the three-layer pattern: a Starlark source function for connector logic, YAML models for configuration, and SQL for downstream transformations.

Uses the Open-Meteo API — no API key required.

Source Function

Create a reusable weather connector in lib/:

# lib/open_meteo.star

def open_meteo(save, latitude="59.33", longitude="18.07", city="Stockholm", past_days="7"):
    """Fetch weather data from Open-Meteo API."""
    resp = http.get(
        "https://api.open-meteo.com/v1/forecast",
        params={
            "latitude": latitude,
            "longitude": longitude,
            "daily": "temperature_2m_max,temperature_2m_min,precipitation_sum",
            "timezone": "auto",
            "past_days": past_days,
        },
        retry=3,
    )

    if not resp.ok:
        fail("Open-Meteo API returned " + str(resp.status_code))

    days = resp.json["daily"]["time"]
    temp_max = resp.json["daily"]["temperature_2m_max"]
    temp_min = resp.json["daily"]["temperature_2m_min"]
    precip = resp.json["daily"]["precipitation_sum"]

    for i in range(len(days)):
        save.row({
            "date":       days[i],
            "temp_max_c": temp_max[i],
            "temp_min_c": temp_min[i],
            "precip_mm":  precip[i],
            "city":       city,
        })

    print("Fetched " + str(save.count()) + " days for " + city)

The function receives save as its first argument for collecting rows. All other built-in modules (http, json, etc.) are available as globals.

YAML Models

Reference the source function from YAML models — one connector, many cities:

# models/raw/weather_stockholm.yaml
kind: table
source: open_meteo
config:
  latitude: "59.33"
  longitude: "18.07"
  city: Stockholm

# models/raw/weather_london.yaml
kind: table
source: open_meteo
config:
  latitude: "51.51"
  longitude: "-0.13"
  city: London

# models/raw/weather_berlin.yaml
kind: table
source: open_meteo
config:
  latitude: "52.52"
  longitude: "13.41"
  city: Berlin

No code to write — the YAML model specifies what to fetch, the source function handles how.

Running

# Run one city
ondatrasql run raw.weather_london

Fetched 14 days for London
[OK] raw.weather_london (table, backfill, 14 rows, 463ms)

# Run all models
ondatrasql run

One connector, many models — each with its own configuration.

Using load() Directly

If you need more control than YAML provides, use load() in a Starlark script:

# models/raw/weather_custom.star
# @kind: table

load("lib/open_meteo.star", "open_meteo")

open_meteo(save, latitude="48.86", longitude="2.35", city="Paris", past_days="14")