> ## Documentation Index
> Fetch the complete documentation index at: https://docs.citino.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Pulling data nightly

> The pattern for a daily warehouse job: ask for yesterday only, with date as a dimension, and follow the cursor to the end.

Citino collects answers every day and rebuilds the figures overnight. A warehouse job that keeps up with it needs one habit: **each night, pull yesterday, and only yesterday.**

## Why one day at a time

* **It always fits.** A report's budget is counted in prompts × engines × days. One day with `date` as a dimension fits on every plan, however many prompts and engines your project has, so the job you write today keeps working as the project grows.
* **It reads each row once.** With `date` as a dimension, pages run date first and each one reads only its own days, so the job does no repeated work.
* **It is light on your rate limit.** One day is only as many pages as that day has rows, and report pages are the kind that count as several requests. See [Rate limits](https://docs.citino.ai/api/rate-limits.md).

A first backfill is the same loop over a longer window: keep `date` in the dimensions and page through. It takes longer, because every page counts as several requests, but up to 366 days it never hits the budget. A longer window is refused with `window_too_large:`, so pull a longer history in windows of 366 days or fewer.

## The job

1. Work out yesterday's date in your project's market time zone (`America/New_York` for a US market), because days in Citino are the market's days, and run the job after midnight there. The samples read the zone from `MARKET_TZ`.
2. Ask for the report with `from` and `to` both set to yesterday and `date` among the dimensions, plus whatever else you break down by (`engine`, say).
3. Follow `next_cursor` until `has_more` is `false`. A page can be short, or even empty, before the end.
4. Write the rows keyed on `date`, the other dimensions and the brand (or domain, or URL), **replacing** any row already stored with the same key.
5. Run report requests one after another. A second report request from your workspace while one is running gets `503 busy`.

```bash curl theme={null}
# One page of yesterday's brands by engine; YESTERDAY is yesterday in your market's time zone.
# Repeat with &cursor=$NEXT_CURSOR until has_more is false.
curl "https://api.citino.ai/v1/projects/$PROJECT_ID/reports/brands?from=$YESTERDAY&to=$YESTERDAY&dimensions=date,engine" -H "Authorization: Bearer $CITINO_API_KEY"
```

```python Python theme={null}
import os
import time
from datetime import datetime, timedelta
from zoneinfo import ZoneInfo
import requests

BASE = "https://api.citino.ai/v1"
key = os.environ["CITINO_API_KEY"]
project_id = os.environ["PROJECT_ID"]
session = requests.Session()
session.headers["Authorization"] = f"Bearer {key}"


def pull(report, day, dimensions="date,engine"):
    """Every row of one report for one day."""
    params = {"from": day, "to": day, "dimensions": dimensions}
    while True:
        response = session.get(f"{BASE}/projects/{project_id}/reports/{report}", params=params)
        if response.status_code in (429, 503) and response.json()["error"]["code"] in ("rate_limited", "busy"):
            time.sleep(int(response.headers.get("Retry-After", "5")))
            continue
        response.raise_for_status()
        page = response.json()
        yield from page["items"]
        if not page["has_more"]:
            return
        params["cursor"] = page["next_cursor"]


market = ZoneInfo(os.environ["MARKET_TZ"])  # e.g. America/New_York
yesterday = (datetime.now(market).date() - timedelta(days=1)).isoformat()
for row in pull("brands", yesterday):
    key_columns = (row["date"], row["engine"], row["brand"]["id"])
    v = row["visibility"]
    # Replace the stored row with this key: upsert(key_columns, row)
    print(key_columns, v["value"], v["band"], v["n"], v["low_sample"])
```

```javascript JavaScript theme={null}
const BASE = "https://api.citino.ai/v1";
const headers = { Authorization: `Bearer ${process.env.CITINO_API_KEY}` };
const sleep = (seconds) => new Promise((r) => setTimeout(r, seconds * 1000));

// Every row of one report for one day.
async function* pull(report, day, dimensions = "date,engine") {
  const params = { from: day, to: day, dimensions };
  for (;;) {
    const query = new URLSearchParams(params);
    const response = await fetch(
      `${BASE}/projects/${process.env.PROJECT_ID}/reports/${report}?${query}`,
      { headers },
    );
    const body = await response.json();
    if (["rate_limited", "busy"].includes(body.error?.code)) {
      await sleep(Number(response.headers.get("Retry-After") ?? "5"));
      continue;
    }
    if (!response.ok) throw new Error(`${body.error.code}: ${body.error.message}`);
    yield* body.items;
    if (!body.has_more) return;
    params.cursor = body.next_cursor;
  }
}

// Yesterday in the market's time zone (MARKET_TZ, e.g. America/New_York).
const today = new Intl.DateTimeFormat("en-CA", { timeZone: process.env.MARKET_TZ }).format(new Date());
const day = new Date(`${today}T00:00:00Z`);
day.setUTCDate(day.getUTCDate() - 1);
const yesterday = day.toISOString().slice(0, 10);
for await (const row of pull("brands", yesterday)) {
  const key = [row.date, row.engine, row.brand.id];
  const { value, band, n, low_sample } = row.visibility;
  // Replace the stored row with this key: upsert(key, row)
  console.log(key, value, band, n, low_sample);
}
```

The same loop pulls the domains and URLs reports: change `brands` to `domains` or `urls`, and key the rows on the domain or the URL instead of the brand. The URLs report has no `prompt` dimension.

## Late answers, and why you replace rather than append

The figures for a recent day can still change after you first pull them. Every night Citino rebuilds the most recent seven days, because a day's late answers and their reading can arrive after the day ends. So:

* **Replace, never append.** Key each row as above and overwrite. Appending would count a revised day twice.
* **If you need settled numbers**, re-pull the last seven days each night instead of only yesterday, with the same loop and `from` set seven days back. With `date` as a dimension the budget allows it, and it picks up every revision the nightly rebuild makes.

Older days change only rarely, when Citino re-reads stored answers with an improved reader. A periodic full re-pull of your history catches that too.

## Keep the band with the number

Store each figure's `band`, `n` and `low_sample` beside its `value`. A dashboard built on your warehouse should show the ± range and mute a change smaller than it, exactly as Citino's own screens do. See [Metric objects](https://docs.citino.ai/api/metric-objects.md).

## Related

> [card] [Reports](https://docs.citino.ai/api/reports.md)
>
> Dimensions, filters, budgets and paging.

> [card] [CSV output](https://docs.citino.ai/api/csv.md)
>
> The same rows as a file, for a spreadsheet.

> [card] [Rate limits](https://docs.citino.ai/api/rate-limits.md)
>
> Budgets, weights and `busy`.

> [card] [How answers are collected](https://docs.citino.ai/guides/methodology/collection.md)
>
> What happens each day before the figures exist.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.