An AI research agent that grades its own calls three days later

Asking a language model what it thinks of a token is easy and nearly free. The answer arrives in confident prose, sounds plausible, and nobody ever goes back to check it. That describes most generated market commentary: a great deal of output and no accountability attached to any of it.

sift produces that analysis too, through a five stage pipeline, with the model chosen per role at each stage that uses one. The part worth building is what happens afterwards. Every call is written down in a form specific enough to be judged, and three days later a scheduled job asks what the price actually did and records whether the call was right.

3 days

Between a call and its verdict

268

Unit tests, plus browser smoke tests on the real build

$0.003

Typical cost of one analysis, by the token prices it records

A model will produce a market view for any asset you name, at any length, in a voice that sounds like it knows. There is no natural moment where that view meets a result. By the time the price has moved, the conversation is gone, the claim was vague enough to fit either outcome, and the next question starts from zero.

Making that loop exist turns out to constrain almost every other decision in the system. A claim has to come out of the model in a shape a job can compare against a number, which rules out free prose. It has to carry the price it was made at, which means the market call and the storage cannot be separate concerns. And the rule that decides right or wrong has to be fixed in advance, because a rule chosen after seeing results is not a measurement, it is a story.

The plan assumed a per-coin news endpoint. Testing showed the category filter on the provider I had chosen returns general crypto news whatever category you ask for, and no free source covers assets outside the majors. Pretending otherwise would have meant an analysis quietly built on headlines about something else.

So coverage became a value the system carries rather than an assumption it makes. Headlines that actually name the asset are one tier, market-wide context is another, nothing is a third. The prompt is told which tier it is working in and told to cap its confidence in the weaker two, and the card tells the reader the same thing. The honest version is better than the complete-looking one.

The second finding came from the price series. The first version asked the model for levels and gave it seven daily closes, which cannot locate a level, so it made them up. The same response already carried a week of hourly points that were being thrown away. Those get reduced by a plain function to the bands where the price actually spent its time, and a band has to hold well over its even share of the week to be named at all, so a week that drifted evenly reports nothing rather than inventing concentration that was not there.

Follow one question through. A message becomes an intent, an intent becomes a pipeline run, and the run writes one row that the evaluation job will come back for. Dependencies point one way: providers talk to the outside world, the pipeline orchestrates and never imports the interface, and the routes are thin.

Solid lines are the request path. Dashed lines run on their own clock: the scheduled job that judges old calls, and the verdict it writes back against each one. Only market data and synthesis can end a run; news, sentiment and extraction may each fail and the analysis continues with less evidence, saying so.

Every stage that matters returns a structured object validated against a schema rather than prose that something later has to parse. This is not about developer convenience. It is the thing that makes the evaluation possible at all: a trend has to be one of three values for a job to compare it against a price move, and a parser over model output would be a second thing that can be wrong. When generation fails the schema check it retries once and then returns a typed error, because retrying three times on a metered allowance spends the requests that would have served real users.

Models are chosen by role rather than by name. Intent, extraction, sentiment and synthesis each have an ordered list, each model carries a daily request limit, and usage is counted from what the provider actually reported rather than what was hoped. When one is spent the next takes over and the switch is recorded on the analysis, so a fallback run cannot quietly pass itself off as a normal one.

Cost stays near a third of a cent per analysis through three habits. The cheap model reads fifteen headlines and hands the expensive one five bullets, so the expensive prompt is small. Reasoning is turned down to the floor the model allows, because on this task most of the budget went to thinking rather than answering. And repeat work is cached per stage on short windows, so a second question about the same coin inside the window makes no external calls at all.

This is the part the rest of the system exists to serve. When an analysis is written it stores the direction it called, the confidence it claimed, and the price at that moment. Nothing else about it is opinion: those three fields are enough for a machine to decide later whether it was right, with no human reading the prose and deciding what it meant.

Three days on, a scheduled job wakes up, finds every call that has reached its horizon, and asks the price feed what actually happened. Bullish is correct if the price rose past the same threshold a neutral call has to stay inside, bearish if it fell past it, and a move that stays inside the band scores only the neutral call correct. The three verdicts cannot both be right about one move, and none of them is graded on magnitude, because sift called a direction and scoring it on size would grade a claim it never made.

1. 01

   The claim is stored in a gradeable shape

   A direction from a fixed set of three, a confidence number, and the price at the time of the analysis. The horizon and the neutral threshold in force are written onto the row too, not read from configuration at judging time.

   On failureIf synthesis cannot produce an object that satisfies the schema, there is no analysis and nothing is stored. A half-formed claim is worse than none, because it would be graded as though it meant something.

2. 02

   The row waits out its horizon

   Three days by default, configurable, and stored per evaluation so changing it later cannot silently rewrite what older calls were judged against.

3. 03

   A daily job collects what is due

   It reads the calls that have reached the horizon, then asks for one price per coin rather than one per call, so a backlog of thirty calls on the same asset is a single question to the feed. Failures are cached for the run too; otherwise one unreachable coin costs a slow request per row.

   On failureThe endpoint refuses to run at all when its secret is unset, rather than defaulting to open. A scoring job anyone can trigger is a scoring job anyone can pollute.

4. 04

   The verdict is computed by rules fixed in advance

   Percentage change from the stored price, then direction compared against it, with the neutral threshold from that row. The rule lives in one small module on its own so it can be read, tested and, most importantly, not quietly adjusted after a bad week.

   On failureIf the stored price is missing or zero, the row is left unscored rather than recorded as flat. Returning zero would have avoided a division and invented a verdict the system never earned.

5. 05

   A missed call is left unscored rather than guessed

   If the price lookup fails, the row stays due and the next day's run picks it up. Nothing in the loop is allowed to write a number it is not sure of, because the whole value of the number is that it was not chosen to flatter anyone.

6. 06

   Wrong calls get a post-mortem, and it is never read back

   The cheapest capable model writes two or three sentences on what the reasoning leaned on that did not hold, and that note is stored. It is deliberately never fed into a later analysis.

   On failureA post-mortem that cannot be written does not fail the evaluation. The verdict is the thing that has to land; the note is commentary on it.

That last decision is the one people ask about. Feeding the lessons back in is the obvious next feature and it is exactly what would break the measurement: the accuracy figure would stop describing one system and start describing a moving one, and this month would no longer be comparable with last. It becomes worth doing at a few tens of scored calls, and only with the lesson sets versioned so a track record can say which version produced it.

The public page is deliberately unflattering about its own sample. It reports directional accuracy over a window, a breakdown by coin and by trend, and the confidence gap between calls that were right and calls that were wrong. With a handful of scored calls it says that this is a working loop with almost nothing in it yet, rather than showing a triumphant number that is really one lucky guess. Directional accuracy starts meaning something in the tens, and calibration, where calls made at eighty percent confidence should be right about eighty percent of the time, needs a few hundred.

Four decisions did most of the work, and in each case the easier option would have produced a system that looked the same and meant less.

What makes that last one tolerable is that there is nothing to escalate to. No model call in sift defines a tool, nothing a model returns triggers a fetch or a write, and a successful injection buys exactly one wrong analysis, which gets scored three days later and shows up in the track record as a miss.

There is also a decision about conversation. History is never sent to a model: each analysis stands alone, and the only thing carried between messages is the last coin discussed, which is what makes a follow-up like asking for the risks resolve to the same asset. The cost is real, sift cannot answer a question about its own previous answer, and the interface says so under the composer rather than letting somebody discover it.

Deciding up front that the output had to be gradeable changed the design more than any model choice did. It is what forced schemas instead of prose, one stored price per call, a fixed rule in its own module, and a refusal to write a verdict that could not be computed. A system that has to prove itself later is built differently from one that only has to sound right now.

The second lesson is that honesty is mostly a set of small refusals: leaving a row unscored rather than guessing, reporting a coverage tier rather than implying the headlines were about this asset, naming no price bands when the week had none, and publishing a sample size that undercuts the number next to it. None of those are hard to build. They are just easy to skip, and skipping any one of them would make the headline figure meaningless while leaving it looking exactly the same.