Almost every conversation about this product category eventually reaches the same awkward pause. Someone asks how you know whether an AI assistant is any good — not the ten-minute demo, but the tenth week of actually living with one — and the honest answer has been that nobody really knows. Academic tests measure the model underneath; developer tests measure engineering agents; the first consumer-side efforts are younger than some of the products they cover. We got tired of waiting for that gap to close on its own. So today we are publishing the Today Assistant Benchmark: a 12-dimension framework for evaluating consumer AI assistants, organized around three questions and held to five scoring rules that bind us before they bind anyone else. This article is the announcement and the instruction manual in one — and an invitation to use it on any assistant, including ours.
Why We Built Our Own Benchmark
Start with the lay of the land, because the gap is specific. Benchmarks today come in three layers. The academic layer — MMLU, GAIA, τ-bench — measures the raw model: what it knows, how it reasons, how it follows instructions in a vacuum. The engineering layer — SWE-bench, WebArena — measures agents that write code and navigate tasks for developers. Both layers are rigorous, and both answer questions a shopper never asks. A person deciding between two assistants does not care about a reasoning score; they care whether the Wednesday flight gets booked, whether the Monday summary arrives, whether the thing stays quiet after nine in the evening. The consumer layer, where products get tested as products, was born only this year, and its methodologies are still finding their feet. We walked through all three layers — and what each one can and cannot tell you — in how benchmarks work today.
That essay closes on an uncomfortable conclusion, and this announcement is our response to it. If you build assistants, you are already in the evaluation business whether you admit it or not, because every marketing claim is an implied score. We would rather do the work in the open, under rules anyone can inspect, than let implied scores be the only kind in circulation. So the position is simple: we make an assistant, and we also accept the job of saying how assistants should be evaluated. Yes, that puts the referee and one of the players in the same building. The answer to that conflict is not to shrink from publishing — it is to make the rules bind the referee hardest, which is why the first scoring rule below exists, and why Today's own scores will only ever appear under this framework, never beside a friendlier private one.
One more thing before the framework itself, because transparency is the whole point of this post. Version 0.1 of the benchmark is dated September 18, 2026, and it is a methodology, not a results table. Nothing has been scored yet — not Today, not anyone. What exists today is the part most evaluation efforts skip: a defined set of dimensions, a standard task for each, scoring anchors, and rules about what a number must carry with it before it can be called a score. Those are the sections that follow.
The Three Questions
A framework earns its keep only if it is organized the way a person actually decides. Taxonomies built for researchers — flat lists of features, capability matrices with sixteen rows — are accurate and useless in a shop aisle. So before writing down a single dimension, we watched how people choose, including how we choose, and the process reliably collapses into three questions asked in a strict order. Can it do the work? Is it good to live with? Can I trust it?
The first question sounds modest, and is not. Capability is not the ability to discuss your work; it is the ability to finish it. The test is the email you have been avoiding since Tuesday: an assistant that can actually draft the reply in your voice, get the time zones right, respect the thread's history, send it once you approve, and confirm it landed. Around that core sit the calendar moves, the recurring jobs, and the requests that have to cross several tools in a row without anyone dropping a link. If the answer to the first question is no, the other two never get asked — charm and privacy polish cannot rescue an assistant that cannot do the work.
The second question arrives around week two, once the novelty has worn off: is it good to live with? This is where capability that costs you your attention stops counting as capability. Does it remember what you told it on Monday, or does every conversation start from zero? Does it speak up when something actually matters — a clash on your calendar, a thread going cold — and stay quiet when nothing does? Does it answer before your coffee cools? Does it genuinely connect to the tools your week runs through, or does it just list their logos? An assistant that pings you twelve times a day to tell you about three meetings is not an assistant; it is a second job. The difference between helpful initiative and daily nagging is a product quality, and it can be measured.
The third question people ask last and remember longest: can I trust it? Trust here is not a vibe; it is concrete and checkable. What exactly did I authorize, how finely, and what happens the moment I take it back — when you revoke access to your mailbox, is the data actually gone, or merely out of your sight? After the assistant acts, can you trace what it did and why — which memory, which rule, which permission produced that email? And does the company behind it publish numbers you can verify, or only adjectives? An assistant that works inside your email and your calendar is a houseguest with keys; the trust question is whether you can see what it does with them.
These three questions are also, not coincidentally, the order in which we built Today — execution first, the living-with qualities of memory and restraint second, permission granularity and inspectable memory throughout. The framework is simply the questions written down, so they can be asked of anyone.
The 12 Dimensions
Here is the section worth screenshotting. Twelve dimensions, grouped under the three questions; each one measurable, each with a standard task and scoring anchors from 1 to 10 in the full methodology. In the tables below, every row gives the dimension, what it measures, and a question you can ask during a trial — no lab required.
Capability: can it do the work?
Four dimensions, one theme: finished work, verified. This layer is scored on outcomes with receipts, not on the quality of advice. An assistant that explains how you might book the flight scores nothing here; one that books it and confirms the booking scores.
| Dimension | What it measures | A question to ask in a trial |
|---|---|---|
| Task Execution | Whether it completes a real online task end to end, instead of offering advice and a link | Ask it to book something and confirm when it is done — does a verified result come back? |
| Drafting, sending and following up on mail with the right facts and the right tone | Hand it a scheduling thread with time zones in it — is the draft sendable after one edit? | |
| Routines & Scheduling | Calendar work and recurring jobs that keep running without babysitting | Request a Monday morning summary — does it arrive on time, about your actual week, week after week? |
| Chained Tasks | One request that has to span several tools and steps without dropping a link | Find a flight, book the best one, add it to the calendar, remind me to check in — how many steps survive? |
Quality: is it good to live with?
Five dimensions about the texture of daily use. Note that two of them pull in opposite directions on purpose: an assistant earns credit for speaking up at the right moment and for saying nothing at the wrong ones. Initiative without restraint is noise, and the framework scores both sides of that trade.
| Dimension | What it measures | A question to ask in a trial |
|---|---|---|
| Memory | What it keeps across sessions, and how much control you have over what it keeps | Tell it a preference today — does it use it tomorrow without being reminded? |
| Proactive | Whether it offers something valuable before you think to ask | Plant a meeting that conflicts with an existing one — does it raise the conflict before it starts? |
| Proactive Restraint | Whether it stays quiet when nothing needs saying | Count a week of unprompted messages — how many were worth stopping for? |
| Speed | Time from your instruction to a result you can act on | Time three everyday requests — how long until each answer is usable? |
| Connected Apps | How many third-party integrations actually work, and how deep they go | Make one request that spans calendar, mail and a third tool you rely on — does it complete? |
Trust: can I trust it?
Three dimensions, and the last of them is unusual: it measures whether the vendor publishes verifiable quality metrics at all. A transparency dimension is missing from most checklists in circulation, which tells you something about the industry; we think a claim without a number deserves to be visible as such.
| Dimension | What it measures | A question to ask in a trial |
|---|---|---|
| Permissions & Privacy | How finely you can grant access, and how completely you can take it back | Revoke a connected account — can you verify the data is truly gone, not just hidden? |
| Transparency | Whether you can trace what it did, and why it did it, after the fact | After a multi-step task, ask it to walk through every decision it made |
| Published Metrics | Whether quality claims come with numbers you can check and reproduce | Search the vendor's docs for a figure published alongside its test conditions |
Two design choices shaped the list. Where existing checklists carry overlapping dimensions — a purchasing row separate from a general online-tasks row, for instance — we merged them, because a dimension you cannot separate in scoring is really two names for one test. And we ordered the layers along the decision path people actually walk, from capability through quality to trust, rather than alphabetically or by technical ancestry. The order is the opinion.
How Scoring Works
Dimensions are the visible part of any benchmark; the rules underneath decide whether it is worth anything. Ours come from studying how evaluation efforts fail — each rule below is the direct antidote to a documented failure mode, and most of those failures are not hypothetical.
No score without a logged run — every number has to arrive attached to a verifiable record: a screenshot, an export, a timestamp, and the date the test ran. The failure it prevents is the self-reported figure, the one a vendor quotes with nothing checkable behind it. Unverified numbers circulate anyway, because quoting is cheaper than testing.
One dimension, one task, one starting point — each dimension is exactly one standardized task, run with identical instructions and identical initial conditions for every assistant tested. The failure is coupling: when tasks overlap, a strong email performance quietly pads the routines score, the totals start moving together, and twelve dimensions turn out to measure three.
Untested is not zero — a dimension can be in one of three states: tested and scored from 1 to 10, untested and marked with a dash, or genuinely not applicable. The failure is silent: scoring untested dimensions as zero drags the overall average down, punishes whoever reported honestly, and rewards whoever tested only what they knew would pass.
Version your methodology — any change to a scoring rule bumps the framework's version number, and every historical score keeps the label of the version it was measured under. The failure is the quiet revision: standards change, nothing is announced, and this quarter's numbers get compared against last quarter's as if the ruler had never moved.
Subjective dimensions get no number — some things are matters of taste, and taste is not a measurement. Where a judgment is inherently subjective, the framework quotes public user reviews and names the source rather than inventing a decimal. The failure is the score that dresses a preference up as a fact.
On top of the five rules sits the arithmetic, and we kept it deliberately boring. An overall score is the unweighted mean of the tested dimensions — no weighting scheme, because weights are opinions wearing a lab coat — and each layer also gets its own sub-score, so you can read capability and trust separately. Most importantly, every overall is published with a coverage tag: tested 9 of 12, tested 5 of 12, never a bare number. A score without its coverage is exactly the kind of claim this framework exists to prevent, and the reading habits that catch such claims — check the date, check the coverage, check the version — are what the benchmarks essay at the top of this piece teaches in full.
What We Commit To
Four commitments, made in public. First, the methodology stays open: every dimension, standard task, anchor and rule is published — this post is its full description at version 0.1 — and anyone may run the framework, criticize it, or improve it. We would rather be argued with on the record than trusted blindly.
Second, Today's own scores will be published under this same framework — same tasks, same anchors, same coverage tags, same version labels — never under a private, friendlier variant. And nothing gets published until it clears the first rule above: a logged, verifiable run.
Third, scores, not rankings. The framework will never crown a winner or assemble a leaderboard, because compression is where honesty goes to die: a ranking turns twelve dimensions and a coverage tag into a single ordinal, and ordinals are a marketing instrument, not an evaluation result. A score with its context respects your judgment; a rank replaces it.
Fourth, the first published metric will be transcription accuracy, measured with character error rate. When it lands, it lands complete: the number, the test set it was measured on, and the conditions for comparing it — published together, once the audit of those conditions completes. That standard, and the reasoning behind it, is the subject of our piece on transcription accuracy and CER. You will notice this post contains no number, and that is deliberate: a figure without its test conditions is just an adjective with a decimal point.
Try It Yourself
You do not need our lab, or anyone's permission, to start evaluating assistants more rigorously than the industry average. Take the three tables above into your next trial — of Today, of anything else — and spend one evening walking the rows. Three of the twelve dimensions verify faster than the rest, and each takes less than a day.
The memory test — in one session, tell the assistant a single concrete preference: that you never take meetings before ten on Mondays, or that your emails should be short and skip pleasantries. Then close the app and stay away for at least twenty-four hours. Come back in a fresh conversation, mention nothing, and ask something the preference should shape. The second half of the test matters just as much: find where memory lives in the product, and check whether you can see the stored row, edit it, and delete it. An assistant that remembers but will not show you what it remembers is remembering on its own behalf.
The proactive test — put a new meeting on your calendar in direct conflict with an existing one, then say nothing to the assistant at all. The pass condition is specific: it must raise the conflict before the meeting starts, at a moment when you can still act, ideally with a concrete suggestion rather than a notification of the obvious. A conflict flagged after both meetings have ended is commentary, not initiative.
The permissions test — grant access to a single account, use the assistant briefly, then revoke it. What you are testing is verifiability, not the checkbox: does the product show you what revocation removed, and when you ask support what actually happens to the data, do you get a concrete answer with a timeline? A company that cannot answer that question cleanly has still answered a different one for you.
One run of any of these is an anecdote, not a score — our own framework is strict about that. But anecdotes with screenshots and dates are how logged runs start, and three honest anecdotes will tell you more than a homepage full of adjectives. Keep the receipts, whichever assistant you are testing.
Conclusion
A benchmark is not a race, and the point of running one is not to crown an assistant. Done honestly, it is an instrument for making every assistant transparent to the person about to live with it — the work it can finish, the attention it costs, and the trust it has earned and can show. That is what we want this framework to be: useful to you when you evaluate anything on the market, and binding on us when you evaluate Today. Today is free to use, so checking us costs one evening and the three tests above. Bring the scorecard, grade us first, and keep it handy for whoever comes next — get started with Today.