How we test AI agents

Updated on

This page explains how we test AI agents and Grok Bot templates. It describes the method only. We have not published any agent comparisons yet. Guides show dated runs of the scripts they teach, such as SEC EDGAR API in Python and how to read a 10-K with AI.

Same task, same inputs

When we compare agents or templates, each one will get the same task and the same inputs. If a template adds setup, we will also run that same task without the extra setup, so the difference is the setup and not a different job. One run is not a result. We will run each test task at least three times and report how many runs finished, not only the best one.

Templates we may test later are the public ones in the template library. A page in that library is not itself a test.

Where this method is used

The first jobs on the research bench follow these rules. Grok Bot vs Claude Code gives both agents the same stock research job, and Can AI pick stocks? grades an AI analyst's logged calls. Both pages show the method now and will show results once the runs are done. Every task is listed on Tasks, and every comparison on Compare.

How we score

We will score the end state, not the wording. A run will count when the outcome is real: the task is finished, the file is in place, or the check we set has passed. The criteria are:

We will check the output against the source, and we will look for failures that look like success. On top of those checks we will ask one plain question: would we hand this job off and use the result without checking how it got there? The verdict we publish will be one of three: Worth installing, Situational, or Skip. This page does not give that verdict for any agent or template, and it does not report scores.

Every result is dated

Every result we publish will say “results as of <date>”. Products and templates change, so a dated result goes stale. The date is part of the result.

When we re-test

We will re-test when a product or template changes notably. We will review each tested page at least every 90 days.

Which account and plan

Each test notes the account type and plan used, so you can compare like with like.

How we scrub screenshots

Screenshots we publish will not include account names, emails, holdings, positions, or personal information. We will remove those before a screenshot goes on the site.

No paid rankings

Nobody can pay to change a test result or a ranking. If we ever add affiliate links, we will label them, and they will never change a result or a ranking.

What the status labels mean

Each template page shows one status. The label links to this page. The words mean:

As of , every template on this site is “Not tested yet”.

To install a template yourself, read how to add a template. For what the product is, read what Grok Bot is. What bests.one is, and where things stand, is on About.

Ideas we borrowed

The commitments on this page draw on public posts. A short line credits each idea. Those authors have not endorsed bests.one.

Follow the tests