Can AI pick stocks? How we grade an AI analyst
Updated on
Educational only, not investment advice.
This page asks one question in plain words: can AI pick stocks? An AI research agent has logged its stock calls in a ledger since 2026-10-03, and each call is graded when its own horizon ends. This page explains how. It does not give the answer.
How the grading works
The ledger is kept by an AI research agent that runs on Hermes Agent. It covers China A-shares and ETFs listed in Shanghai and Shenzhen, including cross-border QDII and gold ETFs, starting on 2026-10-03. Every call is logged before the outcome, one row per call in a dated daily file. A logged row is never edited: reviews and follow-ups are added below it. Grading uses daily prices from after the call was logged, up to the end of its horizon. That is point-in-time data.
Horizon
Each call sets its own horizon in trading days: 5, 10, 20 or 60. A one-month call is 20 trading days and a three-month call is 60. Days are counted on that security's own trading calendar. The agent reviews calls as they reach T+5 and T+20, and a weekly script scores every call at the end of its own horizon.
Benchmark
The CSI 300 index (000300.SH) is the default benchmark, for QDII and gold ETFs too: the question is whether a call did better than simply holding the CSI 300. A single call can name a different benchmark when it is logged. If the benchmark data is missing, the call is scored on its own return and the gap is noted.
How a call is scored
The script, not the model, does the arithmetic. Stops are checked against closing prices.
| Type of call | When it counts | Hit | Miss |
|---|---|---|---|
| Buy-type: OPEN_PROBE, OPEN_STANDARD, NEXT_TRANCHE_PER_PLAN, WATCH | Only once triggered: some day's low inside the horizon comes within 0.5% of the entry reference price, or goes below it. Otherwise the call ends as Not triggered and no return is counted. | At the end of the horizon, a gain that is at least the benchmark's. | A close below the stop after the trigger, or at the end of the horizon a loss that trails the benchmark. |
| Sell-type: REDUCE_REVIEW, TRIM_HALF_TIGHT_EXIT, EXIT_DISCIPLINE | Always, measured from the anchor close on the day it was logged. | A loss over the horizon, or trailing the benchmark. | A gain above 3% while beating the benchmark. |
| Hold: HOLD_NO_ADD | Always, measured from the anchor close. | Matching or beating the benchmark. | A close below the stop, or trailing the benchmark by more than 3 percentage points. |
| No view: NO_POSITION_VIEW, MANUAL_REVIEW | Not scored. | ||
Anything that is neither a hit nor a miss is Neutral. Until its horizon ends, a call is Pending. A call with no price data is marked Insufficient data. The daily review can also note a call as Extended, but the weekly score still uses the horizon the call was logged with.
Duplicates and follow-ups
Before logging, the agent checks the last 20 trading days. If the same security already has an open call with the same stance, it does not log a new call. It adds a follow-up line that cites the original call ID, and any change to the entry or the stop goes there. The score still uses the original row. A changed stance, or an original call past its horizon, gets a new call ID that cites the old one. When scoring, the script also merges any duplicate it finds into the earliest call and reports the counts before and after merging.
Hit rate
When we publish results, the hit rate will be hits divided by hits plus misses, counted after merging duplicates. Neutral and Not triggered calls are left out. There is no hit rate on this page yet.
The ledger format
Each call is one row with these 16 fields. The table defines the fields. It is not a list of calls.
| Field | What it records | On this site |
|---|---|---|
| call_id | A stable ID such as C20261009-01, so a later review points at the same call. | Shown |
| date | The decision date (T0). | Shown |
| lane | new, holding, portfolio or watch. | Shown |
| account | Which account the call is for. | Not shown |
| symbol | The security's code. | Masked |
| name | The security's name. | Masked |
| stance | One of a fixed set, such as OPEN_PROBE, HOLD_NO_ADD or REDUCE_REVIEW. | Shown |
| confidence | High, medium or low. | Shown |
| entry reference price | One price. A buy-type call counts only after the price reaches it. | Field name only |
| stop | A closing price that would invalidate the call. | Field name only |
| size rule | The position-size limit, in words. | Field name only |
| horizon | 5td, 10td, 20td or 60td. | Shown |
| catalyst | What should move the price, and when. | Not shown |
| anchor | The close on the decision date, written as close@date. | Not shown |
| strongest counter-case | The best argument against the call. | Not shown |
| sources | Where the call came from. It can also name a different benchmark. | Benchmark only |
Reviews are added in their own section, never over the call row: call_id, review date, T+n, close, change against the entry or anchor, change against the benchmark, result and a note.
Ways to run an AI stock-research agent
The ledger on this page is kept by an AI research agent that runs on Hermes Agent. Other ways to run the same kind of job, which we have not run for this page:
- Grok Bot with a general research template (Grok Bot)
- Self-built with Claude Code (Claude Code)
- A SaaS research tool (various vendors)
- ChatGPT scheduled tasks (ChatGPT)
We will compare them only after running the same job on each, as described in how we test. The results will be published on this page.
Setup outline
This is an outline, not a tested recipe.
- Pick a data source you can cite.
- Write the logging rules into the agent's instructions. Use the fields above. Log before the outcome. Never edit past calls.
- Schedule the agent. For Grok Bot scheduled routines, read how routines run.
- Keep the ledger in a file or sheet the agent can append to but not rewrite.
- When each call's horizon ends, grade it against its benchmark with a script, not by hand.
- Review the mistakes.
Sample instruction text for the agent:
Log each call before the outcome, one row per call:
call_id | date | lane | account | symbol | name | stance | confidence | entry_ref | stop | size_rule | horizon (5td/10td/20td/60td) | catalyst | anchor (close@date) | countercase | sources (bm=<code> if the benchmark is not the CSI 300)
Never change a logged row. To update a call, append a follow-up line that cites its call_id.
Before logging, check the last 20 trading days: an open call on the same symbol with the same stance gets a follow-up, not a new row.
Limits and failure modes
- Look-ahead bias. An LLM may already know what happened in a past period, so backtests on old data can look better than they are. That is why we only grade calls logged before the outcome.
- Survivorship bias. Testing only on companies that still exist leaves out the ones that do not.
- Small samples. A few dozen calls cannot show skill. One lucky month proves nothing.
- Made-up numbers. Models can invent figures. Every number in a thesis must link to a source.
- Benchmark choice. A call is only meaningful against the benchmark stated at logging time.
- Costs and slippage are not captured by a paper ledger.
FAQ
Can AI pick stocks?
That is the question this ledger is designed to test. We have not published results. No AI tool should be treated as a reliable stock picker without a long, honest track record.
How do you grade an AI's stock picks?
Each call is logged before the outcome with its stance, confidence, horizon, entry reference price, stop and benchmark. When its horizon ends, a script scores it against the benchmark as a hit, a miss, neutral or not triggered. A logged row is never edited, and wrong calls stay in the ledger.
How long is each call graded over?
Each call sets its own horizon in trading days when it is logged: 5, 10, 20 or 60. A one-month call is 20 trading days. The horizon is fixed in advance, so a call can't be judged on a convenient date.
Why are the stock symbols hidden?
So this page can't be read as a list of picks. We publish how each call is graded and, later, how the calls turned out, not which securities to buy or sell.
Is this investment advice?
No. This page is educational only.
Related on bests.one
- How we test — the rules for running the same job more than once.
- How Grok Bot routines run — what a schedule does.
- How to create a Grok Bot — the setup before any routine.
- Research — a general research template, not built for investing.
- Fact Check Bot — for checking claims against sources.
Ideas we borrowed
A short line credits each idea. These authors have not endorsed bests.one.
- LLMs may have already seen the test period, so old-data backtests can mislead — @gemchange_ltd (https://x.com/gemchange_ltd/status/2099115980026704123)
- Classic look-ahead and survivorship bias, plus the model-memory version — @LLMQuant (https://x.com/LLMQuant/status/2072655143540064662)
- Use point-in-time data — @GfI_Himmelreich (https://x.com/GfI_Himmelreich/status/2098671353838641203)