Putting trading claims to a harder test

A research paper released on September 28 proposes a benchmark for evaluating AI trading agents across historical tests, forward paper trading and trading with real money. Its title asks a direct question: Can AI Make Money in Crypto?

The framework focuses on the gap between a strategy that looks profitable on recorded prices and one that can execute successfully in a live market. It is a proposed evaluation method, not proof that an AI system can reliably produce profits.

Why simulated returns can mislead

A backtest can assume an order trades at a price that was briefly visible. Real orders encounter delays, limited liquidity, fees and market impact. A strategy that trades frequently may lose much of its apparent advantage once these frictions are included.

Historical testing also creates opportunities to overfit. Repeatedly adjusting a model until it performs well on one dataset can reward memorization rather than a durable trading signal.

Three stages answer different questions

A historical test checks behavior against past data. Forward paper trading tests decisions on information arriving in real time. Real-money execution then reveals whether the model can obtain the prices and fills its results require.

Comparisons are more informative when they report costs, failed orders and risk alongside returns. One strong period is not enough to establish reliability.

A framework for scrutiny

The paper gives researchers a structured way to examine AI trading claims. For readers, the central question is whether a reported result survived realistic execution conditions, rather than how impressive its headline return appears.