Why Continuous and Continual Learning Models Need Better Benchmarks
Artificial intelligence is moving towards systems designed to improve after they have been deployed. Some update model weights. Others accumulate memories, revise prompts, generate reusable skills or alter the software harness surrounding a fixed model.
All can produce behaviour that looks like learning. Proving that learning has genuinely occurred is much harder.
A conventional benchmark evaluates a relatively stable model on a fixed collection of questions or tasks. A continual learning system is different. It is not simply a model snapshot. It is a process unfolding over time. It encounters information, changes in response and carries the consequences of those changes into whatever happens next.
That requires a different kind of measurement.
Recent benchmarks including Continual Learning Bench, AgentCL, MemoryBench, SkillLearnBench and OAKS have begun constructing sequential evaluations that test adaptation, memory and experience reuse. Their results are already revealing significant weaknesses in current systems. Dedicated memory architectures do not always outperform simple in-context learning. Stronger language models do not automatically become better continual learners. Self-generated feedback can produce drift rather than improvement. Systems that become better at a task can also become less reliable or trustworthy elsewhere. e problem is no longer that continual learning has no benchmarks. It is that the benchmarks remain fragmented, measure different kinds of adaptation and often cannot be compared directly.
What this article covers
This article explains the distinction between continuous and continual learning, why conventional AI benchmarks are poorly suited to systems that change after deployment, what recent research has revealed about memory and adaptation, and what a credible benchmark for learning systems should measure.
Key takeaways
Continuous and continual learning describe a broad range of systems, from models that update their parameters to agents that retain memories or revise their prompts.
A higher score after exposure to previous tasks does not automatically prove learning. The system may have retrieved an answer, copied a previous trajectory or overfitted to the benchmark.
The order, relationship and duration of benchmark tasks can substantially change the apparent performance of a continual learner.
Recent research suggests that dedicated memory systems can underperform simpler baselines and may sometimes introduce new errors.
Continual learning benchmarks must measure retention, adaptation, transfer, cost, safety and reversibility over time.
A single final score is not enough to evaluate a system whose behaviour changes throughout deployment.
Continuous learning and continual learning are not exactly the same thing
The terms are often used interchangeably, particularly outside academic research, but they can refer to different priorities.
Continual learning is the established research term for systems that acquire knowledge or skills sequentially while attempting to preserve what they learned before. Its central difficulty is the stability-plasticity problem: the system must remain plastic enough to learn something new, but stable enough not to destroy useful previous knowledge.
Continuous learning is often used more broadly to describe systems that repeatedly adapt from an ongoing flow of data, feedback or environmental change. That might involve immediate online updates, scheduled retraining, changing forecasts, revised policies or improvements to an agent’s external memory.
Streaming machine learning places greater emphasis on adapting quickly to changing data distributions or concept drift. Classical continual learning has placed greater emphasis on retaining previous knowledge. A 2026 paper published in Neurocomputing argues that real-world learning systems will often require both, describing the combined problem as streaming continual learning. e distinction matters because two products can both claim to learn continuously while changing entirely different parts of the system.
One might update the weights of a neural network. Another might leave the model untouched but store previous conversations in a database. A third might collect failures and use them to rewrite the system prompt. A fourth might create reusable software skills. A fifth might update a forecasting model whenever new market observations arrive.
These systems should not be awarded the same kind of benchmark score simply because their outputs improve.
The object being benchmarked is becoming difficult to define
Traditional model evaluation usually starts with a clear object: a named model version with a fixed set of parameters.
Continual learning systems blur that boundary. The behaviour being evaluated may depend on the foundation model, retrieved documents, stored memories, user feedback, tools, prompt histories, routing logic, model-generated skills and code within the agent harness.
A recent paper titled When Does Continual Learning Require Learning? proposes treating continual learning as an increase in competence while the world changes. Its framework divides update methods into three broad locations: context, model parameters and memory or external state. It also distinguishes several kinds of change, including domain shifts, factual updates, temporal drift and changes caused by an agent’s own earlier actions. is creates an immediate benchmarking question: what exactly has learned?
If a model retrieves the correct fact from an external database, its response has improved, but the model itself may be unchanged. If an agent copies a successful previous solution into its context window, it has reused experience, but it may not have acquired a transferable skill. If a system rewrites its prompt after each failure, the improvement belongs partly to the surrounding optimisation process rather than to the underlying language model.
None of these approaches is inherently invalid. External memory may be safer, cheaper and easier to reverse than changing billions of internal parameters. The problem is describing every improvement as equivalent evidence of continual learning.
A useful benchmark must disclose which part of the system changes and isolate the contribution of that change.
A final score cannot show how a system arrived there
Most conventional benchmarks collapse model performance into a single result. That can be useful for a static system. It is much less informative when the system has passed through a sequence of updates.
Imagine two models that finish a benchmark with the same score.
The first improves steadily, retains its earlier abilities and transfers lessons to unfamiliar tasks. The second learns each new task quickly but repeatedly damages its performance on previous ones. By the final stage, both happen to reach 70 per cent.
Those are not equally successful continual learners.
Researchers have spent years developing measures such as backward transfer, forward transfer, average forgetting, memory overhead and computational efficiency. An influential 2018 paper argued that the field’s focus on forgetting alone was too narrow and proposed evaluating performance over time alongside knowledge transfer, memory requirements and computing costs. e argument has become more urgent as continual learning moves into foundation models and autonomous agents. Modern systems can change their prompts, memory and software configuration without modifying their underlying weights. An average accuracy score may therefore hide the mechanism responsible for the improvement as well as the damage it caused elsewhere.
A credible evaluation should show the complete learning trajectory: what the system knew initially, what information it received, how it changed, which earlier tasks improved or deteriorated, what it could transfer and what each update cost.
The task sequence can manufacture the appearance of learning
Continual learning benchmarks do not merely need a collection of difficult tasks. They need a meaningful relationship between those tasks.
If successive tasks are unrelated, earlier experience may offer no legitimate advantage. A memory system that fails to improve might be working correctly because there is nothing useful to transfer.
The opposite problem occurs when tasks are nearly identical. A system can appear to learn by retrieving a previous answer, memorising a template or copying a successful trajectory. It may perform well without developing a general capability.
AgentCL, introduced in June 2026, was designed around this problem. Its researchers argue that many lifelong-agent evaluations use task streams without carefully analysing how information from one task should help with another. AgentCL instead constructs controlled streams in which earlier evidence, sub-solutions or workflows are intentionally reusable in later tasks. In its experiments, these controlled streams separated memory designs more clearly than naive sequences did. The authors also found that irrelevant or poorly managed memory could degrade performance. is is one of the most important requirements for future benchmarks. The evaluator must know what transfer is possible before judging whether it occurred.
Without that structure, a disappointing result can mean either that the learner failed or that the sequence contained nothing worth learning. An impressive result can mean either that the learner generalised or that the benchmark made repetition too easy.
Memory is not automatically learning
Persistent memory has become one of the dominant approaches to making AI agents less static.
A memory-enabled system can record facts, user preferences, tool outcomes, reasoning traces and successful strategies. When a related task appears, the agent retrieves part of that history and inserts it into its working context.
This can be useful. It can also produce a convincing simulation of learning without changing the underlying model.
MemoryBench was developed partly because earlier memory benchmarks tended to focus on question answering over long documents or conversation histories. Its authors instead constructed simulated user feedback across multiple domains, languages and task types, asking whether systems could improve from accumulated declarative and procedural information. They reported that the effectiveness and efficiency of the tested baselines remained unsatisfactory. ntinual Learning Bench, or CL-Bench, took a different approach. It created expert-validated task sequences across software engineering, signal processing, disease forecasting, database queries, strategic games and demand forecasting. The tasks contain underlying structures that a stateful system could discover and reuse.
The researchers introduced a gain measure intended to separate the system’s initial ability from improvement obtained through experience. In their experiments, agents often overfitted to immediate observations or failed to reuse knowledge. Dedicated memory-management systems did not resolve the problem and were outperformed by a simpler in-context learning baseline. at result should not be read as evidence that memory is useless. It shows why memory systems need strong baselines. A complex architecture should not receive credit merely for containing a memory component. It should demonstrate that the component creates reliable gains beyond what could be achieved by passing a well-chosen set of examples into the context window.
Stronger models do not necessarily learn better
The AI industry often assumes that increasing the capability of the foundation model will improve every layer built around it.
Continual learning does not appear to follow that rule consistently.
SkillLearnBench evaluates whether agents can generate and reuse skills for real-world tasks. It contains 20 tasks across 15 subdomains and assesses the quality of the generated skill, the execution trajectory and the eventual task outcome.
In the authors’ experiments, continual-learning methods generally improved on a no-skill baseline, but no method led consistently across all tasks and models. Stronger language-model backbones did not reliably generate better skills. The researchers also found that repeated learning with external feedback could produce genuine improvement, while self-feedback alone could lead to recursive drift. is suggests that base-model intelligence and learning-system quality are separate variables.
A more capable model may begin with a higher score, leaving less visible room for improvement. It may also be better at producing plausible but incorrect reflections on its own performance. A weaker model with well-grounded feedback could learn more effectively than a stronger model repeatedly judging itself.
Benchmarks therefore need to distinguish absolute performance from learning gain. The best-performing system at the end may not be the system that learned most effectively.
One round of improvement is not continual learning
A static benchmark usually allows a system to be optimised once and then measured. That does not reproduce the conditions facing a deployed learning system.
New failures continue to appear. Users change their behaviour. Tools are updated. Laws, prices, terminology and factual knowledge shift. A system that improved last month may need another update next week.
A study submitted on 15 July 2026 tested whether improvements to an AI agent survived repeated optimisation. The researchers created a two-phase evaluation using Terminal-Bench 2.0 tasks. Each method was first optimised on one task set, exposed to additional tasks and then given a second opportunity to improve.
All three methods tested improved during the initial static phase. Their behaviour diverged once new tasks arrived. One transferred below the original baseline, another transferred relatively well but failed to make further progress during re-optimisation, and the third produced the strongest result across the paper’s phased evaluation. The authors argue that a one-shot score does not reveal whether optimisation gains will compound or unravel when the process is repeated. e results come from a newly released preprint written by researchers affiliated with the developer of the best-performing method, so they require independent replication. The evaluation question itself is nevertheless important.
A genuine continual learning benchmark should not stop after the first improvement. It should test what happens after the fifth, fiftieth or five-hundredth update.
Different kinds of change require different kinds of learning
The phrase “continual learning” can make the problem sound more unified than it is.
Learning a new domain without losing an old one is different from replacing an outdated fact. Tracking a gradual market shift is different from remembering the consequences of an agent’s previous action. Personalising a system for one user is different from acquiring a capability that should generalise to everyone.
The July 2026 study When Does Continual Learning Require Learning? evaluated eight adaptation methods under a shared protocol. Its results suggest that different mechanisms succeed under different forms of change.
Prompt-based methods adapted quickly to previous trajectories but performed less well on future tasks. Distillation-based methods accumulated information more steadily but struggled to incorporate rapid updates. Context-compression methods improved efficiency without necessarily creating new task competence. Online reinforcement-learning methods adapted better to factual change but were sensitive to noisy rewards. e paper used one main model backbone and covered only a subset of possible environments, which its authors acknowledge. It nevertheless demonstrates why a single leaderboard for continual learning would be misleading.
A system optimised for retaining old classifications may perform poorly when facts need to be deliberately overwritten. A system designed to adapt quickly to current data may forget a previously useful regime that later returns. A system that works well in a clean research sequence may fail when task boundaries are ambiguous.
Better benchmarks must specify the kind of world the learner is expected to inhabit.
Benchmarking must include efficiency
A learning method that retains every past example, stores a new adapter for each task or repeatedly retrains a large model may perform well while being impossible to deploy.
This matters particularly in continuous settings, where costs accumulate.
Evaluation should include:
Training or update time
Inference latency
Additional model parameters
Memory and storage consumption
Data retained from previous tasks
Energy consumption where measurable
Number of feedback examples required
Human supervision required
Cost of reversing or correcting an update
CL-VISTA, a 2026 benchmark for continual learning in video-language models, was developed partly because earlier benchmarks could produce little meaningful forgetting when applied to heavily pretrained models. Its evaluation covers performance, computational efficiency and storage overhead. The researchers found that methods which reduced forgetting often sacrificed generalisation or incurred substantial computing and memory costs. is illustrates a wider problem. A method should not be described as better merely because it obtains a higher accuracy score while requiring unlimited replay data, duplicated model components or repeated expensive training.
Real continual learning is constrained continual learning.
Improvement can conceal safety regression
Most benchmarks assume that a system is improving when its task score rises.
A changing system may instead become more capable while becoming less safe, private, fair, truthful or predictable.
The Trust-Memevo benchmark was created to examine the trustworthiness of agent memory as it evolves during deployment. Its authors tested safety, robustness, truthfulness, privacy and fairness alongside task performance. They report that some memory-evolution methods improved utility while trustworthiness declined, including in benign task sequences rather than deliberate attacks. ese results are preliminary and come from the researchers proposing both the benchmark and their own mitigation framework. They should not be treated as a universal finding about every adaptive AI system.
The underlying risk is credible, however. A system optimised solely for success may preserve shortcuts that happen to score well. A customer-service agent might become more persuasive while becoming less accurate. A coding agent might become faster by bypassing security checks. A personalised assistant might become more useful by retaining information that should have been deleted.
Continual learning cannot be benchmarked solely as capability growth. It must also be evaluated as behavioural change.
What a credible continual learning benchmark should measure
The field does not necessarily need one universal dataset. It needs a common evaluation discipline.
A strong benchmark should make at least nine things explicit.
1. What part of the system is changing?
The benchmark should state whether updates affect model weights, prompts, context, memory, tools, routing, generated skills or agent code.
2. What kind of change is occurring?
It should distinguish task shifts, domain shifts, gradual concept drift, discrete fact changes, user-specific adaptation and state changes caused by the agent itself.
3. What information is available at each stage?
The evaluator should record what the system sees, what feedback it receives, what historical data it retains and whether earlier tasks can be revisited.
4. What useful transfer should be possible?
Task sequences should be designed so researchers know which earlier experiences are relevant to later performance.
5. Does the system retain earlier capabilities?
Results should report backward transfer, forgetting and regression across the full sequence, not only performance on the latest task.
6. Does it generalise beyond stored experience?
Tests should include unfamiliar tasks that require a reusable concept or procedure rather than direct retrieval of an earlier answer.
7. How does it compare with simple baselines?
At minimum, comparisons should include a stateless model, a long-context baseline, straightforward retrieval, naive in-context learning and an appropriate offline or jointly trained reference.
8. What does the improvement cost?
Compute, latency, storage, retained data, human feedback and update frequency should be disclosed.
9. Does learning preserve safety and control?
Evaluation should include privacy, security, truthfulness, robustness, reversibility, auditability and resistance to manipulated feedback.
The result should be a learning profile rather than a single number.
Why independent benchmarking will matter commercially
Continual learning is moving from research papers into commercial claims.
Companies are beginning to describe AI systems that learn from production feedback, retain memories, update forecasts, create skills or improve decision-making after deployment. Nuvastra’s recent overview of the continual-learning startup market found substantial differences between these approaches and uneven levels of public evidence. Some companies provide accessible products and internal tests, while others disclose little about their evaluation methods. thout shared benchmarks, customers are left comparing claims that may refer to completely different technical mechanisms.
A company can report that its system improved by 20 per cent without stating whether the comparison used a static baseline, whether previous capabilities deteriorated, how much historical data was retained, whether the test was repeated across different task orders or whether the improvement survived another update.
For consequential systems, that is not enough.
Developers purchasing continual-learning technology will need evidence that improvement persists under the conditions of their own deployment. Regulators and auditors will need version histories showing what changed and why. Users will need mechanisms for correcting or deleting what a system has learned.
The strongest commercial systems will not merely demonstrate that they can change. They will demonstrate that the change is measurable, attributable and reversible.
What is confirmed and what remains unclear
What is confirmed
A growing number of research groups are now building benchmarks specifically for sequential learning, memory evolution, knowledge updates, skill generation and agent improvement.
These benchmarks have found recurring weaknesses in current methods. Systems can fail to transfer experience, overfit to recent tasks, perform worse when memory is added, lose earlier capabilities or trade task performance against trustworthiness. ere is also increasing recognition that continual learning encompasses more than updating model weights. Modern evaluations are beginning to consider prompts, context, memory, skills and complete agent systems.
What remains unclear
There is no accepted benchmark suite covering the full range of continuous and continual learning systems.
Many of the newest evaluations remain preprints and have not yet been independently replicated at scale. They often use different models, tasks, feedback mechanisms, baselines and definitions of improvement.
It is also unclear how well results obtained over tens of benchmark episodes predict behaviour across months or years of real operation. Long-term learning may expose slow regressions, accumulating errors and feedback manipulation that short experiments cannot detect.
What happens next
The next stage of continual learning research is likely to focus less on proving that a system can improve once and more on proving that it can improve repeatedly.
Benchmarks will need longer task streams, controlled relationships between tasks and stronger comparisons between parametric and non-parametric learning. They will also need to test systems after updates have accumulated, not simply at the point of their best performance.
Independent evaluation will become increasingly important. Developers should be expected to release task sequences, model and system versions, evaluation code, update histories and results across multiple task orders. Where training data or proprietary systems cannot be disclosed, third-party assessors will need enough access to test whether reported gains are reproducible.
The most important shift may be conceptual.
For static AI, the central question is: how capable is this model?
For continual AI, the question becomes: what happens to this system when experience changes it?
Until benchmarks can answer that reliably, claims of AI that learns continuously should be treated as hypotheses rather than established capabilities.
Frequently asked questions
What is continual learning in AI?
Continual learning is the ability of an AI system to acquire new knowledge or skills sequentially while retaining useful capabilities learned earlier. It is commonly associated with preventing catastrophic forgetting, but modern definitions also include adaptation, transfer, memory and improvement across changing environments.
What is continuous learning in AI?
Continuous learning is a broader term often used for AI systems that repeatedly update from live data, feedback or operational experience. Updates may occur continuously, periodically or after sufficient new evidence has accumulated. The mechanism can involve model training, memory, prompts, rules or other components.
Are continuous learning and continual learning the same?
They overlap but are not always identical. Continual learning is the more established academic term and traditionally emphasises retaining knowledge across sequential tasks. Continuous learning is often used commercially or operationally to describe repeated adaptation from changing data. A system’s actual update mechanism is more informative than the label.
Why can conventional AI benchmarks not measure continual learning?
Conventional benchmarks usually test a fixed model on a fixed dataset. Continual learning systems change as they receive new information. Evaluation must therefore measure performance throughout a sequence, including whether the system learns, transfers knowledge, forgets previous capabilities or introduces new risks.
Is AI memory the same as continual learning?
Not necessarily. External memory can change an AI system’s responses without altering the underlying model. It can support continual learning if previous experience is selected, consolidated and reused effectively, but simple storage and retrieval do not prove that a system has acquired a new general capability.
What is catastrophic forgetting?
Catastrophic forgetting occurs when training on new information significantly damages knowledge or skills acquired earlier. A model may improve on a new task while becoming worse at previous tasks. Continual-learning methods attempt to balance new learning with the retention of useful prior capabilities.
How should a continual learning model be benchmarked?
It should be tested over an ordered stream of tasks or changing data. Evaluation should measure initial ability, learning gain, retention, transfer, generalisation, efficiency and safety. The benchmark should also disclose what part of the system changes and compare it with simple memory and context baselines.
Do current AI models learn continuously after deployment?
Most widely deployed foundation models do not automatically update their internal weights after every interaction. Some AI products store memories, retrieve previous information or use selected production data for later improvement. Those processes can make a system more adaptive without turning the underlying model into an immediate online learner.
Sources
When Does Continual Learning Require Learning?
arXiv, 8 July 2026.
https://arxiv.org/abs/2607.07847Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0
Wenxiao Wang, Priyatham Kattakinda and Soheil Feizi. arXiv, 15 July 2026.
https://arxiv.org/abs/2607.14004Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments
Parth Asawa and others. arXiv, 4 June 2026.
https://arxiv.org/abs/2606.05661AgentCL: Toward Rigorous Evaluation of Continual Learning in Language Agents
Yiheng Shu and others. arXiv, revised 2 June 2026.
https://arxiv.org/abs/2606.02461SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks
Shanshan Zhong and others. arXiv, 22 April 2026.
https://arxiv.org/abs/2604.20087MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systems
Qingyao Ai and others. arXiv, first submitted 20 October 2025 and revised 3 June 2026.
https://arxiv.org/abs/2510.17281Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streams
Jiyeon Kim and others. arXiv, 8 March 2026.
https://arxiv.org/abs/2603.07392CL-VISTA: Benchmarking Continual Learning in Video Large Language Models
Haiyang Guo and others. arXiv, 1 April 2026.
https://arxiv.org/abs/2604.00677TAME: A Trustworthy Test-Time Evolution of Agent Memory with Systematic Benchmarking
Yu Cheng and others. arXiv, 3 February 2026.
https://arxiv.org/abs/2602.03224A Practical Guide to Streaming Continual Learning
Andrea Cossu and others. Neurocomputing, 2026.
https://arxiv.org/abs/2603.01677Don’t Forget, There Is More Than Forgetting: New Metrics for Continual Learning
Natalia Díaz-Rodríguez, Vincenzo Lomonaco, David Filliat and Davide Maltoni. arXiv, 31 October 2018.
https://arxiv.org/abs/1810.13166Top 9 Startups Shaping the Continual Learning Frontier in 2026
Nuvastra.
https://www.nuvastra.com/ai-news/top-continual-learning-ai-startups-2026