Since our last blog post, “Evaluating Memory Models,” Giacomo and I have been implementing and evaluating existing memory models and, more recently, designing and training new ones. We are excited to share some initial results.
[!warning] Preview status
We are fixing some consistency issues in the validation dataset, so the results below may change slightly. The broad pattern has been stable across the checks we have run so far, but the numbers should still be read as preview results rather than final benchmark claims.
With that disclaimer out of the way, here is what we have done.
Dataset
We are working with a large AnkiHub review dataset built from real Anki review logs. After cleaning, the current dataset contains:
-
53,586 users
-
107,304 reviewed cards
-
28,903 reviewed notes
-
4.76 billion reviews
-
3.91 billion successful reviews, where success means
rating >= 2 -
847 million failed reviews, where failure means
rating == 1
The overall success rate is 82.2%. The rating mix is heavily concentrated in “Good,” which accounts for about 65% of all reviews, followed by “Again” at about 18%, “Hard” at about 12%, and “Easy” at about 5%.
We use a temporal cutoff to split training data from validation and test data. Reviews before January 18, 2026 are used for training. Reviews on or after January 18, 2026 are held out, and users are split 50/50 into validation-eligible and test-eligible groups. All results in this note are validation-only. We are reserving the test data for the final reporting at the end of the project.
The model comparison below mostly uses the full validation split: 95,846,768 post-cutoff reviews from validation users. The validation base rate is 80.3%, so even the held-out data is strongly success-skewed. The ERT prediction parquets currently have a slightly different row count because of the same validation consistency issue noted above; we include them here because they are useful preview results, but the exact comparison should be rerun after cleanup.
We were pleased to see that forgetting is observable in the data.
Existing Models
We started by implementing and evaluating several baselines.
Averaging Baselines
Before looking at memory models, we include three simple averaging baselines to anchor the scale.
avg_global predicts the same probability for every validation review: the global success rate in the pre-cutoff training data.
avg_per_user predicts each user’s own pre-cutoff average success rate for all of that user’s validation reviews.
avg_moving_per_user is an online moving-average baseline from srs-benchmark.
IRT
Item Response Theory is the simplest useful baseline in this set. The IRT model predicts:
where \alpha is the global base rate, \theta_{\text{user}} is user ability, and \beta_{\text{card}} is card difficulty. This model does not know how long it has been since the last review, and it does not model the card’s review history. It only asks: how skillful is the user in general and how hard is this card?
IRT is not a dynamic memory model in that it predicts the same p-recall regardless of interval, but we include it as an intelligent baseline.
FSRS v6
FSRS is the memory model currently used in Anki. It tracks same-card state variables, most importantly stability and difficulty, and predicts recall as a function of elapsed time since the previous review. Unlike IRT, FSRS takes the interval into account, outputting forgetting curves.
We evaluated several FSRS v6 variants: the published default parameters from GitHub, per-user fitted parameters, the median of those per-user fits, and a median over the top 10% highest-data users. Our evaluation setup is deliberately stricter than the usual “periodically re-optimize FSRS for each user” setup. We train on pre-cutoff history and evaluate on a fixed post-cutoff validation period without re-optimizing during validation. For sequential models like FSRS, we replay each user’s pre-cutoff history to build state as applicable, then score the post-cutoff reviews.
FSRS v6 has an important practical limitation: it was not designed to produce meaningful probabilities for same-day reviews (delta_t = 0 days). We therefore keep all validation rows, but for same-day rows we emit the most recent inter-day prediction while still updating the FSRS state. That is, all reviews for the same (card, user, day) get the same probability of recall.
KAR3L
KAR3L is the strongest published memory model we evaluated. To our knowledge, it is also the first content-aware, general-purpose memory model. It embeds card text, retrieves similar cards from the user’s history, and uses both ordinary same-card review statistics and statistics for similar cards to predict retrievability.
Our implementation follows the KAR3L paper’s baseline philosophy: bert-base-uncased, frozen embeddings, CLS pooling, and top-k retrieval from the user’s prior review history. For Anki cloze cards, we embed the card front with the clozed answer replaced by [...], so the representation reflects what the student actually sees rather than revealing the answer. We evaluated various retrieval depths (i.e. the number of most similar cards’ data to use) K = 0, 5, 10, 25; increasing retrieval depth consistently helped, with K = 25 best among the KAR3L variants in the current run. In other words, looking at the history of other, similar cards is helpful.
New Models
We now have two related families of new model architectures. Both are based on the same broad approach: feed the model a window of the user’s previous reviews across many cards. Each review is an event vector that includes information about the card content, time of the review, and the outcome. The model predicts the next review from the preceding event stream.
ERT stands for Event Retriever Transformer. The ERT models retrieve the last few review events for the top k most similar cards to the query card, then feed this event history to a transformer with a more explicit memory head. In ERT v4, the head separates short-term memory (fast decay), long-term memory (slow decay), and consolidation boost terms (discrete increases each day for the first few days after a review). The first ERT model, ert_v4_b000, is a single-bucket training run. The second, ert_v4_b000_001_cont, continues training into buckets 0 and 1. This continued run is the stronger of the two.
The first memory-transformer version, memory_transformer_v1, is a content-free cross-card baseline. It represents cards by random identity vectors and uses the user’s cross-card review history, timing information, and outcomes. This was designed as a control: how far can we get from full-history modeling alone, before adding semantic content?
The second version, memory_transformer_v2, replaces those random identity vectors with frozen Qwen3 Q+A embeddings of the card version active at the time of review. That is the first direct test of the core content-aware claim: does card text add predictive power on top of full-history behavior?
The fourth version, memory_transformer_v4, makes the content path trainable with a per-version delta table added to the frozen embeddings. On smaller training subsets, this overfit, but at full scale it improved over v2. That suggested there is card-specific signal not captured by the frozen text embedding alone.
The current best model, memory_transformer_v5, keeps the content encoder frozen but uses a much larger 2048-wide, 6-layer delta-rule backbone. Importantly, it does not use the v4 per-card delta table, so it is less reliant on memorizing individual card IDs. It is closer to the actual content-aware goal: generalize from the text and history structure rather than just learning a lookup table for every card.
These models are still works in progress, but on this validation benchmark they already appear to be the strongest models we have evaluated.
Evaluation Refresher
Every model outputs a probability of success for each held-out review. We binarize Anki ratings as:
-
Again→ failure -
Hard,Good,Easy→ success
We report several metrics because each answers a different question.
Log loss is the main proper scoring rule. It strongly penalizes confident wrong predictions and rewards calibrated probabilities. Lower is better.
Brier score is the squared error of the predicted probability. It is also a proper scoring rule, but less harsh than log loss for very confident mistakes. Lower is better.
SmECE is a smooth expected calibration error. It asks whether events predicted at probability 0.8 happen about 80% of the time, events predicted at 0.9 happen about 90% of the time, and so on. Lower is better.
AUC measures ranking ability: does the model assign higher probabilities to successes than to failures? Higher is better. AUC is useful, but it does not by itself tell us whether the probabilities are calibrated.
Precision and recall at 0.90 evaluate a scheduler-relevant operating point. If the model predicts a success probability of at least 90%, precision asks how often the review actually succeeds. Recall asks what fraction of all successful reviews are above that threshold.
Results
Here is the current full-validation comparison.
The heatmap prints the raw value in each cell, while the color shows each model’s standing within that metric. Green is better, red is worse, and the bar chart on the right summarizes each model’s mean normalized standing across metrics.
