Why async video needs Mental State Tracking

When we started building Virtual we assumed the hard part would be the AI, getting the model to answer viewer questions accurately from a transcript and a sparse set of frames. It turned out to be the easy part. Modern LLMs are very good at that.

The hard part was the metric. The thing we are actually optimising is comprehension, did the viewer understand what they just watched, and there is no clean signal for that in any existing video tool.

Why attention is the wrong target

Every analytics tool measures attention. View counts, play rates, average watch time, drop-off curves. These metrics exist because they are easy to measure, not because they are useful.

Attention is a confound. A viewer rewinding the same segment three times is "highly engaged" by every traditional metric. They are also profoundly confused. The metric is firing for the opposite of the reason you want it to fire.

Worse: a viewer who understood your video perfectly and watched it once at 1.5× looks "less engaged" than the viewer who rewound twice and dropped at minute three. By the time you compare cohorts, the data is noise.

Questions are the closest thing to truth

The signal we landed on is questions. A viewer who asks a question at a specific timestamp is telling you, unambiguously, that they did not understand the previous twenty seconds. No interpretation required.

Better still: the question itself is structured data. The text reveals the gap. "What does API mean?" tells you the vocabulary failed. "Why does this option exist?" tells you the UX rationale failed. "Wait, is the heatmap real-time?" tells you the spec was unclear.

Compared to "average watch time dropped to 62%", which could mean anything, a real viewer question is a high-resolution signal. It points at the exact moment, and it tells you what to fix.

The noise problem

The catch: not every question is a comprehension signal. "Great video!" is a question with a question mark, technically, but it tells you nothing about what the viewer understood. Neither does "first!" or "are you a bot" or "what are you doing today", all messages we have actually seen in our viewer logs.

These conversational messages crowded out the content doubts in our early prototypes. Creators looked at the Master Doubts queue and saw fifty greetings and three real questions. The signal-to-noise was bad enough that creators stopped reading the queue.

Mental State Tracking

The fix we shipped is a content classifier that separates content doubts from conversational talk. Greetings, small talk, AI-identity questions, one-word reactions, all filtered out of the Master Doubts queue and the confusion heatmap, but still counted in the raw question total.

The classifier is conservative: it tags about 30-40% of inbound messages as conversational. That number is roughly stable across our customer base, about a third of inbound viewer messages, regardless of niche, are not content questions.

We call this Mental State Tracking because the question itself is a window into what the viewer was thinking, were they trying to understand the content, or were they just being friendly? Both are valid, but only one is actionable. The Master Doubts queue is now made entirely of the actionable kind.

What it changes

Three months in, the data is clear: creators who use Virtual answer about 60-70% fewer DMs and ticket-style messages, because the AI handles the conversational tail. Master Doubts queues went from ~40 mostly-noise items per recording to ~5-12 high-signal items. The heatmap stopped flagging false hotspots driven by chit-chat.

For anyone shipping async video at scale, courses, onboarding flows, sales demos, the lesson generalises: measure comprehension, not attention. The technical hard part is the classifier. The product hard part is committing to a metric that does not look as good in marketing charts but actually reflects whether your video worked.

Try it on a real recording

The free trial gives you 100 viewer questions on one recording. Upload an MP4, share the link, watch the heatmap fill in over the first 24 hours. Pay attention to the Master Doubts queue, what you see there is what the rest of your dashboard should have been showing you all along.