Dear YouTube!
In a Nutshell
YouTube is introducing video AB testing that lets creators upload different cuts of the same video and have YouTube automatically test which version holds viewer attention best. This feature breaks the shared viewing experience that makes content culturally meaningful, since viewers won't know they're watching one version of potentially multiple cuts, and comments/timestamps will become unreliable across versions. The speaker argues creators should reject this retention-optimization approach, as endlessly testing creative decisions produces less interesting content than standing by your choices and learning from publishing multiple videos.
These notes were generated by AI and may contain inaccuracies.
The speaker thinks about YouTube's strategic direction often, specifically that the platform spends too much time chasing other platforms and competitors when they should focus on what makes YouTube unique. Over the past couple years, there have been many changes to YouTube, some user-facing, some creator-facing. Some changes are great, others not so much, but the latest change is considered their craziest, riskiest, and possibly worst idea yet.
YouTube announced a feature called video AB testing during their annual Made on YouTube event. Other features revealed included custom homepages, new live stream features, and AI shorts editing tools. While some features might be useful, video AB testing is viewed as a straight-up bad idea.
Creators have been able to AB test thumbnails and titles for a little while, starting last year. YouTube shows one thumbnail to one part of the audience and the other thumbnail to the other part to see which performs better, then lets you pick the winner. The same testing capability exists for titles. However, AB testing the video itself seems like it goes too far.
The feature was presented as solving a classic problem: shooting two different opening hooks and not knowing which would perform better. With video AB testing, creators can upload different cuts (shorts or long form), and YouTube tests them with real viewers to help select the video cut that holds audience attention best. At the end of the test, a chart shows three different retention graphs from all versions, allowing the uploader to choose the winner based on which holds viewer attention longest.
Several questions remain unanswered by the presentation. Viewers won't know if a video is part of an AB test. Comment sections become problematic since readers won't know which version of the video commenters watched, and timestamped comments may break across different versions. Questions include how long after publish creators can AB test a video, whether sending a video link could result in different versions being viewed, and how different the tested videos are allowed to be from each other.
Experience with thumbnail testing revealed that small differences between thumbnails (slightly different text size, arrow placement, or even arrow versus no arrow) usually don't produce statistically significant results. The most valuable learning comes from testing completely different concepts. Applying this logic to video cuts, small differences like two or three shots in a cold open likely won't make meaningful statistical differences.
The presentation example showed videos of drastically different lengths: 17 versus 19 minutes. This raises concerns about timestamps breaking across versions. YouTube has had limited video replacement capability behind the scenes for specific reasons like factual corrections, but the replacement video must be the exact same length as the file it's replacing.
YouTube engineers explained that video AB testing will use a semantic analysis model that examines both video files to determine their similarity, only allowing tests with high enough similarity scores. This theoretically prevents completely different videos from being tested against each other. However, confidence in this approach is limited, as AI systems like Gemini often default to transcripts when analyzing YouTube videos.
AB testing will only be allowed at video launch, not for old videos. Creators cannot AB test and replace existing videos with different cuts. The rollout is planned for early next year to everyone.
Comments with timestamps will likely be confusing and broken. Uploaders won't know which video version commenters watched. Viewers won't know they're watching an AB tested video. People might watch a cut that loses the test, then return later to find a different video. The effectiveness of the similarity test score remains uncertain - whether one different sentence that changes the video's conclusion would flag it, or if out-of-order cuts would be detected.
The core concern is that YouTube's community relies on an unspoken rule that everyone has the same experience watching the same video. This shared experience is what makes content a cultural phenomenon. Allowing multiple versions of the same video at once fundamentally breaks this principle.
This video AB testing represents a broader trend of YouTube responding to competitors rather than focusing on unique strengths. The public video dislike counter was removed, framed as protecting creators, but Facebook, Instagram, and TikTok don't show dislikes either. The like-to-dislike ratio was a uniquely useful YouTube feature. View counting has also changed to count views when the first frame loads, making views less valuable compared to the previous sophisticated measurement system.
Most creators won't need this feature and probably shouldn't use it. The feature is framed as helping improve videos, but it actually helps find which version has the best retention, not necessarily which is a better video. Creative decisions that aren't retention-optimized can still be fun, creative, interesting, new, entertaining, and characterful.
The top 0.1% of channels focus on mass appeal, but much of what makes YouTube fun is niched-down content by definition. If channels became super retention-optimized, they would become less fun to watch. Optimizing exclusively for retention trims out creative elements that aren't the most efficient storytelling methods.
The skill of being a video creator involves finding creative ways to share, teach, explore, explain, or review content, not necessarily the most efficient optimized approach. Making and publishing multiple videos is better than making one video with multiple drafts of titles, thumbnails, and cuts to optimize for marginal view duration improvements.
Creators should stand by their creative choices and share the video they chose to make. Learning comes from both wins and failures, then applying those lessons to the next creation. The creative process involves having conviction in what you want to share with the world rather than endlessly optimizing for retention metrics.
Keep Marques Brownlee in your library
Save the videos and channels worth coming back to, and find them again in one place.
