I Ran 56 Experiments to Find the Best Way to Make AI Watch Videos

Cross-posted from marcindudek.dev I Ran 56 Experiments to Find the Best Way to Make AI Watch Videos I wanted a simple thing - feed a video to an AI running on my Mac and get back useful descriptions of what's happening in each frame. Not a cloud API. Not a $200/month subscription. Just a local pipeline that actually works. Three days and 56 experiments later, the biggest finding was counterintuitive: telling the model what the speaker is saying matters more than any vision trick, OCR injection, or bigger model. The Problem With Video Understanding Most "AI video tools" are wrappers around OpenAI's API. You upload your video, pay per minute, and get back generic summaries. That's fine for some use cases, but I wanted something that runs locally, processes any video, and extracts specific details - option names, numbers, UI labels, before/after states. Think screen recordings of software, tutorials, product demos. The kind of video where "a person is showing a WordPress admin panel" is u