Real ones know I’ve never used AI tools to write this blog because it’s my baby and I will always give my children my whole heart. I’ve made one (1) exception for Shreya Shankar’s new writing IDE DocWriter (“Claude Code for writing”), which I regard as far more ambitious than an average AI tool because it is explicitly designed in service of the quality- and voice-conscious human writer. I’ve found DocWriter particularly useful for suggesting enhancements via in-line prompts from me as I write without being overly prescriptive or forceful. DocWriter integrates seamlessly with Claude and gives me the flexibility to offload mundane tasks as I write and maintain word-level control over DocWriter’s prompted suggestions with a human (me)-first feedback system. I encourage any technical, research and/or hobby writer to reach out to learn more. Thanks Shreya!
Detecting everything, everywhere
Hot on the heels of successfully vibe-coding (in an hour) an on-screen facial recognition software suite that detected Daniel Molloy in Interview with the Vampire and played SexyBack in real time, I set out to solve a more complex problem.
As we started Company Retreat, Jenny (the inspiration behind Molloy Detector) pointed out a number of items on screen that she was interested in purchasing as party favors, including these chili pepper necklaces:
She was forced to manually search for them on Amazon, yet another glaring platform problem begging to be automated.
aside: Prime Video’s X-Ray
Prime Video’s X-Ray feature already overlays, in real-time, a panel of actor pages and trivia provided by IMDb:
This is an impressive feat of engineering but not actually in the ways you might suspect. As we demonstrated with Molloy Detector, facial recognition is a solved problem and Amazon can afford a high quality standard via its ownership of IMDb and AWS’s AI suite. The harder problem is delivering this much temporal information consistently and dynamically to many concurrent viewers. Perhaps Netflix, recently measured at twice Prime Video’s viewership, hasn’t implemented a version of X-Ray for more reasons than not owning a comprehensive catalog like IMDb.
But if the hard part isn’t facial recognition, how hard is it to extend this to brand and product detection on screen?
No-context window shopping
The closest solutions I could find deployed at scale were Shop Your TV and WornOnTV, which seem to asynchronously pull and publish screenshots labeled with product details and, at least on WornOnTV, direct links to buy them.
However, neither Shop Your TV nor WornOnTV had updated their catalog to include Company Retreat, indicating a data lag and popularity bias. Shop Your TV and WornOnTV could be using completely end-to-end human workflows to identify and label clothing- and jewelry-containing screenshots from popular shows to put on their websites, but I hope for their collective sanities that they are at least using some kind of automation to routinely collect and filter through screenshots to annotate and serve to their customers. Based on specific annotations it looks like they may also be relying on social media sleuths to help label this data.
Overall, I found the on-screen clothing detection market to be smaller and more handmade than I initially assumed.
Another hour with Claude: prototyping ShopMyScreen
Seed prompt: “build a clothing detection software for tv and film that allows me to click links to buy clothing i see on screen” / “name it
shopmyscreen”v0.0: Claude one-shots a UI similar to Molloy Detector that will make calls to Claude Vision, which will cost money. This is reasonable because I never added “don’t use Claude Vision it costs money” to
CLAUDE.md. Still, I have to think of the end user (“ok is there a free method”), to which Claude proposes:BLIP (Bootstrapped Language-Image Pretraining) requiring a 1GB local download, which will caption images that can be used as input to retailers
FashionCLIP, built on OpenAI’s CLIP (Contrastive Language-Image Pretraining) “adapted for the fashion industry”, requiring a smaller 600MB local download. The FashionCLIP paper correctly calls out “the huge operational costs of training and developing models” leading to “the value of ML innovations… mostly captured by a few players” motivating their decision to open source their model.
v0.1: Since we’re running this locally, BLIP and FashionCLIP both require some boot-up time which Claude doesn’t initially incorporate into UI. I guide Claude to add progress bars and additional instructions into the UI so users don’t confusion-quit out of the application before local models have a chance to load.
v0.2: Open-source BLIP and FashionCLIP can only identify at the granularity of “blue shirt”. Functionally, BLIP and FashionCLIP are both downloadable locally and ready to run on screenshots and Claude picks a few retailers to link off to (Google Shopping, Amazon, ASOS, Shopstyle) that I wouldn’t have picked myself, but it’s a start. Claude did warn me (paraphrased) that I would get what I paid for; these open-source alternatives to Claude Vision were lacking in their ability to recognize specific clothing brands.
v0.3: I ask Claude to add an option to image search. Claude proposes Google Lens on the cropped part of the screenshot containing the clothing item, but Google Lens does not have a public API. There is a workaround: send the image in POST request to Google Lens reverse image search endpoint. On macOS this still requires some user action to click into Google Lens results, but brand identification accuracy is much higher than BLIP or FashionCLIP:
v0.4:
ShopMyScreencan identify categories of clothing on screen and direct link to Google Lens. From there, the user can adjust the image to search on and find exact match with high success rate guaranteed by Google Lens.
Show-stopper or show-flopper? Prime Video’s Shop the Show
Amazon recently launched Shop the Show on Prime Video for customers to “discover toys, apparel, and other products related to what they’re watching”, only available on the mobile app, which is definitely capitalizing on watching-TV-and-scrolling-phone behavior.
For example, Company Retreat’s Shop the Show features a huge number of t-shirts and merchandise that I will probably be buying at some point, but that doesn’t solve the ShopMyScreen problem: I could not find the chili pepper necklaces anywhere on this Shop the Show page.
ShopMyScreen final boss: chili pepper necklaces
Using ShopMyScreen, I was able to identify the chili pepper necklaces via Google Lens. Admittedly there was not much use for FashionCLIP or BLIP for this necklace detection task because I had to crop the screenshot myself, but at least it’s more specific than Prime Video’s Shop the Show.
Tailoring ShopMyScreen is hard to do all at once
At this point, we have a functional prototype (the software can detect clothing on screen and I can click off to determine where exactly to buy it), but there are a few major scalability and usability concerns:
Thin UI layer / minimal improvement over manual Google Lens search. Without Claude Vision, our options for accurate brand detection are limited to Google Lens, which (without unauthorized calls via some proxy API) still requires users to click off, crop the screenshot to the clothing or item they want to image search, and find the retailer themselves. This isn’t too different from the completely manual flow of taking a screenshot and pasting it into Google Lens yourself. In fact, it’s basically a “Google Lens wrapper” with some categorical detection from FashionCLIP / BLIP.
FashionCLIP / BLIP without Google Lens is basically useless, unless you really just want to buy a blue shirt on Amazon.
Training open-source clothing brand detection models without logos is a hard problem. There are open-source solutions to many subproblems that could collectively feed into a reasonable
ShopMyScreensolution (logo recognition DeepLogo, brand/logo Brand Eye), but clothing poses an often logo-less problem. These open-source models should have no issue identifying Coca-Colas and BMWs on screen, but unlabeled clothing is far less generalizable and will also intuitively have less training data, putting open-source clothing brand detection models at a disadvantage. Google Lens can do it, Claude Vision can probably do it, and this is where the FashionCLIP paper’s observation that ML innovation is captured by a few big players in the market (because of prohibitively high training and operational cost) becomes particularly salient.Integrating
ShopMyScreeninto video streaming platforms would be exponentially more complex. This prototype relies on an initial download of up to 1GB local models (FashionCLIP / BLIP) and runs locally by taking screenshots and updating links to Google Lens and retailers. The advantage is that any video streaming platform that can be screenshotted will work with this local software. But if we tried to build a browser extension or feature for any video streaming platform to do this, we would need to compute model outputs and serve them to the viewer alongside the actual video content, the same way X-Ray does for Prime Video. InShopMyScreen’s case, the number of detectable products is at least an order of magnitude higher than IMDb’s actor database, so to scale this we would probably have to significantly limit the types of products that would trigger a detection and link off to buy.ShopMyScreenis currently tuned to clothing recognition (e.g. FashionCLIP), but could be more effective as a simple logo detector. As referenced earlier, logo detection is an easier problem than facial recognition, so if product logos (brands, cars) are visible on screen, there are sufficiently good open-source models that we wouldn’t need to jump through all the hoops of using Google Lens to get around Claude Vision.
Next up for ShopMyScreen
v0.4 is up on GitHub, but it’s not a viable product beyond single user at this point. I’d like to push the limits of real-time brand recognition and explore serving model results via video streaming overlay next. I could also explore models tuned to identify logo-less clothing brands, but the most pertinent engineering bottlenecks in developing both ShopMyScreen and Molloy Detector are in linking images to model results and building a cost-effective way to stream these to viewers in real-time. I’d like to get to a point beyond locally run software and downloaded models processing periodic screenshots before exploring accuracy improvements to the actual models.
I would like if ShopMyScreen could identify the type of chili pepper so that if search results are inconclusive, I can make my own chili pepper necklace at home