automated pipeline for turning long-form content into youtube shorts. speech-to-text for captions, nlp for highlight detection, video summarization for segment selection. the goal was scale — produce engaging short-form video without sitting through hours of editing.
the pipeline: transcribe the full video with whisper, feed the transcript to an llm with instructions to identify the 3-5 most engaging segments, extract those segments from the video, generate captions, add styling, output as shorts. the transcribing and extracting are solved problems. the identifying is where it gets hard.
the hard part was making the output feel intentional rather than chopped. most automated shorts look like they were cut by a machine because they were. the ai picks the segment with the highest "engagement score" based on keyword density and sentence structure, but engagement isn't a text property — it's a pacing property. the best clips have a setup, tension, and payoff. the ai can identify the payoff sentence but it can't tell you if the 10 seconds before it provide enough context for the payoff to land.
the best results came from treating the ai as a first pass editor, not a final one. let it identify candidate segments, then manually review and trim. saves 80% of the time while keeping the 20% that actually matters — the human judgment about what feels right. fully automated shorts are technically possible and emotionally empty. semi-automated shorts are the sweet spot.