How to use Video-Use: let Claude Code, Codex, and other AI coding agents edit video with natural language

Type “cut the ums” into a chat box and, if the agent’s only move is to dump the whole mp4 into a vision model, the bill and the latency blow up first. browser-use’s video-use takes another path: coding agents that can run commands, such as Claude Code and Codex, read the transcript, write an edit decision list (EDL), then call ffmpeg to render. The repo states it is 100% open source. The entry point is github.com/browser-use/video-use. This article follows the README, install.md, and SKILL.md in that repo as of 2026-09-23. Commands and directories follow those files that day. If a pull changes them, your local SKILL.md wins.

The kinds of pieces the project lists: talking-head, montage, tutorial, travel, interview. There is no preset menu. What you want is an agent session that can run to the end, plus a machine that will not sleep halfway through the render after transcription.

What video-use actually cuts

You do not open a timeline. You put the raw takes in a folder, start the agent in that directory, and say the goal in plain language. The official example is one line: edit these into a launch video. The agent inventories the footage, proposes a strategy, waits for you to agree, then writes edit/final.mp4 next to the footage. The skill repo itself stays clean. Outputs must not be written into video-use/.

The finished-film capabilities listed in the README are things ffmpeg and a few optional animation engines can already do. They are not a new model:

• Cut filler and dead air between takes: umm, uh, false starts, silence
• Grade per segment: a warm cinematic look warm_cinematic, a neutral punch neutral_punch, or your own ffmpeg filter chain
• A 30ms audio fade on every cut, so you do not get a pop
• Burned-in subtitles: two words, all caps, by default. The style can change
• Overlay animation: HyperFrames, Remotion, Manim, or PIL. Several animation slots run as parallel sub-agents
• A self-check after the render, and only then does it show you the result. Session memory lives in project.md, so you can pick up the cut next week

It is the same family of idea as browser-use reading the web: instead of filling the context with screenshots, give the model structured text. For video, that text is the transcript.

How the model “reads” video

The SKILL is blunt: the LLM never watches the video. It reads the video. Two layers are enough for it to cut on word boundaries.

Layer one is always loaded: the audio transcript. Each source file goes through ElevenLabs Scribe once, for word-level timestamps, speaker diarization, and audio events such as (laughter), (applause), and (sigh). Every take is packed into one ~12KB takes_packed.md. A phrase breaks when silence reaches 0.5s or the speaker changes. The model mostly reads lines like this:

## C0103  (duration: 43.0s, 8 phrases)
  [002.52-005.36] S0 Ninety percent of what a web agent does is completely wasted.
  [006.08-006.74] S0 We fixed this.

Layer two is on demand: a picture composite. timeline_view exports a filmstrip, a waveform, and word-label PNG for one time window. Use it only at decision points: an ambiguous pause, which of two retakes to keep, whether a cut is clean. It is not a scanner. Do not call it every second.

The project compares a naive path: 30,000 frames times about 1,500 tokens per frame, roughly 45 million tokens of noise. The video-use path is 12KB of text plus a few PNGs. That number is the repo’s own comparison, not a bill measured on this site. Remember the direction: the context should be words and times, not extracted frames.

Install: Claude Code, Codex, OpenClaw

The machine needs four things at once: a clone on a stable path, ffmpeg and ffprobe on $PATH, an ElevenLabs key in .env at the repo root, and the current agent able to find SKILL.md. Python dependencies in pyproject.toml: requests, librosa, matplotlib, pillow, numpy. There is no console command. Scripts run directly as python helpers/….py.

Hand install to the agent

The project suggests pasting the block below to Claude Code, Codex, Hermes, OpenClaw, or any agent with a shell. It clones, installs dependencies, registers the skill, and asks you for the key once when it needs it:

Set up https://github.com/browser-use/video-use for me.

Read install.md first to install this repo, wire up ffmpeg, register the skill with whichever agent you're running under, and set up the ElevenLabs API key — ask me to paste it when you need it. Then read SKILL.md for daily usage, and always read helpers/ because that's where the editing scripts live. After install, don't transcribe anything on your own — just tell me it's ready and wait for me to drop footage into a folder.

The key is on the API keys page at elevenlabs.io. The install check only looks at timeline_view.py --help and ffprobe -version. Do not transcribe a real clip just to prove the install worked. Scribe is billed per call. After the key is in ~/Developer/video-use/.env, run chmod 600. Do not echo it. Do not commit it.

Clone and register it yourself

If you want to do it by hand, follow the repo’s manual install. The symlink is the whole directory, because helpers/ has to sit next to SKILL.md:

git clone https://github.com/browser-use/video-use ~/Developer/video-use
ln -sfn ~/Developer/video-use ~/.claude/skills/video-use
# Codex: ln -sfn ~/Developer/video-use ~/.codex/skills/video-use

cd ~/Developer/video-use
uv sync                         # or pip install -e .
brew install ffmpeg             # required; ffprobe comes with it
brew install yt-dlp             # optional, only when pulling footage from a URL

cp .env.example .env            # ELEVENLABS_API_KEY=...

ffmpeg must be 4.x or newer. A static build is fine. On Linux, use the distro package. Do not guess a sudo password. HyperFrames, Remotion, and Manim are installed the first time an animation slot needs them. Do not globally prefer one of them on install day. HyperFrames currently wants Node.js 22+. yt-dlp is lazy-installed the same way.

To update, git pull --ff-only. The symlink picks up the new code on the next session. If the dependency list changed, run uv sync again.

The first finished cut

After install, you are in the footage directory, not the skill repo:

cd /path/to/your/videos
claude    # or codex, hermes

The first sentence is not “start rendering.” Say: inventory these takes and give me a strategy in plain language. The official flow is ask until it is clear, confirm, then execute, then self-check, then write it into project.md. Until the strategy is agreed, it must not cut.

A normal session goes through questions that change with the footage. There is no fixed checklist: the kind of piece, target length and aspect, tone, moments that must stay, moments that must go, whether you want animation and a grade, whether you want subtitles. Then the agent writes the shape in 4 to 8 sentences: which take, where to cut, animation, grade, subtitles, expected duration. After you reply that you agree, it writes edl.json, grades per segment, and calls render.py.

Preview with render.py --preview (720p, fast). The self-check looks at the finished picture: a timeline_view about ±1.5s around every cut, plus the head, the tail, and two or three spots in the middle, and ffprobe for duration. If it fails, it revises, renders again, and looks again, at most 3 self-eval passes. If something is still wrong after 3 passes, it must flag you. It must not loop forever. By the time you see the preview, the self-check should already have passed.

The project also offers a hosted try-out (Browser Use Cloud) and an always-on entry from a VPS or Telegram through Browser Use Box. On your own machine or a Cloud Mac, the path is the one above: a footage directory plus a registered skill.

Folders, scripts, and the EDL

Originals do not move. The whole session lands in <videos_dir>/edit/:

The pipeline is a straight line. Self-eval can fall back to render, but there is a cap:

Transcribe → Pack → LLM reasoning → EDL → Render → Self-Eval
                                              └ if something is wrong, revise, at most 3 rounds

The scripts you actually use live in helpers/, with paths relative to the directory that holds SKILL.md:

• transcribe.py: Scribe one file. You can add --num-speakers. The result is cached
• transcribe_batch.py <videos_dir>: 4 workers in parallel. Use this when there are many takes
• pack_transcripts.py --edit-dir: fold the JSON into takes_packed.md
• timeline_view.py: a filmstrip PNG at decision points
• render.py <edl.json> -o …: extract per segment, concat, overlays, and burn subtitles last. --preview is 720p. --build-subtitles writes master.srt along the way
• grade.py: a preset, or a custom chain via --filter. Grading happens when the segment is extracted. Do not re-encode the whole piece after the concat

edl.json is roughly this shape. Times and paths come from your footage. Do not copy the seconds in the sample:

{
  "version": 1,
  "sources": {"C0103": "/abs/path/C0103.MP4"},
  "ranges": [
    {"source": "C0103", "start": 2.42, "end": 6.85,
     "beat": "HOOK", "quote": "...", "reason": "cleanest take"}
  ],
  "grade": "warm_cinematic",
  "overlays": [
    {"file": "edit/animations/slot_1/render.mp4", "start_in_output": 0.0, "duration": 5.0}
  ],
  "subtitles": "edit/master.srt",
  "total_duration_s": 87.4
}

When several takes contain the line you want, another edit sub-agent opens. It assembles by beat (HOOK, problem, solution, and so on) rather than by file order, and in and out must land on word boundaries. The structure can be a launch film, a tutorial, an interview, or something you invent. Do not lock it to one template.

Hard rules that silently ruin a cut

The SKILL leaves artistic judgment to the model and nails down only the 12 rules that silently ruin a picture if you ignore them. Below are the ones you actually hear and see while cutting. The rest (cached transcripts, parallel animation, confirm before you cut, outputs only in edit/) are already covered above.

A local Whisper SRT is explicitly an anti-pattern: it is slow, and it normalizes filler away. Transcription goes through hosted Scribe and is cached per source file.

How footage is handled

Cuts come from speech and breaths. The picture follows. Silence of ≥400ms is usually the cleanest cut. A phrase boundary at 150–400ms can work, but look at the picture. Under 150ms is usually mid-sentence and unsafe. A speaker handoff can keep a 400–600ms breath. Tighten it for a fast cut, loosen it for documentary. That is taste, not a law. Leave a tail on laughs, punchlines, and emphasis. The laugh itself is the beat. An event such as (laughs) is a signal. Do not place the cut just before it.

Every cut has to work in sound and picture together. Fixing only the audio or only the picture is a failure mode the SKILL names.

Grade per segment. Look at one frame with timeline_view, change one thing, then look again. Presets are only a start: warm_cinematic is the safer one on talking-head, neutral_punch only adds contrast and does not move hue, and none is the default when the user never mentioned a grade. If you have not tested skin tones, do not push the slope hard.

Subtitles have three knobs: how many words on a line, letter case, and how far from the bottom edge. A ready style for a short launch video is two words, all caps, Helvetica 18 Bold, white with a black edge, MarginV=35. A documentary can switch to one line of 4–7 words, normal case, and a larger margin. Either way, burn subtitles last, and timecodes use the output timeline.

Animation picks an engine per slot. Do not default to Remotion just because it “looks like the web”:

• HyperFrames: HTML/CSS/GSAP, product-UI motion, transparent WebM. In the slot directory, npx --yes hyperframes. It wants Node 22+
• Remotion: only when the user names React, or a Remotion design system already exists
• Manim: formulas, state machines, and morphing graphs. The repo ships skills/manim-video/
• PIL + PNG sequence: counters, a typewriter, a single reveal. The launch-film example uses this path. It is fast

If color and type are not clear, stop at the strategy stage and draw only after confirmation. Cards locked to narration run about 5–7s when they are simple and 8–14s for a complex diagram, and the settle holds at least 1 second. Reveals use a cubic ease-out, not a linear one. Several animation slots open sub-agents at once. Wall-clock time is about equal to the slowest one. Each sub-agent prompt has to be self-contained, because they cannot see the parent session.

Aspect follows the source unless you specify otherwise. Common targets: cinematic 1920×1080 at 24fps, screen content at 30fps, vertical 1080×1920, square 1080×1080. render.py scales whatever it receives to 1080p by default. Delivery aspect is worth one question in the strategy.

Natural-language prompts

Useful prompts constrain the strategy. They do not write ffmpeg filters for the agent. The lines below are ones you can say in the same flow. Chinese or English both work. The agent reads intent:

• “Do nothing beyond transcription for now. Inventory filenames, durations, and whether there is dialogue. Give me a strategy in six sentences, and wait for my reply before you write the EDL.”
• “This is a product-launch talking-head. Target 60–90s, 16:9. Cut the ums, ahs, and false starts. Keep the line We fixed this and the laugh after it.”
• “Both takes cover pricing. Use the one without a flub. Cuts land on word boundaries, with ~80ms of pad on each side.”
• “No grade. No subtitles. --preview first. I will agree before you make final.mp4.”
• “Vertical 1080×1920. Subtitles as two capitalized words. The only animation is a 5-second title card. Confirm the colors before you make it.”
• “The source files did not change. Do not re-transcribe. From my last note, only change the HOOK out-point.”

When notes come back, use the natural-language loop: restate the strategy, re-render, and do not re-transcribe. The final comes out only after you confirm the preview. Each session appends the strategy, the decisions, and the leftovers to project.md. The next time you enter the directory, the agent should summarize the last session in one sentence, then ask whether to continue.

When not to use it

It is an editing skill on a coding agent, not a color suite.

• A music montage with almost no dialogue. Cut design is built mainly on speech boundaries and silence. You have to write the picture intent into the strategy yourself, or the model has no main text to read.
• You want to hand-drag keyframes, switch multicam live, or do fine secondary color. Leave those in Final Cut, Premiere, or DaVinci. video-use handles the rough cut, filler removal, burned-in subtitles, and simple overlays. The result can go into an NLE afterward.
• The transcription budget. Scribe spends ElevenLabs credit. The cache exists so you do not pay twice. Running a long clip through transcription during install is the most common waste.
• Self-check is not a review. It samples PNGs and waveforms near the cuts and catches jump cuts, pops, and covered subtitles. It does not guarantee the joke lands, or that the brand color is right. Taste still needs your yes.
• Rights. Only process footage you have the right to cut. The skill can pull a URL with yt-dlp. That is a footage inlet, not a reason to bypass copyright.
• The key. Keep it only in the skill repo’s .env. Do not write it into the footage directory, and do not paste it into a chat log that syncs somewhere else.

Why long sessions belong on a Cloud Mac

A launch film is often this: batch transcription, strategy revisions back and forth, a 720p preview, then one or two animation slots, then 1080p. ffmpeg and the sub-agents both take time. Close the laptop lid and the agent process sleeps with it. edit/ stops on a half-finished preview.

• Do not drop the session. Open tmux on a Cloud Mac and run Claude Code or Codex inside it. After a disconnect the render is still going, and project.md is still there.
• The toolchain is brew. The official macOS path is brew install ffmpeg. The same machine also has Python, uv, and the Node 22 that HyperFrames wants. You do not switch to another Linux machine mid-render to chase filter differences.
• Disk follows the footage. Originals and edit/ live on the disk where the agent is. The symlink points only at the skill repo, not at the footage.
• Splitting it from coding work is fine. One machine is simpler. On the same Mac, Codex edits a repo in the morning and video-use cuts the launch talking-head in the afternoon. Both are long sessions, and both hate sleep.

The model may change next week. The skill directory and ffmpeg will not. Put the agent on a Vuncloud Cloud Mac. The laptop only has to watch preview.mp4 and reply “make the HOOK two seconds shorter.”

FAQ

Is video-use editing software?

No. It is an open-source agent skill from browser-use. You talk in the footage directory. The agent reads the word-level transcript, writes an EDL, and renders edit/final.mp4 with ffmpeg. There is no timeline UI.

Which agents can use it?

Any agent that can run a shell and read SKILL.md. The official paths cover Claude Code, Codex, Hermes, and OpenClaw. Symlink the whole repo. Do not symlink only a markdown file.

Do I have to use ElevenLabs?

For transcription, yes. Scribe provides word-level timestamps, speaker diarization, and events such as laughter. The key goes in the repo .env with mode 600. Without the key, the pipeline cannot start. Do not test transcription on a real clip during install.

Does the model watch the whole video as pictures?

It does not keep watching. The main text is takes_packed.md. Picture is sampled only at decision points and during the finished-cut self-check, through timeline_view PNGs.

Where is the finished file, and are the originals changed?

Originals stay where they are. Outputs live in edit/ under the footage directory. The final is final.mp4. It should not cut until you confirm the strategy.

Can it replace Premiere or Final Cut?

For a rough cut of talking-head and tutorials, removing filler, and burning subtitles, it is fast. Keyframes, multicam, and fine grading still go to an NLE. For a piece with almost no dialogue, you have to state the picture strategy yourself.

Conclusion

Three lines: video-use makes a coding agent read the transcript instead of extracted frames; it should not render before you agree; the finished file is edit/final.mp4 next to the footage, not inside the skill repo. Get ffmpeg, the key, and the whole-directory symlink right first. The first sentence only asks for a strategy. Talk about grade and animation after the preview has passed.

Filler and breaths belong on this pipeline. Taste and rights stay with you. Do not let the machine sleep in the middle of a render.

Related reading

• Best AI coding tools in 2026: Claude, Cursor, Copilot, Codex, Gemini
• Codex weekly limit exhausted: fixes and alternative APIs
• OpenAI DevDay 2026: Agents API, Hosted Sandbox, and Codex