Motion Hack

Overview

Turn a short talking-head video into a Seedance 2.0 prompt for an animated motion-graphics reel: the speaker is re-rendered in a stylized look that changes per beat, lip-synced to the original audio, with motion graphics that illustrate the words, animated transitions, animated cut-out text and timestamped SFX. Cut-paper collage is the flagship style, but any look works (the user's references define it). The reel is generated by Seedance from the original video alone, so the styles are described in the prompt and the creator uploads only their video.

When to use

• A creator has a short talking video (a hook, a tip, a mini tutorial) and wants a viral collage / motion-graphics version.
• They want the visuals (styles, text, effects) to land exactly on the words.
• They want a ready-to-paste Seedance 2.0 prompt in English and Chinese.

Setup (one time, free and local, no API keys)

• Python 3.8+
• ffmpeg on PATH (Whisper reads audio through it)
• pip install -r requirements.txt (installs openai-whisper)
  First transcription downloads a Whisper model (~140MB) once. See README.md for per-OS install commands.

Inputs

• Required: the talking-head video (any language).
• Optional but recommended: a few collage style reference images (e.g. Pinterest finds). Look at them to describe each style faithfully. With no refs, propose strong distinct collage looks or ask the creator.

Workflow

1. Transcribe the video word-level:
   python scripts/transcribe.py <video> base transcript.json
   Every word comes back with start/end seconds.
2. Show the creator the full transcription with timestamps, as a clean visual table (every word with its start and end time, grouped into the detected beats). This is the timing backbone and lets the creator see exactly where each word lands. Example row format: word | start to end.
3. Auto-detect beats from the transcript: split on natural pauses and phrase boundaries. Aim for 1.4s or more per beat so each visual can breathe and lip-sync. If the speech is too fast for the number of beats, use fewer beats and tell the creator.
4. Assign one style per beat (collage is the flagship). With refs, describe each faithfully and spread them across beats; without, pick distinct easy-to-render looks.
5. Build the prompt with references/prompt-template.md, filling the real timestamps from the transcript.
6. Deliver all three to the creator:
   • the full transcription table with timestamps,
   • the beat + style + motion-graphic breakdown table,
   • the prompt in both English and Chinese (Chinese tends to adhere better on Seedance; English to read or to try on Gemini Omni). Speech and on-screen text stay in the creator's language.
7. Tell the creator: upload only the video to Seedance, then paste the prompt.

Non-negotiable prompt rules

• Person consistency: every shot is the same person from the video; only the style changes.
• Full-frame: each style fills the whole frame edge to edge, never a centered patch with empty borders.
• Lip-sync on every shot, copied 1:1 from the reference video, including abstract styles.
• Motion graphics illustrate the words (literal, in cut-paper material), entering on the matching word. Match the MG to the active style (e.g. shattered style means numbers that assemble from shards).
• Text never covers the face. Captions sit above the head or below the chin only.
• Animated transitions, never hard cuts (morph, paper shuffle, spray wipe, tear).
• SFX only, no background music. Every transition and motion graphic gets its own timestamped SFX.

Common mistakes

• Text over the mouth (kills lip-sync readability). Keep it above or below the face.
• Too many beats for a short clip (rushed cuts). Fewer beats, each 1.4s or more.
• Asking Seedance to render the real footage inside a motion graphic (it glitches). Use the look only (a film strip, an old camera), never embed the actual video.
• Long baked-in text (it warps). Keep on-screen text short.

See references/prompt-template.md for the full bilingual prompt skeleton, the motion-graphic idea library, and a worked example.