Warum machst du das?

How the video is made

Every step since the first idea, including the images that were rejected and why. The previews from 1 November onwards have sound.

1. 30 OctAfternoon

   Concept: behind every number

   The song is told from the point of view of an abandoned animal. That becomes the concept: a black-and-white documentary intro with real figures, then three fates (Sammy the dog at the service station, Mia the cat in a back yard, Flocke the rabbit in the forest), intercut with Keks and Krümmel performing in an abandoned animal shelter. At the end, a cage door opens.

2. 30 Oct22:56

   Analysing the song

   So that every cut lands on the right word, the song is transcribed locally with faster-whisper. Because the PC wasn’t allowed to download the speech model by itself, it was placed in the project folder by hand. The result: the start and end of every line to within about 0.3 seconds, checked against the lyrics. The song runs 3:20.6 and has twelve sections.

   IntroInstrumentalVerse 1Pre-chorusChorusTransitionVerse 2BridgeFinal chorusEcho & pauseOutroFade-out

   [  17.62 -   20.00] Und hinter jeder Zahl steht ein Lebewesen.
   [  20.64 -   22.00] Ein Tier, das vertraut.
   [  23.08 -   25.00] Ein Tier, das fühlt.
   [  76.66 -   78.12] Ich bin dein Freund
   [  95.52 -   98.54] Sag mir: Warum machst du das?

3. 30 Oct23:12

   Script and storyboard

   50 shots, each with a sketch, timecode, sung line, action, camera movement and opening frame. 24 shots need lip sync. Every clip is rendered at 10 seconds, and the edit only uses the best part. Several shots share one clip, which saves render time.

   All 50 shots

4. 30 Oct23:30

   The three animals get a face

   Before a single scene is made, each animal gets a reference sheet with four views. Every later image is generated with this sheet, so Sammy stays the same dog in every scene. Flocke ended up with brown rings around her eyes that weren’t in the prompt. They stay, because they make her unmistakable.

   

   

   

   

5. 30 Oct23:50

   Reference images: locations, memories, performance

   16 images are made in one go: the abandoned shelter, the performance shots of Keks and Krümmel, the happy memories, the three places where the animals are left behind, and the black-and-white motifs for the intro.

   

   

   

   

   

   

   

   

   

   

   

   

   

   

6. 30 Oct23:56

   Review and fixes

   Every image is checked against the references. Three needed fixing, each with an edit prompt rather than a fresh roll of the dice, so the rest of the image stays as it was.

   

   

   BeforeAfter

   

   

   BeforeAfter

   

   

   BeforeAfter

7. 31 Oct09:12

   14 new opening frames

   14 more opening frames are made for the shots in the storyboard. One was fixed: in the night scene the car was still driving away, although it should be long gone by then. One planned image was dropped because the clip of the service-station scene ends on exactly that image.

   

   

   

   

   

   

   

   

   

   

   

   

   

   

   

   BeforeAfter

8. 31 Oct09:30

   Scene descriptions for the clips

   Every clip gets a detailed scene description in English: the action second by second, camera, light, colour, atmosphere, and what must not change. A prompt tool turns that, together with the opening frame, into the video prompt for ComfyUI. Part 1 with the intro (S01–S09) is done, and all nine video prompts have been generated.

   ACTION TIMELINE
   0–2s: The eye is open and calm, looking slightly past the camera.
   2–4s: Sammy blinks once, slowly and naturally.
   4–6s: The eye shifts its gaze slowly toward the camera, as if he noticed someone.
   
   CONSISTENCY – MUST NOT CHANGE
   Eye color (warm amber-brown), the tan fur pattern around it ...

9. 31 Oct10:30

   Part 2: verse 1 and pre-chorus

   The scene descriptions for S10–S18 are done. New this time: every lip-sync prompt now states to the second who sings when and when the mouth stays closed, derived from the song analysis. In the duet part, Keks and Krümmel often sing only half lines in turn; without this, both would move their lips the whole time. All nine video prompts for this part have been generated.

   AUDIO TIMING (German lyrics)
   0.0–2.5s: Krümmel sings "ich dachte, dass sich daran niemals was ändern kann."
   2.7–5.1s: Krümmel sings "Doch jetzt steh ich hier an diesem grausamen Ort,"
   5.3–6.4s: Krümmel sings "du gehst einfach weg –"
   6.4–7.8s: Krümmel is silent, his mouth stays closed.

10. 31 Oct11:15

    Part 3: chorus and second verse

    The scene descriptions for S19–S33 are done: the chorus with both singers in turn, the swinging light bulb as a transition, and Mia’s story, from waiting at the front door to the back door being pulled shut. That covers 32 of 44 clips. All 14 video prompts for this part have been generated.

11. 31 Oct12:00

    The storyboard becomes part of the page

    The storyboard is now built right into the page. Each shot has its sketch and the finished opening frame, with a switch between the two views. Every image can be clicked and browsed at full size.

    All 50 shots

12. 31 Oct12:20

    Part 4: bridge, finale and ending

    The last scene descriptions are done: Keks alone with Sammy’s collar in the bridge; the final chorus, where the camera makes the two look smaller and smaller in the empty corridor; the light going out; the outro sitting down; Sammy at dawn; and finally the hand that opens his cage door. All 44 clips are now described.

13. 31 Oct12:50

    All 44 video prompts checked

    Before rendering, every prompt goes through one more check. Three problems turned up and were fixed directly in the text: in five prompts the prompt tool had split one continuous clip into several numbered shots, which can cause hard cuts in the video. Some duet clips lacked the note that the other singer keeps his mouth closed. And some prompts were over 5,000 characters; they were shortened in the sound descriptions, which the song replaces in the final video anyway.

14. 31 Oct12:55

    Ready to render

    A checklist sets the order in which the 44 clips are made: first 25 clips without lip sync to find the right settings in ComfyUI, then 13 lip-sync clips with one person, and finally the 6 duets. The matching 10-second excerpts of the song were cut automatically for every lip-sync clip. ComfyUI numbers every attempt, so every version is kept for the making-of.

15. 31 Oct13:58–16:44

    The first clips

    S01 to S06 are rendered, each shot four times with a different seed, in full HD at 48 frames per second. A clip takes about 7 minutes, the 24 attempts together just under three hours. For review, each attempt is turned into a contact sheet of 12 stills and compared. Most attempts were usable and differed only slightly. One outlier: in S04, version 2, the dog turns its head away and suddenly there are three eyes in the frame. The selected clips:

    

16. 31 Oct18:30

    S08 and a change of system

    The title shot through the empty shelter is done, rendered under Windows 11 this time instead of CachyOS. The difference is clear: around 10 minutes per clip instead of about 7. Version 3 was rejected because the model cuts hard to a close-up of the leash in the middle of the camera move. Version 2 was used.

    

17. 31 Oct – 1 Novuntil 09:50

    Phase A: all clips without lip sync

    Seventeen more shots, again four attempts each, again under CachyOS with the RTX 3060 Ti unlocked: a good 6 minutes per clip on average. For review there’s a comparison sheet per shot with all four versions stacked. Typical outliers: in S17 one version zooms all the way in on the dog unasked, in S14 the dog’s face changes colour, and in S35 a figure suddenly appears in the background. In S43 the light doesn’t go out at the planned second in any version, so a later part of the clip is used. S39 is still missing.

    

    

18. 1 Nov09:37–17:30

    Lip sync: phase B and the first duets

    64 clips in just under eight hours straight, about 7:20 minutes per clip on average. Lip-sync clips take a little longer than the scenes without singing. Because no screenshots of the render times were taken here, the times were calculated from the file timestamps. For review, each check sheet shows when the song has singing and when the mouth must be closed. The lip-sync model sticks to that surprisingly well. Two problems: in S48 Keks’s opening frame was used by mistake, and the model invented a stranger with different glasses. In S19 Keks briefly opens his mouth in three of four versions while Krümmel is singing. These previews have sound.

    

    

19. 1 Nov19:47–22:32

    Last clips: everything rendered

    S48 was rendered again with the right opening frame, along with the remaining duets and the reserve shot S42. That completes all 45 clips: 44 for the script plus one reserve. In the final check, checklist and files were compared: every shot has a signed-off version, and every file is there. In total there were 184 attempts in just under 22 hours of render time.

20. 4 Nov09:25–09:34

    Edit: timeline for DaVinci Resolve

    From the storyboard, the clip checklist and the hand-over, a script builds the timeline as FCPXML for DaVinci Resolve: all 50 shots without gaps across 3:20.6, the song as the only audio track, clip audio muted, plus the title in S08 and the black screen with text at the end. Every lip-sync clip starts on its cue, frame-accurate. The storyboard had three tiny gaps of 0.2 to 0.3 seconds before the chorus, the bridge and the final chorus; there, the previous shot now stays a little longer. As a check, a rough-cut preview with the song was rendered from the same edit list. That revealed that the child’s face becomes visible towards the end of the excerpt planned for the S44 flashback. So S44 now comes from seconds 4 to 6.5 of the puppy clip, stretched to half speed. Still open for the fine cut in Resolve: S04 in black and white with colour spreading in, the fade-ins, the text design, and a lip-sync check with sound for S19, S37, S40 and S45.

    S18 | 1:12,4–1:16,7 | S18_Clip_00001 | 0,00–4,31 s | bis 76,7 s verlängert
    S23 | 1:32,4–1:35,4 | S13_Clip_00001 | 7,00–10,00 s
    S25 | 1:38,5–1:43,8 | S25_Clip_00003 | 3,00–8,29 s
    S44 | 2:48,3–2:53,2 | S09_Clip_00002 | 4,00–6,46 s · Tempo 50 %
    S50 | 3:17,5–3:20,6 | Schwarzbild mit Text

    

21. 6 Nov20:18

    Lip sync: second attempt

    After the rough cut, the lip-sync clips aren’t convincing. A likely reason: in the solo shots the model also heard the other singer’s voice and moved the lips to it. So the pure vocal track now goes into rendering, with other voices and echoes muted in eleven excerpts. The cues have been re-measured word by word, and all 20 scene descriptions got their own section on mouth shapes: lips closed on M, B and P, wide open on A and O, completely still in the pauses. In the six duets both voices stay in the audio; there the prompt has to make sure only one of them sings. Rendering the new versions is next.

    Stummgeschaltete Stellen in: S07, S12, S15, S18, S24, S29, S32, S36, S42, S47, S48
    Duette mit beiden Stimmen:   S07, S19, S22, S37, S40, S45

22. 8 Novuntil 18:11

    Lip sync v2: 14 of 20 shots rendered and checked

    With the pure vocal track and the new scene descriptions, the lip-sync shots run a second time, again with four attempts each. For the first time, rendering happened on two machines: S07, S10 and S12 on my own PC under CachyOS with an unlocked RTX 3060 Ti, S15 to S36 at Ascot Media under Windows 11 with an RTX 5070 Ti (16 GB), for which there is no unlock. Even so, the new card is a little faster: a median of about 7:00 minutes per clip versus 7:44 minutes on the 3060 Ti. The first attempt of every shot always takes longest, between 8:47 and 11:20 minutes, because that’s when the models are loaded. One run of S12 crashed. Together that makes 56 attempts in just over 7 hours of render time. For sign-off there is a check sheet per shot: the face every 0.2 to 0.3 seconds, with the sung words above it. The result is clearly better than the first attempt. In the pauses the mouth stays closed almost everywhere, and in the duets S07 and S22 only the one whose turn it is sings. Three weak spots remain: in S36 Keks closes his mouth while the long “vertrauen” is still ringing. In S18 the held “kalt” ends a little too early. In the wide shot S19 the faces are too small and too vague. Still missing: S37, S40, S42, S45, S47 and S48. The previews here have the song as audio.

    Shot | System | Versuch 1–4 (Minuten)
    S07 | CachyOS · 3060 Ti | 10:41 · 7:44 · 7:49 · 7:55
    S10 | CachyOS · 3060 Ti | 9:26 · 7:39 · 7:28 · 6:41
    S12 | CachyOS · 3060 Ti | 9:27 · 7:17 · 7:44 · 6:48
    S15 | Win 11 · 5070 Ti | 11:20 · 6:59 · 6:57 · 6:46
    S16 | Win 11 · 5070 Ti | 9:07 · 6:45 · 6:47 · 7:01
    S18 | Win 11 · 5070 Ti | 8:47 · 6:55 · 7:04 · 6:55
    S19 | Win 11 · 5070 Ti | 8:56 · 7:05 · 6:57 · 6:57
    S22 | Win 11 · 5070 Ti | 8:58 · 7:27 · 6:56 · 6:55
    S24 | Win 11 · 5070 Ti | 9:06 · 6:59 · 7:07 · 6:45
    S27 | Win 11 · 5070 Ti | 9:10 · 7:10 · 7:19 · 6:54
    S29 | Win 11 · 5070 Ti | 9:04 · 6:55 · 6:55 · 7:18
    S32 | Win 11 · 5070 Ti | 9:18 · 7:25 · 7:17 · 7:09
    S34 | Win 11 · 5070 Ti | 9:06 · 7:12 · 7:04 · 6:56
    S36 | Win 11 · 5070 Ti | 9:17 · 6:56 · 7:20 · 6:56

23. 8 Nov20:30

    Duets: new attempt with reference imagesin progress

    For the solo shots, the second attempt worked. The duets are still struggling with the faces, above all in the wide shot S19: when Keks and Krümmel are small in the frame, their faces get blurry and lose their likeness. S19 has therefore been dropped. All six shots in which both can be seen (S07, S19, S22, S37, S40, S45) get new scene descriptions. On top of the opening frame, the model now also receives portraits of the two singers as references, so it knows what the faces look like up close. The three remaining solo shots S42, S47 and S48 have been sharpened based on what the sign-off showed: on long, held words the mouth should stay open until the sound ends, and nobody should smile on sad lines.

    Duette mit Referenzen:  S07, S19, S22, S37, S40, S45
    Bilder je Duett:        1. Startbild  2. keksprofil.png  3. kruemmelprofil.png
    Offene Solo-Shots:      S42, S47, S48