While putting together the rough cut at the start of November, I noticed something you can't unsee once you've seen it: the lips in the singing shots don't quite match. In some of them the wrong person is even singing. So: take two.
What went wrong
The likely cause is pretty mundane. In the solo shots the model also heard the other singer's voice and dutifully moved the lips along with it. The AI has no idea who's meant – all it hears is "someone's singing".
So now the model only gets the isolated vocal track. I muted other voices and echoes in eleven sections. Every entry was re-timed word by word, and all 20 scene descriptions got their own section on mouth shapes: lips closed on M, B and P, wide open on A and O, completely still during pauses. Sounds like speech therapy for pixels. It pretty much is.
Two machines, 56 attempts
Every shot runs again with four attempts. What's new is that, for the first time, rendering happened on two machines: S07, S10 and S12 on my own PC under CachyOS with an unlocked RTX 3060 Ti, everything from S15 onwards at Ascot Media under Windows 11 with an RTX 5070 Ti (16 GB), which has no unlock available.
The newer card is still a bit faster: a median of about 7:00 minutes per clip versus 7:44 on the 3060 Ti. The first attempt of every shot always takes longest, between 8:47 and 11:20 minutes, because that's when the models get loaded. One run of S12 crashed. All in all: 56 attempts in just over seven hours of render time.
Sign-off with check sheets
So I don't have to watch every clip twenty times in slow motion, each shot gets a check sheet: the face every 0.2 to 0.3 seconds, with the sung words and the vocal track above it. At a glance you can see whether the mouth stays shut during pauses and whether the right person is singing. Whether the lips land on every syllable still needs one pass with sound.
The result is clearly better than the first attempt. The mouth stays closed in pauses almost everywhere, and in the duets only the person whose turn it is sings. 13 shots are signed off.
S07, their first appearance together (version 4): Keks stays still during Krümmel's "Ein Tier" and then sings cleanly.
S22, chorus (version 4): the hand-off Keks, Krümmel, Keks works, and whoever isn't singing keeps his mouth shut.
What's still off
Three weak spots remain:
- S36: None of the four versions holds the long "vertrauen". Keks closes his mouth from about 3.2 seconds while the note is still ringing. Version 1 holds it longest.
- S18: The sustained "kalt" ends a little too early in every version.
- S19: In the wide shot the faces are too small and too vague. The shot has been dropped.
S36 (version 1): the head lifts, but on the long "vertrauen" the mouth closes too soon.
S18 and S36 are fine for now. I'll make the final call in the edit.
What's next
For the solo shots, take two worked. The duets are still struggling with the faces. When Keks and Krümmel are small in the frame, their faces get blurry and stop looking like them. So all six shots that show both of them (S07, S19, S22, S37, S40, S45) get new scene descriptions. On top of the start frame, the model now also receives portraits of both singers as references, so it knows what the faces look like up close.
I've tightened up the three remaining solo shots S42, S47 and S48 based on what the sign-off taught me: on long, held words the mouth should stay open until the sound ends, and nobody should smile on sad lines. Sounds obvious. Apparently not to a video AI.
Still to render: S19 (again), S37, S40 and S45 as duets, plus S42, S47 and S48 solo. Then it's on to the fine cut.
As always, all clips, render times and the full diary are on the project page.