the joys and pains of changing frame rate

If you're not planning to print your carefully-constructed movie project to 35mm film, but if you want to work with approximately 24 frames per second, then I recommend working at 23.976 frames per second. It seems that as of this writing, HD digital video standards all seem to privilege 23.976, which means a frame rate change is inevitable if you want to put your 24fps work to tape.

"The Incident at Tower 37" was produced at a native fps of 24. When it came time to stick it on DVD, HDCAM stock, DigiBeta, etc., we needed 23.976 (which I will refer to from now on as 23.98).

23.98 and 24 aren't so far apart. What gives? Why is this such a big deal?

Think of it this way: after your film has played for 100 seconds, you'll either be on frame 2400 or frame 2398 depending on your frame rate. That's a 2 frame difference. After another 100 seconds, you're going to be off by 4 frames. These small deltas matter if you have sync sound. If you don't, then stop reading here and work at whatever framerate you want (though keeping the DV standard rates in mind is probably a good idea).

Okay, so take Tower 37. It has carefully synced sound with cracking glass, foot falls, a gunshot, etc. It runs just under 10 minutes long. For the purposes of this post, let's call it an exact 10 minutes. 600 seconds. At 24 fps, that's 14,400 total frames.

If I just play the 14,400 frames more slowly, at 23.98 instead of 24 fps, the film runs 600 and one half seconds. This means that any sound synced to the end of the film will hit a half second early, which is very noticeable. In other words, I've got 12 "extra" frames beyond the duration of the sound.

Let's also assume that the sound is locked and cannot change. Given my limited mastery of audio tools, this is more than an assumption. It's a given.

So to solve this problem I need to play the original frames more slowly AND somehow toss 12 frames along the way. And do so in such a way that preserves sync as much as possible.

When you retime a QuickTime movie with Compressor, you are given choices in how you should create the new frames. The fastest is simple: just choose the nearest frame. What this means is that Compressor keeps track of what the ideal frame should be, presumably as a fractional number, and then it rounds it in some way when choosing which frame to present. Think of it this way: at 100 seconds, Compressor needs to pick a frame from our original movie to use for frame 2398 of the new 23.98 fps movie. The most logical choice is 2400, since frame 2400 was exactly at 100 seconds in the original movie. So that's an easy choice.

But what about frame 2397? Some simple math shows that frame 2397 would correspond to frame 2398.999 from our original movie. But there was no frame 2398.999 in our original movie, was there? There were just frames 2398 and 2399. I think that all of you would probably choose 2399 as the "closest" to 2398.999. That's what Compressor does, too.

Keep going backwards. At what frame of the new movie will the ideal frame lie 50% between two frames from the original movie?

(go ahead, take some time and try to figure it out)

Basically, I compute that at every 600 frames Compressor is going to have to make a harder choice. So I guess my answer is 1800. Let's see:

frame 1800 of new movie * (1/23.98fps) = 75.0626 seconds into the movie
75.0626 seconds into the old movie * 24 fps = frame 1801.5

Yes - so what should Compressor do now? What's the better choice for frame 1800 of the new movie? Frame 1801 or 1802? No matter how you pick, you can probably see that somewhere between frames 1800, 1801, and 1802 of the new movie we're going to skip a frame of the old movie.

This is good: it keeps the picture in sync with the sound. It is the method by which we periodically drop frames in order to keep the sync throughout.

It is also bad: if the frame dropped just happens to be in the middle of a smooth camera and/or object motion, you will perceive a jump. When I first did this with Tower 37, I noticed about 4 jumps. The other dropped frames happened to occur where the motion was so subtle I didn't catch it.

The other Compressor options don't please me. They end up making new frames by averaging or blending or warping old frames, which is slow to do and/or looks bad. Cripes, I'm looking to lose 12 frames out of 15000, I don't need to do much.

So what did I do?

I dragged two versions of Tower 37 into Final Cut Pro, the version that has every frame but plays at 23.98 fps (and therefore runs long), and the 23.98 fps version that Compressor made which syncs to the sound but drops frames in bad places.

I made the longer version partially transparent and walked through the film, finding (every 600 frames or so - of course) where a frame had been dropped to make the shorter version.

What I did was instead of cutting out a frame at that spot, I'd find the next nearest edit point and drop a frame there. This seems so easy to do I cannot believe it's not a default option in Compressor. People have been detecting scene breaks for a long time now, it should be a fairly simple algorithm to connect the two and I suspect others are already doing this. Please comment if you know of a publication that mentions this approach.

So I spent the better part of an evening walking in 600-frame jumps through the movie, chopping out frames from scene breaks until I had eliminated exactly enough to keep the audio synced. I wrote this while waiting for the final movie to export!

Fun, wasn't it?

a few key phrases to guide curious web searchers here:
conforming 24 fps to 23.98 fps
conforming 24 fps to 23.976 fps
changing frame rate from 24 to 23.98
changing frame rate from 24 to 23.976