Friday, February 26, 2010

Making a DCP entirely with open source tools


this technique is now obsolete - try the latest method instead!


Here are the steps I followed to produce a DCP for the 11-minute short film The Incident at Tower 37. It has ingested and played properly on a Doremi DCP-2000. The projector was a Barco model. Everything below was done on a Linux machine.

Credit where it is due: as I mentioned in an earlier post, I'm building off the work of the people behind asdcplib and OpenCinemaTools, and the users on the Red user forums who have published software and instructions that they have used/created while traversing this bumpy path. Big thanks to Wolfgang, Pavel, Marc, Samson, and Josiah.

In the interest of brevity, I'm not going to explain why I chose program X over program Y -- those interested can read my previous posts where I go into more of those details. Also, I'm not trying to present the be-all, end-all DCP manual here; this is just what worked for me!

Step 0: prepare sources

My source video was a 24fps lossless QuickTime movie, 1920 x 1080 (16:9). For the purposes of this document, call it INMOVIE.mov.

My source audio comprised of six wav files for Dolby 5.1 surround, each 48kHz mono.

Step 1: export PNGs from source QuickTime movie

Tool: ffmpeg
Version: SVN-r21770 (compiled locally)
Command:

ffmpeg -r 24 -i INMOVIE.mov -f image2 %08d.png
This marches through the video file outputting a lossless, compressed PNG 24 times per second.

Step 2: convert to 12 bit per channel XYZ color images
Tool: ImageMagick's convert
Version: 6.5.9 (compiled locally)
Must be run once per PNG frame.
Command:

convert FRAME.png -type TrueColor -alpha Off \
-background black -extent 2048x1080-64 -depth 12 \
-gamma 0.454545 \
-recolor "0.4124564 0.3575761 0.18
04375 \
0.2126729 0.7151522 0.0721750 \
0.0193339 0.1191920 0.9503041" \
-gamma 2.6 FRAME.tif
This takes a 1920 x 1080 PNG from step 1, removes any alpha, pads the image with 64 pixels of black on each side so that it becomes 2048 x 1080, sets the bit depth to 12 bits per pixel, and does a linear-space color shift from RGB->XYZ. Then it writes the image out as a TIFF.

Upon further reading, I interrogated my own decision to force the 1920 x 1080 images into a 2048 x 1080 container by padding with black. Apparently, you only need one dimension of the source imagery to match either 2048 or 1080 and the OpenJPEG encoder will do the right thing.

The new conversion line I used today omits the padding and looked stellar at the theater just moments ago:
convert FRAME.png -type TrueColor -alpha Off -depth 12 \
-gamma 0.454545 \
-recolor "0.4124564 0.3575761 0.18
04375 \
0.2126729 0.7151522 0.0721750 \
0.0193339 0.1191920 0.9503041" \
-gamma 2.6 FRAME.tif
This must be run one frame at a time, but it's very easy to distribute ranges of frames to multiple processors if you have them.

Step 3: convert TIFFs to JPEG-2000
Tool: OpenJPEG's image_to_j2k
Version: 2.1.3.0(compiled locally from subversion revision 543)
Must be run once per TIFF frame.
Command:
image_to_j2k -cinema2K 24 -i FRAME.tif -o FRAME.j2c
This takes longer per frame than convert, in my experience. About 2x longer. Also easy to distribute to many machines, however.

Step 4: make all the necessary DCP files
Tools: OpenCinemaTools
Version: 1.1.2 (compiled locally)

This step has five substeps, each of which generates one or more of the files needed for the DCP. Naming conventions are very important in DCP land, so I highly recommend that you read about them yourself and make informed choices. When you see DCPFULLNAME below, that's shorthand for what I used, which was the mouthful:
TOWER37_FTR_F_EN-XX_US-XX_51_2K_ST_20100212_OV
Command A (create a video MXF file from the folder of J2K files):
asdcptest -v -L -c PROJ.video.mxf FOLDER_OF_J2KS
Command B (create an audio MXF file from the six wavs):
asdcptest -v -L -c PROJ.audio.mxf left.wav right.wav \
center.wav sub.wav surrLeft.wav surrRight.wav
Command C (create an XML composition playlist):
mkcpl --kind feature --title DCPFULLNAME \
--annotation DCPFULLNAME --norating \
PROJ.video.mxf
PROJ.audio.mxf > PROJ.cpl.xml
Do check the output of command C and confirm that the durations of both video and audio are identical. The XML tag for this is "IntrinsicDuration."
Command D (create an XML packing list):

mkpkl --issuer BitFilms --annotation DCPFULLNAME \ PROJ.video.mxf PROJ.audio.mxf PROJ.cpl.xml > PROJ.pkl.xml
Command E (create the ASSETMAP and VOLINDEX XML files):
mkmap --issuer BitFilms \PROJ.video.mxf PROJ.audio.mxf PROJ.cpl.xml PROJ.pkl.xml

I used a bit of shorthand above since I did not identify enclosing folders. I suggest that if you make a DCP this way, you use temporary folders for all the inputs (and the results of steps 1-3), then create a finalDCP folder which you populate with only the files that will be part of the eventual DCP itself (namely, the 2 mxfs and the 4 XML files that come from step 4).

Step 5: Get the DCP onto a Hard Drive
Tools: regular linux shell

I need to get the specific instructions from Josiah, but basically we formatted a UDF disk solely for the purpose of holding the DCP (partition type 0x83 if I remember correctly). Then I copied over the 6 files that emerged from step 4 right to the root of the new disk. And it's done!

Some quick notes:
  • It is important that you check your XML files to make sure the file paths don't contain any incorrect information: if you aren't careful during step 4, your file paths in the XML may be incorrect! Here's what one of my entries looks like in my final ASSETMAP.xml file as an example: file:///tower37_complete3.video.mxf
  • I don't know what happens if your audio and video durations are different, so I was careful to make sure mine were not. Check them in the CPL (the tag is ).
  • Work with a small movie file for starters, a minute or less, until you get the steps ironed out. Then move to your full film.
I hope this is useful for some of you! If you see any mistakes above, or have suggestions to improve or generalize this pipeline, please let me know.

32 comments:

Photo Blog said...

Hi Chris,
Thanks for the great blog. It's a shame it's so hard to make an open source DCP, should be easier. We followed you steps and got some issues with bitrate. It seems like the files we are getting have a really high bitrate, definitely not under 125 Mbps. Any idea how we can control the bitrate during the process?

Thanks in advance,

Ariel

Chris Perry said...

Hi Ariel,
Where did you get your 125 Mbps number from? I was unaware of that as some kind of limit (and the spec seems to hint at average bit rates ranging from 80-250 in table 9, section 7.5.3.7).

Looking at the openjpeg dox (http://www.openjpeg.org/index.php?menu=doc#encoder), it seems that the Cinema2K profile by default limits the bit rate to approximately 1.3Mb per frame. That's just under 250Mbits/sec. I'm not sure why that would be the standard for Cinema2K if 250Mbits/sec were too high.

Let me know more about your 125Mbps number and I'll see if I can come up with some answers for you.

- chris

Photo Blog said...

Hi,
I forgot to mention we are working on a 3D stereoscopic DCP. So each eye can't exceed a bitrate of 125 Mbps.
Any idea how to lower the bitrate?

Thanks,

Ariel

Chris Perry said...

This might sound hacky, but change the -Cinema2K arg on image_to_j2k to 48 instead of 24. This should effectively cut the bitrate in half. Let me know if it works!

- chris

FreFred27 said...

If you are using the DcpMaker.exe GUI for creating a 3D DCP, there is a problem. It tells image_to_j2k.exe to use "-cinema2K 24" instead of "-cinema2K 48". You can fix this in two ways:

1.a.Fill in and start creation in DcpMaker.exe.
b.Stop the process.
c.Open the created file _makedcp.bat in text form (notepad, etc.).
d.Use "find replace" to replace all instances of "-cinema2K 24" with "-cinema2K 48".
e.Run the corrected _makedcp.bat.

or

2.Get the source code and in DcpMakePanel.cpp change the two instaces of "-cinema2K 24" to "-cinema2K 48" and build for your platform. This will create a "3D Only" version of DcpMaker.exe. You will still need to tick the 3D option in the GUI.

Benjamin said...

> I don't know what happens if your audio and video durations are different,

It's simple: some players don't start movie ;-)

paul said...

MacSoftBlog / Ariel.
We're in the 3D space ourselves. We need to make DCP ourselves from time to time. Did you have any success ?
Chris - I'm not a big blogger guy but I would love to get hold of Ariel to ask her some questions about this.
paul@venture3d.com
Thanks

Martin Bonnici said...

Hi Chris,
Thanks for posting so much information about your process. I'm trying to create a DCP for a project of mine, and whilst following your steps I got into some errors.

First, when encoding using TIFFs, I get an error saying that there is a problem with the 8-strip decoding, and the resulting J2Cs are just black.

So I switched to using PPMs, but the J2Cs turn out a mess, just a corner of the actual image is visible (although it keeps the resolution) and it's lost all colour data and full of noise.

Had you come across these problems? Do you know what could be causing them?

Thanks again for posting all this information and I'm appreciate it if you could help me out.

Martin

Chris Perry said...

Martin - I haven't personally encountered your problem before, but I would start by looking more deeply at those TIFFs. How did you generate them? Can you run tiffinfo on one and diagnose what might be wrong?

Let me know how I can help,

- chris

Martin Bonnici said...

Hi Chris,
thanks for your reply. I tried different methods to create the tiffs. I tried rendering them out from After Effects and I also tried using the code your provided here, got an error both times. I'll look into diagnosing with tiffinfo, but the tiffs (and the ppms) both load fine in any player or software I open them in. And again, I'm using the code you used to generate the J2Cs....

Chris Perry said...

Martin - In the method I posted, I used TIFFs only in step 2. Is that where you're having difficulties, too? Or are you trying to use TIFFs further upstream in the process? If the latter, what format is your source movie in and what tools are you using for step 1?

Don't forget, the code I used to generate the j2c images relies on a very specific format of TIFF image, so if you've skipped any steps I will have a harder time helping you debug the problem. Also, your claim that the TIFFs look good in other packages has me nervous, since if they have the proper RGB->XYZ colorspace conversion, they will look incorrect (color-wise) on a tool that is expecting RGB color.

Good luck,

- chris

Martin Bonnici said...

Hi Chris,
I've tried it a number of ways. But saying that they look good I meant they looked like what I would expect from the X'Y'Z' colour space.

When using your code step by step, I get an error as the TIFF to J2C routine starts operating saying "LibTiff: 8 Strip Scheme Compression not implemented". And the J2Cs turn out black.

That's why I tried using PPMs instead, it doesn't give any errors, but the image just turns out being junk.

Would it be possible for you to send me a couple of your TIFFs (even one would do) so that I see if I get an error on that as well and compare it with mine using TiffInfo?

Martin

Chris Perry said...

Martin - get in touch directly with me and I can email you an image, and we can take this conversation offline. Thanks.

- chris

Daniel said...

Hey, is there any solution for the "8 Strip Scheme Compression not implemented"-problem. I did the same way, which was posted.

It would be nice.
Thanks.

Daniel

Daniel said...

For those who want to know how: The 8 Strip Scheme Compression not implemented problem.

at Step 2: there should be the parameter -compress None.
Because 8 = Deflate ('Adobe-style') is not supported at J2K. So I turned off the compression and it works.

Chris Eller said...

Chris,

Thanks so much for doing the grunt work and gathering all of the knowledge together to write up a concise method of DCP creation. The commercial packaging systems, easyDCP Creator+ in my case, are very nice and elegant, but this harnesses other resources and is very appealing for a variety of reasons.

Unknown said...

Hi to all,
I've been following forums related to this. I followed the steps here but my DCP won't ingest on DC20 server. I managed to ingest it on a Dolby DSS200 but it says file corrupted when tried to play it.

And also, I'm having trouble compiling the newest build of asdcplib. Can anyone upload the newest binaries for win32.

Thanks in advance.

Unknown said...

Yay you guys got it :)

jamiegau said...

ALso, after you have the DCP, you can use dcpPlayer at www.dcpPlayer.net
The Demo version will let you, with a lot of annoyance until you purchase it, test the file.
Also better the the easyDcp Player, in demo mode, will do the XYZ to RGB, let you see the full DCP, not only the fist 15seconds, AND does 3D playback (converting it to common 3D formats.)

Would really add the final key to making a DCP for free.
Ie making, then being able to TEST it ;)

James

@flipsideza said...

Hi Chris,

Are you aware or know a way to allow the mkcpl & mkpkl to create cpl and pkl files for interlop stereoscopic footage?

With modern hardware it seems more viable to create interlop instead of SMPTE

Regards,
Chris

Chris Perry said...

Chris - I think that the interop vs SMPTE choice happens at the asdcplib level. Check out the documentation for asdcp-test.

http://www.cinecert.com/asdcplib/asdcp-test.tt2

Good luck

- chris

Toby T said...

Hi Chris,

Cheers for this. I think I've successfully made a DCP, I'm testing it tomorrow at a cinema on a Doremi server.

I deviated somewhat from your guide to get this all to work in OS X. The only think I couldn't get to work in OS X was mkcpl. Also, opposed to using OpenCinemaTools to make the mxf files I used asdcp-test from http://www.cinecert.com/asdcplib/ as I was having diffculty compiling OpenCinemaTools at all on OS 10.6.

My friend Chris also combined steps 2 and 3, deleting the TIF's as they are converted to J2C, which really helps with the storage problem of 1000's of Tif's. We also paralleled this step, and on an 3.8GHz i7 and 7 threads we were getting 1fps on the PNG>J2K conversion (-; The ffmpeg process to PNG can also be sped up if you split the video into multiple files (one for each thread) and then batch rename the PNG's to preserve the sequential naming.

I'm trying the DCP tomorrow on a HFS formatted USB stick, as I couldn't find a way to format it to UDF or EXT2 on OS X or Windows 7. Any ideas? I think it will work with the server tomorrow but I would like to have more guaranteed compatibility in the future of course.

Thanks again! Its been a tough few days but I think I've cracked it (It play well in EasyDCP Player at least)...but tomorrow will tell!

Hope you're well,

Toby

Chris Perry said...

Toby - glad to have your input on the process!

But I'm worried about your memory stick. I think the UDF thing is pretty important.

I was able to do some rudimentary UDF stuff on OS X with hdiutil and mount_udf. I don't have time now to post explicit instructions, but see if you get anywhere with those and keep us all posted!

- chris

Chris Perry said...

Quick update:

I, too (like poster Vincent) had troubles with the DCP showing as corrupt on a Dolby DSS-100.

I'm looking into it and will post anything I learn about the error or what to do about it.

- chris

Toby T said...

It worked! (-;

Quick question though- What sound level should the wav's be at (what should be peak etc?)?

Thanks again for everything,

Toby

Cristian Bote said...

Hello Chris,

You have a nice tutorial on doing DCPs, congrats!
Anyway, I have a problem. The thing is that converted the RGB TIFF with the code you've mentioned but the output it looks bad - not only on the computer but also on the screen.

It looks like the color depth is very low. I made a color chart(separate gradients for primary colors) and it doesn't look good.
What is odd is the fact that the non-converted DCP looks perfect on screen.

Any suggestions would be very welcomed.

PS: I've tested the DCP on Dolby DSS200

Chris Perry said...

Cristian -
How did you generate your TIFFs? Have you used tiffinfo and checked that they are RGB, a good bit depth, etc? Also, how did you make the color chart? If you can send me some of those images I can try to look at them for you. Contact me off of this thread and we can discuss.

- chris

Anonymous said...

Dear Chris,

I run a small independent film festival and need to convert all the amateur films submitted in various formats into DCP so we can play them in a nice big commercial theater that only accepts DCP on Barco 2k projectors.

Do you do any consulting or have an email address I can contact you?

What version of Linux and hardware do you have? How do you get the converted DCP onto the right hard drive that we can just plug into the theater system? How long does it take to convert a 1.5 hour HD video?

We want to be able to go between our film festival TITLE on the screen then some credits then commercials then to the DCP film, then back to the title and repeat this cycle throughout the film festival.

Any help is appreciated since I've been quoted minimum $3,000 to create a DCP feature length.

dasca said...

Hi chris,
Thanks so much for your amazing knowledge!
i have a question, it seem you mention that the tiff file converted using the script cant be viewed normally?
you are correct, but i think the finished dcp file is also isnt in the right color as in the PNGs.
is it because your color preference?
is it possible that i omit the -recolor script?

Chris Perry said...

Dasca - you need to keep the recolor because the DCP expects XYZ colors, not RGB. If you use a TIFF viewer that doesn't recognize the colors are in XYZ, it will display them as RGB and they will look wrong. Anything you have to view DCPs should (hopefully!) recognize the XYZ colors and transform/display them correctly.

- chris

Henry Huberts said...

Hi Chris,
thanks for your blog
do you have an Idea how to extract (or unwrap) the content from a dcp?
I have a 3d trailer (without encription) and I would like to make a 3D movie out of it, but looks like all the tools to extract form the dcp are very expansive.
any suggestion?
thank you very much... ;)

Henry

Chris Perry said...

Henry - I don't have experience extracting content from DCPs. Sorry!

And since this is now a very old thread (and I want people to see the updated details), I'm going to turn off commenting for this post.

Go here instead for updated DCP info.