Saturday, November 20, 2010

Making a DCP entirely with open source tools (update)

Here I detail my most up-to-date pipeline for producing a DCP with open source tools. I have used these techniques to produce a DCP for The Incident at Tower 37 that has (to date) ingested and played properly on Doremi and Dolby servers. This technique is an updated version of the original pipeline I posted earlier this year, but with many small but important changes (mostly related to audio and proper formatting of the hard drive).

Step 0: prepare sources

My source video was a 24fps lossless QuickTime movie, RGB color, 1920 x 1080 (16:9). For the purposes of this document, call it INMOVIE.mov.

My source audio comprised of six wav files for Dolby 5.1 surround, each 48kHz mono and 24 bits per sample. This bit depth is critical! Dolby DSS servers considered the DCP corrupt until the audio sources were 24 bit.

Step 1: export PNGs from source QuickTime movie

Tool: ffmpeg
Version: SVN-r21770 (compiled locally)
Command:

ffmpeg -r 24 -i INMOVIE.mov -f image2 %08d.png
This marches through the video file outputting a lossless, compressed PNG 24 times per second.

Step 2: convert to 12 bit per channel XYZ color imagesTool: ImageMagick's convert
Version: 6.5.9 (compiled locally)
Must be run once per PNG frame.
Command:
convert FRAME.png -type TrueColor -alpha Off -depth 12 \
-gamma 0.454545 \
-recolor "0.4124564 0.3575761 0.18
04375 \
0.2126729 0.7151522 0.0721750 \
0.0193339 0.1191920 0.9503041" \
-gamma 2.6 FRAME.tif
This takes a 1920 x 1080 PNG from step 1, removes any alpha, sets the bit depth to 12 bits per pixel, and does a linear-space color shift from RGB->XYZ. Then it writes the image out as a TIFF. According to a comment I saw here, only one of the dimensions of the image need to match 2048 x 1080.

Step 3: convert TIFFs to JPEG-2000Tool: OpenJPEG's image_to_j2k
Version: 2.1.3.0(compiled locally from subversion revision 543)
Must be run once per TIFF frame.
Command:
image_to_j2k -cinema2K 24 -i FRAME.tif -o FRAME.j2c
This takes longer per frame than convert, in my experience. About 2x longer.

Step 4: make all the necessary DCP filesTool: OpenCinemaTools
Version: 1.1.2 (compiled locally)
Tool: asdcplib
Version: 1.6.37 (compiled locally)

NOTE: the next time I revisit this process I'm going to try using the new openDCP library instead of OpenCinemaTools, which is not actively supported and was quite challenging to build.

This step has five substeps, each of which generates one or more of the files needed for the DCP. Naming conventions are very important in DCP land, so I highly recommend that you read about them yourself and make informed choices. When you see DCPFULLNAME below, that's shorthand for what I used, which was the mouthful:
TOWER37_SHR_F_EN-XX_US-XX_51_2K_ST_20101105_OV
Command A (create a video MXF file from the folder of J2K files):
asdcptest -v -E -L -c PROJ.video.mxf FOLDER_OF_J2KS
Command B (create an audio MXF file from the six wavs):
asdcptest -v -l 5.1 -E -L -c PROJ.audio.mxf \
left.wav right.wav center.wav \
sub.wav surrLeft.wav surrRight.wav
Command C (create an XML composition playlist):
mkcpl --kind short --title DCPFULLNAME \
--annotation DCPFULLNAME --norating \
PROJ.video.mxf
PROJ.audio.mxf > PROJ.cpl.xml
Do check the output of command C and confirm that the durations of both video and audio are identical. The XML tag for this is "IntrinsicDuration."
Command D (create an XML packing list):

mkpkl --issuer BitFilms --annotation DCPFULLNAME \ PROJ.video.mxf PROJ.audio.mxf PROJ.cpl.xml > PROJ.pkl.xml
Command E (create the ASSETMAP and VOLINDEX XML files):
mkmap --issuer BitFilms \PROJ.video.mxf PROJ.audio.mxf PROJ.cpl.xml PROJ.pkl.xml
I used a bit of shorthand above since I did not identify enclosing folders. I suggest that if you make a DCP this way, you use temporary folders for all the inputs (and the results of steps 1-3), then create a finalDCP folder which you populate with only the files that will be part of the eventual DCP itself (namely, the 2 mxfs and the 4 XML files that come from step 4).

Step 5: get the DCP onto a Hard DriveTool: gparted
Version: 0.7.0-4 "live" CD image

I found the best guidance for this step at the Inter-Society Digital Cinema Forum website.

In short, you want an MBR partition table and an EXT3 file system. I booted gparted from the live CD and made a single partition on my portable USB hard drive. NOTE: I tried an HP SimpleSave drive and regretted it. This drive comes with an uneditable partition that mounts like a read-only CD-ROM in addition to its actual storage. My Lacie Rakiki worked great.

Once the disk is formatted, just copy over the 6 files that emerged from step 4 right to the root of the new disk. And it's done.

Some quick notes:
  • It is important that you check your XML files to make sure the file paths don't contain any incorrect information: if you aren't careful during step 4, your file paths in the XML may be incorrect! Here's what one of my entries looks like in my final ASSETMAP.xml file as an example: file:///tower37fullDCP.video.mxf
  • I don't know what happens if your audio and video durations are different, so I was careful to make sure mine were not. Check them in the CPL (the tag is ).
  • Work with a small movie file for starters, a minute or less, until you get the steps ironed out. Then move to your full film.
Good luck!

Thanks:
Wolfgang Woehl
John Hurst
Dave Samson
Bryan Dennis

24 comments:

Andy said...

very good tutorial this was what got me going on my own DCP project but I've been having trouble with pixel formats, any chance you could post the output of ffmpeg -i (your input video)? would really help. Also I believe the -l trigger on asdcptest is new? what's it do?

Chris Perry said...

Hi Andy.

Here is the ffmpeg -i output on my source video. I had used the lossless PNG codec to make the master:

FFmpeg version SVN-r21770, Copyright (c) 2000-2010 Fabrice Bellard, et al.
built on Feb 12 2010 10:19:41 with gcc 4.1.2 20071124 (Red Hat 4.1.2-42)
configuration:
libavutil 50. 9. 0 / 50. 9. 0
libavcodec 52.54. 0 / 52.54. 0
libavformat 52.52. 0 / 52.52. 0
libavdevice 52. 2. 0 / 52. 2. 0
libswscale 0.10. 0 / 0.10. 0

Seems stream 0 codec frame rate differs from container frame rate: 2400.00 (2400/1) -> 24.00 (24/1)
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/PATH/tower37_24_PNG_picture_2pops.mov':
Metadata:
major_brand : qt
minor_version : 537199360
compatible_brands: qt
Duration: 00:10:56.45, start: 0.000000, bitrate: 408162 kb/s
Stream #0.0(eng): Video: png, rgb24, 1920x1080, 408159 kb/s, 24 fps, 24 tbr, 2400 tbn, 2400 tbc

The -l trigger on asdcp-test is new. I haven't found documentation online for it yet, but if you type asdcp-test -help you will see the following:

-l label:

Use given channel format label when writing MXF sound files. SMPTE 429-2 labels: '5.1', '6.1', '7.1'. Default is no label (valid for Interop only).

- chris

Anthurium said...

Awesome job. Thanks for the detailed tutorial. ffmpeg can be frustrating for people (like me) who don't know what they're doing.

Unknown said...

Hey Chris,

I'm not sure if you are using OSX or not, but I wanted to point out a software package that allows you to read/write EXT3 disks directly from OSX. Its called Paragon ExtFS and it is $40, but it has been working really well for me so far. I had been running Linux under Vmware, but this is much better.

http://www.paragon-software.com/home/extfs-mac/

Andy said...

Thanks for the ffmpeg info Chris it turned out to be very useful when you know a reference point. One thing though, I find my DCPs are washed blue when on screen until I run -recolor "1.0478112 0.0228866 -0.0501270 0.0295424 0.9904844 -0.0170491 -0.0092345 0.0150436 0.7521316" after the first recolor, did you experience the same problem? that recolor converts the whitespace from d65 (computer screen) to d50 (closer to xenon). Im testing on DCP2000 and type2 NECs.

Rodney Recor said...

Hello, Chris.
Thank you for this work. It is oustanding. Please permit me to clarify the statement in Step 2, "...only one of the dimensions of the image need to match 2048 x 1080."

When preparing a 2k(2048) DCP, if the dimensions of the tiffs (and pngs) are 1998x1080, the aspect ratio will be 1.85 or 1.77 and, if the dimensions of the tiffs (and pngs) are 2048x858, the aspect ratio will be 2.39.

Should you be making a 4k(4096) DCP, the tiffs (and pngs) should be 3996x2160 or 4096x1716.

(Reference section 2.4.2.3 of the DCI.) "The PNG file is required to be mastered at the same resolution as the DCDM".

Chris Perry said...

Rodney - I'm not sure your comment is 100% accurate.

First of all, there is no 2.4.2.3 -- you must have meant section 3.4.2.3. But this is the Subpicture section of the spec, not the Image section.

In the image section (3.2.1.2 Image Structure) it says:

The maximum number of horizontal and vertical pixels shall be constrained to fit within one of the following image pixel arrays: The number of active pixels (e.g., the pixels of the image structure that are intended to be displayed) shall extend to the maximum in either the horizontal or vertical direction of the defined level of operation as shown in Table 1.

(And table 1 shows 2048 and 1080 for 2K, 4096 and 2160 for 4K)

Can you please elaborate if I'm missing something?

Thanks,

- chris

Chris Perry said...

Andy - I did not notice the blue tinting that you saw. I will keep my eyes open for it, thanks for the heads up.

- chris

Coeur Noir said...

Hi... I'm also trying to create DCP fot the theatre I'm working for (in France).

I'm not very used in command line - and my english is poor.

I tried this program : DCP Creator you can reach here http://cinema.terminal-entry.de/
wich does the same through a gui on winXP.

But ever since I try to convert a 3 minutes duration film, I get a system error after all images have been converted into jpeg2000.

So I'm looking for other way making it !

What is your system/OS/hardware specs ?

___


As far as I know, DCI recommendations accept 3 image ratio :
FLAT 1998x1080 px
SCOPE 2048x858 px
HD 1920x10820 px
I don't know what happens if other ratio are used through DCI projectors...

Chris Perry said...

All the above was done on a Linux box, specifically: a Visionman 1U fileserver running Ubuntu 8.04LTS server.

- chris

Unknown said...

Hello Chris. I followed your instructions on creating DCP. Got positive result. So thanks for that a lot. The only difference was that I converted tiff files instead of png.
The problem is in X'Y'Z' color space convertion. The red color seems to be dissapiared. I am not shure what the exact problem is. I read in some article that it is OK. But when I watch the video in easyDCP player or stereoscopic player red color is still gone.
Can you give some advice, what should I do?

Chris Perry said...

ghostas -

I'd be nervous about the Red channel vanishing. If you'd like, you can email me one of your frames and I'll take a look. Best would be to send the original PNG and then the J2K as well so I can study them both.

- chris

Unknown said...

Hello again Chris. Today I played the video on Doremi server. And it played just fine. All the colors were at their places including red channel. So the problem was only in my copy of stereoscopic player probably. But thanks for respond anyway.

- Stas

Anonymous said...

Hey,

Does anyone know of a free (as in beer or speach) DCP player for linux?

Cheers,

Andrew

Coeur Noir said...

Well using different information sources - yours amongst - I managed to create a dcp with ubuntu 10.04

Many thanks for sharing your experience.

Hence this french tutorial very inspired by you and aproche, member of forum du repaire :

http://www.projectionniste.net/forums/viewtopic.php?f=7&t=3247&p=55458#p55458

yellow said...

Chris, very interesting read. I'd just like to comment that for anyone considering converting YCbCr video to .png's that they don't use ffmpeg, it stretches levels and poor interpolation.

I currently use a free 3D LUT based conversion from YCbCr to RGB called yCMS combined with AVISynth. Explained a little on my blog here: www.blendervse.wordpress.com.

I'm getting xvYCC from Canon DSLR video but currently struggling to get it into an image format with a wide gamut and am looking to Imagemagick for a solution.

I think it's probably possible to use yCMS to go from YCbCr to whatever DCP requires in one conversion?

Would just need yCMS author to implement the LUT?

First steps for me on this road so a lot to learn. :-)

Anonymous said...

I run a small independent film festival and would like to do everything in DCP.

Questions:

1) does this work if you have a video and audio combined in one mov or avi file? if not how do you separate the files?

2) does this work if you only have 2 channel audio? many independent films aren't 6 channel.

3) does this work for all pixel format sources, i.e. does this work if your source is 720x480 (or any other pixel format other than an HD format) or other format that is smaller than 1080 pixels?

Do you have a way I can email you?

Chris Perry said...

Answers to the most recently-asked questions:

1) You need to extract the audio into separate channel wav files. The tools depend on your platform. I sometimes use QuickTime Player for this. I'll ask around as to the best open source option (maybe ffmpeg and/or mplayer?).

2) I think the first two channels of audio are always L and R, so my guess is that you can provide those two and only those two and still get things to work. But I don't have the DCP spec in front of me so I can't say for sure.

3) You would have to conform your images to one of the DCP-accepted resolutions. But that should be easy no matter what pipeline you follow.

If you want to email, try info@bitfilms.com.

Loïc said...

Hi,

Thank you so much Chris for that amazing tutorial.

I run my opensource DCPs on a Doremi DCP-2000 server for a year now and it's working really well :-).

I was wondering if from your experience your DCPs are compatible with all the cinema server (Dolby etc...) ?

Many Thanks

-Loïc

Chris Perry said...

As I wrote at the top of the post, I've had DCPs made this way play successfully on Dolby and Doremi servers. I think we had a test run successfully on an XDC Cinestore Solo, too. Hope that helps.

- chris

Coeur Noir said...

As a result of some months of testings, here is a place to share some tips about DCP :

http://home-made-dcp.over-blog.fr

There are some command lines for FFMPEG, ImageMagick and OpenDCP and also step-by-step for Audacity and DCP Creator...

It's a work in progress but what's on this blog has been tested on doremi dcp2000 and christie projectors cp2000X, cp2000ZX and cp2210.

Errr it's in french, sorry !

Unknown said...

http://converseen.sourceforge.net/index.php

GUI for imagemagick batch processing. might help.

Carlos Vidal said...

CHRIS AMAZINGGGG TUTORIAL! Very useful!!!!

I've new commands for batch processing folders:

ALL IMAGES IN FOLDER TO TIFF XYZ (warning this command write the originals!):
mogrify -type TrueColor -alpha Off -depth 12 \-gamma 0.454545 \-color-matrix "0.4124564 0.3575761 0.1804375 0.2126729 0.7151522 0.0721750 0.0193339 0.1191920 0.9503041" \-gamma 2.6 *.tif

ALL IMAGES IN FOLDER TO JP2:
image_to_j2k -cinema2K 24 -ImgDir . -OutFor j2k

Thanks for your help!
Carlos (from Buenos Aires, Argentine)

Anonymous said...

I've been having a very hard time with my DCPs. Sources are an HD 23.976 Avid project on my Mac Pro and 6 mono tracks for my 5.1 mix. I'm using QubeMaster Xport plug in for compressor and Paragon ExtFS to get the external drives formatted at EXT2 and copy the DCP package folder over.
I'm not sure why, but doremi servers are having problems ingesting the DCP. It plays fine on Dolby servers, but just won't ingest on doremi it seems like. I tried various tests, also demo versions of EasyDCP player and doremi's cineplayer - both have no problem reading the original encoded DCP, but once it's copied to the external drive, it's very touch and go. Ideas, anyone?