Lacq’ing in Depth

The Levels node in Conduit is basically equivalent to the Levels adjustment in Photoshop, so it’s easy to think of it as an image operation foremost. Yet Levels actually performs a very general-purpose math operation that can come in handy in many situations.

Consider an effect that takes a value from the user using the Slider node. The slider’s value is usually in the range 0 to 1 (what is often called the “standard range” in Conduit because it’s the range for visible pixel values). What if we want this slider to drive a value that needs a larger range — for example, a 2D Transform node? If we want to use this slider to move the image up to 100 pixels in either direction, we’ll need to change the slider’s input value’s range from [0, 1] to [-100, 100].

This could of course be accomplished by using a few basic math nodes:

But that looks rather unweildy for such a simple operation… So let’s use Levels instead:

See what happens in Levels? Input black and input white specify the original range (in this case, 0 to 1). Output black and output white specify the output range. This performs exactly the same math as the Multiply+Subtract nodes in the first screenshot.

Don’t forget that the Clip to 0-1 setting is enabled by default when a Levels node is created, so you need to uncheck that box. (If you don’t, Levels will clip all negative values to zero and all >1 values to one.)

The Mini Converters by BlackMagic Design have been around for a few months, but I don’t think I’ve written about them before. BlackMagic is the company behind the popular DeckLink capture boards which are a very good option if you need to get an SDI signal into Conduit Live affordably. Their new HDMI->SDI mini converters seems to be a nice solution for the other end of the computer’s video pipeline.

Until now DVI->SDI converters have cost something in the range of several thousand USD, making them out of reach for many people building a Conduit Live setup on a budget. BlackMagic’s converter box is only $495. Unless your video card has HDMI output, you’ll need a DVI->HDMI converter on top of that, but luckily that’s a very common piece of electronics you can buy in any computer store (DVI and HDMI are designed to be compatible without expensive signal conversion acrobatics).

Caveat lector: I haven’t tried the BlackMagic box myself. But if you do, I’d be intrigued to hear whether it’s as good as it seems!

As you may know from having read the Conduit Live manual (page 13 onwards), Conduit Live supports two capture modes: “raw” and “rendered”. Superficially they both do the same thing: both end up writing a QuickTime movie to disk, after all. But that’s pretty much where the similarity ends, in terms of implementation and performance.

In this post I’d like to explain in some more depth why exactly “rendered capture” is so much more performance-intensive. I’m also going to show the alternative: how to capture raw and render the final result from Conduit Live, without switching apps and with a reasonable overhead.

An essential part of Conduit Live’s capture performance is the involvement of QuickTime. As I wrote in my previous post, QuickTime is not just a media player, but a sprawling API that aims to provide a solution to all your media processing needs. Of course it’s impossible to satisfy everyone, but there are a lot of things that QuickTime does really well (partly because it’s been around for so long that its implementation has matured).

One of those things is that it handles the entire process of writing a captured video stream to disk as a .mov file. As long as Conduit Live doesn’t need to touch the video data, it can rely on QuickTime to do the disk writing in the background while Conduit Live receives a copy of the video data for its own display needs – it’s efficient multitasking. Here’s an illustration of the process:

So that’s for the “raw capture” mode. For “rendered capture”, we can’t rely on QuickTime on handling the diskbound video data behind the scenes anymore. Instead we need to have a compression connection and keep feeding the .mov file ourselves. To further complicate matters, the rendered image resides on the GPU, so before it can be compressed and written to disk, it needs to be pulled back to main memory. Here’s the illustration — comparing with the previous image, it’s easy to see why this mode introduces more performance bottlenecks:

Because the image needs to be recompressed, you must choose the QuickTime compression settings before capture can proceed. This is why Conduit Live pops up the QuickTime settings dialog when starting rendered capture, if you haven’t yet specified any compression settings. (A lot of QuickTime codecs are not suitable for this kind of realtime compression – please see manual page 14 for more information and some suggestions of codecs to use.)

So if rendered capture turns out to be too heavy, what can you do? One solution is to consider the whole pipeline: usually you’re going to want to edit the footage anyway; maybe add some graphics and so on. Wouldn’t it be nice to retain the ability to finetune the effect you made in Conduit later, instead of setting it in stone while capturing?

The Conduit plugins make this possible. When editing in Final Cut Pro, import the footage you captured in Conduit Live and then use the Conduit filter to apply the effect(s) used during capture. The Conduit filter offers the same node-based user interface as Conduit Live, so you can tweak everything at this stage before exporting the final result from FCP. (Conduit also works in Apple Motion and Adobe After Effects.)

If you don’t need to edit but would like to quickly render out the composited result, you can do it easily inside Conduit Live. Here’s how…

When you’re finished capturing, go to the Control Panel and switch the Source image input from “Live video” to “Image/movie”. Load the video file that you just captured:

Now you’ve got the freshly captured video in Conduit Live in timeline mode, so you can play it with the effect applied, go over the video frame by frame, mark In/Out points to trim it, etc. (If timeline mode doesn’t ring a bell, please read “Working with images” on page 4 in the manual!)

When you’re ready, just export the composite to a QuickTime movie. Conduit Live should render pretty fast, so hopefully this workflow can be practical even in high-pressure environments. If you disagree, I’d love to hear your insights!

Conduit Suite’s latest 1.6.2 update (available for download through your dvGarage account) adds another application to the suite. It’s called Conduit Live for Leopard, and its raison d’être is somewhat complicated: it’s not merely an update to Conduit Live, but more like a new branch that will exist from now on alongside Conduit Live for Tiger – that is, the application that previously was simply “Conduit Live”.

Confused yet? Please bear with me as we take a quick look into Mac operating system internals…

Conduit Live uses Apple’s QuickTime programming interface (abbreviated API) for capturing live video. First released in 1991, QuickTime was a breakthrough in desktop video: it offered a standardized interface for creating, editing and playing all sorts of media formats. Since then, QuickTime has accumulated all sorts of functionality, from video capture to subtitles to live streaming to Flash playback to image filters… It has become such a sprawling and uneven collection of programming interfaces that it would deserve the title “Los Angeles of media APIs”.

As Mac OS X has evolved rapidly, Apple has been trying to evolve QuickTime along with it. The QuickTime API has a lot of quirks and old-fashioned Mac OS Classic weirdness that would deserve to be eliminated in favour of a modern approach. Unfortunately such a modernized QuickTime would ideally need to both offer all the functionality of the old QuickTime and be compatible with the enormous array of software that interfaces with QuickTime on a lower level, e.g. drivers and codecs. (The situation is further complicated by QuickTime’s cross-platform nature: it is widely used on Microsoft Windows, and in fact perhaps the most popular downloadable Windows application, iTunes, relies heavily on it.)

For these reasons, Apple has never embarked on a complete rewrite of QuickTime. Instead the company appears to have decided to introduce the new QuickTime in a piecemeal fashion. Mac OS X 10.5 (a.k.a. Leopard) contains the first new QuickTime functionality that is only available through a brand new interface and only on Mac: a new video capture API that’s officially called “QTKit Capture”.

(The old QuickTime video capture API was called “Sequence Grabber”. Considering that Apple usually gives their APIs names like Quartz, Core Image or Cocoa Touch, I have to guess their marketing department was simply too burdened to come up with more interesting names for video capture…)

The new QTKit Capture API is highly appealing on paper: not only does it offer a simplified way for programmers to access video devices, but it also allows support for the highly compressed formats such as MPEG-2 which is used in HDV and Sony XDCAM recorders. The old Sequence Grabber API was too dated to ever accomodate these formats, so QTKit Capture answers a genuine need in the Mac video community. (Final Cut Pro has previously used private device control APIs to capture HDV/XDCAM video, but these were not made available to 3rd party developers before.)

So that then is the origin of Conduit Live for Leopard: it’s the same application as before, but reworked to use the new QTKit Capture API available on Mac OS X 10.5. This allows Conduit Live to support HDV cameras – I’m really happy to be able to offer this highly requested feature!

Unfortunately the situation is not clear-cut, as the new API doesn’t fully replace the old one. The main problem is that a number of devices (cameras, capture boards, etc.) that were supported under the old Sequence Grabber API don’t work correctly under QTKit Capture. Many other devices do work superficially but their functionality is limited: the new API doesn’t give access to the Settings dialog which may be necessary to change between the card’s inputs, adjust brightness, or modify any other device-specific settings.

For this reason I can’t recommend that everyone start using Conduit Live for Leopard immediately. You should adopt Conduit Live for Leopard only if your capture device is correctly supported by the new API. (It goes without saying that if you want to use a device like HDV which was previously unsupported, then the Leopard version is the only way to go.)

For devices that previously worked fine, I would encourage you to keep using Conduit Live for Tiger even if you’re on Mac OS X 10.5.

In an attempt to clarify the situation, I’m maintaining a device compatibility chart. The URL is:
http://lacquer.fi/conduit/live-compatibility

If you have a video device that’s unlisted, it would be greatly appreciated if you could try it with both versions of Conduit Live and report your findings back to me.

(Still on the topic of naming: I realize the potential for confusion that arises from having a “Leopard” and “Tiger” version even though the Tiger version might be the one you want to use even on Leopard… I just couldn’t come up with better names for these. Clearly, calling them “Conduit Live for QTKit Capture” and “Conduit Live for Sequence Grabber” wouldn’t have helped the situation at all…)

It’s been a busy summer. I’ve been particularly occupied by a bug in the latest releases of Final Cut Pro (6.0.3 and 6.0.4) that causes FCP to ignore the Conduit plugin’s requests when the Canvas window needs to be updated. This makes it look like Conduit isn’t working, when in fact FCP is simply failing to correctly update its windows. (The same update operations do work when Conduit is running in Motion and previous FCP versions, so something has changed in these latest FCP updates.)

Unfortunately the fix for this seemingly simple bug has turned out elusive. A fix doesn’t seem to be forthcoming from Apple, so I’ve spent quite some time trying to come up with a workaround, but have been unable to find a solution that would work for all users. The only solution guaranteed to work was to basically circumvent FCP’s Canvas window entirely by providing a preview inside the Conduit plugin, so that’s what I ended up doing. That new feature will be shipping in the Conduit 1.6.2 update very soon. Hopefully it will provide a relief from FCP 6.0.4 woes to everyone – apologies to all those affected!

With the FCP fix and the 1.6.2 update wrapped up, I’ve finally had more time to work on the Conduit Live manual — here is the latest complete draft. As always, any feedback is most welcome!

A major omission in the Conduit Suite is that we didn’t have a proper manual for Conduit Live. There’s the manual for the Conduit plugin that you can download at dvGarage’s Samples page, but it only covers the common functionality (Conduit Editor, node reference, etc.), not the workings of the standalone Live application.

Understandably this has been testing your patience, and I apologize. I’m writing a User Manual and will release it in pieces as soon as they’re ready for consumption. Here’s the latest version (updated 2008.08.11):

Conduit Live User Manual

If there’s anything you specifically want to see covered, please don’t hesitate to post a comment.

There are two separate full-screen outputs in Conduit Live. The difference between them is subtle but potentially important if your computer setup includes multiple video cards.

The controls for both outputs are located in the Control Panel:

“Main output” is the one that you’re normally watching in the Output Viewer window. To move the main output onto a separate display of its own, just select the display from the pop-up and press Cmd+F (or choose the “Enter Full Screen” option from the Output menu). That’s all there is to it!

If you want to view the main output simultaneously on a separate display and on the main display, just open the Monitor window.

The scaling options are hopefully self-explanatory: the output can be displayed with 1:1 pixels (i.e. no scaling is done, so the image may be cropped or bordered); stretched to fill the screen while maintaining aspect ratio (which will result in borders if the aspect ratio is different, for example when viewing a widescreen image on a 4:3 display); or stretched to fill the entire screen regardless of aspect ratio.

The pixel aspect ratio settings can be useful if the display in question is something else than a computer screen. For example, an NTSC TV-out display is commonly 720*480, which means that pixels are stretched on the monitor. (How much they are stretched depends on whether the monitor in question is a 16:9 widescreen or a traditional 4:3 screen…) Choosing the correct pixel aspect ratio will allow Conduit Live to take this into account when rendering the image.

So what is Aut Ouxput then? It’s a feature that allows you to display two different images on two different displays. When you enable it, a new AuxOut node is created in the Conduit Editor (if there isn’t one already). It works just like the regular Output node: you can pipe anything into it and it’s rendered on the display that you specified.

The render size options for Main and Aux outputs allow you to specify the resolution used for rendering each output image. This can be useful to improve performance: for example, you might have a 1920*1080 HD image that is rendered at full resolution for the main output, with an alternate preview version simultaneously rendered on a smaller display using Aux output. As the smaller display doesn’t need the full resolution, you might specify something like 640*360 as the render resolution to ensure that both images render at full framerate.

Here’s an example situation that shows AuxOut:

What’s happening here is just colorspace conversions — not very exciting certainly, but often crucial. The original image in this case is an sRGB image; it has a gamma of 2.2. The main output is a display with a gamma of 1.8 (this is the standard Mac calibration), so we convert the gamma from 2.2 to 1.8. An easy way to accomplish this is simply to convert to linear (1.0 gamma), then back to the required gamma value. (As Conduit always combines nodes when possible, there’s absolutely no loss of quality from this method.)

The Aux output is on a different display which uses the Adobe RGB colorspace. If sRGB colors are viewed on a display that expects Adobe RGB, they will appear much too saturated. The Convert RGB node is convenient for converting back and forth between the most commonly used colorspaces. It can also be used for white point conversion (e.g. if you have photos that were saved using a warm D50 white point and you want to display them on a video monitor with a cool D93 whitepoint — or just for artistic purposes).

There’s one more thing to cover about AuxOut: it’s a non-obvious but important difference between how the main output and aux are rendered in full-screen mode. If you have multiple video cards in your computer, you may need to be aware of this…

Multiple video cards means multiple GPUs (graphics processors), and that poses a problem because GPUs can’t share data easily. Conduit Live always renders on the main GPU in your Mac. Typically the main GPU is much faster than any additional GPUs that might be on the system: for example, if a Mac Pro is outfitted with both Radeon X1900 and a Geforce 7300 video cards, the Radeon is several times faster than the Geforce — it makes sense to render everything on the main video card when possible.

But there’s a catch: in this example situation, the main output can’t be rendered on a display that’s connected to the Geforce because it’s on a different GPU. If you try to enter full-screen on such a display, it won’t render.

Luckily AuxOut comes to the rescue. It works on any connected display device, regardless of whether it’s actually rendered by the main GPU. Conduit Live will still render on the main GPU (the fast video card) as usual, and the AuxOut image data will be to the other GPU if necessary. In theory this feature allows AuxOut to work with devices that look like displays to the operating system but aren’t really GPUs at all, such as SDI Out on certain capture cards and other weird video outputs. (…But I haven’t actually tried it, so I can’t guarantee compatibility. If you do try it, please let me know how it goes!)

In my previous post, I wrote: “The Conduit Suite gives you access to Conduit in 7 different applications: FCP, Motion, AE/Mac, PS/Mac, AE/Win, PS/Win, Conduit Live.”

Make that eight, as the Conduit plugin also works in Apple’s Final Cut Express 4. The latest version of this excellent mid-range ($199) video editing application added support for the FxPlug plugin standard, so Conduit now works just as it does in FCP and Motion.

There’s still a few days left of dvGarage’s $149 sale on the new Conduit Suite. If you can accept the limitation of only working with “prosumer” video formats, the combination of FCE+Conduit makes for a tremendous editing system: you get the fluid and familiar Final Cut interface, HDV and AVCHD support, FCE’s new open format timeline… And Conduit delivers advanced effects directly on the timeline – flexible keying, infinitely customizable color effects, custom transitions between clips, and guaranteed high-end color precision through pervasive HDR support.

Total cost: $348 – and you’re also getting Conduit Live and all the other Conduit plugins completely free.

So, the big update on which we’ve been working for quite a while has been released. It’s called Conduit Suite, and it significantly expands Conduit’s reach: in addition to the existing plugin which works in Final Cut Pro and Motion, the Suite includes plugins for After Effects and Photoshop, as well as a brand new standalone application, Conduit Live. (To see Conduit Live in action, check out this video tutorial by dvGarage.)

The After Effects and Photoshop plugins are also available for Windows. These are not crippled ports: they offer the same Conduit Editor user interface. Conduit’s rendering backbone, the Conduit Pixel Engine, has been ported to Windows’s native graphics interface Direct3D, so you’re getting the best GPU rendering performance available on the platform just as with Conduit on the Mac.

The Conduit Suite gives you access to Conduit in 7 different applications: FCP, Motion, AE/Mac, PS/Mac, AE/Win, PS/Win, Conduit Live. Of course the same effect setups (.conduit files) work in all these applications, so it doesn’t matter where you build an effect, you can deploy it practically anywhere.

Interested yet? Our partner dvGarage is running a pretty hot sale: the entire Suite is only 149 USD until February 20, 2008. (For only $50 more, you can also get dvMatte Pro 3 for true keying nirvana!)

- -

Anyway, enough with marketing and on to the real content: to celebrate the occasion, I’d like to show something simple with Conduit inside Photoshop. Let’s start with this snapshot I took some years ago:

We’d like to place the lion in another background, so in practice we’ll just key out the sky:

I’m using the 3D Keyer because it’s easy and quick for keying out an arbitrary color. Levels is used to tweak the matte’s black/white points, and Exposure is used to tweak the layer’s color — nothing special going on there.

So what happens at the end of the conduit? Set Matte is used to apply the key matte to the image, and then it’s fed to the output. This means we’re giving back to Photoshop an image with an alpha channel. The actual compositing of the keyed lion over the background happens in Photoshop. Here’s what it looks like in Photoshop’s layers panel:

IMO this technique makes a lot of sense because it combines the strengths of Conduit and Photoshop.

Another thing: notice the label “Smart Filters” in the screenshot above? In Photoshop CS3, Conduit works as a non-destructive smart filter: you can go back and edit the effect in the Conduit Editor simply by double-clicking on the Conduit label in the layer’s properties. It’s like Adjustment Layers in Photoshops of the past… Except that Conduit has a hundredfold more capabilities than Photoshop’s simple built-in adjustments. (Thank you Adobe, I love smart filters!)

So here’s what the composite looks like, with the roughly keyed lion over a very plain gradient background:

Hmm. I think the lion and those white clouds might look nice with some glow, and maybe I could experiment with the color a bit. Well, those smart filters definitely come in useful; I’ll just open the Conduit Editor and add some nodes…

(Figuring out what’s happening here is left as an exercise to the reader.) Here’s the final image with a text layer added:

To recap, this Photoshop composite contains three layers: the background gradient, the text, and the lion. The only filter used is Conduit, which was applied to the lion layer. The node tree shown above does all the hard work: keying (sky removal), glow, color adjustments.

Are you seeing the benefits yet? We could have accomplished something similar with a combination of adjustments, filters and selection tools in Photoshop… But by building it in Conduit, everything happens within a single interface.

This way, the effect is also eminently reusable. Just save it as a .conduit file and it can be applied to another layer in another Photoshop document… Or a layer in an After Effects composition… Or a video clip in Final Cut Pro… Or even a live HD video stream in Conduit Live.

This is something of a FAQ: how does one apply a mask (a.k.a. matte) to an effect in Conduit, so that the effect is applied only to certain parts of the image?

There are a few ways to accomplish this. If the effect to be masked is a single node and it has an Effect opacity parameter, you can simply plug the mask image into this parameter input:

But if the effect is more complex, maybe consisting of multiple nodes, you’ll need to use a node to combine the original image with the effect. You could use the Blend node for this purpose:

The Blend node’s Bias parameter controls the blending of the two input images. This parameter only accepts a scalar (=greyscale) image as input (as indicated by the light-blue color), so I’ve added a Separate RGBA node to separate the alpha channel from the mask image.

Alternatively, you can achieve the exact same result by using an Over node:

The light-blue middle input on the Over node is the mask that controls where the top image is composited over the bottom image. In this case, the top image is the one with the Bezier Curve applied.

Whether to use Blend or Over is entirely up to personal preference. (I prefer Blend because it somehow makes more sense to me to think of this operation as blending two images, rather than compositing one on top of another. But like I said, the end result is exactly the same either way.)