Inefficient Alpha-Blended Windows: Layered Windows

This is the preamble to the tutorial on DirectComposition. If you just want efficient transparent windows without the history lesson, skip straight to that article. In this article, I break down why the popular GDI approach just doesn't cut the mustard any more.

Recently I found myself wanting to render 3D content to a transparent, borderless window, and have that window blend seamlessly with the window(s) behind it. Imagine, if you dare, something that resembles our old friend Clippy from Office, only hopefully much more modern and much less irritating.

RIP Traditionally, transparent widgets of this kind have been achieved through using Layered Windows, i.e. using the WS_EX_LAYERED extended window style (which makes the client area transparent), removing WS_BORDER and WS_CAPTION, and blitting an image onto the window or desktop using GDI.

The Layered Windows APIs provide some cool (in a 90's kind of way) but limited options for controlling how the final result is composited: you can specify a color key (which means that the compositor ignored that color and treated it as fully transparent); specify a uniform alpha blend factor for the entire window; or do a per-pixel alpha blend.

If you're happy using a color key to achieve transparency (who likes fuchsia anyway?), you can use the SetLayeredWindowAttributes and BitBlt APIs to blit your image. You can also use this API to adjust the translucency for all visible pixels too, but you won't get different opacity values per-pixel. As I mentioned above, per-pixel alpha-blending is possible and we'll cover that in a moment.

How you generate the image that you want to composite is up to you, as long as it ends up in a Device Independent Bitmap (see CreateDIBSection). Commonly people will render using software, either via their own routines or GDI/GDI+. However, if you want to do some really cool stuff (which we do, obviously), you could conceivably use a 3D API to render a super-duper 3D alpha-blended scene—perhaps a spinning 3D model surrounded by a translucent fire effect.

Let's take the example of using OpenGL, which is a common choice for this scenario. To do this, you would create a 32-bit DIB, specify the PFD_SUPPORT_OPENGL|PFD_DRAW_TO_BITMAP flags in your PIXELFORMATDESCRIPTOR and bind your OpenGL context to that bitmap's HDC. Once you've rendered your scene into the DIB, you can blit that to a Device Context.

Now, let's circle back to per-pixel alpha blending. We want see our other application windows through the fire, but we want to keep our spinning model fully opaque. We can't achieve this using color keys and SetLayeredWindowAttributes, so instead we use UpdateLayeredWindow. Like the previous APIs this allows you to composite an image onto a window or the desktop itself (i.e. GetDC(NULL)) but it also understands what an alpha-channel is. In fact, you can instruct it exactly how to blend your pixel values through a BLENDFUNCTION structure. The only prerequisite is that your image has the alpha premultiplied into your color values.

Unfortunately, this doesn't work as well as one would hope it would. There is an irritating quirk in OpenGL-to-DIB interoperability: even if you set up a 32-bit DIB, rendering OpenGL to a DIB will completely discard the alpha values. As MSDN puts it:

Each DWORD in the [DIB data] represents the relative intensities of blue, green, and red for a pixel. The value for blue is in the least significant 8 bits, followed by 8 bits each for green and red. The high byte in each DWORD is not used.

Even if you disable GL_BLEND or use glColor4f(1,1,1,1), your alpha values will always be zero! As such, you'll have to fill in your alpha channel yourself as a post-process. So close, yet so far.

However, the problems with Layered Windows go beyond the DIB alpha-channel limitation.

For example, rendering OpenGL straight into a DIB forces OpenGL to render in software rather than using the GPU—which a very serious performance pitfall. We can try and mitigate this by rendering into an off-screen buffer instead, such as an OpenGL Frame Buffer Object or Direct3D Render Target. This leverages the GPU so we get the performance we expect when rendering our 3D scene—great!

But, wait, now we have to transfer that image back from the GPU to system memory. Depending on the size of the image, this could cost us non-trivial time or PCI bandwidth. Once we have it in system memory, we might have to copy it again out of our graphics API into the DIB, possibly even having to swizzle the channels to be compatible with the DIB (ARGB). At least we'll have an alpha channel this way, I guess.

However, even when you ignore performance, things still aren't rosy. In my experiments I saw some pretty nasty flickering when moving or resizing my transparent window. When you consider what GDI is and how it works, this isn't much of a surprise. GDI is a mechanism that has existed in Windows since the 80's and, at the end of the day, it's a software renderer that simply isn't designed to be alpha-blending huge regions of your desktop.

Throughout my struggles with OpenGL and GDI, it rankled that I was stuck using a software renderer when Windows already seemed to be abusing some kind of HAL to render fancy border shadows and other effects. And don't even get me started on Vista's Aero Glass: that thing wasn't just translucent—it even blurred the background of windows! Surely there must be some way...

Go to Part 2 →