5

This is mostly relevant for augmented reality type applications. Apple provides information on how to capture video frames (and save them as images if need be) with AVCaptureSession here:

http://developer.apple.com/library/ios/#qa/qa2010/qa1702.html

I know that it is possible to create an openGL texture out a captured video frame and then use that as a background in the openGL view over which to overlay other graphics.

I am wondering if there are any alternatives to this method? The method mentioned above may be the best (I don't know if it is) but if there are alternatives to try it would be good to know. For example, is there a way to overlay the openGL view directly over the AVCaptureVideoPreviewLayer?

ajeetdl
  • 1,254
  • 1
  • 13
  • 17

2 Answers2

6

You can indeed layer OpenGL content over something like AVCaptureVideoPreviewLayer, but your performance will suffer. Apple highly recommends that you not overlay non-opaque OpenGL ES content on top of other display elements. From the OpenGL ES Programming Guide for iOS:

For the absolute best performance, your application should rely solely on OpenGL ES to render your content. To do this, size the view that holds your CAEAGLLayer object to match the screen, set its opaque property to YES, and ensure that no other Core Animation layers or views are visible. If your OpenGL ES layer is composited on top of other layers, making your CAEAGLLayer object opaque reduces but doesn’t eliminate the performance cost.

If your CAEAGLLayer object is blended on top of layers underneath it in the layer hierarchy, the renderbuffer’s color data must be in a premultiplied alpha format to be composited correctly by Core Animation. Blending OpenGL ES content on top of other content has a severe performance penalty.

Honestly, it really isn't that hard to pull in the video as a texture and then display that as a billboard behind your 3-D overlay. My sample application here does passthrough of camera video to an OpenGL ES (2.0) texture for display to the screen. With only a few modifications, you could place 3-D content on top of that. This will give you far better performance than trying to draw non-opaque 3-D content on top of an AVCaptureVideoPreviewLayer.

However, if you are just wanting to display simple static UIViews over OpenGL ES content, that can be done without much of a performance penalty (~5% reduction in framerate in my experience).

Brad Larson
  • 170,088
  • 45
  • 397
  • 571
  • Thanks. I did not have a problem with pulling in the video as a texture but I wasn't sure which technique would yield the best performance (creating/bind texture vs layering). Your links to Apple's document and your own code are very helpful. I am doing something quite similar to what your code does and have most of that in place. Thank you. – ajeetdl Dec 18 '10 at 01:56
  • @BradLarson is there any performance issue if we place AVCaptureVideoPreviewLayer on top of OpenGL ES content ? – onmyway133 Jun 06 '13 at 08:34
  • 1
    @entropy - If you're talking an opaque OpenGL ES hosting view, that's not going to be significantly different from placing other UIViews in front of it. You'll of course have to profile this first to see what the true impact is, because the extra compositing workload might slow things down. You'd probably be best off by feeding your camera video into an OpenGL ES texture and displaying that in your scene, though. My GPUImage framework can do this, for example, but it's pretty easy to write code to go from AVFoundation to a texture. – Brad Larson Jun 06 '13 at 16:46
1

Sure, views can be layered together, regardless of content. Layering GL over Video is no different than layering 2D over 2D.

Just about the only catch is that you need to render your GL content so that the image produced is premultiplied by alpha (just like all other transparent content on iOS is premultiplied).

Frogblast
  • 1,671
  • 10
  • 9