December 10, 2009

Flash in QGraphicsView

Over the past couple of months, I have been working losing sleep over getting Flash to work in QGraphicsView on all platforms.  The good news is that flash now works in QGV on all platforms. Yes, that means that not only can QGV display Flash but can also rotate and transform flash when using QGraphicsWebViewIMPORTANT: Flash embedded in a QWebView using QGraphicsProxyWidget is NOT supported.

Youtube in QGVLauncher
Youtube in QGVLauncher

What follows is a brief summary of my work and the list of patches that you will need to get Flash working with Qt 4.6.0. Note that when I say plugin below, I mean NPAPI plugins like Flash. They have nothing to do with QWidget based plugins. QWidget based plugins should work in Qt 4.6.0 when using QGraphicsWebView. I have also simplified the concepts (in favor of being completely correct) in the post below so that it is readable.

Windowed and Windowless plugins

Flash gets embedded into web pages using an API called NPAPI (Netscape Plugin API). NPAPI provides two mechanisms to embed plugins into a web page – Windowed mode and Windowless mode.

Windowed mode – the idea is that the plugin gets a native widget handle and does it’s own mouse, keyboard processing. For example, on Windows, we give the plugin a HWND. The plugin will install a window proc and process events directly (i.e the events are never seen by the browser). On X11, the plugin embeds itself into the Window using XEmbed. When the browser scrolls, the browser moves the native window appropriately. Mac doesn’t support windowed mode.

Windowed mode should have been enough but it has a limitation that it interferes with the z-ordering of html elements.  Some sites like to put html _over_ their flash content (for example, menus). This can be achieved by making the menu html have a higher z-order than the plugin. Can you spot the conflict interest here between z-ordering of native widgets and z-ordering of HTML elements? The plugin native window is a child of the browser’s window and we have a situation where the browser wants to paint HTML after the child has painted itself! This is not possible since a parent window always draw before it’s child. There are other disadvantages too – it is also impossible to transform plugins (native widgets are rectangular), making them transparent is tricky and printing plugins requires lots of hacks.

Windowless mode – The idea here is that plugins just paint to a pixmap – platform’s graphics context to be more precise (HDC on Windows, Pixmap on X11, ContextRef on Mac). It is now the browser’s responsibility to pass (forward) mouse and keyboard events to the plugin. Since the browser now contains a screenshot/snapshot/pixmap of the plugin, the browser can transform plugins, draw it in the correct z-order, make is transparent even. The main disadvantage here is that it can tend to be a bit slow.

Evan has a better explanation of the modes. There’s also one by Robert from which I originally learnt about these modes but I cannot find the link anymore).

Flash

I keep saying Plugin but really we care only about Flash ;-) At least, for the moment that is our priority. All version of Flash supports Windowed mode. Only Flash 10 supports windowless mode on linux.

Flash lets the html/plugin author control the mode of operation using the wmode parameter (i.e pass it as part of html <object> tag). By default, it operates in windowed mode. Setting wmode to ‘transparent’  makes the html below visible through the flash. If wmode is set it to ‘opaque’, the html will not be seen through and the flash will have some solid background color (think of it as Qt::WA_OpaquePaintEvent, it allows for optimizations). transparent and opaque mode are implemented using ‘windowless’ mode. Here’s a great site to test out these modes in action.

Important: When in windowed mode, Flash expects a native window handle/id. A browser cannot force windowless mode – the mode is decided by the plugin. A browser can, in theory, say that it supports only windowless mode but this makes Flash crash (at least) on Linux. Hence, when we have a site like youtube (which uses Flash in windowed mode) in QGV, we need to come up with a way of providing flash with a window handle. There are two ways to deal with this problem – we can either give Flash a fake native window, we then grab the contents of this fake native window and paint it wherever we want OR we can inject the wmode ‘opaque’ parameter when loading flash in QGV. The former approach is platform specific and requires lots of work. The latter is a 10 line patch.

Plugins in QGraphicsView

Items inside QGraphicsView are not native widgets. They are just items on a canvas that can be transformed and visualized using multiple views. This means that windowed mode of plugins is not feasible (apart from the fact that we have no native widget handle for items on canvas). To make plugins work in QGV, we need to implement windowless mode.

X11 (Linux)

Windowed mode on Linux has already been implemented in Qt 4.5. What was missing is the windowless mode support. One nice feature is that Flash is able to fall back windowed mode when the browser says it cannot support windowless mode (vice versa crashes as I noted before). This is the reason that sites that use windowless mode work in QWebView with Qt 4.5.

To support Flash in QGV, I started out adding windowless mode support. In theory, implementing windowless mode is a matter of passing a X Pixmap to the plugin, but there were lots of problems. The entire history is on bugzilla bug 20081.

  • Flash uses a different X Display connection – the default one provided by gtk/gdk and not the one that Qt provides it. Ouch. This means that when flash paints something, we have to XSync to get changes reflected in the Qt connection and vice versa. The uglier part is that Qt somehow needs to get hold of gdk’s Display without linking to gdk. It’s all very hairy. (r49158).
  • Flash uses a different X Visual (the default system visual) rather than the one provided by Qt. This means that even if we created a 32-bit pixmap, Flash won’t be able to draw into it because it uses a 24-bit visual. Since people really wanted transparency to work, the solution is to grab the contents of the backing store to fake transparency. Very hairy stuff. This solution works in QWebView but not with QGraphicsWebView (r49169). We have turned off transparency in QGVuntil this Flash bug is fixed.
  • The NPAPI requires X Pixmap. QPixmap may or may not be backed by a X Pixmap depending on thegraphicssystem.
  • Print preview implementation of Qt holds a reference to the X Pixmap (because of QPicture). This meant that when the user prints we have to ‘let go’ of the X Pixmap because the print preview has to remain unaffected by on screen changes. At the same time, someone needs to destroy this X Pixmap when the print preview dialog close. This one was very tricky to solve. Actually, I didn’t solve it, Samuel did :-) (r50123)

All windowless mode changes for Linux are part of Qt/WebKit 4.6. QGV on linux can display Flash only in windowless mode. If Flash is in windowed mode (like in youtube), it does not work.  As mentioned there are two methods to fix windowed mode in QGV . Method 1 – we create a fake window and grab it’s contents. This can be achieved using X Composite and X Damage (as done in Fennec). I have a patch at 31232 but it requires more work. Method 2 – we can inject wmode opaque. This has been committed as part of 32059. If you need youtube to work in QGV, you need to apply that the patch r51759 on top of Qt 4.6.0.

Mac OS X

On the Mac, plugins operate in Windowless mode _always_.  There is no such thing as windowed mode on the Mac. NPAPI  on Mac supports various event models and painting models. Event model dictates how we pass the mouse/keyboard events to Flash (i.e which data structure is used to report events – NSEvent or the classic EventRecord).  Qt/WebKit only supports only the carbon event model and the CoreGraphics drawing model (there’s a open gl drawing model, core animation and a QuickDraw drawing model). ATM, only Carbon based apps can display flash. Qt/Cocoa apps (32 or 64 bit) cannot display Flash in QWebView or QGV. (Confusingly, this has nothing do with Qt/WebKit not supporting cocoa event model. It is possible for Qt/Cocoa apps to display using the carbon event model. This is tracked using 32376)

Now to the carbon event model – how it works is that the browser provides the plugin it’s window handle (a WindowRef), a ContextRef and the position (a rectangle) inside the window and Flash draws itself there.

Simple, no? You wish :-) Unlike in QWebView, one cannot provide a rectangle as the position of Flash since the Flash could be transformed. For those new to Mac, ContextRef is like a QPainter (it can be over any paint device – a widget, pixmap, image). What we do is to get the ContextRef of a QPixmap and pass the position as (0, 0, width, height). Unfortunately, Flash requires the WindowRef parameter to be valid because it seems to be using it to detect if the window is ‘active’ for processing mouse move events. To workaround, we create a fake hidden window. In addition, we also fake the fact that this fake hidden window is the active window (else Flash won’t process mouse move events)! If you thought this was insane, you should check out what the chrome devs do (they intercept carbon calls of Flash by injecting a library using DYLD_INSERT_LIBRARY at runtime). See 311833179431979 for more history.

None of the patches are in Qt 4.6.0. You need to apply  r51234 for painting; r51105r51412 for mouse handling andr51485 for context menu position. Currently, printing does not work on the Mac. This is tracked at 31975. Flash transparency works fine in QGV.

Windows

Windows has always supported Windowed and Windowless mode (Thanks to Apple). Printing already works thanks to some awesome hacks. Flash transparency works in QGV. However, the problem (as mentioned above) is that Flash expects a HWND when in windowed mode. So, youtube inside QGV requires a HWND. Fix is to inject wmode opaque. You need to apply r51979 on Qt 4.6.0

Symbian

The hardwork here was done by Yael. See 29302 for more details.

Gitorious

I have done the necessary cherry-picking and published a branch (based of 4.6.0) athttp://qt.gitorious.org/~girish/qt/girishs-qt/commits/4.6.0-gv-flash. With that branch, Qt should support Flash in QGV on all platforms in all modes.

More information

If you need more information on internal working of plugins, Yael, Torarne and I have writtenhttp://trac.webkit.org/wiki/QtWebKitPlugins. Drop by on irc channel #qtwebkit on freenode for any clarification. Performance needs some love, we are working on it.

My work would not have been possible without the tireless reviews of SimonHolgerKenneth. Thanks guys! Simon, in particular, who  kept pushing me to try out various possibilities/hacks. I cannot thank him enough! And it’s a matter of great pride that I am now a webkit commiter! :-)

The guys at Linden Lab have made ubrowser support Qt/WebKit (llqtwebkit). With the above changes, flash works in ubrowser.

Qt/WebKit in ubrowser

18 Responses to “Flash in QGraphicsView”

  1. LS says:

    Staggering the amount of work one closed source plugin can cause. About time they opened this beast up to really keep Silverlight at bay.

  2. Karellen says:

    Wow. I have such conflicting emotions.

    1) That’s awesome! Thanks so much for figuring all that crap out, and the effort to get it to work.

    2) Yuck! No-one should ever have to wade through that amount of bogosity to get stuff to work properly.

    3) Wouldn’t the time be better spent improving Gnash?

    Number (1) is definitely the most prominent here. Even though I don’t use proprietary crap (and pretty much anything from Adobe for Linux really does deserve the moniker “crap”) on my system, it’s awesome in general for other people who don’t particularly care about “free”, but still want to use Linux and have things Just Work(tm). Try not to let (2) and (3) (especially 3) take away from that too much.

    Thanks.

  3. Livio says:

    Tell me… What about stability ^^ ? What happens when Flash crashes?

  4. Girish says:

    @Karellen : Agree with you; most of the pain is caused by Flash being closed source and it is hard to guess what actually Flash is doing. As for gnash, I intend to test it out soon.

    @Livio: If flash crashes, everything goes down with it :-) No work is being done to move Flash to a separate process AFAIK.

  5. Leo says:

    Hi Girish, slightly off topic question, but I think you are the right guy to estimate this. In Qt3 there was a nsplugin extension which offered the ability to run Qt inside Mozilla.
    Any thoughts how much work this would require to port it to Qt4?
    I looked a bit at the code, but its pretty hairy platform dependent and I could not see anything about Mac (and I’m on Mac…)
    Kind Regards, Leo

  6. Girish says:

    @Leo – I would say one has to write large portions of code to make it work for the Mac. I am guessing ~1 month to have most of the stuff working on the Mac.

  7. Thanks a ton for your explanation here. Very helpful indeed. I’m running into an issue on X11 with the forced “mode=opaque” patch, where it doesn’t appear to be forced at all. I’m wondering if it has to do with how I’m organizing my QWebView.

    Anyway, I don’t want to bother and have you do the debugging for me. I was hoping to do the debugging in QtWebKit on my own, but I don’t care to wait 2 hours in between Qt builds to check my changes. Can you point me in the direction of how you setup your development environment for debugging? Any pointers on how you’ve got your change/build/test process put together would be much appreciated.

    Thanks!

  8. Thijs says:

    I’m running into issues with Flash apps not being able to get mouse focus on osx 10.5 and 10.6 using qt 4.6.3, wonder if it’s fixed in 4.7.0 rc1 by now..

  9. Thijs says:

    Looks like 4.7 has most of the patches for OSX mentioned in this article, except the http://qt.gitorious.org/~girish/qt/girishs-qt/commit/6179e185892192aba52fc5b4c54aa031f2827e04 commit. 4.6.3 doesn’t have any as far as I could see, so using Flash apps with those builds seems like a dead end anyway, unless you patch it.

  10. Thijs says:

    Can confirm the Flash Player (10.1) works normally with 4.7 cocoa on osx 10.5.8 :) I wonder if I should still apply that patch in http://trac.webkit.org/changeset/51105 which supposedly brings down the cpu 20-30%.. But thanks for all the info, was pulling my hair out.

  11. Girish says:

    Thijs,
    51105 does reduce CPU usage but was reverted later because it was required for some Flash :( See http://trac.webkit.org/changeset/55332

  12. Thijs says:

    Ah, good to know. Perhaps the article should have a note about the upcoming 4.7.0 that contains your patches (at least for osx)?

  13. Thijs says:

    @Livio: Flash Player 10.1 introduced ‘out-of-memory management’ which supposedly should only take down a single flash player instance, and not the whole browser, more in http://kb2.adobe.com/cps/860/cpsid_86018.html

  14. bill says:

    I have a question please. since your good knowledge of the matter.
    is there any method to make possible flash player support
    on qt embedded.

  15. pump says:

    I want use QWebView play swf live ,but embedded fail ,please help me !
    os : 2.6.32.9-dove-5.4.2 #1 Mon Aug 22 22:18:20 PDT 2011 armv7l GNU/Linux

    bug etc:

    https://bugs.webkit.org/show_bug.cgi?id=72232

  16. fast weight loss diet plan says:

    I do not even understand how I stopped up right here, however I thought this post was good. I do not recognise who you’re but definitely you are going to a famous blogger when you are not already. Cheers!

  17. Johnd825 says:

    This actually answered my drawback, thank you! kdafaecadcda

Leave a comment

(email will not be published)