disconnected VM operation and 3D
So one of the stumbling blocks on my road to getting 3D emulation in a VM is how most people use qemu in deployed situations either via libvirt or GNOME boxes frontends.
If you use are using libvirt and have VMs running they have no connection to the running user session or user X server, they run as the qemu user and are locked down on what they can access. You can restart your user session and the VM will keep trucking. All viewing off the VM is done using SPICE or VNC. GNOME Boxes is similar except it runs things as the user, but still not tied to the user session AFAIK (though I haven't confirmed).
So why does 3D make this difficult?
Well in order to have 3D we need to do two things.
a) talk to the graphics card to render stuff
b) for local users, show the user the rendered stuff without reading it back into system RAM, and sticking it in a pipe like spice or vnc, remote users get readback and all the slowness it entails.
No in order to do a), we face a couple of would like to have scenarios:
1. user using open source GPU drivers via mesa stack
2. user using closed source binary drivers like NVIDIA or worse fglrx.
How to access the graphics card normally is via OpenGL and its window APIs like GLX. However this requires a connection to your X server, if your X server dies your VM dies, if your session restarts your VM dies.
For scenario 1, where we have open source kms based drivers, the upcoming render nodes support in the kernel will allow process outside the X server control to use the capabilities of the graphics card via the EGL API. This means we can render in a process offscreen. This mostly solves problem (a) how to talk to the graphics card at all.
Now for scenario 2, so far NVIDIA has mostly got no EGL support for its desktop GPUs, so in this case we are kinda out in the cold, until they have at least EGL support, in terms of completely disconnecting the rendering process from the running user X server lifecycle.
This leaves problem (b), how do we get the stuff rendered using EGL back to the user session to display it. My first initial hand-wave in this area involved EGL images and dma-buf, but I get the feeling on subsequent reads that this might not be sufficient enough for my requirements. It looks like something like the EGLStream extension might be more suitable, however EGLstream suffers from only being implemented in the nvidia tegra closed source drivers from what I can see. Another option floated was to somehow use an embedded wayland client/server somewhere in the mix, I really haven't figured out the architecture for this yet (i.e. which end has the compositor and which end is the client, perhaps we have both a wayland client and compositor in the qemu process, and then a remote client to display the compositor output, otherwise I wonder about lifetime and disconnect issues). So to properly solve the problem for open source drivers I need to either get EGLstream implemented in mesa, or figure out what the wayland hack looks like.
Now I suppose I can assume at some stage nvidia will ship EGL support with the necessary bits for wayland on desktop x86 and I might not have to do anything special and it will all work, however I'm not really sure how to release anything in the stopgap zone.
So I suspect initially I'll have to live with typing the VM lifecycle to the logged in user lifecycle, maybe putting the VM into suspend if the GPU goes away, but again figuring out to integrate that with the libvirt/boxes style interfaces is quite tricky. I've done most of my development using qemu SDL and GTK+ support for direct running VMs without virt-manager etc. This just looks ugly, though I suppose you could have an SDL window outside the virt-manager screen and virt-manager could still use spice to show you the VM contents slower, but again it seems sucky. Another crazy idea I had was to have the remote viewer open a socket to the X server and pass it through another socket to the qemu process, which would build an X connection on top of the pre opened socket,
therefore avoiding it having to have direct access to the local X server. Again this seems like it could be a largely ugly hack, though it might also work on the nvidia binary drivers as well.
Also as a side-note I discovered SDL2 has OpenGL support and EGL support, however it won't use EGL to give you OpenGL only GLES2, it expects you to use GLX for OPENGL, this is kinda fail since EGL with desktop OpenGL should work fine, so that might be another thing to fix!
David, question, I get the bonus stability from running qemu as something other than the non-current user... but I feel like its a little counterproductive if we are hitting snags like this. What's wrong with virt-manager being just a standard user application? Don't need root access to run it (I wouldn't think anyway), Virtual machines are saved to "~/My Virtual Machines", no fighting with the graphics server since it'd be tied to the user session. We'd get the exact same situation as we have with VMWare or Virtualbox, which is frankly pretty good.
Could still keep the client-server model around as an option, just like VMWare has, but it seems like a feature like 3D is more of a user-friendliness feature (games, videos, desktop effects) and therefore if QEMU/KVM is now going towards the user friendliness and the 'average' user who wants a VM for Netflix or gaming, then we should just push all the way towards that goal.
Well boxes mostly does that it just still runs things disconnected from the user so if the server dies the VM's dont, granted I expect I should fine some way to offer 3D first in that scenario, but there are lot of people stuck in the current model and I don't really have the time or power to spend persuading them otherwise.
We also get benefits like SElinux and sandboxing from running VMs in this way.
Really qemu isn't the problem, if people were able to run it directly, but qemu developers insist on command lines making a GUI kinda pointless as you have to spend days writing the command line, so I think libvirt is the only decent interface to qemu we deploy currently so whatever I do should integrate with that in the end. I've already got 3D working on qemu with SDL and GTK3 backends, its just not really something a user would ever figure out.
Sorry about the double post, first post this morning shot back an error, so I retyped it tonight.
By sandboxing I'm assuming you mean we dont potentially screw up the User's home dir, which is understandable. What's the SELinux benefits from running it like this vs running it as a standalone app?
Also, can you elabroate on what ya mean by "but qemu developers insist on command lines making a GUI kinda pointless as you have to spend days writing the command line"?
yeah the VMs running under SELinux are locked down pretty well, so someone escaping the VM via bugs in qemu can't interfere with anything else the user is running or any other VMs running.
QEMU does all its configuration through the command line, libvirt generates the command line for all the cases it supports. There has never been much upstream interest in using a configuration file to replace the command line, or at least it hasn't been achieved as of yet. So you have a GUI but to get it started you have to specify all the things like disks and networking and devices on the command line, hardly user friendly.
Interesting about libvirt, I knew what it was but I hadn't really delved into it while I was on Fedora and Arch, when I jump back to Fedora after the F20 release next week I'll have to check it out more. Does make sense then that any GUI tool would just target libvirt rather than QEMU/KVM directly.
How tied to QEMU and KVM is Virgil3D? Phoronix ran an article about a new linux hypervisor a few weeks ago, I'm blanking on the name, but could your Virgil3D stuff work on there? Or others? Could it eventually replace the VMWare and VirtualBox graphics (in theory at least, obviously they'd have to choose to do that)?
It's up to NVidia to make their proprietary drivers work within the OSS software stack. With any luck, one day they'll realise it's easier for them to get nouveau into shape than to have to keep their binary blobs working against the ever changing OSS landscape. [one can dream] Compromising the design, just to support current binary blobs would be a mistake IMO.
Using render-nodes ties things in nicely with where the Linux software stack is going. When there is no (graphical) terminal connection, the configured graphical virtualised output should keep running, just as with qxl, perhaps triggering a monitor hot-unplug.
Its mainly for personal benchmarking I want the nvidia driver :)
since I can compare against vmware using it and get more accurate results, also its good to test the renderer on multiple OpenGL implementations.
David, just out of curiosity... Why is QEMU/KVM set up like client-server? Sure the stability is great but it just seems like a weird default. Why not do Virt-Manager like VMWare and Virtualbox? Standard user app, Virtual Machines get stored under "~/My Virtual Machines" no weird security issue to deal with, more user friendly, gets run with standard user permissions. For those that need client-server keep it available and configurable, just the same as VMWare does with its Server product.
I make extra note about User Friendliness since 3D, to me, would be mostly targeted at them-- Plymouth splash during startup, desktop graphics, movies, games, the likes. I realize there's other possible uses but are any of them excluded from happening as either a client-server or standalone app model? The current model just seems to have some fairly large hurdles at this point that you (seem to) imply wouldnt be an issue if virt-manager was just a standard user app and qemu/kvm was just some general backend library
There is quite a lot of state in the GPUs, so even if you suspend a VM, wouldn't getting it running again be unpleasant?
I'd have thought that you be better off just hot-plugging the GFX card in and out, and telling the Guest OS to deal with it. If you're going to eventually support live migration (especially between GPUs of different capability), you'll need quite a few smarts in there anyway.
I'm not very knowledgable about hardware - can you hibernate the entire graphics stack and recover later?