On wl_keyboard.key/modifiers, we're just forgetting about currently pressed mouse buttons. Fix this by storing button and key modifiers separately, and put these together when creating the GdkEvents.