This was being done so only on pointers. Internally, a GdkDeviceGrabInfo is kept for each of the master pointer/keyboard, failing to do this for keyboards results in a stuck keyboard grab. https://bugzilla.gnome.org/show_bug.cgi?id=748892