I'm curious where negative values come from exactly?

The calling code could pass just current_group - 1 to switch to previous layout.

Right now, we have to do the wrapping ourselves here in KWin, which duplicates the code above:
https://lxr.kde.org/source/plasma/kwin/src/xkb.cpp#0606

I see, thanks.

As you can see from the XkbWrapGroupIntoRange code you linked to, XKB has a concept of "Out of range group action", which determines what happens when the group index goes of range. It can be configured per-key using the following values in the XKB symbols file:

• groupswrap - wrap
• groupsclamp - saturate, i.e. stop at zero if underflow and at last group if overflow
• groupsredirect=<groupnumber> - set to <groupnumber>

Like many things in XKB, I'm pretty sure this functionality is entirely unused, but it's supported by libxkbcommon and you can never know what configs people have.

So question is, when switching to prev/next in KWin UI, do you want the "Out of range group action" to be consulted, or always wrap?

Right, I was confusing the "global" active layout (which always wraps as we don't support the global "XKB Controls" like X11 does), and the per-key derived layout. So that's fine.

The next problem is that xkb_state_update_mask is not really meant to be used in the way KWin uses it in XKB::switchToLayout. As the xkb_state_update_mask docstring says:

 * This entry point is intended for window systems and the like, where a
 * master process holds an xkb_state, then serializes it over a wire
 * protocol, and clients then use the serialization to feed in to their own
 * xkb_state.
 *
 * All parameters must always be passed, or the resulting state may be
 * incoherent.
 *
 * The serialization is lossy and will not survive round trips; it must only
 * be used to feed slave state objects, and must not be used to update the
 * master state.
 *
 * If you do not fit the description above, you should use
 * xkb_state_update_key() instead.  The two functions should not generally be
 * used together.

that is, it is only meant to be used as a counterpart to xkb_state_serialize_mods and xkb_state_serialize_layout.

Of course, KWin is entirely justified in using it this way, as xkbcommon doesn't provide any other way for a UI toggle to switch the layout in the master state.

From this lens, xkb_state_update_mask is good as it is, and what we really need is a new function to update the active layout in the master state (like xkb_state_update_key for updating key state in the master state).

If we design such a function, would you be able to use it?

Yes I think we could use it, just thought it maybe feasible to change the type for xkb_state_update_mask, too - because right now it already allows wrapping "forward", but doesn't allow "backward" because of that unsigned type. So that looks like a symmetrical and at the same time less invasive change, backward-compatible withe the current API.

If we design such a function, would you be able to use it?

On a second thought, do we really need a new function here if the same can be achieved with the existing one?

I've created a proposal for a new function for doing this - #314. Not implemented yet, just the API, but let me know if you have any thoughts on it.

Related: #72