For what it’s worth, I have since then replaced my YubiKey and the problem doesn’t seem to occur anymore.
To anyone else having such issues, make sure you can’t reproduce the problem on another computer first! I haven’t actually tried that, but it does look like the newer YubiKey doesn’t suffer from those issues…