It seems to only happen in Gauntlet. Every wave I switch my skills via the skill map: I replace my FB to curse at the boss, and replace curse to FB on mini rooms (every single wave).

After some time it doesn't work anymore, when I go press the fireball it doesn't deploy anymore but I do see it refresh its cooldown. It also doesn't let me switch to another skill, for instance, I can't change back to my curse. So I'm effectively only able to use 3 skills at gameplay instead of 4. And no I am not experiencing high ping or spikes.

Temporary Fix: The only remedy I'm doing to make it work again is by loading my 2nd skill loadout, then loading the 1st skill loadout again. But it would occur again indefinitely.