I've had my pet go hungry while the cooldown timer for its AA was still counting down. I would then go into my stable to try and change pets but having to wait for CD to expire. If this was in the middle of a tomb run, this wouldn't end very well. So I would suggest keeping a cooldown timer over the sad face so we could still fight while waiting for the cooldown to expire and switch pets when able to.
