I could imagine that it just doenst work, when you slot in other pets. Maybe thats wanted as it would give more luck? Could also be a bug though. Did you try with different pets? If its the same with different pets in your slots, then its probably wanted.

Maybe a dev can answer that?