Why do I lose dps and dmg points when i put on argyle or hunter sets?

I usually wear serpent set.

I didnt know vanity can affect your overall point status

MM