So, I went ahead and did this (as well as a respec) and wow.. it was so worth it. Since I did it, I have held aggro on every boss 100% of the time no matter what classes I was with. I rarely have to use a single potion on elite jarl runs, unless there is no mage healing, and even then I only use 2 or 3. I don't even have the best armor/helm yet, so my defense will still get better!