I have done a fair amount of testing with & without passive damage and I can say your auto attack and skill damage does do more with passive damage. Where passive damage doesn't help is where the skill formula would only take stats into account. Life giver or shield for example. 5/5 Dex gives you +125 HP so this is the main advantage and you only get a small boost in damage. And 5/5 STR gives you +250 HP.