I can see the whining already from people in smaller guilds.

I don't really see this working as is, it seems like a broken idea and it doesn't really address an existent issue. Just invite 4 guild members to a party and you'll get an xp bonus.

Guild vanities - maybe. Extra xp? No