There is an option to join a guild you've been invited too. But no option to decline.