Actually you do not have to attack either.
If you just stand there with a rogue, either will he leave the map and another one will enter the map, or you leave and join another party.
Its as simple as that.
Those rogue whiners are 12 y/o kids anyways so you shouldnt give them too much attention.
Rogues blame me too, but I dont care since the map can be done with 3 players too so yeah...