I'm guessing you are on OSX?

Most likely, you are holding down your "A" key.
I dunno how you could remove those, but when moving, try clicking 2 directional keys at the same time (A+W, W+D, S+D,...