I was able to turn it off after my first level up... And it never gave me exp while offline :'(