Jewels are the only ones removable (except Lazarith). Gems are permanent once you socket them.