When returning from Elite Gnomish Ruins back to [Elite] Cryogem Mines, you actually return to normal Cryogem Mines instead of elite.

Edit: Note that this is within the map and not through the world map.