It looks like someone has to be in costume when going down the hole, and have walked to it before using, or the portal doesn't show up.

It always worked for me since I walked to it.
I'll fix it to work even if you don't walk close enough to the hole.