Pocket Legends Downtime 2/3/2015

[TzatOps]

Pocket Legends continues to experience poor database performance that is causing extremely long load times for many players. We have brought the game down again to upgrade the disks and apply some other optimizations to the database server that we believe will resolve the problem.

I expect the downtime to be no more than two hours and we'll post here when we bring the game back online.

Once we've had a chance to review the entire maintenance process we'll post some more detailed information about what went wrong and what we're doing to prevent it in the future for this and all our other games.

[lelandbroome04]

I and im sure others have lost 3x blessings and missed our daily blessings because of this incident will there be anything to compensate for that?

[lelandbroome04]

Sincerely,
Littlemangoby

[Writing]

yes... My daily blessing has been missed, any solution to that ^^^

[Vexmaw]

And how about those of us who are currently running a week long 2x elixir? Can we get a few hours back?

[sorristhatall]

9hrs from L77 , I planned this journey since the cap increased . I wouldve been a few hrs away from me goal . Ugh highly disappointed

[Waug]

Fix this server issue, ping stability was horrible during last three days.

[Mutilating]

Waug ping is caused on your end not Stg.

[lelandbroome04]

The maintnece is done but GAME IS STILL DOWN

[Befs]

Fix this server issue, ping stability was horrible during last three days.

I've also been experiencing this and yet I have 4 bar connection all the time. So just please fix the issue as long as it's done soon.

[Mutilating]

Please stop breaking PL. Ever since you guys released the app update on android many people have been having issues. You already discontinued the game now let us play out our last few months

[TzatOps]

We've upgraded the disks in the server, and applied some system settings that should improve performance but we are still seeing long load times for many players. We are continueing to work on the problem. I will post here again as soon as we have new information.

[dansmage]

We are continueing to work on the problem.
will pl be up by tomorrow tho?

[TzatOps]

well I hope so, we don't go home till its working properly, so....

[Writing]

well I hope so, we don't go home till its working properly, so.... now i feel bad ;( right now ive been lying down watching tv with a bowl of ice cream :3

[usmcporch]

now i feel bad ;( right now ive been lying down watching tv with a bowl of ice cream :3

I don't think telling him those specifics makes him feel much better z.z lol

[usmcporch]

Tzat, what was the original maintenance about? Before the other issues occuried that is. And did the "wall glitch" have anything to do with lag?

[Waug]

Waug ping is caused on your end not Stg.

Not really, it can be caused on device and also server's end.

@Dev it's getting stuck in 'Authenticating' loading screen.

[Draebatad]

We've upgraded the disks in the server, and applied some system settings that should improve performance but we are still seeing long load times for many players. We are continueing to work on the problem. I will post here again as soon as we have new information.

Moved PL from the cloud to co-located servers in a local datacenter?

[Waug]

And it's back, and better though it would be too early to say that.

[TzatOps]

We believe the problem to be fixed now, please post here in this thread if you see long load times or abnormal delays when changing maps.

[TzatOps]

Moved PL from the cloud to co-located servers in a local datacenter?

from ec2 classic to vpc, with some upgrades to memory and disk io.

[Draebatad]

from ec2 classic to vpc, with some upgrades to memory and disk io.

Nice :)

TzatOps and everyone in the team, thank you for your hard work all day and most of tonight for getting PL back on-line. We truly appreciate it!

Everyone in my guild and everyone I've spoken to reports that PL is back to normal.

[Remiem]

Big round of applause for our Ops guys who literally worked day and night to get the game back up and running right! Those guys are some of the hardest working people at STS, no exaggeration.

[Bazinga!]

We believe the problem to be fixed now, please post here in this thread if you see long load times or abnormal delays when changing maps.

Pl has been updated. Now I'm having long loading time when changing maps and also really high ping / no ping / lag spikes in the map itself.

EDIT:
On computer I am still facing these problems. I tried on Android, everything is working perfectly fine.

[Krato5]

Hey everyone!

We just wanted to provide you with a little more information about the issues that were experienced with PL yesterday. We performed an upgrade of the database and operating system on the database servers. Once we started up the game, we noticed that our database was performing poorly and under an unusually high load. We also noticed that players were experiencing significantly long load times. This was a result of our database queries taking an extremely long amount of time to complete. We performed several tasks, such as, upgrading the volumes of the database to obtain higher IOPS and executing some clean up/performance optimization actions on the database tables, in an attempt to resolve the problem. Ultimately, the root cause of the problem was that our database was executing a certain query in a very inefficient manner. Instead of using an index scan on our tables, which completed the query in seconds, it used a sequential scan that took minutes to complete. In order to resolve this issue, we made a configuration change that would tell the database to prefer an index scan when executing queries. Once that was implemented, our queries finished quickly and performance returned to normal.

Just want to say thank you all for being patient with us! Apologies for the extremely long downtime, as I know it must have been frustrating having the game down for so long. Thanks again for bearing with us while we worked to find a solution! Have fun playing PL!

[wvhills]

Hey everyone!

We just wanted to provide you with a little more information about the issues that were experienced with PL yesterday. We performed an upgrade of the database and operating system on the database servers. Once we started up the game, we noticed that our database was performing poorly and under an unusually high load. We also noticed that players were experiencing significantly long load times. This was a result of our database queries taking an extremely long amount of time to complete. We performed several tasks, such as, upgrading the volumes of the database to obtain higher IOPS and executing some clean up/performance optimization actions on the database tables, in an attempt to resolve the problem. Ultimately, the root cause of the problem was that our database was executing a certain query in a very inefficient manner. Instead of using an index scan on our tables, which completed the query in seconds, it used a sequential scan that took minutes to complete. In order to resolve this issue, we made a configuration change that would tell the database to prefer an index scan when executing queries. Once that was implemented, our queries finished quickly and performance returned to normal.

Just want to say thank you all for being patient with us! Apologies for the extremely long downtime, as I know it must have been frustrating having the game down for so long. Thanks again for bearing with us while we worked to find a solution! Have fun playing PL!

ok thanks! that's what i thought was the problem!

[Veryhihpriest]

How come my game won't go to connect server? It just kicks me out?