Update on Arcane Legends stability issues

[null_void]

Hello all - I thought I'd take a moment to let you know what steps we're currently taking in order to address the disconnection issues in Arcane Legends that are impacting players. On one hand, we want to fix these issues to the best of our ability, and as quickly as possible. On the other hand, some of these changes are rather invasive, and we want to make sure that our fixes do not create any unintended side-effects that would impact your play experience further. In addition to the disconnect problem, a number of you have noticed that the overall bandwidth usage of the game has steadily increased over time. As you may suspect, these two problems are not unrelated. So without further ado, here's what we've done and what we're working on.


Server-to-client message buffering - Until recently, the server would send information to the game client multiple times per server frame (any time a game system decided it needed to send a message). We've switched the server to use a buffered mode, meaning that the server will send out messages to the client once per frame. This is a somewhat temporary fix to prevent a lot of the disconnects that we were originally seeing. The deal was that under certain circumstances, the server would send so many individual messages that the client could not acknowledge them quickly enough, leading to disconnects. We were not able to determine whether this inability to keep up with these ACKs was due to some sort of intermediate throttling or a result of overall bandwidth issues, but the easiest (and quickest) solution was to reduce the number of messages.
Server-only effects - We've done some analysis on the types and numbers of game messages we're sending from the server to the client. This has lead us to give things a once over to ensure that those effects that are purely "logical" rather than having client-facing effects that are marked as server-only. You may have noticed some of the experiments over the past week - cases that we've had to fix where client effects weren't showing up even though the skills were working correctly on the server. We believe we've ironed out most of those issues at this point.
Mob clusters - The number of enemies in any given fight tends to be a multiplier on the network traffic that has to be sent from the server to the client. When an enemy is impacted by a skill you use, the server has to notify all observing clients of this effect (so that they can display visual effects or overhead text, etc). This means that if you can group up two dozen enemies and you have a full party, the server has to send down lots of data to every nearby player. A solution we're exploring to this is to prevent mobs from grouping up quite as much. This is accomplished by reducing leash ranges and altering the placement of enemy groups so that their ranges don't overlap quite as much. As you can probably guess, this sort of rearrangement has an impact on gameplay, so we're going to be very careful with it. We're going to start with the reorganization of a couple of zones, and push that change out to you guys after it's been through our internal QA process. Once we've received feedback on those changes, we'll measure their effectiveness and (assuming it's a good effect), start making changes to other zones that experience this problem. This will be an ongoing effort, so don't expect every single zone to change at once. We will maintain open communication with you all during this process to measure its impact on overall gameplay as we go.
Network traffic compression - One reasonably easy way to reduce bandwidth usage is to compress the outgoing traffic from the server. It's not a magic bullet (at least, we're not expecting it to be one), since compression requires additional processing on both the client and server end. There are, however, lightweight compression mechanisms that we're experimenting with. This sort of change will require a new client and it's something we want to be pretty careful with, so it's going to need a hefty QA pass before we can push it out to you all.


I hope that gives you some insight into what steps we're taking to deal with bandwidth issues in general. The next question you might have is, "so when will it be fixed?" As I've mentioned these are pretty invasive changes that will require time to test and deploy. We don't want to be patching up issues with chewing gum and paper clips. So, we will be rolling out changes incrementally as they are ready to go. The short answer is that we will take as long as we need to make sure we're fixing the issue without any further impact on your gameplay, and will make sure to keep you updated along the way. We appreciate your feedback, and especially the efforts that certain players have made such that the problem was brought to our attention in the first place. I'll keep you updated!

[Puppys]

First lmao

Nice to see this is getting fixed :)

Thanks null

[Azemen]

2nd!

[Artoholics]

Nice to hear that [emoji4] Thank you for noticing and correcting it

Gesendet von meinem GT-I9300 mit Tapatalk

[Kakashis]

That's great, but you'll be putting shorter leashes on already short ranges? It will literally take an eternity to finish elite maps

[Valkiryas]

Out of thread, Someone el se is having lag inssues? My ping can't down ... over 1500 always

[bruceboster]

When Mob crowded, i cant touch everything.the game crashed and slow motion.

[marfibagheera]

Ty null ;-)

[ColdBlahd]

Hot.

[Consequently]

Please fix this asap. /:
Disconnected twice while fighting Syrillax.

[GoodSyntax]

To be honest, I'm a bit surprised that you aren't already using compression. Gzip or LZW are lightweight enough that I think most platforms can handle it.

I agree that there is an overload of information, and I think it is a good approach to reduce the number of messages in large pulls. Perhaps a fair middle ground would be to communicate status ailments received player dmg and aggro target in real time, with AA, positioning, mob dmg received etc. in a buffered, bursting fashion.

My concern is that major changes to leash lengths will have a dramatic, adverse impact on elite strategies. These newer maps are long, require many mob kills and take a considerable amount of time to complete. As it stands now, planar tombs are relatively slow runs, even in pro parties. To reduce leash lengths could easily double the run time, resulting in higher costs in elixirs (in that players can't complete as many maps on he same amount of time).

Another reasonable alternative could be to reduce the feedback rate as mob count increases. To be honest, if I have 30 mobs on screen, I don't care who is affected by Nox, who is rooted, which ones are under curse, who has been snared, etc. This type of regressive feedback would help scale bandwidth without hurting gameplay.

I appreciate the new sense of transparency! I'm sure we can all work together towards a viable solution that everyone can be happy with.

Thanks Null!

[Carapace]

That's great, but you'll be putting shorter leashes on already short ranges? It will literally take an eternity to finish elite maps

This post, along with parts of Good Syntax's post are exactly why Null talks about these being incremental changes, testing in QA, and open discussions with the community. We want to preserve the integrity while also reducing message sending overloads and improving overall network traffic.

As an initial pass we're going to play with things like smaller group sizes, that are more typical of a "pull" scenario in other MMO's. Groups of 3-5 for example, that may overlap with one other group of 3-5. On top of that not allowing these mobs to run as far as they were before may have an impact as well. We can tune and scale as necessary to determine best practices moving forward that still accommodate the intended design and flow of an elite map. Not being able to do immense pulls is an example of how the community plays these maps, and an important game play component we would like to keep from changing the experience of too much if that is what the community would like to see. In some cases it may not be possible to maintain a 1:1 experience while also improving our network traffic.

As Null mentions, it will take some time to do and require internal testing on top of other things and we will communicate as it becomes community relevant to do so.

Thanks all!
- Carapace

[Mehmet Tuncay]

[arcanexmen]Hello all - I thought I'd take a moment to let you know what steps we're currently taking in order to address the disconnection issues in Arcane Legends that are impacting players. On one hand, we want to fix these issues to the best of our ability, and as quickly as possible. On the other hand, some of these changes are rather invasive, and we want to make sure that our fixes do not create any unintended side-effects that would impact your play experience further. In addition to the disconnect problem, a number of you have noticed that the overall bandwidth usage of the game has steadily increased over time. As you may suspect, these two problems are not unrelated. So without further ado, here's what we've done and what we're working on.


Server-to-client message buffering - Until recently, the server would send information to the game client multiple times per server frame (any time a game system decided it needed to send a message). We've switched the server to use a buffered mode, meaning that the server will send out messages to the client once per frame. This is a somewhat temporary fix to prevent a lot of the disconnects that we were originally seeing. The deal was that under certain circumstances, the server would send so many individual messages that the client could not acknowledge them quickly enough, leading to disconnects. We were not able to determine whether this inability to keep up with these ACKs was due to some sort of intermediate throttling or a result of overall bandwidth issues, but the easiest (and quickest) solution was to reduce the number of messages.
Server-only effects - We've done some analysis on the types and numbers of game messages we're sending from the server to the client. This has lead us to give things a once over to ensure that those effects that are purely "logical" rather than having client-facing effects that are marked as server-only. You may have noticed some of the experiments over the past week - cases that we've had to fix where client effects weren't showing up even though the skills were working correctly on the server. We believe we've ironed out most of those issues at this point.
Mob clusters - The number of enemies in any given fight tends to be a multiplier on the network traffic that has to be sent from the server to the client. When an enemy is impacted by a skill you use, the server has to notify all observing clients of this effect (so that they can display visual effects or overhead text, etc). This means that if you can group up two dozen enemies and you have a full party, the server has to send down lots of data to every nearby player. A solution we're exploring to this is to prevent mobs from grouping up quite as much. This is accomplished by reducing leash ranges and altering the placement of enemy groups so that their ranges don't overlap quite as much. As you can probably guess, this sort of rearrangement has an impact on gameplay, so we're going to be very careful with it. We're going to start with the reorganization of a couple of zones, and push that change out to you guys after it's been through our internal QA process. Once we've received feedback on those changes, we'll measure their effectiveness and (assuming it's a good effect), start making changes to other zones that experience this problem. This will be an ongoing effort, so don't expect every single zone to change at once. We will maintain open communication with you all during this process to measure its impact on overall gameplay as we go.
Network traffic compression - One reasonably easy way to reduce bandwidth usage is to compress the outgoing traffic from the server. It's not a magic bullet (at least, we're not expecting it to be one), since compression requires additional processing on both the client and server end. There are, however, lightweight compression mechanisms that we're experimenting with. This sort of change will require a new client and it's something we want to be pretty careful with, so it's going to need a hefty QA pass before we can push it out to you all.


I hope that gives you some insight into what steps we're taking to deal with bandwidth issues in general. The next question you might have is, "so when will it be fixed?" As I've mentioned these are pretty invasive changes that will require time to test and deploy. We don't want to be patching up issues with chewing gum and paper clips. So, we will be rolling out changes incrementally as they are ready to go. The short answer is that we will take as long as we need to make sure we're fixing the issue without any further impact on your gameplay, and will make sure to keep you updated along the way. We appreciate your feedback, and especially the efforts that certain players have made such that the problem was brought to our attention in the first place. I'll keep you updated!

[CAK Vader]

[arcanexmen]Hello all - I thought I'd take a moment to let you know what steps we're currently taking in order to address the disconnection issues in Arcane Legends that are impacting players. On one hand, we want to fix these issues to the best of our ability, and as quickly as possible. On the other hand, some of these changes are rather invasive, and we want to make sure that our fixes do not create any unintended side-effects that would impact your play experience further. In addition to the disconnect problem, a number of you have noticed that the overall bandwidth usage of the game has steadily increased over time. As you may suspect, these two problems are not unrelated. So without further ado, here's what we've done and what we're working on.


Server-to-client message buffering - Until recently, the server would send information to the game client multiple times per server frame (any time a game system decided it needed to send a message). We've switched the server to use a buffered mode, meaning that the server will send out messages to the client once per frame. This is a somewhat temporary fix to prevent a lot of the disconnects that we were originally seeing. The deal was that under certain circumstances, the server would send so many individual messages that the client could not acknowledge them quickly enough, leading to disconnects. We were not able to determine whether this inability to keep up with these ACKs was due to some sort of intermediate throttling or a result of overall bandwidth issues, but the easiest (and quickest) solution was to reduce the number of messages.
Server-only effects - We've done some analysis on the types and numbers of game messages we're sending from the server to the client. This has lead us to give things a once over to ensure that those effects that are purely "logical" rather than having client-facing effects that are marked as server-only. You may have noticed some of the experiments over the past week - cases that we've had to fix where client effects weren't showing up even though the skills were working correctly on the server. We believe we've ironed out most of those issues at this point.
Mob clusters - The number of enemies in any given fight tends to be a multiplier on the network traffic that has to be sent from the server to the client. When an enemy is impacted by a skill you use, the server has to notify all observing clients of this effect (so that they can display visual effects or overhead text, etc). This means that if you can group up two dozen enemies and you have a full party, the server has to send down lots of data to every nearby player. A solution we're exploring to this is to prevent mobs from grouping up quite as much. This is accomplished by reducing leash ranges and altering the placement of enemy groups so that their ranges don't overlap quite as much. As you can probably guess, this sort of rearrangement has an impact on gameplay, so we're going to be very careful with it. We're going to start with the reorganization of a couple of zones, and push that change out to you guys after it's been through our internal QA process. Once we've received feedback on those changes, we'll measure their effectiveness and (assuming it's a good effect), start making changes to other zones that experience this problem. This will be an ongoing effort, so don't expect every single zone to change at once. We will maintain open communication with you all during this process to measure its impact on overall gameplay as we go.
Network traffic compression - One reasonably easy way to reduce bandwidth usage is to compress the outgoing traffic from the server. It's not a magic bullet (at least, we're not expecting it to be one), since compression requires additional processing on both the client and server end. There are, however, lightweight compression mechanisms that we're experimenting with. This sort of change will require a new client and it's something we want to be pretty careful with, so it's going to need a hefty QA pass before we can push it out to you all.


I hope that gives you some insight into what steps we're taking to deal with bandwidth issues in general. The next question you might have is, "so when will it be fixed?" As I've mentioned these are pretty invasive changes that will require time to test and deploy. We don't want to be patching up issues with chewing gum and paper clips. So, we will be rolling out changes incrementally as they are ready to go. The short answer is that we will take as long as we need to make sure we're fixing the issue without any further impact on your gameplay, and will make sure to keep you updated along the way. We appreciate your feedback, and especially the efforts that certain players have made such that the problem was brought to our attention in the first place. I'll keep you updated!
And you posted this why?

[Hamid Ahfir]

Haha funny , just for tell us , if we can help them , for play better the mor fast than possible :)

[vawaid]

What's wrong? I only feel 2 dc's a week.

[Zeus]

This post, along with parts of Good Syntax's post are exactly why Null talks about these being incremental changes, testing in QA, and open discussions with the community. We want to preserve the integrity while also reducing message sending overloads and improving overall network traffic.

As an initial pass we're going to play with things like smaller group sizes, that are more typical of a "pull" scenario in other MMO's. Groups of 3-5 for example, that may overlap with one other group of 3-5. On top of that not allowing these mobs to run as far as they were before may have an impact as well. We can tune and scale as necessary to determine best practices moving forward that still accommodate the intended design and flow of an elite map. Not being able to do immense pulls is an example of how the community plays these maps, and an important game play component we would like to keep from changing the experience of too much if that is what the community would like to see. In some cases it may not be possible to maintain a 1:1 experience while also improving our network traffic.

As Null mentions, it will take some time to do and require internal testing on top of other things and we will communicate as it becomes community relevant to do so.

Thanks all!
- Carapace

Alternatively, if developers create shorter leashes, they should also reduce the mob count it takes for a boss to spawn. This way, developers can reduce network traffic and at the same time, maps should take around the same time that they have before (even if it means that pulls are less dense).

[Madnex]

Alternatively, if developers create shorter leashes, they should also reduce the mob count it takes for a boss to spawn. This way, developers can reduce network traffic and at the same time, maps should take around the same time that they have before (even if it means that pulls are less dense).
Exactly! Timed runs are heavily reliant on leashes because larger pulls = faster completion times. If you want to lower the mob density, you have to lower the mob count for boss spawn too.

+1

[vawaid]

New update gave me 2x dc in mobile, on google chrome, no dc.

[Serancha]

Exactly! Timed runs are heavily reliant on leashes because larger pulls = faster completion times. If you want to lower the mob density, you have to lower the mob count for boss spawn too.

+1

Where is the rule that says speed runs "must" be a certain time? Maybe shorter leashes will make people have to actually think and use new techniques. I don't think lowering counts is necessary. Eliminating parking is a dam good idea.

[Madnex]

Where is the rule that says speed runs "must" be a certain time? Maybe shorter leashes will make people have to actually think and use new techniques. I don't think lowering counts is necessary. Eliminating parking is a dam good idea.
It's not really a "must" but it's a large factor in speedrunning on both low and high level maps (as long as we have sorcerers, the more mobs hit by the same AoE skills, the faster the time will be). The techniques are there, not too much strategy apart from picking how many to kill and where though. Parking, although it can be considered as a "cheese" move, is unrelated - not sure if anyone even uses it anymore but no time run has been set involving parking.

[lethaljade]

Where is the rule that says speed runs "must" be a certain time? Maybe shorter leashes will make people have to actually think and use new techniques. I don't think lowering counts is necessary. Eliminating parking is a dam good idea.

Well records are already set so would be unfair to people who try after its added and unfair for board reset so far into season as many have set stable legitimate recs already n worked incredibly hard for them.

[Zeus]

Let us not even talk about records but farming in general. If the farming time per map is increased, players will get frustrated and may not even want to run! So, if developers are going to mess with leashes, then they should also lower the kill count it requires the boss to spawn so I'm not taking more time to get the same reward.

[Serancha]

Well records are already set so would be unfair to people who try after its added and unfair for board reset so far into season as many have set stable legitimate recs already n worked incredibly hard for them.

They'll undoubtedly wipe the boards when they put timers back in anyways.

Wouldn't decreasing kill counts be more unfair? Not sure which way you're arguing.

[nun]

zues is u apoollo?

[Carapace]

Let us not even talk about records but farming in general. If the farming time per map is increased, players will get frustrated and may not even want to run! So, if developers are going to mess with leashes, then they should also lower the kill count it requires the boss to spawn so I'm not taking more time to get the same reward.

Yes, so far in local tests of the current map we're playing with run times increased in speed. The required kills were certainly reduced to accommodate the new number of enemies.

[Zeus]

Yes, so far in local tests of the current map we're playing with run times increased in speed. The required kills were certainly reduced to accommodate the new number of enemies.

Awesome, Cara! Any update on the fossil mystery? Players have run it daily without even one fossil that's been looted from arena.

[GoodSyntax]

awesome, cara! Any update on the fossil mystery? Players have run it daily without even one fossil that's been looted from arena.

^ this!

[raw]

They'll undoubtedly wipe the boards when they put timers back in anyways.

Wouldn't decreasing kill counts be more unfair? Not sure which way you're arguing.

Sera, can you provide more insight on this theory? Are you implying that all of the boards will be wiped when timers are re-implemented? Can a Mod address this...

Thanks

[Madnex]

Sera, can you provide more insight on this theory? Are you implying that all of the boards will be wiped when timers are re-implemented? Can a Mod address this...

Thanks
I'm afraid it means what it means. I'm guessing they will focus on the harder maps from Nordr and up - although Nordr mobs have very small leashes already so not sure if they'll do anything there. Any campaign maps below that will probably remain untouched.