Asakai Post-Mortem: An Interview with CCP Veritas

The Asakai battle was huge in terms of both ISK lost and pilots involved. As a result, it was also undisputedly laggy, as pilots on both sides have confirmed. By and large, however, the server did not break; instead, it channeled the little engine that could and kept chugging along. We reached out to CCP for some answers to questions about the server in relation to the fight. CCP Veritas, of Team Gridlock, the team that coded TiDi, was kind enough to set aside the time to reply.  

(Author's note: this interview took place via email.  Special thanks to CCP Manifest for setting this up.)

Firstly, how long did it take CCP to become aware of the fight, after it had started? A good metric would be how many people were in system at the time.  

No idea, I was asleep ;) The red flag went up at 03:20 from Customer Service, which is about an hour after festivities had begun. That isn’t indicative of when it was noticed, it’s just when the decision was made that it was an event worth waking people up about.

Is there an automated system in place that alerts someone that a node is being overloaded? Or is there some other way that CCP found out?

GMs monitor the cluster with an internal tool. Given that there’s not a whole lot we can do once a node has become superloaded, the need for big flashing lights and such just isn’t really there.

Once CCP was aware that something major was occurring, what steps did they take to help with the lag?

Once it was clear that the fight was large, in charge, and not going anywhere, we took the only action that we really have – we moved other solar systems away from that node. Moving a system like this disconnects everyone in it, so moving the fight system itself isn’t acceptable, as those in the non-favorable position simply won’t log back in. This didn’t make a big difference on the performance of the fight, as those systems were mostly empty, but it did at least make those other systems fine after the move.

There is a rumor that CCP took down the Caldari DUST server to attempt to help the lag on the fight. Is this true?

Nope, DUST 514 servers were unaffected by the fight. The DUST team was conducting some in-beta server tweaks though this weekend, although they were unrelated.

During the fight, TiDi was maxed the whole time at 10%. Is there a reason it’s capped at 10% and not lower?  (It certainly felt a lot slower than some of the other 10% fights I had been in)

The 10% cap is there to cover cases where the kind of load that TiDi doesn’t alleviate becomes dominate. Consider that the rate of commands isn’t necessarily slowed down by TiDi – you can still double click around in space or do orbits and approach and such as you normally can. The server needs to still process those as they happen – so slowing down TiDi doesn’t really help that. Things like that are usually a small amount of load, but as the numbers get crazy, it adds up and becomes significant. Capping TiDi to 10% allows the game to advance, ever so slowly, but at least advance. Without that cap, it’s possible to get into a situation where no amount of time slowing fully allows the server to catch up, which leads to an unresolvable situation.

The reason this fight felt slower than other 10%-type fights is that this is the most load we’ve seen on a server since Time Dilation was introduced. Congratulations to all involved!

There was a titan that was lost to ‘black screen,’ where they crashed and were not able to log in and thus were helpless. Their ship, due to the changes made was still in space, however, getting shot. Is CCP planning on addressing this issue in the future, perhaps putting logins to the front of the queue on calls to the server?

No. Prioritizing one type of load over another is very risky business, and something we don’t have architectural support for currently. I’d much rather spend time optimizing things so we’re past the 10% cap less frequently or looking at making the login process more tolerant of delays. Most likely the optimization bit, as that benefits more situations.

CCP did a lot of work with missiles and their performance on the server which improved the way they work on Tranquility. Are there any plans to do the same with drones?

There are no plans currently to specifically target drones for optimization. There are plans to attack fundamental performance bottlenecks which will help lots of things, including drones.

The CFC Skyteam claims that Low Sec has the worst hardware out of all of EVE, is this true?


Currently, it is only possible to reinforce a node at downtime. In the aftermath of this fight, Rydis, the CFC Sky Marshal said that in the future strategic assets, i.e. supercapitals, will not be committed to an unreinforced node in the future, no matter what. The implication is that spontaneous fights of this nature will almost certainly not occur in the future. The player base seems to like these spontaneous fights. Is CCP going to be able to do anything to allay the concerns of supercapital fleet commanders to the point where they will be committed, even if the Fleet Fight Notification Form was not filled out in advance?

If that’s how the players want to play the current situation that’s up to them. It seems like a strange position to make publically though, as it allows opposition to put their supers out without risking CFC counter drops.

If there are specific concerns, those should be brought up.

I’ve seen the graph on ships destroyed in Asakai. Would there be a way to get a graph with the number of pilots in system, perhaps broken down by class (titans, supercarriers, carriers, dreadnoughts and subcapitals) over time?

We’ve got a dev blog coming out today with lots of graph porn.

How does CCP feel their server and their team handled the load that was put on it?

Customer Service and Operations reacted perfectly, taking appropriate action at the correct time.

The server could of course handle things better.  That can be said of any occasion where Time Dilation has to kick in though.  It’s obvious that this event was more load than we can comfortably handle, even with the TiDi cushion in place.

Does CCP have anything they wish to say about the fight?

Hurray for things being blown up!  [Emphasis theirs.]

I concluded by thanking CCP for an amazing Eve experience and expressed my best wishes for their efforts in the future, because fights like these are only going to get bigger. Thanks to CCP Veritas for taking the time to get back to me and for all his work so far in improving server performance for the playerbase.  It is not a stretch to say that without the hard work Team Gridlock and the rest of CCP has done in improving the server, the Asakai battle would have ended in an hour-with a black screen for everyone involved.

On hiatus. On Twitter @AlizabethVea