Asakai Post-Mortem: An Interview with CCP Veritas

Avatar
I see people complain about TiDi and lag and such. They have NO clue that without TiDi these kinds of fights would be simply impossible. I have been around a long time and remember large scale battles which would be one side not loading at all, resulting in a turkey shoot. No fun. And of course trying to make critical decisions during a slide show. No fun.With TiDi, everyone is on an even playing field with command and the server. TiDi deserves much praise and I'm very happy with it. Sure explore ways to lessen the need for TIDi, but so far I love it. Great job CCP.
Avatar
I'm not really sure how the system works but isn't a reinforced node simply a server with better hardware than the normal server in the cluster + it just handles one solar system? If that is true isn't it "just" a matter of upgrading more servers to make fights like this go more smoothly? I'm probably wrong somewhere beacause it soounds too easy :-P
Avatar
The existence of Tidi got a fight that made headlines (and the Eve subreddit is full of wide-eyed newbies drawn into the game from it) as well as an article on penny-arcade about tidi technology. I cannot stress enough how good this situation has worked out for Eve players and CCP alike.
Avatar
CFC decides to not comit supers to not-reinforced systems and makes such fights imposible in future, but does CCP care? Nope, better blame CFC for missed opportunity. Typical reaction, which haven't changed for last 5-6 years. Probably the only case where manufacturer doesn't care for its customer reaction...and CCP stop making excuses of missing server functionality, virtualization has been arround for years you can simply add more server power for this software node you use. It is done by a mouse click for god sake! Invest in some eve hardware and stop redirecting eve money to fail project as dust!
Avatar
People see fights of almost three thousand and look at their (whatever PVP game) and realize that it doesn't even come close on scale. Small gang/solo PVP has it's place, but the huge fleet fights are what makes news and what gets lots of people into this game. The more CCP works at making giant battles playable and enjoyable, the better for the game.
Avatar
No, you're right. It's just not economically feasible and entirely unnecessary to do this 'just incase'.
Avatar
Just wanna say that Veritas and his team are heros. Before TiDi the node would have black screened, ate a few billion ISK for no known reason, sent half the participants home in clones and then crashed. GM Horse would still be reading petitions for reimbursements. Sure, in an ideal world we wouldn't need it, but TiDi makes the game playable and lets us push the limits of what is possible in an MMO. I'm looking forward to see what these guys can pull out to make things even better in future.
Avatar
Two words ....Single. Threaded.now f*** off
Avatar
I would say in an idea world we will need it (TiDi). The I hope that the scale of battles continues to grow at the pace of hardware. So, when CCP gets a server that can handle three thousand ship fights with no sweat, I hope the community has moved to the ten thousand ship fights.
Avatar
of all the mistakes CFC skymashals did that day, commiting caps to a unreinforced node(which after all is even playing ground) was the least of its mistakes.
Avatar
Guess server upgrades comes natually with time, so an ever increasing capacity will occur.
Avatar
For once i'm not bitter about CCP. When it comes to mass-management they are still lightyears ahead of everyone else in the business. What interests me about this article instead is their still continued defiance to look at the lag-issue from any other angle and that no one seem to elaborate properly on the commonly hailed interjection that this was a spontaneous fight.It was a botched Titan-bridge turned supercapital-gank that escalated, everyone keeps saying it, but no one seem to elaborate on the fact that even with (well-implemented) TiDi in place we seem to have found the mass-cap by now (some 3k players) and the current political climate allow us to spontaneously break that. It took a couple of hours, but what happens when we have pre-planned timers in peak-hour timezones where both EU and US can play? This was a spontaneous even well into US TZ and way past EU prime. One could even argue it was past US/E prime.So what am i saying here?Anyone who complain about TiDi or server performance in large-mass gameplay of EVE in general, they are of course idiots. We play in thousands other games can't even handle hundreds, some can hardly handle tens.However, anyone who settles for calling out people complaining about TiDi and reminding them how it would be without TiDi also seem to avoid the obvious issue that we (the community) are breaking 100% load and the second layer of full TiDi with relative ease already...... and those who recognize that do not elaborate on it and discuss the issue at depth. This includes CCP.
Avatar
GSF committing caps to an unreinforced node kicking it into 10% tidi gave N3 et al a lot of time to burn across the map (iirc someone mentioned they had to use 6 cynos) while not much at all was happening in the battle.As the availability of reinforcements was not equal the playing ground favored HBC/N3.
Avatar
Enough good words cannot be said about TiDI. Anyone who looks down on it never truly played Blackscreen Online.
Avatar
Correct, this negated the CFC home field advantage. Asakai was only one or two jumps from the GSF capital staging system and close to FA's as well.
Avatar
Hardware-wise, it's not feasible to fit the hardware that will relatively solve these problems, it's not something game companies can afford but yes, as the computer industry progresses, it becomes more feasible over time, but until it's feasible, other technologies like TiDi need to be made to compensate till that point. It's also why EVE's physics isn't newtonian, there is no feasible hardware at the moment to support a 2-3k battle with newtonian physics, not even 500 or so. Fortunately CCP plans to continuously iterate EVE way into the future, so as technology progresses, so does EVE. :)
Avatar
Pre-planned timers imply sufficient notice to get a node reinforced
Avatar
I'm sorry CCP Veritas, but you're wrong on one point; there is ALWAYS a need for big flashing lights.And klaxons.And battle-stations red lighting.
Avatar
No, its not that simple....The servers are already pretty good. Spending more for better hardware will not give significant gains for the money spent.The fundamental issue is that modern processors expect that code can be run in "parallel" ie. two unrelated calculations can be done at the same time in two different CPU cores - thereby halving the time required to run a program. That's why you are buying dual core, quad core processors etc. But processor core speeds themselves are no longer increasing significantly.Eve's problem is that the vast majority of the calculations that must be done cannot be done in parallel. They must be done sequentially on a single CPU. This is what we mean when we say "single threaded". Changing this requires a fundamental re-write of the core server code in a new language with new formulas for everything from DPS to missile trajectories, to bombs, to skills, to lazors, in fact to Everything in the physics engine - and it simply is not possible to do this and end up with a new server that works the same as the existing one. You would basically be throwing away 10 years of development and starting again from zero.Team Gridlock have written some pretty good blogs on how they're addressing these issues - including re-writing core parts of the server code to make at least some of the calculations run in parallel. But progress is always going to be slow.
Avatar
TiDi is an amazing piece of technology, That being said massive complaints from the playerbase are what led to its creation.EVE players should both applaud CCP for doing technologically amazing shit, while also pressuring them for further improvements. The fact that no other MMO comes within an order of magnitude of EVE's mass battle potential says a lot, but player whining was a big part of what pushed CCP in this direction
Avatar
"It seems like a strange position to make publically though"CCP Veritas? More like CCP FuckingBurn
Avatar
" There was a titan that was lost to ‘black screen.’ " I remember getting a 'black screen' myself. Had to close client and relog, took like 10-15 minutes, like everyone else. This dude died not to a 'black screen' but to being probed and aggressed. I had the same client issues and I still got this nerd's titan kill-mail. Nice Try.
Avatar
Afaik It was DBRB, not Rydis that commited supercaps.
Avatar
CCP already has some very impressive hardware behind Tranquility, I don't understand your bitterness towards CCP as the game itself has improved in leaps and bounds since the Bad-Old-Days when the game was nearly unplayable with 200 in local.This is just one instance where things really got escalated, Veritas said that this was the heaviest load on the servers since TiDi's inception, so this isn't some common event, just a one-off stress test.
Avatar
"The CFC Skyteam claims that Low Sec has the worst hardware out of all of EVE, is this true?Nope."In CCP Quant's Devblog. CCP Masterplan chimes in with "...the Crimewatchsystem in low-sec adds a significant amount of load to the server"Which may explain why the CFC Skyteam thought lowsec suffered from poor hardware.
Avatar
I can confirm, those systems they moved away from the node were not empty. In a TS with 30 people frantically clicking their character trying to log back in and jump the gate.
Avatar
but that CCP did a great job, regardless.
Avatar
The mistake was not fighting in lowsec, but using titans in lowsec.....If CFC thinks taking fights is a mistake then so be it, but taking the fight itself wasn't the mistake made that day.
Avatar
Rydis was out walking blind poodles. I'm not making this up.
Avatar
agreed, that and inadequate hic fleet and bad target calling (focusing supers before the much lighter dread/carrier fleet). If you expect to only 30-40min of shooting time it makes sence to focus the supers for maximum damage right away, however if you are planning to win the battle itself you wanna get rid off all the dreads and carriers which CFC didnt, thus over time this would come back to haunt them, as 50 dreads do the same dps as 50 supers while beeing beeing 1/10 as easy to kill, but since they got left alone ,the HBC dreads(and bl dreads) could provide supercap like dps while the CFC dread fleet got blown up. Same with carriers, supricingly few HBC/n3 carriers dies, while they accually rep more than your avarage supercarrier.(god knowns how many that accually triaged).It may not sound that important in a supercap slugfest , but over time it does matter, specially when your starting to loose.
Avatar
I mean dreads have prob 1-2million ehp with blap fit. cant be remote repped assuming they triage and do about 10k dps. vs a supercarrier with 35million ehp , good resist and is gonna recive tons of remote reps and only do 10k dps if its a nyx.
Avatar
siege*
Avatar
LXQ2 Never Forget.
Avatar
To add to what some others have said (especially Bitter Vet), the rank & file servers that CCP uses are well beyond "pretty good". The reinforcement servers are top of the line pimped out machines that would make any hardware geek giddy. So don't think that they are just running a bunch of crappy pizza boxes and calling it good.
Avatar
CCP Meta-tas
Avatar
The difference being that if this happens to a subcap, you aren't the immediate priority of every prober in system. If it happens to a super, you're dead. Everything with a probe launcher fitted is immediately trying to get a warpin, and without tank modules even a titan goes down fast.Is this a problem CCP can fix - not really. Is it something that is a much bigger worry for a titan than say, a loki, yes it is.
Avatar
HBC carriers were not triage, IIRC
Avatar
What could possibly go wrong?
Avatar
In an interview with the mastermind behind Time Dilation, CCP representatives refer to an event that resulted in the destruction of virtual assets worth approximately $20k USD as "festivities". Love this game.
Avatar
Nothing. I'd get some more supercap kills, get to write an awesome article, again. Actually, yes. Can someone please make this happen? (Remember in Tomorrow Never Dies where the bad guys starts a war to sell newspapers. Same thing)
Avatar
and your issue with this is?
Avatar
I actually requested the number of fighter bombers and fighters used in the fight. Unfortunately, CCP was not able to get that number to me.
Avatar
That would be interesting. Its a completely different dynamic on server load with fighter bombers for some reason. I doubt Veritas will really get into it due to him still being sad about brain in a box getting shittcanned. During tribute, it was pretty noticeable on a day when we flipped 6 stations. I use that as my benchmark. He's a smart guy, but he really doesn't talk about certain things like that. I would love for him to go over what happens, as far as server interaction goes, specifically when a fleet launches fighterbombers.
Avatar
I was there (on the other side I would assume from the current local) but epic times were had by all. The only fight I was ever a part of the that felt more epic than that was the battle for FAT-32 during the IAC/GOON/RA vs BOB/MC/GBC era of the first great war. I know we all hate the old style doomsday but that was one time where fleets rose to the challenge and fought it out medieval-style.
Avatar
PS I know the lag sucked but I for one looked wide eyed as I finally warped into the egagement (before all hell broke loose) and thought; this is why video games were invented.
Avatar
They took all out network admin toys like that away when we all got smart phones :(
Avatar
Question from someone doesnt completely understand - why arent all nodes "reinforced"?CCP removed other systems from the overloaded unreinforced node.assuming a normal node can hold say 30 systems under normal load. and a reinforced node could hold say 300 systems. having all nodes "reinforced" normally would allow CCP to remove the 299 uninvolved systems from the overloaded but pre-reinforced node. instead of removing 29.
Avatar
They should have the same 'red alert' event like in Star Trek at their houses.

The Asakai battle was huge in terms of both ISK lost and pilots involved. As a result, it was also undisputedly laggy, as pilots on both sides have confirmed. By and large, however, the server did not break; instead, it channeled the little engine that could and kept chugging along. We reached out to CCP for some answers to questions about the server in relation to the fight. CCP Veritas, of Team Gridlock, the team that coded TiDi, was kind enough to set aside the time to reply.  

(Author's note: this interview took place via email.  Special thanks to CCP Manifest for setting this up.)

Firstly, how long did it take CCP to become aware of the fight, after it had started? A good metric would be how many people were in system at the time.  

No idea, I was asleep ;) The red flag went up at 03:20 from Customer Service, which is about an hour after festivities had begun. That isn’t indicative of when it was noticed, it’s just when the decision was made that it was an event worth waking people up about.

Is there an automated system in place that alerts someone that a node is being overloaded? Or is there some other way that CCP found out?

GMs monitor the cluster with an internal tool. Given that there’s not a whole lot we can do once a node has become superloaded, the need for big flashing lights and such just isn’t really there.

Once CCP was aware that something major was occurring, what steps did they take to help with the lag?

Once it was clear that the fight was large, in charge, and not going anywhere, we took the only action that we really have – we moved other solar systems away from that node. Moving a system like this disconnects everyone in it, so moving the fight system itself isn’t acceptable, as those in the non-favorable position simply won’t log back in. This didn’t make a big difference on the performance of the fight, as those systems were mostly empty, but it did at least make those other systems fine after the move.

There is a rumor that CCP took down the Caldari DUST server to attempt to help the lag on the fight. Is this true?

Nope, DUST 514 servers were unaffected by the fight. The DUST team was conducting some in-beta server tweaks though this weekend, although they were unrelated.

During the fight, TiDi was maxed the whole time at 10%. Is there a reason it’s capped at 10% and not lower?  (It certainly felt a lot slower than some of the other 10% fights I had been in)

The 10% cap is there to cover cases where the kind of load that TiDi doesn’t alleviate becomes dominate. Consider that the rate of commands isn’t necessarily slowed down by TiDi – you can still double click around in space or do orbits and approach and such as you normally can. The server needs to still process those as they happen – so slowing down TiDi doesn’t really help that. Things like that are usually a small amount of load, but as the numbers get crazy, it adds up and becomes significant. Capping TiDi to 10% allows the game to advance, ever so slowly, but at least advance. Without that cap, it’s possible to get into a situation where no amount of time slowing fully allows the server to catch up, which leads to an unresolvable situation.

The reason this fight felt slower than other 10%-type fights is that this is the most load we’ve seen on a server since Time Dilation was introduced. Congratulations to all involved!

There was a titan that was lost to ‘black screen,’ where they crashed and were not able to log in and thus were helpless. Their ship, due to the changes made was still in space, however, getting shot. Is CCP planning on addressing this issue in the future, perhaps putting logins to the front of the queue on calls to the server?

No. Prioritizing one type of load over another is very risky business, and something we don’t have architectural support for currently. I’d much rather spend time optimizing things so we’re past the 10% cap less frequently or looking at making the login process more tolerant of delays. Most likely the optimization bit, as that benefits more situations.

CCP did a lot of work with missiles and their performance on the server which improved the way they work on Tranquility. Are there any plans to do the same with drones?

There are no plans currently to specifically target drones for optimization. There are plans to attack fundamental performance bottlenecks which will help lots of things, including drones.

The CFC Skyteam claims that Low Sec has the worst hardware out of all of EVE, is this true?

Nope.

Currently, it is only possible to reinforce a node at downtime. In the aftermath of this fight, Rydis, the CFC Sky Marshal said that in the future strategic assets, i.e. supercapitals, will not be committed to an unreinforced node in the future, no matter what. The implication is that spontaneous fights of this nature will almost certainly not occur in the future. The player base seems to like these spontaneous fights. Is CCP going to be able to do anything to allay the concerns of supercapital fleet commanders to the point where they will be committed, even if the Fleet Fight Notification Form was not filled out in advance?

If that’s how the players want to play the current situation that’s up to them. It seems like a strange position to make publically though, as it allows opposition to put their supers out without risking CFC counter drops.

If there are specific concerns, those should be brought up.

I’ve seen the graph on ships destroyed in Asakai. Would there be a way to get a graph with the number of pilots in system, perhaps broken down by class (titans, supercarriers, carriers, dreadnoughts and subcapitals) over time?

We’ve got a dev blog coming out today with lots of graph porn.

How does CCP feel their server and their team handled the load that was put on it?

Customer Service and Operations reacted perfectly, taking appropriate action at the correct time.

The server could of course handle things better.  That can be said of any occasion where Time Dilation has to kick in though.  It’s obvious that this event was more load than we can comfortably handle, even with the TiDi cushion in place.

Does CCP have anything they wish to say about the fight?

Hurray for things being blown up!  [Emphasis theirs.]

I concluded by thanking CCP for an amazing Eve experience and expressed my best wishes for their efforts in the future, because fights like these are only going to get bigger. Thanks to CCP Veritas for taking the time to get back to me and for all his work so far in improving server performance for the playerbase.  It is not a stretch to say that without the hard work Team Gridlock and the rest of CCP has done in improving the server, the Asakai battle would have ended in an hour-with a black screen for everyone involved.

Writer. Twitter: @AlizabethVea