Ever Notice....

[Allia]

ever notice how the dalaran server is always one iof the last few servers to come up after the tuesday maintenance....

ever notice how there are constant problems with our server...
ie. this tuesday night's mass boot from the eastern kingdoms and then the server shutdown???

What;s up?

[LardGibs]

There are constant problems with all the servers, some far more serious than Dalaran. My impression is that Dalaran actually is running on top of the line hardware, and the realms that are in trouble are hardware used for the beta.

This interview was the most revealing as to why, internal bandwith issues with Blizzard's network infrastructure. The PvP doesn't help actually, crashes the continent.




I mentioned last week that we sent a fairly terse set of questions over to Blizzard. They're a lot more direct than developers are used to receiving. The crazy thing is, the answers we got back were a lot more direct than the interviews out there usually receive. Funny, that.

Without further ado:

1. You say that you sold six hundred thousand units. Is the game not capable of supporting this many users?

The short answer is "The game is capable of supporting this many players," but it would probably be helpful to provide some background information. Based on our market analysis, we made some initial calculations about the size of the massively multiplayer online games market in the United States. We then accounted for new customers to the genre based on our previous games. Looking over this data, we did believe that there was the potential for an extremely sizable interest in a Blizzard MMOG. According to our research, other successful MMOGs in the U.S. had achieved roughly 300,000 subscribers after 12 months of operation. What ended up happening with World of Warcraft is that we achieved double these numbers in approximately the first six weeks of launch. We absolutely can support the number of copies we put on shelves, but we believed it would take us longer to get to this number in terms of players purchasing the game and logging on.

We had not anticipated this amount of growth in such a short time; however, we did have a backup plan that was deployed rapidly. In the first week of launch, we more than doubled our number of game servers and server infrastructure to accommodate the demand. The fact that we had planned to grow the service over the first 12 months of operation was evident, as we had server hardware waiting to be deployed. We just anticipated that this server rollout would be gradual. Copies of the game were being purchased at a much faster rate than anticipated, so we had to abandon our slower-paced plan and go into rapid deployment to accommodate these additional customers. This meant we also had to advance our timetable for additional server purchases.

With such a rapid growth of the network, we started to see several bottlenecks in the infrastructure that exposed themselves very quickly when the expanded hardware immediately took on massive load. These bottlenecks were solvable, but they required additional upgrades to the backend systems to accommodate the load—which, again, we hadn’t planned to see, even with the extreme estimates, until later in the year. Regardless, server stability has remained our number-one priority, and so we acquired and deployed even more equipment as part of the process of addressing these issues. All of this new hardware also required additional software and operating system upgrades on the backend. The problems that some players on the 20 or so most populated servers (out of the current total of 88 servers) have been experiencing are related to some of the upgrades not functioning as desired. We are working diligently with our vendors and internal technical staff to get as quick of a resolution to the problems as possible, and we believe there should be noticeable improvements soon. When our community team commented that people are working 24/7, they weren't exaggerating.

2. If it's true that the server problems are related to the overwhelming number of players, why was no effort made to better distribute players evenly across realms, or allow players and guilds to transfer to less populated servers?

We actually did have a number of checks in place at launch to distribute players as evenly as possible across realms. When a new account logged in, the game would ask what realm rule set and time zone the player preferred, and then it would suggest the realm with the lowest population that matched the selected preferences. That said, we're definitely working on resolving the overpopulation problems that ended up occurring on some realms despite our preventative measures. A realm-transfer option that would allow players to move from their high-population realm to one with a low population is one of the things we’re investigating. We’re exploring this option fully and hope to be able to communicate more detailed information about it to our customers in the coming weeks.

3. Currently, large scale player raids involving large groups of players experience a huge amount of latency. How do you plan to compensate for this in your upcoming PvP Battlegrounds feature?

The player raids often have hundreds of people per side in one area; that area is on a server that is also running the rest of the continent, and that can result in the latency you describe—depending, as well, on the total population of that server. We’re continuing to look into the issues surrounding this dip in performance. Battlegrounds, on the other hand, will run on the instance server, so there should be no such issues. Additionally, players will be unable to "zerg" in Battlegrounds; there will be a limit to the number of players per side.

4. What accounts for the frequent "emergency" maintenance downtime? What issues are you attempting to resolve?

The emergency maintenance periods are to restore stability while we continue to narrow down the cause of the problems. Some of them are also to deploy temporary fixes to various in-game systems while we continue to develop a longer term, more stable solution. World of Warcraft delivers many complex features that are unique to MMOGs. Features such as the in-game mail system, auction houses, player inventories, flight paths, quest states, etc. use a lot of server bandwidth, which makes pinpointing problems on the server infrastructure much more complicated.

Recently, the extended emergency downtime for a certain number of realms was needed in order to better accommodate our growing player base. Some of the upgrades that we planned for all of the realms were made to these realms first, as they are among the most populated and thus most in need of aid. We set the realms up on the latest top-of-the-line hardware and made the software upgrades accordingly, but some unforeseen issues cropped up with the database that resulted in the problems players currently see. This is no fun for our player base, of course, and we don't want to keep the realms running in a condition that frustrates our customers when we can attempt to fix things . So, these downtimes have been used to change hardware and apply fixes that will hopefully alleviate the issues. We have not yet resolved the problems, but we're working on this around the clock.

5. What issues are you experiencing with your login/authentication servers? It is often the case for myself and the people I play with that we cannot access realms our friends are already logged into.

These types of issues stem from the problems described above. Conflicts occur between some of the internal applications running in the background, and the end result can take the form of temporary login issues. We're working to resolve these conflicts so that they are no longer a factor.

6. When do you expect to have the worst of these problems resolved?

We’ll be constantly working on these issues each day moving forward until they’re resolved, but we don't currently have a set date for when that will be. We're doing all we can to make sure these problems no longer occur -- it's our top priority, and we hope to have the issues fixed as soon as possible. We’ll continue to provide players with regular updates on our progress.

7. Will the European launch utilize the same realms, or will these players be hosted on all new equipment? If they are hosted on new servers, what have you done to ensure that the launch will be free of the problems mentioned above?

They will be on their own set of hardware, as with our Korean release. Our teams are learning from the experience of our North American launch and are applying that knowledge to the servers in Europe. We hope to provide them with a smooth launch.

8. What would you have done differently? It would be easy to speculate about what we could have done differently, but that wouldn’t turn back the clock. Right now we’re extremely focused on the issues at hand, and this focus is helping us methodically chase down the problems that are causing frustration for some of our players. The foundation of our company is based on providing a top-notch game experience and an equally top-notch level of customer satisfaction; we won’t be happy until we feel we’re consistently meeting those standards.


much <3 to Blizzard for not putting extra CDs on the market.

[ebev]

YEa but i'm sure they are getting much pressure from there publisher to release more copies. The publisher gets diddly from the Subscription fees. Especially with the Finatial issues vivalidi is haveing. WOW is the only thing makeing them money right now.
Last night kinda sucked. me and Ellora wanted to log on but server down. With NO explination! So we tried another server and made new carectures. I gotta admit I like the Shaman, hehe But dont worrie we'll be back on our server, (when it's availible! lol)

[LardGibs]

The problem they should solve is character/guild transfer IMO. But for me the bottom line is at least on Dalaran, downtime excepted, the game is still fun and it seems the AH is consistently usable, but slow. Right now I AH in the morning for maximum efficiency, and better deals too.

[2biT]

Yes i agree, im a uk player and i hardly notice much lag (IF AH is always but i never use it anyway). I think this server is fdairly well run, btw i have seen a server shutdown counter, it shutdown for about 5 mins then was back up again. Just check next time that its not just a hardware switch, not a complete reboot.

[Allia]

well, it happened again....

[LardGibs]

http://forums.worldofwarcraft.com/thread.aspx?fn=wow-realm-status&t=418&p=1&tmp=1#post418


I gave up after the second drop... guess they don't feel the same way about it.

[ebev]

I made it back on and I think after the third drop it got very stable. We shouldent complain too much I think Dalaran has had the least amount of downtime of any of the servers. From what i can read Burning Legion has the worst down time.
Hiccups will happen. Not too upset about it. They will get it fixed. Blizzard allways done

[Allia]

So,

I couldn't log off tonight for like 5 minuntes. The i tried to complain to you guys and it said i wasn't in a guild.

If i am somehow booted, can you invite me again?

[LardGibs]

you're not booted it did the same thing to me last night.

just gave up on lagglethorn vale again.

[Allia]

Seems like everything is working fine again..YaY!!!

[ebev]

Yea the server has been great all weekend.

[tom]

*sits comfortably on his stable Eve server* :)

[MVB]

heh, stealth; eve doesn't have nearly as many players as wow; not even close .... 17k+ on LOWpop servers, and like 200 servers.

[ebev]

*sits comfortably on his stable Eve server* :)


Umm and eve has what. maybe 1% the users of WOW?

[Lama Master]

Cause dalaran is so stuffed up usualy im gonna make a horde char on Silverhand. There is already a post about it but im gonna be a tauren shaman called kairaque or kairarch if kairaque is taken :) ill start a guild called 'defenders of the warcheif'. anyone wanna join me?

[MVB]

As stated in the other post, please don't siphon people off of the main guild in the primary server for a non-TLSC guild in another server altogether; Dalaran is a lowpop server anyway, and will have generally less lag than some of the higher pop ones.

[LardGibs]

Yeldarb!
I didn't give youa 5g enchant for free so you could make a bunch of alts and waste time!111!
Battlegrounds Baby!

[Lama Master]

sorry sir :(


:):):) BATTLEGOUNDS... DW MVB ill wait there for you ;)

[ebev]

Yeldarb!
I didn't give youa 5g enchant for free so you could make a bunch of alts and waste time!111!
Battlegrounds Baby!

HUH?? wheres my free enchant??? hehe

[Lama Master]

you havnt been playing madly to get to level 50 for battle grounds :P (level 35 in under 3 weeks isnt bad) hehe. especialy when you have to share a computer with 5 other people!

ohh and btw.. i was thinkin... /falls asleep

[LardGibs]

yeah ebev if you beat me in a duel like yeldarb did, I'll consider it.

seriously ifyou want an enchant I can make it affordable, got my mount, just low on shards at the moment.

[Lama Master]

im close to getting mounted.. i got about 50gp and lvl 36 (soon)

[ebev]

yeah ebev if you beat me in a duel like yeldarb did, I'll consider it.

seriously ifyou want an enchant I can make it affordable, got my mount, just low on shards at the moment.

Heh be careful us leval 38 warlocks bring friends like fellwalkers that make life hard for mages. heheh

Ok i'll ask you about it in game.

[Lama Master]

To get to Lard you gotta beat me :)

warlocks are easy beat with intercept and immune to fear (Zerk stance) :) ill chew you up n spit you out.

[LardGibs]

Not to mention I could cast mind control on you and make you kill your pet, then DOT you while you're feared, then a few 100+ whacks from the staff to your puny non-melee arse :P

And dispel magic on anything you stuck to meh.

But lets get you the enchant you want.

[2biT]

stun lock + cold blood lvl 5 evis = all dead.

All i need is that magic 21 pnts in assin' tree :[