Where Realtime Worlds went wrong, part 2

Last time, I touched on some of the crazy walls we put up between different groups in the organisation.  I focussed in on the Community team because it happened to fit with the other stuff I had to say about our relationship with our gamers, but it would be unfair to give the impression that the problem was limited to their group.

From the moment we took the investment, our organisational structure developed in a number of unhealthy ways.  These problems detracted from the quality of product and experience we were able to deliver, destroyed internal motivation and passion, and also caused us to burn through our cash much faster than necessary – all significant factors in our ultimate failure.

Spend it all!

Let’s start with our attitude to the investment we raised.  We fell foul of some kind of modern version of Parkinson’s Law:

A Realtime Worlds project will spend the funding available to it.

When we received the initial $30m to develop MyWorld, management literally reverse-engineered a “hiring curve” (a graph of team size against time) from 3 parameters: the budget available, the desired launch date (set by the investors), and our internal figure for the maximum rate we were able to hire people at (this was the only good part of the plan – Dundee put the brakes on for us!). There are obviously far better ways to plan a project, and I could spend a whole post discussing just that, but for now I just want to focus in on the unquestioned assumption that we should set out to spend all the money.

This attitude infected our company culture at many levels. Almost everything we did, we sought to throw people at, and our hiring created inefficiencies all over the place.

More people in one team created knock-on effects elsewhere: more programmers needed to support more artists, more IT, QA and admin staff needed to support everyone else, more project managers needed to manage everything, and more recruiters to hire all these staff.  We hired a whole “business development” group that did, well, nobody else in the company got told what they did, except they hassled development constantly for “executive” progress reports (of course, making reports takes time, so this probably contributed to further hiring).  Then we hired a director of development, who while certainly helping us focus on on-time delivery, was sadly forced to spend much of his time fending this senior management layer off our backs.  Then we added a “program manager”, reporting to business development, to fulfil a nebulous floating cross-company communication role.  Someone above us came up with a “patent strategy” initiative; the engineers dragged along to the meetings managed to fend that off long enough for management to get distracted and forget the whole thing.  We hired a “live production” team, whose entire job seemed to be to pass messages between operations (the folks who run the servers) and engineering, on the basis that these two groups were struggling to communicate.  Unfortunately, they struggled to communicate with either group, and spent a lot of time creating Processes for how to pass these messages around.

All these layers, of course, generated extra meeting upon meeting.  When I worked on APB, my Outlook calendar looked like a game of Tetris, the day stacked full of meetings, usually with a triple-booking somewhere and several double bookings.  Hardly any of them held sufficient value for the time spent.

I don’t mean any of that to be a criticism of the specific people in those roles.  Many of these folks were wonderful, talented people, and many of them realised the problems they were part of and fought hard against them.  I was as much to blame as anyone.  I built a technology team that was too big for its goals.  I also spent a year in a nebulous “technical manager” role on APB, and didn’t do enough to fight the cultural problems, including the question of why I was there at all (and why I had so many meetings).

Good vs Evil

The layers and walls between groups all came from a smallish group of powerful people (who we’ll label ‘Red’ for the sake of argument), that wanted Realtime Worlds to become more “corporate”, lazily imitating big companies in a superficial way, at the expense of critical thinking and focussing on our true goals.  Here’s a typical Red quote (I may not have the precise wording down but you get the idea – and I’m omitting the name of his previous employer because my point here is not personal):

Producers are how you drive accountability and make sure things get done.  <Company X> [who they worked for previously] understood this.  If we were adding a feature to the game, the first question would be “who’s producing this?”  If the floor needed cleaning, the first question was “who’s producing that?”

Opposed to Red was a group that for the sake of argument we’ll call Blue, with diametrically opposed views.  Quietly and subtly, perhaps without many in the organisation noticing, these two groups fought for the company’s culture.  Ultimately the Blues were destroyed.  While probably numerically greater, they held less org-chart power and were forced to work hard for even small concessions.  And while the Red relished the meetings and political fighting, the Blue were passionate about getting on with real work, about making our product better, and for the most part gave up the fight to focus on that.  The Red weren’t averse to dirty tricks either, such as paying a key Blue to leave (that’s org-chart power for you).

So a key piece of advice to any company receiving a large investment would be this: be very, very careful about who you hire to manage your fast growth.  You need someone who “gets” lean startup culture and is not trying to turn you into an identikit corporation where everyone talks bullshit management-ese just for the sake of it.  If you become big and successful, you might turn into that anyway, but if you try to be that way first, you won’t become big or successful.

Don’t get me wrong – I’m not against big corporate culture per se.  It has its proper place, as the natural result of refining a successful business to reproduce and continue that success as efficiently as possible.  But copying it cannot possibly be a driving force towards success in the first place.

Chalk and cheese

One of the most harmful organisational walls we built was between “business” and “development”.  Once we took that investment, responsibility for anything with a £ sign in front fell to a small group of “business” people, most of them in our US office.  We’d ask these people how much gameplay bandwidth cost us on APB, in case we needed to optimise our usage to make the game profitable.  We’d ask how much patch bandwidth on our CDN cost, in case it was worth adding bittorrent support to our patching system.  We’d ask how much running a server cost, in case it made business sense to spend more time on server code optimisation and reduce the number of servers we needed.

Every time, we’d be met with a condescending pat on the shoulders.  “Don’t you worry about that, son, you just get on with making the game.  Let us take care of the money.”  We asked, and we asked, and we asked, and were eventually point-blank refused, and later just ignored, on these and other important questions.

It wasn’t just about the questions we were asking.  The questions we didn’t ask were just as harmful and more.  How much are we actually spending on this game per month?  If we launch on this date, how much cash will be left in the bank?  We got so fed up of the small battles that we gave up and stopped asking important questions.  We wouldn’t question the financial prudence of an additional hire because we were tired of being shut out of that conversation.  We just asked upwards for permission and washed our hands of financial responsibility.  Of course, the people we were asking didn’t understand the importance of the hire, so we had to justify it to them.  But we did so without any context of how important it might be to save money.  It turns out, if your goal is simply to justify a hire to someone who doesn’t understand its importance, you easily find reasons to convince them.

We built a culture of treating development as completely divorced from the business we were running.  It doesn’t help that game developers are not used to thinking about the money.  Traditional publisher-developer relationships mean that the publisher worries about the money, and all a developer has to do is hire the number of people the publisher says they’ll pay for.  Everything else is taken care of.  We let ourselves sleepwalk into this attitude when we were spending our own money.

When we were asked to talk about money, the conversations were ridiculous.  Our finance guys thought we should cut the free fruit.  I had to wait 2 months for a pinboard because everything went for approval to the very top level for a while, apparently to help get a grip on our spending.  We cut all conference visits for my last 18 months there – a paltry amount next to our salary bill.  I spent 3 months arguing to upgrade some of the APB developers’ PCs in the name of working more efficiently; our wonderful build engineer, Phil, even put some hooks into the build system to automatically track how much time each developer spent compiling code in a day and collect all the data centrally.  We could prove that the whole thing saved us money in a couple of months, but nobody was interested.  They wanted to stick to a supposedly pre-agreed budget, which it then turned out they could change at will to suit their position.  I wasted weeks arguing about this.

With hindsight, I guess we were about to run out of cash and any expenditure was a problem.  But if this was the case, why on earth weren’t we trimming down our massive salary base at that point, or earlier?  Why did we waste time debating pointlessly small amounts of money?  Why did we force our talented development leads to spend their time on silly internal arguments instead of focussing on the product?

Developers might not be experts on money, but they’re incredibly smart people, and the parallel between spending in an organisation and optimising code is obvious.  Measure where your resources are going and focus on the areas where you can achieve the most benefits.  You don’t talk about cutting fruit bowls when you have 300 staff and are still hiring more.  I also don’t see how you can spend intelligently unless you hold plenty of open, intelligent conversation between the two groups – or have someone who understands both development and finance run the show.

If wonder if the finance guys were equally hamstrung.  Maybe they wanted to cut the staff size but were told to stay clear of that.  I can’t think why else they appeared to be so interested in the petty savings (the only other logical explanation is personally unpleasant towards them, and I’m trying to stay clear of that, especially for people I hardly know).

I wonder what would have happened if we’d had a different relationship with finance.  Or if we’d been given financial responsibility ourselves.  I’m pretty sure we wouldn’t have come up with a plan that involved letting our cash reach almost zero at the point of APB’s release, with ongoing costs of 300ish salaries.  I’m also pretty sure we would have trimmed our feature set differently, and structured our teams to be leaner.

What I’ve said so far sounds so unbelievably incompetent that you’d think we were all just a bunch of idiots.  But I still contend that Realtime Worlds had an unbelievably smart, talented staff.  I can’t prove that with words, but even if you accept it only hypothetically for a moment, it raises this question: why would a bunch of smart people put up with this crap?

Next time, I’ll try to answer just that.

(Part 3 now up).

This entry was posted in Realtime Worlds. Bookmark the permalink.

31 Responses to Where Realtime Worlds went wrong, part 2

  1. Nick McCrea says:

    Really great stuff, Luke.

  2. Matt Doar says:

    Congratulations, this is a well-written and helpfully critical article without the tedious personal attacks that usually come up in these situations (I’ve been there).

    > why would a bunch of smart people put up with this crap?

    Well, now that’s the question, and it applies to so much more than just our businesses. To all of human endeavour, really. My take on it is that just being smart is not enough to understand how to persuade people to work well together. Money can be used to lubricate human interactions for a while but not forever.

  3. Nathan H says:

    > why would a bunch of smart people put up with this crap?

    From what I’ve seen the amount of crap people are willing to put up with tends to be directly proportional to how much they actually care about what they are producing. Money or not…

  4. Thank you for these open and intelligent articles, Luke. I’ve seen similar projects before, and how they almost killed off the company of talented individuals that made those products, so I’m happy someone steps up and writes about how it is. Good company culture is incredibly important. I may be a developer, but I took management classes in college, and the first thing they talk about is how important company culture is. I guess most future managers just shrugged during those sessions and wondered when we’re getting to the bit where everyone gets a big car. And hookers.

    Having to fight for that fruit bowl? Check. I ended up paying for fruit out of my pocket until it shamed someone in management into reinstating it. Conferences? We used to sent all the marketing folks and one engineer to help them set up demonstration machines. Even for GDC. None of the grunts ever got to go for educational purposes.

    Meanwhile, everyone still soldiered on and wanted the games to succeed, even when we saw that we were headed for the cliff. Because years of our lives, hours of unpaid overtime, all our weekends and years of our career were invested in this. I’m looking forward to part three, I have a feeling I’ll again be nodding my head as I read it.

  5. Nick McCrea says:

    Speaking of culture, someone commented on the Guardian article here. As MyWorld found out, we had cultural differences that grew and emerged simply from our team structure, never mind the amplifying effect of actually being in different time zones. Quote below:

    “To compound this, opening offices on three continents (Dundee, Boulder, Korea), separating teams geographically for no good reason. Utter, utter failure. Many teams will develop their own insular culture if they’re in a different ROOM, never mind an ocean away. “

  6. Martin Wilson says:

    Excellent article, it’s really fascinating to hear all this. My brother worked under you, and for years he would say things like “Oh, I can’t tell you about this, I can’t tell you about this…” and seemed almost as if he were under the official secrets act. It’s nice- and more importantly interesting- to learn what was actually going on all that time. (the secrecy bit you mentioned in your first post is what I’m thinking of in particular- he never talked about APB or Project MyWorld until long after the fact)

    While I find the whole mess disheartening, I’m more worried that the whole thing is going to repeat over and over again with sub-par executives re-hired without learning their lesson.

    I understand this might stray into fingerpointing territory, but my brother told me that some of the management staff in RTW’s past experience was Star Wars Galaxies. I’d never hire those guys! It takes a special talent to run a Star Wars MMO into the ground.

    • Nick McCrea says:

      If you could only hire people who’ve worked only on successful games, there wouldn’t be a lot of hiring going on :) Most long-standing industry people have a few skeletons in the closet here and there…

      Plus, there’s valuable lessons to be learned in failure. Certainly, APB didn’t repeat the mistakes of SWG – it made whole new ones :)

  7. This is really great stuff, Luke. I’d like to point you to two places that help answer your questions here.

    The first is from Steve Blank, an amazing writer/thinker on startups. This one is a must read: http://steveblank.com/2010/09/13/job-titles-that-can-sink-your-startup/

    The other is a post I wrote. I think that the artificial distinction between “management” and “doers” is dangerous anywhere. It’s dangerous in hospitals, it’s dangerous in schools, it’s dangerous in the creative arts, it’s dangerous in development.

    http://www.gamesbrief.com/2010/08/why-creatives-and-business-types-must-mix/

    We need more mutual respect between creativity and management. really we do.

  8. Tacroy says:

    When the simple consumable free stuff goes away (sodas, fruit, chips, pens, legal pads, etc) , that’s when you know the company is in trouble. It’s not necessarily running out of money, mind you – no, what it means is that someone in finance has decided that it is more important to save ~$5.00 per employee per day (if that!) than it is for that same employee to work more efficiently. The math on this is really simple: if you make $60,000 per year, and the free stuff makes you at least 3% more efficient, then the company is more than repaid.

  9. jurio says:

    That red/blue thing is bullshit. You need both.

    Because the picture you described of endless meetings, bloat and bureaucracy arose because of a laissez faire attitude where you have a spiders web instead of an org chart.

    Good management gets rid of that shit. They give the Blues deadlines, they tell them to cut the fluff. Lean “startup” organisation does not scale up, you need structure.

    If the upper management were competent (which they clearly weren’t), they’d have cut a swathe through this. Set up actual org charts and streamline reporting.

  10. Pingback: APB Closing Today

  11. anon says:

    Its more of the “doers” being considered too lowly to care for budget issues other than what upper management explicitly told them. Any talk of money was probably met by condescendence in an attempt to make sure they didn’t infringe on their duties (or rather, control).

  12. MikeD says:

    The best quote I got from a producer when I expressed how concerned I was with the spending was:-

    “Don’t worry about money, it’s not your job. We’re backed by some big players and they won’t let us go down. we’ll just get more if we need it.”

    Classic. Thats really not the kind of thing you want to hear from the exec producer…. It shows just how bad the culture was in the upper management.

    And to… “Good management gets rid of that shit. ”
    You hit the nail on the head. We didn’t HAVE good management. You need a “little” Red, and a lot of Blue. They wanted ALL Red. Our producers were too busy being game designers and not doing their jobs; producing and keeping crap away from the teams. Team leads ended up spending most of their time doing production work, not coding and leading the team, all because production was in yet another meeting about design or something.

    And don’t get me started on allowing juniors to reorginise the entire code base and overall design….

  13. Pingback: Realtime Worlds and Runic Games: Vastly Different Strategies « Double Buffered

  14. Pingback: Keen and Graev’s Gaming Blog » Blog Archive » APB’s Failure and why I think it’s a good sign

  15. fuck you says:

    why dont you give me my 50 fuckin bucks back asshole

  16. fuck you says:

    a monkey can run a business better than you

  17. Pingback: The Sunday Papers | Rock, Paper, Shotgun

  18. generalnio says:

    Well written comment. Well, to me Your company just got captured in enormous ego games. And it absorbed so much of the potential available in finance and energy, that you lost focus on the goal: making a good game. And there didn’t seem to be anybody who was had a vision of the whole and who had the final decisive word. Nice lesson you learned for the future. Our ego is a tricky thing, you know.

  19. Otis says:

    Thank you for this detailed description of RTW’s issues. What’s shocking, perhaps, to people outside the game industry is that RTW is NOT unique. Not at all.

    I work for an American development studio that was bought out some years ago by another corporation. We are right now experiencing some symptoms and corporate shenanigans highly similar, astoundingly similar, to what is described for RTW. It’s as if the people in the “Red” corporate culture all work form the same playbook and have exactly the same responses to external stimuli.

    I’m guessing based on internal scuttlebutt that we have about another four months of funding left. We’d have released at least one or two of our products by now, but the “Reds” in our company are actually interfering with development (in the name of making it “go faster”, which is having the opposite effect). If they’re wise, our investors are getting smart enough to not keep throwing money down the same hole, which of course spells our doom.

    Wish us luck.

  20. Pingback: Eckyman.com » What went wrong?

  21. michael says:

    So.. What am I supposed to do? I know it isn’t easy for you guys to close such a big project, but you make it sound like a piece of cake and compared to all of us (the players) , we also loss money wich we would very much like to get back in someway, wheither we can get to play the game we payed for or get cash back at our bank account. I bought this game a month ago, i am on an education so i dont have alot of money, so therefore I feel pretty damn stupid to have spend my hard earned money on a product wich is unusable after a month. I think i speak on the behalf of many many people, so find out a way to make your previous costumers happy again.

  22. Pingback: APB’s Failure and why I think it’s a good sign | Play Game Now

  23. Pingback: A Look at Realtime Worlds | GameTaco

  24. Ash_F says:

    Luke. Thank you for the postmortem. It is eerily similar to one I read by a developer for Tabula Rasa.

    Best wishes to you and I hope you continue to bring the world more games.

  25. Panda says:

    Sounds like your management had some sorta odd “cargo cult” mentality going on there, I mean it seems they thought that if they look like a big corporation with all the bells and whistles and inefficiency, they automatically become big rich corporation.

    Just like cargo cultists doing those creepy re-enactments of air cargo droppings, with all their best effort. Of course nothing ever materialized.

  26. Pingback: Game Retail Store » This Year In Video Game Criticism: A Look Back At 2010

  27. Pingback: When to Jump Ship | Karl Katzke

  28. Pingback: Great MMO Company Collapses of Our Time – RealTime Worlds | Vicarious Existence

  29. DB says:

    More IT? We did pretty good running that whole gig with 2 engineers and 2 sysadmins (irrespective of what someone would have the company believe). Took a lot of hard pushing to even get the sysadmins.

    • lukehalliwell says:

      You’re underselling yourself there – you guys did an amazing job :) I certainly didn’t mean that we had too many in IT. I still think logically, that more staff require more of other supporting departments (and I include teams I worked in there, programmers and testers supporting artists). Maybe not a lot in any one place but it adds up across them all. And btw, you forgot to mention the sequence of IT managers we had ;)

Comments are closed.