chantz explains regression testing

Asmodean · December 14, 2002, 15:31

Full story here: http://www.ina-community.com/forums/...hreadid=242812

Quote:

***So, basically, every time we get a new gold candidate build we put it through rigorous testing (see above), weigh the pros and cons and DECIDE whether it is good enough to go gold. Right now, it is not. And that is our decision to make an no one else’s.***

Lastly, I *did* make a mistake telling you folks that it would go gold Nov 26th. I honestly thought it would because THAT WAS THE DIRECTIVE from above. Period, no matter what, it goes gold the 26th. That changed for reasons i cannot and will not go into. Am i happy that i got the extra time? YOU BET YOUR BOOTY I AM because the game will only be better for it. Am i bummed it isn't out already? You bet i am, because no one wants it finished, out the door and into your hands more than me and the folks at QSI. Catch 22 eh?

Okay. So MoO3 was going to be like Civ3, in that they were going release an unfinished game. I, for one, am extremely happy that they didn't. I am a strong believer in finished products.

Asmodean

moomin · December 14, 2002, 15:55

And now inquiring minds would love to find out just what those reasons that Chantz "cannot and will not go into" are.

Chantz, my pal, feel free to mail the full disclosure to Firaxis, at least. I'm sure they'd love to know how to convince IG to allow them to ship useable products...

Proteus_MST · December 14, 2002, 16:42

But maybe by shipping unfinished Products Firaxis just made it possible for Quicksilver to ship a useable product

MarkG · December 14, 2002, 17:08

full copy for the time the server is busy

Quote:

A bit of detail about testing MOO3

Folks.

Some people are confused about how regression testing works. I'm going to take minute to hopefully give you some insight into why it takes so long to test moo3. Note that companies have varying methodology but the end result is the *same*; shake out all the major bugs.

1: MOO3 is huge. HUGE. Did i say how huge it is? There are so many permutations and so many ways to "skin a cat" that testing them all takes a team of 20 people working roughly 16 hours a day 3 straight weeks to slog through it. That's with people assigned to specific areas of focus. Did i say how huge it is? IT'S HUGE.

2: Re: Regression testing. See the above statement? Every time we get a new build, we start from scratch and test every known permutation. Bugs come up, they always do. The important thing to note here is that *are the bugs that come up during a regression test show stoppers and the ones that we're marked as showstoppers in the previous build fixed?*

3: What's a showstopper? Many, many things. A repeatable crash. Crashing after 2-4-6 hours of gameplay. Running out of memory and crashing. Save game corruption. Multiplayer hangs. Missing feature or a feature not working properly. Missing art; the list goes on and on. We have an enormous checklist we go through. Every time we fix a bug, we take a chance that we might break something else. To mitigate this, especially during the final weeks of testing, we enter into extreme paranoia mode. We call this Code Review. Every time a programmer makes a change, it is reviewed by at least 2 other programmers who share some knowledge of that particular piece of code. A line-by-line review is done, questions are asked, flags are raised (or not, as the case may be). That bug fix then gets tested for at least a few hours before it gets "checked in" to the main source tree. It gets compiled into the build (along with any other fixes) and is then sent off for proper (see above) testing.

4: Source code control. Know it, love it, it is your best friend. When a programmer makes a change to a source or header file (or creates a new one), it is done outside the main "tree", which resides on a SERVER that all team members have access to. In other words, it is done locally on that programmer’s machine. So, if i create a file or modify a file called spies.cpp (c++ extension), i'd do it locally to ensure that i'm not stomping on code in the main tree and causing dumb errors and stopping other people from working. A file can only be checked out by one programmer (or designer or artist for that matter) at a time, for obvious reasons. This is really the condensed version, as there is much more to it, but this sums it up in a nutshell.

5: Sometimes we know right away that a build doesn't cut the mustard (a crash 20 turns into it), but we are *now* at the point where most problems are occurring very LATE GAME in MP on Large maps with many races; in other words, we are exercising and stress testing areas of the game we couldn't 6 weeks ago because we couldn't GET there due to crashes and errors. Game balance also figured into this. Heck, there was a time when you could *not* win against the new orions no matter how many ships you could throw at them (turn after turn). That means no Sole Superpower Victory. That's a Showstopper because you cannot win the game - at all. No fun.

***So, basically, every time we get a new gold candidate build we put it through rigorous testing (see above), weigh the pros and cons and DECIDE whether it is good enough to go gold. Right now, it is not. And that is our decision to make an no one else’s.***

Lastly, I *did* make a mistake telling you folks that it would go gold Nov 26th. I honestly thought it would because THAT WAS THE DIRECTIVE from above. Period, no matter what, it goes gold the 26th. That changed for reasons i cannot and will not go into. Am i happy that i got the extra time? YOU BET YOUR BOOTY I AM because the game will only be better for it. Am i bummed it isn't out already? You bet i am, because no one wants it finished, out the door and into your hands more than me and the folks at QSI. Catch 22 eh?

Anyway, hopefully this gives you some insight into how regression testing (at least in my little world) works. No questions please, just enjoy (or not, as the case may be).

Thanks,
Constantine

Main_Brain · December 14, 2002, 19:51

mhh extensive Testing..
I conclude that there are Cheats in the Game :P

Gooberman32 · December 14, 2002, 22:26

I was right then bugs are late game

H Tower · December 15, 2002, 00:34

hard to get rid of those

Skanky Burns · December 15, 2002, 02:33

Quote:

Running out of memory and crashing.

Memory leaks are evil. At least that is one advantage of using java over, say, c++.

moomin · December 15, 2002, 10:10

Actually, I'm not overly impressed by this explanation. So Chantz tells us Moo3 is extremely complicated (Uh huh. Compared to moo2? Perhaps. Compared to Neverwinter Nights? Not bloody likely. Compared to luggage logistics in a major airport? No way José!). He goes on to explain details such as CVS and QA procedures. Like QS invented these, or something.

Here's a newsflash. Software gets written using methods like these all the time. Some of it orders of magnitude more complex than any PC game. And while it would be unfair to say many software projects deliver on time, Moo3 current troubles are somewhat exeptional - but bad project management seems to be the standard in this particular industry. Still, that is what we're talking about. Bad project management. In fact, from the outside, it seems like atrocious project management.

Unspeakable Horror · December 15, 2002, 11:27

Quote:

Catch 22 eh?

What does it mean?

TCO · December 15, 2002, 11:50

A self-defeating requirement. For instance if the only place to complain about internet connectivity problems is a website. Or if you have to have experience to get a job...but the only way to get experience is to have a job.

Unspeakable Horror · December 15, 2002, 12:01

Thanks

habadacus · December 15, 2002, 19:54

It's nice we got bit of information, it really is, although it doesn't include a release date, and even if it did I'd be reluctant to put any faith in it, but it doesn't really seem to shed any light. I mean, yeah when you code something, you have to fix bugs. Knowing how you fix the bugs doesn't help me. That it's a long arduous process isn't news to anybody, that's why programmers and developers get paychecks. I would symathize so much more with the developer's plight if this game hadn't been in development FOREVER. YEARS EVEN!!!
He says how HUGE the game is, well thats nice, and that it takes a team of 20 people working 16 hours for three weeks straight to test it. Um, am I mising something here? I'm no Human Resourses Administrator, but maybe you should get ANOTHER 20 GUYS. How hard is it to hire game testers? I know from being one at EA they don't exactly make 100k year, in fact I got paid salary, so the all the extra hours in the crunch time didn't cost them anything more.
My final nitpicking point is he says it was a mistake to say the release date would be Nov 26th, but that was a a command from on high, and under mysterious circumstances that directive changed.
OK well, they knew the game was HUGE, they knew they only had 20 OVERWORKED GUYS , they knew they had to do SOURCE CODE CONTROL and other exciting and arcane sounding things and that the game had bugs and missing art and everything else. So given all that, and the huge sigh of relief everyone must have enjoyed after the Borg or whatever said they didn't have to go gold on the 11/26, then what the hell was up with the Dec 4th release date? They were so glad to to have extra time and be released from the demands of thier demon masters. Who got the bright idea that 8 days was gonna be the ticket they needed to release what by now ought to be the BEST GAME EVER?
Well, anyway, I feel better after my little rant. Although I whine and moan, it's only because I'm DYING to play this game already. I took a week off from work starting the 18th becasue I was hoping the game would be out by then, and again it looks like I will be disappointed. (and bored... Hegemonia sucks) And this is the best they can do by way of explaination? Felgercarb! If they miss December altogether, and I recant and actually do buy this game it had better transform my computer into and actual ALIEN BATTLECRUISER. Anything else will seem anti-climactic.

Asmodean · December 16, 2002, 07:07

@habadacus

You very nearly turned me around with your post. But then I realized that QS and IG have promised us a flawless game. And that's what makes the waiting time durable for me. OK...so they've made mistakes, what with the release dates and all. So what. People make mistakes. Deal with it.

I, for one, would much rather wait untill january or february, and then play a bug-free game, than playing a game that crashes in turn 258 in my christmas holiday.

Asmodean

Zed-F · December 16, 2002, 13:31

Habadacus, adding more people at this point will not solve the problem, it will worsen it. Adding more people 6 months or a year ago when they might have had time to train them might have helped, but not now, and 6 months or a year ago they either (a) didn't realize that they would need more people or (b) far more likely, realized that they needed more people but couldn't afford to hire any more. So, they had to make do with what they had.

Moreover, there are natural limits to how much you can effectively subdivide any given set of work. I don't know if the regression team has reached that point, but throwing more people at a problem does not always help even if they have lots of time to train before crunch time. Nine women cannot make a baby in one month.

Grumbold · December 16, 2002, 13:48

Momin the nice thing is that they are even using rigorous QA methods and sticking to them. Many othr developers either don't employ them or allow the publisher to publish and be damned despite the results. I'm sure MoO3 will still have an issues list when it ships, but hopefully nothing that should have us tearing our hair out waiting for the patch to arrive...

Xentax · December 16, 2002, 15:02

Memory leaks can be hard to track down, especially if you don't have the right tools (like Purify); there are also various ways to program defensively to reduce the chance of them cropping up in the first place (auto pointers, making objects reference counted, etc.).

Usually, they end up being a contractual thing -- block of code A calls block of code B, object(s) created, B thinks A will be responsible for cleaning it/them up, A thinks B will, and voila, memory leakage.

Still, they're usually fairly easy to fix once located, unlike, say, some sort of unanticipated threading issue.

Moomin: While software engineering (and the subset, testing) practices might not be a "newsflash" to everyone, an overview of what's involved probably IS new information for a majority of the people reading about MOO3 (even moreso on the IG boards than here, perhaps).

Lots of people that SHOULD know better don't, even -- for example, the next pointy-haired type that says "It's just software" in my presence won't live long enough to make another such baseless assumption...

moomin · December 16, 2002, 15:29

Quote:

Originally posted by Xentax
Moomin: While software engineering (and the subset, testing) practices might not be a "newsflash" to everyone, an overview of what's involved probably IS new information for a majority of the people reading about MOO3 (even moreso on the IG boards than here, perhaps).

Perhaps. Given the amount of impatient whining about release dates all over the net, some explanation was clearly in order. It jsut rubbed me the wrong way. I get the impression Chantz thinks this is something terribly unique being done. And it ain't. Except, perhaps, in the gaming industry (or, PC gaming indutry - I bet the console shops do their programming and testing by the book).

Quote:

Lots of people that SHOULD know better don't, even -- for example, the next pointy-haired type that says "It's just software" in my presence won't live long enough to make another such baseless assumption...

As a former IT slave who worked the trenches and propelled myself into pointy-haried-hood by means of MBA, I feel your pain. I just hope you aren't implying that there's plenty of these guys around at QS?

Ozymandous · December 17, 2002, 01:01

Quote:

Originally posted by moomin
Actually, I'm not overly impressed by this explanation. So Chantz tells us Moo3 is extremely complicated (Uh huh. Compared to moo2? Perhaps. Compared to Neverwinter Nights? Not bloody likely. Compared to luggage logistics in a major airport? No way José!). He goes on to explain details such as CVS and QA procedures. Like QS invented these, or something.

Here's a newsflash. Software gets written using methods like these all the time. Some of it orders of magnitude more complex than any PC game. And while it would be unfair to say many software projects deliver on time, Moo3 current troubles are somewhat exeptional - but bad project management seems to be the standard in this particular industry. Still, that is what we're talking about. Bad project management. In fact, from the outside, it seems like atrocious project management.

Funny, I don't seem to remember people making fan sites and/or frothing at the mouth on message boards daily anticipating the next installment of the MS Office suite do you? Which means that MS can put no release date what-so-ever and people will just pick it up when they see it. This is SELDOM the case with any game, people want it NOW and usually post negatively when they can't have their instant gratification.

Easy to throw stones when you have *absolutely* no idea what sort of PM QSI and IG have isn't it? Tell you what, since you seem to have all the answers, why don't you start your own company and develop the perfect game since you obviously

know how to do everything perfectly.

Better yet, try to be positive for a change, heaven forbid you aren't absolutely pissing your pants with joy when MOO comes out or I guess we'll have to call you " " and you'll have to go back to the "perfect" Civ2.

Ozymandous · December 17, 2002, 01:06

Quote:

Originally posted by moomin

Perhaps. Given the amount of impatient whining about release dates all over the net, some explanation was clearly in order. It jsut rubbed me the wrong way. I get the impression Chantz thinks this is something terribly unique being done. And it ain't. Except, perhaps, in the gaming industry (or, PC gaming indutry - I bet the console shops do their programming and testing by the book).

Ah, comparing console to PC game, like apples and oranges... Consoles are *so* hard to develop for when you know exactly what software & hardware you're writing for right off the bat.

Console games = kit cars, you know where each part goes because they are all the same.
PC games = hand crafted and built-to-order concept car where you have an idea of what you want but have to figure out just *how* to make this spiffy piece fit around the gas tank, between the shocks, axel and drive shaft.

They are a tad different (not better or worse, just moe to developing a PC game.)

Craig P. · December 17, 2002, 02:19

Quote:

Originally posted by moomin
Actually, I'm not overly impressed by this explanation. So Chantz tells us Moo3 is extremely complicated (Uh huh. Compared to moo2? Perhaps. Compared to Neverwinter Nights? Not bloody likely. Compared to luggage logistics in a major airport? No way José!). He goes on to explain details such as CVS and QA procedures. Like QS invented these, or something.

I think you are completely missing the context for chantz' post. IMO, it's a necessary response to the whinery that has been a substantial portion of the postage on the official boards.

Note that in a vacuum, or in the context of this forum, I think your response is reasonable.

moomin · December 17, 2002, 06:10

Quote:

Originally posted by Ozymandous
Easy to throw stones when you have *absolutely* no idea what sort of PM QSI and IG have isn't it?

My, such hostility. Can't be good for the blood pressure, you know. Now, I have a pretty good idea of the measures QS employs - after all, everybody who isn't illiterate is perfectly able to read about them in some detail in Chantz's posting!

Quote:

Tell you what, since you seem to have all the answers, why don't you start your own company and develop the perfect game since you obviously

know how to do everything perfectly.

I already head a development shop. True, we don't do games, we do medical software, but so far we've never missed a delivery date, and believe me, our QA procedures leave anything in the gaming industry in the dust, if only for liability reasons. Now, mr Friendly, what exactely are your qualifications, except for ranting?

moomin · December 17, 2002, 06:14

Quote:

Originally posted by Craig P.
I think you are completely missing the context for chantz' post. IMO, it's a necessary response to the whinery that has been a substantial portion of the postage on the official boards.

That's quite possible. I haven't frequented those. But I still can't help but to think that some boilder-plate CB would have done just as well as an explanation and left a lot of stakeholders happier.

Xentax · December 17, 2002, 09:59

Oh, I wasn't referring to pointy-hairs at QSI, rather the group at one of our (former) customers.

As I've said before, the PC gaming industry seems to lack the techniques a lot of the rest of the software industry uses to plan projects that get done on time and on budget; personally, I think it's a response to short-cycle pressure that's become a real trend, hopefully one that will be reversed soon.

I mean, the short cycle is really a *myth*, nowadays. Companies keep promising games out in a year or 18 months, and end up being a year late. If they'd given themselves the 2 or 3 years they needed, up front, they'd end up with a BETTER product at the SAME end date, because MOST (not necessarily all) people, including engineers, work better when they're not laboring against an impossible deadline. When you're trying to do the impossible, corners are invariably cut, shortcuts are taken, and the product suffers as a result.

Gooberman32 · December 17, 2002, 10:39

Im not sure about recent advances in console technology, but can console games be patched?

moomin · December 17, 2002, 12:25

Quote:

Originally posted by Xentax
I mean, the short cycle is really a *myth*, nowadays. Companies keep promising games out in a year or 18 months, and end up being a year late. If they'd given themselves the 2 or 3 years they needed, up front, they'd end up with a BETTER product at the SAME end date, because MOST (not necessarily all) people, including engineers, work better when they're not laboring against an impossible deadline. When you're trying to do the impossible, corners are invariably cut, shortcuts are taken, and the product suffers as a result.

This seems pretty obvious, yes. Given that you are something of an industy insider, would you care to guess about what the reasons behind this seemingly self-destructive behaviour is? Is it because the customers accept it? Is it because it has become a norm noone seeking funding dare deviate from? It's hard to understand it from the outside.

Grumbold · December 17, 2002, 13:19

The only positive side to it that I can see is that it forces a serious assessment of the product 40-50% of the way into development. All too often companies are making claims about what it can or cannot do at 95% of the wya to release that suddenly get cut or massively revised before release.

It would be a lot more honest to write the original project plan with a crunch point at 50%, but that just doesn't happen in most cases. Its easy to keep putting it off, thinking features are still achievable given a bit more work. There also may be an issue of Producers not wanting to take on new projects knowing that it has a minimum 3 year life cycle when many of them are financially hard pressed. I doubt many execs get promotions and bonuses for telling it like it is rather than shooting for ambitious targets then chasing the developers hard if they miss.

LDiCesare · December 17, 2002, 13:50

One thing strikes me about the regression testing techniques chantz talks about.
What about automated non-regression tests?
Surely if he talks about code control, he should speak about that, but does not. Don't they use any? That would save them time. Of course that doesn't replace manual testing, but still, seeing no mention of it is pretty frightening.
And the definitions of huge are not the same for everyone. 20 testers? My, we have many more where I work. And we don't delay our products that much. But, sometimes, for the sake of quality, maybe we should...

Plus I wonder whether code reviews are as efficient as pair programming. Anyone has opinions/experience about that?

Zed-F · December 17, 2002, 15:29

Automated testing works best in an environment with limited churn. Otherwise the (non-trivial) cost of writing and rewriting the testcases as things change exceeds the value you get out of them. We typically only consider introducing automated testing after the first release of a product, once there is a relatively solid base from which to work and once the time pressure has abated. MOO3 is not yet in that position, but I would think that automated testing is something that they would consider implementing as they start working on the first patches, to help ensure the patches don't break existing functionality.

Straybow · December 18, 2002, 00:47

Moomin, is that medical database or equipment control, or maybe something else entirely? What kind of AI does that use? Does it have cool white blood cell battle graphics?