My Letter to Brian About A Public Alpha/Beta: Your Thoughts...

[yin26]

May 22, 1999

Brian,

When Sid released Civ1, it changed the computer gaming world forever. Now, 2.5 million fans later, the fate of this legacy rests in your hands. But if one thing has become clear to me after pouring over hundreds of posts for the Civ3 list, it is that these 2.5 million fans don’t want Civ2+. They want a new era to dawn on this game.

I’d like to share an idea that I am convinced will take Civ to a higher level and ensure its future:

A PUBLIC ALPHA/BETA.

The advantages to this strategy are numerous and profound:

** AN HONEST VIEW OF YOUR PRODUCT:

If I may be so bold, I believe the reason that you and your team were so surprised at people’s criticism of SMAC’s graphics and theme is that SMAC was primarily an in-house product. Simply, you and your team became used to your own ideas. You became so comfortable with the graphics and technology that you could no longer see the game as an average gamer would. A public alpha ensures this won’t happen again. By putting a Civ3 alpha into gamers’ hands, the relevance and the timeliness of the feedback loop skyrockets exponentially, and there are no surprises for anybody.

** QUALITY ASSURANCE:

This point cannot be stressed enough. This point on its own merits should form the basis of a winning public relations campaign. SMAC had problems. Some of them embarrassing. A public alpha/beta program puts the game on machines very early, at a time when crashes and major bugs are to be expected. This system would allow you to test the wild world of system configurations on a much larger scale than you could ever hope to do at your headquarters. And, I must remind you, Jeff’s insistence that SMAC’s problems were often due to system configurations and not the program itself has, to this day, left a bitter taste in many people’s mouths. No matter how true his statements are, a public alpha/beta would virtually eliminate this need to make excuses.

** GENERATING FAN INTEREST:

You should not rest on Civ’s laurels. As I have said, fans are growing tired of the current Civ formula. As I’m sure you are aware, a number of people simply no longer have faith that any company is capable of taking the risks it will take to move this game to the next level. A few fans are writing their own version as we speak. A public alpha/beta will help squash the counter-momentum. It will give all fans tangible proof that Firaxis doesn’t want to publish ITS game but the GAMERS’ game. The positive hype this will generate among fans is hard to overestimate.

** A SHOW OF COMPETENCE AND DARING:

Few companies have the stomach to put their work out in such an early stage. This is, essentially, vanity and/or a lack of faith. It’s akin to an artist refusing to show his early sketches to the patron. True, if the finished piece is accepted, he’s a hero. If it’s not, he’s looking for work somewhere else. Firaxis is a young company and should set a new paradigm in this industry. Putting your work out early into gamers’ hands shows immeasurable confidence in your own ability to take suggestions. It also shows Firaxis’ commitment to making games for gamers. You simply have no better way of demonstrating these qualities than through a public alpha/beta program. Perhaps "Furby's Mad World of Lawn Bowling" doesn't have fans who care enough to work through rough versions of the game. But Civ, we must admit, is no Furby.

** THIS WILL MAKE MONEY:

Finally, a public alpha/beta program will make you a great deal of money. First, each applicant for the alpha program will pay $10 for the disk (plus shipping and handling). The same for the beta stage. People who participated in the alpha and beta stages qualify to buy the finished game for $10. TOTAL SPENT ON CIV3: $30—a fair price, and you don’t have to wait months to collect it. People who participate in only the alpha or beta stage qualify to buy the finished game for $20. TOTAL SPENT ON CIV3: $30. Again, you get this money without having to wait weeks for these gamers to maybe buy the game. You also get the money from the alpha and beta stages without having to go through the very expensive process of publishing the game. Those who do not participate, of course, pay the full $50 price or wait for the price to naturally fall. But the momentum and positive buzz of the alpha/beta program will no doubt be so positive, and so many “Well, I doubt Firaxis can do this right” questions will have been laid to rest, that the average Civ fan will rush to buy the game, confident that gamer input and quality assurance have been your priority from day one.

I would suggest opening a password-access web site just for those who participate in the public alpha/beta program. A List room should be set up—similar to the one I’m running on Apolyton but limited to people participating in the program. The feedback on the Alpha becomes the roadmap for the Beta. The feedback on the Beta becomes the roadmap for the final version. The final version becomes the killer game that takes Firaxis and Civ to a whole new level of expertise and admiration.

Of course, you can do this in-house and risk putting all your eggs in one basket. Or you can put this for the most part into the gamers’ hands—and watch the money and guaranteed success come in.

I recently read an article in Business Week in which Bill Gates’ right-hand man—I can’t recall his name—talked about the most important thing he learned as he tries to restructure Microsoft and free it from its bureaucracy and bad public image. He said:

“I have become a leader of leaders.”

He has learned to let go enough so that his clenched fist becomes an open hand. The people below him are now empowered to make decisions and contribute. Brian, I encourage you to do the same.

Please let me know your thoughts on this,

Yin
(K. McLaughlin)

------------------
CIV3 DEVELOPMENT LIST COORDINATOR

**(un)Officially Making Lists for Firaxis Since SMAC Enhancement 3!**


[This message has been edited by yin26 (edited May 22, 1999).]

[Koyaanisqatsi]

'Over the top' at the end comes to mind, but hey, sometimes that's good.

I should say, though, that I don't think a public beta would be the best idea for Firaxis, for piracy reasons. I would certainly be in favor of one for all the reasons you mentioned (as well as for just because I want to play it) but I don't think it would be good for the game or the company.

But that's certainly no reason not to ask . . . I may just be too pessimistic for my own good . . .

[russellw]

I love this idea! But...

1. Wouldn't this seriously screw with the retail/distribution model that game companies go through? I don't think Hasbro would really sign off on something like this. I know that getting more money upfront is good, but not if it messes with their way of business. I don't really think this is Firaxis's call to make. (I know Microsoft charges for their betas, but don't the testers still have to pay normal price for the final?)

2. Expect thousands of people to be in on this. I haven't done any beta testing, but I assume it gets more confusing when you add on more and more testers. There will be a great deluge of save game files, people throwing out error codes, and a big headache with getting all of the versions straight.
The product would have to be at the stage of Ultima Online when UO went to beta. (Was anyone here a tester for that?)

Sorry to be so negative here. If there is a way to make it work, I'm all for it. If not, maybe Brian can give us a public preview of some of the graphics, the interface (with clickable buttons), and details of the design.

[yin26]

Bell,

LOL Yeah, I had some Top Gun music going in the background toward the end! Well, maybe it will inspire the man to let us gamers in on a larger scale. As for the secrets, if they've done the right job from alpha to beta, nobody will doubt that Firaxis is the place for this game. It's just too damn hard to simply rip off these ideas. Game balance in a game like this takes years to learn. Anyway, thanks for the commentary.

------------------
CIV3 DEVELOPMENT LIST COORDINATOR

**(un)Officially Making Lists for Firaxis Since SMAC Enhancement 3!**

[yin26]

russellw,

Good points. Firaxis would still have to go through all the publishing channels, of course, for the final version. And if anybody were to buy the game, it's no doubt going to be the alpha and beta testers (assuming Firaxis has been actively taking their feedback into consideration). So it's a very good trade off to give them a discount on the final game. The only way Firaxis could lose on this is if they aren't really committed to the player's ideas and just push on with their own stuff. In which case, all the publicity backfires in a BIG way. So they would have to do it right, there's no doubt about that.

[tfs99]

Good letter.

But wasn't it written by the same guy who resented the fact that gamers had to pay to beta test?

Ted If I volunteer to pay and see a sneek preview, that's a whole other story! *Yin*

Shoot for the stars, maybe we can end up hitting the moon!

Civ3 n ... Ted S.


[This message has been edited by yin26 (edited May 22, 1999).]

[the Octopus]

I don't want to rain on anybody's parade, but there are significant problems with this idea.

Firaxis design process is to have "working prototypes" that iteratively evolve into a game. Now, that is usually a pretty bad ideas from a Software Engineering perspective, which means there are likely to be lots of bugs in their alpha and beta phases. So, instead of having a core group of people who are familiar with: a bug reporting procedure, debugging methodology, and the general state of game development you have any schmoe off the street who can give you ten bucks. This will be more trouble than it is worth for the programmers at Firaxis, because they will get tons of conflicting and confusing reports, and will need to wade through them all, working out which problems are real problems, which are due to innappropriate use, which are known bugs, etc., etc., etc. I've read on a website somewhere that the person who managed the SMAC beta thought it was worthwhile, but found that the biggest problem was that the non-in-house beta testers were unfamiliar with the process. What does this mean from a quality perspective? It means that people are spending their time debugging bug reports instead of debugging software.

As has already been said, this also undercuts the traditional retail distribution chain. No development house can expect to survive if it undersells both its publisher and the retail market. It can't work from a business perspective. Firaxis does not have the financial or production capacity to be its own publisher (and I doubt that they want to be their own publisher), therefore they can not engage in any shenanigans.

For the SMAC beta-test, they actively recruited and interviewed people that they thought would be good beta-testers. I remember when they posted their initial call on the SMAC forums, and there were plenty of "dammit, I'm not qualified" posts. Remember, a beta-tester isn't somebody who gets a sneak preview, they are somebody who is working to make the game better.

I personally dislike the software industry's attitude toward bug-finding. In general, bug-finding for them is an afterthought, done by "lowly" individuals. I personally think that's stupid. The solution to buggy software is a commitment to bug-finding up front, though, not a whole bunch of people pounding on things at the back-end.

I do not think that a wide public beta test will happen. They will probably repeat what they did for SMAC, which is to gather a small number of people that they think is manageable. There are good reasons for them to do it this way, from their perspective. There are good reasons for them to totally overhaul the way they look at bug-finding as a part of the process of software design, but that is another issue for Firaxis to deal with internally.


------------------
CIV3-THE MASTER LIST-TECHNOLOGY "THREAD MASTER"

[Rong]

In support of Yin's letter, I suggest everyone read the paper: <a target="_blank" href="http://www.tuxedo.org/~esr/writings/cathedral-bazaar/">The Cathedral and the Bazaar</a>.

Actually I have been thinking about petitioning for Open Source Civ3. But I haven't come up with a business model that could work. Yin, maybe you can after reading that paper.

------------------
The radical invents the views. When he has worn them out the conservative adopts them.
- Mark Twain


[This message has been edited by Rong (edited May 22, 1999).]

[Druid2]

Rong: I think that "OpenSource Civ" is asking them to give away the store and the candy at the same time. "Open Source AI" would be more than enough to put Civ3 on top of the heap for the forseeable future.

Yin: I dont think that an "open" alpha / beta testing procedure would provide useful info. Or, more accurately said, the volume of data would be overwhelming. They need to have a *larger* group of alpha & beta testers, but not an unlimited group. As for pay... well your plan changes their cash flow to the positive, so I'm thinking they'll like that. And it *does* lower the testers eventual cost, so there is an reward to the tester.

[LordStone1]

Perhaps an open-alpha test is a bit excessive - so how about only an open-beta test.

No, wait, it'd be better the other way around. Have an open-alpha, then a closed-beta. This way, Firaxis will have more time to wade though all the suggestions of the alpha test, PLUS, have a good pool of candidates to choose from for the beta stage. I know that no matter what, I would not want to be Firaxis in the middle of an open-beta test. It'd just be too much.

[UCK]

I've been a tester for both alpha and beta software. An open-public model wouldn't necessarily work for the following reasons. A lot of "alpha" software is just that, the first build of the program. Instead of graphics, lots of numbers, crappy BMP graphics usually in 16 colours, most of the options aren't working, and quite often, it doesn't even run at all!

An "alpha" build usually only concentrates on getting the main game engine up and running. I was on an alpha test of a civ-type game (which got scrapped) and there was no AI, only YOU were on the world. All you could build were settlers and militia, no researching, no diplomacy, nothing. All the alpha did was click through the years and move settlers and militia around. No barbarians, no improvements, no wonders, no gold. In fact the years weren't "years" but just turn numbers. There were no "screens" like city status or anything. You couldn't do anything. Would 2.5 million (Yin's figure) accept paying $10 for an alpha that you can't do anything in? AND then have to fill in wads of bug report sheets?

But in essence, an open-beta may be successful. Any programmer knows that the best way to bug-check a piece of software is give it to the "dumbest" computer user. By dumbest, I don't mean it literally, but more like computer illiterate. In the public, what better mix do you have? You have (like me) highly qualified computer technicians, software developers, people who don't even know how to use a mouse properlly, quite simply, the entire mix (not meaning any offence ). Also, a beta is usually with most (if not all) the options turned on.

So while IMO an open-alpha test would NOT work, an open-beta test may actually provide the developers with good feedback. Plus, in reality it'll shut those naggers who like with CTP, ordered Activision to publish the game before it was complete, cuz they have something to get their hands dirty with.

[meowser]

Everyone is making good comments for their opinions. I'm glad you started this post Yin.

Firaxis whatever type of testing scheme you use, I have one request: listen to your testers and make sure they have no GRIPES before releasing the game. I remember quite a few post from some of the beta testers for CTP remarking how a lot of their suggestions, concerns and bugs they found were not addressed at all. You don't have to implement any of their suggestions, but you BETTER address all BUGS that they find.

Public testers, if it appears Firaxis is giving you guys the run a round, I implore you to hurt them where counts, by posting and posting to forums such as this.

[evil conquerer]

I think that a public beta would give Firaxis too much information to be useful. Imagine several hundred thousand people all submitting lenghty e-mails about the game.

How about a slightly more limited beta test? Such as just, say, all the thread masters for the suggestion list?

[This message has been edited by evil conquerer (edited May 24, 1999).]

[Sieve Too]

Having been on both ends of the s/w testing stick, I would have to recommend against open alpha testing and allow only limited public beta testing. An organization can only absorb only so much data, and some of it will be facetious or incomplete, e.g. "I don't like the graphics" or "The AI sucks" or "I could have spent a better day fishing than playing this game" - all gripes lacking the necessary details needed to address the concerns. These are all followed by the later "told-ya-so"s. "See, I told them the AI sucked and they didn't fix it!"

[Blade Runner]

IMHO we need an public early beta to get feeling about the end product. After we can send back our opinion about the game, so Firaxis can use our feedback to impove the game play. They can send the later betas (which include already the changes) to the beta testers to rid off all the bugs. This part must be closed (Not everybody can or wants to write a correct bug report).

Blade Runner

[Shining1]

Hmmm. Basically, it sounds good okay - BUT there are issues. That's not to dismiss the idea - but to be realistic about the effort manifest in this idea.

Numbers in a public beta: the last one they did had 25. That's enough of a minor management crisis to contend with. Having 100's of people in a logicistal nightmare.

Quality: Apparently, the game was improved ENORMOUSLY - graphically, gameplay, the whole works. People criticise SMAC now - I agree, incidentally - but by the sounds of it the current version is an angel compared to the dog we might have received.

But most of the problems concerned the infrequent crashes and the end game bugs. Having more people involved doesn't help these things much - bugs can creep into a gold candidate as easily as they can a beta build. And it took me about a month to get properly into the endgame of CivII. Unless you make a concerted effort to get people to test the whole thing, you end up with a lopsided product with a flawless beginning and a nominal endgame.

Greasy letter: I think you should start collecting signatures now. This kind of this requires a major groundswell of support, rather than a couple of greasy letters (although I quietly suspect they're planning one anyway - so you might score a major coup if you get your timing right ).

The money thing: I dunno. If I'm going to test a game, I don't want to pay. I want to be paid - at least with a free game. Beta testing is hard work, and often not fun.

"Hi Brian,

CivIII crashed again. I was 5 turns into a polynesian game. Here's the save game file and the windows error message. Hope this helps!

Cheers,
Ned."

Not the best way to spend a weekend.

Never the less, if we can assemble a list of ideas, we're qualified to test the game too. Add my name to the list.

Shining1

[the Octopus]

I'll chime in again about the actual content of the letter: The basic premise is "you screwed up SMAC bigtime, don't screw up again, or else!". You catch more flies with honey...

I truly doubt that having a large number of people as alpha or beta testers is realistic. Firaxis is a small company, with a few programmers on a project. The kind of thing you want them to organize is a logistical nightmare, and of questionable utility from a bug-finding perspective. There are better ways to produce a quality product. If you simply want them to give you a vehicle so you can criticize the game engine, etc., then ask for a demo, not an alpha version.


------------------
CIV3-THE MASTER LIST-TECHNOLOGY "THREAD MASTER"

[yin26]

The idea of a limited beta does sound better. Of course, the importance of our list should be stressed in such a scenario: This list will be one of (if not THE) best indications for Firaxis what we would like to see BEFORE they get so far along that it's too hard to turn back and 'fix' it.

Thanks for all the good replies.

Yin

------------------
CIV3 DEVELOPMENT LIST COORDINATOR

**(un)Officially Making Lists for Firaxis Since SMAC Enhancement 3!**

[Blade Runner]

Shining1,

I agree with the problem to manage 1500 people is a nightmare. But my idea was the next:
1. Early beta to everybody -> we can send feedback about the CONCEPTS, INTERFACE of the game.
2. Closed beta to rid off the bugs.

Firaxis can send a kind of compatibility list (let say pentium proc, matrox G200,..., EIDE or SCSI HD ...) which parts are OK to run this early beta. If your machine is compatible with the game you can use it, if not, you need to wait for the end product.

Blade Runner

[Sieve Too]

Blade Runner: I'm still not convinced. I guess I've seen too many, "I don't like the sound" type comments to believe that a large group will be disciplined in their criticisms. Again, you open yourself up to a backlash when you don't respond the way people want you to. And even if you had a mind reader to figure out everyone's ramblings, some ideas will contradict. In a large group there is no way to get consensus. Someone will inevitably get pissed off and tell their friends not to bother buying the game since those "arrogant programmers" don't listen to good ideas. Once a negative buzz gets on the net, you might as well cancel the project.

[kmj]

Of course, they could organize some sort of forum for people to post their complaints/bugs/etc. They could have different sections for each different type of thing. This way they could weed out the more important posts. If it was organiized at all, instead of "send all comments to Brian Reynolds' desk", it shouldn't be that bad. And the fact is, the larger the pool of testers, the more likely you are to find problems. I refer you to Rong's earlier post about the Cathedral and the Bazaar.

I like the idea, but that may just be because I want to get my hands on it.

[Brother Greg]

A public beta test will never work. One reason? A beta test usually goes through multiple stages. SMAC I believe went through about ten. So, that's 10 CDs to 25 beta testers. 250 CDs with some simple math.

And guess what - they were only for playbalancing, not for bug finding - this was a mistake.

Multiply that by about, um, one hundred thousand to a million or more for an open beta (we are talking about CIV here). And then try and make money from it. You'd have to charge $200+ for the price of being a beta tester.

Plus, do any of you have any idea of the amount of crap that is actually involved in testing? It ain't just "play this game through 100 times, and tell us what you think". It is stuff like when you discover a bug, document everything you did in the last turn, if you could recreate it, and if you could, if you could avoid it, and how. Testing isn't fun, and a lot of people here seem to think that it is.

And as for anybody wading through all of the replies, trying to correlate them, and so on and so on, forget it. It would take hundreds of people.

Nice idea, but it will never happen. My reasons are just some of the more obvious.

[Travathian]

Brother Greg, you're ignorance is appalling.

Do you think any software maker would do a full public beta? Not a chance. Maybe a few hundred people, but nothing like a hundred thousand to a million. And as far as all of your cds. Know how much it costs to make a cd? Less than a buck. And if the update was small, they'd just put it on a secure site for downloading, not just send out a new disk. Granted they might do that for major revisions, but not every little thing.

Now I will agree about the playtesting for game balance only is a shame on their part.
Beta testing should be for bug fixing, playtesting, stability, etc.

[Dominique]

Please forget about that Beta thing, tempting as it might seem.

Reasons:

a) AFAIK no game company has done a PUBLIC beta until now - why should this change now?

b) Let's a ssume a public beta is out - what could happen?
- the beta is bad, and Firaxis gets negative feedback... baaaaaaaad marketing
- the beta is good, and people enjoy it so much that the tension (aka "motivation to buy") is lessened - after all, why pay $40 for some slight improvements when you could play the core game for $10?

c) Activision showed us how to successfully sell a game (sadly so): Make a big hype, sell a public beta as release, and THEN patch it (after the people have paid full price).

So?

[Shining1]

Personally, I think that firaxis will handle the beta the same way they handled SMAC - a small committed group of americans(?) to do the hard yards, and then a demo release to generate widespread public feedback.

Okay, it didn't work for SMAC. My reasoning - the game wasn't really up to scratch when it was released to the testers. Thus you had the 'improvement' process problem - the game did get a hell of a lot better while being tested, thus making everyone very happy and hence missing the fact that it wasn't yet 100%.

To get around this, I suggest a two stage public beta - 25 people to start with and then another 25 a month or so later. Hence you don't get the 'added value' problem with the second round of players, who see a markedly improved version of CivIII as the starting point, and not as a mountain of hard work and argument.

And you get brian to include the sea borders, under threat of physical violence, if nothing else works.

Travathian: I don't get it. Didn't BG just say the same thing? How is then appallingly ignorant?

The 'flame BG sh*tless over nothing' tactic seemed to work for Yin - I guess it's worth a try .

Shining1