Not enough beta testers? - Alpha Centauri Forums

  Alpha Centauri Forums
  The Game
  Not enough beta testers?

profile | register | prefs | faq | search

Author Topic:   Not enough beta testers?

TheHelperMonkey posted 02-23-99 08:57 PM ET
I'm not bashing firaxis, but was 25 people enough to test the product? I mean, they could not have possibly tested every game condition. Like, running SMAC using a Hatachi dvd drive while using Dx6.1 and a RivaTnT card. It seems the some people are having ALOT of problems running this game (If you go to the troublshoot forum, there is at least a bug complaint posted every hour). I think there should've have been at lease 100 beta testers testing for a about a year (?) to get *most* of the bugs out.
What's you oponion? Too little or just right?
Brother Greg posted 02-23-99 09:48 PM ET
No matter how many beta testers you have, you will NEVER get all of the bugs. Well, unless you use every single person. The point is that Firaxis write their program to utilize DirextX. Now, the fact that people's drivers are not compliant (like my Sound Blaster Live) isn't a Firaxian bug. Frankly, if the program won't run on System A, but will run perfectly on System B, then it is a fault of the system, no the game.
That is the great advanatage of DirextX - weite the code, and as long as the drivers are tested and certified, the game will work. So, most of the time, the problem comes down to the syetem.
Of course there was the bug implemented when they put out the patch, but beta testing wouldn't have helped that.
So, to answer your question, the difference between 25 people and 100, beta testing for a year (rather than 3 months) would have made very little difference.
DanS posted 02-23-99 11:02 PM ET
If they would have had 100 testers, it would have been more difficult to keep any of the surprising and good features secret. Everybody would be getting their information second-hand, which would just cause impatience and anger (when the info was wrong).
DanS posted 02-23-99 11:10 PM ET
Further, to answer your broader question. I don't mind being a guinea pig, if it will get me a good game faster. This method is much better if patches are sure to be released because I will have my chance to make suggestions about changes. Just think Netscape and the endless beta. It is a more enlightened way to put out a piece of software, in my opinion.
Besides, Firaxis is doing much better than about 99% of other companies selling a $35 piece of software.
I have nightmares about how many complaint posts there would be if this game sucked...
Scrubby posted 02-23-99 11:13 PM ET
In order to be effective beta testing (like any quality assurance) has to have a strictly defined goal. Firaxis determined that 25 people were enough to achieve their goal which may or may not have anything to do with technical compatibility (for example, they could have been trying to break game balance solely...). As others have pointed out technical bugs are impossible to eliminate and sometimes, as mentioned are not even really "bugs" just a product of all the bizarre combinations of peripherals and hardware (and software). Also sample size and coordination of product testing is problematic when too large. There's also the law of diminishing returns, Firaxis felt they could get the best results out of a group of 25 so they did it. This also isn't a matter of generating hype as someone mentioned. I would point to Everquest or other large "beta" testing in which hype generated by participants invites interest in a game rather than quenching it. I'm sure Firaxis would love to generate buzz about their games via public beta test etc. But it just wasn't deemed necessary in this case (hell, you guys bought the game anyways right? )
Freddz posted 02-24-99 12:01 AM ET
Unfortunately I have to agree with Monkey, there are just too many bugs. The game really should have been tested more, not neccessarily by a lot more people, but longer.
Blizzard's Starcraft is a good example on what a lot of testing can accomplish. When the game came out there wasn't a whole lot of bugs, and the game was excellently balanced(at least as good as it one can demand). We will see more complaints in the future when MP gets rolling on that some factions don't do good, and other do too good and probably a few of those posts will be right.
Blizzard spent more than a year to balance Starcraft.
The game works good for me as I like it overall and that some of the features are eally great, but even so I see too many strange bugs, some of them which was reported before the game went gold and which are still there.
But worse than the bugs, are for me the Unit Design screen, not that it isn't good and a fun feature, more that it could have been both a great and easy to use feature with the comp remambering your favorite units when the Design Automatically was on and suggesting only them, and with easier upgrading so that when you get Particle Impactor and had laser before the comp could ask you if you would replace those units even tho prototype wasn't finished and so on
Firaxis could have made a real statement if they had fixed most of those bugs
Personally, I don't think they should have released it before they had corrected the game fully, but then again maybe they were getting scared people would tire of it as the 100 turn limit was blown
Still I very much doubt that that any turn-based game will come close to this one for quite a while, but then again - what a game it could have been
Glak posted 02-24-99 01:24 AM ET
Yeah you're right about Starcraft. I was going to post the same thing earlier but I had to get an account.
Blizzard knows how to do beta tests right. They got 1000 people for six weeks to test the product over battle.net, that is just the right way to do it. There have only been a few bugs found ever and all but one of them involved pressing bizzare combinations of buttons or situtations which are extremely rare (like they won't happen anywhere near 1 out of a million games). I also think that NDA agreements are silly unless the beta is going to be longer than 2 months. There is no reason to hide the game unless its quality is lacking.
Fortunately Blizzard fixes them quickly. That is what keeps company loyalty up. If I just came out of a coma I would go right to the store and buy all Blizzard games without even looking them over first (well I haven't played Diablo but I hear Diablo 2 will be good).
mouse posted 02-24-99 12:27 PM ET
I have to disagree that Firaxis should have used more beta testers, or that Blizzard always does it right. This is the second game the Firaxis has released, Starcraft is Blizzard's fourth/fifth title. I certainly don't remember a huge beta test of Warcraft2 or Diablo. Brian Reynold's has an excellent record of support. Give the company time to grow before grumbling about big beta tests.
Abdiel posted 02-24-99 12:48 PM ET
As per the Firaxis design policy, Brian would have had a prototype of the game up and running as soon as possible, which means that all through development (how long is that now?) would have been the balancing phase. That would, I think put even Blizzard's efforts to shame.
TheHelperMonkey posted 02-24-99 04:29 PM ET
Brother Greg: If my video card is incompatible, whose fault is it? Mine, becuease I shouldnt've bough the game, or the designers for not making my system incompatible with the game. It's the designers fault obviously. Should I have bought a good 35$ game or a good 150$ video?
I think that if you make a product that only supports certain hardware, then that would get alot of people pissed. I'm not saying they should support every combination. But most of them.
I also think that if they would have released the game earlier if they had more beta tester.
Glak posted 02-24-99 04:38 PM ET
mouse: you are right about that. It is a new company so I expect them to make mistakes, just like blizzard still makes some mistakes.
The important thing I think it to realize that it is a mistake and to try to improve. I'm sure eventually they will adopt a better method or maybe even invent a better way to do things (and then all the other companies will copy it).
I think that you are right about war2 not having a big beta test (I didn't hear about war2 until early 97 so I wasn't around) but I think that if blizzard could go back that they would have a big beta for all of their games.
I am not one of those people in favor of open beta tests though. Once you get too many people it becomes hard to manage the thing and people start hacking the product. I think that 1000 sounds about right.
Abdiel: Your message was kind of short so I am not sure what you meant. I am sure that firaxis had an early version designed fairly early into the design process. Blizzard did the same (they actually started with the warcraft engine but scrapped it because everyone made fun of them). I am not saying that you should have a big beta test throughout the design process. I think that is should be a final step to perfect the game.
Follol posted 02-24-99 04:41 PM ET
THM: Firaxis coded for DirectX, which does support "most combinations" of hardware. They did their job.
The fault lies either with the video card manufacturer for not supporting DirectX, or Microsoft for not using the information the video card company supplied them.
Oh, I almost forgot. It could be your fault, if you aren't running the latest DirectX (assuming that the latest DirectX fixes the problems). The preceding sentence is NOT NOT NOT a personal attack!
BikeDude posted 02-24-99 05:01 PM ET
Lets not forget the UK copyprotection scheme "bug" ("feature" if you work for EA).
I had the demo up and running without any incidents for weeks. And now the fullversion doesn't run properly under Win2000 (which despite being a beta is my main OS at home).
Now, _that's_ annoying (when some dildohead of a distributor in Europe adds stuff that breaks the functionality you had in the demo).
DirectX is a nice abstraction from tedious compatibility problems that haunted some games during the good ol' DOS days. Anyone remember when "HiColor" graphics adapters first came out, and when John Carmack stated that Doom would never utilise 16 bit colors due to compatibility issues? <G>
However, the cheap hardware these days usually reflect the cheap support you usually get when reporting a problem. Unless its a known issue, then they either fix it or tell you to go away. (I know about two reproducable bugs with ATI XPert@Work that ATI doesn't bother with)
--
Rune
Imran Siddiqui posted 02-24-99 05:35 PM ET
Yes, I agree with mouse. Firaxis isn't a huge company. It doesn't have that much capital and so forth, so have 100 beta testers might have been a little hard.
Look at Civ2. Microprose was a big game designer and had a lot of cash. It could use more beta testers and moer testing time. As a result, Civ2 had almost NO bugs!
Imran Siddiqui
DanS posted 02-24-99 05:47 PM ET
But Firaxis' size has nothing to do with whether they "should" have had 100 beta testers.
Again, give the imperfect game to me as soon as possible and give me a forum in which to express my opinion, offer suggestions, and get help when difficulties arise. Like this one.
Andrew Goldstein posted 02-24-99 06:19 PM ET
I was a one of the beta testers.
On my system, the initial release version 1.0d was extremely stable (only one game freeze). It seems that a lot of "bugs" described here is due to hardware. The system I beta tested SMAC on is a homebuilt system (PII-333, Assus PB2-LS m/b, 64 MB PC100 ram, Millenium G200 AGP vid card, Seagate SCSI-II 4.5 GB hd, a soundblaster Awe 64. I made sure that the components wheren't some no-name crap stuff). I don't know what the other testers used, but I suspect that they used systems better and worse than the one I used. FIRAXIS did ask for a detailed system list.
A lot of the testing we public beta testers did was more in playbalancing, internet multiplayer game snags and overal game impressions. The heavy duty bug hunting was done by a highly respected professional group (in the game design circles) called Absolute Quality Inc.
The other point you you may not be aware of is the fact that in the past, Sid & company never ever had a public beta test of any of their games (under FIRAXIS or MPS) until SMAC (FIRAXIS staff were divided on the issue of having any public betatest in the first place - fortunately it worked out very well all around). Basically, they dumped various beta versions of SMAC on us (with NO manual), and did not send us any explicit instructions other than to write a "first" impressions document and to (later) play some multiplayer games over the net connected to a FIRAXIS server.
There was a private forumn where we could write about the various "bugs" found and ask questions and make suggestions. I think the forumn was the most useful tool (I heard it was even more useful than the database the professional group used which a few of the public beta testers had access to). The major problem I found during the beta test was that definitons of what is a "bug" "crash" "freeze" "logic flaw" and names of the various menus and windows area were not given to me, which made it harder for me to exactly describe the problems I had. It is quite possible that one of us found a bug that the posters of this forumn are struggling with, but becuase we didn't konw the "lingo" used by the FIRAXIS programmers, didn't describe them quite right so didn't get fixed (Also, bug resolutions were not divulged to us either). But again, please keep in mind, FIRAXIS never did a public beta before, so this was a "beta" process to them also. Hopefully the next public beta test will be a bit better designed.
To close this thread, click here (moderator or admin only).


Hop to:

Contact Us | Alpha Centauri Home

Powered by: Ultimate Bulletin Board, Version 5.18
© Madrona Park, Inc., 1998.

Author	Topic: Not enough beta testers?
TheHelperMonkey	posted 02-23-99 08:57 PM ET I'm not bashing firaxis, but was 25 people enough to test the product? I mean, they could not have possibly tested every game condition. Like, running SMAC using a Hatachi dvd drive while using Dx6.1 and a RivaTnT card. It seems the some people are having ALOT of problems running this game (If you go to the troublshoot forum, there is at least a bug complaint posted every hour). I think there should've have been at lease 100 beta testers testing for a about a year (?) to get most of the bugs out. What's you oponion? Too little or just right?
Brother Greg	posted 02-23-99 09:48 PM ET No matter how many beta testers you have, you will NEVER get all of the bugs. Well, unless you use every single person. The point is that Firaxis write their program to utilize DirextX. Now, the fact that people's drivers are not compliant (like my Sound Blaster Live) isn't a Firaxian bug. Frankly, if the program won't run on System A, but will run perfectly on System B, then it is a fault of the system, no the game. That is the great advanatage of DirextX - weite the code, and as long as the drivers are tested and certified, the game will work. So, most of the time, the problem comes down to the syetem. Of course there was the bug implemented when they put out the patch, but beta testing wouldn't have helped that. So, to answer your question, the difference between 25 people and 100, beta testing for a year (rather than 3 months) would have made very little difference.
DanS	posted 02-23-99 11:02 PM ET If they would have had 100 testers, it would have been more difficult to keep any of the surprising and good features secret. Everybody would be getting their information second-hand, which would just cause impatience and anger (when the info was wrong).
DanS	posted 02-23-99 11:10 PM ET Further, to answer your broader question. I don't mind being a guinea pig, if it will get me a good game faster. This method is much better if patches are sure to be released because I will have my chance to make suggestions about changes. Just think Netscape and the endless beta. It is a more enlightened way to put out a piece of software, in my opinion. Besides, Firaxis is doing much better than about 99% of other companies selling a $35 piece of software. I have nightmares about how many complaint posts there would be if this game sucked...
Scrubby	posted 02-23-99 11:13 PM ET In order to be effective beta testing (like any quality assurance) has to have a strictly defined goal. Firaxis determined that 25 people were enough to achieve their goal which may or may not have anything to do with technical compatibility (for example, they could have been trying to break game balance solely...). As others have pointed out technical bugs are impossible to eliminate and sometimes, as mentioned are not even really "bugs" just a product of all the bizarre combinations of peripherals and hardware (and software). Also sample size and coordination of product testing is problematic when too large. There's also the law of diminishing returns, Firaxis felt they could get the best results out of a group of 25 so they did it. This also isn't a matter of generating hype as someone mentioned. I would point to Everquest or other large "beta" testing in which hype generated by participants invites interest in a game rather than quenching it. I'm sure Firaxis would love to generate buzz about their games via public beta test etc. But it just wasn't deemed necessary in this case (hell, you guys bought the game anyways right? )
Freddz	posted 02-24-99 12:01 AM ET Unfortunately I have to agree with Monkey, there are just too many bugs. The game really should have been tested more, not neccessarily by a lot more people, but longer. Blizzard's Starcraft is a good example on what a lot of testing can accomplish. When the game came out there wasn't a whole lot of bugs, and the game was excellently balanced(at least as good as it one can demand). We will see more complaints in the future when MP gets rolling on that some factions don't do good, and other do too good and probably a few of those posts will be right. Blizzard spent more than a year to balance Starcraft. The game works good for me as I like it overall and that some of the features are eally great, but even so I see too many strange bugs, some of them which was reported before the game went gold and which are still there. But worse than the bugs, are for me the Unit Design screen, not that it isn't good and a fun feature, more that it could have been both a great and easy to use feature with the comp remambering your favorite units when the Design Automatically was on and suggesting only them, and with easier upgrading so that when you get Particle Impactor and had laser before the comp could ask you if you would replace those units even tho prototype wasn't finished and so on Firaxis could have made a real statement if they had fixed most of those bugs Personally, I don't think they should have released it before they had corrected the game fully, but then again maybe they were getting scared people would tire of it as the 100 turn limit was blown Still I very much doubt that that any turn-based game will come close to this one for quite a while, but then again - what a game it could have been
Glak	posted 02-24-99 01:24 AM ET Yeah you're right about Starcraft. I was going to post the same thing earlier but I had to get an account. Blizzard knows how to do beta tests right. They got 1000 people for six weeks to test the product over battle.net, that is just the right way to do it. There have only been a few bugs found ever and all but one of them involved pressing bizzare combinations of buttons or situtations which are extremely rare (like they won't happen anywhere near 1 out of a million games). I also think that NDA agreements are silly unless the beta is going to be longer than 2 months. There is no reason to hide the game unless its quality is lacking. Fortunately Blizzard fixes them quickly. That is what keeps company loyalty up. If I just came out of a coma I would go right to the store and buy all Blizzard games without even looking them over first (well I haven't played Diablo but I hear Diablo 2 will be good).
mouse	posted 02-24-99 12:27 PM ET I have to disagree that Firaxis should have used more beta testers, or that Blizzard always does it right. This is the second game the Firaxis has released, Starcraft is Blizzard's fourth/fifth title. I certainly don't remember a huge beta test of Warcraft2 or Diablo. Brian Reynold's has an excellent record of support. Give the company time to grow before grumbling about big beta tests.
Abdiel	posted 02-24-99 12:48 PM ET As per the Firaxis design policy, Brian would have had a prototype of the game up and running as soon as possible, which means that all through development (how long is that now?) would have been the balancing phase. That would, I think put even Blizzard's efforts to shame.
TheHelperMonkey	posted 02-24-99 04:29 PM ET Brother Greg: If my video card is incompatible, whose fault is it? Mine, becuease I shouldnt've bough the game, or the designers for not making my system incompatible with the game. It's the designers fault obviously. Should I have bought a good 35$ game or a good 150$ video? I think that if you make a product that only supports certain hardware, then that would get alot of people pissed. I'm not saying they should support every combination. But most of them. I also think that if they would have released the game earlier if they had more beta tester.
Glak	posted 02-24-99 04:38 PM ET mouse: you are right about that. It is a new company so I expect them to make mistakes, just like blizzard still makes some mistakes. The important thing I think it to realize that it is a mistake and to try to improve. I'm sure eventually they will adopt a better method or maybe even invent a better way to do things (and then all the other companies will copy it). I think that you are right about war2 not having a big beta test (I didn't hear about war2 until early 97 so I wasn't around) but I think that if blizzard could go back that they would have a big beta for all of their games. I am not one of those people in favor of open beta tests though. Once you get too many people it becomes hard to manage the thing and people start hacking the product. I think that 1000 sounds about right. Abdiel: Your message was kind of short so I am not sure what you meant. I am sure that firaxis had an early version designed fairly early into the design process. Blizzard did the same (they actually started with the warcraft engine but scrapped it because everyone made fun of them). I am not saying that you should have a big beta test throughout the design process. I think that is should be a final step to perfect the game.
Follol	posted 02-24-99 04:41 PM ET THM: Firaxis coded for DirectX, which does support "most combinations" of hardware. They did their job. The fault lies either with the video card manufacturer for not supporting DirectX, or Microsoft for not using the information the video card company supplied them. Oh, I almost forgot. It could be your fault, if you aren't running the latest DirectX (assuming that the latest DirectX fixes the problems). The preceding sentence is NOT NOT NOT a personal attack!
BikeDude	posted 02-24-99 05:01 PM ET Lets not forget the UK copyprotection scheme "bug" ("feature" if you work for EA). I had the demo up and running without any incidents for weeks. And now the fullversion doesn't run properly under Win2000 (which despite being a beta is my main OS at home). Now, _that's_ annoying (when some dildohead of a distributor in Europe adds stuff that breaks the functionality you had in the demo). DirectX is a nice abstraction from tedious compatibility problems that haunted some games during the good ol' DOS days. Anyone remember when "HiColor" graphics adapters first came out, and when John Carmack stated that Doom would never utilise 16 bit colors due to compatibility issues? <G> However, the cheap hardware these days usually reflect the cheap support you usually get when reporting a problem. Unless its a known issue, then they either fix it or tell you to go away. (I know about two reproducable bugs with ATI XPert@Work that ATI doesn't bother with) -- Rune
Imran Siddiqui	posted 02-24-99 05:35 PM ET Yes, I agree with mouse. Firaxis isn't a huge company. It doesn't have that much capital and so forth, so have 100 beta testers might have been a little hard. Look at Civ2. Microprose was a big game designer and had a lot of cash. It could use more beta testers and moer testing time. As a result, Civ2 had almost NO bugs! Imran Siddiqui
DanS	posted 02-24-99 05:47 PM ET But Firaxis' size has nothing to do with whether they "should" have had 100 beta testers. Again, give the imperfect game to me as soon as possible and give me a forum in which to express my opinion, offer suggestions, and get help when difficulties arise. Like this one.
Andrew Goldstein	posted 02-24-99 06:19 PM ET I was a one of the beta testers. On my system, the initial release version 1.0d was extremely stable (only one game freeze). It seems that a lot of "bugs" described here is due to hardware. The system I beta tested SMAC on is a homebuilt system (PII-333, Assus PB2-LS m/b, 64 MB PC100 ram, Millenium G200 AGP vid card, Seagate SCSI-II 4.5 GB hd, a soundblaster Awe 64. I made sure that the components wheren't some no-name crap stuff). I don't know what the other testers used, but I suspect that they used systems better and worse than the one I used. FIRAXIS did ask for a detailed system list. A lot of the testing we public beta testers did was more in playbalancing, internet multiplayer game snags and overal game impressions. The heavy duty bug hunting was done by a highly respected professional group (in the game design circles) called Absolute Quality Inc. The other point you you may not be aware of is the fact that in the past, Sid & company never ever had a public beta test of any of their games (under FIRAXIS or MPS) until SMAC (FIRAXIS staff were divided on the issue of having any public betatest in the first place - fortunately it worked out very well all around). Basically, they dumped various beta versions of SMAC on us (with NO manual), and did not send us any explicit instructions other than to write a "first" impressions document and to (later) play some multiplayer games over the net connected to a FIRAXIS server. There was a private forumn where we could write about the various "bugs" found and ask questions and make suggestions. I think the forumn was the most useful tool (I heard it was even more useful than the database the professional group used which a few of the public beta testers had access to). The major problem I found during the beta test was that definitons of what is a "bug" "crash" "freeze" "logic flaw" and names of the various menus and windows area were not given to me, which made it harder for me to exactly describe the problems I had. It is quite possible that one of us found a bug that the posters of this forumn are struggling with, but becuase we didn't konw the "lingo" used by the FIRAXIS programmers, didn't describe them quite right so didn't get fixed (Also, bug resolutions were not divulged to us either). But again, please keep in mind, FIRAXIS never did a public beta before, so this was a "beta" process to them also. Hopefully the next public beta test will be a bit better designed.