Surprise attack bonus - Page 3

DrSpike · January 31, 2003, 05:16

Oh and as for the independence query........I take your point, because I expressed myself badly. Check out the info: combat thread for the real formula, and to see why the results from any one combat isn't bog-standard binomial.

rah · January 31, 2003, 09:39

I figure the test will include 4 combats a turn.
Human vs. Human.

4 warriors against 4 warriors all on neutral terrain.

1. warrior 1 surprise attacks enemy warrior1
2. warrior 2 attacks enemy warrior2.

make peace.

3. warrior 3 surprise attacks enemy warrior 3
4. warrior 4 attacks enemy warrior 4.

reload, and repeat all 4 attacks for a minimum of 25 times.

if the results aren't obvious in 25 rounds I'll be shocked.
I'll bet, on the first surprise attack, that the attacking unit will win 18 or more of the 25 encounters, and 22 or 23 won't surprise me. I'll be real interested to see if the results of attack 3 match attack 1 or if they're consistent with 2 and 4.

RAH

DrSpike · January 31, 2003, 10:18

Quote:

Originally posted by rah

if the results aren't obvious in 25 rounds I'll be shocked.
I'll bet, on the first surprise attack, that the attacking unit will win 18 or more of the 25 encounters, and 22 or 23 won't surprise me.

RAH

The point is you have picked 18 completely arbitrarily, with no idea what the probability of observing 18 would be even without a bonus.

DrSpike · January 31, 2003, 10:19

Oh and there is absoutely no point to providing an attack whilst at war for control purposes.......since we have perfect knowledge in that state anyway.

rah · January 31, 2003, 10:47

No to both posts.
18 was not quite so arbitrary. I'm assuming a similar bonus as barbs attacking humans at raging (50%)

I using a control because I'm betting that EVERY test previous was not done Human vs Human, but Human vs AI. I'm not expecting any difference, but why take any chances?

RAH

DrSpike · January 31, 2003, 10:52

18 is arbitrary, unless you have calculated the effect of the number of repititions on the variance in the proportion of wins within the sample......which you haven't.

And the second point was the probability of a warrior beating another with no bonus can be derived exactly from first principles......there is no need to sample this probability.

But whatever......it's your test, do it whatever way you feel is best.

rah · January 31, 2003, 11:06

18 was based on assumptions and years of experience so it's more of a swag. Researchers find it hard not to anticipate/reason a possible result prior to the test.
Why don't you make a guess and we'll see who's closer.

WHAT PRINCIPLES. Those have all been developed and tested human vs AI. Those principles have never been proven to be valid for human vs human. NO ONE has ever published a test on human vs human. Again, my guess is that there will be no difference, but until it's tested, it's just another assumption.

RAH

DrSpike · January 31, 2003, 11:16

Quote:

Originally posted by rah

18 was based on assumptions and years of experience so it's more of a swag. Researchers find it hard not to anticipate/reason a possible result prior to the test.
Why don't you make a guess and we'll see who's closer.

RAH

Your missing the point. Say the warrior wins 17 - what will your conclusion be? And what about 19? And the absolute critical point is that the number you choose as your cutoff point for evidence of a bonus effect must depend on the number of repititions.

Somehow I thought you'd be more open to others' expertise in this area Rah.

rah · January 31, 2003, 11:25

I'll wait for the result before I crunch the statistics.
The 18 was an educated guess. Previous experience is the entire reason why I'm doing the test. People have said there isn't one, and experience tells me there is. There's a big difference between arbitrary and educated guess.

I've worked in Market research for years. I have programmed and analyzed a considerable statistical runs. Your comments lead me to believe that you don't have similar experience in this area. (based on your comments about a control not being necessary)
I'll be more that willing to listen to a statistical expert when one shows up.

Ming · January 31, 2003, 11:29

Considering that rah does research for a living... and that he's a professional at it.. I think he's the one with some expertise

rah · January 31, 2003, 11:31

And, if it's 17, i'll probably test more cases. If it still remains 17 after that, it will still prove my theory that there is a suprise bonus. It just not be the same bonus that I guessed. And no, I didn't run the numbers to see how many observations were necessary to get it into the 95% confidence level. But I did ask a more qualified statistician here at work to run it and let me know what it would be. When he gets back to me, I'll post it. But I'm going to start testing prior to his answer because my curiosity is getting the better of me.

RAH

DrSpike · January 31, 2003, 11:42

I build and estimate cutting edge econometric models for a living. I have a ph.d in econometrics (statistical economics). However, screw that, your comment about my expertise is not what bothers me. What bothers me is your reaction to my suggestions........it really doesn't seem in keeping with the picture I have built up of you.

Still, because I respect you, one last try:

The problem with the control you propose here is that it itself is noisy from your proposed sample.........a far better control would just be to use the formula we know holds (and the formula for bog standard comabt is the same for MP, we can be virtually certain of this - it is just the bonus modifiers that may be different against a human rather than an AI).

As to the arbitrary 18, the point is not that your educated guess is a bad one, but that you need a concrete decision rule. Your statement:

"I'll bet, on the first surprise attack, that the attacking unit will win 18 or more of the 25 encounters, and 22 or 23 won't surprise me. I'll be real interested to see if the results of attack 3 match attack 1 or if they're consistent with 2 and 4"

Now 'consistent' is the key bit........how much of a divergence are you going to allow before it affects your conclusions? Do you know the probability of the warrior winning more than 18 even if a bonus does not exist. No, you don't. Do you know the probability of observing 17 when a bonus does exist? No, you don't. Both of these are ways your decision rule fails.......now potential failure in hypothesis testing is endemic, but you have to know the associated probabilities or results are just glorified junk.

It's funny, I always stay out of these stats debates, because it is far easier to let the person do it by brute force, with a badly constructed experiment but lots of repititions than try and explain a more cogent testing structure. But you said you were going to talk to stats people about confidence, so I chirped up, and thought you would be receptive.

Still, happy testing........hope it goes well.

rah · January 31, 2003, 12:05

I guess I overreacted to your belief that human vs humanl combat need not be tested and that previous ai vs human results would be valid despite never having been tested. Hence my desire to have a control. (but I do agree that they will probably be the same, but paranoia when testing is never a bad thing)

And your comment that 18 was arbitrary. It was not.
An educated guess is quite different than arbitrary.

And to your nitpicking on 17 or 19. WE DON"T KNOW WHAT IT"S GOING TO BE, that's why were testing.

The initial hyposthesis is that there is a difference.

And I'll do the statistical run, AFTER the test. Having data has been known to make it easier.

Since the testing will consume some time, I wanted a statistician to give me a ballpark of number of observations needed so I wouldn't not get enough or waste my time doing hundrends.

If I overreacted, I apologize, but I recommend that in the future, if you don't want people to get defensive, don't start by attacking there initial position distorting what they said. And your use of abritrary and misinturperting my hypothesis (there is a difference, the amount of difference was just a guess) was exactly that.

RAH

DrSpike · January 31, 2003, 12:24

Quote:

Originally posted by rah

And your comment that 18 was arbitrary. It was not.
An educated guess is quite different than arbitrary.
RAH

Again, sure you used knowledge to select 18, but you don't know any of the properties of that decision rule or you would have posted them to shut me up by now. Maybe arbitrary is a strong word, but it's not far off.

I doubt anyone really cares, but this is how I would do it. Far from being clueless in this area as you suggested I do this (well this is analogous to the most basic econometrics I would do) every day, and lecture other people at undergraduate and postgraduate level on how to do it.

I would use 50 repitions (or 25*2 as you suggest is the same, as long as making and breaking peace gives you the same state as the initial one before the civs have met). I would derive the proportion of expected wins for the attacking warrior without a bonus. Your null hypothesis is then.

H0: p hat = derived number

with

H1: p hat is not derived number.

Then you just derive the variance of phat in your sample of 50......it is just {p(1-p)}/n, where n is the number of repititions Under the null the distribution of phat is normal (from various statistical theorems) with a mean of 'derived number' and a variance as above. You then choose a level of significance, which also fixes the probability of you rejecting the null when it is true.

You then compare the standard normalised value for the proportion of wins you observed in your testing with the associated critical value from the normal distribution, which tests the null at the 5% level, the level you said you wanted.

Easy. Concrete. And as the icing you can derive the power of the test (related to the probability of accepting the null when it is false) under the assumption of a bonus of 50%. YOU CAN EVEN DERIVE THE MOST LIKELY SPECIFICATION FOR THE BONUS GIVEN THE DATA.

Please feel free to show this post to your stats guys if you doubt my expertise as you said above. Any competent statistician would carry out the test the same way.

rah · January 31, 2003, 12:54

Quote:

Originally posted by DrSpike
I would use 50 repitions (or 25*2 as you suggest is the same, as long as making and breaking peace gives you the same state as the initial one before the civs have met). I would derive the proportion of expected wins for the attacking warrior without a bonus. Your null hypothesis is then.

I'm doing 25*2 to test if the different states make any difference. (can you get a second suprise bonus) But I'm glad you mentioned that because I was going to just line up 4 warriors against 4 warriors, now after reading this, yes the first combat must be run right after the initial contact notice, So I'll start with the four warriors 1 square away as the intial base point (i probably would have remembered that when I set it up but I might have wasted time

).

As to your statitiscal analysis. Yeah I could have copied crap from my SAS book, but I have statisticians working for me and their expertise there is greater than mine. I have never once claimed expertise there, just understanding. My expertise is the test methodology and data collection/manipulation. Analysis comes after the data is collected. And if you want, I'll give you the raw data and you can save my staff some effort

But, if the result is > 18, I'll claim there is a bonus prior to the stat run, but I'll wait for the analysis to tell me how much and how confident it is. My initial intention was too disprove all of those that claim that the sneak attack bonus is a myth. How much has always been secondary.

And the difference in win percentage is all MP players are really interested in so they can develop some rule of thumb guidelines for on the fly attacks. When the clock is ticking all you really have time for is hmmm 4attack vs 2def, good, or 4att vs 4 def, I'd better have more units attacking than he has defending. or 4att vs 6def, I'd better have more than twice the units attacking.

RAH

-Jrabbit · January 31, 2003, 13:37

This looks interesting. Thanks for starting this thread, AH.

* -Jrabbit posting his +1 just so it'll show up on "my threads"

*

DrSpike · January 31, 2003, 13:39

Even if you only want to establish the existence or not of the bonus my method is far superior. Ask your staff. But never mind......we have exhausted potential discussion I think.

I don't think the textbook gibe is fair........formulating the test procedure above is not trivial........so I posted it......not least to show you how it would be done by someone who actually has experience in this area, and that I wasn't criticising your formulation without proposing an alternative. (because it p1sses me off when people do that

)

I admit your claiming experience in the area whilst disparaging mine annoyed me and made my responses prickly......I hope it is all in the past now.

Happy testing.

rah · January 31, 2003, 13:56

yep, I'm not a fan when people stick equations under my face and say, see I'm right. I have books full of equations and I'm never right

So sorry, if I jumped on you there.

But since you do have an interest and the skill, I'll post the raw data results here, and you can take it from there. (while I'm boasting that there is a surprise bonus while I'm waiting for the final analysis, since that's what market research firms are good at.)

RAH

DrSpike · January 31, 2003, 14:31

Quote:

Originally posted by rah

But since you do have an interest and the skill, I'll post the raw data results here, and you can take it from there.
RAH

Agreed.......though for the basic test all I need do is supply the decision rule, since we don't like arbitrary ones

I just did some back of the envelope calculations (I used the shortcut so they are at present a touch off, but not much). They confirm the need to analyse tests properly. It turns out that with exactly 50 repititions that 32 or more wins is sufficient to reject the null at the 5% level. This is the same as saying you would only observe more than 32 wins with no bonus 5% of the time.

18 (from 25) is *much* too strict a decision rule, since 36 or more wins from 50 repititions is only observed a fraction of 1% of the time. Consequently the probability of accepting the null when it is false (precisely what you wish to avoid) is needlessly large. I should have done this earlier......it makes my points far better than anything else possibly could. Hope you are convinced now.

rah · January 31, 2003, 15:15

Yes the 18 was more based on playing experience and guessing how much 50% bonus would impact it.

Your example that 32 is necessary only to prove that there is some type of bonus, not taking into account how much. The bigger the bonus, the more wins will be necessary to prove it. (and this is assuming a 50-50 norm human vs human, which has yet to be verified even though i'm sure it will)

My stat guy asked me how much of a bonus I was trying to prove.

Bloody Monk · January 31, 2003, 17:30

All this technical stuff makes my hair hurt

But, I'm sure all of us await , with great fascination, your conclusions.

On another note..."Let the word go forth from this time and place, to all the known worlds, that rah and DrSpike are two very smart guys."

Monk

Scouse Gits · January 31, 2003, 18:29

Glad the flames have died down guys...

I guess it just shows the difference of approach between the academic and (dare I say it) the real worlds. We both need each other - but sometimes we resent it!

SG[1] - an academic

DrSpike · January 31, 2003, 21:08

Quote:

Originally posted by rah

My stat guy asked me how much of a bonus I was trying to prove.

Having returned from the local weekly gathering of the......uh....strategic gaming society I hope I can retain my usual level of clarity.

Yeah, if we knew the bonus exactly we could formulate the test the other way around, with the null as "there is a bonus". Since we don't the way around I suggested is natural.......alternative hypotheses are rarely exactly specified since it would lead to the test having very bad properties if the specification is even slightly wrong. Plus it is more natural to ask the data to prove the existence of the bonus rather than disprove it.

Hence what your guy was after was a measure of the power of the test (related to the probability of accepting the null when it is false).......which does depend on the specification for the bonus. Incidentally the power of my formulation has almost twice the power of your original formulation.......I checked that earlier in case I needed it to settle any arguments.

Any results yet?

DrSpike · January 31, 2003, 21:17

Quote:

Originally posted by Scouse Gits
Glad the flames have died down guys...

I guess it just shows the difference of approach between the academic and (dare I say it) the real worlds. We both need each other - but sometimes we resent it!

SG[1] - an academic

There weren't any flames......I can't usually be assed to insult people........I just bridled a little when Rah assumed I was bullsh1tting and didn't know what I was talking about......when in fact I do.

I admit when Ming made his post about Rah being a professional and hence knowing better that my keyboard came close to extinction. All will be pleased to know it is safe and sound.

If I am perfectly honest (and I am

) Rah could carry out this test by brute force (the way most stats that 'real world' people do is) with little loss of precision.......it would just take longer. And if the result happens to be semi-conclusive that is when an academic with a more rigorous procedure will win every time.

Six Thousand Year Old Man · January 31, 2003, 21:54

I hate to say this, but - why use Warriors?

Warriors are A:1/D:1 - if there is a bonus, it's going to be pretty hard to see it, if rounding is used by the Civ program. For example, a 20% bonus is significant if you're attacking a Riflemen with a Dragoons - but the .2 bonus may be rounded to nothingness when using Warriors to test for it.

Sorry if that point has already been considered

I hate to get involved in a high-level statistical discussion, anyway - I just use stats to confuse my clients

rah · February 1, 2003, 01:48

warriors, for two reasons.
One, it takes faster to set up on a random start, I'll just do it on a small world and it won't take war and i more than 10 minutes to line em up, and i won't have to use a scenario that could bias the results.
Two, my initial intent was just to prove there was a bonus. so any equal units would do. IF there is one, we'll work on the amount.

With Spike, we're fine, we were just feeling out each others backgrounds, and I took early offense to a few of his choice of words, and comments about not needing a control group. I overreacted and fueled the flames. My initial thought was a simple test, but it's kinda gotten bigger.

I believe we're fine and will share on the discovery. I'm glad there's a statistician available since I really don't like using my staff for my personal concepts(even though I know they'd be happy to do it) This way we can keep the entire thing in house.

RAH

And everyone showed up early to play so I didn't get a chance to test it, War and I will do it tomorrow before we play.

rjmatsleepers · February 1, 2003, 05:42

Quote:

Originally posted by DrSpike

Sorry but it isn't.

Having demonstrated my statistical naivity once

, I probably shouldn't post again on this subject but ...

Surely we only need to test the hypothesis that there is a "sneak attack bonus" that is large enough to increase the attack strength by 1 (against the null hypothesis that the bonus is not big enough to increase the attack strength). If you're using warriors to run the test, that would mean changing a 50% win chance into something pretty close to 100%. I think either Dr Spike or Rah's stat guys could show that the sample size needed for that test is pretty damn small.

(RJM retires behind SDI shield in case of nuclear attack from any direction)

RJM at Sleepers

DrSpike · February 1, 2003, 07:01

DrSpike uses nuclear weapons!

SDI defence blocks attack!

Bugger.

Hehe, um, as to your query I don't understand why we should test an increase in attack strength of one.

The way hypothesis testing is done is to have 2 hypotheses you wish to test, and derive the distribution for your statistic (in this case its the proportion of wins). Then you can calculate whether or not the value from your sample (that Rah is going to provide) could reasonably have come from the postulated distribution. Your hypotheses are a little rough.....they look tricky to test to me.

Of course it is natural to want (as Rah did) the test to be stricter the bigger you believe the bonus to be....but unfortunately this is the wrong reaction........all that a bigger bonus does it make the job easier by increasing the power of the test.....since the probability of rejecting the null when it is true is fixed by the researcher anyway. This also shows why large sample sizes are not necessary for this test if the test procedure is well formulated.......though they are if there is no formal test procedure.

As to sample size all we really need is to be sure that the statistic has a normal distribution.......and for that we need in excess of 30. I am perpetually paranoid so I said 50.

SCG · February 1, 2003, 10:47

Quote:

Originally posted by Scouse Gits
Glad the flames have died down guys...

I guess it just shows the difference of approach between the academic and (dare I say it) the real worlds. We both need each other - but sometimes we resent it!

SG[1] - an academic

You know - Maquis de Sodaq never did run for trade advisor

*ponders whether it would still be worth while trying to set up a ticket stand for the 'heated debate'*

this joke brought to you by the new Democracy Game Trade department

rah · February 1, 2003, 18:01

Quote:

Originally posted by SCG
this joke brought to you by the new Democracy Game Trade department

Don't you guys get enough posts in that forum?