Dataset
The following information is based on examining many thousands of games by expert NetHack players. "Expert" players, here, is generally defined as:
These are perhaps fairly arbitrary criteria, but I think it is safe to say that any player meeting any of those criteria could be safely called an "expert" player. They attest to wide experience with and deep knowledge of the game either directly (many wins, long streaks) or indirectly (unusually fast or efficient play, difficult conduct play).
The games were analyzed first by downloading them in ttyrec format. Most of the ttyrecs were downloaded from NAO, but many came from other public servers, including nethack.fi, acehack.de, Nostalgia House, and nethack.eu. Every .ttyrec was run through the ttyconcat tool, which can log many interesting facts about NetHack games.
Games of 10 turns or less were not considered in the dataset. There may be a handful of legitimate games that simply ended that quickly (an early trap, say) but the vast majority of these were start-scummed games. Escaped and quit games were also excluded -- many of these were failed conducts.
The dataset contains 35,131 games, 5,574 of them ascensions. All of the games are NetHack 3.4.3, with interface or bug-fix patches only (no gameplay changes). The ascension rate for the games included in the data, overall, is 15.9%. The ascension rate for games that would continue a streak (i.e., the player's previous game was also an ascension) is 47.3% (2027/4288). The ascension rate for games in which the player's previous two games were ascensions is 64.6% (1071/1659), and the ascension rate for games in which the previous three were ascensions is 70.7% (650/919).
In most of the comparisons below, I will calculate overall ascension rate as well as ascension rate after one, two, or three previous ascensions. I'll do this because there are some players in this dataset that are not necessarily trying to ascend every game, but that are trying to achieve a difficult conduct, execute a speed run, perhaps just goof off, etc. (NETHACK IS SERIOUS BUSINESS.)
The dataset used is hardly exhaustive -- not every player meeting those conditions is represented. For example, most of the NAO ttyrecs are from March 2008 or later; a hard drive failure caused the loss of most earlier NAO ttyrecs. There are exceptions: for example, thanks to his 23-game ascension streak and generally excellent play, mrivan's NAO games were widely circulated as examples of expert play and were available to include in this data.
On this page, I will use this data to attempt to answer questions (or at least make some piquant observations) about some common NetHack strategy questions.
1. What's the better Sokoban prize?
Before looking at the data, I would have said the amulet of reflection was certainly the better Sokoban prize (in terms of improving a game's ascension chances), and by quite a bit; after all, a bag of holding is very useful, but it can't stop a death ray.
Let's take a look at the data. 7,208 of the games in the dataset both reached the top level of Sokoban and picked up the prize. In 3,780 of these games the prize was a bag of holding, in 3,428 of these games the prize was an amulet of reflection (apparently it is more common, in this dataset, for the player to have reflection by the end of Sokoban than it is for the player to have a bag of holding -- this makes sense as an early wish is more likely to go for SDSM or an amulet of reflection than a bag of holding.)
| Prize | Ascension Rate (#games) | prev. game ascended (#games) | prev. 2 games ascended (#games) | prev. 3 games ascended (#games) |
|---|---|---|---|---|
| bag of holding | ||||
| amulet of reflection | ||||
| (either) |
We can't say that the overall ascension rate differs based on the Sokoban prize with any statistical significance. The "previous game ascended" rates may be different, but with a rather low confidence level (about 84%). Although the differences in ascension rate between prizes for the 3-game streaking players are the largest, they are not different from the overall mean with statistical significance (thanks to the much smaller sample size). The amulet of reflection is probably a little better for survivability, but for most expert players it appears that the Sokoban prize makes little or no difference.
2. How much does an early wish help an expert player?
For this study, we'll first split the data into three categories: games with no wishes (this will include a lot of the early deaths), games with at least one wish before obtaining the Castle wand, and games where the first wish came at (or after) the Castle. We expect that games with wishes will have very high ascension rates (since most early game deaths are screened out) but does an early wish make a significant difference?
| Category | Ascension Rate (#games) | prev. game ascended (#games) | prev. 2 games ascended (#games) | prev. 3 games ascended (#games) |
|---|---|---|---|---|
| No wishes | ||||
| Early wish | ||||
| No early wish |
Wait a minute! Why does "no early wish" have a higher ascension rate than "early wish"? Simple -- there are the games in which a player gets an early wish, but dies before making it to the Castle. So how about "early wish and makes it to the Castle" against "no early wish and makes it to the Castle"? We'll compare those numbers next.
One other interesting thing to note in the above table is that some serial ascenders are quite happy to ascend a wishless conduct game in the middle of a multiple-game streak. Nine players have even done it more than once.
| Category | Ascension Rate (#games) | prev. game ascended (#games) | prev. 2 games ascended (#games) | prev. 3 games ascended (#games) |
|---|---|---|---|---|
| Early wish(es) + Castle wand | ||||
| just Castle wand |
Here are a few things to take away from this data:
3. The protection racket: when (if ever) is it optimal?
The protection racket (an opening in which the player heads to Minetown to buy protection while at low experience level) always invites some controversy; there are some expert players who commonly play this opening, while others opine that it is far riskier and results in far more death than a "standard" opening, in which the player levels up normally.
In a pacifist game, the protection racket is probably the correct opening, and it is played most of the time; you are relying on your pets to kill anyway and the only meaningful early experience source is the Oracle, which may be nearly as risky to reach as Minetown. But what about in "normal", non-conduct play? Is there a situation in which the protection racket is the best opening?
For the purposes of this study, I considered any game in which protection was bought early in the game at XP levels 1 or 2 as a protection run. (Occasionally, much later, players level-drain themselves to buy cheap protection; by screening the protection turn count these games are eliminated.)
There are 1,348 successful protection rackets in the dataset; of these, 1,190 (88%) bought protection at XP level 1 and 158 (12%) bought protection at XP level 2. Every role was represented; in order of frequency they are Hea (636), Arc (145), Tou (140), Wiz (74), Cav (50), Pri (49), Kni (48), Ran (42), Mon (39), Rog (38), Val (37), Sam (29), Bar (21).
(The idea of a Barbarian running the protection racket amused me so much I examined each of those 21 games. Here they are for you to examine, as well.)
Of the 1,348 successful rackets, 184 games ended as pacifist (140 in death and 44 in ascension); these games are excluded, as we are trying to evaluate the protection racket in non-pacifist conduct play. (We've already eliminated the "escaped" and "quit" outcomes, which gets rid of most of the failed pacifist runs ending in an accidental monster kill rather than death.)
Of these 1,164 remaining games, 581 ascended, for a rate of 49.91%. Now, what we're going to do is compare that to typical games, and show ascension rates by the AC level achieved by the character:
| AC level reached | Ascension rate (#games) | Ascension rate for non-PR Healers (#games) |
|---|---|---|
| 2 or better | 34.89% (15,943) | 30.21% (427) |
| 1 or better | 38.10% (14,600) | 34.22% (377) |
| 0 or better | 41.52% (13,396) | 38.51% (335) |
| -1 or better | 48.13% (11,556) | 45.58% (283) |
| -2 or better | 52.06% (10,681) | 49.81% (259) |
| -3 or better | 56.22% (9,889) | 56.58% (228) |
| -4 or better | 60.51% (9,181) | 61.43% (210) |
| -5 or better | 64.23% (8,644) | 65.82% (196) |
With each additional AC point, the early-game deaths melt away. Notice the big jumps as you go into negative AC (as you start reducing damage from monster attacks). And notice how quickly the ascension percentage goes up -- the difference between "AC 0" and "AC -1" turns out to be, in absolute percentage-point terms, as large as the difference between "player reaches Minetown" and "player reaches Minetown and finds a magic lamp there". Every AC point is precious!
(And now, incidentally, you know why I test practically every piece of armor in the Mines.)
Now, the improved ascension chances with improved AC is of course not entirely due to the AC itself; the lower your AC, the further along in the game you probably are generally. For example, as you get to AC -10 or below, the odds that the player has magic resistance and reflection greatly go up (because of all of the characters wearing DSM). But for relatively high AC numbers, we're early in the game and the primary difference between most characters is AC, so that is the big survivability factor at least for this chart. (I could produce a similar chart mapping "deepest dlvl reached" with "ascension rate", and there would be a clear positive correlation there, but that does not mean we can conclude that the "dig for victory" is the best opening strategy. Speaking of which... that's a good topic for another section!)
Here's one possible confounding factor here: role. Some roles (Val Sam Kni for example) start with pretty good armor (although the Valk only gets a +3 small shield, that's an excellent piece in itself) and are generally considered "easy" or "easier" roles. Since most protection racket runs are done with Healers, we can compare apples to apples and look at AC-to-ascension rate for Healers specifically in the chart as well. We exclude the successful protection racket runners in the above chart -- note that this means those ascension rates include any failed protection racket run Healers that reach the specified AC class (though many failed PR Healers will not get to AC 2).
Let's be clear: the 49.91% quoted above is the ascension rate of successful protection racket runs, i.e. the protection was actually bought. It leaves out pacifist conduct attempts, and it does not take into account any failed protection rackets, i.e. the character is killed on the way to Minetown. Players in this dataset who successfully bought protection in Minetown ascended 50% of the time, which is somewhere between "all games that reached AC -1" and "all games that reached AC -2" overall. For non-PR, non-pacifist Healers, this rate is about the same as "games that reach AC -2". But there's the rub: any successful protection racket should always result in AC better than -2, and often substantially better (between the guaranteed points of protection, non-guaranteed protection, your starting armor, and whatever you found along the way, you could be at AC -10 or better after a successful protection racket.)
In other words, a protection racket run, even if successful and in non-conduct play, results in substantially lower ascension ratios than players who achieve the same AC levels normally. It even underperforms Healers specifically who don't purchase protection at XP level 1-2, including failed protection runners. That is the opposite of good.
Why would this be so? One possibility is that, even after you succeed in buying protection, you're too deep for your level. You normally have too few hit points. It might be interesting to see how a protection racket run compares to a standard opening for combat classes, with beefier hit points and greater survivability in general, but as you can see in the role break-down above, the sample size for such games is quite small.
It's also possible that part of the sub-standard performance of protection racket games is on the players; it may be that "less expert" players play the protection racket more often, while "more expert" players try to improve their AC the usual way. Most of the expert players in the dataset have made a successful protection racket run at least once, however.
The early protection run is an extremely popular opening with Healers, but looking at this data I find it hard to justify it as a general opening -- in conduct play perhaps, but not in a game without conduct restrictions. There might exist rare lucky circumstances (an early mithril-coat? speed boots perhaps?) that make an XP 1 protection run "worth it", but in the general case it is not the percentage play.
4. What's the better early wish: magic resistance or reflection?
Here's a classic: an early player is offered a single wish by a grateful djinni or water demon. He has neither magic resistance nor reflection. Should he go for magic resistance (either in artifact or armor form) or should he go for reflection?
For this study, I examined games in which a player was offered a single wish while lacking both magic resistance and reflection. There were 3,084 such instances in the dataset. In 1,308 cases (42%), the player wished for something granting magic resistance; in 972 cases (32%), the player wished for something granting reflection. 804 times the player wished for neither, though many of those wishes (about 200) were for magic markers or scrolls of genocide (through which an appropriate DSM could be made.)
Here are the ascension rates after the early wish:
| Wish | Ascension rate (#games) | prev. game ascended (#games) | prev. 2 games ascended (#games) | prev. 3 games ascended (#games) | |
|---|---|---|---|---|---|
| early MR | 55.43% (1,308) | 80.71% (368) | 87.35% (166) | 86.87% (99) | |
| MR from GDSM/cloak | 56.32% (673) | 82.72% (191) | 85.71% (84) | 88.24% (51) | |
| MR from artifact | 54.49% (635) | 78.53% (177) | 89.02% (82) | 85.42% (48) | |
| early reflection | 61.73% (972) | 79.85% (258) | 87.69% (130) | 91.57% (83) |
Nobody wishes for the Longbow of Diana, so we don't break out "reflection from an artifact". The most common MR artifacts wished are the Eye of the Aethiopica (281 times) and the Sceptre of Might (144 times).
Although survival rates are (slightly) lower for players obtaining magic resistance from an artifact, compared to players who wish for GDSM or a cloak, this is not statistically significant. Some people might expect the "MRtifact" to yield higher ascension rates, under the theory that these artifacts provide useful benefits besides magic resistance (hungerless conflict, charging, power regeneration, astral vision, etc.) While that is true, the AC from dragon scale mail is an enormous benefit in itself, and we've already seen what improving AC does for a character's ascension rate. It's more or less a wash; if you can't reliably wish for an artifact, don't ever feel bad about wishing for (or better yet, making) DSM.
It is sometimes said that the ability to wish for Quest artifacts is "overpowered" or somehow "game-breaking"; this data speaks to the contrary. Indeed, if you think about it, the ability to wish for gray dragon scale mail (best base AC armor, low weight, gives magic resistance and does not hinder spellcasting) is surely comparable (in terms of in-game benefit) to wishing for the Eye of the Aethiopica or the Sceptre of Might. If Quest artifacts were genuinely game-breaking, expert play would reveal a higher ascension rate when an early wish yielded them. It does not.
Something that might surprise other people is that, in this dataset, reflection trumps magic resistance in terms of winning percentage. (The difference in overall ascension percentage is statistically significant at the 99% confidence level.) This is clearly not true in the late game -- in the late game, instadeaths due to lack of magic resistance are far more common than instadeaths due to lack of reflection, and while there are (relatively) many reflection-less ascensions, there are few magic resistance-less ascensions. But the late game is not important. Almost everybody survives the late game, and if you're running around without either MR or reflection in the late game, it's your fault. It's the early and midgame that matters, and there, magic resistance, while helpful, isn't necessary. There are far more early deaths due to lack of reflection than to lack of MR (MR does nothing against wands of fire, cold, lightning, or sleep, after all, or most monster breath attacks, or floating eye accidents, and even with the appropriate resistances, exploding equipment can still kill a player).
But if reflection is so good, you ask (heck, I ask), why is there no or almost no difference in winning rates between Sokoban prizes? I have to assume that intrinsic resistances play a role here -- by the time a character completes Sokoban he often has multiple resistances, and hence the danger from early attack wands or breathing monsters is substantially lower. It may also be that a bag of holding is almost as good (in terms of improving survival) as an amulet of reflection; you can carry more helpful items, wear heavier armor, etc. Most deaths in expert play occur before Sokoban is complete; it stands to reason, then, that the greatest threats and dangers come before then.
It is worth pointing out that, although the difference between MR and reflection is significant overall, it is not for 1-game and 2-game streakers, and only marginally significant (86% confidence level) for 3-streakers. Since streaking players need to be careful of early wands, this is perhaps indicative that a more careful play style makes early magic resistance and early reflection closer in value.
...more to come, this page is a WIP