Under-Estimating Difficulty
It is well understood in the industry that a game developer is completely incapable of judging the difficulty of any game they make. Game teams are notoriously poor at understanding and predicting how new players will respond to their first experience of a game. No matter how smart a game maker is, it is simply impossible for them to blank out the intimate knowledge they already have of their game and see it through the eyes of a newbie.
My predecessor at EA Montreal knew this very well, and had the foresight to hire a trained expert in user-experience evaluation. This person worked separately to the game teams to avoid any subconscious bias, and put early prototypes into the hands of the general public. Using video footage, questionnaires and recorded conversation, it was possible to draw immediate and genuine reactions to each game during its development. And it was a constant reminder of how game teams, regardless of talent and experience, would invariably underestimate the difficulties people would have in playing their games.
Yet knowing this, I still failed terribly in getting the Smart Numbers difficulty curve right, out of the gate. Let me share with you what I’ve just seen…
The first week of data has been SHOCKING!! Even the very 1st level !! Level 1 of any game should be super-simple and have 100% success rate. Level 1 of Smart Numbers recorded a tawdry 66% win-rate. On average one third of players were finding failure as their first experience of the game. Not good. Sure, most played a second time and won, but I was seeing a 10% fall-off of players who never even tried level 2!
The next shock was level 6. I saw nothing unusual about level 6 - my own playing stats showed this to be mildly more difficult than levels 1-thru-5, but still ‘falling-off-a-log’ easy. Not so the 100+ players who tried it last week! Only 25% of their attempts to win level 6 resulted in a win. People were likely to fail, 3 times out of 4. Bear in mind levels 1 thru 15 are in the ‘BASICS’ section of the game, and the player can see 150 levels lying ahead of them. If level 6 was already feeling too tough, any thought of going on to play the rest of the game was likely way too daunting. And sure enough, the drop-off rate of players at level 6 was a whopping 30%. (Interestingly my D7 retention for the same period was comfortably north of 35%, so people were sticking at it despite getting stuck on early levels. But there is a high risk of player frustration bringing that number down. Not a risk I want to be taking!)
There were a few unexpected difficulty ‘bumps’ after that, notably levels 13, 18 and 21. All had win-rates below 50%. I really still want them at 75% at this point.
But then at level 24 the game just BROKE! A disastrous 5% win rate for level 24, with levels 26 to 30 all averaging at the same low low level. Players were getting stuck on these levels big time. It prompted me to look at the REPLAY and QUIT data for those levels. They were through the roof!! For example, level 26 had been played 321 times by 43 players - an average of 8 replays per player. And more likely it had been abandoned after a couple of tries by some players, meaning the rest played many more than 8 replays. Only 25 of the 321 attempts resulted in a win! This was what I wanted to see around level 40 - not happening in every level from 24 onwards…
So my lesson is learned - I’ve already put out an update with better balanced difficulty across those 30 levels, and we shall see the comparative results in a weeks time. I also have some ability to tweak difficulty per level from the cloud - hopefully that will allow for fine-tuning over the coming weeks / months.
One last thing that leapt out of the data - For a game of strategy and mathematical thinking like this, the difficulty of ‘timed levels’ is even harder to balance. There is such a HUGE variation in thinking-speed between different players, that to set an appropriate ‘challenging-but-doable’ time that works for everyone looks to be near-impossible. For now, I just keep increasing the time for those levels, to ensure people don’t get blocked. But I think I’ll write a subsequent blog about the perils of having time-based challenges in puzzle games, and share with you the solution I finally hit on for Smart Numbers.
