What is playtesting and why do devs screw it up?

User research firm Player Research tells VG247's Dave Cook why advances in playtesting tech are winning the war against poorly-designed games.

Not to be confused with publisher surveys or focus-group polling, user-research is a growing trend in the games industry. It's a way for developers to craft smarter, more enjoyable titles based on raw data at an early stage, rather than relying on pure guesswork and assumptions.

Chances are you've played a game and encountered a particularly frustrating section or flawed mechanic, and you've started to ponder just how on Earth these problems weren't spotted by the developer before release.

It's a valid question, and it's something studios have to think about to a greater degree given the rising cost of development today. Rectifying a busted game component long into its production cycle can prove expensive, while such issues might put people off spending money in a freemium title, effectively cutting off a studio's revenue stream.

I decided to speak with Graham McAllister, founder of leading user research studio Player Research to get a handle on how advancements in usability tech and methods can help save a game from such pitfalls, and the insight he gave me revealed an unseen side of the development process I hadn't considered until now.

"Play testing is getting players in front of your game and this is not about asking them how they feel, it's about watching them actually play a game," McAllister explained. "It's about observation of behaviour, understanding their emotions - because we use sensor technology as well - and overall it's building a picture of the overall player experience. Did they like what they played? And if they didn't like what they played, why?"

Valve is said to be using sweat-tracking and VR, both of which are used in modern testing.

This may sound like a common-sense practice that every developer should be employing as standard, but McAllister told me he's constantly amazed at how few companies actually use this method of testing. Getting in as early as a game's concept or pre-alpha stage is the key to rooting out flawed design before it becomes too disruptive or costly to rectify.

The playtesting process uses methods such as sweat, eye and Galvanic Skin Response-tracking, emotion monitoring and simple direct observation of behaviour to identify a range of problems that include poor sign-posting, miscommunicated mechanics, clumsy controls, unfair checkpoints and other blockages that might stunt a person's enjoyment.

"All of our staff here come from either psychology, interaction design or computer science. So we’re well grounded in analysing behaviour and emotion," McAllister explained, stressing the need for a qualified approach to playtesting. "We're not to be confused with QA, which is bug-hunting. I know nothing about that, and we're not to be confused with market research, which is how to sell stuff to people."

Player Research and other playtesting labs exist is to show studios where an experience is stumbling. It's clear that studios aren't missing these things because they're stupid, because creating games requires great intelligence. Instead, the problem typically stems from a developer's attachment to a project.

"The purpose of playtesting is to help the designer make the game that's inside their head," McAllister explained. "So they have this great vision, and one of the problems is they're too close to it, because after month or years on a game you start to not see some of the core issues because you've learned them.

Plants Vs Zombies is Player Research's opinion of a perfectly usable experience.

"You've learned the rules of the game, you've learned the UI, you've learned the complex interactions and the narrative. When they played the game they assume all of that has been understood and digested. So it’s interesting when we put people new to a game in front of it and they say, ‘Why would I do this, and how would I do this?’

McAllister believes that any game should be able to convey its core mechanics, UI structure and controls to players in a meaningful way which helps them learn. If it fails the chances of people sticking around becomes reduced, and when it comes to free games, frustrated players might be reluctant to spend a penny if they don't understand how to use a game properly.

The first few minutes of a game can literally make or break a player's engagement, and it's within this window that developers should be conveying the rules and base mechanics of the experience clearly.

"If we're analysing the early experience, there's so many concepts that have to be communicated in that space of time. It's quite overwhelming. The trick is how do you do it in the best possible way? Our go-to game for the best possible game design is Plants Vs Zombies.

"OK, it's not the most complicated of games but everything it does is pretty much perfect. It's genius. In terms of the casual game space people say for example, 'our game is very accessible', but that's rarely the case. It's usually quite demanding, and we've got a simple model for comparing things like how much of your visual attention does it take, how much cognitive load, or how many interactions are you doing?

Dark Souls knew its audience well and its tough design was well-considered

"In all the most successful games they're low for each of these scores, each of these metrics. In other words say, Candy Crush Saga; it doesn't actually make you think that much. You always have to scan the page with your eyes a little bit which is medium to high as you're always looking for something. The interaction is quite low. It's just the odd tap here and there."

It's telling that the most successful games from a usability standpoint always demand less from players. It goes a long way to explaining why casual mobile games appeal to those on the go, or to players without a long experience of gaming. This accessibility is also why titles like Angry Birds and Doodle Jump sell so well.

But what of the 'hardcore' pack for lack of a better word? McAllister feels that complexity in games should come from depth and strategy itself, not from trying to master an over-complicated user interface or annoying mechanics. Making a game more usable does not make a game easier by any means, but it does make for a more accessible, intuitive and functional experience.

Giving me a practical example McAllister said, "If you're in a strategy game and you want to change character on the fly or something like that, and if it's a bit too cumbersome then frustration quickly settles in. That's frustration from the wrong part of the game.

"Frustration should come from, 'Oh man I got beaten but that was my fault. I should come up with a better strategy, I want to play that again.' So if people get frustrated with games we sometimes ask them, 'Do you feel that frustration was your fault, or the games fault?'"

I flagged up Dark Souls to McAllister for the way it is both punishing and fair in equal measure, and that nine times out of ten frustration comes from something the player did, or whenever they are careless and rush into battles without strategy.

McAllister feels Modern Warfare 2's snow level is a masterclass in pacing and sign-posting

"That game knew its core audience pretty well," McAllister added. "They were going for that sort of player, the one who wants the challenge, might throw their controller down in frustration, but ultimately when they beat it, when they get that strategy right they're going to feel amazing. So that was fine, that was probably the designer's intent."

An understanding of the common theories and principles of psychology helps Player Research and developers seeking guidance truly understand how people are responding to a game, regardless of genre, target audience or difficulty. For example, if a player forgets a key plot point or mechanic could it be communicated better so it sticks in the mind?

Game pacing is also important to a person's enjoyment and Player Research often sends what McAllister calls 'Biometric Storyboards' to studios. These charts give companies a clear image of a player's psychophysiology, and how they experienced a game on a second-by-second basis.

The data shows how a player responded both mentally and physically to boss battles, scene changes, levels and more. Perhaps one of the best case studies used by Player Research is to compare the psychophysical reaction to Haze and Call of Duty: Modern Warfare 2. I'll leave you to discern which game fared better in testing.

The Modern Warfare 2 level in which Soap MacTavish and the player scale a Russian ice face, sneak through a heavily-guarded compound using stealth, and then make their getaway on snowmobiles is regarded by McAllister to be a masterclass of design. The speed and intensity of the mission rises and falls so frequently that players are constantly hooked in to the experience.

Angry Birds is another intuitive, expertly-designed title from a playtesting perspective

By this point in our conversation the benefits of advanced playtesting were clear in my mind, so I had to ask McAllister why he felt publishers are still falling back on assumptions and old-form methods of data capture such as the focus group and questionnaire.

"I think they're coming around," he stressed." I've been talking about this in the UK for five years and when I first started it was a real uphill struggle I can tell you. The first thing was awareness; people just didn't know what it was and they'd say, 'Oh we can just do that internally'.

The reality was that these developers were getting it wrong, or simply sat someone from the QA department down in front of the game and took some feedback after they were done. McAllister said these companies are just wasting their time.

"If you're just playtesting, it's too late," he warned. "You need to involve users - not staff or friends - as early as possible or have an understanding of users as early in development as possible. The most successful games we work on; they start right at the start of the project and I don't think that's a coincidence."

The process typically starts by engaging with the target audience, in particular to discuss to ask them all what it is they like about a specific genre, and to show them potential concepts and art samples. The data is then applied to every step of the design process, right down to UI design, and all the way through playtesting once working code is available.

McAllister closed by saying that this process is employed by almost every entertainment and design industry in the world to make better products, be it movies, cars, gadgets or what have you but that games are in danger of falling behind.

Given the advancements in biometric tech and data capture, you'd like to think that publishers are staring to wake up to the idea. It’s been proven scientifically that these techniques and theories really do make a difference, so it's probably a good idea for publishers to take heed and use them to better understand what it is we as players want from our gaming experience.

What do you make of playtesting and user-research? Are focus tests redundant in the face of new data capture techniques or can they still prove useful? Let us know what you think below.