Usability, Focus Testing and Playtesting

04 May Usability, Focus Testing and Playtesting

Tameem Antoniades

As you may know we are currently working on our Vertical Slice, the details of which were covered recently here

In summation, it is a small section of the game to near-finished standard that allows us to see if the game we imagined in our heads actually works!  The vertical slice covers everything from combat, exploration, puzzles, visuals, audio, music…the full shebang.

I am very aware that we are all fallible human beings and one big weakness of the human ape is our optimistic outlook: to overestimate how good something we create actually is in the eyes of others.

When you’re too close to a project, you cannot trust your own judgement and so you need to actively work against your preconceptions and just assume that what you think is great is probably… a little bit shit.

You often see creators extolling the virtues of their upcoming project confidently proclaiming it to be amazing only to be genuinely shocked that the world doesn’t appreciate their efforts.  From the outside it’s baffling.  It’s just human nature and I’ve fallen into this trap myself.

So what can you do about it? Well, you put it in front of others and WATCH and LISTEN!  In games, you’ll hear about focus testing, playtesting and usability testing.  If you’re not sure what separates the three, you’re in good company, many seasoned developers cannot tell you the difference with any great certainty.  To complicate matters, the terminology is often used interchangeably. Here’s my take on it based on my personal experience:

Focus Testing: Best for washing liquid packaging

Typically done by publishers, this is one to avoid like the plague.  The process involves getting a bunch of random people off the street to examine a piece of concept art, or character, or gameplay description and give their judgement to a moderator.

A report is written up, the publisher panics and asks you to change said gameplay, character or concept art because someone who knows nothing about such things doesn’t like it. In effect, the design process is taken away from experts and put into the hands of a layman.

The worst offending publishers actually design their games through this process.  I’ve witnessed had a room full of people at a publisher tell me that they designed the perfect game, rated off-the-charts through this process, if only they could find a developer to make it.  The games designed this way are usually something like: GTA meets COD meets WOW.  All the BEST GAME EVER’s combined into THE BEST GAME EVER!!!

And it’s not just games: both New Coke and the London Olympics mascots were all designed using focus groups:

The-London-2012-Olympic-m-011

Play Testing: Best for Balancing

This is where you get a large group of people to play your game.  The players are vetted to be the types of people who you want to sell to.  So if you are doing a driving game, you might find people who casually play driving games, and those that love driving games.  The more people the better.

You then let them play the game and track data, either through questionnaires or in-game statistical gathering, or both.  These days, it’s possible to make this group very large as you can do it online through closed and open beta testing.

Largely though, you do this when you have a game that is fully featured and you need to iron out the peaks and troughs of difficulty, progression, balancing issues and so on.

We experimented with a free to play game called Fightback a couple years ago.  F2P is a much maligned concept for many but just forget about the monetization aspect for a moment.  The ability to track millions of players’ progress through the game, where they drop out, where they get stuck, and so on using very sophisticated tools is playtesting taken to an extreme scale.   There are vital lessons to be learnt there for all game developers.

Usability Testing: Best for Improving the Game

Usability testing is my favourite of the three types of testing and the one that has the most dramatic impact on the quality of the game you are making.

It’s simply watching a small handful of people play your game and gently asking them to talk about what they are experiencing along the way.  And you always make the full team watch the play through.

With about 3 people playing your game for around an hour each, you can usually identify 80% of issues you had no idea existed.  You do this regularly throughout development until you have nailed the experience you wanted people to have.

This process only works if you start it early enough – a year or more before you are due to finish the game.  If you wait until the back end of production, it’s probably too late to make a great game, but still worth doing nevertheless.

We first came across usability testing when working with Microsoft Games Studios who have it down to an art form with massive dedicated usability labs often run by psychologists.

Once you experience a project that goes through good usability testing, there is no going back.  It’s brutal, sometimes shocking but always honest and useful.  Best of all, it doesn’t take a lot to do this yourself.

We always ask our publishers to run extensive usability sessions but most aren’t too familiar with the process and, to my amazement, some are against it, citing cost, distraction or confidentiality concerns.

Given we are independently making Hellblade, we no longer have to ask permission to do this so we have renovated a tiny room set up with a sofa, a camera, mic and a little water tank with what admittedly looks like a dead fish in it. It does the job.

Q&A

The Vertical Slice Sessions

We now have a first pass Vertical Slice up and running so we asked a few of our internal staff from other projects to have a go while the full Hellblade team sat in a meeting room and watched.  It was horrific to watch.  So many assumptions we inadvertently made were utterly destroyed by this first test.   The team was very despondent at this point.  In fact, developers generally fear this kind of testing because it shatters your best laid plans and with it, your confidence.

But however brutal the process is, it is nothing compared to how vociferous and critical your players will be once the game is out (believe me I know!).

We made some big changes after that session and were pretty confident for our next round.  Once again, everything unravelled before our eyes but not nearly as bad as the first time.   The mood was less bleak this time and, after a pub meeting, we had a new plan of attack.  The next one will be the one!

Well, no it won’t but this is exactly what game development is all about: incremental iterative improvements that puts the player front and centre.  Nothing focuses the team on the player experience more sharply than usability testing.

We will be doing this every month until we ship.  However painful it feels at the moment, I know it will get better and better and that, by the end of it, we will have created a game we can feel proud of and one that will hopefully meet your expectations.

 

8 Comments
  • Sean McLaughlin
    Posted at 06:34h, 04 May Reply

    Would you consider a limited alpha? I really agree with player feedback during the process of playing the aforementioned. I think a select group playing a limited slice of your game, mic’d/camered up a la Twitch, would be a great way to demonstrate constructive feedback from a demo. You really never know how a player experiences your piece, until you see visual and audible feedback from the player, during their experience with your work; whether it be polished or raw. Thanks for keeping people in the loop on your dev processes; it is very refreshing. And I, for one, would love to give you input, in any way, shape or form, if you are so inclined to request it.

  • André Vila Franca
    Posted at 11:18h, 04 May Reply

    The best way to go guys! Keep up the great work!

  • Malik Robinson
    Posted at 18:51h, 04 May Reply

    Really excited to see what the public demo is going to be like.

  • Jay
    Posted at 06:48h, 05 May Reply

    Design by committee is the worst thing ever!
    Getting honest feedback is priceless though.
    Love the insight in your development process. Keep it up!

  • RodyHW
    Posted at 16:17h, 05 May Reply

    I may be repeating myself (looks like I’m doing iterative processes too haha) but I couldn’t agree more with your way of making a game.

    I can barely imagine how hard it is to watch a handful of people destroy your hard work but don’t give up !This game has everything he needs to be awesum.

    Keep moving forward !

  • Vartul
    Posted at 14:53h, 06 May Reply

    Larian’s Swen Vincke once said that game development is ‘enlightened despotism’. So while it’s a great idea watching people play your game, while taking their suggestions one should be a lot more cautious. Players will often find out that something is wrong, less often they’ll find out exactly what is wrong, and rarely, if ever, will they make good suggestions towards the solution of the problem. All the best, I hope Hellblade turns out wonderful. 🙂

  • Link Ebanks
    Posted at 23:43h, 08 May Reply

    Looking forward to E3 more and more. Hopefully you’all will have something to show – during Sony’s press conference lol

  • Andrew Mitchell
    Posted at 10:27h, 11 May Reply

    Hellblade looks pant-wettingly good . Elements of skyrim mixed with shadow of mordor , celtic myths , viking myths, absolutely amazing .It ticks all the right boxes for me and I cannot wait for it to be released .Visually it looks stunning , the music and sound track sounds like it is spot on . The enemy design looks ace . This looks like it could well be the best game ever . Keep up the work chaps and chapesses .

Post A Comment