Info - Game testing

[Cain]

http://www.gamasutra.com/features/20050623/laitinen_02.shtml

 

Better Games Through Usability Evaluation and Testing

 

No one wants to play games that are either frustrating or difficult for the wrong reasons. The best way to make sure that unintended problems do not hinder enjoying the game is to take usability into account in game development. This article presents how this can be done and what kind of results to expect.

 

Usability is an integral part of software development and has been so for the past 20 years. For one reason or another, usability has not gained similar popularity in game development. This, however, is about to change. Ease of use and optimal user experience are already important in games and will become even more so in the future.

 

What is Usability?

Usability is about maximizing effectiveness, efficiency and satisfaction. This definition originates from the traditional software industry, but it translates well to game development. In games, usability is about delivering a better and deeper experience with less unnecessary interruptions or challenges that have not been designed by the developers.

 

Why is Usability Important?

There are many reasons why usability is important in games. For one, playing games is voluntary. If the player has to struggle with problems that make playing less fun than doing something else, then there is nothing to stop the player from switching off the console. This is a serious risk as the user experience is very sensitive to usability problems. Even the smallest glitch or hiccup in the user interface may render an otherwise good game into a rather annoying experience. For example, if managing the inventory in a role playing game is not fluent enough or restarting a race in a driving game is tedious the player is not likely to enjoy playing the game.

 

Another reason why usability is important in games is the competition. Competition in the market is fierce. The gamers can choose which game to buy from a wide variety of titles; if the controls are not fluent in one soccer game, there are five more titles left from which to choose. Usability is one of the key factors that make the game stand out of the crowd.

 

The delicacy of the user experience and heavy competition actually make usability more important in games than it is in other software. There are not too many word processors to choose from, and having fun at work is not usually a top priority.

 

There are also other reasons why ease of use is important. One of them is that modern games are large and complex programs. In even the most focused games there are tons of menus and ways to interact within the game, not to mention games like the Grand Theft Auto series. Usability is important when making a game as easy and intuitive to play as possible. Good examples of complex games made easy are World of Warcraft and Xbox Live games. Both have succeeded in making traditionally difficult multiplayer gaming easy.

 

Usability is also important for the future of gaming. For gaming to continue to increase its popularity, the ease of setting up games and a fluent gaming experience are of the essence. This is because newcomers are not familiar with the conventions and common pitfalls of gaming. For them, learning the peculiarities that old gamers are already familiar with can be too much.

 

This Article

 

The focus of this article is not only to tell why usability is important in games, but also to introduce two usability methods and the kind of results they yield. The methods are expert evaluation and usability testing.

 

It will be presented how these were applied in the development of Frozenbyte's Shadowgrounds (www.shadowgroundsgame.com) game. Adage Corporation (www.adage-usability.com) was responsible for the usability related activities in the project. Before moving on to the methods, the game will be presented briefly.

 

Frozenbyte's Shadowgrounds Game

 

Frozenbyte Inc. is an independent games studio, based in Helsinki, Finland, founded in year 2001. Frozenbyte is currently working on a PC game called Shadowgrounds, which is to be released in the third quarter of 2005.

 

Shadowgrounds is an action game viewed from the top-down/3rd person perspective (see Figure 1). The main features of the game are adrenaline-pumping old-school action, realistic lighting, destructible environment and upgradeable weaponry.

 

 

 

 

Figure 1. Intense battles, lightning effects and upgradeable weapons are key elements of Shadowgrounds.

Usability Expert Evaluation

 

In a usability expert evaluation, usability experts review the game and search for potential usability problems. Based on both their knowledge and experience they review the game systematically and report the findings.

 

In a typical expert evaluation three usability experts review the game. The experts first evaluate the game independently. After that the experts have an evaluation meeting together. In this meeting, the usability problems found are discussed and rated for severity. After that a report is written where the usability problems found are listed systematically. For each problem there is a title, severity rank, and a detailed description of the problem. A solution to each problem is also suggested.

 

This process takes approximately a week. Preliminary results, however, can be given in a few days. After handing in the final report, a meeting between the usability experts and the game developers is arranged. In this meeting, the findings are discussed in more detail, and the usability experts can answer questions that come about from the report.

 

Expert evaluation is a very flexible method, and it can be done at almost any point of game development. In the case of Shadowgrounds the expert evaluation was done approximately six months before the planned deadline of the game. In the version evaluated there was one playable level and the basic gameplay mechanics were implemented. Everything was, however, not ready. For example, the destructible environment was not fully implemented yet, voice acting was missing, and many smaller bugs were still present.

 

It is also possible to conduct an expert evaluation even in earlier stages of game development. For example, menus and displays can already be evaluated on the basis of paper prototypes, and potential usability problems affecting gameplay can be spotted from the design document. The earlier the expert evaluation is done, the easier and more cost efficient it is to make changes.

 

The depth and scope of an expert evaluation are also easy to change. For example, if there is a desire for constant input from the usability experts to the development process, then conducting several smaller usability evaluations with less experts and faster reporting may be a good idea.

 

Results of the Expert Evaluation

In total 135 usability problems were found in the expert evaluation of Shadowgrounds. Out of these problems 2 were classified catastrophic, 30 severe and 60 intermediate. The remaining 43 problems were either minor or cosmetic. The distribution of the problems is illustrated in Figure 2.

 

 

 

 

Figure 2. Classification of the usability problems found in the expert evaluation.

 

The number of usability problems found may seem high, but in reality it is not exceptional by any means. Expert evaluation gives feedback to almost every area of game development. In a typical evaluation, usability problems are found in menus, in-game displays, controls, gameplay and level design. Because of this, the total number of problems found can be quite high. Here are some examples of usability problems along with the game developers' view on the problem.

 

Problem

One color has multiple meanings in the map display.

Rating:

Catastrophic

 

Description:

The light blue color in the map display has three different meanings. It marks the area visited, unknown area, and areas that cannot be accessed (e.g. mountains).

 

Using the same colour to symbolize three different things makes understanding the map difficult. Every time the user opens the map, s/he must stop for a second and think what the colors mean in different locations. There is also a danger that the users do not understand what the colors mean or misinterpret the map.

 

 

Solution

Use different colors for displaying visited, unknown and inaccessible areas.

 

Developers' comment

This problem could have been addressed earlier if the design document had covered the map screen more thoroughly. In the final game, a map legend will be displayed, and only one color (shade) will be used for one meaning.

 

 

Problem:

No feedback is given if the player cannot pick an item.

Rating:

Severe

 

Description:

Sometimes it happens that the player cannot pick up an item because there is no room in the inventory. If this happens, the user is not given any feedback.

 

This is problematic as the user may not know why s/he cannot pick up the item. It is likely that the user will figure it out eventually, but the confusion and extra effort required are likely to cause frustration.

 

 

Solution

Give the user proper feedback in every situation where the user interacts with the environment. If the item cannot be picked up, inform the user about this with a sound and/or textual feedback.

 

Developers' comment

This was addressed only vaguely in the design document, and was overlooked by the programmers at the time. It is likely that this feature would have been implemented in due time, and goes to show that sometimes developers leave many of the smaller usability issues to the final stretch of development.

 

 

Problem

When fixing an item, the feedback is displayed too far away from the location being fixed. Rating

Intermediate

 

Description

The player often fixes items in the game. Fixing an item is done by moving close to the item and holding down E. A progress bar is displayed and the item is fixed when the bar reaches the end. The progress bar is displayed too far away from the item fixed.

 

Presenting the feedback closer to the location at which the action takes place makes it easier to understand that the feedback and action are linked to each other. It also reduces the need for the user to move his/her attention between the feedback and the action.

 

 

Solution

Display the progress bar close to the location it refers to.

 

Developers' comment

This seems like a no-brainer but it went unnoticed for a long time by the development team, because it had grown so accustomed to the game and its features. The feature had only a vague description in the design document (loosely translated to "a repair bar is displayed on the screen"). Had the document been given more thought or had it been read by usability experts before implementing the feature, the feature might have been done correctly from the get-go.

 

 

Problem

It is not immediately obvious whether the upgrades presented next to the weapon are installed or not.

Rating

Intermediate

 

Description:

The upgrades available for the weapons look active even before purchasing them. This is because they are marked with a similar color scheme as the weapons next to them, which are active.

 

This is especially problematic early in the game. Because the upgrades look active, the user may not understand that this menu can be used to purchase new upgrades.

 

 

Solution

Make the upgrades that have not been purchased yet clearly inactive. This can be done for example by using a darker shade of blue to present the upgrades that have not been bought yet.

 

Developers' comment

The weapon upgrade feature had not been fully implemented by the time of the testing, but regardless, this particular problem had not been taken into consideration. Later user-testing showed that this was definitely a problem with gamers, especially in the beginning of the game.

 

 

Problem

Red lights in the computers are easily mixed with enemies displayed in the radar. Rating

Minor

 

Description

Enemies are marked with small red spots in the radar. The radar is semi-transparent: the scenery is visible through it.

 

In some buildings there are computers and other equipment that have small red lights. When it happens that such a light is visible through the radar it is easily thought to be an enemy.

 

 

Solution

Do not use small red lights in levels, or change the method of displaying enemies in the radar.

 

Developers' comment

Sometimes the developer can freely ignore the comments or suggestions made by the usability report. Here, Frozenbyte decided that this problem occurs too rarely to warrant any changes to the user interface.

 

 

After the expert evaluation some of the most central usability problems were fixed to prepare the game for the usability testing that was scheduled to start two weeks later. For example, the map problem discussed earlier and minor usability problems related to the radar were fixed.

 

Better Games Through Usability Evaluation and Testing

 

Usability Testing

 

Usability testing is the most fundamental usability method. In a usability test, the game is tested with players that represent the target group of the game. Just like in an expert evaluation, the goal is to pinpoint the challenges in the game that were not intended by the game developers.

 

The reason why testing is important is that it provides direct and objective information about how real players play the game and what are the exact usability problems that players face when playing the game. This data is irreplaceable when developing the game and making it easy to use.

 

In a typical usability test 3–6 players from each target group of the game are recruited to come and play the game in a usability laboratory (see Figure 3). The players come to the laboratory one at a time and play the selected parts of the game for 1–2 hours.

 

 

 

 

Figure 3. A usability laboratory consists of two rooms with a one-way mirror and a sound proof wall in-between. The user and the test instructor are in the laboratory room. The test is recorded in the observing room and the developers can follow test in the nearby meeting room.

 

In the laboratory, there is a test instructor together with the player. The role of the expert is to give tasks to the player and observe playing the game. The player is instructed to think aloud while playing. In practice this means that the player tells what s/he is doing while playing the game. Sometimes the usability expert may interrupt the player and ask questions about what the player was doing and why. The purpose of these questions is to get in-depth information about the problems and the reasons behind them. Game developers are also invited to come and observe the tests. The onscreen action and an overall view of the laboratory are displayed in a meeting room where the developers can follow the tests and discuss the findings with a usability expert.

 

After the tests, each recording is carefully analyzed to find all the usability problems and to understand the reasons behind them. Then the final report is created. In the final report, all the problems found are prioritized and solutions for the problems are provided.

 

Typically usability testing takes 2–3 weeks from the start to delivering the final report. Initial findings can be provided even earlier, and it is also possible to change the version tested between the players. This makes it possible to fix the problems during the testing and thus to verify the changes later on in the same test.

 

In the case of Shadowgrounds the usability test was conducted two weeks after the expert evaluation. The focus of the testing was broad, and all the aspects but the main menu of the game were tested. The testing was done with a version similar to the version used in the expert evaluation. Like mentioned earlier, some usability problems were fixed between the expert evaluation and the test. Next, examples of the problems found will be presented.

 

Results of the Usability Test

In the usability tests 97 usability problems were found. Finding less problems in usability testing than in expert evaluation is typical: players do not necessarily use every feature of the game, and some of the minor problems found in the evaluation do not necessarily show in the tests. One of the problems found was classified catastrophic, 26 severe, 28 intermediate, and 42 minor or cosmetic. This is illustrated in Figure 4.

 

 

 

 

Figure 4. Classification of the usability problems found in the expert evaluation.

 

Next, five examples of problems classified as severe are presented. In addition to presenting the problems and their suggested solutions there is also game developer's comments after each problem.

 

Problem

Picking up a new weapon went unnoticed.

Rating

Severe

 

Description

It happened twice in the usability tests that a user picked up a laser rifle but did not notice it before entering the weapon upgrade menu later on. As acquiring new weapons is an important reward in the game this should not go unnoticed.

 

 

Solution

Give clear feedback when the user finds or acquires a new weapon.

 

Developers' comment

There was feedback for picking up a new weapon - the information bar in bottom of the screen displayed a text message. In this case the feedback was not noticeable nor rewarding enough, and will be improved in the final game.

 

 

Problem

The character moves too slowly. Rating

Severe

 

Description

Two out of six users commented spontaneously that the character moves too slowly. The movement speed was considered especially problematic in open spaces. Inside the buildings the speed was considered to be good. See also the survey results.

 

 

Solution

Consider implementing a running option.

 

Developers' comment

This could be one of the biggest issues in games - the movement speed of the units/characters. In our case, the problem was mostly outdoors, which would suggest that the outdoor areas were not interesting by themselves, and the player wanted to get to the next interesting area (building). It will be interesting to see if this is the case, when the final game is tested

 

 

Problem

The users did not understand how the radar works.

Rating

Severe

 

Description

The radar displays only the enemies that are moving. None of the users understood this. Most of the users thought that the enemies are shown in the radar when they are close enough.

 

Many users also thought that the radar cannot see through walls. In reality, the radar does not care about obstacles.

 

 

Solution

Inform the users about the radar and how it works in the tutorial of the game. Consider labelling the radar as motion sensor.

 

Developers' comment:

The radar will be renamed to motion sensor (which it is). Apparently the aliens theme wasn't enough to make the players understand that the circular element in the top-right corner worked like a motion sensor, even if it had the familiar beeps.

 

 

Problem

Finding doors is difficult.

Rating

Severe

 

Description

The users had difficulties in finding doors, and it happened often that the user thought that either a window or some other structure was a door.

 

 

Solution

If the door is not hidden on purpose mark it clearly. Symbols and/or light effects should be considered. Change the shape of the windows and the buildings' support structures so that they are not mistaken as doors.

 

Developers' comment

This is a tough problem, and hard to predict in the design phase. Additional lights could be added, and the buildings could be remodeled, but this takes a lot of effort. In the end, this problem seemed big enough to warrant changes, and each door was scrutinized individually to make sure it worked alright.

 

 

Problem

Changing of the mission objective was not always noticed

Rating

Severe

 

Description

During the tests it happened a few times that the users did not notice that the mission objective had changed. This was because the users did not always pay close attention to the dialogue between the characters.

 

 

Solution

Provide the users a clear notification about a new mission objective. It is not enough to inform the users about the new objective in the in-game dialogue.

 

Developers' comment

Part of this problem was caused by unpolished dialogue (sometimes the texts went by too fast). English wasn't the users' mother tongue, which could have affected the outcome as well. The new mission objectives (and the target point) were presented in the map screen, but not all users had understood this very well (due to the lack of tutorial before playing the game). " Mission objectives updated" was displayed in the information bar at the bottom of the screen, and also as a icon on the right side of the screen, but the icon was not understood clearly. In the final game, all of these problems will be fixed.

 

 

Benefits of Usability Expert Evaluation and Testing

 

Usability expert evaluation and testing are systematic methodologies that provide information to support game development. Together they provide both the experts' view on usability and experimental data of usability problems in the game. The usefulness of the methods is illustrated by the following quotes from Joel Kinnunen, the development director of Frozenbyte:

 

On expert evaluation:

 

"Expert evaluation is a fast and effective way to check the usability of a game. In our case, the results arrived in a couple of weeks, and they helped us solve some major design issues. We were also able to fix numerous smaller usability problems, and avoid a couple of potential pitfalls in designing and implementing new features.â€