What We Learned From Testing At AwesomeCon 2017

Hey everybody, it’s Frank! I just got back from a trip to Washington D.C. for AwesomeCon 2017, a comic convention that’s expanding its selection of gaming exhibits. We were invited by the wonderful team that hosts the MAGFest Indie Videogame Showcase to take part in their giant indie booth – thanks to Lexi Urell and her team for allowing us to take part in such an awesome con!

 

20170617_211819

Why Did Frank Go To AwesomeCon?

That’s kind of a weird question, right? Is there ever a reason not to go to a convention? Besides, we were invited! Do you even need to ask?

Now that the Game Revenant official coffers are looking a little emptier, it’s important to evaluate every large expense. Travel is certainly one of them. While I’d love to go to every show on planet Earth that’s even remotely related to gaming, we don’t have that kind of cash to spend. Besides that, there’s the time cost involved. If I’m standing at a table showing off Where Shadows Slumber for 3 days straight, that’s 3 days I’m not spending doing animations or environment art for the game. Was it worth it?

We decided that the best way to get a return-on-investment for our time and money was to focus on one very specific thing during AwesomeCon 2017 – testing. Conventions are a great way to show your game to a lot of people. It may seem like this is purely a marketing activity where indies promote their game, but that’s a shallow view of what conventions can do for you. When you’re given the opportunity to sit down with nearly 100 people and focus on your game, that’s a great time to ask them critical questions about your work and get their honest feedback.

So before I left, Jack created a build of our Where Shadows Slumber alpha that had all 17 of our test Levels in it, along with a basic menu for easy navigation. I resolved to show this early alpha to as many people as possible, with a specific focus on these key issues:

  1. If I don’t tell Players how to play the game, what will they do?
  2. What do Players think of the first three Levels, which are meant as a tutorial?
  3. How far will Players go before they get stuck or bored?

 

20170616_182723

My Testing Procedure This Time Around

As you test your game at a convention, you begin to find a consistent testing method that works. Halfway through the first day (Friday), I had a pitch ready to go once people sat down at the Where Shadows Slumber table.

I was really straightforward with people. I told them that I wasn’t going to teach them how to play because I wanted to see how they performed on their own. (No one seemed to mind!) Then I told them that they could ask me questions if they got really stuck. I told them that the game’s artwork was a placeholder. The only information they were allowed to know was that it was a puzzle game called Where Shadows Slumber. With that, I just watched them play through Level 0-1 and noted their progress. This pitch accomplished a few key things.

This Is Only A Test: Setting up expectations right away is key. By telling people that the game is being tested (and not them) it put them in the proper mindset. They weren’t here to be entertained – they were here to break the game if possible, and try to beat it. I think that increased people’s enjoyment actually, and definitely led to finding some serious bugs.

Ask Me Questions: Getting people to talk while they play is really hard, but it’s very important. You can only glean so much from watching people. I didn’t give anyone that much information, but allowing them to ask questions is helpful. After all, if they ask a question, it means they don’t understand something. That “something” is what Jack and I have to go back and add to our tutorial.

Don’t Tell Me The Art Sucks: It’s important that you tell people what you don’t want to hear. Setting up this expectation decreased the amount of people who would complain about the art. Seeing this alpha next to screenshots of our beautiful demo was probably  a bit jarring, but once I explained it to testers it wasn’t an issue anymore. When you’re testing, you don’t have much time with each person so you need to make it count. Make sure that people know what you already know, so they focus on different issues.

 

20170616_173920

The Results

To my surprise, people loved the alpha! I only say I am surprised because this is the first time I’ve seen people play it with my own eyes. And although the artwork is all just placeholders and the Levels are brand new, people gave it glowing reviews:

Thor

Having said that, not everything is sunshine and rainbows. We found a few bugs over the weekend, and there are some Levels that may need to be redesigned or cut from the game entirely. Here are all my notes from AwesomeCon 2017:

 

  • People don’t realize they can’t drag something if the Player is in the way. Draggable objects should smack into the protagonist to give them feedback on this matter.
  • Someone suggested a mechanic where torches (lights) are only on for a fixed amount of time before they shut off.
  • Someone requested a Reset button (which our demo has, but the alpha does not – even though you can just re-select the current Level from the menu).
  • MAJOR ISSUE: People didn’t realize they could drag red objects. Many suggested that they “shimmer” when they are dormant to encourage dragging. Perhaps there should be a handle on the Draggable object to indicate that it is interactive, and show the direction it moves. They should glow when they are being dragged as well.
  • Someone suggested a UI indicator that shows how a Draggable moves, since some objects rotate but others slide across the floor.
  • When the Player is following closely behind a Walker, he stutters and stops, producing an awkward floating animation.
  • The protagonist’s light should grow out from him and stop at the predetermined radius needed to solve this Level.
  • MAJOR ISSUE: Every single Player (with few exceptions) dragged-to-move if I didn’t tell them the controls. Our game is tap-to-move, so dragging is not an optimal way to play. People assume the controls are bad, but they’re just doing it wrong. Without a way to correct them, they make it harder on themselves.
  • Someone suggested charting a path (like in StarCraft) when you drag-to-move, a possible solution to those who find that way more comfortable. This would basically be like connecting the dots between every space you dragged over.
  • IDEA FOR A LEVEL: Level 1-3’s “Lock”, but the Light Switches are connected to some of the Rotating Draggable blocks.
  • MAJOR ISSUE: People tried to drag the purple blocks, but couldn’t. This stopped them from trying things in the future.
  • Glyphs are really just buttons that can be pressed infinite times, right?
  • Draggable Light Switches need to be turned off when they’re off. They still appear on, which is impairing people’s understanding of the light mechanic.
  • The age when players seem able to understand the game is 12 – younger children could trudge through it by trial and error, but with limited understanding.
  • MAJOR ISSUE: “Why is there a shadow?” People do not realize the main character has a lantern with a massive radius and it’s the only light in the scene. This is understandable because our game is super weird. We need to find a way to show this constantly, or they’ll think the shadows have a mind of their own.
  • Someone suggested a mechanic where the main character’s lantern is a spotlight, instead of a point light, for a few Levels.
  • Someone suggested a mechanic where the main character can lower their lantern’s light radius and then reset it, for a few Levels.
  • A businesswoman with knowledge of the Indian market suggested that we lower the price from $5 for that particular market. She felt strongly that Indian mobile gamers wanted free games or something much cheaper.

Here are my notes that are specific to each Level in the alpha.

 

0-1.PNG

Level 0-1, Fallen

There’s a bug in this level where there bridge (which should fall after you press a trigger) stays exactly where it is. Players who drag-to-move skip right over the trigger, and they never trigger the bridge sequence, so basically they miss the puzzle.

The Draggable box on this level doesn’t have much weight to it. People fling it around like crazy. They also really want to drag it down (onto the dirt path), up (onto the dirt path), or onto the bridge to drop it into the water as a makeshift bridge. None of that is possible but there’s no feedback for that and they don’t know how shadows work yet so it doesn’t register.

Half of the people who play this Level don’t quite understand that the shadow makes the bridge appear.

It’s possible to walk past the Goal Space, and go to a spot on the Level that is beyond the door.

This level is not idiot-proof, like the first Level in our demo.

I think this is our weakest Level. I suggest cutting it and replacing it with a walking tutorial similar to the first Level in the demo. This Level is just throwing way too much at Players all at once.

 

0-2

Level 0-2, Bridge

An excellent Level. This serves as a perfect introduction to 3 key mechanics: walking, shadow revelation, and dragging.

The Rotatable bridges here should probably wobble after a while to indicate they can be dragged. I can also make a circular pivot point in the center, cut into the stone. That would be a good indication that these are on a swivel.

Draggables can also have parts on them that suck in when Players hold them down. Having parts of the stone depress inward is a good sign that you’re controlling the object with your finger.

 

0-3.PNG

Level 0-3, Monolith

This Level is perfect teaching. It’s a great gateway – you will never beat this if you do not understand how shadows work in our game.

“The purple box moved!” We need to make sure people don’t think the shadows merely move things. They make things appear and disappear… the visual style of the purple box makes it seem like it’s jumping around.

Why can’t Players make the farthest purple block appear if they are standing all the way at the entrance of the Level?

The Draggable Block here should be on some kind of a flagpole so that the vertical movement appears to be a natural fit to Players. (Many tried to move it horizontally.)

 

1-1

Level 1-1, Recovery

The name of this Level ought to be “Protection” or even just “Light”.

Why is the Light Switch casting a shadow? Does that shadow do anything? That may be a visual error.

 

1-2.PNG

Level 1-2, Detour

This Level can be broken to make both Goal Spaces appear at the same time. Players usually move the Draggable Block back and forth so rapidly that it causes both to be visible. However, the fake Goal Space does not work. If we can’t fix this bug… we should make it work! Why not reward Players for their trickery?

If there was a Light Switch near the space where the Goal Space is revealed, this Level would be a bit harder. You’d have to make sure the Light Switch was off. That may make it more interesting for the Players who figure it out in two seconds – and it keeps the World’s atmosphere consistent, since we use a lot of lights here.

The shadow needs to change more of the Level when it swipes across the screen, to give Players a clue that something weird is going on.

There ought to be two Shadow Eyes on the Draggable Block.

 

1-3.PNG

Level 1-3, Lock

Let’s make the sides of the Rotating Blocks sloped here, or at least spiked. People consistently try to walk on the sides of them when they are down, but that would break the Lights. It must appear unwalkable.

 

1-4.PNG

Level 1-4, Pressure

Extremely hard Level. That’s a good thing to have at this point in the game.

“I didn’t know I could stand on the box and rotate it.” Are we being consistent with when Players can do this and when they cannot?

How will Shadow Eyes work here? How can we align them with the object they are changing?

Someone found a bug where both buttons were pressed and they beat the Level, but they could not walk on the green path. (This is a soft crash I guess, since the Level is broken but the game still works fine.)

 

1-5.PNG

Level 1-5, Wolf

This Level should be renamed to something that indicates how to solve the puzzle, like “Doors” or “Black” or “Pitch”.

People don’t know they can drag these pillars.

The effect of pressing a Button here was not always obvious. I need to make an animation and we ought to have a clear sound attached to it.

On the iPhone, there was a bug where the sliding pillars could not be dragged. We had to reset the Level. I suspect Glyphs have something to do with this.

 

2-1.PNG

Level 2-1, Docks

Literally every tester thought the Walkers would hurt them and everyone called them “zombies”. My use of the color green was foolish!

We should start this Level with a Walker coming toward you that you can’t avoid, so people see that they aren’t bad.

People LOVE the reveal with the pillar sweeping across the Level. We should do more.

People tried to reverse the reveal and they couldn’t do that, which upset them. I think they wanted to see it more than once. When we get it set up properly, let’s consider this. It’s about consistency and Players enjoying the game for its toys rather than its puzzles.

 

2-2.PNG

Level 2-2, Test

We can call this Level “Elevator” or something. Maybe “Switch”, because you press a switch, but you also need to change places with the Walker.

Walkers flip around when you rotate Draggable Bridges, and this really annoys Players who are trying to guide his path. Also sometimes the Walkers float, breaking immersion.

 

2-3.PNG

Level 2-3, Guide

Pressed Buttons really ought to look pressed. I need to redo the art and then I’ll need help setting the states properly. We can also drain them of color once pressed.

For some reason I think buttons should be octagons. Why did I write this?

 

2-4.PNG

Level 2-4, Ebb

These Walkers cast a light, but they don’t have an obvious light source. I can make them holding torches, but what happened to their little light bulbs? Did I delete them?

 

3-2

Level 3-2, Tradeoff

The main light in this Level looks like it’s off because it’s so dark. The Player’s lantern doesn’t always need to be the brightest light in the scene! This sliding light is way more important to the mechanics of the Level. We can dim the Player’s light in favor of the other one.

 

3-3

Level 3-3, Anchor

Rectangles can pass through each other.

The right side Button node was briefly unwalkable, due to a multiple reality error.

After leaving a node, the state of a Button was still pressed. This made the Level unbeatable.

 

3-4.PNG

Level 3-4, Torus

“Is that it?” Torus looks more intimidating than it is. Can we bring up the difficulty on this one somehow? I think people are disappointed that you don’t need to find a way to navigate back and forth using the rotating segments. It is solved quite easily.

 

3-5.PNG

Level 3-5, Island

This Level can be broken by drag-dashing back and forth until the pillars remain upright. Then, walk into the island, the pillars lower, and you beat the puzzle without really solving anything.

 

It’s incredible how much insight you can get from just a few days of testing! These kind of testing moments are hard to come by, so it’s important to make the most of them. I hope you appreciated seeing how your feedback will impact the game, and this gave you an insight into what indie developers are looking for from testers.

We’ve got a lot of work cut out for us this month, so expect to see these changes reflected in my post at the end of June where I update you on the state of the game’s artwork.

 

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =

We hope you enjoyed this insight into our testing methods. Do you have any feedback for us about the game’s alpha? You can reach out to us at WhereShadowsSlumber.com, tweet at us on Twitter (@GameRevenant), message us on Facebook, leave a comment on itch.io, jump into chat on Twitch, and email us directly at contact@GameRevenant.com.

Frank DiCola is the founder of Game Revenant and the artist for Where Shadows Slumber.

Advertisements

Where Shadows Slumber: Testing

This week marks the deployment of our first batch of test levels, assuming I haven’t totally screwed up and delayed them. Throughout the past few months, we’ve been building a list of testers, and it is finally time to put it to good use!

If you’re on the list and you have an iOS device, you should have received an invitation to our TestFlight over the weekend. If you’re on Android, you can expect a similar email later this week. If you’re not on the list (or don’t know if you are), just let us know! We haven’t hit the limit on testers yet, and every pair of eyes helps us make a better game.

Frank already wrote about our testing, and why we’re doing it. This post touches on a lot of the same things, but I want to delve a little bit more into the benefits of testing.

 

Mommy, look what I made!

A child returns home, an exuberant look on her face. “Mommy, look what I made!” she exclaims, holding up a piece of construction paper covered in marker lines and dried macaroni. Her mother reaches down and takes the paper. “Isn’t it great?”

“Why, yes, darling, it’s the most beautiful piece of art I’ve ever seen!”

macaroniwithjack

The original concept art Frank made for Where Shadows Slumber

Now, you and I know that the mother in this story is not telling the truth. Obviously, Where Shadows Slumber is the most beautiful piece of art she’s ever seen. But her daughter doesn’t know that – her daughter planned out the art, decided exactly how she would approach the task, and executed flawlessly.

Frank and I are the daughter in this metaphor, and Where Shadows Slumber is the macaroni masterpiece. We look at our game and we see something beautiful – but who knows what it actually looks like?

You do! You, our adoring public, are our mother. However, we need you to tell us the truth! We will never grow up to be a wonderful artist if you tell us that our macaroni levels are beautifully designed when they’re not.

This is the concept behind a lot of what we have been doing over the past year. We created a demo for the sole purpose of showing it to people and getting feedback about the game. We’re sending out test levels to get feedback about our level design. We even write this blog, in part, to get feedback about our process!

We use all of this feedback to help make Where Shadows Slumber a better game. That is the benefit of testing.

 

The Power of the POC

If you’re a game developer, and your friends and family know you’re a game developer, then there is a phenomenon I’m sure you’re quite familiar with. If not, allow me to explain.

Most people don’t understand the amount of time and effort that goes into the development of a game. Therefore, if someone thinks of a half-decent game idea, they come to you with it. If I had a dollar for every time someone has said “Jack, listen to this idea for a game – you’ll be a millionaire!”, I actually would be a millionaire.

So, you end with a lot more game ideas then you can possibly make. Some of them might actually be pretty good, but you’re just one person – how can you tell if an idea will end up working out? Do you have to just pick a concept, make the whole game, and then just hope that people like it?

This is exactly what proof of concept projects are for! Can you imagine if we had spent two years making the full game for Where Shadows Slumber, only to release it and find out that nobody enjoys shadow-based puzzles? What a waste! But spending two months working on a project that could become something big is totally worth it, even if people don’t like it.

POC

A shot from one of our earliest POCs

That’s exactly how Where Shadows Slumber started. I came up with the idea, and I immediately spent two months or so developing a POC, a very, very basic version of the game, that would just be used to tell if the game had any merit. I showed it to Frank and a few other friends, and they liked it, so we decided to make the full game. If they hadn’t liked it, then we would have scrapped the project. I would have wasted two months, but I would have known that it was a project not worth pursuing, without wasting even more time.

 

Testing Design

In my opinion, the hardest part of game development is design. Programming is easy enough, once you know what you want to program. Art, on the other hand, would be the hardest part if it weren’t for Frank, but that’s just because I’m bad at art.

Once you know what you want to make, programming and art are mostly execution on that vision. Coming up with that vision is the hard part. What is your game mechanic? How does it work? How do you explain how it works to the player? What does your difficulty curve look like? On and on, there are thousands of questions like this that you can apply to game design, and they’re all important.

This is one of the main reasons we try to get as many people as possible to play our game. There are certain realizations about design that you can only get by showing it to a lot of people and getting feedback. We never would have discovered how much people dislike ‘randomness’, which is something that could have played a major part in our game, if we hadn’t shown it to a bunch of people.

But even once you’ve made all of the decisions regarding your game’s mechanics, you still need that feedback on the last giant piece of the game design puzzle: level design. If you have an awesome mechanic, but your levels are boring and easy, or way too hard, nobody’s gonna want to play.

In order to prevent this, we’re doing some alpha testing! We have all of the levels designed, and we have all of the mechanics half-implemented, so we’re sending out test levels!

Crumble

This is what it would look like if I were in charge of the art…

These levels are ugly – but we’re not testing the art! These levels are buggy – but we’re not testing the code! These levels just might be poorly-designed, and that’s what we want to know. Our testers will tell us what they like and don’t like about these levels, and we will update them accordingly.

The important part of this process is that it’s happening as early as possible. If we sent out fully-complete levels, and then we had to change one, we would end up either scrapping the art, which is a huge waste, or trying to change the level without changing the art, which just makes it look weird. That’s why these levels look all dull – it’s all part of the plan!

 

Testing Philosophy

The last thing you want is to work hard for two years on a project, only to release it and find that it’s not as great as you thought it was. In particular, no matter how great you think it is, your audience might not enjoy it. While there will always be people that don’t like your game, it’s important to make sure that your target audience does like it. And how do you ensure that will happen? Testing!

It’s not a great experience to hear someone say they don’t like your game, and it’s perfectly natural to shy away from that. However, hearing that from a dozen people and still having time to make changes is a whole lot better than hearing it from a thousand people after you’ve already released the game.

Whether it be unit tests (small tests to make sure that one part of your code works) or beta testing (sending an almost-complete game out to fans to look for small bugs and last-minute fixes), testing is an important part of development. Don’t get caught up in your own little game-dev world; make sure you find out what the people want!

As always, let us know if you have any questions or comments about testing (or anything else)! You can always find out more about our game at WhereShadowsSlumber.com, find us on Twitter (@GameRevenant), Facebook, itch.io, or Twitch, and feel free to email us directly with any questions or feedback at contact@GameRevenant.com.

 

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =

Jack Kelly is the head developer and designer for Where Shadows Slumber.

Calling All Testers!

I speak now to our legions of adoring fans. *ahem*

Lend us your talents! Your eyes that see microscopic faults, your ears that hear the lies in the truth! Lend us your hands that stumble over screens and fingers that are too large! Lend us your tongue that licks the phone for some reason (??), and most importantly your mind, which is not satisfied until there are 999 levels in the game!

john_f_kennedy-h

“Ask not what Game Revenant can do for you, but rather what you can do for Game Revenant!”

[The crowd goes wild]

The call has gone out to Android Revenants and Apple Revenants alike. Your Supreme Chancellor has need of you!

c4pi-04vyaausr_

Part of a paper concept for the upcoming level “Ramparts”.

Jack and I have been hard at work the past two months designing every level in the game on paper. Now we need Volunteer Revenants who are willing to take 15 minutes out of their day to test these new levels. We’ll be sending you an early, near-prehistoric version of the final game. Levels will be blocky, ugly, and impossibly grey. They will have either no sound, or limited sound. You probably won’t enjoy playing them.

Sold yet? Listen, we need your perspective. We think these levels are perfect. Do you know why we think that? Because we designed them! Of course we think they are perfect. They are not. We need you to download the test version of the game and then tell us via Facebook, Twitter, or private email (contact@GameRevenant.com) what you think. Be honest!

greygrongus

I played the demo available on the App Store. Does that count?

Thanks so much, but no. We’re moving past that now — we’re talking about new levels for the final game. Not many demo levels will make it to the real game.

How do I become an iOS tester?

You’ll need the app “TestFlight” on your modern iOS device (iPad, iPhone, iPod Touch) and you’ll need to send us your email address to opt into the program. We’ll add you to the list and send out levels in waves, probably one World each month.

How do I become an Android tester?

You must send us your email address so we can figure something out. TestFlight does not work with Android so we need a different solution. We may just email you an .apk file with some levels in it.

What’s the deal with Kindle Fire?

I don’t really know… we aren’t on that store yet, it’s a long process. Hang tight!

I have a Windows Phone…

Windows Phone is kind of a small market and we don’t have one of those devices to use for testing. We will probably not publish on the Windows platform for a while, if at all.

You guys get free quality assurance from this. What do I get out of it?

You get the satisfaction of knowing you’re helping us make a great game! You get to see the game early, before anyone else. You get to tell us what you think and make a real impact on the game. Plus, if you pester us enough and your feedback is helpful (instead of just annoying) we can probably put you in the Credits as one of our “Worldwide Quality Assurance Experts”. Play games in your pajamas and get into the Credits… what’s better than that?

tanuki2b-2bi2bwill2bdo2bit2bi2bwill2btake2bthe2bring2binto2bmordor

OK

OK?

Yes

Great! Send your information to contact@GameRevenant.com. All we need is your email, device operating system, and device version/name. You’ll be sorted and placed in the appropriate email list. Expect a brief from us soon about how you can be a Test Revenant!

 

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =

Although I don’t recommend it, you can also give us your email through the Game Revenant official Facebook Page or Twitter Handle. I also have a Twitch game development stream, so I guess you could hop into the chat and message me? Don’t do that, though. Spammers are always looking for information in the chat! Email the official address at contact@GameRevenant.com.

Frank DiCola is the founder of Game Revenant and the artist for Where Shadows Slumber.