Full Solo Playtest

#FF644E
#5E5E5E

---

Situation:

I was tasked with leading my first playtest. Solo. A few weeks prior I was only assigned to solo execution, with a senior researcher crating materials. But as the weeks went by and workloads piled up eventually everything but recruitment came down to me. This was both exciting and daunting, but I was ready for the challenge! I had led projects in the past ant NetEase, like creating monthly meetings, reports, and parts of playtests, but never an entire playtest.

Research Objectives:

The research objectives had been updated and in some cases completely changed after communicating with product team. This was pretty last minute, but these updated ones were a lot more specific and gave me more to work with, even if I didn’t have much time. The research objectives covered 3 main categories:

Initially, there was meant to be 4 sessions, but time constraints on the rented facility reduced this to 3 sessions, with 3-4 players each.

Challenges:

I had a number of challenges going into this project. First off, before starting prepping for this project it was back to back playtests, and working overtime every day just for those projects. By the time those were all wrapped, I had 4 days before flying out for this playtest. On top of this, some plans had changed as workload piled up for the team, and I was now in charge of all the material prep. The original plan was for me to only do protocol and execution. Needless to say, this would be an uphill battle.

Additionally, this would be my first time fully leading a playtest. I had lead projects in the past, and playtest executions before, but never the full front to back prep, execution, and analysis of a playtest. This brought it’s own challenges, including having to muster up a lot of confidence.

The prep, execution, and analysis where also mine and mine alone to accomplish. A totally solo mission, only the recruitment thankfully being done by my manager. This had an upside of not having to delegate work, where I could focus on the learning experience. The material and protocol creation, and then executing and analyzing my own work. Of course it also had down sides. Having to be confident, decisive, and just having to do a whole lot of work all by myself. Luckily, I had the mentorship of a senior coworker and manager I could ask advice from, and of course the ability to think back to my experience leading different parts of projects and stitching them together into this one.

Lastly, communicating with the product team had a few challenges. First, we were on such a short time frame but there were still last minute changes requested and things needed to be clarified. Being a solo project made pivoting when requested easier, but far from easy. Secondly, the 12 hour time zone difference meant that I could have questions that might not be answered until after my work day is over. This meant that if I played my cards wrong I could ask a question, work off an assumption, and then realize all that work was wasted once product team got back to me. I had to make sure to avoid this.

Tasks:

Being a solo project, I was responsible for a lot of tasks. As mentioned earlier, I was on a short timeline and the timeline looked something like this:

• 4 days of material prep
• 2 days of tech setup (solo in rented facility)
• 2 days of execution (3 sessions in total)
• 1 day for analysis

The first and largest obstacle I needed to face was the material prep. The first part of this was getting the research objectives set in stone, as what was first handed to me was very vague. Following that I was tasked with creating the protocol, interview guides, questionaries, observation sheet, and marketing print outs.

Secondly, I had to fly out to Los Angeles to the facility and setup the room and all the tech for the playtest. We did not have our own lab so we would frequent various faculties to do the setup, and usually with 3-4 researchers to do the setup and execution, but this time it was just me.

After the tech setup I of course did the execution, 3 playtests over 2 days, with about 10 participants total. Again all solo, including the tech tear-down and cleaning on the second day. Since we were renting lab space from vendors, I always tried to leave the rooms spotless to keep a good business relationship with them.

Following all this was one day for analysis to create a summary to hand off. Of course a summary is not much to hand off, but the amount of qualitative data from the execution combined with the wide breathe of the research objectives made this a challenge.

And to accomplish any of the above I was also responsible for communication. This was not only communicating with the stakeholders to nail down the research objectives, updating them on progress, and handing off the analysis. I also had to communicate with my team members when I could to get some advice as this my first full project lead experience. Even though it was a solo project doesn’t mean I couldn’t have help, and I wanted to make the most out of it.

Actions:

Part 1: Prepare

One of the very first things I did was ask for advice from a senior team member who I regarded as a close friend. She was great about letting me work independently while also being really receptive to any questions I had. From her I learned a few key things that helped this project’s success:

Guidelines to questionnaire design in the games industry.
My coworker was already working on a slideshow to present to the entire team about the major do’s-and-dont’s of questionnaire design. Luckily she let me take a look at it early and let me ask some questions. This made the prepping of the questionnaire a lot easier for me as I had specific guidelines to follow.

Asking the product team for previous studies on this game.
Another tip she let me in on that up being extremely important was to ask the product team for past studies on the game that had already been conducted and reported on. This is because the product teams often use them as a reference point when looking at your analysis, and it can give more detailed insight into what they are really looking for.

Set product team’s expectations with their last minute requests.
One thing that kept happening over my 4 days of material prep was the product team coming back to me with more requests, changes, and questions they wanted me to ask the participants. Being a one man project I could pivot quickly, but still could not do everything they ask. My coworker let me know I need to be transparent with this. Before I take up a whole day making a change, I need to let them know I won’t be able to make every change they want, and also might have to localize some of the questions they sent me to ask. As well, I might have to change some things to keep research validity. Luckily the PR was really lenient and understanding with this.

Trusting myself as a researcher.
Finally, she gave me a lot of confidence in myself, which was extra important in this case. With only for 4 days for material prep and the PR on a 12 hour time difference, I simply did not have the time to wait for every decision of mine to be approved my product team my manager. Therefore confidence in my ability to make the right call and move forward with it was a necessity.

I had to communicate with the product team to accomplish all my tasks, and the above advice proved extremely useful in navigating this potential minefield. One of the first things I did was ask for any recent previous studies and corresponding reports done on the game. The product team happily gave me this, and along with it a much more clear and specific idea of what they wanted. With this in hand the product team told me that what they really wanted was to see if North American Players responded the same way players from the [REDACTED] region had.

With the clear research objectives from the product team received, I had to rescope the project. I had to marry what we had already planned for with the updated objectives, and mostly toss the original research objectives I was provided to the side. I was left with the 3 research objectives discussed early: Validating findings for the North American market, identify any major understanding barriers in the FTUE, and evaluate the usability and UI of several major screens. Although now much more clear, this was still a lot to cover. So I made sure to ask the product team what objectives took priorities over others.

With these new research objectives and corresponding priorities, I had to transform the basic structure that recruitment had already started for into something that could meet the new expectations. This was going to be a very iterative process, and things would have to change as things fleshed out. Actually creating the research material and getting feedback from the product team would show what would and wouldn’t work. With this in mind I started with a basic skeleton that would both provide somewhere concrete to start, but also be quick and easy to change.
I started with a simple protocol plan of:

1. Basic intro
2. Long gameplay section
3. Short Survey
4. Player Break
5. In-depth focus group

With this basic protocol idea in mind, I was able to start really fleshing out the material. This included the focus group guide, the observation sheet, survey questionnaire, and marketing print outs. The basic structure of the protocol gave me estimated times for each to aim for, which influenced the amount of questions and probes included.

As mentioned this was all a very iterative process. A big part was getting feedback and updates from the product team. I would send in my work at the end of the day and ask for feedback. I would get feedback and updates overnight, and wake up early enough so I could respond while they were still awake. Again my mentor’s advice proved very useful here, as I had to say I couldn’t change some things already decided upon, and had to modify some of what the product team sent me.

Through all of this I kept modifying the protocol to keep up with the product team requests and any snags or issues found while fleshing out the material. I think I did a good job of balancing the product team’s last minute requests, research validity, and my own sanity to end up with this basic protocol:

1. Basic intro
2. Short pre-gameplay survey
3. Short pre-gameplay focus group
4. Medium length gameplay mainly covering FTUE
5. Player Break
6. Short post-gameplay survey
7. In-depth post-gameplay focus group

The survey and focus group prior to the gameplay mostly covered the validation of certain responses of the North American players compared to players from another region. After the gameplay, we focus on the FTUE, usability, and UI. The surveys covered the more numerical aspects of both of these (satisfaction rating, etc.) while the focus groups went more in depth and sometimes piggy-backed off topics covered in the surveys.

Now, all that was left to do was fly out to LA and actually execute the playtest.

Part 2: Execution

Prep really is half the work when it comes to a playtest, and that doesn’t end with writing the protocol and focus group questions. I came with materials ready to print out, the game files ready to transfer, player list, extra cables and chargers for the rental devices, backup pens, all that stuff. I also made sure to dress like a researcher, with nice business casual clothes for each playtest day, a fresh haircut, and even my own clipboard. Being the only researcher there, I didn’t have the presence of any coworkers to buy me an air of legitimacy in the eyes of the players. This made appearing like a researcher at the player’s first glance extra important.

Luckily, my manager was able to negotiate for me to have 2 days of teach setup. Again, being the only researcher on site, this was absolutely necessary. Especially because there was an issue with one gaming device, and I had to make some calls to the rental vendor to get a replacement the following day. I can’t go into too much detail because of my NDA, but this was an in-depth setup that could have a lot of possible failure points. Again being the only researcher, I couldn’t have anyone step in for me while I fixed something. I thought ahead and planned redundancies and used both days implementing them. I am happy to say the execution went off without a hitch, and my built-in redundancies came in handy!

During the tech setup and playtest days, I made sure to keep tabs with the product team and regularly update them. This was first to just keep them informed that things where going well, players were arriving, and the research was being accomplished. I also made sure to send them some basic insights and trends as they were appearing, which proved very useful when the product team saw some they were particularly interested in.

After the first session on the first day, I updated the product team with some basic trends from it. They were very interested by one in particular, where one of the main features of the game could not be clearly identified by the players when they were asked how they felt about it. So I talked with the product team and we decided I could focus in on this more, and see what players thought this feature was referring to and what was confusing them about it. Then after that, ask them what they thought about it. This extra probe proved extremely useful in untangling one particularly complicated usability issue during analysis. It did take up more time, but I was able to take less time in discussing the lower priority questions. I still got all players out on time!

Part 3: Analysis

After flying back from LA, I only had one day to do the analysis. The end product was just a summary, but the research objectives covered a large variety of data, and there was a lot to go through. So I started with qualitative coding, and has to smartly choose what to fully analyze. I looked at what was the top priority for the product team and started there. After that, I made sure to do a quick coding on the biggest points for the less important research objectives.

After the qualitative coding was done, I wanted to put some numbers together. This was in no way a quantitive study as we had only 10 players, but I knew the product team loved to see graphs. I put these together from the survey responses, and averaged out the player responses to overall satisfaction and some material they were shown. I also plotted a graph to show the basic demographics, like main game platform, of the 10 players.

Finally I had to put all of this analysis into a summary to hand off. Being a summary, it can only cover so much, and I did not have time to waste on analysis that couldn’t be handed over. So I picked what to do carefully based on what objectives had the most valuable meat on their bones, and on what the product team cared the most about. Looking back on the daily summaries I sent to the product team was very useful for this, and I combined my qualitative coding and some basic graphs into a 2 page summary to send over to the product team.

Result:

After everything was submitted, I did not have much time to see my summary and effort affect anything as layoffs affecting my team were announced the day of my summary handoff. However, I do know for a fact it was a great learning experience that both my skills and confidence as a researcher. I tackled a big project, met all the goals, and learned a lot from my coworkers in the process.

What response I did get from management was positive! My direct manager said I did a great job, and the product team very thankful for my summary, insights, and ability to think my feet and make sure the playtest answered their questions in such a short time frame.

I think the most impactful result however was the building of a bridge to another UX team at NetEase. There was another North American UX team, more focused on design than research, that we had not really worked with before. However, they were working on this game at the same time as me. Because of this we communicated and I made sure to gather extra insights here and there to help them with their project. After the summary handoff I met with one of the members on a call. After this I was not satisfied with all my hard work ending only in a summary, so over the next couple days I dove deeper into the qualitative coding to pull out some major and minor insights that aligned with the NA UX’s goals. This lead to the first real connection with this team, and even though I am not longer at NetEase to see that connection blossom, I hope it has turned out well.

Reflection:

As mentioned, this was a fantastic learning experience. I learned a lot from my coworker as she taught me all this in’s and out’s of working directly with the product team, which I had little chance to do before then. I also learned a lot more about questionnaire design from her, and it has made me much more confident to use that skill in future projects. Speaking of confidence, this was a huge boost in that. Until this point I had mainly worked on protocols after being provided specific survey and interview guidelines, but had not really had a chance to design my own in the corporate world. But this time I did all of it, all the questions, and timings and structure, and then even the tech setup and execution. And it all went off without a hitch! This can be rare at NetEase playtests as there are so many people and moving parts to come together, that something usually went wrong that we had to work around, But this went extremely smoothly. I got all the objectives answered, got all the players out on time, and pleased the product team. Part of this was definitely that it was a solo project, which let me be very detail orientated and made everything was connected and working in order. It also helped me focus in on learning experiences of questionnaire design, observation sheet design, focus group design, and stakeholder communication. I wasn’t distracted by much else, and even though all this work was stressful and daunting, it felt like an entire semester of college in 2 weeks of work. However, it being a solo project had some downsides as a learning experience. I did not get the chance to try out skills in work delegation to other coworkers and see what it was like to lead a project with multiple people supporting you, which comes with it’s own challenges and pitfalls. I wish I was able to learn more of these skills, but overall I’m glad it was solo as you can only learn so much at a time.