North American Gamers Focus Groups

#FF644E
#5E5E5E

---

Situation:

I was tasked with the facilitation of 4 focus groups, just the execution really. The questions and power point are written for me, and I’m just taking notes and doing a summary to pass onto someone else for full analysis. Sounds easy right? It was not, in fact just the moderation of these focus groups would be one the biggest challenges of my communication and interview skills I have ever faced.

Research Objectives:

The below research objectives below are heavily redacted to comply with NDA:

The focus group were setup into 4 sessions, with 4-5 participants each. Being a small sample size and a focus group format where other participants can influence discussion, this was heavily qualitative. There were 3 main categories of participants, which I will refer to group A, B, and C. The First focus group was mostly group A, second mostly B, third was comprised of category C, and the final was comprised of a mix of all 3 groups. This allowed me to compare and contrast what each group thought about the topic at hand without fear of being judged by gamers from a different group, and allow for more interesting inter-participant discussion in the final group.

Challenges:

Even though my work was limited to execution and basic analysis, the focus group protocol I was given was an uphill battle. The focus groups took place from 8pm EST to 12 AM EST each night so some of the China based stakeholders could listen in and observe live. You might see 2 issues with this.

Firstly, this was extremely late at night for me (who was located in EST) and for participants who were tuning in from all across the US. Even for someone on the west coast it would be ending at 9pm. Secondly, this is a 4 hour long focus group. That is extremely long time to ask for any kind of testing, especially a session that is not a playtest and involves no gameplay to keep the player engaged, survey to give them a speaking break, or lunch break to just keep them alive.

Additionally, the PPTX provided to me to show the participants was bare. Just black text with the questions verbatim on a plain white background. No visuals, no videos, no graphs, nothing besides blocks of text to show participants for 4 hours. Needless to say, I couldn’t rely on this pptx to keep participants engaged. I even learned later that the pptx would sometimes do less to keep the participants engaged than showing nothing at all.

Finally, even though the focus groups were 4 hours long, I still had nowhere near enough time to touch on all the questions and probes provided. It is understandable that stakeholders wants to fit in as many questions as possible to get as much value out of expensive research sessions as possible, and that was nothing I wasn’t used to. But the scale and scope of the focus group guide provided to me was immense in a way I had never seen before, and I knew I would have to really prioritize the most important topics.

Tasks:

I knew I had some basic analysis to do after the focus group sessions, but the challenges spelled out above were in the front of my mind. I knew I had to figure out had to handle the execution before anything else. To make sure the notes and data I gathered were detailed, in-line with stakeholder expectations, and held research validity I had to do a few key things:

With the sheer amount of questions being handed to me, I knew I could not cover them all. Additionally, many were unclear or worded strangely. To combat both of these issues, I would have to ask the product team about them. I had quickly learned at NetEase going right to the source was the best way to clear these things up. Luckily I was able to ask a lot of questions during a kick-off meeting setup with the product team, and I maximized this opportunity.

Last, and certainly not least, I would have to keep the participants engaged throughout a 4 hour late night session with nothing but words. This was important because if they’re not engaged they’re not gonna be detailed for honest, and will just follow the past of least resistance (AKA telling me what they think I want to hear.) They were also discussing with other people in the zoom call. I had to make sure they felt comfortable being honest with me and the other members, and weren’t afraid to disagree with one another. This would be the hardest task of all considering the challenges, and is the one I prioritized with my following actions.

Actions:

With how many questions and probes there were to cover, I knew I couldn’t cover them all. I expressed this to the stakeholders and asked what were the most important topcis I should cover. This proved to be very useful as they provided me an updated guide with color coding to show which were the most important questions and probes, which differed slightly between participant categories.

In the same meeting from above I also dived deeper into the research objectives and went over some specific wording with the product researcher. This gave me more knowledge about what their objectives were so I could more easily probe into certain discussion points on the fly. It also let me reword some of the more important questions with the PR’s approval to make them more understandable for the North American audience. This included fixing some translation errors, reducing leading questions, and just better tailoring the questions to a North American gamer’s vocabulary.

Now that I had this deeper understanding it was time for the long late night focus groups, and I had planned out some specific strategies just for this challenge.

Now the focus groups were starting, 4 hours long and late into the night. I had a Mocha from McDonald’s to keep me awake, opened Zoom, and performed the following actions:

Of course we all have a list of disclaimers and reminders for participants at the start of a focus group or interview, and this was no different. However, because of the format and length, I made some special editions. First I reminded the participants that disagreeing with each other was acceptable and even encouraged at the start. Furthermore, over the 4 hour time span, I would remind them to be honest, feel free to disagree, and speak on their own individual opinion multiple times. I usually slotted this in when there was an example of a participant doing this well. This made the reminder both feel more natural and have a tangible example attached to it, which shoudl have made it stick more in the participant’s minds.

One of the first things I did was find out everyone time zones. I did this by including it as a part of mine and everyone else’s intros. This accomplished 2 things: First it established that I was on EST and the focus group was ending at midnight for me. This meant that the participants knew no one was staying up later than the host, and meant they wouldn’t be disgruntled seeing someone fresh and lively at 8PM while it was 11PM for them. Secondly it let me note down how late it was for each of the participants and I could keep an eye on the EST and CST participants’ fatigue first. I kept this in mind as we neared the end of the focus group, going a little extra to keep the late night particpants engaged.

One thing I realized in the second half of my first session was my mistake of sharing the PPTX screen of the exact question I was asking to my participants while it was being dicussed. In the second half, where fatigue was starting to set in, I noticed my participants would not listen to each other or even think of their answer ahead of time while I did this. Instead, when I asked them that question they would be taken by surprise because they were spaced out and not listening. They would then read the question from the PPYX being shown and then take a minute to think of their answer. This took up valuable minutes in this already packed focus group. It also meant they did not listen to each other, so would not bounce of each other discussion points’. To fix this I still showed the PPTX, but kept it on the title slide of the general topic at hand instead of the exact question. This meant they had something to remind them of the general topic, but also kept them aware of each other and not relying on the PPTX. This led to more alert particpants, deeper discussions between them, and a more time effecient focus group.

Having my webcam on was not a requirement from the product team, which is honestly appreciated. Webcam being on for the researchers wasn’t the norm at NetEase, and even if it was, the late night timing of these focus groups meant the expectation was not there. However, I thought it was one of the most important parts to make this successful and keep everyone engaged. Not only did it give the participants a visual to grab onto, but I think seeing the effort I was putting into this focus group made them more receptive to me as a host. Even if I was keeping them there until midnight. They were able to see the sun go down on me through my apartment windows and know I was in the same camp as them, not just a faceless researcher asking them a bunch of nosey questions. Some even volunteered to turn on their webcam after seeing I had mine on! Also being able to smile, laugh, and nod along with them as they spoke also helped build rapport.

Keeping with the theme of showing my effort to the participants, I made sure to dress business casual for the focus groups. If I am going to be on camera might as well look nice! This had the usual effect of making me look more like an authority figure and therefore someone to listen to, but helped me ingratiate myself to the participants over the 4 hours. With these remote focus groups you only get out what you put in, and every little bit helped. If I had just worn a T-shirt I woudl have been more comfy and less sweaty, but I would not have gained the same level of respect from the participants.

Result:

The results from these focus groups were positive! All 4 focus groups were completed within the allotted time frame, which was not the norm at NetEase. We even did some follow up interviews with returning individual participants from deeper questions that were spurred from the focus groups. (Thankfully these were only 30 minutes long.) The product team passed on positive words about me to my manager, saying I did a “great job as host”. I also must have done a good job prioritizing what questions to ask because I was told I “understood the purpose of the research well.” This was really important, as it was also only a few months into my job at NetEase. Afterwards I was see as a kind of “go to guy” for challenging interviews or focus group were a nuanced host was needed.

Additionally, through all 16 hours of focus groups and a few more from follow up interviews, I was able to gather a lot of notes, quotes and insights. One big upside to this format was that it allowed some exploration and noting down of “unspoken” attitudes that North American gamers held. I cannot disclose the details here but having these insights in writing with qualitative evidence helped verify some of the cultural differences between NA and Chinese gamers. This could help further reports and studies into the NA Market, as you could point to specifci evidence to back up your claims that these attitudes and differences existed.

Reflection:

This project felt like a big win for me. The product team presented me with quite the challenge and I conquered itn even though it wasn’t perfect. The long hours were exhausting for me and the participants and despite my best efforts I’m sure it had some effect on their answers towards the tail end of their focus groups. I also did not understand all the product team’s expectations for this focus group. Afterwards learned that the product team expected the notes from the focus group to be exactly verbatim, which was news to me. But besides that I felt like I tackled this challenge well and came out of it with both improved hosting skills, improved communication skills, and a higher confidence in those skills! Thankfully they never asked me to do such long sessions again, which was great for me and the very concept of research validity’s sake.