Case studies

How To Make Peer Reviews Work, An Example From Automattic

Automattic, the makers of WordPress.com and WooCommerce, continuously deliver CSAT of 95% and more. Yet, 56.6% of their customer conversations fall under the “Room for Improvement” category in internal assessments.

With a sharp focus on the quality of their interactions, Automattic conducts regular conversation reviews with Qualitista. Grigorij Urasov, Happiness Engineer at Automattic, shares insight into the benefits of doing internal assessments in the form of peer reviews, and how to come to terms with giving negative feedback to one’s teammates.

Here’s Grigorij’s take on conversation reviews in his own words.

Advantages of peer reviews

Team Orion [ed note: the internal name for Grigorij’s team] has been using Qualitista for peer reviews for over a year now. Every week, each member of my team reviews at least 10 interactions of another agent - whom we call Happiness Engineers (HE). We usually choose a new HE for reviews each week.

Since peer reviews can feel intimidating, mostly due to the feeling of being graded, I'd like to point out how I managed to get into the right mindset, and how peer reviews have helped me personally.

When we first started out, peer reviews were mostly positive, but according to the latest Feedback Analysis report in Qualitista, 56.6% of reviewed tickets fell under "Room for Improvement". I see this as a good thing indicating that the system is working and HEs are actually learning a lot from each other.

I will not deep-dive into results of the above feedback analysis but I'd like to focus on the psychological part of this process.

The responsibility that comes with being a judge of someone else's interaction might be daunting at first. The one thing that helps me relax when giving or receiving the "Thumbs down" is the realization of the fact that without these thumbs down we might continue to make the same mistakes over and over again.

In my personal opinion, giving and receiving peer reviews is especially important for relatively new Happiness Engineers, just as important as pair ticket and chat sessions and other means of learning. This helps us get to the level of other Happiness Engineers more quickly, provide more consistent customer experience, and learn about our products.

I felt the lack of peer reviews when I starting working on the WooCommerce Mobile queue. This happened simply because there were just two of us working on that queue, meaning that in some cases, neither of us knew what the right answer was. Luckily, most of us work on big queues with a lot of peers and we can (and should) be using a peer review process to constantly improve our performance.

When reviewing tickets of seasoned Happiness Engineers, especially those of Team Ohana (they work with complicated, escalated tickets), I almost always have “Today I Learned” moments. Quite often the only thing left to do is to give 4 thumbs up and add a #great tag.

Giving negative reviews

It does sometimes happen that you have to give a thumbs down to 4 or 5 tickets (out of the 10 you’re expected to review) of a particular HE. This isn't easy because the reviewer cannot predict the reaction of the reviewee. Here's what I'm striving for when giving negative reviews: I imagine that it's my own interaction which is being reviewed and try to think if I would feel that the feedback was fair.

I'm trying to avoid phrases like "You should have.." or plain "this is wrong". Instead, I usually explain how I would have handled this interaction starting with "I usually..." or "On top of that, I'd add...". Shifting your mindset from "you are wrong, I am right" to "This is how you handle it, this is how I usually handle it, let's find the perfect answer" helps a lot.

It isn't about sugarcoating versus being straight to the point. It's more about realizing that everyone has their own approach. Our task is to level it out for the sake of our customers.

For example, some of us might be giving quick replies without going into too much detail. At the same time, another person could be providing answers that cover all the possible scenarios to all users, including the non-paying ones.

It might feel like the latter did a better job, but that's not necessarily the case. What if the time spent on that free customer could have been spent on a paying customer who needed urgent help? What if the SLA suffered because of repeated situations like this?

Once I was about to give the fifth thumbs down in a row to a fellow HE when I realized that our styles are so different that I was simply not seeing the right answers in the replies. After a closer look, I saw that the answers were there: short and to the point. After that, I changed four out of the five thumbs down to thumbs up.

It's best not to assume that you're 100% right when doing peer reviews. You'll enjoy the process a lot more.

Read more about how Automattic fosters a culture of feedback with Qualitista and Zendesk, and why Valentina Thörner, Happiness Team lead at Automattic, prefers peer reviews.

Also, check out Qualitista - the tool that Automattic uses for conversation reviews.

Get the Support Quality Blog posts via email

No spam, maybe also an occasional cat picture.