Approximately 23 million Americans have adopted a pet since the beginning of the COVID-19 pandemic. The surge created a wave of demand for Chewy. Though most people know Chewy through their large ecommerce store, the pet care giant also offers critical services like virtual veterinary care, pet pharmacy, PetMD, and their adoption finder. But no matter how pet parents connect with Chewy, they know they’re protected by Chewy’s famous unconditional satisfaction guaranteed policy. This policy forms the backbone of Chewy’s mission: to be the most trusted online destination for pet parents everywhere.
For Chewy Director of Engineering Russell Schwager, that trust extends to the quality of his organization’s digital properties. Ensuring that Chewy customers have a quality experience on every Chewy website is essential, but doing so at the scale needed post-pandemic posed significant challenges. Russell shared how his team answered this challenge and created a sustainable quality strategy with mabl’s test automation platform at mabl Experience.
How Chewy Uses Mabl to Make Testing Scalable, Efficient
At Chewy’s scale, an effective quality engineering strategy must be efficient. Russell’s team needed to move quickly without sacrificing quality. Before mabl, they relied on a software testing strategy that focused on manual testing, which wasn’t able to keep up with the surge in demand. They knew they needed a testing strategy that allowed them to re-engineer their entire process for scalable, sustainable quality.
Manual Testing, Multiple Sites Created Testing Challenges
Russell had a single quality engineer dividing her time across four teams and manually testing functionality across three different sites for five different browsers on desktop and mobile. “As functionality completed, our quality engineer would go in, validate it, and then move on to the next piece. She did an amazing job, but it was easy to see we were putting her in a challenging position. And our aggressive release cadence — weekly for one site and daily for the others — also created a challenge,” Russell said. “It looked to me like we were not going to be able to expand out our testing without hiring more quality engineers.”
Different code bases and rotating release managers also created additional challenges. “Like most startups that have become successful, we started as a monolith and are moving to single-page applications across the Chewy.com ecosystem — and so the code that we would have to test would multiply across different code bases. In addition, the team rotated release managers — and the release manager wasn’t focused on quality, they were simply focused on wrangling 25 different people to sign off on the release, which was a nightmare. Our team looked at this and realized it wasn’t scalable — we needed to do better.”
A Solution with Mobile Web Functionality, Analytics Integrations, and Strong Support
Russell and his team knew they wanted a solution with solid analytics that supported the company’s mobile-first approach. After testing multiple products, Chewy chose mabl due to its mobile web functionality, analytics integrations and high level of support.
“Mabl came out ahead as the clear winner,” Russell said. “The approach we took was to roll it out to a single team and identify the base functionality that we wanted to test. The SEO team was perfect, as we had test cases already written associated with our weekly build. We built a test suite in both desktop and mobile and ran it for a while to make sure that we were happy with how it was running.”
The quality team also focused on shifting the team culture to embrace a more agile approach to development. “Instead of relying on the quality engineer to do all of the testing, we’ve been trying to get the engineering team to write tests and own that instead of throwing it over the fence. Once we felt successful with one team, we moved on to the second and the third. We count this as a success,” Russell said.
Mabl Cuts Testing from Three Engineering Days to One Hour
Chewy has achieved impressive results since it began automating its software testing with mabl. “We became much more efficient, and we’re covering more functionality,” Russell explained. “The amount of testing we needed to do our weekly release was about two to three engineering days. With mabl, it’s down to about an hour per week to run that base core functionality. We’ve recouped about half a week of engineering time to work on other things, meaning we’re covering more functionality on a weekly basis. Our quality engineer is working on other things in other programs that she was unable to do.”
In addition, Russell and his team have successfully changed Chewy’s internal culture. “Our teams have become self-sufficient in that they don’t need a day-to-day quality engineer — they’re writing tests, they understand the value of it. And they’re thinking more end-to-end in terms of their responsibility as engineers — and some of them have taken to the tool and are looking for ways to integrate it further,” Russell noted.
In terms of lessons learned, Russell offered the following advice to other teams looking to implement automated testing: “One of the things we’ve learned is, interest is high, but adoption is inherently slower. Teams are set in their ways, and there’s an emotional impact to change. Some of the engineers were a little bit resistant to getting up and running on this. It’s important to think about how you roll things out emotionally. People’s roles do change, and changing the way things have always operated takes some effort. Continue to think big, and take a long-term view on it.”
Watch Russell's mabl Experience 2021 presentation
Is your organization growing quickly? Sign up for mabl’s 14-day free trial to see how intelligent test automation can help your team scale software testing.
