How We Test Robot Vacuums
See How We Test Other Products:
As they come down in price and get better at cleaning, more consumers are interested in buying a robot vacuum cleaner.
Most of the products we review have industry-standard testing protocols. Since robot vacuums are an emerging market, however, there are no standardized protocols for testing their performance—so we created our own.
The testing team at Reviewed interviewed consumers and engineers, and visited some of the world’s best robot vacuum design centers to develop a series of objective tests and subjective assessments. This way, we can provide readers with a well-rounded report on each robot vacuum. Here’s how it works.
Prior to testing
Once we receive the vacuum, we unbox it and take note of all of the accessories in the box. Typical accessories include a filter, the charging dock, and components of a virtual fence, among others. Once the robot vacuum is assembled, we measure its weight in pounds. We weigh the vacuum and the dirt receptacle prior to testing because it will gain weight over time, as dirt particles accumulate in the vacuum (both within and outside of the dirt receptacle). Lastly, before the tests begin, we ensure that each robot vacuum is fully charged.
The testing arena
We designed and built an obstacle course that closely replicates the challenges a robot vacuum would face in the average home. The course consists of a laminate floor, carpet squares, wooden risers, a room threshold, wooden shelves, wooden stands, a gap, and a mirror. While it may not look like any room in your home, it puts a robot vacuum through a series of tests that are representative of what a real living space has to offer.
Laminate floor and carpet squares
The testing platform is covered in a laminate floor that’s similar to the kind found in kitchens and bathrooms worldwide. Three carpet squares lie on top of the laminate: a black high pile carpet, a white patterned medium pile carpet, and a dark blue doormat that represents low pile carpet.
The wooden risers represent common household obstacles that sit close enough to the ground that the robot vacuum might try to climb them, with heights of 2/3”, 2”, and 4/3”. These risers help our reviewers to determine how well the robot vacuum will clean on an uneven surface or with flat, low-lying obstacles such as board games or planks of wood.
We installed one wooden room threshold in the middle of our testing arena to ensure that the robot vacuum can successfully transition from one room to another, as it would have to in the average home.
The wooden shelves in the obstacle course are similar to the wooden risers mentioned earlier, except that these shelves are located 2”, 4”, and 6” off the ground. An additional shelf in the shape of a wedge has a second wooden component that meets the ground at an angle of 45°. Observing how a robot vacuum behaves around these shelves tells our reviewers how well the vacuum can clean beneath the toekick of cabinetry, underneath couches, and near other elements of a living space that jut out from the wall or are slightly elevated off the ground.
The three wooden stands—with widths of 6”, 12”, and 18”—have four legs each, and are meant to test the robot vacuums’ ability to clean in the presence of (and beneath) furniture like tables and chairs. Additionally, one wooden stand is equipped with a sensor that records the force of the impact if/when the robot vacuum collides with it. This can tell us whether the force of a robot vacuum could damage delicate furniture or baseboards.
Our testing arena is almost entirely enclosed. A gap in the wall is intentionally left open to test the vacuum’s ability to sense and deal with abrupt changes in floor height, such as at the edge of a deck or at the top of a stairway. An inability to recognize such drop-offs could result in serious damage. We deliberately did not install a stairway, as some robot vacuums can sense a nearby stair but cannot sense a larger drop-off.
We’ve propped a full-length mirror against a wall in our testing arena, because some robot vacuums get confused when encountering their own reflection. Why? One theory holds that it’s because mirrors have the potential to widely scatter a vacuum’s sensing beam, which can bring it to an indecisive halt. A second school of thought holds that the sensing beam could be reflected unchanged, making the vacuum believe it’s the sensing beam from another, different vacuum and leading it to pause until the “other” vacuum trundles away. Regardless of which theory is correct, we make sure each vacuum we test doesn’t lose its mind when it sees itself.
The dirt on the dirt
The two types of detritus that the robot vacuum cleaner encounters in our tests are ground cork (with a density of 84 g/L), and pet hair. The cork is ground into pieces with a diameter of 2-3 mm. Reviewers with feline and canine companions generously donate the pet hair we use in our robot vacuum tests. These two types of messes are representative of the most common use cases for robot vacuum cleaners: cleaning up pet hair and larger debris. Grittier messes, like dust and pollen, are more appropriately handled by full-size or cordless vacuum cleaners.
Each robot vacuum takes three passes at the obstacle course. For the first two runs, we distribute 0.5 ounces of ground cork along or beneath the surface of the aforementioned obstacles. For the third run, we put 1 cubic inch of pet hair in each of the four corners of the test arena, including a pet hair patch on both the high- and medium-pile carpets.
Once the robot vacuum is fully charged, we move the docking/charging station into our testing arena so that each vacuum starts and ends in the same location. In the vacuum’s menu, we select the cleaning setting recommended in the user manual and let the robot vacuum get to work. In between runs, the vacuums charge up at their docking stations, and we completely clean the obstacle course of cork and pet hair with a cordless vacuum to prevent dirt contamination from one run to the next.
After the robot vacuum finishes its cleaning run, we judge the thoroughness of cleaning in two ways.
First, we weigh the amount of cork collected in the dirt receptacle—the higher the weight, the better the score. The pet hair score is much simpler: we just count how many tufts of fur the robot vacuum completely removed from an obstacle, how many it blew into other parts of the test area, and how many it left behind.
After that, we assess the cork or pet hair left behind at each obstacle. Typically, this involves measuring the width of the cleaned area within the boundaries of each obstacle, relative to the edge of the carpet square or the edge of the wooden risers. Larger cleaned swaths produce a higher score.
The ability to surmount or circumvent the wooden threshold is a pass/fail test. The number of vacuum attachments (such as extra spinning brushes) and the vacuum’s weight also contribute to the overall score, although we give more weight to more useful attachments. Overall, more attachments and a lower weight tend to achieve higher scores.
Because repeated collisions can be very hard on furniture, we also incorporate the impact force on the wooden stand into our objective scoring. The larger the force, the lower the score. Because some consumers use robot vacuums to clean multiple rooms in a single cleaning run, we measure the dirt receptacle capacity by filling it to the maximum recommended load (sometimes denoted by a “fill line”) with the ground cork. A higher capacity gets a higher score.
Lastly, because consumers want vacuums that work quickly and unobtrusively, we also score them on how long it takes to complete a cleaning run. We apply penalties for any events that deviate from a regular run, such as manual resets, getting stuck on obstacles, hesitating or delaying indefinitely at the mirror obstacle, or falling out of the testing arena through the gap in the wall.
Half the battle with new, connected appliances is figuring out how they work, and how to get them up and running. Accordingly, we rate each vacuum for ease of use when it comes to the dirt receptacle and filters. If those parts are easy to find, remove, and replace, the vacuum will score well. Another factor is the controls: Are they easy to operate, with or without guidance from the manual? Are they intuitive and responsive?
Robot vacuums are neat (and sometimes stylish) little gadgets, but we don’t always want to leave them out when they’re not in use. We also rate the storability of the robot vacuum by posing and answering these questions: Is it easy to store the robot vacuum and all of its accessories into one location? Can extra parts be stored on the vacuum itself?
We spend a lot of time testing these products so we can confidently tell readers which robot vacuums are the best for their needs. Want to know more? Check out our roundup of the best robot vacuums, or browse our full list of reviews.