Open methodology
Every day, a program reads each publication's public feeds and sitemaps and picks out new phone reviews. It identifies itself as BestSmartphoneBot, obeys every site's robots.txt, and fetches one page at a time. For each review it records the publisher's own score and any award, plus a short summary in our words of the verdict, strengths and weaknesses. The text of the review is not stored or republished; every review is linked.
Publishers use different scales, and they use them differently: some give most phones four stars out of five, others spread their scores widely. A simple average would let the most generous publisher decide the rankings. So each score is first converted to 0–100, then compared with that publisher's own history of scores. A phone that scores well above a publisher's typical score counts for more than one that matches it. While a publisher has few scores on record, this adjustment is small.
The consensus score is the average of those adjusted scores, with more recent reviews weighted more heavily (a review loses half its weight after a year). It needs at least 2 scored reviews. Publications that give picks rather than scores count towards a phone's review and award totals.
Next to every score we show how far apart the reviewers are. “Reviewers agree” means the adjusted scores sit within about five points of each other; “Reviewers split” means they are more than ten points apart, and that phone is worth reading about in more than one place.
A phone joins the rankings once at least 2 publications have reviewed it. It leaves them when it has had no new review for about eighteen months; its page stays up.
Specs — launch price, form factor, display, cameras, battery and so on — come from the spec boxes in the reviews we cite. Each review is read separately; where they differ, we show the value most of them give, and implausible values are discarded. Prices are US launch prices for the base storage model, not today's street price.
Lists such as “best phones under $500” are filters over the same consensus ranking. The camera and battery lists are different: they rank by the camera or battery scores publishers give in their reviews, because a spec like megapixels or battery capacity says little about how a phone actually performs. A list is published only once it has at least three phones in it.
Anything the program is unsure about — an unfamiliar name that might be an existing phone, a score that doesn't match the publisher's own page data, a large change in a publisher's score — is held back until a person has looked at it.