Multi-Factor Decision-Making Math Applied Example - Headphone Ratings Site

Disclaimer: I’m not an expert on philosophy. I’m just a person trying to figure things out for myself. I speak for no one but myself.

RTings.com

I have been rereading Elliot Temple's (ET) article on Multi-Factor Decision-Making Math. (I made some flash cards.) I thought it would be fun to look at an example of something that uses an approach that ET's article criticizes and try to apply some of the ideas in the article to it. Quotes from ET are from the Multi-Factor Decision-Making Math article unless otherwise specified.

RTings.com is a website that contains ratings of headphones. They seem to do a lot of thorough testing and collect a lot of data. They give ratings for various categories of headphone performance, such as "Neutral Sound", "Sports/Fitness" and so on.

These categories are themselves based on a weighted assessment of various factors. As an example, here's a weighting breakdown for the "Phone calls" category (along with scores for the Sony WH-1000XM4 Wireless headphone):

RTings also apparently uses these scores in order to come up with overall evaluations of headphones in particular categories (like "Best Wireless Earbuds Under $100"), though they don't try to make these evaluations an issue of math.

Notably, the factors considered may be weighted in more than one category, though the weighting may be different for different categories. For example, the "comfort" factor gets a 15% weighting in the Commute/Travel category, an 8% weighting in Sports/Fitness , a 26.1% weighting in Office, a 16.8% weighting in Wireless Gaming, an 18% weighting in Wired Gaming, and a 3% weighting in Phone Calls. It's not considered in the Neutral Sound category (so that's effectively a 0% weighting).

Here's an example of RTings.com's description of a factor, Comfort:

Comfort relates to how pleasant and enjoyable headphones are to wear over an extended period of time. There are a couple of different types of headphones: over-ears and on-ears have frames that sit on top of the head and earcups that rest around or on the ears, while earbuds and in-ears are inserted into the ear to a varying degree of depth and sometimes have neckbands to keep them in place. Regardless of the type, comfortable headphones shouldn’t cause stress, fatigue, or pain, even after prolonged periods of use.
Comfort is subjective and tends to vary between individuals; however, there are certain design elements that most people tend to keep an eye out for. Weight, padding, headband tension, and frame tightness tend to influence how comfortable over-ear or on-ear headphones are, especially over long listening sessions. For earbuds and in-ears, how soft and flexible the earbuds are, how deep they protrude into the ear canal, and how much pressure they exert on the ear generally impact comfort as well.
We use a mixture of objective measurements and subjective tests in order to test comfort. We measure weight and clamping force objectively to give us data on how heavy and tight they are and then test the headphones on various people with different characteristics to estimate how pleasant the headphones might be to wear for a while. The sum of these experiences is taken into consideration and a comfort score is subjectively determined.

I note that they're not very clear about how precisely the different elements are taken into account when arriving at the subjective determination of comfort score.

For some factors, like Bass Frequency (a component of the Neutral Sound category), the factor itself has weighted subfactors:

Based on RTings.com's own description of the "Bass Std. Err." subfactor, it seems like that factor is based on objective measurements, though even with that factor RTings notes that "a perceptual weighting filter is applied prior to the Std. Err. calculation since humans are less sensitive to accuracy in the sub-bass region." So that sounds like they are making judgments about which values are more or less important to humans in a certain range (that's not a criticism, just an observation).

Applying Multi-Factor Decision-Making Math Ideas

Weightings

One thing that really stuck out to me was the different weightings of factors. I think you can see the intuition behind at least some the weightings. Like, consider the "Comfort" factor. It gets a 26.1% weighting in the Office category, and an 8% weighting in Sports/Fitness. I think the idea there is something like: in an office, you're sitting around all day, so you need something that you can wear comfortably all day. When doing sports/fitness, you're only doing that for a limited amount of time, so while comfort matters, it matters less in that context. But why 26.1% and 8% in particular? These are arbitrary weightings. ET:

Any method involving arbitrary choices (like what unit conversions or weights to make up) runs into a major problem: You have no good way to make an arbitrary choice unless you have pre-existing knowledge of what a good answer is.

RTings is trying to convert the comfort factor into different categories like "Office" or "Sports/Fitness" and trying to do the same with other factors as well. ET criticizes this sort of approach:

They multiply by weighting factors that get results they think are reasonable. But that isn’t actually a way of making decisions. How do they know what’s reasonable? They must be using their intuition, common sense or something else other than weighted factor summing. So the weighted factor summing method doesn’t work as a self-contained solution to decision making. It relies on pre-existing opinions reached some other way.

RTings frames their method as very mathematical, but the basic approach is relying a lot on intuitive judgment calls about things like weightings and whatnot. So it's much less mathematical than it seems. It's closer to making intuitive judgment calls than it is presented as. (They also do a bunch of actual rigorous testing and give some useful information, so this is not an attempt to attack them. They put a lot of work and effort in. But there are philosophical issues with their approach.)

Headphone Comfort Factors

What are the dimensions that RTings.com is taking into account when coming up with its evaluations? ET says "[a] dimension is a type, kind or category of factor, such as length, weight, time, cuteness or color." Comfort makes intuitive sense as a category. I have had comfort issues with both earbuds and over/on-the-ear headphones.

But let's look at the details of RTings.com's description again:

Weight, padding, headband tension, and frame tightness tend to influence how comfortable over-ear or on-ear headphones are, especially over long listening sessions. For earbuds and in-ears, how soft and flexible the earbuds are, how deep they protrude into the ear canal, and how much pressure they exert on the ear generally impact comfort as well.

(emphasis added).

Note that both over-ear/on-ear headphones and earbuds get the same "comfort" score. But the actual specific issues that may affect the perceived comfort are pretty different.

I'm particularly interested in this point because I've had a lot of experience trying out different kinds of headphones. In my experience, even the most comfortable over-the-ear or on-the-ear headphones produce at least mild discomfort after a relatively short amount of time. (I seem to have sensitive ears.) OTOH, there are earbuds (such as the Apple Airpods Pro) that I can wear indefinitely without comfort issues. If I were to put the two types of headphones on the same 10-point Comfort scale, then any over-the-ear or on-the-ear headphones would max out at around a 5, whereas there are some earbuds that can be a 9 or 10. The RTings.com ratings actually give the AirPods Pro a lower comfort score than some over-ear headphones I've tried.

Are earbud comfort and over-ear/on-ear headphone comfort different dimensions? On the one hand, there is a way of describing the comfort issue in a general way that would seem to apply to both: headphone comfort is about the extent to which you can wear the headphones for an extended period of time without experiencing discomfort or pain. On the other hand, the specific factors real-world factors that you take into consideration when assessing headphone comfort seem dramatically different based on the type of headphones. I know that in my own case, essentially any headphone pressure on the ears ("clamping force"), is a problem, which basically rules out headphones that aren't earbuds.

Maybe instead of thinking about this issue in terms of dimensions, there's a way to think about it in terms of factors. E.g. "clamping force" is definitely a factor in headphone comfort. In putting earbuds (which don't have clamping force by their nature) and other headphones with clamping force on the same "Comfort" scale, RTings.com is assuming there's some way to convert clamping force, or its lack, into some sort of common scale of comfort (according to what conversion factor?). But as I've already said, at least in my case, clamping force vs no clamping force is a qualitative difference in terms of comfort. I can speak loosely in terms of putting different headphone types on the same 1-10 scale, but in terms of having pain vs not having pain, the difference is binary (at least potentially – some earbuds pass the no-pain test, whereas ~all other types fail). So the issue here may be some inappropriate attempt to convert between different types of factors into a common "comfort" factor (sort of like a "goodness" factor). I'll need to think more about this.

To be continued.