Methods for Improving Speaker Testing

Although this speaker comparison produced interesting results, I believe we all learned a lot from this process regarding how to improve our testing methodology for producing more consistent testing results.  I have listed my observations below in efforts to improve our methodology for our next annual Faceoff.

Find Less Lossy Grille Cloth Covering
I believe one of the largest biasing factors in this test was the extremely lossy grille cloth we utilized to cover the speakers which seemed to shift the listener preference to the brighter speakers.  Since it's virtually impossible to stretch the cloth more taut, we either need to blindfold our listeners or find less lossy material to use in future comparisons.   Ian Colquhoun from Axiom Audio commented that perhaps the grille cloth we used for our testing was only acoustically transparent when stretched over a grille cover.  Axiom uses specific cloth that is less lossy for these type of tests and will be happy to supply us samples for our next blind faceoff comparison testing.

Use Better Randomization of Testing
Rather than assigning each speaker a letter designation, I will follow in Tom Andry's foot steps where he only assigned an "A" and "B" designation for the speakers and simply kept track which was which during the testing to always keep the listeners guessing.

A                      B
Klipsch             JBL
JBL                   EMP
EMP                 Axiom
Axiom               Klipsch
JBL                   Axiom
Klipsch             EMP 

I believe towards the end of the tests, people already had preconceived ideas of what each speaker would sound like based on simply calling out its letter designation.  Some of the listeners reported it was too confusing remembering four letter designations when taking notes.

Reduce Number of Listeners per Session and Increase Number of Listening Sessions
For two-channel listening, it makes little sense to sit any particular listener off axis from the stereo sweet spot.  I believe we partly confirmed the research Dr. Sean Olive conducted regarding listening position can have a more dramatic impact on listener preference than the actual audible differences in the loudspeakers themselves.  Our data showed that people sitting in the back row preferred the speakers with the most bass and high frequency bias.  Their preferences also seemed to have the most variability in ratings from test to test.  It's inconclusive however if these preferences were influenced by them also being the least experienced listeners or simply changing their opinion of the speaker under test when compared to a different speaker each time.  More study is needed to make an accurate correlation.   Perhaps in our next round of testing we will trick the listeners into comparing the same speaker during our A/B testing as a control to determine the variability in their preferences. 

One thing that’s important to realize is in order to have reasonably accurate results a greater number of tests and sampling of listeners is needed in order to produce significantly meaningful results (>95% confidence).  Unfortunately it is often costly and impractical to conduct blind tests in this fashion like it's typically done in the medical field.  Sorry we are just an A/V publication, NOT a pharmaceutical company with billions of dollars in financial backing.  This doesn't, however, mean we can't improve our methodology by restricting our listening sessions to the best seats in the room and spanning the test out for multiple days to include more listeners.  If there are any volunteers to host such tests, drop me an email :)

Use the Most appropriate Listening Venue
An important factor to how a loudspeaker performs is the type and size of the room it is placed in.  My primary theater room is quite large (almost 6,000ft^3).  It's also acoustically treated with an asymmetrical right sidewall.  Small speakers have a hard time producing high SPLs in this room as they tend to get drowned out in such a large listening venue.  I suspect if we used my Family room which is much more lively thanks to its wood floors, glass door and high ceilings, listener preferences would have been much different, perhaps favoring the more laid back speakers in the high frequencies.  It would have be interesting to conduct these listening tests in two different rooms to determine how those results change based solely on room acoustics.   Future comparisons may be conducted in two different listening venues to determine how much the room influences listener preference.

Better train inexperienced listeners
The variability both in comments and ratings was highest among the least experienced listeners.  I was a bit surprised by all of the negative comments for all of the speakers.  Perhaps I didn't do a good job explaining to the listeners what to listen for and to put more focus on the positive attributes they were hearing.  

I also observed that the trained listeners exhibited preference towards the more tonally neutral speakers while the untrained listeners appeared to have higher preferences for the speakers with the most bass and treble.  Perhaps it would be beneficial to better prepare the inexperienced listeners on what to listen for when evaluating loudspeaker performance.  On the flip side it does provide insightful data to see what untrained listeners prefer when walking into a showroom floor trying to select the best sounding speaker. 

A Note About Listening Tests

When choosing a loudspeaker it's important to first decide on what purpose it will serve to you as well as what type of room you will be listening to it in - and at what SPL levels.  All of the blind test results in the world won't determine how a particular loudspeaker will sound in your listening environment reproducing the music you like to listen too.  It's also important to NOT judge a loudspeaker's performance solely on one or two short listening sessions.  Casual buyers tend to gravitate towards the most colored sounding speaker on a busy showroom floor only to find weeks later they suffer from buyer's remorse caused by listening fatigue.  When judging a loudspeaker performance in your own home, ask yourself "does this sound like an actual reproduction or just a colored representation?"