2010 $1k Floorstanding Loudspeaker Faceoff Test Methodology

Since the days when I founded Audioholics.com in 1999, we've  conducted  annual speaker Faceoffs.  These Faceoffs ranged from bookshelf to floorstanding speakers of various designs and budgets.  Over the years we managed to beef up the brand variety while also increase the panel of listeners and refine our testing process.  Last year we conducted our first blind listening test headed up by Tom Andry.  The results were both enlightening and intriguing to say the least. 

I wanted to continue in this tradition by hosting a blind Faceoff at the Audioholics Showcase Home premier theater room.  Because of the enormous amount of time and resources it takes to conduct a blind comparison such as this, I limited it to four speaker pairs, each with a maximum price of $1k/pair.

The Contenders

• Axiom Audio M60v3               $990/pair
• EMP E55ti                          $795/pair
• JBL ES80                           $999/pair
• Klipsch RF-62                     $880/pair
  

With the exception of the relative newcomer EMP, all of the other speakers in this comparison are some of the biggest names in the industry and also most accessible to consumers both online and/or in brick and mortar stores.  It was important that our selection included easily accessible speakers, as some of the smaller brands tend to be hard for consumers to locate on their own to conduct a proper demo.

Test Methodology

After selecting the speakers, I assembled a panel of listeners which included two trained listeners and four casual listeners.  One of the casual listeners was a female with proclaimed excellent hearing whom also had no knowledge of any of the brands under test.   The local Florida Klipsch rep also showed up for our Faceoff and sat through a few trials with us.  I was happy to capture his experiences while listening to the Klipschs faced off against the Axioms and the JBLs. I noted some of his impressions in the comments section of this report.

The entire front of the room was draped in speaker cloth furnished by RBH Sound with no backlighting behind the cloth so none of the listeners were able to determine what speakers were under test at any given time.  Speaker pairs were tested, two at a time, staggered in the following pattern to ensure equidistant spacing of the pairs:

AB                                AB

The grille covers were removed from each speaker to minimize any additional losses.  The speakers were placed about one foot behind the grille cloth where positions for two pairs at a time were marked off on the floor to ensure consistent lateral spacing between left and right speakers of each pair.  If speaker A was placed towards the left wall for one trial, its position was reversed for the next comparison and so on.

Pictured from left to right (JBL ES80, Klipsch RF-62, Axiom M60v2, EMP E55ti)

Three songs were selected for the blind test which included:

• SACD:        Patricia Barber "Company"
• CD:            Dianne Reeves "Never too far"
• CD:            Usher CD Sampler "Don Juan"
  

Each song was played in its entirety with 20-30 second sampling between the speaker pairs under test.  After switching between pairs more than two consecutive times, I called out the speaker under test as "A, B, C, or D" to ensure the listeners knew what they were listening to at any given time. 

Each listener was asked to pick a designated seat to be used for all of their listening tests.  The listeners were instructed NOT to discuss what they wrote or how they felt about each speaker until the end of the day when all of the listening tests were completed.  Each listening session lasted about 15 minutes.  At the end of the session, the listeners were instructed to leave the room to consume mass quantities of food and drinks while I set up the next test.  Even during our sighted tests, the listeners were never allowed an up-close look at the speakers to reduce aesthetic biases, and I still identified them by their letter designation instead of name brand.  We entertained NO discussions about aesthetics of any of the speakers it order to reduce any listener bias based on appearance alone during the sighed listening tests.  

I personally did NOT participate as a listener in this Faceoff nor did I interject any of my opinions or biases towards each speaker to our listening panel before, or during any of the listening tests. But, I did provide my listening experiences for all of these speakers within this report since I spent several weeks with most of them listening and analyzing prior to conducting this comparison.  It's important to note that I was likely most critical in my comments about the Axioms and EMPs since I spent the greatest amount of time listening to them compared to the other speakers in this Faceoff.

The tests were paired as follows:

• AB
• CD
• AC
• BD
• AD
• BC
  

Where:
A = Axiom M60v3
B = EMP E55ti
C = Klipsch RF-62
D = JBL ES80

Audioholics Speaker Test Form

Participant Name:                      __________________________________

Date: May 7th, 2010

Seat Location (circle one):          FR, FL, BR, BC, BL, Dog

Comparison# (circle one): AB & CD; AC & BD; AD & BC

Rating Scale

1 - poor
2  - fair
3 - Good
4 - Very Good
5 - Excellent 

Describe in detail what you like and dislike about each of the speakers:
 

Speaker A:
 

Speaker B:
 

Speaker C
 

Speaker D:
 

Test Equipment & Calibration

I utilized my Denon AVP-A1HDCI Zone 2 and Zone 3 outputs connected to my Axiom A1400-8 multi channel power amplifier.  The sources included the Denon DVD-A1UDCI for two-channel SACD playback and the Yamaha MCX-2000 for two-channel PCM playback.  All interconnects were Bluejeans 1694A coax and the speaker cables were 20ft lengths of 8PR furnished by Kimber Kable. 

I used the Sencore SP495 to generate wideband pink noise and used C-wt to calibrate each speaker to 75dB SPL at the primary listening position which happened to be center front row.  Level matching speakers isn't a trivial matter.  I found that simply using internal test tones found in A/V receivers / processors isn’t sufficient since they aren’t wideband making the bassier speaker seem less efficient than the speaker with less bass.  In fact, I found that when using the internal test tones of my processor, I had to set the bassier speakers 3dB louder than the less bassy speakers which sounded very different in level when doing my own listening comparisons.  Thus I switched to using wideband pink noise signals to level match and achieved much more balanced results.  Sonically, all the speakers seemed level matched both to myself and the panel of listeners during the testing.

I pinged Dr. Sean Olive and Dr. Floyde Toole from Harman on this very topic and here is what they had to say:

Editorial Note on Level Matching and Listening Position

We use wideband pink noise measured at the listening seat and adjust to B-weighting. More recently we started using a software-based CRC loudness meter that follows the ITU_R 1770 loudness standard for matching loudness of programs.

The combination of wideband noise and B-weighting should give a better loudness match for speakers with larger bass differences. The CRC loudness meter uses a modified B-weighting that extends the low frequency HP filter even further down.  

My meters don't have B-wt capability but I found C-wt was close enough.  I also pulled measurements of each speaker at the primary listening position to get a better sonic picture on what was going on.  In further discussions with Dr. Olive/ and Dr. Toole, they made me well aware of the fact that listening position can have even more impact on preference than the actual speaker itself.  Readers interested in this subject are should reference the white paper submitted to AES by Sean Olive.

Realizing that only two seats were optimal for two channel listening and the other four listeners weren't getting the full sonic picture of the speakers they were listening too.  I didn’t really have a solution for this that wouldn't require multiple days of testing which we simply didn’t have.   Harman conducts their blind tests in mono as do many other loudspeaker companies.  I personally feel this leaves out critical information on how the loudspeakers interact with each other and setup soundstage and imaging characteristics which also leads to very dry listening sessions.  But, I can understand why they test in mono since it reduces another source of variability in listening tests.  Dr. Toole suggests if a speaker wins a blind test in mono, it will almost always win in stereo as well.  Further study on this topic should be explored which is beyond the scope of this article.

Test Biases

No test is without bias, even those that claim to be scientifically controlled and double blind.  In fact, most companies whom claim to do double blind tests are in fact only doing single blind testing since the actual tester is aware of the equipment under test. Our testing isn’t exempt from this either and I did my best to tabulate all of the biases below.

• Tester (me) was aware of all loudspeakers under test at all times
• Tester (me) analyzed results unblind knowing what data corresponded to each speaker
• Some of the listeners were aware of the brand of speakers under test
• Pre-selected music that listeners may be unaware of as a personal reference for judging sound quality
• The listeners were aware that designations for Speaker A, B, C, D never changed
• The grille cloth used to cover the speakers significantly attenuated high frequency response of the speakers under test
• 4 of the 6 listeners were not located in the stereo sweet spot
• Towards the end of the day, listeners experienced confusion and sensory overload from too much testing
• Only a limited number of test trials (6 blind)  were conducted over a single day period which resulted in reduced statistical sampling accuracy
  

The point about the grille cloth losses is perhaps the most significant.  I was able to identify when the grille cloth was up or down using pink noise testing while I was blindfolded.  This concerned me but at the time I had no better way of obscuring the speakers under test for the listeners.  I measured one of the speakers (Axiom M60v2) with and without the grille covering to better understand the losses. 

Axiom M60v3 Frequency Response Comparison
Blue trace - no grille cloth
Red trace - grille cloth 

Using LMS, I measured the Axiom M60v3 with and without the grill cloth covering the speaker and plotted the results for comparison.  As you can see the losses with the grille cloth in place in the 6-7kHz and above 10kHz were pretty substantial (about 2dB). 

I did my best to ensure the cloth was as taut as possible but in hindsight it was placed about 1 foot in front of the speakers under test which may have caused it to act as an absorber of direct and reflected sound. 

At first glance I wanted to chastise RBH Sound for using such lossy grille cloth until we both confirmed with measurements that once the cloth was stretched onto a grille cover and placed in close proximity to the speaker, the losses were less substantial (about 1dB) as shown below.

RBH Sound 441-SE Frequency Response
Blue trace - no grille cover/cloth
Red trace - grille cover/cloth

This does however give me pause about ever draping grille cloth for conducting a blind test.  I also hypothesize that these losses will lead the listeners to prefer the brighter speakers over the more tonally neutral speakers as a result.