“Let our rigorous testing and reviews be your guidelines to A/V equipment – not marketing slogans”
Facebook Youtube Twitter instagram pinterest

2010 $1k Floorstanding Loudspeaker Faceoff Test Methodology

By

speakers2.jpgSince the days when I founded Audioholics.com in 1999, we've  conducted  annual speaker Faceoffs.  These Faceoffs ranged from bookshelf to floorstanding speakers of various designs and budgets.  Over the years we managed to beef up the brand variety while also increase the panel of listeners and refine our testing process.  Last year we conducted our first blind listening test headed up by Tom Andry.  The results were both enlightening and intriguing to say the least. 

I wanted to continue in this tradition by hosting a blind Faceoff at the Audioholics Showcase Home premier theater room.  Because of the enormous amount of time and resources it takes to conduct a blind comparison such as this, I limited it to four speaker pairs, each with a maximum price of $1k/pair.

The Contenders

With the exception of the relative newcomer EMP, all of the other speakers in this comparison are some of the biggest names in the industry and also most accessible to consumers both online and/or in brick and mortar stores.  It was important that our selection included easily accessible speakers, as some of the smaller brands tend to be hard for consumers to locate on their own to conduct a proper demo.

Test Methodology

listeners.jpgAfter selecting the speakers, I assembled a panel of listeners which included two trained listeners and four casual listeners.  One of the casual listeners was a female with proclaimed excellent hearing whom also had no knowledge of any of the brands under test.   The local Florida Klipsch rep also showed up for our Faceoff and sat through a few trials with us.  I was happy to capture his experiences while listening to the Klipschs faced off against the Axioms and the JBLs. I noted some of his impressions in the comments section of this report.

The entire front of the room was draped in speaker cloth furnished by RBH Sound with no backlighting behind the cloth so none of the listeners were able to determine what speakers were under test at any given time.  Speaker pairs were tested, two at a time, staggered in the following pattern to ensure equidistant spacing of the pairs:

AB                                AB

The grille covers were removed from each speaker to minimize any additional losses.  The speakers were placed about one foot behind the grille cloth where positions for two pairs at a time were marked off on the floor to ensure consistent lateral spacing between left and right speakers of each pair.  If speaker A was placed towards the left wall for one trial, its position was reversed for the next comparison and so on.

speakers2.jpg

Pictured from left to right (JBL ES80, Klipsch RF-62, Axiom M60v2, EMP E55ti)

Three songs were selected for the blind test which included:

  • SACD:        Patricia Barber "Company"
  • CD:            Dianne Reeves "Never too far"
  • CD:            Usher CD Sampler "Don Juan"

Each song was played in its entirety with 20-30 second sampling between the speaker pairs under test.  After switching between pairs more than two consecutive times, I called out the speaker under test as "A, B, C, or D" to ensure the listeners knew what they were listening to at any given time. 

Each listener was asked to pick a designated seat to be used for all of their listening tests.  The listeners were instructed NOT to discuss what they wrote or how they felt about each speaker until the end of the day when all of the listening tests were completed.  Each listening session lasted about 15 minutes.  At the end of the session, the listeners were instructed to leave the room to consume mass quantities of food and drinks while I set up the next test.  Even during our sighted tests, the listeners were never allowed an up-close look at the speakers to reduce aesthetic biases, and I still identified them by their letter designation instead of name brand.  We entertained NO discussions about aesthetics of any of the speakers it order to reduce any listener bias based on appearance alone during the sighed listening tests.  

I personally did NOT participate as a listener in this Faceoff nor did I interject any of my opinions or biases towards each speaker to our listening panel before, or during any of the listening tests. But, I did provide my listening experiences for all of these speakers within this report since I spent several weeks with most of them listening and analyzing prior to conducting this comparison.  It's important to note that I was likely most critical in my comments about the Axioms and EMPs since I spent the greatest amount of time listening to them compared to the other speakers in this Faceoff.

The tests were paired as follows:speakers-back2.jpg

  • AB
  • CD
  • AC
  • BD
  • AD
  • BC

Where:
A = Axiom M60v3
B = EMP E55ti
C = Klipsch RF-62
D = JBL ES80

Audioholics Speaker Test Form

Participant Name:                      __________________________________

Date: May 7th, 2010

Seat Location (circle one):          FR, FL, BR, BC, BL, Dog

Comparison# (circle one): AB & CD; AC & BD; AD & BC

Rating Scale

1 - poor
2  - fair
3 - Good
4 - Very Good
5 - Excellent 

 

Speaker A

Speaker B

Speaker C

Speaker D

Highs

 

 

 

 

Midrange

 

 

 

 

Bass

 

 

 

 

Soundstage

 

 

 

 

Imaging

 

 

 

 

Dynamic Range

 

 

 

 

Overall Rating

 

 

 

 

Would you Own (Y/N)

 

 

 

 

 

Describe in detail what you like and dislike about each of the speakers:
 

Speaker A:
 

Speaker B:
 

Speaker C
 

Speaker D:
 

Test Equipment & Calibration

racks.jpgI utilized my Denon AVP-A1HDCI Zone 2 and Zone 3 outputs connected to my Axiom A1400-8 multi channel power amplifier.  The sources included the Denon DVD-A1UDCI for two-channel SACD playback and the Yamaha MCX-2000 for two-channel PCM playback.  All interconnects were Bluejeans 1694A coax and the speaker cables were 20ft lengths of 8PR furnished by Kimber Kable. 

I used the Sencore SP495 to generate wideband pink noise and used C-wt to calibrate each speaker to 75dB SPL at the primary listening position which happened to be center front row.  Level matching speakers isn't a trivial matter.  I found that simply using internal test tones found in A/V receivers / processors isn’t sufficient since they aren’t wideband making the bassier speaker seem less efficient than the speaker with less bass.  In fact, I found that when using the internal test tones of my processor, I had to set the bassier speakers 3dB louder than the less bassy speakers which sounded very different in level when doing my own listening comparisons.  Thus I switched to using wideband pink noise signals to level match and achieved much more balanced results.  Sonically, all the speakers seemed level matched both to myself and the panel of listeners during the testing.

I pinged Dr. Sean Olive and Dr. Floyde Toole from Harman on this very topic and here is what they had to say:

Editorial Note on Level Matching and Listening Position

We use wideband pink noise measured at the listening seat and adjust to B-weighting. More recently we started using a software-based CRC loudness meter that follows the ITU_R 1770 loudness standard for matching loudness of programs.

The combination of wideband noise and B-weighting should give a better loudness match for speakers with larger bass differences. The CRC loudness meter uses a modified B-weighting that extends the low frequency HP filter even further down.  

My meters don't have B-wt capability but I found C-wt was close enough.  I also pulled measurements of each speaker at the primary listening position to get a better sonic picture on what was going on.  In further discussions with Dr. Olive/ and Dr. Toole, they made me well aware of the fact that listening position can have even more impact on preference than the actual speaker itself.  Readers interested in this subject are should reference the white paper submitted to AES by Sean Olive.

Realizing that only two seats were optimal for two channel listening and the other four listeners weren't getting the full sonic picture of the speakers they were listening too.  I didn’t really have a solution for this that wouldn't require multiple days of testing which we simply didn’t have.   Harman conducts their blind tests in mono as do many other loudspeaker companies.  I personally feel this leaves out critical information on how the loudspeakers interact with each other and setup soundstage and imaging characteristics which also leads to very dry listening sessions.  But, I can understand why they test in mono since it reduces another source of variability in listening tests.  Dr. Toole suggests if a speaker wins a blind test in mono, it will almost always win in stereo as well.  Further study on this topic should be explored which is beyond the scope of this article.

Test Biases

No test is without bias, even those that claim to be scientifically controlled and double blind.  In fact, most companies whom claim to do double blind tests are in fact only doing single blind testing since the actual tester is aware of the equipment under test. Our testing isn’t exempt from this either and I did my best to tabulate all of the biases below.

  • Tester (me) was aware of all loudspeakers under test at all times
  • Tester (me) analyzed results unblind knowing what data corresponded to each speaker
  • Some of the listeners were aware of the brand of speakers under test
  • Pre-selected music that listeners may be unaware of as a personal reference for judging sound quality
  • The listeners were aware that designations for Speaker A, B, C, D never changed
  • The grille cloth used to cover the speakers significantly attenuated high frequency response of the speakers under test
  • 4 of the 6 listeners were not located in the stereo sweet spot
  • Towards the end of the day, listeners experienced confusion and sensory overload from too much testing
  • Only a limited number of test trials (6 blind)  were conducted over a single day period which resulted in reduced statistical sampling accuracy

The point about the grille cloth losses is perhaps the most significant.  I was able to identify when the grille cloth was up or down using pink noise testing while I was blindfolded.  This concerned me but at the time I had no better way of obscuring the speakers under test for the listeners.  I measured one of the speakers (Axiom M60v2) with and without the grille covering to better understand the losses. 

1grille.JPG

Axiom M60v3 Frequency Response Comparison
Blue trace - no grille cloth
Red trace - grille cloth 

Using LMS, I measured the Axiom M60v3 with and without the grill cloth covering the speaker and plotted the results for comparison.  As you can see the losses with the grille cloth in place in the 6-7kHz and above 10kHz were pretty substantial (about 2dB). 

I did my best to ensure the cloth was as taut as possible but in hindsight it was placed about 1 foot in front of the speakers under test which may have caused it to act as an absorber of direct and reflected sound. 

At first glance I wanted to chastise RBH Sound for using such lossy grille cloth until we both confirmed with measurements that once the cloth was stretched onto a grille cover and placed in close proximity to the speaker, the losses were less substantial (about 1dB) as shown below.

441-SE grill on off.JPG

RBH Sound 441-SE Frequency Response
Blue trace - no grille cover/cloth
Red trace - grille cover/cloth

This does however give me pause about ever draping grille cloth for conducting a blind test.  I also hypothesize that these losses will lead the listeners to prefer the brighter speakers over the more tonally neutral speakers as a result. 

 

Confused about what AV Gear to buy or how to set it up? Join our Exclusive Audioholics E-Book Membership Program!

Recent Forum Posts:

gene posts on January 14, 2012 12:04
Time of Reconcile our Differences

This constant back and forth between us, forum members and Axiom isn't productive for anyone involved. I honestly don't have it in my heart to sustain negative energy like this. I myself am to blame because I allowed it to progress on this forum rather than objectively removing or stopping the threads just like I would have done for any other manufacturer. For that, I apologize to Axiom.

I think its better to agree to disagree on design aspects of loudspeakers and just be done with it. Everyone has their own reasoning that may work best (in their minds) for them. 100% truth can never be realized in something as subjective and emotional as the audiophile experience.

There are MANY satisfied Axiom customers in the marketplace and it is obvious Axiom is meeting their needs. There are also many competing brands for those wanting something else. The free market rules!

Going forward, I'd like to keep a more open mind and positive attitude towards all manufacturers and let the consumers decide based on our reviews and their experiences with the products if said products are right for their needs.

Mods please note this and lets put the brakes on any future threads that turn out like this.

I am closing this thread on a high note with hopes we can continue to cover new Axiom products for the benefit of readers interested in learning more about them.
MinusTheBear posts on January 13, 2012 22:31
Paul_Apollonio, post: 857427
I followed the link to the Axiom site, and would like to make an engineering sidenote. First of all by eliminating the series high-pass capacitor (at the VERY minimum) needed to protect the midrange from dangerous levels of peak LF content, this lowers the impedance of the system in a range where the output of the mids add NOTHING to the output of the Woofer; hence lowering system sensitivity. First bad. Second bad = pretending the worst of this can be found with a distortion sweep looking for 2nd and 3rd Harmonics. RUBBISH. The problem is that by eliminating the High Pass (HP) filter on the mids, the LF content will move the VC about, possibly out of the gap and thereby allow the LF to modulate (read distort) the midrange the speaker produces. With a sweeping test (one frequency at a time) there is NO WAY to see this effect, and Axiom is well aware of that. To see this effect, one must put in two frequencies simultaneously and view the output on a spectrum analyzer. (One can see distortion products as sum and difference frequencies) This is a simple process and one all audio engineers are familiar with. Sadly, not all customers are, so the charade continues.

Even if the Midrange driver is made INCREDIBLY stiff, and placed in a very very small sealed enclosure minimizing excursion and hence this distortion, subjecting the midrange VC to the heat caused by the LF content is NEVER better than saving the price of the series capacitor. Unless, I guess, it is your money, and you don't really care about stressing an amp or drive unit you get paid to replace.

There is such a thing as recommended practice and procedures, and the practice of eliminating the high pass filter, even if only a single series capacitor from the midrange driver is not a good idea by any stretch of imagination. In fact, it is a sign the designer is clueless or could care less about the result.

As for the listening tests, there are never any shortage of people willing to claim a given distortion is inaudible. Of course, if you limit the input power to very low levels, you won't hear this problem. You won't hear many speaker distortions as most only show up on high drive levels.

Certain physical principles apply to design, and it does not matter the brand or the politics involved. Allowing the large peak amplitudes of low frequency content to get to a midrange speakers voice coil is a terribly bad idea period.

This is not a new concept. (At least to competent engineers who are not counting nickles and pennies). - Paul Apollonio

Axiom did this with the M80v1 as well with not so successful results.

Emonatics - Emotiva Fans Site
Paul_Apollonio posts on January 13, 2012 16:57
gene, post: 856611
{Heavily edited by me}

Nobody really wants unfavorable comments written about their products and proud owners of said products don't want to see negative comments either.

The bottom line is Axiom has always had a problem with our critical review process. They don't like negative comments about products. They don't like face offs. They seemingly don't like measurements or blind tests done outside their facilities. They have told me this many times in the past in person and via email. They instead prefer consumers to accept their science and testing as gospel that spending more money than what they sell their speakers for is simply purchasing cosmetic upgrades.

I've spent 12+ years comparing speakers (sighted and blind) from virtually every manufacturer and my experiences don't always match what Axiom preaches.

In my experience Axiom is NOT up to the level of legitimately well engineered cost no object speakers however.

While there are some outlandish speaker designs out there, there are also some incredible sounding products that happen to cost a lot too.

Performance is all over the map but it doesn't necessarily stop progressing at a magic price point.

I'd honestly like to stop the bantering back and forth on this topic. It's been beaten to death and nobody gains from it.

I won't be commenting any further about Axiom on the forums. I've said everything that needed to be said.

I followed the link to the Axiom site, and would like to make an engineering sidenote. First of all by eliminating the series high-pass capacitor (at the VERY minimum) needed to protect the midrange from dangerous levels of peak LF content, this lowers the impedance of the system in a range where the output of the mids add NOTHING to the output of the Woofer; hence lowering system sensitivity. First bad. Second bad = pretending the worst of this can be found with a distortion sweep looking for 2nd and 3rd Harmonics. RUBBISH. The problem is that by eliminating the High Pass (HP) filter on the mids, the LF content will move the VC about, possibly out of the gap and thereby allow the LF to modulate (read distort) the midrange the speaker produces. With a sweeping test (one frequency at a time) there is NO WAY to see this effect, and Axiom is well aware of that. To see this effect, one must put in two frequencies simultaneously and view the output on a spectrum analyzer. (One can see distortion products as sum and difference frequencies) This is a simple process and one all audio engineers are familiar with. Sadly, not all customers are, so the charade continues.

Even if the Midrange driver is made INCREDIBLY stiff, and placed in a very very small sealed enclosure minimizing excursion and hence this distortion, subjecting the midrange VC to the heat caused by the LF content is NEVER better than saving the price of the series capacitor. Unless, I guess, it is your money, and you don't really care about stressing an amp or drive unit you get paid to replace.

There is such a thing as recommended practice and procedures, and the practice of eliminating the high pass filter, even if only a single series capacitor from the midrange driver is not a good idea by any stretch of imagination. In fact, it is a sign the designer is clueless or could care less about the result.

As for the listening tests, there are never any shortage of people willing to claim a given distortion is inaudible. Of course, if you limit the input power to very low levels, you won't hear this problem. You won't hear many speaker distortions as most only show up on high drive levels.

Certain physical principles apply to design, and it does not matter the brand or the politics involved. Allowing the large peak amplitudes of low frequency content to get to a midrange speakers voice coil is a terribly bad idea period.

This is not a new concept. (At least to competent engineers who are not counting nickles and pennies). - Paul Apollonio
haraldo posts on January 13, 2012 03:50
agarwalro, post: 857098
Perhaps I was a little curt in my last post. I sensed something that was valuable and worth saving and perhaps overreacted. Moving on…

$5 says Jim Salk can match the finish on any of those Lansche veneers .

My remark was not really about Gene but the credibility of the AH reviews and their process
Now we move on!!!!!!!!

Some remarks been said that the Lansche 5.1 is the best dynamic speaker some people heard…. (Obviously they didn't listen to the upscale models from same producer) well they better be considering the price tag…. but still…. need to make sure I deliver my lotto this week
tom67 posts on January 12, 2012 20:03
So lets end this…..

where it began…..Klipsch won the contest and it was a fair fight…and none of these contests are perfect…..so, perhaps the bottom line is that you might want to look at their products within given price points and not believe the stale, lame comments on websites by people who have never owned them…and Yamaha receivers are not “bright” either and on and on…
Post Reply