2010 $1k Floorstanding Loudspeaker Faceoff Test Methodology
Since the days when I founded Audioholics.com in
1999, we've conducted annual speaker Faceoffs. These Faceoffs ranged from bookshelf to
floorstanding speakers of various designs and budgets. Over the years we managed to beef up the
brand variety while also increase the panel of listeners and refine our testing
process. Last year we conducted our
first blind listening test headed up
by Tom Andry. The results were both
enlightening and intriguing to say the least.
I wanted to continue in this tradition by hosting a blind Faceoff at the Audioholics Showcase Home premier theater room. Because of the enormous amount of time and resources it takes to conduct a blind comparison such as this, I limited it to four speaker pairs, each with a maximum price of $1k/pair.
The Contenders
- Axiom Audio M60v3 $990/pair
- EMP E55ti $795/pair
- JBL ES80 $999/pair
-
Klipsch
RF-62 $880/pair
With the exception of the relative newcomer EMP, all of the other speakers in this comparison are some of the biggest names in the industry and also most accessible to consumers both online and/or in brick and mortar stores. It was important that our selection included easily accessible speakers, as some of the smaller brands tend to be hard for consumers to locate on their own to conduct a proper demo.
Test Methodology
After selecting the speakers, I assembled a panel of
listeners which included two trained listeners and four casual listeners. One of the casual listeners was a female with
proclaimed excellent hearing whom also had no knowledge of any of the brands
under test. The local Florida Klipsch
rep also showed up for our Faceoff and sat through a few trials with us. I was happy to capture his experiences while
listening to the Klipschs faced off against the Axioms and the JBLs. I noted
some of his impressions in the comments section of this report.
The entire front of the room was draped in speaker cloth furnished by RBH Sound with no backlighting behind the cloth so none of the listeners were able to determine what speakers were under test at any given time. Speaker pairs were tested, two at a time, staggered in the following pattern to ensure equidistant spacing of the pairs:
AB AB
The grille covers were removed from each speaker to minimize any additional losses. The speakers were placed about one foot behind the grille cloth where positions for two pairs at a time were marked off on the floor to ensure consistent lateral spacing between left and right speakers of each pair. If speaker A was placed towards the left wall for one trial, its position was reversed for the next comparison and so on.
Pictured from left to right (JBL ES80, Klipsch RF-62, Axiom M60v2, EMP E55ti)
Three songs were selected for the blind test which included:
- SACD: Patricia Barber "Company"
- CD: Dianne Reeves "Never too far"
-
CD:
Usher CD Sampler "Don Juan"
Each song was played in its entirety with 20-30 second sampling between the speaker pairs under test. After switching between pairs more than two consecutive times, I called out the speaker under test as "A, B, C, or D" to ensure the listeners knew what they were listening to at any given time.
Each listener was asked to pick a designated seat to be used for all of their listening tests. The listeners were instructed NOT to discuss what they wrote or how they felt about each speaker until the end of the day when all of the listening tests were completed. Each listening session lasted about 15 minutes. At the end of the session, the listeners were instructed to leave the room to consume mass quantities of food and drinks while I set up the next test. Even during our sighted tests, the listeners were never allowed an up-close look at the speakers to reduce aesthetic biases, and I still identified them by their letter designation instead of name brand. We entertained NO discussions about aesthetics of any of the speakers it order to reduce any listener bias based on appearance alone during the sighed listening tests.
I personally did NOT participate as a listener in this Faceoff nor did I interject any of my opinions or biases towards each speaker to our listening panel before, or during any of the listening tests. But, I did provide my listening experiences for all of these speakers within this report since I spent several weeks with most of them listening and analyzing prior to conducting this comparison. It's important to note that I was likely most critical in my comments about the Axioms and EMPs since I spent the greatest amount of time listening to them compared to the other speakers in this Faceoff.
The tests were paired
as follows:
- AB
- CD
- AC
- BD
- AD
- BC
Where:
A = Axiom M60v3
B = EMP
E55ti
C = Klipsch
RF-62
D = JBL
ES80
Audioholics Speaker Test Form
Participant Name: __________________________________
Date: May 7th, 2010
Seat Location (circle one): FR, FL, BR, BC, BL, Dog
Comparison# (circle one): AB & CD; AC & BD; AD & BC
Rating Scale
1 - poor
2 - fair
3 - Good
4 - Very
Good
5 -
Excellent
|
Speaker A |
Speaker B |
Speaker C |
Speaker D |
Highs |
|
|
|
|
Midrange |
|
|
|
|
Bass |
|
|
|
|
Soundstage |
|
|
|
|
Imaging |
|
|
|
|
Dynamic Range |
|
|
|
|
Overall Rating |
|
|
|
|
Would you Own (Y/N) |
|
|
|
|
Describe in detail what you like and
dislike about each of the speakers:
Speaker A:
Speaker B:
Speaker C
Speaker D:
Test Equipment & Calibration
I utilized my Denon AVP-A1HDCI Zone 2 and Zone 3
outputs connected to my Axiom A1400-8 multi channel power amplifier. The sources included the Denon DVD-A1UDCI for
two-channel SACD playback and the Yamaha MCX-2000 for two-channel PCM
playback. All interconnects were Bluejeans
1694A coax and the speaker cables were 20ft lengths of 8PR furnished by Kimber
Kable.
I used the Sencore SP495 to generate wideband pink noise and used C-wt to calibrate each speaker to 75dB SPL at the primary listening position which happened to be center front row. Level matching speakers isn't a trivial matter. I found that simply using internal test tones found in A/V receivers / processors isn’t sufficient since they aren’t wideband making the bassier speaker seem less efficient than the speaker with less bass. In fact, I found that when using the internal test tones of my processor, I had to set the bassier speakers 3dB louder than the less bassy speakers which sounded very different in level when doing my own listening comparisons. Thus I switched to using wideband pink noise signals to level match and achieved much more balanced results. Sonically, all the speakers seemed level matched both to myself and the panel of listeners during the testing.
I pinged Dr. Sean Olive and Dr. Floyde Toole from Harman on this very topic and here is what they had to say:
Editorial Note on Level Matching and Listening Position
We use wideband pink noise measured at the listening seat and adjust to B-weighting. More recently we started using a software-based CRC loudness meter that follows the ITU_R 1770 loudness standard for matching loudness of programs.
The combination of wideband noise and B-weighting should give a better loudness match for speakers with larger bass differences. The CRC loudness meter uses a modified B-weighting that extends the low frequency HP filter even further down.
My meters don't have B-wt capability but I found C-wt was close enough. I also pulled measurements of each speaker at the primary listening position to get a better sonic picture on what was going on. In further discussions with Dr. Olive/ and Dr. Toole, they made me well aware of the fact that listening position can have even more impact on preference than the actual speaker itself. Readers interested in this subject are should reference the white paper submitted to AES by Sean Olive.
Realizing that only two seats were optimal for two channel listening and the other four listeners weren't getting the full sonic picture of the speakers they were listening too. I didn’t really have a solution for this that wouldn't require multiple days of testing which we simply didn’t have. Harman conducts their blind tests in mono as do many other loudspeaker companies. I personally feel this leaves out critical information on how the loudspeakers interact with each other and setup soundstage and imaging characteristics which also leads to very dry listening sessions. But, I can understand why they test in mono since it reduces another source of variability in listening tests. Dr. Toole suggests if a speaker wins a blind test in mono, it will almost always win in stereo as well. Further study on this topic should be explored which is beyond the scope of this article.
Test Biases
No test is without bias, even those that claim to be scientifically controlled and double blind. In fact, most companies whom claim to do double blind tests are in fact only doing single blind testing since the actual tester is aware of the equipment under test. Our testing isn’t exempt from this either and I did my best to tabulate all of the biases below.
- Tester (me) was aware of all loudspeakers under test at all times
- Tester (me) analyzed results unblind knowing what data corresponded to each speaker
- Some of the listeners were aware of the brand of speakers under test
- Pre-selected music that listeners may be unaware of as a personal reference for judging sound quality
- The listeners were aware that designations for Speaker A, B, C, D never changed
- The grille cloth used to cover the speakers significantly attenuated high frequency response of the speakers under test
- 4 of the 6 listeners were not located in the stereo sweet spot
- Towards the end of the day, listeners experienced confusion and sensory overload from too much testing
- Only a limited number of test
trials (6 blind) were conducted
over a single day period which resulted in reduced statistical sampling
accuracy
The point about the grille cloth losses is perhaps the most significant. I was able to identify when the grille cloth was up or down using pink noise testing while I was blindfolded. This concerned me but at the time I had no better way of obscuring the speakers under test for the listeners. I measured one of the speakers (Axiom M60v2) with and without the grille covering to better understand the losses.
Axiom M60v3 Frequency
Response Comparison
Blue trace - no grille cloth
Red trace - grille cloth
Using LMS, I measured the Axiom M60v3 with and without the grill cloth covering the speaker and plotted the results for comparison. As you can see the losses with the grille cloth in place in the 6-7kHz and above 10kHz were pretty substantial (about 2dB).
I did my best to ensure the cloth was as taut as possible but in hindsight it was placed about 1 foot in front of the speakers under test which may have caused it to act as an absorber of direct and reflected sound.
At first glance I wanted to chastise RBH Sound for using such lossy grille cloth until we both confirmed with measurements that once the cloth was stretched onto a grille cover and placed in close proximity to the speaker, the losses were less substantial (about 1dB) as shown below.
RBH Sound 441-SE
Frequency Response
Blue trace - no grille cover/cloth
Red trace - grille cover/cloth
This does however give me pause about ever draping grille cloth for conducting a blind test. I also hypothesize that these losses will lead the listeners to prefer the brighter speakers over the more tonally neutral speakers as a result.
Confused about what AV Gear to buy or how to set it up? Join our Exclusive Audioholics E-Book Membership Program!
See also:
Recent Forum Posts:
This constant back and forth between us, forum members and Axiom isn't productive for anyone involved. I honestly don't have it in my heart to sustain negative energy like this. I myself am to blame because I allowed it to progress on this forum rather than objectively removing or stopping the threads just like I would have done for any other manufacturer. For that, I apologize to Axiom.
I think its better to agree to disagree on design aspects of loudspeakers and just be done with it. Everyone has their own reasoning that may work best (in their minds) for them. 100% truth can never be realized in something as subjective and emotional as the audiophile experience.
There are MANY satisfied Axiom customers in the marketplace and it is obvious Axiom is meeting their needs. There are also many competing brands for those wanting something else. The free market rules!
Going forward, I'd like to keep a more open mind and positive attitude towards all manufacturers and let the consumers decide based on our reviews and their experiences with the products if said products are right for their needs.
Mods please note this and lets put the brakes on any future threads that turn out like this.
I am closing this thread on a high note with hopes we can continue to cover new Axiom products for the benefit of readers interested in learning more about them.
Paul_Apollonio, post: 857427
I followed the link to the Axiom site, and would like to make an engineering sidenote. First of all by eliminating the series high-pass capacitor (at the VERY minimum) needed to protect the midrange from dangerous levels of peak LF content, this lowers the impedance of the system in a range where the output of the mids add NOTHING to the output of the Woofer; hence lowering system sensitivity. First bad. Second bad = pretending the worst of this can be found with a distortion sweep looking for 2nd and 3rd Harmonics. RUBBISH. The problem is that by eliminating the High Pass (HP) filter on the mids, the LF content will move the VC about, possibly out of the gap and thereby allow the LF to modulate (read distort) the midrange the speaker produces. With a sweeping test (one frequency at a time) there is NO WAY to see this effect, and Axiom is well aware of that. To see this effect, one must put in two frequencies simultaneously and view the output on a spectrum analyzer. (One can see distortion products as sum and difference frequencies) This is a simple process and one all audio engineers are familiar with. Sadly, not all customers are, so the charade continues.
Even if the Midrange driver is made INCREDIBLY stiff, and placed in a very very small sealed enclosure minimizing excursion and hence this distortion, subjecting the midrange VC to the heat caused by the LF content is NEVER better than saving the price of the series capacitor. Unless, I guess, it is your money, and you don't really care about stressing an amp or drive unit you get paid to replace.
There is such a thing as recommended practice and procedures, and the practice of eliminating the high pass filter, even if only a single series capacitor from the midrange driver is not a good idea by any stretch of imagination. In fact, it is a sign the designer is clueless or could care less about the result.
As for the listening tests, there are never any shortage of people willing to claim a given distortion is inaudible. Of course, if you limit the input power to very low levels, you won't hear this problem. You won't hear many speaker distortions as most only show up on high drive levels.
Certain physical principles apply to design, and it does not matter the brand or the politics involved. Allowing the large peak amplitudes of low frequency content to get to a midrange speakers voice coil is a terribly bad idea period.
This is not a new concept. (At least to competent engineers who are not counting nickles and pennies). - Paul Apollonio
Axiom did this with the M80v1 as well with not so successful results.
Emonatics - Emotiva Fans Site
gene, post: 856611
{Heavily edited by me}
Nobody really wants unfavorable comments written about their products and proud owners of said products don't want to see negative comments either.
The bottom line is Axiom has always had a problem with our critical review process. They don't like negative comments about products. They don't like face offs. They seemingly don't like measurements or blind tests done outside their facilities. They have told me this many times in the past in person and via email. They instead prefer consumers to accept their science and testing as gospel that spending more money than what they sell their speakers for is simply purchasing cosmetic upgrades.
I've spent 12+ years comparing speakers (sighted and blind) from virtually every manufacturer and my experiences don't always match what Axiom preaches.
In my experience Axiom is NOT up to the level of legitimately well engineered cost no object speakers however.
While there are some outlandish speaker designs out there, there are also some incredible sounding products that happen to cost a lot too.
Performance is all over the map but it doesn't necessarily stop progressing at a magic price point.
I'd honestly like to stop the bantering back and forth on this topic. It's been beaten to death and nobody gains from it.
I won't be commenting any further about Axiom on the forums. I've said everything that needed to be said.
I followed the link to the Axiom site, and would like to make an engineering sidenote. First of all by eliminating the series high-pass capacitor (at the VERY minimum) needed to protect the midrange from dangerous levels of peak LF content, this lowers the impedance of the system in a range where the output of the mids add NOTHING to the output of the Woofer; hence lowering system sensitivity. First bad. Second bad = pretending the worst of this can be found with a distortion sweep looking for 2nd and 3rd Harmonics. RUBBISH. The problem is that by eliminating the High Pass (HP) filter on the mids, the LF content will move the VC about, possibly out of the gap and thereby allow the LF to modulate (read distort) the midrange the speaker produces. With a sweeping test (one frequency at a time) there is NO WAY to see this effect, and Axiom is well aware of that. To see this effect, one must put in two frequencies simultaneously and view the output on a spectrum analyzer. (One can see distortion products as sum and difference frequencies) This is a simple process and one all audio engineers are familiar with. Sadly, not all customers are, so the charade continues.
Even if the Midrange driver is made INCREDIBLY stiff, and placed in a very very small sealed enclosure minimizing excursion and hence this distortion, subjecting the midrange VC to the heat caused by the LF content is NEVER better than saving the price of the series capacitor. Unless, I guess, it is your money, and you don't really care about stressing an amp or drive unit you get paid to replace.
There is such a thing as recommended practice and procedures, and the practice of eliminating the high pass filter, even if only a single series capacitor from the midrange driver is not a good idea by any stretch of imagination. In fact, it is a sign the designer is clueless or could care less about the result.
As for the listening tests, there are never any shortage of people willing to claim a given distortion is inaudible. Of course, if you limit the input power to very low levels, you won't hear this problem. You won't hear many speaker distortions as most only show up on high drive levels.
Certain physical principles apply to design, and it does not matter the brand or the politics involved. Allowing the large peak amplitudes of low frequency content to get to a midrange speakers voice coil is a terribly bad idea period.
This is not a new concept. (At least to competent engineers who are not counting nickles and pennies). - Paul Apollonio
agarwalro, post: 857098
Perhaps I was a little curt in my last post. I sensed something that was valuable and worth saving and perhaps overreacted. Moving on…
$5 says Jim Salk can match the finish on any of those Lansche veneers .
My remark was not really about Gene but the credibility of the AH reviews and their process
Now we move on!!!!!!!!
Some remarks been said that the Lansche 5.1 is the best dynamic speaker some people heard…. (Obviously they didn't listen to the upscale models from same producer) well they better be considering the price tag…. but still…. need to make sure I deliver my lotto this week
where it began…..Klipsch won the contest and it was a fair fight…and none of these contests are perfect…..so, perhaps the bottom line is that you might want to look at their products within given price points and not believe the stale, lame comments on websites by people who have never owned them…and Yamaha receivers are not “bright” either and on and on…