Setting up the Comparison
The first step in doing a comparison is picking the room. For consumers looking to make a purchase this should be YOUR room. Showrooms are NEVER, EVER like your room. They are either much better or more likely much worse (if you take even the smallest amount of our advice on this site).Try to do in-home auditions as much as possible. Remember, a store with a good return policy (check for those restocking fees) is just begging you to take items home for comparison. These days, even the internet direct companies are starting to loosen up their shipping policies offering free shipping (at least one way, sometimes both). Check the cost of the return shipping - it very well may be worth it for the piece of mind that you bought the right equipment.
In a professional setting, you'll want to pick a room where you can easily fit the gear, the participants, and everything else that is required. An acoustically treated room is best - preferably something that has a fairly flat frequency response. This ensures that changes will be more audible than in a non-treated room.
Yes, I said more audible and not just audible. Why? The room is affecting all the items equally - at least in theory. So if there is a 75Hz suckout; it is there for all the items. If the room has been measured, you probably already know where the problems are and can warn the participants or modify the results afterwards. So any negative (or positive for that matter) effects will be applied to all the components equally. This is also why it is important to equalize all the other components in the system. Any affect any one of them has on the sound will be the same for both items under comparison. As long as the effect is the same, it shouldn't stop the participants from determining the differences in the comparison units. So, getting the best room possible is definitely the goal but having a less than perfect room is NOT a valid reason to discount a comparison's results.
Author's Note - This idea of a negative effect of a room or component not really mattering assumes a minor effect. Small lapses here or there will not overly taint the results of a comparison. Since all the comparison items are affected equally, the participants shouldn't even be aware of them. Larger effects (or for that matter small effects at critical points) can taint the entire thing and make the results suspect. For example, if you are comparing subwoofers and the room you are in has a suckout at 30Hz, subs that cut off around 28Hz will sound like they die far before that while subs with lower output will have time to kick back in. Generally though, a few smaller dips or spikes shouldn't make much of a difference.
The next step is to set up the components under comparison. For electronics, it is fairly straightforward in that you just have to have everything accessible. Displays might present a bit more of a problem but as long as the lighting conditions are about the same and the displays equidistant from the observers, you should be okay. Speakers, as you might imagine, present a special case.
Where a speaker is placed in a room can make a big difference sonically. Distance from walls, toe in, distance from each of the listeners, acoustical treatments, wall materials and more can all make a fairly substantial difference in imaging, soundstage, perceptions of brightness, etc. The accepted method of combating this is to place speakers so that the pairs are staggered. So if the right speaker is on the outside, the left speaker is on in the inside (rather than having one pair on the outside and one on the inside). Personally, I'm not convinced this is the best solution but it obviously looks to be the best. If I had a research grant, that'd be one of the first things I'd look at. I'd suggest if you have the time, using the staggered method but switching the speakers at least once to see if the listener's perceptions are any different.
I'll talk more about blind/double blind comparisons later, but when you are setting up your room, you'll want to consider whether or not you are going to use some sort of screen to hide the components. For displays, at the very least you should hide the logo though some might be able to tell which is which from the bevel. Amps, receivers, cables, other sources and electronics should all be screened off from the listeners. Speakers, again present a bit of a problem.
Some people believe that using a screen, even one that is designed to be acoustically transparent, attenuates the high end. Again, personally, I'm not convinced. Sure, you may be able to measure a bit of a difference but without credible proof I'm not buying that it is an audible difference. That being said, nearly every speaker manufacturer on the planet makes a grill for their speakers and most make them out of fabric. Remove the grill and put up a screen. While I, like many enthusiasts, do critical listening with the grills off, most of a speaker's duties will involve a grill. If nothing else, you haven't unfairly hamstringed anyone.
The last thing to remember (and this is a biggie) - level match. Nothing will unduly skew a comparison like having one component louder than the other. It is well documented that people associate loudness with quality (and for that matter, brightness with quality). If one component is louder than the other, it will consistently be rated more favorably. As an aside, making sure that that your listeners switch seats during each and every comparison is a way to offset any placement issues that may arise from being nearer to one speaker than another.
1)That's a good idea but it bring up a question of how different size speakers will interact and affect each others sound in free space. Baffle step compensation is designed differently by different manufacturers.
I'd go with turntable. It's not hard to manufacture. Basically it's a large Lazy Susan with carpeted base.
Apparently the turntable method at the labs are enough not to have interference between the speakers when they are 180 behind in the off mode.
2)Ehh, yes if we want to be perfectly precise. How do we account for differences in off axis FR between different speakers? It will affect SPL matching unless the reviewer is in anechoic chamber.
Personally, I'd go with pink noise within 0.5db. Differences in FR between the speakers are very obvious in comparison to amps, CD players and other electronics.
The point of level matching is not to make it harder to differentiate, it is to eliminate a volume difference that the listener may misconstrue as being better. You don't EQ the hole frequency band, perhaps at one frequency, like 1kHz. And the other parameters are really what will tell one speaker apart from the other and stand it out, that is the whole point, which is the better speaker.
3)I was being a bit sarcastic about it.
On another note, I think a reviewer should use real MLS measurements and not a Micky Mouse analyzer.
What's wrong with Mickey Mouse?
They might be the best sounding speakers in the world.
But if their ugly, they won't be going in my home.So yeah, there is a bias in sighted tests. But we are talking about comparisons, not strictly testing for performance.
Besides, if you do it right there will be a non-voting person controlling the a/b switch and not telling you which pair your listening to. .
Yes, but, if you can see the speakers I am sure you will have a good probability of knowing which one is on
The visual impact should play a role, of course, but in my estimation after your sonic experimentation and see which has more weight for you.
Even with that switcher person not telling you, it still matters and can affect the bias responses. That is why DBT is used when it matters.
In this case though...
I've conducted a few comparo's myself, and to be honest, the whole single-blind, double-blind stuff is totally unnecessary. When a test is set up properly, you'll find that speakers differ SO MUCH that you'll laugh at the notion of DBT's. ...
Go to town and be amazed how quickly and easily you can tell differences.
The issue is not that there are still sonic difference between speakers but where does one's bias will take them; the one that visually impresses and affects the sensory inputs or, in fact the one that one really prefers due only to the sound and nothing else.
If DBT was not necessary, even with speakers, research labs would not use them. After all, that is what they used to find out what people of all backgrounds prefer, flat frequency response, wide dispersion, etc. And demonstrated that without it, the other sensory inputs do a number on preferences.
You cannot and don't think the labs EQ the speakers so the levels are matched, but you do need to level match and perhaps, with speakers, you do need to use a very sensitive spl meter that is capable of such level.
Come to think of it, the same voltage may not be enough as when the speakers are the constant and the components are swapped; then, it is imperative to use a volt meter as the speaker will output the same level with the same input voltage.
I made this post in another thread, but considering the disucssion here I feel it is worthy of being copied:
They might be the best sounding speakers in the world.
But if their ugly, they won't be going in my home.
So yeah, there is a bias in sighted tests. But we are talking about comparisons, not strictly testing for performance.
Besides, if you do it right there will be a non-voting person controlling the a/b switch and not telling you which pair your listening to. The people listening just need to sit back, relax, close their eyes and listen. Once your ears pick out a particular pair you like, it will become very obvious when that pair is playing. It worked out quite well at my house last year when Gene ran the show. We came up with the top two bookshelfs being the Status Acoustic Decimos and a pair of AV123 x-ls speakers. Definitely not in the same ballpark with appearance and price range. And for what it's worth, the worst performers were the SVS bookshelves.
Unfortunately the Decimos sounded so good I had to buy a pair. And at this point, I don't want to find anything that sounds better. My wallet couldn't stand it and wife would kill me.
So what's the lesson to learn from this?
DON'T KEEP EXPENSIVE GEAR THAT YOU DON'T OWN IN YOUR HOME FOR VERY LONG.
YOU MIGHT END UP WANTING TO BUY IT.