Setting up the Comparison
The first step in doing a comparison is picking the room. For consumers looking to make a purchase this should be YOUR room. Showrooms are NEVER, EVER like your room. They are either much better or more likely much worse (if you take even the smallest amount of our advice on this site).Try to do in-home auditions as much as possible. Remember, a store with a good return policy (check for those restocking fees) is just begging you to take items home for comparison. These days, even the internet direct companies are starting to loosen up their shipping policies offering free shipping (at least one way, sometimes both). Check the cost of the return shipping - it very well may be worth it for the piece of mind that you bought the right equipment.
In a professional setting, you'll want to pick a room where you can easily fit the gear, the participants, and everything else that is required. An acoustically treated room is best - preferably something that has a fairly flat frequency response. This ensures that changes will be more audible than in a non-treated room.
Yes, I said more audible and not just audible. Why? The room is affecting all the items equally - at least in theory. So if there is a 75Hz suckout; it is there for all the items. If the room has been measured, you probably already know where the problems are and can warn the participants or modify the results afterwards. So any negative (or positive for that matter) effects will be applied to all the components equally. This is also why it is important to equalize all the other components in the system. Any affect any one of them has on the sound will be the same for both items under comparison. As long as the effect is the same, it shouldn't stop the participants from determining the differences in the comparison units. So, getting the best room possible is definitely the goal but having a less than perfect room is NOT a valid reason to discount a comparison's results.
Author's Note - This idea of a negative effect of a room or component not really mattering assumes a minor effect. Small lapses here or there will not overly taint the results of a comparison. Since all the comparison items are affected equally, the participants shouldn't even be aware of them. Larger effects (or for that matter small effects at critical points) can taint the entire thing and make the results suspect. For example, if you are comparing subwoofers and the room you are in has a suckout at 30Hz, subs that cut off around 28Hz will sound like they die far before that while subs with lower output will have time to kick back in. Generally though, a few smaller dips or spikes shouldn't make much of a difference.
The next step is to set up the components under comparison. For electronics, it is fairly straightforward in that you just have to have everything accessible. Displays might present a bit more of a problem but as long as the lighting conditions are about the same and the displays equidistant from the observers, you should be okay. Speakers, as you might imagine, present a special case.
Where a speaker is placed in a room can make a big difference sonically. Distance from walls, toe in, distance from each of the listeners, acoustical treatments, wall materials and more can all make a fairly substantial difference in imaging, soundstage, perceptions of brightness, etc. The accepted method of combating this is to place speakers so that the pairs are staggered. So if the right speaker is on the outside, the left speaker is on in the inside (rather than having one pair on the outside and one on the inside). Personally, I'm not convinced this is the best solution but it obviously looks to be the best. If I had a research grant, that'd be one of the first things I'd look at. I'd suggest if you have the time, using the staggered method but switching the speakers at least once to see if the listener's perceptions are any different.
I'll talk more about blind/double blind comparisons later, but when you are setting up your room, you'll want to consider whether or not you are going to use some sort of screen to hide the components. For displays, at the very least you should hide the logo though some might be able to tell which is which from the bevel. Amps, receivers, cables, other sources and electronics should all be screened off from the listeners. Speakers, again present a bit of a problem.
Some people believe that using a screen, even one that is designed to be acoustically transparent, attenuates the high end. Again, personally, I'm not convinced. Sure, you may be able to measure a bit of a difference but without credible proof I'm not buying that it is an audible difference. That being said, nearly every speaker manufacturer on the planet makes a grill for their speakers and most make them out of fabric. Remove the grill and put up a screen. While I, like many enthusiasts, do critical listening with the grills off, most of a speaker's duties will involve a grill. If nothing else, you haven't unfairly hamstringed anyone.
The last thing to remember (and this is a biggie) - level match. Nothing will unduly skew a comparison like having one component louder than the other. It is well documented that people associate loudness with quality (and for that matter, brightness with quality). If one component is louder than the other, it will consistently be rated more favorably. As an aside, making sure that that your listeners switch seats during each and every comparison is a way to offset any placement issues that may arise from being nearer to one speaker than another.
