What’s really interesting is to repeat the test on different equipment such as headphones and loudspeakers, or different DAC’s – or even different listening levels.
(The questions are randomised each time you re-load the page – but of course there’s a learning effect that might influence your results)
Bart says:
I’ve done this test using my 15″ apple macbook pro retina and my sennheiser hd 595 headphones.
The volume was set a +/- 70%.
Only on the piano concert and the coldplay song I could identify the uncompressed wav, on all the others I got it wrong and identified them as 320kbs mp3. (I don’t have good ears)
Although it is a very nice test I question the validity of it.
First of all we don’t know the quality of the conversion equipment they used to put the files on the website.
Second, we don’t know if the songs where properly level matched.
Also, all these songs have a different mastering (more or less dynamic range compression) so they will sound inadvertently different, some record companies even use different masters between file formats.
What also bothers me is that they don’t tell us the bitrate of the uncompressed wav file.
This is my subjective (and thus properly not 100% correct) opinion and it would be nice to know what you feel about such tests.
cheers
Bart
Christian Klit says:
3/6 on my H6’s on my iMac.
The difference is really difficult to hear!
geoff says:
Hi Bart,
My opinion is that tests like this are fun to do, and can help people to learn how to hear artefacts (in this case, of the MP3 CODEC). They are “ecologically valid” but not “scientifically valid” – however there is no intention that this be a scientifically valid listening test – it’s merely a demonstration – a bit like attending a wine tasting evening in a local restaurant.
The details of the demonstration like (1) which encoder was used, (2) what the settings of that encoder were, (3) whether the CODEC versions of the signals were from different masterings (I seriously doubt that this was the case in this test – it would make its creation unnecessarily complicated), (4) what decoder you’re using, (5) whether or not your decode is sample rate converting (whether you know it or not), (6) your playback level, (7) the performance of your playback hardware (DAC, amp, transducers, listening room, etc…), and so on and so on… are kind of irrelevant. The point of the article was to make a “big paint brush” demo – not a detailed investigation into the various setup parameters of a LAME encoder.
If you are interested in such a thing, then I suggest getting the disc from the AES called “Perceptual Encoders: What to Listen For” (http://www.aes.org/publications/technical/AudioCoding.cfm). It is also interesting to read the procedural recommendations for running “real” listening tests on perceptual CODEC’s in ITU’s BS. 1116 (https://www.itu.int/rec/R-REC-BS.1116/en).
As to the bitrate of the wav’s – this is not a big deal for me. I assume that they are 44.1/16, since that’s most probable.
Cheers
-geoff
Bart De Bie says:
Thanks for the links, I will certainly check these out.
I hope that the wav is not 44.1/16 otherwise we have a problem.
320kbs mp3 (=uncompressed mp3) is the same as cd quality audio (44.1/16) so if the wav is also of cd quality (44.1/16) then we are supposed to hear a difference between 2 files of the same sound quality.
Not that I am an expert at the matter but this seems a bit strange to me.
Cheers
Bart
geoff says:
now I’m confused. why do you say that 320 kbps MP3 is the same as 44.1/16?
bart de bie says:
I always thought this was the case because that’s what they say at every music review site.
cheers
bart
geoff says:
Ah! This equation (of 320 kbps MP3 with 44.1/16 PCM) is the result of listening tests – but you have to consider that a listening test will not necessarily cover all possible signals that may reveal artefacts. I won’t claim to be able to hear the difference – but it’s not hard to find a signal that reveals the difference between the two. Whether or not that will be audible in the NPR comparison is debatable. However, it gives me an idea of a possible blog posting! :-)
Cheers
-geoff
Casper says:
I did this test a while back and was able to identify all samples except the Suzanne Vega outtake. I listened to my home setup using my laptop as a source.
Laptop = HP Elitebook
Amp = Marantz PM-6005
Speakers = Focal Aria 906
Could have been I was just lucky though and I too had to listen to all the outtakes more than once.