Ogg Vorbis, Lame Mp3, WMA, FLAC - PepsiTest

Read this threadie and got inspired to challenge my neuroses w. some science and created a blind-test... and frankly, to my surprise, I couldn't tell the difference.

Thought I'd share it w. you to see if the results were replicable in a large group, thus...


Ogg Vorbis, Lame Mp3, WMA, FLAC - PepsiTest

Objective:

1) To determine if there's any possibility to subjectively distinguish between the three most popular lossy formats (Ogg Vorbis aoTuV, Lame Mp3, WMA 9,2) and a lossless format (FLAC).

2) To determine which format has a higher likelyhood of being favoured in a blind-test.

3) To potentially rid some of the neuroses associated w. "lossy" formats.


Method:

All songs were compressed from their original wave using the respective codec-library* for each format - retagged to hide any traces of the original tag - trimmed to stay within copyright boundaries and keep the filesize down - and finally converted (FLAC) to make the files undistinguishable from one another.

(This procedure allows for keeping any format-specific artefacts intact whilst hiding clues of the format from the test-subjects... ehhr... mainly you)

*Codecs and compression-settings used:
LAME Mp3 v. 3.97 beta 2 - V1 --vbr-new- (target bitrate=210 kbps)
Ogg Vorbis aoTuV beta 5 - q6.56 (target bitrate=210 kbps)
WMA v. 9,2 - VBR 135-215 kbps (target bitrate=approx. 210 kbps)

(These settings produce "close to similar" outputs and album-filesizes between eachother and are considered to have a high tranparency to size-ratio. Output and filesize may vary on individual songs though)


How to participate:

1) Download the rar-file from rapidshare or megaupload.

2) Place the "PepsiTest"-subfolders in your DAP (if you place the entire folder you won't get the albumart and some samples may magically disappear).

3) Turn off any sound-enhancements in your DAP and set your EQ to flat (+-0 on all levels).

4) Listen thru the albums and artists and choose which sub-sample you prefer on a scale of 1-4 (1 being the best, 4 being the worst).

If you can't make a distinction between two sub-samples, give them an equal value and skip the next number on the scale. Making instinctive decisions works fine and will give an overall representation of the average user. For more "precise" (slightly less random) opinions you should however try to section the samples into key-elements (ie: cymbals, echoes, fade-outs, a specific note, a guitar-string etc... in various parts of the sub-samples - and then compare them).

5) When you're done... publish your results along with info on which player (or if it's on your computer...) and the headphones you use.

Example on how to rate:

A1=3
A2=4
A3=1
A4=2

B1=4
B2=1
B3=2
B4=3

Example on how to rate when impossible to distinguish between two or more formats:

A1=1
A2=1
A3=3
A4=4

B1=4
B2=1
B3=1
B4=1

C1=1
C2=2
C3=2
C4=4

(There are seven different samples - each divided in four sub-samples, one 35-sec. trim for each codec. The test will take you approx 30 mins. or more depending on your level of ambition. You may submit test-results for as many or few samples you like as long as all the sub-samples in that specific sample are included)

Note: I've made a conscious decision to include clips from wave-samples with various degrees of statics and natural artefacts (present on the master itself) to make it a fair and average cross-section (in case you're wondering).

I'll leave the test open until next Sunday 29/7 (or until at least 10 people have done the test) before I publish which format is which along with some useful statistics and as a nice little bonus (a treat)... I'll be handing out blow-jobs to whoever finds the authentic FLAC in all samples.

Cheers/

Milo Fo







PepsiTest2 (128 kbps)

Info - codecs and compression-settings used:
LAME Mp3 v. 3.97 beta 2 - V5 --vbr-new- (target bitrate=130 kbps)
Ogg Vorbis aoTuV beta 5 - q4 (target bitrate=128 kbps)
WMA v. 9,2 - VBR 85-145 kbps (target bitrate=128 kbps)

@ Rapidshare

@ Megaupload

How is this a blind test?

If I were a born again, bow to the east 4 times, take no prisoners, all .wma users are wankers "FlacHead" won't I be influenced by being able to see the file types?

Originally Posted by /\/\@D/\/\@/\/

How is this a blind test?

If I were a born again, bow to the east 4 times, take no prisoners, all .wma users are wankers "FlacHead" won't I be influenced by being able to see the file types?

They're all converted to wav --> FLAC after original conversion. Since FLAC isn't a lossy format it won't alter the original artefacts, merely compress the wav.

In short... no... you won't. They're all FLAC.

Here are my results (with Dane-Elec miniPlayer S2.002.3 & Panasonic RP-HTX7) :

A1 1
A2 2
A3 3
A4 4

B1 2
B2 4
B3 1
B4 3

C1 3
C2 2
C3 4
C4 1

D1 4
D2 2
D3 3
D4 1

E1 4
E2 3
E3 2
E4 1

F1 2
F2 3
F3 1
F4 4

G1 1
G2 3
G3 4
G4 2

Sincerely, if there are differences between the tracks, they are so minor...

and these are the results from Germany :

la France : 0 points. :D

okay, it was really difficult since you have used very high bitrates. Of course all of them sounded quite similiar. My only way to contribute something serious was to use my instincts, and zapping from version to version to "feel" the best, then the next best, and so on. When I was almost finished, I realized that you said it's possible to make ties between contestants, but in my amusement for this pepsi test - I forgot it. So I forced myself to give all rankings from 1-4. But to be honest, my number 1 was often easy to find, but 2-4 was .. not rational in any way.

A1 3
A2 4
A3 2
A4 1

B1 1
B2 4
B3 3
B4 2

C1 2
C2 4
C3 1
C4 3

D1 1
D2 4
D3 2
D4 3

E1 1
E2 4
E3 2
E4 3

F1 2
F2 1
F3 3
F4 4

G : I have zapped between them for several minutes but I couldn't hear ANY differences, even not small ones. But then again, it might be the crappy recording.

I used M6 with v-moda vibes, everything turned off.

without having taken the test my options about the formats are these:

Lame Mp3: if used on a higher bit-rate and VBR setting produces sound quality that's very good for portable devices. NO form of lossy format is suitable for use with a descent full blown home audio set

WMA: Microsoft's MP3: while not noticeably worse or better there just doesn't seem to be much use for this format as Mp3 will always play and is the standard format and WMA does not offer anything extra whatsoever.Again, lossy so not suitable for home stereo use

OGG: Sounds slightly better than Mp3 or WMA. However, not supported much. therefor an Mp3 VBR at a higher bit-rate is more useful. A shame OGG isn't used more often. Id like to see OGG with the popularity of Mp3. Lossy so not suitable for home stereo use.

FLAC: nice sound quality but on an mp3 player hardly noticeably better then a well encoded MP3 VBR. The slight improvement in sound is imo not worth the 5 fold in storage space it takes. FLAC is excellent for back up purposes and it would be great if homes stereos supported FLAC as its loss-less and so will sound good on even a home stereo.
For mobile use however FLAC files are too large.

Originally Posted by BobbyQ

and these are the results from Germany :

la France : 0 points. :D

Hello from Sweden!

Germany: 0

France: 0

Originally Posted by Bobby Q

okay, it was really difficult since you have used very high bitrates.

Yeah.

I thought about using a lower bitrate (were the difference would be more apparent), but these setting produce about ~75 MB for your average album (and would therefor be interesting in terms of replacing FLAC for us die-hards).

Originally Posted by Bobby Q"

G : I have zapped between them for several minutes but I couldn't hear ANY differences, even not small ones. But then again, it might be the crappy recording.

Neil Young - Live at the BBC 1972.

Threw it in there cause acoustic gigs are often used as an argument against the lossy formats... when in fact, as you say, the recordings are often less than great and impossible(?) to tell the difference from.

Anyways...

Good of you guys to do the homework(!).

Without revealing anything too much it's an interesting read at my end and you'll be amazed when I drop the key.


Be assured though - none of you guys could do any worse than me. I failed horribly (and you would think I'd at least stored some of it in my memory while doing the work). I got one (1) FLAC right... and it was more of an accident than anything else (I shuffled the songs and couldn't pick it out of the crowd a second time).

Originally Posted by Job

[lossy=dap, lossless=stereo]

It'd be great if you could try the samples on your stereo. I've been using the same arguments and it would be nice to see if there's any substance in that claim.

hehe, this test is a nice idea and I am really curious to know if some of my 1s have hit the FLACs. I apparently had a totally different opinion on the test tracks than xx-c.
I would be interested in another test as well that has 128 kbit samples, since this is widely agreed to be close to the CD.

Originally Posted by BobbyQ

hehe, this test is a nice idea and I am really curious to know if some of my 1s have hit the FLACs. I apparently had a totally different opinion on the test tracks than xx-c.
I would be interested in another test as well that has 128 kbit samples, since this is widely agreed to be close to the CD.

A 128 kbps version. I re-used the same wavs to save some time (I guess it could have some value in being able to compare the samples as well, but mostly... I'm lazy ):

@ Rapidshare

@ Megaupload


---PepsiTest2---

Info - codecs and compression-settings used:
LAME Mp3 v. 3.97 beta 2 - V5 --vbr-new- (target bitrate=130 kbps)
Ogg Vorbis aoTuV beta 5 - q4 (target bitrate=128 kbps)
WMA v. 9,2 - VBR 85-145 kbps (target bitrate=128 kbps)

Originally Posted by MiloFo

It'd be great if you could try the samples on your stereo. I've been using the same arguments and it would be nice to see if there's any substance in that claim.

well.. you can simply try by connecting your mp3 player to a decent home stereo... a 265 vbr mp3 will sound flat and empty while the CD original where you got the mp3 from (WAV = equal to FLAC) sounds rich and full...

losy is fine for portable use... not home usage

Originally Posted by Job

decent home stereo

define "decent home stereo" ! On mine I don't hear the difference between a well coded mp3 (>192) and the CD.

Originally Posted by Job

well.. you can simply try by connecting your mp3 player to a decent home stereo... a 265 vbr mp3 will sound flat and empty while the CD original where you got the mp3 from (WAV = equal to FLAC) sounds rich and full...

Cool. Then it'll be no prob whatsoever for you to pick out the FLACs. :D


Seriosly... thx for the input...and I do agree with your op. (same procedure I use - FLAC for archiving/wav-conversion, a mixture of FLAC and Vorbis for my Meizu)...

But with that said...

It is however impossible for me to know if the tinny sounds are my mind playing tricks on me... or if it, in fact, really is a tinnyer sound.

After having experienced occasions where I've played Vorbis... thinking it was FLAC... I'm not so sure about the latter anymore. It could be my mind telling me what to hear... the same way it tells me what to taste and what to feel.

And the way it works... you have to challenge those conceptions in order to shift senses (ie: taste a food you don't like) before your perception tags along.

Thus...........



It would be really nice if you could muster the energy and go out on a limb on this, but I'm not gonna hold a gun to your head.

Originally Posted by MiloFo

Cool. Then it'll be no prob whatsoever for you to pick out the FLACs. :D


Seriosly... thx for the input...and I do agree with your op. (same procedure I use - FLAC for archiving/wav-conversion, a mixture of FLAC and Vorbis for my Meizu)...

But with that said...

It is however impossible for me to know if the tinny sounds are my mind playing tricks on me... or if it, in fact, really is a tinnyer sound.

After having experienced occasions where I've played Vorbis... thinking it was FLAC... I'm not so sure about the latter anymore. It could be my mind telling me what to hear... the same way it tells me what to taste and what to feel.

And the way it works... you have to challenge those conceptions in order to shift senses (ie: taste a food you don't like) before your perception tags along.

Thus...........



It would be really nice if you could muster the energy and go out on a limb on this, but I'm not gonna hold a gun to your head.

I understand what you mean, sometimes it might actually be your mind playing tricks on you In that case the differences are too small to be interesting
I guess if you encode the Mp3 at a high enough bitrate you wont hear the difference much
however, this defeats the porpose of mp3 because an mp3 encoded at a high enough bitrate will approach FLAC size and quality... in that case...why not use FLAC and go lossless all the way?
Mp3 was meant to have smaller files and more compression

On a higher end system (top of the line Yamaha system with good speakers) i can clearly hear the difference between Mark knopfler original and one encoded in 265 vbr mp3.

in the end its a trade off between quality and size
for me 265 vbr mp3 for protable use if fine
at home however i use lossless or cd/dvd original

and there's more to quality than just the bitrate. Just test this :
encode with lame @320 kbit/s, encode again with 320 kbit/s, repeat 4 or 5 times. It starts sounding awful ..
With flac you can convert, convert and convert and it stays always 100% the same.
Converting once with a lossy codec is alright, but not more than that.

Originally Posted by BobbyQ

and there's more to quality than just the bitrate. Just test this :
encode with lame @320 kbit/s, encode again with 320 kbit/s, repeat 4 or 5 times. It starts sounding awful ..
With flac you can convert, convert and convert and it stays always 100% the same.
Converting once with a lossy codec is alright, but not more than that.

thats very correct
lossy to lossy conversion should be avoided at any cost cause it quickly kills the quality of your sound
this doesn't go for FLAC or WAV to lossy of course

besides stuff like that the quality of the encoder itself also has great influence on sound quality

anyone who has ever used the crappy mp3 encoder in Itunes can vouch for that

Originally Posted by Job

I understand what you mean, sometimes it might actually be your mind playing tricks on you In that case the differences are too small to be interesting

Ok. I'm not gonna argue w. you - you do as you please.


For the record though:

- It was a two-step conversion/compression wav>lossy>FLAC in the case of the lossy-formats and a one-step compression, wav>FLAC, in the case of the lossless. Whatever distortions may have occured in the lossy formats (:P ) should only make it easier to identify the genuine FLACs.

- My wav-extractor of choice is EAC due to it's ability to fix discread-errors, but you would have identical results using CDex or Foobar given there were no discread-errors. The codec-libraries I've used are stated in the info.

- If you don't fancy ~210 kbps there's a ~128 kbps there as well (courtesy of Bobby Q). ~265 kbps is however a new one to me since I use Hydrogen Audios recommended settings.

- There is no 'trick' to it (i.e: them all being FLACs or similar) - all the samples are legit... there is however one sample which has a different conversion-method in two of the sub-samples to try the lossy>lossy-theory.

Also, here's the key:

@Rapidshare

@Megaupload

... so you guys know it wont get altered afterwards.

I'll release the password coming next Sunday, see you guys then.

Cheers.

[QUOTE=MiloFo;27442]Ok. I'm not gonna argue w. you - you do as you please.


sorry if my post wasnt clear
i didnt mean to say its juts your mind that plays tricks on you but "your mind" as in general, for everyone, including me, you and everyone else

I wish there were more people participating in this interesting test.

Originally Posted by BobbyQ

I wish there were more people participating in this interesting test.

Yeah. It'd been fun to see if anyone actually could tell the difference.

Anyways...

I asked a friend of mine to swap the samples so I could do the test too, so here's my results:

M6 2003.2b - Sony MDR-E931 & Ultimate Ears SuperFi3 (the test was done with my Sony's since they're my pick of choice, but I tried it with my UE's as well - apart from the sore ears I couldn't tell much difference though)


A1 - 4
A2 - 2
A3 - 1
A4 - 3

B1 - 1
B2 - 4
B3 - 3
B4 - 2

C1 - 4
C2 - 3
C3 - 2
C4 - 1

D1 - 2
D2 - 1
D3 - 4
D4 - 3

E1 - 4
E2 - 1
E3 - 2
E4 - 3

F1 - 4
F2 - 3
F3 - 2
F4 - 1

G1 - 4
G2 - 2
G3 - 3
G4 - 1

H1 - 3
H2 - 1
H3 - 4
H4 - 2

I1 - 1
I2 - 3
I3 - 4
I4 - 2

J1 - 3
J2 - 2
J3 - 1
J4 - 4

K1 - 1
K2 - 3
K3 - 2
K4 - 4

L1 - 2
L2 - 4
L3 - 1
L4 - 3

M1 - 3
M2 - 2
M3 - 4
M4 - 1

N1 - 3
N2 - 4
N3 - 1
N4 - 2


There's no use in copying my results btw.