The thing is this. All of the ogg-files I've encoded myself works perfectly. The only ogg-files I have problem with are those that I've bought online. But.. here comes the interesting part.. If I remove all tags from the files I buy and then retag them with my tag-program(MP3BookHelper), they work like a charm.
If we could make a small database of working/not working ogg-files and where they come from, then maybe we could find the source of the evil ones