Hi Frank,
I have the same problem - when CSS optimization is enabled, first page load is extremely slow (like a minute or so), but next load is quick (I use Super Cache).
Beside of that everything works great.
Really awesome plugin, just have to solve this issue and you've got my 5 stars :)
Ondrej