Hmm ... Especially with WP Super Cache your site should fly, so something is not right.
Do you have waterfall charts (as produced e.g. by webpagetest.org) with and without AO that I could have a look at and compare?
Feel free to mail me at futtta at gmail dot com if you prefer not to give that kind of info in public.