Hi Felix;
Google's Pagespeed Insights is just an "appreciation" of you performance, but real performance (gmetrix or webpagetest.org) is clearly more important.
Kind of surprised your site is that much slower with "inline & defer" though, would you care to share the GTmetrix-report (here or via futtta-at-gmail-dot-com)?
frank