As for how to detect XHTML, that may be a can of worms if you don't use the doctype
The detection in HTML-minify.php (which is not mine, I just use it in AO) uses the doctype, but can be overridden to enforce XHTML and that's what the new filter ('autoptimize_filter_html_forcexhtml') does. So I'm leaving the can of worms very much closed ;-)
Anyway, thanks for looking into this. The more i think about it the less i think that html standards matter any more in html5, as long as it's html of some sort.
Standards are important and validation is a tool (nothing more) to check adherence to those standards, but one should not be too ... religious about them ;-)
frank