I don't think you'll get a definitive answer to does slow respone time affect rankings because during the time taken for the experiment to run its course (weeks or months depending on your site) then Google will have updated its algorithms many times.
Slow downloads are a killer if you are doing Google Adwords and it was announced that it could impact on your seo rankings a couple of years back, and at the same time Google started issuing very specific advisories on how you can speed up your site in the webmaster tools and later via analytics.
My gut is that anything more than 5 seconds per page average can impact on your bounce rate(s), which is bad for rankings (indirectly) but to impact on your Google quality score it would have to be quite a severe problem because I have slow sites that rank quite well for sompetitive terms c. 10 s per page.
If someone has a page that ranks and can afford to fill it with shite, do a fetch as Google bot and get back to us, then that might help seal this one
