The hard part here, is what do you set a timeout too? If the site is down after 10 seconds? 30 seconds? Its a difficult one to work out the best answer for. Any delay will make your site look bad whilst that feed is attempted to be retrieved.
No easy answer, so I just disabled RSS feeds from my sites