Why you should always Dispose your HttpWebResponse objects

I very recently came across a "hard to solve" issue.


Well, it certainly felt like one...


The problem showed itself via all my HttpWebRequest's blocking - they didn't timeout, they didn't exception - they just blocked.


I couldn't work out what was causing this - and the situation was complicated because the servers I were talking to were quite complicated - they were PHP apps running on Apache, but hiding behind nginx running node.js middleware.


The one thing I could see was that just before things went wrong, these servers were returning "chunked" HTTP 1.1 responses.


The problem also looked more complicated because when I used Fiddler to try to debug the issue, then the problem disappeared (this is because Fiddler alterred the way the responses were received - it doesn't quite mirror responses perfectly - for chunked responses it concatenates them)


Some research led me to various blog posts, stackoverflow answers, etc which all suggested issues - e.g. things like http://blog.degree.no/2011/06/webclient-httpwebresponse-problems-with-chunked-transfer-encoding/


I wrote a test harness, got the case reproducible and then reached out to some friends in MS to see if I had found a genuine issue.


I had, of course, not found one....


Instead, one of the MS guys answered quickly and from his gut:

my gut feel is that you aren’t emptying the body of the first request when using HttpWebRequest.  Can you ensure you are reading the entire body of the first request before sending the second one with HttpWebRequest?

I checked all our calls....


... and in one of them we weren't reading the response at all.


... so I added a `Dispose()` to the response there (instead of reading its stream)


... and that caused the Http stack to start behaving again


... :)


So, it seems like what was happening was that when this "Chunked Transfer Encoding" occurred, then the HttpWebRequest stack got "stuck" waiting for code to read the incomplete response - and this prevented future calls from working.


So the moral of the story is: Dispose() your HttpWebResponse objects - or, if not, at least read them in their entirety.


It's also worth noting: this problem didn't really show up when we switched the code to the newer HttpClient - so, if you are looking at new code, then please do use HttpClient instead!


Thanks especially and hugely to @dsplaisted and @theesj for pointing out (using just their guts) the error. Hopefully by posting this here, it will help others....