IIS / ASP.NET Recycling “Deadlock Detected”

I like long walks, especially when they are taken by people who annoy me.   - Noel Coward

For some time I’ve been working on an ASP.NET web site issue where almost like clockwork, once an hour, the app recycles and then once again about six minutes later, it recycles again.

It took a long time to find it, but it turns out that “we’re our own worst enemy”. No, it wasn’t some external process like Task Scheduler on the box, running once per hour and hogging threads. It was me having so much fun doing FireAndForget pattern RPC server pings on every search that I was shooting myself in the foot! This is what happens when we put in some new cool “thing” and then six months later, when we start to see problems, we can’t remember what we did!

The ASP.NET “Deadlock detected” shows up in your Application Event log. It’s kind of cryptic, but here’s the general cause: You have a Threadpool per AppDomain. You may have a number of different operations going on (including just serving pages) – all of which use up a ThreadPool thread. Usually, whatever it is that’s happening gets done pretty quickly and so that thread becomes immediately available, and there is never a problem.

But if it doesn’t – you can quickly end up using all of your ThreadPool threads and then what will happen is that the very next Page request that comes in is plum “out of luck” – and your AppPool can thus cryptically recycle.

It could be unclosed connections, sockets, or in this case, the FireAndForget pattern which is a non-blocking way to send off some operation on a background thread.

The problem is, its still a real thread and it comes from the ThreadPool. And if it takes it’s sweet time, you could end up starving yourself out of ThreadPool threads and blow up your app. While the error reported is “Deadlock detected”, my theory is that it may not actually be a deadlock at all  – just simple thread starvation.

There is an elegant fix, the general pattern of which looks like so:

public static void DoPingOMatic(string title, string url)
{
int workers = 0;
int completion = 0;
ThreadPool.GetAvailableThreads(out workers, out completion);
if(workers> 10)
FireAndForget(new PingOMaticDelegate(PingOMatic),new object[] {title, url});
}

What I’m doing above is very simple and happens at the method body level. I simply check to see if there are at least 10 worker threads available before I allow it to attempt to use a Threadpool thread. Of course, you can get more sophisticated (waiting, for example) but in this case, just checking to see that I’ve got at least “X” Threadpool threads before we let – er- rip is sufficient.

Recursion: See “Recursion”

Doing this kind of RPC Service pinging every time you get a search or other request can get you into an unwitting “internet recursion” folly with your website if you aren’t careful. Here’s what I mean: You decide to ping a bunch of RPC Servers every time you get a search on your blog or website.  What happens is that the RPC Ping service (Pingomatic, whatever – see article on eggheadcafe.com) dutifully goes out and makes a request of the url you’ve pinged it with (e.g., http:/www.yourblog.com?q=whatever ). Your blog treats this RPC Service request as a new search, and then pings the RPC Servers with it, and …. you get the picture. Talk about thread starvation!

So, the next time you see this kind of IIS ASP.NET Application Pool recycling, search through your code carefully. The bottom line is, if you have any kind of background thread processing going on, your ThreadPool only has 50 worker threads by default – some of which may already be being used behind the scenes even though your code doesn’t explicitly engage them. It’s a good idea, therefore, to throttle whatever you’re doing -- especially if it is likely to happen many times per minute.