ASP.NET 2.0 Speed Tests: Where's the Fire?
I read a very nicely done piece on MSDN about how the ASP.NET 2.0 pipeline, even though it is more complex than the ASP.NET 1.1 pipeline, has been optimized to be 30 percent faster.
"Great!", I thought. "Let's do a couple of tests."
So I whipped up a page in ASP.NET 2.0 that just outputs "Testing...", and removed all extraneous HTML from the ASPX portion except the FORM Tags. Did the same in ASP.NET 1.1. Pages were substantially identical, being designed primarily to test the pipeline itself - get a request, set up and process the Page, send it out.
I did two sets of tests on each using "Homer", 1 with 100 threads, no delay. and another with 100 threads, a 10X multiplier (the equivalent of 1000 simultaneous requests), and no delay between requests.
In both cases, the ASP.NET 2.0 pipeline was SLOWER, not faster. Both sets were compiled and run in Release, not Debug mode.
I suppose I should consider the fact that it is still BETA, but - SLOWER? Guess we'll be running more tests. . .
Anybody want to comment on this? I have plenty of experience doing web stress testing, having several weeks experience at the MS Testing Lab in Charlotte, NC. This is with IIS 6.0 on Windows Server 2003 64-Bit running on an AMD Athlon x64 3400+ with 2 G RAM.
NOTE: I did have a brief interchange with MS about this and after talking with them, I believe that the discrepancy has a lot to do with ASP.NET 2.0 running under 64-bit WOW. Unfortunately as of right now, the only other machines I've been able to test this on are Windows XP with a 10 connection limit, where it appears that there is less of a discrepancy between the 2 frameworks when both are running under 32 bit IIS. However, I certainly did not notice a "30%" faster pipeline. Having said that, I suppose that anybody really interested in pursuing this should do 2 things:
1) Run different kinds of tests - with both simple and complex pages. Run them on 32 bit only, and then if possible, 64-bit under wow64 and compare the results.
2) Ultimately, wait until RTM since that's really where the "HTTP Request meets the Browser" so to speak.
"Great!", I thought. "Let's do a couple of tests."
So I whipped up a page in ASP.NET 2.0 that just outputs "Testing...", and removed all extraneous HTML from the ASPX portion except the FORM Tags. Did the same in ASP.NET 1.1. Pages were substantially identical, being designed primarily to test the pipeline itself - get a request, set up and process the Page, send it out.
I did two sets of tests on each using "Homer", 1 with 100 threads, no delay. and another with 100 threads, a 10X multiplier (the equivalent of 1000 simultaneous requests), and no delay between requests.
In both cases, the ASP.NET 2.0 pipeline was SLOWER, not faster. Both sets were compiled and run in Release, not Debug mode.
I suppose I should consider the fact that it is still BETA, but - SLOWER? Guess we'll be running more tests. . .
Anybody want to comment on this? I have plenty of experience doing web stress testing, having several weeks experience at the MS Testing Lab in Charlotte, NC. This is with IIS 6.0 on Windows Server 2003 64-Bit running on an AMD Athlon x64 3400+ with 2 G RAM.
NOTE: I did have a brief interchange with MS about this and after talking with them, I believe that the discrepancy has a lot to do with ASP.NET 2.0 running under 64-bit WOW. Unfortunately as of right now, the only other machines I've been able to test this on are Windows XP with a 10 connection limit, where it appears that there is less of a discrepancy between the 2 frameworks when both are running under 32 bit IIS. However, I certainly did not notice a "30%" faster pipeline. Having said that, I suppose that anybody really interested in pursuing this should do 2 things:
1) Run different kinds of tests - with both simple and complex pages. Run them on 32 bit only, and then if possible, 64-bit under wow64 and compare the results.
2) Ultimately, wait until RTM since that's really where the "HTTP Request meets the Browser" so to speak.
Hi Peter,
ReplyDeleteCan you send me an email (scottgu@microsoft.com) with a copy of the test you ran? I can then investigate to try and better understand why you were seeing slower numbers.
Thanks,
Scott
I'm glad I'm not the only person that needs to create their own tests before they believe marketing hype. You get 2 points from a fellow obsessive compulsive benchmarker.
ReplyDeleteBack in my PHP days I ran many tests to disprove all the myths of speed and the preg_ functions (Perl inspride) vs the standard, less featured PHP functions.
It's my tests along with ranting to the manual writers, that I believe led to the manuals recommending the improved functions.
I'd like to also point out that most benchmarks show Windows 64bit to be slower at running 32bit code than normal 32bit Windows. But you already hinted at the fact that the 64bit server was affecting your results.
Peter,
ReplyDeletedid you manage to do any further tests with 32 bit. Also did you get a reply from Scott at Microsoft?
I ask because I have just done some simple speed tests myself with a near-real-world site and the speed from 2.0 was disappointing. I gave it a good chance by making sure all the pages were pre-compiled (using aspnet_compiler) in advanceand it was still slower than 1.1 on initial page hits! The only place it did outperform was when there was a slow DB response, so not really a good metric.
I have been putting the speed issue down to the framework being beta, but I would be interested to hear if you have got any further
Cheers
Richard
I did correspond with Scott, and we more or less decided that is was due to the 64 bit running 32 bit mode. Unfortunately I haven't been able to do any real 32 bit testing except on Windows Xp and there we have the 10 connection limit. I wouldn't be concerned about it at this point.
ReplyDeletehave you enabled tracing in the page to determine if lag is occurring at a specific stage?
ReplyDelete