RSS--> JSON--> On-Demand Script Tags and ASP.NET [AJAJ]

"You can observe a lot just by watching." --Yogi Berra, Berra's Law

I've been experimentng with some server side code that I already had, namely, a method that will get any RSS Url and turn it into a DataTable.

What I've done with that is to add the ability to convert that DataTable into a JSON data object, wrapped in a callback. With that done, it is a relatively simple matter to create a javascript source file that will "inject" not only the call to the aspx page that serves as the dynamic javascript "src" generator, but also the the UI for the DOM to display the results, and the callback method itself. It really doesn't take a lot of code to do this; in fact I used Jason Diamond's Manager class from the Anthem.Net framework to generate the JSON from the datatable more or less "out of the box".

The neat thing about this sort of approach is that there are now so many services and sites that return content and news via RSS that it's becoming almost ubiquitous. Even if they don't return RSS, you can still use Libraries like HtmlAgilityPack to parse out what you want and still put it in the format of a DataSet or DataTable. The end goal might be to create a library that is capable of retrieving or even web scraping most any kind of content whether RSS or not, and massaging it into a consistent "Delivery Vehicle" that always behaves the same.

The Hypocricy of Web Standards

Let's look at the hypocricy of so -called "Web Standards" for a moment. We are gonna tighten up security on the XMLHTTP object in modern browsers to prevent it from requesting content from any domain other than the one that the page has been requested from, yet we freely allow a Javascript <Script> element to access content from anywhere! Does that make sense? Sure it does, in a strange way: If you were consistent, and did the same thing with script tags, Google, Yahoo and every other contextual advertising service would go APESHIT! So, go figure.

Developers all over, in virtually every discipline from ASP.NET to PHP to Ruby, are using "AJAJ" - Asynchronous Javascript and JSON. Probably it should be referred to as ODJAJ - On - demand Javascript and JSON. But then, we have enough acronyms already, wouldn't you say?

This is an experiment to play with the concept. You can enter any search text in the "SEARCH" textbox below, and either TAB out or click outside the textbox and you should get your search results. This will work in most browsers, including IE 6 and 7 and Firefox.

In modern browsers, XMLHttpRequest is constrained by what is called "Same Origin Policy". This allows the interface to only connect with the server that delivered the base page, and the rule deals with some common, long-standing security flaws in web architecture. If this policy were not in place, then the user could be harmed by XSS (cross site scripting) attacks.

The Same Origin Policy frustrates these attacks, but it also frustrates a larger class of legitimate uses - developers like you and me wanting to provide quality, performant "Web 2.0" - like applications. It should be possible for a script in a page to access data from other servers without compromising the security of the user or his organization. In particular, it should be the USER who gets to make this choice as an informed choice, NOT for it to be made in a sledgehammer fashion for him by the browser instead!

Surprisingly, the Same Origin Policy does not apply to scripts and that is why so many developers have begun to dynamically generate <script> tags to connect to any server. The server sends back a script which delivers some data. Unfortunately, this script runs with the same authority as a script from the originating page, allowing the script to steal cookies or directly access the originating server, or do virtually any of the things that the flawed "Same Origin Policy" attempts to thwart. This is unsafe.

Douglas Crockford, the originator of the JSON data format, proposes a standardized approach to solve this problem here. I like Crockford's approach, although I do not believe it will ever be implemented.

What you see below this textarea is completely the result of this single script tag:

The point of all this is that by using JSON, combined with the option of an on-demand-script tag (e.g., adding a script tag and its dynamic src property to the DOM programmatically) we get around the lousy cross-domain security restrictions of the XMLHTTP object in modern browsers. When the script tag is added to the DOM, the script is automatically executed in the client browser. You are welcome to stick the above script tag into a page of your own and experiment.

For those who are still "JSON-Challenged", JSON is built on two structures:

A collection of name/value pairs. In various languages, this is realized as an object, record, struct, dictionary, hash table, keyed list, or associative array.
An ordered list of values. In most languages, this is realized as an array, vector, list, or sequence.
These are universal data structures. Virtually all modern programming languages support them in one form or another. It makes sense that a data format that is interchangable with programming languages also be based on these structures.

In JSON, they take on these forms:

An object is an unordered set of name/value pairs. An object begins with { (left brace) and ends with } (right brace). Each name is followed by : (colon) and the name/value pairs are separated by , (comma).
An array is an ordered collection of values. An array begins with [ (left bracket) and ends with ] (right bracket). Values are separated by , (comma).
A value can be a string in double quotes, or a number, or true or false or null, or an object or an array. These structures can be nested.
A string is a collection of zero or more Unicode characters, wrapped in double quotes, using backslash escapes. A character is represented as a single character string.

Except for a few encoding details, the above entirely describes the notation. Unfortunately, people are already referring to JSON as a "language"; (including Crockford himself, on his site). It is not. JSON is a javascript - parseable data format for encoding objects. A programming language consists of the grammar used to construct a computer program; you cannot create a program with JSON, you can only describe objects that can be interpreted by Javascript, which is a programming language.

Please understand that I sometimes use my UnBlog as a sandbox for my experiments. I got one comment on the last post complaining about this that and the other thing; it had nothing at all to do with the subject of my post. If it doesn't render correctly in "FireFart" or whatever browser you are using besides "Internet Exploder", try to get over it, OK?

I'll have more soon, including some nice ASP.NET generic source code for a sample "Serving" page, an HTML form that will convert your script creation into all the "document.writeln" statements to create your script src file including any DOM elements you want and a Javascript callback method, and simple instructions on how you can "roll your own". Probably, I'll post this as an article on eggheadcafe.com and when I do, I'll update this post with the link.


You people don't get it, do you? I don't know why you would bother going through all the CAPTCHA stuff to post your "Mortgage Rates Online" crap. It only takes me a second to delete them. Get a life, Scum!


Coding Standards Redux

I complained today about having to work on two web projects where there are now not one or two, but FOUR separate generic SQL Server Data Access classes, all of which do essentially the same thing!

They all do it differently, because individual developers have taken it upon themselves to "roll their own" without consulting with the team about best-practices coding standards and what we already may have, or not have, to work with.

In addition, developers have "cobbled in" references to things like Anthem.NET, a WAV COM Library, and other dependencies into these Data Access assemblies that have NO BUSINESS being in a generic data access class.

When I worked at a banking software firm back in early 2001, doing our first big .NET project, we had 17 developers in two teams working together. We all sat down in the conference room at the beginning and argued, and we all agreed to use the "SqlHelper" class from the Microsoft Data Access Application Blocks sample code for our basic "talk to SQL Server" needs. We chose it because it (at least, at that time) exemplified one of the most important fundamental principles of framework design: it had been constructed by focusing on a set of common usage scenarios to enable power and expressiveness, and especially, ease of use. The learning curve, even for beginners, was fast, the class was well-documented, and it offered features (such as automatic static caching of SqlParameters) that I wouldn't have thought of.

I cannot describe how much easier that made things. Everybody was doing Sql calls the same way; you always knew exactly what their code was doing, and you never had to look for assemblies that you had missing references to, because THERE WAS ONLY ONE.

Now if you wanted to build a specialized DAL, you could still have it use the SqlHelper class under the hood for your basic data access, and so you weren't restricted in any way. It saved us literally dozens of hours of developer time and, you know what? I still use it today (even the more advance Provider model version) - five years later.

But -- It doesn't have to be "SqlHelper" - it can be any basic data access class you want. The point is, it should be the same for everybody, and if it needs to be able to do something new, then everybody should get together on that before making the change. And the framework should be designed properly. If you want to study good framework design basics for class libraries, a good place to start is Framework Design Guidelines by Krzystof Cwalina and Brad Abrams.

Man, I hate to sound like a broken record, but really - this is such basic stuff, it just blows me away how people don't seem to think that this is important. It's VERY IMPORTANT! We do not do our development in "a vacuum" -- even if we are the only developer on a project. Other people have to be able to use our code!


Lessons Learned: Selective Reading and Human Nature

Psychologists will tell you that humans "see what they want to see", "hear what they want to hear". They also "read what they want to read".

That's why police identification lineups of suspects are notoriously flawed. When I publish my articles on our developer site, eggheadcafe.com, I almost always put one of my little article-specific discussion boards at the bottom of each article. I do this because I want to encourage feedback. It gives the reader a sense that if they have something to say, they are free to do so, and I've gotten a lot of good suggestions from some of their posts, many of which have helped me to become a better writer. Once a month or so, I run a little cleanup SQL Script that lets me get rid of the relatively few "ASDF" "ASDF" posts left by mindless curiosity seekers (I say mindless because they never quote Shakespeare or Chaucer, it's just "ASDF").

On one recent article, "ASP.NET: Obfuscating the QueryString", I got a surprising number of negative comments. By way of explanation, I had answered a newsgroup post at the MS C# Language group to somebody who wanted to pass a "connection" on the querystring to a new application, and I had answered that they could pass the connection STRING, and use that on the receiving end to recreate the same connection. Another MVP posted that it wasn't a good idea to try to pass connection strings on the querystring for security reasons, which is of course, correct.

I got interested in the concept and wrote this article about obfuscating the querystring. The fact is, you couldn't pass a connection string as a querystring item anyway because they contain "=" equals signs which would totally bomb out your querystring.

So after some experimentation I came up with an ASCII-Hex / Hex-ASCII arrangement that allows you to pass whatever you want on the querystring, obfuscating both the names and the values, so all you would see is something like:


That's not encryption (although there is nothing to stop you from say doing 3DES encryption and then obfuscating the result). But, it is sufficient to keep your "stuff" from prying eyes, and it has a number of uses that don't necessarily involve anything to do with sensitive data that shouldn't be on the querystring. I took care to write a whole paragraph in this article explaining in plain English that it is not a good idea to put something like a connection string on the querystring or any other sensitive data for that matter. In fact, in this one paragraph, I actually made this point not once but two times:

"If you truly need strong encryption, my suggestion is -- don't use the querystring at all; find another way." and,

"But the bottom line is, it's best not to put sensitive data on the querystring at all -- there are other ways to pass the data."

One would think that after making this point not once but two times very clearly in plain English that I DO NOT recommend puttiing connection strings or similar sensitive data on the querystring, that the point would get across, right? Nope! Regardless, there were several nasty comments from readers who pretty much had decided in advance (although possibly it was subliminal) that no matter what I wrote, they were going to "put me down". Some Examples:

"Encouraging bad solutions"
In what scenario could you possibly want to pass a connection string via querystring? This seems so ridiculous to me, and justifying it by obfuscating the querystring is worse. I would expect more from an MVP with 20 years of experience.

"Decent idea for non-sensitve data, but otherwise you're a joke"
wow, what a joke of a mvp.

"You are actually a complete joke (Part II)"
Hi, How is it possible for an MVP with a PH.D. to have so little regard for security. This is the most rediculous [sic] article I have read in years. Regards C.M.

There were a much larger number of positive comments, some "Thank you"''s and a few that felt it necessary to respond to the nasty people by defending me, the poor author. Plus, I had some fun jabbing in a couple of retorts of my own. But, I wasn't nasty; I was respectful.

The Bottom line? People approach certain situations in print, as they do in social environments, with certain predispositions that prevent them from seeing -- or at least comprehending -- what is in front of their own eyes. We as writers and developers need to be mindful of this so that these contextual choices don't inadvertently throw off the reader onto a "slanted perspective" that prevents them from absorbing the intended meaning or information.

The next time I write an article about a subject like this, you can be sure I won't use a connection string as the example case, even if it represents the useful purpose of bringing the article into the proper evolutionary context, as it did with the above-captioned article.


Is your development process "Broken"?

Recently I was asked to add something to an internal web project that has gone through a number of iterations and also a number of developers. I spent the better part of an afternoon, and part of the next morning just getting everything out of Sourcesafe to build. I complained, because most of this was due to either developer lack of knowledge, lack of professional standards, or just plain "don't care". This is a list of some of the initial recommendations I made when asked:

List of items that should be fixed with development process:

1) Developers should always arrange their projects, solutions, and the way they are source controlled with a view toward making it easy for new developers to be able to begin work on the solution with the minimum amount of hassle. This includes:
a. Put 3rd party assemblies in the solution as Solution Items and have them checked in with the solution so that people don't need to set mapped drives to shares containing libraries. These can be easily updated and developers can Get Latest to stay in sync.
b. Make all references PROJECT references. If the developer already has this project from Sourcesafe on his hard drive, it will synchronize automatically from Sourcesafe, they will not need to get a separate copy. Keep all the PROJECTS that are used WITHIN THE VISUAL STUDIO SOLUTION whenever possible. Project references allow you to debug through different projects in the solution, and they also enable you to keep changes made by members of your team synched-up.
c. Name your solutions appropriately in Sourcesafe so that new developers will not mistakenly get the wrong version or one that is not being used.
d. Name your Projects and the namespaces of your classes in a project consistently. don't have namespace Foo.Bar in project named Blat.Bork, and then have another project in the solution that looks like "Foo.Bar"!
e. Do not alter the default build outputs of a project. Project output is normally directed to the /bin/debug or /bin/release folder.
f. Correct any and all compiler errors and warnings, even if they do not prevent successful compilation. If the compiler says Not all code paths return a value then FIX IT before checking in your code. If the compiler has a warning about an ambiguous cast, FIX IT. If the compiler says a variable was declared but never used, then FIX IT. There is NO REASON why a professional developer cannot have a Solution that has NO WARNINGS after it is built.
g. Check in all code but only if it builds and has been tested first. Use comments so other developers can see what changes you did. Did you test to see if OTHER PARTS of the Solution haven't been broken by your changes? You'd better!
h. Set all Assembly versions to in the assemblyinfo.cs file, not 1.0.*.*. This prevents overwrite errors because of automatic build number incrementation. If you really have a New Version then hard code it in.

2) A new developer should be able to tell Visual Studio to open Solution from source control, download all the projects, compile and build the project within 20 minutes. If this cannot be done, either your source control arrangement is not acceptable, or your developer team has not been educated in the above concepts, or both.

3) Whenever possible ambiguities and misunderstandings about projects, naming conventions and other similar items can cause problem, it is incumbent on each developer to think about the other people who may need to work with their code and take appropriate steps to ensure that their development experience goes smoothly. Usually this is simple common sense, and asking the question. "What does this affect?"

There is nothing that PISSES ME OFF more than a shoddy development team effort that reflects a lack of planning, discipline and basic rules of the road. If I were the boss and you did this stuff on my team, YOU WOULD BE GONE.


And Another Thing: Apple, IPod, ITunes and DRM

I probably never would have gotten to the point of this rant if it weren't for the totally insulting experience I had to go through to get the latest version of Quicktime for a program that is completely unrelated to digital music.

Apple now package ITunes with Quicktime, and there doesn't seem to be a legitimate way to get around the 35 MB download of Quicktime with ITunes piggybacked onto it. Or should I say, ITunes with a "free bonus" of Qucktime?

So you go to install it, thinking you'll see a checkbox that says "Install ITunes" that you can uncheck, right? Nope. Those BASTARDS are gonna make you install the WHOLE FRICKIN' THING. Only when it's done can you finally go into Control Panel, Add/Remove, and get rid of their stupid bloatware ITunes thing and hopefully, QuickTime will be left alone.

What's wrong with this? I'll tell you what's wrong. I don't like software being shoved down my throat, that's what's wrong! I believe the consumer should be treated like a real human being, and given a choice. Apple calls iTunes "revolutionary" but record companies are simply using the service to force the same exploitive and unfair business model onto a new medium. iTunes AAC files don't sound as good as CDs. AAC is a "lossy" compression format, high bitrate MP3's have much better quality. So does WMA.

Apple says iTunes is "better than free" because it's "fair to the artists and record labels." That's BULLSHIT! Apple gets 3 times as much money as musicians from each sale. Apple takes a 35% cut from every song and every album sold. Record labels receive the other 65% of each sale. Of this, major label artists will end up with only 8 to 14 cents per song, depending on their contract.

If you really want to support the musicians you love, the best way to begin is by downloading the song for free on a filesharing network. Then send them what you want to give, whatever you feel is right, no middleman. Weed ( weedshare.com ) is a new system where songs can be distributed on p2p networks but must be paid for after 3 plays.

There's even an open-source project called hymn that gives you your rights back, enabling you to unprotect Apples DRM-infested music files that you paid for. French lawmakers voted 296 to 193 in support of a law that would stop Apple, plus any other firm selling music downloads, using proprietary software to limit what people can do with tracks they have bought.

You couldn't pay me any amount of money to get an IPod or ITunes and pay Apple money for this DRM-infested corporate greedware masquerading as a music service. I have an AudioVox SMT5600 with Windows Media Player 10 on it, and a 2GB mini-sd card that holds all the high quality WMA or MP3 I want. I can burn them to a CD, whatever I want. And the phone, which has Outlook, Tasks, and custom .NET Compact Framework software that I wrote myself, cost less than a comparable IPod. It plays video too.

Get real, Apple. BTW, here is the link to their "Without ITunes Quicktime." I found it quite a bit later. It's still 21 megabytes full of crap. Jeesh! Even Windows Media Player is only about 10 MB or so.


.NET Debugging Tips Redux; Javascript Context Search

There are two major deficiencies I've noticed with newer developers that seem to be rampant, based on my unscientific analysis of newsgroup and forum post questions that I've read or answered.

First, many new developers (and some older ones too, unfortunately) haven't learned how to use the Try / Catch / Finally block semantics to catch an exception and examine the results. A simple example pattern would be:

// Your buggy code here
catch(Exception ex)
// place a breakpoint on the next line:
System.Diagnostics.Debug.WriteLine(ex.Message + ex.StackTrace);

This will save you hours of time wasted on making reallyreallyDUMB newsgroup posts and waiting for answers, because 9 times out of 10, the Message and StackTrace will tell you not only WHAT happened, but exactly WHERE it happened, no matter how big the "Your buggy code here" block of code is. The modus operandi is "teach a man to fish". By simply replacing the word Debug with "Trace" for an ASP.NET app, and turning on Page Tracing in a deployed application, you can actually do some debugging of release-built apps in production - provided you've instrumented your code with enough of the above. As Joel Spolsky reports, this is a programming habit you want to firm up fast, and learn to do right -- not just for debugging, but for handling ACTUAL EXCEPTIONS in production code. Read Joel's piece and either laugh - or weep!

Another important concept is how to "interpret" exception messages. When the runtime says "Object reference not set to an instance of an object" you need to take this LITERALLY, not post to newsgroups asking "What's wrong"! The runtime is telling you that your code is attempting to use an object variable that does not contain an instance of the object it is supposed to represent. If that's the case, then it should (under ordinary circumstances) be a relatively simple matter to work your way backwards and find out why when you thought "SelectedValue" was supposed to be holding something, it is not! Yet, hordes of developers post mindlessly to newsgroups and forums asking "what is wrong?". Jeesh! The Common Language Runtime JUST TOLD YOU what is wrong, in PLAIN ENGLISH!
Second, few developers except the most professional have taken the time to really study the art and science of DEBUGGING. I certainly don't profess to be an expert, but there are a number of books on the subject that have helped me immeasurably. Here are a few:

  • Debugging Applications for Microsoft .NET and Microsoft Windows by John Robbins (MS Press)

  • Debugging Strategies for .NET Developers by Darin Dillon (aPress)

  • Comprehensive VB.NET Debugging by Mark Pearce (aPress)

Also, consider looking over the following documents:

Debug.Doc and DebugRef.Doc
which can be found in folder:
c:\\Program files\Microsoft Visual Studio .NET 2003\SDK\v1.1\Tool Developers Guide\Docs
and in the Documentation or on MSDN Online in the section "Building Debugging and Testing:

Here is a list of some of my favorite learned tips that increase the power of the developer and the debugger!

Tip: Call Stack Window Debugging: Many developers are not aware that you can set breakpoints in the Call Stack Window (except when doing SQL debugging). Just highlight the call you want to stop on and either press F9 or right-click on the line and select Insert Breakpoint from the shortcut menu. You can also right-click on any breakpoint in the Call Stack Window to enable, disable or set the properties of the breakpoint. For those unfamiliar with the Call Stack Window, this is how you find out "how did I get here" - it enables you to go back up the call stack to find out "from where" the method that you are in was called. Not only can you see where it was called from, you can actually go "Back there" and trace the exact execution path in the debugger. Not to be an evangelist, but if you haven't discovered it yet, please do so now and "get religion".

Tip: Use Run to Cursor for one-shot breakpoints: All you need to do is right-click on the line you want and choose "Run to Cursor" from the menu, and program execution will immediately continue to the point where you are at. Many developers are not aware that this is available from both debugging and editing. In edit mode, this will actually start the debugger and run your program to the cursor location. As with breakpoints, right-clicking in the Call Stack window pops up the shortcut menu that also offers this feature.

Tip: Set sub-expression breakpoints: For example, if you have the expression:
for (int j=0; m=0;j<activelines.count> you can set a breakpoint in the middle of it, not on the whole line.

Tip: Use the BreakPoints Window: If you know the name of the class and method you want to break on, you can choose New from the Breakpoints window and type it directly into the Function area of the window. Developers can spend 15 minutes wandering through their projects opening files just so they can position the cursor over a line of code and hit the F9 key to set a breakpoint. If you know the name of the class and method, doing it this way will find it and set the breakpoint. In addition, you can click it in the Breakpoints list of the window and go to the location automatically.
You can also set a breakpoint by just entering the method name, if it is unique in your application. And, if it isn't you will actually get a list of ALL the locations where the method can be found, which is extremely useful for setting multiple breakpoints on a commonly used function throughout your project. It also displays the overloaded methods in the same class.

Tip: Use the Find ComboBox: If you enter a method name in the Find Box (that's on the top menu bar, just to the right of the Configuration dropdown that reads Debug, Release, Configuration Manager) and hit F9, this will also find the method and set a breakpoint.

Tip: Use Hit Counts: In the Breakpoints window, when you right-click and choose Properties and then click the Hit Counts button, you can modify a breakpoint with four possible combinations of choices. Experiment with this to see how useful it can be. What makes this especially useful is that when you have stopped in the Debugger, it tells you how many times your breakpoint has executed. You can also create conditional expressions that will trigger the debugger only if your expression evaluates to true or has changed since the last time it was evaluated.

Tip: Use the Watch window (and its brethren): The power offered by the Watch window and its close cousins, the Quick Watch Dialog, Autos window, Locals window, and the This / Me window are what can make the difference between hopping around aimlessly all day looking for a bug and quickly solving a problem. Also, you can use any of these windows to change a variable's value by simply typing in a new value.

Tip: Break only when a specific thread calls a method: To set a per-thread breakpoint, you need to uniquely identify a particular thread that you have given a name with its Name property. You can set a conditional breakpoint for a thread by creating a conditional expression such as "ThreadToStopOn" == Thread.CurrentThread.Name .
You can manually change the name of a thread in the Watch window by watching variable "myThread" and entering a Name value for it in the value window. If you don't have a current thread variable to work with, you can use Thread.CurrentThread.Name to set the current thread's name. There is also a private integer variable in the Thread class, DONT_USE_InternalThread, this is unique to each thread. You can use the Threads window to get to the thread you want to stop on, and in the Watch window, enter Thread.CurrentThread.DONT_USE_InternalThread to see the value of it so you can create the right conditional breakpoint expression.

Tip: Use Assertion Debugging: The idea behind asserts is very simple: When you are writing code, you understand that some condition is always expected to be true. If that condition is ever not true, then there is a bug. Asserts can be written so that if this condition is ever not true, the debugger will be launched at that exact point in your code:
System.Diagnostics.Debug.Assert (myValue >=0) The nice thing about Assert windows is that they not only give you the opportunity to break or not, they also show you the stack trace at that point. A very powerful, yet highly under-utilized feature.

Javascript Context Search

I've installed an experimental script that allows you to highlight any text on a page, right-click and my eggheadcafe.com SearchRelated page will come up with our search results. You can try it anywhere on this page. So far it seems to work with IE 6, IE7, and Firefox. With Firefox I couldn't "select Text" on a page with my mouse; you have to use the "Edit / Find on this page" feature to get the text selected. If you would like to try it out on a page of your own, just add this script block to any web page:

<script src="http://www.eggheadcafe.com/articles/search.js"></script>