HTTP referer spoofing, cookies, User Agent strings

There was a post on one of the groups recently by a developer who was making a number of WebRequests for various pages, claiming that one of them would strangely fail.

Yet, this individual stated that if he would paste the respective URL into his browser, that page would come up just fine.

There are several things that could come into play here with various sites:

1) Many sites will reject a request that doesn't match a particular one or more User Agent strings (Here are some samples, if your memory is rusty). So you can add the UserAgent header to the WebRequest. I've even seen some wise-asses who detect Internet Exploder and give you a nasty message about how immoral you are and you should go download Firefox to become a real person and how dare you try to view my site with ..etc...

(Listen, Pal: I already figured out you are a Webtard, so I'm not going to bother to fire up my copy of Firefox to see your page, which I already know is worthless. Besides, your little shenanigan is currently restricting you to only about 15% of your potential audience. Go take a good course in Marketing.)


2) Many times a site is looking for a cookie. Perhaps they set it when you first visit, and subsequent pages look for it. So if you make a request for a "Subsequent page" without the required cookie, you get "Bupkis". Some cookie container code:

CookieContainer myContainer = new CookieContainer();

            // following line adds a cookie in container, which will be for all urls on the domain myDomainstr

            myContainer.Add(new Cookie("name", "value", "/", myDomainstr));

            HttpWebRequest request1 = (HttpWebRequest)WebRequest.Create(httpUrlString);

            request1.CookieContainer = myContainer; // use this same container for all requests

            HttpWebResponse response = (HttpWebResponse)request.GetResponse(); //you can check cookies on response.Cookies

 

            // next request coming--

 

            //all cookies received on request1 would be automatically included in this request from same Cookiecontainer

            HttpWebRequest request2 = (HttpWebRequest)WebRequest.Create(httpUrlString2);

            request2.CookieContainer = myContainer;

            HttpWebResponse response = (HttpWebResponse)request.GetResponse();

3) Another common issue is redirects. Here are a couple of settings you can use:

webrequest.AllowAutoRedirect = [true|false];
webrequest.MaximumAutomaticRedirections = 30;

You can also capture the redirect url:

 public virtual string GetRedirectURL(HttpWebResponse

                webresponse, ref string Cookie)

        {

            string uri = "";

 

            WebHeaderCollection headers = webresponse.Headers;

            if ((webresponse.StatusCode == HttpStatusCode.Found) ||

              (webresponse.StatusCode == HttpStatusCode.Redirect) ||

              (webresponse.StatusCode == HttpStatusCode.Moved) ||

              (webresponse.StatusCode == HttpStatusCode.MovedPermanently))

            {

                // Get redirected uri

                uri = headers["Location"];

                uri = uri.Trim();

            }

 

            //Check for any cookies

            if (headers["Set-Cookie"] != null)

            {

                Cookie = headers["Set-Cookie"];

            }

 

            return uri;

        }//End method

4) Another common technique (this one is real popular with "those" sites) is to check the HTTP Referer. That's available (in ASP.NET) with the Request.UrlReferer property. They do this as a sort of "poor man's authentication" - the idea being that you got in at some "gateway" page with your credentials, and now they figure that you could only be requesting one of their pages from within one of their sites that you "got into" so they look for one or more referers. Here's some code to handle this in a WebRequest:

public string GetUrl(string url, string referer)

        {

           // assumes a fully-qualified "http://" url

            HttpWebResponse webResp = null;

            HttpWebRequest HTTPGetRequest = null;

            StreamReader sr = null;

            string myString = String.Empty;

            HTTPGetRequest = (HttpWebRequest)(WebRequest.Create(url));

            HTTPGetRequest.KeepAlive = false;

            HTTPGetRequest.Referer = referer;

            webResp = (HttpWebResponse)HTTPGetRequest.GetResponse();

            sr = new StreamReader(webResp.GetResponseStream(), Encoding.UTF8);

            myString = sr.ReadToEnd();

            sr.Dispose();

            webResp.Close();          

            return myString;

        }

Happy Spoofing!

Inspector Rae and the case of the Incredible Shrinking DIV

This one popped up on the asp.net newsgroup, and I think it's good for a chuckle!