mhinze.com archive

using System;
using HtmlAgilityPack;
using System.Net;

class LbHtmlWeb : HtmlWeb
{
    private readonly string USER_AGENT =
        "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)";

    public LbHtmlWeb()
    {
        this.PreRequest += delegate(HttpWebRequest request)
        {
            request.UserAgent = USER_AGENT;
            return true;
        };
    }

    //…
}

Now that was easy!  Pretty spiffy that HtmlWeb exposes that event.

Please let me at the HttpWebRequest.

Don't do this (RSS Toolkit, I'm talking to you):

XmlDocument doc = new XmlDocument();
doc.Load(url);

=(

This entry was posted on Tuesday, November 6th, 2007 at 11:50 am and is filed under .NET, Tips. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.