Thursday, November 22, 2007

Introducing SolrNet

UPDATE 2/19/2009: by now most of this is obsolete, please check out more recent releases

Last month I've been working my a** off integrating Solr to our main site. The first step was to find out how to communicate with the Solr server. Naturally, I came to SolrSharp. But I found it to be really IoC-unfriendly: lots of inheritance, no interfaces, no unit-tests, so it would have been a real PITA to integrate it to Castle. So, instead of wrapping it, I built SolrNet.

Before explaining how it works, a disclaimer: I'm a complete newbie to Solr, Lucene and full-text searching in general. The code works on my machine and does what I need it to do for the task that I have at hand. This project is not, and might never be, feature complete like SolrSharp. Currently it doesn't support facets (UPDATE 8/20/08: I added facet support) or highlights, and maybe some other stuff. If you absolutely need those features right now, either use SolrSharp or write a patch for SolrNet. However, the next step in the integration is implementing faceted search, so I will definitely implement facets sooner or later.

Usage

First we have to map the Solr document to a class (Solr supports only one document type per instance at the moment). Let's use a subset of the default schema that comes with the Solr distribution:

 

public class TestDocument : ISolrDocument {
    private ICollection<string> cat;
    private ICollection<string> features;
    private string id;
    private bool inStock;
    private string manu;
    private string name;
    private int popularity;
    private double price;
    private string sku;

    [SolrField("cat")]
    public ICollection<string> Cat {
        get { return cat; }
        set { cat = value; }
    }

    [SolrField("features")]
    public ICollection<string> Features {
        get { return features; }
        set { features = value; }
    }

    [SolrUniqueKey]
    [SolrField("id")]
    public string Id {
        get { return id; }
        set { id = value; }
    }

    [SolrField("inStock")]
    public bool InStock {
        get { return inStock; }
        set { inStock = value; }
    }

    [SolrField("manu")]
    public string Manu {
        get { return manu; }
        set { manu = value; }
    }

    [SolrField("name")]
    public string Name {
        get { return name; }
        set { name = value; }
    }

    [SolrField("popularity")]
    public int Popularity {
        get { return popularity; }
        set { popularity = value; }
    }

    [SolrField("price")]
    public double Price {
        get { return price; }
        set { price = value; }
    }

    [SolrField("sku")]
    public string Sku {
        get { return sku; }
        set { sku = value; }
    } 
}

 

It's just a POCO with a marker interface (ISolrDocument)[1] and some attributes: SolrField maps the attribute to a Solr field and SolrUniqueKey (optional) maps an attribute to a Solr unique key field. Let's add a document (make sure you have a running Solr instance first):

[Test]
public void AddOne() {
    ISolrOperations<TestDocument> solr = new SolrServer<TestDocument>("http://localhost:8983/solr");
    TestDocument doc = new TestDocument();
    doc.Id = "123456";
    doc.Name = "some name";
    doc.Cat = new string[] {"cat1", "cat2"};
    solr.Add(doc);
    solr.Commit();
}

Let's see if the document is there:

[Test]
public void QueryAll() {
    ISolrOperations<TestDocument> solr = new SolrServer<TestDocument>("http://localhost:8983/solr");
    ISolrQueryResults<TestDocument> r = solr.Query("*:*");
    Assert.AreEqual("123456", r[0].Id);
}

For more examples, see the tests.

DSL

Since DSLs are such a hot topic nowadays, I decided to give it a try to see what happened. I just defined the syntax I wanted in a test, then wrote the interfaces to comply to the syntax and chain the methods, then built the implementations for those interfaces. The result is pretty much self-explanatory:

[SetUp]
public void setup() {
    Solr.Connection = new SolrConnection("http://localhost:8983/solr");
}

[Test]
public void QueryById() {    
    ISolrQueryResults<TestDocument> r = Solr.Query<TestDocument>().By("id").Is("123456").Run();
}

[Test]
public void QueryByRange() {
    ISolrQueryResults<TestDocument> r = Solr.Query<TestDocument>().By("id").Between(123).And(456).OrderBy("id", Order.ASC).Run();
}

[Test]
public void DeleteByQuery() {
    Solr.Delete.ByQuery<TestDocument>("id:123456");
}

Run() is the explicit kicker method [1]. The DSL is defined in a separate DLL, in case you don't want/need it. There are some more examples in the tests.

I TDDd most of the project, so the code coverage is near 75%. I'll add the remaining tests if/when I have the time. Of course, as usual, patches/bugfixes are more than welcome :-)

[1] I might drop this requirement in the future.

Wednesday, November 21, 2007

Poor man's javascript testing

I'm sure someone else has done this, being so simple and old tech... But I couldn't find it anywhere, soo...

Let's say you have just finished writing a great, I mean really ground-breaking javascript library, and you call it lib.js:

Array.prototype.clear = function(){
  this.length=0;
}

And you want to test it. But how? Well, you can forget all about JsUnit, Script.aculo.us BDD, Crosscheck or the others, because now you can test your javascript with... cscript!! Yes, just write a little WSF that includes your great library:

<job>
    <script src="lib.js" language="jscript"/>
    <script language="jscript">
        var a = [1,2,3];
        a.clear();
        if (a.length != 0) {
            WScript.Echo("test failed");
            WScript.Quit(1);
        } else {
            WScript.Echo("test passed");
            WScript.Quit(0);
        }
    </script>
</job>

To execute the test, just invoke cscript on the WSF. Isn't it cool? Ok, ok, it's not cool at all, it only works for non-DOM javascript and only tests for internet explorer, BUT it's really simple and you get the potential to write the tests in another language [1], AND it can be easily integrated to a nant build with a <exec> task. In any case, it beats having no tests at all.

PS: Now seriously, go and check out the tools I mentioned above. GO GO!

[1] Doesn't really work most of the times. I tried ActivePython and ActiveRuby, but they don't see the functions exposed in jscript. Not to mention prototype modifications like the example above, since it doesn't make sense to other languages... VBScript seems to cooperate nicely but only for the most basic scenarios.

Tuesday, October 2, 2007

OpenSearch for OpenGrok

To speed up my searches on OpenGrok, I wrote a little OpenSearch descriptor. You know, that little technology that allows you to put any search engine in that little search box on IE7 and Firefox 2+. Just put the following XML on a file called opengrok.xml, on the root of the OpenGrok context of your servlet container:

<?xml version="1.0" encoding="UTF-8"?>
<opensearchdescription xmlns="http://a9.com/-/spec/opensearch/1.1/">
<shortname>OpenGrok</shortname>
<description>Search in OpenGrok</description>
<tags></tags>
<contact></contact>
<url template="http://YOUR_SERVER_HERE/source/search?q={searchTerms}&amp;start={startPage?}" type="text/html">
</opensearchdescription>

Change YOUR_SERVER_HERE to your actual OpenGrok server.

Then declare the descriptor in index.jsp's head (that's also on the root of the context), so it can be auto-discovered:

<link rel="search" href="opengrok.xml" type="application/opensearchdescription+xml" title="OpenGrok Search" />

And that's it, now Firefox and IE7 will offer you to add the new search plugin.

Have fun grokking your code ;-)

Oh, I should mention that there is also a specific Firefox toolbar extension for OpenGrok, but apparently it's incompatible with Firefox 2+...

EDIT 9/6/2009: This feature is now in OpenGrok's trunk so it will be part of the next release.

Sunday, September 30, 2007

Code search engines mini-review

A couple of weeks ago I got sick of searching our codebase with Total Commander. So I went googling for a code search engine. This is what I found:

  • cs2project: (open-source) developed by Simone as an academic project, using Lucene.Net to index. It's very basic as it's still a new project, but looks really promising. For some reason it only indexed 740 files and then stopped. I'll turn on logging later to see what happened...
  • Koders Pro: (commercial) led by Phil Haack, supports over 30 languages and provides lots of statistics. I installed the demo and after the indexing finished, I went browsing and searching and suddenly it asked me to get a (free) account on koders.com. Hmm, no thanks.
  • Krugle Enterprise: (commercial) seems cool, but I couldn't find a trial version.
  • Fisheye: (commercial) seems more repository-oriented than code-oriented. I mean, you can do full-text queries, but it doesn't give you the exact LoC where the query matched and it doesn't cross-reference classes and types. But when it comes to repository analysis, I think no other product has so many features. It even has a pseudo-SQL language called EyeQL to query the repository! Too bad the trial crashed while indexing our repository...
  • OpenGrok: (open-source) developed in Java, under the wing of OpenSolaris, OpenGrok uses Lucene to index source code. It's only about searching and doesn't offer statistics like the commercial products, but it's very good at what it does. It groks (parses) several languages, including C/C++,Java, Shell Scripts like ksh, bash, Perl, Makefiles, XML/HTML/SGML files, ELF files, Java class files, Java Jar files, archive files like TAR, GZip, BZip2, Zip files, man page formats like troff and more, but sadly, still no .NET languages. For the languages it groks, it provides cross-referencing of classes and types. And it gives you repository history search, too!

 

So I kept OpenGrok, and after installing and configuring, I announced it to the team. But our web designer (one of the coolest guys I have ever worked with) heard "OGrok" instead of OpenGrok (ogro means ogre in spanish). He went on calling it OGrok, and then he even put together an alternative logo, featuring the most famous ogre :-)

It has since become an invaluable tool for us, I can't recommend enough that you install one of these code search engines, it really improves collaboration with your teammates.

Wednesday, September 26, 2007

Keep your CI server on a fast machine

The build for our main site was taking nearly 8 minutes on the CI server, getting dangerously close to the recommended 10-minute limit. And I'm talking about the basic, commit build. However, a "nant all" on my machine took about 2 minutes! WTF?!? Then I remembered that I installed CruiseControl on a crappy old 512mb RAM machine!

I moved it to a newer, faster server, and problem solved :-)

Wednesday, September 19, 2007

HttpInterfaces for ASP.NET 1.1

Inspired by Phil Haack's HttpInterfaces, I wrote a similar set of interfaces for ASP.NET 1.1 (which is what we still use at work, sigh...), so we can better test our huge legacy codebase. The most significant difference is that DuckTyping doesn't work in 1.1 AFAIK... so I had to write adapter classes to wrap System.Web.HttpApplication, etc, which was pretty trivial thanks to ReSharper.

Let's see how we could use these interfaces to test a common legacy WebForm. Suppose you have a page which puts the content of a QueryString parameter in a Label, i.e.:

 

public class MyPage : Page
{    
  protected Label Label1; 
  private void Page_Load(object sender, EventArgs e) {        
  	Label1.Text = Request.QueryString["text"];    
  } 
    
  protected override void OnInit(EventArgs e) {        
  	InitializeComponent();        
  	base.OnInit(e);    
  } 
  
  private void InitializeComponent() {
  	this.Load += new System.EventHandler(this.Page_Load);    
  }
}

We build a BasePage from which MyPage will inherit, that will allow injection of Request, Response, etc:

 

public class BasePage : Page
{    
  private IHttpRequest requesto;    
  public new IHttpRequest Request {        
    get {            
      if (requesto == null)
        requesto = new HttpRequestAdapter(HttpContext.Current.Request);            
        return requesto;        
    }        
    set { requesto = value; }    
  } 
  ...
}

Make MyPage inherit from BasePage instead of Page, and set the Request to whatever you like... Using this, we can test that Label1 effectively gets the QueryString parameter. Just create a stub for the request, assign it to a instance of MyPage, and call Page_Load(). For example:

 

[Test]
public void PageLoad() {
  MockRepository mocks = new MockRepository();
  IHttpRequest req = (IHttpRequest) mocks.CreateMock(typeof (IHttpRequest));
  string text = "hello world";
  NameValueCollection queryString = new NameValueCollection();
  queryString["text"] = text;
  Expect.Call(req.QueryString).Return(queryString);
  mocks.ReplayAll(); MyPage p = (MyPage) ReflectionHelper.CreatePageWithControls(typeof (MyPage));
  p.Request = req;
  ReflectionHelper.InvokeMethod(p, "Page_Load", null, null);
  Label Label1 = (Label) ReflectionHelper.GetPageControl(p, "Label1");
  Assert.AreEqual(text, Label1.Text);
}

Here I used Rhino.Mocks to create the request stub. Note that we have to use reflection to call Page_Load() and get the controls since they are not public... But with minimum changes to the code, we gained a lot of testability!

With TypeMock, a very powerful mocking framework (although not free), we could write the same test without depending on any interface and without making any changes to the original code:

[Test]
public void PageLoad_WithTypeMock() {
    MockManager.Init();
    Mock requesto = MockManager.Mock(typeof (HttpRequest), Constructor.Mocked);
    string text = "hello world";
    NameValueCollection queryString = new NameValueCollection();
    queryString["text"] = text;
    requesto.ExpectGet("QueryString", queryString);
    Mock myPage = MockManager.Mock(typeof (MyPage), Constructor.NotMocked);
    myPage.ExpectGet("Request", new HttpRequest(null, null, null));
    myPage.Strict = false; 


    MyPage p = (MyPage) ReflectionHelper.CreatePageWithControls(typeof (MyPage));
    ReflectionHelper.InvokeMethod(p, "Page_Load", null, null);
    Label Label1 = (Label) ReflectionHelper.GetPageControl(p, "Label1");
    Assert.AreEqual(text, Label1.Text);
}

But, like I said, TypeMock is not free. There is a community edition, though. I think TypeMock is great for initial legacy testing, but in the long run, it pays off to refactor, to add new seams so that legacy code stops being legacy. The beauty of refactoring is that it can be done progressively, so don't be afraid to make changes!

Oh, I almost forgot, here's the code, have fun! :-)

Sunday, September 16, 2007

Duplicate files finder

Oh no, not another duplicate file finder!! There are already hundreds of these... and still, I had to hack my own, since I couldn't find a single one that kept hashes in a database for quick subsequent searches and support of lots of files, and was free. So I wrote dupito (stupid name, I know). It uses SQL Server Compact Edition to store and query hashes (SHA-512 is used). I tried SQLite at first, but it was way too slow... and since I used Castle ActiveRecord, it was just a matter of configuration to change that to SQLServerCE.

Anyway, usage is as follows: when called without arguments, it indexes files in the current directory and subdirectories, then prints a list of duplicate files.

Command-line arguments:

  • c: cleans up database, deleting rows that reference nonexistent files
  • r: rehashes all files in database 
  • l: lists duplicate files currently in database

And here it is.

(As a sidenote, the exe is pretty big because I merged in Castle.ActiveRecord.dll + all of its dependencies...)

Thursday, July 12, 2007

Improving web performance by distributing images among hostnames

It's a known fact that browsers open a very limited amount of concurrent connections to get the components of a page. By components I mean all the resources required to render a page: images, css, javascript, flash, etc. In fact, the HTTP/1.1 standard recommends that no more than two concurrent persistent connections per hostname should be open at the same time. Users can modify these default settings for their browsers, but as developers we have to assume defaults. So, if we can't control the number of connections, we have to increase the number of hostnames (the other variable). How do we do that? Basically, fetching images and/or other resources from a hostname other than the main host. E.g. if your main host is www.example.org, then you could create subdomains and fetch some images from images1.example.org, some from images2.example.org, and so on. So if you have your main host plus two subdomains for images, you get 6 concurrent connections from your visitors.

Now this is all nice and dandy, but if you have a 10000+ page site in ASP.NET, replacing every friggin' src attribute in every img tag is simply not viable. Plus, the guy that used to work before you wrote a lot of image urls on the code-behind, like img.Src = "/img/hello.gif";
And you have to consider browser caching too. If you assign two different hostnames in two pages to the same resource, it will break the cache.

The solution I propose is to write a Stream to plug into Response.Filter. Let's call it ImageDistributionFilter. You would create this ImageDistributionFilter, somehow pass it a list of domains of subdomains available (these parameters could be injected by a IoC container or fetched from web.config or dynamic.web.config). Then you hook it to Response.Filter, using you custom base page class (if you have one) or a HttpModule. The ImageDistributionFilter scans the HTML before sending it to the client, and replaces every src of every img tag with a corresponding subdomain. E.g:

<img src="/img/hello.gif">

becomes:

<img src="http://images1.example.org/img/hello.gif">


Here's a catch: you should always check that Request.Url.Scheme == Uri.UriSchemeHttp before applying the filter (in other words, don't apply the filter in secure pages), otherwise you could end up with mixed http / https content (which generates warnings on the browser) or certificate mismatches triggered by requests such as https://images1.example.org/img/hello.gif (if you have a certificate for exactly www.example.org, it won't work for other subdomains)

Another thing to remember: there's a limit to the number of subdomains that can be used to increase performance. Each subdomain will require a separate DNS request, that's an important overhead to take into account. Yahoo ran an experiment and found out that the optimal number of subdomains lies between two and four. Beyond that, DNS overhead is too high and you actually lose performance. CPU usage increases too. However, you have to think about the most common network connection that your users have (Google Analytics to the rescue!). If 90% of your users are still on dial-up and use Pentiums-133, don't use this technique, it will kill their machines! Ok, maybe I'm exaggerating, but it will make things slower for them. On the other hand, if it's an intranet app (LAN) and users have modern computers, maybe you could use six or eight subdomains.

Note also that it's not neccesary that each subdomain goes to a different server. The connection limit applies per hostname, not per IP. So, you could use wildcard DNS entries to map every subdomain to the same server.

You can find the source for the ImageDistributionFilter along with a little demo here.

Further reading :

Thursday, June 14, 2007

Dynamic.web.config

I don't know about you, but I absolutely hate it when I change something in my web.config and the application restarts. Maybe I just misspelled something in the <appSettings> or changed some custom cache configuration. The app should be able to survive some configuration settings changes without resetting. So I built my own ConfigurationSettings-like class and named it DynamicConfigurationSettings. It reads a web.config-like file (which I call dynamic.web.config, but could be anything) and whenever the file changes, its values are automatically reloaded (courtesy of FileSystemWatcher and Bjarne Lindberg's FileWaiter). FileWaiter was necessary because FileSystemWatcher raises multiple events when a file is changed, a problem described in this thread. DynamicConfigurationSettings is meant to be used as a drop-in replacement for ConfigurationSettings. It was written for 1.1 but it should be fairly easy to port it to 2.0. Like I said, dynamic.web.config has a schema similar to that of a basic web.config:

<?xml version="1.0" encoding="utf-8"?>
<xs:schema attributeFormDefault="unqualified" elementFormDefault="qualified" xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="configuration">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="configSections">
          <xs:complexType>
            <xs:sequence>
              <xs:element maxOccurs="unbounded" name="section">
                <xs:complexType>
                  <xs:attribute name="name" type="xs:string" use="required" />
                  <xs:attribute name="type" type="xs:string" use="required" />
                </xs:complexType>
              </xs:element>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
        <xs:element name="appSettings">
          <xs:complexType>
            <xs:sequence>
              <xs:element maxOccurs="unbounded" name="add">
                <xs:complexType>
                  <xs:attribute name="key" type="xs:string" use="required" />
                  <xs:attribute name="value" type="xs:string" use="required" />
                </xs:complexType>
              </xs:element>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
</xs:schema>

(extracted with Visual Studio). Basically, it supports <appSettings> and <configSections>.
Example:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
 <configSections>
  <section name="customSection" type="MyApp.CustomSectionHandler, MyAssembly"/>
 </configSections>
  <appSettings>
  <add key="MaxResults" value="300"/>
  </appSettings>
 <customSection>
  <default>hello</default>
  <rules>
   <rule source="something" destination="something else"/>
  </rules>
 </customSection>
</configuration>

To use it, just call DynamicConfigurationSettings.load(@"path\to\dynamic.web.config") in your Application_Start and that's it.

Additionally, whenever it can't find some requested config item, it tries to fetch it from ConfigurationSettings. So, if your code uses DynamicConfigurationSettings, you can override your web.config settings with your dynamic.web.config settings. Example: suppose you have defined MaxResults as 200 in <appSettings> in web.config, and you have the following code:

DynamicConfigurationSettings.load("dynamic.web.config");
int maxResults = Convert.ToInt32(DynamicConfigurationSettings.AppSettings["MaxResults"]);
You'd get 300, as it's defined in dynamic.web.config. BUT, if we remove the definition from dynamic.web.config, you get 200, as defined in web.config.
Before going to the code, a piece of advice/disclaimer:

DON'T use this to store critical data! I only use it to store cache sizes and other non-critical stuff. I tried hard to make it as thread-safe and fault-tolerant as possible, but be aware that it's very dangerous to change configuration at runtime. If you edit you dynamic.web.config and save it while it's not well-formed, and your application crashes, it's not my fault (it shouldn't happen, but still).
I recommend that you wrap every access to a strongly-typed value in DynamicConfigurationSettings (like the maxResults example above) with try..catch and provide sensible defaults, so you won't crash everything if you write something like <add key="MaxResults" value="Eoo"/>

You have been warned. Now here's the code, go have fun :-)
As usual, comments and bugfixes are very welcome.

FileWaiter.cs
using System;
using System.Diagnostics;
using System.IO;
using System.Threading;

namespace Utils
{
 /// <summary>
 /// Waits for a file to close
 /// From Bjarne Lindberg (http://www.thescripts.com/forum/post1453548-8.html)
 /// </summary>
 public class FileWaiter
 {
  public string fileName;

  public FileWaiter(string fileName) {
   this.fileName = fileName;
  }

  public void DoWait(int timeout) {
   DoWait(new TimeSpan(0, 0, 0, 0, timeout));
  }

  public void DoWait() {
   DoWait(TimeSpan.MinValue);
  }

  public void DoWait(TimeSpan timeout) {
   Debug.Assert(fileName != null);
   Thread t = new Thread(new ThreadStart(ThreadWait));
   t.Start();
   if (timeout > TimeSpan.MinValue)
    t.Join(timeout);
   else
    t.Join();
  }

  private void ThreadWait() {
   do {
    //Start waiting..
    Thread.Sleep(500);
   } while (File.Exists(fileName) && isFileOpen(fileName));
  }

  protected static bool isFileOpen(string fileName) {
   FileStream s = null;
   try {
    s = File.Open(fileName, FileMode.Open,
                  FileAccess.ReadWrite,
                  FileShare.None);
    return false;
   }
   catch (IOException) {
    return true;
   }
   finally {
    if (s != null)
     s.Close();
   }
  }
 }
}

DynamicConfigurationSettings.cs
using System;
using System.Collections;
using System.Configuration;
using System.IO;
using System.Xml;
using System.Collections.Specialized;
using log4net;

namespace Utils
{
 /// <summary>
 /// Similar to <see cref="System.Configuration.ConfigurationSettings"/>, but accepts changes in configuration file at runtime without restarting the application
 /// </summary>
 public class DynamicConfigurationSettings
 {
  private static readonly ILog log = LogManager.GetLogger(typeof(DynamicConfigurationSettings));
  
  /// <summary>
  /// Gets configuration settings in the configuration section.
  /// </summary>
  public static NameValueCollection AppSettings {
   get {
    lock (padLock) {
     if (_appSettings != null)
      return _appSettings;
     return ConfigurationSettings.AppSettings;     
    }
   }
  }
  
  private static Hashtable configSectionsCache = new Hashtable();

  /// <summary>
  /// Returns configuration settings for a user-defined configuration section.
  /// </summary>
  /// <param name="sectionName">The configuration section to read.</param>
  /// <returns>The configuration settings for sectionName.</returns>
  public static object GetConfig(string sectionName) {
   if (configSectionsCache[sectionName] == null)
    try {
     SectionConfig config = (SectionConfig) sections[sectionName];
     IConfigurationSectionHandler sectionHandler = (IConfigurationSectionHandler) Activator.CreateInstance(Type.GetType(config.handlerName));
     configSectionsCache[sectionName] = sectionHandler.Create(null, null, config.node);
    } catch (Exception) {
     configSectionsCache[sectionName] = ConfigurationSettings.GetConfig(sectionName);
    }
   return configSectionsCache[sectionName];
  }

  /// <summary>
  /// Loads configuration from xml document
  /// </summary>
  /// <param name="config">Document to load</param>
  public static void load(XmlDocument config) {
   XmlNodeList configSections = config.SelectNodes("/configuration/configSections/section");
   
   lock (padLock) {
    sections = new Hashtable();
    
    foreach (XmlNode section in configSections) {
     SectionConfig sc = new SectionConfig();
     string sectionName = section.Attributes["name"].InnerText;
     sc.node = config.SelectSingleNode("/configuration/" + sectionName);
     sc.handlerName = section.Attributes["type"].InnerText;
     sections[sectionName] = sc;
    }
   
    XmlNode nodeAppSettings = config.SelectSingleNode("/configuration/appSettings");
    NameValueSectionHandler nvsh = new NameValueSectionHandler();   
    NameValueCollection nvc = (NameValueCollection) nvsh.Create(null, null, nodeAppSettings);
    _appSettings = new NameValueCollection();
    _appSettings.Add(ConfigurationSettings.AppSettings);   
    foreach (string k in nvc.Keys)
     _appSettings[k] = nvc[k];    
    configSectionsCache = new Hashtable();
   }

  }
  
  /// <summary>
  /// Loads configuration from xml file and reloads when it changes
  /// </summary>
  /// <param name="filename"></param>
  public static void load(string filename) {
   _filename = filename;
   XmlDocument config = new XmlDocument();
   try {
    config.Load(filename);
    load(config);
    if (fsw != null) {
     fsw.Dispose();
    }
    fsw = new FileSystemWatcher(Path.GetDirectoryName(filename));
    fsw.Changed += new FileSystemEventHandler(fsw_Changed);
    fsw.EnableRaisingEvents = true;       
   } catch (Exception e) {
    log.ErrorFormat("Error loading dynamic.web.config: {0}\n{1}", filename, e.ToString());
   }
  }
  
  /// <summary>
  /// Loads configuration from xml stream
  /// </summary>
  /// <param name="s"></param>
  public static void load(Stream s) {
   XmlDocument config = new XmlDocument();
   StreamReader sr = new StreamReader(s);
   try {
    config.LoadXml(sr.ReadToEnd());
    load(config);
   } catch (Exception e) {
    log.Error("Error loading dynamic.web.config:", e);
   } finally {
    sr.Close();
   }
  }
  
  /// <summary>
  /// Loads configuration from xml string
  /// </summary>
  /// <param name="s"></param>
  public static void loadString(string s) {
   XmlDocument config = new XmlDocument();
   try {
    config.LoadXml(s);
    load(config);          
   } catch (Exception e) {
    log.Error("Error loading dynamic.web.config:", e);
   }
  }

  /// <summary>
  /// Handles changes in config file
  /// </summary>
  /// <param name="sender"></param>
  /// <param name="e"></param>
  private static void fsw_Changed(object sender, FileSystemEventArgs e) {
   if (e.Name.ToLower() == Path.GetFileName(_filename.ToLower()) && e.ChangeType == WatcherChangeTypes.Changed) {
    FileWaiter fw = new FileWaiter(e.FullPath);
    fw.DoWait();
    load(e.FullPath);
   }
  }

  private static NameValueCollection _appSettings;  
  private static FileSystemWatcher fsw;
  private static string _filename;
  private static Hashtable sections;
  private static object padLock = new object();
  
  private class SectionConfig {
   public string handlerName;
   public XmlNode node;
  }

 }
}

Sunday, June 10, 2007

Exception Translation with Castle Windsor

Warning: the following code has become obsolete since the release of Castle RC3. I'll update as soon as I can. The concepts still apply, though.

Recently I've been writing an app using Castle Windsor. While I was writing the service classes, the following problem came up: I don't want my services' consumers catching Dao exceptions. I'm using ActiveRecord to persist domain objects, but it could have been directly NHibernate, or iBatis. Either way, whoever uses the service classes should be ignorant about persistance details. Suppose the service interface looks like this:

public interface ISomeService {
 void ProvideService();
}
and its implementation lets an internal exception pass, which we could model like this to make things simple:
public class SomeService : ISomeService {
 public void ProvideService() { 
   throw new InternalException();
  }
}

public class InternalException : ApplicationException {}
Now we'd like to translate InternalException into an ExternalException, which is a service-level exception. We could manually catch InternalException in every method of SomeService, but that's annoying.
Let's use Windsor instead. We need three components: the exception translator (which says what external exception corresponds to each internal exception), the interceptor (which actually does the try..catch) and a contributor (which installs the interceptor and translator).
First, let's write a test that should pass when we finish installing all three components:
[TestFixture]
public class ExceptionTranslatorTests {
  [Test]
  [ExpectedException(typeof (ExternalException), "1")]
  public void ExceptionIsTranslated() {
    IWindsorContainer container = new WindsorContainer();
    container.AddComponent("service", typeof (ISomeService), typeof (SomeService));
    ISomeService svc = container.Resolve<ISomeService>();
    svc.ProvideService();
  }
}

public class ExternalException : ApplicationException {
  public ExternalException(string message) : base(message) {}
  public ExternalException() {}
  public ExternalException(string message, Exception innerException) : base(message, innerException) {}
}
If we run this test now, it will fail, since the exception thrown by ProvideService() is still InternalException instead of the expected ExternalException.
Now let's build the translator:
public interface IExceptionTranslator {
 Exception translate(Exception e);
}

public class SomeExceptionTranslator : IExceptionTranslator {
  public Exception translate(Exception e) {
    return new ExternalException("1", e);
  }
}
Simple, no? Just make sure you wrap the original exception, so you don't lose stack information and stuff.
Now for the generic interceptor:
public class ExceptionTranslationInterceptor<T> : IMethodInterceptor where T : IExceptionTranslator {
  private T translator;

  public ExceptionTranslationInterceptor(T translator) {
   this.translator = translator;
  }

  public object Intercept(IMethodInvocation invocation, params object[] args) {
    try {
      return invocation.Proceed(args);
    } catch (Exception e) {
      throw translator.translate(e);
    }
  }
}
Notice that the actual translation logic will be injected by Windsor into the interceptor. Oh, and make sure theIMethodInterceptor you implement is the one in Castle.Core.Interceptor, not the one in AopAlliance.Intercept.
You may wonder why the interceptor takes <T>, you'll see why in a second when we build the contributor:
public class ExceptionTranslatorContributor<T, T2> : IContributeComponentModelConstruction
 where T2 : IExceptionTranslator {
  public void ProcessModel(IKernel kernel, ComponentModel model) {
    if (typeof (T).IsAssignableFrom(model.Implementation)) {
      kernel.AddComponent(typeof (T2).ToString(), typeof (T2));
      kernel.AddComponent(string.Format("ExceptionTranslationInterceptor<{0}>", typeof(T2)), typeof(ExceptionTranslationInterceptor<T2>));
      model.Interceptors.Add(new InterceptorReference(typeof (ExceptionTranslationInterceptor<T2>)));
    }
  }
}

This contributor takes two types: T, which is the service to intercept, and T2, which is the IExceptionTranslator. It installs the translator and the corresponding interceptor as components, then it installs the interceptor as such. By making the interceptor and the contributor generic, we can easily associate different translators for different services, just by setting up the contributor in the container, like this:
[TestFixture]
public class ExceptionTranslatorTests {
  [Test]
  [ExpectedException(typeof (ExternalException), "1")]
  public void ExceptionIsTranslated() {
    IWindsorContainer container = new WindsorContainer();
    container.Kernel.ComponentModelBuilder.AddContributor(new ExceptionTranslatorContributor<SomeService, SomeExceptionTranslator>());
    container.AddComponent("service", typeof (ISomeService), typeof (SomeService));
    ISomeService svc = container.Resolve<ISomeService>();
    svc.ProvideService();
  }
}
And now the test passes!
Full code shown here is available here.

Monday, May 28, 2007

Xml serialization of enums - System.InvalidOperationException

A few days ago I was writing a web service when all of a sudden a test failed on me with:
System.InvalidOperationException: Instance validation error: '3' is not a valid value for SomeEnumInMyApp
which turned out to be quite a google-unfriendly exception. The exception is thrown when trying to serialize an enum with a value that isn't explicitly in the declaration of the enum. A couple of people dealt with this, but I don't really want to manually edit the WSDL, and I don't care right now about breaking a future version of the web service (not likely to happen in my case)
In case nothing of this makes any sense to you, let's see some code:
Given this enum declaration:
public enum Epopo {
 Uno = 1,
 Dos = 2
}
"Epopo?" Yes, Epopo. Just because.
This test passes:
[Test]
public void SerializeEnumToXml1() {
 Epopo p = Epopo.Uno;
 using (MemoryStream ms = new MemoryStream()) {
  XmlSerializer xml = new XmlSerializer(typeof(Epopo));
  xml.Serialize(ms, p); 
 }
}
But this one fails with the mentioned exception:
[Test]
public void SerializeEnumToXml2() {
 Epopo p = Epopo.Uno | Epopo.Dos;
 using (MemoryStream ms = new MemoryStream()) {
  XmlSerializer xml = new XmlSerializer(typeof(Epopo));
  xml.Serialize(ms, p); 
 }
}
Hmm. What if we try with a LosFormatter instead of XmlSerializer?
[Test]
public void SerializeEnumToLosFormatter() {
 Epopo p = Epopo.Uno | Epopo.Dos;
 LosFormatter los = new LosFormatter();
 using (MemoryStream ms = new MemoryStream()) {
  los.Serialize(ms, p); 
 }
}
Test passes. Damn you, XmlSerializer. What about XmlEnum?
public enum XmlEpopo {
 [XmlEnum("Uno")]
 Uno = 1,
 [XmlEnum("Dos")]
 Dos = 2
}

[Test] 
public void SerializeEnumWithXmlEnumToXml() { 
 XmlEpopo p = XmlEpopo.Uno | XmlEpopo.Dos; 
 using (MemoryStream ms = new MemoryStream()) { 
  XmlSerializer xml = new XmlSerializer(typeof(XmlEpopo));
  xml.Serialize(ms, p); 
 }
}
Doesn't work either... Well, after that I despaired a bit and started trying everything: XmlElement, PowerCollections.Set, Iesi.Collections.ListSet, Dictionary<Epopo, object>, but nothing worked. Ultimately, I used a List<Epopo>, like this:
[Test]
public void SerializeListOfEnum() { 
 List<Epopo> l = new List<Epopo>();
 l.Add(Epopo.Uno); 
 l.Add(Epopo.Dos); 
 using (MemoryStream ms = new MemoryStream()) { 
  XmlSerializer xml = new XmlSerializer(typeof(List<Epopo>));
  xml.Serialize(ms, l); 
 } 
}
It sucks, I know, but at least it works. If any knows a better solution, feel free to post a comment!
By the way, it seems that Microsoft is having this same problem as well...
Full code for these tests is here.

Sunday, May 20, 2007

Introduction 1.1

A good way to know a programmer is to see what tools he uses. If I read "NUnit" on a résumé, I can safely assume he/she at least heard of unit testing (that doesn't mean he's doing it right, though). If I see "MbUnit", it's likely that he did use NUnit at first, but needed more or wasn't satisfied (that was my case) So, here are some of the tools/frameworks I use (in no particular order) Another good way is to see what he's reading. I must confess I'm not much of a book reader (technical books are very expensive here) but I read a lot of blogs (part of my blogroll is in the column on the right). I think the best ideas, patterns, practices, like TDD, DDD, Continuous Integration, etc, are out there in the net anyway. I try not to stay with .NET blogs only, I also read some Java, Javascript and even Lisp blogs (although Lisp is a pending subject)

Wednesday, May 16, 2007

Introduction

Hi, my name is Mauricio Scheffer, and this is yet-another-programming-blog. I live in Argentina and I'm currently working as a developer for a fairly popular US website (around 150000 unique visitors per month) using ASP.NET (still 1.1, we don't have time to migrate yet :-( ) Since I'm pretty much the only developer here (my boss knows her .NET and SQL but she's too busy administering the site) I spend most of the day squashing bugs (hence the blog title). But I'm not complaining, I'm getting a new teammate next week and I'm constantly interviewing new people. Plus, there's zero bureaucracy, so I'm free to implement things the way I want, as long as it doesn't take forever. Well, that should be enough for a first post. :-)