Well I have implemented the first set of Data Translators in my Normalizer (for those who do not know a Normalizer is made up of a Message Router and a set of data translators); it turns out that I had already put a Message Router into my application so all I needed to do is to add the translators. So now I have come to the conclusion that while this is all well and good what I really need is a generic Catonical Data Format ( I really did not want to try to do this; I mean how do you write a format which can represent N different types of data which is represented in (N-X) different formats).
So I decided to go and look if anyone had tried to make a generic data format (XML based preferably); and what do I find but 4 different projects to do just that. They are: NASA's XDF and CDF projects; NCSA's HDF; and the Global Grid Forumns DSDL. Well this made my day all I need to do is figure out how to convert my data into one of these formats and away we go right. Wrong first DSDL is the ultimate in vaporware, the only thing that has been done in that group is argue about what DSDL should be for the last three years. Ok so that leaves three others let's look at them: HDF has some great ideas (Hierarchical data for one) but it is mired in code from the early '90s all of the major functionality is in C not C++ or even Java but C. This makes converting the code to .NET problematic (I have tried to port C code before and some of the concepts in C just do not have an equivalent in .NET or Java for that matter). OK that is fine maybe they have an XML representation defined by a schema so I can deserialize it into an object set; nope just one defined by a DTD; while I could try and reverse engineer a schema from the DTD I have found that to be problematic.
OK so now it is time to look at CDF and it's relative XDF; first thing CDF is old it was started back in the late '80s to deal with the pleiferation of data formats coming out of space science (that sounds familiar (I am dealing with the same problem only in biology and chemistry over 10 years later)). Well now let's look the last update was 2 years ago and it seems to have been a merger between XDF and CDF which allows XDF to represent CDF data as XML. OK thats fine so let us go look at XDF it is really just an XML schema for generic data and low and behold it uses an very well written and thought out XML schema. I can use this. It took me 2 hours to get SQl Server 2005 to accept data into a strongly typed XML column and be able to push and pull XML serialized objects in and out of the DB in XDF. This was way too easy.