Perfection impedes achievement

Fielded Text

At work I often had to write text file parsers to obtain data from information vendors. After a while, I found myself repeating the same patterns whenever I wrote another parser. So (in my spare time), I decided to try and write a general purpose text parser.

This project was only supposed to take a month or so but it just grew and grew. In the evenings and weekends I snatched whatever hours I could to continue on my text parser. (However, being careful that my wife and kids did not think I was too obsessed with it).

As the project evolved, it started to look like a database API. In fact, Fielded Text now associates a small XML file with a text (eg CSV) file. This XML file (the Meta file) describes the structure and format of the data within the file. Once the Meta file is associated with the text file, you can read it in exactly the same way you read a database table. For example, create a DataReader, read a record and then obtain the value for each field in that record.

About halfway through this project, I had to decide where I wanted this project to head. Did I want to make a commercial product out of it? There were already some CSV Parsing libraries/components on the market; did I want to compete with them? I decided I did not want to do that.

Instead, I decided to try and make a standard for text files. Maybe an overly idealised goal and certainly not commercially rewarding - however I was enjoying the technical challenge. Maybe I will get some recognition out of it?

Anyway, that’s how http://www.fieldedtext.org came about. My approach to ordering the world of CSV files and other text files of that ilk. The ultimate aim of the Fielded Text standard is for people (or organisations) which publish data text files, to also publish the corresponding Meta files. Then parsing these files will be so much easier.

In terms of actual software, the outcomes of my project are:

  • Fielded Text - A software component (C# and Delphi) which parses and generates text files using the associated Meta file. It can also be used to create the Meta files.
  • FTEditor - A text editor which visually and interactively lets you create Meta files for text files.
  • FTValid - A small windows command line program which can validate the format/structure of a Fielded Text or CSV file.