Dirty Data

I'm currently working on a business intelligence review and the first test began to run into issues as soon as it started trying to make sense of the data in NWC Inc. Dates that are strings is bad and so is allowing just any old data into the database. BI tools like nice clean data to work with and my dirty data is driving tools crazy.

So I spent the morning cleaning up the data, including adding a new column to handle real dates. Of course then it was necessary to change the widget ordering code to make certain the new dates were entered into the database. And there's always that moment of "should I really hit enter on this SQL statement?" Nerve wracking, to say the least!

Then I added some indexes to some of the commonly used fields for reporting, because churning through more than half a million records can be a pain when you're pulling it across the network and trying to load it into Internet Explorer. I think it's helping... cause I"m finally able to pull data that makes some amount of sense:

Wow. Wonder what happened in 2002 that we only made $5,969? This year's looking good though, isn't it? Too bad it isn't real money, I could buy this, fix it up and still have money left for other things, like gaming books.

Alas, it's back to cleaning up all that dirty data. And you thought your mom wanted you to wash your hands just to keep germs away. She was just worried about dirty data and what problems it would cause later on. So remember, always wash your hands before you enter data...your DBA will thank you for it.