Home > ETL Best Practices > Why validation and standardization are so frustrating?

Why validation and standardization are so frustrating?

We have done the ETL mapping, and acceptance testing with limited data volume. Standardization has been documented. But still, once the system is in production, the most confusing and frustrating issues are validation and standardization.

Take standardization as an example. The project started out with standardized device models and the mapping. But in the middle of the project, we decided that we will allow data that are not on the standard table being loaded and new device models will be created on the fly from the raw data. I guess it’s still unclear to me who actually owns the standardization.

By validation I am talking about validation against the source data. It seems to me that this was not really the focus during UAT. The reason is because the UAT environment has only limited data volume. It’s hard for users to test it against the source data where the full spectrum of data is available.

Categories: ETL Best Practices
  1. No comments yet.
  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: