Why are we so afraid to touch business rules built in our ETL process?
When I got my old job back, naturally I’ve noticed many of the existing ETL processes have been modified with new business rules.
However, many of the business rules were added in a way that existing rules were barely touched. This has caused issues in several ways:
1. Existing rules might be in contradiction with the new rule. Without removing the existing rules, newly added rules are either overwritten or generate mixed/incorrect result.
2. Some times there is no need to add a new rule. The clean way is to modify the existing rules.
This leads to the question, why are we so afraid to touch the existing rules.
Some might say, it’s because existing rules weren’t documented. Others might say, the process is so complicated that it’s better not to touch them.
I have to say that none of these is good enough reason. If you have access to the existing process, you have the rules documented. It might not be in the way you prefer, but it’s documented. Many companies also have project management or change management systems. They are also a form of documentation.
As a matter of fact, the first step we should take is to examine the existing rules and articulate them. No change should be made without a clear understanding of where the new rules stand in terms of the existing rules.
I recently got involved in two issues where business rules got piled on over the time, and the data is no longer valid.
I examined the code, and realized that developers never touched existing rules when new rules were added. In another word, bandages were applied over time, and nobody can articulate the rules anymore.
Don’t be afraid. It’s just code anyway.