SSIS #104 – When Row Count Transformation is invaluable
The Row Count transformation is a pretty straightforward transformation. It does the following two things:
1. It counts rows as data passes through a data flow, and
2. it stores the final count in a variable. (Note that the variable is only populated when the data flow is finished, not while the data flow is still in the middle of transferring data.)
Here is a data flow design pattern I use very often.
At the end of the data flow, my variable @vNewQueueCount is populated with the row count from the data source.
Then I use this variable @vNewQueueCount in a Precedence Constraint expression, as shown below, to decide conditionally whether I need to process my Queue dimension or not.
Configuring the Row Count transformation is simply enough. I’ll show you that later in this blog.
What I want to say first is why the Row Count transformation is invaluable in this specific design pattern.
In this design, I’ve decided to use the Recordset Destination, which is a in-memory ADO dataset and it saves data in memory in a recordset that is stored in an Integration Services package variable of the Object data type. In my next blog, I’ll show you how to use a Foreach Loop container with the Foreach ADO enumerator to process one row of the recordset at a time.
The Recordset Destination is a handy component to use when you do not need to (nor want to) persist your data. One drawback of using the in-memory ADO dataset is that Integration Services does not provide any out-of-box methods to get the row count in the dataset. This is when Row Count transformation becomes invaluable to me. Without too much work on my side, I am able to accomplish the following:
1. Retrieve data from the source, and store the data in a, ADO dataset.
2. Also store the row count in a variable.
3. Finally, be able to use the row count variable to conditionally decide whether I need to process my dimension or not.
Here are a few simple steps to configure the Row Count transformation.
1. Define a variable vNewQueueCount.
2. Create a new data flow
3. Drag the Row Count transformation to the data flow design surface. On the Advance Editor, for VariableName select User::vNewQueueCount.
Finally, you can use the variable in a precedence constraint expression to do conditional processing:
@vNewQueueCount > 0
Have fun with the Row Count transformation!