The consideration of automating the processes in a data warehouse (DWH) as far as possible keeps many project teams busy today.
The desire for a comprehensive automation solution usually arises very quickly.
What expectations are placed on an automation solution, what the vendors promise and what the reality looks like in the end is what I describe in this multi-part blog post. In the first part, I described the project teams' expectations of automation. The second part deals with the promises of automation vendors and a first negative example for the selection of an automation solution.
In the third part of the mini-series, I present more real cases that I have experienced in recent years. First, two more cases that serve as negative examples (the first case was described in the second part of the mini-series). The third part of this mini-series ends with a case as a positive example of a successful automation solution.
Zweiter Fall
In the second case, the project team, let's call it Team B, made a pre-selection of automation solutions. This looked like the following: Search Google for the keyword 'automation'. It is amazing what search results you get there that do not fit at all to a DWH, called data solution in the following.
The result was a huge list of over 30 vendors offering an 'automation solution' found by searching the Internet.
Team B's list included everything that was somehow related to automation. At this point, Team B enlisted external help, significantly reduced the number of vendors, and finally had a shortlist of vendors.
The really good thing about team B's initial startup and expectation phase was that they took the opportunity to exchange ideas with other companies that had already successfully implemented automation tools.
At this point, it seemed that team B was really well on their way to identifying a perfect tool for their requirements and goals. But: Team B expressed the requirements on a very high and abstract level, with a vague definition and a very strong focus on technology. And on top of that, from that point on, they did it without external support.
Team B's real goal was to find an automation solution that would minimize and support all of the developer's manual tasks. Thus, the focus was to be on domain-oriented modeling - conceptual and logical. The physical modeling and thus the creation of tables, loading processes and orchestration were also to be supported by the automation solution.
In practice, team B had great difficulty in identifying, defining and weighting the criteria for the selection process. The reason for this was the vague and abstract requirements formulated in advance.
Before this criteria identification process was completed, team B already invited the first manufacturers to presentations and demonstrations. And again, in case 2, the vendors were very strong in presenting their tools. They were always promising team B: Yes, our tool can do this.
And what happened? In principle, a really sad ending, because team B failed with this project after one year and completely stopped working on the automation solution. The automation solution did not meet the requirements, the expectations. A wrong decision was made due to the vague and incomplete criteria. A lot of money was wasted or burned.
Third case
In the third case, the project team, let's call it team C, has a strong desire for an automation tool.
From the beginning, team C did not create a complete list for selecting an automation solution. There was only a very short list because the pre-selection was based on the personal views and sensitivities of a single person from team C.
On the other hand, team C did a good job by creating a proof of concept (PoC) to gain experience with the automation solutions. On the other hand, there were no specific evaluation criteria for the PoC, only general buzzwords.
In summary, team C wanted to conduct an 'independent' selection process for an automation solution without any real pre-selection of vendors. And this combined with a required PoC, which was to be evaluated according to keyword criteria.
The (unclear) criteria for the PoC as well as (unclear) requirements and goals for the selection process were largely determined by personal opinions and preferences of a single team member. Possible external help for the selection process of independent vendors and the preparation of a PoC were rejected on the grounds that team C has more than enough expertise to do so.
To date, team C has not made a decision on an automation solution. One of the reasons, in addition to the uncertainties mentioned above, was that business problems should be solved with a new technology. This usually does not work, because business problems have to be solved on a business level and not with a new great technology.
Thoughts
A few thoughts (almost) to conclude this series of posts. The most important thought I would like to share is that you always have to consider what challenges you want to solve with an automation solution.
Requirements and objectives
What are the requirements and what are the objectives? Where is the journey to go? When requirements and objectives are clearly defined, they provide some objective orientation.
Far-reaching decision
The choice of an automation solution is a far-reaching decision that will last for several years. Possible changes must not lead to the automation solution having to be replaced.
Proof of Concept
Define a PoC to test the defined requirements and objectives in the smallest possible timebox.
Technology is not a solution
The technology itself is not the solution to the problems. The requirements and goals are decisive for the right selection.
Circle of Competence
Know the actual circle of competence compared to the perceived circle for selecting an automation solution. Do not overestimate the level of competence.
Acting responsibly
Acting responsibly and sustainably for oneself, the project and the company.
Independent expert
Knowing when you need the help of an independent expert. You can't know everything as a project team, especially because software selection doesn't happen every day.
Being critical
Do not (blindly) trust the 'baseless' promises of the salesperson. The salesperson's goal is to sell his product, not to solve your problems.
Selection process
The requirements and objectives as well as the existing circle of competence form the basis for the actual selection process.
Structured
Start structured, use requirements and objectives to find good criteria for the selection process.
Criteria
Finding tangible and trackable criteria to evaluate the automation solution against your requirements and objectives.
Confidence
Confidence in the changes in case adjustments are needed in the future.
Case study - ideal-typical
The first three cases for selecting an automation solution were not successful. In the fourth case, the project team - let's call it team D - made a successful decision at the end of the selection process. The following figure shows the result of the selection process.
What did they do differently? Team D accurately defined and described the requirements and goals, criteria and features they needed to make a decision on an automation solution.
Categories such as metadata, security, data quality were defined and described by team D. All detailed criteria in these categories were weighted according to the requirements.
The result was that team D was able to select the best automation solution for them through the simple visualization. It did not perform best in every category, but overall it was the best result. This allowed team D to objectively select an automation solution. In terms of the requirements and the objectives that team D had, they made a good decision. One of the key success factors was that team D carefully defined all the categories and nearly 200 criteria in advance of the selection process and stuck to them. The subsequent PoC led to success.
This was the third and final part of the mini-series. But not the last of the coaching series! I would be very interested in your experiences on this topic. Write me here in the comments or directly.
Be sure to check back.
Until then
Your Dirk