The industry teaches it wrongly

Now there’s a statement to begin my first blog with! Two weeks ago I had the privilege of attending the Data Vault training by Dan Lindstedt (yes, the master himself!). Believe me, it’s a must have for any BI & DWH-professional. What stroke me in particular, apart from the attractiveness and usefulness of the DV modeling approach, was Dan’s critical attitude towards the industry and what is being told and teached about data warehousing. In his opinion, the industry teaches much of it wrongly. It was refreshing to hear his original thoughts and his motivation behind it. Things like data cleansing (downstream, not upstream), implementing business rules into the DWH-model (DON’T), who’s responsible for data quality (the business stupid, not IT!) are just some of the topics that were discussed. Not only were much of his thoughts original and stimulating, he also introduced the practical experience that backed it up. Now there’s something to think about!

Why don’t we hear, read, write, blog, you name it, much more often with this critical attitude in mind? Don’t get me wrong, with ‘critical’, I do not mean ‘conservative’. But much of the stuff that’s written about BI & DWH is slideware and hardly supported by either tons of practical experience or rock solid scientific proof. Did you know that on average, 60-70% of data warehouse initiatives fail? What does the industry do in response? Instead of investigating root causes and improving current practices and methodologies (like Dan did, thank you!), we introduce new services and products and tell the client he should abandon his current solution and invest $ 15 mio in this great new solution, called ‘Pervasive BI’, or ‘Operational data warehousing’ or We don’t help our customers this way and we don’t help ourselves! It gives the firm a nice revenue on the short term, but in the long run it hurts the industry and damages our image as professionals. Don’t get caught in the bandwagon.


Tagged with: , , ,
Posted in BI industry
5 comments on “The industry teaches it wrongly
  1. Well Wouter,

    Could not agree more with your statements. There are for too much many companies that sells Business Intelligence and data warehousing like a commodity. Well, wake up you all…IT AINT!

    By definition…i would even say…DWH and BI aint a commodity. Correctly implemted it’s the differentiation factor in a truely integrated enterprise. But it’s always tightly connected with your business processes…..and really guys….you can not buy that.

    But no….organisations like to ‘buy’ a BI or dwh solution like they buy there laptops. And bad organisations in our ‘industry’ even sell this, propogate it…market it…Well, these organisations give the trade a very bad name. And in my everyday work I have to explain why we should not buy x-tool or y-tool or go with some ‘powerpoint-specialist in the field’ (e.g. a bi/dwh solution that works on powerpoint).

    …’It sooooo goood’….’It’s soooo easy to implement’,…..’You dont even need a data warehouse, aint that cool’….’ETL is not necessary, we got adapters’…..’we got a service oriented BI environment, u can just buy off the shelf, aint that nice’…..’ETL, data warehousing, nah, not necessary with our solution, we give u a realtime solution!’

    Bullshit….I appeal to our future customers, smart and intelligent organisations; please do not fall for any company that is trying to sell you a commodity called BI or something like that…….

    And ‘the industry’…..well grow up. You’re doing a damn bad job at the moment.

    sorry…got carried away a bit…..

    Ronald Damhof

  2. Wouter (and Ronald) I cannot agree more on your opinions about this current BI world.

    Like Wouter I had the same course (and so did Ronald), you might call us brainwashed then, but hell Dan was right with all his examples about organisations lacking interest in competative advantages with their data warehouse. Data Quality is an issue for the Business. Performance has its roots in the business decision about the importance of data. Business rules are called BUSINESS rules, because they represent the truth of the business and how they intepretate that.

    Busines is the key indeed. They have the power to let the organisation cooperate with us consultants and architects, and to provide them the right solution for their management information wishes. Together as a team this makes sence, but the industry has cooperate as well. Without the tools and techniques we lack the possibility to do our work correctly.

    In my opinion the tool vendors might benefit from review session with the people working with their products. Currently I must admit I have met such a tool vendor that listens to their customers and tries to adapt tools to wishes instead of providing tools and techniques to the organisations they might need (and if not, then they find a way to let organisation think they need this kind of tool…). Let us advise them the right solutions for their needs. A tool vendor can just sell them the product, but when it comes to the business processes and knowledge they seem to have forget reality…

    I favour the Data Vault approach since having had the DV course and must admit it has ‘brainwashed’ me.. I can now think CLEANLY.

    Does this look three musketeers or what ?!?!?

    Walter Smetsers
    BI Consultant / DWH architect

  3. Dan Linstedt says:

    Gents, thank-you so much for the kind words. Please, lest we forget that the Data Vault is not _easy_ to load or query, and the approach to architecture, the methodology takes a lot of thinking and consideration, and like any “modeling” effort or EDW effort: there are pitfalls and downfalls, and issues. I see the Data Vault methods and architecture as the next step in the data modeling evoluion (a very slow evolution), but it certainly isn’t the be-all-end-all.

    I really appreciated the pro’s and con’s everyone added to the end of their tests, I’d like to see a few more of the “cons” published, along with the corresponding mitigation strategies. Why? Because knowing where the pitfalls are, and identifying how to navigate through or around them makes it easier to live with a solution.

    The class as a whole certainly has a lot of experience (between 15+ and 25+ years in the I.T. industry), so it would be good to hear more on what the issues are.

    And as always, please feel free to contact me directly, or sign up for my forums.

    Dan Linstedt

  4. Fred Schenk says:

    I think the statement about DQ being a business-only issue is a bit too harsh. For your ETL/BI you’ll have to agree upon some data definitions, not just the table structures, but the data inside those tables too. To ‘protect’ your products (etl-flows and reports) you better implement some checks to prevent the bussiness ‘upstream’ from changing their definition and not informing you. Otherwise your reports might become incomplete or complete nonsense.

    If I were the customer of those reports, I wouldn’t accept an excuse from a professional BI/ETL firm about not being responsible for the source systems. You are indeed not responsible for the source system, but you’re responsible for checking the interface and preventing garbage to enter your system. You’re responsible for discovering those data related problems and reporting them to the business. The business is responsible for getting those solved, but you can’t expect them to be able to discover those changes all by themselves.

    So you better schedule time to implement those checks into your release schedule. And no, not at the end of the schedule, rather the first thing one should do. I bet the confirmed upon definition isn’t that perfect after all, once you get the first results of those checks…

    With kind regards,

    Fred Schenk

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: