I see things going the same direction. I've always been pretty anti-dashboard because of the lack of long-term utility. If you want to make data-driven decisions, you need to spend the time codifying the decision making process and automating that. Automated actioning off data is far more impactful than automated visualization of data.
When it comes to business stakeholders, the biggest obstacle is trust. If you're not a data person, or a developer, decision logic being gated behind code is a scary black box. "If I can't control it, I can't trust it". That feeling gets even worse when we build systems that are controlled by AI or ML.
I think we need more solutions for data teams that allow business stakeholders to take part in the automated workflow deployment. Let them see how things are connected together. Let them verify that the decisions are being made correctly every day. Let them tweak levers so they have a say in how things are working. That's the only way to move beyond the current environment where everyone wants a dashboard, but no one looks at it.
That's a big problem I'm aiming to solve right now.
Wonderful comment. But trying to automate actions (write/modify underlying data or system) is often a hugely complex task compared to a visualization that allows you to form an insight. Have you found enough common/repeatable examples that you can automate or develop some type of actions template? What is your business domain?
Previous domain I worked in was digital advertising. We built solutions to automate ad and keyword creation from product feeds and promo calendars, increase/decrease bids and budgets based on hourly data, verify all settings were set to best practices, etc. on Google/Bing/Facebook. We built internal systems that allowed us to map any data view in our warehouse to any API endpoint in a templated fashion.
The technology is getting good enough with cloud warehouses that most of the logic can be defined in SQL, with a script only needing to map the results back to an API.
Current domain is SaaS. I've found many of the same types of opportunities on the sales/marketing side of things, but not at quite the same scale.
Agreed that automation it will always take more time than a visualization. If we can shift the conversation towards the expected action that will be taken off the dashboard, we can hopefully save thousands of wasted hours on visualizations that don't get used, redirecting those hours towards more impactful efforts.
I would love to expand my mindset and hear of domains where the complexity is far higher. Feel free to shoot me an email.
The way I'm (currently!) approaching this is by implementing the decision engine in 2 steps separated by an evaluation period. I refer to these to the stakeholders as "user approves" and "user reviews".
Essentially, I first implement the engine with the output being an automated list of 1-click actions (typically a set of links on an email, but could easily be a dashboard button or anything else), so the system default is "detect, but don't do". After an evaluation period of the system actions, we move onto the second part, in which the system performs the actions and produces a log of activity, plus a list of 1-click UNDO actions to the user.
The idea is that a) the system earns trust from the stakeholders due to their direct involvement, b) the system can enjoy some supervised learning from someone other than the dev team and c) worst case, if it never earns enough trust, I've still saved a stakeholder dozens of hours of work sifting through dashboards as opposed to taking a look at a pre-filtered list. A few systems have turned out well enough that they never move from Phase1 while still being considered a massive win.
I think that's a fantastic approach. You're effectively avoiding the pitfall of automating too soon while still shifting the focus towards "driving action with this data". If people click the buttons frequently, it's ripe for full automation - and they already trust it. If not, there's still an easy way to for users to get the job done quicker.
The data driven process app sounds similar to the ideas behind Knime (https://www.knime.com/) to me.
I think the visualisation matters though. It's much easier to convince people that the automated actions are worth doing once the related data is visualised. And at that point, whatever provides you both may be the best solution.
While I'm not super familiar with Knime, it looks like it falls into the legacy BI category that Benn talks about. An "all-in-one" for your data.
Is the platform still easily used by business users? Or does it primarily become a gated way for Data Engineers to build/action on data?
A lot of decisions can't be easily visualized in your standard dashboard. You can forecast the potential of those automated decisions, but that's still black box logic. You really need more of a middle ground that lets you preview "given these inputs, show me what the output would be".
> Is the platform still easily used by business users?
I don't know and honestly I don't believe there's a system that actually achieves it. But this it does go beyond BI in actually being aimed towards running code based on results. Which makes it closer to Labview-with-graphs than to BI.
This. Knime is similar to BI tools, but it is more action oriented. The reports may not be as pretty, but it can handle the entire ETL pipeline and actually do things besides make interactive pivot tables.
I've found that the vast majority of stakeholders want a dashboard either to (1) spend other people's money and feel important or (2) create opportunities for multiple "version of the truth" and thus prepare multiple narratives to pick and personally gain from when needed. Middle managers would rather reserve the right to say X dashboard has a metric that doesn't reconcile with Y other reliable source, than to actually collaborate with competent analysts and build a holistic info platform. We need to stop casting such a wide net and selling the dream of a glamorous all-knowing BI tool and instead, service analytics on a case by case basis. Over time this can inform practical dashboard design, kept up by a small team of experts and largely ignoring the "laundry lists" of every prospective user.
When it comes to business stakeholders, the biggest obstacle is trust. If you're not a data person, or a developer, decision logic being gated behind code is a scary black box. "If I can't control it, I can't trust it". That feeling gets even worse when we build systems that are controlled by AI or ML.
I think we need more solutions for data teams that allow business stakeholders to take part in the automated workflow deployment. Let them see how things are connected together. Let them verify that the decisions are being made correctly every day. Let them tweak levers so they have a say in how things are working. That's the only way to move beyond the current environment where everyone wants a dashboard, but no one looks at it.
That's a big problem I'm aiming to solve right now.