Earlier in my career, as a data analyst, the lag time between their requests and business insight was very frustrating for my marketing clients. The log is extended if the required data has to be taken from a new source. The old ETL framework and associated data management tasks created a long wait for actionable insights to inform marketing strategies.
Throughout my career, despite advances in data strategies and infrastructure, questions remain such as:
- “Why can’t I get my tests sooner?”
- “Why can’t I access my own data?”
- “Why does it take so long to source the new data we need for analytics?”
A recent concept is trying to reduce the stress associated with these questions Data fabric. According to Gartner, the data fabric…
“An architecture pattern that informs and automates the deployment of data objects regardless of design, integration, and deployment platforms and architectural approaches.
“It uses continuous analytics and AI/ML on all metadata assets to provide actionable insights and recommendations on data management, integration design and deployment patterns.
“This leads to rapid, informed and (in some cases) complete automation of access and sharing.“
A very long multi-layered definition, but based on this promise With data fabrics, data analytics teams can now:
- Avoid moving data
- Have better access controls and democratize access
- Deliver faster, more automated analytics through AI
Business managers often think about these issues and their impact on their business:
Faster time to insight (lower cycle times): There is no ETL, or ELT, or other time-intensive data management tasks associated with data analytics. AI/ML tools automate data tasks and data virtualization allows analysts to go directly to source elements avoiding time-intensive access and exploration, with the exciting prospect of eliminating extract tools, schedulers and more.
Cost savings: By going straight to the source, there is no need for intermediary platforms, systems or repositories. So platform/tool costs and recurring maintenance costs are saved.
Why move data? (When you don’t need to!)
The problem many organizations face is storing their data in multiple silos and systems—clouds (lakehouses), warehouses, source systems, ODSs, CRM systems, marketing marts, and in some cases, legacy SAS files. Each of these silos introduces necessary workflows but increases insight from time to time. What organizations are like Make efficient use of all their data?
The idea behind data fabric is to break down data silos and put data into the hands of users. The data fabric is the tapestry that connects data across all platforms to users, creating efficiencies without having to move data. Data automation capabilities in the fabric further increase efficiencies by providing accessible, quality data through AI and other tasks.
One of the key enablers of the data fabric is the virtualization layer, which serves data requirements directly (from transactional and operational systems) without moving or copying data. Data tasks like sourcing, extracting, cleaning/transforming etc. are all automated. Fabric helps manage the data lifecycle, for example, by controlling your data using proactive metadata to enforce policies including access, compliance and quality.
Briefly, data fabrics are still evolving—the architecture has merit, the concept seems more advanced than the data mesh (which is more of a theory), and the foundational tech capabilities are well underway. I see fabrics as having great promise, but as an architectural strategy that focuses on less data movement, fewer silos, greater access to data, and faster analytics. Do fabrics eliminate the need to move data altogether? I don’t think so (at least in the near term), moving and replicating some data will always be necessary. Would also love to hear from vendors in the space on this post.
Is it data fabric or data fabrication? Only further development (and time) will tell, but I’m betting on the fabric’s promise!
I’m betting on you, my readers, too—I’m betting that a discussion with you on this topic will reveal many challenges and opportunities for data fabrics. What would you like to know more about? What is your experience with Data Fabric? Is it fabric or fabrication? Please reply to this post with your comments so we can all delve deeper into the strategic, tactical and business implications of this fascinating area.
From your site articles
Related articles around the web