Getting Greater Enterprise-Wide Adoption of Real-Time Data Processing for a Variety of Use Cases
In this post you will find some of the key questions discussed in the Panel Discussion, the answers are the views of the author and the comments and insights as expressed during the conference.
Q1: When talking about customer engagements, there are 2 critical elements to consideration:
1. The velocity of Data – speed of access
2. Availability of data – in the right format, ready to use – complete, accurate, consistent
What is the right architecture for a segment of 1-1 communication?
Let’s start by recalling a definition by Paul Greenberg about customer engagement:
“Customer Engagement is the ongoing interactions between customer and brand, offered by the brand, chosen by the customer”
This definition lets us see an important aspect of the human-to-brand activity, customer engagement is unpredictable.
On the other hand, the “segment of one” is a marketing concept that indicates the ability to identify a customer as an individual and have all the data resources required to individualize the communications and interactions the brand has with the customer.
Combining these two concepts is the holy grail of customer and marketing operations. However, to achieve this at scale, organizations need advanced technologies and capabilities to do this consistently, efficiently, and in a way that favors the customer.
At scale, this is only possible with the support of real-time data and the architectures that enable them. At a high level, there are four components to this architecture:
- Data Ingestion: from channels and systems
- Data Storage: of customer interactions
- Data Analysis: dynamic segmentation and behavior analysis
- Decisioning: selection of brand offerings and reactions to customer requests.
- And, Activation: value added to the data pipeline.
I explore this architecture and the capabilities required in the Engagement Fabric.
From an ecosystem point of view, the architecture should include the following components:
- Data Ingestion and Integration: This is the first step in the data pipeline where data from various sources, like CRM systems, social media, web analytics, etc., are collected. The architecture should support batch and real-time data ingestion to ensure the timely availability of data.
- Data Storage and Management: A data lake or warehouse is typically used to store and manage large volumes of structured and unstructured data. This component should be scalable and support high-speed data retrieval.
- Data Processing and Analytics: This component is responsible for processing the raw data and transforming it into actionable insights. It should support advanced analytics capabilities like predictive modeling, machine learning, and AI to understand customer behavior and preferences.
- Customer Data Platform (CDP): A CDP is a unified customer database accessible to other systems. It helps create a 360-degree view of the customer by integrating data from various sources. This is crucial for the segment of one marketing as it enables personalized communication.
- Marketing Automation and Engagement Platform: This component delivers personalized messages to the customers based on the insights derived from the data. It should support multi-channel communication (email, SMS, push notifications, etc.) and real-time engagement.
- Feedback Loop: This is an important component that captures the customers' response to personalized communication. The feedback is used to continuously refine and optimize the marketing strategy.
- Data Governance and Security: Given the sensitive nature of customer data, the architecture should have robust data governance and security mechanisms in place. This includes data privacy, quality, lineage, and access controls.
On a more technical level, the data architecture includes different technologies and patterns, such as:
- Lambda Architecture: The Lambda architecture separates the processing into two paths: the speed and batch layers. The batch layer manages the master dataset and pre-computes the batch views. The speed layer compensates for the high latency of updates to the serving layer and deals with recent data only. The serving layer indexes the batch views so that they can be queried ad ad-hoc. It's powerful because it combines batch and real-time processing methods to provide a robust and scalable solution for handling large amounts of data.
- Kappa Architecture: Kappa architecture is a simplification of Lambda where the batch layer is removed, and all data is processed in a streaming manner. This addresses the complexities of managing both a batch and real-time version of the same computation. It is also easier to manage since there's no need to maintain two separate systems.
- Event-Driven Architecture: In this pattern, a system reacts to events generated from within the system or external systems. This type of architecture is particularly useful in real-time data processing because it allows systems to respond immediately to real-time data. For instance, in financial systems, this architecture could be used to respond to changes in market data and make trades in response to these changes.
Q2: Which of these patterns is more suitable for customer engagement in a multi-dimensional world of customer touchpoints and where the irrationality and unpredictable nature of human choice is considered?
All three of the patterns can handle customer engagement data to some degree, but given the specific nuances of your question, an Event-Driven Architecture may be the most suitable.
In an Event-Driven Architecture, data processing systems are designed to react to real-time information. This architecture is particularly useful in environments where you need immediate reaction and where there is high variability and unpredictability - just like in customer engagement scenarios where touchpoints are multi-dimensional and human choices can be irrational and unpredictable.
An event-driven system can capture and process a wide array of customer interactions as they occur, including web clicks, mobile app usage, customer service interactions, social media mentions, purchase transactions, and more. The architecture's ability to handle a high volume of events and react to them in real time allows for a more dynamic, immediate understanding of customer behavior.
Furthermore, an event-driven system can react to these events in real time. For instance, it could trigger a promotional offer for a customer who abandoned their shopping cart, recommend a product based on the customer's browsing history, or alert customer service to a problem that a customer is experiencing.
That being said, no architecture will completely solve the problem of predicting human behavior, which can be highly irrational and unpredictable. However, These systems can provide valuable insights into patterns and trends that inform strategies and drive decision-making.
Lastly, it's also important to remember that a robust data science approach is needed to build models that can handle and make sense of the complexities of human behavior. This typically involves machine learning and potentially more advanced techniques, such as reinforcement learning, which can handle the complexities of a multi-touchpoint environment where the "rewards" of certain actions (i.e., customer purchases or other forms of engagement) might be delayed.
Q3: I read an article where you talked about the Engagement Fabric and the limitations of Enterprise Architecture. Can you talk more about this?
The Engagement Fabric is a concept to model and map the universe of customer and brand encounters where each encounter has a context. The relationship between brand and customer is defined by these interactions and not by pre-defined or dictated steps pre-conceived by the brand.
Solving the complexities of the Engagement Fabric requires a set of advanced technical capabilities that are generally provided and designed by Enterprise Architects. However, the fabric’s main elements, such as the customer activity across touchpoints and the people and processes that support this activity, are understood by other business areas such as CX, Service, and Operations.
In the early days of the digital era, humans and businesses had few interactions across a limited number of touchpoints. But as the internet and communications networks exploded, device fragmentation, proliferation, and adoption skyrocketed, and money and business models erupted, the possibilities for interactions between humans and businesses grew exponentially and multi-dimensional.
Today, Businesses face a real challenge in keeping their customers engaged, and remember, as once defined by Paul Greenberg, customer engagement is the customer's choice and not a mandate of the brand or business.
In the exponential age, businesses that want to thrive must carefully design and architect their capabilities centered around Engagement.
In other words, a solid and mature approach to architecting an enterprise is engagement-led, and the Engagement Fabric is the cornerstone of this approach.
Q4: In a customer communication model, a customer may switch from a single channel to multiple channels; how do we manage large complex calculations in these ecosystems, or how do you navigate the decision-making process when choosing data formats, schemas, and development framework?
The Engagement Fabric and the Data Mesh paradigm share a common principle: decentralization. The Engagement Fabric decentralizes customer interactions, treating each interaction as a unique encounter with its own context. Similarly, the Data Mesh paradigm decentralizes data ownership and management, treating each data domain as a separate product with its own product owner and team.
In an omnichannel customer communication model, the Engagement Fabric's approach to managing complex calculations across multiple channels parallels the Data Mesh's approach to managing data. Just as the Engagement Fabric breaks down customer interactions into individual encounters, the Data Mesh breaks down data into domain-oriented datasets. This decentralization allows for more efficient and scalable management of data, as each team can focus on its specific domain, reducing the complexity of handling large amounts of data.
When choosing data formats, schemas, and development frameworks, both the Engagement Fabric and Data Mesh advocate for context-specific and adaptable decisions. In the Engagement Fabric, this means choosing technologies that can handle real-time interactions and context enrichment. In the Data Mesh, this means choosing technologies that best suit the needs of each data domain, allowing for flexibility and adaptability as those needs evolve.
The Engagement Fabric's emphasis on real-time interaction and context enrichment aligns with the Data Mesh's emphasis on providing real-time, actionable insights. In both paradigms, the ability to process and analyze data in real time is crucial for delivering personalized experiences and making informed decisions.
In conclusion, the Engagement Fabric and the Data Mesh paradigm offer complementary approaches to managing customer interactions and data. By decentralizing and contextualizing these processes, businesses can enhance their customer engagement strategies and effectively use their data. This parallelism between the Engagement Fabric and the Data Mesh provides a comprehensive framework for navigating the complexities of customer communication ecosystems and data management.
Q5: Can you provide some examples where you have used real-time data and what benefits this has provided?
In today’s world, companies have incredibly powerful technologies and advanced frameworks to make processes incredibly fast while using real-time data and architectures.
The first instinctive reaction to such capabilities is to think that real-time is a tool that can solve most problems in customer engagement and communications. However, this is not always the case. Not too long ago, in another data conference, I heard the CTO of a digital bank explain how well their data streaming architecture worked and how easily it broke the customer experience and derived reputational damage to the brand.
There are scenarios, like in the case of loan requests from customers, where it’s expected that the brand “takes time” to consider the customer’s request and carefully analyze the case before providing a “robotic” and immediate response.
This happened to this digital bank, they could streamline the loan assessment process to provide a response to the customer in less than a minute, which seemed a win for the brand. However, in the many cases where the response returned a negative assessment for the customer, they felt unheard and unfairly rejected because of the lack of consideration; the bank didn’t take more than a minute to understand their case! The brand received the impact of negative reviews, which forced the bank to introduce a “lag” in the process and an expectations management process upfront.
It’s quite important to understand that the rational use of any technology should prevail, but also that if we have proper structures to understand the human-to-brand relationships, we’ll have a better understanding of the impact of new processes and new technologies in the customer experience.
Q6: What are the evolving skills sets in that data architecture world – any key themes as closing remarks?
Data architecture is a critical component of modern business strategy, and it has evolved significantly with the advent of new technologies and methodologies. Here are some essential hard or technical skills in today's data architecture world:
- Data Modeling: The ability to design and implement effective data models is crucial. This includes understanding concepts like normalization, entity-relationship models, and schema design.
- Data Governance and Quality: Ensuring data accuracy, consistency, and security is a key part of a data architect's job. This includes understanding data governance strategies and quality tools and methodologies.
- Business Intelligence and Data Visualization: Data architects must understand how to use data to drive business decisions. This includes knowledge of BI tools like Tableau or PowerBI and data visualization techniques.
- Programming and Scripting: Knowledge of programming languages like Python, Java, or R and scripting languages like SQL is important for manipulating and analyzing data.
As for the key themes related to enterprise marketing technology, we could mention:
- Customer Experience Management: This involves using technology to track, oversee, and organize every interaction between a customer and the organization throughout the customer lifecycle.
- Omnichannel Marketing: This is the practice of marketing across multiple platforms, including email, apps, social media, and your website, so that you can connect with customers on more touchpoints.
- Artificial Intelligence and Machine Learning: AI and ML are being used to automate tasks, gain insights from data, and provide personalized experiences to customers.
- Data Analytics and Predictive Analytics: Companies leverage data analytics to understand customer behavior, measure campaign performance, and predict future trends.
- Individualization: This involves using technology to deliver individualized messages and product offerings to current or prospective customers.
- Privacy and Compliance: With regulations like GDPR and CCPA, businesses need to ensure they're compliant in handling customer data.
- Integration: Marketing technologies must be integrated for data to flow between them, allowing for better coordination and efficiency in marketing efforts.
However, from my point of view, one key skill that will become more and more relevant in the near future is using critical thinking and systems thinking applied to problems to find more and better questions instead of the rush of quick-finding of specific answers to problems.
We live in a world of abundance of data and tools to interrogate them, we should utilize our intelligence to make the most out of both.
.png)
.png)
