David Monnerat

Product + AI | Systems Thinker | Enterprise Reality

Tag: customer experience

  • If You Want to Keep Reading This Post

    If You Want to Keep Reading This Post

    If you want to keep reading this, I’m going to need you to type something on your keyboard every two minutes.

    Can you imagine? You’d close the tab. You’d go somewhere else. You’d tell someone about it.

    That’s the experience I had last night trying to activate a new phone through a major telecommunications company — one I actually worked for, about six years ago, when we were identifying these exact kinds of problems and trying to fix them.

    I typed my issue into the chat assistant. It responded with a workflow of pre-set options, none of which matched my situation. I clicked through five or six of them before typing “agent.” It asked me three more questions. Then it connected me to a human.

    The first thing the human said, after the greeting, was that I needed to type a message every two minutes to keep the session alive.

    Not them. Me. The customer who already has a problem. I have to do work to stay connected to the person trying to help me.

    That’s not an AI problem. It’s not even a problem AI can solve. It’s a product decision that someone made, or a technical limitation that someone accepted, and it’s been sitting there making customers feel like a burden since before I worked there.

    The Mute Button That Wasn’t

    While I waited, I started my audiobook on the same phone I was using for the chat. Every time a message came in from the agent, a notification sound interrupted the audio.

    I noticed a mute icon at the top of the chat. I tapped it. The icon changed to show it was muted. I restarted the audiobook.

    The next message came in. The sound played. The audiobook stopped.

    The mute button didn’t mute anything. It changed its icon. Those are not the same thing.

    This is a product problem. A design problem. Someone built a feature, shipped it, and either never tested it in a real use scenario or tested it and accepted the gap between what it looked like and what it did. A customer using a phone to listen to something while they wait on a chat is not an edge case. That’s Tuesday.

    These are the kinds of things you find when you walk through your own product as a customer. Not in a lab. Not with a test account. As a customer, with a real problem, in a real context, trying to get something done.

    An Hour Later

    After about an hour of troubleshooting, the agent, who was genuinely helpful and clearly trying, told me there was a process that needed to run on the back end. They’d put in a ticket. It would be resolved, but not tonight.

    Which meant I’d be without a phone until it was.

    I asked what I should do if there was an emergency. They offered a workaround: set up a new line, activate the phone on that, switch back later. Creative. But if the activation experience was this complicated, kicking it down the road felt like borrowing trouble. I declined and started wrapping up the chat.

    Then the agent, almost certainly prompted by their system, offered to tell me about current internet plan upgrades.

    I want to be precise about what happened there. My problem wasn’t solved. I was going to be without a phone overnight. The chat had taken an hour. And the system fired an upsell.

    That’s not a bug. Someone designed that. Someone looked at the data on upsell conversion and decided the trigger should fire regardless of whether the customer’s issue was resolved. The metric the team is measured on includes upsell attempts. So the system attempts the upsell. The customer experience is not the metric.

    What AI Has to Do With Any of This

    None of these problems are AI problems. The session timeout, the mute button, the upsell timing — none of them require a language model to fix. They require someone to walk through the product as a customer, identify what’s broken, and have the organizational support to fix it.

    Six years ago, when I worked there, we were calling out problems like these. Some got fixed. Some didn’t. The ones that didn’t weren’t technically hard. They were prioritization decisions. There was always something more exciting to work on, something more fundable, something that showed better in a demo. Now that something is AI.

    I work in AI. I know what the technology can do and I believe in it. But companies are investing heavily in making their customer service smarter while the foundational product and design problems that have existed for years go unfixed. The AI layer gets better. The experience layer stays broken. The customer lives in the experience layer.

    Here’s the question worth asking before the next AI investment: if you asked your customers what frustrated them most about your product, would any of them say it doesn’t have enough AI?

    Probably not.

    They’d say the mute button doesn’t work. They’d say they had to type every two minutes to keep the session alive. They’d say someone tried to sell them an internet upgrade after failing to activate their phone.

    Those are fixable. They were fixable six years ago. They’re still broken.

    Not because the technology doesn’t exist to fix them. Because the attention is somewhere else.

    I’m still thinking about the mute button.

  • RAG Against the Machine

    RAG Against the Machine

    Who controls the past now controls the future
    Who controls the present now controls the past.

    Rage Against the Machine, “Testify”

    There is irony in starting this post with a Rage Against the Machine (RATM) quote. They’re not just a band known for relentless protest against oppression, corporate greed, police brutality, war, propaganda, and systemic injustice; they’re a call to action, urging listeners to question authority, challenge the status quo, and fight for change.

    Quoting RATM in a post aimed at helping corporations use AI more effectively would draw the ire of Zack de la Rocha. However, my motivation is to question the blind adoption of a technology without doing the hard work that would improve outcomes for the people, in this case, the users and customers.

    The Status Quo

    In my career, I’ve spent a lot of time focused on improving the customer experience, whether working on self-service tools like chatbots or finding ways to help internal employees better serve customers. In both cases, the goal is to provide the right answer at the right time to help customers get the most out of our products and services.

    A common challenge in customer service, especially in call centers, is finding the answer to satisfy a customer’s need. Often, companies have large collections of documents managed by a content management system (CMS). Different yet overlapping documents may exist across products or business lines or intended for different audiences, such as internal or customer-facing documentation. Errors and updates may inconsistently cascade throughout the documentation, resulting in missing, incorrect, or conflicting information. As a result, the CMS may return the wrong results, causing slower response times, conflicting answers, higher call volumes, and frustration by both employees and customers.

    The RAG

    With the availability of large language models (LLMs) and the introduction of retrieval-augmented generation (RAG), organizations saw a possible solution to improve the search process. RAG offered a way to retrieve a specific answer to a question, rather than returning a document in a search result that would need to be read to find the answer. It was also seen as a way to overcome the challenges of poor documentation.

    Companies quickly discovered, however, that this wasn’t the case. Whether it was due to missing, incorrect, or conflicting information1,2, the adage of “garbage in, garbage out” applied and resulted in poor performance.

    Additionally, the documentation format made it difficult to ingest and maintain the complete context, even if the documentation itself was correct. Key information in images, tables, and complex formatting is often lost or misinterpreted in a RAG system.3

    Ultimately, these challenges caused these systems to be delayed or prevented them from being deployed at scale.

    The Machine

    The “machine” represents the continuation of the status quo. In this case, it symbolizes the tendency to think every problem can be solved with the latest technology. It involves our inclination to blindly trust often-overhyped technology to solve problems without fixing any underlying issues. It also includes grinding through poor implementations and subpar results rather than dedicating resources to evaluating other less-hyped or less-technical solutions.

    I’ve worked on AI products where the underlying data was so bad that it was impossible to make reliable, actionable predictions. But rather than direct teams to fix the underlying data, the machine allows for only one direction–forward. The only acceptable actions were to continue to endlessly iterate and hope for a different result, to lower the bar for success, or look for a problem that we could solve with the bad data, even if it doesn’t align with the original or important business goals.

    It’s happening again with RAG and generative AI. The machine is seeking technical tweaks to improve the results rather than fixing the underlying documentation or converting it to a more RAG-friendly format. Change the chunking strategy. Fix the embeddings. Adjust the prompt. Select a different algorithm. Use reinforcement learning with human feedback. 4,5,6

    Of course, these are great recommendations when the underlying problem isn’t the data or the documentation. But they’re ineffective when the problem is the source. No amount of tweaking will fix fundamentally bad documentation.

    The Resistance

    If the machine perpetuates the existing pattern, the resistance challenges that pattern, takes a different approach, and focuses on the business goal, not the technology.

    At its core, RAG is designed to make searching better. It includes additional benefits such as synthesizing coherent, context-aware responses, but in service of returning the most relevant, useable answers to a user.

    To take advantage of RAG’s secondary benefits, companies should not rush into the technical solution pushed by the machine and instead spend time fixing the documentation. As controversial as that recommendation seems, it’s the most fundamental task likely to improve search results.

    In a world where generative AI initiatives benefit from seemingly unlimited funding, requesting resources to update documentation will likely draw questions.

    Why can’t we solve this with technology? Have you heard about this new feature that might solve the issue? Can we bring in a new vendor?

    It is up to leaders to take a stand and convince the business to take a different path. It requires focusing on the business problem, identifying the root cause, and recommending the best way to address it.

    If the root cause of poor search results is the documentation, addressing issues in the documentation is the best course of action and may show immediate improvements with the existing search solution. It’s possible the improvements will be dramatic enough to not require RAG at all, at least right away. But it leaves the door open to implement a RAG-based solution on top of better documentation to take advantage of RAG’s additional benefits.

    The machine will persist.

    But leaders must resist.

    Conclusion

    The allure of new technology is powerful. The promise of AI and RAG as instant solutions to deeply rooted problems is tempting, especially in corporate environments eager for innovation. But when the fundamental issue is bad documentation, no AI model—no matter how sophisticated—can fully compensate for it.

    The real battle isn’t against the technology itself but against the mindset that technology alone is always the answer. True leadership means recognizing when the most effective solution isn’t the most exciting or trendy one, but the one that addresses the root cause.

    Fixing documentation isn’t glamorous. It won’t make headlines like an AI-powered chatbot or a multimillion-dollar LLM investment. But it’s the foundation of good search, good RAG, and ultimately, good customer experience. Without it, RAG becomes just another iteration of the machine—grinding forward without progress.

    So before blindly deploying the latest AI tool, ask this question: Are we solving the problem, or are we just feeding the machine?

    1. https://www.valprovia.com/en/blog/top-7-challenges-with-retrieval-augmented-generation
      ↩︎
    2. https://www.aimon.ai/posts/top_problems_with_rag_systems_and_ways_to_mitigate_them
      ↩︎
    3. https://arxiv.org/abs/2401.05856
      ↩︎
    4. https://snorkel.ai/blog/retrieval-augmented-generation-rag-failure-modes-and-how-to-fix-them/
      ↩︎
    5. https://medium.com/towards-data-science/12-rag-pain-points-and-proposed-solutions-43709939a28c
      ↩︎
    6. https://doi.org/10.48550/arXiv.2501.13264
      ↩︎