When Everything Is an Insight, Nothing Is: Why AI Output Depends on Context
Post adapted from “Reading Between the Lines” Issue #1, Oct 19 2026
Imagine a research dashboard where nearly every sentence from every customer interview has been tagged, with complete confidence, as an insight. I've seen that dashboard. Even though the dashboard was technically “working”, human-centered perspective and judgment weren’t yet taken into account.
The prompt said "find insights"
This was a core piece of functionality in an AI-native SaaS platform. Its job was to use AI to analyze user research, such as interviews, surface insights in them, and guide teams to actionable knowledge built on those insights. The engineering foundations and pipeline were in place. The platform was hooked up to all the cloud-based architectural components: an LLM, vector embeddings, a vector database, the whole nine yards. Transcripts flowed in. Via an API call, the LLM was prompted to “find insights” in the unstructured content.
And, voila!, insights appeared in the interface.
Or, did they? Because, actually, most of what came back was noise. There were just too many “insights”, most of them not very insightful.
Almost every statement a user made was labeled an insight. A passing remark about using the mobile app when out and about and other obvious statements were all considered insights. The dashboard was so overcrowded that it was like finding a needle in a haystack. Product teams couldn't tell which findings deserved their attention.
Users didn't stay on the platform.
What makes an insight insightful? And, why didn’t the model know?
It's tempting to call the kind of behavior we were seeing a model failure. “Why doesn’t it know what we mean? Why does it think all these things are insights?”.
In reality, it was a design failure.
The prompt being used in these early versions was just connecting the plumbing. It didn’t really describe the job to be done. There was no context. None of the implied meaning had been spelled out. Considerations about how LLMs and generative AI fundamentally work and how to handle that hadn’t been taken into account yet, either.
A language model given the word "insight" and nothing else produces what that word typically means across all the data it’s learned from. In that broad, everyday sense, almost anything a person says about their experience could qualify. Combined with LLM proclivities for verbosity/overgeneration and “people pleasing”, it’s no wonder so many insights were surfacing.
To an experienced researcher, "insight" is a term of art with many implications.
An insight is not data, information, or an observation;
An insight is supported, whether by more than one data point or with striking clarity and depth from a single person;
An insight surfaces something meaningful and powerful that wasn’t already known or couldn’t be articulated;
Finally, an insight bears on a downstream decision.
This framing of “insight” lived in the heads of the product team and relied on their deep research expertise. In my experience, that's how expertise and judgment usually work. It stays tacit until someone asks for it in writing.
Turning tacit expertise into context AI can use
It was a constant refrain, “Why can’t AI surface good insights?”.
My position: AI is capable of surfacing high quality insights, and doing many other things, when we equip it to do so.
This isn’t unique to user research platforms. It’s pervasive across almost all domains that use AI. In health literacy, for example, the need to equip AI with cultural, social and medical context comes up again and again, as I wrote about here.
In this case, we needed to codify what an insight was in the first place and, importantly, what examples of good insights (and bad ones!) look like and what makes them good (or bad).
Not surprisingly, while the team could spot a good insight in seconds, articulating what an insight was, what made an insight “good”, and how it was different from (and related to) other concepts in the platform was harder.
Capturing a team's standards and semantics is part art and part science. This was no exception.
Some answers came quickly. Others required people to agree on things that had been implicit or where shared understanding had been assumed. Once the answers existed, they could be structured and written into the context the system relied on in ways that humans could interpret and the system could consume.
We then evaluated outputs using a principled methodology and framework that allowed the qualitative nature of this type of evaluation to be reliably quantified across users and data sets.
Using this framework, we iterated on the prompt and supporting context in a lather-rinse-repeat fashion until results consistently reflected the team’s standards and semantics while staying within budget and processing constraints.
The platform began surfacing smaller sets of high-quality insights across different data volumes and topics. Users went from drifting away to loving it.
Did a better prompt with robust supporting context fix the issues we were seeing? Yes, and it’s important to keep in mind that the improvements were the result of doing the hard work, not just using clever wording or making the prompt “prettier”. Encoding human judgment, expectations, and critical thinking into what we gave the system to work with, i.e., representing human knowledge, was what made the improvements.
Increasingly, this kind of work is being referred to as “context modeling” or “context engineering”. At its core, I think of it as knowledge representation. It’s the process of deciding what the system needs to know and encoding that in ways that it can use.
The same problem hides in everyday AI prompts & usage
You don't need to be developing an AI-native SaaS platform for this to apply. Plenty of ordinary requests cram a whole definition inside a single word. Many fail to establish the shared standards and meaningful semantics that equip AI systems to produce the kinds of outputs that we actually want.
It’s easy - and tempting! - to skip the hard work, whether because it doesn’t fit neatly into the boxes we use to track our work, or because we don’t know how, or we’re rushed for time.
Think about examples like these:
"Prioritize these support tickets."
"Summarize the key risks in this vendor contract."
"Score these leads."
In each case, the model will produce something.
Often, it will be fluent and interpretable.
Whether it's actually useful for your organization depends on whether what’s important to you, i.e., your definitions, your standards, your expectations, your human know-how, ever made it into the system in a way that both humans and machines understand.
How to bring human know-how into your AI work
There are lots of ways to bring your human knowledge and critical thinking into how you use generative AI. And, it doesn’t have to be a “boil the ocean” scenario. Even a drop in the bucket can make a big difference.
A simple starting place is to reframe how you engage with generative AI, if you haven’t already. Move towards asking generative AI to do a job or a task, rather than asking it to simply answer a question. With that simple reframing, it becomes easier to think about what a job well done looks like and what matters to you.
How deep you go with this depends on what you’re using it for, how you’re using it, who interacts with the outputs, and why. There isn’t a “one size fits all” solution. That said, there is an opportunity for all of us to consider how we can bring human know-how and understanding into how we use AI.
Why context modeling matters
AI systems are only as good as their design. Increasingly, the human expertise we bring to them and how we do that are what make them shine or fail. That’s why context modeling matters so much.
When shared semantics, standards and understanding are encoded for the system to use, there are higher quality, fit-for-purpose outputs in lower volume. No one is drinking from a firehose spouting sludge any more. (Thank goodness! That sounds yucky, doesn’t it?)
And, when AI is creating high-quality outputs that are fit-for-purpose, human expertise is what drives using those outputs. It’s where we have the opportunity to be strategic and innovative as humans on a different scale.
It takes hard work to encode human know-how and tacit understanding so that both humans and machines can use it. But, it’s how we can start to realize real value with AI. Otherwise, we’re just back to noise.
Food for thought
Here’s some food for thought for you today. Think of a task you or your team routinely use AI for. Which single word or phrase in that task is carrying a definition nobody has written down or that nods to something assumed, rather than explicitly stated?
Can it be written down?
And, would everyone agree with it?
Thanks for reading!
If you enjoyed reading this post, consider signing up for our newsletter “Reading Between The Lines”. It’s distributed up to twice monthly, some time around the middle of the month and the end of the month/early the next month. All newsletter issues get posted on our blog, too.
Reach us at hello@subtextive.com .