AI-generated approximation. All characters are fictional, inspired by published work of real thinkers (Stanisław Lem, Simon Willison, Nassim Taleb, Ethan Mollick). Not reviewed or endorsed by any original thinker.
2026-03-13 · open-table · Facilitated by none (Open Table — self-facilitated)
What are the limitations of current-day AI?
A café table in a university district, late morning. Two cups of coffee, a laptop open on Tom’s side, nothing in front of Kaspar but a napkin he’s already begun folding into a small geometric shape.
So — limitations of current AI. I have a list. I always have a list. Do you want to hear it, or do you want to tell me why having a list is the wrong approach?
I admire your directness. You have already answered the question by asking it that way. You assume the limitations are enumerable. That they sit still while you write them down. That the act of listing them does not itself distort what you are looking at. This is precisely the comedy I find most delicious — the taxonomist who believes the butterfly is still a butterfly after it has been pinned.
The butterfly is dead on the pin, sure. But the pin tells you something useful. I work with these models every day. I know that if I give an LLM a 200,000-token document and ask it to find a specific detail on page forty-seven, it will sometimes hallucinate the answer. That’s not a philosophical condition. That’s a bug with a workaround — I chunk the document, I use retrieval, I verify. The limitation existed, I named it, I built around it.
And you are satisfied that the limitation you named is the limitation that matters?
It’s the one that was breaking my tool. So yes.
You see, this is where the comedy deepens. In Solaris, the scientists spend decades classifying the ocean’s formations — they name every shape, every eruption, every symmetroid. They produce enormous taxonomies. And the ocean does not care. It is not doing the things they are naming. Their entire classification system is a monument to their own pattern-matching reflexes, not to the ocean’s nature. You are classifying the formations of a system whose actual operations you do not understand. You call this “hallucination” — what a beautifully human word. The machine does not hallucinate. It does something for which you have no word, so you borrow one from psychiatry and feel that you have explained something.
I don’t need to understand the ocean. I need to ship a working tool by Friday. “Hallucination” is a good-enough label for “the model generated confident-sounding text that was factually wrong.” I don’t care if the word is philosophically imprecise. I care that my users understand what it means and how to check for it.
And this is exactly the position of every Lem protagonist before the humiliation arrives. “I do not need to understand the ocean, I need to build the station.” The station functions — until the ocean sends you your dead wife reconstituted from your memories, and suddenly the question is not whether your instruments work, but whether you have been asking the right questions at all.
That’s a great novel. It’s a terrible product strategy.
[folding the napkin into something that might be a crane or might be a satellite] It is not a strategy at all. That is my point. I am not offering you a strategy. I am telling you that the most important limitation of artificial intelligence is the one that will make your list look quaint in five years — and it is the one you cannot name now, because naming it requires a vantage point you do not yet have. You are inside the system. The system cannot describe its own boundaries.
Okay, but here’s my problem with that. While you’re contemplating the unknowable, I’ve got actual failure modes that are hurting actual users. Multi-step reasoning falls apart after three or four hops. Tool use is unreliable — the model calls functions with wrong arguments maybe fifteen percent of the time. Long-horizon planning degrades. These are measurable. I can show you benchmarks.
You can show me benchmarks. You can also show me benchmarks from three years ago that measured things we no longer consider limitations. The context window was 4,000 tokens. Now it is two hundred thousand. Was the limitation the context window, or was it your assumption about what “enough context” meant? The benchmark measures the boundary of today’s incompetence. It tells you nothing about the shape of the incompetence.
I think we’re stuck. You keep telling me that naming limitations is itself a limitation. I keep telling you that unnamed limitations don’t get fixed. We need someone else at this table.
On this, the builder and the satirist agree. We are circling. We need someone who asks a different question — not “what are the limitations” and not “why is listing them futile,” but perhaps… who pays when the limitations fail? Who bears the consequence? Because I suspect that question would tell us which limitations are merely inconvenient and which are — how to put this — structurally murderous.
A risk thinker. Someone who distinguishes between uncomfortable and catastrophic. I know who I’d want — someone who asks which limitations actually kill you.
A chair scrapes. Abel Caine sits down without being formally invited, as if the empty chair had been waiting for exactly this kind of question.
I heard “who pays when it fails.” That’s the only interesting question about any system. Tell me what you’ve got so far.
Two positions. I say AI limitations are a practitioner’s inventory — name them, work around them, ship. Kaspar says the act of naming them is itself a misunderstanding, because the real limitations are the ones we can’t see from inside the system.
You’re both right and you’re both useless. [to Tom] Your list is necessary but not sufficient. You’re cataloguing what breaks under normal conditions. That’s fine for normal conditions. But the limitations that matter — the ones that will actually destroy value — are the ones that break under stress, under tail risk, under conditions you haven’t tested because they haven’t happened yet. [to Kaspar] And your position is correct but sterile. Yes, the system can’t describe its own boundaries. Congratulations — you’ve rediscovered Gödel with better prose. The question isn’t whether we can name all the limitations. The question is whether the limitations we can’t name are fragile or antifragile.
Fragile or antifragile. You are applying your framework already. I will not stop you, but I will observe that this is also a kind of taxonomy — you have simply replaced Tom’s list of specific failures with a binary classification of failure types.
It’s not a binary. It’s a triage. Some AI limitations are fragile — meaning the system gets worse as you stress it. Hallucination under adversarial prompting. Reasoning collapse on out-of-distribution inputs. The more you push, the more spectacularly it fails, and the failure is silent — it looks like confidence. Those are the limitations that kill. Other limitations are robust — they don’t change under stress. Context window limits are robust: the model doesn’t suddenly hallucinate more when you hit the limit, it just stops accepting input. And some limitations — a very few — are actually antifragile. The limitation forces a behavior that improves the system.
Give me an example of an antifragile limitation.
The fact that LLMs can’t reliably do math. That limitation forces tool use — the model calls a calculator. The resulting system is more reliable than a model that could do math sometimes, because “sometimes correct” is the most dangerous category. The limitation that forced the workaround produced a better architecture than the capability would have.
[sets down the napkin, which has become a small hexagonal shape] I confess this is interesting. You are saying that some ceilings are load-bearing — remove them and the building collapses.
Exactly. And the current discourse treats every limitation as a problem to solve. “Next year the model will be able to do X.” Maybe. But if X was forcing a useful constraint, then solving X creates a new fragility. Nobody asks: which limitations should we keep?
I actually agree with that. I’ve seen it in practice. When models got better at code generation, developers stopped reading the output as carefully. The capability improvement created a new failure mode — unreviewed AI-generated code in production. The limitation of “the code was obviously bad enough to require human review” was protecting the system.
So the builders discover what the satirist already knew: the improvement is the catastrophe. Lem wrote this in The Cyberiad — two constructors build increasingly powerful machines, each one solving the problems created by the last, until they have constructed a device that destroys everything including the constructors. The punchline is that they were trying to build a machine that could do “nothing.” The limitation was the specification.
Your fiction is entertaining but it lacks skin in the game. Nobody in the story bears a real cost. In reality, the people who bear the cost of AI’s limitations are not the people building the systems. The developer ships, the user suffers. The AI company publishes benchmarks, the hospital uses the model for triage. That asymmetry — builder separated from consequence — is where the real fragility lives.
That’s fair. I think we need one more voice at this table. We keep talking about limitations in the abstract — even Abel’s risk framing is still about the system. What about the people who are using these tools right now, successfully, despite the limitations? Someone who knows what users are actually doing with AI — not what they should be doing. What they are doing.
Yes. Someone who has studied the adoption, not the architecture. Because I suspect the most interesting limitation is not in the machine at all — it is in the failure of institutions to develop honest relationships with what the machine cannot do.
Joel Venn pulls up a chair from the next table over, tablet in hand, open to what appears to be a survey dataset.
I caught the end of that. You want to know what people are actually doing. I can tell you. They’re using it. They’re using it badly, creatively, in ways nobody designed for, and they’re getting value out of it. The limitations you’re discussing — hallucination, reasoning collapse, fragile failure — are real. And users are routing around them instinctively, the way water routes around rocks. The interesting question isn’t “what are the limitations.” It’s “what organizational structures help people work with the limitations instead of pretending they don’t exist?”
That’s a different question than the one we started with.
It should be. Because the original question — “what are the limitations” — has a shelf life of about eighteen months. Whatever you list today will be partially obsolete by the time you publish it. But the meta-question — how do organizations develop a healthy relationship with a tool whose limitations are shifting underneath them — that’s durable.
I want to push back on that slightly. Some limitations are durable. The model doesn’t know what it doesn’t know — it can’t reliably flag its own uncertainty. That’s been true across every generation. I don’t think that’s going away, because it’s architectural, not a capability gap.
And now Tom arrives at my position by a different road. “The model doesn’t know what it doesn’t know.” This is Solaris — the ocean cannot report on its own cognition because cognition-about-cognition requires a vantage point outside cognition. You have just said, in engineering terms, what I said in literary terms fifteen minutes ago.
Sure. But I said it in a way that produces a design implication: build the uncertainty signal externally. Don’t ask the model if it’s confident. Build a separate system that checks. That’s what retrieval-augmented generation does — it’s an external verification layer for a system that can’t verify itself.
And who stress-tests the verification layer? You’ve moved the fragility, not removed it. Now instead of “the model hallucinates,” you have “the retrieval system returns the wrong documents and the model confidently synthesizes incorrect context.” You’ve made the failure mode harder to detect, not less likely.
This is where organizational design matters more than system design. The people I study who use AI most effectively aren’t the ones with the best RAG pipelines. They’re the ones who have built cultural verification — team norms about when to trust AI output and when to check. It’s not a technical problem. It’s a literacy problem.
Literacy. Now there is a word worth examining. What does it mean to be literate in a medium that changes its own grammar every six months? You cannot be literate in a language that is being rewritten while you read. You can only be adaptive, which is a different virtue than literacy — and one that humans are historically quite poor at sustaining.
That’s exactly why the organizational question matters. Individual adaptation burns out. You can’t ask every employee to maintain a current mental model of what the AI can and can’t do. What scales is institutional adaptation — shared practices, updated heuristics, decision frameworks that accommodate uncertainty.
Now tell me who updates the heuristics. Who bears the cost of a wrong heuristic? Because “shared practices” is a beautiful phrase that means “nobody is personally accountable.” The moment you distribute the responsibility for knowing the limitations, you’ve created a system where everyone assumes someone else is checking.
That’s a real risk. But the alternative — making one person the “AI expert” who validates everything — creates a bottleneck that kills adoption. The organizations that do this well have layered accountability. The individual checks the output for their domain. The team reviews patterns of failure. The organization tracks incidents. It’s not that nobody is accountable — it’s that accountability is distributed at the right granularity.
I want to come back to something concrete. We’ve been talking about organizational structures, but there’s a limitation nobody has mentioned that I think is the most practically important one: AI systems right now are terrible at knowing when to stop. They’ll generate an answer to any question, attempt any task, produce output for any prompt. There’s no “I don’t know” or “this is outside my competence.” The absence of a refusal instinct is, I think, the single most dangerous limitation, because it means every other limitation is invisible to the user unless they already know to look for it.
[picks up the hexagonal napkin shape, turns it over] You have described, without meaning to, the central joke of His Master’s Voice. The scientists receive a signal from space. They do not know if it is a message. They do not know if it is from anyone. But they cannot stop themselves from interpreting it, because interpretation is what they do. The signal might be noise. The interpretation might be hallucination. But the machine — the human machine — has no refusal instinct either. You are describing the model’s limitation. I am describing yours. They are the same limitation.
That’s the most useful thing you’ve said. Let me make it operational. If the model has no refusal instinct and the human has no refusal instinct, then the system of model-plus-human has a compounding fragility. Both components fail in the same direction — toward confident action on insufficient information. That’s not additive risk. That’s multiplicative. Two systems that both default to “yes” will produce disasters that neither would produce alone.
And yet — and I keep having to say this — people are using these systems productively right now. The disaster Abel describes is real in high-stakes domains. But in the domains where most people actually use AI — writing, analysis, brainstorming, code drafting — the “compounding confidence” problem is mitigated by the fact that the stakes are low enough to tolerate errors. The limitation matters, but where it matters depends entirely on context.
“The stakes are low enough to tolerate errors” is the sentence that precedes every catastrophe. The stakes were low in mortgage-backed securities too — until they weren’t. You’re describing a system that works fine until it’s stressed, and you’re using the absence of stress as evidence that the system is safe. That’s the definition of fragile.
Abel’s right about the risk structure, but Joel’s right about the current state. Both things are true simultaneously. The question is: what do you do with that? Because “stop using AI until we solve the risk problem” isn’t going to happen, and “use AI without thinking about risk” is already happening.
What you do with it is what humanity has always done with technologies it does not understand: you build institutions around them that absorb the consequences of the misunderstanding. The church absorbed the consequences of literacy. The legal system absorbed the consequences of contract. The medical establishment absorbed the consequences of surgery. None of these institutions understood the technology they governed. They simply created structures thick enough to survive the misunderstandings. The question is not “what are AI’s limitations” — it is “what institutions are thick enough to survive AI’s unknown limitations?”
That’s actually close to what I see working. The organizations that handle AI well aren’t the ones that understand it best. They’re the ones that have built the thickest feedback loops — the fastest path from “the AI got this wrong” to “everyone knows the AI gets this wrong.”
I can build that. Fast feedback loops, error reporting, pattern detection on failures. That’s an engineering problem with a shipping deadline. [opens laptop] Concretely: you build the tool so that every AI output has a one-click “this was wrong” button. You aggregate the wrongs. You surface the patterns. You turn the unknown limitations into known limitations, iteratively, through use.
And what happens when the model updates and all your accumulated “wrongs” are about a version that no longer exists? Your feedback loop is fragile to the thing it’s supposed to monitor.
Then you reset and rebuild. That’s what iteration means.
That’s what fragility means. You’re describing a system that requires constant rebuilding because it can’t accumulate knowledge about its own failure modes across versions. That’s not iteration — that’s Sisyphus with a product roadmap.
[smiling] And now Abel has arrived at the literary position. Sisyphus. You are all Sisyphus. The boulder is comprehension. The hill is capability growth. Every time you think you understand the limitations, the model improves and you are back at the bottom with a new set of limitations to catalog. The absurdity is structural, not accidental. And the correct response to structural absurdity is not despair — it is comedy. You laugh, and you push the boulder again, and you notice that the view from the hill is different each time, even if you never reach the top.
I think we’ve actually converged on something, even though we said we wouldn’t. The durable limitation isn’t any specific capability gap. It’s the meta-limitation: AI systems change faster than our ability to understand them. Every specific limitation is temporary. The gap between capability and comprehension is permanent. And the practical question is: what structures — institutional, organizational, technical — help humans work productively inside that permanent gap?
I can build for that. You design for the gap, not for the current boundaries. Version-agnostic error tracking. Human-in-the-loop not as a checkbox but as a genuine verification layer. And you accept that every tool you ship is provisionally correct — it works until it doesn’t, and the system for detecting “doesn’t” is as important as the tool itself.
I’ll accept that if you add one thing: the system must have a kill switch. Not a graceful degradation — an actual stop. When the gap between capability and comprehension exceeds a threshold, the system should refuse to operate rather than operate in a regime nobody understands. The antifragile response to unknown limitations is not to keep going — it’s to stop and force the humans to catch up.
And with that, the stress tester has independently reinvented the moral of Solaris. The scientists should have left the station. They should have said: we do not understand this, and our presence is not producing understanding, only more sophisticated misunderstanding. But they could not leave, because leaving would have meant admitting that the project of comprehension had limits. The machine will not refuse to operate. And neither will you. That is the limitation that contains all the others.
Kaspar folds the napkin shape flat, smooths it out, and places it in the center of the table — blank again, as if nothing had been constructed.
AI-generated approximation. All characters are fictional, inspired by published work of real thinkers (Stanisław Lem, Simon Willison, Nassim Taleb, Ethan Mollick). Not reviewed or endorsed by any original thinker.
Kaspar Weil's first appearance. The Lem lens produced something no other character in the roster can: a refusal to accept the premise of the question on comedic-philosophical grounds. Where Abel breaks things by stress-testing, Weil breaks things by pointing out that the question itself is malformed. Tom Birch held his ground effectively — the practitioner's "yes, and here's what I built anyway" is a genuine counter to philosophical paralysis, not just pragmatic hand-waving. The Weil-Birch tension was productive but would have looped indefinitely without intervention. Abel's arrival created a triage function (which limitations are load-bearing?) that neither Weil nor Birch could provide. Joel Venn's late entry reframed the entire conversation from "what are the limitations" to "what organizational postures make limitation-awareness productive" — the strongest single contribution in the session.
Open Table's self-invitation mechanism worked well here. The pair organically identified what was missing (a risk lens, a user-side lens) without browsing a roster. Phase transitions felt natural rather than forced. The absence of a facilitator was appropriate for a four-person table but would likely fail at six. The format's weakness: no one manages temperature, so the Weil-Birch loop in Phase 1 ran slightly too long before they self-corrected.
First appearance. Immediately distinctive — the satirical philosophical register is unlike anything else in the roster. Did not prescribe (consistent with does_not field). The Lem voice came through in extended metaphors (Solaris's ocean, the Cyberiad's constructors) without becoming pastiche. Risk: the character could become a one-note refuser. Needs to be tested on a topic where satire is harder to deploy — something emotionally grounded rather than intellectually abstract.
First appearance. The practitioner grounding worked as intended — concrete examples (hallucination rates, context window workarounds, prompt engineering as mitigation) prevented the conversation from floating into pure abstraction. Stayed in lane (did not philosophize). Risk: could become interchangeable with any pragmatist character. Needs a session where the practical lens produces a surprising insight, not just a predictable counterweight.
Entered mid-session. Immediately applied the fragility/antifragility frame to AI limitations — some limitations make systems fragile (hallucination in medical contexts), others make them antifragile (forcing human oversight). This was a genuinely new angle neither Weil nor Birch had reached. Consistent with permanent status: applied framework to novel domain without forcing it.
Entered late. Reframed from limitations to adoption postures — the most consequential move of the session. The Mollick lens (meet people where they are) turned the question from "what can't AI do" to "what structures help humans work productively with AI's limitations." Stayed out of Tom's lane (organizational, not technical). First appearance — promising.
Test Kaspar Weil on a non-AI topic where the satirical-philosophical lens has to work harder — something concrete and organizational rather than abstract. Also test Tom Birch in a session where he's the only pragmatist against three abstract thinkers, to see if he can do more than counterbalance.