Articles
AI Explainability: History, Methods, and Implementation in Architectural Drawing Review
Automatically translated from the Japanese original.
Introduction: Does AI Really "Understand"?
The explosive progress of deep learning in modern AI was not driven by any single factor. It was the simultaneous maturation of several elements—models, data, and computing power—that produced a major turning point.
In the 1990s, the convolutional neural network proposed by Yann LeCun and colleagues—known as LeNet—delivered strong performance in handwritten digit recognition and document recognition. Deep learning research was then carried further by Geoffrey Hinton and other researchers. Advances in GPU-based parallel computing made large-scale matrix operations practical to run. On top of that, ImageNet, the large-scale image dataset championed by Fei-Fei Li's team, allowed AI systems to learn from vast numbers of real images and to have their performance benchmarked and compared in competition. Then, in 2012, AlexNet—developed by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton—achieved a landmark result on the ImageNet image recognition task, and modern deep learning began to attract serious attention.
But this raises an important question. Feed an image into a model and it returns a recognition result with high accuracy. At first glance, the AI certainly appears to be "getting it right." Yet do we truly understand why the AI arrived at that answer?
When an AI recognizes a cat, does it really understand the cat's eyes, ears, outline, and body structure? Or has it merely memorized statistical patterns found in the training data? When it assesses risk in a medical image, is it actually looking at the lesion? Or could it be reacting to information it should never have been relying on—a watermark, the quirks of a particular imaging device, a hospital's own markings? It is much like a person who has watched an apple fall from a tree ten thousand times: they may be able to predict correctly that "apples fall downward," but that does not necessarily mean they understand gravity.
In high-stakes domains such as autonomous driving, medical diagnosis, architectural drawing review, and financial risk management, this problem is especially critical. In these fields, an AI error is not like getting one practice question wrong—it bears directly on safety, liability, cost, and trust. That is why explainability is not merely a "nice-to-have technology." It is an essential element in AI's evolution from something that "produces results" into something that humans can understand, verify, trust, and manage.
Executive Summary
Put simply, AI explainability is not about making AI "talk better." It is about enabling humans to understand why an AI made a given decision—what it looked at, what it overlooked, where the uncertainty lies, and how a problem can be traced back when something goes wrong.
The reason this topic has become so important in recent years is not simply that "people dislike black boxes." Regulation, customer trust, quality audits, safety accountability, and cross-functional collaboration are all pushing AI hard to move from a stage of "just giving answers" to one of "giving answers together with their grounds." The EU AI Act requires technical documentation, automatic logging, transparency, and human oversight for high-risk AI systems. China's Provisions on the Management of Algorithmic Recommendation Services and its Interim Measures for the Management of Generative AI Services likewise call for disclosure and explanation of the basic principles, purposes, and main operating mechanisms of algorithms. NIST, too, cites AI's "invisibility and elusiveness" as one of the major sources of AI risk.
Looking back at the research history, early AI centered on rules, decision trees, and expert systems, and the models themselves were relatively easy to understand. The rise of deep learning brought rapid gains in accuracy, but model readability dropped sharply. As a result, the late 2010s saw the emergence of methods for "attaching a manual to the black box"—heatmaps, LIME, SHAP, Grad-CAM, counterfactual explanations, and more. With the arrival of large language models, research has branched in two directions: explaining from the outside (presenting rationales, retrieved evidence, structured outputs) and explaining from the inside (neuron interpretation, sparse features, circuit tracing, mechanistic interpretability).
For an AI agent that reviews architectural drawings, explainability is not just an academic add-on—it is an eminently practical tool for boosting productivity. Building review is a domain with many rules, clearly defined responsibilities, and high sensitivity to both false positives and missed issues. Rather than a bare risk score, the AI's output is far better suited to real-world operation when delivered as a multi-layered evidence package: conclusion + location on the drawings + supporting code provisions + audit log + a human review entry point.
Why the Whole Company Needs to Understand AI Explainability
Start with the most practical point: explanation is not decoration; it is risk management. DARPA's XAI program set as its main goal "enabling humans to understand, appropriately trust, and effectively manage" AI. The U.S. National Institute of Standards and Technology (NIST) likewise points out that AI's "elusiveness" makes risk measurement and risk management harder. In other words, explanation is not "for show"—it is what an organization needs in order to fold AI into a process that is genuinely manageable and accountable.
When it comes to trust, what truly matters is not "getting people to believe AI more" but "enabling people to trust AI appropriately." Without explanation, people tend to drift toward one of two extremes: over-relying on AI because it looks smart, or refusing to use it at all because they don't understand how it works. The XAI literature and research on human-AI collaboration often emphasize "appropriate reliance"—trusting when trust is warranted and doubting when doubt is warranted. That said, adding explanations does not automatically lead to better decisions. This is precisely why companies need to design explanation as a mechanism for calibrating trust, not as rhetoric for persuading people to believe the AI.
From a safety standpoint, explanation helps uncover what "odd shortcuts" a model has learned. During training, many models pick up signals that happen to correlate with the target but have nothing to do with the actual task—watermarks, title blocks, scanning noise, page borders, and so on. This matters especially for architectural drawings, which are full of elements that can confuse a model: layout differences, scan quality, stamps, revision clouds, scale notes, multi-page merges, and more.
Explainability is also underrated as a productivity tool. If the results of AI drawing review can simultaneously show a heatmap, the relevant code provisions, the dimensional evidence, and "why it judged this rule to apply," the system becomes more than a classifier—it becomes on-the-job training material for newcomers. For audits and cross-functional collaboration, if a single conclusion can be unfolded into "the location in question, extracted dimensions, applicable provisions, similar past cases, model version, and a suggested human check," the discussion shifts from abstract unease to concrete evidence.
History and Evolution
The history of AI explainability is easiest to grasp as one in which explanatory tools were retrofitted, after the fact, as models grew more powerful.
| Period | What happened | What it meant for explainability |
|---|---|---|
| The early era of rules and tree models | Expert systems, rule-based systems, and decision trees dominated. With their if-then structure, the models themselves could be read. | Explanation mostly meant "reading the model itself." |
| Deep learning turns models into black boxes | Deep networks made major strides in vision tasks, and research sought to understand CNNs through gradient saliency maps, feature visualization, and the like. | Explanation shifted from "reading the rules" to "giving the black box a health check through visualization." |
| The systematization of XAI | DARPA XAI set out to build explainable models without sacrificing high performance; LIME, SHAP, Integrated Gradients, Grad-CAM, and others went mainstream. | Explainability became a research field in its own right. |
| The era of faithfulness and evaluation | It was recognized that "looking like an explanation" and "actually explaining" are different things, and evaluations such as sanity checks, infidelity, and sensitivity were proposed. | Explanations themselves became something to be scrutinized. |
| The era of LLMs and mechanistic interpretability | With the development of Transformers and LLMs, research advanced in tracing attention, neurons, sparse features, and internal circuits (Anthropic, OpenAI). | Explanation expanded to "what is happening inside the model." |
In a nutshell: early AI was a transparent machine; deep learning was a high-performance black box; the next stage bolted a dashboard and a dashcam onto that black box; and today's frontier is attempting to open up the engine itself. Understanding this arc also reveals the most realistic approach for enterprise implementation. Rather than trying to see all the way through a large model from the outset, putting the dashboard, the evidence chain, and the human review interface in place first is far more likely to translate directly into business value.
An Overview of Methods
Strictly speaking, interpretability refers to a model being easy to understand in itself, whereas explainability refers to being able to explain a model to humans through additional means, even when the model itself is complex. In practice, rather than fussing over terminology, what matters is being clear about whether you want the model itself to be transparent or its results to be verifiable.
Ante-hoc Interpretability
Ante-hoc interpretability—also called "intrinsic interpretability" or "white-box methods"—is the idea of choosing a structure that humans can readily understand right from the design stage. Decision trees, rule lists, and linear models are the classic examples. Later, methods such as generalized additive models, GA2M, and the Explainable Boosting Machine (EBM) were proposed to raise performance while retaining readability. Decision trees are particularly easy to explain: a rule such as "if the effective width of a stair is below a certain value and that area is part of an evacuation route, flag a risk" is something building practitioners grasp intuitively.
Another important direction is constrained learning. This includes imposing constraints such as sparsity, monotonicity, or stability during training, or—as in self-explaining neural networks—explicitly building the "concept → contribution → conclusion" relationship into the model architecture. Here, explanation is not something bolted on afterward; it is part of the learning itself.
Post-hoc Explainability
Post-hoc explainability takes an already-trained model as it is and builds explanatory scaffolding around its outputs. Local explanations answer "why did this drawing get this result?"; global explanations answer "what cues does this model generally rely on across the board?"
| Method | Analogy | Strengths | Limitations | Direct value in architectural drawing review |
|---|---|---|---|---|
| Decision trees / rule lists | A transparent flowchart | Easy to audit; well suited to training and compliance discussions | Weak at expressing complex patterns | Explicit-rule review; high-frequency checks of simple provisions |
| EBM/GA2M | Readable itemized scoring | Combines performance and understandability on some tasks | Not suited to every complex task | Tabular and structured review features |
| LIME | A little tutor next door | Model-agnostic; convenient for explaining a single result | Can be unstable due to sampling | Explaining false positives on a specific drawing |
| SHAP | A fair split of the credit | Offers both local and global views | Computation and interpretation require experience | Feature-contribution analysis; version comparison |
| Counterfactuals | A revision proposal | Easy to translate into action | Must consider whether the change is actually feasible | Suggestions on "how to fix it so it passes" |
| Deep Unrolling | Breaking a craftsman's iterative routine into fixed steps | Powerful and fast; easier to explain than a pure black box | Not suited to every task | Image restoration, drawing vectorization, measurement submodules |
LIME works by generating many similar samples in the neighborhood of a single instance and training a small surrogate model that explains only that local region; think of it as a "small tutor who can only tell you about the neighborhood around that one sample." SHAP distributes credit for the outcome among the individual features using Shapley values, a concept borrowed from game theory. Bear in mind, however, that these contribution values are not causal relationships in themselves, so they must be interpreted with care. Counterfactual explanations show "what would have happened if things had been slightly different." In drawing review, this can take a form close to an improvement proposal: "If this door were 900 mm wide instead of 850 mm, and the missing evacuation-direction arrow were added, this item would most likely pass."
Explainability in image models
Heatmaps are the easiest explainability concept to grasp. At heart, a heatmap is a "map of influence": picture a translucent thermal film laid over the original image, showing where the model was paying attention. Saliency maps, Grad-CAM, Integrated Gradients, LRP and similar techniques differ in how they are computed, but they are frequently visualized in this form.
Yet precisely because heatmaps are so easy to explain, they are also remarkably easy to misread. A heatmap is not a legal basis, not a proof of causation, and not a recording of the model's reasoning process. It is merely a clue as to which regions may have served as major cues for this particular decision. If the model says "there is an evacuation-related problem here," the heatmap should be concentrated on stairwells, doors, dimension annotations, axis relationships and the like. If, on the other hand, the title block, the drawing border or a watermark lights up strongly, the model may have learned a shortcut it was never supposed to learn.
Adversarial examples offer another critical perspective on explaining image models. Adding a perturbation to the input so small that a human can barely see it can be enough to make the model misclassify with high confidence. This suggests that the model may be learning "non-robust features": patterns that are useful for prediction but do not look natural to humans. In drawing AI, scanning, scaling up or down, compression, local occlusion, stamps and revision marks can all become input changes that are minor to the eye but fatal to the model, which is why explainability and robustness need to be considered together.
Explainability in large language models (LLMs)
The attention mechanism at the heart of the Transformer is one of the most natural entry points for thinking about LLM explainability. Tools such as BertViz visualize the attention relationships between different layers, different heads and different tokens. That said, attention is not the same as an explanation. An attention distribution does not always faithfully reveal the evidence the model is actually relying on.
One of the most familiar forms of "explanation" in the LLM era is Chain-of-Thought: having the model write out its steps. But the steps a model writes are not necessarily the chain of reasons that actually operated inside the model. Enterprise implementations should therefore not rely solely on "making the LLM write more detailed reasons" as their approach to explainability. A more solid approach is to tie explainability to external evidence: RAG, structured outputs, tool calls, source citations, page numbers, article numbers and so on. In drawing review, the best explanation a large language model can give is not a monologue on "why I thought so," but a clear account of "which standard was cited, where on the drawing was examined, which values were extracted, and how the result should be verified."
In recent years, the frontier of LLM explainability has moved toward mechanistic interpretability. Anthropic published research using sparse autoencoders to extract more monosemantic features from the activations of Claude 3 Sonnet. OpenAI, for its part, conducted research using GPT-4 to automatically write explanations for the neurons of GPT-2. LLM explainability is progressing from the stage of "making outputs readable" to the stage of "investigating mechanisms, controlling behavior, and pinpointing problems."
Evaluation, trade-offs and limitations
The truly hard question in explainability is what counts as a good explanation. In practice, it helps to break the evaluation of explanations down along the following dimensions.
| Evaluation dimension | Typical question | Representative methods |
|---|---|---|
| Faithfulness | Does the explanation reflect the evidence the model actually relied on? | Infidelity、Sensitivity、Deletion/Insertion、Sanity Checks |
| Usefulness | Does it help people understand, verify and correct faster and more accurately? | User studies, decision accuracy, verification time |
| Robustness | Does the explanation stay stable when the input changes slightly? | Perturbation tests, randomization tests, version comparisons |
| Actionability | Can the explanation be converted into concrete correction proposals? | Counterfactual explanations, rule alignment, evidence cards |
| Compliance | Can the explanation be used for record-keeping, retrospective review and accountability? | Log completeness, source tracing, human verification workflows |
It is often said that "accuracy and explainability are always a trade-off," but strictly speaking this is not necessarily true. Research on EBMs and GA2Ms has shown that interpretable models can achieve very high performance on certain tasks. What matters in practice is which explanation helps which user on which task. A mature explainability system for real-world operations needs to uphold at least three principles: be faithful (do not mislead), be useful (support decisions, verification and improvement), and be traceable (allow responsibility and evidence to be followed back).
Implementation proposals for architectural drawing review AI and the road ahead
My strongest recommendation is not to start by building "flashy explanations," but to build an evidence chain for drawing review. In other words, every AI output should carry at least the following five layers.
Conclusion
Location on the drawing
Basis in standards and code provisions
Uncertainty
Logs and a path for human verification
Suppose, for example, that the AI flags a floor plan with "the clear width of this evacuation door may be insufficient." A useful explanation cannot stop at displaying that one sentence. It should simultaneously show the door's location, the dimension annotations the model focused on, the clear width extracted via OCR or geometric analysis, the applicable code provision along with its version and page, and a counterfactual correction proposal such as "adjusting this width to at least a certain value would likely allow it to pass." Output of this kind serves as a verification card for the reviewer, an explanation card for the customer, and training data for the product and algorithm teams.
Conversely, when the model mistakes a past revision note in the title block for a normative statement on the current page, or misreads a stamp, watermark or revision cloud as a structural member, the system should detect an "explanation anomaly" rather than silently issuing a conclusion. Monitoring for such mismatches between explanation and conclusion (a heatmap concentrated on non-design areas rather than design areas, visual evidence that disagrees with the code evidence, or a confidence level that is too high despite a missing critical dimension) can potentially surface problems earlier than simple accuracy monitoring.
Over the next several years, AI explainability will likely advance along four directions at once: first, verifiable AI (formal guarantees for critical properties); second, standardization of explainability (model cards, explanation cards, logging specifications); third, automation of explainability (automatic generation of code-provision evidence cards, automatic detection of unnaturally high confidence); and fourth, the interplay between explainability and privacy and security (layered explanations).
The clearest recommendation is to define explainability as a product capability rather than an appendage to the algorithm. In architectural drawing review, the ultimate winner will not be the model that is best at scoring points, but the system that best organizes conclusions, evidence, standards and human verification. In this sense, explainability is not an added cost; it is the product's barrier to entry.
The end
Read next ↓