Explainable AI

Explainable AI (XAI) encompasses the methods, frameworks, and design principles that allow human users — including domain experts, regulators, developers, and affected individuals — to understand why an AI system produced a particular output. As machine learning models have grown in predictive power, they have also become more opaque: deep neural networks with billions of parameters, gradient-boosted ensembles, and large language models offer little inherent insight into their reasoning. XAI addresses this opacity by surfacing post-hoc explanations, designing inherently interpretable models, or providing visualisations and statistics that reveal how inputs relate to outputs.^[1] The need for explainability is driven by multiple factors: regulatory requirements that mandate transparency in automated decision-making, operational requirements that allow engineers to identify and correct model errors, and ethical requirements that ensure individuals can contest automated decisions that affect them.

Why Explainability Matters

In high-stakes domains — medical diagnosis, credit lending, criminal justice risk scoring, and hiring — opaque AI models create legal and ethical risks. A loan applicant denied credit by an algorithmic system has a reasonable expectation of understanding the basis for that decision. A clinician relying on an AI diagnosis tool must be able to assess whether the model's reasoning aligns with medical knowledge before acting on its recommendation. A regulator auditing a bank's credit model must be able to verify that the model does not rely on proxies for protected characteristics such as race or gender.^[2] Beyond ethics and compliance, explainability supports debugging and model improvement: an explanation that reveals a model is relying on a spurious feature (such as the hospital name embedded in a medical image) enables practitioners to retrain or re-engineer the model.

Methods

XAI methods are typically categorised along two dimensions: scope (local vs. global) and model dependency (model-agnostic vs. model-specific).

SHAP (SHapley Additive exPlanations)

SHAP is grounded in cooperative game theory. It assigns each input feature a Shapley value representing that feature's average marginal contribution to the model's prediction across all possible subsets of features. Shapley values provide both local explanations (why did the model predict X for this specific instance?) and global explanations (which features matter most across the entire dataset?). SHAP values satisfy desirable mathematical properties — efficiency, symmetry, dummy, and additivity — that make them theoretically principled.^[3] Implementations such as TreeSHAP exploit the structure of tree-based models to compute exact Shapley values efficiently, while KernelSHAP provides a model-agnostic approximation.

LIME (Local Interpretable Model-Agnostic Explanations)

LIME generates explanations by perturbing the input around a specific instance, observing how the model's output changes, and fitting a simple, interpretable surrogate model (such as a linear regression) to this local neighbourhood. The surrogate's coefficients are presented as the explanation for that instance. LIME is model-agnostic — it treats the underlying model as a black box — and supports tabular, text, and image data. However, LIME explanations are inherently local and may not generalise across the input space, and results can be sensitive to the perturbation sampling strategy.^[4]

Attention Visualisation

In transformer-based models, attention weights indicate how much each input token influenced each output token. Visualising attention patterns can provide intuitive explanations of which words or image patches a model focused on, though research has shown that attention weights do not always correspond to feature importance in the causal sense.

Saliency Maps and Gradient Methods

For image classification CNNs, gradient-based methods such as Grad-CAM (Gradient-weighted Class Activation Mapping) produce heatmaps that highlight image regions most influential for a given prediction. These maps are widely used in medical imaging to verify that a model is attending to clinically relevant anatomy rather than image artefacts.

Intrinsically Interpretable Models

An alternative to post-hoc explanation is to use models that are inherently interpretable: decision trees, linear regression, rule-based systems, and generalised additive models (GAMs). Techniques such as Neural Additive Models (NAMs) attempt to bring the predictive power of neural networks closer to the interpretability of additive models.

Regulatory Context

Regulatory pressure has made XAI a compliance requirement rather than merely a best practice in many jurisdictions. The EU AI Act (2024) classifies high-risk AI systems — in medical devices, credit scoring, recruitment, and law enforcement — as requiring transparency, human oversight, and documentation of how the system reaches decisions, with penalties reaching 6% of global annual revenue for non-compliance. The EU GDPR Article 22 grants individuals the right not to be subject to solely automated decisions with significant effects, and the right to receive meaningful information about the logic involved. Similar obligations appear in financial regulations: the Basel Committee on Banking Supervision has issued guidance on responsible AI use in banking that includes model explainability requirements.^[2] Malaysia's regulatory landscape increasingly requires explainability for AI systems deployed in high-stakes sectors. Bank Negara Malaysia (BNM) issued its Principles-based Taxonomy for Financial Sector Technology and the Responsible AI Policy, both of which identify transparency and explainability as fundamental principles for AI systems used in credit assessment, fraud detection, insurance underwriting, and investment advice. Financial institutions regulated by BNM — including Maybank, CIMB, Public Bank, AmBank, and Hong Leong Bank — are expected to document how their AI models make decisions and to provide borrowers with explanations when automated credit decisions are unfavourable. Malaysia's National AI Office, under the Ministry of Science, Technology and Innovation (MOSTI), has published the Malaysia AI Governance Framework, which places transparency and accountability at the forefront of responsible AI deployment. The framework requires organisations deploying AI in public sector services to conduct AI impact assessments and ensure that automated decisions affecting citizens can be explained and contested. This applies to government agencies using AI for social benefit targeting, public health interventions, and immigration processing. The Securities Commission Malaysia (SC) has issued guidelines on digital investment management (robo-advisors) and algorithmic trading that require firms to document and explain the basis for automated investment recommendations. As AI-driven credit scoring, insurance pricing, and algorithmic trading grow in Malaysia, XAI methods such as SHAP and LIME are being adopted by local fintech firms and incumbents alike to satisfy these obligations. Universiti Malaya's Faculty of Computer Science and Information Technology and Universiti Teknologi Malaysia's School of Computing have active XAI research programmes, with particular focus on healthcare decision support. Malaysia's high burden of non-communicable diseases — including diabetes, cardiovascular disease, and cancer — has motivated research into interpretable diagnostic AI that can earn clinician trust in clinical settings where accountability is paramount. The Human Resources Development Corporation (HRD Corp) recognises XAI as an emerging skill area, and providers including Asia Pacific University, Multimedia University, and Sunway University offer postgraduate modules on AI fairness, accountability, and transparency that address XAI methodology. As Malaysia positions itself as a regional AI hub, demand for professionals who can audit, explain, and govern AI systems is expected to grow alongside demand for AI model builders.

See Also

References

1. Gunning, D., Stefik, M., Choi, J., Miller, T., Stumpf, S., & Yang, G. Z. (2019). XAI — Explainable artificial intelligence. Science Robotics, 4(37).
2. Basel Committee on Banking Supervision. (2022). Principles for the effective management and supervision of climate-related financial risks. Bank for International Settlements. [Model risk governance document that includes AI explainability provisions.]
3. Lundberg, S. M., & Lee, S. I. (2017). A Unified Approach to Interpreting Model Predictions. Advances in Neural Information Processing Systems, 30.
4. Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). "Why Should I Trust You?": Explaining the Predictions of Any Classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.