Malay Language NLP

Malay language NLP is the field of natural language processing concerned with enabling machines to read, generate and speak Bahasa Melayu and the informal Malaysian variants people actually type and speak — including Manglish (Malaysian English), social-media slang and regional dialects — despite a digitised corpus that is a small fraction of English's.^[1]

Background

Malay is an Austronesian language spoken natively across Malaysia, Indonesia, Brunei, Singapore and southern Thailand; together with Indonesian it serves well over 200 million speakers, yet the corpus of clean, digitised Malay text available for training has historically been far smaller than for English. Early NLP pipelines were built English-first: tokenisers trained on English split Malay words at awkward subword boundaries, classical classifiers needed heavy pre-processing, and off-the-shelf sentiment and summarisation tools degraded sharply on Malay input. Open-source Malay libraries such as Malaya filled the gap with task-specific tools — stemming, sentiment, summarisation, named-entity recognition — before the large-language-model era arrived.

Key Concepts

Several properties make Malay difficult relative to its apparent simplicity:

• Agglutinative morphology. Productive prefixes and suffixes (meN-, peN-, di-...-an, ke-...-an) generate many surface forms from one root, inflating vocabulary sizes and pushing word-level models toward unseen tokens.
• Code-switching. Urban Malaysian speech routinely mixes Malay and English within a single sentence; chatbots and sentiment tools must handle sentences that switch language mid-clause.
• Dialect and register. Kelantanese, Terengganu and northern dialects differ markedly from standard Bahasa Malaysia, while formal written Malay differs from marketplace (bahasa pasar) usage and misspelt social-media text.
• Evaluation. The benchmark MalayMMLU, introduced at EMNLP 2024, comprises 24,213 questions spanning primary and secondary school curricula across 22 subjects, and exposed large gaps between general-purpose LLMs and models pretrained on Malay data.^[1][3]

Applications or Impact

Modern responses are Malay-pretrained models: MaLLaM, from Malaysian startup Mesolitica, was trained from scratch on around 90 billion tokens of Malaysian text^[2]; ILMU, from YTL AI Labs, is a multimodal model covering Bahasa Melayu, Manglish and regional dialects across text, voice and vision, topping MalayMMLU zero-shot results when published; and Singapore's SEA-LION extends coverage across Southeast Asian languages.^[4] These underpin citizen-service chatbots, banking and telco customer support, voice assistants, translation and classroom tools.

>See Also

• ILMU (Malaysian LLM)
• MaLLaM (Malaysia Large Language Model)
• Mesolitica
• SEA-LION
• Natural Language Processing
• MDEC Malaysia