Arabic content is central to web data in the GCC. Product names, store locations, property descriptions and reviews are frequently published in Arabic, in English or in both. Pipelines built for Latin-script data tend to fail on Arabic in predictable ways. This post covers the problems we see most often and how we handle them.
Encoding: get it right at the boundary
The most visible failure is mojibake, where Arabic arrives as strings of accented Latin characters or question marks. It almost always happens at a boundary: a page served in a legacy Arabic code page and read as UTF-8, or a correct UTF-8 file opened by a tool that assumes another encoding.
- Detect the source encoding from response headers and page metadata, and verify it on a sample.
- Convert to UTF-8 once, at ingestion, and keep UTF-8 throughout.
- When delivering CSV for spreadsheet users, include a byte order mark, or many tools will misread the file.
- Run an automated encoding check on every delivery: count replacement characters and unexpected Latin-1 sequences.
Unicode normalisation
Two strings that look identical can be different sequences of code points. Arabic has presentation forms, which are legacy code points for letter shapes, and some sources still emit them. Normalising to a standard Unicode form, typically NFC for storage and NFKC for matching, removes many of these differences. Normalisation should be applied deliberately and recorded, because NFKC can change characters in ways that matter for display.
Right-to-left and mixed direction
Arabic is written right to left, but product titles often embed Latin brand names, model numbers and Western digits. Display software uses the Unicode bidirectional algorithm to order these runs, and results can surprise. A title such as a brand name followed by an Arabic description and a capacity can appear scrambled in a spreadsheet cell, even though the underlying data is correct.
The key point for data teams is that display order and storage order are different. Never "fix" a string by reversing it to make it look right in one tool. Store the logical order, and address display issues in the presentation layer.
Store Arabic text in its logical order and fix display in the presentation layer. Reversing strings to make them look right in one tool breaks them everywhere else.
Digits
Arabic-language pages may use Eastern Arabic digits (٠١٢٣٤٥٦٧٨٩), Persian variants in some sources, or Western digits. Prices and pack sizes must be converted to Western digits before parsing. The Arabic decimal separator and thousands separator also differ from Latin forms, so number parsing needs explicit handling rather than a generic conversion.
Matching Arabic text
Arabic spelling varies in ways that are harmless to readers and damaging to exact-match logic. For matching purposes, a light normalisation removes most of this variation without changing meaning.
| Variation | Example | Matching treatment |
|---|---|---|
| Alef forms | أ, إ, آ and ا | Map to bare alef |
| Taa marbuta | ة and ه at word end | Map to a single form |
| Final yaa | ى and ي | Map to a single form |
| Diacritics | Short vowel marks, often partial | Remove |
| Tatweel | Decorative letter elongation | Remove |
| Definite article | ال prefix present or absent | Compare with and without |
Apply these rules to a matching key, not to the stored text. The original string is what the source published and what any client-facing output should show.
Transliteration
Names written in Latin script have no single standard form. The same place, brand or company name may be spelt with different vowels, with or without a hyphenated article, or with different letters for sounds that English lacks. Mohammed and Muhammad are the familiar example, but product brands and district names vary in the same way.
- Maintain alias tables for brands, districts and cities, reviewed by a native speaker.
- Use phonetic or character n-gram similarity for candidate generation, then confirm with attributes.
- Where both Arabic and English names are available, match on both and require agreement.
- Do not auto-transliterate for display unless a client specifically asks for it.
Search and tokenisation
Arabic words carry attached prefixes and suffixes, such as conjunctions and pronouns, so whitespace tokenisation produces many variants of the same root word. For search and classification, light stemming helps. For product matching, it is usually better to extract structured attributes, such as brand, size and variant, and match on those rather than on free text.
Quality checks we run
- Encoding check on every field expected to contain Arabic.
- Script detection to confirm Arabic fields contain Arabic and English fields contain Latin script.
- Digit conversion check on numeric fields parsed from Arabic pages.
- Spot review by an Arabic-reading analyst on a sample of each delivery.
- Comparison of Arabic and English attribute values where both exist.
None of this requires exotic tooling. It requires treating Arabic as a first-class part of the pipeline from the start, rather than a special case handled after something breaks.