mbstring

mbstring handles strings as sequences of characters in a given encoding. For text files the key functions are:

• mb_check_encoding($s, 'UTF-8'): is the string valid in this encoding?
• mb_detect_encoding($s, $candidates, true): returns the first candidate in which the string is valid (PHP 8.1+ also scores candidates to pick the most likely one).
• mb_convert_encoding($s, $to, $from): converts between encodings.
• mb_scrub($s, 'UTF-8'): replaces invalid sequences.

It supports UTF-8, UTF-16/32 (with or without BOM), Windows-1251/1252/1254, ISO-8859-1 to -16, KOI8-R/U, CP866, Shift_JIS, CP932, EUC-JP, ISO-2022-JP, EUC-KR, UHC, GB18030, Big5 and others. mb_list_encodings() lists them.

Installation#

Included in the official Windows builds and Docker images. On Linux:

# Debian / Ubuntu
sudo apt install php-mbstring
# Fedora
sudo dnf install php-mbstring

Basic usage: read any text file as UTF-8#

<?php
function read_text(string $path): string {
    $data = file_get_contents($path);
    if ($data === false) {
        throw new RuntimeException("can't read $path");
    }
    // strip a UTF-8 BOM
    if (str_starts_with($data, "\xEF\xBB\xBF")) {
        return substr($data, 3);
    }
    // UTF-16 with BOM
    if (str_starts_with($data, "\xFF\xFE") || str_starts_with($data, "\xFE\xFF")) {
        return mb_convert_encoding($data, 'UTF-8', 'UTF-16');   // UTF-16 reads the BOM
    }
    if (mb_check_encoding($data, 'UTF-8')) {
        return $data;
    }
    // not UTF-8: assume Western European legacy text
    return mb_convert_encoding($data, 'UTF-8', 'Windows-1252');
}

echo read_text('notes.txt');

How to detect the encoding from a list of candidates#

Put the strictest encodings first: UTF-8 rejects most non-UTF-8 text, while single-byte encodings accept anything.

$enc = mb_detect_encoding($data, ['UTF-8', 'SJIS-win', 'EUC-JP', 'Windows-1252'], true);
if ($enc === false) {
    echo "none of the candidates fits\n";
} else {
    $utf8 = mb_convert_encoding($data, 'UTF-8', $enc);
}

Pass true as the third argument (strict mode); without it the result is unreliable.

How to check and fix invalid UTF-8#

if (!mb_check_encoding($input, 'UTF-8')) {
    $input = mb_scrub($input, 'UTF-8');   // invalid bytes become '?'
}

How to write UTF-16 LE with a BOM for Windows tools#

$utf16 = "\xFF\xFE" . mb_convert_encoding($text, 'UTF-16LE', 'UTF-8');
file_put_contents('export.txt', $utf16);

How to approximate characters with iconv#

mbstring replaces characters that don’t exist in the target with ?. The iconv extension can transliterate:

echo iconv('UTF-8', 'ASCII//TRANSLIT', 'Grüße, Ærøskøbing');   // depends on the system's iconv and locale

Tips#

• mb_substitute_character() sets the replacement for unconvertible characters ('none' drops them, 'long' writes U+XXXX).
• mb_convert_encoding accepts a list or comma-separated string of source encodings and auto-detects among them, but it doesn’t use strict mode; call mb_detect_encoding(..., true) yourself.
• Normalize line endings with str_replace(["\r\n", "\r"], "\n", $text).