All formats

encoding_rs

LanguageRust
License(Apache-2.0 OR MIT) AND BSD-3-Clause
Repositorygithub.com/hsivonen/encoding_rs
Documentationdocs.rs/encoding_rs
Depends oncfg-if
Read / writedecodes and encodes all WHATWG encodings; BOM sniffing; no statistical detection

encoding_rs implements the WHATWG Encoding Standard: exactly the encodings, labels and error handling that browsers use. Each encoding is a &'static Encoding constant (UTF_8, WINDOWS_1252, SHIFT_JIS, UTF_16LE, …). The simple API decodes a whole buffer into a Cow<str>; the streaming Decoder/Encoder API converts chunk by chunk without allocation. It’s fast (SIMD where available) and used in Firefox and many Rust crates.

Following the WHATWG rules, it can decode UTF-16 but only encode to UTF-8 and the legacy encodings (encoding to UTF-16 produces UTF-8). For UTF-16 output use str::encode_utf16. To guess an unknown legacy encoding, use chardetng.

Installation#

Shell
cargo add encoding_rs

Basic usage: decode a file to UTF-8#

Rust
use encoding_rs::WINDOWS_1252;
use std::fs;

fn main() -> std::io::Result<()> {
    let bytes = fs::read("old.txt")?;
    // decode() sniffs a BOM first (UTF-8, UTF-16LE/BE) and uses it if present,
    // otherwise decodes as Windows-1252. Invalid bytes become U+FFFD.
    let (text, encoding_used, had_errors) = WINDOWS_1252.decode(&bytes);
    println!("{} chars, decoded as {}, errors: {}", text.chars().count(), encoding_used.name(), had_errors);
    fs::write("old-utf8.txt", text.as_bytes())?;
    Ok(())
}

How to look up an encoding by name#

Rust
use encoding_rs::Encoding;

let enc = Encoding::for_label(b"shift_jis").expect("unknown encoding label");
let (text, _, _) = enc.decode(&bytes);

Labels are the WHATWG ones: latin1 and iso-8859-1 both map to windows-1252, as in browsers.

How to detect a BOM#

Rust
use encoding_rs::Encoding;

match Encoding::for_bom(&bytes) {
    Some((enc, bom_len)) => println!("{} BOM, {} bytes", enc.name(), bom_len),
    None => println!("no BOM"),
}

How to decode without BOM sniffing or with strict errors#

Rust
use encoding_rs::UTF_8;

// ignores a BOM-like prefix and decodes exactly as UTF-8
let (text, had_errors) = UTF_8.decode_without_bom_handling(&bytes);

// returns None on any malformed sequence instead of inserting U+FFFD
let strict: Option<std::borrow::Cow<str>> = UTF_8.decode_without_bom_handling_and_without_replacement(&bytes);

How to encode UTF-8 text to a legacy encoding#

Rust
use encoding_rs::SHIFT_JIS;

let (bytes, encoding_used, had_unmappable) = SHIFT_JIS.encode("日本語のテキスト");
// unmappable characters are written as HTML numeric character references (&#...;)
std::fs::write("sjis.txt", &bytes)?;

How to decode a large file in chunks#

Rust
use encoding_rs::WINDOWS_1251;
use std::io::Read;

let mut decoder = WINDOWS_1251.new_decoder();   // with BOM sniffing
let mut file = std::fs::File::open("big.txt")?;
let mut buf = [0u8; 8192];
let mut out = String::with_capacity(8192 * 3);
loop {
    let n = file.read(&mut buf)?;
    let last = n == 0;
    let (_result, _read, _had_errors) = decoder.decode_to_string(&buf[..n], &mut out, last);
    // out now holds the decoded text so far; process and clear it here
    if last { break; }
}

decode_to_string stops when out has no spare capacity; check the returned CoderResult and read count in production code, or reserve with decoder.max_utf8_buffer_length(n).

Tips#