All formats

chardetng

LanguageRust
LicenseApache-2.0 OR MIT
Repositorygithub.com/hsivonen/chardetng
Documentationdocs.rs/chardetng
Depends onencoding_rs, memchr, cfg-if
Read / writedetects the encoding; returns an encoding_rs::Encoding for decoding

chardetng (“character encoding detector, next generation”) is the detector Firefox uses for HTML and text without an encoding declaration. It’s built for the legacy encodings of the web: Windows-125x and ISO-8859 code pages, KOI8-U, IBM866, Shift_JIS, EUC-JP, ISO-2022-JP, EUC-KR, GBK, Big5 and UTF-8. It’s small and fast, and an optional top-level domain hint (b"jp", b"ru") biases the guess like a browser would.

It always returns an encoding (never “unknown”). It doesn’t detect UTF-16: check for a BOM with encoding_rs::Encoding::for_bom first.

Installation#

Shell
cargo add chardetng encoding_rs

Basic usage: detect and decode a file#

Rust
use chardetng::{EncodingDetector, Iso2022JpDetection, Utf8Detection};
use std::fs;

fn main() -> std::io::Result<()> {
    let bytes = fs::read("unknown.txt")?;

    let mut detector = EncodingDetector::new(Iso2022JpDetection::Deny);
    detector.feed(&bytes, true);   // true = this is the last chunk
    let encoding = detector.guess(None, Utf8Detection::Allow);

    let (text, _, had_errors) = encoding.decode(&bytes);
    println!("{} (errors: {})", encoding.name(), had_errors);
    println!("{}", text.chars().take(200).collect::<String>());
    Ok(())
}

How to feed a large file in chunks#

Rust
use std::io::Read;

let mut detector = EncodingDetector::new(Iso2022JpDetection::Deny);
let mut file = std::fs::File::open("big.txt")?;
let mut buf = vec![0u8; 64 * 1024];
loop {
    let n = file.read(&mut buf)?;
    let last = n == 0;
    detector.feed(&buf[..n], last);
    if last { break; }
}
let encoding = detector.guess(None, Utf8Detection::Allow);

feed returns true once it has seen non-ASCII bytes. To detect from just a sample of a long file, feed the sample with last = false and call guess without feeding the rest.

How to give a domain hint#

Rust
// text from a .ru site: Cyrillic encodings get priority over others
let encoding = detector.guess(Some(b"ru"), Utf8Detection::Allow);

How to check for a BOM first#

Rust
let encoding = match encoding_rs::Encoding::for_bom(&bytes) {
    Some((enc, _)) => enc,
    None => {
        let mut d = EncodingDetector::new(Iso2022JpDetection::Deny);
        d.feed(&bytes, true);
        d.guess(None, Utf8Detection::Allow)
    }
};

Tips#