All formats

juniversalchardet

LanguageJava (version 2.x needs Java 7+; the examples use Java 11 APIs)
LicenseMPL-1.1, or GPL-2.0-or-later, or LGPL-2.1-or-later (your choice)
Repositorygithub.com/albfernandez/juniversalchardet
DocumentationREADME
Depends onnothing
Read / writedetects the encoding; helpers open a Reader with it

juniversalchardet is a Java port of “universalchardet”, the encoding detector of Mozilla. This maintained fork (Maven group com.github.albfernandez) fixed many bugs of the original Google Code version and added UTF-16/32, Windows-1252 and helpers like UniversalDetector.detectCharset(File) and ReaderFactory.

It detects Chinese (GB18030, Big5, EUC-TW, HZ, ISO-2022-CN), Cyrillic (Windows-1251, KOI8-R, ISO-8859-5, IBM866, IBM855, MacCyrillic), Greek, Hebrew, Japanese (Shift_JIS, EUC-JP, ISO-2022-JP), Korean, Unicode and Windows-1252. Central European code pages (Windows-1250, ISO-8859-2) aren’t detected. Names are returned as strings that Charset.forName accepts.

Installation#

Maven:

XML
<dependency>
  <groupId>com.github.albfernandez</groupId>
  <artifactId>juniversalchardet</artifactId>
  <version>2.5.2</version>
</dependency>

Gradle: implementation("com.github.albfernandez:juniversalchardet:2.5.2")

Basic usage: detect the encoding of a file#

Java
import java.io.File;
import java.io.IOException;
import java.nio.charset.Charset;
import java.nio.file.Files;
import org.mozilla.universalchardet.UniversalDetector;

public class Detect {
    public static void main(String[] args) throws IOException {
        File file = new File(args[0]);
        String encoding = UniversalDetector.detectCharset(file);
        if (encoding == null) {
            System.out.println("no encoding detected (ASCII or too little text)");
            encoding = "UTF-8";
        }
        System.out.println(encoding);
        String text = Files.readString(file.toPath(), Charset.forName(encoding));
        System.out.println(text.length() + " characters");
    }
}

How to detect while streaming#

Feed chunks until the detector is sure, then call dataEnd:

Java
UniversalDetector detector = new UniversalDetector();
byte[] buf = new byte[4096];
try (var in = Files.newInputStream(path)) {
    int n;
    while ((n = in.read(buf)) > 0 && !detector.isDone()) {
        detector.handleData(buf, 0, n);
    }
}
detector.dataEnd();
String encoding = detector.getDetectedCharset();   // null if unknown
detector.reset();                                  // before reusing the detector

How to open a Reader with the detected encoding#

ReaderFactory detects the encoding, skips a BOM and falls back to the platform default:

Java
import org.mozilla.universalchardet.ReaderFactory;

try (var reader = ReaderFactory.createBufferedReader(new File("unknown.txt"))) {
    reader.lines().forEach(System.out::println);
}

How to detect the encoding of a byte array#

Java
UniversalDetector detector = new UniversalDetector();
detector.handleData(bytes, 0, bytes.length);
detector.dataEnd();
String encoding = detector.getDetectedCharset();

Tips#