ICU4J

Language	Java (8+)
License	Unicode License v3 (permissive)
Repository	github.com/unicode-org/icu (`icu4j` folder)
Documentation	Character set detection, CharsetDetector API
Depends on	nothing (single JAR, ~14 MB)
Read / write	detects the encoding and decodes; writing uses `java.nio.charset`

ICU4J is the Java edition of ICU, the Unicode Consortium’s library for text processing. Its com.ibm.icu.text.CharsetDetector guesses the encoding of a byte array or stream and returns CharsetMatch objects with the charset name, a confidence from 0 to 100 and the language. A match can decode the text directly with getString() or getReader().

It detects UTF-8, UTF-16/32, ISO-8859-1/2/5/6/7/8/9, Windows-1250 to 1256, KOI8-R, Shift_JIS, EUC-JP, EUC-KR, GB18030, Big5, ISO-2022-JP/CN/KR and some EBCDIC code pages. ICU4J also has transliteration, normalization, collation and more, which is why the JAR is large.

Installation#

Maven:

XML

<dependency>
  <groupId>com.ibm.icu</groupId>
  <artifactId>icu4j</artifactId>
  <version>79.1</version>
</dependency>

Gradle: implementation("com.ibm.icu:icu4j:79.1")

Basic usage: detect and decode a file#

Java

import com.ibm.icu.text.CharsetDetector;
import com.ibm.icu.text.CharsetMatch;
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;

public class Detect {
    public static void main(String[] args) throws IOException {
        byte[] data = Files.readAllBytes(Path.of(args[0]));

        CharsetDetector detector = new CharsetDetector();
        detector.setText(data);
        CharsetMatch match = detector.detect();
        if (match == null) {
            System.out.println("no match");
            return;
        }
        System.out.printf("%s, confidence %d, language %s%n",
            match.getName(), match.getConfidence(), match.getLanguage());
        String text = match.getString();   // decoded with the detected charset
        System.out.println(text.length() + " characters");
    }
}

How to get all candidate encodings#

Java

CharsetDetector detector = new CharsetDetector();
detector.setText(data);
for (CharsetMatch m : detector.detectAll()) {
    System.out.printf("%-12s %3d %s%n", m.getName(), m.getConfidence(), m.getLanguage());
}

How to detect from a stream and get a Reader#

setText(InputStream) needs a stream that supports mark/reset, such as BufferedInputStream:

Java

try (var in = new java.io.BufferedInputStream(Files.newInputStream(path))) {
    CharsetDetector detector = new CharsetDetector();
    detector.setText(in);
    try (java.io.Reader reader = detector.detect().getReader()) {
        // read the text
    }
}

How to give the detector a hint#

Pass the encoding declared elsewhere (an HTTP header, an XML declaration). The detector favors it when the data fits:

Java

detector.setDeclaredEncoding("windows-1252");
detector.setText(data);

How to ignore HTML markup during detection#

Java

detector.enableInputFilter(true);   // skip <...> tags so they don't skew the statistics
detector.setText(htmlBytes);

Tips#

CharsetDetector.getAllDetectableCharsets() lists every name the detector can return.
Confidence values aren’t probabilities; compare them only between matches of the same input.
If ICU4J’s size is a problem, juniversalchardet is a much smaller alternative for detection alone.