Text encoding libraries
Every language reads and writes text files. What the standard libraries usually lack is encoding detection (guessing the encoding of a file that doesn’t declare it) and, in some languages, legacy encodings such as Windows-1252 or Shift_JIS. These libraries fill the gap.
| Language | Library | Detect | Convert | Notes |
|---|---|---|---|---|
| C/C++ | uchardet | yes | no | Mozilla’s universal detector as a C library, used by mpv and many editors |
| C/C++ | ICU | yes | yes | Unicode’s reference library: detection, 200+ converters, normalization |
| C/C++ | iconv | no | yes | POSIX API in glibc and macOS; GNU libiconv elsewhere |
| C# | System.Text.Encoding | BOM only | yes | built into .NET; code pages via CodePagesEncodingProvider |
| C# | UTF.Unknown | yes | no | port of uchardet for .NET |
| Java | java.nio.charset | no | yes | built into the JDK |
| Java | juniversalchardet | yes | no | Mozilla detector port, small |
| Java | ICU4J | yes | yes | CharsetDetector plus ICU’s converters |
| Go | golang.org/x/text/encoding | BOM / HTML | yes | official extended library: code pages, CJK, UTF-16 |
| Go | chardet | yes | no | port of ICU’s detector |
| Python | charset-normalizer | yes | yes | used by requests; MIT |
| Python | chardet | yes | no | the classic detector, rewritten in version 7 (0BSD) |
| Ruby | String#encode | no | yes | built into Ruby |
| Ruby | charlock_holmes | yes | yes | ICU binding, used by GitHub’s Linguist |
| Ruby | rchardet | yes | no | pure Ruby port of chardet |
| Rust | encoding_rs | BOM only | yes | Firefox’s WHATWG encoding implementation |
| Rust | chardetng | yes | no | Firefox’s detector for unlabeled legacy text |
| PHP | mbstring | yes | yes | PHP extension: mb_detect_encoding, mb_convert_encoding |
Detection is a guess. All detectors need a few hundred bytes of non-ASCII text to be reliable, and they confuse closely related encodings (Windows-1252 vs ISO-8859-15, Windows-1251 vs KOI8-R). Check for a BOM and for valid UTF-8 first, and only then run a detector.
By language#
- TXT libraries for C/C++ Detect and convert text file encodings in C and C++ with uchardet, ICU and iconv. Comparison and code examples.
- TXT libraries for C# Read, write and detect text file encodings in C# and .NET with System.Text.Encoding and UTF.Unknown. Code examples.
- TXT libraries for Java Read, write and detect text file encodings in Java with java.nio.charset, juniversalchardet and ICU4J. Comparison and code examples.
- TXT libraries for Go Convert and detect text file encodings in Go with golang.org/x/text/encoding and chardet. Code examples for Windows-1252, UTF-16, Shift_JIS and BOM handling.
- TXT libraries for Python Detect the encoding of text files in Python with charset-normalizer and chardet, then decode with Python's built-in codecs. Comparison and code examples.
- TXT libraries for Ruby Convert and detect text file encodings in Ruby with String#encode, charlock_holmes and rchardet. Code examples.
- TXT libraries for Rust Convert and detect text file encodings in Rust with encoding_rs and chardetng, the libraries used by Firefox. Code examples.
- TXT libraries for PHP Detect and convert text file encodings in PHP with the mbstring extension: mb_detect_encoding, mb_convert_encoding and mb_check_encoding. Code examples.