rchardet

rchardet is a port of the classic Python chardet (the Mozilla universal detector). It has one main function, CharDet.detect, which returns a hash with "encoding" and "confidence" (0.0 to 1.0). Being pure Ruby, it installs anywhere without ICU or a compiler, but it’s much slower than charlock_holmes on large inputs. It’s a dependency of many gems, which is why it has a very large download count.

It detects UTF-8, UTF-16/32, Windows-1252, Windows-1251, KOI8-R, ISO-8859-5/7/8, MacCyrillic, IBM855/866, Shift_JIS, EUC-JP, ISO-2022-JP, GB2312, Big5, EUC-KR, EUC-TW and others. Central European code pages aren’t covered.

Installation#

gem install rchardet

Basic usage: detect and convert a file#

require "rchardet"

data = File.binread("unknown.txt")
cd = CharDet.detect(data)
encoding = cd["encoding"]       # e.g. "windows-1251", or nil
confidence = cd["confidence"]   # 0.0 .. 1.0
puts "#{encoding} (#{confidence.round(2)})"

if encoding
  text = data.force_encoding(encoding).encode("UTF-8")
  File.write("utf8.txt", text)
end

How to detect only the start of a large file#

Detection time grows with input size; a sample is usually enough:

sample = File.open("big.log", "rb") { |f| f.read(64 * 1024) }
cd = CharDet.detect(sample)

How to fall back when confidence is low#

cd = CharDet.detect(data)
encoding = cd["confidence"].to_f >= 0.5 ? cd["encoding"] : "Windows-1252"
text = data.dup.force_encoding(encoding).encode("UTF-8", invalid: :replace, undef: :replace)

Tips#

• Ruby may not know every name rchardet returns; Encoding.find(name) raises ArgumentError for unknown ones, so rescue it.
• ASCII-only input is reported as "ascii", which Ruby accepts as US-ASCII.