All formats

charlock_holmes

LanguageRuby (C++ extension)
LicenseMIT
Repositorygithub.com/brianmario/charlock_holmes
DocumentationREADME
Depends onICU (libicu), a C++ compiler
Read / writedetects the encoding; transcodes with ICU

charlock_holmes wraps ICU’s charset detector and converter. EncodingDetector.detect returns a hash with the encoding, a confidence from 0 to 100, the type (:text or :binary) and sometimes the language. Converter.convert transcodes between any encodings ICU knows. It’s fast and accurate, and the binary check makes it useful for deciding whether to show a file as text, which is what GitHub’s Linguist uses it for.

The gem builds a native extension, so ICU’s development files must be installed first.

Installation#

Shell
# Debian / Ubuntu
sudo apt install libicu-dev g++
# macOS
brew install icu4c

gem install charlock_holmes
# macOS: point the build at Homebrew's keg-only ICU
gem install charlock_holmes -- --with-icu-dir=$(brew --prefix icu4c)

Basic usage: detect the encoding of a file#

Ruby
require "charlock_holmes"

contents = File.binread("unknown.txt")
detection = CharlockHolmes::EncodingDetector.detect(contents)
# => {:type=>:text, :encoding=>"ISO-8859-1", :confidence=>60, :language=>"fr"}

if detection.nil?
  puts "unknown"
elsif detection[:type] == :binary
  puts "binary file"
else
  puts "#{detection[:encoding]} (#{detection[:confidence]}%)"
end

How to convert any text file to UTF-8#

Ruby
contents = File.binread("old.txt")
detection = CharlockHolmes::EncodingDetector.detect(contents)
utf8 = CharlockHolmes::Converter.convert(contents, detection[:encoding], "UTF-8")
File.write("old-utf8.txt", utf8)

How to detect many files efficiently#

Reuse one detector instance:

Ruby
detector = CharlockHolmes::EncodingDetector.new
Dir["docs/**/*.txt"].each do |path|
  d = detector.detect(File.binread(path))
  puts "#{path}: #{d && d[:encoding]}"
end

How to set a string’s encoding from detection#

Ruby
require "charlock_holmes/string"

contents = File.binread("unknown.txt")
contents.detect_encoding!          # sets the string's encoding to the detected one
puts contents.encode("UTF-8")

Tips#