All formats

String#encode (Ruby built-in)

LanguageRuby
LicenseRuby License or BSD-2-Clause (part of Ruby)
Repositorygithub.com/ruby/ruby
DocumentationEncoding, String#encode, Encodings guide
Depends onbuilt in
Read / writeyes / yes; detects BOMs with "bom|utf-8"; no statistical detection

Every Ruby string has an encoding. String#encode converts the characters to another encoding, String#force_encoding relabels the bytes without changing them, and valid_encoding? checks them. File.read, File.open and IO.readlines accept encoding: "external:internal" to decode while reading, and the special "BOM|UTF-8" encoding strips a BOM and switches to UTF-16/32 when the BOM says so.

Ruby supports over 100 encodings, including Windows-125x, ISO-8859-x, IBM437, KOI8-R, Shift_JIS, CP932, EUC-JP, GBK, GB18030, Big5 and EUC-KR (Encoding.name_list lists them).

Installation#

Built in.

Basic usage: read a Windows-1252 file as UTF-8#

Ruby
# decode while reading: file is Windows-1252, strings come back as UTF-8
text = File.read("old.txt", encoding: "Windows-1252:UTF-8")
puts text.encoding   # UTF-8

File.write("old-utf8.txt", text)   # UTF-8 bytes, no BOM

How to read a file that may have a BOM#

Ruby
text = File.read("export.txt", mode: "r:BOM|UTF-8")
# UTF-8 BOM removed; a UTF-16/32 BOM switches to that encoding
text = text.encode("UTF-8") unless text.encoding == Encoding::UTF_8

How to check and repair invalid UTF-8#

Ruby
data = File.binread("mixed.txt").force_encoding("UTF-8")
unless data.valid_encoding?
  data = data.scrub("�")   # replace invalid bytes
end

How to convert, replacing characters the target can’t represent#

Ruby
ascii = "Grüße, 世界".encode("US-ASCII", invalid: :replace, undef: :replace, replace: "?")
# => "Gr??e, ??"

sjis = "日本語".encode("Shift_JIS")
utf16 = "hello".encode("UTF-16LE")

How to normalize line endings#

Ruby
unix = text.encode(universal_newline: true)   # \r\n and \r become \n
dos  = unix.encode(crlf_newline: true)        # \n becomes \r\n

Tips#