All formats

golang.org/x/text/encoding

LanguageGo
LicenseBSD-3-Clause
Repositorygo.googlesource.com/text (GitHub mirror)
Documentationpkg.go.dev/golang.org/x/text/encoding
Depends onnothing outside the Go project
Read / writeconverts to and from UTF-8; detects BOMs; no statistical detection

Go strings are UTF-8 and the standard library has no legacy encodings. The golang.org/x/text/encoding packages, maintained by the Go team, add them:

Package Encodings
encoding/charmap Windows-874, 1250-1258, ISO-8859-1 to -16, KOI8-R, KOI8-U, IBM code pages (437, 850, 866, …), Macintosh
encoding/unicode UTF-8 (with BOM handling), UTF-16 LE/BE with or without BOM
encoding/unicode/utf32 UTF-32
encoding/japanese Shift_JIS, EUC-JP, ISO-2022-JP
encoding/simplifiedchinese GBK, GB18030, HZ-GB-2312
encoding/traditionalchinese Big5
encoding/korean EUC-KR
encoding/htmlindex, encoding/ianaindex look up an encoding by name ("windows-1252", "shift_jis")

Each encoding has a NewDecoder() (to UTF-8) and NewEncoder() (from UTF-8). They work on byte slices, strings, or as transform.Transformers wrapped around an io.Reader or io.Writer for streaming.

Installation#

Shell
go get golang.org/x/text

Basic usage: read a Windows-1252 file and write UTF-8#

Go
package main

import (
	"bufio"
	"fmt"
	"log"
	"os"

	"golang.org/x/text/encoding/charmap"
	"golang.org/x/text/transform"
)

func main() {
	f, err := os.Open("old.txt")
	if err != nil {
		log.Fatal(err)
	}
	defer f.Close()

	// wrap the file: everything read from r is UTF-8
	r := transform.NewReader(f, charmap.Windows1252.NewDecoder())
	sc := bufio.NewScanner(r)
	for sc.Scan() {
		fmt.Println(sc.Text())
	}
	if err := sc.Err(); err != nil {
		log.Fatal(err)
	}
}

How to read UTF-16 or UTF-8 files with a BOM#

unicode.BOMOverride switches to the encoding given by a BOM, and otherwise uses the fallback decoder:

Go
import (
	"io"
	"os"

	"golang.org/x/text/encoding/unicode"
	"golang.org/x/text/transform"
)

func readText(path string) (string, error) {
	f, err := os.Open(path)
	if err != nil {
		return "", err
	}
	defer f.Close()
	dec := unicode.BOMOverride(unicode.UTF8.NewDecoder()) // UTF-8 if no BOM
	b, err := io.ReadAll(transform.NewReader(f, dec))
	return string(b), err
}

The BOM is removed from the output.

How to write UTF-16 LE with a BOM#

Go
enc := unicode.UTF16(unicode.LittleEndian, unicode.UseBOM).NewEncoder()
out, err := enc.String("Grüße\r\n")
if err != nil {
	log.Fatal(err)
}
os.WriteFile("unicode.txt", []byte(out), 0o644)

How to get an encoding by name#

Go
import "golang.org/x/text/encoding/htmlindex"

enc, err := htmlindex.Get("shift_jis") // WHATWG names and labels
if err != nil {
	log.Fatal(err) // unknown name
}
utf8, err := enc.NewDecoder().Bytes(sjisBytes)

How to detect the encoding of an HTML page or a file#

golang.org/x/net/html/charset (a separate module) applies the WHATWG rules: BOM, then the Content-Type header, then a <meta> tag, then a guess:

Go
import "golang.org/x/net/html/charset"

enc, name, certain := charset.DetermineEncoding(content, "text/plain")
fmt.Println(name, certain)
text, _ := enc.NewDecoder().Bytes(content)

For text without these hints use chardet.

Tips#