All formats

chardet (Go)

LanguageGo
LicenseMIT (ICU-derived parts under the ICU license)
Repositorygithub.com/saintfish/chardet
Documentationpkg.go.dev/github.com/saintfish/chardet
Depends onstandard library only
Read / writedetects the encoding; no conversion

This package ports ICU’s CharsetDetector to pure Go. NewTextDetector analyzes plain text and NewHtmlDetector skips HTML tags first. It returns the charset name, the language and a confidence from 0 to 100. It recognizes UTF-8, UTF-16/32, ISO-8859-x, Windows-125x, KOI8-R, Shift_JIS, EUC-JP, EUC-KR, GB18030, Big5 and ISO-2022 variants.

The code is stable but rarely updated (last change in 2023); github.com/gogs/chardet is a fork with the same API. Names follow ICU (windows-1252, Shift_JIS), which golang.org/x/text’s htmlindex.Get understands.

Installation#

Shell
go get github.com/saintfish/chardet

Basic usage: detect the encoding of a file#

Go
package main

import (
	"fmt"
	"log"
	"os"

	"github.com/saintfish/chardet"
)

func main() {
	data, err := os.ReadFile("unknown.txt")
	if err != nil {
		log.Fatal(err)
	}
	det := chardet.NewTextDetector()
	res, err := det.DetectBest(data)
	if err != nil {
		log.Fatal(err) // chardet.NotDetectedError
	}
	fmt.Printf("%s (language %q, confidence %d)\n", res.Charset, res.Language, res.Confidence)
}

How to detect and convert to UTF-8#

Go
import (
	"github.com/saintfish/chardet"
	"golang.org/x/text/encoding/htmlindex"
)

func toUTF8(data []byte) ([]byte, error) {
	res, err := chardet.NewTextDetector().DetectBest(data)
	if err != nil {
		return nil, err
	}
	if res.Charset == "UTF-8" {
		return data, nil
	}
	enc, err := htmlindex.Get(res.Charset)
	if err != nil {
		return nil, fmt.Errorf("unsupported charset %s: %w", res.Charset, err)
	}
	return enc.NewDecoder().Bytes(data)
}

How to list all candidates#

Go
results, err := chardet.NewTextDetector().DetectAll(data)
if err == nil {
	for _, r := range results {
		fmt.Println(r.Charset, r.Confidence)
	}
}

How to detect the encoding of HTML#

Go
res, err := chardet.NewHtmlDetector().DetectBest(htmlBytes) // ignores markup

Tips#