All formats

charset-normalizer

LanguagePython (3.7+)
LicenseMIT
Repositorygithub.com/jawah/charset_normalizer
Documentationcharset-normalizer.readthedocs.io
Depends onnothing (optional compiled speedups in the wheels)
Read / writedetects the encoding and decodes; converts to UTF-8

charset-normalizer decodes the input with each plausible encoding and scores the results for “chaos” (odd character mixes, unprintable characters) and “coherence” with the letter frequencies of known languages. The best result is the most readable decoding. It supports all 99 encodings that Python’s codecs provide, detects BOMs, and reports the language.

It’s the encoding detector of the requests library, and offers a chardet-compatible detect() function and a normalizer command line tool.

Installation#

Shell
pip install charset-normalizer

Basic usage: detect and decode a file#

Python
from charset_normalizer import from_path

results = from_path("unknown.txt")
best = results.best()
if best is None:
    raise SystemExit("not text, or no encoding fits")

print(best.encoding)          # e.g. 'cp1252'
print(best.language)          # e.g. 'French'
print(best.bom)               # True if the file starts with a BOM
text = str(best)              # the decoded text
print(len(text))

How to convert a file to UTF-8#

Python
from charset_normalizer import from_path

best = from_path("old.txt").best()
if best is not None:
    with open("old-utf8.txt", "wb") as f:
        f.write(best.output())     # bytes, re-encoded as UTF-8 (default)

How to detect the encoding of bytes#

Python
from charset_normalizer import from_bytes

data = "Привет, как дела? Это проверка кодировки.".encode("cp1251")
best = from_bytes(data).best()
print(best.encoding, best.language)    # cp1251 Russian

How to restrict the candidate encodings#

When you know the file is one of a few encodings, say so; it’s faster and avoids wrong guesses:

Python
results = from_bytes(data, cp_isolation=["utf_8", "cp1252", "cp1250"])

cp_exclusion removes encodings instead.

How to replace chardet#

Python
from charset_normalizer import detect

result = detect(data)    # {'encoding': 'windows-1251', 'language': 'Russian', 'confidence': 0.98...}

How to detect from the command line#

Shell
normalizer unknown.txt          # JSON with encoding, language, alternatives
normalizer -m unknown.txt       # just the encoding name
normalizer -n unknown.txt       # write a UTF-8 copy next to the file

Tips#