All formats

UTF.Unknown

LanguageC# (.NET 8+, .NET Standard 2.0)
LicenseMPL-1.1, or GPL-2.0-or-later, or LGPL-2.1-or-later (your choice)
Repositorygithub.com/CharsetDetector/UTF-unknown
DocumentationREADME
Depends onnothing
Read / writedetects the encoding; returns a System.Text.Encoding for reading

UTF.Unknown is a .NET port of Mozilla’s universal charset detector, based on Ude and uchardet. It recognizes BOMs, UTF-8, UTF-16/32 and the legacy single-byte and CJK encodings of about 40 languages, and returns a ranked list of guesses with confidence values.

The result includes a System.Text.Encoding object when .NET has that encoding. On .NET Core and .NET 5+, register CodePagesEncodingProvider first, or legacy encodings come back with Encoding == null and only a name.

Installation#

Shell
dotnet add package UTF.Unknown

Basic usage: detect and read a file#

C#
using System;
using System.IO;
using System.Text;
using UtfUnknown;

Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);

DetectionResult result = CharsetDetector.DetectFromFile("unknown.txt");
DetectionDetail best = result.Detected;
if (best == null)
{
    Console.WriteLine("no encoding detected (empty or binary file?)");
    return;
}
Console.WriteLine($"{best.EncodingName}, confidence {best.Confidence:P0}, BOM: {best.HasBOM}");

Encoding enc = best.Encoding ?? Encoding.UTF8;
string text = File.ReadAllText("unknown.txt", enc);
Console.WriteLine(text.Length);

How to see all candidate encodings#

C#
var result = CharsetDetector.DetectFromFile("unknown.txt");
foreach (DetectionDetail d in result.Details)
    Console.WriteLine($"{d.EncodingName,-15} {d.Confidence:F2}");

How to detect the encoding of bytes or a stream#

C#
byte[] bytes = await File.ReadAllBytesAsync("data.csv");
var fromBytes = CharsetDetector.DetectFromBytes(bytes);

using var stream = File.OpenRead("data.csv");
var fromStream = CharsetDetector.DetectFromStream(stream);

How to detect asynchronously#

C#
DetectionResult result = await CharsetDetector.DetectFromFileAsync("big.log", cancellationToken);

How to convert any text file to UTF-8#

C#
static void ConvertToUtf8(string path)
{
    var best = CharsetDetector.DetectFromFile(path).Detected;
    var enc = best?.Encoding ?? Encoding.GetEncoding(1252);   // fallback
    string text = File.ReadAllText(path, enc);
    File.WriteAllText(path, text, new UTF8Encoding(false));
}

Tips#