System.Text.Encoding

Language	C# / .NET
License	MIT (part of .NET)
Repository	github.com/dotnet/runtime
Documentation	Character encoding in .NET, Encoding class
Depends on	built in; legacy code pages need the `CodePagesEncodingProvider` (built into .NET Core 3.0+ and .NET 5+, NuGet `System.Text.Encoding.CodePages` for .NET Standard)
Read / write	yes / yes; detects BOMs only

.NET strings are UTF-16. System.Text.Encoding converts between them and bytes. File.ReadAllText, File.WriteAllText, StreamReader and StreamWriter take an Encoding argument, and the readers detect UTF-8, UTF-16 and UTF-32 BOMs by default.

On .NET Core and .NET 5+, only the Unicode encodings, ASCII and Latin-1 are available out of the box. Register CodePagesEncodingProvider.Instance once at startup to get Windows-1252, the other Windows and ISO code pages, Shift_JIS, GBK, Big5, EUC-KR and the rest. .NET Framework on Windows has them all without registration.

Installation#

Built in. For .NET Standard 2.0 libraries that need code pages:

Shell

dotnet add package System.Text.Encoding.CodePages

Basic usage: read and write text files#

using System;
using System.IO;
using System.Text;

// UTF-8 without BOM is the default for File.WriteAllText
File.WriteAllText("out.txt", "Grüße, 世界\n");

// ReadAllText detects a BOM (UTF-8, UTF-16, UTF-32) and falls back to UTF-8
string text = File.ReadAllText("out.txt");

// read a legacy Windows-1252 file
Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);
Encoding cp1252 = Encoding.GetEncoding(1252);   // or "windows-1252"
string old = File.ReadAllText("old.txt", cp1252);

// write UTF-16 LE with a BOM, as Windows tools expect
File.WriteAllText("unicode.txt", old, Encoding.Unicode);
Console.WriteLine(old.Length);

How to find out which BOM a file has#

StreamReader reports the encoding after the first read:

using var reader = new StreamReader("file.txt", new UTF8Encoding(false), detectEncodingFromByteOrderMarks: true);
reader.Peek();                                 // triggers BOM detection
Console.WriteLine(reader.CurrentEncoding.WebName);   // utf-8, utf-16, utf-16BE, utf-32

CurrentEncoding stays the one you passed if there is no BOM.

How to write UTF-8 with or without a BOM#

File.WriteAllText("no-bom.txt", text, new UTF8Encoding(encoderShouldEmitUTF8Identifier: false));
File.WriteAllText("with-bom.txt", text, new UTF8Encoding(encoderShouldEmitUTF8Identifier: true));
// note: Encoding.UTF8 writes a BOM

How to check that a file is valid UTF-8#

A UTF8Encoding created with throwOnInvalidBytes: true throws instead of inserting U+FFFD:

static bool IsValidUtf8(byte[] bytes)
{
    var strict = new UTF8Encoding(false, throwOnInvalidBytes: true);
    try { strict.GetString(bytes); return true; }
    catch (DecoderFallbackException) { return false; }
}

On .NET 8+, System.Text.Unicode.Utf8.IsValid(bytes) does the same without allocating.

How to convert a file between encodings#

Encoding.RegisterProvider(CodePagesEncodingProvider.Instance);
var shiftJis = Encoding.GetEncoding("shift_jis");
string s = File.ReadAllText("sjis.txt", shiftJis);
File.WriteAllText("utf8.txt", s);   // UTF-8, no BOM

How to normalize line endings#

string unix = text.ReplaceLineEndings("\n");   // .NET 6+: handles \r\n, \r, \n and Unicode separators

Tips#

Encoding.Default is UTF-8 on .NET Core and .NET 5+, but the system ANSI code page on .NET Framework. Pass an explicit encoding.
Encoding.GetEncoding(0) returns the system ANSI code page (e.g. 1252) once the code pages provider is registered.
To guess the encoding of files without a BOM, use UTF.Unknown.