Note, especially for UTF-16 detection, there is always an element of ambiguity. This post<\/a> by Raymond shows that however you try and detect encoding there will always be some sequence of bytes that will make your guesses look stupid.<\/p>\n\n\n\n Here are the detection methods I’m currently using for the various types of text file. The order of the checks I perform are:<\/p>\n\n\n\n The C# and C++ library can be downloaded from GitHub here: https:\/\/github.com\/AutoItConsulting\/text-encoding-detect I assume that if I find a BOM at the start of the file that it is valid. Although it’s possible that the BOM could just be ANSI text, it’s highly unlikely. The BOMs are as follows:<\/p>\n\n\n\n UTF-8 checking is reliable with a very low chance of false positives, so this is done first. If the text is valid UTF-8 but all the characters are in the range 0-127<\/strong> then this is essentially ASCII text and can be treated as such – in this case I don’t continue to check for UTF-16.<\/p>\n\n\n\n If a character is in the range of 0-127<\/strong> then it is a single character and nothing more needs to be done. Values above 127<\/strong> indicate multibyte encoding using the next 1, 2 or 3 bytes.<\/p>\n\n\n\n These additional bytes are in the range 128-191<\/strong>. This scheme means that if we decode the text stream based on this method and no unexpected sequences occur then this is almost certainly UTF-8 text.<\/p>\n\n\n\n UTF-16 text is generally made up of 2-byte sequences (technically, there can be a 4-byte sequence with surrogate pairs). Depending on the endianness of the file the unicode character 0x1234 could be represented in the character stream as “0x12 0x34” or “0x34 0x12”. The BOM is usually used to easily determine if the file is in big or little endian mode. Without a BOM this is a little more tricky to determine.<\/p>\n\n\n\n I use two methods to try and determine if the text is UTF-16 and the endianness. The first is the newline characters 0x0a and 0x0d. Depending on the endianness they will be sequenced as “0x0a 0x00” or “0x00 0x0a”. If every instance of these characters in a text file is encoded the same way then that is a good sign that the text is UTF-16 and if it is big or little endian. The drawback of this method is that it won’t work for very small amounts of text, or files that don’t contain newlines.<\/p>\n\n\n\n The second method relies on the fact that many files may contain large amounts of pure ASCII text in the range 0-127. This applies especially to files generally used in IT like scripts and logs. When encoded in UTF-16 these are represented as the ASCII character and a null character. For example, space, 0x20 would be encoded as “0x00 0x20” or “0x20 0x00”. Depending on the endianness this will result in a large amount of nulls in the odd or even byte positions. We just need to scan the file for these odd and even nulls and if there is a significant percentage in the expected position then we can assume the text is UTF-16.<\/p>\n\n\n\n The C# and C++ library can be downloaded from GitHub here: https:\/\/github.com\/AutoItConsulting\/text-encoding-detect Using C# as the example, the two main public functions are:<\/p>\n\n\n\n\n
Downloads<\/h2>\n\n\n\n
![\"download_zip_106x51@2x\"](\"https:\/\/www.autoitconsulting.com\/site\/wp-content\/uploads\/2018\/06\/download_github_106x51@2x.png\")
<\/a><\/p>\n\n\n\nBOM Detection<\/h2>\n\n\n\n
Encoding<\/th> BOM<\/th><\/tr> UTF-8<\/td> 0xEF, 0xBB, 0xBF<\/td><\/tr> UTF-16 Little Endian<\/td> 0xFF, 0xFE<\/td><\/tr> UTF-16 Big Endian<\/td> 0xFE, 0xFF<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n UTF-8 Detection<\/h2>\n\n\n\n
First byte<\/th> Number of bytes in sequence<\/th><\/tr> 0-127<\/td> 1 byte<\/td><\/tr> 194-223<\/td> 2 bytes<\/td><\/tr> 224-239<\/td> 3 bytes<\/td><\/tr> 240-244<\/td> 4 bytes<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n UTF-16 Detection<\/h2>\n\n\n\n
The Library<\/h2>\n\n\n\n
<\/a><\/p>\n\n\n\n