That blog post was written in 2015. At the time, the suggested solution might have been the best way to test for UTF-8 encoding.

Today, it’s easier to use the unicode module or binary matching:

  def is_valid_utf8?(<<_ :: utf8, rest :: binary>>), do: is_valid_utf8?(rest)
  def is_valid_utf8?(<<>>), do: :true
  def is_valid_utf8?(<<_ :: binary>>), do: :false

(Using the unicode module as shown in my previous post is probably faster, but will build more garbage.)

Hey I’m all about using language built-ins. I learned something.

Thank you all. This implementation (slightly tuned from Björn’s answer) seems to do what I need it to do:

def decode(b) when is_list(b) do
  raw = :erlang.list_to_binary(b)

  case :unicode.characters_to_binary(raw) do
    utf8 when is_binary(utf8) -> utf8
    _ -> :unicode.characters_to_binary(raw, :latin1)
  end
end

It is impossible to infer the enccoding just from the bytes.

<<64, 65, 66>> is probably valid in any 8bit encoding that ever existed, you will never know if this is meant to be ASCII, latin, or EBCDIC without someone telling you! If its ASCII or latin1, you can both treat the same, but if its EBCDIC you are lost.

Yes, latin-1 is a subset of UTF-8 in the sense that all latin-1 characters can be expressed in UTF-8. However, the encodings are significantly different for code points >127.

Everything in latin-1 is a single byte; some of those single-byte values (>127) are not legal by themselves in UTF-8. To continue using Björn’s example (his name), the ö character is encoded in latin-1 as the single byte 246. In UTF-8 the byte 246 marks the first byte of a four-byte character sequence; if those bytes (which must each be in the range 128…191) do not follow, the UTF-8 sequence is invalid and the decoder must return an error.

If that error occurs when attempting to decode as UTF-8, we then fall back to latin-1.