Hey @caslu I think @ken-kost’s answer is more accurate here. Matching isn’t a search of a binary rather it’s a “destructuring” of the binary where you’re pulling out a specific offset from the start into a value. This requires knowing what that offset is. Technically this does NOT need to be at compile time, you can do variable length offsets eg:

iex(2)> n=1; <<stuff::binary-size(n), rest :: binary>> = "hello world"; stuff
"h"
iex(3)> n=2; <<stuff::binary-size(n), rest :: binary>> = "hello world"; stuff
"he"
iex(4)> n=3; <<stuff::binary-size(n), rest :: binary>> = "hello world"; stuff
"hel"

However the point is that at the time the match executes the offset into the binary is known.

A smarter compiler might be able to do what you want; however, allowing variable at the beginning of a binary match will open a big can of worms. For example:

x <> "b" <> y = "abba"

Which one is correct?

• x = "a", y = "ba"
• x="ab", y="a"

I don’t think it would be impossible per se, here is a proof of concept where you can achieve something similar:

message = "hello world"
suffix = "world"

case message do
  <<prefix::binary-size(byte_size(message) - byte_size(suffix)), ^suffix::binary>> -> prefix
end
# "hello "

(I’m cheating because I’m using the message variable from the parent scope, so this wouldn’t be viable as an implementation strategy).

The real problem I assume would be to be able to do it efficiently especially when dispatching over multiple clauses. Binary matching is very optimized in its current implementation, nimble_parsec is a great example leveraging it.
I found this paper when looking for resources explaining how it is being optimized, quite interesting.

@benwilson512

That use of dynamically sized binaries is super interesting! Your observation that matching isn’t searching but is destructuring is a very helpful one.

I did the next logical thing and broke it on purpose to see the error message with your use of binary-size:

iex(2)> n=3; <<stuff::binary-size(n), rest :: binary>> = "hello world"; stuff
"hel"
iex(3)> n=3; <<rest :: binary, stuff::binary-size(n)>> = "hello world"; stuff
error: a binary field without size is only allowed at the end of a binary pattern, at the right side of binary concatenation and never allowed in binary generators. The following examples are invalid:

    rest <> "foo"
    <<rest::binary, "foo">>

They are invalid because there is a bits/bitstring component not at the end. However, the "reverse" would work:

    "foo" <> rest
    <<"foo", rest::binary>>


└─ iex:3

** (CompileError) cannot compile code (errors have been logged)

What a lovingly crafted error message.

@derek-zhou

Ok, this is what I was getting at. I sensed there was some deeper “this is a Hard Problem ” type of thing going on and you certainly highlighted an example.

x <> "b" <> y = "abba" is obviously impossible to destructure “correctly”.

But, I wonder, is x <> "ba" = "abba" impossible to destructure? Could something be programmed in the language itself to use negative offsets or is there some CS theory going on where backtracking is too potentially resource intensive etc.

No, but there is another can of worms that you may encounter. What if there is a multi-byte UTF-8 sequence end with the byte “b”? After matching you could have a x that is not a valid UTF-8 string.

For strings, you always want to match from left. You could do right side match with regular expression but that should be frown upon.