Use binary pattern matching together with list comprehension:

for <<chunk::size(chunk_size)-binary <- input>>, do: chunk

A relevant library that came to my attention recently is text_chunker_ex by Revelry (thanks to @hugobarauna for sharing this in Elixir Radar 420).

The aim of the library is to split text to be fed to AI models. As I understood, it mimics some functionality of the LangChain. I have not used it (yet), but seems interesting!

The stackoverflow is likely to be quite a bit slower.

handling utf8 is slow, but if not needed:

Here is a version based on the list comprehension but which takes the leftover into account/

defmodule Chunker do

  alias Chunker

  def chunk(string, size \\ 5), do: chunk(string, size, [])

  defp chunk(<<>>, size, acc), do: Enum.reverse(acc)
  defp chunk(string, size, acc) when byte_size(string) > size do
    <<c::size(size)-binary, rest::binary>> = string
    chunk(rest, size, [c | acc])
  end
  defp chunk(leftover, size, acc) do
    chunk(<<>>, size, [leftover | acc])
  end


  def stackoverflow(string, size \\ 5) do
    string
    |> String.codepoints
    |> Enum.chunk_every(size)
    |> Enum.map(&Enum.join/1)
  end

  def withstream(string, size \\ 5) do
    string
    |> String.codepoints
    |> Stream.chunk_every(size)
    |> Enum.map(&Enum.join/1)
  end

end

And here is the benchee run:

{:ok, data} = File.read("./data/alice29.txt")
Benchee.run(
  %{
    "chunk" => fn -> Chunker.chunk(data, 5000) end,
    "overflow" => fn -> Chunker.stackoverflow(data, 5000) end
  })

Here is the result:

Operating System: Linux
CPU Information: Intel(R) Xeon(R) CPU E3-1245 v3 @ 3.40GHz
Number of Available Cores: 8
Available memory: 15.55 GB
Elixir 1.10.4
Erlang 22.3

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
memory time: 0 ns
parallel: 1
inputs: none specified
Estimated total run time: 21 s

Benchmarking chunk...
Benchmarking overflow...
Benchmarking withstream...

Name                 ips        average  deviation         median         99th %
chunk          586726.39     0.00170 ms  ±1225.64%     0.00145 ms     0.00350 ms
withstream         17.50       57.15 ms    ±14.33%       54.27 ms       81.87 ms
overflow           15.71       63.64 ms    ±16.49%       61.00 ms       94.93 ms

Comparison:
chunk          586726.39
withstream         17.50 - 33528.78x slower +57.14 ms
overflow           15.71 - 37339.36x slower +63.64 ms

well, you could use Stream.chunk_every, so it would lazily chunk your string instead of eagerly. For long strings like you said you are working with, it should improve performance a lot already.

One question though, where are you getting this string from?

I see, yeah, I was just curious cause normally big amounts of data like that come from files or uploads, that could be streamed themselves, avoiding to load the whole thing to memory. But in your case, as it’s a ported lib function that takes a loaded string, you don’t much control over that, so yeah, binary pattern matching seems like the way to go then.