@darnahsan I fixed the rope scaling, so you should be able to get pass that error on Bumblebee main : )

Bumblebee 101

Mix.install([
  {:bumblebee, [github: "elixir-nx/bumblebee", override: true]},
  {:nx, "~> 0.8.0", [override: true]},
  {:exla, "~> 0.8.0", [override: true]},
  {:kino, "~> 0.14.0", [override: true]},
  {:kino_bumblebee, [github: "livebook-dev/kino_bumblebee", override: true]},
  {:kino_db, "~> 0.2.12"}
])

Nx.global_default_backend({EXLA.Backend, client: :host})

Untitled

hf_token = System.fetch_env!("LB_HF_TOKEN")
{:ok, bert} = Bumblebee.load_model({:hf, "google-bert/bert-base-uncased"})
{:ok, tokenizer} = Bumblebee.load_tokenizer({:hf, "google-bert/bert-base-uncased"})

serving = Bumblebee.Text.fill_mask(bert, tokenizer)

text_input = Kino.Input.text("Sentence with mask", default: "The capital of [MASK] is Paris.")

text = Kino.Input.read(text_input)

Nx.Serving.run(serving, text)

phi3_5 = {:hf, "microsoft/Phi-3.5-mini-instruct"}
t5_flant = {:hf, "google/flan-t5-large"}
smollm = {:hf, "HuggingFaceTB/SmolLM-1.7B"}
llama_minitron = {:hf, "nvidia/Llama-3.1-Minitron-4B-Width-Base"}
repo = phi3_5

{:ok, model} = Bumblebee.load_model(repo)

{:ok, tokenizer} = Bumblebee.load_tokenizer(repo)

{:ok, generation_config} = Bumblebee.load_generation_config(repo)

generation_config = Bumblebee.configure(generation_config, max_new_tokens: 256)

serving = Bumblebee.Text.generation(model, tokenizer, generation_config,
        compile: [batch_size: 1, sequence_length: 256],
        stream: true,
        defn_options: [compiler: EXLA]
)

Kino.start_child({Nx.Serving, name: Model, serving: serving})

prompt = "Complete the paragraph: our solar system is"

Nx.Serving.batched_run(Model, prompt) |> Enum.each(&IO.write/1)

It was added after the last release, so try bumblebee main : )

Llama3.1-Minitron and SmolLM models are outputting garbage

I tried nvidia/Llama-3.1-Minitron-4B-Width-Base and it worked fine, but SmolLM has another option related to parameters that we didn’t respect, hence the debug message. I’ve just fixed it on main and HuggingFaceTB/SmolLM-1.7B-Instruct also works.

For the record, here’s the notebook:

# llama3

```elixir
Mix.install([
  {:bumblebee, github: "elixir-nx/bumblebee"},
  {:nx, "~> 0.8.0", override: true},
  {:exla, "~> 0.8.0", override: true},
  {:kino, "~> 0.14.0"}
])

Nx.global_default_backend(EXLA.Backend)
```

## Section

```elixir
# repo = {:hf, "nvidia/Llama-3.1-Minitron-4B-Width-Base"}
repo = {:hf, "HuggingFaceTB/SmolLM-1.7B-instruct"}

{:ok, model_info} = Bumblebee.load_model(repo)
{:ok, tokenizer} = Bumblebee.load_tokenizer(repo)
{:ok, generation_config} = Bumblebee.load_generation_config(repo)

:ok
```

<!-- livebook:{"output":true} -->

```

12:14:16.182 [info] Loaded cuDNN version 90400

```

<!-- livebook:{"output":true} -->

```
:ok
```

```elixir
generation_config =
  Bumblebee.configure(generation_config,
    max_new_tokens: 256
    # strategy: %{type: :multinomial_sampling, top_p: 0.6}
  )

serving =
  Bumblebee.Text.generation(model_info, tokenizer, generation_config,
    compile: [batch_size: 1, sequence_length: 256],
    stream: true,
    defn_options: [compiler: EXLA]
  )

Kino.start_child({Nx.Serving, name: Llama, serving: serving})
```

<!-- livebook:{"output":true} -->

```
{:ok, #PID<0.261.0>}
```

```elixir
prompt = "Complete the paragraph: our solar system is"

Nx.Serving.batched_run(Llama, prompt) |> Enum.each(&IO.write/1)
```

<!-- livebook:{"output":true} -->

```
 a vast and complex system of celestial bodies that orbit around the sun. The planets in our solar system are divided into two categories: terrestrial planets and gas giants. The terrestrial planets are rocky and have a solid surface, while the gas giants are composed mostly of hydrogen and helium gases. The four terrestrial planets are Mercury, Venus, Earth, and Mars, while the four gas giants are Jupiter, Saturn, Uranus, and Neptune. The gas giants are much larger than the terrestrial planets, with Jupiter being the largest. The gas giants are also much more massive than the terrestrial planets, with Jupiter being over 10 times larger than Earth. The gas giants are also much more distant from the sun than the terrestrial planets, with Jupiter being over 10 times farther away than Earth.
```

<!-- livebook:{"output":true} -->

```
:ok
```

LIVEBOOK_HF_TOKEN

All env vars with LIVEBOOK_ prefix are used to configure Livebook itself and we remove them from env on startup, that’s why it is not propagated to the runtime. You can make it LB_HF_TOKEN and it should work. You can also set the token using Livebook secrets, see the lock icon in the session sidebar.

Thanks, managed to have the models run on CPU though they tend to get stuck in a loop after couple of sentences .

If you can provide a notebook that reproduces it, I can have a look : )

Regarding ollama, if there is quantization is involved then that is definitely a major memory usage reduction. There was a recent work to support quantization in Axon, but currently it still requires loading the full-precision parameters first and converting to quantized version.