whoiscoder

whoiscoder

Unable to add items to saved index of Hnswlib

Hi All,

I am trying to store text embeddings of 384 dimensions in index of hnswlib (GitHub - elixir-nx/hnswlib: Elixir binding for the hnswlib library.).

def load_index() do
{:ok, saved_index} = HNSWLIB.Index.load_index(:l2, 384, "my_index.bin")
end

idx = load_index()
    IO.inspect idx
    #{:ok, idx} = HNSWLib.Index.new(:l2, 384, 1000)  #works fine
    IO.inspect HNSWLib.Index.get_current_count(idx) #gives error {:error, "The number of elements exceeds the specified limit"}

    data = Embed.similarity(text)
    IO.inspect Nx.shape(data[0]) # {384}

    IO.inspect HNSWLib.Index.add_items(idx, data[0])
    IO.inspect HNSWLib.Index.get_current_count(idx) {:ok, 0}

It works fine if I try to add items to newly created index, however throws error : “The number of elements exceeds the specified limit” when I try to load the existing index and try to add same embeddings.

First Post!

NickTaylor

NickTaylor

This is being tracked in HNSWLib.Index.new should not accept 0 for max_elements · Issue #5 · elixir-nx/hnswlib · GitHub, and I am hopefully going to push up a PR soon.

Where Next?

Popular in Questions Top

New
quazar
How to set Jason to encode all fields in ecto schema, I don’t care about security and implementing only is taking long list of attributes...
New
jononomo
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set? Thanks.
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
New
johnnyicon
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
New
ovidiubadita
Hey all, I discovered Elixir and I love it. I always wanted to learn a functional programming and I intended to go for Haskell, but afte...
New
sabri
Can someone explain the settings of pool_size of Ecto in config file? and what is the recommend size? Thanks
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New

Other popular topics Top

JakeBecker
TL;DR: I’ve just released an implementation of Microsoft’s IDE-independent Language Server Protocol for Elixir. It adds language support ...
1140 51847 244
New
chrismccord
As promised, the first release candidate of Phoenix 1.3.0 is out! This release focuses on code generators with improved project structure...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
danschultzer
None of the current solutions worked well for me, so I went ahead and built a user management system from scratch. This project took far...
548 27727 240
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
fayddelight
I tried installing elixir 1.11.2 erlang 23.3.4 via asdf in my zsh shell. Enabled the versions locally and globally. When I list them ...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 42633 214
New
johnnyicon
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
New
vrod
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New

We're in Beta

About us Mission Statement