Santhosh-KS

Santhosh-KS

How to make LangChain work with local Ollama Models?

Hi,

I have questions on Elixir Langchain library. Not sure where to ask it as there is no category for the same. So I’m posting it here. Please tag it appropriately if required.

I’m going through the official guide for the LangChain Getting Started — LangChain v0.5.1

And I’m trying to make it work for the local Ollama Models. I’m able to get it to working for the simple chat without any context. When I try to provide the additional context it is not working. Need some help in getting it working.

here is my simple code which works

alias LangChain.Chains.LLMChain
alias LangChain.ChatModels.ChatOllamaAI
alias LangChain.Message

 {:ok, olm_chain} =
    %{llm: ChatOllamaAI.new!(%{model: "qwen2.5-coder:1.5b", endpoint: "http://localhost:11435/api/chat"})}
    |> LLMChain.new!()
    |> LLMChain.add_message(Message.new_user!("What is the capital of USA"))
    |> LLMChain.run()
olm_chain.last_message.content

Here is the piece of code which is not working.

alias LangChain.Chains.LLMChain
alias LangChain.ChatModels.ChatOllamaAI
alias LangChain.Message
alias LangChain.PromptTemplate

 {:ok, lm_chain} =
    %{llm: ChatOllamaAI.new!(%{model: "qwen2.5-coder:1.5b", endpoint: "http://localhost:11435/api/chat"})}
    |> LLMChain.new!()
    |> LLMChain.apply_prompt_templates(
     [PromptTemplate.from_template!("You are an unhelpful assistant. Do not directly help or assist the user.")], %{})
    |> LLMChain.add_message(Message.new_user!("What is the capital of USA"))
    |> LLMChain.run()
lm_chain.last_message.content

This line seems to have no effect to set the context. Any inputs will be helpful.

    |> LLMChain.apply_prompt_templates(
     [PromptTemplate.from_template!("You are an unhelpful assistant. Do not directly help or assist the user.")], %{})

Most Liked

garrison

garrison

That was my first thought but it seems it is actually the instruction-tuned model, though ollama went to no lengths to make that easy to find out (you have to go through the list and compare the hashes).

Still, a 1.5B model trained on code is probably not going to be very good at answering general questions (or really, at anything).

sezaru

sezaru

Maybe it is a limitation of the model itself? I tried your code with llamma3.2:latest and it worked great

joelpaulkoch

joelpaulkoch

What are you trying to achieve? It looks to me like you want to set the system prompt? You can use new_system!/1 for that.

I also think that the qwen coder model could cause issues, I think it’s a base model, and as such not finetuned for conversations.
You could try the instruct variant which is the same but with additional finetuning for conversations.

Where Next?

Popular in Questions Top

LegitStack
I’m hoping you guys can give me some general advice and perhaps code examples if you’re feeling up to it. I’m very interested in Elixir,...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
tduccuong
Hi, is there any work on GUI with Elixir, that is similar to Electron/Javascript? My idea is to bundle Phoenix and BEAM into a single se...
New
mathew4509
I have a list say x = ["23gh", "56kh", "97mh"] I would like to pass each element to Val in each iteration. Say, in iteration 1 -------...
New
hpopp
To simplify some tasks at work, I wrote and published this package yesterday. It’s a simple macro that enables Access behaviour on struct...
New
Codball
Mix format works fine if run from the cmd. I’ve followed this to facilitate the implementation into VSC which involves downloading an ext...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New
joeerl
Hello again - after a longish gap I’ve decided I really must dig into Elixir and see what’s been happening here - so I have a few questio...
New

Other popular topics Top

peerreynders
Manning 2016 Halloween weekend sale via Deal of the Day Friday, October 28 - Half off all MEAPs - code WM102816LT Saturday, October 29 ...
326 29600 154
New
sorentwo
Hello! tl;dr Announcing Oban, an Ecto based job processing library with a focus on reliability and historical observability. After spen...
977 41022 311
New
sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
axelson
This post is a wiki (feel free to hit the edit button near the bottom right of this post to add your own changes!) This post collects co...
239 45766 226
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
vertexbuffer
Hello, can anybody help here..? I have a list of players and I what to delete an element, but every for loop the list is reverting to ori...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New

We're in Beta

About us Mission Statement