Santhosh-KS
How to make LangChain work with local Ollama Models?
Hi,
I have questions on Elixir Langchain library. Not sure where to ask it as there is no category for the same. So I’m posting it here. Please tag it appropriately if required.
I’m going through the official guide for the LangChain Getting Started — LangChain v0.5.1
And I’m trying to make it work for the local Ollama Models. I’m able to get it to working for the simple chat without any context. When I try to provide the additional context it is not working. Need some help in getting it working.
here is my simple code which works
alias LangChain.Chains.LLMChain
alias LangChain.ChatModels.ChatOllamaAI
alias LangChain.Message
{:ok, olm_chain} =
%{llm: ChatOllamaAI.new!(%{model: "qwen2.5-coder:1.5b", endpoint: "http://localhost:11435/api/chat"})}
|> LLMChain.new!()
|> LLMChain.add_message(Message.new_user!("What is the capital of USA"))
|> LLMChain.run()
olm_chain.last_message.content
Here is the piece of code which is not working.
alias LangChain.Chains.LLMChain
alias LangChain.ChatModels.ChatOllamaAI
alias LangChain.Message
alias LangChain.PromptTemplate
{:ok, lm_chain} =
%{llm: ChatOllamaAI.new!(%{model: "qwen2.5-coder:1.5b", endpoint: "http://localhost:11435/api/chat"})}
|> LLMChain.new!()
|> LLMChain.apply_prompt_templates(
[PromptTemplate.from_template!("You are an unhelpful assistant. Do not directly help or assist the user.")], %{})
|> LLMChain.add_message(Message.new_user!("What is the capital of USA"))
|> LLMChain.run()
lm_chain.last_message.content
This line seems to have no effect to set the context. Any inputs will be helpful.
|> LLMChain.apply_prompt_templates(
[PromptTemplate.from_template!("You are an unhelpful assistant. Do not directly help or assist the user.")], %{})
Most Liked
garrison
That was my first thought but it seems it is actually the instruction-tuned model, though ollama went to no lengths to make that easy to find out (you have to go through the list and compare the hashes).
Still, a 1.5B model trained on code is probably not going to be very good at answering general questions (or really, at anything).
sezaru
Maybe it is a limitation of the model itself? I tried your code with llamma3.2:latest and it worked great
joelpaulkoch
What are you trying to achieve? It looks to me like you want to set the system prompt? You can use new_system!/1 for that.
I also think that the qwen coder model could cause issues, I think it’s a base model, and as such not finetuned for conversations.
You could try the instruct variant which is the same but with additional finetuning for conversations.







