hakarabakara

hakarabakara

Replacing placeholders in a string from CSV values

Hi,

My application accepts a string input and dynamic values from an uploaded CSV file and should generate a response based on the csv.

The string has placeholders have corresponding values in the CSV under the headers.

I was wondering whether there is a more efficient way of reading the stream once to perform the whole operation as opposed to fetching columns which are in the first row then streaming through the file again replacing the occurrence of the header in the string.

Most Liked

amnu3387

amnu3387


template =
      "hi #name, you have scored #physics in physics, #chemistry in chemistry and #maths in maths"

input = """
    name,maths,chemistry,physics
    James Ortega,44,14,49
    Eula Garrett,26,74,16
    Emily Marsh,47,22,73
    Tommy Jenkins,37,54,55
    Ronnie Nash,54,04,31
"""

[headers] =
      input
      |> NimbleCSV.RFC4180.parse_string(skip_headers: false)
      |> Enum.take(1)

replacement_hs =
      headers
      |> Enum.map(fn header -> "#" <> header end)

input
|> NimbleCSV.RFC4180.parse_string(skip_headers: true)
|> Enum.map(fn row ->
      Enum.zip_reduce(row, replacement_hs, template, fn col_val, header, acc ->
        String.replace(acc, header, col_val)
    end)
end)

The only reasons to use NimbleCSV here is because sometimes there can be escaped separation characters and if you have a streamable source it’s just easier to get it working. Using @mindok example but with a string I would just map the headers into their replacement tokens ("#some_header") and then brute force the replacements unless it turned out to be slower than acceptable at which maybe compiling a custom regex could be faster.

Sebb

Sebb

Definitely use a CSV-parser. I just wanted to keep it simple.

I like your little template-engine, but maybe its not really beginner-friendly (and it doesn’t address the selection of the template by CSV header).

my solution with nimble

def load(csv) do
  [header | rows] = NimbleCSV.RFC4180.parse_string(csv, skip_headers: false)
  Enum.map(rows, fn row -> sentence(row, header) end)
end

defp sentence([name, email, balance], ["name", "email", "balance"]) do
  "#{name}, you owe us #{balance} this month. Please see your statement here #{email}"
end

defp sentence([other, header], ["other", "header"]) do
  "#{other}, ... #{header} ..."
end

@mnussbaumer’s solution is better because you don’t need to write code for the templates, could even put them in a json if you want. Just create a map header -> template from which to select the template you need.

Sebb

Sebb

Still not sure what you want, is it this?

test "sentences" do
  data = """
  name,email,balance
  Francis Waters,jolir@jalih.mz,$1810.08
  Ina Thomas,duzzigip@hizjos.cl,$5639.13
  George Cortez,siw@jijol.ma,$222.81
  Oscar Nguyen,ov@rici.nu,$7167.56
  Wayne Campbell,nad@tuj.jp,$964.14
  """
  
  rows = String.split(data, "\n")
  Enum.map(rows, fn line -> line |> String.split(",") |> sentence() end)
end


defp sentence([name, email, balance]) do
  "#{name}, you owe us #{balance} this month. Please see your statement here #{email}"
end
defp sentence(_), do: ""
["name, you owe us balance this month. Please see your statement here email",
 "Francis Waters, you owe us $1810.08 this month. Please see your statement here jolir@jalih.mz",
 "Ina Thomas, you owe us $5639.13 this month. Please see your statement here duzzigip@hizjos.cl",
 "George Cortez, you owe us $222.81 this month. Please see your statement here siw@jijol.ma",
 "Oscar Nguyen, you owe us $7167.56 this month. Please see your statement here ov@rici.nu",
 "Wayne Campbell, you owe us $964.14 this month. Please see your statement here nad@tuj.jp",
 ""]
mindok

mindok

Using Enum.take(1) on a stream to get the headers should be pretty efficient. Using Nimble CSV:

  alias NimbleCSV.RFC4180, as: CSV

 ...
 # Creates a list of header names
 [header] 
  = input_csv_stream 
    |> CSV.parse_stream(skip_headers: false) 
    |> Enum.take(1)

 # Then you can do what you need to with the main CSV. If you are a bit lazy (like me) you can zip
 # the header with the value into a map to allow easy lookup for processing your string template
 data 
  = input_csv_stream
    |> CSV.parse_stream(skip_headers: true)
    |> Stream.map(fn row -> Enum.zip(header, row) |> Enum.into(%{}) end)
    |> Stream.map(fn row_map -> template_wrangling_stuff(template, row_map) end)
    |> Enum.take(10) # or however you want to deal with the output

Sebb

Sebb

It takes about 1 sec to load and convert a 1.000.000 lines CSV.
So I’d just stick with the naive implementation for this one (and - having about 1 Billion revenue in that file - just buy a supercomputer to speed things up if need be).

Where Next?

Popular in Questions Top

Brian
What is the proper way to load a module from a file in to IEX? In the python world, doing something like this pretty standard: from ....
New
lessless
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
dokuzbir
Hello, I am trying to convert my lists to string without losing brackets.For start i have 3 map. They look like these buyer = %{ id: ...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
gonzofish
I’m currently trying to understand how to join three tables using Ecto. All the examples I’ve seen use 2, so maybe I’m just missing somet...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
New
LegitStack
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
New
Fl4m3Ph03n1x
About me? ( if you have nothing better to do than reading about some random guy in the internet :stuck_out_tongue: ) Hello all, this is ...
New

Other popular topics Top

chrismccord
Phoenix 1.4.0 released Phoenix 1.4 is out! This release ships with exciting new features, most notably with HTTP2 support, improved deve...
688 30048 115
New
William
I would like to know that is there any online source for learning Phoenix Framework for building E-Commerce Store? Any advantage on build...
New
yawaramin
In the Dialyzer docs ( http://erlang.org/doc/man/dialyzer.html#requesting-or-suppressing-warnings-in-source-files ), there is a way to tu...
New
AstonJ
You’re a programmer, so you don’t need spoon feeding with the conventional drivel about “this is an integer.” No. You need to know what’s...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
stefanluptak
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
New
WestKeys
Currently suffering from paralysis by [HTTP client] analysis. This is rather unusual in Elixirland as there tends to be consensus on the ...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New

We're in Beta

About us Mission Statement