zhanjingbaobao

zhanjingbaobao

Merging two lists of maps

I currently have two list data.

data1 = [%{id: "111", name: "name1"},
           %{id: "222", name: "name2"},
           %{id: "333", name: "name3"},
           %{id: "444", name: "name4"}
          ]

data2 = [
  %{
    mapData: %{
      "111": "data1.com",
      "222": "data2.com",
      "333": "data3.com",
    }
  },
  %{
      mapData: %{
        "111": "data11.com",
        "222": "data22.com",
      },
      mapData: %{
        "111": "data111.com"
      }

  }
]

Now I want to obtain data in this format.

 [
           %{id: "111", name: "name1",address:"data1.com"},
           %{id: "111", name: "name1",address:"data11.com"},
           %{id: "111", name: "name1",address:"data111.com"},
           %{id: "222", name: "name2",address:"data2.com"},
           %{id: "222", name: "name22",address:"data22.com"},
           %{id: "333", name: "name3",address:"data3com"},
           %{id: "444", name: "name4",address:""}
          ]

I don’t know how to implement it now.I am a beginner and I hope everyone can help me. Thank you.

Most Liked

Eiji

Eiji

Yes, as this was not mentioned originally.

No worries.

No, it’s not. As above you have added a special cases not mentioned before. Here is the current error:

** (FunctionClauseError) no function clause matching in Example.merge/3    
    
    The following arguments were given to Example.merge/3:
    
        # 1
        []
    
        # 2
        [{"666", "testName666"}]
    
        # 3
        false

We can see that we have done with data, but still have extra names list elements. This means that we only need to modify a strict condition that finishes our recursive process i.e.

# in case no extra names we can strictly pattern match here
defp merge([], [], _last_matched), do: []
# in case we do not care about more items in `names` list
defp merge([], _names, _last_matched), do: []

Also I have simplified my example a bit:

defmodule Example do
  def sample(data, names) when is_list(data) and is_list(names) do
    # a list of names
    # reduce it with an empty list
    names
    |> Enum.reduce([], fn %{mediaTailoId: ids_map, name: name}, acc ->
      # take a map value under mapData key
      ids_map
      # change atom id to string
      |> Enum.map(fn {id, _value} -> {Atom.to_string(id), name} end)
      # concat the result with acc
      |> then(&(acc ++ &1))
    end)
    # the original names are sorted partially i.e. only in specified mediaTailoId
    # to fix that we simply need to sort the flat list result by id
    |> Enum.sort_by(&elem(&1, 0))
    |> then(&merge(data, &1))
  end

  # function head is required when using a default argument(s)
  # in functions with multiple clasule
  defp merge(data, names, last_matched \\ false)

  # when done simply finish a recursive call returning an empty list as result tail
  defp merge([], _names, _last_matched), do: []

  # if id matches add a new map to result and continue with same data and the rest of names
  defp merge([%{channelId: id} = map | _data_tail] = data, [{id, name} | names], _last_matched) do
    [Map.merge(map, %{name: name}) | merge(data, names, true)]
  end

  # if id does not match and previous clause matched for this id
  # we simply continue with rest of data and same names
  defp merge([_data_head | data], names, true), do: merge(data, names, false)

  # otherwise if id still does not match and previous clause do not matched for this id even once
  # add a new map with an empty name and continue with rest of data and same names
  # the combination of last_matched boolean argument and this clause
  # is required in case we do not have any names for specific data like it is for "444" id 
  defp merge([map | data], names, false) do
    [Map.merge(map, %{name: ""}) | merge(data, names, false)]
  end
end

data1 = [
  %{
    channelId: "1",
    # fix: describtion -> description
    description: "dev-test_001",
    tags: %{pipelineId: "1001", pipelineTemplate: "live-drm"}
  },
  %{
    channelId: "2",
    # fix: describtion -> description
    description: "dev-test_002",
    tags: %{pipelineId: "1002", pipelineTemplate: "live-drm"}
  },
  %{
    channelId: "3",
    # fix: describtion -> description
    description: "dev-test_003",
    tags: %{pipelineId: "1003", pipelineTemplate: "live-drm"}
  },
  %{
    channelId: "4",
    # fix: describtion -> description
    description: "dev-test_004",
    tags: %{pipelineId: "1004", pipelineTemplate: "live-drm"}
  }
]

data2 = [
  %{
    mediaTailoId: %{"1": "channel1_data", "2": "channel2_dadta", "3": "channel3_dadta"},
    mediaTailorRegion: "us-east-1",
    name: "testName1"
  },
  %{
    mediaTailoId: %{"1": "channel1_data", "2": "channel2_dadta"},
    mediaTailorRegion: "us-east-2",
    name: "testName2"
  },
  %{mediaTailoId: %{"1": "channel1_data"}, mediaTailorRegion: "us-east-3", name: "testName3"},
  %{mediaTailoId: %{"666": "channel1_data"}, mediaTailorRegion: "us-east-3", name: "testName666"}
]

expect_data = [
  %{
    channelId: "1",
    description: "dev-test_001",
    tags: %{pipelineId: "1001", pipelineTemplate: "live-drm"},
    name: "testName1"
  },
  %{
    channelId: "1",
    description: "dev-test_001",
    tags: %{pipelineId: "1001", pipelineTemplate: "live-drm"},
    name: "testName2"
  },
  %{
    channelId: "1",
    description: "dev-test_001",
    tags: %{pipelineId: "1001", pipelineTemplate: "live-drm"},
    name: "testName3"
  },
  %{
    channelId: "2",
    description: "dev-test_002",
    tags: %{pipelineId: "1002", pipelineTemplate: "live-drm"},
    name: "testName1"
  },
  %{
    channelId: "2",
    description: "dev-test_002",
    tags: %{pipelineId: "1002", pipelineTemplate: "live-drm"},
    name: "testName2"
  },
  %{
    channelId: "3",
    description: "dev-test_003",
    tags: %{pipelineId: "1003", pipelineTemplate: "live-drm"},
    # fix: "testName2" -> "testName1" 
    name: "testName1"
  },
  %{
    channelId: "4",
    description: "dev-test_004",
    tags: %{pipelineId: "1004", pipelineTemplate: "live-drm"},
    name: ""
  }
]

iex> Example.sample(data1, data2) == expect_data
true
gregvaughn

gregvaughn

I’m joining this thread late and to be honest, have not read all the correction details. I offer another approach with get_in that may or may not be of interest. I like these sorts of data manipulation puzzles though :grin:

iex(11)> data1
[
  %{id: "111", name: "name1"},
  %{id: "222", name: "name2"},
  %{id: "333", name: "name3"},
  %{id: "444", name: "name4"}
]
iex(12)> data2
[
  %{mapData: %{"111": "data1", "222": "data2_test", "333": "data3_test"}},
  %{mapData: %{"111": "data11_test", "222": "data22_test"}},
  %{mapData: %{"111": "data111_test"}}
]
iex(13)> Enum.flat_map(data1, fn %{id: id} = d1 ->
...(13)>   id2 = String.to_atom(id)
...(13)>   addrs = data2 |> get_in([Access.all(), :mapData, id2]) |> Enum.reject(&is_nil/1)
...(13)>   addrs = if addrs == [], do: [""], else: addrs
...(13)>  Enum.map(addrs, &Map.put(d1, :address, &1))
...(13)> end)
[
  %{address: "data1", id: "111", name: "name1"},
  %{address: "data11_test", id: "111", name: "name1"},
  %{address: "data111_test", id: "111", name: "name1"},
  %{address: "data2_test", id: "222", name: "name2"},
  %{address: "data22_test", id: "222", name: "name2"},
  %{address: "data3_test", id: "333", name: "name3"},
  %{address: "", id: "444", name: "name4"}
]
cloudytoday

cloudytoday

Could you please post the sample data exactly the way you have it? From this example it’s not clear why in data2 the keys are quoted atoms ("111": ...) and not usual string keys ("111" => ...) and whether this is intentional.

Regardless, you could look for something like this:

iex(39)> for %{id: id, name: name} <- data1,
             %{mapData: m2} <- data2,
             is_map_key(m2, String.to_atom(id)),
             do: %{id: id, name: name, address: m2[String.to_atom(id)]} 
[
  %{address: "data1.com", id: "111", name: "name1"},
  %{address: "data111.com", id: "111", name: "name1"},
  %{address: "data2.com", id: "222", name: "name2"},
  %{address: "data3.com", id: "333", name: "name3"}
]
Eiji

Eiji

Copying these over from the Elixir chat room.

Originally sent in Elixir Chat
zhanjingbaobao

I currently have two list data.

data1 = [
  %{id: "111", name: "name1"},
  %{id: "222", name: "name2"},
  %{id: "333", name: "name3"},
  %{id: "444", name: "name4"}
]

data2 = [
  %{
    mapData: %{
      "111": "data1",
      "222": "data2_test",
      "333": "data3_test"
    }
  },
  %{
    mapData: %{
      "111": "data11_test",
      "222": "data22_test"
    },
    mapData: %{
      "111": "data111_test"
    }
  }
]

# Now I want to obtain data in this format.
mergeData = [
  %{id: "111", name: "name1", address: "data1_test"},
  %{id: "111", name: "name1", address: "data11_test"},
  %{id: "111", name: "name1", address: "data111_test"},
  %{id: "222", name: "name2", address: "data2_test"},
  %{id: "222", name: "name22", address: "data22_test"},
  %{id: "333", name: "name3", address: "data3_test"},
  %{id: "444", name: "name4", address: ""}
]

I don't know how to implement it now.I am a beginner and I hope everyone can help me. Thank you.

Eiji

Your data2 map is invalid as it's keys are not unique and therefore only last would be used.

There are 2 other issues … In data2 the first 111 have data1 address, but should have data1_test. In mergeData you have name22 instead of name2.

After applying all fixed this is a final example code:

defmodule Example do
  def sample(data, addresses) when is_list(data) and is_list(addresses) do
    # a list of addressess
    flat_addresses =
      addresses
      # reduce it with an empty list
      |> Enum.reduce([], fn %{mapData: addresses}, acc ->
        # take a map value under mapData key
        addresses
        # change atom id to string
        |> Enum.map(fn {id, address} -> {Atom.to_string(id), address} end)
        # concat the result with acc
        |> then(&(acc ++ &1))
      end)
      # the original addresses are sorted partially i.e. only in specified mapData
      # to fix that we simply need to sort the flat list result by id
      |> Enum.sort_by(&elem(&1, 0))

    # in order to recursively pattern match key by key we need to change a list of maps
    # to a map with id as key and name as value
    data |> Enum.map(&{&1.id, &1.name}) |> merge(flat_addresses)
  end

  # function head is required when using a default argument(s)
  # in functions with multiple clasule
  defp merge(data, addresses, last_matched \\ false)

  # when done simply finish a recursive call returning an empty list as result tail
  defp merge([], [], _last_matched), do: []

  # if id matches add a new map to result and continue with same data and the rest of addresses
  defp merge([{id, name} | _data_tail] = data, [{id, address} | addresses], _last_matched) do
    [%{address: address, id: id, name: name} | merge(data, addresses, true)]
  end

  # if id does not match and previous clause matched for this id
  # we simply continue with rest of data and same addresses
  defp merge([_data_head | data], addresses, true), do: merge(data, addresses, false)

  # otherwise if id still does not match and previous clause do not matched for this id even once
  # add a new map with an empty address and continue with rest of data and same addresses
  # the combination of last_matched boolean argument and this clause
  # is required in case we do not have any addresses for specific data like it is for "444" id 
  defp merge([{id, name} | data], addresses, false) do
    [%{address: "", id: id, name: name} | merge(data, addresses, false)]
  end
end

data = [
  %{id: "111", name: "name1"},
  %{id: "222", name: "name2"},
  %{id: "333", name: "name3"},
  %{id: "444", name: "name4"}
]

addresses = [
  %{
    mapData: %{
      # fix: "data1" -> "data1_test"
      "111": "data1_test",
      "222": "data2_test",
      "333": "data3_test"
    }
  },
  # fix: split map wih duplicate keys into two separate maps
  %{
    mapData: %{
      "111": "data11_test",
      "222": "data22_test"
    }
  },
  %{
    mapData: %{
      "111": "data111_test"
    }
  }
]

expected = [
  %{id: "111", name: "name1", address: "data1_test"},
  %{id: "111", name: "name1", address: "data11_test"},
  %{id: "111", name: "name1", address: "data111_test"},
  %{id: "222", name: "name2", address: "data2_test"},
  # fix: "name22" -> "name2"
  %{id: "222", name: "name2", address: "data22_test"},
  %{id: "333", name: "name3", address: "data3_test"},
  %{id: "444", name: "name4", address: ""}
]

iex> Example.sample(data, addresses) == expected
true
zhanjingbaobao

Thank you very much Eiji,but when I tested the code, I didn't get the data structure I expected.

I have two pieces of data: data1 and data2,

data1 = [
  %{
    channelId: "1",
    describtion: "dev-test_001",
    tags: %{pipelineId: "1001", pipelineTemplate: "live-drm"}
  },
  %{
    channelId: "2",
    describtion: "dev-test_002",
    tags: %{pipelineId: "1002", pipelineTemplate: "live-drm"}
  },
  %{
    channelId: "3",
    describtion: "dev-test_003",
    tags: %{pipelineId: "1003", pipelineTemplate: "live-drm"}
  },
  %{
    channelId: "4",
    describtion: "dev-test_004",
    tags: %{pipelineId: "1004", pipelineTemplate: "live-drm"}
  }
]

data2 = [
  %{
    mediaTailoId: %{"1": "channel1_data", "2": "channel2_dadta", "3": "channel3_dadta"},
    mediaTailorRegion: "us-east-1",
    name: "testName1"
  },
  %{
    mediaTailoId: %{"1": "channel1_data", "2": "channel2_dadta"},
    mediaTailorRegion: "us-east-2",
    name: "testName2"
  },
  %{mediaTailoId: %{"1": "channel1_data"}, mediaTailorRegion: "us-east-3", name: "testName3"}
]

Here are my expected data.Expected results do not need to be sorted, as long as the content can be aligned.

expect_data = [
  %{
    channelId: "1",
    description: "dev-test_001",
    tags: %{pipelineId: "1001", pipelineTemplate: "live-drm"},
    name: "testName1"
  },
  %{
    channelId: "1",
    description: "dev-test_001",
    tags: %{pipelineId: "1001", pipelineTemplate: "live-drm"},
    name: "testName2"
  },
  %{
    channelId: "1",
    description: "dev-test_001",
    tags: %{pipelineId: "1001", pipelineTemplate: "live-drm"},
    name: "testName3"
  },
  %{
    channelId: "2",
    description: "dev-test_002",
    tags: %{pipelineId: "1002", pipelineTemplate: "live-drm"},
    name: "testName1"
  },
  %{
    channelId: "2",
    description: "dev-test_002",
    tags: %{pipelineId: "1002", pipelineTemplate: "live-drm"},
    name: "testName2"
  },
  %{
    channelId: "3",
    describtion: "dev-test_003",
    tags: %{pipelineId: "1003", pipelineTemplate: "live-drm"},
    name: "testName2"
  },
  %{
    channelId: "4",
    describtion: "dev-test_004",
    tags: %{pipelineId: "1004", pipelineTemplate: "live-drm"},
    name: ""
  }
]

Thank you for reading this message.

Eiji

Again, you had issues in your data. Firstly there is descri**b**tion instead of descri**p**tion. Secondly 2nd last expected map should have name testName1 instead of testName2.

Here goes the complete example with fixed data:

defmodule Example do
  def sample(data, names) when is_list(data) and is_list(names) do
    # a list of names
    # reduce it with an empty list
    flat_names =
      Enum.reduce(names, [], fn %{mediaTailoId: ids_map, name: name}, acc ->
        # take a map value under mapData key
        ids_map
        # change atom id to string
        |> Enum.map(fn {id, _value} -> {Atom.to_string(id), name} end)
        # concat the result with acc
        |> then(&(acc ++ &1))
      end)
      # the original names are sorted partially i.e. only in specified mediaTailoId
      # to fix that we simply need to sort the flat list result by id
      |> Enum.sort_by(&elem(&1, 0))

    # in order to recursively pattern match key by key we need to change a list of maps
    # to a map with id as key and name as value
    data |> Enum.map(&{&1.channelId, &1}) |> merge(flat_names)
  end

  # function head is required when using a default argument(s)
  # in functions with multiple clasule
  defp merge(data, names, last_matched \\ false)

  # when done simply finish a recursive call returning an empty list as result tail
  defp merge([], [], _last_matched), do: []

  # if id matches add a new map to result and continue with same data and the rest of names
  defp merge([{id, map} | _data_tail] = data, [{id, name} | names], _last_matched) do
    [Map.merge(map, %{name: name}) | merge(data, names, true)]
  end

  # if id does not match and previous clause matched for this id
  # we simply continue with rest of data and same names
  defp merge([_data_head | data], names, true), do: merge(data, names, false)

  # otherwise if id still does not match and previous clause do not matched for this id even once
  # add a new map with an empty name and continue with rest of data and same names
  # the combination of last_matched boolean argument and this clause
  # is required in case we do not have any names for specific data like it is for "444" id 
  defp merge([{_id, map} | data], names, false) do
    [Map.merge(map, %{name: ""}) | merge(data, names, false)]
  end
end

data1 = [
  %{
    channelId: "1",
    # fix: describtion -> description
    description: "dev-test_001",
    tags: %{pipelineId: "1001", pipelineTemplate: "live-drm"}
  },
  %{
    channelId: "2",
    # fix: describtion -> description
    description: "dev-test_002",
    tags: %{pipelineId: "1002", pipelineTemplate: "live-drm"}
  },
  %{
    channelId: "3",
    # fix: describtion -> description
    description: "dev-test_003",
    tags: %{pipelineId: "1003", pipelineTemplate: "live-drm"}
  },
  %{
    channelId: "4",
    # fix: describtion -> description
    description: "dev-test_004",
    tags: %{pipelineId: "1004", pipelineTemplate: "live-drm"}
  }
]

data2 = [
  %{
    mediaTailoId: %{"1": "channel1_data", "2": "channel2_dadta", "3": "channel3_dadta"},
    mediaTailorRegion: "us-east-1",
    name: "testName1"
  },
  %{
    mediaTailoId: %{"1": "channel1_data", "2": "channel2_dadta"},
    mediaTailorRegion: "us-east-2",
    name: "testName2"
  },
  %{mediaTailoId: %{"1": "channel1_data"}, mediaTailorRegion: "us-east-3", name: "testName3"}
]

expect_data = [
  %{
    channelId: "1",
    description: "dev-test_001",
    tags: %{pipelineId: "1001", pipelineTemplate: "live-drm"},
    name: "testName1"
  },
  %{
    channelId: "1",
    description: "dev-test_001",
    tags: %{pipelineId: "1001", pipelineTemplate: "live-drm"},
    name: "testName2"
  },
  %{
    channelId: "1",
    description: "dev-test_001",
    tags: %{pipelineId: "1001", pipelineTemplate: "live-drm"},
    name: "testName3"
  },
  %{
    channelId: "2",
    description: "dev-test_002",
    tags: %{pipelineId: "1002", pipelineTemplate: "live-drm"},
    name: "testName1"
  },
  %{
    channelId: "2",
    description: "dev-test_002",
    tags: %{pipelineId: "1002", pipelineTemplate: "live-drm"},
    name: "testName2"
  },
  %{
    channelId: "3",
    description: "dev-test_003",
    tags: %{pipelineId: "1003", pipelineTemplate: "live-drm"},
    # fix: "testName2" -> "testName1" 
    name: "testName1"
  },
  %{
    channelId: "4",
    description: "dev-test_004",
    tags: %{pipelineId: "1004", pipelineTemplate: "live-drm"},
    name: ""
  }
]

iex> Example.sample(data1, data2) == expect_data
true

@AstonJ Could you please move our messages to a separate topic, please?

Note: Fixed code formatting.

Where Next?

Popular in Questions Top

aalberti333
As the title describes, I’m trying to run Enum.map() over a list of key/value pairs, where the value is a map. My data looks like this: ...
New
pgiesin
This should be a simple problem but I just can’t seem to figure it out. I have a standalone Elixir app that won’t find the database. Dep...
New
hariharasudhan94
I would like to know what is the best IDE for elixir development?
New
lastday4you
I wanted to check elixir version in phoenix because i found that my elixir is 1.5 but when i use Enum.chunk_by it said the function is un...
New
Phillipp
Hey, I have a NanoPi-M3 and try to install Elixir on their Ubuntu image. I followed the Raspberry Pi installation instructions from the ...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
sabri
Can someone explain the settings of pool_size of Ecto in config file? and what is the recommend size? Thanks
New
siddhant3030
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
New
idi527
I’ve been re-reading swift book again and noticed that multiline strings there don’t have a trailing line break, unlike in elixir iex(2)...
New

Other popular topics Top

dotdotdotPaul
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
New
script
If I have a string “1000 cfu/ml” . I want to remove the characters and / and space . So the string is like this "1000" What is the ...
New
myronmarston
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lis...
New
chensan
I have a User schema with a :from_id field set to type :string: defmodule TweetBot.Repo.Migrations.CreateUsers do use Ecto.Migration ...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
skosch
To my knowledge, put_in, Map.update etc. all have the one limitation of not automatically creating intermediate keys when needed (for exa...
New
romenigld
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
New
aesmail
Hello guys, I have finally made it. I created an admin interface for a framework. It’s been on my todo list for years and with the curre...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

We're in Beta

About us Mission Statement