The tricky part is dealing with cards that could be in either a set or a run - for instance if you have 5/6/7/8 of diamonds, 5 of clubs and 5 of hearts.

I’d implement this check by first rewriting it into a recursive definition:

* A hand is a gin if:
  * there is a set of 4 cards, and one of the following:
    * the remaining cards are a gin
    * the remaining cards plus one of the 4 from the set are a gin
  * there is a set of 3 cards, and one of the following:
    * the remaining cards are a gin
    * the whole hand is a gin ignoring this set of 3 cards
  * there is a run of at least 3 cards, and:
    * the remaining cards are a gin
  * the hand is empty

Some of these clauses are trickier than others:

• the case where a candidate set of four cards isn’t used as a set is ignored, since having four runs would require 12 cards!
• the “not a set” branch for 3 cards needs to track which values have been tried as a set, or else the definition will loop forever

There’s a corresponding clause in GinCalc.gin? for all the bullet points above:

defmodule GinCalc do
  def gin?(hand, ignored \\ [])
  def gin?([], _), do: true
  def gin?(hand, ignored) do
    case split_set(hand, ignored) do
      {run, rest} when length(run) == 4 ->
        if gin?(rest, ignored) do
          true
        else
          Enum.any?(run, &gin?([&1 | rest], ignored))
        end
      {[{_, v} | _] = run, rest} when length(run) == 3 ->
        if gin?(rest, ignored) do
          true
        else
          gin?(hand, [v | ignored])
        end
      {nil, rest} ->
        case split_run(rest) do
          {nil, _} -> false
          {_, without_run} -> gin?(without_run, ignored)
        end
    end
  end

  def split_set(hand, ignored) do
    grouped = Enum.group_by(hand, &elem(&1, 1))

    set_key =
      grouped
      |> Map.keys()
      |> Enum.reject(& &1 in ignored)
      |> Enum.find(fn k -> length(grouped[k]) > 2 end)

    if set_key do
      set = grouped[set_key]

      {set, hand -- set}
    else
      {nil, hand}
    end
  end

  def split_run(hand) do
    sorted = Enum.sort(hand)

    candidate_run =
      Enum.reduce_while(sorted, [], fn
        {suit, value}, [{suit, prev_value} | _] = acc when value == prev_value + 1 ->
          {:cont, [{suit, value} | acc]}
        _, acc when length(acc) > 2 ->
          {:halt, acc}
        {suit, value}, _ ->
          {:cont, [{suit, value}]}
      end)

    if length(candidate_run) > 2 do
      {candidate_run, hand -- candidate_run}
    else
      {nil, hand}
    end
  end
end

GinCalc.split_set([{:h, 5}, {:h, 6}, {:h, 7}, {:s, 5}, {:d, 5}], []) |> IO.inspect()
GinCalc.gin?([{:h, 5}, {:s, 5}, {:d, 5}, {:c, 5}, {:h, 6}, {:h, 7}]) |> IO.inspect()

I used tuples to represent cards because they Enum.sort in a reasonable way (grouped into suits, then by number) and are easy to pattern-match, but using structs would only require light modifications.

What have you tried?

You might find this approach to ranking poker hands helpful. Playing Poker with Elixir (pt. 1)