janp
How to connect nodes in Kubernetes before application startup?
I am wondering whether it is possible to use distributed Erlang plus libcluster in kubernetes? 
I tried it with the following setup:
StatefulSet (with 3 replicas)
example@example-0.example-headless.default.svc.cluster.localexample@example-1.example-headless.default.svc.cluster.localexample@example-2.example-headless.default.svc.cluster.local
Cluster.Strategy.Kubernetes
kubernetes_ip_lookup_mode: :podsmode: :hostnamekubernetes_namespace: defaultkubernetes_selector: app=appkubernetes_service_name: example-headless
config/runtime.exs
config :kernel,
sync_nodes_optional: [
:"example@example@example-0.app-headless.default.svc.cluster.local",
:"example@example@example-1.app-headless.default.svc.cluster.local",
:"example@example@example-2.app-headless.default.svc.cluster.local"
],
sync_nodes_timeout: 5000
mix.exs
def releases do
[
example: [
reboot_system_after_config: true,
# ...
]
]
end
My problem with this setup is, that the nodes cannot connect to each other before starting the application, because their DNS records will only be available when the pods are Ready, which requires the readinessProbe (GET /health/readyz) to be successful, but the readyinessProbe already requires the application to be started
So a chicken and egg problem, but maybe I am just doing something wrong 
Currently, when all pods are started after exceeding the sync_nodes_timeout, the application is running in a cluster, but I would like to have the nodes being connected with each other before the application gets started.
Marked As Solved
janp
@mudasobwa Thanks for your answer, but I think I found the issue with my setup 
TLDR;
- changed from
Cluster.Strategy.KubernetestoCluster.Strategy.Kubernetes.DNSSRV - set
publishNotReadyAddresses: truefor the headless service - set
podManagementPolicy: Parallelfor theStatefulSet
$ kubectl explain service.spec.publishNotReadyAddresses
KIND: Service
VERSION: v1
FIELD: publishNotReadyAddresses <boolean>
DESCRIPTION:
publishNotReadyAddresses indicates that any agent which deals with
endpoints for this Service should disregard any indications of
ready/not-ready. The primary use case for setting this field is for a
StatefulSet's Headless Service to propagate SRV DNS records for its Pods
for the purpose of peer discovery. The Kubernetes controllers that generate
Endpoints and EndpointSlice resources for Services interpret this to mean
that all endpoints are considered "ready" even if the Pods themselves are
not. Agents which consume only Kubernetes generated endpoints through the
Endpoints or EndpointSlice resources can safely assume this behavior.
Setting publishNotReadyAddresses: true allows the pods to find each other via the headless-service before they are ready.
As mentioned in the explaination for the publishNotReadyAddresses option, I also switched from Cluster.Strategy.Kubernetes to Cluster.Strategy.Kubernetes.DNSSRV.
Additionally I set podManagementPolicy to Parallel which allows the StatefulSet to schedule all pods in parallel. The default is OrderedReady which requires one pod to be ready before Kubernetes schedules the next pod, so they are only started one after the other.
Now my application is clustered and started within 15 seconds 
Popular in Questions
Other popular topics
Latest on Elixir Forum
Sponsor Spotlight
Enabling companies to succeed by building software people love.
Categories:
Sub Categories:
Forums
Our Sponsors
Build Elixir applications with speed and confidence.
Supporting innovation across the BEAM ecosystem.
We build reliable cloud platforms for business-critical systems.
Catch errors, track performance, monitor hosts and more.
Practical resources that improve the lives of professional developers.
Develop your skills with books, videos, and courses.
Enabling companies to succeed by building software people love.
Courses that'll move you from confusion to "Aha, now I get it!"