Jaypee

Jaypee

Secure files transfers servers in Elixir?

Dear Elixir Community,

First, I hope I choosed the right thread to post…

After Clojure last year, I’m currently studiying Elixir.
I’ve read several books, seen tons of videos and also struggled with some exercices on Exercism/Elixir ( a bit less “beginners friendly” than its Clojure counterpart “4Clojure” !).

It happens that the advantages of Elixir/OTP could benefit to the project I’m working on at the moment, so I would like to do some tests/benchmarks on some basics componants of this project:

I want to implement small servers in Elixir, that could be located on several computers/VM which simply send and receive files and could also verify their integrity (MD5 checksum ?)

The benchmark will mainly focus on scalability (multiply the servers and test a lot of simultaneaous file transfers) and robustness (test the transfer of huge files between two servers).

At the moment, I must confess that I’m a bit confused with the Elixir/OTP Genservers, Supervisors, Workers and others and how to correctly articulate them.
So I would like to have some advices/suggestions/hints on the correct way to use and organise these entities in order to implement the most efficient file transfer server in Elixir, regarding the goals of the benchmark.

I’m also open to suggestions concerning tips and tricks I could use to optimize my FT servers.

To my knowledge, Elixir is not used in my (big) company and I’d really like to succeed in providing a “proof of concept” ! :wink:

Best regards and thanks in advance for your help.

P.S.: Please forgive my English as long as you roughly understand what I write !

Marked As Solved

Jaypee

Jaypee

I redevelopped an entire new application (“Simple SFTP Client Server in Elixir”), based on things I learned this year:

Here’s the SscsEx GitHub page

NB: You can only use this repository as an inspiration for your own code as SscsEx will NOT be updated: I’m now on my way to the next Elixir step: learning Phoenix + LiveView ! :wink:

Also Liked

joeerl

joeerl

Creator of Erlang - Fondly Remembered

Well you have sockets, and something to compute an MD5 checksum - which is actually all you need.

Forget all the generic this and that and make a simple socket client and server (homework - find out the smallest socket client and server code)

Making a client/server over a socket is virtually the first thing I do in any language when learning - very instructive

Repeat in C, TCL, JS, Java, Ruby, Python, Perl, C#

Seriously, the round trip client -> server -> client with a check that you transfer all data and loose nothing is an essential programming technique - learn to do this first THEN add JSON/XML (whatever) on top.

Once you can get two programs in different languages communicating raw bytes over a socket then you can start having fun - Use the low level socket libraries and NOT fancy frameworks - this is the slow way to program - BUT in the long term the best way - understanding is the key to programming libraries which hide what you are really doing should only be used once you understand what’s really happening

Cheers

joeerl

joeerl

Creator of Erlang - Fondly Remembered

Well a while back I wanted to transfer some files so …

:slight_smile:

AstonJ

AstonJ

What is your experience on which you base such comments?

As shown in the benchmarks by @brightball any performance hit by Phoenix for the majority of apps would generally be considered to be negligible.

easco

easco

If your goal is file transfers only, then it seems like Erlang should contain mechanisms to handle that already. I believe Erlang implements an sftp server (http://erlang.org/doc/man/ssh_sftp.html). I also see this article:

https://ninenines.eu/articles/ranch-ftp/

Which talks about building an ftp server on top of Ranch.

Accessing those things from Elixir to help with distributed coordination shouldn’t be too hard.

OvermindDL1

OvermindDL1

At that scale on a single server then you are looking at pure socket performance, so C/C++ or Rust with specific libraries, or Nim (I’m still heavily impressed at its significantly loaded socket performance). I’d not recommend Go or OCaml, neither perform as well as C/C++/Rust/Nim when finely tuned (though they do well when not everything is finely tuned). You will be looking at memory pools, no-alloc structures, etc… etc… Honestly just adding more servers as better as redundency is always better for such things, at which point Elixir is just fine.

Still, hundreds of thousands per second is crazy high, I’m exceptionally curious what this is. ^.^

Where Next?

Popular in Questions Top

bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
JorisKok
I have a server on AWS, and was running a load test using artillery. When looking at the Phoenix dashboard I see the Ports going to 100% ...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
fireproofsocks
I’m working on defining a simple Ecto schema for a table (in PostGres), but I don’t see where I can define a column as NOT NULL. Conside...
New
polypush135
As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a ...
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
baxterw3b
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
New
Patoshizzle
After calling mix ecto.create I get this error: 17:00:32.162 [error] GenServer #PID<0.412.0> terminating ** (Postgrex.Error) FATAL...
New
Exadra37
Sometimes I want to check if the input into a function is not a blank string. My first approach: defmodule Example do def do_stuff(s...
New

Other popular topics Top

sergio
I couldn’t find any guides that worked well with Phoenix 1.6.0 and esbuild. I hope this helps people test the waters and eases you into t...
New
axelson
This post is a wiki (feel free to hit the edit button near the bottom right of this post to add your own changes!) This post collects co...
239 45766 226
New
KronicDeth
Elixir plugin for JetBrain’s IntelliJ Platform (including Rubymine) This is a plugin that adds support for Elixir to JetBrains IntelliJ...
289 35421 110
New
lk-geimfari
What is most correct way to open, read and parse JSON file with poison? For example if we have example.json file in root of some projec...
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
msaraiva
Surface is an experimental library built on top of Phoenix LiveView and its new LiveComponent API that aims to provide a more declarative...
564 42633 214
New
chrisalley
ExUnit now has describe blocks which is a welcome addition coming from RSpec. In the docs, it states that nested hierarchies of describe ...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
lucidguppy
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
New

We're in Beta

About us Mission Statement