Questions & Help>Questions

Process a large gRPC client response as it is being streamed

#grpc#stream#bitcoin#lightning-network
RooSoft

RooSoft

Process a large gRPC client response as it is being streamed

Hi!

Trying to efficiently parse a 111MB JSON gRPC response that is bound to exponentially grow over time for a while. Expecting it to hit gigabytes mid-long term… talking about the Lightning Network graph straight from a local node. Here’s what it looks like visually…

Currently using this lib: {:grpc, github: "elixir-grpc/grpc"}

As these lines are being written, it seems there’s no way to access that data until it’s been completely parsed and downloaded in RAM. Not sure about that solution’s scalability…

If possible at all, I’d decode the JSON as it is being streamed from the server. That would make this process even possible on the smallest computer such as a raspberry pi, which is super popular hardware used by node operators.

Thanks for any insight on how to achieve this!

Marc

3 comments
#grpc#stream#bitcoin#lightning-network
3 1150 3

Most Liked

dimitarvp

dimitarvp

Just today I found myself wondering the same. There’s Jaxon but I am not sure I can invest the time to rewrite certain code in the projects I work on. Replacing a simple Jason.decode with who knows how many lines that partially ingest a JSON stream can bring about metric tons of complexity.

Maybe a better question for you would be: is there a way to consume the API in a one-JSON-record-per-line fashion?

1
Post #2
dimitarvp

dimitarvp

Shame the service doesn’t offer alternatives. :frowning:

Forking the library is always an option though. If you get desperate, go for that. In my case, I have to parse no more than 20-30 MB for the moment and the API server is slow – sometimes takes 15-20 seconds to deliver data, but our app isn’t real time for 99% of the data so it’s okay.

Hope you can find a solution.

1
Post #4

Where Next?

View thread on forum (has 3 responses!)
grpc
stream
bitcoin
lightning-network
Home Questions & Help>Questions
#grpc#stream#bitcoin#lightning-network
3 1152 3
Last post

Popular in Questions Top

pmjoe
Questions & Help>Questions

What do you think of Gleam compared to Elixir?
I have a relationship of love and hate with Elixir. Lots of things are just absolutely right, but there are some things that are kind of ...
#programminguages#gleam
24 16555 10
New
gshaw
Questions & Help>Questions

Idiomatic guard clause for checking not nil
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
#guard-clauses
13 43178 3
New
quazar
Questions & Help>Questions

How to encode ecto schema with Jason with all fields
How to set Jason to encode all fields in ecto schema, I don’t care about security and implementing only is taking long list of attributes...
/phoenix#json#jason
17 13698 9
New
hariharasudhan94
Questions & Help>Questions

What is the best IDE for elixir?
I would like to know what is the best IDE for elixir development?
#code-editors
33 49503 23
New
stefanchrobot
Questions & Help>Questions

How to decode a JSON into a struct safely?
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
#structs#json
29 20780 26
New
myronmarston
Questions & Help>Questions

Typespecs - best way to spec keyword lists?
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lis...
#dialyzer
52 14258 3
New
stefanluptak
Questions & Help>Questions

Using VSCode on multiple monitors
Hello everybody, usually, I use a 29" ultra-wide monitor for VSCode which can easily accomodate explorer (files panel) + file with code ...
#vscode#vscode-elixir
7 17615 6
New
johnnyicon
Questions & Help>Questions

How do I use the Postgres JSONB / Postgrex JSON extension?
Hi all, I've just started learning Elixir and Phoenix Framework, so please pardon my n00bness at this stage. I'm trying to use Postg...
#ecto
22 17588 9
New
romenigld
Questions & Help>Questions

Docker run Error loading shared library libstdc++.so.6 and libgcc_s.so.1
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
/phoenix#deployment#docker-compose
30 19932 15
New
siddhant3030
Questions & Help>Questions

How can I write a raw sql query?
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
/phoenix#ecto
13 19061 20
New

Other popular topics Top

dotdotdotPaul
Questions & Help>Questions

Ecto: Validating belongs_to association is not nil?
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
#ecto
29 17806 25
New
lessless
Questions & Help>Questions

How are you dealing with CSV files that uses CR line breaks?
I believe there are people here who are dealing with CSV files import on the daily basis, and since Excel is a really popular tool there ...
#how-to-question
10 18353 17
New
gshaw
Questions & Help>Questions

Idiomatic guard clause for checking not nil
What is the idiomatic way of matching for not nil in Elixir? E.g., First way: defp halt_if_not_signed_in(conn, signed_in_account) when...
#guard-clauses
13 43178 3
New
itssasanka
Questions & Help>Questions

Why use `utc_datetime` over `naive_datetime` for Ecto?
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: https://hexdocs.pm/ecto/Ecto.Schema.html#module-...
#ecto#timestamps#datetime#timezones
115 14814 19
New
grych
Your Libraries & Projects>Libraries

Drab: remote controlled frontend framework for Phoenix
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
/phoenix#web-frameworks#frontend#javascript#drab
639 49522 488
New
mcarvalho
Questions & Help>Questions

System.get_env vs. Application.get_env
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
#environment
13 17021 6
New
fireproofsocks
Questions & Help>Questions

Ecto delete a record WITHOUT selecting first
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
#ecto#optimization#delete-method
22 16810 8
New
Qqwy
Questions & Help>Questions

Updating structs: Map.put vs %Foo{oldfoo | new: value} vs put_in
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
#maps#structs#pipeline#access
115 28410 31
New
Nvim
Chat & Discussions>Discussions

Elixir vs Python
Elixir appears to be a superior language to Python. I don’t see any advantage of Python over Elixir. Are there any?
#discussion#python#elixir
82 17872 30
New
vrod
Questions & Help>Questions

Starship (cross-shell prompt) error - (starship::utils): Executing command "elixir" timed out
I am using the Starship cross-shell prompt – it seems pretty nice, but I get some errors: [WARN] - (starship::utils): Executing command ...
#starship
8 17167 3
New
Back to top

Questions & Help>Questions

Questions Questions ❯

Latest on Elixir Forum

Elixir Forum ❯

Sponsor Spotlight

Dashbit

Build Elixir applications with speed and confidence.

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Manning

Develop your skills with books, videos, and courses.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

WyewWorks

Enabling companies to succeed by building software people love.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

Categories:

Sub Categories:

Forums

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Manning

Develop your skills with books, videos, and courses.

ElixirCasts

Producing high quality Elixir screencasts since 2017.

WyewWorks

Enabling companies to succeed by building software people love.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

We're in Beta

About us Mission Statement