benonymus

benonymus

Delete file from aws(arc) fom postgres trigger

Hey there,

I am using arc for file processing with aws, and I am trying to delete images from aws once an item is deleted
from my db.

I am using postgres triggers(https://blog.lelonek.me/listen-and-notify-postgresql-commands-in-elixir-187c49597851), so I am getting the item form the database, arc for the files only saves a string to the database such as:

test_document_fEfRY-pR2JX3jIF9CrXPC.pdf?63754075895

My problem is, since my trigger is upon deletion I cannot get the item by id from Repo to have the file taken care of by arc.
The attachment inside my item instead of a map is just a string.
I am usually using arc’s delete from the avatar with the image and entity but without having the item going through ecto it just won’t delete by that string that I mentioned above.

Any thoughts how could I get the attachment deleted with the string or how to load that attachment manually in order to be able to delete it?

Most Liked

benwilson512

benwilson512

Author of Craft GraphQL APIs in Elixir with Absinthe

I would probably invert this whole process. Don’t delete the record from the database first. Mark the database record as deleted_at pending_deletion or something, and then enqueue a job in something like Oban which can try to hit AWS to delete the record. Only after that has succeeded do you delete the record in the DB. The flag lets you filter it out from UI lists immediately, but you want the record around to reason about the file until it’s actually gone.

The job part here is critical. AWS S3 has good uptime, but not perfect up time. Requests to delete files will fail periodically and will need to be retried.

baldwindavid

baldwindavid

If you like Arc, you might take a look at Waffle (I've forked Arc to Waffle and merged all open PRs). Arc seems to have gone unmaintained and Waffle is a fork with fixes. I prefer the transparency and control of working directly with ex_aws_s3 and orchestrating uploads and deletions via Ecto.Multi, but understand the allure of an opinionated package.

dimitarvp

dimitarvp

Hm, I see. In this case I recommend you switch approaches. You’re better off putting a “please delete attachmend with ID 123” command on a message queue (RabbitMQ, Kafka, you can also use your DB as well) and then have a background worker that periodically works through those and just have that delete the files from AWS.

Where Next?

Popular in Questions Top

JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
bsollish-terakeet
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
ycv005
I have followed this StackOverflow post to install the specific version of Erlang. And When I am running mix ecto.setup then getting fol...
New
New
Werner
Hi, I’m using Ubuntu 18.04 and after updating to OTP-24.0 yesterday i have this warning when I run “mix local.hex”: 14:57:30.512 [warn] ...
New
dokuzbir
Hello, I am trying to convert my lists to string without losing brackets.For start i have 3 map. They look like these buyer = %{ id: ...
New
mcarvalho
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
New
Phillipp
Hey, I have a NanoPi-M3 and try to install Elixir on their Ubuntu image. I followed the Raspberry Pi installation instructions from the ...
New

Other popular topics Top

Tee
can someone please explain to me how Enum.reduce works with maps
New
openscript
Hello! Sorry for this astonishing simple question, but I’m really stuck. I try to set up the intellij-elixir plugin, but I don’t know ho...
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
chrismccord
As promised, the first release candidate of Phoenix 1.3.0 is out! This release focuses on code generators with improved project structure...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
myronmarston
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lis...
New
minhajuddin
I have seen a lot of code which picks the first element from a list using Enum.at(0) instead of List.first. Is there a reason why people ...
New
electic
Hi, I am new to Elixir. I am trying to use the DateTime component to insert a date into MySQL however the there seems to be no way to fo...
New
josevalim
Hi everyone, One of the features added to Elixir early on to help integration with Erlang code was the idea of overridable function defi...
New
lanycrost
Hi everyone! I need implement if…else if…else condition from my elixir code, and anymore of this control flow structures not work proper...
New

We're in Beta

About us Mission Statement