Questions & Help>Questions

Why this Crawly app doesn't write to disk parsed items nor logs?

#crawly
protoboolean

protoboolean

Why this Crawly app doesn't write to disk parsed items nor logs?

I’ve implemented my first web-scraper with Crawly following the tutorial published on the official docs.

I’m at the point where I start a single crawler module from iex and it seems it’s successfully crawling but it’s not writing to disk.

I’ve checked it’s crawling the website and extracting some items using IO.inspect.
I’ve limited the total number of items to make sure it completed gracefully a run.
The Crawly.Pipelines.WriteToFile module is on the pipeline and it’s configured to write to the user-writable folder /tmp/crawly, but at the end of the run it’s as empty as ever.
I’ve enabled logging pointing in the same folder but that’s not written either.

I’ve never debugged an Elixir/Erlang app and I still find the idea a bit daunting.

Anyone has a clue why it’s not writing to disk?

https://git.sr.ht/~protoboolean/unicrawl

1 comment
#crawly
1 526 1

Marked As Solved

kip

kip

ex_cldr Core Team

I haven’t use crawly so have no direct experience. But I do notice your configuration file is called config.ex when it should be config.exs which may be part of the issue.

1
Post #2

Where Next?

View thread on forum (has 1 response!)
crawly
Home Questions & Help>Questions
#crawly
1 526 1
Last post

Popular in Questions Top

JDanielMartinez
Questions & Help>Questions

** (RuntimeError) could not lookup Ecto repo Repo because it was not started or it does not exist
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
#umbrella#testing#repo#runtimeerror
11 15040 14
New
dotdotdotPaul
Questions & Help>Questions

Ecto: Validating belongs_to association is not nil?
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
#ecto
29 17806 25
New
bsollish-terakeet
Questions & Help>Questions

Checking if an enum is empty - Credo vs Compiler
Credo is smart enough to check for (something like) this: assert length(the_list) == 0 with this response: Checking if an enum is empt...
#compiler#credo
8 22889 6
New
albydarned
Questions & Help>Questions

Erlang and Elixir on Apple Silicon/M1 Chip
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
#erlang#troubleshooting
121 24599 65
New
itssasanka
Questions & Help>Questions

Why use `utc_datetime` over `naive_datetime` for Ecto?
Hi all, Trying to get some more clarity over utc_datetime and naive_datetime for Ecto: https://hexdocs.pm/ecto/Ecto.Schema.html#module-...
#ecto#timestamps#datetime#timezones
115 14814 19
New
dokuzbir
Questions & Help>Questions

How to convert list to string without losing brackets
Hello, I am trying to convert my lists to string without losing brackets.For start i have 3 map. They look like these buyer = %{ id: ...
26 14426 10
New
shahryarjb
Questions & Help>Questions

How to convert map to string (separated with ,)
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
#maps#strings
15 15491 2
New
baxterw3b
Questions & Help>Questions

Anonymous functions with multiple body
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
19 21396 4
New
lucidguppy
Questions & Help>Questions

How do I read a file and enumerate its lines
I have a super simple question about elixir - how would I take a file like this foo bar baz and output a new file that enumerates th...
#how-to-question
20 15454 3
New
siddhant3030
Questions & Help>Questions

How can I write a raw sql query?
Hi, I have to write a raw query for one of my project. But till now I have used ecto queries and don’t have much experience writing raw ...
/phoenix#ecto
13 19061 20
New

Other popular topics Top

freewebwithme
Questions & Help>Questions

Failed to run 'elixir' command error in vs code
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
#vscode#elixir-ls
48 15871 39
New
jononomo
Questions & Help>Questions

Where / How does the Mix environment variable get set?
I am trying to figure out how Mix knows whether the environment is test, dev, or prod -- where is this set? Thanks.
#mix
31 22791 9
New
grych
Your Libraries & Projects>Libraries

Drab: remote controlled frontend framework for Phoenix
Hi folks, Few months ago I have announced the proof-of-concept of the library to manipulate the browsers DOM objects directly from Elixi...
/phoenix#web-frameworks#frontend#javascript#drab
639 49522 488
New
romenigld
Questions & Help>Questions

Docker run Error loading shared library libstdc++.so.6 and libgcc_s.so.1
I am trying to run a deploy with docker and I successfully runned with this command: docker build -t romenigld/blog-prod . but when I t...
/phoenix#deployment#docker-compose
30 19932 15
New
quazar
Questions & Help>Questions

How to encode ecto schema with Jason with all fields
How to set Jason to encode all fields in ecto schema, I don’t care about security and implementing only is taking long list of attributes...
/phoenix#json#jason
17 13698 9
New
nsuchy
Questions & Help>Questions

IEX in Windows Powershell?
Hi. I’ve noticed that Windows Powershell has it’s own IEX command and you cannot access Elixir’s IEX due to the conflict. This isn’t a cr...
#iex#microsoft-windows#windows#powershell
15 29309 4
New
vac
Questions & Help>Questions

Formatting a string to PEM with headers and line breaks
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
2 14555 7
New
mcarvalho
Questions & Help>Questions

System.get_env vs. Application.get_env
What is the difference between System.get_env and Application.get_env? For example, what are best practices to use one versus another.
#environment
13 17021 6
New
fireproofsocks
Questions & Help>Questions

Ecto delete a record WITHOUT selecting first
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
#ecto#optimization#delete-method
22 16810 8
New
baxterw3b
Questions & Help>Questions

Anonymous functions with multiple body
Hi guys, i’m new in the Elixir world, and i have to say, that i love it! i’m having some problem to understand anonymous functions with ...
19 21396 4
New
Back to top

Questions & Help>Questions

Questions Questions ❯

Latest on Elixir Forum

Elixir Forum ❯

Sponsor Spotlight

Dashbit

Build Elixir applications with speed and confidence.

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Manning

Develop your skills with books, videos, and courses.

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

Categories:

Sub Categories:

Forums

Our Sponsors

Dashbit

Build Elixir applications with speed and confidence.

EEF

Supporting innovation across the BEAM ecosystem.

Coders51

We build reliable cloud platforms for business-critical systems.

AppSignal

Catch errors, track performance, monitor hosts and more.

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Manning

Develop your skills with books, videos, and courses.

Potions

The deployment platform built for Elixir: PaaS ease, VPS control.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

We're in Beta

About us Mission Statement