adamwight

adamwight

MediaWiki Client - read and write Wikipedia, follow recent edits, access machine learning analytics

Seeking early adopters and reviewers of the mediawiki_client library (docs), which intends to wrap all of the major access methods for Wikipedia content.

The supported APIs are functionally complete and the library includes tests. My current goal is to get feedback on the exposed interfaces, and stabilize on an idiomatic Elixir style to begin the 1.x releases.

Examples

Watch all Wikimedia sites and print a link to the next 6 articles as they are created:

Wiki.EventStreams.start_link(streams: "page-create")
Wiki.EventStreams.stream()
|> Stream.take(6)
|> Enum.each(fn event -> IO.puts(event["meta"]["uri"]) end)
# https://www.wikidata.org/wiki/Q111963009
# https://en.wikipedia.org/wiki/Rhinella_xerophylla
# https://fr.wiktionary.org/wiki/stachst_aus
# https://commons.wikimedia.org/wiki/File:St_Swithin,_Quenington_-_Wall_monument_-_geograph.org.uk_-_3514076.jpg
# https://commons.wikimedia.org/wiki/File:Nebria_brevicollis_(Carabidae)_-_(imago),_Henshuisterveld,_the_Netherlands.jpg
# https://commons.wikimedia.org/wiki/Category:Protected_areas_of_Russia/5630230

Get current site statistics for German Wikipedia:

|> Wiki.Site.get!("dewiki")
|> Wiki.Action.new()
|> Wiki.Action.get!(
  action: :query,
  meta: :siteinfo,
  siprop: :statistics
)
# %Wiki.Action.Session{
#   ...
#   result: %{
#     "batchcomplete" => true,
#     "query" => %{
#       "statistics" => %{
#         "activeusers" => 19687,
#         "admins" => 188,
#         "articles" => 2583285,
#         "edits" => 211219883,
#         "images" => 130199,
#         "jobs" => 0,
#         "pages" => 7163473,
#         "queued-massmessages" => 0,
#         "users" => 3715461
#       }
#     }
#   },
#   ...
# }

Fetch all information about Douglas Adams contained on Wikidata:
```elixir
Wiki.Site.get!("wikidatawiki")
|> Wiki.Action.new()
|> Wiki.Action.get!(
    action: :wbgetentities,
    ids: "Q42"
)
# result: %{
# 	"entities" => %{
# 		"Q42" => %{
# 			"aliases" => %{
# 				"uk" => [
# 					%{"language" => "uk", "value" => "Дуглас Ноел Адамс"},
# 					%{"language" => "uk", "value" => "Адамс Дуглас"}
# 				],
# 				"pt" => [
# 					%{"language" => "pt", "value" => "Douglas Noël Adams"},
# 					%{"language" => "pt", "value" => "Douglas Noel Adams"}
# 				],

Request ORES scoring for an edit to predict whether it was spam or not:

Wiki.Ores.new("enwiki")
|> Wiki.Ores.request!(
  models: ["damaging"],
  revids: 456789
)
# %{
#   "enwiki" => %{
#     "models" => %{
#       "damaging" => %{"version" => "0.5.1"}
#     },
#     "scores" => %{
#       "456789" => %{
#         "damaging" => %{
#           "score" => %{
#             "prediction" => false,
#             "probability" => %{
#               "false" => 0.9784615344695441,
#               "true" => 0.021538465530455946
#             }
#           }
#         }
#       }
#     }
#   }
# }
"""

Most Liked

adamwight

adamwight

I’ve spun out a streaming bzip2 codec as part of the work to support Elixir processing of Wikipedia dumps.

Example usage:

Mix.install([:bzip2])

:inets.start()
:ssl.start()
tmpfile = '/tmp/articles.xml.bz2'
url = 'https://dumps.wikimedia.org/testwiki/20220501/testwiki-20220501-pages-articles.xml.bz2'

{:ok, :saved_to_file} = :httpc.request(:get, {url, []}, [], [stream: tmpfile])

File.stream!(to_string(tmpfile), [], 900 * 1024)
|> Bzip2.decompress!()
|> Enum.into("")
#  ...
#  <page>
#    <title>Wikipedia:Help</title>
#    <ns>4</ns>
#    <id>80</id>
#    <redirect title="Main Page" />
#    <revision>
#      <id>438551</id>
#      <parentid>438171</parentid>
#      <timestamp>2020-07-01T07:11:36Z</timestamp>
#      <contributor>
#        <username>JohanahoJ</username>
#        <id>37147</id>
#      </contributor>
#      <comment>Changed redirect target from [[Wikipedia:Main Page]] to [[Main Page]]</comment>
#      <model>wikitext</model>
#      <format>text/x-wiki</format>
#  ...

I would have liked to give a fully streamed example, but :httpc requires more than a dab of glue to do so, and I can’t find any other currently working library to stream HTTP.

adamwight

adamwight

Now with Livebook examples.

Also, at day job my team is writing a complementary library to parse wiki article HTML dumps, which works already but will be finalized in a few weeks, then we run it on all available dumps.

We’re looking at article footnotes for our particular use case, but it can be generalized to run whatever other parsing modules.

Where Next?

Popular in Libraries Top

woutdp
Hi! I wanted to introduce my latest project LiveSvelte. It allows you to render Svelte inside LiveView with end-to-end reactivity. It’s ...
New
benlime
LiveMotion enables high performance animations declared on the server and run on the client. As a follow up to my previous thread A libr...
New
Qqwy
Hello everyone, I wrote a small library today called MapDiff. It returns a map listing the (smallest amount of) changes to get from map...
New
zorbash
I created Kitto a framework for dashboards inspired by Dashing. [demo] The distributed characteristics of Elixir and the low memory foo...
New
ahamez
Hi everyone, I’ve been working on this protobuf library for 3 years. We use it in the company I work for, EasyMile, to communicate with ...
New
wmnnd
Hi there, for my project DBLSQD, I needed a file storage solution that is a bit more flexible than Arc. Because I thought others might f...
New
engineeringdept
I’ve just released the first version of Snap, an Elasticsearch client. It borrows ideas about application structure and process managemen...
New
bryanjos
Hi, I wanted share a small library we at Revelry Labs made for rendering react components from the server side. There are instructions fo...
New
mischov
import Meeseeks.CSS html = HTTPoison.get!("https://news.ycombinator.com/").body for story &lt;- Meeseeks.all(html, css("tr.athing")) do...
New
wojtekmach
Hey everyone! Req is an HTTP client for Elixir that I’ve been working on for quite some time. There is already a lot of HTTP clients out...
New

Other popular topics Top

chrismccord
Phoenix 1.4.0 released Phoenix 1.4 is out! This release ships with exciting new features, most notably with HTTP2 support, improved deve...
688 30048 115
New
yurko
Here are few pieces of (common) Linux knowledge that we use for reasonably small one server apps. We use Ubuntu but this should work for ...
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
stefanchrobot
What’s the safe way to decode a JSON string into a struct? I want to avoid calling String.to_atom. Jason.decode can give me a map with st...
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
freewebwithme
Using vs code and installed ElixirLS: support and debugger. And I got an error popped up on start up says Failed to run ‘elixir’ comma...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
shahryarjb
Hello, I have map which I want to convert it to string like this: the map: %{last_name: "tavakkoli", name: "shahryar"} the string I ne...
New
vonH
When I run the Plug and I recompile I wind up having to use Ctrl C to quit iex and start again. Witht the help of rlwrap I can use the cu...
New
9mm
I am constructing a JSON object (map) and I need to conditionally set a field. I’m trying to write proper elixir-way code… and I’m at a l...
New

Sub Categories:

We're in Beta

About us Mission Statement