cgrimmett

cgrimmett

Creating a BEP3 compatible percent encoder (URI encoder)

I would like to announce torrents to my opentracker using an Elixir program. I have a test qbittorrent and opentracker running and I’ve identified how qbittorrent sends it’s announces to opentracker.

METHOD: GET
URL
http://localhost:8000/announce?info_hash=%ac%c3%b2%e43%d7%c7GZ%bbYA%b5h%1c%b7%a1%ea%26%e2&peer_id=-qB5020-r3FSX0qNU6Oo&port=4993&uploaded=0&downloaded=0&left=0&corrupt=0&key=F7F879A3&event=started&numwant=200&compact=1&no_peer_id=1&supportcrypto=1&redundant=0
HEADERS
Accept-Encoding:
gzip
Connection:
close
Host:
localhost:8000
User-Agent:
qBittorrent/5.0.2

I would like to build this same GET request into my own code. My confusion comes from the percent encoded info_hash. Neither URI.encode/2 nor URI.encode_www_form/1 encode the URI the same way that qbittorrent does it.

Let me show you what I’ve tried so far.


# We start with the `Info hash v1` of the torrent, as copied from qBittorrent.
info_hash_v1 = "acc3b2e433d7c7475abb5941b5681cb7a1ea26e2"

# Next we decode the hexadecimal representation into binary.
binary = Base.decode16!(info_hash_v1, case: :lower)

I’m not sure what comes next. Digging through Elixir URI source code, I can see the method of percent encoding that creates a binary in almost the right format.

The problem here is that the hex/1 function only outputs 16 possible values, A-F, 0-9. There is no lowercase! The way qbittorrent is doing their percent encoding, they have lowercase too-- a-f, A-F, 0-9.

I read qbittorrent source code to see how they’re percent encoding, but I couldn’t find the code that does that. I found some info_hash references though. I can barely read C++, but I think the url encoding might be abstracted away in a request library.

I also looked through transmission-qt code. That code is greek to me, but I was able to find their percent encoder implementation.

Seems similar to what I read in Elixir URI source code related to unreserved and unescaped characters. RFC 3986 - Uniform Resource Identifier (URI): Generic Syntax

I bounced some ideas off of ChatGPT, borrowed code from Elixir URI, and I got a bep3_encode/1 function put together.


  @doc """
  Encodes `string` as BEP3's weird URL encoded string.

  ## Example

      iex> bep3_encode("a88fda5954e89178c372716a6a78b8180ed4dad3")
      "%A8%8F%DAYT%E8%91x%C3rqjjx%B8%18%0E%D4%DA%D3"

  """
  @spec bep3_encode(binary) :: binary
  def bep3_encode(string) when is_binary(string) do
    string = Base.decode16!(string, case: :lower)
    URI.encode(string, &URI.char_unreserved?/1)
  end

Given an Info hash v1 input of acc3b2e433d7c7475abb5941b5681cb7a1ea26e2, the expected output should be as follows.

%ac%c3%b2%e43%d7%c7GZ%bbYA%b5h%1c%b7%a1%ea%26%e2

However, I haven’t figured out how I preserve the case on the various letters. I assume I need to have exactly the same case-sensitive output as qBittorrent because an ASCII A is not the same as a.

It seems like the way qbittorrent does it, and I apologize for not having the language to communicate this… hex values that can be displayed as ASCII are output as ASCII. Otherwise, the hex representation is displayed.

I wrote this out to make a visual comparison of expected input and output values, because this is the only way I could think of understanding what is happening during the encoding.

 a8   8f   da   59   54   e8   91   78   c3   72   71   6a   6a   78   b8   18   0e   d4   da   d3
%A8  %8F  %DA   Y    T   %E8  %91   x   %C3   r    q    j    j    x   %B8  %18  %0E  %D4  %DA  %D3

On the second line (expected output) See the ASCII Y? That matches up with Hex 54. The three values before that were all not able to be displayed as ASCII, so the hex value was used instead.

Later on, we can see lowercase x, r, q, j, j, x. From what I can tell, Elixir’s built-in URI.encode/1 can’t do this, because like I mentioned earlier, that can only output A-F,0-9. No lowercase!

By the way, https://www.asciitohex.com/ has been very helpful.

Anyway, I am very confused. I’m going to rest now and pick this up in the morning.

Marked As Solved

cgrimmett

cgrimmett

Eureka!

defmodule App.BittorrentUrlEncoder do
  @moduledoc """
  URL encoding for Bittorrent Info hash v1. Designed to be compatible with qBittorrent's percent encoding.
  """

  import Bitwise

  @doc """
  Encodes `string` as a Bittorrent-flavored percent-encoded string.

  ## Example

      iex> encode("a88fda5954e89178c372716a6a78b8180ed4dad3")
      "%a8%8f%daYT%e8%91x%c3rqjjx%b8%18%0e%d4%da%d3"

  """
  @spec encode(binary()) :: binary()
  def encode(hex_string) when is_binary(hex_string) do
    hex_string
    |> Base.decode16!(case: :lower) # Decode from hex to raw bytes
    |> encode_bytes()
  end

  defp encode_bytes(<<>>), do: ""

  defp encode_bytes(<<byte, rest::binary>>) do
    percent_encode(byte) <> encode_bytes(rest)
  end

  defp percent_encode(byte) when byte in ?0..?9 or byte in ?a..?z or byte in ?A..?Z or byte in ~c"~_-.!" do
    <<byte>>
  end

  defp percent_encode(byte) do
    "%" <> <<hex(bsr(byte, 4)), hex(band(byte, 15))>>
  end

  defp hex(n) when n <= 9, do: n + ?0
  defp hex(n), do: n + ?a - 10
end

The secret sauce was changing ?A to ?a in the hex/1 function

-defp hex(n), do: n + ?A - 10
+defp hex(n), do: n + ?a - 10

Also Liked

al2o3cr

al2o3cr

This is not true, as trying it will demonstrate:

iex(1)> s = <<0xA8, 0x59, 0x72>>
<<168, 89, 114>>

iex(2)> URI.encode(s)
"%A8Yr"

Where Next?

Popular in Questions Top

JDanielMartinez
Hi! May someone helps me, please! I have two apps into an umbrella project: the first one is Database, which manages queries, and the se...
New
Tee
can someone please explain to me how Enum.reduce works with maps
New
LegitStack
I’m hoping you guys can give me some general advice and perhaps code examples if you’re feeling up to it. I’m very interested in Elixir,...
New
Werner
Hi, I’m using Ubuntu 18.04 and after updating to OTP-24.0 yesterday i have this warning when I run “mix local.hex”: 14:57:30.512 [warn] ...
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
myronmarston
The Elixir Typespec docs show the following syntax for keyword lists in typespecs: # ... | [key: type] # keyword lis...
New
fireproofsocks
Forgive me if this is obvious, but how does one delete a database record WITHOUT selecting it first? https://hexdocs.pm/ecto/Ecto.Repo.h...
New
LegitStack
I’m trying to make a websocket server in Phoenix or raw Elixir. I heard about gun, I think I could use cowboy, but since I’m not that sma...
New
Mooodi
Given a string, how can I get access to its character by index? Enum.at("my_string", 2) doesn't work. Or rather, not char, but a substr...
New
Qqwy
Original source of discussion: This topic on the Pragmatic Programmers' Functional Web Development with Elixir, OTP, and Phoenix forum. ...
New

Other popular topics Top

yurko
Here are few pieces of (common) Linux knowledge that we use for reasonably small one server apps. We use Ubuntu but this should work for ...
New
dotdotdotPaul
Okay, I'm having a heck of a time trying to figure out how to best handle the validation of belongs_to associations in Ecto. I'm sure I'...
New
AstonJ
You’re a programmer, so you don’t need spoon feeding with the conventional drivel about “this is an integer.” No. You need to know what’s...
New
_russellb
I want to try my hand at web scraping. What tools/libraries do I need to use. I’m hoping to turn this into something professional so don’...
New
polypush135
As many of you may have realized by now (sorry for all the posts here) I’ve been working on a db problem where I’m trying to aggregate a ...
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
ashish173
I am using Ecto timestamps with postgres, I can see the timestamps() use the :naive_dateime but for my use case I wanted to store the ti...
New
qwerescape
Is there a way to get the call stack or stack trace at any point in the code? Not from exceptions, but an expression that returns how the...
New
AstonJ
by Lance Halvorsen Elixir and Phoenix are generating tremendous excitement as an unbeatable platform for building modern web application...
460 27162 124
New
jay1
Why is it that the mnesia database isn’t the most preferred database for use in Elixir/Phoenix?
New

We're in Beta

About us Mission Statement