josevalim

josevalim

Creator of Elixir

Proposal: Private modules (implementation specific) (closed)

The goal of private modules is to define a module that cannot be trivially accessed by other modules where they are not visible to.

In this proposal, private modules work by declaring exactly which other module prefixes can access it:

defmodulep MyApp.Private, visible_to: [MyApp] do
  def hello do
    IO.puts "hello world"
  end
end

In the definition above, only MyApp and modules nested under it can access MyApp.Private. In this other example:

defmodulep MyApp.Nested.Schema, visible_to: [MyApp.Nested] do
  def hello do
    IO.puts "hello world"
  end
end

only modules in MyApp.Nested and under it can access MyApp.Nested.Schema.

To access a private module, you must explicitly require and alias it:

defmodule MyApp.Other do
  require MyApp.Private, as: Private
  Private.hello
end

The require is necessary to validate the visibility rules. The alias is required to give the private module a proper name (as we will learn later on, private modules live in different namespaces).

Private modules can be arbitrarily nested too:

defmodulep MyApp.Private, visible_to: [MyApp] do
  defmodulep Nested, visible_to: [MyApp] do
    def hello do
      IO.puts "hello world"
    end
  end
end

Requiring MyApp.Private does not automatically require MyApp.Private.Nested. It still need to be explicitly required either directly:

require MyApp.Private.Nested, as: Nested

If you have already required Private, you can also require Nested from the Private alias:

require MyApp.Private, as: Private
require Private.Nested, as: Nested

Nesting

defmodulep works as defmodule as it can be accessed directly following its definition:

defmodule Foo do
  defmodulep Bar, visible_to: [MyApp] do
    ...
  end

  Bar # We can access bar here even if not in visible_to
end

In other words, a more correct description of defmodulep is that it is visible to any following module declared in the same file or to any module declared in visible_to. In fact, :visible_to may be skipped for nested private modules which means they are only accessible to the following modules in the same file.

Testing

In order to test a private module, you need to make sure the private module is visible to the test module. Since most private modules are visible to their own rootname, testing just works if you follow Elixir’s testing conventions. For instance, a private module MyApp.Foo.Bar is likely visible to MyApp or MyApp.Foo, which means the default test module, which is MyApp.Foo.BarTest, should have access to the private module. In other words, the following code should work just fine:

# lib/my_app/foo/bar.ex
defmodulep MyApp.Foo.Bar, visible_to: MyApp.Foo do
  ...
end

# test/my_app/foo/bar_test.exs
defmodule MyApp.Foo.BarTest do
  use ExUnit.Case

  require MyApp.Foo.Bar, as: Bar
  ...
end

Inspecting private modules

Private modules work by being assigned a different naming structure. If you define a private module Foo.Bar, it will actually be compiled as :"modulep_DDD_Elixir.Foo.Bar", where DDD will be a arbitrarily assigned number, instead of the usual Elixir.Foo.Bar. The number is arbitrary to discourage developers from accessing the underlying module directly, as this number may change at any time. The only way to safely access a private module is by requiring and aliasing it first.

Proof of Concept

I have written a proof of concept that is "ready to use today"™ for those willing to try this idea out:

However, the proof of concept has certain limitations:

  • Since we can’t change the behaviour of require, the library introduces a requirep to require private modules.

  • If you define defmodulep Foo and then defmodule Foo, the proof of concept won’t warn.

  • If you invoke SomePrivateModule.foo without requiring it, the error message says the module does not exist, without giving any hints the module is actually private (this may or may not be a feature).

  • Private modules appear literally as :"modulep_DDD_Elixir.Foo.Bar" but it could show up as Foo.Bar when inspected by updating the Inspect implementation for atoms

  • If you define a module defmodule Public nested inside defmodulep Private, Public cannot be accessed directly but only via requirep Private, as: Private and then by calling Private.Public. This will be fixed if we add this to Elixir by making Module.concat/1 to be aware of modulep_DDD_ prefixes.

All of those limitations could be addressed by adding defmodulep
to Elixir.

Your turn

I would love to hear feedback on:

  1. The feature

  2. Implementation details and concerns on this area

  3. The proof of concept

Also, I would love examples of how other languages tackle private modules. A common implementation is to have the visibility of your modules associated to the “idea of a package” but Elixir does not quite have the concept of a package. Elixir does provide the idea of “applications” but they are define only after the code is compiled. That’s why the “package” approach has been ruled out in favor of a explicit visible_to control.

Most Liked

JEG2

JEG2

Author of Designing Elixir Systems with OTP

I find myself not wanting this change. It feels like a lot of special cases, for mild protection. I feel like people who use code with something like @moduledoc false on it today know what they are doing and will not be deterred by this feature. The protection is easily defeated as shown in this thread.

Maybe I just haven’t felt the pain this feature is targeted at though. Perhaps a good example use case would help me see the value.

13
Post #7
josevalim

josevalim

Creator of Elixir

The problem with this line of thought is that it gives an impression of instability since packages break whenever there is a new release of something they were using a private API of. It also discourages communication in favor of quick work arounds. Well, if you need private functionality, why not start a discussion on the best way to expose it?

My experience coming from the Ruby community which (at the time) did not value contracts and visibility that much is that this leads to a lot of pain down the road, especially as systems grow in complexity. Updating only a small part of the system becomes impossible, because a minimal change breaks many unwarranted things along the way. It usually goes like this: let’s update Elixir! Unfortunately, updating Elixir breaks package X because X used a private API. So we have to update package X too but wait! That breaks Y and Z. And so on and so on.

Also beware of “truths made along the way”. It is very likely a community ends-up accepting that “being able to call privates is a good thing” because this behaviour was there since the beginning and it is impossible to change it now, so the best they can do now is to focus on the pros despite the cons. Note this is not a criticism to Python nor I am implying it is the case here, as I am not that familiar with the Python community, but it is an effect we see in all communities, including Elixir’s.

josevalim

josevalim

Creator of Elixir

Answering many comments at once…

First of all, in regards to the feature as a whole being necessary: it absolutely is. For an example, just look at the Elixir v1.7 release which broke many packages that were using Elixir private modules. When talking to developers who were using these APIs, most of times, they simply did not notice it wasn’t documented. On large systems, this leads to a cascade effect that makes it very hard to update only parts of the system, because in order to update Elixir, you also need to update dependency X, which may break Y and Z due to private APIs, and so on.

In regards to the feature having workarounds: yes, there are workarounds and even simpler than the ones posted on this thread. But let’s be honest here: there is no implementation that will forbid someone from bypassing the visibility boundaries if someone really wants to do it. All languages that I have explored while writing this proposal has this “flaw”. In a nutshell, we mostly need a better way to document intent.

That said, I think @hauleth does provide a good point: all of the features above could be achieved with just warning. So we need to do a choice between a hard failure (this proposal) or a warning (as mentioned by @hauleth). Implementing it via a warning would be much simpler but such warnings would be a “best effort” and they would be quite easy to bypass. For example, by doing:

mod = SomePrivateModule
mod.foo()

Is the warning worth it even if it is not guaranteed? Thoughts?

mkaszubowski

mkaszubowski

I’m absolutely for this change :heart:

Most people focus on library code, but I think it’s also important inside a single codebase, maintained by one team. Currently, we have no language support for setting up boundaries and establishing clear interfaces between different parts of the application, which can lead to issues with maintainability.

Sure, with enough discipline you can make sure that nobody uses these “private” modules, but the same could be said about private functions, and yet we have them in a language. Unfortunately, disciple and conventions fades when we face a deadline, are tired, unexperienced or new on the team. That’s why having a help from the compiler is essential. If we care about visibility of a function, why shouldn’t we care about visibility of a module? After all, the purpose of both of them is to organise code.

hauleth

hauleth

I have mixed feelings about it. On the one hand private modules are nice idea, on the other hand I think that this proposal is quite “too much”. What I would like to see instead is rather compiler warning and module attribute (maybe even with EEP), so this would not be a hard error, but compilation warning, something like:

defmodule MyApp,Private do
  @private [MyApp]

  def hello do
    IO.puts "hello world"
  end
end

This would be less strict than Your proposal, but I think it would be more “in line” with rest of the platform.

Where Next?

Popular in News Top

josevalim
NOTE: this is a focused thread, so we appreciate if everybody stayed on topic. Feel free to comment anything in regards to calendar forma...
New
Elixir
Elixir v1.18 is an impressive release with improvements across the two main efforts happening within the Elixir ecosystem right now: set-...
New
Elixir
Release: Release v1.13.1 · elixir-lang/elixir · GitHub 1. Bug fixes Elixir [Code] Do not show code snippets in SyntaxError and TokenMis...
New
Elixir
Release: https://github.com/elixir-lang/elixir/releases/tag/v1.11.1 1. Bug fixes Elixir [Code] Ignore tracers if lexical tracker is dea...
New
Elixir
This release includes type inference of patterns to provide warnings for an initial set of constructs (binaries, maps, and atoms) within ...
New
Elixir
1. Enhancements Elixir [JSON] Encode any JSON key to string [Kernel] Allow <<_::3*8>> in typespecs Mix [mix loadpaths] Sup...
New
Elixir
1. Enhancements Elixir [Code] Add :emit_warnings for Code.string_to_quoted/2 [File] Add :offset option to File.stream!/2 [Kernel] Auto i...
New
Elixir
Type system improvements Type checking of protocol dispatch and implementations This release also adds type checking when dispatching and...
New
Elixir
Official announcement: Elixir v1.16 released - The Elixir programming language 1. Enhancements EEx [EEx] Include relative file informat...
New
josevalim
moderators note: A conclusion by @josevalim has been drawn in Proposal: Private modules (general discussion) - #143 by josevalim While...
320 12877 159
New

Other popular topics Top

Qqwy
Update: How to use the Blogs & Podcasts section You can post links to your blog posts or podcasts either in one of the Official Blog...
3268 119930 1237
New
sergio
In Ruby, I can go: User.find_by(email: "foobar@email.com").update(email: "hello@email.com") How can I do something similar in Elixir? ...
New
albydarned
Hello all! I am typing this post from my new MacBook Pro with the M1 chip. I’m loving it so far, and will probably use it as my daily dr...
New
jerry
Good day to you all. I have been struggling to get a query involving like and ilike to work. Can anyone assist me on this, please? pro...
New
Harrisonl
We have an ECS cluster with 4 services, where each task joins a single cluster, via discovery ECS discovery service. Currently when I de...
New
chrismccord
This release brings a number of exciting features, including integration with the new Phoenix LiveDashboard and Phoenix LiveView. There h...
New
vac
Hi, I'm quite new in Elixir and I'm trying to format a string to a PEM format. I have the certificate value like MIIDBTCCAe2...... and ...
New
alice
Hey, Just curious what are the main benefits of Elixir compared to Clojure? When is Elixir more useful than Clojure and vice versa? Th...
New
belgoros
I’m not a pro in using Regex and can’t figure out why the following behaviour happens, especially if we take into account the difference ...
New
AstonJ
by Lance Halvorsen Elixir and Phoenix are generating tremendous excitement as an unbeatable platform for building modern web application...
460 27162 124
New

We're in Beta

About us Mission Statement