An alternative is to copy what haskell has done. They’ve got a library that parses kate syntax XML highlighting files and generates lexers from them at compile time. The format of the XML files is quite simple, but most of the XML files are LGPL, which is probably too restrictive for most people.

Another alternative is to parse textmate’s syntax highlighting files, but their grammar is more confusing because they do funny things with regexs…

The obvious answer is that yes, you can port the lexers as long as you port the Python code to Elixir.

The practical answer is: Yes, but you have to rewrite the weird Pygments’ state-machines into a grammar for a PEG parser (which is way simpler than the original state-machines.

The crazy answer is that tah Pygments lexers are somewhat data-driven, if you squint a little (a lot), so it’s probably possible to write a Python program that compiles most of the Lexers into Elixir. Writing such program would be highly non-trivial, and my library is technically superior (and more powerful) than Pygments in a fundamental way, so I think that it’s better to just port the lexers.

I’d like to write a guide on how to write the lexers and crowdsource it to the polyglot members of the Elixir community, but I have no time for that now…

The elixir lexer had some problems too. I’ve corrected them in my implementation. Pygments in general is not a very good piece of software. The real killer feature of pygments is the enormous number of man-hours put into writing lexers for all those languages. The lexers themselves are not necessarily very good.

Hm… I don’t want to criticise Pygments too much. Pygments is very versatile, and even though I don’t like the architecture for its lexers, it’s a fact that people have been writing new lexers. Some people might even prefer their way of writing lexers. If I were to start Pygments today, I’d use a PEG parser like Arpeggio, but they surely had their reasons to start with a state-machine based approach… Maybe there were no good PEG parsers for python in 2006? Maybe despite my opinion egexes + state-machines are easier to use than PEG parsers? Maybe people started writing things that way and then there was already too much code to change it to a different architecture? I don’t know.

My own design borrows a lot from Pygments: it has lexers, formatters and styles. The styles themselves were shamelessly copied from Pygments (by writing a python program that introspects the styles and turns them into Elixir modules), and my HTML formatter is not as feature rich as theirs. I can copy all of their features given enough time, but I’m only one person and have limited time… Have you looked at their list of contributors? They mention 212 people… My project is a joke in comparison xD