There’s a lot of things I learned (and still learning!) about implementing an Ecto Adapter that aren’t written down anywhere. I’ve been meaning to write a blog post about the experience, but I don’t even have a blog yet, so that’s still on the project list.

One of those things was realizing I shouldn’t use DBConnection for my particular backend. I don’t know enough about LanceDB to make a call, but if it were me, I’d start without it and add it back in later if needed.

A bit off topic, but this is actually something I’m wrestling with right now on my Ecto Adapter, so I’m curious about what’s in your future for your project.

Aside: I don’t know if folks would be interested in connecting on the EEF Slack for more realtime collab but I’d be up for it.

Hot diggity. Yeah - users do not connect to LanceDB over tcp at all. You can have basically limitless reads and appends, and you only want to throttle concurrent update/delete (mutation) operations to a recommended 10 max. The exqlite discussion you linked is really useful because it seems like similar dynamics are at play.

I am not sure where I got the impression that I needed to implement DBConnection to create an Ecto Adapter but I’m going to regroup and try to attack this directly as an Adapter.

It’s not a noSQL DB, fwiw. “LanceDB supports a growing list of SQL expressions.” including DataFusion math functions. However it’s not relational, i.e. there are no joins.

It also has operations like “nearest_to” for vector similarity search, as well as full-text search, and hybrid “vector + full-text” search with configurable reranking strategies. Very good for RAG. So if I can make an Ecto adapter that supports that kind of stuff either via options or some DSL, that will be ideal.

I am just finishing preparing my Rust NIF wrapper library for Ecto integration. This has mandated a number of changes that I couldn’t easily guess by myself e.g. supporting savepoints and finding the primary key value of a just-inserted record.

Full disclosure: I made heavy use of Gemini Pro v2.5 for this particular task and asked it to analyze the existing first-class Ecto integrations (PostgreSQL, MySQL, SQL Server). It provided a very nice checklist that I am still working through and I am almost done – only schema introspection is left, after which I will proceed with writing another library that is using my Rustler NIF library but adds all the necessary Ecto boilerplate (Elixir code of course) to make it a good Ecto citizen. My goal is a 99% drop-in replacement for the other 3 main contenders (well, and for the other SQLite NIF library that is using C code: exqlite).

This hybrid approach – asking an LLM for a discussion and an analysis, making it generate some code, rigorously checking it against what I wrote (or would write), and asking to check if I missed corner test cases, and a few other such tension items – so far has netted me fantastic results and mildly increased productivity, while keeping me firmly on the wheel (I want an LLM to be an enabler and a productivity booster, not a brain replacement). Most code I wrote came from me with some exceptions like e.g. super long list of error mappings from an underlying Rust library’s error enum to my own type of error (also a Rust enum) that gets encoded to {:error, reason} tuples. I have only used the LLM for such laborious and annoying tasks that would waste me a lot of time. (And still checked them manually by eyeballing, top to bottom).

…oh, and I just used it to generate me a README. It did so surprisingly well.

If you like, we can pair on your library one of the following days, weeks, months. Most of that Ecto integration checklist is still warm & fresh in my head and I’ll start working my way through it VerySoon™. We could do screen share sessions or, if you are not comfortable with that, we can chat via text here or somewhere else. I believe we can help each other. Or you can choose to go your own way if that helps you learn in a way that sticks with you for longest.

I could paste you that checklist here but I am afraid the thread could become quite big and noisy.

I’m wondering if you’re to deep into the weeds here. Yes there is low level details of connection handling. That’s not the only thing to deal with though when ecto is meant to interact with a db. I don’t know much about lancedb, but my point about looking at the higher level was mostly about the question if sql is actually sufficient or if there needs to be other querying apis and stuff like that. If it’s sql, then what sql statements are required to be supported, does Ecto.Query support building up all the necessary statements, do migrations support what is needed, …

DBConnection is also not at all required to build an ecto adapter. It just happens to encapsulate a lot of pooling requirements that you encounter when connecting to a database “server” over tcp. It might or might not be the right tool for a local database, which comes with different tradeoffs and limitations.

The future for generating sql will be GitHub - elixir-dbvisor/sql: Brings an extensible SQL parser and sigil to Elixir, confidently write SQL with automatic parameterized queries.

By using the basic engineering principle of separation of concern you can built something very powerful with Enumerable and Collectable protocols.