I’ll tell you what my real experience is, using LLMs has allowed me to write more well tested code, as well as more well documented code. For example, we’ve been having discussions about some issues w/ Ash’s documentation.

1. the search results didn’t seem ordered by the most helpful stuff
2. there is so much content that “cross referencing” is important, i.e putting links to things you were probably looking for if you got here and its not what you want.

I spent about 2 hours w/ Claude working on our four most commonly used modules, Ash, Ash.Changeset, Ash.Query and Ash.ActionInput. We have a solid usage rules file and other things that helped guide its output. I tasked it with expanding the docs on the most common functions (and many of the less common ones) ensuring that we have example usage, cross links to other relevant guides, and typespecs. It wasn’t “vibe coding”, i.e I reviewed all of the work, made many tweaks of my own, etc. But it turned a multi-day project into a 2-3 hour project. And when we have other docs issues in the future I will likely feed it into Claude and have it generate some nicely formatted output. i.e “this user had trouble with X, please look at the current docs for Y and produce something that would have prevented this confusion”. It also figured out why the search results were suboptimal before I could figure it out myself. The order of modules in the sidebar controls to some degree the order of modules in the search. We reordered the sidebar, and search got that much better

If you’re familiar with the issues around Ash docs, the improvement to this module is effectively night and day: Ash — ash v3.5.13

To me, it’s not a replacement for my brain, it’s just a way for me to exert more leverage than I would have before. When you know how to use it, it can make you faster, help you automate the boring stuff so you can focus on the important stuff, and get you past the blank-page problem.

Having spoken to Zach at ElixirConf EU and heard some of the challenges he has seen with supporting users that have completely fictional code. Coupled with his general skepticism of the AI hype. I am glad he wrote something up.

This is a pragmatic take on what we can usefully do around adoption, and being on a slimmer end of the bell curve when people start using bell-curve-machines for information and code generation.

Without trying to go to the moon or saying the cost of programming will go to zero. That stuff is speculative. This is mostly addressing current reality, whether you like the situation or not. I don’t love dealing with LLMs but I don’t particularly want to close my eyes because I don’t think it will entirely blow past.

treat them as if they are only capable of doing things like summarization and pattern transformation

They are not even capable of this, only of giving the appearance of being so. It’s on the prompter to check the accuracy.

I think the biggest direct risk for most users of LLMs are that the LLM companies are aiming to become gatekeepers. All these efforts to integrate with LLMs are free work for these companies, who will increase the rents once they get enough people dependent on them.

I’ve only just started using AI and I have yet to use an agent. I’ve also never really chat as there is something something I find so off-putting out it. This article has some nice tips and some good insights that I hadn’t thought of. I particularly like the tip of asking it to first find the docs. This is something many people clearly take for granted.

Ultimately you should not be relying on an LLM as a source of “knowledge”.

I’ve seen several questions here, on reddit, and watching people IRL where they simply refuse to use anything other than a chatbot to figure things out. When they give up they don’t look to the docs, they ask someone. I know I’m not the only one wondering this but I haven’t seen too many satisfactory answers: what happens when people stop writing documentation? Are we going to start seeing libraries where the authors haven’t looked at a line of code? If people are producing software like this, how do the LLMs learn to use it? They’d have to start analyzing the code, of course, but how is that not going to degrade the slop even further?

Not trying to be a doomer just wondering, and wondering is all we can do.

Best case AFAIC is that we get flooded with so many quick-to-market apps that carefully crafted software with performance and security as first class citizens becomes cool again. And by “cool” I mean every day users will feel a noticeable difference.

Ask Claude for a TLDR