I’m finding LLMs pretty useful across a broad array of tasks. The TL; DR is that it works very much like a junior developer that I pair program with. There are mistakes (conceptually and in details) and I’m not saying “go write an accounting system with inventory control”… I’m very much taking things function-by-function… but it’s useful.

For me, LLMs come to mind/into play when I’m doing the following:

• Autocomplete while writing code. While I’m not sure it’s the most important thing I use the LLM for, it’s definitely the most common interaction for me because it happens without my stopping to make it happen. Not unlike standard autocomplete anticipating the next variable or keyword you’re starting to type, the LLM is anticipating the expression or statement (or even block) that you’re going to write. Results can vary here quite a bit and this is probably an area where the integration tool has the biggest influence; Cody does a meh job with this task and Cursor does a very good job with it (both Claude Sonnet 3.5). When this works well, boilerplate-like blocks just appear and you can accept them and move onto the next. Context available to the LLM here is clearly key. Often times I can accept the autocomplete suggestion without changes, there’s a decent amount of suggestions that can be accepted with minor modifications, and some of which just aren’t right. Interactivity with the autocomplete functionality varies from tool to tool as well, but most will show you the suggestion and then you can use keystrokes to accept/ignore it/retry it. Cursor will also allow you to incrementally accept the suggestion (crtl + arrow I think) word by word.

• Initial Documentation. I’m finding it useful too, once I have an API that should have ExDoc strings written, I let the LLM write the initial documentation. It does a pretty good job of getting things like parameters and return values at least represented and it also will often times include even examples. It’s terse and sounds like it was written by a marketing department minion, but it gets a lot of the form right. Again, it does well with boilerplate-like bits like assigning sections if there’s enough context from other parts of the code to set the example. I do go back and clarify or re-write portions, but its better than just starting from scratch. I also find it less mentally taxing to act in the editor/reviewer capacity than in the author capacity.

• Writing Tests. Recently I had to write some tests for some of my application components which were created before my testing strategy was fully thought out. For each new test, I just told the LLM that I needed a test and gave it the file and function name being tested. This worked well and was more thorough than I might have been in some cases. There were a fair number of times where I had to tweak the tests to be correct, but it still was a significant time saver. Again, context availability was key here and if I had similar tests already existing, the output quality would go up as the generated test incorporated norms within the testing corpus.

Other less frequent uses are:

• Supplying expertise I lack. Some of my recent tests were dealing with network addresses and related bit twiddling… I’m much more facile with financial and accounting operations than I am with bitwise operations. The LLM was able to correctly do the bit manipulation and evaluation; all I had to do was validate that it was correct. This also manifests in being able to recall the APIs of common libraries across a broader range of topics than I’m usually commonly working with.

• Interpreting Difficult to Read Code. A colleague tried to use an LLM to convert a MSSQL stored procedure into a PostgreSQL function. The LLM failed and they eventually called me in. I do understand the PostgreSQL just fine, but I’ve never worked with TSQL and the original code given to me was “a little obscure”… and they couldn’t tell me even anything about the workings of the code. After a couple of “close but not quite attempts” on my part… I finally broke down and just asked the LLM to tell me in plain English what the original code did. And it did so perfectly and understandably. I could see the places where I misunderstood or let confirmation bias cloud my view and I was able to immediately produce correct and simpler/saner code in PostgreSQL. (That was my first LLM experience and moved me from skeptical to enthusiastic).

• Writing Shell Scripts. The last time I looked at the bash man page I think I saw that the Marquis de Sade was in the author’s list. I hate shell scripts with a passion: I write them rarely and typically need a drink immediately after writing one. These tend to be small programs well within the scope of a decent LLM, and so far the experience of letting the LLM deal with this when needed as increased my personal joy a lot.

• Writing regular expressions. See “Writing Shell Scripts”.

What I do not use the LLM for:

• Search. When I’m searching, I want something much more mechanical than an LLM is designed to produce. LLMs could excel doing contextual searches with inferred matching, but they end up failing in completeness or end up including mistakes (or making stuff up). They designed to produce credible language like a human might produce in a similar situation… faults and all… not mechanically testing for thoroughness or even correctness. Outside of basic “does this thing exist” kind of questions, I’d tend to avoid it for this case and even then I tend not to trust it.

• Brainstorming. I’d think it would be useful in a case like this, but too often I’m just getting a conventional wisdom that I’m often times already aware of. This isn’t surprising.
  There exist parameters to tweak how “far out” the models can stray from the most common kind of response… but that’s more time investment than I’m willing to take so I don’t go there.

Anyway… you asked …

FWIW, it has helped me to add “write the most concise and idiomatic elixir code possible” to any prompt (or to the project’s general instructions). Especially when learning Elixir, I often was writing code based on old habits (too many if statements etc), and the AI code really accelerated my learning. Also, asking it to write my own “bad” code using more idiomatic Elixir was often enlightening.

Yeah, I agree with much (all?) of what you say here and would go further in that one of the big concerns for me is that even experienced developers’ knowledge/ability can atrophy without regular exercise. While I find that I get to my larger goals faster with the LLM, in part that’s because I don’t have to work or think as hard… but I am losing that regular exercise and expect that writing without the LLM will result in my being slower than I was before just because I’d have to rebuild that knowledge of language minutiae to be as effective as I was pre-LLM.

I think what it means to be a developer will be an interesting topic over the coming years as these tools and workflows improve. How do juniors obtain experience and will the art become more and more about extracting a given result from the tool rather than not needing the tools at all? Are there analogies from other industries about how, for example, we moved from designing things like airplanes very manually with actual pen and paper drawings and slide rules to using advanced CAD systems which can even simulate the complex aerodynamics of the system? Surely, we lost skills when these very manual tasks of drafting and calculation were taken on by the computer, but we also gained a lot too. (And to be clear, I’m think in terms of professional software development, not recreational software development where a more “artisanal” approach to coding can very well be more desirable.)

Anyway, interesting times ahead and it will be interesting to watch as it evolves. In the meantime, I’ll continue to use the LLM because it allows me a real shot of completing a project that otherwise is too foolishly large for an indy developer like me to complete without such aids.

Posting here my slightly angle for potential newcomers to Elixir like myself.

Context
I’ve just started to pick up Elixir (had a go with Dave Thomas Elixir for Programmers a couple of years ago, loved it but stopped half way because life and work (Python) got in the way) for the second time.
This time is more serious as I quit my job and have full on time to go at it if I’d like.
I’m trying to create my own internet products (nearly all in Python) and have deeply dived into using LLMs (via Cursor) in pretty much all of the ways described before - to great benefit, speed and boilerplate-skipping.
More specifically, I like the autocomplete which gets me 90% of the way there and I’m immediately able to assess whether the code will behave as intended or is structured in my intended way.

Opinion for learners

• For learning specifically I’m choosing to not let AI write any code for me (not even tests) for the time being (for as long as I can, hopefully the first year at least).
• This is because I want to ingrain in my muscle and visual memory the language, libraries, idioms and just feeling it - I feel like I’d be missing out on some good fun/dopamine if I just let AI do it all. After all I’m going into Elixir with the expectation that it becomes my new favourite language - so I want to maximize the pleasure of writing the actual code.
• For documentation, interpreting and explaining new code and as @sbuttgereit does for delegating away ancillary tools/code (read bash, deployment/configuration tooling) that you’d rather not deal with it - I’m generally all in.

Hope this is somewhat useful

Wise choice, you have my respect for it. You seem like one of those Elixir beginners that I had a blast tutoring some 4-5 years ago – eager, willing to learn, not lazy, and willing to put in the work to get the right habits ingrained in the brain. Bravo.

As you yourself noticed, it’s important to learn which is which, and why. The current breed of what people call “AI” should be limited to boilerplate generation for the moment, because in the observations of my colleagues and acquaintances they are not good for much else. And the nature of “AI” does not help with this because statistically the bigger dataset (JS, Python) yields better results and frak everyone else I guess – solid philosophy but I expect nothing more than VC-funded companies so it’s all par the course.

Elixir has high chances of becoming your new favorite language. Immutability and generally the isolation of side effects help with developer experience much more than JS / Python programmers realize (they only understand it after the fact). Throw iex and ex_unit in the mix and you have an awesome success recipe.

Shout out if you need help here on the forum.