Why do your expectations need to change? Naively I‘d say you don‘t want to assert on the response you get from a third party, but you want to assert on what those responses mean to your systems state. Otherwise you‘re testing the third party system not yours.

okay i’ve thought about this long and hard. i keep feeling that i’m doing something wrong. so i spent days refactoring and simplifying the architecture and i think i got a clear picture now.

problem was that my requests were messy and there was no clear point to differentiate business logic from request stuff. so here’s how i’m thinking about it now:

i think this is ultimately the correct approach regardless how i test the business logic and API requests. i’ve simplified so i do have a single module for most of the requests now (that calls the complex stuff from there). and for a start i’ll just not run the API tests automatically and mock the business logic that happens after the reqests instead.

this is an interesting one. i do in fact want to test that my whole system works even if the 3rd party API changes, but you’re right that i don’t need to test that very often.

another issue here is that before business logic happens the API communication has complexity in itself already. say one function makes 3 requests and puts a single struct together that way. i do want to test this. but again, keeping business and API tests separate here i think is the solution.

oh my god this is awesome! however i don’t think it caches any requests like ExVCR so it’s not quite on topic. But I may use this in the future anyway, looks like a great idea.

this is a good idea in general, i think. but then it adds a lot of complexity that i don’t really want to spend my time on fixing later. also this is basically what ExVCR does (but they probably do it better) and yet i can’t really get that thing to work properly. and even when i do it’s kind of a black box and i’m not sure my own solution would be any better.

currently with my new clear way of separation of business and API concerns, i think i will need solution like this eventually. like if i want to run my tests more often.

edit: oh, i should add. i think i’m going add the flag (as a module property) to enable printing of responses from my central module, because i still need to write expectations to the tests.

thanks for chipping in everyone, you really sped up my thinking here. i will probably refer back here again and again in the coming years

Probably but it relies on mocking where what I propose justs needs a if somewhere, which is much more simple and even if you are not confident in the fact that you can pull it out I’m sure you will very quickly see that you actually can.

Something like that could be a good start :

defmodule MyApi do
  case Application.compile_env(:my_app, :api_request_mode, :real) do
    :real ->
      def request(url, method, body, headers) do
        do_request(url, method, body, headers)
      end

    :record ->
      def request(url, method, body, headers) do
        file = hash_req(url, method, body, headers) <> ".json"
        resp = do_request(url, method, body, headers)
        record_response(resp, file)
      end

    :mock ->
      def request(url, method, body, headers) do
        file = hash_req(url, method, body, headers) <> ".json"
        mock_response(file)
      end
  end
end

No mocking library or macros to get in the way.

What about writing a test that hits the real thing/a sandbox? You’d load real keys, make it hit the real endpoints. These tests would only be run locally and would be excluded from a normal mix test run (what CI would run).

It’s basically capturing what you did in your IEx sessions/running the real code (however you were using the IO.inspect/1 calls). I’ve seen these types of tests save a lot of time for teams. Mostly because it’s documentation and something that anyone on the team can run. No passing scripts around, etc.

More info: Mocks and explicit contracts - Dashbit Blog, the

Along these lines (the article does a great job at showing these):

defmodule IntegrationTests do
  @moduletag :integration_tests

  ...
end

# test_helper.exs
ExUnit.start(exclude: [:integration_tests])

Very much that. Keep the code directly depending on the API shallow, e.g. just pulling out certain values out of individual requests and let the business logic/composition of those values work only with those known to exist values.

The more business logic you can decouple from the actual requests being made the simpler it will be to test.