Hi @pdgonzalez872,
the answer I have in mind is very general, and has to do with debugging more than with specifically analyzing logs, but I hope it can still be useful.

In my experience, debugging is an activity that is best approached with the scientific method.

Imagine you have a weird bug. Something that is not immediately obvious, but rather puzzling and defying explanation. Say that application instances crash, for no apparent specific reason. Assume that you collected the initial evidence, but you are still clueless about the root causes. Here is where the scientific method comes into play:

• First of all, before you even start digging through the logs, formulate hypotheses. A good hypothesis is one that produces testable predictions. For example, if I suspect that the crash is related to a hardware problem, I should predict that all crashing instances are located on the same physical machine. If you instead suspect a memory leak, you might predict that we should find “out of memory” log entries. These are things we can practically validate.

• Even more important, hypotheses are falsifiable: if we observe instances crashing on different hardware nodes, my hypothesis about a hardware failure becomes extremely unlikely, so we can archive it and move on to the next one. I should not get attached to my hypothesis: if it turns out to be false, we still made a step forward in understanding.

• Only after knowing what you are looking for, you look at your logs, metrics, etc. It’s very important to stick to testing hypotheses, instead of spending too much time randomly looking for patterns. Even though serendipity can sometimes help in simple cases, our brain is too prone to see some pattern even when there is really none. If we really suspect that a pattern is more than a coincidence, we need to find a way to test that.

• Typically, this goes in cycles: a hypothesis suggests a test, which brings some answers, which in turn bring more questions, more hypotheses, more tests, etc. If the test result is inconclusive, we need to go back to the drawing board, with either new hypotheses or new ways to test. This testing cycle drives your investigation.

• The best engineers I had the pleasure to work with are egoless: they won’t focus on proving themselves right, or on showing off their knowledge. They can use this method so efficiently, that it might seem they can always guess right. The truth is often the opposite: they are good precisely because they don’t guess, but rather follow a method.

Does this make sense?

Here is my effort to share in short what I learned by experience about logging. The most important thing, in my opinion, is that your log messages should be:

1. Informative: while concise, they should contain all the important information about the event, not just say “event X happened”. It helps to think: should I even look for this event in the logs, what information would I probably want to know? A practical example: a log event about a purchase might include the user ID that made the purchase.

2. Discoverable: it is useless to log a lot of information, if it is not possible to easily find it when you most need it. This is not only a problem of having the right tools, but has actually to do with logging in such a way to enable future retrieval. Should you need to search for logs related to a certain occurrence, what search terms are you likely to use? Also, when debugging, you often start looking for some event in the log, but soon want to correlate it with other events happening around it. A practical example: in a web app, including a unique “request ID” in all logs pertaining to the same HTTP request makes it possible to easily correlate all log events that happened within the same request. Including a user ID, or a hash of it, makes it possible to find everything logged regarding a specific user (just pay attention not to log sensitive user data though). Tagging logs by service is also a common strategy. Multi-service architectures often pass around a “correlation ID” for the same request, that all services include in log messages, in order to follow the request flow across services.

3. Reviewed and maintained: when debugging an issue, take note of what would have helped you, and what could be improved. Every application is different in terms of what is interesting to log and what is not. Keep improving it, add missing information or events, remove noise, and debugging will get better and better for you and your team.

What do you think? I hope this helps

Ciao @lucaong, that’s great advice. It talks about how to log and maintain logging sane, this is critical to this task.

Now, help me think: imagine you have great logging already since I followed your instructions. Now what? There is something to be said about what to do with the logs after you have them. I feel that it is almost like looking at a painting: some people know what they are looking at and others see some paint on a canvas. (sorry for the analogy , best I could do)

As usual, it depends on many factors. But, in general, I do think there is a process, some type of step by step that we do, maybe even without knowing we do it. Maybe they are as follows:

• Do a first read in chronological order to see if anything jumps out as you are looking for clues (cc: @fhunleth)
• Anything weird? Warnings, errors?
• Use clues in logs to build a timeline of events
• If not enough logging, follow @lucaong’s advice and add. Deploy and monitor.
• (Imagine this is a bug for a second) Find error in log. Look at codebase for said log message. Go up the call stack and find root cause.

Something like that? Anyways, I don’t want to make this into an esoteric discussion, I just wanted some guidelines on how to do it properly, I want to understand that painting!

Thank you for the reply, really good feedback, exactly the type of discussion I was looking for.

Wow, fantastic answer man.

I do think deep down there is a process. After reading your answer, I agree, this is an awesome way to approach issues and I bet you that the really good ones do this, whether they realize it or not. Sometimes people forget how they got to where they are

Thank you for spending the time to write it. This is awesome.