Production debugging of Java applications with dbg4j

Debugging - is it good or bad? I have participated in many discussions related this topic (mostly as a listener :) ), and I learned that in most cases there are 2 different approaches of handling and investigation of the defects:

1. Verbal logging. The main idea is that every action should be logged somewhere, so in case of error situation we can reproduce system behavior and identify root cause of the problem by just reading the logs.
2. Debugging. Very simple - application is running with some king of debugging support (very often done on JVM level - f.e JDWP/JPDA), we establish session with the application's process, trying reproduce the bug and perform step-by-step debugging to identify nature of our failure.

Of course, there is no "magic recipe". Both approaches have pros and cons, and should be used depend on particular situation. Let's go through main features of each approach.

Verbal logging. In general, this is good approach and there are situations, when extended logging is required by the nature of the application (not just for debugging). For example, in payment gateway applications, which process customer orders (especially if they perform asynchronous processing based on some queues implementation) verbal logging is "must have" thing.
The main disadvantage of this approach is that you have to deal with the huge amount of logged data. So costs for storage, processing and querying information sometimes are very high, and often it doesn't worth to set-up whole BigData infrastructure just to be able to investigate defects.

Debugging. I love it - you pause application at any place and play with code, review application state and do many-many other things.
But of course there are some cons:

• that can be comfortably done only when you run application(s) locally; usually you have to deal with some troubles trying to connect to the applications running on pre-prod environments (test, stage, integration etc.) and I have never seen anyone who established debugging session with prod environment. So if you have to deal with defect that reproduces only on production - you're in troubles :).
• some troubles of debugging multi-node applications. So for example, there is a bug in web application on stage environment, and you're able to establish debugging session with it. But what to do if there are several (2, 4 or 8 or whatever) instances of the application you have to debug, and all requests go through load balancing app? In that case you have to shut down all the app instances except that one that you've connected to, or establish debugging sessions with ALL instances. You can also set up session with just one instance and wait for needed request - but I'm not sure that is very efficient way of debugging.
• troubles debugging in multi-user environment. Let's get back to the previous example - bugged web application on stage environment. Let's say you have just one app instance. After setting up a debug session, every request from any user on stage (manual testers, automated scripts, other developers etc.) is intercepted by your debugger, which is big PITA. There are ways how to deal with it (f.e. "conditional breakpoints"), but they don't address all possible cases.
• step-by-step debugging of multithread functionality is very hard and (sometimes) impossible to perform.

So, what to do? As I mentioned before there are no "magic recipe" that works for everyone. However, when my team was running into multiple environment-specific issues with our applications, I decided to create lightweight debugging framework to make out life easier. Hope it will be useful for someone else too.

DBG4J to rescue!

What is dbg4j? The main idea behind this tool is to have very verbal and extended logging but on demand (so not for every request/functionality etc.). And that is exactly what this tool does.
Some key features:

• collects detailed information on method invocations;
• is managed by debugging activators, so you have full control of when and what to debug;
• annotation-driven - just mark methods with @Debug annotation and you're done;
• different ways of representation of the debugging information - as a part of response, web page logs etc.
• customizable - you can create your own customized debugging components for your needs.

To simplify integration, this tool heavily uses AOP concepts. To integrate with this thing you have to do the next:

1. (optional, but recommended) compile your app with aspectj compiler to support debugging aspect; set-up aspectj viewer;
2. configure debugging options - debugging activators (when to debug), appenders (where and how to represent debugging info) etc. - default implementations can be used oor you can customize any components you want.
3. annotate methods, classes and fields with specific annotations (if you're using AOP approach) or use other debugging tools
4. enjoy!

Detailed information can be found on project's page in github. There is also an example of integration with simple spring application (exposed as separate commit).

I found this library also useful for speedup learning the new stuff that I have to deal with. So when I have to support the application developed by another team, first thing I do is I integrate app with dbg4j, and try to play with application (as a user). Then I analyze dbg4j output  and get myself familiar with what application does under the hood for each functionality it provides. Even if I won't need dbg4j in a future - it's very easy to "un-integrate" it.

And a short demo video at the end:

Have a great day and enjoy coding!