Production debugging of Java applications with dbg4j

Debugging - is it good or bad? I have participated in many discussions related this topic (mostly as a listener :) ), and I learned that in most cases there are 2 different approaches of handling and investigation of the defects:

1. Verbal logging. The main idea is that every action should be logged somewhere, so in case of error situation we can reproduce system behavior and identify root cause of the problem by just reading the logs.
2. Debugging. Very simple - application is running with some king of debugging support (very often done on JVM level - f.e JDWP/JPDA), we establish session with the application's process, trying reproduce the bug and perform step-by-step debugging to identify nature of our failure.

Of course, there is no "magic recipe". Both approaches have pros and cons, and should be used depend on particular situation. Let's go through main features of each approach.

Verbal logging. In general, this is good approach and there are situations, when extended logging is required by the nature of the application (not just for debugging). For example, in payment gateway applications, which process customer orders (especially if they perform asynchronous processing based on some queues implementation) verbal logging is "must have" thing.
The main disadvantage of this approach is that you have to deal with the huge amount of logged data. So costs for storage, processing and querying information sometimes are very high, and often it doesn't worth to set-up whole BigData infrastructure just to be able to investigate defects.

Debugging. I love it - you pause application at any place and play with code, review application state and do many-many other things.
But of course there are some cons:

• that can be comfortably done only when you run application(s) locally; usually you have to deal with some troubles trying to connect to the applications running on pre-prod environments (test, stage, integration etc.) and I have never seen anyone who established debugging session with prod environment. So if you have to deal with defect that reproduces only on production - you're in troubles :).
• some troubles of debugging multi-node applications. So for example, there is a bug in web application on stage environment, and you're able to establish debugging session with it. But what to do if there are several (2, 4 or 8 or whatever) instances of the application you have to debug, and all requests go through load balancing app? In that case you have to shut down all the app instances except that one that you've connected to, or establish debugging sessions with ALL instances. You can also set up session with just one instance and wait for needed request - but I'm not sure that is very efficient way of debugging.
• troubles debugging in multi-user environment. Let's get back to the previous example - bugged web application on stage environment. Let's say you have just one app instance. After setting up a debug session, every request from any user on stage (manual testers, automated scripts, other developers etc.) is intercepted by your debugger, which is big PITA. There are ways how to deal with it (f.e. "conditional breakpoints"), but they don't address all possible cases.
• step-by-step debugging of multithread functionality is very hard and (sometimes) impossible to perform.

So, what to do? As I mentioned before there are no "magic recipe" that works for everyone. However, when my team was running into multiple environment-specific issues with our applications, I decided to create lightweight debugging framework to make out life easier. Hope it will be useful for someone else too.

DBG4J to rescue!

What is dbg4j? The main idea behind this tool is to have very verbal and extended logging but on demand (so not for every request/functionality etc.). And that is exactly what this tool does.
Some key features:

• collects detailed information on method invocations;
• is managed by debugging activators, so you have full control of when and what to debug;
• annotation-driven - just mark methods with @Debug annotation and you're done;
• different ways of representation of the debugging information - as a part of response, web page logs etc.
• customizable - you can create your own customized debugging components for your needs.

To simplify integration, this tool heavily uses AOP concepts. To integrate with this thing you have to do the next:

1. (optional, but recommended) compile your app with aspectj compiler to support debugging aspect; set-up aspectj viewer;
2. configure debugging options - debugging activators (when to debug), appenders (where and how to represent debugging info) etc. - default implementations can be used oor you can customize any components you want.
3. annotate methods, classes and fields with specific annotations (if you're using AOP approach) or use other debugging tools
4. enjoy!

Detailed information can be found on project's page in github. There is also an example of integration with simple spring application (exposed as separate commit).

I found this library also useful for speedup learning the new stuff that I have to deal with. So when I have to support the application developed by another team, first thing I do is I integrate app with dbg4j, and try to play with application (as a user). Then I analyze dbg4j output  and get myself familiar with what application does under the hood for each functionality it provides. Even if I won't need dbg4j in a future - it's very easy to "un-integrate" it.

And a short demo video at the end:

Have a great day and enjoy coding!

JGit in short example

JGit is the great java library, that provides functionality to perform basic git operations. I used it when I decided to implement release cutting helper for my team. One of the requirements was to make this tool cross-platform (part of the team uses Windows PCs, and another one - Macs), that's why I decided to use java.

The very first search in google pointed me to JGit as "one of the best solutions to work with git from java".

Despite its popularity, it was hard to find some useful code examples for beginners, instead I've found a lot of general descriptions of how cool this library is.

Finally I've got some working examples in stackoverflow, that allowed me to start building my tool.

So trying to fix that situation - listing basic git operations using JGit:

public class GitUtils {
    private Git git;

    public GitUtils(String localPath) throws Exception {
        if(!localPath.endsWith(File.separator)){
            localPath += File.separator;
        }
        git = new Git(new FileRepository(localPath + ".git"));
    }

    public void pull() throws Exception {
        String currentBranch = git.getRepository().getBranch();
        StoredConfig config = git.getRepository().getConfig();
        if(config.getString("branch", currentBranch, "remote") == null){
            config.setString("branch", currentBranch, "remote", "origin");
            config.setString("branch", currentBranch, "merge", "refs/heads/" + currentBranch);
        }
        git.pull().call();
    }

    public void commit(String message) throws Exception {
        git.commit()
                .setMessage(message)
                .setAll(true)
                .call();
    }

    public void push() throws Exception{
        git.push().call();
    }

    public void checkout(String branchName, String sourceBranchName, boolean createNew) throws Exception {
        CheckoutCommand command = git.checkout()
                .setName(branchName)
                //.setUpstreamMode(CreateBranchCommand.SetupUpstreamMode.TRACK)
                .setForce(true)
                .setCreateBranch(createNew);
        if(StringUtils.isNotBlank(sourceBranchName)){
            command = command.setStartPoint(sourceBranchName);
        }
        command.call();
    }
}

Enjoy coding!

Google Guava: missing cache entries

Recently I had to create new caching layer for our web applications. Nothing fancy, just usual multi-level read-through cache. I used Cache from Google Guava library for one of the caching levels, and faced interesting issue. Some tests intermittently failed b/c of the cache data misses, even despite that data was loaded into the cache during test data initialization.

Playing with different cache configurations I found out that data misses happens in a case of full cache population.

Below are the test that demonstrates the issue. It creates Cache, populates it with the random data, then reads the data from the cache and and counts number of misses (google guava v. 14).

public class GuavaTest {

    public static void main(String[] args){

        for(int i = 1000; i <= 50000; i += 1000){
            System.out.println(MessageFormat.format("count = {0} missCount = {1}", i, String.valueOf(test(i))));
        }

    }

    public static int test(final int cacheSize){

        String[] urls = new String[cacheSize];
        com.google.common.cache.Cache<Object, Object> cache = CacheBuilder.newBuilder()
                .maximumSize(cacheSize)
                .build();

        for(int i = 0; i < cacheSize; i++){
            urls[i] = UUID.randomUUID().toString();
            cache.put(urls[i], new Object());
        }

        int missCount = 0;

        for(int i = 0; i < cacheSize; i++){
            Object o = cache.getIfPresent(urls[i]);
            if(o == null){
                ++missCount;
            }
        }

        return missCount;
    }
}

Below are results I've got (they're slightly different every time, but trend and range are approx. the same):

I was really surprised with results. Maybe that's the price for some kind of optimization or something. I'm still using guava cache, it is great tool, but always keep in mind that I have to reserve a bit more slots than required number to avoid data loss.