Production debugging of Java applications with dbg4j

Debugging - is it good or bad? I have participated in many discussions related this topic (mostly as a listener :) ), and I learned that in most cases there are 2 different approaches of handling and investigation of the defects:

1. Verbal logging. The main idea is that every action should be logged somewhere, so in case of error situation we can reproduce system behavior and identify root cause of the problem by just reading the logs.
2. Debugging. Very simple - application is running with some king of debugging support (very often done on JVM level - f.e JDWP/JPDA), we establish session with the application's process, trying reproduce the bug and perform step-by-step debugging to identify nature of our failure.

Of course, there is no "magic recipe". Both approaches have pros and cons, and should be used depend on particular situation. Let's go through main features of each approach.

Verbal logging. In general, this is good approach and there are situations, when extended logging is required by the nature of the application (not just for debugging). For example, in payment gateway applications, which process customer orders (especially if they perform asynchronous processing based on some queues implementation) verbal logging is "must have" thing.
The main disadvantage of this approach is that you have to deal with the huge amount of logged data. So costs for storage, processing and querying information sometimes are very high, and often it doesn't worth to set-up whole BigData infrastructure just to be able to investigate defects.

Debugging. I love it - you pause application at any place and play with code, review application state and do many-many other things.
But of course there are some cons:

• that can be comfortably done only when you run application(s) locally; usually you have to deal with some troubles trying to connect to the applications running on pre-prod environments (test, stage, integration etc.) and I have never seen anyone who established debugging session with prod environment. So if you have to deal with defect that reproduces only on production - you're in troubles :).
• some troubles of debugging multi-node applications. So for example, there is a bug in web application on stage environment, and you're able to establish debugging session with it. But what to do if there are several (2, 4 or 8 or whatever) instances of the application you have to debug, and all requests go through load balancing app? In that case you have to shut down all the app instances except that one that you've connected to, or establish debugging sessions with ALL instances. You can also set up session with just one instance and wait for needed request - but I'm not sure that is very efficient way of debugging.
• troubles debugging in multi-user environment. Let's get back to the previous example - bugged web application on stage environment. Let's say you have just one app instance. After setting up a debug session, every request from any user on stage (manual testers, automated scripts, other developers etc.) is intercepted by your debugger, which is big PITA. There are ways how to deal with it (f.e. "conditional breakpoints"), but they don't address all possible cases.
• step-by-step debugging of multithread functionality is very hard and (sometimes) impossible to perform.

So, what to do? As I mentioned before there are no "magic recipe" that works for everyone. However, when my team was running into multiple environment-specific issues with our applications, I decided to create lightweight debugging framework to make out life easier. Hope it will be useful for someone else too.

DBG4J to rescue!

What is dbg4j? The main idea behind this tool is to have very verbal and extended logging but on demand (so not for every request/functionality etc.). And that is exactly what this tool does.
Some key features:

• collects detailed information on method invocations;
• is managed by debugging activators, so you have full control of when and what to debug;
• annotation-driven - just mark methods with @Debug annotation and you're done;
• different ways of representation of the debugging information - as a part of response, web page logs etc.
• customizable - you can create your own customized debugging components for your needs.

To simplify integration, this tool heavily uses AOP concepts. To integrate with this thing you have to do the next:

1. (optional, but recommended) compile your app with aspectj compiler to support debugging aspect; set-up aspectj viewer;
2. configure debugging options - debugging activators (when to debug), appenders (where and how to represent debugging info) etc. - default implementations can be used oor you can customize any components you want.
3. annotate methods, classes and fields with specific annotations (if you're using AOP approach) or use other debugging tools
4. enjoy!

Detailed information can be found on project's page in github. There is also an example of integration with simple spring application (exposed as separate commit).

I found this library also useful for speedup learning the new stuff that I have to deal with. So when I have to support the application developed by another team, first thing I do is I integrate app with dbg4j, and try to play with application (as a user). Then I analyze dbg4j output  and get myself familiar with what application does under the hood for each functionality it provides. Even if I won't need dbg4j in a future - it's very easy to "un-integrate" it.

And a short demo video at the end:

Have a great day and enjoy coding!

JGit in short example

JGit is the great java library, that provides functionality to perform basic git operations. I used it when I decided to implement release cutting helper for my team. One of the requirements was to make this tool cross-platform (part of the team uses Windows PCs, and another one - Macs), that's why I decided to use java.

The very first search in google pointed me to JGit as "one of the best solutions to work with git from java".

Despite its popularity, it was hard to find some useful code examples for beginners, instead I've found a lot of general descriptions of how cool this library is.

Finally I've got some working examples in stackoverflow, that allowed me to start building my tool.

So trying to fix that situation - listing basic git operations using JGit:

public class GitUtils {
    private Git git;

    public GitUtils(String localPath) throws Exception {
        if(!localPath.endsWith(File.separator)){
            localPath += File.separator;
        }
        git = new Git(new FileRepository(localPath + ".git"));
    }

    public void pull() throws Exception {
        String currentBranch = git.getRepository().getBranch();
        StoredConfig config = git.getRepository().getConfig();
        if(config.getString("branch", currentBranch, "remote") == null){
            config.setString("branch", currentBranch, "remote", "origin");
            config.setString("branch", currentBranch, "merge", "refs/heads/" + currentBranch);
        }
        git.pull().call();
    }

    public void commit(String message) throws Exception {
        git.commit()
                .setMessage(message)
                .setAll(true)
                .call();
    }

    public void push() throws Exception{
        git.push().call();
    }

    public void checkout(String branchName, String sourceBranchName, boolean createNew) throws Exception {
        CheckoutCommand command = git.checkout()
                .setName(branchName)
                //.setUpstreamMode(CreateBranchCommand.SetupUpstreamMode.TRACK)
                .setForce(true)
                .setCreateBranch(createNew);
        if(StringUtils.isNotBlank(sourceBranchName)){
            command = command.setStartPoint(sourceBranchName);
        }
        command.call();
    }
}

Enjoy coding!

Google Guava: missing cache entries

Recently I had to create new caching layer for our web applications. Nothing fancy, just usual multi-level read-through cache. I used Cache from Google Guava library for one of the caching levels, and faced interesting issue. Some tests intermittently failed b/c of the cache data misses, even despite that data was loaded into the cache during test data initialization.

Playing with different cache configurations I found out that data misses happens in a case of full cache population.

Below are the test that demonstrates the issue. It creates Cache, populates it with the random data, then reads the data from the cache and and counts number of misses (google guava v. 14).

public class GuavaTest {

    public static void main(String[] args){

        for(int i = 1000; i <= 50000; i += 1000){
            System.out.println(MessageFormat.format("count = {0} missCount = {1}", i, String.valueOf(test(i))));
        }

    }

    public static int test(final int cacheSize){

        String[] urls = new String[cacheSize];
        com.google.common.cache.Cache<Object, Object> cache = CacheBuilder.newBuilder()
                .maximumSize(cacheSize)
                .build();

        for(int i = 0; i < cacheSize; i++){
            urls[i] = UUID.randomUUID().toString();
            cache.put(urls[i], new Object());
        }

        int missCount = 0;

        for(int i = 0; i < cacheSize; i++){
            Object o = cache.getIfPresent(urls[i]);
            if(o == null){
                ++missCount;
            }
        }

        return missCount;
    }
}

Below are results I've got (they're slightly different every time, but trend and range are approx. the same):

I was really surprised with results. Maybe that's the price for some kind of optimization or something. I'm still using guava cache, it is great tool, but always keep in mind that I have to reserve a bit more slots than required number to avoid data loss.

"I hate Android!" or my first android app

Despite having development experience for different mobile platforms, I have never worked with Android (so shame!). However last month good candidate for my first android app was found!

The company where I'm working has started using 2-step authentication. So every time any employee wants to connect to f.e. VPN, he/she has to:

• sign-in using username and password
• if credentials are valid, system calls to the employee's cellphone and asks to press pound ("#") sign to complete authentication.

This second step is a bit annoying - you have to search your phone every time you want to login. Especially it annoys at home, where my phone usually hides from me in different places. Also if you forgot your phone somewhere (in the car, at home etc.) it is hard to connect to corporate network.

"It's time for Android development"- I have decided to create android app, that should:

1. Pick up the phone
2. Press pound key
3. Hang up
4. Bring the beer (joking :) )

Sounds not very scary (huh, if I knew!), simple enough for the first app. Let's rock!

Before start

Development Tools

I tried to use pre-released version of AndroidStudio, but for some reasons  it didn't work on my OS X 10.8.5. So I fall back to "standard" ADT Bundle. Unfortunately its built-in Eclipse-based IDE has disappointed me - it is slow thing with "bugged" debugger, that often disconnects from emulator without any obvious reasons. I'm not saying it is ugly, but that is definitely worst dev tool for mobile platforms I've ever tried (I'm comparing with dev tools I used for development for iPhone, BlackBerry and Nokia). Looking forward to trying release version of AndroidStudio - I hope guys from JetBrains won't break good tradition and will deliver high-quality product soon.

Documentation and tutorials

Opposite to the toolkit, documentation is very good. Here some good resources for beginners:

• Android Development Tutorial - good overview of android platform, provides basic understanding of android's app and its components.
• UI Overview - must read - provides overview of the main UI components and recommendations how to use them to improve apps' usability.
• Devices and Displays, Metrics and Grids, Iconography - basics about different UI interface aspects.
• Best practices for Performance, Best UX - highly recommended to read.
• Firs app - if you're building your first app
• Android Performance Case Study - very good post about performance of android's apps - even if you do not try to profile your app, article provides basic understanding of the apps' performance and ways for optimization.
• SDK Samples, Google I/O App, photup - learn best practices by reading source code.

Huh, finally - development!!!

Preferences Screen

I didn't want that my app intercept every call, only from some certain number. I also would like to have ability to turn application off. So I started building my preferences view from scratch, but then I realized that there is mechanism building standard preferences screens. I won't pay a lot of attention of how to work with preferences, it is well described here. All that added to it is displaying preference value in the preferences main screen:

    @Override
    protected void onPause() {
        super.onPause();
        // Unregister the listener whenever a key changes
        getPreferenceScreen().getSharedPreferences()
                .unregisterOnSharedPreferenceChangeListener(this);
    }
 
    @Override
    protected void onResume() {
        super.onResume();
        //update phone preference summary with actual value
        SharedPreferences preferences = getPreferenceScreen().getSharedPreferences();
        phoneNumberPreference.setSummary(preferences.getString(PHONE_NUMBER_PREFERENCE, getResources().getString(R.string.phone_number_desc)));
        preferences.registerOnSharedPreferenceChangeListener(this);
    }

    @Override
    public void onSharedPreferenceChanged(SharedPreferences preferences, String key) {
        //update phone preference summary with actual value
        if(key.equals(PHONE_NUMBER_PREFERENCE)){
            phoneNumberPreference.setSummary(preferences.getString(key, getResources().getString(R.string.phone_number_desc)));
        }
    }

Pick up the phone

Surprise - there is no easy way to do it in Android. Traditional approach - use hidden ITelephony API via reflection does not work in v. 2.3+ (details here and here). Finally (after spending many hours trying to find solution) I applied approach used in this great application - I just send bluetooth keypressed event, and it works even if there is no bluetooth headset installed! Magic!

    // Simulate a press of the headset button to pick up the call
    Log.d("AuthenticatorAutoAnswerIntentService.pickUp", "Intent.ACTION_MEDIA_BUTTON Down");
    Intent buttonDown = new Intent(Intent.ACTION_MEDIA_BUTTON);
    buttonDown.putExtra(Intent.EXTRA_KEY_EVENT, new KeyEvent(KeyEvent.ACTION_DOWN, KeyEvent.KEYCODE_HEADSETHOOK));
    context.sendOrderedBroadcast(buttonDown, "android.permission.CALL_PRIVILEGED");

    // trigger on buttonUp instead of buttonDown
    Log.d("AuthenticatorAutoAnswerIntentService.pickUp", "Intent.ACTION_MEDIA_BUTTON Up");
    Intent buttonUp = new Intent(Intent.ACTION_MEDIA_BUTTON);
    buttonUp.putExtra(Intent.EXTRA_KEY_EVENT, new KeyEvent(KeyEvent.ACTION_UP, KeyEvent.KEYCODE_HEADSETHOOK));
    context.sendOrderedBroadcast(buttonUp, "android.permission.CALL_PRIVILEGED");

Press pound key

I was not able to do it programmatically. There is no API to do it. There is open issue to provide it, but it is still open. Here what I tried:

• Send "#" pressed event - no luck (good overview of different ways how to do it is here)
• Use hidden android.telephony.Phone class (as suggested here)
• Call the "#" number using intent (recipe from here)
• During experiments I've found out, that when I put my android device close enough to the powerful sound speaker that plays"#" tone (300ms long), it was recognized by caller as "#" keypress. But when I tried to play "#" DTMF tone (using ToneGenerator) by android audio system (after my application picks up the phone), it didn't work. I think that my phone's sound is just not powerful enough.  However when my app makes android audio to play long DTMF beep (5 seconds instead 300ms), caller hangs up - so it "hears" long beeps, but ignores short ones (again - in a case of external sound source, short DTMF beeps worked well - magic!).

I was not able to make my app send "#" DTMF to the caller, and left "beeps" variant as "almost work" one.

Hang up

The most easy case - that was done by using ITelephony hack, which (in this particular case) for some unbelievable reasons does not need android.permission.MODIFY_PHONE_STATE permission.

Dessert - notification

In addition, application shows notification in the status bar when auto-answer is activated. I found that very useful.

Project Structure

Project lives here

• SettingsActivity - main settings screen
• AuthenticatorAutoAnswerNotifier - main guy who takes care about apps' notifications in the status bar
• AuthenticatorAutoAnswerBootReceiver - starts on device boot complete, all that it does - just calls AuthenticatorAutoAnswerNotifier to update notification
• AuthenticatorAutoAnswerReceiver - receiver that activates on incomming call, checks settings, and in case if app is activated and it is "authenticator" phone number, initiates async call to the auto-answer service.
• AuthenticatorAutoAnswerIntentService - service that does all the "auto-answer and authenticate" work. All process takes several seconds, so we cannot perform it in AuthenticatorAutoAnswerReceiver, which is running in main application thread.