Debugging Kerberos Issues

Kerberos is one of those items that I can’t imagine anyone ever actually enjoys. It’s a necessary evil if you want to have a secure Hadoop cluster setup. Without it, the permissions checks are trivially easy to sidestep.

When you’re running into an issue with an application running on this type of setup, the first thing you’ll want to look at is to turn on debug logging. If you start your application with the parameter:

-Dsun.security.krb5.debug=true

Then you’ll get a whole lot of debug output to stdout. Often issues will come up with ticket renewal, and in most setups, ticket renewal will happen every 24hrs. So it helps to pipe the console log to a file so you can go over it later. An example of how you can start an application to accomplish this is:

nohup ./bin/app-start >> logs/app.log 2>&1 &

This command will start your application and pipe the stdout and stderr to the app.log file. It will also append to the log instead of rewriting it, this way you won’t lose the log on restarts.

When developing applications that communicate with a secure Hadoop cluster, I’ve found it to be helpful to change the default ticket renewal time to be 10minutes. Less than this and you’ll start to see unrelated errors, but this will make it easier to verify that ticket renewal is happening correctly.