Changing Kerberos Expiration To Test Ticket Renewal

If you’re testing a Kerberos enabled hadoop cluster and want to make sure that ticket renewal is working as expected, you’ll probably want to change the ticket renewal time so that you don’t have to wait 24 hours for each test.

Using a “krb5-server” as an authentication source for a Hadoop Cluster. You can run the following commands to change the default ticket lifespan. 

kadmin.local: getprinc krbtgt/EXAMPLE.COM@EXAMPLE.COM

Principal: krbtgt/EXAMPLE.COM@EXAMPLE.COM
Expiration date: [never]
Last password change: [never]
Password expiration date: [none]
Maximum ticket life: 1 days 00:00:00
Maximum renewable life: 365 days 00:00:00
Last modified: Wed Nov 19 00:09:37 UTC 2014 (quick/admin@EXAMPLE.COM)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 7
Key: vno 1, aes256-cts-hmac-sha1-96, no salt
Key: vno 1, aes128-cts-hmac-sha1-96, no salt
Key: vno 1, des3-cbc-sha1, no salt
Key: vno 1, arcfour-hmac, no salt
Key: vno 1, des-hmac-sha1, no salt
Key: vno 1, des-cbc-md5, no salt
Key: vno 1, des-cbc-crc, no salt
MKey: vno 1
Attributes:
Policy: [none]

To change the default ticket expiration from 1 day to 30 minutes, issue the following command:

kadmin.local:  modprinc -maxlife “30 minutes” krbtgt/EXAMPLE.COM@EXAMPLE.COM

You should then be able to verify that the settings have taken effect:

[root@host ~]# kinit -k -t hdfs.keytab hdfs
[root@host ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: hdfs@EXAMPLE.COM
Valid starting     Expires            Service principal
08/22/16 20:24:18  08/22/16 20:54:18  krbtgt/EXAMPLE.COM@EXAMPLE.COM
renew until 08/29/16 20:24:18

You should see that the “Expires” time is 30 minutes in the future. This keytab will be renewed every 30 minutes for 7 days.

The interesting point that isn’t well documented is that there is a hierarchy to the settings in Kerberos. You can modify each individual principle’s maxlife and maxrenewlife, but if a higher level principle has stricter settings then they will be used. The krbtgt principle is the top level principle. Changes made here will apply to all other principles.

 

Debugging Kerberos Issues

Kerberos is one of those items that I can’t imagine anyone ever actually enjoys. It’s a necessary evil if you want to have a secure Hadoop cluster setup. Without it, the permissions checks are trivially easy to sidestep.

When you’re running into an issue with an application running on this type of setup, the first thing you’ll want to look at is to turn on debug logging. If you start your application with the parameter:

-Dsun.security.krb5.debug=true

Then you’ll get a whole lot of debug output to stdout. Often issues will come up with ticket renewal, and in most setups, ticket renewal will happen every 24hrs. So it helps to pipe the console log to a file so you can go over it later. An example of how you can start an application to accomplish this is:

nohup ./bin/app-start >> logs/app.log 2>&1 &

This command will start your application and pipe the stdout and stderr to the app.log file. It will also append to the log instead of rewriting it, this way you won’t lose the log on restarts.

When developing applications that communicate with a secure Hadoop cluster, I’ve found it to be helpful to change the default ticket renewal time to be 10minutes. Less than this and you’ll start to see unrelated errors, but this will make it easier to verify that ticket renewal is happening correctly.

Trying out AWS Workspaces

I’ve wanted to get my hands on a Linux laptop again. I used to work on one all the time, but most of my work these days is on a MacBook Pro. I’ve got a great Thinkpad sitting at home, but it’s running win7 because there are a couple of applications that I need that require windows.

Over the holiday break, I started playing with AWS Workspaces. For $35 / month I can have a win7 VDI setup.  Along with workspaces, I’ve started testing Zocolo, Amazon’s file syncing solution.

So far, I’m incredibly pleased with the solution. Once I’ve got everything up and running in the workspace, I’ll be able to reinstall Linux on the Thinkpad and have the best of both worlds.

SSL Test Sites

Now that security on the net is becoming more of a concern, here are two websites for testing the soundness of your web connections. SSL/TLS is a negotiation between client and server, so you need to see what settings each endpoint will accept.

For the Server side, there is SSL Labs. With this site, you can put any URL in and get a score for how well configured the settings of that server are.

For the client side, there is How’s My SSL. This site will give you a readout on the browser that you’re currently using.

Doing some drive characterizations

As an Engineer at a storage company, I’m often working to characterize how different drives will perform in an environment. As you go down the stack from application to OS to hardware, there are a lot of different factors that come into play. It’s amazing to see what types of differences in performance, you’ll see with varying drives and workloads.

Here are some example results from a testbed looking at a single Seagate 15K 600G drive connected to an LSI 2008 HBA on a CentOS 6.5 machine.

The Random Read tests are using fio with 100% random reads of the specified block size and queue depth against the entire raw drive. There is no filesystem caching in these tests. The Random Write test is the same, but with 100% writes. The Mixed tests are 65% read and 35% writes.

randomReadThroughput

randomWriteThroughput

mixedThroughput

randomReadIOPs

randomWriteIOPs

mixedIOPs

readLatency

writeLatency

mixedLatency