Bringing back the blog

Hiking with the kids

It’s been a while.

Life has been crazy.

My focus these days has been on the kids and making the most of their education while we’re all trapped sheltering in place. It’s a very interrupt-driven environment, which means that it’s tough to sit and focus on development work. So, I figured I’d do a little easy sysadmin work and get the blog back in decent shape.

This site has been living on a t2.small AWS instance for years. I had used a Bitnami installer to package up a whole WordPress installation. It was super easy to set up but was getting harder to maintain. The VM was running Centos 6, which is just prehistoric these days, and with apache, PHP, WordPress, and MySQL all bundled, it was hard to make sure that everything was up to date.

So, I broke everything back apart and did clean installs. Got the OS upgraded to the latest Ubuntu, which is dramatically easier to keep up to date. I moved off of Mysql and instead used AWS RDS Aurora. Did a clean vanilla install of WordPress.

One last thing that I had wanted for a long time so that I could take snapshots of the server and run multiple instances if necessary, was to move the media libraries and other files to S3 and use a CDN. This turned out to be easier than I had expected. Just required 2 CloudFront distributions for all of the sites hosted on this installation.

Now, I’m ready to get back into the habit of posting some thoughts and ideas. Not that I think anyone will read or care, but I’ve got to keep practicing writing.

Configuring a spring application to use https behind an ELB

I have a couple of sites that run on AWS and take advantage of Elastic Load Balancers to handle proxying to servers in a private VPC as well handling https security. While CloudFront distributions can be configured to upgrade requests to https if they come in as HTTP, I haven’t found a way to configure ELB connections to do the same. What I wanted is for the ELB to listen on both port 80 and 443 and if a connection comes in on 80, then to redirect to the same request on 443. This way if a user just enters the URL into their browser, they’ll automatically be upgraded to https even if they didn’t remember to specify it.
It’s fairly simple to accomplish this with an Interceptor. Create a class that will handle the redirects.
 

@Component
public class SecurityInterceptor extends HandlerInterceptorAdapter {
  private static Logger logger = LoggerFactory.getLogger(SecurityInterceptor.class);
  public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
    String proto = request.getHeader("X-Forwarded-Proto");
    if (!Strings.isNullOrEmpty(proto)) {
      logger.debug("Found proto: {}", proto);
      if ("http".equalsIgnoreCase(proto)) {
        StringBuffer redirectString = createRedirect(request);
        logger.debug("Redirecting to {}", redirectString);
        response.sendRedirect(redirectString.toString());
        return false;
      }
    }
    logger.debug("No proto header found");
    return true;
  }
  private StringBuffer createRedirect(HttpServletRequest request) throws IOException {
    StringBuffer result = request.getRequestURL();
    result.insert(4, "s");
    String queryString = request.getQueryString();
    if (!Strings.isNullOrEmpty(queryString)) {
      result.append("?");
      result.append(queryString);
    }
    return result;
  }
}

Since encryption is handled by the ELB and all traffic arrives at the server as HTTP we have to rely on the x-forwarded headers to know if we should redirect the request back to a secure page.
Then I have a config class to load the Interceptor.

@Configuration
public class WebMVCConfig extends WebMvcConfigurerAdapter {
  @Autowired
  private SecurityInterceptor securityInterceptor;
  @Override
  public void addInterceptors(InterceptorRegistry registry) {
    registry.addInterceptor(securityInterceptor);
  }
}

This Interceptor is specific to the x-forwarded headers that ELBs will send to the backend instance.
 

Hiring for culture fit can have unintended consequences

An interesting post on “culture fit.” I’ve been catching up on the SAASTR podcasts, and a lot of the interviewees sound like they’re falling into this trap. I agree with Matt’s assertion that by trying to hire people that you would want to hang out with, you will end up hiring a lot of people that look like you. Instead, you should focus on culture agnostic characteristics:

  • Are they kind and empathetic?
  • Do they care about their work?
  • Do they have good communication skills?
  • Do they have good self-management skills?

Finally, also look for “Do they bring a valuable new perspective?”

Terminal hack to make logging into a cluster easier

I’ve been looking for a way to make it simpler to quickly login to all nodes of a cluster from a mac terminal. Discovered a small npm module that helped me do it called ttab. With this npm module I can then write a bash script:

#!/bin/bash
TYPE=$1
rs6 () {
ttab -w ssh bryan@example1.com
ttab ssh bryan@example2.com
ttab ssh bryan@example3.com
ttab ssh bryan@example4.com
ttab ssh bryan@example5.com
ttab ssh bryan@example6.com
}
if [ “rs6” = $TYPE ]
then
rs6
fi

This will open one new terminal window with 6 tabs all logged into the different servers.

Spring Boot with JSP and React Template Application

Awhile back I was playing around with moving some personal applications to use spring boot. There are a lot of really nice aspects to this project. You get a whole lot of functionality without having to write boilerplate code.

  1. Executable jar file.
    1. packages entire application as a single executable file.
  2. Service Support
    1. link to your executable jar from /etc/init.d and you get full Linux service support (start, stop, restart, status)
    2. logs go to /var/log/{application name}.log
    3. PID status support in /var/run/
  3. Embedded web container
    1. tomcat or jetty is upgraded with your spring boot version, so you don’t have a separate set of binaries to maintain.

The biggest downside that I could see was that JSPs are not supported out of the box because of a tomcat issue. Spring boot is perfect for developing REST microservices but somewhat limited as a general purpose web application replacement. You need to convert your display code to use Thymeleaf templates, which could be a significant amount of work.
I came across a workaround to get JSPs working as expected. It involved moving files out of the /WEB-INF/ directory and putting them in /resources/META-INF/resources/WEB-INF/. It’s a little weird, but when you do it, everything works as expected. You’re even able to use things like JSP tags just as you would have before.
Another thing that I’ve been spending time thinking about is how to integrate a Java development project with a current Javascript setup. Maven works as well as anything for a large multi-module structure, but how does the Javascript code incorporate? There are a lot of different options. I wanted a build pipeline that would allow you to use the more popular Javascript tools, not break for non-Javascript developers, and allow fast iteration of Javascript changes (doesn’t require a full ‘mvn clean package’ to see changes).
After all of this experimentation, I believe I’ve found a good balance of tools for both Java and Javascript developers. It uses Spring Boot with full JSP support as a base. Gulp and Webpack for the Javascript build pipeline. Has a development mode that uses the Webpack dev server to iterate quickly. It also includes support for development with React, so you can play with the cutting edge of developing web applications.
I’ve done all of the fiddlings with different settings and created a working template application.  Check it out and let me know what you think.

Changing Kerberos Expiration To Test Ticket Renewal

If you’re testing a Kerberos enabled hadoop cluster and want to make sure that ticket renewal is working as expected, you’ll probably want to change the ticket renewal time so that you don’t have to wait 24 hours for each test.
Using a “krb5-server” as an authentication source for a Hadoop Cluster. You can run the following commands to change the default ticket lifespan. 
kadmin.local: getprinc krbtgt/EXAMPLE.COM@EXAMPLE.COM

Principal: krbtgt/EXAMPLE.COM@EXAMPLE.COM
Expiration date: [never]
Last password change: [never]
Password expiration date: [none]
Maximum ticket life: 1 days 00:00:00
Maximum renewable life: 365 days 00:00:00
Last modified: Wed Nov 19 00:09:37 UTC 2014 (quick/admin@EXAMPLE.COM)
Last successful authentication: [never]
Last failed authentication: [never]
Failed password attempts: 0
Number of keys: 7
Key: vno 1, aes256-cts-hmac-sha1-96, no salt
Key: vno 1, aes128-cts-hmac-sha1-96, no salt
Key: vno 1, des3-cbc-sha1, no salt
Key: vno 1, arcfour-hmac, no salt
Key: vno 1, des-hmac-sha1, no salt
Key: vno 1, des-cbc-md5, no salt
Key: vno 1, des-cbc-crc, no salt
MKey: vno 1
Attributes:
Policy: [none]

To change the default ticket expiration from 1 day to 30 minutes, issue the following command:
kadmin.local:  modprinc -maxlife “30 minutes” krbtgt/EXAMPLE.COM@EXAMPLE.COM
You should then be able to verify that the settings have taken effect:

[root@host ~]# kinit -k -t hdfs.keytab hdfs
[root@host ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: hdfs@EXAMPLE.COM
Valid starting     Expires            Service principal
08/22/16 20:24:18  08/22/16 20:54:18  krbtgt/EXAMPLE.COM@EXAMPLE.COM
renew until 08/29/16 20:24:18

You should see that the “Expires” time is 30 minutes in the future. This keytab will be renewed every 30 minutes for 7 days.

The interesting point that isn’t well documented is that there is a hierarchy to the settings in Kerberos. You can modify each individual principle’s maxlife and maxrenewlife, but if a higher level principle has stricter settings then they will be used. The krbtgt principle is the top level principle. Changes made here will apply to all other principles.

 

Debugging Kerberos Issues

Kerberos is one of those items that I can’t imagine anyone ever actually enjoys. It’s a necessary evil if you want to have a secure Hadoop cluster setup. Without it, the permissions checks are trivially easy to sidestep.
When you’re running into an issue with an application running on this type of setup, the first thing you’ll want to look at is to turn on debug logging. If you start your application with the parameter:

-Dsun.security.krb5.debug=true

Then you’ll get a whole lot of debug output to stdout. Often issues will come up with ticket renewal, and in most setups, ticket renewal will happen every 24hrs. So it helps to pipe the console log to a file so you can go over it later. An example of how you can start an application to accomplish this is:

nohup ./bin/app-start >> logs/app.log 2>&1 &

This command will start your application and pipe the stdout and stderr to the app.log file. It will also append to the log instead of rewriting it, this way you won’t lose the log on restarts.
When developing applications that communicate with a secure Hadoop cluster, I’ve found it to be helpful to change the default ticket renewal time to be 10minutes. Less than this and you’ll start to see unrelated errors, but this will make it easier to verify that ticket renewal is happening correctly.

Trying out AWS Workspaces

I’ve wanted to get my hands on a Linux laptop again. I used to work on one all the time, but most of my work these days is on a MacBook Pro. I’ve got a great Thinkpad sitting at home, but it’s running win7 because there are a couple of applications that I need that require windows.
Over the holiday break, I started playing with AWS Workspaces. For $35 / month I can have a win7 VDI setup.  Along with workspaces, I’ve started testing Zocolo, Amazon’s file syncing solution.
So far, I’m incredibly pleased with the solution. Once I’ve got everything up and running in the workspace, I’ll be able to reinstall Linux on the Thinkpad and have the best of both worlds.