Executive summary: Setting up automated UI tests for an GWT/AppEngine application using TeamCity for CI and Cucumber for the tests.

Settle in folks. I’ma feelin’ verbose…

Sometimes this “best tool for the job” schtick is a royal pain in the raccoon when it comes to putting them together. We chose Google Web Toolkit because it is designed specifically for the type of app we want to build. Google App Engine made sense because of its integration with GWT. We went with Ruby/Cucumber for the UI tests because we recognized its power in this area and we were lucky enough to have a resource available to help guide us through the learning curve. For our CI server, we have TeamCity because…well, okay, I have no idea if it’s the best. It works well and I know how to use it. There’s only so much time you can spend for R&D (outside of family reunions, at least).

I’ve been happy with all four pieces. GWT has proved its worth and we’ve been CI’ing for many months, happily compiling and running unit tests. The UI tests have been a bit slower to catch up but they work really well, especially lately since we’ve made a shift towards (what I think is) BDD.

We’ve been running the UI tests against a deployed version of the app on AppEngine, one specifically set aside for UI tests because from experience, trying to show someone the app while the UI tests are constantly clearing and resetting the data makes for a poor demo experience.

Recently, I’ve switched the tests over to run against devmode, which is a way of running a GWT/AppEngine app locally. Usually it’s used for debugging as it allows you to step through Java client code in an IDE.

Why devmode?

Running against a deployed version works reasonably well but has led to a good number of scenarios like:

Given I’m on the user form
And I wait 2 seconds so the watermark has time to load
And I fill in the password
And I press “Save”
And I wait 5 seconds so the screen can finish loading
Then hopefully I’ll see a message at the top. But even if you do, mark the test failed anyway ‘cause the hillbilly has nothing better to do than rerun failing tests and watch them pass locally.

(<Random deity> bless Capybara for allowing me the freedom to write tests to express my *real* intent.)

That is, the tests needed to be told to pause every once in a while to allow things time to load. This isn’t to say the app is slow, just that computers are fast. And until we find a way to instantaneously populate a cascading dropdown, computers will continue to have an edge.

Devmode, on the other hand, is crazy fast if you don’t have it configured for debugging (i.e. you don’t add the gwt.codesvr parameter to the URL). It requires you to GWT compile the app first but hey, it’s a CI server; it can wait the extra three minutes to do that.

Another issue with this: using a deployed version requires the latest code to be deployed first. Granted, we have automated this in the CI server but it still adds fifteen to forty-five minutes to the process depending on how much bandwidth GoDaddy deems appropriate for our server on any given day.

Finally, there are quotas to consider. Every action on our application is logged and we have to be mindful of how much CPU time we’re using, how much bandwidth, how many emails get sent out, etc. At this stage, it’s unlikely we’ll reach those quotas but an errant test could potentially have costs associated with it.

So between the slow tests, the deployment time, and the potential cost, an obvious solution was to run the tests in devmode.

Hosted mode/devmode

If you create a GWT app with the built-in webAppCreator, you get a nice little Ant build file with a target that launches the app in devmode (nee hosted mode). I had to do some finagling to get it to work though. Here’s the final target in Ant (formatted to fit the width):

<target name="devmode" depends="" description="Run development mode">
  <java fork="true" classname="com.google.gwt.dev.DevMode" 
    dir="${basedir}/war" spawn="true">
    <classpath>
      <pathelement location="src" />
      <path refid="project.class.path" />
      <path refid="tools.class.path" />
    </classpath>
    <jvmarg value="-Xmx512M" />
    <jvmarg value="-javaagent:${appengine.folder}/
          lib/agent/appengine-agent.jar" />
    <jvmarg value="-Duser.dir=${basedir}/WAR" />
    <arg line="-war" />
    <arg value="${basedir}/war" />
    <arg line="-logLevel" />
    <arg value="INFO" />
    <arg value="-server" />
    <arg value="com.google.appengine.tools
        .development.gwt.AppEngineLauncher" />
    <arg value="net.bookedin.bam.BAM" />
  </java>
</target>

Most of this you can glean yourself if you look at the arguments tab of the GWT run Configconfiguration in Eclipse. Some notes on discrepancies ‘twixt this and the default one:

AppEngine support

The second <jvmarg> and the second and third last <arg> values are included to support AppEngine. In some recent upgrade to AppEngine, the javaagent JVM argument was added.

Changing the working directory

If you leave the working directory as is, you’ll get a warning that the working directory doesn’t match the WAR directory. In most cases, this won’t matter. For us, we have files we wish to access so we need the working directory to match in order to access them.

To do this, we added the dir attribute to the <java> task and added the last <jvmarg> argument. We also updated the argument value for the –war parameter.

Logging

When you launch the app, the logLevel parameter needs to match what’s in your web.xml file.

Spawn=”true”

I’ll come back to this.

Running the UI tests

This wasn’t much of an issue. We run our UI tests with cucumber in Ruby. From what I can tell, it’s a standard implementation. And TeamCity supports it natively. Set Rake as the runner and run your cucumber task like you normally would.

Killing dev mode

The reason we added spawn=”true” to the <java> task above is to allow the Ant task to actually finish once devmode is launched. Without it, the process waits until devmode is finished before allowing you to continue. Which means it won’t go on to the next step in the CI process: running the UI tests.

Because of this, we needed a way of killing devmode after it had been started:

<target name="kill">
      <exec executable="taskkill">
          <arg value="/fi"/>
          <arg value='"Windowtitle eq GWT Development Mode"'/>
      </exec>
</target>

 

Putting it together

This was harder than one might expect and we still don’t quite have it right. We now have three TeamCity build configurations, one for each step above:

  • Launch devmode
  • Run tests
  • Kill devmode

“Run tests” is configured to run upon a successful “Launch devmode” build. And “Kill devmode” runs upon any build, successful or otherwise, of “Run tests”.

Note that as of TeamCity 6, you can have multiple build steps per configuration. And indeed, my first pass at this was one configuration with three build steps. But as per the documentation: If a build step fails the rest are not executed.

This automatically means “Kill devmode” has to be a separate configuration because I don’t want devmode hanging around active if any of the tests fail. (Alternatively, I could configure the build not to fail if any tests fail but…c’mon!)

The reason we separated the first two steps is that the Java code lives in a separate repository than the UI test scenarios. If we combine the two steps, then we need to pull both repositories. This is fine but I also want to configure the build to launch when code is checked in on only one of them, not both. By separating the steps, the first runs on one repository and the second on another. So I can configure the first to run whenever any Java code is checked in.

But this doesn’t work either. The UI tests take about forty minutes to run (which, I should note, is half an hour shorter than they did against the deployed version of the app). If any code is checked in during that forty minutes, “Launch devmode” will kick off again and fail miserably.

That said, I’ve just now thought of a better solution that I will try the minute I click Publish. We already have a build configuration that runs the unit tests. I’m going to combine the first two steps into a single configuration again and make it contingent on a successful build of that configuration. That way, even if someone checks in code while the UI tests are running, it won’t launch devmode until the UI tests are done because they’re both part of the same configuration.

I’m skipping a whole lot of details here so if you’re interested, ping me in the comments or offline.

As Joe Jackson says in Happy, Loving Couples: Right, that’s enough.

Kyle the Configurated