Dja miss me? Of course not. You’re reading this in an RSS reader and if I hadn’t asked the question, you wouldn’t have noticed I was gone. If you’re like me, you’re scanning through until you can find any reason to skip this post and move on to the next one. Here, I’ll help your little mental filter:

Executive summary: An example of when UI/integration tests succeed where unit tests don’t.

But I’m warning ya, you’ll miss out on the amusing delivery if you go by this alone.

Pre-amble

We are tantalizingly close to coming out of trial mode with BookedIN. My business partner had a nice little debate for our development process after its out. His stance: we need to write UI tests before we write code to avoid a lot of the issues that were arising. My stance: We unit test the ever-living raccoons out of the app now (thanks in no small part to Jukito). And we do have a UI testing process that was used for major features. While I agreed that we should have more, I was leery that it would make us any more productive. And I doubted it would help us discover any more issues than we already were…

(Ed. note: Did you notice the ellipsis at the end of that last paragraph? That’s what we in the legitimate journalism field call foreshadowing. Or, as is more often the case in my writing, foreboding. Note that you won’t see classy tropes like this from those hacks at Los Techies.)*

The bug

When someone logs into the application, there are a number of notifications that could pop up in various scenarios:

  • Owner logging in for the first time: Enter company details, following by a welcome message
  • New staff member: welcome message
  • Owner after the first time: You haven’t verified your email address yet (assuming it hasn’t, in fact, been verified)
  • Anyone after the trial period has expired: Message that says “Just kidding, you don’t have to pay! Tee hee to those that did.”
  • Existing users after a major release: What’s new!

There might be more, you can’t expect me to keep all this &*^% in my head.BookedIN

We covered all of this with a NotificationController. It would determine what conditions were met and start piling messages into a queue and display them in order. All of it very thoroughly tested with unit tests to make sure repositories were called and messages sent and users updated and so on and so forth.

The bug as listed was: When they log out and re-log in, users are seeing the what’s new message again.

As with most bugs, this was an oversimplication. As I tested, I discovered that this would happen intermittently. Sometimes it would happen as stated. Sometimes a new owner would enter her company details, log out, log in again, then see the company details window once more. Sometimes it would just work.

The analysis

“How can this be?” says I. The tests clearly outline what is supposed to happen and the code clearly passes all the tests.

As mentioned, we have a UI testing framework in place and used it on occasion. The reason it is not used more frequently is a topic that is already half-written but the short answer is: Google Web Toolkit adds some hurdles we haven’t had time to overcome yet. But given our recent debate, I fired up Notepad and banged out a capybara scenario. Nine times out of ten, it would fail. Except when I ran through the debugger in which case ten times out of ten, it would succeed.

The culprit

A good, old-fashioned race condition. In our client piece, the code for the first notification scenario above reads like this (it’s in pseudo-code because I don’t actually code anymore…):

if (logging_in_for_the_first_time) {
    showWelcomeMessage( );
    markWhatsNewMessageAsSeen( );
}

Judicious use of logging revealed the following exchange on the server:

[MarkingWelcomeMessageSeen] Starting execution
[MarkingWelcomeMessageSeen] Retrieving user details
[MarkingWelcomeMessageSeen] Marking welcome message seen
[MarkingWhatsNewMessageSeen] Starting execution
[MarkingWhatsNewMessageSeen] Retrieving user details
[MarkingWhatsNewMessageSeen] Marking what’s new message seen
[MarkingWelcomeMessageSeen] Saving user
[MarkingWhatsNewMessageSeen] Saving user

After making a mental note to perhaps differentiate my method names a little better, it was painfully obvious what the problem was.

The solution

That’s not the topic of this post. Focus, Hillbilly!

The moral

As I said, all unit tests passed. Our tests on the client proved that we were marking the messages seen in the appropriate conditions. Tests for the server component show that we are calling repositories and setting properties (or the Java equivalent: setting properties, but with more code) properly.

But the problem didn’t manifest itself until we did the UI test (or integration test, if you prefer).

So folks, heed my warning. A lack of decent integration/UI tests can lead to a smug and self-righteous business partner.

Kyle the Humbled

* shiny nickel for those of you who recognize this little bit being ripped off from Bloom County