Jeff's Blog

Too Much Incremental Refactoring

2008-08-14T11:02:00.007-05:00

Having reasonably comprehensive tests is great. I currently have a good-sized codebase for an application I'm working on. In one key subsystem, representing about 1/6 of the whole application so far, there are about 25 classes and 86 code-level tests. Most of the classes are small, one to three methods. Coverage is in the very high 90s. Sixteen of the 86 tests are necessarily "slow" tests (centered around persistence), so I run them in a separate suite. Combined, they take 3/4 of a second; the "fast" tests run almost instantly.

I've been very happy with the design and implementation of this subsystem so far, with the exception of one central class for which the tests are a bit ugly. Up until now, I've been able to introduce significant new features in very short order, with minimal shotgun surgery and no fear of breaking anything (the sixteen integration tests are a key part of eliminating this fear).

But I just introduced one would-be small change that introduced significant pain. There were two causes of the pain: one, I chose to use an array to represent one significant collection, and two, I chose to expose that implementation through parameter passing. Why did I choose an array? Well, it represented a fixed collection of data for a class, which means it was easiest to code as an array (think initialization). It also made for effective tests, particularly given the expressiveness of varargs.

My new change pushed me to prefer a parameterized collection instead of an array. So I bit the bullet and figured I'd just incrementally refactor my way away from the array. Worked great! In an incremental fashion--with no one step taking more than 5 minutes before getting a green bar (and most taking 30 seconds)--I got to my desired solution steady and sure. I made a couple simple mistakes along the way, and relished the way the red bar immediately informed me of my stupidity. The problem, though, was that there were far too many of these wonderfully short steps, adding up to about an hour of work. "Ok, made the change here...ohhh, there, it ripples to a half dozen other methods!"

The lesson that I was reminded of was that I would have been much better off having encapsulated the collection early on. Instead of passing about Widget[] or List<Widget>, this change would have been considerably simpler had I created a type named WidgetList. I would have had to expose some overloaded "bridge" methods while I fixed the tests that used varargs, but in 20/20 hindsight, the change would have taken me maybe 10 minutes.

Of course I knew better, even in the heady moments of thinking the arrays made things a bit simpler. There's plenty of literature that covers this, and I've read it. I've chastised others for the same thing plenty of times, even recently. But for whatever reason, I didn't bother. Laziness, not paying enough attention, not caring enough. Of course it's a violation of encapsulation, but technically I can also look at my problem as having allowed the duplication of my choice for the collection's implementation to run rampant throughout the system.

A pair would probably have squashed the potential headache early on. Maybe. Take a look at your code--where are you passing around exposed collections?

Request for a Blog Pair

2008-07-29T09:11:00.005-05:00

I encountered a bit of code recently that screams out for refactoring. Problem is, I'm not sure of the most effective way to do this.

The context: this is a tic-tac-toe game that Paul Nelson and I paired on, in order to explore mobile device programming (Paul has an iPhone, I have a BlackBerry). Paul's not here right now to pair, unfortunately, hence the blog request (a really slow way of virtual pairing).

Here's the relevant code:

private boolean isRowWinner(int row) {
   for (int column = 0; column < SIZE; column++)
      if (grid[row][column] != currentPlayer)
         return false;
   return true;
}

private boolean isColumnWinner(int column) {
   for (int row = 0; row < SIZE; row++)
      if (grid[row][column] != currentPlayer)
         return false;
   return true;
}

private boolean isDescendingDiagonalWinner() {
   for (int i = 0; i < SIZE; i++)
      if (grid[i][i] != currentPlayer)
         return false;
   return true;
}

private boolean isAscendingDiagonalWinner() {
   for (int i = 0; i < SIZE; i++)
      if (grid[i][SIZE - 1 - i] != currentPlayer)
         return false;
   return true;
}

The field grid is simply a 3x3 matrix of Player references; Player is an enum; currentPlayer is a Player reference.

What's the most effective way to simplify these four methods, seemingly rampant with duplication? I'm not looking for a "clever" solution; I'm looking for one that still retains high levels of expressiveness.

UPDATE: tests and code

Scrumalicious

2008-07-24T15:08:00.002-05:00

The Scrum Yahoo! group is again undergoing some minor turbulence about proper use of the list. Someone from the Scrum Alliance laid out the law with a number of rules about what could and could not be posted to the list. Many people reacted poorly--supposedly this is the poster's first post, and he's not even a Certified Scrum Master (CSM). (gasp!)

Oddly, a year or so ago when there was similar agonizing over the use of the list, it appeared as if L. Ken Schwaber "owned" the Yahoo! group and thus controlled (an important word in the Scrum community) appropriate content. Not that there's anything wrong with that. For those who like command & control, with the additional potential of being part of a good multi-level marketing machine, the Scrum system provides a wonderful starting point.

The nice thing is that you can still do Scrum and talk about Scrum without succumbing to the controlled community that is S©rum. As far as I know, there's no licensing scheme required to be able to say, "I'm doing Scrum." And that's exactly what I'd recommend. No annual maintenance fee required! No required trainer tiering! Yes, you need a Scrum Master in order to successfully execute Scrum. But I don't believe they must be "certified." (I could be wrong--perhaps I need a lawyer here!)

Where might you find a good Scrum Master? Well, you should probably read L. Ken Schwaber's book, Agile Project Management With Scrum (that link is my piece of the Scrum machine's gravy train, by the way). You'll quickly discover, through a number of case studies that L. relates, that Scrum and S©rum are mostly a matter of executing to common sense, best done by someone with good experience in people-oriented problem solving and leadership. Find someone who can do that well, and hire them. It really doesn't matter if they're a CSM.

The Realities of Test-After Development (aka Why POUT Sucks)

2008-07-22T14:19:00.008-05:00

"Plain Old Unit Testing," or "POUT," implies that developers are coding unit tests, but they are not doing test-driven development (TDD). In other words, they are following the classic pattern of writing the code, then coming back and coding unit tests to verify the correctness of the code. Instead of POUT, I prefer the acronym TAD, or Test-After Development, which is always a "TAD" too late.

Why does it matter whether tests are written first or last? Theoretically, someone doing TAD could easily produce as high-quality a system with as comprehensive test coverage as produced via TDD. In reality, however, it never happens.

Here are some of the reasons why TAD sucks:

Doesn't promote a testable design	Most of the time, testability is in concert with notions of "good," or sustainable, design (coupling and cohesion and all that). Usually, developers discover too late that their design isn't testable. Most often in this case, they quickly give up on trying to test the code, Over time, sure, developers figure out how to produce a more testable design out of the gate, but it's still never a certainty. The real problem is that almost no developer will go back and fix an untestable design in order to be able to write tests.
Is undisciplined	No rule says that a developer must write tests. Good developers might write enough (usually they do not), but there are plenty of not-so-good developers that will use the lack of a standard as a quick excuse to write as few tests as possible.
Falls victim to ego	"My sh*t don't stink, I don't need to bother with writing unit tests, so I'll do the fewest that I can get away with." Enough said. This attitude doesn't belong in a professional development team, but it's sadly common.
Appears unnecessary	"I already proved that the code works, I loaded it up in the app server and tried it myself. Why would I bother writing unit tests now?"
Clashes with deadlines	When it comes down to deadlines, managers will quickly say, "Never mind with the unit tests, we just have to ship this software."
Decreases average coverage	For many of the reasons already stated, the practical coverage ceiling for TAD is about 70%. This is based on lots of anecdotal evidence, often stated by TAD proponents themselves. I met with Agitar reps about 18 months ago; they confirmed that their TAD-tool (which generates unit tests automatically) has a comparable ceiling. No matter how TAD tests get there, the result is that up to 30% of your app remains unverified at the unit level. Maybe that's not such a big deal if you have higher-level (integration) tests, but...
Doesn't afford adequate refactoring	If up to 30% of your app is not well-covered, that means you cannot safely and effectively refactor almost a third of your application. A third of your system will progressively get worse almost by definition.
Results in difficult tests	Many TAD tests are closer to integration tests in nature. "Who cares," say some of the TAD proponents, "about the distinction between unit tests and integration tests?" I don't care about the semantic distinction either, but integration tests are harder to maintain, harder to comprehend, harder to use to pinpoint a problem, and slower.
Isn't very enjoyable	Few people enjoy writing unit tests; many people will claim that they love TDD. Honestly, it's not very satisfying to spend time on unit tests when you're pretty sure the code already works.
Questionable return on value	TAD is expensive. If you have to choose between TAD and more end-to-end functional tests, dump the TAD.
Does nothing to advance the craft	TAD is a haphazard approach. TAD proponents insist that we're all smart enough to figure out what should be verified at the unit level, and what can be safely ignored. Reality tells us otherwise, but still, that choice may be acceptable in some shops. However, the industry has been screaming for a more disciplined approach to software development that can consistently produce higher code quality. Perhaps TDD isn't it, but TAD most certainly ain't it.

A TDD Contest

2008-07-21T12:55:00.003-05:00

I love competition, and I suspect the majority of programmers do, too. I've met only a few very good programmers who weren't into games or other forms of competition. I've also known a few developers whose egos exuded in everything they did, whether or not they'd admit it. These are the kinds who want to argue virtually every point that is made, not that there's anything wrong with that. (Well, maybe there is--no one wants to feel like every one of their comments is possibly wrong.)

In listening to someone else's contrarian nature take over, I often sit there and think, "We should just argue our respective positions in code." Of course, I'm also thinking that "I can code this better than you can."

The site TopCoder provides a place where money can be put in someone's big mouth, but it's not what I'm looking for. TopCoder is more a highly competitive marketplace, a great idea that appeals to my capitalistic nature. But it doesn't appeal to the ego in me, nor does it appeal to my interest in TDD.

Rather, I'd prefer a competition that offers bragging rights. Not that I need to brag, but I would love some validation to my claims. I think as geeks we all like numbers, and I'd like to know if I am the best, or if I am only the 17th best.

So, a contest. Who is the most effective coder? Here are my thoughts for such a contest:

Devise a problem whose test-driven solution might take about 90 minutes
Insist that the problem be solved using some sort of solution recorder (maybe an IDE plugin); either that, or provide a proctoring mechanism. Unfortunately I'm not sure what the implication and cost of this is.
Award bonus points for early completion; subtract bonus points
Deduct points for deficiencies in solution--e.g. lack of test/code expressiveness, unnecessary duplication, etc.
Score tests using a panel of three or more judges

Could such a contest be used to determine a champion between TDD proponents and the rest of the world?

If I can get enough interest, I'll happily run such a contest and offer up some kind of prize, although I'd just as soon participate in one. Step one involves getting feedback--are there enough fearless programmers who would participate? How would the judges come to at least some consensus on judging criteria?

Are you the best? Do you have any ideas about how to judge such a competition? Let me know.

Fine-Grained Incrementalism

2008-07-10T09:16:00.006-05:00

Seeing a first-time demo of TDD, students inevitably ask, "do you really code that way?" The answer is yes, even after almost ten years of doing TDD.

They are of course asking, do I always code the simplest possible solution first, instead of just introducing the obvious solution, the one that I know I'm absolutely going to need in another five minutes? And the answer is still yes.

For example, when testing a container, the first test inquires whether or not it's empty:

assertEquals(0, container.size());

and the passing implementation is:

int size() {
   return 0;
}

The next test usually involves adding an element to the container and asserting its size (which is of course only part of the complete verification):

container.add(element);
assertEquals(1, container.size());

Yes, I know I need a [hash map|array list|linked list|etc] in about 5 minutes. I still code:

void add(T element) {
   count = 1;
}

int size() {
   return count;
}

Not even a ++. That comes from writing a subsequent, failing assertion.

I offer many reasons:

It helps deeply ingrain the notion of incrementalism, which is at the heart of agile
It allows sustaining the fail-first, red-green rhythm, and thus keeps you out of trouble more often
Similarly, it allows adhering to the rule that insists on green bars every ten (five) minutes
It forces the addition of more assertions
It keeps with the spirit of YAGNI--sometimes you don't end up needing what you think you will

This fine-grained incrementalism involves continual movement from the specific to the generalized: from hardcoded solutions to progressively-more-generalized solutions.

Fine-grained incrementalism is, as many things in software development, a tradeoff. It results in continual waste product, and thus a slower initial pace. A hardcoded value is discarded, and replaced with a counter, which too is soon discarded. But on the other side of this trade is the significant benefit of continual and consistent forward progress--"slow and steady wins the race."

For the new TDD practitioner, and for even experienced TDD practitioners, it is very tedious and annoying. Often, developers simply discard rigid adherence to the practice. For an experienced TDD developer, that's their prerogative, and I'll admit that I have done so at times too.

For everyone else, I think it's essential to first learn the value of something before discarding it. So I'll define my use of "experienced TDD developer" in the prior paragraph as "someone who has not done enough fine-grained incremental development to understand how it helps." Perhaps I'm slow--I still prefer to work this way.

A final justification (aka "the new thought" that entered my head this morning):

Yes, it's slower, tedious, and produces continual waste, particularly as you build the rudiments of a class. But these complaints are focused at the early minutes in the lifetime of a class--a fraction of the effort and struggle involved with maintaining it. There's a benefit I didn't explicitly mention above, and that's the ability to more easily and safely maintain a system, because of the increased number of assertion increments. The small amount of initial tedium is worth it.

Less Refactoring

2008-06-09T14:13:00.003-05:00

If you could have only one refactoring, which would it be? Which refactoring could you least afford to give up?

To me the obvious answer is the rename refactoring. It's an activity that I do very frequently. In fact, the way I code is dependent upon my ability to do a rapid rename.

I try to follow Uncle Bob's mantra "never be blocked" at even the smallest granularities. I won't stew for more than a few seconds to derive a name for a class, field, method, or other identifier. Instead, I type. If my brain is really frozen, I'll surge forward by typing an "x" or the word "something." No worries--I usually think of a better name shortly thereafter. I then do a ctrl-1/rename, type the new name, press enter, and Eclipse completes the job for me very safely and quickly.

The name isn't final at that point. I'll continue to update an identifier's name as my understanding of what it represents gets refined (or changes).

Recently I committed myself to working more in-depth with Groovy. I'm happy with its improved simplicity, consistency, and expressiveness over the tedium that is Java. As always, I spent a while using Groovy's bare-bones tools (groovyc, groovyConsole, etc.) so that I understood them enough to survive if I had to. Done enough of that! Now I'm looking to my productivity experience by introducing an IDE, so I downloaded and installed the Groovy plugin for Eclipse.

Harumph, and that's even after ignoring (a) a couple bizarre errors that I received because I hadn't forced a clean and (b) the obtuseness of the exceptions. No, the big harumph is that there are no ctrl-1 or refactoring options anywhere to be found. I've heard IDEA is better (and hope to try it out here soon), but as I understand, the rename challenge will never be completely resolved: safe rename refactorings require type information. You can provide that in Groovy, but figure on most developers not bothering.

And for now, the Eclipse plugin isn't even trying. I created a private, typed field--something for which a refactoring operation could be very safe--but now I can't change its name without resorting to search/replace. The fact that I may need to discard my newer programming style of "continual rename" makes me feel dirty.

Even with some forethought, programmers will always choose some names that end up less than ideal. And since developers are notorious for not changing things that might break, the poor names will remain as long as the IDE can't do it safely. Maintenance costs will be higher.

The main point of my post is that after years of using a strongly-typed language, I am now resisting languages like Groovy and Ruby to an extent. Not because I'm worried about the defects that could happen at runtime that wouldn't otherwise-my use of TDD generally means that's not an issue. Instead, my complaint is about lack of productivity, and worse, the decreased amount of refactoring that will likely result.

The question is, do benefits accruing from the increased simplification of the code outweigh the negatives of more difficult refactoring?

Opinions on Ubuntu Revisited

2008-05-23T13:58:00.002-05:00

OK, Ubuntu is still pretty cool, but man, sound driver support is sad. My son, who purchased a SoundBlaster card for his machine, got his sound to work but could not get the microphone to work. I ended up in the same situation, except that I have an on-board Intel HDA sound processor.

The good thing is that lots and lots of people have posted on this topic. That's also the bad thing, because (a) lots and lots of people have similar problems and (b) the resolutions are all over the map. I tried everything recommended, including downloading the latest ALSA drivers, rebuilding, and reinstalling, at which point my sound refused to work at all. Several hours later, I got the sound to work (I found a better posting which properly covered the arguments to pass to the configure script before making) again. I tried a few more things, but my microphone still does not work (it works fine under Vista).

OK, so I'll have to boot into Windoze to use Skype. But otherwise, all else appears to be running well. Next up, I intend on exploring Wine to run a few of the rare Windows programs that are outstanding.

One of the nicer aspects was the ease of keeping Windows around, which I need for a few reasons. While the Vista partition manager sucked (it would not do what it claimed it could do with respect to shrinking my partition), the Ubuntu installation made that a breeze. An important note, though: The partition resizing operation showed no feedback for a couple hours, so I wasn't sure whether or not it was doing anything. I waited patiently, and was probably going to reboot in a few more minutes, but the app must have suspected my anxiety and finished promptly.

I can get at files from either side of the fence, meaning that I don't have to worry about redundancies between Vista and Ubuntu. From Ubuntu, you can see NTFS drives by default, although you have to activate the ability to change them to be writeable. Under Windows, a quick search revealed the existence of Explore2fs, a nice little utility that can read ext2 and ext3 file systems. Unfortunately, it looks to involve copying to local temp files, but it's better than nothing, and should cut down the number of reboots, particularly until I get everything settled in.

So Ubuntu is not going anywhere. Mentally, it's now my primary OS, and Windows is just here as a necessary evil. I'm satisfied.

Ubuntu

2008-05-21T10:35:00.006-05:00

I finally got up the energy to spend several hours cleaning and doing a reasonably full backup of my (inexpensive) laptop, which had shipped with Windows Vista. Glutton for punishment that I am, I figured it would be a good idea to learn enough about Vista to survive when working in a place where they happened to use it.

Well, it turns out that no one is using it, or at least no one really wants to use it, as far as I can see. And enough is enough--I learned what I need to know, which is that Vista adds almost nothing to the advancement of operating systems or the user experience, as far as I'm concerned.

So after backing up, I decided to load Ubuntu, which I'd previously installed the prior weekend on my son's machine. Tim's machine had been running Windows XP, but somehow got screwed up, at which point he realized he did not have a Windows install disk (and images were nowhere to be found on the machine). The experience was surprisingly better than my last attempts to install Red Hat or Debian--up and (mostly) running in less than an hour.

Still, it reminded me why Ubuntu is far from being something that I can recommend to anyone other than a computer geek. I thought Vista was annoying with its continual interruptions; at least it didn't require me to enter a password everytime I su'did something (maybe there's a way around it, and it had better not involve removing the password).

The bigger problem is that no one, except us of course, wants to muck with the command line. Having to deal with my son (who's not stupid and is on the computer all the time) who is frustrated over figuring out to configure his machine reminded me of that. Finding the right drivers, doing chmods as appropriate, sudoing things where necessary, and so on. God love it and 'em, Ubuntu is not for anyone but a geek. I'm glad I switched--Ubuntu is great, and I guess that makes me a true geek.

Revisiting BDD in Java

2008-04-28T22:49:00.004-05:00

My initial reaction a few years ago to BDD implementations in Java was less than stellar. Since then, I revisited them a couple times, and now I'm digging into them again. They're still around, so someone must be getting value out of them. I certainly have gotten some mileage around naming and organizing my tests in a more BDD fashion.

Right now, I'm being lazy, and haven't done diligent research. But my initial internet searches suggest that there are a handful of tools, none of them the de facto standard. I remember my doctor once saying, "We have about a hundred ways to treat this [a plantar wart]. Usually when there are so many ways, it means we don't know what we're doing."

I started looking at one framework in more depth, and coded some straightforward examples. Didn't seem so bad, until I looked to code a more involved assertion. Here's one (contrived but simple) example:

Ensure.that(name, and(contains("Langr"), startsWith("J")));

One of the core principles of BDD is "getting the words right." I think this example demonstrates that sometimes we try too hard, and perhaps this construct isn't a great idea for Java. The "classic" approach is easily more expressive:

assertTrue(name.contains("Langr") && name.startsWith("J"));

or better...

assertTrue(name.contains("Langr")
assertTrue(name.startsWith("J"));

or even...

assertFirstInitialAndLastName(name, "J", "Langr");

I suppose my complaint could be tempered with the fact that these kinds of compound assertions don't occur so frequently.

Yes, sure, the failure messages are better. That's not significant to me, given that I don't drive my development off of failure messages. Sure, all tests fail at least once, but I usually know why they're failing the first time, so I don't need that message initially. Later, when they fail for unknown reasons, more information might be useful (sometimes it is, not always), but if I need that, I just enhance the assertion I already have and rerun the tests. No big deal.

Writing custom "matchers" is also an example in ugliness, given the limitations of Java. (I need to speed up my move to Groovy or Ruby, so I can stop whining about Java so much...)

I won't give up on the Java BDD tools completely, but I'm not much happier now than I was three or so years ago.

What's the Next Test?

2008-04-23T13:29:00.003-05:00

I coded the "Roman numeral kata" for the first time last night. I thought it was a reasonably straightforward exercise, but then I looked at the solutions of others by searching the web.

In agile, there's this notion of "doing the simplest thing that could possibly work," and I think it applies to TDD just fine. It helps keep me in the red-green-refactor rhythm. More importantly, it prevents me from pre-building complexity. It also keeps me in a mode of extreme incrementalism: DSTTCPW keeps teaching me how to truly grow a system incrementally, as agile demands.

From looking at the solutions on the web, however, I suspect that some people translate "do the simplest thing" to "don't think at all." Specifically, I'm referring to their progression of tests. At all times, once I complete a single test method, I spend some time thinking about what the next test should be. I don't spend too much time, but nor do I blindly jump into the next apparently obvious test.

The task in the Roman numeral kata is to convert a positive integer to a Roman numeral. Doing this kata in a TDD fashion, the first test is converting 1 to I, which is a hard-coded return. The second test is converting 2 to II, which is simply done with an if statement. One might refactor at this step to a loop/count construct, or one might wait until 3 (III).

From 3, though, I noted that many of the coders insisted on jumping directly to 4 (IV). Certainly, this works, but most of the solutions I saw derived in this fashion either ended up more complex, or resulted in a fairly excessive solution that scaled back dramatically with some last-step refactoring.

One could argue that this is in the true spirit of incrementalism. After all, we should be able to tackle requirements in any order. Perhaps we need only support the numbers 1 through 4 initially, and later someone will insist on adding 5+.

An argument currently brewing on the XP list goes into the whole YAGNI thing, with one poster suggesting that if you have comprehensive information, there is nothing wrong with taking that into account. Kent Beck subsequently updated his take on YAGNI: "Here's what I do: consider [forthcoming feature] a little, but try not to get so caught up in it that I freeze."

In a similar fashion, TDD doesn't mean "don't think." Determining the next test to write should incorporate some thought about what represents the simpler path to a more complete solution. Sometimes this is a guess, but with more experience, the wiser choices are more apparent. Sometimes you make the wrong choice, but you can still end up in the right place, particularly if you refactor with diligence.

With respect to the Roman numeral solution, the better answer is, try 5 (V) before 4 (IV). Then add combinations of V and I (6, 7); then finalize this logic with X, which pretty much demands a table-driven solution.

A last intuitive leap can dramatically improve the solution, but is not essential: Table entries don't have to represent a single Roman letter. Thus you end up with an entry in the table for 4 (IV), 9 (IX), 40 (XL), 90 (XC), and so on. The logic stays clean and simple:

import java.util.*;

public class Roman {
  private final int number;
  private static final TreeMap<Integer, String> digits = new TreeMap<Integer, String>();
  {
     digits.put(1, "I");
     digits.put(4, "IV");
     digits.put(5, "V");
     digits.put(9, "IX");
     digits.put(10, "X");
     digits.put(40, "XL");
     digits.put(50, "L");
     digits.put(90, "XC");
     digits.put(100, "C");
     digits.put(400, "CD");
     digits.put(500, "D");
     digits.put(900, "CM");
     digits.put(1000, "M");
  }

  public Roman(int number) {
     this.number = number;
  }

  @Override
  public String toString() {
     StringBuilder roman = new StringBuilder();
     int remaining = number;
     for (Map.Entry<Integer,String> entry: digits.descendingMap().entrySet()) {
        int decimalDigit = entry.getKey();
        String romanDigit = entry.getValue();
        while (remaining >= decimalDigit) {
           roman.append(romanDigit);
           remaining -= decimalDigit;
        }
     }
     return roman.toString();
  }
}

If you don't make the intuitive leap, the final solution is a bit more muddy (as I saw in a few solutions out there). But you have a second chance: Since you have all those wonderful tests, you can refactor the heck out of your muddy solution, looking for a more concise expression, and often you'll find it.

Clean Code

2008-04-08T22:25:00.002-05:00

Uncle Bob Martin just completed a draft of Clean Code, which the publisher site lists as being available August 22, 2008. The book is a code-intensive collection of thoughts on how to keep, well, your code clean. I was honored to submit two chapters, one on "clean classes" and one on emergent design. Uncle Bob got the last word, putting in even more code than I'd laid out on these chapters.

Many of the chapters are contributions by other current-and-ex-Object Mentors, but Bob put in a significant effort to go through the compiled book as a whole, to make sure everything coheres well and put his personal stamp on it.

The announcement got me stoked again about writing. Developer.com is also looking to compile an e-book of my many articles on design patterns (most with a TDD bent), so look for that in the near future. Still, I think it's time I went back and started on a whole, real book of my own. The C++ tome is still a potential if Mr Koss and I can find mutual time together for it.

Open Source Pairing Scheduler

2008-04-03T16:30:00.003-05:00

Rather than whine about not having enough good pairing experiences, I've decided to do something about it. Recently I've had a few pairing sessions online, and I decided that the software to build would start around the simple notion of scheduling online pairing sessions.

The idea is to slowly start building an online pairing community. I imagine there are many kinds of participants, people who want to:

build quality software via pairing and TDD
promote their services and ability to mentor via pairing
get some work done on a real product
get in touch with other members of the agile community
learn about either pairing or TDD
learn about the effectiveness of distributed pairing
learn a specific technology (Java, Rails, JMock, RSpec, etc.)
improve their Ward number for bragging purposes

I hope that the scheduler software will be all that and many more. Initially, I created a wiki (see http://jlangr.wiki.zoho.com/) as a scratch pad area, to help set up sessions, to hold an initial backlog of stories around the scheduler itself, and to record experiences.

I chose Java, but only because of interests in my pairing partner. I'm not sure that the pairing scheduler should be Java, but for now it is. If Java, what represents the best choice for front end (web) technologies?

Please feel welcome to join up and post your recommendations and experiences at the wiki site. Interested developers should send me an email with their intents, and I'll set up their SourceForge account to have appropriate access.

Upside-Down Problem Solving

2008-03-31T08:54:00.002-05:00

In doing Sudoku puzzles, most of the time the next solution is staring you in the face. But you can't see it for all the other numbers in the way.

I got stumped on a puzzle Saturday, one that I'd stared at for about ten minutes without success. I have a ten-minute rule, so it was time for a new tack. (Others might call this ADD, but I follow Weinberg's Bolden Rule: "If you can't fix it, feature it.") I turned the book around to let my son look at it, but continued to look at the upside down numbers. Thirty seconds later, the very obvious solution almost started blinking at me. Duh.

Often, all we need is a different perspective on the problem. Many of us have experienced code enlightenment when having someone read over our shoulder. The usual explanation is that we're now trying to perceive how others are reading our code. Another thought: Sometimes I suspect that a very brief mental break is all that's needed.

But maybe it's just that we're shifting in our chair and catching the code at an odd angle. :-) Next time I'm stumped on code, perhaps I'll invert my display. Or change my font.

Dogs and Bugs

2008-03-23T08:06:00.003-05:00

My dog kept wandering around the neighborhood, so I recently put up a fence around my back yard. Now I'm experiencing some regret: after all that expense and effort, the dog no longer gets out of the yard! Was it all a waste of time?!!?

(I think Ward Cunningham puts it much more nicely, but hey, I'm not Ward and will never be. Ward says, "How about FIND BUGS PROMPTLY?")

TDD Hype

2008-03-04T20:43:00.006-06:00

A trend I've noticed in the software development community is that there are a large number of developers who resist the new wave. Many of those who spent their formative development years doing procedural development went kicking and screaming into the OO age; some suggested that it was an inappropriate tool for many problems. Many C++ developers looked at Java as a toy, not useful for any real work. And I've met enough Java developers who thought that Java was the last word in programming languages.

Certainly we should question newfangled tools and techniques, and insist on putting them through their paces. OO is not the best tool for every problem, Java did suck big time when it first came out, and yet there are now some significant advantages to continue doing business with it. Challenges ultimately help the tools & techniques improve.

Experimentation and innovation is at the heart of the software industry, not active resistance. But it's far easier to sit in a corner and make as many possible excuses for not doing something. I know, because I've done so myself. There are plenty of jobs out there using old technologies, and places where people are happy with the status quo.

So, TDD. Some of us strongly believe in it, only because we've seen benefits firsthand--not just from toy Stack examples, but from working on systems that range from tens of thousands, to hundreds of thousands, and millions of lines of code. Ahh, but the stack example is all the people sitting in the corner see, because that's all they want to believe is behind TDD.

I agree with the vocal dissenters that TDD should not always be used, but not for the lame list of excuses they often come up with. Primarily, the reason is that the typical American software team just isn't that good. TDD is a reasonably tough discipline to get a handle on, and most developers are still struggling with OO, language, and tool basics.

What's interesting is that TDD would ultimately help these people the most--some of its best potential is its use as a learning tool. The vocal dissenters, on the other hand, are usually reasonably sharp developers who already have a good notion of design. Thus they can't quite understand why a technique that promotes more decoupled and cohesive designs is important--"duh, we already get it." Particularly, for example, if they don't happen to work with lots of typically underachieving coders.

Is TDD overhyped? Not a chance. There are still mountains worth of learning that TDD will promote in the minds of the majority of the developers out there. For those who don't want to do TDD, don't do it.

Can TDD be dangerous, and lead to disasters? Yes. This is true for any tool in the wrong hands. That also includes insisting on emphasizing test-after--I've seen that lead to plenty of failures and wasted time. TDD, like OO or any other tool/technique, is not for everyone or every project. But in the right hands, I've seen TDD work wonders on software projects.

Choices, Choices

2008-03-02T15:14:00.003-06:00

Who else has this quandary?

I'm long, long weary of Java and C++ (which I thought I had been weary of over well over a dozen years ago, but here I am again struggling with it). Where next? Sure, I've been looking at Ruby for over five years now, and Rails a bit as well, but I've not had an opportunity to put it to any real use in a production environment (except for an odd pairing session here and there).

Recently I've also looked at D, Erlang, and Scala. "Looked at" means that I've downloaded it, gotten it running, run through at least a tutorial or two, maybe read the book, and then moved on to experimentation, perhaps starting to re-implement pieces of systems.

Seems like the most interesting stuff happening is going on in Ruby--it's a rapidly maturing environment. There are things going on around the other three languages I mentioned, but it doesn't to be nearly as much. Maybe when they've reached "critical mass," there will be more going on around them.

In any case, there are scads of new languages and tools vying for attention. It seems like there are more than ever, making for an exciting time. It also means that my stack of what I don't know keeps piling up. I think I need to pick one and invest some real, significant effort into it.

So, the decision: ride on the bandwagon, or blaze a trail? What are your reasons for putting your money where it currently is? What else should I be looking at that I'm not?

Pair Programming Withdrawal

2008-02-07T16:02:00.000-06:00

Once again, I find myself without peers. I'll spare the details, but I used to work with some great agile technical mentors. I had some (never enough) opportunities to just sit and pair on a solution, not as a mentor, but as a peer on a programming effort. Unfortunately they've all disappeared, one way or another.

I'm reminded of my feelings when I left a 7-month programming gig a few years ago. During that seven months, I paired almost continually. I looked forward to going to work every morning. There's a significant social aspect to pairing, but mostly I looked forward to learning new things.

Some of my favorite pairing sessions in the past half-year were with Tim O.**, and more recently with a guy named Rick. I learned, and I think they may have learned too. I improved in capability and thus effectiveness on vi while working with Tim. With vi, a very powerful editor, I think there are a number of rungs: 1, where it just beeps at you, 2, where you've figured out how to survive, 3, where you're reasonably proficient, 4, where you know quite a bit and are very effective, and 5, a true master that knows everything in there. Tim is certainly at 4 and approaching 5 (there were a few things he had to look up). The nice thing about our pairing sessions is that Tim helped me to step up from rung 2 to 3.

Now, I'm all alone (*sob*). Sympathy cards are welcome, but smacks upside the head for my whining are also welcome. The way I look at it is that every day that I'm not pairing, there are many, many things that I could be learning that I'm not. That's a shame. I now second-guess my work. "This sucks. There has to be a better way. Is there a better way?" I'm pretty sure that I also develop more slowly.

As an aside, I've heard suggestions that expert-to-expert pairings are a waste of time. That's absolutely not true. A good tennis player and friend once told me that the best way to hone your game and take it to the next level is to play with someone as good as you or better. It works in programming, too.

**Tim is an Object Mentor consultant who coaches agile teams in just about everything. I think he'd like it that I added this information.

Stories and Code Quality

2008-01-30T10:11:00.000-06:00

Doling stories out to individual developers not only makes for more boring iteration planning meetings, but it also impacts the system's overall quality.

One of the most overlooked practices in software development today is review. If you don't pair, then you really must find a way to consistently review work product. Unfortunately, most companies pay the cost of not doing this: code that is increasingly more difficult to maintain, and solutions that are questionable at best.

Insisting that a team finish a story every couple of days or fewer will often mean that two or more developers must collaborate on a story. In order to collaborate, they must agree on some part of the design and a plan to implement it. The developers will need to continue talking as the implementation progresses. That's a good thing. Much of the resulting design won't be one developer's wonderfully clever or woefully inadequate brainchild.

Such collaboration brings a bunch of siloed developers one step closer to being a true team. Initially, velocity will decrease a bit: developers will have to learn how to talk to each other and coordinate things. Developers will learn more about other parts of the system, increasing their value to the project as well as their personal worth. It's not quite as good as pairing, but it does go to the heart of agile.

Stories and the Tedium of Iteration Planning Meetings

2008-01-25T10:35:00.000-06:00

In "Tools, Iterations, and Stories," I said you should "focus on implementing and completing stories, not iterations. If your iteration is 10 days, and you have 10 stories of 1 point each, you should have delivered one story by the end of the first day, and two stories by the end of the second day, and so on."

Then, in a blog post titled "Stories and the Tedium of Daily Standups," I talked about how a focus on completing stories, not tasks and iterations, would make daily stand-ups interesting enough to bother with (anew--since many teams that initiate daily stand-ups often find them boring and thus slowly stop doing them).

There's another important side-effect of organizing an iteration around incrementally completing stories: Iteration planning meetings are more interesting.

Implicitly, to be able to complete stories as an iteration progresses requires team members to actually collaborate on a story. Imagine that. It's at the heart of the emphasis on continual communication in agile, yet many teams tend to miss this key point. In my next blog posting I'll talk about the side-effect of improved quality. For now, I'll restrict my thoughts to the impact of story planning on iteration planning meetings.

If an iteration planning covers four stories to be implemented by a team of four developers, there are two primary ways this meeting could go: 1) The customer details each story by focusing on each individual that will work it, or 2) The customer details all stories, discussing them with all four team members. The downside of option #2 is that it will probably take a little longer to work through all the stories. In fact, if you went with option #1, you really wouldn't need a team meeting. The customer could just meet individually with each developer.

Effectively, what you have with option #1 is a meeting in which 3 out of 4 developers have little interest in the story discussion at any given point in time. Information about other developers' stories is mental clutter, and consciously or subconsciously, each developer will tune out to some extent for 75% of the meeting. They will view iteration planning meetings as mostly a waste of time.

In contrast, option #2 requires everyone to be attentive. We all have to understand every story, and we most certainly have to start figuring out how each one breaks up. We probably need to do a little bit of design. People attending these sorts of iteration planning meetings are engaged. They leave the meeting with the impression that it held considerable value for them.

Blocked!

2008-01-24T20:36:00.001-06:00

Jerry told me about frustrations at his new company. Here's a possible conversation from my next interview (if there ever is one):

Jeff: Do you block web sites or other things like IM?
Interviewer: We filter on naughty words, we block web mail, and we also disallow use of instant messaging.
Jeff: Interesting. Why do you block web mail and IM?
Interviewer: Because we've found that some people tend to abuse it at the office. They also raise some concerns with respect to security.
Jeff: So you also block phone calls, then?
Interviewer: Of course not.
Jeff: Why not?
Interviewer: Well, people have legitimate business reasons to make phone calls.
Jeff: OK, first, there are legitimate business reasons to use IM. Second, you could block calls to individuals' homes if you really wanted to.
Interviewer: Sometimes employees have legitimate need for personal calls. We find that's ok if they don't abuse the "privilege."
Jeff: So, if talking on the phone all the time was a problem, a manager might intervene and have a discussion with the employee. That seems reasonable. It also seems reasonable that an employee might have a similar, legitimate need for personal emails from time to time.
Interviewer: But we block webmail.
Jeff: Thank you, and have a nice day.

Let's face it, companies rarely change abusive policies like these. Sometimes, the best we can do is voice our disapproval and vote with our feet.

Test-First vs. Test-After, Round 4

2008-01-02T10:41:00.000-06:00

I submit two articles per month to Developer.com. It's often tough coming up with new article ideas, so coming up with a theme that will produce a long stream of ideas is always a good idea (as long as the publisher is willing to accept it). Design patterns is something I've done about a dozen articles on.

Over the nice, long Christmas holidays (vacation for me), I started writing an article featuring the memento design pattern. For me, the hardest part of writing any of these articles is coming up with a good example. I try to avoid the common examples that are already out there, and I don't think abstract examples are very useful. In any case, once I've thought of and coded an example, the article almost writes itself.

Right now, I'm on my third example attempt for memento, which for me is a fiasco. I've probably burned eight hours on something that usually only takes me a few. Memento is not something I've (explicitly) used, so that's not helping.

Each example attempt has been centered around the typical use of memento, to support undo/redo capability. I floundered through two examples, discarding each in turn when I decided that it wasn't a good representation of "real" use of memento.

The other real problem was that I figured I'd wing it. I've coded the examples for every other design pattern article using TDD. But over the holidays, I guess I got lazy. For memento, I banged out a solution and figured I'd refactor to something that I could easily unit test after the fact if I felt compelled to.

It's been a while since I've lazed off and not done TDD. As usual, it was a good reminder that it's just a crappy, less-effective way of working. The code I produced was a mess, with logic entwined in the UI class, as tends to happen when there's no interest in unit testing. Worse, I kept getting stuck on a problem with BigDecimal and scaling that wouldn't have given me such trouble had I used TDD.

Maybe I'm just not a very good developer, after all. Or maybe, just maybe, test-after is a less mature way to code.

Test-First vs. Test-After, Round 3

2007-12-21T09:30:00.000-06:00

I post occasionally to JavaRanch, a "friendly" forum (it's policed by moderators who cleanse any material deemed offensive). JavaRanch features a book-of-the-week discussion and giveaway. This week, it's a pitch for the book, Next Generation Java Testing. The book so far appears to largely be a pitch for TestNG and an expected couple of rants against JUnit (version 3.8, for whatever odd reason, and not version 4) and TDD. I stopped reading it for now (I'll finish it later) because I found Lasse Koskela's book Test Driven more useful.

As expected, the JavaRanch discussions contain some allusions to the fact that it doesn't matter whether or not you write (unit) tests first or last. I've written elsewhere on this subject many times, so I'll add only a brief thought or two here.

I suppose it doesn't matter, from a conceptual standpoint, if you do test firsts or tests after if you are an experienced professional developer. If you are good at design and structuring your code well, then there's no reason you shouldn't be able to write as many useful tests after you've already built the code.

But that's not what really happens. In the real world**, 8 out of 10 developers aren't so experienced or professional, and so most don't care enough or know enough about design to structure their system right. And as soon as someone says, "you don't really have to write tests for everything, and they can break and that's ok," the 80% end up thinking that these unit tests are largely a waste of time. So they--the developers and the tests--don't get better.

The other reality is of course time. Without a disciplined approach, deadlines always get in the way of tests. Interruptions push unit tests outward, and they just don't get done.

I believe that test-after is only for the true software development master who is close to expert level. For anyone else, it's a recipe for wasted time and frustration. A better investment would be in automating your acceptance criteria.

That's opinion. Test-after is unscientific and undisciplined, and that's fact. I'm willing to accept that there may be something better, but test-after when-you-feel-like-it-and-can-figure-out-how-to-do-it is not it, and TDD is closer to following a scientific method. That too is fact. I'd rather look to advance the craft than make excuses why I'm too set in my ways to change.

** Sorry Tim.

Stories and the Tedium of Daily Standups

2007-12-06T19:29:00.000-06:00

In "Tools, Iterations, and Stories," I talked about how you should "focus on implementing and completing stories, not iterations. If your iteration is 10 days, and you have 10 stories of 1 point each, you should have delivered one story by the end of the first day, and two stories by the end of the second day, and so on."

Today I helped deliver training on agile estimation and planning. One topic that arose was the daily stand up. We quickly surmised, no surprise, that this group's experiences with daily standups were negative. We discussed some of the reasons why these meetings might be so poorly received. As typical, their standup meetings went on much longer than 5-10 minutes. Reasons? Also typical: people didn't literally stand up, the meetings lacked focus, and people tried to use them to solve problems instead of just identify them. Most people, rightly so, viewed these meetings as very costly.

In response, we talked about why daily standups might be useful. We discussed the importance of getting the entire team to communicate more frequently. Daily should be a bare minimum!

The other part of our discussion was around what should be said during a standup. One of the complaints was that the standups were boring and tedious. The teams were using the typically recommended elements: what did I accomplish, what am I working on, what might I need help with? You know the drill.

Then it struck me: a focus on completing iterations, not stories, was part of the problem, contributing to the tedium.

The classic mistake in an initial stab at doing agile is to treat each iteration as a mini-waterfall. Spend a day discussing requirements, spend a day or two on design, code it, then test it, then integrate it. A team will open most of its stories on day one, and on the last day of the iteration, most of these same stories will still be open. The team will struggle to close them out prior to iteration end. The article I wrote (see link above) talks about a better approach: initiate work on one or two stories, and move on only after these stories are completed. That allows for incremental delivery of business value throughout the iteration, and usually minimizes the amount of work at risk for the iteration.

A team following the first approach, where an iteration is a mini-waterfall, will have very little useful information to say during a daily standup:

"What did you work on yesterday?"
"The flim flam widget."
"Is it going ok?"
"Sure, I guess so."
"What are you doing today?"
"Same thing as yesterday. Probably tomorrow, too."

Zzzz. The daily standup is drudgery, because not much is changing, and none of the information is reliable anyway. When asked where they're at, a developer uses the old 80/20 rule. Week one in a two-week iteration, they're probably 20% done, and week two, they're 80% done. And during week one, most developers have the confidence that they'll be done in time, even if they're stumbling. We won't find out that there are real problems with their commitment until (too) late in week two.

In contrast, here's how a standup might go in a team that looks to complete stories, not iterations:

"What did you work on yesterday?"
Joe: "I started on the frizzbang story."
"That's supposed to be done sometime today, right? Are you on target?"
"Well, maybe not, I'm stuck on a strange exception. I could use help from someone who's worked in this area."
Jane: "OK, well, I helped complete the flim flam widget by finishing the GUI component, so I'm available."
"So the flim flam is ready for testing?"
"Yup. Today I'll go work with Joe, before I start on the festivus tool."

The more I do agile, the more I find that a primary focus on (really) completing stories is the best path to success.

Testing Getters and Setters?

2007-12-05T14:37:00.001-06:00

You've heard many developers claim they don't test getters and setters. Correct--you should probably not write direct tests for these things. The primary reason is that they do not (usually) represent behavior; a setter or getter is (usually) just a means of exposing an attribute.

Let me touch on a few points.

"Getters and setters can't possibly break." Wrong! Been there done that. Try misspelling the formal argument name in a setter, e.g. "X" instead of "x." this.x = x; will compile (you might get a warning), but it will do nothing on execution. If you assume that this sort of code can never break, you will spend lots of time looking elsewhere for the problem--been there done that! :-) Still, this isn't a compelling reason to write a direct test.
"Don't test getters and setters" should probably be quantified with "directly." More or less everything, in effect, should be test-driven. That is, code should not come into existence without first having a failing test. So, perhaps tests against a method in another class drive the need for the getter, and that's fine. But you should never blindly add getters/setters to a class without a reason. (The code generator utilities in Eclipse and other tools should probably be painful to use, to dissuade you, and they sort of are.) If you use a dumb framework that insists on you creating getters/setters that aren't directly called by your code, consider a better framework.
Many getters have interesting logic (e.g. lazy initialization), as do setters (although this is often a riskier design choice). I'm a firm believer in verifying that these "trivial" idioms actually work; I've seen enough broken initialization attempts. You should test-drive these, and probably directly so (and maybe even ensure they are comprehensively re-tested within the context that they are used).
I always write tests for constructors, for a few reasons. First, it's a good, simple place to start. It allows you to get a quick test in place and build an instantiable class within a minute or two. Second, a class without tests often suggests to other developers that maybe we don't need to worry about writing any tests for that class. "Oh yeah, I guess we need to add to the tests for this class." Third, having to write constructor tests can discourage people from putting any complex logic in constructors, something that's not usually a good thing to do. :-) Fourth, and most importantly, tests for constructors document all of the possible means of creating instances of a class, information that can be difficult to derive otherwise. You're cheating your fellow developers by skimping on these tests.

Having tests for my constructors means that I effectively test-drive getters, as they are the best means to verify construction.

So, should you test-drive getters and setters into existence? Yes, usually indirectly. As long as they get test-driven into existence, you'll eventually find the problem if they hide a defect. Directly, only if the getters and setters are interesting.