So what is it that we do? This is the question that I find myself constantly answering. My kids ask me, my neighbors ask, and our customers ask me all the time; I think I have finally found an appropriate, although simplistic answer to the question: “We prevent the accumulation of Technical Debt”.

Yes, it is true we write great software.

Yes, we are a Test Driven organization.

Yes, we focus on bridging the gap between Business and IT.

Yes, we focus on Value first.

Yes, we embrace a set of practices that drive our daily and weekly disciplines. But we do so for one simple reason – the prevention of Technical Debt.

When I compare what we do at Pillar with the many other organizations, it usually comes down to the set of practices that allow us to identify and eliminate Technical Debt before it becomes a burden.

Technical Debt is the cancer of the IT world. It is in the database schema in form of tables, indexes, columns that are not used. It is in the extra code that was written that was not needed. It is found in every method or object that was written without a test harness. It is most often found in the 800-page requirements document that embodies the vain attempts of humans to capture business requirements. Technical Debt is the nuisance of society. It is the reason for batch programs to fail at night. It is the reason many programmers loose their hair. Yes, Technical Debt is the evil cancer of our profession.

So it comes down to this simple goal – the early identification and elimination of Technical Debt.

The Agile Practices that we embrace allow us to gain a competitive advantage over those who do not share this world-view. Our understanding of Technical Debt allows us to keep our attention on this age-old problem. If you don’t care about the consequences of Technical Debt, you tend to not worry about its impact. Some people prefer to just allow the next generation of developers to worry with the debt. We cannot assume this position – Our devotion to error free code will not allow it.

Pillar’s success is rooted and established upon the belief that the early identification and elimination of Technical Debt is in fact the pathway to lower overall cost of software construction and longevity.

So there you have it. That’s our secret. I wish I could have a better explanation, but I really believe that it comes down to this simple fact: We achieve our overall success by focusing on the elimination of Technical Debt.

FINALLY – the industry is awakening to one simple truth: VALUE IS WORTH PAYING FOR.

As you know, over the past several years the business community has been tempted to try to find a cheaper way to produce software. This search for “cheaper software” almost destroyed the software industry and if you look close enough – the entire country. In many cases, this search led companies down the traditional Waterfall path (invented here in the US). In short, this process involves death by analysis and the compilation of volumes of nearly worthless “requirements documents”.

The promise of “cheaper software” was often held out like a carrot in front of a donkey – you can see it but its always just out of reach. Such is the case for companies that ship their requirements documents to some distant software sweat shops to produce various renditions of software that always seem to just miss the mark – again and again.

To be fair, the concept of finding cheaper labor was an option at one point, but the true costs of software is not found in the cost of labor. The truly alarming hidden costs of software development is a cancer within the process itself. The “Waterfall” approach is prone to failure because human beings and software engineers (humans too) are not properly equipped to communicate requirements via pictures and or Word documents. We are far too relational, sensual, and creative to expect paper to carry the proper requirements.

Agile to the rescue: As an industry, we are awakening to the reality that we can reverse the trends of the past. We can actually write software in a new and exciting way. We have the technology, the motivation, the passion, and a new approach.

Now for my primary point: Companies that once leaned on offshore companies in search of the holly grail of savings are beginning to benefit from Agile Approaches that Pillar embraces. They have tried and failed at various approaches. Failure on one path often leads to a willingness to try new approaches. For this I am grateful.

Three weeks ago I received a call from a large food distribution company. They were ready to try our “Agile Approach”. They were in a mood to try our fixed priced Agile Services. They had struggled with the offshore model for years, but in the final analysis they want someone to write the software right the first time. They want high quality code. They want to know the budget. They want to deliver on time and on budget. Those are the promises that we make.

I am happy we won the business. But the most important factor is the fact that we did not win on the basis of price. We won the business on the strength of several factors: We are an Agile Company – We write great software – Our software has great test coverage – We are truth tellers – We hold our commitments – We deliver great software products that work.

Moving to Agile does lots of things, but one of the most important is that it puts the focus back on writing code. We spend a whole lot less time, energy and worry on producing design documents, requirements specs and architecture diagrams. Instead, we get to the point of writing code and testing it as quickly as possible.

And wasn’t that what the developers always wanted to do anyway? Weren’t developers always champing at the bit trying to write code “too early?” It’s interesting that, instead of finally figuring out how to hold back our developers, we’ve gone the other way and flipped our processes around to fit how developers want to work.

In an Agile team room, development is king. The team is there to make sure that great software emerges quickly and correctly. Business analysts write cards and organize tasks to assist development. Testers write tests to provide feedback to the developers. Iteration managers clear roadblocks that the developers encounter.

In fact, being a coach of many of these roles, and often playing the roles myself, I’ve realized that I am doggone jealous of the coders. They make stuff happen! They write code and things come together. They are the center of our attention. This isn’t to say that the other team members aren’t equally important, but it is a switch from the days when project managers thought they were “senior” to developers or testers. No more!!

In waterfall-land, I would never have considered going back to writing code in my spare time, or sitting with a developer on my team and figuring out how they are doing what they do. But with Agile, I find myself more and more drawn to the code. How is it working? How did you build that? Why did you pick that tool instead of this one?

I think it’s cool that Agile has flipped my career ambitions upside down. Instead of being a manager of a small project hoping someday to be a manager of a super-big project, I’m now a manager wishing I was a coder.

(Excerpt from the upcoming book “Agile in the Bloodstream” by Daryl Kulak and Dr. Hong Li)

Agile is an exciting prospect. It is usually pretty easy to see that an Agile project is physically different than other ways of developing software (team rooms, pair programming, etc.). Moving to Agile can feel like jumping across a chasm into a whole new territory.

Once you’re there, and it starts working well, Agile can be exhilarating. We’ve seen teams experience true joy from producing results quickly and being so deeply in touch with their customers using Agile.

But often a strange thing happens with Agile teams inside non-Agile organizations. They don’t expand. The Agile team may be able to produce great successes but the rest of the organization doesn’t recognize the team’s victories or, worse, start working against the team to ensure its failure.

Is this crazy? Is the rest of the organization insane with jealousy because the Agile team has found something new and useful? Yes, this is possible. But, in our experience, there is more going on. Introducing Agile into a non-Agile organization creates all kinds of issues that the organization and the Agile team need to address upfront if the team wants to be successful in the long term. We group these issues into a category we call illth.

Agile is Like Wal-Mart (Sort Of…A Little)

Illth. Nineteenth century English art critic John Ruskin created this word to describe the messy bits that came along with wealth. In Ruskin’s time, he saw that wealth was being created in various ways in England, with businesses and factories, but with those advances came the “side effects” of wealth, which were not as desirable. Underpaid and overworked laborers’ dismal lives could have been considered illth. The pollution and grime created by the factories in the community would have been illth. And the unrelenting movement of money from the poor to the rich was certainly illth in Ruskin’s mind.

Here is the definition of illth in Ruskin’s own words:

Wealth, therefore, is ‘The possession of the valuable by the valiant’; and in considering it as a power existing in a nation, the two elements, the value of the thing, and the valour of its possessor, must be estimated together. Whence it appears that many of the persons commonly considered wealthy, are in reality no more wealthy than the locks of their own strong boxes, they being inherently and eternally incapable of wealth; and operating for the nation, in an economical point of view, either as pools of dead water, and eddies in a stream (which, so long as the stream flows, are useless, or only to drown people, but may become of importance in a state of stagnation should the stream dry); or else, as dams in a river, of which the ultimate service depends not on the dam, but the miller; or else, serve as mere accidental stays and impediments, acting not as wealth, but (for we ought to have a correspondent term) as ‘illth,’ causing various devastation and trouble around them in all directions; or lastly, act not at all, but are merely animated conditions of delay, (no use being possible of anything they have until they are dead,) in which last condition they are nevertheless often useful as delays, and ‘impedimenta.’

Although Ruskin’s definition sounds overly damning toward the rich, we still feel there are aspects of this definition that can be quite useful.

Since Ruskin’s time, illth has been used only sparingly in books and speech, mostly notably by George Bernard Shaw in his essays on socialism. But it seems like a good word. Certainly, often when we do something good in our community, there are bad things that come with it that we need to deal with. Pollution is the best example in our own time. Our incredible engines of growth in North America, Europe and Asia have generated great wealth for millions of people. But the scourge of pollution has inevitably followed that wealth, darkening its splendor.

Wal-Mart is a great example of wealth and illth. When Wal-Mart comes to town, there are great things that happen. People can get stuff for good prices and there are more jobs, lots more jobs. But there is also illth. Wal-Mart has a tendency to have a bad effect on the local smaller stores, putting them out of business because they can’t compete with the low prices.

What does illth have to do with Agile? Please don’t be offended that we are comparing Agile to Wal-Mart, and please don’t think that we’re saying that Agile team members are very like the idle rich that Ruskin seems to detest so much.

When you look at a highly-functioning Agile team, you can see their success as a form of wealth. An Agile team is often somehow magically able to accomplish mountains more work than a comparable waterfall team, and yet they work fewer hours and are happier with their work. The wealth flows.

So where is the illth? It might not be within the team itself, but it could be leaking into the teams surrounding the Agile team. How does the Program Management Office (PMO) feel about the Agile team? Are they “feeling the wealth” or “feeling the illth?” What is the effect of Agile on the team that deploys applications into production? How about the competency centers? Or the human resources teams?

You don’t have to scratch the surface very hard to find people in a large organization that are actually upset about the Agile team’s successes. Yes, this might sometimes be due to their jealousy of the success, but there is another aspect to it.

Think of your large organization like the body of an animal or a person. What happens when your body ingests something that it doesn’t recognize? White blood cells attack the new thing and try to kill it. War is on, and either the intruder is flushed out of the system or the body dies trying.

Is that necessary in your organization? It doesn’t have to be if you, the Agile team, can manage your illth.

First, let’s examine the problems that an Agile team can create for its neighbors in the large organization. After that section, we’ll look at some solutions that can help Agile and non-Agile co-exist.

Here are some types of Agile illth:

• Agile vanity
• Agile project within Non-Agile program
• Incremental impedance mismatch upstream
• Incremental impedance mismatch downstream
• Clash with the corporate methodology
• Clash with the corporate project management structure and tools
• Clash with competency centers

Here is a bit about Agile vanity. (We’ll discuss the other topics in a later post.)

Having seen Agile projects in many, many situations and variations, we’ve had the chance to observe a peculiar Agile tendency that we call “Agile vanity.”

In fact, it is quite natural, once you’ve discovered a new and better way of doing something to feel a bit full of yourself for a while. The only trouble is that this can cause great strain in your organization. Let’s look at some of the things that happen when Agile proponents turn into “Agilistas.”

Inside the Agile team, the observation of results helps to create enthusiasm and a willingness to commit time and energy. The team feels like they are being productive and are working well together.

When someone in the team is discussing their project with people outside the team, they may give a sunny picture of what is happening due to their own happiness with their work. The outsider is likely to question these new methods, especially if they haven’t seen it work themselves. This may put the Agile team member on the defensive a bit, feeling like they need to give the most articulate answers or this person might think the Agile team is just being duped by the latest software development “fad.”

Each time this happens, the Agile team members may feel like they just shouldn’t bother interacting with those outside the team, or if they do, maybe they shouldn’t discuss Agile too much for fear of being in a defensive situation again. The other aspect that can arise is that the team members discuss the successes of Agile among themselves but decide that the others “just can’t understand the Agile way.” This creates a barrier between the “insiders” and “outsiders.” This is what we call “Agile vanity.”

The danger here is that, as soon as the team erects this barrier, those outside instantly feel that they need to defend themselves against being viewed as stalwarts of the “old way.” The outsiders begin to come up with lists of reasons why Agile can’t possibly work and why the Agile team’s successes are due to factors other than the Agile practices they are using.

As you can imagine (or possibly as you’ve experienced), this is not productive. The outsiders can be other teams or they can also be the executive managers of the Agile team. The direct and indirect managers of the team can cause a great deal of havoc for the team, at worst hemming them in with random directives that are, subconsciously, being done to ensure the failure of the team.

The same things that help the team stick together can be the very things that distance them from the outside world. And those outsiders are people the Agile team needs to be successful, whether direct managers, co-workers, upstream or downstream projects, or other connecting teams. Agile vanity, although it feels good at the time, can be quite destructive.

It is worth paying equal attention to what is happening “outside the Agile team room” as well as inside. Too many of our Agile practices and principles deal with the goings-on inside the room, but it is what’s happening “out there” that can kill the project too.

So what is database migration?

Managing database schema changes between multiple environments.

In most of the agile projects database schema evolves along with the code. Its follows the same incremental change code goes through, we add a table, rename column and so on (Evolutionary database design). Database by itself doesn’t have any versioning information. One of the great thing about versioning code is if the current version doesn’t work the way we want we can always rollback to the prior version. But what about the database schema changes? Having a tool automate and manage database migration helps teams to address that problem.

If you have worked with grails then you would know that we don’t necessary deal with database when working with grails domain objects. Instead of creating new table we create new domain object, we don’t add column instead we add a new property to an existing domain object. But we still need to deal with database migration issues when make incremental releases. We cannot create/recreate database every time we make release. We have to have a mechanism to version and keep track of all the scheme changes. As of writing grails as 3 viable possible tools to manage database migration and in this blog entry I will discuss about my little playing with them

DbMigrate

Well I didn’t find any descend documentation about how to use dbmigrate with grails. When I installed the plugin it don’t work and gave me a file not found exception. After going into the plugin source code I figured that it is looking for the resource inside my grails project not in the location where plugin is actually install {dbmigratePluginDir}. And on top of that asking to create version table (db_version) manually is not very developer friendly. Let me know if anyone has used dbmigrate and had better experience than me.

Liquibase

This is a very matured database migration tool but unfortunately has some gotcha’s when it comes to use it with grails. If you haven’t thought about database migration from beginning of the project it’s becomes harder to start with one. When you will run the liquibase plugin for the first time it will show the entire database as change because none of the changes are logged with liquibase. So to register all the changes with liquibase we had to recreate the database schema. Now if that is not an option, taking backup and reloading the data back again is possibly the only way. Well now is the fun part. My spike involved adding a property to an existing domain object and then removing it again, simple isn’t? We test-drive almost all our changes. So in any typical scenario our test database will change before any other environment. So one of the expectations we had liquibase is to detect that change and propagate that to dev and QA environment.

Liquibase plugin does ship with a db-diff tool but unfortunately it compares between dev and test environment. I think I need to explain that little bit more. So according to Liquibase db-diff tool dev is more current than test schema. What that means is if we add a property to a domain object(that means a new column is added in test schema) liquibase will think that the column needs to be dropped from test schema and will generate a drop column change log, weird right? So I had to change the plugin source.

In this case I changed DbDiff.groovy to swap dev and test database between source and target and it worked. But when we tried dropping a property for some reason it got the schema name wrong in it. I had to modify the generated change log. But overall it’s a working solution if you are ready to deal with xml.

Autobase

Autobase is the new and shinny database migration tool for grails. It is actually build over liquibase and has a nice little DSL in groovy. If you have an experience with rails migration you will find this very closely related. The biggest problem with autobase is to make to run successfully over and over again. When I tried to run it first time I got all sort of class not found exception. It turns out that autobase plugin relies on some groovy source that needs to be compiled before using it. And it doesn’t happen automatically. So I basically had to clean the scriptCache (<home>/.grails/scriptCache) and rebuild the entire grails project before running it (I had to do that every time before running grails create-migration). Autobase doesn’t ship with db-diff tool like liquibase so we have to create our own migration scripts. Even though it might sound bad but having little cool groovy dsl is nice and easy.

//Dropping column

changeSet(id:’DropColumnAddedForTest’, author:’Nilanjan’) {

dropColumn(tableName:’company’, columnName:’added_for_test’)

}

Looking at all the things I went through so far I like Autobase better than any other database migration tool. Who doesn’t like DSL these days