Efficiency considered harmful

This is a post about a pattern which I`ve been noticing for quite a while on a subconscious level, but which recently became painfully obvious. It happened, as usual, at a client location. I`m talking about the fact that completing a project ahead of schedule is bad. Very bad. Let me explain. We all already know (and love) the typical atmosphere that management at most organizations create - an atmosphere that is super-conducive to getting real work done. Not. Further, from a project management perspective (and I`m not being facetious anymore), estimation is hard. It just is. Whatever methods and models you employ to try and make estimates accurate (how`s that for an oxymoron?), you just have to ultimately rely on heuristics. Bring in a bunch of experienced programmers/architects (after all, architects know best) into the room and have them give you an idea of how long the project will take and how many people need to be on it. Despite my biting sarcasm, I`m actually being serious about one thing - the fact that programmer heuristics is the best way to estimate. I say this more than just because I think that the people who are actually going to do the job should say how long it will take, but because they factor in the environment they`re working in. Load factors or project velocity depends on more than just the skills of the team members. If the environment (management, company policies, politics, "the system") make it hard to get work done, well then - less of it will get done. People working in such typical soul-crushing environments adapt to it - as any human will when faced with something less than perfect (and unchangeable.) They throw in the towel, and become part of the system. They behave the same way the system wants them to - and become risk averse and rule-following sheep with no enthusiasm left for anything other than doing what makes them look best for the year-end review. They adapt to the measurement process and also to the rule-keeping process. If this were an IT department of a large company (who would have made that connection?), what do you think will happen when a project comes in ahead of the schedule? Let me tell you what happened at a recent client - they were crucified. Why? Because it was clearly a case of rigging the schedule. Dishonesty. Everyone knows that projects can either be late, or they can be on time. Early? What? Huh? Everyone knows what early means - the project manager (and therefore the team) had bloated the estimates to begin with so that they can look good to upper management when they complete "on time". They should now be crucified because of this sin. And they do. So what happens next? The adaptation process kicks in. A project just can not come in early - a couple of days late (just fashionably so, leaving scope for last minute "heroics") becomes the norm. Or late, of course, for that is always allowed and accepted (with a knowing, friendly shrug and perhaps a nod). Scheduling and estimates are still bloated, but work expands to fill the available time, and the finish lines are always met appropriately. Works out nicely. Just never forget the old software development pattern - being efficient and coming out ahead of the schedule is considered harmful.

Recruiting and the Butterfly Effect

Everyone has heard of the Butteryfly Effect. (Maybe you`ve even seen the movie by that name - I`m sorry, if you have!). Anyway, according to WikiPedia - "The Butterfly Effect is a phrase that encapsulates the more technical notion of `sensitive dependence on initial conditions` in chaos theory. The idea is that small variations in the initial conditions of a dynamic system produce large variations in the long term behavior of the system." When a startup is founded, it is typically done so by a few friends. They`re like-minded or at the very least, focused on the same goal. They are also pretty good at what they do (despite the fact that most startups fail). Without me having to list a whole bunch of wonderful characteristics of this set of people, lets just agree that they all have some desirable characteristics. Not only that, but these folks make the bar that was set on the standard around these characteristics. When this company grows to the point where they need to hire more people, they are extremely careful in making sure that they hire people like themselves. They hire all the new folks themselves, because they want to preserve the quality of the people that work with them and are like the "model-employee" (geeky, brilliant, marketing genius, a combination of these, whatever). Being personally involved in hiring (interviewing - I use the term hiring to mean interviewing from here on) can continue upto a point - and then they can no longer do this. Already, because of the fact that no two human-beings are perfectly alike, they have no real model employees, just very smart people who have most of the desirable properties in sufficient amounts. Let us call this deviation from the model, d1 and label this set of employees who were personally picked out as wave-1. The moment someone other than themselves start doing the hiring (wave-2 and so on), the deviations start compounding. This is because everyone in their mind creates a mental model of the ideal employee that differs, atleast ever-so-slightly, from what they understood was the "model-employee" from the people that hired them. This is inevitable - these are all smart people who are empowered to draw inferences and make decisions. We now therefore have wave-2 and deviation d2. Total deviation at this point is not just d2, but d1d2. If wave-2 starts hiring further employees, deviations start compounding - d1d2d3...dN. There comes a time, when the total deviation from what was desired is now more than a tolerable threshold and things are different from what they were meant to be. This is a manifestation of the butterfly effect. This is because, in the end, everything (for eg. working at a company) can be reduced to the people involved. This is what probably happened to most large companies of today. What is my conclusion? People with as low a wave-number as possible, must stay involved in hiring. You can not expect wave-Z folks with a deviation of d1d2*d3...*dZ to be able to hire people who are like wave-1 or wave-2. ThoughtWorks as a company is at times, guilty of doing this and I think a part of the blame is attributable to the fact that its growing too fast - it has become less feasible to apply this conclusion. Of course, a lot of caveats apply - things change over time - models change and desirabilities change. But I think this hypothesis still holds good. At the very least, it is useful to recognize this phenomenon. To collaboratively create something that is as subjective and personal as "a great company" and to be able to stay as close as possible to this vision , it is imperative to keep people who buy-in and fit-in to the vision the most, completely involved in the inevitable expansion process. Easier said than done.

On becoming a master

For a while now, I`ve been convinced that it is becoming increasingly important for regular software developers to really know Computer Science. Yes, the internals that you don`t really care about `cause all you care about is "business value". Yes, I`m talking about finite automata, algorithms, operating systems, compiler theory, artificial intelligence, functional programming... Without this knowledge and the deep understanding of where to apply it, it is very hard to progress to beyond journeyman. I`ve also realized that what many people claim is master-level of competence, is really not quite the truth. If someone who is supposed to be a master programmer doesn`t understand computer science internals, then he or she is really not all that masterful. Those who think that these are merely details left to the systems guys are just plain wrong. While you may not write the next RSA, the enlightenment that all this knowledge gives you, when coupled with actually applying this theory to "application development" will take you to the next level.

On the high cost of everything

I`ve decided that I have figured out why everything is so very expensive. IT is a major part of every industry today. This is because IT promises levels of productivity previously unimaginable. It also promises more reliable and easier ways of conducting business. However, IT costs a lot. People in IT are one of the highest paid lot in the world of salaried employees. Perhaps rightly so - IT is a knowledge driven industry. In theory. Here`s the thing. From the past few years of experience as a consultant and therefore having worked with IT people at clients` from different industries, I can say one thing with a fair amount of confidence: most IT people suck. Big time. Not only do they suck, but since most of them suck, they form a sort of syndicate of people who engage in pretend-work and who like to spend time in meetings. They create policies to help create the need for this type of non-work. They revel in getting nothing done and making it look like work. The net result, is that tens of billions of dollars is spent every year in keeping this non-work-force on the payrolls of these organizations. And since these people are incompetent and love non-work, in an ironic attempt at saving costs, they outsource to equally incompetent consulting organizations that do the same non-work (or only slightly better) and bill them more money. I have personally seen tens of millions of dollars go down the drain in failed projects. Where does all this money come from? In the end, from the consumers of the services or products of the orgnization, of course! That means, from you and me. This drives the cost per unit of these services or products up. This is what I mean. P.S. All qualifications apply. I know not all organizations are like this and that not all people working at these companies are like this. 95% are, though.

CVS Quick Reference

I`ve looked up this information in the Open Source Development With CVS book so many times that I decided to put it up here so others who might want it can get at it just as easily as I can. Creating a repository: cvs -d init Importing a module: cvs import -m Checking a module out: cvs checkout Adding and checking files in: `nbsp;`nbsp;`nbsp; `nbsp;`nbsp;`nbsp;`nbsp;`nbsp;`nbsp;cvs add -kb `nbsp;`nbsp;`nbsp;`nbsp;`nbsp;`nbsp;cvs -q commit -m `nbsp;`nbsp;`nbsp; Updating a module - be in the same directory as project: cvs update -dR More information can be got either from the cederquist manual or from the book mentioned above - its quite good, actually.

Goal Driven Development

There is something to be said about an almost new kind of thinking about software development. Agile folks probably think of this in many forms subconsciously already for it almost forms the basis of the agile philosophy. This "new" idea is that of business goals - the somewhat quaint idea that any business software that is being developed is to meet some... well, business need. Or goal. Why quaint? Because in the everyday details of requirements, development, QA, deployment, bugs, bug-fixing, maintenance, meetings and what-not, people simply forget what the intent of the system was. Software is written to help improve business. This is the business goal - make more money by doing something in a more efficient manner or making certain people more productive. There is an important difference between these goals and the actual software that is being developed, and it is useful to keep this in mind. Why Agile works, is because of this unconscious essence of Goal Driven Development. The idea that people (business people) do not know what they want (quoted as a reason for changing requirements) is not entirely true. They are clear and quite knowledgeable about their business - it is hard to doubt that. What they may not be entirely clear about is merely the software - which is as it should be, because the software really is subservient to the end-goal - that of greater profitability, productivity or efficiency. They don`t care if a certain story card is implemented one way or another as long as it meets their needs - their business needs. This is fine - business people know their business well and technologists (developers, testers, UI folks, etc.) know technology. I am not implying that techies (used in an umbrella sense of the word - it includes all roles on a software development team) shouldn`t or can`t tell what is good for the business - they can and should. This is because they need to help make the software as good as possible to meet the business goals. What I am saying is that they shouldn`t get caught up in the software itself - so much so, that they forget the intent - the business goals. From the various consulting gigs I`ve been on, I can say with certainty that in many IT shops, this is precisely what happens. When business people (with the help of business "analysts") write out requirements - developers stick to these (or at least try to) to the letter. QA folks test against these and again, log bugs unless the software does everything as specified in the requirements spec. UI guys aren`t happy until the interface does things exactly as their plans had laid them out. They all make their activities about the software - what it looks like, what it does very specifically, etc. They forget about the actual goals. Do all the members on your team know what the business goals of the project you`re working on are? Do they know how it will improve the business of the stakeholders? Do they know what the real goal is? I bet you a dollar, most don`t. They probably don`t even care - since they`re not the business, they`re IT. This is why Agile works out better. The constant involvement of the customer makes it easier to perform constant checks that the software is going to (or already is) deliver business value. Not just that it is matching up to the spec. This is the reason you don`t need to do complete upfront design - in fact, it explains why you shouldn`t. The reason is that business folks don`t know about software! And why should they? They don`t care about much more than getting the desired results from the business point of view. It is the reason they have an IT department or the reason they hire highly paid consultants to come in and do the software. They can and do need help with the details of the software and how it should do what it is meant to. And that is really the industry you and I are in - it is our job to improve the usability of the software and leave it maintainable and extensible and all the good stuff - but most of all so that it achieves the goal it was meant to. Once again, that is why Agile is more successful than other methods of building software. The quick feedback loops with everyone (especially the customer, who doesn`t know much about software) helps in defining and refining the "requirements". This is the cause of changing requirements - they aren`t "changing", they are going through the natural and obvious process of discovery. (And most often, changes are really made to the way the software functions and not to the underlying business process and certainly not to achieve and entirely different business goal). This is what IT teams should allow and encourage, for that is their real jobs - not complaining about how requirements are constantly changing under their feet and that they can not get anything done. Any process that attempts to freeze requirements is therefore inherently flawed. This is what I try to do on project teams - encourage client IT folks to think in the systems-thinking 1 kind of way. It is the bottom-line that matters most, and everyone - the business and IT, needs to work together as a team, to make the company more profitable so that everyone wins. And by the way, the software may help to do that. It is more about the business and less about the system itself. That`s why Agile makes the difference - it helps this kind of thinking, making it possible to create business value faster and of higher quality, with everyone pushing in the same direction. P.S. All disclaimers apply - this is not always the case and there are business folks who understand software and there are technical people who get the business and all that stuff. The description applies to many typical IT shops, not all. I`m also saying only that Agile makes it easier - not that it is the panacea to all software development problems. [1] The Fifth Discipline by Peter M. Senge

IRB tab completion

The ruby IRB comes with tab completion. Here`s how to enable it - Create a file (if it doesn`t already exist) in your home directory called ".irbrc". In it, cut and paste the following lines - IRB.conf[:AUTO_INDENT] = true IRB.conf[:USE_READLINE] = true IRB.conf[:LOADMODULES] = [] unless IRB.conf.key?(:LOADMODULES) unless IRB.conf[:LOAD_MODULES].include?(`irb/completion`) `nbsp;`nbsp; IRB.conf[:LOAD_MODULES] `lt;`lt; `irb/completion` end This should have you all set. When you start IRB the next time, you can press TAB and it will provide code completion. Woo hoo! Troubleshooting - If IRB spits out an error referring to its inability to load readline, then do the following - make sure you have libreadline5 and libncurses5 (dev for both) installed. You can do this using apt-get. Go into you ruby source code directory and then into the ext/readline directory. Type ruby extconf.rb; then make; then make install

Improving team communication - Or on teams and snacks

I was having a discussion with a friend who asked what in my experience had been an effective way to improve team communication and enthusiasm in an agile project room. We discussed various ideas, but later that evening, I remembered this - Here`s a quick and easy way to improve team communication on a local team. On one side of the room, place a table. Place a supply of snacks - candy, cakes, drinks and coffee, fruits etc. Replenish this every time levels diminish. That`s it. People will gather around it to grab snacks and they will talk. People will talk as they eat, they`ll throw candy wrappers at each other and they`ll gel. The level of trust will go up and as people make the transition from colleagues to even friends, aided by the banter and the food, the level of noise in the room will increase somewhat (usually a good sign) and the team will become more effective. It worked for me...!

On DSLs

I`ve been busy, working on two very interesting projects these past several weeks (on the side - my current project for work is unbelievably mind-numbing). Both these projects are in Ruby and use Ruby On Rails. There, got that out of the way. I`ve always been in search of software designs that result in a clean DSL (Domain Specific Language)... lisp style, bottom up, bring the language up to the problem, rather than the other way around, etc. What I`ve managed so far is a set of mini-DSLs in my applications. One for each module, so to speak. And perhaps, that is the way things work in general; rather than the unrealistic goal of building one DSL which is supposed to capture your entire domain - build several, smaller ones, which each capture aspects of the domain.

Yet another Getting Real review

I recently finished reading Getting Real by the 37Signals gang. After which, of course, I decided to chime in with yet another review. I enjoyed reading the book - and I agreed with most of what they said. I have, after all, been immersed in Agile for over 4 years now - and I`m also a firm believer in Interaction Design. An example of a part I disagreed with was about it being okay if team-mates are not co-located. I don`t care how they claim to have worked it out (DHH no longer lives in Copenhagen FWIW), I think team-mates should be in the same room - if possible. Anyway - it is a pleasant enough read, even if it simply re-iterates what many other books already say - the agile series, interaction design series, lean software, theory of constraints etc. If you want actual theory and a deeper understanding, read the full list. If you want instant gratification without the longer term benefits of understanding the concepts behind this stuff, read Getting Real. Heck, read it anyway - its kind of a fun book.

Interaction Design

The Inmates Are Running The Asylum is a book that all software developers ought to read. It is about design - and don`t think architecture - but think interaction design. Interaction design is one of the most neglected subjects within the field of software engineering - and yet is the crucial piece that can make or break a consumer software product. Interaction design is a process of researching users, their needs, their likes and dislikes, their work patterns, their business needs, their goals, profiling all of this, to build software that is cognitively friction-free. Without interaction design, software usually feels raw and clunky - with users having to adapt to the way it works. Software should be soft - it should be malleable enough to adapt to its users. Interaction design provides a way to fulfill that requirement. Alan Cooper does a remarkable job in explaining this - and this book is a great prologue to his somewhat more prescriptive About Face. Check this book out, I`m hoping that if enough software people read it, software will stop being so darn frustrating.

Ruby on hiring Java developers

I thought of one more use of Ruby. If you want to hire Java developers, hire ones who know Ruby. (Or Python (Definitely LISP))

NTFS vs FAT

(or why does Windows complain of low disk space when there is enough available?) This was an issue I was running into, trying to copy a large 17GB file onto my portable, external hard drive. It had about 60 gigs free, and yet, everytime I tried to copy the file, Windows would complain about lack of disk space. It was frustrating. Until I realized, that the external drive was formatted with FAT, which has a 4GB limit on file sizes. Really stupid error message, but what else could one expect from Windows? Anyway, changing that to NTFS fixed the problem.

Look Ma, no Ruby IDE!

Or why the "lack" of an "IDE" is not a big deal for Ruby programmers. Don`t get me wrong - I`d love to see an IntelliJ like IDE for Ruby. However, I`ve been working on a (personal) Ruby/Rails project for a while now - and I really haven`t felt that the lack of refactoring support or navigation support was holding me down in any way. The only thing I can attribute this to is that the design/structure of my application makes the importance of these tools somewhat diminished. I`m just able to design the application in ways that the limited OO capabilities of a language like Java doesn`t allow me to. Meta-programming and features like mixins and real closures increase the expressiveness of code making it possible to break down functionality into extemely small and very clear aspects. This makes the application code-base simply a collection of exremely local peices of functionality. And that makes all the difference. Update 1: One aspect of the above is that design doesn`t have to be restricted to a combination of a single-inheritence-based class hierarchy and composition-based delegation for reuse. Think about that. Update 2: I use Eclipse + RDT (+ RadRails) for editing the code.

SQLite with RubyOnRails

That was nightmarish! All I wanted was SQLite working with Rails. That`s all. And it was quick and easy - on my Ubuntu workstation at home, that is. And then, began 2 hours of gruesome agony - as I tried desperately to get the damn thing to work on my RHEL staging box. I only have shell access to this machine - so no fancy graphical package manager. If I had a package manager, I would have selected libsqlite0-dev (the missing item) as well as libsqlite3-dev just to be safe, and would have saved myself the afore-mentioned two hour hair-pulling. Here are the steps to get SQLite working with Rails - Download sqlite from http://www.sqlite.org/download.html. Compile and install the usual way. Or just get the compiled versions. Install libsqlite-dev (both libsqlite-dev3 AND libsqlite-dev0-> AAARGH!) Download, compile and install swig -> http://www.swig.org/download.html Install the sqlite3-ruby gem Downlad ruby-dbi and follow instructions on http://ruby-dbi.rubyforge.org/ Everything should work now! I tested this with Typo 4.0.3 and it works. Phew!