I’ve been listening to an inspiring 4-part series on natural wisdom by Juan Vereecken and Roberto Bautista. In this post, I’m going to summarize some of the things that I learned about the wisdom related to ants. If you understand Spanish and want to listen to the messages yourself, they can be downloaded here.

“Four things on earth are small,
yet they are extremely wise:
Ants are creatures of little strength,
yet they store up their food in the summer;
hyraxes are creatures of little power,
yet they make their home in the crags;
locusts have no king,
yet they advance together in ranks;
a lizard can be caught with the hand,
yet it is found in kings’ palaces.” (Proverbs 30:24-28 – NIV)

Ants

Ants are wise because they store up their food in the summer so they’ll have enough in the winter. What is the principle here? Preparation. They get ready for the winter that eventually will come, before it comes. By preparing ourselves for the “winters” that will surely come along, we’re going to be secure.

Now, there are things that we know they’ll come and when they’ll come, such as starting a new job, getting into the university, having a child (for some couples), etc. Besides, there are things we know they’ll come but we don’t know when, such as the loss of a loved one. There are still things that we don’t even know whether they’ll come, such as a rough financial situation, a disease, etc.The question is: what can we do before those things happen? The ants give us the answer: we ought to prepare ourselves beforehand. How can we do that? Some of the ways include:

• Establish firm criteria before things happen – Roberto gave as example two criteria he and his wife established in their own life: one is to not start something new in life with debts and the other is when they can’t seem to come to an agreement over any issue, they’ll look for a third-party opinion. Other firm criteria he mentions include not marrying a non-christian person, not having sexual intercourse before marriage and on and on.
• Have right expectations toward people and toward life – this one is huge! What disappoints us is not the people or the situations of life, what disappoints us is our expectations toward them. Many people tell their spouses or their friends: “You disappointed me!” In fact, they disappointed themselves because of the high expectations they placed upon others. As tough as it may sound, that’s sheer reality. People get frustrated all the time because they have unreal expectations toward others. What’s worse, the people we love the more are the people who let us down the more, because they are the ones we expect the most from. Thefore, let’s learn to have balanced expectations toward people and toward the circumstances of life. This will build up a healthy emotional protection in our lives.
• Eat healthy food and exercise yourself regularly - when you take care of your health by eating appropriately and doing physical exercises, you’re protecting yourself against health problems.
• Save money – it’s important to have a financial reserve to help you when unexpected events happen. Unforeseen things quit being unforeseen when you protect yourself against them in advance.
• Take care of your relationships – with God and with people - there are people who have never bothered to grow their faith, who never spend quality time with God. The time will come when they’ll need God’s help and, by then, will find themselves in trouble because they have not taken time to develop a steady relationship with God beforehand. How can we avoid that to happen in our lives? By spending daily time with God, praying and reading the Scriptures. In addition, we should take good care of our relationships with other people. Choose wise friends and invest time on them as well! Invest time on your main relationships, your family and friends. But do that in a systematic way! For example: every Friday night you’re going to spend with your spouse; every week you’re going to call your loved ones that live away from you, etc.

This simple message, extracted from the life of ants, is full of wisdom. In the next days, take time to think of how you can better prepare yourself in advance against the winters that will eventually come. Don’t forget that the best time for one to do that is in the summer, when things are well, because when the winter comes, it might be too late to do something about it.

Last year I finished reading the excellent book Domain-driven Design, by Eric Evans; but it was not until now that I’ve managed to put aside some time to write about it.

In a time when we have lots of different kinds of cool tools and technologies, it can be easy for us developers to become entangled in those issues and forget to pay close attention to the core problem domain. The approach of Domain-driven Design (DDD) shifts the focus back to the domain, joining together developers, users and domain experts through the use of a common language, spoken and understood by everyone: the ubiquitous language. With the terms of this shared language, we can develop a model which will be used to drive design (hence the term “model-driven design“). The idea is for the code to faithfully reflect this model. This approach enables excellent communication among the stakeholders and empowers the design with the richness of the domain which in turn makes it possible for the system to evolve more easily as the users demand new features.

And that’s Domain-driven Design in a nutshell! Simple, isn’t it?

Yes, that is DDD. However, putting that into practice is a whole different thing. In the book, Eric Evans explains how to apply DDD, what are the some of the challenges and difficulties, how to express the model in code, supple design, patterns, large-scale structures, etc. I definitely recommend reading the book, especially  parts I, II and III.

Just one more comment to finish this post: I came to the conclusion that it is virtually impossible to apply DDD without continuous Refactoring. This happens because the ubiquitous language and the model are constantly evolving as developers and users gain new insights. Actually, coming up with a good model is a strenuous task and, therefore, things are likely to change often (especially at the beginning of the project) and the code needs to easily accomodate these changes. Since it’s unwise to do refactoring without automated tests, some flavor of Test-driven Development is highly desirable as well.

Different from what happens in other countries, here in Brazil the candidates for a job position in public companies must pass these kinds of exams; and they are highly competitive. In the article, we go over the advantages and (a few) disadvantages of working at public companies, with emphasis on the IT industry. At the end, we present some questions of real exams with answers and comments (the questions concern primarily Object-orientation, design patterns, Java SE and Java EE). We appreciate any feedback on the article.

Nick Kallen, Systems Engineer of Twitter, talked about how he designed and evolved solutions to deal with some of the fundamental types of data of Twitter: tweets, timelines and social graphs. Each of these real time data had different datastore needs. He discussed how the original solution to store the data in MySQL did not scale well and then went on to present the datastore strategies they created to meet their specific needs.  They exploited techniques like master-slave replication with memcached for reads, data partitioning, temporal locality and indices. A recurring pattern to achieve efficiency and scalability that was identified was to partition, replicate and index the data.

Guilherme Silveira talked about REST, semantics and the future of the web. He presented some ideas that can be used such as exploiting the power of links, microformats with hypermedia, addition of semantics and the existence of a shared vocabulary.

Randy Shoup shared some of the best practices for large-scale web sites based on his experience as a chief engineer at eBay. Some pieces of advice were: 1) partition everything; 2) asynch everywhere; 3) automate everything; 4) remember that everything fails. He’d already presented this talk at QCon San Francisco in 2008 and it can be watched here.

Douglas Crockford presented about the future of JavaScript (or ECMAScript as he pointed out). He went through some changes that are being proposed to the language, especially better math support.

Rodrigo Yoshima talked about the method war we are witnessing these days: Scrum vs XP; Scrum vs Kanban; PMBok vs Scrum; Lean vs XP, etc. He reminded that the most important value of Agile is to deliver value more rapidly. We can be applying a handful of agile best practices but if we don’t stick to that one principle we won’t truly be Agile. He also pointed out that there is no such thing as a closed scope in software development.

Paulo Silveira talked about the blurry distinction between architecture and design; and also between design and implementation. He quoted several renowned authors and his conclusion was that “architecture is every decision that impacts in a big trade-off and can or not be difficult to change”. It also can be evolutionary (last responsible moment). When questioned about whether it makes sense to be a “java architect”, he replied with a quote by Joel Spolsky that says we need at least one architect with great experience on the chosen platform. Paulo then went on to discuss (with some real examples) the role of  design as a key element to being able to make changes both to the implementation and the architecture of a software system.

The event had several other good talks that I won’t describe here. The organizers are working to make the talks available on InfoQ.

 // TODO: This class is doing too many things. Refactor!
 public class BloatedClass {

    // 40-something methods

}

That’s it! Now you move on with your next task feeling good about yourself because you think you’ve already handled the problem “appropriately”.

“What’s wrong with that?” – you may ask – “after all, I do intend to deal with that later” . Come on, my friend! Let’s be realistic and face the facts! Chances are you’ll forget about that comment and it will remain there forever since neither you nor anyone else on the team will have the guts to carry out that refactoring afterward for fear that something might stop working. This happens because if you remember that comment, you’ll probably already have lost the general context that surrounds that code and thus you may not have enough confidence to refactor. Worse comes to worst when there isn’t a good suite of automated tests to back you up. As a consequence, your fellow workers will have to work and get along with that poor legacy code.

Bottom line: it’s best to try to keep yourself from adding TODO comments to your source code in the first place (in fact, this applies to code comments in general); but, if you do, make sure you don’t forget about them (Eclipse has a “Tasks view”, which can be useful to track those notes). Those comments might be indications of technical debt that you need to be mindful of, yet more often they’ll just be deodorant for poorly written code.

After nearly two years of procrastination, I finally finished reading Fred Brook’s classic “The mythical man-month: essays on software engineering”. It has lots of useful insights on software project management and development. Rather than making a summary, I’ll just go over the pivotal concept found in the book: conceptual integrity.

The author goes at great lengths in how to achieve conceptual integrity in our software products. From my understanding, he considers conceptual integrity as a user-centered view of the software system; and, therefore, it’s something deeply connected to ease of use. For the author, this is the most important thing to consider in system design. There are severe challenges that stem from large programming projects, where division of labor can make it harder to preserve conceptual integrity. And that’s why he suggests the role of the architect, who is the person responsible to envision the system as a whole and make sure the conceptual integrity will be preserved. As I mentioned, Brooks used this concept primarily to talk about the software product as perceived by its users. In my view, the idea of conceptual integrity is one also to be pursued and applied to the more inferior levels of the software system (e.g. software design, code). On a project using Agile practices, such as XP‘s collective code ownership and emergent design, good communication is crucial to achieving conceptual integrity in code.

The book is mandatory reading  for any developer or software development manager. The phrase “there is no silver bullet”, for example, which has become something of a cliché in our industry, comes from an author’s paper “No Silver Bullet – essence and accident” and it was included as a chapter in the book.

Nevertheless, I do consider some parts of the book to be dated (the book was first published in 1975!). If you have little time, in a first reading consider reading chapters 1, 2, 3, 4, 7, 10, 11, 13, 16 (No silver bullet), 17, 18 (this chapter is a nice summary of the whole book) and 19 (this is a review of the original publication after 20 years). After reading this book, you may also consider reading the author’s new publication “The design of design” (I myself haven’t read it yet, but have seen good comments about it).

As a side note, I liked seeing the author use some bible-inspired thoughts to make some of his points in a technical book. A few areas of knowledge (e.g.: management, oceanography, etc.) have already benefited somehow from the wisdom found in the Scriptures. We still have a whole lot to learn in our craft and I really think the Word of God can be a source of inspiration no matter the subject matter. I finish this post with a quote by Brook’s himself:

“We need a God-given humility to recognize our fallibility and limitations”.

Among other benefits, with the blog I force myself to write down book reviews, which makes me review and ponder a little about what I read. I also like to summarize the talks I hear in events I go about software development as well as Christian messages. For this second year, I’d like to increase the number of general posts related to something other than books and events.

The blog still has modest numbers: 22 posts, 30 comments and 1,976 views (tracked by WordPress). I hope to improve on those.

“I will exalt you, my God the King;
I will praise your name for ever and ever.” (Psalm 145:1)

Leandro Silva talked about how a company, leading in its segment, completely failed with the adoption of Scrum. The company’s worst mistakes were: 1) lack of knowledge on agile; 2) lack of support from the IT directors; 3) “hybrid adoption” (they tried to adapt the practices with no support and at the same time satisfy the company’s culture). As Leandro pointed out, if the company’s culture does not change, it’s difficult to be agile. It’s necessary to influence and change the mindset of the people that make up the company.

James Lewis talked about agile adoption anti-patterns, which he defined as the following: “Stuff I see as a practitioner time and time again… and I wish I didn’t”. The anti-patterns he presented are: 1) Flaccid technical lead (lack of technical direction), 2) Inverted servant leader, 3) Novice in charge, 4) Change everything at once, 5) Be careful what you wish for, 6) Novices don’t know what they want, 7) New architecture team, 8 ) Process mavens (you have a team whose job is to create the best process and then tell the others to use it), 9) Stagnant agile, 10) Agile adapter and 11) Hamstrung Scrum (this is taking some practices of Scrum and neglecting the best engineering practices – TDD, CI, etc.). The root causes for these anti-patterns include issues related to organizational change and the fact that people have a hard time changing. James pointed out that there are local and global optimizations, and sometimes incremental change is not enough. People will break, we must deal with it. In addition, he mentioned the Dreyfus Model of skill acquisition and, in particular, a workshop Pat Kua put together, applying the Dreyfus model to pair programming. In regard to changes to teams, James quoted Conway’s law: “organizations which design systems … are constrained to produce designs which are copies of the communication structures of these organizations”.

Klauss Wuestefeld closed the conference talking about the values and practices of XP after his 10 years of experience. He started by using a few analogies to explain the nature of software development, such as software development as a cooperative game, osmosis and fragments, the fractal nature of software and project vs projectile. He then gave a nice metaphor for software: a representation of what the team knew at a specific moment in time. This definition eliminates the view of software as a product of construction and connects it to the concept of learning. Together with learning, he adds coolness as the ‘new’ principles. As to the practices, Klaus sticks with the following: planning on demand,  periscope,  forget feature requests (“let your customer remind you what is important” – Getting Real – 37 signals), continuous deployment and cleaners (this last one means having people to clean things… I myself prefer the Boy Scout Rule).

Overall, this was a nice conference, well-organized, held in a great venue and in a great city.

Hope to see you in Agile Brazil 2011!

I just had the opportunity to participate in Agile Brazil, our national conference on agile methods. It was worth attending, I heard some good talks and also enjoyed traveling to Porto Alegre. In this post, I’ll summarize some of the talks of the first day of the conference.

Martin Fowler opened the event with a keynote encompassing three topics: the essence of agile, the value of software design and continuous integration and delivery. Agile methods appeared as an attempt to change the chaotic situation software development was in, in the 90′s: late projects, unhappy users, buggy software, etc. People from the Smalltalk community, Scrum, FDD, XP and others got together and created the Agile Manifesto for Software Development. Martin mentioned the semantic diffusion problem – how ideas can quickly get lost, so it’s important to remind the key points. He then compared agile with plan-driven approaches to software development.  The former emphasizes adaptive planning (1 - release early and often; 2 – learn), whereas the latter values predictive planning (sucess = being in accordance to the plan). Predictive planning has its strengths, but has weaknesses, and a major drawback of them is depending on requirements stability. But the question is: in software development, how stable can our requirements be? So they try to stabilize requirements by using change control, sign-offs and up-front requirements gathering. As opposed to that, in adaptive planning, the plan is seen as a tool rather than a prediction, “a late change in requirements is a competitive  advantage” (Mary Poppendieck). Adaptive planning gives rise to evolutionary design, which involves self-testing code, continuous integration, refactoring and simple design. Martin also compared the two approaches in regard to people vs process. Agile methods put people first, while plan-driven approaches put processes first. Plan-driven processes design the process first and then “slot the people in”. They don’t realize that “a bad process will beat a good person every time” ( W. Edwards Demming).

Martin went on to his next short talk about the value of software design. People sometimes cut down on quality to add more features to a software in less time. Quality has two dimensions: external quality and internal quality. The former is visible to the users and customers and comprises things like a pleasant user interface and few defects. The latter comprises good modular design. The reason we care about internal quality is referred by Martin is the design stamina hypothesis. It comes down to how much effort it takes to add new features. With good design, over time we can add more features quickly and more cheaply. He also talked about code base complexity (essential complexities and accidental complexities.) and the metaphor of technical debt. In particular, he emphasized the technical debt quadrant and remembered that a mess a not a technical debt. Just like in our finances, sometimes it’s necessary to take a prudent amount of debt. “The very best team will produce technical debt, we need to manage the quality of our debt”, he concluded.

Martin’s last talk was about Continuous Integration and Delivery, which I have already commented in a previous post.

Alisson Vale talked about Kanban, a process based on the toyota production system whose main idea is to limit the Work in Progress (WIP). He talked about the differences between manufacturing and knowledge work, and that the essence of this difference lies in the concept of variability. In manufacturing, variability is always harmful and needs to be avoided. In knowledge work, variability is inherent to the activity and we need to adapt to it. Kanban’s principles can be adapted to software, but the way it is implemented needs to be adapted for the work’s context and nature. When we speak about Kanban in software development, we have three guidelines: 1- make work visible , 2 – limit WIP and 3 – help work to flow. Alisson also stressed that Kanban is not a methodology for process management, it requires a pre-existing process/methodology. Kanban is a starting point to continuous improvement.

David Hussman talked about products and people over process and technology. He gave an informal talk with some good advice. First, he pointed out the idea of learning by comparing and indicated three “agile” books: the Black Swan, Freakonomics and Blink. He also mentioned the Checklist Manifesto – how to get things right. David said we have three kinds of problems: the simple (e.g.: preparing a cake), the complicated (e.g.: sending a rocket) and the complex (e.g.: being a parent – though for this last one we have the biblical advice in proverbs 22:6). The complicated problems are difficult but have a solution. The complex ones have no defined solution. Agile methods help with dealing with complex problems. What do we know about the people we’re making software for? We have two aspects to consider: Discovery (what and why) and Delivery (how and when). Are we building the right thing? Are we doing it in time? Most companies are just concerned about the last question. Our focus should be in continuous product learning. The question we should be continually asking ourselves should be: “why are we doing what we are doing?” David concluded by giving the equation: value = why / how.

Paulo Camara and Daniel Vieira Magalhães, from Ci&T,  talked about how it is possible to have a lean software architecture. The idea is to avoid BDUF and “see the whole”. Is it useful to have an architect? In their view, yes, it is, but what we need is an architect with a different profile. That will be the person who’ll bring the necessary skills (technical knowledge) and experience. This architect should be 100% hands-on, a leader not a boss, an example, a reference and his major ambition should be to become dispensable (this reminded me of this article). They like to do on their projects what they call “setup sprints”, one or two week sprints whose objective is to address the “high stake constraints”. For the production sprints, the focus is on collaborative design. Architecture activities are tasks associated to stories and it needs to be a competence of the team. Lastly, they quoted some lean principles found in Mary and Tom Poppendieck’s book Lean Software Development (eliminate waste, amplify learning, decide as late as possible, deliver as fast as possible, etc.). I really liked to see Agile gaining popularity in a CMMi-5 company.

Rodrigo de Toledo and Daniel Teixeira talked about their experience of three years using Scrum on a Research and Development project for the Oil Industry. The system that was to be developed was for integrated 3D visualization for exploration and production. The project started in 2006 and they spent one year just discussing how the system would be. It was one year with no software! By this time, some team members got the opportunity to learn Scrum and set out to use it on this project. Nevertheless, they had a challenge:  how to do R&D with Scrum? They came up with research stories. These stories had limited scope, they were time-boxed (e.g.: one or two weeks to research) and paper-boxed (e.g.: read the three most relevant papers on the topic). The acceptance criteria for these stories included documentation generated in a wiki and the capability to assess the development story.

Mauricio Aniche talked about common deviations when applying Test-driven development (TDD). He started off by reminding the audience that TDD is about design and for one to do TDD effectively it’s necessary to explicit dependencies. He took a survey asking lots of developers, both experienced and beginners with TDD, whether they followed some TDD’s principles all the time, rarely, never, etc. The principles he used were: 1- not seeing the test fail, 2- forgetting to refactor, 3-refactoring another part of the code, 4-not starting with the simplest test first, 5- running only the current test, 6- writing complex tests, 7- not implementing the simplest code to make the test run, 8- giving unclear names for tests and 9- not refactoring the test code. The most common deviations were: writing complex tests, forgetting to refactor and refactoring another piece of code. He concluded by saying we should take into account our experience when deciding whether to break a TDD principle or not, but we should be careful since breaking the rules all the time can be dangerous. More information about his survey can be found here.

To conclude the first day of the conference, Bruno Pedroso presented his ideas about the mixture of the following techniques: GTD, Scrum, Pomodoro and TDD. After giving an overview of each technique, he described the fractal nature of each one. For him, GTD focuses on “me”, Scrum focuses on “us”, Pomodoro focuses on “now” and TDD focuses on “code”. We should balance static and dynamic quality, we need to balance the period when we think and when we do and the principles are: 1) don’t mixture, 2) make things explicit and 3) conclude often.