Last weekend I took part in QCon Sao Paulo, the Brazilian edition of one of the main worldwide events for software developers and architects. The conference had a strong focus on issues related to scalability, cloud computing and replication in addition to Java, .NET, Ruby/Rails and Agile. Redis and memcached were two of the most mentioned tools used to deal with scalability issues.
Nick Kallen, Systems Engineer of Twitter, talked about how he designed and evolved solutions to deal with some of the fundamental types of data of Twitter: tweets, timelines and social graphs. Each of these real time data had different datastore needs. He discussed how the original solution to store the data in MySQL did not scale well and then went on to present the datastore strategies they created to meet their specific needs. They exploited techniques like master-slave replication with memcached for reads, data partitioning, temporal locality and indices. A recurring pattern to achieve efficiency and scalability that was identified was to partition, replicate and index the data.
Guilherme Silveira talked about REST, semantics and the future of the web. He presented some ideas that can be used such as exploiting the power of links, microformats with hypermedia, addition of semantics and the existence of a shared vocabulary.
Randy Shoup shared some of the best practices for large-scale web sites based on his experience as a chief engineer at eBay. Some pieces of advice were: 1) partition everything; 2) asynch everywhere; 3) automate everything; 4) remember that everything fails. He’d already presented this talk at QCon San Francisco in 2008 and it can be watched here.
Rodrigo Yoshima talked about the method war we are witnessing these days: Scrum vs XP; Scrum vs Kanban; PMBok vs Scrum; Lean vs XP, etc. He reminded that the most important value of Agile is to deliver value more rapidly. We can be applying a handful of agile best practices but if we don’t stick to that one principle we won’t truly be Agile. He also pointed out that there is no such thing as a closed scope in software development.
Paulo Silveira talked about the blurry distinction between architecture and design; and also between design and implementation. He quoted several renowned authors and his conclusion was that “architecture is every decision that impacts in a big trade-off and can or not be difficult to change”. It also can be evolutionary (last responsible moment). When questioned about whether it makes sense to be a “java architect”, he replied with a quote by Joel Spolsky that says we need at least one architect with great experience on the chosen platform. Paulo then went on to discuss (with some real examples) the role of design as a key element to being able to make changes both to the implementation and the architecture of a software system.
The event had several other good talks that I won’t describe here. The organizers are working to make the talks available on InfoQ.