What does it take to become an architect?

Jul 24, 2004

A note on this post: My views have considerably evolved. I will write an update on full stack engineering sometime. Most of the points, and the sentiments in this post are still valid.

How to become a software architect

Every large software system needs an architect. Although many people clamor to become architects, there is still a lack of qualified architects in the industry. In this article, I will list what I believe to be the essential characteristics of an architect. It is relevant for senior developers and junior architects.

[On setting high standards…] Two things emanate from this changed standard. First, the work becomes more fun. It is deeper, it never gets tiresome or boring, because one can never really attain this standard. One’s work becomes a lifelong work, and one keeps trying and trying. So it becomes very fulfilling, to live in the light of a goal like this. But secondly, it does change what people are trying to do. It takes away from them the everyday, lower-level aspiration that is purely technical in nature, (and which we have come to accept) and replaces it with something deep, which will make a real difference to all of us that inhabit the earth. -- Christopher Alexander

Introduction

During the course of my research I was amazed to learn that all the software that controls the aircraft traffic was written by just a few individuals using fewer code than an average website today. That the code stood for so many years is a testimony not to the static requirements, but to maturity of the programmers.

At one time, programming was only for the elite. Programs were written only by smart individuals, perhaps well versed with algebraic notations of mathematics (no wonder that most programs in those days borrowed those notations). Their programs were limited in scope, simple in integration, and cohesive in planning.

Times change. These days, the range of tasks that computers are expected to do are lot more diverse than before. They are expected to work well with other systems and need to be supported and maintained long after they are designed.

However, the average programmer today is less qualified to write code than before.

Sure, programming has become easier -- it takes a few click and select operations to generate a GUI that used to take better part of a week. Advances in basic computer science are good enough to produce a decent quality parser within hours, instead of months.The "Code, compile, debug cycle" is shorter than what it was before. In short, it takes less "smarts" to write code than before.But, has programming become simpler? Coding may have become less difficult, but designing the system is as complex as before, if not more. Confusing array of technologies, competing choices of platforms, increasing needs to integrate applications -- all these factors made software architecture more demanding than ever.

My thesis is this: An architect is not just an experienced programmer; he is a programmer with a special mindset. It is a qualitative jump from being a programmer. In this essay, I will write down what I believe to be the needed qualities in an architect, in particular, a J2EE architect. (I use the term architect for software architect exclusively throughout this note).

What is an architect?

I often get resumes with architect in title. Judging by these resumes, an architect is supposed to have the following qualities:

Knowledge of UML
Being able to draw architecture diagrams
Experience as a developer

Architect: Someone who knows the difference between that which could be done and that which should be done. -- Larry McVoy

The predominant thought seems to be progrmmers with long years of experience in coding either become managers or architects. Typically, all the people with an interest in technology become architects. But, what is the role of an architect in software development?

Defining the problem in technical terms : Normally, the problem is stated in the terms of the domain -- it could be scientific, business, or any other area where the solution is needed. To solve that problem using the computer, you must be able to state the problem in "computerese", which means, in the terms of established technical frameworks. Typically, it takes:
1. Understanding the problem as explained by the customers
2. Translating into the computer solution domain
3. Guiding the customer to well understood problem-solution patterns, and
4. Understanding where true innovation is required
Specifying the technologies needed: A given problem can be solved with diverse combinations of technologies. It is the architect's job to specify what technologies to be used.
Defining the major modules of the system: The architect also must break down the system into modules and define interactions between modules. In fact, she should also specify the guidelines of interactions so that the module dependencies are well understood.
Recommending patterns and frameworks: Since there is nothing new under the sun, each problem, in some fashion must have been solved earlier. It is the architect that should identify the relevant previous patterns to apply in the current project. In addition, she may need to define new patterns to be used in the application.
Establishing code quality measures: For example, code reviews and design reviews fall under the code quality measures. The architect should establish the guidelines and practices in the team.
Establishing Best Practices: Any given task can be done in many ways, some of which are equally good. However, an architect must establish best practices for doing even these mundane tasks so that consistency is maintained, and unnecessary effort is spared in looking for solutions.
Maintaining conceptual integrity of the application: As the project grows, requirements change; new technologies get added; new people come into the project. With all these changes, it becomes imperative to maintain a coherent and consistent vision of the project. An architect should be able to do that.

With these duties in mind, I am going to list the characteristics of a successful architect.

Personality traits

To be successful architect, you must have the following traits:

You should be passionate: Go back to the quote by Christopher Alexander in the beginning of this note. As it said, as long you are doing a job, you should try to do it as best as anybody else in the world can do it. Your pride and passion in your work makes you the strongest critic of your own work.
You should be inquisitive: In these days, there are far too many new developments in the technological front. Unless you are naturally curious, you may miss the right tools for the job. If you do not try these tools by yourself, you will end up taking recommendations from others, worse -- from the vendors.
You should be skeptical: In the era of hype and buzzwords, it takes a special talent to remain inquisitive, passionate and yet the same time skeptical about the technical advances. Understanding the history of software provides a good dose of reality, enough to make anybody a skeptic.
You should be honest: It takes intellectual honesty to acknowledge one's shortcomings and learn. It also helps in serving the customers and your colleagues better.
You should be courageous: Without courage to speak out and confidence to make recommendations, you will never be able to make a difference in the state of the world, especially, in the world of software.
You should be compassionate: Unless you understand the impact of your actions you will not make the correct technical choices. It is easy to lose sight of the impact on people when working with machines. As an architect, the choices you make will effect not only you, but also your colleagues and customers, not only now but also even in future.
You should be quality conscious: Quality of a work reflects in the details. Paying attention to even the smallest details and balancing them with the realities of the project is the hallmark of a true architect.
You should be ready to convince others: As an architect, you should be able to convince your team members, managers, and clients of your vision. It means having the patience to explain to the outsiders, the interest to mentor the junior team members, and the humility to listen and learn from others.
Above all else, you should have good taste: Good taste, while being subjective, is easy to recognize. An architect should develop good taste that leads to consistency of vision, simplicity of design, and cleanliness of implementation.

Technical Experience

It takes experience to become an architect. This sort of experience cannot be counted in the number of years, but in the amount of wisdom gained from it. If you want to become an architect, you should try to acquire and master these skills.

You must know the basics of computers: There are several aspects to the basics of working with a computer. These basics will help you understand the nature of solutions that people are used to. It lets you anchor your ideas with familiar simple patterns.
1. You should master at least one OS: I prefer people to master one version of Unix (or Linux) for it encourages people to explore "under the hood". However, any OS will do for this purpose. To truly master an OS, you should be able to install, maintain, manage, and make it do whatever you want to do.
2. You should master at least one scripting language: Turns out that most of the time, we manipulate text, whether we are writing code, generating code, generating data, analyzing test results, or preparing reports. A scripting language that lets you do these tasks efficiently will make you effective.
3. You should master at least one High Level Language: It goes without saying that at least one main stream HLL (C++, Java, C#, C) is absolutely essential. Of course, it is even better if you know languages like LISP, to understand the possibilities of language design.
You should master the tools of trade: Since you work with these tools no matter what you do, mastering them makes you efficient. Be sure to choose the tools that are ubiquitous or that you can carry with you.
1. You should master at least one editor: Since editor is the most used program, mastering it is essential for an architect. I use Emacs, but any editor with extensibility and macros will do.
2. You should master at least one document producing system: Producing well-written, informative, and aesthetically pleasing documents is a needed skill for an architect. The most popular tools to develop such documentation is MS Office. Several Open Source proponents use SGML/HTML/XML. LaTeX, TeX is the most used system in Academia. The market you are pursuing may best dictate your choice.
You should understand the basics of computer science: Many people learn programming on street corners. While the practice of programming is just that -- practice, it can benefit from good understanding of basics of computer science. For example, I find myself using the data driven abstractions in all my designs.
1. You should understand data models: Most computer programs are concerned with collecting, classifying, managing, and presenting data. As such, understanding data models, schemas, and databases is an important part of an architect's portfolio.
2. You should understand transactions: There are far too many myths associated with transactions. An architect understands where transactions are needed, where they are superfluous, and where they can be simulated in programs and in databases.
3. You should understand three fundamental Programming Constructs: All programs are built up using the same techniques: combination of the basic blocks, abstraction to create new building blocks, and application of these abstractions. Understanding and practicing these fundamental techniques in proofs, code, or documents is an experience that an architect should have.
4. You should understand higher-order functions, and data-driven abstractions:

"Any sufficiently complicated C or Fortran program contains an ad hoc informally-specified bug-ridden slow implementation of half of Common Lisp." - Phil Greenspun

One of the original models of programming languages, lambda calculus has no data; instead it is represented as functions. These days, it is common to abstract programs as declarative data, particularly in programs where flexibility is desired. A proper understanding of these abstractions is necessary for an architect to create good architectural patterns.

You should learn the craft of programming and managing software through practice: Any architect should write code. Without writing code, they cannot meaningfully design the system. In fact, through this practice they should learn a few things:
1. You should have, through practice, a set of preferred solutions that you thoroughly understand: A friend of mine has several standard templates for solving the problems with the tools that he knows well. For example, he could put together any workflow automation with email and procmail. Philip Greenspun used AOL Server, TCL, and Oracle to solve his problems. Joel Splosky uses Excel to solve his problems. Basically, you need to develop trusted frameworks that can help you solve your problems. Since you practiced these technologies already, you would not waste time in interoperability issues; more over, you would have evolved best practices in these frameworks.
One aspect of these technologies is that they enable you to develop quick prototypes. Often as an architect, you need to prototype to validate some ideas. Having a least expensive way of prototyping a realistic application can help you make the right choices.
1. You should understand abstractions for standard problems: Using patterns is an effective way to capture trusted solutions for most common problems. However, patterns are too concrete to capture several problems. There are standard abstractions that any architect must be familiar with. A partial list includes:
  1. Use of caching: Caching occurs in all ORM solutions, and in any solution where you keep data in multiple places. There are standard solutions with different characteristics -- real time coherence, delayed coherence, and on-demand coherence.
  2. Use of views: Most problems are concerned with maintaining different views of data. Even the MVC paradigm is about presenting a different view of the data. Most of the data analysis packages present different views of data.
  3. Use of declarative extensions: To extend a static program, there are different strategies ranging from command line options to dynamically loading extensions to XML based configuration files. Some systems provide scripting extensions.
2. You should understand the standard software development practices: A partial list includes
  1. programming life cycles -- XP practices, Waterfall methods, RUP,
  2. requirements writing,
  3. version control,
  4. development environment configuration,
  5. logging,
  6. debugging,
  7. documentation,
  8. building process,
  9. testing process,
  10. release process, and
  11. bug tracking process.
You should learn to work with a team: Since your decisions as an architect impact the whole team, you should learn to work with the team. I am not going to list out the rudiments of team play, but I would like to point out the technical skills required:
1. You should be able to write well: Most programmers communicate more with people than machines during the course of a project. In fact, even writing programs is predominantly to communicate with people. It is no wonder that for an architect, being able to write well is one of the most fundamental necessities.
"Programs must be written for people to read, and only incidentally for machines to execute." -- Abelson & Sussman, [SICP](http://www- mitpress.mit.edu/sicp/).
1. You should practice good communication mechanisms: Since most of the knowledge about building a system is captured in conversations, it is imperative that architects should have good understanding of the mechanisms for fostering and developing those communications. For example, understanding on how to set up project portals, how to publish code reviews, code metrics results, how to encourage mails to mailing lists -- all these activities help reduce the learning curve for a new member.
2. You should make work interesting: As it is, working with computers can get boring, unless you can make the work interesting. You should encourage junior members to learn and experiment so that they can gain from experience. For example, you could motivate them by offering a chance to do the following:
  1. learn new technologies,
  2. apply best practices,
  3. learn and documenting patterns and frameworks, and
  4. learn good development principles.
3. You should be available for the whole project life cycle: Unless you are ready to accept the consequences of your choices, you should not make architectural decisions. You make sane choices only if you are forced to think about how a technology is used by the developers and how it is supported by the vendors or community, and how it can be upgraded to support new advances in the other technologies and protocols.

J2EE Knowledge

That's one sure tip-off to the fact that you're being assaulted by an Architecture Astronaut: the incredible amount of bombast; the heroic, utopian grandiloquence; the boastfulness; the complete lack of reality. And people buy it! The business press goes wild! -- Joel Splosky.

Several people wanted to know about what makes an effective J2EE architect. In my opinion, there is nothing significantly different from a J2EE architect and a regular software architect. I am going to take ideas expressed in the preceding sections, and concretize them. That is, wherever possible, I will make specific recommendations.

Learn the basics: If you are a J2EE architect, it makes sense to understand the basics behind Java, JVM, and J2EE. That is to say that you should both read the API documentation and experiment with the fundamentals.
1. You should understand the performance characteristics of Java: Quick! How many objects can you generate per second in Java? How many characters can you read per second? How many can you write? How many JDBC calls can you make? All these questions do not have definite answers; they depend on several factors, some of which are not in the control of the programmer. However, knowing these answers for any representational system will let an architect make rational choices.
2. You should understand the basics of JVM: For example, what options does it take? Can you share code between two JVMs? How do you control class loading? How do you do introspection? How do you write JNI? For all these answers, you must have experimented with your own code so that you understand the rationale behind Java language design choices.
3. You should understand the basics of garbage collection: How do you control, measure, and manipulate memory in Java? How do you track memory management in JNI? How do application servers deal with memory management? You should learn the native diagnostic tools of Java and JVM so that you can debug memory issues in Java.
4. You should know HTML, HTTP, CSS, and Javascript: Ultimately, we deliver html to the browser using HTTP protocol. Several senior developers consider these technologies to be beneath them; after all, either the tools or the designers hide them from the developers. However, these skills can make an architect develop good usable solutions.
Master the tools of the trade: There are several tools used to help with Java development, most of them are open source.
1. Development Environment: Proper configuration of an IDE/Development and testing environment is essential for productivity gains. In addition, by standardizing it, you can improve the process of inducting a new developer into the team.I recommend a good open source IDE like Eclipse. It comes well equipped with several plugins that are useful for various purposes. For example, it has facilities for the following:
2. It lets you create your own templates for Java files and classes.
3. It supports integration with indenting engines like Jalopy.
4. It supports integration with style checkers like Checkstyle.
5. It is well integrated with CVS (supports cvs over ssh!) -- with diffs, annotations, synchronizations, cvs ignores, and options to create new projects in cvs.
6. It supports DB browsing as well.
7. It has a plugin to add Logging automatically.
8. It supports profiling an application so that you can improve performance.
9. Build and test environment: To do builds and automated tests, you should learn the following tools:
10. Build templates for automated builds. Choices are ant and maven.
11. Unit tests with junit.
12. Web test automations with either openSTA or curl.
Practice the craft:
1. You should master at least one end-to-end technology set that solves most problems: For example, you can master the combination: Oracle + Hibernate + Struts + Tomcat + Tiles + JSP. Whichever one you take, make sure that the following hold:
  1. The technologies must be well integrated to support a full solution.
  2. They must be easy to install and configure, so that they can be used for quick prototypes.
  3. They must scale to small to medium projects with minimal modifications.
  4. They must follow standards, so that the solutions can easily be transported to another similar solutions.
2. You should develop best practices: Since Java allows for several different ways to express the same thought, and there are several competing ways to handle patterns, you should develop the best practices. In particular:
  1. Logging: You should know logging well. You should understand how to setup logging guidelines for the whole project.
  2. Exception Handling: You should know and understand the role of exceptions in a project. You should create the right exception hierarchy for the project.
  3. Style: You should establish a standard style for the project. You should understand the repercussions of violating the style guidelines and be able to convince the team.
  4. Coding Templates: Java encourages people to use several small files. By establishing the right templates for class files you can increase maintainability of the code. For example, by adding proper keywords, you can have the version information from the version control mechanism itself. By adding the right javadoc comments, you can eliminate long and verbose changelog, which belongs in the version control.
3. You must practice continuous refactoring: While refactoring is important for any software project, good tools make it feasible for a Java project. You should understand the role of software metrics in maintaining software. You should know how to write code at a domain level, instead of language level. You should know how to establish practices for naming, abstraction, and application of the code.
4. You must know at least one code generation technique: Unlike Lisp, Java does not treat code manipulation as a first class activity in the language. However, all the Java based systems such as containers do generate code. Many templating systems generate code. Code generation is becoming even more popular with Aspects. Therefore, for an architect, learning some code generation techniques are important.
Foster Teamwork: Java and its tools make it easy to foster good teamwork. For example, Eclipse has plugins to access most version control systems. The following tools and practices can help a team communicate well.
1. Version Control: A properly configured CVS can help a team manage its source code well without clobbering changes made by others. In addition, it can show who is responsible for what change. Such tracking ability makes developers more responsive towards unit testing and documenting the changes. Moreover, Eclipse has good support for CVS.
2. Project Information Systems: The Java world has tools that support automated building of project portals so that project can be tracked by mangers and clients. In addition, these portals can help a new developer understand the project better. The two leading choices are: maven and Forrest.
3. Automated Code Quality Measures: These tools can help a team communicate well using the code. For example, code that has good code metrics is easy to read and follow. There are good code metric tools integrated into eclipse.

You must become immune to hype: Even though this point has been said once before, it is worth repeating. In the Java world there is a tendency towards claiming solutions for "grand" problems that are difficult to solve. While the dynamic nature of Java has permitted the design of interesting toolkits, there are no silver bullets for the truly difficult problems. Whenever I read about any product or innovation in Java, I try to count ten reasons why it would not work.

Reading Material

The Wizard Book: Generally acknowledged to be the best Computer Science book on programming, it is a tough read, not because it uses Scheme programming language, but in spite of it. Several concepts it introduces are essential to developing right abstractions in solution development.
Ward's Wiki: This mother of all wikis has collective wisdom of several developers and architects. From the well-informed and diverging opinions presented there, you can learn a lot about software development.
Richard Gabriel's Site: He is one of the Lisp gurus, and the author of an influential article called "worse is better". In particular, his book on software patternsmakes an interesting read.
Pragmatic Programmer's Site: Even though the books are more appropriate for a senior software developer, they are useful to an architect as well. In particular, the eponymous book is a must read for any developer.

To understand the Java world better, there are several resources. Some of them are:

Java.net resource: This site is an unofficial hang-out for several Java projects and weblogs.
Serverside.com resource: All things related to application servers is discussed here.
Apache resources: This site contains several packages available under ASF license, to be used in applications readily.
IBM resources: This site contains several tutorials for a Java developer.