(SFSSDP) Suzanne's Fancy Schmancy software development process¶
This page documents the software development process I have been using over the years. It is by no means perfect, and there are always compromises due to the reality of the individual projects.
We usually receive a more or less formal request to develop a software application. This request should be approved by my managers before I spent any time on it.
At this time, I want to understand if it is funded, and how important and urgent it is. If there is a reasonable amount of funding, importance, and urgency, it is worth going to the next phase which is writing the requirements.
Requirements¶The requirements define and limit the scope of the project and ultimately become the contract between the developer and the requester. It is the most important phase, because the most devastating and expensive errors are made here. Usually this is by misunderstanding the intent of the software, the scope, or the users. It is also a very problematic phase because a group of people with very different opinions need to converge. Below are some points to keep in mind.
- The requirements state WHAT a project will do. There is a tendency to include HOW it will be done, but one should leave that to the design. An explicit effort has to be made to prevent design decisions to creep in.
- It is common that the requester has not thought through the project, and is either unwilling or incapable to articulate what exactly this software will do. The requester usually wants to see a prototype, and agree or disagree on the implemented functionality. This is a very expensive way to develop software, it is much quicker to think it through and sketch it on paper (hours vs. weeks) . Changes will be inevitable, but to write down what the vision is helps limit the changes.
- I have learned that there is a correlation between the willingness of a requester to write (or at least review) the requirements and the importance of the project. In other words, if a requester is not willing to spend 2 hours reviewing the requirements, it is not worth months of my time to develop it.
- The requirements give me an idea of how many users will use this project. There is a difference developing software for 2 people than for 200. This is also an indicator of the project’s importance.
- It is advantageous to include people who resist the project or have contradictory opinions to the requester. I want to have these arguments on paper, I don't want to make changes to the code to accommodate contradictory requirements.
- I have settled on a requirements methodology that works for me. I list the authors, and the reviewers. I put the requirements into a table each has a number, the date and requester. I separate requirements if they cover multiple aspects, and a requirement must be specific and testable. I use the word ‘shall’ in all requirements, and I assign them a priority (Critical, Expected, Desired).
- I recommend to wait with the design until the requirements have been reviewed and there is consensus. I found that once the design starts, it is very difficult to go back to separate the design from the requirements. Starting the design is analogous to starting construction on a building without having the contract to actually build it. You first want to make sure you client has agreed to the contract.
- It usually takes a few weeks to write the requirements and about twice as long to have them reviewed. This is time well spent, consensus on requirements is analogous to quality.
- Tools are available for the requirements process. My experience is, it is not a matter of tools, it is a matter of convincing the requester and reviewers to spend the time articulating the requirements and going through the necessary arguments for a consensus.
- The requirements document must have version control.
If the previous two steps have been successful, meaning that we have agreed on the scope, the functionality, and the usefulness of the project, we proceed to write the high level design. Often this is a private document used within the developer team. If it is a critical and high-visibility project (for example Role Based Access for the LHC controls), the high level design needs to be written up formally and available for review. It must be defendable and meet the requirements. On a small project that is not critical (for example Tier 0 Monitor), the design does not need extensive documentation. The developer still needs to be able to explain it and defend it, but it is really up to her to choose a design.
At this time, some prototyping may be useful, however, most of the time, a prototype is a waste of time and is an excuse for rushing into coding.
I usually resist the pressure (and there always is) from management to make a schedule before the high level design is agreed upon. At that time the work can be divided into 'chunks' and each chunk can be estimated to be done parallel or separately. At this time, we can make a schedule to deploy the agreed upon requirements with confidence and realistic benchmarks.
Finally the coding starts, this is when the developer team hunker down and no longer communicates with the outside world. It is the fastest phase because there are very few people interactions. Because of the instant gratification and minimal controversy this is the easiest phase for most developers.
Coding skills vary, a junior person should be teamed up with a more senior person, or given a less important and well defined task. I don't like code reviews, I find them tedious and unproductive. I believe everyone has their own coding style, and am tolerant of curly brackets placements. I want to see exception handling, database connection pooling, and limited public methods (entry points). All public methods should be documented.
I believe good communication within the team at this phase is essential, so that everyone feels free to ask questions. For critical pieces, I may ask a developer to present his code. All code that is to be deployed must be under change control.
Unit test: each Java class should have a main method for testing the public interfaces of that class.
Integration test: Usually there are incremental releases. The first may be an integration of the basic code, at which time two pieces developed independently will meet for the first time. For a large project a test plan for integration testing is appropriate. The test plan should include: system configuration, software version, tester, date, and a list of numbered and reproduce able tests. When executing the tests, the test plan is updated with the results. If there are performance tests to be run, these should be specified ahead of time. If there are requirements, the test plan can be written using the requirements.
System test: Again, a test plan is in order for large systems. It is extremely useful to have reproduce able tests that can be used for subsequent releases and regression testing. For smaller projects, informal testing of all features can be done.
This is usually a very painful phase. It involves learning and following the specific deployment idiosyncrasies of the project requester. For example CMS want an RPM with all the dependencies. There is no choice but to learn how to make a script to build the RMP. The main mistake is to under estimate how long it takes to find the resource to deploy the software and set it up correctly. Initial deployment may take weeks because if this.
At this time, the requirement have probably changed or rather evolved. There may have been some obvious mistakes, or some requirements just could not be met. I usually write a new revision of the requirements document, just for the sake of documenting what we have actually implemented. However, the reviewers are usually no longer interested in this.
Operations¶If we have followed this process we now have a deployed software application, that is being used by real users. In my experience this almost always takes about nine months. I can not explain why, it seems like projects get staffed enough so that the team can deliver the first production release about nine months after the requirements started.
Often months pass between deployment and actual use. It is important that the developer keeps the responsibility for project through this idle time, otherwise it will be abandoned.
When it is really in use, there are many changes to be made. At this time extreme flexibility and good judgment is needed to choose the changes to implement to keep this software viable. There are three categories of change requests:
- The obvious requests for changes will come up over and over again, or they are just plain show stoppers. These need to be addressed as soon as possible.
- Requests for changes that are unreasonable or address only the needs for a single users should be ignored.
- Requests that extend the scope of the project well beyond the requirements should be evaluated for an upgrade with new requirements.At this early stage is it good to have a user willing to work with the application in an alpha stage. However this user may also drive the project in a direction that is too customized for him only.
The application is in operation for a few months maybe a year, it is time for the developer to hand over the application to an administrator. This is always very difficult, usually people do not volunteer to accept support for a software application. Management usually has to agree to this.