Acceptance test driven development is an agile software development method that uses the Acceptance tests to establish when a requirement meets the client’s expectation. It is an advanced practice of the Test Driven Development method.
Acceptance tests or Customer tests are written by the client together with the developers for every requirement that is part of the specification. Whereas a requirement describes specific behaviour the client wants in the finished software, the Acceptance tests make sure that the requirement was implemented in its completeness and according to the client’s input.
As an example we can use the tried and tested password requirement:
Requirement “When a user is creating an account, he/she is required to choose an outlandishly secure password”
Acceptance Test1 Test that the password contains at least 8 characters with at least one upper case letter, a number and a symbol”
Acceptance Test2 Test that the password does not contain a word from the english dictionary”
Acceptance Test3 Test that the user receives different error messages when the password is too short, contains no upper case letter, no number or no symbol. “
In this case the Acceptance tests define now what an “outlandishly secure password” is and increase the shared understanding for this requirement. And after the tenth iteration I can make sure that these Acceptance tests still pass so I have regression testing built right in.
The many benefits of ATDD
- Acceptance Tests increase the shared understanding of a requirement because they are a product of direct interaction between the client and the developers.
- Clarifies requirements and keeps the developers focused on what the client really wants.
- Drives out ambiguity if certain behaviour is ultimately a bug (behaviour violates either requirement or tests or both) or a change request (behaviour is acceptable within the scope of requirement and acceptance tests but ultimately undesirable).
- Software delivery is now dependent on all Acceptance tests passing and with that defines when a project is done.
- Percentage of passed acceptance tests/all acceptance tests acts as a project progress indicator.
- Free regression tests for the future iterations (especially if automated).
Some ATDD drawbacks
- Client interaction is needed which can prove difficult due to time constraints
- More work for the developer if tests are automated.
- Project progress might be slower due to additional effort.
Automating ATDD
There are many tools and frameworks for acceptance test automation, even some that can be used for User Interface testing. Many of them are free and open source, so the framework can be adapted to suit the project. Automating acceptance tests is highly desirable because regression testing is a hugely important step for iterative agile development methods. The immediate drawback is that this also means the developer has even more work to do because test automation is a development task in itself.
Are Unit tests not needed anymore?
Acceptance tests are no replacement for Unit tests. Unit tests must still be written by the developers to make sure that the code they are writing is performing as it should. Unit tests are written to test individual units of source code, whereas Acceptance tests are written to test several units of source code connected by some type of workflow defined by the client.
ATDD and outsourcing
ATDD is very much suited for outsourcing because its previously listed advantages address many of the issues encountered in outsourced software development. But it is important to notice that it will only work if the project is using an agile development method and can’t be shoehorned into a waterfall based development scenario of design first, development second, testing third. ATDD needs commitment from both the project owner and also the agile supplier because the real emphasis is not necessarily on the testing itself but on the increased shared understanding that is a product of the increased contact between the two parties. Including ATDD will also most likely increase the cost of the project at the start as more time is needed by the developers to write the additional tests. The real gain comes over time as the functionality of the delivered software should with every iteration be closer to what the client envisaged in the first place and therefore should reduce the need to add or refine functionality. The Acceptance tests also act as a baseline that can be consulted if a dispute about the scope should arise.
“Should you outsource software development” is a very hotly contested topic on the Internet. And again you have to whole spectrum of opinions, from the “we are all going to loose our job” blog posts to the “we can develop everything at any time at the best rate for you” emails. Reality is that offshoring, near sourcing or using local or even in-house developers are all viable solution in some circumstances. Fact is that you will get a lot more man-hours if you choose to use a lower cost destination. Whether this is going to save you any money is an entirely different question.
The deciding factors for us to outsource the development were the following:
- Lack of local Ruby on Rails expertise
- Limited financial resources and the price was right
- We have developed software before
- We develop alongside them and can see their code
- Due diligence finding an offshore partner (see three part post about our search)
- Matching our developemnt process with theirs
- Solid list of requirements
We can gladly say that for us it worked very well, we got a lot more bang for the buck than if we would have developed here. Here are the key points that we think were crucial to our success:
- We felt comfortable with the company we chose from the start
- We felt especially comfortable with the team lead, a highly skilled software engineer with excellent communication skills
- Skype was available at any time for them and for us. Communication needs to stay open.
- Using IM rather then calls for technical discussion keeps a log that could be referred to later.
- Because we kept on talking with the developers daily, a personal connection formed and they got more involved in the project
- Their ideas were treated the same as ours and many of them have found their way into the endproduct. This motivated them even more
- Using an agile development process also meant that we were all working on the same iteration and as such felt like we were in the same boat
- We responded to their question in the same time frame we were expecting from them
- A shared understanding that we will come up with more requirements as time goes by and a willingness on their side to change requirements and sometimes adding a requirement without charging for it
- Our acceptance of real change requests without a lengthy price renegotiation. It started as a fixed price project and the price and time frame were adjusted according to the extensions
- From early on using the developed software to track the project
And so far we still have not met a single person from that company.
We are not advocating offshoring. What has worked here for us can also work locally but in our case we had no real choice because of lack of local expertise and limited resources. What has to be taken into account in our case is that we only partially offshored as we were part of the development team ourselves. We believe that in order to reap the real benefits of outsourced development, a company needs a mixed team, some developers are local and some offshore and that are able to interact as described above. In the future we will build a local team alongside the offshore team we used so far.
In this post we will talk briefly about how we gathered our requirements for the Outfarm Service. We are using an agile development approach and will be using userstories as the basis for our requirements. A userstory is a software requirement described in one or two sentences using everyday or business language of the user. This enables non-technical business people to formulate the requirements using their own language, because they will the use the new system afterwards. And in case you wondered, yes they have to come up with the requirements, not an analyst who is already removed from their day to day problems.
There are many ways of getting the user to come up with the requirements: interviews, questionnaires and workshops to name a few. Because we build a service platform that we would use ourselves, we chose to organize a workshop. Many of the steps we used are coming from Mike Cohn’s excellent book “User Stories applied for agile software development”.
The workshop has three distinct modules:
- Role-Modeling
- Userstory gathering
- Acceptance test gathering
Role-Modeling: Every application will be used in different ways by users in different roles. The idea of the workshop is to tease out those user roles that will then be used as the basis for the userstories. We started with a brainstorming session ending up with many overlaping user groups. Next step was to consolidate and condese the overlaping roles to a few roles that define these strongly overlapping user groups. And the last act was to think about attributes such as: how often are they using the software, level of domain expertise, general goal for using the software, proficiency with the software developed and general proficiency with computers and internet. We defined our own roles first and then went on to define other roles we thought would be using this software including the person who makes the buy decision for our service.
Userstory gathering: There are many ways to gather user stories: User interviews, Questionnaires, Observations, Story Writing Workshops and working with user proxies such as product manager, sales/markting or business analyst. We used a combination of user interviews and story writing workshop to come up with a simple conceptual prototype on paper and resulting userstories. Our software is a web application so we used high level pages (no page details included) as the basis for the prototype and started with an empty page. For each role we then defined what actions they could take from here and these actions would become a userstory and add pages to the prototype from which we would look for more possible actions and add more pages. While walking through the prototype one role at a time, we asked the following questions:
- What would this role most likely want to do next?
- What mistakes could he make from here?
- What could confuse the user at this point
- What additional information could the user need?
Throughout the workshop we kept in mind to keep the discussion at a high level and to come up with as many stories as possible. And throw away the simple prototype because all the information should now be captured in the usesrstories and the prototype will only create confusion when all of a sudden someone revisits it after a month.
Acceptance test gathering: The last post is already concerned with testing, therefore we only want to add a few more things:
- Write tests before development starts as they give the developer boundary conditions during developemnet and for effort estimation
- The end user/customer needs to specify the acceptance tests with the help of the developers
- Testing is part of the process and happens throughout development and not just at the end
- As long as tests add value, they should be added. Additional tests can and should be written throughout the lifetime of the project
- Automate acceptance testing using frameworks because manual testing is error prone and mind numbing to the testers
- Acceptance tests should cover different types of testing (Functional testing, User interface testing, Usability testing, Load testing)
All requirements that were created in the last step have one issue, they describe in (hopefully) simple words the business requirement that was created using the insight of end user and the developer. But this is a description of functionality as it should happen in the best case. So there will be a good bit of ambiguity in a requirement.
This is where acceptance tests are coming in. At the start they are simply notes about what should be tested for every requirement. Acceptance tests are later often used as the criteria to verify whether a requirement was fully implemented and this makes a lot of sense. Because the initial tests are described again in plain english and outside of a technical framework, the end users should define the bulk of them. Also new tests should be added over time to cover all eventualities.
Acceptance tests should be defined before a line of code is written as they provide a lot of information for the developer that could be interpreted differently due to requirement ambiguity. They also create boundaries for the requirement and limit the amount of work that is needed for a given requirement. Tests should be integrated into a test automation framework so they can be reused for every iteration and with that become automated regression tests that make sure any new iteration did not break what was developed in an earlier iteration. There are many frameworks out there like Fit, Fitnesse and Selenium that can help with this automation.
But besides Acceptance Tests, it is important to request that the developers also write lower level unit tests during development. When combining both lower level unit tests and higher level acceptance tests quite a good amount of test coverage is achievable.
Unfortunately testing comes with a steep price tag. Most outsource companies slapped on between 25-50% for extended repetitive manual testing (poor testers). It has to be said that requesting proper automated unit testing and acceptance testing is a development step in itself because tests have to be written in much he same way as the rest of the software. But as a project goes over several iterations, regression tests will help finding issues early rather then at the last test run. In our opinion paying for testing should mean that the tests are automated and can be reused in later iterations or even releases. Manual testing is error prone and one of the least motivating pieces of work and should be best avoided if possible.
How much time should be spent on a project before starting with development? It can be safely said that this is one of the most talked about topics when people in the software industry come together. There (still) are some who think that everything can be designed before a single line of code is written. But many industry veterans would now go into the other direction and advocate successful communication between developers and the end user throughout the duration of the project as the single most important factor for project success.
Project success can mean many things. For us it means that the developed software is useful to the end user. Useful means that the software actually improves the process the end user is using it for. This can often be quantified by measuring the time saved when using the software to the state of affairs before. We do live in a world of finite resources, therefore on time and cost delivery of a project obviously should not be underestimate. But keep in mind that on time and or cost delivery only does not mean it is actually useful to the end user.
Reality is that every so called development methodology (just a fancy word for method) is pushed by someone in the industry with vested interests. There are the multi-billion dollar companies that offer an extensive and expensive range of integrated software development tools that will tell you that efficient software design can only be done with their brand of tools and a formalised design language. And then there are all the methodology gurus and software consultancies that are charging their clients by the hour and the more upfront design they can sell their clients, the more money they will make. Reality is that many managers feel more comfortable with early upfront design because it creates huge amounts of documentation (nobody reads) and that creates the illusion of a more manageable project.
On the other hand there are the agile development methodologies that put communication between the developer and the end user above everything else (Agile Manifesto) . Communication starts with writing down requirements together in plain mother tongue. A good requirement is small, testable, valuable to the end user and should be independent from other requirements. It is followed by iterative development step that produces running software at the end of each iteration. That is when the developers and the end users come together again and go through the available software and find out what needs to be changed, extended or dropped. Over several iterations they, at least most of the time (or so it is said), end up with software that the end user is happy with. The proponents of agile methodologies claim that changes to the requirements throughout the lifetime of the project are an integral part of the method because nobody can foresee every single requirement (see post about requirements gathering). Some only become apparent during development.
So who pushes agile development? Agile development is pushed by small software houses because it creates a competitive advantage over large slow moving competitors with their armies of formally trained software developers. Small companies are so much more adaptable and are more willing to integrate change into their idea of project management and agile really highlights those strengths. Not to underestimate is the amount of money saved when not using the expensive design tools and training the developers to use them. Because agile is quite a loose approach, there are many different agile methods. Even though they fundamentally are the same, some still claim “moral” higher ground for their agile cult. Things never change.
As you can easily see from the above post, Outfarm is using an agile approach to software development even though we are working with an outsourced development provider. This goes against the “throw it over the wall and see what comes back” mentality of outsourcing which has very little chance of succeeding anyway. We are not advocating against using formalised approaches, but we choose an agile approach because:
- We worked with formalised approaches and didn’t like it
- We can’t afford spending 6 months in a design phase and prefer to talk to each other about requirements
- We are impatient and want to be able to see/touch/feel the software as it develops.
- We are the end users and know that we will introduce more requirements along the way.
- We see our developers abroad as peers, not as code monkeys
- Ongoing communication with the developers gives us greater control over the project because we get a better understanding whether they understand the requirements and because we love their crazy accents
- We have limited resources and believe that an agile approach will get us there cheaper and faster
Requirements gathering is one of the most important early steps in a software development project. Let’s assume company A has an outstanding software product idea and they hire company B to write that software. Requirements communicate what A wants B to create for them. It’s like an architect handing over the blueprint to the construction company and they build it, right?
Unfortunately some characteristics of software make the analogy of the requirements to the blueprint quite a stretch:
- Software is intangible which makes it inherently difficult not only to come up with all detailed requirements before development starts but also to communicate them to another person or group.
- Software represents dynamic workflows. The definition of a workflow consists of what one wants to happen, what one does not want to happen and how to deal with it in case it does happen. That is a lot to think about at once without a feedback mechanism.
Therefore change is an integral part of a software development process. Not to speak of the incredible boredom when spending hours, days and weeks meticulously defining requirements in every possible detail using well known formalized approaches.
We don’t want to get bored with what we are doing, so we decided to use an agile approach. Here requirements are used as a basis for discussion between A and B rather than exact instructions what to do. This approach also takes change into account and change requests can be integrated easily.
We expect the following from this approach:
- Shorter initial requirements gathering phase
- Increased communication with the developers should lead to less misunderstandings and also better control over the development process
- Increased ability to act on changing requirements
- Shorter time to market
Prototyping is a great way of getting a preview of the product or service you are developing. Then again, we are talking about an online service, something that has been developed so many times before. So do we really need a prototype? Prototypes have other disadvantages: they are generally built so the product team/customer can have a first look or it is used to present it to investors. This generally means the prototypes is all show and no substance and should be discarded afterwards; and that is the crux of a prototype: it NEVER really is discarded.A prototype sends the wrong signal: look the product is nearly finished; this increases expectations on development timelines and long and behold the prototype becomes the starting point for the alpha release.
Alpha releases on the other hand implement all necessary and high priority functionality and a simple user interface and, in theory at least, are built with further development in mind. They are used to get your customers to play with the high priority functionality and give you feedback to it early. The feedback then flows into the beta release.
In our case a prototype does not make sense. We already know what we want, we are our own customer and investment is not on the cards yet. Every startup graples with limited resources and the alpha release gets us faster to the customer and the all important feedback.
Still protoypes are important and are often used as a sales tool. Sales contracts were signed on the back of them and many myths are told about the use of prototypes in sales meetings. Just always keep in mind that a prototype is a sales brochure and its rightful resting place, like any other brochure, is in the (recycle) bin.
