Rails 3: Functional controller testing when using devise and declarative authorization gems 28 Jun 11

0 Comments »

One has to give it to the developers of devise and declarative authorization, they have really created extremely useful gems. As we are using these gems extensively in our own product and also for client products and projects, we also need to test them. Now both gems offer very helpful test helpers which need just a little bit of help to work together. The issue arises if a controller takes advantage of both gems where we found ourselves faced with disappearing session values after signing in with devise. But after digging a little bit in the declarative authorization source code, we found how to make the two gems work together for testing. The same post can also be found on Github

Below code example seems to work for Rails 3.0.6, devise 1.1.5, declarative authorization 0.5.3.

The sign_in test helper from devise adds the key/value pair “warden.user.user.key” => ["User",id] to the session and one has to explicitly add the session to the declarative_authorization get_with helper. The *_with methods call the request_with method which defaults the session argument to an empty hash unless one explicitly provides the session.

And without the above mentioned key/value pair in the session when calling get_with, one will not get past devise’s authenticate_user! before filter.

The following test in the users_controller_test.rb works when trying to access the users_controller show action which is included in the filter_resource_access and has an appropriate rule listed in the authorization rules:



User Acceptance Testing with Selenium IDE: How to handle Javascript Alerts, Ajax Popups and Drag and Drop 08 Oct 10

2 Comments »

We have been using Selenium successfully as an automated tool for Acceptance Testing all releases from our outsourcing provider. Even though we have written many of the testcases in ruby, using the IDE to create testcases in Selenese has been enough in many cases. Every release we receive is tested using the old test suites to make sure the new functionality does not break existing one, this step is also called regression testing. Selenium works great for applications with an already advanced user interface that will not change significantly over time any more. Otherwise a lot of time will be spent debugging the testcases themselves. One way around this is to use a fixed id for the elements used in testing, but that might not always be possible.

Tip:If you have access to the database, create a database dump after running a test suite. If you have to do manual tests in between test suites then create a db dump before AND after. This will make your life a lot easier if your tests are dependent on previous tests which they probably will be. Now if one of the tests in a test suite fails, you just reload the database and start the test suite again.

We used Selenium and found it to work very well with our AJAX enabled website, but it took some time to get the head around the topics described below. The following examples worked in Firefox 3.6.10 and Selenium IDE 1.0.7.

The documentation was pretty useful and the API covered most other topics not found in the documentation.
Selenium IDE documentation
Selenium Selenese Commands
Selenium 1.0 API

AJAX/Javascript in general

When you are using AJAX/Javascript, the web page will most likely not actually reload. Therefore Selenium will not know if and when an element in the page is available and the verify* or assert* commands will often fail. This is where the waitFor* commands come in. For all these commands Selenium will repeatedly check whether a certain event came to pass until a set time out is reached. By default the time out is set to 30 seconds but can be manually adjusted using (surprise surprise)

setTimout( timeout in milliseconds) –> setTimeout(5000) –> sets timeout to 5s
Once set, the timeout will stay at 5s.

AJAX or “in browser” popup

Many web applications now use what we call an “in browser” popup, this is NOT a smaller new browser window and NOT a Javascript alert. Most of the time it is a hidden DIV element at a relative or fixed position that is shown and hidden using javascript and might be populated with data from the server using AJAX. It is very often used for data entry and therefore will have to be tested with Selenium.

• Use waitForVisible if the element’s CSS attributes include “visibility:hidden” or “display:none”.
• Use waitforElementPresent if the element did not exist before. This will not work for the above situation because the element itself is present, even though it is not visible.

Tip:use waitFor commands in sequence if you are waiting for an element in a container. For example you have to test an AJAX action that populates a hidden DIV with a page from the server that contains input fields and then makes the DIV visible. As part of the test you most likely will have to populate these inputs fields. In that case use waitForVisible for the DIV container first and then use the waitForElementPresent for the first input field. Even though Selenium verified that the DIV is now visible, it still is not a guarantee that the browser already loaded the form with the inputs fields from the server. So the test will most likely behave slightly erratic, sometimes it passes and sometimes it will fail. Using a sequence of waitFor commands will make sure that the elements are available at the time of the test step.

Javascript alert boxes/confirmations

Javascript alerts are a bit of a problem for Selenium and there are even situations where the box has to be clicked manually. (From the documentaiton: Selenium does NOT support JavaScript confirmations that are generated in a page’s onload() event handler. In this case a visible dialog WILL be generated and Selenium will hang until you manually click OK.)

But most of the time it is possible to deal with this pesky little alert boxes. When recoding the test using Selenium IDE the alert boxes appear as normal and the IDE will happily record a “assertConfirmation” command, but when running the test, no alert box shows up and the test often fails. It is important to know that alert boxes are NOT shown when running the test as noted in the Selenium documentation. And replacing assertConfirmation with waitForConfirmation most of the time allows the test to pass.

All commands use a positive confirm, so if you want to test the negative confirmation use chooseCancelOnNextConfirmation BEFORE the action that results in the alert box.

Drag & Drop

Even complicated actions like drag and drop can be implemented using the Selenium IDE only. In our case we had the following setup to test.

• Two DIVs side by side with ids “account” and “team”, the “account” DIV containing a table of account users not in a project team and the “team” DIV containing the table of the project team users.
• The users were distinguished using their unique email address and the table position for a specific user is dynamic in both DIVs.
• Dragging a user from the “account” into the “team” DIV also opened an “in browser” popup to choose the project roles for that user.
• Once the user was moved over in the “team” DIV, we needed to check that the email address was in the table of the “team” DIV.

Tip:Don’t move the mouse when running a test that contains mouse movement commands, otherwise your mouse movements will interfere with Selenium and the test will very likely fail.

Tip:Use the echo command to check your variables in the log (works if set to info)

Acceptance Test Driven Development (ATDD) 20 Sep 10

1 Comment »

Acceptance test driven development is an agile software development method that uses the Acceptance tests to establish when a requirement meets the client’s expectation. It is an advanced practice of the Test Driven Development method.

Acceptance tests or Customer tests are written by the client together with the developers for every requirement that is part of the specification. Whereas a requirement describes specific behaviour the client wants in the finished software, the Acceptance tests make sure that the requirement was implemented in its completeness and according to the client’s input.
As an example we can use the tried and tested password requirement:

Requirement “When a user is creating an account, he/she is required to choose an outlandishly secure password”
Acceptance Test1 Test that the password contains at least 8 characters with at least one upper case letter, a number and a symbol”
Acceptance Test2 Test that the password does not contain a word from the english dictionary”
Acceptance Test3 Test that the user receives different error messages when the password is too short, contains no upper case letter, no number or no symbol. “
In this case the Acceptance tests define now what an “outlandishly secure password” is and increase the shared understanding for this requirement. And after the tenth iteration I can make sure that these Acceptance tests still pass so I have regression testing built right in.

The many benefits of ATDD

• Acceptance Tests increase the shared understanding of a requirement because they are a product of direct interaction between the client and the developers.
• Clarifies requirements and keeps the developers focused on what the client really wants.
• Drives out ambiguity if certain behaviour is ultimately a bug (behaviour violates either requirement or tests or both) or a change request (behaviour is acceptable within the scope of requirement and acceptance tests but ultimately undesirable).
• Software delivery is now dependent on all Acceptance tests passing and with that defines when a project is done.
• Percentage of passed acceptance tests/all acceptance tests acts as a project progress indicator.
• Free regression tests for the future iterations (especially if automated).

Some ATDD drawbacks

• Client interaction is needed which can prove difficult due to time constraints
• More work for the developer if tests are automated.
• Project progress might be slower due to additional effort.

Automating ATDD
There are many tools and frameworks for acceptance test automation, even some that can be used for User Interface testing. Many of them are free and open source, so the framework can be adapted to suit the project. Automating acceptance tests is highly desirable because regression testing is a hugely important step for iterative agile development methods. The immediate drawback is that this also means the developer has even more work to do because test automation is a development task in itself.

Are Unit tests not needed anymore?
Acceptance tests are no replacement for Unit tests. Unit tests must still be written by the developers to make sure that the code they are writing is performing as it should. Unit tests are written to test individual units of source code, whereas Acceptance tests are written to test several units of source code connected by some type of workflow defined by the client.

ATDD and outsourcing
ATDD is very much suited for outsourcing because its previously listed advantages address many of the issues encountered in outsourced software development. But it is important to notice that it will only work if the project is using an agile development method and can’t be shoehorned into a waterfall based development scenario of design first, development second, testing third. ATDD needs commitment from both the project owner and also the agile supplier because the real emphasis is not necessarily on the testing itself but on the increased shared understanding that is a product of the increased contact between the two parties. Including ATDD will also most likely increase the cost of the project at the start as more time is needed by the developers to write the additional tests. The real gain comes over time as the functionality of the delivered software should with every iteration be closer to what the client envisaged in the first place and therefore should reduce the need to add or refine functionality. The Acceptance tests also act as a baseline that can be consulted if a dispute about the scope should arise.

Release 2: Main development is over 19 Aug 10

0 Comments »

We are happy to announce that the main development for release 2 of the outfarm platform has finished now. This is a major milestone for Outfarm as this brings the date for the roll-out of many improvements to the platform closer. We have listened to our current clients and have added many notable improvements:

1. A brand spanking new “Idea to Paper” Wizard that includes drag and drop actor modelling, drag and drop specification management and the brand new “Non-Functional Requirements Questionnaire”.
2. Complete overhaul of RFP/Proposal wizards to make the whole process a lot easier for both sides. Includes the ability to break down items into tasks and to move attachments from a proposal into the main project when awarding the project
3. Integrated with http://www.gravatar.com to show your global avatar. Go and create one for yourself
4. Hugely improved dashboard that now includes time graphs and better access to past progress data
5. Ability to graphically monitor activity of individual project contributors
6. Improved drag and drop support for adding/removing items to milestones and releases.
7. Estimated time management and status change integration between tasks and items the task is connected to.
8. Changes to the User Interface include streamlining of processes and a much nicer look and feel.
9. And many other smaller changes that will make your life easier

If you want to be notified when release two is ready, then signup here.

Beta service release scheduled for end of March 01 Mar 10

0 Comments »

We have chosen a few companies to take part in the beta. But the beta will be open to everyone

Attending the Web Summit on the 4th of Feb in Dublin 27 Jan 10

1 Comment »

A representative of Outfarm is attending the Web Summit in Dublin which takes place in Trinity College. We are there to spread the word about Outfarm and talk to companies that are facing the problem our service is addressing.

We are very excited about the fact that our service is close to “official” Beta launch (expected in February) and we are welcoming Trial customers to use the service free of charge for a limited period.

Looking forward to meeting you there.

Outsourcing, copyright and intellectual property protection 07 Jan 10

0 Comments »

Whenever there is a discussion about outsourcing any type of software development work, someone will always play the intellectual property card. How can you protect it, is always the question. Software is intangible and the IP is really in how a business process is implemented within the software. Business processes are very hard to patent or to protect otherwise legally. Also the majority of software implements operational processes that are not that different between companies. This post obviously does not cover speciality software that implements other types of processes that might very well be patentable.

Many companies are afraid that if they use a software development service provider, those guys will then either use the software they developed and claim it is their own or use the know-how they acquired when developing software for the next customer. The first is quite unlikely, because most of the service providers are quite happily just that and have neither the knowledge, motivation nor the resources to take the software and sell it as their own or use it to build the next big Internet service. The second is certainly true, the question is to what extent. It is highly unlikely that the next customer will ask for exactly the same piece of software. Sure some parts will be reused, but because developing software is a rather complicated process, most of the time it will be completely rewritten to fit the new application. Reality is that IP theft for software is really hard to prove (not copyright infringement which is easier to prove) and would need a lot of money to actually prove in a court. We used a simple Non-Disclosure Agreement to protect ourselves against the most obvious forms of IP theft and that was it. The biggest threat to a company’s IP is its employees because the can walk away with the software and the market knowledge that lacks most software development service providers.

When it comes to software, then copyright infringement is probably the bigger issue as most large software companies will let you know. There is also an issue with who gets the copyright in different countries. Some countries award the copyright to the contractor and not the company that hired the contractor. So it makes sense to look up the copyright law and make sure the contractor signs a legally binding document stating that he/she does not claim the copyright now and in the future for the software developed.

There is also a very interesting article in The Economist covering some of this topic: http://www.economist.com/sciencetechnology/displayStory.cfm?story_id=15479680

Ruby on Rails: a design decision 30 Nov 09

0 Comments »

We had heard about Ruby on Rails and how efficiently one can create web applications with a handful of good developers and we were intrigued to find out more. Due to budgetary constraints we would not use a platform that we would have to pay for to run our service and we all had now used open source components for years. After playing around with Ruby on Rails for a while we were convinced that Ruby on Rails really is a great language/framework combination. We also felt very quickly at home with the Ruby language itself and found a huge amount of plugins/gems and a vibrant community. Also there are many great online services out there already running on Ruby on Rails on Linux for a while, so the majority of issues running an online service were already ironed out. But what really convinced us was that it implements the tried and tested model view controller pattern in an easy and straightforward way. We were sold.

Once again we are not advocating that Ruby on Rails is the only right choice but at the time we felt that it was the optimal choice given our background. Ruby on Rails in the end is just a tool set and there are many great tool sets out there that have their place in the right context. At the time of the post we are still very enthusiastic about our choice.

Offshoring, near sourcing or local sourcing and how to fit agile 07 Sep 09

1 Comment »

“Should you outsource software development” is a very hotly contested topic on the Internet. And again you have to whole spectrum of opinions, from the “we are all going to loose our job” blog posts to the “we can develop everything at any time at the best rate for you” emails. Reality is that offshoring, near sourcing or using local or even in-house developers are all viable solution in some circumstances. Fact is that you will get a lot more man-hours if you choose to use a lower cost destination. Whether this is going to save you any money is an entirely different question.

The deciding factors for us to outsource the development were the following:

• Lack of local Ruby on Rails expertise
• Limited financial resources and the price was right
• We have developed software before
• We develop alongside them and can see their code
• Due diligence finding an offshore partner (see three part post about our search)
• Matching our developemnt process with theirs
• Solid list of requirements

We can gladly say that for us it worked very well, we got a lot more bang for the buck than if we would have developed here. Here are the key points that we think were crucial to our success:

• We felt comfortable with the company we chose from the start
• We felt especially comfortable with the team lead, a highly skilled software engineer with excellent communication skills
• Skype was available at any time for them and for us. Communication needs to stay open.
• Using IM rather then calls for technical discussion keeps a log that could be referred to later.
• Because we kept on talking with the developers daily, a personal connection formed and they got more involved in the project
• Their ideas were treated the same as ours and many of them have found their way into the endproduct. This motivated them even more
• Using an agile development process also meant that we were all working on the same iteration and as such felt like we were in the same boat
• We responded to their question in the same time frame we were expecting from them
• A shared understanding that we will come up with more requirements as time goes by and a willingness on their side to change requirements and sometimes adding a requirement without charging for it
• Our acceptance of real change requests without a lengthy price renegotiation. It started as a fixed price project and the price and time frame were adjusted according to the extensions
• From early on using the developed software to track the project

And so far we still have not met a single person from that company.




We are not advocating offshoring. What has worked here for us can also work locally but in our case we had no real choice because of lack of local expertise and limited resources. What has to be taken into account in our case is that we only partially offshored as we were part of the development team ourselves. We believe that in order to reap the real benefits of outsourced development, a company needs a mixed team, some developers are local and some offshore and that are able to interact as described above. In the future we will build a local team alongside the offshore team we used so far.

Gathering agile requirements for our service 15 Aug 09

0 Comments »

In this post we will talk briefly about how we gathered our requirements for the Outfarm Service. We are using an agile development approach and will be using userstories as the basis for our requirements. A userstory is a software requirement described in one or two sentences using everyday or business language of the user. This enables non-technical business people to formulate the requirements using their own language, because they will the use the new system afterwards. And in case you wondered, yes they have to come up with the requirements, not an analyst who is already removed from their day to day problems.

There are many ways of getting the user to come up with the requirements: interviews, questionnaires and workshops to name a few. Because we build a service platform that we would use ourselves, we chose to organize a workshop. Many of the steps we used are coming from Mike Cohn’s excellent book “User Stories applied for agile software development”.

The workshop has three distinct modules:

• Role-Modeling
• Userstory gathering
• Acceptance test gathering

Role-Modeling: Every application will be used in different ways by users in different roles. The idea of the workshop is to tease out those user roles that will then be used as the basis for the userstories. We started with a brainstorming session ending up with many overlaping user groups. Next step was to consolidate and condese the overlaping roles to a few roles that define these strongly overlapping user groups. And the last act was to think about attributes such as: how often are they using the software, level of domain expertise, general goal for using the software, proficiency with the software developed and general proficiency with computers and internet. We defined our own roles first and then went on to define other roles we thought would be using this software including the person who makes the buy decision for our service.

Userstory gathering: There are many ways to gather user stories: User interviews, Questionnaires, Observations, Story Writing Workshops and working with user proxies such as product manager, sales/markting or business analyst. We used a combination of user interviews and story writing workshop to come up with a simple conceptual prototype on paper and resulting userstories. Our software is a web application so we used high level pages (no page details included) as the basis for the prototype and started with an empty page. For each role we then defined what actions they could take from here and these actions would become a userstory and add pages to the prototype from which we would look for more possible actions and add more pages. While walking through the prototype one role at a time, we asked the following questions:

• What would this role most likely want to do next?
• What mistakes could he make from here?
• What could confuse the user at this point
• What additional information could the user need?

Throughout the workshop we kept in mind to keep the discussion at a high level and to come up with as many stories as possible. And throw away the simple prototype because all the information should now be captured in the usesrstories and the prototype will only create confusion when all of a sudden someone revisits it after a month.

Acceptance test gathering: The last post is already concerned with testing, therefore we only want to add a few more things:

• Write tests before development starts as they give the developer boundary conditions during developemnet and for effort estimation
• The end user/customer needs to specify the acceptance tests with the help of the developers
• Testing is part of the process and happens throughout development and not just at the end
• As long as tests add value, they should be added. Additional tests can and should be written throughout the lifetime of the project
• Automate acceptance testing using frameworks because manual testing is error prone and mind numbing to the testers
• Acceptance tests should cover different types of testing (Functional testing, User interface testing, Usability testing, Load testing)

Acceptance Test Driven Development and the cost of (not) testing 20 Jul 09

0 Comments »

All requirements that were created in the last step have one issue, they describe in (hopefully) simple words the business requirement that was created using the insight of end user and the developer. But this is a description of functionality as it should happen in the best case. So there will be a good bit of ambiguity in a requirement.

This is where acceptance tests are coming in. At the start they are simply notes about what should be tested for every requirement. Acceptance tests are later often used as the criteria to verify whether a requirement was fully implemented and this makes a lot of sense. Because the initial tests are described again in plain english and outside of a technical framework, the end users should define the bulk of them. Also new tests should be added over time to cover all eventualities.

Acceptance tests should be defined before a line of code is written as they provide a lot of information for the developer that could be interpreted differently due to requirement ambiguity. They also create boundaries for the requirement and limit the amount of work that is needed for a given requirement. Tests should be integrated into a test automation framework so they can be reused for every iteration and with that become automated regression tests that make sure any new iteration did not break what was developed in an earlier iteration. There are many frameworks out there like Fit, Fitnesse and Selenium that can help with this automation.

But besides Acceptance Tests, it is important to request that the developers also write lower level unit tests during development. When combining both lower level unit tests and higher level acceptance tests quite a good amount of test coverage is achievable.

Unfortunately testing comes with a steep price tag. Most outsource companies slapped on between 25-50% for extended repetitive manual testing (poor testers). It has to be said that requesting proper automated unit testing and acceptance testing is a development step in itself because tests have to be written in much he same way as the rest of the software. But as a project goes over several iterations, regression tests will help finding issues early rather then at the last test run. In our opinion paying for testing should mean that the tests are automated and can be reused in later iterations or even releases. Manual testing is error prone and one of the least motivating pieces of work and should be best avoided if possible.

Why we are choosing agile development over a more formalized approach 30 Jun 09

0 Comments »

How much time should be spent on a project before starting with development? It can be safely said that this is one of the most talked about topics when people in the software industry come together. There (still) are some who think that everything can be designed before a single line of code is written. But many industry veterans would now go into the other direction and advocate successful communication between developers and the end user throughout the duration of the project as the single most important factor for project success.

Project success can mean many things. For us it means that the developed software is useful to the end user. Useful means that the software actually improves the process the end user is using it for. This can often be quantified by measuring the time saved when using the software to the state of affairs before. We do live in a world of finite resources, therefore on time and cost delivery of a project obviously should not be underestimate. But keep in mind that on time and or cost delivery only does not mean it is actually useful to the end user.

Reality is that every so called development methodology (just a fancy word for method) is pushed by someone in the industry with vested interests. There are the multi-billion dollar companies that offer an extensive and expensive range of integrated software development tools that will tell you that efficient software design can only be done with their brand of tools and a formalised design language. And then there are all the methodology gurus and software consultancies that are charging their clients by the hour and the more upfront design they can sell their clients, the more money they will make. Reality is that many managers feel more comfortable with early upfront design because it creates huge amounts of documentation (nobody reads) and that creates the illusion of a more manageable project.

On the other hand there are the agile development methodologies that put communication between the developer and the end user above everything else (Agile Manifesto) . Communication starts with writing down requirements together in plain mother tongue. A good requirement is small, testable, valuable to the end user and should be independent from other requirements. It is followed by iterative development step that produces running software at the end of each iteration. That is when the developers and the end users come together again and go through the available software and find out what needs to be changed, extended or dropped. Over several iterations they, at least most of the time (or so it is said), end up with software that the end user is happy with. The proponents of agile methodologies claim that changes to the requirements throughout the lifetime of the project are an integral part of the method because nobody can foresee every single requirement (see post about requirements gathering). Some only become apparent during development.

So who pushes agile development? Agile development is pushed by small software houses because it creates a competitive advantage over large slow moving competitors with their armies of formally trained software developers. Small companies are so much more adaptable and are more willing to integrate change into their idea of project management and agile really highlights those strengths. Not to underestimate is the amount of money saved when not using the expensive design tools and training the developers to use them. Because agile is quite a loose approach, there are many different agile methods. Even though they fundamentally are the same, some still claim “moral” higher ground for their agile cult. Things never change.

As you can easily see from the above post, Outfarm is using an agile approach to software development even though we are working with an outsourced development provider. This goes against the “throw it over the wall and see what comes back” mentality of outsourcing which has very little chance of succeeding anyway. We are not advocating against using formalised approaches, but we choose an agile approach because:

• We worked with formalised approaches and didn’t like it
• We can’t afford spending 6 months in a design phase and prefer to talk to each other about requirements
• We are impatient and want to be able to see/touch/feel the software as it develops.
• We are the end users and know that we will introduce more requirements along the way.
• We see our developers abroad as peers, not as code monkeys
• Ongoing communication with the developers gives us greater control over the project because we get a better understanding whether they understand the requirements and because we love their crazy accents
• We have limited resources and believe that an agile approach will get us there cheaper and faster

Choosing a supplier: Other considerations (Part III) 10 Jun 09

0 Comments »

Does your project management process match?
The best way to find out how a supplier actually manages a software project is with the STAR method. Simply asking do you use an “Agile approach” will in most cases be answered with a yes and is then often followed by a frantic search for documentation if in fact it wasn’t used before. Some processes are simply not compatible and trying to combine them will end in frustration for both sides. Try to combine an agile process with the CMM Level 5 certification. The other extreme is the supplier companies that rely on the process of the client; just think about how long it took the get the process to stick in your own company (if you are actually using a process). In the absence of a shared process or a process in general, many suppliers simply revert to the waterfall model which should never be used! In our case we looked for a company that followed the agile development approach close enough to match our own process.

Consider a small project first
If you can afford it, go with a small project first. Smaller projects, even if they are only one to two weeks in duration, will give you a very good understandig whether you and the supplier can work together. Define milestones, deliverables and acceptance criteria close enough to how you would do it in the larger project.

Don’t be dazzled by software development language
If you have ever dipped your toe into software development, you were, most likely, initially thrown back by the sheer amount of abbreviations and hype words that are used in this industry. The basic rule here is to play dumb and ask everyone to explain it in clear English to you (or your mother tongue if at all possible). The truth is that many people working in that area love to hide behind those big words. The rule here is clear, if someone can’t explain a concept to you in words you understand, then that person probably doesn’t understand it himself and has no business behaving as if. As matter of fact the STAR method is great at exposing someone just talking the talk.

Choosing a supplier: Talk to them (Part II) 29 May 09

0 Comments »

Talking to the supplier and their references is crucial in understanding whether a supplier is suited for your project. Talking to the supplier’s sales contact is a start, but you also need to talk to the people who will work on your project.

Use behavioural interviewing (High) The basic idea of behavioural interviewing is this: future performance of a person is best predicted by understanding past performance in a similar situation. Focus is on experiences, behaviours, knowledge, skills and abilities that are job related. Ask the interviewee to use the STAR method to structure their answer. Using a combination of behavioural questions and the STAR method forces the interviewee to go beyond simple yes/no answers.

• S – Situation; background; set the scene
• T – Task or Target; specifics of what’s required, when, where, who
• A – Action; what was done, skills used, behaviours, characteristics
• R – Result; what was the outcome, what happened

More information here:
http://web.mit.edu/career/www/guide/star.html
http://www.career.caltech.edu/resources/handouts/STAR_Interviews.pdf

Talk to the whole supplier team (High) You want to talk to the people who you will be dealing with on a daily basis first; project manager, team leader or lead developer. But also talk to the rest of the team, if they are already known. Interview topics are: Past projects, technical understanding, project management process used and allow for a bit of friendly chit/chat. By using the STAR method, you should be able to get a pretty clear picture of how the team has approached similar projects in the past.

Do you get along? (High) This can’t be underestimated. If you don’t get along with the project manager or the team lead, request someone else to take the place. Sometimes it only needs is someone else who interfaces with you directly. During our search we encountered two project managers we were not sure about and raised our issues when talking to the references. In both cases our suspicion was confirmed and we went on to use another company.

Talk to references (High) Ask for several references, if possible in your country or area. Out of obvious reasons the supplier wants to connect you with a satisfied client, so asking for several specific references increases the chance of getting a more balanced review. Asking for a reference in our area helped us tremendously. We got honest feedback about suppliers and interestingly enough they got a mixed review. Follow up on all references; chances are that the work done in one of them resembles your own project. Make sure you get references for the type of application that you are planning and, if you already know it, for the technology you are using. Many suppliers are specializing in one technology but happily take other projects also. And again, ask the questions with the STAR method and you should get useful reviews.

Choosing a supplier: looking at the available information (Part I) 15 May 09

0 Comments »

After discarding the bids that are not realistic, the next step is to compare the bids that are still on the table. Here are the criteria we used and the weight we gave them in brackets:

Reviews posted on the freelance market places (Low) should minimize the risk of engaging with a fraudulent company. But these reviews have several issues:

• Reviews work best when the expectations of both parties are clearly defined: for example on Ebay, buyer expects delivery of the right product in-time and in working order, seller expects prompt payment. Deviations are penalized with a bad review.
• Reviews on a freelance marketplace reflect more on the relationship between buyer and supplier. The ability to post a bad review declines the better the parties get along. But this is not equal to a successful project.
• Writing a bad review is equal to admitting that you chose the wrong supplier in the first place.
• Most companies don’t bother to write a review and the standard review questions are too general.
• Suppliers can remove some of the negative reviews.

Profiles on freelance market places (Low) are sales brochures and have to be read as such.

Past work (Low – Medium) Most clients don’t give permission to use their applications as examples of work. Therefore advanced projects are hardly ever shown. Projects that have been developed a few months ago are often much more advanced than at time of delivery due to ingoing developement by the client. Sometimes really bad work examples are shown and that should acts as a deterrent.

Supplier Website (Low – Medium) If you are looking for graphical and user interface work to be done then the website can say everything. Just make sure they didn’t outsource it themselves! The website of a professional supplier company should contain information about who they are, where they are located and the services they offer. A quick call to the phone number in the contact us section can also be revealing. If they also offer completely unrelated products/services on their website, software development projects might not be their main focus.

Certifications (Medium) can give an indication of the processes used to deliver a project and certify the skill level in specific technologies. Certifications are only valuable if they can be verified and are recent enough. Skill certifications are generally per developer but are often used by companies. Certifications are indicators if you are looking for specific skills or process adherence like CMMI or similar.

The project bids are in: A word about low-balling 12 Apr 09

0 Comments »

We had tremendous interest for the project and the 10 bids originated from India, Russia, US, Romania and Australia. They ranged from detailed project delivery documents including time estimates per user story to simple one-liners like “We can do this for $X in the specified timeframe”. Needless to say, we did not follow up on one-liners.

In our case the lowest bid, besides the one-liner offers, was actually well put together. But the document fell apart on closer inspection and even though it seemed that the company had understood the scope of the project, their time estimates were off. During a phone call we were assured that the project duration was correct and that they were confident about their ability to deliver. Their estimated cost was only a quarter of the bid from a company located in the same area of the same country. This was a classic case of “low-balling”.

In this context “low-balling” means that a company deliberately understates the cost of the project with the goal to beat the competition and get the project in the door. Once the project is in the door, every small deviation from the project requirements will be treated as scope extension and the additional effort will be charged accordingly.

The easiest way to spot a “low-ball” is by dividing the cost of the suspected “low-ball” bid with the time estimate of one or more reasonable bids from the same area or country. If the result is an unsustainable hourly rate, in our case it was $4/hour, then you can be pretty sure they either made a mistake or they are using a “low-ball”. The company we are referring to actually had decent enough feedback, no glowing reviews though, so they obviously had delivered in the past. But we simply didn’t get a fuzzy feeling when we talked to them and with that the lowest bid was off the table.

In our opinion, low-balling or low-bidding companies have significantly added to the negative perception of outsourcing. They make it harder for companies like us to find a suited supplier because we don’t want to be constantly hassled regarding rising cost and scope extension. They also make it hard for competing suppliers to justify their higher costs for a well planned project. Unfortunately reviews of past projects have only limited value to spot them. But putting the blame solely on the suppliers is not justified either, some companies see outsourcing as a source of cheap software engineering labour and with that actively encourage low-bidding.

If you don’t want to end up paying a lot more for your project than anticipated, due diligence throughout the selection process is really important. Cost should never be the only criteria.

In the next post we’ll talk about our selection process and list the criteria we used.

Posting the project 30 Mar 09

0 Comments »

We posted our project on the two main freelance websites. As we would like this to be handeled as a project rather than hiring a bunch of developers around the world and do it ourselves, we decided against using a providers that specializes in individual contributors.

After creating an account with both of them, we posted our project. The project description was only 2 lines and included an Excel spreadsheet containing User Stories and some non-functional requirements. This was all the infromation on top of which we expected to receive bids. The only information regarding the User Interface was that we expect it to be Web 2.0 interactive and contemporary looking.

Now this all might sound quite foolish, but we are keeping the information deliberately broad. We are planning on using an agile development approach and as such we don’t want to define the project too narrow from the beginning. In order to find a company that is able to handle an agile development project, we can’t be too prescriptive but have to allow the developers to be invloved in the development process. In another post we will talk about why we think agile development is the right approach in this situation.

All bids have to come back within 2 weeks. More then..

Requirements gathering: Formal or Agile 02 Mar 09

0 Comments »

Requirements gathering is one of the most important early steps in a software development project. Let’s assume company A has an outstanding software product idea and they hire company B to write that software. Requirements communicate what A wants B to create for them. It’s like an architect handing over the blueprint to the construction company and they build it, right?

Unfortunately some characteristics of software make the analogy of the requirements to the blueprint quite a stretch:

• Software is intangible which makes it inherently difficult not only to come up with all detailed requirements before development starts but also to communicate them to another person or group.
• Software represents dynamic workflows. The definition of a workflow consists of what one wants to happen, what one does not want to happen and how to deal with it in case it does happen. That is a lot to think about at once without a feedback mechanism.

Therefore change is an integral part of a software development process. Not to speak of the incredible boredom when spending hours, days and weeks meticulously defining requirements in every possible detail using well known formalized approaches.




We don’t want to get bored with what we are doing, so we decided to use an agile approach. Here requirements are used as a basis for discussion between A and B rather than exact instructions what to do. This approach also takes change into account and change requests can be integrated easily.




We expect the following from this approach:

• Shorter initial requirements gathering phase
• Increased communication with the developers should lead to less misunderstandings and also better control over the development process
• Increased ability to act on changing requirements
• Shorter time to market

Prototype or alpha release? 09 Feb 09

0 Comments »

Prototyping is a great way of getting a preview of the product or service you are developing. Then again, we are talking about an online service, something that has been developed so many times before. So do we really need a prototype? Prototypes have other disadvantages: they are generally built so the product team/customer can have a first look or it is used to present it to investors. This generally means the prototypes is all show and no substance and should be discarded afterwards; and that is the crux of a prototype: it NEVER really is discarded.A prototype sends the wrong signal: look the product is nearly finished; this increases expectations on development timelines and long and behold the prototype becomes the starting point for the alpha release.

Alpha releases on the other hand implement all necessary and high priority functionality and a simple user interface and, in theory at least, are built with further development in mind. They are used to get your customers to play with the high priority functionality and give you feedback to it early. The feedback then flows into the beta release.

In our case a prototype does not make sense. We already know what we want, we are our own customer and investment is not on the cards yet. Every startup graples with limited resources and the alpha release gets us faster to the customer and the all important feedback.

Still protoypes are important and are often used as a sales tool. Sales contracts were signed on the back of them and many myths are told about the use of prototypes in sales meetings. Just always keep in mind that a prototype is a sales brochure and its rightful resting place, like any other brochure, is in the (recycle) bin.

The adventure begins 10 Jan 09

0 Comments »

Hi and welcome to the first entry of this blog. The company Outfarm plans to offer a service that helps companies and their software development suppliers to successfully collaborate on software projects. This blog will follow the development of this online service from the first alpha release up until service launch and should keep you updated about the status and the future development of the service.

Outfarm is a bunch of software product enthusiasts with years of experience developing software and products. We all have worked on successful projects, outright failures, cancelled projects, you name it. Most successful projects we have worked on had one thing in common: Excellent communication between the product and development team. Our goal is to create an online platform that fosters the right communication.

Let the adventure begin