Does your project management process match?
The best way to find out how a supplier actually manages a software project is with the STAR method. Simply asking do you use an “Agile approach” will in most cases be answered with a yes and is then often followed by a frantic search for documentation if in fact it wasn’t used before. Some processes are simply not compatible and trying to combine them will end in frustration for both sides. Try to combine an agile process with the CMM Level 5 certification. The other extreme is the supplier companies that rely on the process of the client; just think about how long it took the get the process to stick in your own company (if you are actually using a process). In the absence of a shared process or a process in general, many suppliers simply revert to the waterfall model which should never be used! In our case we looked for a company that followed the agile development approach close enough to match our own process.
Consider a small project first
If you can afford it, go with a small project first. Smaller projects, even if they are only one to two weeks in duration, will give you a very good understandig whether you and the supplier can work together. Define milestones, deliverables and acceptance criteria close enough to how you would do it in the larger project.
Don’t be dazzled by software development language
If you have ever dipped your toe into software development, you were, most likely, initially thrown back by the sheer amount of abbreviations and hype words that are used in this industry. The basic rule here is to play dumb and ask everyone to explain it in clear English to you (or your mother tongue if at all possible). The truth is that many people working in that area love to hide behind those big words. The rule here is clear, if someone can’t explain a concept to you in words you understand, then that person probably doesn’t understand it himself and has no business behaving as if. As matter of fact the STAR method is great at exposing someone just talking the talk.
Talking to the supplier and their references is crucial in understanding whether a supplier is suited for your project. Talking to the supplier’s sales contact is a start, but you also need to talk to the people who will work on your project.
Use behavioural interviewing (High) The basic idea of behavioural interviewing is this: future performance of a person is best predicted by understanding past performance in a similar situation. Focus is on experiences, behaviours, knowledge, skills and abilities that are job related. Ask the interviewee to use the STAR method to structure their answer. Using a combination of behavioural questions and the STAR method forces the interviewee to go beyond simple yes/no answers.
- S – Situation; background; set the scene
- T – Task or Target; specifics of what’s required, when, where, who
- A – Action; what was done, skills used, behaviours, characteristics
- R – Result; what was the outcome, what happened
More information here:
http://web.mit.edu/career/www/guide/star.html
http://www.career.caltech.edu/resources/handouts/STAR_Interviews.pdf
Talk to the whole supplier team (High) You want to talk to the people who you will be dealing with on a daily basis first; project manager, team leader or lead developer. But also talk to the rest of the team, if they are already known. Interview topics are: Past projects, technical understanding, project management process used and allow for a bit of friendly chit/chat. By using the STAR method, you should be able to get a pretty clear picture of how the team has approached similar projects in the past.
Do you get along? (High) This can’t be underestimated. If you don’t get along with the project manager or the team lead, request someone else to take the place. Sometimes it only needs is someone else who interfaces with you directly. During our search we encountered two project managers we were not sure about and raised our issues when talking to the references. In both cases our suspicion was confirmed and we went on to use another company.
Talk to references (High) Ask for several references, if possible in your country or area. Out of obvious reasons the supplier wants to connect you with a satisfied client, so asking for several specific references increases the chance of getting a more balanced review. Asking for a reference in our area helped us tremendously. We got honest feedback about suppliers and interestingly enough they got a mixed review. Follow up on all references; chances are that the work done in one of them resembles your own project. Make sure you get references for the type of application that you are planning and, if you already know it, for the technology you are using. Many suppliers are specializing in one technology but happily take other projects also. And again, ask the questions with the STAR method and you should get useful reviews.
After discarding the bids that are not realistic, the next step is to compare the bids that are still on the table. Here are the criteria we used and the weight we gave them in brackets:
Reviews posted on the freelance market places (Low) should minimize the risk of engaging with a fraudulent company. But these reviews have several issues:
- Reviews work best when the expectations of both parties are clearly defined: for example on Ebay, buyer expects delivery of the right product in-time and in working order, seller expects prompt payment. Deviations are penalized with a bad review.
- Reviews on a freelance marketplace reflect more on the relationship between buyer and supplier. The ability to post a bad review declines the better the parties get along. But this is not equal to a successful project.
- Writing a bad review is equal to admitting that you chose the wrong supplier in the first place.
- Most companies don’t bother to write a review and the standard review questions are too general.
- Suppliers can remove some of the negative reviews.
Profiles on freelance market places (Low) are sales brochures and have to be read as such.
Past work (Low – Medium) Most clients don’t give permission to use their applications as examples of work. Therefore advanced projects are hardly ever shown. Projects that have been developed a few months ago are often much more advanced than at time of delivery due to ingoing developement by the client. Sometimes really bad work examples are shown and that should acts as a deterrent.
Supplier Website (Low – Medium) If you are looking for graphical and user interface work to be done then the website can say everything. Just make sure they didn’t outsource it themselves! The website of a professional supplier company should contain information about who they are, where they are located and the services they offer. A quick call to the phone number in the contact us section can also be revealing. If they also offer completely unrelated products/services on their website, software development projects might not be their main focus.
Certifications (Medium) can give an indication of the processes used to deliver a project and certify the skill level in specific technologies. Certifications are only valuable if they can be verified and are recent enough. Skill certifications are generally per developer but are often used by companies. Certifications are indicators if you are looking for specific skills or process adherence like CMMI or similar.
We had tremendous interest for the project and the 10 bids originated from India, Russia, US, Romania and Australia. They ranged from detailed project delivery documents including time estimates per user story to simple one-liners like “We can do this for $X in the specified timeframe”. Needless to say, we did not follow up on one-liners.
In our case the lowest bid, besides the one-liner offers, was actually well put together. But the document fell apart on closer inspection and even though it seemed that the company had understood the scope of the project, their time estimates were off. During a phone call we were assured that the project duration was correct and that they were confident about their ability to deliver. Their estimated cost was only a quarter of the bid from a company located in the same area of the same country. This was a classic case of “low-balling”.
In this context “low-balling” means that a company deliberately understates the cost of the project with the goal to beat the competition and get the project in the door. Once the project is in the door, every small deviation from the project requirements will be treated as scope extension and the additional effort will be charged accordingly.
The easiest way to spot a “low-ball” is by dividing the cost of the suspected “low-ball” bid with the time estimate of one or more reasonable bids from the same area or country. If the result is an unsustainable hourly rate, in our case it was $4/hour, then you can be pretty sure they either made a mistake or they are using a “low-ball”. The company we are referring to actually had decent enough feedback, no glowing reviews though, so they obviously had delivered in the past. But we simply didn’t get a fuzzy feeling when we talked to them and with that the lowest bid was off the table.
In our opinion, low-balling or low-bidding companies have significantly added to the negative perception of outsourcing. They make it harder for companies like us to find a suited supplier because we don’t want to be constantly hassled regarding rising cost and scope extension. They also make it hard for competing suppliers to justify their higher costs for a well planned project. Unfortunately reviews of past projects have only limited value to spot them. But putting the blame solely on the suppliers is not justified either, some companies see outsourcing as a source of cheap software engineering labour and with that actively encourage low-bidding.
If you don’t want to end up paying a lot more for your project than anticipated, due diligence throughout the selection process is really important. Cost should never be the only criteria.
In the next post we’ll talk about our selection process and list the criteria we used.
We posted our project on the two main freelance websites. As we would like this to be handeled as a project rather than hiring a bunch of developers around the world and do it ourselves, we decided against using a providers that specializes in individual contributors.
After creating an account with both of them, we posted our project. The project description was only 2 lines and included an Excel spreadsheet containing User Stories and some non-functional requirements. This was all the infromation on top of which we expected to receive bids. The only information regarding the User Interface was that we expect it to be Web 2.0 interactive and contemporary looking.
Now this all might sound quite foolish, but we are keeping the information deliberately broad. We are planning on using an agile development approach and as such we don’t want to define the project too narrow from the beginning. In order to find a company that is able to handle an agile development project, we can’t be too prescriptive but have to allow the developers to be invloved in the development process. In another post we will talk about why we think agile development is the right approach in this situation.
All bids have to come back within 2 weeks. More then..
