SCRUM and Offshore (India)

Agile is the buzzword now days. For some reasons people do not want to wait for a big release to get delivered everthing they want. They want it in increments. Agile methodlogies seem to have an answer for this. Several Agile techniques viz. Xp, Lean, SCRUM, DSDM are availabile and people practice them in projects. But it is getting momentum in offshore (India) now days.

In the last 20 years, outsourcing of IT work to India and other countries has become very common. Many organizations in the developed countries include outsourcing as a major item on their IT strategy. This migration to outsourcing originally started for cost effectiveness, but now is aimed at getting better products, services and expediting the work. This new wave of outsourcing is about becoming thought partners and moving up the value chain. The current offshore model is a result of an evolution. Initially it started with people directly working at onsite, then at offshore under client management and today we are talking about 100% offshore with the help of latest infrastructure. There was (and is) always a demand for the best of everything – the best quality at low cost and within an ideal schedule. With the advent of the latest communication technologies, clients demand and expect transparency. We can meet all these goals by having a disciplined process and managing with the right set of matrices. This presentation will detail how to manage these factors with Agile (SCRUM) working, its challenges in onsite-offshore model and probable answers to them.

SCRUM is designed to deliver higher quality products in quicker and shorter cycles. It talks about collective responsibility including the manager (called as SCRUM master), development team and “the customer” or product owner. SCRUM and traditional project management are two distant practices with SCRUM assuming direct interaction between development team and “end user” or “customer” and SCRUM master is just facilitating instead of taking the “command and control” of project.
Project management with SCRUM is different from traditional project management. SCRUM methodology is adopted to deliver business benefits in better and faster manner instead of prolonged development cycles which produce a product which still needs some changes before it is ready for the customer. SCRUM methodology is about delivering early business benefits
detecting, and fixing defects at early stages close interaction between developers and end users and Many good things for project and not for project managers.
SCRUM assumes co-location. SCRUM welcomes change, it’s rather adaptive than reactive.
In a typical onsite-offshore model, we have an onsite coordinator who is responsible for requirements gathering and acting as a client liaison. Sometimes, we have geographically scattered teams. Setting up and executing any project is quite a task in a distributed environment. We have various groups (marketing, senior management, Solutions group) who are involved in the bid of the project and winning of the contract. Once the contract is signed, a delivery manager is hired who can deliver the commitments made to the client under the contract. Most of the times the he does not have the right mix of team skills. Almost 20% of a project’s efforts are spent on other activities viz. risk management, status reports, Measurements and analysis than development.

This is in complete contradiction with the SCRUM requirements and practices. In this post I am sharing my thoughts on offshore and SCRUM adoption with same case studies. Delivering the right software on schedule has long been one of the most difficult challenges faced by many businesses. The requirements get changed due to various factors viz. market pressure, new technology, business demands, regulatory requirements etc. With Waterfall model, the requirements have to be in “freeze” status before you proceed to design, design to be signed-off for development etc. And when users get the system for usage after 6 months, they needed it in different manner. We all call it “Change Request” if customer wants to change the behavior of the system after requirements are frozen, design is signed off.

With Waterfall, we try to deliver 100% on the day of “Go Live”. It might be the 60 new features, an enhancement across 15 products etc. In the process of design, development, the requirements keep on changing. This leads to mismanagement of requirements, increased number of defects in UAT leading substantial cost / time to fix those as you have to traverse back to requirements and change requests.

Result, more than 60% of medium and large software projects are “failures” in the sense that they will be delayed by six months or more, and many will be cancelled. And the delay means nothing is available to users in those six months. Either you will get the all functionalities or you get none. Due to these reasons we in consultation with client decided to adopt “Agile Working” i.e. SCRUM methodology.

SCRUM Methodology
SCRUM is an iterative approach for delivering business benefits. A project will consist of delivery iterations called as ‘sprints’. Each sprint is of month duration and it creates a complete increment of potentially shippable (releasable) product functionality. The SCRUM methodology consists of phases (planning, staging, development and release) and activities.
Planning and Staging prepare the workload for the Development phase, where all functional development is done iteratively, each Sprint creating a complete increment of potentially releasable product functionality. The activities provide information on what to do, but not how to do it. Scrum is an agile methodology. The people applying Scrum are provided with guidance, but they will require flexibility and experience to manage the project work and complete the activities. Their inspection of the unique project circumstances and characteristics is mandatory to the successful completion of any activity. With this methodology you have new roles and artifacts.
SCRUM is called as lightweight methodology. In reality lightweight is often defined as not heavyweight. Over time, the term lightweight has become another name of agile. But agile and lightweight are not synonyms. Lightweight in this case describes the weightiness of the process and the artifacts. And we talk about the artifacts or processes which do not add any value.

SCRUM Framework

As mentioned above SCRUM projects are delivered in iterations called sprints, and in each 30 (this can be 15, 21, 7) day duration sprint development team deploys a business benefit to the end users. Each sprint starts with Sprint planning meeting. Where product owner along with SCRUM master and the development team review the product backlog to prioritize them and decide the sprint backlog. Once the sprint backlog is agreed it is frozen for the month.

The sprint starts immediately after that. In the development stage a daily standup meeting is conducted which is called as daily scrum to discuss the progress. Only stakeholders are allowed to speak (chicken and pigs concept). With SCRUM the each task is 8 or at the max 16 hours long. Team member develops the module and tests it at the end of day. Automation and testing stubs play key role in this methodology. Formal processes like estimation, design, review etc. are done in this method also but in a different manner.

We do not try to estimate the system or a module which takes at least a week to develop. The focus is on smaller tasks, which later on adds to become a complex functionality. To keep track of the sprint we use the Burndown chart. This will tell you where exactly you are and how much is remained. Velocity is the key while estimating the tasks. Velocity is nothing but productivity and which is a factor of many things.

Prior to SCRUM

It was waterfall. Quarterly drops to production were planned but releases going in ‘RED’ state due to lot of factors. We had the “Quality Gate” processes at every phase but the exit from every gate was always conditional. Releases getting delayed, reverting back them, was happening more and more. Releases for post production fixes had their own path. Version control in the system and across the systems chain also adding the chaos.

In these circumstances, the end user of the system always had to wait n wait. By the time the functionality use to enter the User Acceptance Testing phase, he wanted the system to behave in different manner. But the “Change Requests” again use to take their own path sometimes adding more delay, resources etc.

From developer’s perspective, they always had a question how come the customer always says ‘we wanted this yesterday’. We are putting so much of hard work and when it comes to testing, many defects are always from a typical category ‘changed requirements’ or ‘out of scope’.

We wanted an best answer to this wherein the customer does not have to wait more for everything to be deployed and developers should be able to concentrate on the current set of requirements. We thought ‘SCRUM’ could be answer to this. Because, SCRUM talks about embracing the change instead of resisting it. This was the winning point.

With offshore vendors doing the development remotely, communication management and dependency management also have to be added to the cost, efforts and schedule. The release management gets complex when the client works with multiple vendors and / or geographically distributed teams. With all these challenges we started the first sprint.


Introduction of SCRUM for the enhancement
It all started with me going to client’s place for the traditional Quality gate 2 (QG2) process (i.e. decision has made for a new functionality development and we are ready for design), to understand the requirements and get a High Level Design Document along with the draft schedule. It was told to me that there is no QG2 arranged instead I have to attend a day introduction to SCRUM workshop. When we came out of the workshop, I was partly convinced, partly in the sense I could understand how it works, and partly I could not understand how we are going to adopt this for an enhancement which is impacting 6 systems and only 2 are going to do it SCRUM way. What will happen to the component integration and testing team efforts etc? Also, there were 4-5 teams / different vendors were involved who were also geographically distributed.

Key Challenges in SCRUM Planning
First question, why the entire team is needed to plan a release. Typically we require a project manager along with the design authorities. This release was introducing a new functionality and 5 different systems were affected and to make the matter worst were managed by 6 different teams which were not even co-located. The first question was the co-location demand of SCRUM. Coordination between teams following SCRUM and others was a challenge for the “Release Coordinator”.

The development teams were offshore based and SCRUM master along with the product owner was on a different shore of a different country. The time difference of 5 and half (it was actually 4 and half, thanks to day light saving) was a kind of bottleneck between the daily meetings. Continual availability of product owner for clarifications also was a big question mark.

From, the testing team perspective, they were bit surprised as if this going to be tested by the developers every day and finally gets deployed to live, then what the “Component testing” and the testing team will do.

The infrastructure team had their own work stack; getting the monthly drops in live environment by them needed lot of approvals and explanations.

The next question was raised by our own friends from Quality Department, are you planning for any deviations? Will you be able give us the metrics which form and impact the organization baselines. They were also interested in knowing the benefits to the organization and hence wanted to do benchmarking.

We had worked out answers to all these difficulties. The Sprints were planned in such way that we together deliver functionality to the testing team. We had a testing sprint too. So in all there were 5 SCRUM masters and one was master of these SCRUM masters. We indeed did a SCRUM of SCRUMS.

Offshore Pre-requisites and Approach
This all was sorted out at the enhancement level. Later, when I came back to offshore for starting the project execution it all started with the typical offshore challenges. Mix of senior and junior people, specialist team members (developer, tester to write and execute jUnit test cases, QA for testing the functionality etc.). Also, management wanted to utilize the same team composition as it was the organization framework for such projects.

People were not aware of how SCRUM works (rather how to work using SCRUM), were concerned about tracking of efforts in hours.

We have decided to use the first sprint as a “pilot” for this enhancement.
Actual Implementation
We had to arrange agile training for everybody. The product owners insisted on maintaining the Component Design documents. After all we were going to use the same for carrying out the enhancement. They were of great help to us in understanding the existing system and enhancing it though using ‘SCRUM’.

The component testing team member was absorbed in the development team and believe me it helped us like anything. His knowledge about the system and various scenarios were key in eliminating the defects before they occur.

Testing team had come up with their own sprint plan it started with developing the test strategy initially and as the development team started delivering the drops for testing they use to execute the test cases for that functionality.
Sprint #1 (The proof of concept)
Planning the first sprint was itself an excitement. The developers were enthusiastic about the word ‘estimation’. We had the product backlog with us. Priorities were defined by the product owner. Lot of queries was raised. Everybody knew that this was an enhancement. So we could not think from ‘right from scratch’. It was not like writing the code for a new functionality. We had to understand the existing modules, do the impact analysis and then make the changes. Unanimously we decided for 50% velocity i.e. 4 hours output by each team member in an 8 hour day.

The sprint backlog was developed and the tasks were created. We all had to update a section in the master sprint backlog. Every team use to have their own daily scrum (stand-up) meeting as first thing in the morning. It took almost a week for us to make it a real ‘SCRUM’ meeting. Then in the afternoon (client’s morning) we use to have the SCRUM of SCRUMs.

The team was inspired. For the first time, they though that they have the entire responsibility of delivering the project. They also shared the knowledge of the existing system. The interaction with the SCRUM Master (Project Manager) was considered as confirming the assumptions. The testing team member could suggest better ways of doing the things. The focus was on defect prevention and not detection (could not avoid it though). Luckily the first sprint went through smoothly. There were no leaves or off’s due to unavoidable reasons. We could deliver a tested business benefit to the testing them. And the best part of this sprint was that, even other team delivered the required change to their system and other teams were ready to test.
Sprint Review Meeting
We were now ready for the sprint review meeting. Everybody wanted to know about their performance. The review meeting lasted for almost 2 hours (from planned 45 minutes). There was lot of positive things to be told. The team was happy to interact with people from other teams and with the customer. For the first time they understood the business meaning of what they are delivering, otherwise it was always a Low level design implementation they had delivered. By interacting with other teams, they could understand the system in broader perspective. Their queries were discussed and resolved in a day or two due to the daily meetings. They also wanted to increase the velocity to 75% which was a good thing

As a SCRUM master, the project manager could concentrate on other aspects viz. automation, understanding team’s abilities and having people development plan accordingly. He did not have to ask everybody to update him with the status.

The only concern was about the readiness of the platform for testing, but we were ready for the further sprints.

So far the result was good. Team liked the SCRUM way of working. There was lot of curiosity in organization about SCRUM and they wanted to know how it works. Overall it worked well, but lot of improvement opportunities. And team continued to work on next sprint.
SCRUM of SCRUMS
This is the scenario we come across in Off-shoring model. An enhancement to be delivered by multiple vendors / teams for a business benefit that is implemented across end to end system. Also, involvement of infrastructure group, who had their own priority list, was itself is a big impediment.
SCRUM of SCRUMs is the answer to it. This approach had a product backlog initially defined at enhancement level. Then every team (from various vendors) had their own product backlog and formed different SCRUM team. We always had a role “Release Assurance Manager” to coordinate between various teams to make sure that the enhancement is delivered in tandem. With this approach he played a key role of SCRUM master for big SCRUM and set the priorities for each team. This helped in delivering the increments in required priority and always ensured that together the “SCRUM of SCRUMs” delivers a product increment to the End-to-End system. Every team has their own sprints, sprint backlog, scrum master and daily scrums. Then there is a Parent Sprint and after all teams finish their daily SCRUM meetings, there is this SCRUM of SCRUMs meeting.

Case Study II

Another Organization, with prior experience of driving SCRUM adoption, I was approached by my manager to share my experience with the new SCRUM project team. Again, it was a client mandate. We initially had a training session for all the PMs by an external agency on SCRUM and then there for the project team. I was always there to drive this effort with my experience of implementing the similar methodology.
With different Organization and different client, challenges were obviously different. We have two projects using SCRUM.

Project 1:

With this project, we have offshore SCRUM team and then there is an onsite coordinator. The offshore team does not participate in deciding the backlog items. The backlog and the priority are decided by client and offshore provide the estimates for the same. Onsite coordinator participates in sprint planning meeting to discuss the feasibility in terms of scope. Team participates in daily SCRUM using Video Conference
The team checks in the changes to client repository. The same code base is used by client and also by other vendors who work on the enhancement. Continuous integration is used to track the “code breaking”.
The team has their own challenges viz. absorbing new people to the team and training him on everything i.e. SCRUM, application, business etc. Recently the team had come across a challenge while complying with Organizations drive towards improving code quality. The codebase was owned by client and changed by many people. There were already many violations which were well above the upper limit. Team is working with all these challenges and slowly adopting SCRUM working in real sense.

Project 2:
This is the project where everything is in place i.e. development team participating in planning meeting, deciding the backlog, velocity etc. There is an onsite coordinator, who works with client and gets back to team whenever required. He plays the remote SCRUM master for the team.
This is the Organization where for the first time I have received formal requests to changing some of the Organization level processes for agile working i.e. SCRUM. I worked with concerned people (SEPG) group for tailoring the process and had the guidelines and new template.

This team is doing well with a concern that they are sharing the code base with client and another vendor. Though they are using a continuous integration tool, sometimes it becomes difficult to identify the defect ownerJOffshore perspective
· During the retrospective and discussion with other people who are practicing SCRUM, we could come up with following challenges while using SCRUM in offshore environment.
· SCRUM assumes the scrum team to be matured enough to understand the business, requirements, implementation techniques, have adequate knowledge of the software they are using. Practically, most of the offshore companies prefer to have a mixed set of people. In any typical offshore team you will have few freshers (who do not have any project experience), some good developers, 1-2 SQL developers and then we have a testing team who is reporting to different manager.
· Co-location of the SCRUM team (including product owner) is something very difficult to achieve with off-shoring.
· Time difference does have a significant impact on seeking feedback from client.
· Movement of resources across projects. If a sprint does not have some work for a team member, he will be released from that project and can get allocated to other. Having consistent resources is very difficult if the billing is not assured.
· Many times, it has been observed that the sprint backlog gets changed by client instead of cancelling and re-starting a new sprint.
· Many times, the sprint backlog is decided by the client and offshore has to complete it on time. This becomes Sprint for the client but business as usual for offshore J
· Some organizations still follow the Onsite-Offshore model where we have the onsite coordinator representing the offshore development team.
· SCRUM Master Role is performed by the people who were earlier project manager or the Lead. It has been discussed and proved that it is very difficult to come out of “command and control” working culture.
· Existing processes were not ready handle the SCRUM way of working. Many organizations follow various frameworks viz. ISO, CMM/I and every project needs to maintain certain set of documents, measurements etc. for the same. Tailoring these processes to adopt SCRUM way of working needs good understanding of involved frameworks and the SCUM.
· Organizations use “effort tracking system” to generate various metrics. This still has to be followed along with the burn-down chart.
· People do mention that, it becomes challenge while doing performance evaluation.
· And more……

Conclusion
This was discussed with various SCRUM practitioners (yahoo groups), various groups in Organization and with people who are already using it. The outcome of these discussions and study for me was simple “Rome was not built in a day”. It is not easy to change the habits and there is no “silver bullet” to do this. We have worked with “SEPG (Software Engineering Process Group) to tailor project management processes for SCRUM. Continuous training programs are being conducted for SCRUM in organization. We had to leave with some of the people issues (less experience and long learning curve) but were working on techniques (pair programming) to minimize the effect on deliveries.

Delivering the business benefits every month using the SCRUM methodology was successful. The key to this success was the planning, coordination and the team efforts.

With the use of SCRUM, I have seen the delivery of business benefits every month with reduction of post production defects and increase in team productivity. Other benefits were increased system knowledge of team members, team building, ownership etc.

The best part of this was the defect prevention and the detection. As people started understanding the system very well we came across more defect prevention scenarios than detection.

From the statistics point of view, we had observed that there was gradual increase in productivity and also the estimations were more accurate hence the effort variance was almost negligible.

There were many questions which still remain unanswered viz. following this framework for support projects, involvement of actual team in estimates, the communication gap between the people who submit the RFP (Request for proposal), wins the project and the team who actually delivers it.

For me it will be a paradigm shift for offshore if SCRUM has to be followed.