Using the Agile Framework for BI Development

I’ve become a huge fan of agile development.  Why?  Because we are committed to deliver working functionality every 30 days, and that is rewarding.  In this post I discuss our agile practices and why I believe they work extremely well.  This post is from the perspective of a developer on a team of 10.

What Is Agile?

First, what does agile really mean?  It’s not considered a methodology, a process, or even a set of procedures.  It’s more of a conceptual development framework which allows our team to iteratively prioritize & plan the work, perform the work, and review the work.  As opposed to the classic waterfall approach, agile acknowledges that we don’t have all the answers at the start of a project. 

One approach to agile is the scrum approach, which is discussed here.  Extreme programming is another approach to agile.

 

 

Sprints

The work of a project is divided into sprints.  I have been working in 30 day sprints for the past 9 months, and I believe that iteration works really well for our team size and project size.  A sprint begins with a Sprint Planning session, and it ends with a Sprint Review and a Sprint Retrospective.  We usually work on a calendar 30 day period, although we’ve been flexible a couple of times due to holidays or major milestones.

Working in fixed 30 day periods really prevents scope creep.  Sometimes we can fit in that extra little something, and we do, but sometimes it would jeopardize completing our commitments on time.  Since I can be guilty of scope creep (with good intentions of course!), having the structure of a sprint works extremely well for me.

 

Sprint Planning

At the beginning of each sprint, we take 2-4 hours to go through Sprint Planning.  This is where each member of the team helps plan and prioritize the next 30 days of work.  We each form our own estimates and our fantastic project manager makes adjustments and brings the entire plan together.  The idea is to deliver working functionality at the end of the period, without any heroic efforts.

Because we conduct the task level planning each sprint period, we can adapt to evolving requirements fairly readily.  Of course, our project manager gets to balance what’s acceptable evolution in requirements versus scope creep.  Because things can be a bit vague at the beginning of a sprint, team members need to be willing to accept some ambiguity and embrace change.  However, once the sprint begins the priorities need to remain the same throughout the entire sprint for this process to really work effectively.

 

Daily Scrum Meeting

Every morning we have a scrum meeting.  This is each person’s chance to explain what they are working on, if they need assistance from anyone else on the team, and if they have any impediments.  Although there are some days this is just a rote exercise, there are other days when the lines of communication get opened up because a topic came up that someone otherwise wouldn’t have known about, or someone happens to have valuable information but wouldn’t have thought to ask.  The daily scrum is particularly valuable when there are intra-sprint dependencies between persons (ex: person A needs person B to finish something before they can begin some work).

We’ve changed the scrum format a bit over time to try to ensure the entire team gets benefit from it.  For example, when it used to literally be a stand-up meeting some folks felt such a short discussion was of little use.  With mixed feelings, our scrum master (aka project manager) now allows the team to sit down and take a little longer.  We are not strictly time-boxed to 15 minutes, but we rarely exceed 20 to 30 minutes.

The time of the scrum has been a compromise, considering some of the team members arrive early and some arrive later.  We hold ours at 9:00 a.m.

 

Sprint Review

The sprint review, conducted at the end of each sprint, is fun.  Our product owners attend, and each member of the team literally shows off what they did.  Not only do we really appreciate what our teammates are doing, but the stakeholders are able to see progress firsthand as well.  This is a rewarding time of each sprint.

In the BI and Data Warehousing world, the sprint reviews get more tangible as the project progresses.  At the beginning, the data foundation and ETL is being constructed which is difficult to demo.  As the project progresses, visual components such as portals, reports, and scorecards evolve which are easier to demonstrate working functionality.  Even when most of the airtime goes to visualization in a sprint review, our project manager always makes sure the ETL team and other behind the scenes work gets noticed and appreciated.

 

Sprint Retrospective

The Sprint Retrospective is conducted at the end of the sprint.  In our team room, there are 3 large white sheets:  What Went Well, What Went Wrong, and New Approaches to Consider.  This is basically the “lessons learned” time of a sprint.  All team members get a pack of sticky notes and we go to work filling up those three sheets.  The first one, What Went Well, is just what you think.  We compliment each other & their work.

The second one, What Went Wrong, is our time to constructively discuss what we could have done better during the sprint.  Sometimes we learn valuable information on how something affected a coworker that we wouldn’t have otherwise known. 

The last sheet covers New Approaches to Consider, which is our chance to provide suggestions or opinions on how we might do things differently.  These suggestions cover a wide spectrum, and sometimes are related to the What Went Wrong sheet.  The change in scrum, discussed above, originated from a Sprint Retrospective suggestion.

The Sprint Retrospective is documented and kept for later reference.  The Sprint Retrospective is a really useful way for the team to reflect on how things went and then adjust accordingly.  To be successful, this practice needs a team of people that are really engaged.

Conclusion

One of the basic tenants to working in an agile fashion is we value working functionality over documentation.  While I agree with that philosophy entirely, getting documentation done alongside the work does tend to suffer.  Also, it’s clear the entire process works much more seamlessly when the entire team is devoted 100% to the project.

The above discussion is a bit of an Agile 101 discussion from the perspective of a BI developer.  There’s a lot more to know about agile:  all of the roles, how to handle backlog items, and so on.  We also have a sprint celebration lunch on occasion, to celebrate the particularly larger milestones.  This type of working environment is very collaborative, and it builds trust among the team.  It’s become my preferred development framework.

I’d enjoy hearing other experiences and thoughts about agile development in the comments.