Developing Distributedly, Part 1: Tools for Remote Collaboration

The mobile web engineering team at the Wikimedia Foundation confronts a significant question every day: How does a team stay synchronized and productive when teammates are scattered around the globe? Our team is highly distributed and effective communication is a challenge. We focus on some of the highest priority engineering areas for the WMF, and at any point our team may include members in California, Colorado, Arizona , Russia, India, Poland, and the United Kingdom. To help us clear our geographic and temporal hurdles, we embrace a culture of continuous improvement in the ways we engage with one another and our work.
For many, having a collocated team is critical for success. We, on the other hand, see big advantages in our geographic distribution. The diversity of opinions, world experiences, and cultural knowledge that we can bring to the table through global distribution helps us better cope with some of the challenges we face in developing easy-to-use and accessible tools for a global audience. Of course, we might achieve this by generally hiring people from around the world, but the pool of exceptional recruits is much bigger when you don’t have a relocation requirement.
Similarly, having team members in other locales gives us greater exposure to people using our products on devices you might not commonly find in the US. Our geographic distribution also gives us closer to around-the-clock coverage in case of emergencies. Finally, building support for working remotely into the team gives us all a huge amount of freedom; any of us can travel (for business, pleasure, emergencies, etc) and not have to worry about falling out of sync with the rest of the team.
We make it all work with a multitude of communication tools, organizational practices, and discipline. While sometimes it can feel like there is no real substitute for face-to-face communication, our approach to these challenges has given us great strength, freedom, and resilience. The list of tools we use is long, and many of them are useful even for collocated teams. I will cover a few of the most crucial in detail below.

Video conferencing

Due in part to its simplicity, we rely on Google Hangouts for all of our regular meetings and for the occasional ad-hoc meeting where asynchronous communication will not do. In order to be effective, it’s important for video chat tools to stay out of the way so we can focus on communicating rather than wrestling with technical complications. Of the many tools we use, only one closely approximates in-person, face-to-face communication: group video chat. It’s the highest-bandwidth (yeah, pun intended) tool at our disposal and when used right, it can make all of the distance between us melt away.
Hangouts does this very well; it highlights the video of the person speaking, mutes your microphone when you’re typing, and screen-sharing is very simple. Audio and video quality adjusts automatically based on your bandwidth, and toggling your camera or mic on/off is trivial. The ‘effects’ feature lets you dress up participants in things like pirate hats and mustaches, making meetings more fun and helping the miles between participants disappear. Oh, and it’s free if you use GMail or Google Apps.
We have been iterating to find the best hardware setup for supporting Google Hangouts in the office in as unobtrusive a way as possible. While the mobile web team and other distributed teams have been working hard for improvements, the IT department has been doing an outstanding job helping to find and implement what works. We’ve dedicated one conference room to be our test subject and we are very close to finding an excellent set up. It’s unclear whether this will work well in our other conference rooms (do not underestimate how challenging room acoustics can be!), but we have come a long way from wasting half of our meeting time trying to get the technology to work for remote teammates.

Text

Most of the other tools that we use are text-based and asynchronous. The most important are Internet Relay Chat (IRC) and mailing lists. Both are used throughout the organization, even by teams who are entirely collocated, but they are even more useful for distributed teams.

IRC

Internet Relay Chat , or IRC, if you’re not already familiar with it, is a way for people to have real-time, yet asynchronous text-based chats. It has been used for a long time by Internet-based communities, and particularly for open source software projects. The WMF and its various teams maintain a lot of IRC ‘channels’, or chat rooms. #wikimedia-mobile, the IRC channel for the mobile department at the WMF, is heavily used and open to the public. It serves as the primary mode of day-to-day communication for the mobile department. It also serves as a place for Wikimedia community members to mingle, ask questions and potentially get involved with the projects.
Having a place for general conversation greatly strengthens our camaraderie. Conversations range from strategic and highly technical, to off-topic and regular chit-chat. Its asynchronous nature makes it very useful even for people who are sitting in the same room. When you’re head-down on a project and unable to get up and have a face-to-face conversation, you can still take a second to jot down a question, comment, or response in an IRC channel. While video chat is perhaps our most critical medium, IRC is the bread and butter of our communication tools.

Mailing lists

As useful as IRC can be, it’s really easy to miss a conversation that happened there – particularly if you were disconnected, sleeping, coding, or just not paying attention. While you can log and archive IRC conversations, it’s not particularly easy to catch up on things that were said while you weren’t present. For things that everyone on the team should see, we use mailing lists.
Along with the public mobile mailing list, we have an internal team mailing list that we use to coordinate meetings, document team decisions, and share other important things that may not need to be responded to immediately. In fact, the mobile web team borrowed a maxim from the Apache Community code of conduct: “if it didn’t happen on a mailing list, it didn’t happen.” This means if a decision is made somewhere other than on a mailing list and it is not communicated to everyone on-list, it is an invalid decision. This ensures that critical decisions are made in a place that we can all easily see and comment on, regardless of location or time zone; it can also easily be referred to in the future. This kind of transparency is crucial for team communication, particularly when we are not all in the same room when decisions get made.

Getting things done

We also use some tools to help us manage our work asynchronously, so we can all be aware of what needs to be done and who is working on what, regardless of time or place.

Project management

Like much of the Mediawiki engineering community, we use Bugzilla to track bugs reported in the software we develop. To simplify our workflow, we created Bingle, a tool that pipes bugs reported from Bugzilla into Mingle. This allows us to prioritize bugs against the rest of our planned work and keeps our workflow focussed on one tool, making it easier for all team members to stay in sync.The main tool we use for managing our engineering work is ‘Mingle’. We use it to catalog and prioritize all of the work to be done, and to track who is working on what. It gives visibility into the state of the project both for team members and external stake holders. In Mingle, engineers select what to work on from a prioritized to-do list, so they do not need to ask a manager what they should work on next. The increased visibility into the project and increased autonomy of the engineers simplifies participating in the day-to-day workings of the team for remote teammates.

Code management

Like most engineering teams, we rely on revision control for the software we write and maintain. We use Git, which is a distributed revision control system. Git makes it very easy for engineers to work asynchronously, and even offline, as multiple engineers can work on the same codebase, but independently of one another.
Along with the other WMF projects, we also use Gerrit, a code review system. When a developer submits new changes to the codebase, they must undergo review before being accepted. This ensures incoming changes are of high enough quality for the project and visible to others on the team, making it especially useful for participation of remote teammates and community contributors.

What’s missing?

The tools we currently use go a long way in keeping the team connected and synchronized. One thing that we have not been able to do is find a replacement for the fluidity or spontaneity of working on a whiteboard. In the office, whiteboards get used for brainstorming, diagramming, illustrating, both during ad-hoc in-person meetings and during regular team-wide meetings. There are a few expensive solutions to this problem (e.g. digital whiteboards), but we have not been able to find something that’s simultaneously affordable and easy to use. Leave us a comment if you have any good ideas!

Conclusion

With the right tools, we have been able to work very effectively as a distributed team. But it takes a lot more than tools to make a distributed team work well together. Stay tuned for part 2 where we will cover the less tangible things that keep the mobile web team working well in spite of the geographic distances between us. In the meantime, leave us a comment about any tools that you use that help keep your team in sync, or anything else you might like to discuss!
Arthur Richards, Senior Software Engineer, Wikimedia Foundation