All Articles

Don't be like Boeing

Today news broke that Boeing had outsourced software engineering for components of the deadly 737 Max aircraft to $9 USD per hour contractors based in India (https://bloom.bg/2IWbxOx). When I read this headline a rush of anger initially ran over me: “How could they be so cheap with something so critical”? Unfortunately, it’s far too common for companies, particularly those not categorized explicitly as tech, to see engineering as a cost center that only adds business overhead. This attitude is rampant in business’s where non-technical MBA-type executives call the shots, valuing concepts like efficiency and synergy over correctness and high-quality.

In the software world there is a concept known as technical debt (https://bit.ly/2VT75bF). Technical debt, unlike financial debt, is next to impossible to measure. However, to any semi-experienced engineer the cascading effects of poor leadership and design decisions can derail a projects progress at a super-linear rate. Short cuts, coupled with decisions based on time or financial constraints, often lead to gaps in the underlying codebases that make up a product. These products then become even harder to correct as complexity is added to the system. As someone who already has a mild fear of flying, it is terrifying to think that arbitrary product deadlines and executive pay packages can outweigh building mission critical software. The question therefore, as engineers, how can we prevent this from occurring? How can we make our voice heard if we are not in charge of the decision?

  1. Triple your estimates

    For starters, we engineers are notoriously bad at estimating time to completion on products. This is due to those pesky unknown unknowns that come along with introducing changes to increasingly complex systems. For this reason, software design has lent itself to more modular paradigms (e.g. object-oriented programming or functional programming) versus procedural styles. Focus your design decisions around these paradigms. Keep your sprint items as atomic as possible. And triple your estimated time to completion. It will always take longer than you think. For more on estimates checkout The Mythical Man-Month (https://amzn.to/2XdY5yc)

  2. Stop feature stuffing

    One of the most aggravating things product managers will do is attempt to stuff new features in last second. This can (and eventually will) backfire. Bugs will be introduced, leaving the engineers to be blamed. Don’t allow this. When you receive a project scope, stick to the plan. If a product manager insists on a new feature making it into a release tell them they must sacrifice something to make space.

  3. Ask why

    At Boeing someone should have asked why they are outsourcing engineering teams. Don’t simply take work as it comes. If you receive push back and want to avoid confrontation, simply say you want to learn more about how the business operates. If they tell you it is to cut costs, or to meet some ridiculous deadline, what they are really saying is they do not value engineering talent or high-quality products. I would recommend you leave this type of company.

  4. Show them the math

    Outsourcing to cheaper labor rarely never works for high skill professions. We don’t outsource medical advice. We don’t outsource scientific research. We don’t outsource architectural design of skyscrapers. Why outsource software development? It is equally critical to 9 out of 10 companies. And in some cases, can mean life or death for a company’s customers. Your company will spend as much time reviewing and fixing issues as you would have building the system correctly in the first place. Say a company outsources the development of a new piece of software that takes 500-man hours (https://amzn.to/2XdY5yc). The offshore labor is paid $15 per hour (a generous sum by Boeing’s standards!). This brings the total cost of the product to $7500 (500 x $15). In comparison, engineers based in San Francisco making $100 per hour would cost you $50,000. Woah! You could save your company $42,500.

    This hypothetical situation sounds great to the executive with no real understanding of how software is built. In reality, a project that takes an offshore team 500 hours will almost certainly consume 50% fewer hours for your company’s on-site engineers. This is due to pre-existing domain expertise, understanding of operational overhead and nuance of deploying this new software and consistent development feedback from product and sales teams. Cutting our in-house time estimate in half brings us to a $17,500 difference in cost - still over 3x cheaper to outsource.

    Hmm, maybe we should outsource all of our engineering…

    But what happens in 6 months when our customers want new features? A projects initial development can easily be capped by some agreed upon specification. Once released however, the growth and evolution of a software system can spread in unpredictable ways. This is where domain experts and on-site engineering shine. By being close to product and sales teams in an organization, on-site engineering will have orders of magnitude faster iteration and turn around on specification updates. An outsource team may insist that new features do not fit into the existing spec and require a whole system re-write (It pains me to think of the number of times I have had outsource teams do this with a project I have worked on). Development velocity increases as Knowledge of the underlying system is shared among engineering teams and between business units. However, if a system matures in the hands of an outsourced team then all domain knowledge about the product will be black boxed from the parent organization. Eventually this leads to an organization being handcuffed to the outsourced firm. What follows is years of retainer payments for no, or negative value, work.

What can business leadership and product teams do to prevent becoming the next Boeing? For starters, when thinking about engineering costs, executives should amortize engineering overhead. Up-front costs for software will always be high, but the operational leverage achieved from well-built software compounds over time. Second, listen to your engineering leadership teams. If they push back on features, roadmap estimates or labor allocations, then business leadership needs to listen. While it isn’t clear who made the call to offshore labor at Boeing, I’d place a hefty bet that it was coming from someone with an MBA and with little to no technical experience in the work they chose to outsource.