CIO Rob Carter Transforms FedEx Into A Cloud Centric Enterprise

Original Article

Rob Carter is on the short list of the finest CIOs in history anywhere. I have profiled his accomplishments in an interview published roughly one year ago. A few months ago, Carter spoke at the Forbes CIO conference with his CEO, FedEx founder Fred Smith.  First, it was readily apparent how much Smith values Carter’s opinion, indicating that no strategic decisions are ever made without Carter involved. I was also particularly struck by Carter’s overview of the IT transformation that he has led. As he said then, and as he describes in greater detail in this interview with him, the systems that were at the core of FedEx’s success in recent years in becoming and maintaining its position as a logistics and data analytics leader would not be the same ones that would sustain its success. In fact, as Carter notes, the longer the organization waited to transform, the more those systems would actually impede its success.

Carter graciously agreed to dive into greater detail in describing the transformation, and what he describes as the “Four Horsemen of Dominant Design.” At the heart of this transformation would be a cloud-first strategy. Carter readily admits that this is a decade-long journey that they are roughly in the middle of, but the results so far have already proven the value of simplifying in the ways described herein.

Peter High: You have mentioned that it dawned on you a couple of years ago that the technology that helped FedEx achieve success was not the technology that would help FedEx maintain that success. What led to that insight?

Rob Carter: I started at this role in 2000, and at that time the number one thing that I wanted to do for the business was to become fast and flexible at creating business value, because the world was changing. The needs of FedEx and the expectations of our customers were changing rapidly. The complexity of the system which we had deployed –  which at one time was one of our most powerful assets –  was becoming a challenge to agility for our business.

The reality of the modern computing era – and with a company like ours that has deployed technology over the last 40 years – is that this is the advent of the modern computer era. 40 years isn’t a long time. We’ve deployed unbelievably strategic technology over the years, but some of them were things like 800 MHz spectrum coast to coast so we could stand up radio towers and communicate with data to our vehicles. These weren’t voice communication tools; they were data communication tools. They communicated with the first generation of handheld computers.

Well, as much of a breakthrough as that was in allowing information to flow freely back so that customers could have the ability to track their packages, it clearly became a legacy that burdened us as the modern generation of connected mobile devices began to surface. One of the first ways they surfaced was with cell networks that could communicate with industrial handhelds, not just consumer devices. So that is a window into creative and strategic deployment of technology which becomes a burden over time.

When you’ve invested in an aggressive way, as FedEx has, to further the technology brand that our company is well known for, you wind up with a lot of things that market forces find a way of providing a more general solution for; and often one that is more economical, flexible, and frankly works better than those first generations of technology.

High: What was the process of gaining buy-in, whether from Fred Smith, or your peers across the organization, or your team (who would be undertaking this work)?

Carter: There were two really important things that I brought to the attention of Fred and the broader leadership team. The first was to paint an ugly picture of what we had. I created a systems map which was a real map of all the applications and interfaces that we had. I needed that as a tool and to know what it was we were dealing with. It was such a daunting picture of complexity that I immediately showed it to Fred and to my business partners. Fred ended up naming this thing Hurricane Rob, since it looked like a big swirling hurricane. It became such a powerful tool to clarify the issue.

Mind you, that ugly picture was a powerful set of technologies that we have deployed in this business that automate virtually everything we do. Every time an airplane flies, a truck rolls, a package sorts, or a customer connects with us, it is embodied in this picture. But this picture is not sustainable. We can’t just keep layering complexity on top of complexity and expect a different result.

The fact that the picture was real, and it was kind of mind-numbing to look at was something that scared people. So, we also created another picture – which is the more important picture today – which is the systems model that we have been deploying into rolling through 2010 to where we are today. We looked at that and said, “We need to deploy a common core of technologies”.

IT_Systems_Model_May_2012There is an upper core (we call it the purple core) that is the customer core of all the things we do that face off to the customer. When you look at the FedEx brands – FedEx Express, FedEx Ground, FedEx Freight, FedEx Office – they each have unique properties and unique networks, but customers don’t need to understand that. They want one set of automation, one customer service, one bill, one way of becoming a customer, one account number – all of those things. So the purple core is the customer core.

There is a platinum core in the middle, which is another brand element that we have – the purple “Fed” is in every one of our brands. The platinum core is the common infrastructure, common networks, common data centers, common architectural deployments, and ultimately common services – services which represented a label, a rate, a route, an address, a customer, so we began to engineer durable assets in the middle that represented the fundamentals of our business.

The bottom core is called the multi-core. It represents the uniqueness of the systems that must be present to represent each of the FedEx brands. So, for example,  FedEx Express Company is the only company that is an airline; so it has airline systems embedded that are unique to it. FedEx Office is the only company that does print and copy; so it has print and copy, image management systems. This was all represented in the journey from the hurricane through a set of engineering that would bring you to this well-engineered future. And so, for the business, the important thing was to recognize where we are and recognize where we are going.

High: If I could back up just slightly. When you first created that ugly picture, at the same time you had some of the initial hypotheses about what these changes might entail, so you could have a before and after (ugly and pretty) from the outset from those initial conversations. Is that right?

Carter: We did. But there was a notion of the up and to the right path was going to be a journey. It wasn’t going to be a straight line. So it was kind of a well bracketed journey that says “we’re going to attempt to be less wrong about this journey every step of the way” We knew we didn’t have a perfect line of sight into how to do this, but we knew why we needed to do it, and we knew what it was that we needed to do. It’s just the “how” and the “who” that we have had to spend more time on figuring out how to get there.

High: Once the changes were mapped out, what was the path then? Was there a logic to why you pursued some of the changes earlier than others?

Carter: Yes.  Not so long ago, everything was different. There were different operating systems, different deployment models. Looking forward to the modern era, it wasn’t just different hardware and operating systems; it was network, storage, software deployment. So I coined the phrase, the “Four Horsemen of Dominant Design.” These were servers, networks, storage, and software.

I began to think really hard about how you might start a new company today. The probability of deploying a Z Series or I Series IBM box is near zero. You would deploy into this dominant design of scale-out microprocessors. Virtually every datacenter, every new technology company you can think of, from Google, Facebook, Amazon, Twitter, Zynga, to Rackspace, all have a similar deployment model into scale with big clusters of virtualized servers. And so, I made the conclusion that that was a dominant design factor.

The same thing applies to the network. When I started my career, I was actually a DECnet Administrator at one point, using DECnet to broker all the variety of networks that these computer systems used. There were so many different network protocols and things. Then moving forward to today, you find a dominant design in IT technology Internet Protocols, TCP-IP, which is now the lingua franca of computing. Nobody has to worry anymore about what language to speak on their networks. Everybody speaks TCP-IP and understands that, so there is a dominant design point, and the story goes on and on.

This was an important construct for my business partners too. I put that in some business analogs I used. The evolution of the railroads is a good example. In the US alone, I showed that there were eight different gauges of the railroad as the railroads were coming of age. Everybody was using the same language about it that they now use about computing:- “This changes everything, it’s the future, it’s what connects the world”. But it was very hard work to get to that common gauge of the railroad, which now exists, which is still, to this day, four feet, eight and a half inches. There were many different ones, they didn’t interconnect, creating issues.

A lot of reasons people thought that was the right thing to do at the time. That story was pretty compelling at the time when you say we are at a juncture in the computing revolution here where dominant design is beginning to emerge. We can make pretty big bets about how to deploy the next generation of computing and be reasonably sure that it is going to stick, that it’ll be flexible and adaptable to this cloud world that is emerging around us. Where utility computing, utility storage, utility network, and even utility software is becoming the way that things work.

High: I’m curious about the people implications of this. Obviously these significant changes meant that tasks that people had been doing for a number of years would be either fundamentally changed or even in some cases rendered irrelevant. And obviously the opposite applies, where new skills were necessary to be able to continually deliver against the new reality that you’re describing. I’m curious how you thought of your own team and changes that were necessary. How did you engage the external vendor community in some different ways as a result of some of the changes that were necessary?

Carter: We have an undying respect for people around here. It’s something that we do well. We trust that our people are flexible and capable of moving into the future with us. At the same time, we recognize that anytime you’re changing out a fleet, if you’re retiring all of your old Boeing 727s and you’re launching a bunch of 767s and 777s out there, you can’t afford to say that the 727s are useless, because you still have a lot of men and women out there flying them. That’s the same thing you have to do with the technology.

We needed and still need the people who are flying the hurricane. We didn’t want them to be disenfranchised. So, one of the things that was important for this whole story (and I didn’t just communicate this to my business partners; I communicated it to my team as well) was that I didn’t want it to be a mystery where we were going, so people could make their own choices about their career and what they wanted to do. But at the same time, I wanted to make sure that they understood that we had a lot of work to do between here and there and that there was an awful lot of need for the skill-sets that they represented. There was no wholesale transition there. While it is fairly revolutionary, we handled it like an aggressive evolution.

High: Does that mean new training programs that were necessary though, in terms of just even re-skilling people with existing staff. The methods used to make sure that they were cognizant of what the change would entail, how jobs would change. How did you think about that?

Carter: There was a lot of training available to them. There were job opportunities available to them. One of the best ways for technical people to learn is to transition to a new space, where the work is embodied. Most technologists that have been around for a while have made that transition any number of times from one language or one operating system or methodology to another.

High: How have you gauged success? You mentioned velocity as one of the key ingredients here. One of the things you were hoping to increase was business velocity, and eliminate friction in the process making for a more flexible model for IT, and for the enterprise, more generally speaking. As you think about the value you have gotten, the value you hope to continue to derive from these changes, what are some of the other factors that you’ve been thinking about?

Carter: There are several of them. One of them is something that we’re pretty fanatical about here at FedEx, and that’s the Service Quality Index, SQI. Every aspect of our business is metric based and how effectively and efficiently do we deliver this service that we are about. So we have SQI Indexes against all of the IT systems and capabilities in ways that let us measure whether or not these things are available for our customers and available for our business. One of the things we saw as we began to reduce complexity and increase modularity in these things, was that SQI numbers begin to improve significantly. As we sit here today, we’re about 21 percent ahead of our goals for this year for SQI, and those are goals that you must hit higher levels of performance every year in order to achieve them. So it’s not easy to be 21 percent ahead of SQI goals.

Cost is another big deal in everyone’s world these days. The cost-effectiveness with which we can deliver these modular industry standard kinds of technologies has been pretty remarkable. We’ve regularly lowered the cost on an absolute basis now to the business over the last few years in ways that create margin opportunity for the overall business. If the expense for IT as compared to the revenue of the corporation is a ratio that is decreasing, then that improves margins for the business. So that has been a critical thing.

Resiliency is another critical element. It is linked to SQI, but it is a bit of a different thing. How flexibly can you run these applications in multiple locations and provide better response when you’re running very unique footprints that go on and on as far as the eye can see. In big datacenters, it is hard to be resilient around business continuity, for example.

All of those are factors that we’ve tracked and that we’re paying attention to. And frankly, the hardest one of those is the one that I don’t want to act like we have solved, and that is speed for the business. The Holy Grail in enterprise IT is how to be responsive enough for the business. It is still the thing that we’re chasing. A lot of it has to do with the fact that there is momentum behind building all of these durable assets, whether they’re infrastructure assets or software assets. And we’re still coming through the era of having the complete set of assets that we need to be able to stitch together functionality and capability from the business in a horizontal fashion, much like the world does it today, with apps and things like that. You tap into services that are available instead of building big vertical applications like what the past had.

High: Where are you in the journey now? How far along are you? I’m not sure if this is a journey that is ever absolutely completed, but how close are you to some sort of stasis?

Carter: When I talk about the journey between the beginning state (the hurricane) and the future vision (which is the cores), we’ve always characterized that as a ten year journey across the decade we are currently in, so we are about half-way. But, as far as the deployment and to the infrastructure, the hardware, the networks, those are all new. So Fred can stand up confidently and say these guys have just really done a great job. He’s looking at brand new datacenters where we don’t have all this complexity embodied in these unique hardware instances. Our datacenters look like private cloud datacenters, which look like public cloud datacenters.

We’ve come a long way, and as we’re working our way up the stacks – the first three horsemen, servers, networks, and storage – are pretty well in hand. The fourth one, software, has always been the hard one. So, that takes a long time to architect, build, consume, and decommission. We call this the ABCD’s. We’ve architected and built many of these solutions, and we’re in the process of consuming them, and then decommissioning the old legacy applications. Address is a perfect and understandable example of this. The address service that we created. When we assessed the hurricane, there were 237 applications in the hurricane around the world that in one way or another managed or dealt with addresses –  physical addresses of our customers – whether it was the beginning or end of a journey, whether it was an origin or destination. Well, we built a robust and durable service, SHARE, which is the enterprise address service. It is now the address service of record. It collects about 50 million events per day from all over the world from all of our field personnel, customers, and team members out there. It essentially validates what is an address, what views of that address can you have; sometimes it’s the receiving customer, sometimes it’s a shipping customer.  The address service is now in the consume and decommission part of the journey where many of the 200+ applications that existed have gone away, and the business is now consuming its address needs from SHARE. Therefore, we’ve been able to decommission the old applications that were doing that. That story is having to repeat itself across virtually every business dimension we’ve got.

High: As you think about the vendors that you engage… Is one of the keys to limit that number? To try to get that down to a more manageable group of external resources that you’re engaging, or is there one philosophical point that can encapsulate the way in which you’ve thought about engaging external partners?

Carter: One of the things that we have that makes us a little bit unique, is that almost all of our software is custom software. For example, there was nobody out there creating ERP’s to track packages when we started. So we were dealing with thousands of custom built applications, because there was never anyone around to have these things built for you. To a large extent, there’s still not. That’s not to say that we don’t have key suppliers out there. We obviously do, but increasingly that’s becoming an open stack set of solutions that use industry common hardware and industry standard layered products up and down. That’s how you become a good cloud citizen.

One of the beliefs that I have, in what will ultimately be a hybrid cloud instance, is that in order to be a good cloud citizen, you have to look pretty similar inside in your own private cloud as you would look like when you’re outside in the public cloud. What makes hybrid effective is when it doesn’t have this significant mismatch between trying to tap into broadly available resources that are represented by public cloud services, whether they’re hardware, software, storage, or network, and then what you do inside your own four walls. So as we moved all of our applications to these highly virtualized big arrays, we didn’t just do a physical to virtual conversion, which is pretty easy to do. We moved all that software through a factory, moving it to architectural minimums that changed the way that software worked and allowed it to be highly leverageable and movable around these cloud infrastructures that exist. That’s kind of how that story has played out.

Leave a Reply

Your email address will not be published. Required fields are marked *