- Carl Russo, former Cerent CEO and head of Cisco’s Optical Networking Group, Fiber Optics News, September 24, 2001
As of November 2012, when Doug left Cisco for Cyan (now part of Ciena), there was a total of 16,702 ‘454s’ operating within AT&T’s optical transport network.
Jeff Jacques, who left Cisco in February 2016 adds, “There must have been some 25,000 ‘454’ shelves shipped to AT&T, with some 18,000 in operation at any one time over the years.”
Back to Early 1999
Doug recalls that after his success cracking the IOCs in the northeast for Eric Clelland and then running the cable sales team under Terry Brown before the video option for the ‘454’ was canned, he moved over under John Colvin to work on the AT&T account.
Doug says, “He was sweet talking me that we need to go after the ‘Big Guy’ now, and I need someone on my team to run AT&T. What didn’t happen immediately though is I still had all the cable companies and that became . . . you can’t do that when you’re trying to break into a large account like AT&T. I ended up focusing solely on AT&T.”
“I brought Dan Scatena [into Cerent] from Fujitsu,” Doug recalls. “That was a good hire from two perspectives. He was the competition at Fujitsu and he knew Rochester Tel. I killed two birds with one stone. I set him up to run the IOC part of the NYNEX footprint . . .” Ultimately, Dan followed Doug to work for him on the AT&T account.
Keeping Cerent in the Running with AT&T Despite the Monterey Failing
“The same day that Cerent was acquired by Cisco, so was Monterey Networks,” Doug notes. “AT&T was planning to deploy Monterey Networks’ [wavelength router] in their core network, their optical core long distance network. I was under the understanding that we were actually in the process of taking an order for $30 million and we didn’t tell the customer that we were slipping the program for six or seven months. Once we did, AT&T pulled the order back. And that’s how [Ciena’s] Core Director got the AT&T core business. It’s the same people I had to deal with, like Dennis Morgan.”
Doug had to work through that major setback in the newly budding relationship between AT&T and Cisco’s newly formed optical business unit. “John helped with that,” he says. Cisco quickly got out from under the cloud of its Texas-based startup’s broken commitments in optical switching.
Focus on the Metro Space
“The near term piece for us was the metro, which I worked with Jeff Jacques and some others on. I spent a lot of time on that,” Doug recalls. “The first phone call I got was from John Medamana, who ran AT&T’s Ethernet lab. We had some ‘454s’ in their Ethernet lab, because they liked the Ethernet capabilities on a SONET platform. This was well before Ethernet became really popular. Density and price matter and physical compactness, but one of the deciding factors was having the flexibility for Ethernet. It was one of the big triggers that ultimately got us the win.”
Doug notes that AT&T is typical of the telco mentality where engineers design for worst case. Even though supporting Ethernet functionality on a SONET platform was one of the deciding factors for Cerent to win AT&T’s business, “they deployed maybe $2 million in plug-ins” even as we had sold them “some $600 million of ‘454s’ at this point.”
Achieving this goal was not easy, “We had to work through the contract issues and figure out how to work consignment. Getting the consignment model in place was near impossible, but Jeff Jacques got it done. That paid dividends. That first year of being approved got us about $438,000 and our sales quota was assigned as $50 million by Cisco management, but John [Colvin] didn’t fight this.” Cisco management did not understand the long sales cycle involved with service provider equipment sales.
But Doug and his guys, operating within Cisco’s standalone optical sales team, called on AT&T and persevered. “We got successful in Local Network Services (LNS) and hit $48 million in sales the following year. That was a hell of a growth path.”
It was time for the AT&T optical sales team to grow some more. Doug explains, “I got to bring Jeff Parow into the fold He came over from the XO sales team after the dot.com bubble burst.” Indeed, many CLECs were set to fail, although XO did okay during the telecom meltdown. Stephen Zielke also joined Juers team shortly thereafter, relocating from the Denver area.
“Jeff Parow got AT&T Core up and running,” Doug said of his former colleague in IOC sales and an earlier compatriot at Nortel Networks. “He set that program up.”
“We turned to the UVN side of the house and their unified services offerings. Nortel helped AT&T set up this service and then we came in as a second supplier for that,” Doug recalls. “We ended up kicking some ass and we took 60 percent of the market share.” Doug assigned Stephen Zielke to work it.
Doug is proud of his team’s success at AT&T. “We were the number one supplier ever since. It was tremendous. It was the best experience I’ve ever had. Also the ‘454’ was the best product I had ever sold. I was dedicated to it because I knew it was also a good product.”
Displacing Lucent as Top Dog at AT&T (Or how to beat an incumbent supplier)
AT&T was so dedicated to Lucent, Doug asserts, that “the first PO we got was hard-coded for DDM-2000, so they had to cross it out and put ONS 15454 on there.”
Lucent was desperate. They were about to lose a large amount of business to some upstart startup that was being backstopped by a company – Cisco – that had never been involved in optical fiber transmission. “Lucent tried to play the ‘it’s a small company’ card,” Doug recalls. They can’t scale to meet AT&T’s needs he was told, and so on. A myriad of reasons were propagated across AT&T by both Lucent personnel and AT&T employees who were either loyal to Lucent or had an equity position in the supplier of DDM-2000s. Doug adds, “[Lucent’s] Bill Gartner [1] was there when we started to break in at AT&T. He said, ‘Cerent can’t scale, they don’t have the support, they don’t have the technical ability.’”
Once Cisco announced its acquisition of Cerent, “Lucent played another card. They said, ‘What the hell; Cisco is an IT company? Do you really think they’re going to focus on Layer 1?’”
“That’s what Lucent tried. They obviously weren’t successful and we kept pushing. Lucent couldn’t meet the criteria or the capabilities of the ‘454’ platform. They had two or three new products in development but they just couldn’t get there,” Doug summarized. “This was the biggest challenge for us: to get into the local services [metro] side of the business” and displace AT&T and its legacy DDM-2000 product.
The ONS 15454 incursion into AT&T meant that remaining as a Lucent sales person calling on AT&T was a dead-end job. “After we started kicking some ass, I interviewed some of the first line sales guys, right on up to the VP level sales guy at Lucent,” Doug recounts. “We didn’t hire any of them, of course. I guess they realized, if you can’t beat them, join them.”
“I remember during one job interview with the Lucent vice-president, he mentioned something about Chromatis, like maybe they picked them up or something.” Doug knew he had nothing to fear from a legacy company picking up a startup with its half-baked product idea.
As it turned out, Chromatis had misread the optical transport market on a number of fronts: Metro DWDM was not the fast-growing segment, hundreds of millions of dollars in the sales pipeline was a myth, and Metro DWDM did not offer a cost-effective alternative to next generation SONET. On top of that, Lucent reached a point where it’s business downturn forced the company to prune its product mix and focus on the needs of its thirty largest customers. Chromatis bit the dust in August 2001, after just being acquired in May 2000 – a $4.5 billion mistake.
For every possible application that John Colvin originally mapped out during 1999, and more, Doug adds, “We were the only optical supplier at Layer 1 in all of their services. No other competitor could say that. It was every freakin’ service that they had.” The Cerent 454 inside the AT&T infrastructure was like Intel inside computers [2].
Ethernet was the Key for Cerent and Cisco to Dislodge Lucent
“In total, AT&T bought $600 million worth of ‘454,’ and, maybe, the Ethernet cards made up $2 or $3 million of that,” Doug said. It seems that the Ethernet component had a huge leveraging effect in the ultimate selection of the ‘454.’
“If you think about the mentality of a Tier 1 service provider, who engineers for what could happen,” Doug adds, “they saw Ethernet over SONET as a service. We had the capability in our box for it, thanks to the density. It was a safe decision for AT&T. The price was better than anything out there and we had it integrated.”
“The Ethernet push came from John Medamana in the Ethernet labs,” Doug recalls. “I had it in the Ethernet lab, I had it in their DSL lab, in their International lab. Everyone was doing different tests on Ethernet.”
It turns out that John was a key player for Cisco to support. Within AT&T, he had played a major role in the development of all relevant data networking applications such as Frame Relay, ATM, and then Ethernet technologies and MPLS. By 1996, AT&T had invented virtual private networking (VPN) and Ethernet provided the required access to the network. Ethernet, “became new again,” as a way to transmit IP packets. John was a pioneer for the telecom industry to make Ethernet an access technology and for him to make sure, for his carrier, that Ethernet was not used as a shared medium. Packets had to be separated as they transited the AT&T network and his work was to facilitate that separation.
The lab testing and the demands of a large Tier 1 service provider, like AT&T, “forced us to make a better product, not only from a reliability standpoint but also from a network management perspective,” Doug adds. A great working relationship ensued over the years and it helped us to overcome the “serious issues with the E-Series card,” among others. The ‘454’ would ultimately support a family of Ethernet cards: E-series for Ethernet and Fast Ethernet, ML-series leveraging Cisco’s IOS for a tight coupling with its routers, and G-series for gigabit Ethernet support.
Doug does not know why ATM withered on the vine in the marketplace, especially after much fanfare by the Bell Operating Companies during the 1990s, and so many telecom vendors supporting the ATM standards. At the time, he says, “There was no push for “IP Everywhere” or “IP + Optical” when we broke in [to AT&T].” But it soon became the industry’s buzz words.
Doug adds, “AT&T’s view was the market was going to move forward with Ethernet so they started to focus on Ethernet. I just think that as the market moved, ATM became less and less important.”
Cerent’s Market Mover Inspires AT&T as to What is Possible
The power of innovation was evident in the arrival of the Cerent 454, just as AT&T realized the importance of using Ethernet as the access vehicle to supports its packet-based virtual private network capability. Cerent and Cisco delivered the IP + Optical functionality that AT&T sought, a capability that Lucent never acknowledged was needed by its former customer.
The importance of offering Ethernet as a multi-service interface and pre-positioning the ‘454’ as the platform to support it was crucial to AT&T’s deployment of next generation SONET platforms. For Doug, “It was very memorable not only because I was there, but because it changed the game on how SONET multiplexers were manufactured and made. You could put what used to be an entire bay into a 17-slot shelf that could do OC-3s to OC-192s. It was one of the most innovative developments in [optical] transport in the last 20 years. Nortel, the leader, didn’t have it, Lucent didn’t have it, Alcatel didn’t have it, Fujitsu didn’t have it. These ****ers were the leaders and they were not leading. This little company who looked for a seam in the network, found out what the issues were, and created a box that could scale, without changing anything, just by adding a card. It was the most innovative thing I had ever seen, up until we start talking about SDN today. It was the best product I ever sold in my life. It was one of the most enjoyable times of my life.”
For AT&T’s evolution during the 2000s, Cisco’s IP + Optical enablers matched the nationwide service provider’s growing pains and helped the company become competitive with new service offerings that could support the convergence of voice, data, and later, video traffic.
[1] Bill Gartner was Vice President and General Manager of Lucent's Optical Networking Systems business, responsible for Metro, Access, and Long Haul DWDM Products. In this role, he was responsible for well over $1 billion of revenue, with 500 employees in five worldwide locations. After being competitively pummeled by companies like Nortel and Ciena in the long haul and Cerent (Cisco) in the metro, Gartner left Lucent in 2000 to make a go of it in the world of startups. He held COO jobs with Photuris, Mahi Networks (next door to Cerent in Petaluma), and Meriton Networks, cumulatively for eight years, before, ironically, landing a job with Cisco, heading up the very business unit – Optical Transport – that had been his undoing, some say, at Lucent.
[2] The Cerent 454/Cisco ONS15454s operating inside the traditional AT&T network are operating on software load 9.2.2.1 as of February 2016. The SBC and BellSouth portions of the AT&T network standardized on other software loads. In total, Jeff Jacques tells me, “There must have been some 25,000 ‘454’ shelves shipped to AT&T, with some 18,000 in operation at any one time over the years.”