Time Warner Telecom (tw telecom), a Competitive Local Exchange Carrier (CLEC) visited Cerent’s headquarters in Petaluma, on August 17, 1999, to check out the capabilities of this upstart startup. Visiting executives included Ray Whinery, Executive Vice President (Engineering, Technology and Field Operations), Tony Thakur, Vice President of Technology and Architecture, and Michael Khalilian, Director Technology Applications Development. Cerent sales representatives Dave Cesca and Stephen Zielke, who hosted this customer event, had only been with the company for six months and one month, respectively.
 
Earl Turner, tw telecom’s Chief Architect, who I interviewed for my book project, worked with Tony Thakur on network architecture for the Time Warner “properties.”
 
Todd Murphy, another Cerent sales rep, who worked the Time Warner Cable account in Denver, wrote, on October 22, 1999, “They intend to use the 454 as [an] OC-48 with DS3 drops and some OC-3/OC-12 drops.  Tony has no current plans for TLS, but Earl said he may want to run their TWTC ISP traffic via the Ethernet card.” Murphy added, “Tony said the interoperability testing will be conducted with their Lucent and Alcatel cross-connects.”

Cerent’s Multi-service Provisioning Platform (MSPP) soon became a hit with tw telecom and was rapidly accepted for deployment across most of its metropolitan markets, especially after it passed all of the interoperability testing with other vendors’ equipment.
 
Dave Cesca recalls the time of the telecom meltdown in 2001, “When the market started crashing and the revenues went away, i.e., the voice revenues went away, service providers had to figure out how to deliver cheaper services. To the customer premise, T1 wasn’t cutting it, T3 wasn’t cutting it, and SONET wasn’t cutting it. More service providers were putting Ethernet capability into their buildings because customers were also asking for Ethernet to the building. The cost points for Ethernet came down to a point where it was more cost-effective to deliver Ethernet to the building and do so over a SONET box like the Cerent 454.” 

Cerent’s ‘327’ product arrived, renamed as the Cisco ONS 15327. CLECs, such as tw telecom, helped to define this “baby brother” companion product to the ‘454.’ Dave adds, “The ‘327’ was critical. We needed the central office box, if you will, or the backbone box at the central office and we needed the edge box to connect to the customers and to really scale the business and bring the services onto their network. You could put a ‘454’ out there, but to really get to the smaller offices, you needed the ‘327.’”
 
Dave believes tw telecom was the first ‘327’ customer. However, as he recalls, “The ‘454’ still outsold the ‘327,’ but it was a super-critical component in the smaller offices.”
 
Earl Turner told me during our interview in 2013 that tw telecom bought about one ‘327’ for every ten ‘454s’ across the tw telecom network. More and more landlords were accepting MSPP-based gear located within their buildings.
 
Tony Thakur said on July 19, 2000, at the ICM Conference in Washington, D.C., “Vendors: Please provide old and new services on one platform; multiple boxes to manage is not what CLECs want.” He continued, “You will see metro-DWDM for carriers and next generation SONET for others requiring service aggregation. Overlay networks will be built. [The operators] must be able to point and click so any wavelength can be set-up from A–to–Z.” He highlighted the importance of Next Generation SONET equipment, like the Cerent 454, for data applications.

Stephen Zielke recognized the importance of Ethernet functionality on the Cerent 454 and Cerent 327 platforms too. During our October 2015 interview he noted how the optical team dodged the bullet on some of the failures of the Cerent team’s early Ethernet program (the so-called E-series cards). The Ethernet program became very important to grow the optical business as TDM interfaces waned. He said, “We failed with the E-series and we failed with the ML-series. We got it right with the line-rate G-series cards. Those cards were winners."

Steve’s assertion is seen by many Cerent pioneers as unfair, especially since the E-series program was key to penetrating the CLEC and ISP markets, the very market Steve was hired to tackle, which he did very well.
 
The history of Cerent’s E-Series Ethernet/Fast Ethernet plug-in is a story in itself. In short, the E-series solution was optimized for voice engineers to embrace data traffic in a TDM-based, service provider network. The E-series, Bob Bortolotto writes, “was designed, on purpose, to have nothing to do with IP, which is a layer-3 protocol. The purpose of the E-series was to statistically multiplex Ethernet traffic as transparently as possible (layer 2) over SONET TDM transport. This is exactly what our CLEC customers wanted at that time. They certainly didn't want an IP router inside their SONET ADM. The G-Series plug-in was the same basic card but with gigabit Ethernet interfaces and some other additional capabilities (like link aggregation).” As far as the ML-series Ethernet cards were concerned, Bob adds, “When you are part of Cisco, everything migrates to IOS, so that was the directive, good or bad.”

Stephen has a point though. Unlike the G-series cards, those ML-series cards also required IOS functionality, inside of the Cerent operating system (VxWorks). Although the bigger Cisco team lost some credibility with that approach, since Cisco’s IOS was not a carrier class operating system, Cerent’s development team within Cisco weathered the storm and Ethernet interface sales on the ‘454’ soared.

Case in point. I talked with Curt Kavalo, a network technician, in March 2014. He was working that day in the Ashtabula, Ohio central office, which was once part of the former Alltel footprint. He looked at the front of the original Cerent 454 for me, in operation since 1999, complete with its signature purple shelf cover installed, and saw it functioning with version 7.02 software.
 
He said, “It’s been plugging away as far as I know. We were just working with this thing yesterday; we removed a DS3 in favor of an Ethernet interface. And I love the software. It’s user friendly.”
 
Fifteen years in operation and it’s still upgradable from TDM to Ethernet – not too shabby.
 
2003 Era for tw telecom
By March 19, 2003, the following customers were using tw telecom’s Metro Ethernet services over a Cerent 454 infrastructure. A tw telecom press release boasted, “Customers already experiencing the value of Time Warner Telecom Native LAN services include: First Tennessee National (Memphis), Bank of the West (San Francisco), Epic Imaging (Portland, Oregon), Carondelet Health Network (Tucson, Arizona), New York Unified Court System (Albany, N.Y.), University of Rochester (Rochester, N.Y.), Chase Manhattan Mortgage (Columbus, Ohio), . . .” and many more.
 
A couple of industry analysts praised tw telecom’s capital investments, in spite of the telecom meltdown just a couple of years before: “A lot of carriers are delaying capital expenditures in the hopes of boosting short term financial positions, and that's a huge mistake, because it gives competitors with foresight the opportunity to take your customers away with broader and more cost-effective services,” says Daniel D. Briere, CEO of TeleChoice, a telecommunications industry analyst. “Time Warner Telecom is not only willing to make that CAPEX spend, but they've already done it, and have launched and populated several leading edge metro Ethernet services. This raises the ante in the metro loop in all the markets where they offer service.”

By October 2003, Mike Rouleau, tw telecom’s SVP, Business Development, reported that their network had more than 17,000 local and regional fiber route miles across 44 markets with nearly 3,700 buildings lit with fiber-based services. Their national footprint was interconnected with fiber and an IP backbone in what Cisco would call an IP + Optical solution.
 
Rouleau cited a May 2003 IDC report that tw telecom offered the most comprehensive Metro Ethernet Access speeds in the greatest number of markets (44 of them across 22 states) and with the greatest variety of applications (Transparent LAN Services or TLS, Fibre Channel, Internet Access, and ESCON for storage).
 
“Time Warner Telecom offers one of the most comprehensive metro Ethernet portfolios on the market today, encompassing point-to-point, switched multipoint and premium SONET-based Ethernet flavors, as well as Internet access over Ethernet,” said Ron Kaplan, Research Manager, IDC. "But offering services is not enough. Success in metro Ethernet hinges on fiber availability, and Time Warner Telecom has its own extensive metro fiber networks already in place. Combined, these two elements make Time Warner Telecom well-positioned in the metro Ethernet services market.”
 
“Our Native LAN services deliver Ethernet, Everywhere, Easily. That's tomorrow's network, available today!” Rouleau added. And both the Cerent 454 and the Cerent 327 had plenty of horsepower under the hood to ensure tw telecom met with success in the metro marketplace.
 
2013 Era for tw telecom
Fast forward a decade later: By November 7, 2013, tw telecom continued to grow. The service provider's on-net footprint sported a total of 19,648 buildings connected to its fiber network, up from 3,700 buildings ten years earlier.
 
“By accelerating the expansion of our existing markets using our established operational teams and infrastructure, as well as entering new cities where our customers already have networking needs, this expansion gives us quick access to current demand and an accelerated path to greater revenue opportunities,” said John Blount, tw telecom's chief operating officer, in a press release.
 
The timing of this expansion came as tw telecom reported that it earned $393.2 million of revenue in Q3 2013, up 1 percent sequentially and 6.6 percent year over year.
 
Less than two years later, tw telecom was acquired by its competitor, Level 3, primarily for its Ethernet services offerings and large installed customer base across the United States. Success begets success and Cerent’s evolutionary optical transport platform played a key part of tw telecom and many other service providers’ success introducing Ethernet-based service offerings.
 

​John Colvin was hired by Terry Brown in early 1999, as Cerent’s third sales director, after Eric Clelland and Jeff Santos. Colvin arrived as a lone wolf, a man without a supporting cast, a sales executive who left Alcatel to join Cerent and cover the National Carriers for this northern California Upstart Startup. He accepted the role to introduce Cerent to America’s Tier 1 service providers – AT&T, MCI, Sprint, Worldcom, and the seven Regional Bell Operating Companies (e.g. BellSouth and Southwestern Bell) – and to inform Cerent about the product and network requirements for large carrier class operators.
 
One of his key targets was AT&T and John hit the ground running. 

On March 31, 1999, John Colvin met with AT&T Local Services Division (ALS), the former Teleport Communications Group (TCG), in Staten Island, New York for the purpose of presenting the Cerent 454 and discussing potential applications in their network. Those attending the meeting included Darryl Freeman (District Manager, Local Services Research and Development), Tom Hill (District Manager, Local Services Technology Research and Development), and both Ken Kuboosh and Bob Stickle of Local Services Network Infrastructure. At the time, ALS, since it had acquired the TCG assets, was operating in 60 cities with another 12 cities planned for the balance of the year. It was believed that 85 percent of the equipment installed in their local networks was sourced from Lucent.
 
ALS was apparently quite impressed to see Cisco become an investor in Cerent and stated that level of involvement could help in opening doors within AT&T. Some of the ALS team members were also surprised to see Cisco as an investor since Cisco was supposedly coming out with their own optical transport solutions.
 
The larger AT&T organization itself, was, in the process of implementing major cost reduction programs and one of the main areas of cost savings involved moving, where possible, all DS3/DS1 circuits from the RBOC networks to the legacy TCG network, thereby eliminating the high cost of RBOC access charges. This shift in network utilization would provide substantial savings to AT&T in 1999. However, this program was eating up the legacy TCG's network capacity and infrastructure. And the existing legacy Lucent SONET gear was not very scalable. Many forklift upgrades were needed to migrate from OC-12 to OC-48 capacity, for example. To most observers within AT&T, Cerent appeared as telecom’s white knight in shining armor.
 
Lucent’s equipment shortcomings for metropolitan applications compelled ALS to review next generation platforms and hence plans were set in motion to issue a Request For Proposal (RFP) during the summer of 1999. That RFP would include requests for next generation transport and metropolitan DWDM systems.
 
During John Colvin’s presentation, ALS members fell in love with Cerent 454’s platform and architecture. ALS believed that the Cerent solution was taking the right direction and approach for their network evolution. High-density port cards, as Cerent proposed, were seen in a positive light, but, a battle was expected over the needed mindset change for the maintenance organizations.
 
Darryl Freeman believed that Cerent's main competitor would be the products Pipelinks was delivering and/or coming out with. Once again, it was expressed that Lucent's next generation systems were looked on negatively by ALS and seen as late to market. Nortel's systems were looked upon as “closed” systems, for example, they cited the fact that Nortel's optical transport system must only be used with Nortel's DWDM systems.

Targeting Metro Applications
Three months passed. Then, on June 16, 1999, John Colvin followed up with the AT&T Local Services (ALS) organization, once more, and received further validation of the Cerent 454 value proposition for metropolitan deployment. Ken Kuboosh shared that ALS was very focused on “Data Aware” platforms for all of their next generation products. The Cerent 454, with its planned data interfaces, qualified, and John captured AT&T’s requirements as follows:
     1. Single platforms for voice and data (which fit nicely into Cerent’s view of the optimized SONET infrastructure to carry both voice and data traffic)
   2. Eliminate the need for multiple and sub-tended boxes for offering functionality (an ability that Cerent offered with its multi-service interfaces for optics, voice, and data options)
     3. Better efficiency at the edge of the networks (which Cerent provided in the ability of the Cerent 454 to switch traffic locally, obviating the need to deploy separate 3/1 cross-connects or Layer 2 Ethernet switches)
​    4. Optimum utilization of the transport bandwidth (a core feature of the Cerent 454 with its ability to efficiently pack traffic into STS or OC-n formats) 

​Ken noted that ALS’s focus on next generation transport systems addressed three areas of the AT&T network:

1. Collocation applications
2. Placement at the edge of their larger city networks
3. A one-box, complete solution for their smaller cities

Once again, the Cerent 454 fit these needs perfectly. (Later, the baby brother of the Cerent 454, the Cerent 327, was rolled-out for edge applications called customer premises solutions as a complete solution in a box, but AT&T never deployed them; they stuck with the bigger ‘454’ for ease of sparing and maintenance.)
 
The earlier TCG architecture that ALS inherited, utilized centralized Alcatel and Tellabs cross-connects. ALS was experiencing massive expansion in their networks and they were forced to double all of their wideband cross-connect systems from Tellabs and Alcatel, which in turn rapidly consumed bandwidth across the optical infrastructure, just to switch the traffic. This approach was increasingly becoming too inefficient. To overcome this problem, ALS sought to move some of the cross-connect functionality out to the edge of the networks. ALS was very interested in the Cerent 454's ability to manage and groom VT1.5s with the second release (R2.0) of Cerent’s product.

​Cerent’s value proposition was reinforced with the following attributes recognized as important by Ken:

1. Economics: Saving capital expenditures (CAPEX) was very big within AT&T’s local network buildouts
2. Collocation applications: Reduction of real estate was huge and high-density ports contribute to further space savings. A flexible architecture for TDM, ATM, and IP that Cerent offered, was a big benefit too.
3. The ability to support DS1, DS3, OC-3, and OC-12 services with OC-48 rings was critical
4. Cross-connect functionality, when used at the edge of the ALS network could help reduce port counts needed on wideband cross-connect systems (at those centralized cross-connect locations)
5. “Data Aware” transport for future service offering as the ALS network evolved to become a next generation one

The following day, John Colvin connected with another part of the massive AT&T organization. It was June 17, 1999, and John met with Ahmet Arslan, AT&T’s District Manager, SONET and Optical Network Evolution, to discuss the potential application of and opportunity for the Cerent 454 with respect to concatenated (unframed) SONET payloads. AT&T's Core Network was looking for platforms and solutions that supported the ability to carry STS-48c and STS-192c chunks of bandwidth. AT&T was getting more and more demands for “Cross-Country OC-48c” (city-to-city) services from hand-offs originating from ATM switches, Frame Relay switches, and IP Routers. In these applications the payload had to be transported intact.
 
Prior to the Cerent 454, AT&T was offering OC-48c services to their customers but in an "Un-Restorable" or non-protected mode. Furthermore, OC-48c payloads were launched into AT&T’s DWDM systems in point-to-point applications across their backbone network. On top of that, a DWDM systems’ ability to support optical ring switching or OBLSR was not available. Finally, traditional OC-48 ADM systems from Lucent or NEC did not support the carriage of OC-48c payloads.
 
Cerent came to AT&T’s rescue with a “Restorable OC-48c service” capability. In the near-term, AT&T simply wanted “Restorable OC-48c Services,” where the concatenated payload could be guaranteed by AT&T. This simply meant that the OC-48c was “Ring Protected” for their customers. As John noted, “. . . this capability would allow AT&T to charge more for these services. AT&T had been looking into several ways of accomplishing this with other vendors, including: DWDM OBLSR (Not Available!); Hardware or Software modifications to existing transport ADM solutions (Not Available!); Geographically routed Working-Protect point-to-point solutions (Not Practical!); or Next Generation Transport solutions (the Best Solution and Cerent's Opportunity!)”
 
AT&T was utilizing Lucent and NEC's OC-48 ADMs, which did not support STS-48c through their systems. The highest STS granularity that these systems supported was STS-12c. As a result, AT&T looked into buying OC-192 systems for their network simply to get OC-48c carriage, but the cost of the technology was too expensive and did not economically prove in for them. The OC-48c service offering was the only application that AT&T could target for the proposed OC-192 systems. AT&T might have utilized Nortel as their supplier back then, but AT&T Core did not want to deploy very much of this new technology and preferred to utilize OC-48c over OC-48 rings.
 
John Colvin felt the best solution for AT&T’s needs was their preference for large rings – OC-48 4F-BLSR – as the topology of choice, but they would settle for OC-48 2F-UPSR to support them near-term. AT&T was extremely interested in the Cerent 454's ability to support this OC-48c carriage application and wanted to dig deeper into the product's architecture. AT&T would deploy the Cerent technology in ring topologies but they wanted to understand how many of these configurations could be supported in a single ‘454’ shelf.

Things were looking up as John continued to stoke the fires across many AT&T organizations – ALS and Core, in particular.
 
Three weeks later, on July 8, 1999, John Colvin discussed with AT&T’s Ken Duell, Member Technical Staff, Core Transport Evolution, in Middletown, New Jersey, the Cerent 454 and potential network opportunities within AT&T’s Core Networks.
 
Duell was assigned by Tom Afferton, AT&T District Manager, Core Transport Evolution [1], and Troy Adams, AT&T District Manager, SONET Transport Evolution, to further assess the Cerent 454 for two applications [2]. Cerent’s product was positioned within AT&T as a technology and alternative solution for these two key applications:

1. A low cost solution as an OC-48 aggregator of DS3, OC-3, and OC-12 interfaces.  These systems would launch into higher-level restoration systems, a capability available in the first release of the Cerent 454. (Both Williams and QWEST had already chosen Cerent for this application.)
2. A solution for carrying OC-48c restorable services, with restorable meaning “protected by ring technologies.” 

Within AT&T, these applications were a very high priority and Duell was tasked with completing the assessment of technologies by the end of July and provide a report to management in early August. Cerent was high on his list of companies to assess.
 
In his communication, John Colvin said Cerent needed to cautiously proceed “because if we get the Core Networks team from AT&T truly turned on, then they could literally swallow Cerent up.”
 
A Third AT&T Application Identified
Cerent continued living off the high of its Supercomm buzz from early June and the momentum with customer orders only grew larger. On July 23, 1999, John Colvin and some colleagues met with an enthusiastic crowd of more than 30 members of AT&T’s Core Network organizations. The buzz was all about Cerent, especially after the company’s successful Supercomm launch and continued publicized wins. By now, John had hired Jeff Jacques (pronounced “Jakes”) to assist in staying on top of the overwhelming interest by AT&T. There was now a third identified application that could be immediately addressed by the Cerent 454:

1. Role as an OC-48 Multiplexer of DS3, OC-3 and OC-12 interfaces (aggregator)
2. Delivery of OC-48c restorable services over OC-48 rings
3. Delivery of SDH services within the AT&T US domestic network; Cerent’s system could support the SDH version of optical traffic too.

 
To address item 3., Cerent subsequently informed AT&T in detail, “We plan to provide SDH support initially in Release 2.1: The currently available optics boards (OC-3, OC-12, OC-48) will then also support the comparable STM rates (STM-1, STM-4, STM-16).  A given card will be software-provisionable for either SONET or SDH payloads.  The SDH payload will be mapped onto an STSxc.  This capability will eliminate the need for a separate SDH gateway element."
​
John added, "In Release 3.0, when the OC-192 card becomes available, it will support STM-64 – again, it will be software-provisionable for either SONET or SDH. SDH traffic is mapped onto the comparable STSxc. Through Release 3, the smallest granularity is STM-1. Our future plans include E1 granularity and VT2 cross connect management [in support of international traffic].” 

On industry participation: AT&T was informed that Cerent was an active participant in SIF, the SONET Interoperability Forum. The company was particularly active in the area of SONET conformance. Furthermore, as of this date, Telcordia had completed GR-253 testing against the Cerent 454 (Release 1.0) and issued a report. 4F-BLSR functionality, when released, would conform to GR1230 and the DS3 boards already conformed to GR499.
 
John Colvin received some unofficial positive feedback on Cerent’s pricing information (Quotation No. JNC-071799-01) provided to AT&T on July 19th. He reported, “The pricing was said to have increased the overall interest in the Cerent 454 and caused further investigation by AT&T on whether the product was real or ‘vaporware.’ Overall, [today’s] meeting should be viewed as extremely successful and will generate further discussions, meetings and additional activities that will need to be resourced in order for Cerent to take it to the next step.”

Securing Resources and Executive Support
Certainly John was patient as he addressed each and every concern of the many AT&T stakeholders involved in choosing to switch vendors. Carl Russo, Cerent’s CEO, meanwhile, was being educated on the well-known multi-year sales cycle of engaging a Tier 1 service provider.
 
On August 4, 1999, both Jeff Jacques (Sales Manager) and Ralph Ramsey (Systems Engineer) were formally introduced by John Colvin to ALS in New York (Staten Island) as his expanded account management team. John was no longer a lone wolf and Cerent ramped up its customer-facing resources to attempt to secure AT&T orders. Jeff reported that the meeting went well and positive feedback was given in regard to the upstart startup’s plans for DWDM on the Cerent 454 and the new Cerent 327 product, the smaller version of the Cerent 454 targeted for customer premises.
 
This meeting foreshadowed the need for a scalable management system for deployed Cerent 454s. There was also an application brought up by AT&T for Cerent “to look at in regard to the need to aggregate telemetry at the co-locates nationally back to a central location. This would involve supporting a thousand plus IP network elements spread across a couple of hundred co-locates.” Later, the Cerent CTC would become the Cisco Transport Manager (CTM) to handle thousands of deployed network elements.
 
All of the work completed by John Colvin during 1999, positioned Cerent to enter the evaluation process at a number of AT&T laboratory locations. No one needed a crystal ball to see that Cerent and its mega-star product, the Cerent 454, would soon be part of the transformation of AT&T’s optical infrastructure, replacing Lucent as this Tier 1’s main supplier.
 
However, a crystal ball was needed to see that Cisco would swoop in three weeks later and acquire Cerent, much to the relief of AT&T executives who preferred to deal with vendors that were not startups. The courtship intensified as John Colvin left working on the AT&T account and Doug Juers assumed the mantle to make the sales breakthrough on behalf of Cisco.
 
Notes:
[1] Ken Duell stuck with AT&T for his entire career to date. He migrated through the organization, from the role as a Member of Technical Staff through 2001, well after the Cerent 454 evaluations were completed, to a Director through 2014, to an Assistant Vice-President today.
His colleague, Tom Afferton, reported in 2012 that he joined the OIF Board of Directors, a consistent path with that of his early contributions to the transport evolution of the AT&T network. OIF was launched in April of 1998, just as Cerent was forming from the splinter of Fiberlane. OIF’s objective remains to foster the development of a low-cost and scalable Internet using optical technologies. Tom presented on behalf of the OIF, “It’s the only industry group bringing together professionals from the data and optical communities. It’s an open forum with 250+ member companies, including international players, carriers, component and systems vendors, and testing and software companies.” It’s Mission is to foster the development and deployment of interoperable products and services for data switching and routing using optical networking technologies. From the Cerent perspective, one can argue that the OIF continues the work Cerent-Cisco wrought with their data interface solutions on optical platforms.
[2] Troy Adams played a more direct role then his colleagues. He assessed the Cerent 454 for two pressing and specific applications across the AT&T network. Up until that point though, he developed a network architecture to support AT&T's local service market entry, culminating with a move to local network transport planning once the Telecom Act of 1996 was signed. Next, due to this regulatory relief, Troy led a team to establish AT&T's first entry into local consumer telephony markets. Then, in 1998, Troy joined the new product introduction team to find product solutions for this new market paradigm to enable large carriers like his to deliver local telephony services. He and his team of 10 systems engineers evaluated new products. As Troy writes, “I defined the program for the Multi Services Platform (MSP), selected vendors, assessed alternatives, developed contracts and managed vendor implementation of this program in the AT&T Transport Network. I reduced costs for these elements by more than 70 percent through the introduction of these technologies and features [and I reduced the] overall network budget in this area by over $50M.” One of these vendors was Cerent, and the widescale deployment of the Cerent 454 MSPP (as the Cisco ONS 15454) validates the startups’ claims in how great the CAPEX savings could be. Today, Troy works for Ciena.

​I interviewed Chick Petersen in March 2014 as part of my due diligence for the Cerent book project.
 
He recounts that he was schooled early on at Digital Telephone Systems (DTS), a company Donald Green led in the early 1980s, after arriving in California from the cornfields of Iowa. Chick nurtured his interest in electronics before that, as a youth, while working as a stereo store employee.
 
Chick arrived in the Golden State with a rock band as their soundman. The band was on its way to Los Angeles, but the “motley crew” ended up in San Francisco. They shared their lives in a school bus occupying a parking lot near Pier 39. The lead guitar player had “girl problems” back home so he split for the Hawkeye state. Chick assumed the role as lead guitar player, which he tackled with relish. The band soon found a house in Novato to live and quickly realized they needed day jobs to pay the rent. Ah, the life of a musician.

Chick never received a formal engineering degree. However, he loved to tinker with electronics; so he worked on guitar amplifiers and then found employment in the audio-visual department at a local school. Chick ultimately joined DTS on January 3, 1978, just after the company was bought by Farinon. He started there after completing a competence test as a technician. He fell in love with the job. Initially, Chick was part of the PROM-zapping department, even before PROMs were made reusable as EPROMs, which used ultra-violet (UV) light to erase old programs. DTS’s PROMS held programs in support of DTS’s big-selling product, the D1200 [1].
 
“The D1200 was an amazing PBX,” Chick recalls. It was built out of discrete components and used a homebrewed software language. Earlier PBXs even used paper tape for their programming, well before PROMs arrived on the scene. Chick developed close ties with engineers on the product floor, and after the marketing department wooed him to join their organization, engineering hired Chick away from the manufacturing side of the company. While in his early engineering phase, Chick debugged compilers and then got to write software code on the original D1200. This code was a mix of machine and assembly languages based on Intel 8085 processors.
 
Donald Green believed in-house manufacturing was the way to go. DTS implemented advanced wave-soldering machines and model shops to produce its products. Don even tried to establish his own manufacturing lines at AFC when he later led that company, but this foray was short-lived. Cost pressures to outsource manufacturing likely ended the vertical company models in the late 1990s, especially in telecom. Mike Hatfield closely watched this development in his role at AFC and ultimately adopted a variation of this model of outsourced manufacturing at Cerent.
 
Chick spent 20 years with DTS before joining Cerent on August 12, 1998. (He worked with John Henel at DTS, who later become a Cerent alumnus too.)
 
At DTS, Chick’s career growth saw him evolve from an individual contributor to an engineering lead and then to a supervisor. He felt he was part of a team that acted like a small company, although working in a larger company.
 
Times got tough once DTS prematurely announced its “2020” product. It wasn’t ready for primetime, but orders came in for the new product and this quickly dried up orders for the older D1200 PBX. Revenue dropped. A “bean counter” (controller) was brought in to run the larger Harris operation in California [2], once it acquired DTS-Farinon. 

Going to work was no longer fun. It was time to search for a better work environment, so Chick set off on that quest. By then, DSC had already acquired Optilink, a Donald Green startup. Paul Elliott and Keith Neuendorf swayed Chick to join Cerent instead of toiling at DTS. Chick had previously interviewed at Next Level and while he was mulling that opportunity over, the Cerent one came to his attention. He rejected the Next Level offer and selected Cerent instead.
 
Chick recalls his team interview with Cerent in July 1998. “During the interview process, Ajaib Bhadare initially blew me off, but he brought me a beer. That was pretty cool.” After all, it was Friday afternoon, and even Chick knew it was keg day, which featured Lagunitas brews at Cerent’s main facility. 

Chick’s interviewers that day were Dave Hillard, Paul Elliott, Patrick Bisson (who would become his boss for a short while), and then Ajaib, who finally returned to conduct his portion of the interview [4]. Ron Ostrowski barged in to chat during his interview process, even though he didn’t know Chick. During the Hillard portion of the interview, Chick recalls, “Dave was constantly interrupted since other engineers needed his software build ASAP.”
 
Chick contrasts the Cerent workplace with that of Next Level’s surroundings during his selection process, “Cerent sported commando digs and labs scattered in cubicles while Next Level had amazing [facilities] including offices with doors for its engineering team. I chose the commando digs because the energy level seemed to be so much higher at Cerent.”
 
Chick’s first project was a reworking of the fan try and its alarms for the integrated LCD display. This led to work on the AIC module that many of the telco customers were clamoring for. 

​Chick eventually worked for Humphrey Chin in early 1999, once Humphrey became a supervisor in the software department, “likely due to his DS1 project success,” according to Chick.
​
 
Chick’s key learnings at Cerent can be encapsulated in two bullets:

1. Avoid use of homegrown software code – The DTS development environment seemed antiquated compared to the Cerent experience. At Cerent, a whole new world came about because of the VxWorks operating system versus the use of an Intel development environment where most of the code was homegrown.
2. Maintain a strong focus on what you’re doing. “This is what we need to do and then we went and did it.” Chick adds, “We executed to each and every plan.” 

 
Chick’s view of Cerent’s legacy in the optical transport arena can also be summarized with two bullet points:

1. “Yes, the industry needed another SONET ADM. The fact that it was evolutionary versus revolutionary carried the day.” This is akin to, Apple saying years ago that, yes, we need another mobile phone, one that is user friendly.
2. “Cerent became the #1 OC-48 platform as well as being the first to incorporate data traffic into a SONET multiplexer.” Indeed, Cerent would redefine the entire optical transport space to focus not on optical bit rates, but on multi-service provisioning platforms (MSPPs) that could scale with the bit rate required for the application. By 2006, more than 50 percent of the optical business would use MSPPs.

​As a footnote to Cerent’s Friday afternoon beer socializing, Chick assumed the role of lead guitar player for those end-of-day jam sessions that Paul Elliott instituted as early as the Fiberlane days in 1997. Chick played his 1963 Fender Stratocaster, which he bought for $750 during his early rocker days. 

When we did our interview in 2014, his axe was now worth some $8,000, a great return on investment. Chick had guitarist Steve Cropper sign his Stratocaster too, a musician that Rolling Stone lists as 36th on their list of the 100 Greatest Guitarists of All Time. Mojo ranks Cropper, “The Colonel,” as the second-best guitarist ever.
 
Paul Elliott played bass on Fridays. Keith Neuendorf donated the drums and he and many others took turns playing them to close out the work week. Keith also played guitar and the piano. The engineers in Petaluma contributed money to buy an old Wurlitzer piano too. Often guest musicians came over from other companies in Telecom Valley, such as AFC. Kegs were ever present on Fridays, plus chips and salsa on many occasions. 

​Grant Moulton headed up the OC-192 hardware team for Cerent after being hired by Hui Liu in late 1998. With Dyke Shaffer as his mentor, Grant quickly readied an OC-192 prototype transceiver in early 1999. However, the optical engineers had to wait for two ASICs – the BTC192 and SXC192 – to arrive to allow them to verify module functionality. Martin Roberts and Phu Le, an unparalleled wizard of hardware coding in the digital realm, were busy writing code for these two ASICs. 
 
Even though some within the engineering community believed no one would ever buy OC-192 capacity for the metro, such misplaced sentiments were part of the minority of doubters. Marketing projections showed the next bandwidth bottleneck would soon occur at OC-48 rates. On top of that, OC-192 capacity needs were beginning to appear in large metropolitan centers around the United States. 

​Industry analysts, including Ryan Hankin Kent (r.h.k.) at its STARTRAX ’98 conference, showed a graph of projected SONET spending and it revealed that while capital spending on OC-3, OC-12, and OC-48 speeds would hold steady, OC-192 investment would experience average annual increases of 31 percent for the years ahead. Lucent missed the boat on this capability while Nortel cleaned up in the OC-192 segment, especially for the long-haul market. For Cerent, OC-192 orders for metro applications flooded in once this high bit rate feature on the company’s ‘454’ was announced.

​But all was not rosy with Cerent’s (now Cisco’s) OC-192 development. The first hint of technical trouble occurred in January 2000 when bits coming from Phu’s OC-192 ASIC – the SXC192 – were “swizzled.” Bits were permuted or inadvertently rearranged in the data stream as the ASIC processed them. While mixing cream in your coffee is permissible, mixing up digital ones and zeroes is not. Meanwhile, Martin Roberts made the trek from England to Petaluma, and verified that the BTC192 ASIC performed correctly.

​The schedule for the feature introduction allowed some wiggle room to fix this anomaly, unless, a re-spin of the SXC-192 ASIC was required. Investigations into the movement of bad bits commenced even as Cisco’s operations team balked at making this complex OC-192 board. 

“This is too complicated for manufacturing,” Kevin Smith, Cisco’s operations leader, declared at the conclusion of a Feb 10, 2000 meeting held between engineering and manufacturing to discuss how the OC-192 board would be built. This contentious session marked the last time the Cerent 454 engineering team dealt with this group in San Jose. Carl Russo swooped in and became the OC-192 development team’s “concrete umbrella.” He shielded Grant, Hui, Phu, and other engineers in Petaluma from the Cisco naysayers. Carl trusted his engineering team. The vision was set: OC-192 was needed by customers, and so Carl had Tom Fallon assigned to manage the relationship with the Cisco operations folks in San Jose, with the objective of making 10G operation on the ‘454’ a reality.
 
Little progress was made by March 2000. Only a single useful frame of data emerged from the SXC192 ASIC. A band-aid was needed to salvage the impending roll-out disaster – a missed product introduction of the biggest ‘454’ capability to date.
 
Things went from bad to worse in July 2000 as the sixth revision of the OC-192 boards using an assortment of ASIC-related band-aids failed. Pressure mounted as pending customer orders worth more than $100 million had to be filled, with QWEST leading the growing backlog for 10G capability. Customers were also clamoring for the Cisco (Cerent) solution that would give them an alternative to Nortel as a 10 Gbps supplier.
 
By September 2000, signal loopback on the OC-192 board was achieved, but connections through the ‘454’ backplane from the BTC192 to the SXC192 failed. This meant that traffic could not move into or out of the ‘454’ chassis.
 
What was going on?
 
Phu argued the ASICs were performing.
 
The optical engineers countered, disagreeing with Phu and the ASIC designers.
 
Both groups within engineering were culpable for the stalemate. A lack of communication on two fronts produced a string of failures: physical connectivity of the 10G bandwidth was non-existent and a lack of collaboration between the ASIC and hardware teams failed to define the system level design of the OC-192 feature.
 
First, to get to the root of the problem, Phu was challenged over the algorithm he chose for the ASIC design. Decisions as to whether a digital bit was a “1” or a “0” were made at the edge of the data stream’s eye pattern, not at the center of the eye, as is typically done in transmission systems. This problem, as viewed by the hardware engineers, was not discovered until Phu was compelled to explain to them how he had designed the ASICs. This misstep would have been caught if an early architectural review of the 10G feature had been conducted at the outset.
 
Blind faith was granted to Phu by Hui and consequently, earlier attempts at sharing ASIC design methodology were spurned. After all, ASIC simulation “rules,” so the ASIC designers believed. This thinking was, in part, like the previous mindset held by Fiberlane’s early ASIC designers – if simulation works, the ASIC will work. Designers are not infallible, however, especially when they are working on a systems-based product like the ‘454.’
 
By December 2000, Raghu Belur, one of Cerent’s OC-48 designers, and Martin Fornage, an experienced telecom engineer, were drafted to work full-time on the OC-192 problem. They supported the contention that the ASICs were problematic and a re-spin was needed, in spite of Phu’s protestations. On top of the internal engineering battles of the “Cerent” team, other technical problems plaguing the OC-192 board were solved one by one. A frustrating voltage stability problem was solved by taming the “evil regulator” and the excessive heat produced by the board was dealt with by employing a heat sink the size of the board itself while still keeping the cooling element within the width demands of a single shelf slot. 

​A major step forward occurred with the introduction of the BackPlane Interface ASIC (BPIA), a third (and new) ASIC placed between the BTC192 and the SXC192 ASICs. BPIA mitigated the noise of the other two chips and steadied the jitter of the high-speed data streams in order for an eye pattern to be characterized. Probe points were added to the latest iteration of the OC-192 board, a sign that prudent measurements replaced blind faith to validate the expectations of designers. On April 27, 2001, the first “awful” eye patterns were detected and by September 2001, the ultimate band-aid was applied. A second BPIA appeared and a second ‘454’ backplane was introduced to support the product’s 10G capability. OC-192 came to market almost two years after it had been announced. 

​The 18-month delay hurt Cisco’s credibility, cost the company more than $100 million in missed revenue, and levied over $22 million in costs for scrapped OC-192 boards. Regardless of the setbacks in cost and time, not a single ‘454’ customer was lost to a competitor. Customer faith in the product remained strong during 2001 and 2002, in spite of the dot.com bust and the telecom meltdown. 
 
Chalk it up to lessons learned; lessons made possible by the financial strength of Cisco to stick with Carl Russo, Tom Fallon, and its Cerent-acquired team members.
 
As a startup, this miscue on OC-192 would have likely sunk Cerent as a private company. Fortunately the Cisco cocoon allowed the Cerent 454 to support its OC-192 capability. Phu and Grant persevered through thick and thin, and ultimately, with supportive colleagues in Petaluma, adopted the BPIA solution. In terms of achieving the fastest time to market, Grant says, “It was the correct thing to do.”

The 10 Gbps capability quickly became a big hit for Cisco and it spurred on the optical development that made wavelength division multiplexing in the metropolitan network mainstream. 

​Cerent led the way in the synthesis of hardware and software to produce products of superior customer value. The demand for more cost-effective optical transport to accommodate Internet demand and rapid access to the World Wide Web allowed Cerent to create its first product from scratch. The year was 1998 when the Cerent 454 debuted.
 
A similar event – the synthesis of hardware and software – occurred in 2003, with Tesla, the premiere producer of electric vehicles for the automobile industry. Elon Musk, Tesla’s founder, took advantage of creating a car from scratch, away from Detroit, during a boom era of consumer technology. Like Cerent, Tesla wrote its own code and made its software and hardware the stars of the company.
 
Cerent introduced its Cerent Management System (CMS) to differentiate itself from entrenched telecom manufacturers while Tesla featured its massive touchscreen requiring simple finger swipes as opposed to hunting for and twiddling knobs or pushing buttons as employed by GM, Ford, and Chrysler.
 
Apple, the master of vertical integration, introduced its iPhone in 2007. The company transformed how callers used a cell phone by elegantly combining hardware and software. Its intuitive ease of use makes it trivial for two-year-old children to navigate. Yes, my 2-year-old granddaughter is adept at making phone calls, retrieving voicemail, and snapping photos on my iPhone.
 
Google followed Apple shortly thereafter, in 2008, with its first commercially available software-driven smartphone running Android software. It operated on HTC hardware, but this early development allowed Google to learn, gain market share, and then introduce its own line of phones-become-pocket-computers, in 2010. Tightly integrating software with various types of hardware allowed Google to claim a position of dominance in smartphone operating systems.
 
The rise of Apple and Google in the telephony market allowed the United States to reclaim its leadership position in telecommunications, replacing Nokia (Finland); RIM, the Blackberry company (Canada); and other European and Asian cell phone companies that had briefly led mobile innovation in the mid-2000s.

​In all cases – Cerent, Tesla, Apple, and Google – the tight coupling of hardware and software allowed for rapid market share growth and wealth creation in their respective market segments. Privately-held Cerent would produce wealth for many involved with the company and the telecom industry prior to 2002. Tesla’s high-tech approach would see the company valued as a high-tech company as opposed to an automobile maker. Apple became the world’s most valuable company by 2014 and Google laid claim to global dominance in operating systems with its Android software.
 
“Does software inform hardware or vice versa?” That’s the wrong question. The right question may be: “Is software tightly coupled with hardware?” If so, market leadership is possible and wealth creation follows. The last two decades prove the point and have contributed to American leadership in high-tech.