Introduction to Networking at UCAR

Networking capabilities fundamentally support UCAR's goals for the advancement of science, technology, and education. Research calculations on distributed supercomputers, data assimilation with analysis of multi-gigabyte datasets, distributed visualization, and remote collaboration of UCAR's scientific researchers all require substantial network capabilities, including sufficient bandwidth, reliability, integrity, and scalability. Additional networking objectives include maintainability, adaptability, accommodation of future growth, and cost-effectiveness.

Network Engineering Support and User Feedback

The principal UCAR forum for network-user feedback regarding central network services is the Network Coordination and Advisory Board (NCAB), which consists of appointed technical representatives from the various NCAR and UCAR programs. The purpose of NCAB is to advise NETS concerning central network planning, policy, expansion, and operational issues. NCAB reports to the Director of NCAR's Scientific Computing Division.

The current source of funding for central networking at UCAR is money obtained as part of the UCAR overhead budget in the form of a network "occupancy tax" levied on all UCAR programs based on the square footage of occupied space. With a FY1997 budget of about $1,500,000 and 14 personnel, NETS supported 1,136 UCAR employees with about 3,000 network-attached devices connected to 114 networks.

UCAR Networking Budget

Purchased services consist mainly of purchased hardware/software maintenance service contracts and purchased telecommunications bandwidth. Service contracts are costing more primarily because simple network-equipment maintained by NETS staff is being replaced with complex, intelligent packet-switches that require vendor maintenance contracts to obtain firmware and hardware upgrades and replacements. Telecommunications expenses have increased because more bandwidth has been needed for Commodity Internet access, home-access, and inter-site access.

Though these steep increases are potentially alarming and are one of the drivers for producing this report, the resulting benefits include substantially increased local, metro-area, and wide area networking capabilities that have been vital to the essential business and scientific data communication applications of UCAR.

UCAR Metro Area and Wide Area Networking Initiatives

BRAN is a strategic initiative in which UCAR has joined along with CU-Boulder, NIST, NOAA, and the City of Boulder to construct and operate a private fiber network to interconnect the key facilities of these institutions and access certain common-carrier telecommunications facilities in Boulder. UCAR's portion of the construction costs are estimated to be around $350,000, with capital recovery resulting from cost savings in other areas very conservatively estimated at around 8-10 years. Figure 7 shows a map of the proposed BRAN fiber routes.

The Westnet gigapop evolved from a Boulder-area consortium begun in June 1995 to bring to UCAR a set of communication links that allow participating institutions to interchange information among themselves and the vBNS, and more importantly, to share the costs of one or more high-bandwidth links to the Commodity Internet. Figure 8 illustrates the concept of the gigapop, that is, a situation where several institutions network together to a common location from which shared access to national and other networks is then effected. Figure 10 illustrates the conceptual MAN and WAN connections to the UCAR network, while Figure 11 depicts the actual MAN and WAN connections to the UCAR network.

Internet2 has finally borne fruit in the form of the Abilene network, which will be a national network for Internet2 university members that will be constructed on fiber loaned by Qwest from its new national fiber network. Abilene will initially be constructed at 2.4-Gbps (gigabits/second) speeds and transitioned to 9.9-Gbps speeds as quickly as practical, vastly improving networking to those 48 UCAR universities that are also currently Internet2 members. The cost for UCAR to connect to Abilene is expected to be about $169,000 per year. Figure 15 shows the U.S. Qwest fiber backbone. Figure 12 depicts the Abilene backbone network, while Figure 14 indicates the location of UCAID member institutions.

The vBNS is a very high-speed network provisioned and operated by MCI under the authority of the NSF. In addition to the NSF supercomputing centers, 42 NSF HPC (High Performance Connection) grant universities are currently connected to the vBNS. As of March 1998, 93 universities had been awarded NSF HPC grants, 40 of which are UCAR member universities. It is expected that as many as 130 HPC grant schools will be attached to the vBNS. Figure 16 illustrates the vBNS backbone, and Figure 17 indicates the vBNS-connected institutions.

To continue to effectively communicate with the rest of the world that isn't attached to the vBNS or Abilene, the Commodity Internet service bandwidth will have to be expanded by FY2000. This expansion will result in an additional estimated annual expense of $243,000 for UCAR over and above UCAR's current Commodity Internet expense.

UCAR Local Area Networking

Aggregate UCAR networking now requires about 40 times as much capacity as in the recent past, and will require about 400 times as much capacity in the near future. As a result of this increased demand, probably the biggest networking issue facing UCAR today is to provide ubiquitous, dedicated, full-duplex 100-Mbps (Fast Ethernet) host connections while keeping the backbone LANs and switch/routers capable of handling this increased traffic.

A mass conversion of UCAR computers to Fast Ethernet (100-Mbps) will be required within the next three years. However, the UCAR cabling infrastructure is inadequate to support this required conversion. About 40% of UCAR LAN facilities have been fitted with cabling capable of supporting future needs. The other 60% of the LAN cabling infrastructure is insufficient for future growth and needs to be upgraded as soon as practical.

Probably the most important strategic issue examined in this document is the completion of the LAN cabling upgrade. Strategies to design and implement a flexible LAN structure at UCAR are vital because their design and implementation represents the bulk of NETS's expenditures and efforts, and because the completion of the LAN cabling infrastructure is so vital to UCAR's networking future. The completion of the required cabling upgrades is expected to cost about $2,000,000; recabling needs to be finished by the end of FY2000.

Additionally, a systematic plan for upgrading and replacing network-equipment on an annual basis is recommended. By the end of FY1999, the value of all network-equipment will be about $2,000,000. With an estimated 3-year lifetime, one-third of this equipment should be renewed by upgrade or replacement each year starting in FY2000. Therefore, $670,000 will need to be expended for network-equipment renewal each year from FY2000 and beyond.

With an estimated 10-year lifetime, the completed LAN cable plant should, likewise, be renewed every 10 years. Unfortunately, renewal of one-tenth of the cable plant on an annual basis is not effective because the entire upgraded plant would be no more than 4 years old when annual renewal would start, and such annual renewal would result in parts of the plant being replaced when it was only, say, 5 years old, while other parts would be 15 years old by their time of replacement.

Furthermore, 10 years is too long to predict what will happen with LAN cabling technology and when it will happen anyway, so it's difficult to recommend a specific replacement schedule. Instead, it is simply recommended that the cable plant be replaced when it is apparent it needs replacing.

For the purposes of developing sample budgets, however, an example renewal strategy is to replace the entire cable plant during 4-year periods of construction separated by 6-year periods of only maintenance activity. With an estimated total replacement value of $3,420,000, then for every 10 years, $855,000 would be expended for each of 4 consecutive years, with $50,000 being devoted annually to maintenance during the intervening 6 years.

Possible UCAR Networking Budget Trends

Two network industry research groups in 1996 placed the annual cost of network support at $3,275 to $8,000 per desktop. Central UCAR network support cost will be $781 per network-device in FY1999. Even with increased costs, by almost any measure, networking support at UCAR is a bargain compared to industry estimates.

Conclusion to Executive Summary

Increased expenditures nonetheless, UCAR should move forward with a state-of-the-art networking foundation to support its 21st century information technology requirements, and thereby assure that it can continue its scientific and community leadership roles in a world in which data access and dissemination capability is increasingly fundamental to an institution's intrinsic community value.

To these ends, the following five strategies regarding UCAR networking have been identified as being the most critical for meeting the required objectives:

1. Assure that the network cabling for all of UCAR is capable of supporting switched Fast Ethernet (100-Mbps) to all UCAR workspaces by the end of FY2000.
2. Before the end of FY2001, transition from Ethernet (10-Mbps) to switched Fast Ethernet (100-Mbps) as the standard network service available to all UCAR computers.
3. Renew one-third of all UCAR network-equipment each year by replacement or by upgrade.
4. Renew the UCAR network LAN cabling plant every 10 years by replacement or by upgrade when it is apparent that the cabling plant needs to be upgraded.
5. Continue to actively participate in metropolitan, regional, national, and international networking initiatives.

1. INTRODUCTION

The networking capabilities and strategies presented in this document fundamentally support UCAR's goals for the advancement of science, technology, and education. Research calculations on distributed supercomputers, data assimilation with analysis of multi-gigabyte datasets, distributed visualization, and remote collaboration of UCAR's scientific researchers all require substantial network capabilities, including sufficient bandwidth, reliability, integrity, and scalability.

The strategic networking plan contained herein was originally formulated because UCAR management was concerned about rapidly escalating networking expenditures occurring without reference to a strategic context. The network strategies which follow are therefore intended as an advisory, management, and maintenance resource for making institutional decisions regarding networking.

Most of the ideas and strategies in this plan have previously existed, though they have never been articulated together in a fully self-contained document. The document itself has been written for a broad audience, and therefore contains various sections that some readers might wish to skip. The major sections are briefly described below to aid the reader in making reading selections.

The context for UCAR networking strategy is established in "2. NETWORKING AS AN ESSENTIAL ENABLING TECHNOLOGY", "3. THE OBJECTIVES OF NETWORKING", and "4. IMPORTANT BOUNDARY CONDITIONS OF NETWORKING".

Past budget trends are analyzed in "5. AVAILABLE RESOURCES: THE BUDGET", while future budget trends are presented in "11. ANALYSIS OF FUTURE BUDGETS". These two sections can be read without reference to the rest of the document if the main interest is in raw budget numbers.

UCAR's metro-area and wide area network initiatives are presented in "8. MAN AND WAN ISSUES AND STRATEGIES", while local area networks are extensively discussed in "9. STRATEGIES TO DESIGN AND IMPLEMENT THE FLEXIBLE LAN" and "10. BUILDING A LAN FOUNDATION FOR THE FUTURE". These three sections contain most of the technical networking information, and are important if one wants to know how the technical issues translate into strategic capability.

The "softer" side of networking is dealt with in "6. OPERATIONAL STRATEGIES" and "7. PRODUCTIVITY STRATEGIES", where the application of human resources to networking support is explained.

Note that while this document discusses a variety of strategic issues and strategies related to networking at UCAR, probably the most important strategic issue examined is the completion of the LAN (Local Area Network) infrastructure upgrade. A special emphasis has therefore been placed on this issue throughout this document.

Finally, note that throughout the remaining portions of this document, data communications networking will be referred to simply as networking, while UCAR will often be referred to as the organization or the institution. Note also that the term "Division" is used to refer to any subpart of the National Center for Atmospheric Research (NCAR) and the UCAR Office of Projects (UOP). The discussion of networking within this document is generally with respect to the specific context of UCAR, even though such is usually not explicitly stated hereafter.

2. NETWORKING AS AN ESSENTIAL ENABLING TECHNOLOGY

2.1 Why Networking Is An Essential Enabling Technology

In fact, networking is an essential enabling technology, meaning that one can no longer carry on without it because the intrinsically useful applications enabled by it are themselves deemed indispensable, and therefore essential. Right now, for many people at UCAR, the network is more important than the telephone. When a recent network hardware component failure seriously disrupted network service for most employees of NCAR's Scientific Computer Division, many employees simply gave up trying to get anything done and went home out of frustration after a couple hours. A similar reaction would have been unlikely had the Divisional telephones failed instead.

That networking is perhaps more essential than the telephone is actually a startling thought. It indicates that the nature of UCAR's work is shifting; that UCAR employees are becoming "knowledge workers". So if the network is essential, then it must enable essential applications. What are some of these essential applications?

One of the most important of these is e-mail. National Public Radio recently reported that more pieces of e-mail were sent in the U.S. in 1997 than pieces of U.S. Postal Service mail. Worldwide networked e-mail has rendered the telephone nearly obsolete for many business purposes. In fact, so many people are so busy these days, that it's very common to have to use e-mail to schedule a telephone conversation. For the purpose of simple information exchange that doesn't require significant interaction between communicating parties, e-mail is superior to the telephone because e-mail conforms to ones schedule, while the telephone generally requires a conformance to its schedule. E-mail makes time fluid and its data is also both easily archivable and transformable.

The World Wide Web (the Web) is another essential network application that is increasingly rendering the telephone obsolete. In fact, the Web is quickly overtaking even the ubiquitous fax machine, which is a stopgap marriage of the telephone and printer that will be rapidly supplanted by the marriage of the Web browser and printer. It's believed that within three years, almost all business now conducted by fax will be conducted via the Web, and likewise, almost all query/response business telephone conversations will be supplanted by customer-initiated Web interaction. Web access will be cheaper, more accurate, more informative, more timely, more secure, and more satisfying than calling someone to ask questions.

Because email, the Web, and other applications are deemed essential, the networking that makes them possible must also be deemed essential.

2.2 Why Networking Is Infrastructure

This surface comparison might foster the belief that networking can be treated, from a management and planning perspective, in the same straightforward and simple fashion that power distribution is treated. From a strategic standpoint though, networking is substantially different than electrical power distribution, as illustrated in the next section.

2.3 Why Networking Is Strategic

On the other hand, no simple formula yet exists for the design, implementation, operation, and utilization of networks. To achieve success with networking requires the prediction of an unknown future of changing probabilities in which the values of the variables are not known, and even many of the variables themselves are not known. Strategic planning is necessary because no simpler method that works is available.

2.4 Summary

3. THE OBJECTIVES OF NETWORKING

At UCAR, the essential network applications divide into two distinct types: scientific applications fundamental to the core mission of UCAR, and business applications that enable the institution itself to function. In some cases these two types of applications have different networking needs, but in most cases the needs are the same.

Both types of applications require universal availability of networking services throughout the institution. Both types require a robust, reliable, and dependable network, and both types require timely responses from the network. And finally, both types require the aforementioned qualities with respect to access to the world of networking outside of UCAR. Perhaps the main difference in requirements is that scientific network applications generally require much more network capacity (bandwidth) than do business applications.

Accommodating future networking needs is also another critical objective. Networks must be built to be expandable, flexible, adaptable, and maintainable.

And finally, all of the above objectives must be qualified by the objective of cost-effectiveness. While the network must be effective, controlling costs by obtaining good value is always undertaken.

4. IMPORTANT BOUNDARY CONDITIONS OF NETWORKING

4.1 Introduction

4.2 Institutional Environment

In FY1997, UCAR employed approximately 1,136 people and expended about $142,000,000.

UCAR's own facilities are located in 9 buildings at 4 sites centered around Boulder County.

Approximately 3,000 end-user network-attached devices are connected to 114 UCAR networks, an average of about 2.6 such devices per employee.

Additionally (as one measure of external service to the university community), NCAR's Scientific Computing Division (SCD) reports that about 700 university users actively accessed SCD computing facilities in FY1997 in support of 345 projects located at 90 university facilities.

The UCAR budget for central networking support in FY1997 was about $1,500,000. This budget funded salaries for 14 permanent and casual network support personnel and funded networking equipment, materials, supplies, travel, and purchased services.

About 1.14% of the total FY1997 UCAR budget was therefore spent for central network support. Each network support employee was responsible for networking support for an average of 214 network-attached user-devices and an average of 81 UCAR employees. Additionally, these 14 employees were also responsible for ongoing major network construction projects and the networking support necessary for providing external access between UCAR and the rest of the Internet.

4.3 UCAR Networking Environment

4.3.1 Networking fundamentals: LANs, MANs, and WANs

4.3.1.1 LANs

LAN infrastructure consists of the existing fixed cabling infrastructure that reaches to all workspaces, as well as most of the network devices themselves, including routers, switches, repeaters, and so on. The LAN structure within UCAR currently consists of about 114 logical LAN networks, built from approximately 210 pieces of network-equipment, with approximately 3,000 network-attached user-devices.

The costs of building, maintaining, upgrading and operating the UCAR LAN structure are the most significant cost components of the networking budget. LAN expenditures for both labor and capital greatly exceed expenditures for MAN and WAN services. Practically all capital costs are LAN costs, which means almost all depreciation, interest, and vendor maintenance expenses are LAN expenses as well.

4.3.1.2 MANs

Also included in the MAN category are facilities that supply residential access to UCAR computing facilities and the Web. Currently, home-access is provided by centrally supported modem pools at NCAR connected to dialup circuits leased from US West.

MAN costs consist mainly of purchased telecommunications services, though home-access can have an important capital component as well.

4.3.1.3 WANs

These WAN networks are the principal means by which UCAR accesses external data resources, including access to and from UCAR facilities and the facilities of UCAR's member institutions.

WAN costs consist mostly of purchased telecommunications services.

4.3.2 Key LAN, MAN, and WAN Network Support Activities

1. Network hardware and software troubleshooting.

2. Network error and load monitoring.

3. Network performance tuning.

4. Network configuration updates.

5. Expanding networks to meet increased numbers of users and increased bandwidth requirements.

6. Reconstructing and reconfiguring networks to meet changing needs and standards.

7. Consulting with network users regarding performance problems with applications, hosts, and network connections.

8. Consulting with network users regarding optimal network usage.

9. Cabling-plant re-engineering and construction.

10. Evaluating new networking technologies and equipment.

11. Maintaining network documentation and databases.

12. Tracking and coordinating all network project activities via a work-request/work-tracking system.

13. Inventory management.

4.4 Network Policy Environment: Taxation with Representation

4.4.1 NCAB

NETS is the engineering group responsible for all centralized UCAR network support. NETS is also responsible for certain specialized SCD and national networking activities; however, these specialized activities lie outside of the scope of this document, and funding sources, personnel, and expenditures for these activities are not included in this document.

Within UCAR, NCAB is currently a unique combination of a UCAR-wide policy body that operates in tandem with a central UCAR engineering group that has the resources to fully implement policy in a consistent fashion across UCAR. NCAB acts in a highly visible and interdisciplinary manner, and is able to articulate the importance of networking to the mission of UCAR.

NETS and NCAB have cooperated to maintain and develop a long-term working relationship that transcends any particular issue or crisis. Such an established relationship produces a stronger response during a crisis, but equally as valuable is the resulting continuous communication and feedback. The formation of the NCAB and NETS combination has been indispensable to the rapid progress of networking at UCAR.

4.4.2 Network Service Categories

1. Standard services

2. Premium services

3. Special services

4. Divisional Services

The main purpose for these service categories is to define which services are funded by the UCAR occupancy tax, which services must be funded through chargebacks by NETS, and which services must be directly funded by the Divisions themselves. These service categories are therefore fundamental to the nature of centrally funded and centrally provisioned network service at UCAR, and it's important to understand the engineering and administrative implications of these categories.

4.4.2.1 Standard Services

The cabling component specifies that one standard telecommunications outlet (TO) be provisioned for each workspace. Each standard TO connects four Cat5 copper cables, two Cat3 copper cables, and two pairs of multi-mode fiber optic cable to the nearest telecommunications closet (TC). Only 40% of UCAR facilities currently meet this definition of standard service.

The bandwidth-delivery component specifies that 10-Mbps Ethernet be delivered to each workspace, but does not currently indicate whether such bandwidth is shared among multiple computers, or whether each computer is delivered a dedicated 10-Mbps via a dedicated packet-switch port. However, recent engineering practice has in fact been to deliver dedicated bandwidth, and it is expected that by the end of FY1999 such dedicated bandwidth-delivery will have been provided to almost all UCAR computers.

Except for certain special circumstances, Fast Ethernet (100-Mbps) is not yet considered to be a standard service, though sometime during the next three years Fast Ethernet will be required as a standard service for all computers because Fast Ethernet will become the de facto standard NIC (Network Interface Card) on workstations and PCs. However, without major cabling upgrades, the ubiquitous provision of Fast Ethernet will be problematic because 60% of the institution can not support Fast Ethernet without some type of ad hoc stopgap cabling and network-equipment solution.

The current definition of standard service also states that NETS's official responsibility for networking ends at the workspace TO. This means that the Divisions are responsible for anything past the TO. Usually this simply implies Divisional responsibility for the end-user equipment attached to the network at the TO. However, it is possible for a Division to attach their own network-equipment to a TO, in which case such a Division may become responsible for a "private" portion of the network.

There are some situations when NETS's responsibilities extend beyond the TO because of complex host-based network configuration issues. For example, there are a variety of configuration issues in which the host's connection to a network-switch requires configuration parameters in the host to be consistent with configuration parameters of the connection's network-switch port.

4.4.2.2 Premium Services

Note that new premium service types will be added as technology evolves. For instance, the availability of both Gigabit Ethernet (1000-Mbps) and ATM OC-12 (622-Mbps) are imminent.

4.4.2.3 Special Services

Another special service category includes "emergency" services. The standard service definition states that NETS has up to a week from the time of request-notification to perform the requested service. If a service is required sooner than that, then it is considered to be a special service, even if the service ordinarily would have been considered to be a standard service.

Chargebacks and support levels for special services are negotiated on a case-by-case basis.

4.4.2.4 Divisional Services

Divisions require the services of their own systems administrators who can optimally configure and operate Divisional systems in a complex networking environment. Even if the network is properly engineered and operated, optimal performance of end-user systems requires optimal internal system configurations, and such configurations can be difficult to achieve.

Interaction with NETS regarding a wide variety of complex and sophisticated network-related issues is aided when Divisional engineers are network-savvy. Such interaction takes place directly between NETS and the Divisions, as well as through Divisional representation on NCAB. The technical expertise of Divisional engineers is vital to NCAB's advisory role.

Divisions are also free to augment their portion of the central networks at their own expense and even build their own networks at their own expense if they so choose. However, unless a Division has worked closely with NETS to assure that such private construction meets NETS's support standards, NETS may be unable to deliver operational support, and the Division may have to undertake this support at their own expense. Divisions are sometimes under pressure to supply quick and low-budget network augmentation when UCAR-funded resources are insufficient. While such augmentation can be beneficial to individual users in the short term, too many instances of uncoordinated construction can be quite disruptive to the long-term needs and goals of the institution as a whole.

5. AVAILABLE RESOURCES: THE BUDGET

5.1 Where The Money Comes From

1. As part of the UCAR overhead budget in the form of a network "occupancy tax" levied on all UCAR programs based on the square footage of occupied space.

2. Chargebacks to Divisions when requested services are outside the standard services funded from the UCAR overhead budget.

The occupancy tax supplies by far the largest portion of funding, and in 1998 provided slightly over $2,000,000. All budget discussions in this document will consider only occupancy tax funds because chargebacks produce very little funding, and do so in an unpredictable manner.

Direct funding by UCAR has given central networking higher visibility within the organization, and the consistent funding of a central support group has been essential for providing quality networking to the individual NCAR/UCAR programs.

5.2 Where the Money Goes

1. Purchase networking materials and supplies.

2. Pay for network-equipment via depreciation and interest charges.

3. Purchase telecommunications, hardware maintenance, and other services.

4. Pay for the salaries of UCAR's networking personnel.

5. Pay for training and travel needs.

Table 1 shows the networking budgets for FY1994 - FY1999, and Table 2 shows staffing trends since FY1995. Budgets include only money budgeted from the UCAR occupancy tax as other amounts are trivial and unpredictable.

5.2.1 Budget Explanation

The "Depreciation" expenditure is an accounting artifice that mimics the accounting standards normally used by for-profit businesses to account for the effects of capital investments on tax liability and on financial calculations that show net asset value and net earnings. For networking at UCAR, "capital investment" expenditures occur whenever network-equipment or fixed network cabling is installed that exceeds $5,000 in cost. A commercial loan is obtained to pay for the full expense of capital equipment during the first year, and this loan is then repaid over a period of 4 fiscal years for network-equipment and 11 fiscal years for fixed cabling. (Half payments are made during the first and last fiscal years, so depreciation actually spans only 36 months and 120 months respectively.) The payment of principal on these loans is designated "Depreciation." The "Interest" expenditure is the interest on these loans.

5.2.2 Budget Trend Analysis

5.2.2.1 General Trends

5.2.2.2 Staffing Trends

For instance, the U.S. Department of Commerce's recently released report, "America's New Deficit: The Shortage of Information Technology Workers", explores America's potential for a shortage of information technology workers in the United States and its critical impact on the nation's economy. Quoting from a description of this report:

Although a leader in the Information Age, it seems evident that America may soon lack a supply of qualified "core" IT workers, e.g., computer scientists and engineers, systems analysts, and computer programmers. Since the shortage of IT workers is becoming a global problem, U.S. employers will face tough competition to get and keep highly skilled IT employees. As a result of newly created jobs and other vacancies, it is estimated that the United States will require more than one million IT core workers by the year 2005! This shortage in IT workers will affect not only computer and software industries but also manufacturing and services, transportation, health care, education, and government. If not reversed, these shortages could undermine the U.S.'s economic performance.

On a more local basis, the Rocky Mountain News reported on 5/25/98 that the demand for technology workers in Boulder County is expected to increase by at least 17% before the end of 1999 according to a survey by the Boulder Economic Council. This does not even include the expected demand from Iomega, Sun Microsystems, Level 3 and other new high-tech companies moving to the area.

Finally, there is no guarantee that the productivity gains of UCAR networking staff in recent years can be continued. Therefore, strategies that produce productivity gains and that help attract and and retain key networking professionals will be important cost mitigation strategies.

5.2.2.3 Purchased Services Trends

5.2.2.3.1 Maintenance Services Trends

Maintenance costs have been climbing rapidly, and will continue to climb, primarily because simple network-equipment maintained by UCAR staff is being replaced with complex, intelligent packet-switches that require vendor maintenance contracts to obtain firmware and hardware upgrades and replacements.

Two strategies can be applied to try to minimize maintenance expenses. One is to use as few vendors as possible and negotiate bulk discounts with those vendors. The other is to select the level of maintenance service appropriate to the institution's need for service uptime.

5.2.2.3.2 Telecommunications Services Trends

A major factor that will affect this budget item is the increase in demand for new telecommunications services and facilities. Three such demands are currently known. One is the increasing demand to access the Web and UCAR facilities from employees' homes. Another is the need to join new national network initiatives such as Internet2's new national Abilene network for universities. A third is the need to increase Commodity Internet bandwidth. These issues and purchased telecommunications strategies in general will be discussed in more detail in later sections of this document.

On the plus side, rapid changes have recently begun to occur within the U.S. common-carrier telecommunications industry. Competition is heating up, and alternatives are becoming available. It's possible that wide-area bandwidth will be available in the future for decreased unit pricing.

5.2.2.4 Depreciation and Interest Trends

    .5(FY1995 equipment cost)/3 +
      (FY1996 equipment cost)/3 +
      (FY1997 equipment cost)/3 +
    .5(FY1998 equipment cost)/3 +

FY1998 and FY1999, then, are the only two comparable data points, and while it's hard to draw much trend analysis from two data points, the increase from FY1998 to FY1999 is significant. Furthermore, as will be discussed in more detail later, unless capital expenditures are increased significantly to quickly complete the upgrade of the fixed cable plant, serious network performance problems are going to result.

Another new factor is the need to fund network-equipment renewal on a regular basis. Finally, special projects can increase this budget item. For example, the BRAN (Boulder Research and Administrative Network) project is roughly estimated to cost $350,000 for UCAR's portion of construction expenses.

Given these factors, depreciation and interest expenses are likely to see substantial increases in the near term.

5.3 Budget Flexibility

5.3.1 The Need For Some Budget Flexibility

An example of an elective enhancement would be a network statistics system that would improve understanding of traffic patterns. The network is working fine without such a system, though it could probably be made to work even better with such a system.

An example of a non-elective enhancement is a router upgrade that is necessary because the router CPU load is approaching the router's operational maximum. If the router is not upgraded, then it will begin to discard an increasing number of packets that will seriously erode network function, and this will be clearly visible to users in the form of perceived delay in application function.

It should be noted that in both cases it would have been prudent to budget for these items. However, three things can prevent this from happening. First, the necessary item might not have existed at the time the budget was set. Second, there might not have been a need for the item at the time the budget was set, and it wasn't possible to anticipate the need. Finally, there may have been a need for the item and the item existed, but the budget request for the item was denied.

The first two matters illustrate difficulties of dealing with unanticipated networking needs. Many aspects of future networking needs are hard to anticipate, particularly if sufficient tools and personnel cannot be devoted to the task because of previous budget considerations. The result is that often the best thing that can be done is to look at gross historical trends. For example, based on the need to upgrade routers in the past, one can expect that router upgrades will be necessary in the future, though exactly when and how much is hard to determine. Routers are just one example out of dozens of similar issues. As a result of the difficulty in making exact growth extrapolations, usually some amount of money based on growth trends is requested, and it's hoped that this will be enough.

5.3.2 What Happens When Needs Exceed Budgets

The biggest danger is if a major predictable elective item suddenly becomes non-elective, but the budget isn't increased to accommodate the non-elective expense. This is the case with facilities re-cabling, which recently has moved into the non-elective category. The importance of recabling is discussed in more detail later, but briefly, two factors have contributed to the urgency of this situation, both of which relate to the need for increased bandwidth to the desktop computer. The first factor is that the older cabling requires that multiple desktop computers compete with each other for a finite bandwidth on a shared-media cable. The second factor is that the old cable is of such a type that the bandwidth cannot be increased, even if the shared-media problem could be resolved.

The solution for this situation is to deliver dedicated bandwidth for each desktop, and to optionally increase the dedicated bandwidth-magnitude of specific connections as well. Unfortunately, increasing bandwidth magnitude, and in some cases providing dedicated bandwidth at any speed, is not possible without cable upgrades. More and more desktop computers are reaching the point where the bandwidth demand outstrips the cabling capability, and the only way to meet that demand is to replace the cabling.

Ultimately the most important issue regarding the network budget is whether the budgeted amount can fund sufficient services to meet legitimate demands. If the budget can't grow to meet such demands, then the only alternative is to supply a service that is known to be insufficient.

6. OPERATIONAL STRATEGIES

1. Selection Of Economical Maintenance

2. Management of Spares and Materials Inventory

3. Dealing With The Network Marketplace

4. Staffing Strategies

5. Security Issues

6.1 Selection Of Economical Maintenance

The simplest strategy might be to just purchase the highest possible maintenance service level for all network components; however, this approach isn't feasible since the highest possible maintenance service level can be made arbitrarily expensive by the vendor. Furthermore, such high maintenance expenditures are not economical, since after a certain level of response, each additional incremental expenditure buys a smaller and smaller service improvement. An economic decision must be made that weighs maintenance costs against the needed service level.

The compromise that NETS has chosen is to stock our own set of spare parts on-site and perform our own diagnosis and repair, while purchasing vendor maintenance that furnishes 24-hour parts replacement. NETS also chooses maintenance plans that include embedded-software upgrades and telephone support in addition to parts replacement. Because embedded-software upgrades and fixes are so frequent, so expensive, and so vital to network-equipment operation, maintenance that covers these expenses is critical. Essentially, parts replacement and software coverage is the least expensive vendor maintenance option that allows a network to remain operational.

More expensive maintenance options pay for on-site vendor maintenance personnel to perform problem diagnosis and repair as well as instant parts replacement from local vendor stocks, but such options are several times more expensive than the "do-it-yourself" options chosen by NETS.

The downside of the less expensive maintenance services utilized by NETS is that the probability is higher that longer service outages may occur. However, these are calculated risks that are judged acceptable when compared with the higher costs of more expensive vendor maintenance plans.

6.2 Management of Spares and Materials Inventory

6.2.1 Spares

Spare parts inventory can be minimized by minimizing the number of vendors, by minimizing the number of different kinds of network-equipment used, and by keeping production equipment at the same vendor hardware and software maintenance release levels. Another strategy is to keep the older parts remaining from an upgrade as backup parts for the new, upgraded parts. This isn't always possible because of compatibility issues or because the old parts have to be traded for the new ones, but it's a great money saver when feasible.

When possible, NETS prefers to keep its spare parts "hot," that is, the parts are attached to the network and are monitored for proper functionality, but are not used for production purposes. Should such a part need to be used, it's felt that it is more likely to be functional than if it were to be pulled off a shelf in a storage room after setting there untouched and untested for months. Also, by keeping parts hot, it's much easier to keep them at the same vendor hardware and software maintenance release levels as production parts.

NETS's biggest problem with spare parts is the temptation to move them from "hot" status to production status without quickly replacing the spares because of lack of funding. When a serious network performance problem can be relieved by putting a spare part into production, its immediate replacement may not weigh too heavily. That this happens as often as it does raises the concern that the budget may be running too lean. So far no harm has been done, but sooner or later this kind of activity could result in a network outage of serious consequence.

6.2.2 Materials Inventory

About the only danger of ordering too much of some type of material or part is getting stuck with unusable inventory if the material or part becomes obsolete. Usually approaching obsolescence is obvious, and inventory draw-down can be done intelligently.

6.3 Dealing With The Network Marketplace

6.3.1 Welcome to the Used Car Lot

6.3.2 Let's Make A Deal

The important issues here are to remain cognizant of how the marketplace operates, and to maintain the skills necessary to exploit this knowledge as best as can be done within institutional limitations.

6.3.3 Knowing What To Buy

Marketplace education is a major component of the necessary general continuing education required by networking staff, and a commitment of time and money must be made to avoid marketplace ignorance.

6.3.4 Knowing When To Take Advantage of Special Deals

6.4 Staffing Strategies

6.4.1 Hiring Strategies

6.4.2 Competitive Salaries

6.4.3 Flexibility

Accordingly, it has been necessary to manifest an unprecedented flexibility to retain qualified employees. For example, it is now routine to consider part-time employment. One employee is a full-time commercial airline pilot, but he can also work as a .6 FTE network engineer. Another new employee has been given exceptional flexibility to finish an almost-completed Master's degree. While this much flexibility may be slightly more difficult to manage, it provides an economically competitive advantage for hiring and retaining employees who may have special lifestyle circumstances.

6.4.4 Continuous Training

It's also important to select staff that recognize that ongoing training is a constant, and it's equally important that management foster an environment that supports continuous training, as well as budget for the time and resources it takes to effect such continuous training. Many network engineers will say that it would be a career-threatening move to be removed from the world of networking for as little as six months. It does no good to hire a valuable network professional, and then watch that value go to zero due to a lack of ongoing training. Not only that, the best people will leave if they don't have proper training opportunities, leaving only those employees who don't value continuing education.

6.5 Security Issues

The major problem with security filtering is that as the filters become more complex, it takes more of a router's internal computing resources to implement the filters. These router resources are then unavailable for performing the data routing function itself. The net effect is that the more complex the filter, the more network performance can suffer. Thus there is a tradeoff between security and performance. Routers must be upgraded or replaced with more powerful routers if the desired level of security exceeds the existing routers' capabilities.

In fact, based on UCAR's Computer Security Advisory Committee (CSAC) security recommendations, it will be necessary to substantially augment the basic UCAR security filters already in place. Fortunately, it is expected that router upgrades already being planned will compensate for the increased demand on router resources in the near term. However, it is important to recognize that recommendations for ever-more-complex router filters will ultimately require a greater expenditure of network funds for ever-more-powerful routers.

An important ongoing responsibility of NETS will be to apprise CSAC of the network implications of possible security decisions, including network performance and budget impacts, as well as more general network functionality limitations that could result from a variety of security recommendations.

7. PRODUCTIVITY STRATEGIES

1. Excellence in Human Communication: Change Notification

2. Teamwork

3. The Adaptable Organization

4. The Pervasive Use of Tools

7.1 Excellence in Human Communication: Change Notification

A good example of the butterfly effect occurred in March 1998 when suddenly, for unknown reasons, four out of six NCAR Cray supercomputers started simultaneously crashing (halting). Ultimately, it was determined that a very minor and valid change made on a network router at UCAR's Foothills Lab triggered a cascade of several bugs in Cray code that resulted in the crashes.

A big part of the resulting problem was that diagnosis and resolution took a long time because there was not adequate human communication regarding the triggering event. While good communication is unlikely to have prevented the initial failures, it certainly would have accelerated diagnosis and resolution. Since it's so difficult to know which changes are significant, notification of all changes is necessary. However, simply notifying everyone of all changes is not a practical solution, since most people don't care about most changes, and because too many irrelevant notices soon cause people to start ignoring all notices.

Therefore, a judgment must be made regarding the scope of notification for each change. Essentially, a judgment is made regarding the probability of a disruption and how many network users are likely to be disrupted. If the probability is high and the effect broad, then there are many relevant parties to notify. For example, scheduled maintenance outages have a 100% probability of effect, and maintenance of, say, the gateway router to the Internet will potentially affect all UCAR users. In such cases, the widest notification is sent. Within this broad group are e-mail lists of Divisional system administrators and Divisional network administrators. The staff on these lists are responsible for issuing broader notifications to their users if, in their judgment, such broader notifications are called for.

The great majority of changes, however, are very small in effect and scope. Notification of these types of changes is usually confined to the NETS engineering staff. An example of such a change would be the addition of a new workstation to a network.

All change notices are logged to a common audit file, so that backtracking of changes can be made should some problem suddenly arise that looks like it was probably caused by an engineering change.

The second major notification dimension concerns the timeliness of a change notification relative to the expected magnitude of the disruption. A change such as the scheduled maintenance of the Internet gateway router, which could affect almost all users, needs a long lead time notification, say as long as a week. On the other hand, simply attaching a new workstation to a network is generally unnoticed and thus requires little advance notice; simply noting the change at the time it is made is usually sufficient. Once again, a judgment call must be made regarding the appropriate notification lead time for each change.

The most important factor in all of this is that a commitment must be made to human communication. The discipline required to frequently communicate must be reinforced at every turn, and change notification must become second nature.

7.2 Teamwork

7.2.1 Team Consensus

Achieving consensus means that common goals get developed and that more people accept and understand the common goals. During the implementation phase, these common goals help focus effort on productive activities and reduce the amount of time wasted on counter-productive activities because ambiguities will have been diminished.

When success is finally achieved, everyone will have had a hand in that success, and this will reinforce the value of teamwork for the next task.

7.2.2 Team Professionalism

7.2.3 Team Recognition

Recognition is easy and costs next to nothing to implement. It's even best not to have a formal program, which usually limits to whom and how often recognition can be given and can also sometimes generate resentment. It's best to just tell the members of a team when they've done a good job. Tell this to each person, e-mail each person, and/or tell the assembled team. Peer-initiated recognition with token awards such as candy bars or "certificates of recognition" are also very effective motivators.

7.2.4 Summary of Teamwork

7.3 The Adaptable Organization

There are two types of projects: standing projects and ad hoc projects. Standing projects represent ongoing tasks that will generally never cease, and most of which usually involve some type of operational or production activity. Two examples of standing projects are IP route engineering and supercomputer networking operation. Currently, there are about 60 standing projects.

All other projects are ad hoc projects, and make up the bulk of the work performed by NETS. Ad hoc projects vary greatly in size and complexity. A simple project, such as attaching a new computer to a network, might involve one or two people and take a few hours. A complex project, such as recabling the Mesa Lab, might involve most of NETS and take longer than a year to complete. Large projects (such as the Mesa Lab recabling project) spawn a stream of simpler and smaller projects to accomplish individual tasks as the large project moves forward. In a typical year, NETS will complete over 700 projects, with around 50 in the complex-project category.

NETS also prioritizes all of its ad hoc projects as being "Urgent," "High," "Medium," or "Low." Standing projects all relate to operational activities, so their de facto priority essentially exceeds "Urgent" since the highest priority activity for NETS is to maintain all production networks in an operational state.

This adaptable organization is valuable to NETS for several reasons:

1. It allows NETS to quickly and efficiently respond to large numbers of diverse tasks with exactly the right set of skills needed to accomplish each task.
2. It builds leadership skills for all NETS employees, since everyone gets a chance to be a team leader.
3. It encourages a cooperative approach to all work, since each person who may be a team leader on a few projects will also be a team member on a larger number of projects. Team leaders have no special authority, so all work must occur in a cooperative fashion. Any authority a team leader may possess derives solely from any professional respect that person may have developed among their peers. Such respect can only be obtained by demonstrating competency, hard work, fairness, human warmth, and other attributes that people prize in their leaders. In fact, development of this kind of professional respect is an outgrowth of the adaptable organization, and helps to minimize a person's job title and salary level as a discriminator of that person's worth to the group.
4. It specifically encourages people to take ownership of problems since each project has a team leader who is responsible for shepherding the project, and all people are team leaders for at least some projects.
5. Improved communication usually results, because the chief responsibility of the team leader (and also the team leader's most important tool) is to receive and distribute information to make sure that everyone on the team has the necessary information and understanding to undertake the project.
6. Because of the diverse backgrounds and skill sets of team members, a broader understanding among team members regarding all aspects of the project results because each team member must impart an understanding to the team of the factors peculiar to their part of the project. Eventually, a considerable amount of cross-training is inherently accomplished, and everyone's teaching skills are improved.

7.4 Tools

7.4.1 E-mail and the Web

E-mail is used as the optimal method for disseminating straightforward information to groups of one or more people, though face-to-face contact is still far superior when intense interactive communication is necessary. E-mail is even a useful enhancer of personal conversation since it can be used to formulate considered responses and/or to summarize and document important aspects of a conversation.

In fact, the telephone has increasingly lost favor as a means for simple business communication, and its best use now is as a substitute for intense interactive communication when distance prevents face-to-face contact. Should widespread two-way or n-way IP-network-based personal audio or video communication ever prove popular, the last important business use of the telephone may disappear as well.

Almost all information relating to the conduct of NETS's business has found its way onto the Web, and this information is thus readily available to anyone who needs it. By using e-mail, it's simple to forward a URL pointing to the Web location of the required piece of information. Upon receiving the URL, the information requester can easily review the requested information and print a local hardcopy if needed. Since most equipment and material suppliers operate in the same fashion, it's easy for NETS to obtain technical information from the Web as well. By conducting commerce on the Web in this fashion, NETS's use of the FAX machine has dwindled to almost nil.

NETS continues to find new uses for e-mail and the Web, and usage is automatically adopted whenever it's apparent that the new way of doing business is more efficient than the old way. E-mail and the Web thus continue to offer new opportunities for improved productivity, and are therefore critical strategic tools for NETS.

7.4.2 Meeting Tools: Electronic Calendars and Room Schedulers

7.4.3 Project Tracking Software

7.4.4 Project Planning Tools

7.4.5 Network Testing and Diagnostic Tools

7.4.6 Network Monitoring Tools

7.4.7 Documentation Tools

Recently, however, several documentation tools have appeared in the marketplace that may greatly reduce the cost of documentation, improve the quality of the documentation, and make the document content more easily available to a large number of people.

7.4.8 Conclusion to Use of Tools

However, extensive use of these tools is consistent with several basic strategies:

1. Purchasing hardware and software in lieu of expenditures for staff increases.

2. Using productivity tools in lieu of additional staff.

3. Paying close attention to the marketplace for significant new developments.

4. Investing in staff training to enhance the value of existing staff.

8. MAN AND WAN ISSUES AND STRATEGIES

8.1 Strategies Common to MANs and WANs

8.1.1 Purchased services for the MAN and WAN

However, one exception to this strategy would be to develop a long-term relationship with a vendor if some kind of best-price guarantee can be made. Such a relationship would be the best of both worlds, in which both stability and optimal price would be possible. Such a long-term approach in fact has served UCAR quite well regarding the multi-year contract UCAR has with MCI for Commodity Internet access. This contract will expire within a year, and because NETS took advantage of initial Commodity Internet pricing uncertainty and negotiated an exceptional contract price, it is likely that UCAR will actually have to pay more for equivalent Commodity Internet service when a new contract is required.

It should be noted that, based on usage statistics, the DS-3 (45-Mbps) Commodity Internet link will in fact have to be expanded by FY2000. The cost of this expansion is expected to require an additional $243,000 each year for UCAR over and above UCAR's current expense for Commodity Internet.

8.1.2 Participation in Regional and National Networking Initiatives

It's also important to note that strategic MAN and WAN collaborative opportunities arise in a completely unpredictable fashion, and it is necessary that both NETS and UCAR remain sufficiently adaptable to participate in these important programs. In particular, NETS and UCAR should continue their visionary stance regarding the potential payoff of such programs even if the benefits of participation are not immediately manifest. Internet2's Abilene network is an excellent example of a program whose payoff was not readily predictable at the time UCAR's participation first began.

8.2 MAN Issues and Strategies

8.2.1 Remote-Working and Home-Access

8.2.1.1 Current Provision of Remote-Working and Home-Access

8.2.1.2 Future Provision of Remote-Working and Home-Access

1. Continue as a de facto private ISP and:
   1. Retire the non-upgradable modem pool and obtain another increment of ISDN/V.90-capable equipment and telecommunication links.

   2. If home-access speeds faster than voice-grade lines are needed, then each Division can also pay for US West ISDN or ADSL service for their employees. NETS would then have to obtain and operate the equipment required at UCAR to terminate the resulting ADSL ATM PVCs.

2. Get out of the private ISP business and:
   1. Each Division directly pays for access to a commercial ISP for those Divisional employees that require such service. If home-access speeds faster than voice-grade lines are needed, then the Division also pays for US West ISDN or ADSL service.

   2. A variation on this option would be to have a single contract with an ISP to service all UCAR employees for a single fixed fee.

One advantage of going to a commercial ISP is that an evaluation of bona fide home-access needs of each individual employee would be likely. Due to the lack of such evaluation to date, there has been some concern that much of the home-access service growth is simply a convenient zero-cost option for UCAR employees to access the Web from home.

NETS's strategy is to determine the most cost-effective alternative that meets the needs of UCAR's users.

8.2.2 Integration of Voice and Data in the MAN

8.2.3 BRAN

BRAN will include fiber that transits both the US West Boulder CO (central office) and the ICG Boulder POP (point of presence). UCAR could greatly reduce tail circuit charges to US West for access to the Boulder Main CO (BMCO), and could access ICG without any US West charges at all. ICG is a major US CLEC (Competitive Local Exchange Carrier), and direct access to its facilities has major implications for cheaper access by UCAR to national telecommunications facilities in Denver. Figure 7 shows a map of the proposed BRAN fiber routes.

Recovery of capital costs is very conservatively estimated to be 8-10 years based only on existing usage. Savings after this cost recovery is the most tangible benefit of BRAN for UCAR. However, the intangible benefits are enormous. UCAR's Boulder-area facilities and all other Boulder-area federal laboratory facilities can be connected at arbitrary speeds with only a minor concern for costs. High-speed competitive access to both local and national telecommunications facilities will be possible with a complete US West bypass, bringing true competition for a variety of telecommunications services to UCAR. Right now UCAR is at the complete mercy of US West for access to the world, and for all practical purposes, UCAR is isolated from the rest of the world by US West. The importance of direct access to ICG's Boulder POP can't be overstated.

BRAN is a tremendous opportunity for UCAR to continue to exercise its historical community leadership, and to provide benefit to both to itself and its local community peers. UCAR management is strongly supportive of UCAR's participation in BRAN and has given funding approval to develop accurate construction costs as well as initiating an effort to develop a management structure for operating BRAN once it's built.

8.3 WAN Issues and Strategies

8.3.1 The Westnet GigaPop

The dynamics of the Westnet gigapop have always been fluid. Right now, NCAR supplies a means for Westnet members to attach to itself, the vBNS, the MCI Commodity Internet, and the BBN Commodity Internet via SNI (SuperNet Incorporated, formerly Colorado SuperNet). Figure 8 illustrates the concept of the gigapop, that is, a situation where several institutions network together to a common location from which shared access to national and other networks is then effected. Figure 10 shows the conceptual MAN and WAN connections to the UCAR network, while Figure 11 depicts the actual MAN and WAN connections to the UCAR network. Included on these diagrams are the Westnet-related MAN and WAN network connections.

Those Westnet universities that find it advantageous to connect to NCAR for these services do so and help share in some of the costs, while other Westnet universities find the costs to connect to NCAR to be too great to justify any possible benefits. For many universities, the cost to attach to NCAR can't be justified even though the cost to reach Denver is justifiable. It doesn't make a whole lot of sense to incur the costs of first moving data to Denver and then to Boulder, only to have to send the data back to Denver again. Because Denver is the regional nexus for attaching to national networks such as the vBNS, MCI, and SNI/BBN, it is actually the most natural location for the Westnet gigapop.

With the advent of the Qwest-based Abilene network of Internet2, such a location could become even more compelling. The CU Denver Auraria campus has space to house a Westnet gigapop in one of its buildings at 14th and Lawrence, which is only a couple of blocks from several POPs and COs, including US West, MCI, ICG, Qwest, and SNI. This location therefore has excellent proximity to most major Colorado telecommunication access facilities, and interconnect costs would be extremely attractive.

Westnet and NETS are in the process of exploring the possibility of splitting the Westnet gigapop between NCAR and CU Denver. The NCAR portion of the gigapop would allow access to institutions north of Boulder, while the Denver portion would provide access for the rest. Such a split could mean that NCAR equipment would be located in Denver and that NETS would become responsible for equipment at yet another remote site. The advantages and disadvantages to NCAR of such an arrangement will have to be carefully considered. Figure 9 is a diagram depicting this proposed distributed gigapop architecture.

NCAR's participation in Westnet is a strategic initiative that has enabled NCAR to take full advantage of opportunities flowing from Abilene and other future national and regional networking initiatives. Participation in Westnet is also an important instance of NCAR helping to provide leadership in one of its communities of interest.

8.3.2 Internet2/Abilene and the vBNS

8.3.2.1 Internet2/Abilene

NCAR's longstanding participation in Internet2 and Westnet will help enable NCAR to take full advantage of Abilene's capabilities to vastly improve networking to those 48 UCAR universities that are also currently Internet2 members. Figure 13 depicts the location of proposed Abilene gigapops, while Figure 14 illustrates the location of UCAID member institutions.

It's important to note, however, that Abilene members will have to pay the costs to operate the network, even though the underlying facilities have been donated. Estimated costs for a gigapop such as Westnet to join Abilene at OC-3 speeds would be an annual subscription fee of $110,000 plus an annual fee of $20,000 per individual Westnet institution. To join at OC-12 speeds would cost require an annual subscription fee of $320,000 per gigapop plus the $20,000 per institution. Westnet institutions would also pay for their own telecommunications link to the Westnet gigapop, and the Westnet gigapop's link to Qwest. It has yet to be determined how these fees would be split among Westnet institutions.

8.3.2.2 The vBNS

By virtue of being a high-speed node on the vBNS, NCAR obtains good capability to exchange data with the other supercomputing centers as well as the HPC universities. This connectivity provides opportunities for remote supercomputing, both for university scientists accessing NCAR and for NCAR scientists accessing universities. The vBNS supplies exceptional connectivity to other government agencies such as NASA, DOD, etc. The vBNS STARTAP facility near Chicago has has been designated as the preferred vBNS attachment point for international research and educational network connections, and is the attachment point for the COSMIC network's access to NCAR.

However, it is important to note that NSF funding for the vBNS terminates in March 2000. Equally important is the fact that MCI has been heavily subsidizing the operation of the vBNS. Qwest's five-year contribution of a free national OC-192 fiber backbone for Internet2's Abilene network is therefore vital to the continuance of affordable national networking for the U.S. university community. MCI no longer has an implied monoply vis-a-vis the vBNS, and it will now be forced to continue its subsidized pricing if it wants to attempt to continue the vBNS in a commercial form.

The key strategic issue for UCAR will be to maintain NCAR's ability to take advantage of the capabilities of both the Internet2/Abilene and vBNS networks.

9. STRATEGIES TO DESIGN AND IMPLEMENT THE FLEXIBLE LAN

9.1 Introduction to LANs

9.1.1 Three Network Layers

9.1.1.1 Layer1 Networks

9.1.1.2 Layer2 Networks

Today, logical networks are implemented mainly with network-equipment called packet-switches. Since almost all packet-switches are actually Ethernet packet-switches, the term "packet-switch" is usually understood to mean "Ethernet packet-switch".

9.1.1.3 Layer3 Networks

9.1.2 LAN Types

Host LANs are LANs that have computers connected to them, while backbone LANs generally do not have computers connected to them. The main purpose of host LANs is to let the connected computers talk to each other. The main purpose of backbone LANs is to carry traffic between host LANs. Because backbone LANs can interconnect many other LANs, backbone LANs typically have to have much more capacity than host LANs.

Both backbone LANs and host LANs can be constructed using packet-switches and ATM switches. An important component of host LANs is the network interface card (NIC) inside each computer which is used by the computer to attach to the network, usually via a packet-switch or ATM switch.

A qualification of the two types is made only if the distinction is important. An unqualified use of the term "LAN" implicitly refers to both types. Often, the term "backbone" is used instead of "backbone LAN".

9.1.3 LAN Components

9.2 LAN Design Strategies

9.2.1 General LAN Design Strategies

9.2.1.1 Strategic Criteria Discussion

1. Adaptability to ever-increasing network bandwidth loads from existing staff.

2. Adaptability to sudden growth of UCAR staff.

3. Adaptability to big moves and changes of connected user-devices.

4. Adaptability to future network technologies and applications.

5. Avoidance of premature network-equipment obsolescence.

6. Robustness and reliability of service.

These criteria are discussed in detail below, followed by general strategies relating to these criteria.

9.2.1.1.1 Accommodating Bandwidth Growth

9.2.1.1.2 Accommodating Sudden Staff Growth

9.2.1.1.3 Accommodating Sudden Staff Movement

These moves and changes along with the growth mentioned earlier are often referred to as the "move, add, change" problem. The LAN must adapt to moves, adds, and changes of both the steady variety and the exceptional variety.

9.2.1.1.4 Accommodating Future Network Technologies

Likewise, Internet technology quietly existed for 15 years with little application beyond government and academic usage, yet during the last three years Internet technology has become poised to restructure the economic and social fabric of the world.

Networking technology is changing at an extraordinarily rapid rate and prediction of the future is exceedingly difficult. Nonetheless, today's LAN design must accommodate tomorrow's technologies.

9.2.1.1.5 Avoiding Premature Network-Equipment Obsolescence

9.2.1.1.6 Robustness and Reliability of Service

9.2.1.2 The Strategies

9.2.1.2.1 Risk Avoidance

9.2.1.2.2 The Just-In-Time Decision-Making (JITDM) Principle

9.2.1.2.2.1 Avoiding Premature Obsolescence

Another result of JITDM network-equipment purchasing is that, as network re-engineering takes place, each new project can receive the latest and most capable network-equipment. JITDM is also valuable because waiting to make a purchase will always save interest costs of money borrowed to pay for network-equipment.

However, JITDM purchasing sometimes conflicts with special trade-in or upgrade deals offered by a vendor, and so the long-term value of the deal must be weighed against the risk of premature obsolescence.

9.2.1.2.2.2 Adopting New Technology No Sooner Than It Is Needed

As an example, right now NETS has a choice of whether to continue to use ATM links in its backbone or to look at the possibility of using Gigabit Ethernet instead of, or in addition to, continuing with ATM exclusively. There are a large number of factors that affect this decision, and these won't become clear for several months, so for now NETS will avoid making expenditures that might lock us into one choice or the other.

Note that the JITDM strategy regarding new technology isn't always applicable. Sometimes it's necessary to adopt a new technology as soon as possible because it will have such far-reaching implications regarding future networking design that early adoption is paramount. The early adoption of VLANs (virtual LANs) and packet-switching by NETS is an example of necessary early adoption.

9.2.1.2.3 KISS Principle

9.2.1.2.4 Adherence To Standards

Standards are particularly important for Layer1 cabling infrastructure and Layer3 protocol support; such standards are discussed in more detail in later sections.

9.2.1.2.5 Network-Equipment End-Of-Life (EOL) Strategy

It's also probably not practical to consider replacing network-equipment faster than three years anyway, even if it was affordable and even if a legitimate demand existed. Each replacement cycle requires a certain amount of user disruption, and it takes a certain amount of time to plan and carry out each replacement. More importantly, it takes a while before new network-equipment stabilizes in a production mode. This stabilization period includes training and understanding by the engineers, optimization of configuration by the engineers, installation of a sufficient number of bug fixes to achieve operational robustness, discovery and elimination of fragile hardware components, and the understanding of optimal usage by the users. Too rapid of a replacement cycle could result in a network that was perpetually in stabilization mode, with no time occurring in which stable production usage actually takes place.

Cyclic network-equipment replacement also leads to a natural strategy for choosing which network-equipment should be upgraded next. As service demands rise, the oldest and least capable technology is replaced with the newest, and the users with the worst service now have the best service. This approach automatically defines an end-of-life (EOL) strategy. One could call this the "ladder" strategy, because you can imagine technology generations climbing up a ladder with new technologies starting at the bottom rung, pushing all others up a step, with obsolete technologies being pushed off the top rung.

A consequence of network-equipment life cycles is that multiple generations of networking technologies must be concurrently supported in the production network. This is the main drawback to the "ladder" strategy, but it is unavoidable.

9.2.1.2.6 Sacrificing Redundancy

Sacrificing redundancy is a deliberate engineering decision to avoid the very high costs necessary to achieve a certain incremental improvement in reliability. If reliability is sufficient to meet the service requirements of the institution without expensive and complex redundancy, then the considerable cost savings achieved with diminished redundancy is justifiable.

9.2.1.2.7 Minimizing The Number Of Vendors

A good analogy for the construction of networks using network-equipment from many different vendors is to try to assemble an optimal set of golf clubs by selecting the best individually performing club of each type from all available sets. Each club selected is the "best" of its type, and all of the assembled clubs meet the required specifications. Wouldn't such a set then be optimal? In fact, such a set of golf clubs could be about the worst set you could assemble because they weren't designed to function as a set. Each club has its own unique and subtle characteristics, and it could be difficult to achieve any consistency of usage among such a set. A similar situation exists with sets of network-equipment. Even though a multi-vendor set of network-equipment may meet all of the necessary standards, the great number of small but subtle differences among diverse network-equipment will usually outweigh any apparent performance or cost benefits.

Network-equipment has to be considered as a system, and given the complexity of networking, sticking with only one or two vendors has several important advantages over simply selecting the cheapest or the "best" box of each type from many different vendors. Integrated network-equipment components have a very high need for commonality in the following areas:

1. User configuration interfaces should be the same.

2. User access interfaces should be the same.

3. Management support functions should be the same.

4. Parts like power supplies and fans should be the same across equipment types.

5. Integrated network management packages should require no engineering effort from the customer (plug and play).

The following advantages accrue to an integrated product line:

1. With integrated systems, engineers can concentrate on learning one set of skills very well, rather than learning a different set of skills for each different piece of network-equipment. Training is therefore better and less expensive for an integrated product line. It's also easier to cross-train engineers when fewer skill sets are necessary. Problem diagnosis is also easier when fewer types of similar network-equipment must be dealt with.

2. Stocking spare parts is more economical for an integrated product line than stocking multiple sets of spares for many different types of network-equipment.

3. For an integrated product line, there is only a single integrated maintenance contract, instead of many different contracts, one for each vendor. Maintenance discounts can also be obtained for large volumes of network-equipment from one vendor, which will often be less expensive than many full-priced maintenance contracts from many different vendors.

4. Purchasing is easier for an integrated product line. Only the intimate details of one set of products must be learned, and there is only one set of salespeople to deal with.

5. Keeping abreast of new product developments within an integrated product set is much easier than keeping track of developments among several vendors' product lines.

6. Network-equipment upgrades and embedded firmware upgrades are much easier to track and manage in an integrated product line, as opposed to tracking hardware and software upgrades for many different product sets.

Whenever individual Divisions install and operate their own networks, NETS strongly encourages the purchase of network-equipment consistent with that chosen by NETS for UCAR-maintained networks. If NETS-recommended network-equipment is chosen, then it's possible for the network-equipment to eventually be folded into the UCAR-maintained network some time in the future if that becomes desirable, and it's also possible that NETS could deliver assistance on a time-and-materials basis in the meantime.

However, should NETS's recommendations not be accepted, then it's not always feasible for NETS to render assistance, and it's also not always feasible for NETS to later accept responsibility for maintaining such network-equipment. The situation is very similar to water systems, wastewater systems, and roadways constructed by housing developers as part of residential housing developments. Municipalities won't accept these systems into their maintenance domains unless these systems meet municipal standards.

9.2.1.2.8 Over-Engineering And Excess Capacity

Another important reason for network over-engineering is that the bulk of network capacities usually exist to handle stochastic loads. Should peak loads become chronic, networks often collapse, producing nil throughput. Thus a network must always remain over-engineered to ensure headroom for load bursts, as well as capability to absorb load growth.

Another form of "headroom" is infrastructure that is installed but unused. In the case of cabling, this means that installed, but unused, cables are available to meet expansion demands, such as the connection of a new user-device. It should never be the case that normal service demand growth must wait for new cabling to be installed. This means that at some point all of the installed cabling was used, and a new installation project was not started prior to the new demand. The proper approach for dealing with anticipated demand is to always have reserve cabling available and to start the addition of new cable before the old cable is used up.

A similar situation exists with network-equipment headroom. For example, to connect a new user-device to the network requires the availability of an unused port on some type of network-equipment (generally a packet-switch). An effort should always be made to make sure unused ports are available because it takes too long to install a new port-board containing a new set of unused ports. When the unused ports on the last port-board diminish to a certain number, then another port-board should be installed before all of the ports are used up. In fact, the packet-switch should also have unused board-slots for new port-boards, and when a port-board is installed in the last unused board slot, it's time to install a whole new packet-switch before that last port on the last port-board is used because it takes even longer to install a new packet-switch than a new port-board.

Of course network-equipment headroom applies to all network-equipment, not just packet-switches. ATM switches and routers require similar considerations regarding headroom.

9.2.1.2.9 Management Of Packet-Switch Port-Boards

Port boards are added to packet-switches with port counts generally ranging from 12 to 48 ports per port-board, depending on port type. Common port types are copper 10-Mbps Ethernet, fiber 10-Mbps Ethernet, copper 100-Mbps (Fast) Ethernet, and fiber 100-Mbps (Fast) Ethernet. Usually there is only one port type per port-board.

Currently, there are about 30 packet-switches in use, and if each packet-switch had to provide some unused ports of each of the four common types, that would mean there could be as many as 120 port-boards in partial use. Furthermore, port densities range from 12 to 48 ports per board. Dense boards are good because the number of board-slots is limited for each switch. Dense boards are also cheaper per port, but on an absolute basis a 48-port-board is more expensive than a 12-port-board. The most efficient board arrangement is usually to start off with less-dense boards, until board-slots become limited, and then swap them for denser boards. The less-dense boards can then be moved to other switches that have plenty of board-slots still left. This constant juggling of boards of different types and densities among large numbers of packet-switches makes it imperative that all switches and boards be compatible. Such compatibility can only be achieved if all network-equipment belongs to the same equipment family obtained from the same vendor. Purchasing a few switches and port-boards from each of several different vendors quickly becomes a losing strategy.

9.2.1.2.10 Adaptable Network-Equipment: Modularity and Scalability

9.2.2 Layer1-Specific LAN Design Strategies

Until the last few years, UCAR network cabling was installed in an ad hoc fashion with little regard for long-term maintainability, usability, growth capacity, or industry standards. Recently, a key strategy has been the development of a structured networking cabling design and the implementation of this standard via the retrofit of UCAR buildings. This design is described below.

9.2.2.1 Telecommunications Outlets

9.2.2.2 Telecommunications Closets

9.2.2.3 TO and TC Cabling

Cat5 copper cables provide high-speed short-distance network connections, with distances no longer than 100 meters and speeds currently as high as 100-Mbps. Cat3 copper cables deliver low-speed short-distance connections, with distances usually not exceeding 100 meters and speeds not over 30-Mbps. Cat3 copper cables are also used to connect telephones in workspaces to centrally located telephone switches.

Multi-mode fiber optic cable provides very-high-speed intermediate-distance network connections, with distances around 2,000 meters and speeds in the multi-gigabit range. Copper cable is really only useful between workspace equipment and TC equipment because of its distance limitations. Multi-mode fiber optic cable is the workhorse for all other connections.

Copper cabling has been installed because the copper-based end equipment, both user equipment in the workspaces and the network-equipment in the TCs, is significantly less expensive than equivalent-bandwidth fiber-based equipment. Fiber has been installed to take advantage of its flexibility in long-distance applications and to take advantage of its superior potential for greatly increased bandwidths for future growth.

The fiber plant has been designed to be very flexible, and to take advantage of the long end-to-end distances allowed by fiber connections. TCs typically connect 48 pairs of multi-mode fiber optic cable back to the central location. By patching TC fiber together in the central location, and/or patching workspace fiber to TC fiber, an end-to-end fiber connection can be made from any workspace to any other workspace, to any TC, or to the central location. Closet-to-closet runs are also easy to patch together.

One of the big advantages of this fiber plant design is that the location of fiber-based network-equipment is flexible. For instance, network-equipment that connects to fiber-attached user-devices is usually fairly expensive. ATM switches and FDDI concentrators are examples of such relatively expensive fiber-based network-equipment. Furthermore, demand to connect to such network-equipment can be sparse, with few fiber-based user-devices connecting through any given closet. Location of the smallest available ATM switch (8 ports) in every TC that has at least one ATM connection is very expensive and wasteful. With the flexible fiber plant, it is very easy to patch all ATM user-devices all the way through to the central computer room, and use a single ATM switch at this central location. Should a particular TC ever have enough ATM connections to warrant its own ATM switch, then it's a simple matter to repatch the connections to that closet to terminate them at a local ATM switch.

It's very important that the network cabling infrastructure be consistent throughout the UCAR buildings. While it might be tempting for a given organization in UCAR to install a less expensive but non-standard cabling infrastructure, ultimately the resulting cable plant becomes a maintenance nightmare as more and more of the UCAR plant becomes less and less compatible. You basically end up with a Balkanized network. In fact, this is the situation in much of the existing cable plant that NETS has been struggling to fix for years, and effort needs to be directed at reducing such problems, not increasing them.

One of the big problems with the Balkanized approach is that one group's non-standard cabling is unlikely to meet the needs of the next group to utilize the space, and given the amount of space turnover at UCAR, the result would be constant cable plant changes to meet the needs of the next group. Allowing the installation of network cabling that is less capable than the standard makes about as much sense as allowing each person to select cheaper electrical wiring tailored to their particular needs at that time so as to save a bit of money on the electrical wiring costs. In the case of electrical power, wiring standards are enforced to ensure the existence of surplus capacity to meet future needs without the unnecessary expense to rewire each workspace every time someone moves, and without the need to maintain a multitude of constantly changing and differing capacities. A similar need for standardization is necessary for the network cabling infrastructure.

The network cabling infrastructure is depreciated over a 10-year period. In the world of networking, any projection past a couple of years falls into the area of prophesy, so the 10-year time period is simply a best guess at when the cable infrastructure will become technologically obsolete and must be replaced (renewed) with some newer technology. This guess is based on the fact that no new cabling standards have emerged in recent years, the fact that Cat5 can handle at least 100-Mbps, and the fact that multimode-fiber can handle multiple gigabits. The cable plant won't actually physically wear out, so replacement will be necessary only if some completely unknown technology emerges that handles data communications in such a vastly superior manner that Cat5 and multi-mode fiber equipment are no longer available or are too costly to maintain.

The future of the cabling infrastructure at UCAR is discussed in great detail in a later sections of this document.

9.2.3 Layer2-Specific LAN Design Strategies

9.2.3.1 VLANs and ELANs

9.2.3.2 Ethernet and ATM

9.2.3.2.1 In the Backbone

ATM in the backbone is more flexible than Ethernet because the same ATM switches and links can be used to build LAN backbones, to build WAN and MAN interconnects, and to attach end-user computers. ATM's PVC (permanent virtual circuit) and SVC (switched virtual circuit) capability also allows easy construction of arbitrary LAN, MAN, and WAN Layer2 networks independent of the underlying physical topology of the switches and links. This flexibility in Layer2 network design also greatly aids the design of routing engineering, and greatly reduces the cost of necessary routing equipment.

On the other hand, Ethernet backbone technology is less expensive than ATM, though it currently requires the use of proprietary trunking protocols and has less flexibility than ATM for providing certain advanced network functions such as real-time voice. But perhaps Ethernet's biggest drawback as a backbone technology is the lack of a function similar to ATM PVCs and SVCs.

9.2.3.2.2 For Connecting End-User Computers

Computers can also directly attach to the ATM fabric with ATM NICs. The advantage of direct ATM attachment is that ATM-attached computers can participate on multiple logical networks with a single ATM NIC. File servers in particular can benefit from ATM attachment if they must serve client computers on different logical networks. By participating directly in multiple logical networks, such servers can serve clients directly on different logical networks without having to go through a router.

9.2.3.3 FDDI End-Of-Life

9.2.4 Layer3-Specific LAN Design Strategies

9.2.4.1 Protocols

At this point, the only non-routed traffic appears to be a few SAMBA-related applications. Use of SAMBA is one method of providing NFS file sharing between Windows NT and UNIX systems, and part of the SAMBA function uses non-routable protocols.

One of NETS's strategies is to promote standards-based services whenever possible and to avoid services that require extra support such as non-IP protocols and non-routable protocols. Non-IP protocols and non-routable protocols require specialized networking configurations, and it is in UCAR's best interests to avoid such maintenance-intensive singularities whenever possible.

9.2.4.2 Routing Technology

9.2.4.2.1 The Impact of ATM on Routing Engineering

This ATM design is substantially less expensive than older network designs that used to require each logical network to terminate on its own physical router port. Fewer router ports also means that NETS can use fewer, but more powerful, routers for simple and flexible routing engineering. This small set of powerful ATM-capable routers handles all traffic sent between LANs, MANs, and WANs at UCAR.

9.2.4.2.2 Router Upgrade Strategies

1. Continue to upgrade the five existing Cisco 7500/7000 routers.

   There are a few more upgrade options that can give added routing performance to the existing Cisco routers. The main question will be whether the money is better spent on a new line of routers with significant growth potential, or whether extending the life of the older routers is worth the upgrade costs.

2. Replace the 7500 routers with next-generation routers.

   Several vendors, including Cisco, have new routers that are substantially more powerful than the Cisco 7500 routers. However, these routers are very expensive, and generally route only the IP protocol. Such IP-only routers are therefore not suitable for all UCAR applications, as some non-IP protocols such as AppleTalk must still be supported on local LANs. Still, use of these big new routers could be appropriate for gateway functions to the outside world, since all but one computer at UCAR uses the IP protocol to network with the external world.

3. Distribute the routing functionality by adding routing engines to some of the packet-switches.

   Distributing the routing function into some of the packet-switches could be appropriate for some inter-LAN routing. However, the number of packet-switches is relatively large, and the main issue will be whether inter-LAN routing is concentrated enough in a small enough number of packet-switches to warrant the addition of routing engines.

4. Utilize MPOA to avoid routing.

   MPOA is a new ATM-based protocol designed to allow inter-LAN traffic to avoid going through a router for ATM-based LANs. However, Cisco won't be supporting MPOA on existing ATM boards in the Cisco packet-switches because Cisco wants to include it only on a new generation of ATM cards that have hardware acceleration for MPOA. Therefore, if MPOA is ever to be used on the existing packet-switches at UCAR, expensive ATM board replacements will have to be made for these packet-switches. The question is whether the amount of inter-LAN routing off-loaded with MPOA will justify the large expenditure for ATM board upgrades.

Right now, none of these options stand out as an obvious choice. They aren't even necessarily mutually exclusive; a mixture of routing upgrade options might be the best way to go. The most important issue at this point is to avoid decisions that prematurely limit the available options.

9.2.4.3 Avoid Routing When Possible

Routing can also be avoided by using packet-switching technology coupled with VLANs and ELANs to reduce the number of LANs and increase their size, thus reducing the need to route between larger numbers of smaller LANs, as used to be the case prior to this new technology. The recognized strategy is: "Switch when you can, route when you must."

10. BUILDING A LAN FOUNDATION FOR THE FUTURE

This section discusses the factors contributing to increased future LAN loads, what needs to be done to the LAN so it can function in the future, and what will happen if the LAN is not improved.

10.1 Factors Contributing to Increased LAN Loads

10.1.1 LAN Network Technology Itself Has Improved

Also, with the advent of less expensive packet-switches, shared-media Ethernet networks are rapidly disappearing. Each host now has it own dedicated full-duplex bandwidth Ethernet and does not have to share this bandwidth with another computer. Shared-media Ethernet networks are therefore no longer a bottleneck.

10.1.2 WAN Network Technology Has Improved

10.1.3 Desktop Computing Power Continues to Grow

Two of the biggest factors contributing to the power of desktop computing are CPU performance and disk performance.

10.1.3.1 CPU Power Continues to Grow

10.1.3.2 Disk Storage Continues to Improve

10.1.4 Three Examples of What This All Means

10.1.4.1 Example of Yesterday's Network Utilization

10.1.4.2 Example of Today's Network Utilization

10.1.4.3 Example of Tomorrow's Network Utilization

10.2 Meeting the Demand: The Wrong Way and the Right Way

A mass conversion of UCAR computers to Fast Ethernet (100-Mbps) will be required within the next three years. However, the UCAR cabling infrastructure is inadequate to support this required conversion. About 40% of UCAR LAN facilities have been fitted with cabling capable of supporting future needs. The other 60% of the LAN cabling infrastructure is insufficient for future growth and needs to be upgraded as soon as practical. Attempts at ad hoc conversion without structured cabling upgrades would be a serious impediment to the long-term maintainability and scalability of UCAR LAN networking.

10.2.1 The Wrong Way

A similar situation existed with respect to networking infrastructure prior to the central funding and support of networking by UCAR. In the early days of networking at UCAR, there were a great number of individual ad hoc efforts to provide this infrastructure. The effects of such efforts could be observed in the network cabling strung haphazardly throughout dropped ceilings. Ultimately this approach to providing network connectivity was not scalable. However, it was very difficult to break out of this haphazard mode of operation and build a completely new structured network that, once built, would accommodate growth in an organized fashion. The problem was that all personnel and material resources were being consumed by the stringing of network cables to meet immediate needs. And of course, as more cable was strung in this fashion, the whole structure became more unsupportable. The only way to stop this death spiral was either to cease meeting peoples' immediate needs for a while by reprogramming available resources into a new infrastructure or to maintain the existing effort and find new funds for the new effort.

In fact, a combination of both methods has been used to reach where we are today, which is that about 40% of UCAR facilities are capable of 100-Mbps to the desktop via a structured cabling plant. The problem is what to do about the other 60%. These folks have the same need for higher speeds as everyone else, and if their needs aren't met by the existing infrastructure, then they'll find a means to bypass the inadequate infrastructure. Bypass activities essentially involve going back to the mode of everyone running their own cabling haphazardly in dropped ceilings. This kind of bypass has actually started to happen, and is a serious warning sign that the existing infrastructure is inadequate.

In an attempt to head off the haphazard bypass, NETS has examined possible stopgap options that use the existing non-upgraded infrastructure. NETS has already provided one stopgap solution by upgrading much of the older copper cabling so that it is at least sufficient to handle dedicated 10-Mbps Ethernet. However, no further upgrade of the existing copper is possible without complete replacement, and people are clamoring for 100-Mbps (Fast) Ethernet. The only other stopgap option is to connect equipment to the network with installed workspace fiber.

There are several problems with the fiber option. First, approximately 60% of the installed TO fiber is not terminated, and it is costly to terminate it in an ad hoc fashion, particularly in terms of labor. Second, either a copper-to-fiber repeater would be required for each copper-based user-device, or the user-device would have to be directly fiber-capable. Either case is much more expensive than simply using an all-copper environment, and neither option would integrate well into a copper design once such a design was implemented. Third, the necessary network-equipment can't be purchased in increments smaller than 12 ports, and there isn't enough TC fiber to concentrate sparse usage back to a central location, so many expensive and sparely used port-boards would have to be installed. Finally, all of the labor and supplies expended on ad hoc office-fiber termination are ultimately wasted because it is necessary to first remove all old cabling from existing office conduits to make room for the new, upgraded cabling when it is installed. This removal requires the termination parts to be cut off all cables to be removed, and since it's impossible to remove some cables and leave others in small-diameter conduit, this means all existing terminations must be destroyed.

However, the worst thing about all of this is that all available resources are being devoted to meeting current needs with wasteful ad hoc activity that is ultimately not contributing to the actual solution of the problem. This is the same death spiral mode of operation just described above. The more sensible course is to expend additional resources up front and/or delay meeting immediate needs for a while rather than getting caught up in frantic, but ultimately futile activity.

10.2.2 The Right Way

10.3 Reinforcing The LAN Foundation During The Future

11. ANALYSIS OF FUTURE BUDGETS

11.1 Introduction

Note that even though budgets have been projected until FY2014, this has been done mainly to illustrate calculation principals and to show possible budget trends. Obviously such long-term budgets should not considered to be etched in stone because of long-term uncertainties. However, it is hoped and intended that the projected budgets represent maximum spending scenarios for at least the next few years.

11.2 Fixed Cabling Plant

11.2.1 UCAR Cabling Completion Project

Cabling costs have historically run about $1,500.00 per TO. This cost encompasses all portions of the physical cabling plant including the TCs (telecommunications closets) and cabling from the TCs to a central location.

Ordinarily, network-equipment costs would add about $750.00 per TO to the $1,500.00 cabling costs. However, all 3,000 UCAR user-devices are expected to attach to packet-switches at 10BaseT speeds before FY2000 and within current equipment budgets, and therefore no new network-equipment expenses need be incurred for the 60% cable plant buildout.

Thus, with 1,360 TOs to build out, the total capital costs to complete the LAN infrastructure for current UCAR facilities are estimated to be about $2,000,000 dollars depreciated for 10 years for cabling improvements, and zero dollars depreciated for 3 years for network-equipment. This $2,000,000 dollars is shown in Table 6 in line "Recabling" as being borrowed in the year FY2000, and its depreciation and interest are shown in Table 5 in line "Recabling D/I".

It is very important to note that when the cabling plant construction is complete, the cabling for all 2,280 TOs will be capable of speeds of 100BaseT (Fast Ethernet) and higher, though almost all 3,000 user-devices will still be connected to packet-switch ports capable only of 10BaseT (Ethernet). The costs of upgrading the packet-switch network-equipment to supply Fast Ethernet packet-switch ports are expected to come from annual network-equipment budget items allocated for renewing existing network-equipment on a 3-year cycle. These network-equipment expenses are discussed below in "11.3.2 Systematic Network-Equipment Renewal".

It is also important to note that this budget does not include funds for re-cabling any remodel of existing UCAR facilities or any new facilities that UCAR may lease or purchase. Cabling costs for such remodeled or new facilities can be estimated using $1,500.00 per TO. Network-equipment costs for new facilities can be estimated at $750.00 per TO, plus annual hardware maintenance costs.

11.2.2 Systematic Cable Plant Renewal

In making depreciation assumptions, it has been assumed that the cabling plant has an average lifetime of 10 years, and must be renewed before the end of that lifetime.

Renewal could be effected by replacing an average of one-tenth of the plant each year for 10 years, where $342,000 of new money could be borrowed each year, while annual depreciation and interest would be devoted to paying back portions of previous years' loans.

Unfortunately, renewal of one-tenth of the cable plant on an annual basis is not effective because the entire upgraded plant would be no more than 4 years old when annual renewal would start, and such annual renewal would result in parts of the plant being replaced when it was only, say, 5 years old, while other parts would be 15 years old by their time of replacement.

Furthermore, 10 years is too long to predict what will happen with LAN cabling technology and when it will happen anyway, so it's difficult to recommend a specific replacement schedule. Instead, it is simply recommended that the cable plant be replaced when it is apparent it needs replacing.

For the purposes of developing sample budgets, however, an example renewal strategy is to replace the entire cable An example replacement strategy is to replace the LAN cable plant during 4-year periods of construction separated by 6-year periods of only maintenance activity. With an estimated total replacement value of $3,420,000, then for every 10 years, $855,000 would be expended for each of 4 consecutive years, with only $50,000 being devoted annually to maintenance during the intervening 6 years.

A sample phasing of a four-year construction cycle for replacing LAN cabling in 6 years, and which assumes only existing facilities, is shown below in Table 3b:

Money borrowed for this example LAN cabling renewal is shown in Table 6 in line "Cabling Renewal".

The corresponding example of cabling renewal depreciation and interest is shown in Table 5 in line "Cabling Renewal D/I".

11.3 Network-Equipment

11.3.1 Abilene: A New Project Requiring New Network-Equipment

This expense is a 3-year depreciation capital item borrowed in FY2000, with an initial one-half year's repayment in FY2000. The borrowed money is shown in Table 6 in line "New Equipment" for FY2000, and its depreciation and interest are shown in Table 5 in line "New Equipment D/I".

11.3.2 Systematic Network-Equipment Renewal

Network-equipment consists mainly of highly-specialized computers, so Moore's Law has been applicable to network-equipment. Assuming that a switch-port will cost a constant $500 is similar to assuming that a business-class PC/workstation will cost around $3,000. Using the $500 value for the 3,000 ports estimated to be in service by the end of FY1999 yields a network-equipment valuation of $1,500,000. The set of UCAR core routers, core ATM switches, terminal servers, testers, and miscellaneous networking equipment is estimated to have cost another $500,000. The total value of network-equipment is estimated to be about $2,000,000 by the end of FY1999.

In making depreciation assumptions, it has been assumed that network-equipment has an average lifetime of 3 years, and must be renewed by replacement/upgrade before the end of that lifetime. Renewal could be effected by replacing/upgrading an average of one-third of the network-equipment each year for 3 years. Thus each year, $670,000 of new money would be borrowed, while annual depreciation and interest would be devoted to paying back portions of previous years' loans.

If a constant amount is borrowed each year, a steady state situation is reached after 4 years, in which the depreciation and interest to pay back previous loans reaches a constant $767,000, assuming 6% annual interest, with equal-principal payments and half payments on the first and last years.

These figures do not include possible future growth in numbers of switch-ports, which could result from the addition of new staff and the acquisition of additional desktop computers by existing staff. Inflation is also not included because of the assumed operation of Moore's Law.

Even though the full $2,000,000 of network-equipment will already exist, the first full increment of network-equipment renewal money won't be requested until FY2000 because the FY1999 budget has already been submitted. The FY1999 budget does already include $350,000 for this item, and though this analysis shows that this smaller amount would be inadequate over the long term as an annual amount, the smaller amount during FY1999 simply slightly lengthens the initial 3-year replacement cycle.

The borrowed money is shown in Table 6 in line "Equipment Renewal" as being borrowed for every year starting with FY2000, and its depreciation and interest are shown in Table 5 in line "Equipment Renewal D/I".

11.4 New/Expanded Telecommunications Services

11.4.1 Commodity Internet Growth

Measured Commodity Internet usage has peaks of 98% of the maximum DS-3 (45-Mbps) link speed. This means that UCAR's Commodity Internet provider now has the option of charging for the full DS-3 link instead of just for the 10-Mbps. The second column shows the costs to UCAR and its Westnet partners for the full DS-3 link, assuming the same allocation proportion between UCAR and its Westnet partners.

Finally, given the 98% peak DS-3 usage, it's clear that even greater Commodity Internet bandwidth will be needed shortly. The third column in Table 4 shows the projected costs of two DS-3 Commodity Internet links, again assuming the two-to-one cost allocation ratio. The cost of two DS-3 links has been chosen for a couple of reasons. First, it's not clear that fractional link speeds will be available from Commodity Internet providers. Second, it's likely that when additional Commodity Internet bandwidth is added, that the total bandwidth will be allocated between at least two different major providers, thereby providing both redundancy and direct access to more parts of the Commodity Internet.

Based on usage statistics, the DS-3 Commodity Internet link will have to be expanded by FY2000. As indicated in Table 4, providing two DS-3 links for a full year will result in an additional estimated expense of $243,000 for UCAR over and above UCAR's current expense for 10-Mbps, again assuming equivalent participation by UCAR's Westnet partners. These new expenses would start in FY2000, and are added into Table 5 line "New Pur. Srv.".

11.4.2 Joining Internet2/Abilene

It's estimated that UCAR's portion of network-equipment expense for the Abilene hookup would be about $50,000; this is discussed above in "11.3.1 Abilene: A New Project Requiring New Network-Equipment".

11.4.3 Building and Operating BRAN

UCAR's share of BRAN's construction costs are currently roughly estimated to be about $350,000. This expense is shown as a 10-year depreciation capital item borrowed in FY2000, starting with an initial one-half year's repayment in FY2000.

The borrowed money is shown in Table 6 in line "BRAN", and its depreciation and interest are shown in Table 5 in line "BRAN D/I".

11.4.4 Telecommunications Summary

New network-equipment for Abilene is estimated at $50,000 (3-year depreciation), while new construction for BRAN is estimated at $350,000 (10-year depreciation).

11.5 Salary Trends

11.6 Budget Tables

Note that 6% interest per annum has been assumed when calculating all interest expenses, while 3% has been used for the rate of inflation.

11.6.1 Table 5: Projected NETS Budgets

The latter six lines show new items not previously budgeted. Notes on individual Table 5 lines follow.

11.6.1.1 Table 5: Line "Travel"

11.6.1.2 Table 5: Line "Mat. & Supl"

11.6.1.3 Table 5: Line "Depr. & Intr."

11.6.1.4 Table 5: Line "Prchsd. Srvcs."

11.6.1.5 Table 5: Line "Salry. & Bnft."

11.6.1.6 Table 5: Line "New Pur. Srv."

An additional $60,000 is added to these amount starting in FY2000, for a total of $492,000 respectively. This $60,000 is for hardware maintenance for new equipment and for maintenance for new equipment in Table 6 line "New Equipment".

These amounts are inflated by 3% annually.

Theoretically, this maintenance plus the maintenance already in Table 5 line "Prchsd. Srvcs." should cover renewal equipment purchased with funds in Table 5 line "Equipment Renewal" because renewal equipment replaces or upgrades existing equipment.

11.6.1.7 Table 5: Line "Cabling Renewal D/I"

11.6.1.8 Table 5: Line "Equipment Renewal D/I"

11.6.1.9 Table 5: Line "Recabling D/I "

While it has been recommended that the full re-engineering should take place in FY2000, this work could be performed over multiple years to avoid an expenditure of $2,000,000 in a single year, though the network support consequences are likely to be quite serious.

However, it's not clear that such multi-year funding would substantially reduce the budget impact since these costs are 10-year depreciation items and the actual expenditures are for depreciation and interest. For example, the D & I expense for a full FY2000 buildout would be $224,000, $320,000, and $308,000 for the first three years, decreasing thereafter. Alternatively, assuming construction was split between FY2000 and FY2001, the combined D & I for the two $1,000,000 amounts would be $112,000, $272,000, and $314,000 for the first three years, decreasing thereafter. The total budget savings for the two-year buildout would be only $154,000 during the first three years, with this "savings" decreasing to only $60,000 over the entire 12 year payback period, i.e., the cost to borrow $1,000,000 for a year at 6% annual interest.

11.6.1.10 Table 5: Line "New Equipment D/I "

11.6.1.11 Table 5: Line "BRAN D/I "

11.6.2 Table 6: Capital Expenses for NETS

11.6.2.1 Table 6: Line "Cabling Renewal"

11.6.2.2 Table 6: Line "Equipment Renewal"

11.6.2.3 Table 6: Line "Recabling"

Question marks are shown for future years to indicate that major remodeling of existing facilities and/or acquisition of new facilities would require additional investment in cabling infrastructure.

11.6.2.4 Table 6: Line "New Equipment"

Question marks are shown for future years to indicate that new and unknown networking projects in future years could require the purchase of new networking-equipment.

11.6.2.5 Table 6: Line "BRAN"

11.6.3 Table 7: Projected NETS Budgets vs. UCAR Budgets

The Gartner Group, Inc. reported in the February 1996 "Data Communications" journal that the average annual corporate networking support cost per desktop is $3,275, based upon a corporate LAN cost-of-ownership model they developed. Forrester Research, Inc. reported in the May 1996 "Data Communications" journal that each networked desktop costs about $8,000 a year to maintain, with that amount expected to increase as networks become bigger and more complex.

Central UCAR network support cost will be $781 per network-device in FY1999. Even with increased costs, by almost any measure, networking support at UCAR is a bargain compared to industry estimates.

11.7 Summary of Future Budgets

12. CONCLUSION TO STRATEGIC PLAN

The fundamental potential of electricity to dramatically improve productivity wasn't released until the network of leather belts was replaced with a network of electrical cables powering individual electric motors on individual machine tools. The competitive advantage of this new system was so great that those not making the shift eventually went out of business. However, it took vision for those first industrial leaders to make the large investments required to shift to the new paradigm. Because there was no formula that predicted or guaranteed success, these leaders made a strategic change based on their judgment that that such a change was required.

Data communications networking is the electricity of the 21st century, and it's necessary to replace UCAR's leather-belted network so that the required power can be supplied to the desktop engines of UCAR's information workers of the new century. UCAR should move forward with a state-of-the-art networking foundation to support its 21st century information technology requirements, and thereby assure that it can continue its scientific and community leadership roles in a world in which sophisticated electronic data access and dissemination is increasingly fundamental to an institution's intrinsic community value.

To these ends, the following five strategies regarding UCAR networking have been identified as being the most critical for meeting the required objectives:

1. Assure that the network cabling for all of UCAR is capable of supporting switched Fast Ethernet (100-Mbps) to all UCAR workspaces by the end of FY2000.
2. Before the end of FY2001, transition from Ethernet (10-Mbps) to switched Fast Ethernet (100-Mbps) as the standard network service available to all UCAR computers.
3. Renew one-third of all UCAR network-equipment each year by replacement or by upgrade.
4. Renew the UCAR network LAN cabling plant every 10 years by replacement or by upgrade when it is apparent the cabling plant needs to be upgraded.
5. Continue to actively participate in metropolitan, regional, national, and international networking initiatives.