Back on track to the NII?

by Pat Burns, Ed Sharp, and Darel Eschbach

Burns

Pat Burns, above, originally presented these ideas in a talk at the Colorado Computational Science Fair, hosted in May 1996 by NCAR.

This article presents our opinion of how we have gotten into our present sad state regarding networking in this country. We have organized the information into the following sections:

All the acronyms used in this article are expanded in an accompanying glossary.

What is the NII?

The NII is the National Information Infrastructure, the activity the government tends to term the "Information Superhighway." The everyday network user thinks of the NII as the Internet. WolffIndeed, Steve Wolff (photo, right), ex-director of the National Science Foundation's (NSF) Division of Networking and Communications Research and Infrastructure, has stated publicly and definitively, "The Internet is the NII."

The important aspect here is that currently, the Internet is all we have for our National Information Infrastructure. Our ability to intercommunicate nationally relies solely on the Internet, and our future relies on its working--and working well.

Why networking is important in education

It is axiomatic that networking has been of tremendous benefit to higher education. Today faculty, staff, and students expect excellent Internet connectivity. In fact, the Internet is often used to recruit faculty and students.

The Internet is an enabling technology, providing access to:

Hypertext--traversal of an information "tree" constructed by the author(s)--provides many ways of presenting information, permitting tremendous latitude in the way one learns. Indeed, it permits what some term "hyperlearning." Multimedia electronic textbooks are beginning to emerge, available for free (e.g., http://csep1.phy.ornl.gov/csep.html), that provide motion as well as interaction. Forward thinkers in higher education perceive that we have completed only half of the transformation into the modern information age; the remainder of the transition will involve multimedia.

The next great need is for multimedia applications, including personal desktop videoconferencing. Recently, traffic statistics were taken on the Fiber Distributed Data Interface (FDDI) ring at Moffett Field, California--a national exchange point--over a ten-minute period. At that time 16 of about 400,000 total Internet Protocol (IP) sessions were CU-SeeMe sessions, and they were observed to consume between 1% and 2% of the total capacity. This illustrates the incredibly high capacity required for multimedia.The infrastructure of and "behind" the Internet (i.e., Wide and Local Area Networks--WANs and LANs) will require tremendous upgrading before we can fully use the Internet for multimedia. However, the impact upon learning is likely to be so far-reaching that it is incumbent upon us in education to pursue this next step aggressively.

The tenet of this article is that networking for higher education is in terrible shape right now. The remainder of this article explores whether our present information infrastructure will facilitate or serve as an impediment in taking this next step.


rule

How did we get into this mess?

The "good" times: The history of NSFNET

A bit of history will establish perspective for where we are today regarding networking. Recall that in 1986, the National Science Foundation (NSF) issued a solicitation for regional networks to form consortia to connect to the National Science Foundation Network, which at that time was at 56 kilobits per second (kbps). Even though few nodes then existed on NSFNET, NSF had to delay the implementation of the program due to the lack of adequate capacity in the 56-kbps backbone. We remember with some dismay those early days of the Internet: When attempting to telnet into a CRAY 2 at NASA Ames, after about a minute one would receive a response with both the login prompt and the time-out message--apparently returned in the same packet!

In 1987, NSFNET was upgraded to T1 at 1.544 megabits per second (Mbps), a 24-fold increase in capacity. Figure 1 shows the original 13 nodes. The upgrade was accomplished under the auspices of Merit, and involved MCI and IBM in the joint venture. Many, myself included, suffered qualms about the partnership--after all, IBM was known at that time for Systems Network Architecture (SNA) and not Transmission Control Protocol/Internet Protocol, or TCP/IP. However, the transition went smoothly and the regional networks became connected, for the most part, in mid-1987. During this time, users and regional networks received extremely good service from Merit, and exceptionally good relations developed among all participants. The motto at that time seemed to be, "Just get the job done."

The T1 backbone worked well for about four years, during which time the traffic grew steadily until saturation was imminent on several of its links. In 1991, Merit, MCI, and Advanced Network Services (ANS, a spin-off of IBM), undertook to upgrade the backbone to T3 (45 Mbps)--a 30- fold increase in capacity. The transition was not smooth, but the problems were solved over a period of several quarters. Initially there were 19 nodes, as shown in Figure 2. Merit retained overall responsibility for the effort, and things worked smoothly for about another four years.

Many believed that to catalyze the spread of the Internet into the private sector, commercial entities had to be involved in the activity. From the early days of the T1 NSFNET, commercial organizations were allowed to connect to their backbone, provided they were sponsored by a research or educational institution; the commercial organization had to be doing collaborative work that required Internet access. This policy required NSF to approve every single commercial connection--an unmanageable approach. Then, under the new contract for the T3 backbone, ANS began offering commercial access directly, because they had contracted with the NSF to provide only a portion of their network infrastructure--there was "spare" capacity available, which was offered by ANS to commercial entities via an activity termed CO+RE (Commercial plus Research and Education).

However, other network providers were emerging. They argued with the NSF that the ANS CO+RE activity represented unfair competition with the private sector, since the cost to ANS to provide such service was only an incremental cost--the bulk of the costs were being paid by the NSF under the contract with Merit. Rumors abounded that lawyers were retained to pressure the NSF into disallowing commercial connectivity to NSFNET via ANS. NSF's lawyers advised the NSF that there were indeed significant liabilities, and that NSF should cease and desist the activity.

One alternative was for the NSF to disallow commercial traffic on the Internet. However, it was impossible to delineate definitively the boundary between commercial and noncommercial traffic. Besides, dismissing all the commercial sites, especially those that were participating in NSF sponsored Centers of Excellence, was unpalatable to the NSF. Indeed, it was always NSF's stated intention to promote the Internet until it developed to the point of self-sufficiency, then withdraw support. A solution that seemed to satisfy all constraints was for the NSF to privatize the Internet.

The "bad" times: Privatization

A new solicitation to this effect was issued by the NSF mandating three "priority" Network Access Points (NAPs) to which consortia would connect directly or via an Internet Service Provider (ISP). A Routing Arbiter (RA) was funded to maintain a database of routes that could be used by network providers.

(More detail on the architecture is available at http://www.merit.edu/. See also "NSFNET moves toward a new network infrastructure" in the Fall 1994 SCD Computing News, and "NSF privatizes Internet" and "A new era for the Net" in the Summer 1995 SCDCN.)

To receive funding for connectivity, regional networks had to propose to the NSF to connect to a single ISP. NSF funded most regional networks at a "flat" rate of about the cost of a single T3 connection.

In Westnet, this left the region in a quandary, since four fractional T3 connections were actually needed from our six states; this exceeded our available resources. Instead, Westnet decided to implement 13 T1 connections to Sprint (more expensive than a single T3, but much less expensive that four fractional T3s and associated interconnects), based upon a competitive bid. The transition was scheduled for August 1994, but was delayed eight to nine months because two of the three priority NAPs initially did not work. It was fortunate that two Federal Internet eXchanges (FIXes) and a Metropolitan Area Exchange (MAE) existed to augment the NAPs.

Merit decommissioned NSFNET April 30, 1995 (see http://www.merit.edu/nsfnet/nsfnet.retired). The ten-year project of NSFNET had been tremendously successful, beyond the wildest dreams anyone had in 1986.

Initially, the major beneficiaries of the new architecture were the ISPs of national scope: ANS, Sprint, MCI, and UUNet; in late 1994 and early 1995 these ISPs had partial T3 and T1 networks.

However, many problems (in addition to the delay derived from the NAPs) existed in the new model. Capacity problems immediately existed in the T1 portion of the ISP networks. It took the ISPs about six months to get their circuits upgraded to the point where their networks were usable--much longer than it should have. During this time, network service was extremely poor. Also, the ISPs were using Cisco routers as their backbone routers, and capacity problems appeared in the backbone routers during peak traffic periods. It took Cisco and the ISPs about a year to resolve these problems. Additionally, over the period of the first year, the NAPs became very congested.

Due to these and other problems, there were multiple occasions when selected national networks were down for minutes (sometimes tens of minutes). This was a serious problem for regional networks, which initially had been mandated by the NSF to select a single ISP. Furthermore, the good relations and high quality of service we enjoyed during NSFNET were nowhere to be found.

In retrospect, it should have been easy to foresee and avoid some of these problems:


rule

Analysis: Why has this happened?

Some of the reasons for our present sad state of networking are obvious from the previous section. Most notably, when we should have been increasing the national network capacity by a factor of 20-30, we instead concentrated all of our efforts on privatization; this resulted in an increase in national network capacity of only a factor of 3-5. Some reasons that may not be so obvious are covered in this section.

Lack of responsibility. No single entity is responsible for the national network as Merit was before. All too frequently we see problems between national networks, mostly at the exchange points. When attempting to debug these problems we typically see circular "finger pointing." For example, provider A blames the problem on provider B, which in turn denies it is their problem and blames it on provider A. Current information and historical reports about problems are not available, as the vendors are reluctant to admit having problems at all (ergo, finger pointing), perhaps due to a possible negative impact on sales. Previously, accurate information was available from NSFNET, which would allow us to engineer appropriate remedial action. Now the problems are kept internal to the ISPs; to those of us on the "outside," the network now appears as a series of loosely interacting "black boxes."

Lack of focus. The Internet vendors serve thousands of customers, at rates from 56 kbps up to T3, with far too little support. Previously, the structure of NSFNET was hierarchical--the managers of NSFNET had to deal with only 19 regional networks, who in turn dealt with their users. This made support manageable for the backbone network engineers. Good working relations were developed among a small community of dedicated knowledgeable personnel. Today, ISP backbone engineers deal with thousands of customers, some of whom have little technical expertise. This problem is exacerbated because there are more problems today, and on average each is more difficult to isolate and resolve due to the increased complexity.

Profit. Vendors cannot afford to upgrade their networks by a factor of 30 to accommodate future growth only to have their utilization be low for the first year or so. Instead, we have seen some vendors delay upgrades far past the time when needed, and then upgrade capacity by only a factor of 2-3.

Inability to react quickly. It is not in the telcos' culture to plan for such massive growth as exists on the Internet. One ISP has reported 7% growth per week (a factor of 31 annually)! Indeed, it may be beyond any company's ability to plan for and react to this incredibly high growth rate.

Lack of vision. Personnel at the ISPs cannot see why much higher network speeds are required, and they expect us to manage our growth internally. In fact, we in education have been accused in a public meeting by an employee of a major ISP of frivolous use of the Internet. The case in point was an accusation that some users in education transmit real-time "video clips of trees growing." Philosophically, the point is that until the end user pays directly for Internet access (just as we do for telephone service), we can expect to have frivolous usage.

We can see no way have the end user pay directly for Internet access (termed "charge back") without counting every packet--an impossible task due to the huge volume of traffic among millions of users. Also, restricting usage during the early phases of a new technology is just what we as educators should not do at this time, as it will have a negative impact upon learning before we can assess the true, long-term potential upon learning.

One must question how successful the transition to the private sector has been. It is my belief that it has been extremely successful for the ISPs (those who originally pressured the NSF for privatization). However, we are taking the viewpoint of higher education. The quality of our networking has severely degraded, and costs have risen dramatically. Stated unequivocally:

The transition to the private sector has not been successful in maintaining a robust, high-quality network at prices that the majority of higher education can afford.


Strategies for self-protection

Problems

Four significant problems with networking we now face are:

  1. Problems in the ISP backbones and at the exchange points
  2. Understanding the network and its problems, utilization, and growth
  3. Insufficient capacity in the tail circuits "behind" national networks
  4. Pricing Six months ago, the major problems were (1) and (2) above. Significant upgrades of the NAPs and the ISP national networks have occurred since then, so that problem (1) is not currently the most significant. Indeed, additional exchange points are emerging among the vendors. Thus, the greatest problems are now (2), (3), and (4). Most of the problems we see currently arise due to multiple vendors being involved (the "finger-pointing" problem). Also, we continue to observe congestion on many links to individual sites and to aggregates of sites.

    These problems arise for several reasons:

    • The ISPs have been slow to process orders--in some cases because they have had insufficient capacity in their backbones, in other cases for other reasons.

    • The cost of connecting to the Internet has risen dramatically, and at T3 speed is usage-based. (As mentioned previously, one vendor now charges $780,000 per year for the highest traffic level at T3 speed.)

    • In some cases, individual sites do not know what their traffic level is (since we now have no monitoring of traffic), so they cannot plan and budget quickly enough to avoid severe congestion.

    Solutions

    The first obvious strategy is to aggregate traffic behind a single connection in order to minimize cost (without overloading the connection).

    A second strategy is to minimize the traffic that must be exchanged among vendors. This is accomplished by purchasing Inter-Regional Connectivity (IRC) from the vendor with the most customers.

    A third, related strategy is to purchase services from multiple vendors so that a network outage on a single ISPs backbone will not be debilitating. In this case, the two different sites should be connected to different vendors' backbones and interconnected at the same speed as the connections to the backbones. Traffic to each vendor should flow directly to that vendor (possibly across the high-speed interconnect) so as not to traverse an exchange point. Of course, to maintain good service during an outage, each IRC connection should be capable of carrying the full traffic load. This means that under this arrangement, the cost essentially doubles.

    Finally, all sites that are "local" should be well connected, perhaps through a local NAP, so that local traffic need not traverse a national network. This provides protection from a backbone outage and problems at the NAPs. It also reduces IRC cost by reducing the traffic to the ISP.


rule

How do we get out of this mess?

Note that some of our networking problems can be solved simply with more money. This represents a significant problem for higher education, as financial resources are not plentiful in higher education today.

The more difficult problems, such as interactions among vendors, are cultural and may not be resolved by market forces. Indeed, supply and demand are supposed to result in a "fair market value."

Unfortunately, the demand for networking services continues vastly to outstrip the supply offered by the vendors. Not surprisingly, prices have risen dramatically (by a factor of 2-3 in the last year), with no slacking off due to a decreased demand. We in higher education have been told that we are networking's "problem children" because we complain about quality of service far more than other sites and are unwilling to pay as much (i.e., are unwilling to pay fair market value).

Two important questions are:

The answer to the first question, in my opinion, is: "Not for at least two to three years."

The answer to the second question is now unequivocally, "No. We cannot even afford the prices we are being charged now." We would be delinquent in our responsibilities to our institutions were we not to consider what the real costs are for installing, operating, and maintaining a national network. It may be far cheaper (and better) were we to do the job ourselves.

Assessing the costs of networking

To assess real costs for networking is difficult. Winston Churchill's quotation about Russia is appropriate here: "This is a puzzle, within a mystery, all wrapped up in an enigma." The ISPs claim they are losing money at current pricing. Critics from higher education state that this may be due simply to internal accounting. To understand this requires some knowledge of corporate structure.

First of all, most ISPs are owned by an Inter-eXchange Carrier, or IXC (a long- distance telephone company or telco) parent company. All the IXC parent companies have created separate subcorporations for networking (the ISPs). Each ISP purchases circuits from circuit subcorporations within the parent company at rates set by the circuit subcorporation. If the circuits are highly overpriced, then indeed it is possible for the networking subcorporation to be losing money and the parent company to be making money overall.

The problem is that we don't have any idea how much it costs to provide high-speed digital long-distance circuits. We have seen digital circuits provided by Competitive Access Providers (CAPs) at one-third the cost of the circuits available from the IXCs, and circuit costs represent the majority of cost for a national high-speed network.

By undertaking a design exercise for a national network, higher education could determine real costs, provided "real" circuit costs can be obtained (still an issue in question, but at least fair market value for circuits could be determined).

However, higher education has rarely if ever executed projects of this scope by itself. In 1986, it was NSF, by issuing solicitations for the backbone and for regional networks, that acted as a catalyst for this to happen. In our opinion, the federal government should once again undertake this cause; however, the legal issues mentioned above may be insurmountable.

An answer from IBM?

Fortunately, on April 19, 1996, IBM stepped forward and offered to undertake this exercise jointly with the National Telecommunications Task Force (NTTF, an Educom activity) and the Federation of American Research NETworks (FARNET). Immediately the national attitude changed from gloom and doom to wonder and skepticism.

The thrust of the IBM proposal is to build the next-generation research and education network, initially for higher education only, with "no holds barred" on voice and video. Multiple vendors are to be involved, with a commitment to interoperability. Funding is expected to derive primarily from universities and federal research labs, with start-up funding from the NSF. Technology donations from the private sector are expected. Initially, the network is to be constructed at Optical Carrier-3 speed (OC-3, or 155 Mbps) and involve a voice-replacement strategy with a radical video infrastructure.

IBM is ideally suited to provide industry leadership for the partnership because of their existing relationships with the telecommunications industry, their relationships with other industries and vendors, and their advanced video and voice applications. A major emphasis initially will be management and diagnostics. Early application and management/diagnostic pilots are anticipated, together with a next-generation network pilot project.

The projected stages of participation are that there are about 35 universities that will initially buy from the package, although about 350 need immediate improvement in networking. It is anticipated that 1,000 universities will participate by the year 2000. The project eventually may be extended to other not-for-profit entities (such as K-12, research, government, and communities), although this point remains moot.

To date, an organizational steering group has been formed under the auspices of the NTTF and IBM. NSF has been an active participant. A meeting called by NTTF/FARNET took place 7-9 August 1996 in Colorado Springs, Colorado, to involve the broader higher education community.


rule

The IBM proposal: Elements for success

This section is devoted to comments on the proposal by IBM. First, we are exceptionally pleased that IBM has stepped forward with this proposal. IBM accomplished the network engineering and design for NSFNET and performed superbly in that activity. We have thought deeply about whether there is another company we would trust to do this important job, and we have come up empty.

Those of us in higher education in Westnet unequivocally and enthusiastically endorse the IBM proposal. Some of us regard this as our last chance to achieve high-quality, high-speed networking at an affordable cost.

However, we do have some minor suggestions that may be beneficial to the effort:

  • Merit should be involved, due to their valuable experience in running NSFNET and because they are trusted implicitly to represent the interests of higher education.

  • The design must result in real circuit costs, and many alternative circuit vendors should be encouraged to bid on the effort. All costs should be made public so that we as a country understand and accept them. If networking does indeed cost the amounts the vendors are charging today, then we need to accept and budget for these costs. This can proceed with a clear conscience, bolstered by our knowledge that these costs are indeed "real."

  • A restricted Acceptable Use Policy (AUP) must be implemented and enforced so that we avoid the legal issues mentioned previously. Participation by for-profit entities must not be allowed.

  • The project must permit consortia of higher education to participate. Cost sharing of large expenses makes this network affordable by many institutions, as opposed to only the elite institutions.

  • A design at OC-3 speed should be considered only as a first step. We believe the project should deploy OC-12 (622 Mbps) as soon as practicable.

  • Policy committees should be formed to determine the parameters to which all universities must agree in order to participate. For example, one condition for participation might be that each university must provide a specific number (say 24) of dial-out circuits on its Public Branch eXchange (PBX, or phone switch) made available to the other participants. Another might be the parameters for connecting institutional video. Yet another might be the terms and circumstances under which traffic throttling will be done, and whether any guaranteed bandwidth will be available to participants.

  • NSF should aggressively support this endeavor. In particular, NSF should determine the terms and conditions under which it could migrate the VBNS services to this activity.

  • Higher education must rally around this activity, endorsing it wholeheartedly and unambiguously to state and federal governments. All too often, we in higher education have been guilty of sending uncoordinated and ambiguous messages to our governments.

  • Time is of the essence. This project must be given the highest priority. There are terrible problems now with networking in this country. Each day this effort is delayed negatively impacts learning and research in higher education.


rule

Summary

We have argued that the existing National Information Infrastructure (NII) will be either of insufficient quality or too costly to support the next generation of multimedia information that is so critical to the future of higher education. Since the NII has been privatized, we have experienced that we can influence the evolution of the NII very little, if at all. Therefore we have no choice but to explore a separate infrastructure, an Educational Information Infrastructure (EII). The proposal by IBM is auspicious in this regard. In fact, we could well have titled this article, "On Track to the EII."


Acknowledgments

We gratefully acknowledge the efforts of David C.M. Wood and Chris Garner of Westnet, who reviewed this article. This is but one of the small ways in which they and dedicated others like them have made networking in this country flourish.

Back to contents