Large-scale transportation networks such as airport systems or interstate highways typically are built around the concept of hubs, which are strategically located to maximize the effectiveness and reach of the network.

If such a network were used to transport IP packets, however, then the "hub" would also be known as a point of presence (POP). POPs form the foundation of today's IP infrastructure; the design of the POP influences the overall scalability and services mix of a service provider's network.

POPs have evolved from the early 1990s, and some key technologies and trends will influence future POP architectures. Service providers must employ these POPs to build a robust IP infrastructure.

The early days The early Internet consisted of a set of regional networks that connected U.S. research and educational facilities to share primarily supercomputing resources.

The regional networks, in turn, connected to each other at network access points or Internet exchange points (IXPs), as they are known today. The location of the IXPs coincided with the major traffic junctions of the networks that constituted the Internet. After these networks were privatized and consolidated, most of the large service providers enabled access to the Internet through established voice network facilities. Therefore, the early POPs were co-located within central offices at major traffic centers nationwide and were interconnected using ATM permanent virtual circuits (PVCs).

The phenomenal growth of the Internet fueled an explosion of service providers at all levels of the Internet, resulting in a hierarchy of providers. Tier 1 service providers are mostly facilities-based service providers that deploy nationwide (in some cases, international) networks, with extensive coverage of most traffic centers. Tier 2 and Tier 3 providers maintain non facilities-based networks frequently located in specific regions or metropolitan areas.

The Tier 1 service provider infrastructure settled around the concept of six to 12 major POPs, which aggregated traffic from a large number of secondary POPs that covered most major cities.

Most Tier 2 and Tier 3 service providers buy access from the larger Tier 1 providers to obtain access to other providers and to the Internet, with connectivity initially concentrated around the various IXPs. With the exponential increase in IP traffic and the number of service providers, congestion at the IXPs has increased steadily. This scenario has driven large providers to establish private peering arrangements at key POP locations to bypass the congested IXPs (Figure 1).

The traffic transported initially consisted of simple data transfers and applications such as simple mail transfer protocol, Telnet and FTP. With the advent of HTML and the World Wide Web, however, the Internet was transformed in terms of scale and content.

The framework Routing protocols play a key role in all POP designs. The IP control plane consists of two types of protocols: interior gateway protocols such as open shortest path first (OSPF); intermediate system-to-intermediate system protocol; enhanced interior gateway routing protocol; and external gateway protocols such as border gateway protocol version 4 (BGPv4).

The routing architecture revolves around the concept of autonomous systems, which are defined as entities with an independent routing domain such as large enterprises and service providers. Each of these entities is assigned an autonomous system number that uniquely identifies it in the global IP routing architecture. In general, an interior gateway protocol such as OSPF builds a database of the routers and IP prefixes within a provider's network and takes precedence in packet forwarding decisions with an autonomous system. BGPv4 provides external connectivity to the autonomous system by externally announcing all the prefixes known to the autonomous system and receiving similar information from other autonomous systems.

As the interior gateway protocol provides internal connectivity between the routers in the service provider's network, each router in the network must use the interior gateway protocol. Good network design seeks to minimize sources of instability; therefore, customer-controlled customer premises equipment (CPE) devices are excluded from the provider's interior gateway protocol domain to isolate the provider network from CPE instability.

BGPv4 is deployed in two instances: internal BGP and external BGP. Internal BGP is used to propagate prefix information from other autonomous systems within the provider's autonomous system. Routers participating in internal BGP are typically backbone routers, which connect to other POPs, and routers connected to other autonomous systems such as peering and some aggregation routers. External BGP runs exclusively on routers connected to other autonomous systems to exchange interprovider routing information.

Evolution of the IP POP The architecture of a generic POP includes a backbone layer, an aggregation layer and, in larger POPs, a peering layer.

The backbone layer consists of routers with high-speed interfaces connecting the POP to other POPs in the service provider's network. Aggregation routers offer connectivity to customers, including other service providers, enterprises and Web hosting providers. Peering routers are used for inter-provider connectivity and connections to IXPs.

The function of a POP is to bring in customer traffic via aggregation routers and to transport it to other POPs via backbone routers or to peering routers that offer connectivity to other service provider networks. The final component of the architecture is the POP interconnect (within the POP) that uses packet-over-Sonet, gigabit Ethernet or ATM interfaces to connect the various layers together.

The generic POP underwent a significant redesign with the rise of e-commerce and the growth of a new industry of content providers such as Yahoo and Amazon.com, which attracted huge amounts of traffic to their Web servers.

To handle this traffic, some content providers co-located their Web servers within the service provider's POP, while others built their own networks and connected to the service provider networks as customers. These changes resulted in the rapid growth of aggregation routers in the POP to connect the content providers, as well as the addition of Web servers; this rapid growth of aggregation routers, in turn, created a new type of POP called the Internet data center (Figure 2).

The availability of rich content and services pushed access speeds beyond T-1 to T-3, OC-3 and, in the case of application service providers (ASPs), OC-12 and OC-48. This increase in access speeds increased traffic rates at the edge of the POP, thereby driving the backbone speeds from OC-12 to OC-48 rates and, eventually, to OC-192.

One of the key factors in POP design is the "over-subscription" factor, which determines how much access bandwidth can be offered for any available backbone bandwidth. For example, a typical over-subscription factor of one-to-three implies that for every 10 Gb/s of deployed backbone bandwidth, 30 Gb/s of access bandwidth can be offered to customers.

Driven primarily by the services mix offered in the POP, this ratio varies among service provider networks. To help scale the speed of the POP interconnect in relation to the growth of access and backbone bandwidth and the over-subscription factor, service providers chose newer POP interconnects, including packet-over-Sonet dedicated access to backbone bandwidth - or gigabit Ethernet, which offered cheaper, multiaccess bandwidth.

POP interconnects affect the robustness of POP designs because they play a key role in determining how many intra-POP paths are available for an aggregation router to reach the backbone or peering routers. Scaling the POP interconnect is a challenge for service providers today as they grow the POP to accommodate more customer demand.

The essence of good POP design lies in choosing the technology and systems that scale gracefully. For instance, the internal architecture of backbone routers deployed in the Internet has evolved over time. Distributed system architectures have demonstrated better scalability and services flexibility than rigid centralized systems based on complex shared-memory architectures. Systems with distributed architectures eliminate single points of failure and distribute the forwarding function to provide lookup characteristics to handle the real Internet packet mixes, resulting in routers whose complexity and performance scale linearly with increasing traffic.

Service providers must pay close attention to system architectures given the increasing deployment of latency-sensitive applications such as IPv6, the newest version of IP that has 128-bit addressing, offers new security features and supports real-time communications and multicasting.

Scaling the POP infrastructure In the traditional POP, interfaces on the backbone routers were based on the broadband ISDN model of Sonet+ATM+IP. The interfaces connected to transcontinental Sonet rings provided the Layer 1 connectivity while ATM switches provided the Layer 2 transport for IP and existing frame-based services such as frame relay and ATM. PVCs provided over-lay networks for service providers sharing the same Sonet infrastructure. IP at Layer 3 was treated as another traffic source, which was overlaid on top of the ATM+Sonet infrastructure.

The rapid growth of IP traffic, however, soon exposed the limitations of the time division multiplexing (TDM)-centric ATM+Sonet model: its inability to scale beyond OC-12 rates and the excess header overhead it imposed. In addition, characteristics of IP traffic, its connectionless nature and the robust transport layer protocols such as TCP meant that IP applications worked well without requiring the granular quality of service features and the TDM-like connections of ATM.

Packet-over-Sonet allowed service providers to eliminate the ATM layer in favor of transporting IP directly over the Sonet infrastructure. Packet-over-Sonet interfaces on high-end routers allowed IP routers to be directly connected to the Sonet transmission systems via concatenated, statistically multiplexed interfaces at OC-12/STM-4 and OC-48/STM-16 speeds for the first time.

Today, the IP+Sonet architecture is deployed among service providers with Sonet infrastructures, particularly those that provide leased-line services in addition to IP services over the same network.

Another key IP transport technology in the POP is Ethernet. The de facto choice for access has become 10/100 Ethernet, with millions of PCs using it for connection to the Internet. Similarly, gigabit Ethernet is making substantial inroads as the interconnect of choice for content hosting where server farms of work stations need low-cost and data-friendly interfaces for attaching to routers within POPs.

With a standard for 10 Gb/s Ethernet expected from the Institute of Electrical and Electronics Engineers next year, the stage is set for gigabit Ethernet to not only scale the POP but also move outside the POP and potentially displace some Sonet-based transport in the metropolitan area network and WAN. The attraction of this model is the expected lower cost of gigabit Ethernet relative to OC-192 Sonet interfaces. To what extent gigabit Ethernet will be deployed in the WAN is yet to be seen; it depends as much on new functionality for managing 10 Gb/s Ethernet in the WAN operations, administration, maintenance and provisioning as it does on the interface cost.

The optical revolution Although the Sonet infrastructure offered reliability for transport of traditional voice and frame relay services, it proved to be slow to provision and difficult to engineer with the constantly changing IP traffic patterns. Bandwidth inefficiency is also a key concern because Sonet dedicates about half its bandwidth to restoration. Moreover, many IP customers are more than willing to use their own Layer 3 restoration techniques that offer rapid, service-level restoration times more applicable to their IP-based applications.

With the arrival of dense wave division multiplexing (DWDM) technology and the availability of OC-48c, OC-192c and line cards on backbone routers, service providers can connect their backbone routers directly to the fiber via transponders in the DWDM transmission systems. This scenario resulted in the total elimination of the TDM-oriented Sonet equipment in predominantly IP backbone networks (Figure 3).

The cost of interconnecting routers to DWDM equipment has been further reduced by the introduction of very short reach optical links that can cost-effectively interconnect equipment within 400 meters in the POP at OC-192 rates. DWDM offers two major benefits: It can accommodate the scalability required by IP, and it can further operational efficiencies in service provider networks by allowing them to add bandwidth on a point-to-point basis without having to re-traffic engineer the network as was frequently required in ring-based networks.

Accepted today as the transport technology of choice, DWDM is widely deployed in most national-scale, infrastructure-based service providers. During the last few years the number of wavelengths deployed has increased from about four to 40, and DWDM vendors now are marketing systems with up to 160 wavelengths.

But DWDM offers more than near-limitless capacity. Soon DWDM-based transport systems are likely to add two new technologies: ultra long-haul capabilities and optical cross-connects (OXCs).

Ultra long-haul capabilities will extend the reach of DWDM from a typical range of 500 kilometers to more than 3000 kilometers before requiring expensive electrical signal regeneration. This reach capability will allow POPs on the West Coast to be connected directly at the optical layer to POPs on the East Coast with no intervening electronics and, hence, lower interconnection costs.

OXCs offer service providers a new tool for managing the enormous bandwidth offered by DWDM at the wavelength level. They allow networks to migrate from the ring topology of Sonet toward a more flexible mesh topology. Mesh-based networks - where bandwidth is deployed on a point-to-point basis as needed - offer service providers increased flexibility, simpler provisioning and better bandwidth efficiency.

As the optical layer grows to include connection management through OXCs and other connection-oriented elements, control of the optical layer becomes important. First generation solutions demonstrated useful benefits such as rapid provisioning and mesh-based restoration but by necessity have been proprietary implementations with limited interoperability. Standardization activities currently are under way in a range of standards bodies and forums, including the Internet Engineering Task Force, Optical Internetworking Forum, Optical Domain Service Interconnect and ITU-T.

Two approaches are emerging. One is based on an overlay model in which the optical layer is an opaque cloud, and upper-layer devices such as routers are treated as clients to the optical layer. The other model is the peer model, in which IP routing algorithms are extended to provide for control of the optical layer.

The peer model is the most general and supports a spectrum of implementations that not only allow upper-layer devices to request connections from the optical layer but also allow for the potential for upper-layer devices to select their paths through the optical layer. The model that prevails will again be determined by the market in the coming years. Whichever form the optical control plane takes, it is likely to be an important technology for coordinating and integrating the optical layer with the routing layer in next generation networks.

On with the service In response to phenomenal growth in their customer base, increased competitive challenges and the need to offset rising infrastructure costs, service providers began considering services flexibility as a key design criteria for their POP architectures. E-commerce, combined with the ubiquitous reach of the Internet, enabled service providers to offer enhanced services at the POP, including virtual private networks (VPNs), voice over IP and content delivery. This capability has allowed service providers to turn their massive infrastructure investments into revenue. A range of new technologies such as class of service, IP multicasting and traffic engineering facilitated the deployment of these new services on the IP infrastructure.

Class of service, which is deployed today using the three precedence bits in the IP header, allows service providers to treat different types of IP traffic with different levels of priority as they are forwarded by the routers. One of the key features that enables class of service is committed access rate, which is primarily a packet-classification mechanism deployed on the aggregation routers that face the customer.

Committed access rate can be used to mark the precedence bits to differentiate between different applications and source IP addresses. When marked, the packets are forwarded into the network where backbone routers enforce the class of service using weighted random early detection (WRED) and modified deficit round robin (MDRR) algorithms.

WRED uses the fact that most of the Internet traffic is TCP, and it selectively drops TCP packets based on the precedence bits set when congestion occurs. By using the sliding window behavior of TCP in congestion environments, the pre-emptive dropping of packets is used for avoiding congestion.

MDRR is basically a queuing mechanism implemented in routers to prioritize the way a router queues packets. Once again, the precedence bits are used by MDRR to queue higher-priority traffic ahead of the lower-priority traffic. Class of service is expected to expand beyond the present eight classes supported by the precedence bits when the DiffServ standard is finalized by the IETF.

IP multicasting is another key technology that offers the potential to deliver content to multiple users in the Internet in a scalable and efficient manner. By setting up multicast groups in which multiple receivers can subscribe to an IP address, this technology builds customized topologies overlaying the IP infrastructure.

IP multicasting, however, strains the routers in the infrastructure because it requires packet duplication; therefore, it is important for service providers to deploy systems with multicast-efficient architectures. The dynamic nature of Internet traffic is driving some service providers to try to optimize the traffic flows between POPs in the backbone. Traffic engineering allows packets to be routed around hot spots in the network by using pre-provisioned paths in the backbone; it also offers flexibility in managing long-haul link bandwidth. Key technologies that enable traffic engineering include multi-protocol label switching (MPLS) along with modifications to traditional interior gateway protocols, such as IS-IS and OSPF, and path setup protocols, such as resource reservation protocol.

The wide reach of IP networks and the associated superior economics also are driving service providers to deploy traditional real-time services such as voice and video across the infrastructure (Figure 4). Other service models include offering enterprise customers MPLS-based VPNs with bundled voice and Internet services.

Realizing this vision and offering technologies such as class of service, IP multicasting and MPLS-based VPNs, however, requires the deployment of distributed routing that offers a range of services with strong performance that will not degrade as services are turned up, unlike centralized architectures.

To build a robust IP infrastructure, service providers must deploy scalable POP architectures that can accommodate the exponential IP traffic growth rates. This deployment requires inclusion of the newest advances in the optical arena, including ultra long-haul DWDM systems, optical switches, very short reach optics for OC-192 interfaces and 10 Gb/s Ethernet that will enable flexibility and scalability in a traditionally rigid transport infrastructure. It also requires incorporating into the future POP rapidly advancing high-speed routers with robust, future-proof, distributed architectures and interfaces for direct connectivity to the optical infrastructure.

Finally, it is important to design flexible services so that service providers can offer revenue-generating, value-added IP services without compromising efficiency and performance. This unique mix of growth, technological advance, customer demand and services is driving scalable POP architectures, and this scenario will enable a services-rich next generation IP infrastructure.