Simplicity sold the world on IP-based networks: all the world in a data stream. But well enough could not be left alone, and the world, wanting more, heaped new complexity into the stream and left test companies to sort things out.

Any notion service providers or their vendors had that network management would be on a lesser order of magnitude in a packet environment must have been — on whatever scale of magnitude used — a short-lived thought. IP packet-based networks on their own may be easier to manage, but the growing number of applications and services they support are proving to be quite a handful.

Traditionally, test equipment vendors have worked at the network's physical layer to test, measure and correlate metrics on the performance of the circuit, the network elements and the signaling gear. They since have moved up the stack to begin providing service layer metrics. But all of a sudden, services are beginning to work in tandem. Wireless and wireline services have coalesced into fixed/mobile convergence (in some cases); voice and data have commingled in each other's mailboxes and unified communications (UC) systems; and video has emerged as the new voice and is being incorporated into an interactive broadband session through our TVs.

And the network that many, but not all, experts agree will support these interrelated services, IP multimedia subsystem (IMS), has blossomed into a broader array of network elements than first imagined. Add in the emerging capability for location- and presence-based services, and you have quite a complex IP packet network.

Test companies face many challenges in delivering service assurance solutions to providers engaged in these complex services — which is all of them. They range from the seemingly simple, such as where to put a test probe, to the obviously complex, such as how to track hundreds of thousands of real-time presence updates.

“The first challenge is where to stick probes into the network,” said Chad Hart, product marketing manager for Empirix, a test and monitoring provider from Bedford, Mass. “The monitoring system is only as good as the traffic it sees, and if you put your probe in the wrong spot, you won't see the traffic.”

Hart has written a book, “Ensuring a Quality IMS Experience: A Practical Guide to Testing and Monitoring IP Multimedia Subsystem and Services,” in which the complexity of IMS and the challenges it poses to test companies tasked with validating it become apparent. In it, Hart acknowledged that the main driver of IMS was to simplify the deployment of new applications and services: “However, from the quality assurance perspective, the challenges of increased complexity, standards ambiguity, multivendor interoperability and increasing quality of experience expectations are amplified at the service level.”

Hart cites UC as an example of a challenge not just for test vendors, but for service providers as well. “When you have multiple entry points, you need to have a probe that can cover all those points of entry. It sounds simple, but in a real network architecture, you might have a dozen entry points, and you might not be able to afford a dozen probes,” he said.

And that's just a question of where to stick the probes. UC is one of the services that purports to make use of presence data, and presence adds a whole new dimension to monitoring service quality. Historically, when monitoring a call or session, testers had to worry only about the signaling for setting it up or tearing it down. With presence enabled, messages are being sent all the time regarding a user's location and availability.

“Obviously, you want to see those presence updates and you need to know how to interpret them, but the more messages you have, the more difficult it becomes,” Hart said. What's needed, he said, is a correlation engine that can correlate messages across multiple hops in the network.

However, presence is done primarily through signaling, and there is plenty of industry expertise in testing and monitoring signaling. Presence just adds another order of magnitude. And while quality measurements can be gleaned from signaling data, presence itself is not an application — it enhances other applications — and so test companies are still left with the challenge of testing the quality of actual applications and services themselves.

Why is that so hard? As Mike Stoos, director of business development in the service assurance group for Spirent, said, “When you start talking about video applications, on-demand applications, interactive applications and combinations of the three, we don't even know what the problems are yet.”

While test companies have made headway with video and other content applications and have test tools in their portfolios to prove it, there are still questions about what exactly to measure to ensure a quality experience. “What do you need to measure?” Stoos asked. “Where do you need to look?”

He said there are two kinds of applications: those that users can't live without for which they're willing to endure the pain of early adoption and those that better be rock-solid when they are delivered to the customer. IPTV, he said, better be rock-solid.

“With IPTV, they won't get the second chance they have had with other technologies,” Stoos said. “It's not just about losing a customer if things don't work; they could lose a whole neighborhood, and there's no recovery from that.”

At Spirent, Stoos said, there are three philosophies about measuring IPTV performance. The first is to test it in the network core — that's where the early focus was. The second is to get as close to the customer as economically possible. “That's where we come in,” he said.

Spirent's SmartSight test system measures to the drop side of the DSL in a fiber-to-the-node configuration. It tests physical layer metrics such as the readiness of the DSL loop, but it also tests the customer experience. “I don't care if you are getting good network quality if that means you are getting good video but can't hear anything,” Stoos said.

To accomplish this, Stoos said a hybrid third philosophy is taking shape that tests both content and the core. That includes all the interactive features promised in IPTV such as voice-on-demand, caller ID, click-to-call and merchandising — the ability to click on an advertisement and link to a company's Web site to purchase goods.

All this seems to answer questions about where to look and what to look for; however, test companies are working, as they often do, with emerging technologies that are long on features but short on standards. So the answers for where to look and what to look for could change.

A best-practices white paper on IPTV testing and measurement, released by Alcatel-Lucent in May, offers a definitive answer to these questions. Although the white paper was derived from work done on a Microsoft IPTV Edition software platform, Alcatel-Lucent's goal was to define a recommendation for three operations/engineering decisions for service assurance that would be generic to any video platform.

The decisions will determine a definition of the generic demarcations where measurements should be made in a live IPTV network, recommendations on measuring methodologies and recommendations of a minimum set of measurements to gain visibility and troubleshoot IPTV quality issues.

Alcatel-Lucent worked with Agilent, IneoQuest, JDSU and Spirent to formulate these definitions. They did so by creating what they called an IPTV service assurance calibration environment.

They determined that the key measurement demarcations for probes are at the headend in the serving home office, the transport network, the last mile, the DSL loop and in the home. These demarcation points may seem intuitively obvious, but as more content providers and more third-party applications join the party, the number of demarcations, both logical and physical, can skyrocket.

While standards bodies hash out which definitions they'll use, Stoos said that at each point, service providers will be looking to test all three instruments of quality: network performance, content and signaling.

Brix Networks supplies part of the hybrid solution through its service assurance probes and has been active in the standards work. The company can plug its probes just about anywhere the industry decides they should go and has been focusing its standards efforts on helping to define quality. Kaynam Hedayat, the company's chief technology officer, attended an Internet Engineering Task Force meeting in Toronto last month where he pressed for the definition of the media quality index (MQI) standard. MQI is analogous to the mean opinion score for traditional voice services in that it would establish a metric based on subjective measurement.

Another standard Brix supports is one currently used for voice over IP (VoIP) that the company feels can be applied to IPTV as well. The RTP Control Protocol Extended Reports (RTCP XR), of which Hedayat is co-author, currently defines a set of standard metrics for assessing VoIP call quality and diagnosing problems. John Burnham, vice president of marketing for Brix, said RTCP XR is a well-established standard for VoIP supported by companies such as Linksys and Texas Instruments, and other companies are getting behind it for IPTV as well because it is a lightweight client that provides a lot of rich information about performance.

“Because our origins are in IP, we have been about standards since Day 1 during our days of the LAN wars,” Burnham said. “And IPTV exists as a service based on the availability of an open network, open systems and open standards.”

He said a lot of the interest in end-point monitoring — the kind that can take advantage of standards such as RTCP XR and TR-069 — is driven by IPTV and VoIP. “These services will be heavily standards-dependent because as the subscriber numbers reach the millions, it will no longer be economical to put dedicated measurement gathering devices on the premises [along with the set-top box in the case of IPTV],” Burnham said.

That's why companies are pushing hard for standards. When the big push comes for IPTV and the IP-based services that will ride along with it, service providers will need that third component to go along with core and content monitoring; they will need end-user experience monitoring and other standardized test tools that make testing and monitoring IP-based applications and services as easy as checking for a dial tone.

DEFINING THE DEMARC

Alcatel-Lucent's definitions of the four major demarcation points for testing and monitoring an IPTV service:

First monitoring point:
THE HEADEND

Between the traditional cable headend and the network interface, typically between the MPEG 2/H.264 encoders and the video software platform

Second monitoring point:
TRANSPORT

At the VHO/regional center egress from the transport network. It represents issues that can occur in the transport of the video from the SHO to the VHO.

Third monitoring point:
LAST MILE

On the VDSL or FTTH line. It is expensive and space is limited, but this can automate DSL loop qualification as well as reducetruck rolls.

Fourth monitoring point:
IN-HOME NETWORK

This is the set-top box. It resides behind the residential gateway and is connected through a LAN.

Source: Alcatel-Lucent