WebRTC Streaming is Still King

Why HESP and Media over QUIC Won’t Supplant WebRTC Anytime Soon

Surging demand for multidirectional real-time video streaming infrastructure is driving two major initiatives aimed at creating an HTTP-based path to the future at a moment when advanced WebRTC platforms are already taking the market where it needs to go.

While the perpetual search for better ways to do things is baked into our tech society’s DNA, so, too, is the idea that, as the old saying goes, waiting for perfection can be the enemy of progress. To that we would add that the damage to progress is even worse when misleading information about what’s already doable is used to rally support for new solutions.

Proponents of High Efficiency Streaming Protocol (HESP) and Media over QUIC (MoQ) have come up with intriguing but very different concepts which they hope will transform the one-way, high-latency HTTP streaming architecture into a conduit for real-time interactive streaming (RTIS). HESP is meant to do this through continued reliance on TCP (Transmission Control Protocol) while MoQ depends on the QUIC standard’s use of UDP (User Datagram Protocol). 

But these efforts are still a long way from replicating, let alone supplanting what’s already in operation with the use of WebRTC worldwide. This is especially the case in comparison to how WebRTC streaming is executed on the massively scalable Red5 Experience Delivery Network (XDN) platform, which is setting the pace with deployments worldwide that are consistently registering end-to-end latencies at 250ms and under over any distance in any direction.

Moreover, even if at some future date these new protocols could somehow match what’s being done with WebRTC, both solutions would require costly ecosystem-wide adjustments in the processes that underpin the current one-way HTTP streaming infrastructure. This undermines claims that these protocols represent easy-to-implement alternatives to the architectural changes mandated by WebRTC streaming.

A more honest assessment of where things stand is to acknowledge it would be great if these new solutions pan out to where there’s eventually no need for WebRTC. Implementations of RTIS would then be backward compatible with the legacy infrastructure. But at this point there’s no guarantee these solutions will pan out as anticipated, let alone how fast the ecosystem would adopt them to the point of ubiquitous availability. 

Meanwhile, given what can be done with WebRTC, it makes no sense to put the dawn of a new era in user experiences and the revenues they’re sure to generate on hold. The truth is, a lot of the WebRTC commentary emanating from the HESP Alliance and some participants in the initiative undertaken by the IETF’s MoQ Working Group relies on performance metrics long outdated by Red5’s XDN and other WebRTC-based solutions. 

Misleading claims distorting what’s really underway with the use of these platforms is a disservice to the growing legions of purveyors in multiple industries who need RTIS solutions now. So let’s set the record straight. 

Our response to this tide of disinformation is based on what we know best, i.e., the proven performance of XDN architecture. But, in general, we also know that to one degree or another there are other WebRTC-based solutions as described in this analysis of the real-time streaming market that rebut such commentary.

Whether the discussion is about latency, multidirectional video communications, scalability, quality, feature versatility, ease of use, content security, support for dynamic advertising or costs, nothing touted for HESP or MoQ surpasses and much of it falls short of the proven performance of XDN streaming infrastructures now in operation across the globe. This is true regardless of whether XDNs are activated automatically on the Red5 Cloud platform or through DIY applications of XDN SDKs and Red5 TrueTime toolsets. 

HESP Doesn’t Meet Basic Real-Time Streaming Requirements

We begin with a look at HESP, which is much further along in development than MoQ, even to the point that there’s been a smattering of commercial activity involving early iterations of the technology introduced with the THEOlive platform supplied by HESP inventor THEO Technologies. Since 2020 when THEO and Synamedia co-founded the HESP Alliance, they and their alliance partners have been refining the software stack as a standardized, albeit proprietary offering that any vendor or service provider willing to pay the usage royalties can deploy under their own brands.

Cost implications aside, which we’ll return to momentarily, there’s a fundamental technical drawback to HESP born of its focus on reducing the buffering time consumed by TCP to accommodate the retransmission of dropped packets. The technique not only fails to achieve RTIS-caliber latency performance; it requires ecosystem-wide distribution of server and client player software to accomplish what amounts to an inadequate though significant reduction in end-to-end latency.. 

As described by the HESP Alliance, the protocol’s “special sauce” involves use of an Initialization Stream generating a string of Initialization Packets. These packets convey all the information essential to executing the start of a streaming segment, including decoder and DRM configurations, full picture I-frame and identification of the frame that begins the next segment in the currently viewed primary or “Continuation” stream.

There can be any number of Initialization Packets in the Initialization Stream up to one for every frame in the video, which enables instant shifts to a new segment at any point in the stream, thereby avoiding long buffering delays incurred with dropped packets. The design also supports instant shifts to lower bitrate profiles as dictated by network conditions and allows users to seamlessly switch to another linear stream. 

The upshot is that “HESP can perform stream startup within a few hundred milliseconds, allowing for stable playback at latencies as low as 700ms–900ms,” the alliance says. Latencies reported by users of THEOlive typically reflect upper ranges at 1 second or above.

For example, Luxembourg-based global cloud CDN operator GCore tells customers video can be streamed via THEOlive on its infrastructure with delays “that don’t exceed 2 seconds.” Sky Racing, an Australian provider of live streaming horse and greyhound racing with support for betting, says it is benefitting from enabling last-minute pre-race online wagering through use of THEOlive to reduce latency to “about one second.”

These are impressive improvements on the multi-second latencies common to other HTTP streaming modes, including Low-Latency HLS (LLHLS) and QUIC-based HTTP3, which is the commercially available predecessor to MoQ.  But they don’t achieve the imperceptible 250ms A/V latencies that are essential to enabling the vast realm of applications that rely on interactive video communications,

This includes everything from video conferencing to sports watch parties, micro-betting, multiplayer game competition, live shopping, orchestrating live multi-camera surveillance, remote collaboration in live event productions, networked implementations of immersive extended reality experience, etc., etc. In other words, pretty much everything that comprises the next generation video streaming experience. 

Why would anyone think HESP can support multidirectional streaming that would supplant WebRTC, which brought video conferencing to life by virtue of its reliance on the Real Time Transport Protocol (RTP), which underlies the digital voice systems that replaced switched-circuit telephony?  Even if every user device player enabled for HESP reception were to be equipped with HESP server software, a highly impractical prospect, the latencies would prevent viable interactive video experiences. 

The Dolby Factor: Implications of THEO’s Acquisition

Recent developments in the streaming technology landscape have further complicated the future of HESP. In a significant move, Dolby Laboratories acquired THEO Technologies, the company behind HESP, in 2024. This acquisition raises important questions about the future direction and accessibility of HESP technology.

Notably, the press release announcing the acquisition repeatedly mentioned Dolby Millicast, Dolby’s existing WebRTC-based real-time streaming solution. The emphasis on integrating THEO’s technologies with Millicast suggests a potential shift in focus that could impact HESP’s future.

Given Dolby’s existing investment in Millicast, it’s not far-fetched to speculate that HESP might become less relevant in Dolby’s overall strategy. There’s a possibility that Dolby might slow down or even cease the development of HESP in favor of the Millicast technology.

Even if HESP development continues, Dolby’s track record with patents and royalties raises concerns about future licensing terms. HESP was already a proprietary technology requiring licensing fees, but under Dolby’s ownership, these terms could become even more onerous, potentially limiting adoption, especially among smaller players in the streaming market.

While the integration of THEO’s technologies with Dolby Millicast could enhance capabilities, it might also lead to a more closed ecosystem, potentially limiting interoperability with other streaming technologies.

This acquisition and its potential implications highlight a stark contrast with WebRTC, which remains an open, royalty-free standard. WebRTC’s open nature has been a key factor in its widespread adoption and continuous improvement by the global developer community.

As we’ve already noted, despite HESP’s claims of low latency, it still falls short of the sub-250ms latency consistently achieved by WebRTC implementations like Red5’s XDN platform. The acquisition by Dolby and potential integration with Millicast doesn’t change this fundamental limitation of HESP for truly interactive applications. Perhaps that’s the point: they see the Millicast WebRTC solution as the truly interactive one, and they only acquired THEO for their player business. In this scenario, the HESP technology is a pure throwaway. 

Either way, the existing licensing requirements and the potential for even more restrictive terms under Dolby’s ownership stand in sharp contrast to the accessibility and flexibility offered by WebRTC. While HESP may find its niche in certain streaming applications, it’s clear that for truly interactive, real-time communication, WebRTC remains the superior choice.

This development serves as a reminder of the importance of open standards in driving innovation and accessibility in the streaming industry. As the market continues to evolve, solutions built on open, flexible technologies like WebRTC are likely to maintain their advantage in meeting the diverse and rapidly changing needs of real-time interactive streaming applications.

The Abundance of Inaccuracies in Claims about WebRTC

Beyond the latency disparities, it’s instructive to look at other comparisons HESP supporters are making with WebRTC to rationalize the need for HESP. Virtually all these descriptions are based on the root specifications of the WebRTC protocol rather than on how it is used today.  

Taking these claims about WebRTC one by one as articulated in an article penned by Pieter-Jan Speelmans, technical working group chair for the HESP Alliance, here’s a brief look at what’s going on in the real world with use of XDN architecture:

• Scalability – Speelmans says that while, by tapping into existing CDN infrastructure, HESP “allows for seamless scaling to thousands, even millions of viewers,” WebRTC by virtue of requiring “each client to have a direct connection with the backend,” can only be scaled with deployment of additional media server instances, where, on average, most solutions “initiate a new server for every 500 to 1,000 new viewers.” 

The truth: XDN architecture achieves scaling of WebRTC-distributed content with multidirectional real-time streaming support to millions of users at any distance over one or more cloud computing platforms with no need for client plug-ins. Automated scaling to fit the needs of any given use case is accomplished though software-managed orchestration of a hierarchy of virtually expandable cloud node instances. These include Origin Nodes that ingest content in multiple protocols for conversion in most cases to WebRTC; Relay Nodes used when a large base of end users is involved, and intelligent Edge Nodes serving as the points of dedicated per-user conformation of streams for local distribution.

XDN streaming utilizes WebRTC in all instances where user devices have access to any of the major browsers that support WebRTC, including Chrome, Edge, Safari, Firefox and Opera. At the same time, the XDN node hierarchy can be tuned to pass through content targeted to mobile devices via the widely supported Real-Time Streaming Protocol (RTSP). And ingested content encapsulated for transmission via Real-Time Messaging Protocol (RTMP), Secure Reliable Transport (SRT), and MPEG TS can be transported over XDN infrastructure via RTP when client devices matched to any of those protocols are unable to rely on any of the major browsers that support WebRTC.

There’s also a backup option to this multi-protocol approach to enabling unlimited RTIS scalability. In rare instances where targeted devices are not compatible with any of these protocols, XDNs can be configured to support handoff of content for transmission at much higher latencies over legacy HTTP architecture via HLS. 

• Device Support – Here there’s an attempt to make the lack of browser support for HESP and the consequent need for player software an advantage over WebRTC by suggesting that WebRTC’s support in browsers means that only the Opus audio codec and H.264 and VP8 video codecs natively supported in current iterations of the protocol can be used to encode content. 

• The truth – With higher hardware processing power taking hold across the connected device ecosystem, the state of browser support for codecs beyond H.264 and VP8 is a fast-moving target. In the case of HEVC/H.265, support in browsers has been slow in coming but is picking up in tandem with improving device hardware and growing demand for the codec in traditional streaming. The codec now has the greatest support in Safari covering the vast majority of iOS and Mac devices. It has more limited support with the latest versions of Chrome, Edge, and Opera but no support in Firefox. 

As for HEVC usage with WebRTC, there are other issues beyond device hardware processing power and the degree to which the codec is supported in browsers. One impediment has to do with the high royalty fees, which are out of sync with the royalty-free benefits associated with the open WebRTC standard. More significantly, HEVC support in WebRTC requires new specifications that would make the RTP protocol underlying WebRTC compatible for use with HEVC. 

Now, in the aftermath of some mitigation of royalty issues and ongoing advances in device hardware, the picture is changing. The IETF Audio/Video Transport Core Maintenance (AVTCORE) working group devoted to refining RTP specs in tandem with market developments is working on modifications suited to making RTP and WebRTC compatible with HEVC. As for browsers, Google appears to be preparing to bring HEVC support to WebRTC in Chrome. But, overall, it looks like pervasive plugin-free support for HEVC in WebRTC will take a while.

The situation is very different with AV1, which is already technically compatible for use with WebRTC. With ongoing improvements in chipsets with built-in hardware acceleration, impediments to encoding and decoding live transmissions at acceptable latencies are disappearing. At this writing as reported by this tracker, AV1 has gained support in multiple versions of all the major browsers, with the exception of Safari, which is limited to devices embedded with hardware decoders, such as the iPhone 15 Pro and M3 MacBook Pro.

In light of these developments, Red5 is moving rapidly toward enabling support for both AV1 and HEVC on the XDN platform. We’ll be communicating more about this in the near future. 

• Network Resilience – While Speelmans acknowledges WebRTC implementations “at times feature server-side” implementations of adaptive bitrate (ABR) profiles, which is the case with XDN architecture, he disparages such approaches as being vulnerable to packet losses and frame drops associated with WebRTC’s reliance on UDP transport. Of course, any impact frame losses might have on ABR-formatted profiles would also apply with WebRTC implementations that don’t rely on ABR support. 

The Truth – Such quality-of-performance issues traditionally associated with UDP have been overcome through advanced forward error correction (FEC) and other mechanisms in wide use, not only in the WebRTC domain but also with the widely adopted use of QUIC in low-latency HTTP distribution, including the exclusively QUIC-based HTTP/3. In Red5’s case, the mechanisms we use are delivering proven UDP performance on par with TCP continuity, but, of course, without the buffering delays.

•  Content Protection – Here again an acknowledgement of progress in WebRTC applications lends an aura of credibility to an inaccurate claim, to wit: “While some WebRTC solutions provide proprietary DRM implementations, they lack cross-platform support, for example, not supporting iOS Safari.” 

The Truth – Levels of content protection afforded by WebRTC-based platforms vary widely, leaving it to users to choose RTIS solutions that surpass the level of security described by Speelmans. That said, we can honestly report that there’s no set of protection solutions available with any type of streaming system that matches XDN architecture in providing users the flexibility to achieve the most robust protection possible with or without DRM support.

XDN architecture is integrated with castLabs’s advanced multi-DRM platform, which utilizes the three top DRMs, Google’s Widevine, Microsoft’s PlayReady and Apple’s FairPlay, to deliver best-of-breed multi-DRM protection across all types of connected devices. There are also other multi-DRM suppliers such as EZDRM that are prepared to work in the WebRTC environment.

In cases where customers don’t want to incur the costs of commercial DRM platforms,  Red5 has taken steps to provide DRM-caliber protection through reliance on the 128-bit AES encryption and private key management enabled by the Secure Realtime Protocol (SRTP), which is an integral component of the WebRTC protocol stack. We have made this a viable browser-implemented alternative to DRM with the aid of our RoundTrip Authentication validator, which is a remote server authorization mediator that applies the business logic needed to execute server-to-server validation of publishers and end users. 

• Quality of Experience – One of the strengths of HESP in comparison to legacy TCP-based HTTP streaming stems from its use of the Initialization Stream innovation to support a frame-based approach to channel changes. While this achieves TV-caliber channel surfing speeds that are hard to reach with HTTP streaming, Speelmans is dead wrong when he asserts use of WebRTC inevitably leads to “a trade-off between live latency and channel change time due to a GOP (Group of Pictures) based” approach that leads to “higher channel change times for ultra-low latency streaming.”    

The Truth – XDN architecture supports lightning-fast channel switching tuned to frame breaks as opposed to multi-frame GOP segments defined by I-frames. This was vividly on display at the recent NAB Show in Las Vegas, where Red5 employed its TrueTime Multiview solution to demonstrate instant switching across a 4×6 matrix of stored and live camera feeds. Whether the payload streamed over XDN infrastructure to Edge Nodes is multiple camera feeds from one event or a lineup of live-streamed TV channels, the same Edge Node intelligence is used to instantaneously deliver whatever option is chosen by each user in the node serving area. 

• Captions, Subtitles and Time Metadata – While it’s true that “WebRTC lacks specifications for the delivery of captions, subtitles, ad insertion markers, and other time data,” such specifications aren’t required insofar as any such applications can be activated through use of the WebRTC data channel. Further clouding the picture, Speelmans asserts that these capabilities can only be supported by the use of the WebRTC data channel to activate them with delivery of “proprietary messages.” 

The Truth – The WebRTC data channel doesn’t require proprietary messaging to frame-accurately match metadata, closed captioning, subtitles, graphics or anything else delivered over the channel with the core content elements. XDN architecture actually provides a far simpler, superior approach to adding any type of enhancement to a primary content flow on a personalized or multi-user basis compared to the error-prone complexities of just-in-time packaging that goes into enabling ancillary enhancements with live-streamed content over traditional HTTP infrastructure. 

In such cases XDN SDKs automatically apply the Action Message Format (AMF) commonly used in all types of streaming environments to associate metadata elements delivered in real-time from any source with time-coded metadata markers embedded in the appropriate primary content frames. This invisible code triggers the rendering of the overlaid metadata elements through the processes supported by device browsers in the case of WebRTC or via the use of WebSocket technology when other protocols are used to connect with end devices.

As to the data channel payload, it can be based on standardized approaches to these applications or proprietary variations, as preferred by the service provider. For example, the Red5 HTML5 SDK enables the synchronized rendering of all the types of applications supported by HTML5 extensions. 

However they’re formatted, all metadata elements that aren’t directly incorporated into primary A/V stream frames are delivered as overlays frame-accurately synchronized with the primary streams. These overlays can be captured in the live production process to be delivered with the content to all recipients from Edge Nodes or they can be delivered as individually personalized enhancements that are rendered in sync with the primary content by receiving devices.

The multi-cloud XDN platform is executing perfectly synced multi- and per-user enhancements to real-time streams in one-to-many, many-to-many and many-to-one iterations across a vast range of consumer, business, government, and institutional operations. Cases in point include but are by no means limited to: 

• AI-assisted content recommendations.
• In-stream e-commerce purchase options.
• Responses to linguistic and regulatory nuances impacting globally distributed video.
• On-the-fly matchups of stored clips with live content.
• Multiscreen viewing options.
• Dynamic ad insertion.
• Seamless integrations of real action with animation.
• In-stream access to text and video chat feeds.

• Backward Compatibility – A big selling point for HESP is the fact that backward compatibility with traditional HLS and DASH streaming infrastructure “enables reuse of HESP output for live-to-VOD” applications, generation of “additional HLS or DASH” streams, and “facilitates the reuse of existing encoding and distribution (CDN) infrastructure.”

The Truth – Touche! While XDN architecture is equipped with transcoding support for live-to-VOD content storage, happily the platform is not backward compatible with HTTP infrastructure. As the inadequacies of HESP demonstrate, backward compatibility with the traditional high-latency one-way streaming infrastructure can be a major liability.

Our discussion of XDN capabilities here has been focused on responses to the HESP Alliance claims, leaving unsaid the many other advantages the Red5 platform delivers as a superior approach to RTIS. Our website is rich with resources offering a more extensive overview of what can be done through Red5 Cloud or DIY implementations of XDN infrastructure.

The Timing Issue Plaguing Media over QUIC

Here we turn to discussion of the other major attempt at creating an HTTP-compatible RTIS infrastructure, Media over QUIC. Of course, all the points made in our response to the HESP Alliance claims apply anytime similarly erroneous disparagements of WebRTC capabilities are used to justify the need for MoQ. But in the case of MoQ the reasons it is highly unlikely to eliminate market reliance on WebRTC for RTIS are different.

Basically, the problem with MoQ is two-fold. Even if a protocol is developed with the potential to match what can be done with WebRTC, and there’s no guarantee that will be the case, completion of the standard, let alone its implementation, will take too long and likely be too costly to be of much use in a market replete with next-gen streaming agendas that need to get off the ground immediately.

As described in the Media over QUIC Charter, MoQ and sub-protocol working groups are pursuing development of an IETF standard that would supplant HLS, DASH and WebRTC with a unified QUIC-based approach to supporting one-way and multidirectional streaming for distribution as well as contribution and other backend streaming needs. While most MoQ documentation is vague on latency parameters, a blog post on the NETINT site quoting Will Law, chief architect at Akamai and a founding participant in the MoQ initiative, says the platform will support tunable latencies ranging from as high as 3 seconds to as low as sub-400ms.

Written by NETINT senior video marketing director and Streaming Media contributor Jan Ozer, the article says, “CDNs will play a critical role in scaling sub-3-second latency by processing requests at the edge. Expect gradual adoption driven by live sports and competitive market pressure driven by end users.”

But comments by key participants in the project make clear it could be years before MoQ gets off the ground. “It’s going to take years,” says Discord engineer Luke Curley in a blog posted on a MoQ website. “It’s going to take a lot of idiots like myself who want to replace WebRTC. It’s going to take a lot of companies who are willing to bet on a new standard.”

In his blog summarizing Law’s take on MoQ, Ozer says, “Media over QUIC shows promise but faces adoption and deployment challenges across players, devices and CDNs. Given the years-long adoption cycles, a QUIC-driven overhaul of streaming is far from imminent. But long term, the flexibility and performance of QUIC are likely to drive the next evolution in streaming architectures.”

The Implementation Challenge Posed by MoQ Architecture 

With that evolution already well underway driven by widescale use of XDN and other WebRTC-based infrastructures, it remains to be seen what role, if any, MoQ will have in moving things along even if it achieves status as a standard. Fundamentally, it will take considerable investment with widescale cooperation of CDN vendors to configure an MoQ infrastructure suited to full RTIS capabilities with live content. 

MoQ operates as a software overlay to QUIC that speeds the retrieval of lost packets critical to a smooth user experience by pulling them from replications of streamed segments that are temporarily stored close to end users in what are known as relay caches. This approach, which would require installing new software on hardware resources across the CDN ecosystem, also provides a way to support regional “fanning” of a single content stream to reach all engaged users over their local access networks. 

The Real Cost Picture

In weighing the costs and hassles that would go into enabling either HESP or MoQ, it’s important to recognize that advocates of these solutions who claim they are less costly than reliance on WebRTC are ignoring the cost-saving efficiencies that come with implementations of XDN infrastructure. Notably, the introduction of Red5 Cloud as a curated real-time platform service has reduced what was already the leading cost/benefit equation in the WebRTC market.

Red5 has for a long time employed a success-based pricing structure that departs from commonly employed usage-based pricing schemes by charging fixed monthly fees based on the node server instances used to accommodate the maximum number of connections associated with the plans chosen by customers. Red5 Cloud adds major cost reductions to implementing XDN infrastructure by eliminating the time and staff resources that would otherwise be expended on calculating, configuring, and setting up the streaming infrastructure. By inputting basic use case requirements on the Red5 Cloud dashboard, customers gain immediate access to real-time streaming infrastructure matched to their needs with the ability to trigger adjustments in their set-ups whenever needed. 

It’s also important to note that engaging with XDN architecture doesn’t occur in a standalone WebRTC vacuum requiring customers to put together all the processes that go into bringing the infrastructure to life. Instead, Red5 puts customers in touch with a large ecosystem of best-of-breed applications fully integrated with the architecture to enable seamless operational support for DRM protection, hardware-based encoding, graphics enhancement and captioning, forensic watermarking, surveillance analytics, applications of AI, enhanced GPU processing innovations, immersive multiplayer game-caliber rendering, and much else.   

With integrations across multiple public cloud computing platforms, there’s also virtually no limit to the reach of XDN-enabled RTIS infrastructures. Seamless multi-cloud operations are supported by pre-integrations with numerous cloud platforms, including AWS, Digital Ocean, Google Cloud, Microsoft Azure, and Oracle Cloud, and through engagement with any of more than a dozen other cloud providers whose facilities can be quickly tied into the XDN via the widely used Terraform multi-cloud toolset. 

No matter how many cloud platforms are involved, set-up configurations and ongoing orchestration of all the XDN nodes performed by the platform’s Stream Manager work in real time applying automated scaling mechanisms to add or lower virtualized server capacity in response to fluctuations in traffic demand or the need to add new broadcasters and end users. Similarly, orchestration capabilities that provide fail-safe redundancy in the event of any problems with hardware resources ensure adjustments are made as readily in a cross-platform environment as they are within a single cloud implementation.

————————————————————————   

Marketing messages aimed at justifying commitments to developing and using HESP and MoQ with false claims about what can be done with WebRTC need to end. There’s no room for impeding progress with self-serving disinformation when legions of enterprises, institutions and individuals who depend on video streaming are looking for ways to exploit the power of RTIS.

HESP is not a legitimate RISP solution. And if MoQ ever turns out to be, it will be long after the train has left the station. For a realistic assessment of the possibilities, give us a call or contact us at info@red5.net.