Subject: [rtcweb] WebRTC and Real-time Translation

From: Adam Sobieski <[email-masked]>
Date: Thu, 27 Sep 2018 01:10:40 +0000

IETF RTCWEB Working Group,

Greetings. I opened an issue on WebRTC and Real-time Translation at the GitHub repository for WebRTC version next use cases (https://github.com/w3c/webrtc-nv-use-cases/issues/2).

Introduction

Real-time translation is both an interesting and important use case for a next version of WebRTC.

Speech Recognition, Translation and Speech Synthesis

Approaches to real-time speech-to-speech machine translation include those which interconnect speech recognition, translation and speech synthesis components and services. In that regard, we can consider client-side, on-prem, server-side, third-party and cloud-based components and services. In that regard, we can also consider both free and priced components and services.

We can envision post-text speech technology and machine translation components and services. Speech recognition need not output to text; we can consider speech-to-SSML. Machine translation need not input from nor output to text; we can consider SSML-to-SSML machine translation. Components and services may provide various options with respect to their input and output data formats.

Connecting Components and Services by Constructing Graphs

We can consider APIs which facilitate the construction of graphs which represent the flow of data between components and services. As these graphs are constructed, users could be apprised of relevant notifications, requests for permissions and options for payments. As these constructed graphs are activated, a number of protocols could be utilized to interconnect the components and services which, together, provide users with real-time translation.

Hyperlinks

WebRTC Translator Demo<https://www.youtube.com/watch?v=Tv8ilBOKS2o>
Real Time Translation in WebRTC<https://www.youtube.com/watch?v=EPBWR_GNY9U>

Best regards,
Adam Sobieski
http://www.phoster.com/contents/

---

From: Bernard Aboba <[email-masked]>
Date: Wed, 26 Sep 2018 21:58:08 -0700

One of the key questions for "Next Version Use Cases" is what
WebRTC-deficiencies are preventing these use cases from being
satisfactorily implemented today.

For example, speech transcription cloud services have been implemented over
Websockets, where a snippet of speech is uploaded, and a transcription is
provided in reply.  The latency is satisfactory for some uses cases.
Improvements can perhaps be made by sending an audio stream and receiving a
transcription via the data channel, but this is also within the
capabilities of the existing RTCWEB protocols and WebRTC-PC API.

What seems to differentiate *next version* scenarios are situations where
the processing is best done on the device, in order to lower latency or
enhance privacy.  On-device processing brings in discussion of
workers/worklets, access to raw audio/video, etc.  However, so far I'm not
aware of on-device implementations of transcription or translation.

On Wed, Sep 26, 2018 at 6:10 PM Adam Sobieski <[email-masked]>
wrote:

> Greetings. I opened an issue on *WebRTC and Real-time Translation* at the
> GitHub repository for WebRTC version next use cases (
> https://github.com/w3c/webrtc-nv-use-cases/issues/2).
>
>
> Introduction
>
> Approaches to real-time speech-to-speech machine translation include those
> which interconnect speech recognition, translation and speech synthesis
> components and services. In that regard, we can consider client-side,
> on-prem, server-side, third-party and cloud-based components and services.
> In that regard, we can also consider both free and priced components and
> services.
>
> We can envision *post-text* speech technology and machine translation
> components and services. Speech recognition need not output to text; we can
> consider speech-to-SSML. Machine translation need not input from nor output
> to text; we can consider SSML-to-SSML machine translation. Components and
> services may provide various options with respect to their input and output
> data formats.
> We can consider APIs which facilitate the construction of graphs which
> represent the flow of data between components and services. As these graphs
> are constructed, users could be apprised of relevant notifications,
> requests for permissions and options for payments. As these constructed
> graphs are activated, a number of protocols could be utilized to
> interconnect the components and services which, together, provide users
> with real-time translation.
> Hyperlinks
>
> WebRTC Translator Demo <https://www.youtube.com/watch?v=Tv8ilBOKS2o>
> Real Time Translation in WebRTC
> <https://www.youtube.com/watch?v=EPBWR_GNY9U>
>
>
>
>
>
> Best regards,
>
> Adam Sobieski
>
> _______________________________________________
> rtcweb mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/rtcweb
>

---

From: Ted Hardie <[email-masked]>
Date: Thu, 27 Sep 2018 08:37:22 -0700

Hi Adam,

Are there specific work items for the IETF which you believe arise from
this use case, or was this an FYI?

regards,

Ted

---

From: Adam Sobieski <[email-masked]>
Date: Fri, 28 Sep 2018 00:50:02 +0000

Bernard Aboba,
Ted Hardie,

Client-side Transcription and Translation

With respect to client-side speech recognition, transcription, translation and speech synthesis scenarios, we can consider GPGPU approaches.

HYDRA [1][2] is a “hybrid GPU/CPU-based speech recognition engine that leverages modern GPU-based parallel computing architectures to realize accurate real-time recognition with extremely large models.” In 2012, Professor Ian Lane indicated that HYDRA performs 20x faster than other approaches [3].

Deep Speech [4][5][6] is a deep-learning-based approach to speech recognition which “outperforms previously published results on the widely studied Switchboard Hub5'00, achieving 16.0% error on the full test set” and “handles challenging noisy environments better than widely used, state-of-the-art commercial speech systems.”

Articulatory synthesis can be accelerated by graphics cards [7].

WaveNet [8][9] is a deep generative model of raw audio waveforms including speech audio.

Facebook AI Research recently advanced machine translation [10], advancing performance metrics by 10 BLEU points.

With respect to desktop-based translation, vendors such as SYSTRAN [11] offer desktop-based, server-based and cloud-based solutions.

There are some desktop-based transcription and machine translation solutions [12] and it is expected that real-time client-side solutions for transcription and translation, processing speech audio, will exist in the upcoming years, at least for desktop computing if not mobile computing.

On-premises Transcription and Translation

In addition to client-side solutions, on-premises solutions can deliver lowered latency and enhanced privacy.

Server-side and Cloud-based Transcription and Translation

For a number of scenarios including mobile computing, server-side and cloud-based transcription and translation services make sense.

Major software vendors such as Amazon, Facebook, Google, IBM and Microsoft offer priced cloud-based services which include speech recognition, machine translation and speech synthesis.

Post-text Speech Technology

I am an advocate of post-text speech technologies. Speech-to-text is too lossy. Information pertaining to prosody, intonation, emphases and pauses are discarded in text output. Such information can be useful, for example informing machine translation components and services. In addition to speech-to-SSML speech recognition and SSML-to-SSML machine translation scenarios, we can envision new, intermediate data formats beyond SSML.

The inputs and outputs of speech recognition, translation and speech synthesis components and services could be multiple formats – formats other than text.

API Sketch: Dataflow Graphs

Sketches with respect to APIs include the declarative construction of dataflow graphs which interconnect abstract components. Such APIs can abstract away whether the interconnectable components are client-side, on-prem, server-side, third-party or cloud-based. Such APIs can abstract away whether the interconnectable components are for free or priced to end-users. Considerations to such API include the data formats and stream specifications of components’ various inputs and outputs to be interconnected.

Dataflow graphs can be an intuitive abstraction layer, one which provides intuitive and convenient programming while interconnecting arbitrary numbers of components and services. Dataflow graphs can interconnect client-side and remote speech recognition, translation and speech synthesis components as well as any other components which could reasonably be interconnected or pipelined.

When such dataflow graphs are prepared for activation, it is envisioned that users will be provided with notifications, requests for permissions and options for payment.

Potential IETF Work Items

When such dataflow graphs are activated, it is envisioned that computer networking protocols will be utilized to notify remote components or services of proper data routings, e.g. daisy-chain or pipeline configurations, in a secure manner.

That is, there may be new protocols and computer networking topics with regard to implementing the APIs for interconnecting WebRTC peers with speech recognition, translation and speech synthesis components and services.

Conclusion

Tight WebRTC integration is important for envisioned efficient, low-latency, high-performance, scalable real-time translation scenarios.

While there exist some ad hoc approaches to providing real-time translation with WebRTC, standardizing new APIs and protocols can convenience developers, convenience end users, and create new markets with respect to real-time translation scenarios.

Thank you for considering adding real-time translation to the use cases for a next version of WebRTC. I look forward to any discussion on these topics.

References

[1] http://www.cs.cmu.edu/~ianlane/hydra/
[2] https://www.youtube.com/watch?v=73rQ0lRx2aY
[3] https://www.youtube.com/watch?v=Y7Jlj7QYrcg
[4] https://arxiv.org/abs/1412.5567
[5] https://devblogs.nvidia.com/deep-speech-accurate-speech-recognition-gpu-accelerated-deep-learning/
[6] https://github.com/mozilla/DeepSpeech
[7] https://open.library.ubc.ca/media/stream/pdf/24/1.0348751/3
[8] https://deepmind.com/blog/wavenet-generative-model-raw-audio/
[9] https://devblogs.nvidia.com/nv-wavenet-gpu-speech-synthesis/
[10] https://www.forbes.com/sites/williamfalcon/2018/09/01/facebook-ai-just-set-a-new-record-in-translation-and-why-it-matters/#205b9e5b3124
[11] https://store.systran.us/lp/storeSystran?Langue=en_US
[12] https://en.wikipedia.org/wiki/Comparison_of_machine_translation_applications

From: Bernard Aboba<mailto:[email-masked]>
Sent: Thursday, September 27, 2018 12:58 AM
Subject: Re: [rtcweb] WebRTC and Real-time Translation

One of the key questions for "Next Version Use Cases" is what WebRTC-deficiencies are preventing these use cases from being satisfactorily implemented today.

For example, speech transcription cloud services have been implemented over Websockets, where a snippet of speech is uploaded, and a transcription is provided in reply.  The latency is satisfactory for some uses cases.
Improvements can perhaps be made by sending an audio stream and receiving a transcription via the data channel, but this is also within the capabilities of the existing RTCWEB protocols and WebRTC-PC API.

What seems to differentiate *next version* scenarios are situations where the processing is best done on the device, in order to lower latency or enhance privacy.  On-device processing brings in discussion of workers/worklets, access to raw audio/video, etc.  However, so far I'm not aware of on-device implementations of transcription or translation.

On Wed, Sep 26, 2018 at 6:10 PM Adam Sobieski <[email-masked]<mailto:[email-masked]>> wrote:
IETF RTCWEB Working Group,

Introduction

Hyperlinks

Best regards,
Adam Sobieski

_______________________________________________
rtcweb mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/rtcweb