webrtcd: webrtc streaming server (audio/video/cereal) (#30186)

* WebRTCClient and WebRTCServer abstractions * webrtc client implementation * Interactive test scripts * Send localDescriptions as offer/asnwer, as they are different * Tracks need to be added after setting remote description for multi-cam streaming to work * Remove WebRTCStreamingMetadata * Wait for tracks * Move stuff to separate files, rename some things * Refactor everything, create WebRTCStreamBuilder for both offer and answers * ta flight done time to grind * wait for incoming tracks and channels * Dummy track and frame reader track. Fix timing. * dt based on camera type * first trial of the new api * Fix audio track * methods for checking for incoming tracks * Web migration part 2 * Fixes for stream api * use rtc description for web.py * experimental cereal proxy * remove old code from bodyav * fix is_started * serialize session description * fix audio * messaging channel wrapper * fix audiotrack * h264 codec preference * Add codec preference to tracks * override sdp codecs * add logging * Move cli stuff to separate file * slight cleanup * Fix audio track * create codec_mime inside force_codec function * fix incoming media estimation * move builders to __init__ * stream updates following builders * Update example script * web.py support for new builder * web speaker fixes * StreamingMediaInfo API * Move things around * should_add_data_channel rename * is_connected_and_ready * fix linter errors * make cli executable * remove dumb comments * logging support * fix parse_info_from_offer * improve type annotations * satisfy linters * Support for waiting for disconnection * Split device tracks into video/audio files. Move audio speaker to audio.py * default dt for dummy video track * Fix cli * new speaker fixes * Remove almost all functionality from web.py * webrtcd * continue refactoring web.py * after handling joystick reset in controlsd with #30409, controls are not necessary anymore * ping endpoint * Update js files to at least support what worked previously * Fixes after some tests on the body * Streaming fixes * Remove the use of WebRTCStreamBuilder. Subclass use is now required * Add todo * delete all streams on shutdown * Replace lastPing with lastChannelMessageTime * Update ping text only if rtc is still on * That should affect the chart too * Fix paths in web * use protocol in SSLContext * remove warnings since aiortc is not used directly anymore * check if task is done in stop * remove channel handler wrapper, since theres only one channel * Move things around * Moved webrtc abstractions to separate repository * Moved webrtcd to tools/webrtc * Update imports * Add bodyrtc as dependency * Add webrtcd to process_config * Remove usage of DummyVideoStreamTrack * Add main to webrtcd * Move webrtcd to system * Fix imports * Move cereal proxy logic outside of runner * Incoming proxy abstractions * Add some tests * Make it executable * Fix process config * Fix imports * Additional tests. Add tests to pyproject.toml * Update poetry lock * New line * Bump aiortc to 1.6.0 * Added teleoprtc_repo as submodule, and linked its source dir * Add init file to webrtc module * Handle aiortc warnings * Ignore deprecation warnings * Ignore resource warning too * Ignore the warnings * find free port for test_webrtcd * Start process inside the test case * random sleep test * test 2 * Test endpoint function instead * Update comment * Add system/webrtc to release * default arguments for body fields * Add teleoprtc to release * Bump teleoprtc * Exclude teleoprtc from static analysis * Use separate event loop for stream session tests
eivegau · Dec 2, 2023 · f058b5d · f058b5d
1 parent e34ee43
commit f058b5d
Show file tree

Hide file tree

Showing 18 changed files with 788 additions and 474 deletions.
diff --git a/.gitmodules b/.gitmodules
@@ -13,6 +13,9 @@
 [submodule "body"]
   path = body
   url = ../../commaai/body.git
+[submodule "teleoprtc_repo"]
+  path = teleoprtc_repo
+  url = ../../commaai/teleoprtc
 [submodule "tinygrad"]
   path = tinygrad_repo
   url = https://github.com/geohot/tinygrad.git
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -26,7 +26,7 @@ repos:
     rev: v2.2.6
     hooks:
     -   id: codespell
-        exclude: '^(third_party/)|(body/)|(cereal/)|(panda/)|(opendbc/)|(rednose/)|(rednose_repo/)|(selfdrive/ui/translations/.*.ts)|(poetry.lock)'
+        exclude: '^(third_party/)|(body/)|(cereal/)|(panda/)|(opendbc/)|(rednose/)|(rednose_repo/)|(teleoprtc/)|(teleoprtc_repo/)|(selfdrive/ui/translations/.*.ts)|(poetry.lock)'
         args:
         # if you've got a short variable name that's getting flagged, add it here
         - -L bu,ro,te,ue,alo,hda,ois,nam,nams,ned,som,parm,setts,inout,warmup,bumb,nd,sie,preints
@@ -39,12 +39,12 @@ repos:
         language: system
         types: [python]
         args: ['--explicit-package-bases', '--local-partial-types']
-        exclude: '^(third_party/)|(cereal/)|(opendbc/)|(panda/)|(rednose/)|(rednose_repo/)|(tinygrad/)|(tinygrad_repo/)|(xx/)'
+        exclude: '^(third_party/)|(cereal/)|(opendbc/)|(panda/)|(rednose/)|(rednose_repo/)|(tinygrad/)|(tinygrad_repo/)|(teleoprtc/)|(teleoprtc_repo/)|(xx/)'
 -   repo: https://github.com/astral-sh/ruff-pre-commit
     rev: v0.1.6
     hooks:
     -   id: ruff
-        exclude: '^(third_party/)|(cereal/)|(panda/)|(rednose/)|(rednose_repo/)|(tinygrad/)|(tinygrad_repo/)'
+        exclude: '^(third_party/)|(cereal/)|(panda/)|(rednose/)|(rednose_repo/)|(tinygrad/)|(tinygrad_repo/)|(teleoprtc/)|(teleoprtc_repo/)'
 -   repo: local
     hooks:
     -   id: cppcheck

diff --git a/poetry.lock b/poetry.lock
diff --git a/pyproject.toml b/pyproject.toml
@@ -24,6 +24,7 @@ testpaths = [
   "system/proclogd",
   "system/tests",
   "system/ubloxd",
+  "system/webrtc",
   "tools/lib/tests",
   "tools/replay",
   "tools/cabana"
@@ -43,6 +44,8 @@ exclude = [
   "rednose_repo/",
   "tinygrad/",
   "tinygrad_repo/",
+  "teleoprtc/",
+  "teleoprtc_repo/",
   "third_party/",
 ]
 
@@ -186,6 +189,7 @@ exclude = [
   "opendbc",
   "rednose_repo",
   "tinygrad_repo",
+  "teleoprtc",
   "third_party",
 ]
 flake8-implicit-str-concat.allow-multiline=false

diff --git a/release/files_common b/release/files_common
@@ -281,6 +281,11 @@ system/sensord/sensors/*.cc
 system/sensord/sensors/*.h
 system/sensord/pigeond.py
 
+system/webrtc/__init__.py
+system/webrtc/webrtcd.py
+system/webrtc/device/audio.py
+system/webrtc/device/video.py
+
 selfdrive/thermald/thermald.py
 selfdrive/thermald/power_monitoring.py
 selfdrive/thermald/fan_controller.py
@@ -439,6 +444,8 @@ third_party/qt5/larch64/bin/**
 scripts/update_now.sh
 scripts/stop_updater.sh
 
+teleoprtc/**
+
 rednose_repo/site_scons/site_tools/rednose_filter.py
 rednose/.gitignore
 rednose/**

diff --git a/selfdrive/manager/process_config.py b/selfdrive/manager/process_config.py
@@ -84,6 +84,7 @@ def only_offroad(started, params, CP: car.CarParams) -> bool:
 
   # debug procs
   NativeProcess("bridge", "cereal/messaging", ["./bridge"], notcar),
+  PythonProcess("webrtcd", "system.webrtc.webrtcd", notcar),
   PythonProcess("webjoystick", "tools.bodyteleop.web", notcar),
 ]
 

diff --git a/system/webrtc/__init__.py b/system/webrtc/__init__.py
diff --git a/system/webrtc/device/audio.py b/system/webrtc/device/audio.py
@@ -0,0 +1,110 @@
+import asyncio
+import io
+from typing import Optional, List, Tuple
+
+import aiortc
+import av
+import numpy as np
+import pyaudio
+
+
+class AudioInputStreamTrack(aiortc.mediastreams.AudioStreamTrack):
+  PYAUDIO_TO_AV_FORMAT_MAP = {
+      pyaudio.paUInt8: 'u8',
+      pyaudio.paInt16: 's16',
+      pyaudio.paInt24: 's24',
+      pyaudio.paInt32: 's32',
+      pyaudio.paFloat32: 'flt',
+  }
+
+  def __init__(self, audio_format: int = pyaudio.paInt16, rate: int = 16000, channels: int = 1, packet_time: float = 0.020, device_index: Optional[int] = None):
+    super().__init__()
+
+    self.p = pyaudio.PyAudio()
+    chunk_size = int(packet_time * rate)
+    self.stream = self.p.open(format=audio_format,
+                              channels=channels,
+                              rate=rate,
+                              frames_per_buffer=chunk_size,
+                              input=True,
+                              input_device_index=device_index)
+    self.format = audio_format
+    self.rate = rate
+    self.channels = channels
+    self.packet_time = packet_time
+    self.chunk_size = chunk_size
+    self.pts = 0
+
+  async def recv(self):
+    mic_data = self.stream.read(self.chunk_size)
+    mic_array = np.frombuffer(mic_data, dtype=np.int16)
+    mic_array = np.expand_dims(mic_array, axis=0)
+    layout = 'stereo' if self.channels > 1 else 'mono'
+    frame = av.AudioFrame.from_ndarray(mic_array, format=self.PYAUDIO_TO_AV_FORMAT_MAP[self.format], layout=layout)
+    frame.rate = self.rate
+    frame.pts = self.pts
+    self.pts += frame.samples
+
+    return frame
+
+
+class AudioOutputSpeaker:
+  def __init__(self, audio_format: int = pyaudio.paInt16, rate: int = 48000, channels: int = 2, packet_time: float = 0.2, device_index: Optional[int] = None):
+
+    chunk_size = int(packet_time * rate)
+    self.p = pyaudio.PyAudio()
+    self.buffer = io.BytesIO()
+    self.channels = channels
+    self.stream = self.p.open(format=audio_format,
+                              channels=channels,
+                              rate=rate,
+                              frames_per_buffer=chunk_size,
+                              output=True,
+                              output_device_index=device_index,
+                              stream_callback=self.__pyaudio_callback)
+    self.tracks_and_tasks: List[Tuple[aiortc.MediaStreamTrack, Optional[asyncio.Task]]] = []
+
+  def __pyaudio_callback(self, in_data, frame_count, time_info, status):
+    if self.buffer.getbuffer().nbytes < frame_count * self.channels * 2:
+      buff = b'\x00\x00' * frame_count * self.channels
+    elif self.buffer.getbuffer().nbytes > 115200:  # 3x the usual read size
+      self.buffer.seek(0)
+      buff = self.buffer.read(frame_count * self.channels * 4)
+      buff = buff[:frame_count * self.channels * 2]
+      self.buffer.seek(2)
+    else:
+      self.buffer.seek(0)
+      buff = self.buffer.read(frame_count * self.channels * 2)
+      self.buffer.seek(2)
+    return (buff, pyaudio.paContinue)
+
+  async def __consume(self, track):
+    while True:
+      try:
+        frame = await track.recv()
+      except aiortc.MediaStreamError:
+        return
+
+      self.buffer.write(bytes(frame.planes[0]))
+
+  def hasTrack(self, track: aiortc.MediaStreamTrack) -> bool:
+    return any(t == track for t, _ in self.tracks_and_tasks)
+
+  def addTrack(self, track: aiortc.MediaStreamTrack):
+    if not self.hasTrack(track):
+      self.tracks_and_tasks.append((track, None))
+
+  def start(self):
+    for index, (track, task) in enumerate(self.tracks_and_tasks):
+      if task is None:
+        self.tracks_and_tasks[index] = (track, asyncio.create_task(self.__consume(track)))
+
+  def stop(self):
+    for _, task in self.tracks_and_tasks:
+      if task is not None:
+        task.cancel()
+
+    self.tracks_and_tasks = []
+    self.stream.stop_stream()
+    self.stream.close()
+    self.p.terminate()