Audio stream detected but not saved to video recordings

[s-fairchild]

Hello,

The recorded video files do not contain any audio.

Here is a section of the logs at LOG_LEVEL=9. I see that audio is detected, but no other audio related logs were found. I wasn't able to find any documentation concerning audio configuration other than what's in the main README.md.
I have 3 cameras connected via rtsp, the other 2 aren't recording audio either but I see the same message for the other cameras.

motionplus[318243]: [INF][NET][04:nc04:frontdoor] netcam_connect: Norm: An audio stream was detected.  Capture_rate increased to compensate.

motionplus[318243]: [NTC][NET][04:nc04:frontdoor] netcam_connect: Norm: Camera (frontdoor) connected
motionplus[318243]: [NTC][NET][04:nc04:frontdoor] netcam_connect: Norm: Netcam capture FPS is 30.
motionplus[318243]: [NTC][NET][04:nc04:frontdoor] netcam_connect: Norm: Camera source is 20 FPS
motionplus[318243]: [INF][NET][04:nc04:frontdoor] netcam_connect: Norm: An audio stream was detected.  Capture_rate increased to compensate.
motionplus[318243]: [INF][NET][04:nc04:frontdoor] netcam_open_sws: Norm: Pixel format 13 will be converted.
motionplus[318243]: [INF][ENC][04:nc04:frontdoor] movie_avcodec_log: deprecated pixel format used, make sure you did set range correctly
motionplus[318243]: [INF][ENC][04:nc04:frontdoor] movie_avcodec_log: unable to decode APP fields: Invalid data found when processing input
motionplus[318243]: [INF][ENC][04:nc04:frontdoor] movie_avcodec_log: CSeq 6 expected, 0 received.
motionplus[318243]: [INF][ENC][04:nc04:frontdoor] movie_avcodec_log: Above message repeats 38 times
motionplus[318243]: [INF][NET][04:nc04:frontdoor] netcam_interrupt: Norm: Camera reading (frontdoor) timed out
motionplus[318243]: [INF][NET][04:nc04:frontdoor] netcam_read_image: Norm: Interrupted
motionplus[318243]: [INF][NET][04:nc04:frontdoor] netcam_interrupt: Norm: Camera reading (frontdoor) timed out
motionplus[318243]: [ERR][NET][04:nc04:frontdoor] netcam_handler_reconnect: Norm: Reconnecting with camera....
motionplus[318243]: [INF][NET][04:nc04:frontdoor] netcam_set_options: Norm: Setting rtsp/rtmp
motionplus[318243]: [INF][NET][04:nc04:frontdoor] netcam_open_context: Norm: Opened camera(frontdoor)
motionplus[318243]: [INF][ENC][04:nc04:frontdoor] movie_avcodec_log: Received packet without a start chunk; dropping frame.
motionplus[318243]: [INF][ENC][04:nc04:frontdoor] movie_avcodec_log: Above message repeats 1 times
motionplus[318243]: [INF][ENC][23:motionplus] movie_avcodec_log: unable to decode APP fields: Invalid data found when processing input
motionplus[318243]: [INF][ENC][04:nc04:frontdoor] movie_avcodec_log: Received packet without a start chunk; dropping frame.
motionplus[318243]: [INF][ENC][04:nc04:frontdoor] movie_avcodec_log: Above message repeats 61 times
motionplus[318243]: [INF][NET][04:av04:frontdoor] netcam_init_swdecoder: Norm: Initializing decoder
motionplus[318243]: [NTC][NET][04:av04:frontdoor] netcam_init_swdecoder: Norm: Using decoder mjpeg
motionplus[318243]: [INF][NET][04:av04:frontdoor] netcam_open_codec: Norm: Decoder opened
motionplus[318243]: [NTC][NET][04:nc04:frontdoor] netcam_connect: Norm: Camera (frontdoor) connected
motionplus[318243]: [NTC][NET][04:nc04:frontdoor] netcam_connect: Norm: Netcam capture FPS is 30.
motionplus[318243]: [NTC][NET][04:nc04:frontdoor] netcam_connect: Norm: Camera source is 20 FPS
motionplus[318243]: [INF][NET][04:nc04:frontdoor] netcam_connect: Norm: An audio stream was detected.  Capture_rate increased to compensate.
motionplus[318243]: [INF][NET][04:nc04:frontdoor] netcam_open_sws: Norm: Pixel format 13 will be converted.
motionplus[318243]: [INF][ENC][04:nc04:frontdoor] movie_avcodec_log: deprecated pixel format used, make sure you did set range correctly
motionplus[318243]: [INF][ENC][04:nc04:frontdoor] movie_avcodec_log: CSeq 6 expected, 0 received.
motionplus[318243]: [INF][ENC][04:nc04:frontdoor] movie_avcodec_log: Above message repeats 46 times
motionplus[318243]: [INF][ENC][23:motionplus] movie_avcodec_log: unable to decode APP fields: Invalid data found when processing input

0. Reviewed guide and contributing documents? (Yes/No): Yes
1. version [x.y.z, hash, other]: MotionPlus version 0.0.0-git-20230102-ca68d76
2. installed as a package or compiled from sources [deb, rpm, git, other]: Compiled from source, running inside podman ubuntu container. Dockerfile below.

FROM docker.io/ubuntu:23.04
WORKDIR /tmp/motionplus
COPY . .

RUN apt-get update && \
  DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
      autoconf automake autopoint pkgconf libtool libjpeg8-dev build-essential libzip-dev gettext libmicrohttpd-dev libavformat-dev libavcodec-dev libavutil-dev libswscale-dev libavdevice-dev git-all ffmpeg && \
  apt-get clean && \
  rm -rf /var/lib/apt/lists/ && \
  autoreconf -fiv && \
  ./configure && \
  make && \
  make install && \
  cd .. && \
  rm -rf /tmp/motionplus

WORKDIR /usr/local/bin/

ENTRYPOINT [ "/usr/local/bin/motionplus" ]
CMD [ "-n" ]

3. video stream source [V4L2, network camera(http, rtsp, etc), mmal]: rtsp cameras, v4l2rtspserver source project
4. hardware [x86, ARM, other]: x86_64
5. operating system [18.04, Stretch, etc, FreeBSD, other]: Fedora CoreOS

cat /etc/os-release 
NAME="Fedora Linux"
VERSION="37.20221225.3.0 (CoreOS)"
ID=fedora
VERSION_ID=37
VERSION_CODENAME=""
PLATFORM_ID="platform:f37"
PRETTY_NAME="Fedora CoreOS 37.20221225.3.0"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:37"
HOME_URL="https://getfedora.org/coreos/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora-coreos/"
SUPPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
BUG_REPORT_URL="https://github.com/coreos/fedora-coreos-tracker/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=37
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=37
SUPPORT_END=2023-11-14
VARIANT="CoreOS"
VARIANT_ID=coreos
OSTREE_VERSION='37.20221225.3.0'

Attached are my configuration files.
Motionplug config for github issue.md

[Mr-Dave]

It needs to use the movie_passthrough on option since the motion detection process disconnects the audio and video. (Something I'll think about trying to keep together as an enhancement).

From what I can see in your setup, it would be specifying the same url for the netcam_high_url since it looks like it isn't a dual stream source. You could also probably add netcam_params allowed_media_types=video which would lighten the load on the primary detection.

[s-fairchild]

@Mr-Dave I've added those settings, unfortunately there still doesn't appear to be any audio recorded.

Here's ffprobe of some of the recorded movies.

ffprobe version 5.1.2 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 12 (GCC)
  configuration: --prefix=/usr --bindir=/usr/bin --datadir=/usr/share/ffmpeg --docdir=/usr/share/doc/ffmpeg --incdir=/usr/include/ffmpeg --libdir=/usr/lib64 --mandir=/usr/share/man --arch=x86_64 --optflags='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' --extra-ldflags='-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 ' --extra-cflags=' -I/usr/include/rav1e' --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libvo-amrwbenc --enable-version3 --enable-bzlib --enable-chromaprint --disable-crystalhd --enable-fontconfig --enable-frei0r --enable-gcrypt --enable-gnutls --enable-ladspa --enable-libaom --enable-libdav1d --enable-libass --enable-libbluray --enable-libbs2b --enable-libcdio --enable-libdrm --enable-libjack --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libilbc --enable-libmp3lame --enable-libmysofa --enable-nvenc --enable-openal --enable-opencl --enable-opengl --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librav1e --enable-librubberband --enable-libsmbclient --enable-version3 --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-version3 --enable-vapoursynth --enable-libvpx --enable-vulkan --enable-libshaderc --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-libxml2 --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-avfilter --enable-libmodplug --enable-postproc --enable-pthreads --disable-static --enable-shared --enable-gpl --disable-debug --disable-stripping --shlibdir=/usr/lib64 --enable-lto --enable-libmfx --enable-runtime-cpudetect
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
Input #0, matroska,webm, from 'driveway_2-01-2023:01:19_09:04:31.mkv':
  Metadata:
    ENCODER         : Lavf59.27.100
  Duration: 00:00:13.63, start: 0.000000, bitrate: 6007 kb/s
  Stream #0:0: Video: mjpeg (Baseline), yuvj422p(pc, bt470bg/unknown/unknown), 1280x720 [SAR 1:1 DAR 16:9], 30 tbr, 1k tbn
    Metadata:
      DURATION        : 00:00:13.628000000
[steven@r10 Videos]$ ffprobe driveway_2-02-2023\:01\:19_09\:07\:17.mkv 
ffprobe version 5.1.2 Copyright (c) 2007-2022 the FFmpeg developers
  built with gcc 12 (GCC)
  configuration: --prefix=/usr --bindir=/usr/bin --datadir=/usr/share/ffmpeg --docdir=/usr/share/doc/ffmpeg --incdir=/usr/include/ffmpeg --libdir=/usr/lib64 --mandir=/usr/share/man --arch=x86_64 --optflags='-O2 -flto=auto -ffat-lto-objects -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 -fstack-protector-strong -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection' --extra-ldflags='-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -Wl,--build-id=sha1 ' --extra-cflags=' -I/usr/include/rav1e' --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libvo-amrwbenc --enable-version3 --enable-bzlib --enable-chromaprint --disable-crystalhd --enable-fontconfig --enable-frei0r --enable-gcrypt --enable-gnutls --enable-ladspa --enable-libaom --enable-libdav1d --enable-libass --enable-libbluray --enable-libbs2b --enable-libcdio --enable-libdrm --enable-libjack --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libilbc --enable-libmp3lame --enable-libmysofa --enable-nvenc --enable-openal --enable-opencl --enable-opengl --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librsvg --enable-librav1e --enable-librubberband --enable-libsmbclient --enable-version3 --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libsvtav1 --enable-libtesseract --enable-libtheora --enable-libtwolame --enable-libvorbis --enable-libv4l2 --enable-libvidstab --enable-libvmaf --enable-version3 --enable-vapoursynth --enable-libvpx --enable-vulkan --enable-libshaderc --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxvid --enable-libxml2 --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-avfilter --enable-libmodplug --enable-postproc --enable-pthreads --disable-static --enable-shared --enable-gpl --disable-debug --disable-stripping --shlibdir=/usr/lib64 --enable-lto --enable-libmfx --enable-runtime-cpudetect
  libavutil      57. 28.100 / 57. 28.100
  libavcodec     59. 37.100 / 59. 37.100
  libavformat    59. 27.100 / 59. 27.100
  libavdevice    59.  7.100 / 59.  7.100
  libavfilter     8. 44.100 /  8. 44.100
  libswscale      6.  7.100 /  6.  7.100
  libswresample   4.  7.100 /  4.  7.100
  libpostproc    56.  6.100 / 56.  6.100
Input #0, matroska,webm, from 'driveway_2-02-2023:01:19_09:07:17.mkv':
  Metadata:
    ENCODER         : Lavf59.27.100
  Duration: 00:00:06.80, start: 0.000000, bitrate: 12520 kb/s
  Stream #0:0: Video: mjpeg (Baseline), yuvj422p(pc, bt470bg/unknown/unknown), 1280x720 [SAR 1:1 DAR 16:9], 250 tbr, 1k tbn
    Metadata:
      DURATION        : 00:00:06.799000000

Here's the most recent logs: motionplus_logs_2023-01-19.log

Here's my updated config files:

1. camera1.txt
2. camera2.txt
3. camera3.txt
4. motionplus.txt

[Mr-Dave]

It looks like it could be the source. All the movie_avcodec_log messages are from ffmpeg. You may want to validate that ffmpeg can create a mp4 file with just a copying of the codec provided from the source without any transcoding.

[s-fairchild]

Is it possible audio is being recorded intermittently?
I checked my recordings for the past few days and I'm getting audio successfully recording now, although I haven't done anything different from my last post.

ffprobe version 5.1.2 Copyright (c) 2007-2022 the FFmpeg developers
...
Input #0, matroska,webm, from 'dustins_room_3-142-2023:01:24_08:18:58.mkv':
  Metadata:
    ENCODER         : Lavf59.27.100
  Duration: 00:02:14.68, start: 0.000000, bitrate: 3249 kb/s
  Stream #0:0: Video: mjpeg (Baseline), yuvj422p(pc, bt470bg/unknown/unknown), 1280x720 [SAR 1:1 DAR 16:9], 250 tbr, 1k tbn
    Metadata:
      DURATION        : 00:02:14.683000000
  Stream #0:1: Audio: pcm_s16be, 48000 Hz, 1 channels, s16, 768 kb/s
    Metadata:
      DURATION        : 00:02:14.674000000

[Mr-Dave]

Certainly. The logs show Motionplus trying to record everything but many issues with the source and formats

[Philius]

I ddin't see anything in motionplus_logs_2023-01-19.log that metioned anything besides

netcam_open_sws: Norm: Pixel format 13 will be converted.
movie_avcodec_log: deprecated pixel format used, make sure you did set range correctly

There's no mention of an audio stream, unless the cameras have ones built in.

[Philius]

I set up an rtsp server using rtsp-simple-server(they want to change the projects name to MediaMtx).
It mediates media producers and consumers (in my case) running on localhost:8554.
I used ffmpeg to combine audio+video as an rtsp source.

ffmpeg -thread_queue_size 1024 -probesize 10000000 -i /dev/video0 -f pulse \
-i alsa_input.usb-HD_WEB_CAMERA_HD_WEB_CAMERA_SN0001-02.mono-fallback \
-map_channel 1.0.0 -ac 1 -c:a mp3 -vcodec libx264 -preset ultrafast -tune zerolatency \
-r 30 -s 1920x1080 -g 30 -b:v 6M -f rtsp -buffer_size 10000000 -rtsp_transport tcp \
rtsp://localhost:8554/stream &

...then waited 6 seconds for things to settle before running mpv rtsp://localhost:8554/stream to see it.
The audio is recorded along with the video only if "movie_passthrough on" is configured for the camera.
This is a problem as the throughput is about 2fps even though 30fps is requested at every place in the chain I can think of.
Plus there is a 1-2 second delay from the input to the output.

I'm no ffmpeg expert so my configuration may be sub-optimal.
For example I don't know if it's possible to use ffmpeg directly for this role in motionplus.

But my other problem is, why not have a camera option for an audio data source directly.
Since the audio packets are captured anyway for sound detection, the stream would be available for all cameras configured to use it.

Then there's the issue of two cameras with microphones, viewing the same scene but from different angles.
Let's call them "left" and "right".
You'd want the "left" camera to record its microphone audio to its left channel and the "right" cameras audio to its right channel.
Same for the "right" camera.

So now we're getting into audio mixing or ffmpeg filter graphs.
It would be more efficient to provide a "mixer" audio channel that does the L+R once for all the cameras that want it.

I've cloned the repo and all of this can be found in github.com/Philius/motionplus in the testing/rtsp-simple-server directory.

ffmpeg-cpp2 looks promising as a way to incorporate filter graphs and C++-ify a lot of the code, although I remember reading that stream timestamps aren't done yet or some such - I could be wrong.

[Philius]

I've cloned ffmpeg-cpp2.
It turns out that someone cloned it to github already so I cloned that one.