Fix snap build - TypeError: expected string or bytes-like object

[dasgarner]

The snap build currently fails with the following error:

Pulling gstreamer 
+ snapcraftctl pull
Cloning into '/build/xibo-player/parts/gstreamer/src'...
[28/Jan/2022:08:39:18 +0000] "CONNECT github.com:443 HTTP/1.1" 200 252984123 "-" "git/2.17.1"
+ git checkout 1.16.2
Note: checking out '1.16.2'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at 1294936877 Release 1.16.2
'gst-plugins-base' has dependencies that need to be staged: gstreamer
Skipping pull gstreamer (already ran)
Building gstreamer 
+ snapcraftctl build
Sorry, an error occurred in Snapcraft:
expected string or bytes-like object
Traceback (most recent call last):
  File "/snap/snapcraft/7010/bin/snapcraft", line 8, in <module>
    sys.exit(run())
  File "/snap/snapcraft/7010/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/snap/snapcraft/7010/lib/python3.8/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/snap/snapcraft/7010/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/snap/snapcraft/7010/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/snap/snapcraft/7010/lib/python3.8/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/snap/snapcraft/7010/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/snap/snapcraft/7010/lib/python3.8/site-packages/snapcraft/cli/lifecycle.py", line 303, in pull
    _execute(steps.PULL, parts, **kwargs)
  File "/snap/snapcraft/7010/lib/python3.8/site-packages/snapcraft/cli/lifecycle.py", line 102, in _execute
    lifecycle.execute(step, project_config, parts)
  File "/snap/snapcraft/7010/lib/python3.8/site-packages/snapcraft/internal/lifecycle/_runner.py", line 145, in execute
    executor.run(step, part_names)
  File "/snap/snapcraft/7010/lib/python3.8/site-packages/snapcraft/internal/lifecycle/_runner.py", line 199, in run
    self._handle_step(part_names, part, step, current_step, cli_config)
  File "/snap/snapcraft/7010/lib/python3.8/site-packages/snapcraft/internal/lifecycle/_runner.py", line 213, in _handle_step
    getattr(self, "_run_{}".format(current_step.name))(part)
  File "/snap/snapcraft/7010/lib/python3.8/site-packages/snapcraft/internal/lifecycle/_runner.py", line 255, in _run_pull
    self._run_step(step=steps.PULL, part=part, progress="Pulling")
  File "/snap/snapcraft/7010/lib/python3.8/site-packages/snapcraft/internal/lifecycle/_runner.py", line 330, in _run_step
    self._prepare_step(step=step, part=part)
  File "/snap/snapcraft/7010/lib/python3.8/site-packages/snapcraft/internal/lifecycle/_runner.py", line 315, in _prepare_step
    self.run(prerequisite_step, dependency_names)
  File "/snap/snapcraft/7010/lib/python3.8/site-packages/snapcraft/internal/lifecycle/_runner.py", line 199, in run
    self._handle_step(part_names, part, step, current_step, cli_config)
  File "/snap/snapcraft/7010/lib/python3.8/site-packages/snapcraft/internal/lifecycle/_runner.py", line 213, in _handle_step
    getattr(self, "_run_{}".format(current_step.name))(part)
  File "/snap/snapcraft/7010/lib/python3.8/site-packages/snapcraft/internal/lifecycle/_runner.py", line 266, in _run_build
    self._run_step(step=steps.BUILD, part=part, progress="Building")
  File "/snap/snapcraft/7010/lib/python3.8/site-packages/snapcraft/internal/lifecycle/_runner.py", line 333, in _run_step
    getattr(part, step.name)()
  File "/snap/snapcraft/7010/lib/python3.8/site-packages/snapcraft/internal/pluginhandler/__init__.py", line 599, in build
    self._do_build()
  File "/snap/snapcraft/7010/lib/python3.8/site-packages/snapcraft/internal/pluginhandler/__init__.py", line 691, in _do_build
    self._do_runner_step(steps.BUILD)
  File "/snap/snapcraft/7010/lib/python3.8/site-packages/snapcraft/internal/pluginhandler/__init__.py", line 280, in _do_runner_step
    return getattr(self._runner, "{}".format(step.name))()
  File "/snap/snapcraft/7010/lib/python3.8/site-packages/snapcraft/internal/pluginhandler/_runner.py", line 91, in build
    self._run_scriptlet(
  File "/snap/snapcraft/7010/lib/python3.8/site-packages/snapcraft/internal/pluginhandler/_runner.py", line 189, in _run_scriptlet
    raise error
  File "/snap/snapcraft/7010/lib/python3.8/site-packages/snapcraft/internal/pluginhandler/_runner.py", line 178, in _run_scriptlet
    self._handle_builtin_function(
  File "/snap/snapcraft/7010/lib/python3.8/site-packages/snapcraft/internal/pluginhandler/_runner.py", line 233, in _handle_builtin_function
    function(**function_args)
  File "/snap/snapcraft/7010/lib/python3.8/site-packages/snapcraft/plugins/v1/autotools.py", line 109, in build
    self.run(["env", "NOCONFIGURE=1", script])
  File "/snap/snapcraft/7010/lib/python3.8/site-packages/snapcraft/plugins/v1/_plugin.py", line 190, in run
    cmd_string = " ".join([shlex.quote(c) for c in cmd])
  File "/snap/snapcraft/7010/lib/python3.8/site-packages/snapcraft/plugins/v1/_plugin.py", line 190, in <listcomp>
    cmd_string = " ".join([shlex.quote(c) for c in cmd])
  File "/snap/snapcraft/7010/usr/lib/python3.8/shlex.py", line 325, in quote
    if _find_unsafe(s) is None:
TypeError: expected string or bytes-like object
You can find the traceback in file '/tmp/tmpfwk4ahr_/trace.txt'.
Build failed
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/lpbuildd/target/build_snap.py", line 202, in run
    self.pull()
  File "/usr/lib/python3/dist-packages/lpbuildd/target/build_snap.py", line 172, in pull
    env=env)
  File "/usr/lib/python3/dist-packages/lpbuildd/target/operation.py", line 50, in run_build_command
    return self.backend.run(args, cwd=cwd, env=full_env, **kwargs)
  File "/usr/lib/python3/dist-packages/lpbuildd/target/lxd.py", line 539, in run
    subprocess.check_call(cmd, **kwargs)
  File "/usr/lib/python3.6/subprocess.py", line 311, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['lxc', 'exec', 'lp-bionic-amd64', '--env', 'LANG=C.UTF-8', '--env', 'SHELL=/bin/sh', '--env', 'http_proxy=http://10.10.10.1:8222/', '--env', 'https_proxy=http://10.10.10.1:8222/', '--env', 'GIT_PROXY_COMMAND=/usr/local/bin/lpbuildd-git-proxy', '--env', 'SNAPPY_STORE_NO_CDN=1', '--env', 'SNAPCRAFT_LOCAL_SOURCES=1', '--env', 'SNAPCRAFT_SETUP_CORE=1', '--env', 'SNAPCRAFT_BUILD_INFO=1', '--env', 'SNAPCRAFT_IMAGE_INFO={"build-request-id": "lp-68918776", "build-request-timestamp": "2022-01-28T08:33:45Z", "build_url": "https://launchpad.net/~build.snapcraft.io/+snap/1c6db12c6030af028126a96db3b89dcb/+build/1656544"}', '--env', 'SNAPCRAFT_BUILD_ENVIRONMENT=host', '--', '/bin/sh', '-c', 'cd /build/xibo-player && linux64 snapcraft pull']' returned non-zero exit status 1.

[gamenfs]

Hi
I'm sorry, I'm not good at compiling snap applications, but I was able to customize the compilation for my purposes.
I couldn't deal with the problem of a locally installed snap file (the x1 revision problem #243 (comment)), so I pushed the app to the snap repository to install it.

OS: Ubuntu 18.04.6 LTS
snapcraft 6.0.2
multipass 1.8.0

below is modified snapcraft.yaml file:

base: core18
version: 1.8-R6
summary: Xibo Linux Player
description: Xibo for Linux Digital Signage Player
#grade: stable
grade: devel
#confinement: strict
confinement: devmode

architectures:
  - build-on: amd64

#slots:
#  dbus-daemon: # name this whatever you want
#    interface: dbus
#    bus: session
#    name: org.gtkmm.xibo # adjust accordingly

apps:
  xibo-player:
    command: bin/snap_run.sh xibo-watchdog
    plugs: [gsettings, network, home, audio-playback, screen-inhibit-control, framebuffer, opengl]
#    slots: [dbus-daemon]
    extensions: [gnome-3-28]

  disabled-watchdog:
    command: bin/snap_run.sh xibo-watchdog --disable-restart
    plugs: [gsettings, network, home, audio-playback, screen-inhibit-control, framebuffer, opengl]
#    slots: [dbus-daemon]
    extensions: [gnome-3-28]

  options:
    command: bin/xibo-options
    plugs: [gsettings, network, home]
#    slots: [dbus-daemon]
    extensions: [gnome-3-28]

parts:
  base:
    plugin: nil
    build-packages:
      - software-properties-common
      - apt-transport-https
      - ca-certificates 
      - gnupg 
      - curl 
      - wget
      - unzip
      - libgtkmm-3.0-dev 
      - libwebkitgtk-3.0-dev
      - libxss-dev
      - libglibmm-2.4-dev
      - libssl-dev

  g++-8: # Ubuntu 18.04 has outdated gcc so we have to install it from PPA
    plugin: nil
    override-pull: |
      add-apt-repository ppa:ubuntu-toolchain-r/test
      apt-get update
      apt-get install -y g++-8
      update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-8 90
      update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-8 90
    after: [base]

  cmake:
    plugin: nil
    override-pull: |
      wget -nv https://apt.kitware.com/keys/kitware-archive-latest.asc -O Release.key
      apt-key add - < Release.key
      apt-add-repository 'deb https://apt.kitware.com/ubuntu/ bionic main'
      apt-get update
      apt-get install -y cmake
    after: [g++-8]

  zmq: # Ubuntu 18.04 has outdated ZMQ so we are pulling it from official stable repo
    plugin: nil
    override-pull: |
      sh -c "echo 'deb http://download.opensuse.org/repositories/network:/messaging:/zeromq:/release-stable/xUbuntu_18.04/ /' > /etc/apt/sources.list.d/network:messaging:zeromq:release-stable.list"
      wget -nv https://download.opensuse.org/repositories/network:messaging:zeromq:release-stable/xUbuntu_18.04/Release.key -O Release.key
      apt-key add - < Release.key
      apt-get update
      apt-get install -y libzmq3-dev
    after: [cmake]

  boost: # Ubuntu 18.04 has outdated boost so we have to install it from PPA
    plugin: nil
    override-pull: |
      add-apt-repository ppa:mhier/libboost-latest
      apt-get update
      apt-get install -y libboost1.70-dev
    after: [zmq]

  spdlog: 
    source: https://github.com/gabime/spdlog/archive/v1.4.1.tar.gz
    plugin: cmake
    after: [cmake]

  gtest: 
    source: https://codeload.github.com/google/googletest/tar.gz/release-1.8.1
    source-type: tar
    plugin: cmake
    after: [cmake]

  cryptopp:
    plugin: nil
    override-pull: |
      curl -o cryptopp.tar.gz -SL https://github.com/weidai11/cryptopp/archive/CRYPTOPP_8_1_0.tar.gz && \
      curl -o cryptopp_pem.zip -SL https://github.com/noloader/cryptopp-pem/archive/095f08ff2ef9bca7b81036a59f2395e4f08ce2e8.zip && \
      tar -zxvf cryptopp.tar.gz && \
      unzip -a cryptopp_pem.zip && \
      cp -r cryptopp-pem-095f08ff2ef9bca7b81036a59f2395e4f08ce2e8/. cryptopp-CRYPTOPP_8_1_0 && \
      cd cryptopp-CRYPTOPP_8_1_0 && \
      make -j4 && \
      make install

  gstreamer:
#    override-pull: |
#      snapcraftctl pull
#      git checkout 1.16.2
#    source: https://github.com/GStreamer/gstreamer.git
    source: https://gstreamer.freedesktop.org/src/gstreamer/gstreamer-1.16.2.tar.xz
    plugin: autotools
    configflags: ['--prefix=/usr', '--disable-gtk-doc']
    build-packages:
      - bison
      - flex
      - gettext
      - liborc-0.4-dev
      - libpulse-dev
      - libxv-dev
      - libtag1-dev
      - libwavpack-dev
      - libgl1-mesa-dev
      - libglu1-mesa-dev
      - libglew-dev
    stage-packages:
      - liborc-0.4-dev
      - libasound2-dev
      - libjpeg-dev
      - libvpx-dev
      - libvorbis-dev
      - libogg-dev
      - libpulse-dev
      - libpulse0
      - libwavpack-dev
      - libtag1v5
      - libopus-dev
      - libtheora-dev
      - libxv-dev
      - libxdamage-dev
      - libcdparanoia-dev
      - libgl1-mesa-glx
      - libglu1-mesa
      - libglew2.0
    stage:
      - -usr/lib/x86_64-linux-gnu/libLLVM-10.so.1
      - -usr/share/doc/libllvm10/changelog.Debian.gz
      - -usr/share/doc/gcc-11-base/changelog.Debian.gz
      - -lib/x86_64-linux-gnu/libgcc_s.so.1
    after: [g++-8]

  gst-plugins-base:
#    override-pull: |
#      snapcraftctl pull
#      git checkout 1.16.2
#    source: https://github.com/GStreamer/gst-plugins-base.git
    source: https://gstreamer.freedesktop.org/src/gst-plugins-base/gst-plugins-base-1.16.2.tar.xz
    plugin: autotools
    configflags: ['--prefix=/usr', '--disable-gtk-doc']
    build-packages:
      - liborc-0.4-dev
      - libfribidi-dev
      - libpulse-dev
    after: [gstreamer]

  gst-libav:
#    override-pull: |
#      snapcraftctl pull
#      git checkout 1.16.2
#    source: https://github.com/GStreamer/gst-libav.git
    source: https://gstreamer.freedesktop.org/src/gst-libav/gst-libav-1.16.2.tar.xz
    plugin: autotools
    configflags: ['--prefix=/usr', '--disable-gtk-doc']
    build-packages:
      - yasm
      - liborc-0.4-dev
      - libpulse-dev
      - libva-dev
    stage-packages:
      - libpulse0
    after: [gst-plugins-base]

  gst-plugins-good:
#    override-pull: |
#      snapcraftctl pull
#      git checkout 1.16.2
#    source: https://github.com/GStreamer/gst-plugins-good.git
    source: https://gstreamer.freedesktop.org/src/gst-plugins-good/gst-plugins-good-1.16.2.tar.xz
    plugin: autotools
    configflags: ['--prefix=/usr', '--disable-gtk-doc']
    build-packages:
      - liborc-0.4-dev
      - libpulse-dev
    stage-packages:
      - libpulse0
    after: [gst-libav]

  date-tz:
    source: https://github.com/HowardHinnant/date/archive/v3.0.0.tar.gz
    plugin: cmake
    build-packages:
      - libcurl4-gnutls-dev
    configflags:
      - -DBUILD_TZ_LIB=ON
      - -DBUILD_SHARED_LIBS=ON
      - -DUSE_SYSTEM_TZ_DB=ON
    after: [cmake]

  sqlite-orm:
    source: https://github.com/fnc12/sqlite_orm/archive/refs/tags/1.6.tar.gz
    plugin: cmake
    build-packages:
      - libsqlite3-dev
    stage-packages:
      - libsqlite3-0
    configflags:
      - -DBUILD_TESTING=OFF
    after: [cmake]

  player:
    source: player
    plugin: cmake
    configflags:
      - -DCMAKE_BUILD_TYPE=Release
      - -DAPP_ENV=SNAP
    disable-parallel: true
    stage-packages:
      - freeglut3
      - libzmq5
      - libboost1.70
      - libgtkmm-3.0-1v5
      - libcanberra-gtk3-module
      - libwebkitgtk-3.0-0
      - libgpm2 # gstreamer warning
      - libslang2 # gstreamer warning
    after: [zmq, boost, spdlog, gtest, gstreamer, date-tz, sqlite-orm] ```

[dasgarner]

It may also be more transparent if we build snap packages using a Github Action instead of via the snap store. We could use https://github.com/snapcore/action-build for this.

This would have the added benefit of creating a release when we tag and uploading a snap to that release which could be manually installed.

[dasgarner]

Fixed in #264

[dasgarner]

Closing this was premature as further testing reveals that even though we can build referencing gstreamer archives instead of git checkout - video doesn't actually play.

[dasgarner]

TypeError: expected string or bytes-like object

The above error is fixed by this PR: canonical/snapcraft#3628, which unfortunately did not quite make it into 6.1. I have tried a build with snapcraft from edge (which is version 7) and the build is successful. However with v7 I then can't login to snapcraft.

I think the solution will be to wait for the next release of snapcraft, which contains the fix we need.

It may be that this gets solved out by either of these two issues:

• Wayland: Video won't play. Unable to create player: gtkglsink is missing. #268
• Snap: build against 20.04 #267

[dasgarner]

A final update (I hope).

The video issue seen when referencing the gstreamer archives in snap actually turned out to be a video issue under Wayland and a false lead.

Further testing with the snap built using the gstreamer archives (rev 238 in edge) shows that it is functional, video plays and all seems well.

With that I think this issue can be closed with a big thanks to @gamenfs for pointing us in the right direction.