wireshark: build with QT 6

[lorenz]

Description of changes

Wireshark 4 supports Qt 6 so let's use that.

Tested on GNOME 44 Wayland, nothing was broken and the window decorations now look better.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 23.05 Release Notes (or backporting 22.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

[yu-re-ka]

Builds and runs fine on aarch64-darwin

[yu-re-ka]

CC @NixOS/darwin-maintainers is this a regression for x86_64-darwin and if yes how do we fix it?

[teutat3s]

Can reproduce the same failure on x86_64-darwin. Tried building wireshark from nixos-unstable on the same machine and it fails with a different error.

FAILED: capture/CMakeFiles/caputils.dir/capture-pcap-util.c.o

It looks like it did still build fine here #227852

[Tungsten842]

CC @NixOS/darwin-maintainers is this a regression for x86_64-darwin and if yes how do we fix it?

I don't think that this is a regression, it was already broken: https://hydra.nixos.org/eval/1794647?filter=wireshark&compare=1794631&full=#tabs-still-fail

[wegank]

20 packages built:
compactor credslayer credslayer.dist dbmonster haka hfinger hfinger.dist ostinato python310Packages.pyshark python310Packages.pyshark.dist python311Packages.pyshark python311Packages.pyshark.dist qtwirediff termshark tshark tshark.dev wifite2 wifite2.dist wireshark wireshark.dev

[quentinmit]

I just found this PR after I updated to 23.05; I was successfully using Wireshark on x86_64-darwin before this PR, but it now fails to build with an error in Qt 6. For now I'm just using an overlay to revert to qt5 - would you maybe consider having the package work with both?

[lorenz]

I don't think this is related. Wireshark was broken on x86_64-darwin before this PR (see comment above), also the error message (on Hydra, don't have a macOS handy to test) indicates that it is likely related to a macOS SDK issue (see also #210558).,

[quentinmit]

I don't think this is related. Wireshark was broken on x86_64-darwin before this PR (see comment above), also the error message (on Hydra, don't have a macOS handy to test) indicates that it is likely related to a macOS SDK issue (see also #210558).,

@lorenz I'm not sure what was happening on Hydra, but Wireshark built for me before this PR, and it builds again for me with this overlay:

  wireshark-qt5 = (prev.wireshark.overrideAttrs (old: {
    pname = "wireshark-qt5";

    cmakeFlags = prev.lib.lists.remove "-DUSE_qt6=ON" old.cmakeFlags ++ [
      "-DCMAKE_BUILD_WITH_INSTALL_NAME_DIR=ON"
    ];
    preFixup = old.preFixup + final.lib.optionalString final.stdenv.isDarwin ''
      # https://bugreports.qt.io/browse/QTBUG-81370
      qtWrapperArgs+=(--set QT_MAC_WANTS_LAYER 1)
      # Remove the executable bit from plugins so that Nix doesn't try to wrap them
      find $out/Applications/Wireshark.app/Contents/PlugIns ! -type d -executable -exec chmod a-x {} \;
    '';
  })).override {
    qt6 = final.qt5 // {
      qt5compat = null;
    };
    stdenv = if final.stdenv.isDarwin then final.darwin.apple_sdk_11_0.stdenv else final.stdenv;
    buildPackages = final.buildPackages // final.lib.optionalAttrs final.stdenv.isDarwin {
      stdenv = final.buildPackages.darwin.apple_sdk_11_0.stdenv;
    };
  };

Note that most of this is fixing unrelated runtime problems (the app still builds and runs without the cmakeFlags and preFixup, you just can't load all the plugins and there are graphical glitches).