[BUG] Shebangs replaced by non working ones on install

[AlexisTM]

setuptools version

setuptools==78.1.0

Python version

python 3.10

OS

Ubuntu 22.04 ARM64

Additional environment information

I am building rosmaster

It creates a package then calls setup(...) for the install part.

Source: https://github.com/ros/ros_comm/tree/noetic-devel/tools/rosmaster

Description

The shebangs #!/usr/bin/env python are being replaced by #!python making the scripts non working.

This seems to be related to

setuptools/setuptools/_distutils/command/build_scripts.py

Lines 105 to 113 in 6ead555

	if shebang_match:
	log.info("copying and adjusting %s -> %s", script, self.build_dir)
	if not self.dry_run:
	post_interp = shebang_match.group(1) or ''
	shebang = f"#!python{post_interp}\n"
	self._validate_shebang(shebang, f.encoding)
	with open(outfile, "w", encoding=f.encoding) as outf:
	outf.write(shebang)
	outf.writelines(f.readlines())

Expected behavior

The shebangs are not modified

How to Reproduce

pip install rosinstall_generator

rosinstall_generator --rosdistro noetic --deps --tar \
    rosmaster    > rosmaster.rosinstall && \
    mkdir -p src && \
    vcs import src < rosmaster.rosinstall 

ROS_VERSION=1 ./src/catkin/bin/catkin_make_isolated --install \
      -DCMAKE_BUILD_TYPE=Release \
      -DPYTHON_EXECUTABLE=/usr/bin/python3

Output

$ cat install_isolated/bin/rosmaster
#!python
[...]

[AlexisTM]

Reverting setuptools with pip install setuptools==75.8.2 fixes it.

The commit is c712663

[dvzrv]

@jaraco since you have added the commit in question: What is the planned way forward for distributions?

We're currently hit by this bug on Arch Linux as well (https://gitlab.archlinux.org/archlinux/packaging/packages/python-setuptools/-/issues/4), which breaks all upcoming package builds that have an entrypoint.
Looking at the pypa/installer project - which we use for installation of wheels - it seems they don't want to implement a feature like this (custom setting of interpreters/shebang): pypa/installer#107

Downstreams can currently only work around this issue by adding custom wrapper scripts around setuptools or installer to have functioning scripts (however, this does not scale) 🤔

[jaraco]

That commit mentions #4863, where Setuptools is has been asked by downstream installers (uv in particular) to provide a consistent, predictable shebang so it can be rewritten by the installer to match the target Python environment.

breaks all upcoming package builds that have an entrypoint.

I believe this statement is incorrect. If you have a project where entry points are affected, please report that separately. The issue reported in this issue (and I believe affected by the indicated commit), pertains to user-provided scripts (such as declared here).

For the record, rosmaster probably shouldn't be using scripts with hard-coded, unix-specific scripts. Instead, that project should use entry points, which are portable and require less redundancy. That said, while scripts are discouraged in favor of entry-point-based console scripts, they're still supported.

I note also that rosmaster is using the explicitly deprecated direct call of setuptools.setup(). Setuptools is meant to be accessed as a PEP 517 build backend. Unfortunately, it's difficult to deprecate the use of setuptools.setup() since the build backend accesses it, but it really shouldn't be invoked by other programs.

How to Reproduce

I'm not comfortable with this reproducer. While I respect that it causes the undesirable behavior, it's using tools (rosinstall/catkin) with which I'm unfamiliar and that may have unknown effects on any system I might use to replicate the issue. Can you come up with a reproducer that distills the problem down to something that only involves Setuptools?

Don't spend too much time on it, however, if all it does is call setup.py install. The value in supporting standards-based installation by tools like pip and uv are going to outweigh any compatibility with a custom installer wrapper.

My suspicion is that a tool like rosinstall is going to need to adapt and perform the same transformations that tools like uv and pip do when installing scripts.

[stefanor]

@dvzrv: This caused fallout in Debian too. I raised this on discord, and filed #4952 to request a revert.

[stefanor]

@jaraco: #4863 was specifically requesting the behaviour change for the wheel that build_editable produces, but it has applied to setuptools build directory, which impacts other flows.

[stefanor]

@jaraco: You want a minimal reproducer? Here: https://gist.github.com/stefanor/c6a59e920655c95406f9b221dfefb733

(I'm not sure if that's the same issue as the the ROS toolchain hits, but it's the one I described in #4952)

[lilydjwg]

@jaraco entrypoints are indeed not affected, but scripts are. The installer does indeed not have this problem. (I just rechecked; I don't know why my result was different yesterday.) So this just means directly calling setup.py is finally broken?

@stefanor I remember that #!/usr/bin/env python doesn't reproduce this issue; it needs to be #!/usr/bin/python.

[FFY00]

That commit mentions #4863, where Setuptools is has been asked by downstream installers (uv in particular) to provide a consistent, predictable shebang so it can be rewritten by the installer to match the target Python environment.

That should only apply to wheels. not direct setup.py invocations.

@jaraco, consider the following reproducer (thanks @lilydjwg — https://gitlab.archlinux.org/archlinux/packaging/packages/python-setuptools/-/issues/4#note_264683):

$ cat setup.py
#!/usr/bin/python

from setuptools import setup

setup(
  name = 'test package',
  scripts = ['script'],
)

$ cat script
#!/usr/bin/python

print('script!')

$ python setup.py build
running build
running build_scripts
creating build/scripts-3.13
copying and adjusting script -> build/scripts-3.13
changing mode of build/scripts-3.13/script from 644 to 755

$ python setup.py install --root=destdir/
running install
/usr/lib/python3.13/site-packages/setuptools/_distutils/cmd.py:90: SetuptoolsDeprecationWarning: setup.py install is deprecated.
!!

        ********************************************************************************
        Please avoid running ``setup.py`` directly.
        Instead, use pypa/build, pypa/installer or other
        standards-based tools.

        See https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html for details.
        ********************************************************************************

!!
  self.initialize_options()
running build
running build_scripts
running install_egg_info
running egg_info
creating test_package.egg-info
writing test_package.egg-info/PKG-INFO
writing dependency_links to test_package.egg-info/dependency_links.txt
writing top-level names to test_package.egg-info/top_level.txt
writing manifest file 'test_package.egg-info/SOURCES.txt'
[04/22/25 06:33:47] ERROR    listing git files failed - pretending there aren't any                                                                                                                                                                                                           git.py:26
reading manifest file 'test_package.egg-info/SOURCES.txt'
writing manifest file 'test_package.egg-info/SOURCES.txt'
Copying test_package.egg-info to destdir/usr/lib/python3.13/site-packages/test_package-0.0.0-py3.13.egg-info
running install_scripts
creating destdir/usr/bin
copying build/scripts-3.13/script -> destdir/usr/bin
changing mode of destdir/usr/bin/script to 755

$ cat destdir/usr/bin/script
#!python

print('script!')

[FFY00]

breaks all upcoming package builds that have an entrypoint.

I believe this statement is incorrect. If you have a project where entry points are affected, please report that separately. The issue reported in this issue (and I believe affected by the indicated commit), pertains to user-provided scripts (such as declared here).

FWIW, I was unable to reproduce the issue for entrypoints, only for user scripts. Perhaps the two mechanisms were being confused.

[AlexisTM]

I'm not comfortable with this reproducer. While I respect that it causes the undesirable behavior, it's using tools (rosinstall/catkin) with which I'm unfamiliar and that may have unknown effects on any system I might use to replicate the issue. Can you come up with a reproducer that distills the problem down to something that only involves Setuptools?

Don't spend too much time on it, however, if all it does is call setup.py install. The value in supporting standards-based installation by tools like pip and uv are going to outweigh any compatibility with a custom installer wrapper.

I agree, the reproducer was not good. It is indeed calling setup.py install under the hood.

[lazka]

Just for completeness, that commit also removed/broke the "--executable" feature of build_scripts. The tests for this feature didn't catch it though:

setuptools/setuptools/tests/test_install_scripts.py

Lines 23 to 33 in 56962ec

	def _run_install_scripts(self, install_dir, executable=None):
	dist = Distribution(self.settings)
	dist.script_name = 'setup.py'
	cmd = install_scripts(dist)
	cmd.install_dir = install_dir
	if executable is not None:
	bs = cmd.get_finalized_command('build_scripts')
	bs.executable = executable
	cmd.ensure_finalized()
	with contexts.quiet():
	cmd.run()

because they happen to test with the now hardcoded value.

[FFY00]

Yes, I pointed that out in pypa/distutils#332.