-
|
I'll caveat this question by saying I'm not sure it's a pex issue. I'm seeing some odd behavior with import collisions between the libraries This fails: (so does a variant like: This succeeds: Also interesting, when I specify interpreter constraint of python 3.12, it starts failing again (which leads me to believe this isn't a pex issue)... With the python 3.12 constraints, I'm able to get around the problem by also adding I guess my questions are:
Thanks! |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 3 replies
-
Yes. The prerequisite knowledge is 2-fold:
Addressing point 1: Since Pex never breaks existing users it only adds features and never removes them or changes defaults. As such, the best options are almost certainly not the default ones. You dug and found the best options, namely: Addressing point 2: You left out a juicy detail in your second console output section. Probably it had happened already and did not show on subsequent runs of the same venv: Namely: That's terrifying. Indeed, I find: diff -u /home/jsirois/.cache/pex/installed_wheels/0/08347d8610a5e1d4cb435e6c39ea7f39e1c2f294c829b21f4f5b18eb58a37208/google_cloud_alloydb_connector-1.6.0-py2.py3-none-any.whl/google/api/field_behavior_pb2.py /home/jsirois/.cache/pex/installed_wheels/0/d7abcd75fabb2e0ec9f74466401f6c119a0b498e27370e9be4c94cb7e382b8ed/googleapis_common_protos-1.66.0-py2.py3-none-any.whl/google/api/field_behavior_pb2.py
--- /home/jsirois/.cache/pex/installed_wheels/0/08347d8610a5e1d4cb435e6c39ea7f39e1c2f294c829b21f4f5b18eb58a37208/google_cloud_alloydb_connector-1.6.0-py2.py3-none-any.whl/google/api/field_behavior_pb2.py 2025-01-08 15:15:59.611728042 -0800
+++ /home/jsirois/.cache/pex/installed_wheels/0/d7abcd75fabb2e0ec9f74466401f6c119a0b498e27370e9be4c94cb7e382b8ed/googleapis_common_protos-1.66.0-py2.py3-none-any.whl/google/api/field_behavior_pb2.py 2025-01-08 15:15:59.489732402 -0800
@@ -1,3 +1,5 @@
+# -*- coding: utf-8 -*-
+
# Copyright 2024 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
@@ -12,11 +14,8 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-# type: ignore
-# -*- coding: utf-8 -*-
# Generated by the protocol buffer compiler. DO NOT EDIT!
# source: google/api/field_behavior.proto
-# isort: skip_file
"""Generated protocol buffer code."""
from google.protobuf import descriptor as _descriptor
from google.protobuf import descriptor_pool as _descriptor_pool
@@ -32,7 +31,7 @@
DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(
- b"\n\x1fgoogle/api/field_behavior.proto\x12\ngoogle.api\x1a google/protobuf/descriptor.proto*\xb6\x01\n\rFieldBehavior\x12\x1e\n\x1a\x46IELD_BEHAVIOR_UNSPECIFIED\x10\x00\x12\x0c\n\x08OPTIONAL\x10\x01\x12\x0c\n\x08REQUIRED\x10\x02\x12\x0f\n\x0bOUTPUT_ONLY\x10\x03\x12\x0e\n\nINPUT_ONLY\x10\x04\x12\r\n\tIMMUTABLE\x10\x05\x12\x12\n\x0eUNORDERED_LIST\x10\x06\x12\x15\n\x11NON_EMPTY_DEFAULT\x10\x07\x12\x0e\n\nIDENTIFIER\x10\x08:Q\n\x0e\x66ield_behavior\x12\x1d.google.protobuf.FieldOptions\x18\x9c\x08 \x03(\x0e\x32\x19.google.api.FieldBehaviorBp\n\x0e\x63om.google.apiB\x12\x46ieldBehaviorProtoP\x01ZAgoogle.golang.org/genproto/googleapis/api/annotations;annotations\xa2\x02\x04GAPIb\x06proto3"
+ b"\n\x1fgoogle/api/field_behavior.proto\x12\ngoogle.api\x1a google/protobuf/descriptor.proto*\xb6\x01\n\rFieldBehavior\x12\x1e\n\x1a\x46IELD_BEHAVIOR_UNSPECIFIED\x10\x00\x12\x0c\n\x08OPTIONAL\x10\x01\x12\x0c\n\x08REQUIRED\x10\x02\x12\x0f\n\x0bOUTPUT_ONLY\x10\x03\x12\x0e\n\nINPUT_ONLY\x10\x04\x12\r\n\tIMMUTABLE\x10\x05\x12\x12\n\x0eUNORDERED_LIST\x10\x06\x12\x15\n\x11NON_EMPTY_DEFAULT\x10\x07\x12\x0e\n\nIDENTIFIER\x10\x08:U\n\x0e\x66ield_behavior\x12\x1d.google.protobuf.FieldOptions\x18\x9c\x08 \x03(\x0e\x32\x19.google.api.FieldBehaviorB\x02\x10\x00\x42p\n\x0e\x63om.google.apiB\x12\x46ieldBehaviorProtoP\x01ZAgoogle.golang.org/genproto/googleapis/api/annotations;annotations\xa2\x02\x04GAPIb\x06proto3"
)
_globals = globals()
@@ -41,12 +40,10 @@
DESCRIPTOR, "google.api.field_behavior_pb2", _globals
)
if _descriptor._USE_C_DESCRIPTORS == False:
- google_dot_protobuf_dot_descriptor__pb2.FieldOptions.RegisterExtension(
- field_behavior
- )
-
DESCRIPTOR._options = None
DESCRIPTOR._serialized_options = b"\n\016com.google.apiB\022FieldBehaviorProtoP\001ZAgoogle.golang.org/genproto/googleapis/api/annotations;annotations\242\002\004GAPI"
+ field_behavior._options = None
+ field_behavior._serialized_options = b"\020\000"
_globals["_FIELDBEHAVIOR"]._serialized_start = 82
_globals["_FIELDBEHAVIOR"]._serialized_end = 264
# @@protoc_insertion_point(module_scope)
That's hard to know. If they support very old Pythons, using pkgutil namespace packages could be the way to go. That said, if they don't, they should probably just use implicit namespace packages. But they also have a bigger problem with their namespaces as the conflicting file diff shows above. They should prevent that from ever happening, but I'm not sure what advice to offer them there. EDIT (see below for experiment details): You could reccomend upstream - namely google_cloud_alloydb_connector 1.6.0 - just removes I'm not sure if that will get traction since almost no-one uses multiple |
Beta Was this translation helpful? Give feedback.
-
|
@jcrobak I guess I left out why using the PEX venv mode works here - it works because most libraries and applications in Python land assume a single sys.path entry where all user code is installed - namely a site-packages dir. A traditional PEX uses a sys.path composed of one entry per installed wheel which breaks that assumption. Likewise, a |
Beta Was this translation helpful? Give feedback.
-
|
Even more detail follows. The why behind symlinking top-level packages vs creating a directory tree with symlinked files: Lines 430 to 477 in 991883c The symlinked And looking at those dirs you see: The google.api packages comes from 2 dists: One is googleapis_common_protos 1.66.0: The other is google_cloud_alloydb_connector 1.6.0: One uses an implicit namespace package ( If I kill the non-namespace package When |
Beta Was this translation helpful? Give feedback.
-
|
Thanks, John. I really appreciate the detailed explanation and going the step further to suggest the fix upstream! |
Beta Was this translation helpful? Give feedback.
Yes. The prerequisite knowledge is 2-fold:
Addressing point 1: Since Pex never breaks existing users it only adds features and never removes them or changes defaults. As such, the best options are almost certainly not the default ones. You dug and found the best options, namely:
--venv prepend --venv-site-packages-copies. This produces a venv nearly identical to the one you would construct by hand using whatever tools you'd normally use - for examplepython -mvenvandpip installin that venv. The…