FEAT: Universal Mac Builds and Support (#88)

### Summary  
This pull request introduces significant updates to the build pipeline
and codebase to support universal2 binaries for macOS and improve
artifact organization. The changes include adding a new macOS build job,
updating architecture handling for macOS to prioritize universal2
binaries, and refining artifact naming and placement for better clarity.

### Build Pipeline Updates:
* Added a new `BuildMacOSWheels` job in
`eng/pipelines/build-whl-pipeline.yml` to build universal2 binaries for
macOS. This includes steps for setting up Python, installing
dependencies, building `.so` files, and publishing artifacts.
* Updated artifact paths and names in the build pipeline to reflect the
new structure, such as renaming `all-pyds` to `ddbc-bindings` and
updating artifact names to `mssql-python-ddbc-bindings`.
[[1]](diffhunk://#diff-a871d64acfb78366b4b077045db53551c2df7d622854569f5f6780ad2ec6f911L126-R125)
[[2]](diffhunk://#diff-a871d64acfb78366b4b077045db53551c2df7d622854569f5f6780ad2ec6f911L157-R164)

### macOS Universal2 Support:
* Modified `CMakeLists.txt` to always use universal2 architecture for
macOS, ensuring compatibility with both `arm64` and `x86_64`. Added a
new CMake setting to enforce universal builds.
[[1]](diffhunk://#diff-dbb5892fbbb28149d1639664797cf3adb48ced28ec11aba95f2e2b338ca46badL31-R32)
[[2]](diffhunk://#diff-dbb5892fbbb28149d1639664797cf3adb48ced28ec11aba95f2e2b338ca46badR43-R48)
[[3]](diffhunk://#diff-dbb5892fbbb28149d1639664797cf3adb48ced28ec11aba95f2e2b338ca46badL72-R70)
* Updated `build.sh` to simplify the build process for macOS by removing
architecture-specific logic and focusing on universal2 binaries. Added
diagnostics to verify the universal binary output.
[[1]](diffhunk://#diff-7cfcd9b5499a176c28ca4d028e9b2b060238ec79c4bea5b062d4b39929f9fb04L2-R31)
[[2]](diffhunk://#diff-7cfcd9b5499a176c28ca4d028e9b2b060238ec79c4bea5b062d4b39929f9fb04L87-R85)

### Codebase Adjustments for Universal2:
* Updated `ddbc_bindings.py` to prioritize universal2 binaries on macOS,
regardless of local architecture. Added error handling for unsupported
platforms.
* Adjusted `setup.py` to always use the `macosx_15_0_universal2`
platform tag for macOS wheels, ensuring consistent naming and
compatibility.
[[1]](diffhunk://#diff-60f61ab7a8d1910d86d9fda2261620314edcae5894d5aaa236b821c7256badd7L41-R44)
[[2]](diffhunk://#diff-60f61ab7a8d1910d86d9fda2261620314edcae5894d5aaa236b821c7256badd7L75-R78)

### Documentation and Examples:
* Updated the `README.md` in `mssql_python/pybind` to reflect the new
universal2 binary naming convention (e.g., `.so` files named with
`universal2`).

These changes streamline the build process, enhance cross-platform
compatibility, and improve artifact organization, aligning the project
with modern macOS development practices.

### Issue Reference  
Fixes
[AB#37761](https://sqlclientdrivers.visualstudio.com/c6d89619-62de-46a0-8b46-70b92a84d85e/_workitems/edit/37761)

Author: bewithgaurav

eng/pipelines/build-whl-pipeline.yml

@@ -187,19 +187,19 @@ jobs:
   displayName: 'Build macOS - '
   strategy:
     matrix:
-      # Python 3.13 (only x86_64 for builds)
-      py313_x86_64:
+      # Python 3.13 (universal2 for both arm64 and x86_64)
+      py313_universal2:
         pythonVersion: '3.13'
         shortPyVer: '313'
-        # targetArch is not used anywhere since build.sh does auto-detection, kept for cross-compilation of ARM64 Builds
-        targetArch: 'x86_64'
+        # Always use universal2 for macOS
+        targetArch: 'universal2'
   steps:
     # Use correct Python version and architecture for the current job
     - task: UsePythonVersion@0
       inputs:
         versionSpec: '$(pythonVersion)'
         addToPath: true
-      displayName: 'Use Python $(pythonVersion) (x86_64)'
+      displayName: 'Use Python $(pythonVersion) (Universal2)'
 
     # Install CMake on macOS
     - script: |
@@ -218,7 +218,7 @@ jobs:
     - script: |
         echo "Python Version: $(pythonVersion)"
         echo "Short Tag: $(shortPyVer)"
-        echo "Architecture: Target=$(targetArch)"
+        echo "Building Universal2 Binary"
         cd "$(Build.SourcesDirectory)/mssql_python/pybind"
         # Call build.sh to build the .so file
         ./build.sh
@@ -232,13 +232,12 @@ jobs:
         TargetFolder: '$(Build.ArtifactStagingDirectory)/ddbc-bindings'
       displayName: 'Place .so file into artifacts directory'
 
-    # Build wheel package for the current architecture
+    # Build wheel package for universal2
     - script: |
         python -m pip install --upgrade pip
         pip install wheel setuptools
-        set ARCHITECTURE=$(targetArch)
         python setup.py bdist_wheel
-      displayName: 'Build wheel package for Python $(pythonVersion) ($(targetArch))'
+      displayName: 'Build $(pythonVersion) universal2 whl'
     
     # Copy the wheel file to the artifacts
     - task: CopyFiles@2

mssql_python/ddbc_bindings.py

@@ -5,15 +5,25 @@
 
 # Get current Python version and architecture
 python_version = f"cp{sys.version_info.major}{sys.version_info.minor}"
-if platform.machine().lower() in ('amd64', 'x86_64', 'x64'):
-    architecture = "amd64" if sys.platform == 'win32' else "x86_64"
-elif platform.machine().lower() in ('arm64', 'aarch64'):
-    architecture = "arm64"
+
+platform_name = sys.platform.lower()
+architecture = platform.machine().lower()
+
+# On macOS, prioritize universal2 binary regardless of the local architecture
+if platform_name == 'darwin':
+    architecture = "universal2"
+elif platform_name == 'win32':
+    if architecture in ('amd64', 'x86_64', 'x64'):
+        architecture = "amd64" if platform_name == 'win32' else "x86_64"
+    elif architecture in ('arm64', 'aarch64'):
+        architecture = "arm64"
+    else:
+        raise ImportError(f"Unsupported architecture for mssql-python: {platform_name}-{architecture}")
 else:
-    architecture = platform.machine().lower()
+    raise ImportError(f"Unsupported architecture for mssql-python: {platform_name}-{architecture}")
 
 # Determine extension based on platform
-if sys.platform == 'win32':
+if platform_name == 'win32':
     extension = '.pyd'
 else:  # macOS or Linux
     extension = '.so'

mssql_python/pybind/CMakeLists.txt

@@ -28,17 +28,8 @@ if(NOT DEFINED ARCHITECTURE)
         # Default to x64 for Windows
         set(ARCHITECTURE "win64")
     elseif(APPLE)
-        # Check if we're on Apple Silicon or Intel
-        execute_process(
-            COMMAND sysctl -n machdep.cpu.brand_string
-            OUTPUT_VARIABLE CPU_BRAND
-            OUTPUT_STRIP_TRAILING_WHITESPACE
-        )
-        if(CPU_BRAND MATCHES "Apple")
-            set(ARCHITECTURE "arm64")
-        else()
-            set(ARCHITECTURE "x64")
-        endif()
+        # Always use universal2 for macOS
+        set(ARCHITECTURE "universal2")
     else()
         # Default to x64 for other platforms
         set(ARCHITECTURE "x64")
@@ -49,6 +40,12 @@ endif()
 add_definitions(-DARCHITECTURE="${ARCHITECTURE}")
 add_definitions(-DPLATFORM_NAME="${PLATFORM_NAME}")
 
+# For macOS, always set universal build
+if(APPLE)
+    set(CMAKE_OSX_ARCHITECTURES "arm64;x86_64" CACHE STRING "Build architectures for macOS" FORCE)
+    message(STATUS "Setting universal2 build for macOS (arm64 + x86_64)")
+endif()
+
 # Get Python version and platform info
 execute_process(
     COMMAND python -c "import sys; print(f'{sys.version_info.major}{sys.version_info.minor}')"
@@ -69,12 +66,8 @@ if(WIN32)
         message(FATAL_ERROR "Unsupported Windows architecture: ${ARCHITECTURE}. Supported architectures are win32, win64, x86, amd64, arm64.")
     endif()
 elseif(APPLE)
-    # macOS architecture mapping
-    if(ARCHITECTURE STREQUAL "arm64")
-        set(WHEEL_ARCH "arm64")
-    else()
-        set(WHEEL_ARCH "x86_64")
-    endif()
+    # macOS always uses universal2
+    set(WHEEL_ARCH "universal2")
 else()
     # Linux architecture mapping
     if(ARCHITECTURE STREQUAL "arm64")
@@ -209,6 +202,7 @@ else()
     set(MODULE_EXTENSION ".so")
 endif()
 
+# Set the output name based on the Python version and architecture
 set_target_properties(ddbc_bindings PROPERTIES 
     PREFIX ""
     OUTPUT_NAME "ddbc_bindings.cp${PYTHON_VERSION}-${WHEEL_ARCH}"

mssql_python/pybind/README.md

@@ -60,5 +60,5 @@ This README provides instructions to build the DDBC Bindings PYD for your system
    - Detect system architecture
    - Configure CMake with appropriate include/library paths
    - Compile `ddbc_bindings_mac.cpp` using CMake
-   - Generate the `.so` file (e.g., `ddbc_bindings.cp313-arm64.so`)
+   - Generate the `.so` file (e.g., `ddbc_bindings.cp313-universal2.so`)
    - Copy the output SO file to the parent `mssql_python` directory

mssql_python/pybind/build.sh

@@ -1,76 +1,34 @@
 #!/bin/bash
-# Build script for macOS to compile the ddbc_bindings C++ code
+# Build script for macOS to compile a universal2 (arm64 + x86_64) binary
 # This script is designed to be run from the mssql_python/pybind directory
-# Running this script will require CMake and a C++ compiler installed on your MacOS system
-# It will also require Python 3.x, pybind11 and msodbcsql18 and unixODBC to be installed
-
-# Usage: build.sh [ARCH], If ARCH is not specified, it defaults to the current architecture
-ARCH=${1:-$(uname -m)}
-echo "[DIAGNOSTIC] Target Architecture set to: $ARCH"
-
-# Clean up main build directory if it exists
-echo "Checking for main build directory..."
-if [ -d "build" ]; then
-    echo "Removing existing build directory..."
-    rm -rf build
-    echo "Build directory removed."
-fi
 
 # Get Python version from active interpreter
 PYTAG=$(python -c "import sys; print(f'{sys.version_info.major}{sys.version_info.minor}')")
 
 echo "==================================="
-echo "Building for: $ARCH / Python $PYTAG"
+echo "Building Universal2 Binary for Python $PYTAG"
 echo "==================================="
 
 # Save absolute source directory
 SOURCE_DIR=$(pwd)
 
-# Go to build output directory
-BUILD_DIR="${SOURCE_DIR}/build/${ARCH}/py${PYTAG}"
+# Clean up main build directory if it exists
+echo "Checking for build directory..."
+if [ -d "build" ]; then
+    echo "Removing existing build directory..."
+    rm -rf build
+    echo "Build directory removed."
+fi
+
+# Create build directory for universal binary
+BUILD_DIR="${SOURCE_DIR}/build"
 mkdir -p "${BUILD_DIR}"
 cd "${BUILD_DIR}"
 echo "[DIAGNOSTIC] Changed to build directory: ${BUILD_DIR}"
 
-# Set platform-specific flags for different architectures
-if [ "$ARCH" = "x86_64" ]; then
-    # x86_64 architecture
-    echo "[DIAGNOSTIC] Detected Intel Chip x86_64 architecture"
-    CMAKE_ARCH="x86_64"
-    echo "[DIAGNOSTIC] Using x86_64 architecture for CMake"
-elif [ "$ARCH" = "arm64" ]; then
-    # arm64 architecture
-    echo "[DIAGNOSTIC] Detected Apple Silicon Chip arm64 architecture"
-    CMAKE_ARCH="arm64"
-    echo "[DIAGNOSTIC] Using arm64 architecture for CMake"
-else
-    echo "[ERROR] Unsupported architecture: $ARCH"
-    exit 1
-fi
-
-echo "[DIAGNOSTIC] Source directory: ${SOURCE_DIR}"
-
-# Special handling for macOS ODBC headers and string conversion issues
-if [ "$(uname)" = "Darwin" ]; then
-    # Check if macOS-specific source file exists
-    if [ -f "${SOURCE_DIR}/ddbc_bindings_mac.cpp" ]; then
-        echo "[DIAGNOSTIC] Using macOS-specific source file: ddbc_bindings_mac.cpp"
-    else
-        echo "[WARNING] macOS-specific source file ddbc_bindings_mac.cpp not found"
-        echo "[WARNING] Falling back to standard source file ddbc_bindings.cpp"
-    fi
-    
-    # Configure CMake with macOS-specific flags
-    echo "[DIAGNOSTIC] Running CMake configure with macOS-specific settings"
-    cmake -DCMAKE_OSX_ARCHITECTURES=${CMAKE_ARCH} \
-          -DARCHITECTURE=${ARCH} \
-          -DMACOS_STRING_FIX=ON \
-          "${SOURCE_DIR}"
-else
-    # Configure CMake for other platforms
-    echo "[DIAGNOSTIC] Running CMake configure for non-macOS platform"
-    cmake -DARCHITECTURE=${ARCH} "${SOURCE_DIR}"
-fi
+# Configure CMake (architecture settings handled in CMakeLists.txt)
+echo "[DIAGNOSTIC] Running CMake configure (universal2 is set automatically)"
+cmake -DMACOS_STRING_FIX=ON "${SOURCE_DIR}"
 
 # Check if CMake configuration succeeded
 if [ $? -ne 0 ]; then
@@ -87,32 +45,41 @@ if [ $? -ne 0 ]; then
     echo "[ERROR] CMake build failed"
     exit 1
 else
-    echo "[SUCCESS] Build completed successfully"
+    echo "[SUCCESS] Universal2 build completed successfully"
+    
     # List the built files
     echo "Built files:"
     ls -la *.so
 
     # Copy the built .so file to the mssql_python directory
     PARENT_DIR=$(dirname "$SOURCE_DIR")
-    echo "[ACTION] Copying the .so file to $PARENT_DIR"
+    echo "[ACTION] Copying the universal2 .so file to $PARENT_DIR"
     cp -f *.so "$PARENT_DIR"
     if [ $? -eq 0 ]; then
-        echo "[SUCCESS] .so file copied successfully"
+        echo "[SUCCESS] Universal2 .so file copied successfully"
 
-        # Only on macOS, run the dylib configuration script to fix library paths and codesign
-        if [ "$(uname)" = "Darwin" ]; then
-            echo "[ACTION] Configuring and codesigning dylibs for macOS"
-            chmod +x "${SOURCE_DIR}/configure_dylibs.sh"
-            "${SOURCE_DIR}/configure_dylibs.sh"
-            if [ $? -eq 0 ]; then
-                echo "[SUCCESS] macOS dylibs configured and codesigned successfully"
-            else
-                echo "[WARNING] macOS dylib configuration encountered issues"
-                # Don't exit with error, as the build itself was successful
-            fi
+        # Configure dylib paths and codesign
+        echo "[ACTION] Configuring and codesigning dylibs for macOS"
+        chmod +x "${SOURCE_DIR}/configure_dylibs.sh"
+        "${SOURCE_DIR}/configure_dylibs.sh"
+        if [ $? -eq 0 ]; then
+            echo "[SUCCESS] macOS dylibs configured and codesigned successfully"
+        else
+            echo "[WARNING] macOS dylib configuration encountered issues"
         fi
     else
-        echo "[ERROR] Failed to copy .so file"
+        echo "[ERROR] Failed to copy universal2 .so file"
         exit 1
     fi
 fi
+
+# Check if the file is a universal binary
+SO_FILE=$(ls -1 *.so | head -n 1)
+echo "[DIAGNOSTIC] Checking if ${SO_FILE} is a universal binary..."
+lipo -info "${SO_FILE}"
+
+# Check if the file has the correct naming convention
+if [[ "${SO_FILE}" != *universal2* ]]; then
+    echo "[WARNING] The .so file doesn't have 'universal2' in its name, even though it's a universal binary."
+    echo "[WARNING] You may need to run the build again after the CMakeLists.txt changes are applied."
+fi

mssql_python/pybind/ddbc_bindings_mac.cpp

@@ -410,7 +410,13 @@ DriverHandle LoadDriverOrThrowException() {
 #elif defined(__APPLE__)    
     // macOS: Build path to libmsodbcsql.18.dylib - first check in lib subdirectory
     // We are supporting both Intel and Apple Silicon architectures, so we need to check the architecture
-    std::string macosDriverPath = moduleDir + "/libs/macos/" + ARCHITECTURE + "/lib/libmsodbcsql.18.dylib"; 
+    std::string runtimeArch = 
+    #ifdef __arm64__
+        "arm64";
+    #else
+        "x86_64";
+    #endif
+    std::string macosDriverPath = moduleDir + "/libs/macos/" + runtimeArch + "/lib/libmsodbcsql.18.dylib";
 
     // Check if file exists using traditional C file functions instead of std::filesystem
     FILE* file = fopen(macosDriverPath.c_str(), "r");

setup.py

@@ -38,9 +38,10 @@ def finalize_options(self):
             print(f"Setting wheel platform tag to: {self.plat_name}")
 
             # Force platform-specific paths if bdist_dir is already set
-            if self.bdist_dir and "win-amd64" in self.bdist_dir:
-                self.bdist_dir = self.bdist_dir.replace("win-amd64", f"win-{platform_dir}")
-                print(f"Using build directory: {self.bdist_dir}")
+        elif sys.platform.startswith('darwin'):
+            # For macOS, always use universal2
+            self.plat_name = "macosx_15_0_universal2"
+            print(f"Setting wheel platform tag to: {self.plat_name} (universal2)")
 
 # Find all packages in the current directory
 packages = find_packages()
@@ -72,20 +73,9 @@ def finalize_options(self):
         f'mssql_python.libs.{arch}.vcredist'
     ])
 elif sys.platform.startswith('darwin'):
-    # macOS platform
-    import platform
-    arch = os.environ.get('ARCHITECTURE', None)
-    
-    # Auto-detect architecture if not specified
-    if arch is None:
-        if platform.machine() == 'arm64':
-            arch = 'arm64'
-            platform_tag = 'macosx_15_0_arm64'
-        elif platform.machine() == 'x86_64':
-            arch = 'x86_64'
-            platform_tag = 'macosx_15_0_x86_64'
-        else:
-            raise Exception("Unsupported architecture for macOS.")
+    # macOS platform - always use universal2
+    arch = 'universal2'
+    platform_tag = 'macosx_15_0_universal2'  # Use macOS 15.0 (Monterey) as minimum for universal2
 
     # Add architecture-specific packages for macOS
     packages.extend([