On Mac, run in container ...

if podman is configured with krunkit, or if Docker is installed. Signed-off-by: Daniel J Walsh <[email protected]>
rhatdan · Oct 11, 2024 · 50f0c49 · 50f0c49
1 parent 6226362
commit 50f0c49
Show file tree

Hide file tree

Showing 2 changed files with 35 additions and 5 deletions.
diff --git a/docs/ramalama.1.md b/docs/ramalama.1.md
@@ -26,6 +26,11 @@ behaviour.
 
 RamaLama supports multiple AI model registries types called transports. Supported transports:
 
+Note:
+
+On Macs with Arm support and Podman, the Podman machine must be
+configured to use the krunkit VM Type. This allows the Mac's GPU to be
+used within the VM.
 
 ## TRANSPORTS
 
@@ -117,7 +122,7 @@ store AI Models in the specified directory (default rootless: `$HOME/.local/shar
 
 
 ## SEE ALSO
-**[podman(1)](https://github.com/containers/podman/blob/main/docs/podman.1.md)**
+**[podman(1)](https://github.com/containers/podman/blob/main/docs/podman.1.md)**, **docker(1)**
 
 ## HISTORY
 Aug 2024, Originally compiled by Dan Walsh <[email protected]>
diff --git a/ramalama/cli.py b/ramalama/cli.py
@@ -30,14 +30,41 @@ class HelpException(Exception):
     pass
 
 
+def ai_support_in_vm():
+    conman = container_manager()
+    if conman == "":
+        return False
+
+    if conman == "podman":
+        conman_args = [conman, "machine", "list", "--format", "{{ .VMType }}"]
+        try:
+            output = run_cmd(conman_args).stdout.decode("utf-8").strip()
+            if output == "krunkit":
+                return True
+        except subprocess.CalledProcessError:
+            pass
+        perror(
+            """\
+Warning: podman needs to be configured to use krunkit for AI Workloads,
+running without containers
+"""
+        )
+        return False
+    # Assume this is running with Docker and return true
+    return True
+
+
 def use_container():
     transport = os.getenv("RAMALAMA_IN_CONTAINER")
     if transport:
         return transport.lower() == "true"
 
-    if in_container() or sys.platform == "darwin":
+    if in_container():
         return False
 
+    if sys.platform == "darwin":
+        return ai_support_in_vm()
+
     return True
 
 
@@ -375,9 +402,7 @@ def run_cli(args):
 
 def serve_parser(subparsers):
     parser = subparsers.add_parser("serve", help="serve REST API on specified AI Model")
-    parser.add_argument(
-        "-d", "--detach", action="store_true", dest="detach", help="run the container in detached mode"
-    )
+    parser.add_argument("-d", "--detach", action="store_true", dest="detach", help="run the container in detached mode")
     parser.add_argument("-n", "--name", dest="name", help="name of container in which the Model will be run")
     parser.add_argument("-p", "--port", default="8080", help="port for AI Model server to listen on")
     parser.add_argument(