Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Need to reinstall reticulate whenever I want to access arcpy #1618

Closed
andrew-caudillo opened this issue Jun 14, 2024 · 13 comments
Closed

Need to reinstall reticulate whenever I want to access arcpy #1618

andrew-caudillo opened this issue Jun 14, 2024 · 13 comments

Comments

@andrew-caudillo
Copy link

andrew-caudillo commented Jun 14, 2024

Hello,
When I try to access arcpy (using the python.exe from ArcPro), I receive an error. Here is my workflow:

library(reticulate)
reticulate::use_python('C://ArcGIS//Pro//bin//Python//envs//arcgispro-py3/python.exe',required = T) #set python
Sys.setenv(PATH = paste("C://ArcGIS//Pro/bin", Sys.getenv("PATH"), sep = ";")) #manually ensure variables are loaded correctly
py_config() #check to see if python environment exists
sa=reticulate::import("arcpy.sa") #try to load spatial analyst
#receive error: Error in py_module_import(module, convert = convert) : ImportError: DLL load failed while importing #_arcgisscripting: The specified procedure could not be found.

But, if I restart R, reinstall reticulate, and follow the same steps, "sa" runs succesfully.

install.packages('reticulate')
library(reticulate)
reticulate::use_python('C://ArcGIS//Pro//bin//Python//envs//arcgispro-py3/python.exe',required = T) #set python
Sys.setenv(PATH = paste("C://ArcGIS//Pro/bin", Sys.getenv("PATH"), sep = ";")) #manually ensure variables are loaded correctly
py_config() #check to see if python environment exists
sa=reticulate::import("arcpy.sa") #succesfully import acrpy.sa

Is this something that has happened before with anyone? Both my colleague and I are experiencing this issue, so it is not isolated to my machine. We both have to restart R and reinstall reticulate every time we need to use it. We both have Windows machines.

Thank you,

@t-kalinowski
Copy link
Member

t-kalinowski commented Jun 17, 2024

Thanks for opening. I haven't seen this error before.

My best guess is that reinstalling is forcing some dll's to be reloaded, which is fixing the error.

If you set PATH or any other envvar after initializing Python, the Python session will not see those env vars. You would need to independently also set in Python with import("os")$environ$update(...), or call R's Sys.setenv() before initializing Python.

Does swapping the order of your initialization commands fix the issue?

@andrew-caudillo
Copy link
Author

@t-kalinowski ,
Thank you for getting back.

If I understand correctly, you are saying to do this:

reticulate::use_python('C://ArcGIS//Pro//bin//Python//envs//arcgispro-py3/python.exe',required = T) #set python
Sys.setenv(PATH = paste("C://ArcGIS//Pro/bin", Sys.getenv("PATH"), sep = ";")) #manually ensure variables are loaded correctly
library(reticulate)
sa=reticulate::import("arcpy.sa") #try to load spatial analyst

if that is the case, then no, the issue persists after rearranging my order of operations.

@t-kalinowski
Copy link
Member

I'm sugesting something different actually:

# make sure you're in a fresh R session.
# in RStudio IDE: rstudioapi::restartSession(clean = TRUE)

library(reticulate)

# set env vars before initializing Python
Sys.setenv(PATH = sprintf("%s;%s", normalizePath("C:/ArcGIS/Pro/bin"), Sys.getenv("PATH"))

# initialize Python
use_python('C:/ArcGIS/Pro/bin/Python/envs/arcgispro-py3/python.exe')

# Confirm the correct Python was initialized
py_config() 

# import desired module
sa <- reticulate::import("arcpy.sa")

@andrew-caudillo
Copy link
Author

@t-kalinowski thank you for the clarification.

The order of operations that you have written worked for me-- there was no error. I did not know about the order of operations for reticulate.

I appreciate your quick response on this matter!

@andrew-caudillo
Copy link
Author

hi @t-kalinowski ,
I am encountering this same issue again, even when following your order of operations. If I reinstall reticulate, it fixes the issue.

@t-kalinowski
Copy link
Member

The next step for diagnosing "module not found" issues is to inspect some values, like:

import("sys")$path
Sys.getenv("PATH")
Sys.getenv("PYTHONPATH")
import("os")$environ$get("PATH")
import("os")$environ$get("PYTHONPATH")

@andrew-caudillo
Copy link
Author

andrew-caudillo commented Nov 7, 2024

I'm not sure how I should be diagnosing these outputs, but they are all valid filepaths.

However, the first output for import("sys")$path and Sys.getenv("PYTHONPATH") are blanks (" ").

@t-kalinowski
Copy link
Member

Can you provide them here?
Do the values change between when things work and when they don't?
Where is the missing _arcgisscripting actually located?

@t-kalinowski
Copy link
Member

Note that you can also insert print statements directly in the python module. E.g., if you see that a Python statement like import _arcgisscripting is raising an exception, you can navigate to that Python file, and edit it in place by inserting above that line something like:

import sys; print(f"{sys.path=}")
import _arcgisscripting

@andrew-caudillo
Copy link
Author

andrew-caudillo commented Nov 7, 2024

arcgisscripting is located within the ArcGIS Pro folders.

Here is an example of what outputs I am getting, specifically from the first import() line you suggested for diagnosing. It seems like it is bringing up a lot of file paths that use python?

C:/ArcGIS/Pro/bin/Python/envs/arcgispro-py3\\Library\\bin;
C:\\ArcGIS\\Pro\\bin;
C:\\ArcGIS\\Pro\\bin\\Python\\envs\\ARCGIS~1\\Library\\bin;
C:\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3;
C:\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3\\Library\\mingw-w64\\bin;
C:\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3\\Library\\usr\\bin;
C:\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3\\Library\\bin;
C:\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3\\Scripts;
C:\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3\\bin;
C:\\Users\\foo~1.BUG\\AppData\\Local\\Temp\\_MEI102962\\pywin32_system32;
C:\\ArcGIS\\Pro\\bin\\Python;
C:\\ArcGIS\\Pro\\bin\\Python\\Library\\mingw-w64\\bin;
C:\\ArcGIS\\Pro\\bin\\Python\\Library\\usr\\bin;
C:\\ArcGIS\\Pro\\bin\\Python\\Library\\bin;
C:\\ArcGIS\\Pro\\bin\\Python\\Scripts;
C:\\ArcGIS\\Pro\\bin\\Python\\bin;
C:\\ArcGIS\\Pro\\bin\\Python\\condabin;
C:\\Program Files\\ArcGIS\\Pro\\bin\\Python\\condabin;
C:\\Program Files\\ArcGIS\\Pro\\bin\\Python\\Scripts;
C:\\Program Files\\ArcGIS\\Pro\\bin\\Python\\Library\\bin;
c:\\rtools43\\x86_64-w64-mingw32.static.posix\\bin;
c:\\rtools43\\usr\\bin;
C:\\Program Files\\R\\R-4.3.2\\bin\\x64;
C:\\Program Files\\Microsoft MPI\\Bin;
C:\\WINDOWS\\system32;
C:\\WINDOWS;
C:\\WINDOWS\\System32\\Wbem;
C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0;
C:\\WINDOWS\\System32\\OpenSSH;
C:\\Program Files\\Git\\cmd;
C:\\Program Files\\Amazon\\AWSCLIV2;
C:\\Program Files\\dotnet;
C:\\GDAL;
C:\\Program Files\\GDAL;
C:\\Program Files\\TauDEM\\TauDEM5Exe;
C:\\Program Files\\WindowsPowerShell\\Scripts;
C:\\Users\\foo\\AppData\\Local\\Microsoft\\WindowsApps;
C:\\Users\\foo\\.dotnet\\tools;
C:\\Program Files\\RStudio\\resources\\app\\bin\\quarto\\bin;
C:\\Program Files\\RStudio\\resources\\app\\bin\\postback;
C:\\ArcGIS\\Pro\\bin\\Python\\envs\\ARCGIS~1;
C:\\ArcGIS\\Pro\\bin\\Python\\envs\\ARCGIS~1\\Scripts

@andrew-caudillo
Copy link
Author

if it helps diagnose any issues, this is the line that actually caused my script to fail, according to reticulate.
py_module_import(module, convert = convert)

@andrew-caudillo
Copy link
Author

Could the issue stem from how python was installed? I did not install from python's website, as I have to use ESRI's version of python to use these modules/functions. I chose this version of python as my interpreter for my R project, as well.

@andrew-caudillo
Copy link
Author

arcgisscripting is located within the ArcGIS Pro folders.

Here is an example of what outputs I am getting, specifically from the first import() line you suggested for diagnosing. It seems like it is bringing up a lot of file paths that use python?

C:/ArcGIS/Pro/bin/Python/envs/arcgispro-py3\\Library\\bin;
C:\\ArcGIS\\Pro\\bin;
C:\\ArcGIS\\Pro\\bin\\Python\\envs\\ARCGIS~1\\Library\\bin;
C:\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3;
C:\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3\\Library\\mingw-w64\\bin;
C:\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3\\Library\\usr\\bin;
C:\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3\\Library\\bin;
C:\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3\\Scripts;
C:\\ArcGIS\\Pro\\bin\\Python\\envs\\arcgispro-py3\\bin;
C:\\Users\\foo~1.BUG\\AppData\\Local\\Temp\\_MEI102962\\pywin32_system32;
C:\\ArcGIS\\Pro\\bin\\Python;
C:\\ArcGIS\\Pro\\bin\\Python\\Library\\mingw-w64\\bin;
C:\\ArcGIS\\Pro\\bin\\Python\\Library\\usr\\bin;
C:\\ArcGIS\\Pro\\bin\\Python\\Library\\bin;
C:\\ArcGIS\\Pro\\bin\\Python\\Scripts;
C:\\ArcGIS\\Pro\\bin\\Python\\bin;
C:\\ArcGIS\\Pro\\bin\\Python\\condabin;
C:\\Program Files\\ArcGIS\\Pro\\bin\\Python\\condabin;
C:\\Program Files\\ArcGIS\\Pro\\bin\\Python\\Scripts;
C:\\Program Files\\ArcGIS\\Pro\\bin\\Python\\Library\\bin;
c:\\rtools43\\x86_64-w64-mingw32.static.posix\\bin;
c:\\rtools43\\usr\\bin;
C:\\Program Files\\R\\R-4.3.2\\bin\\x64;
C:\\Program Files\\Microsoft MPI\\Bin;
C:\\WINDOWS\\system32;
C:\\WINDOWS;
C:\\WINDOWS\\System32\\Wbem;
C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0;
C:\\WINDOWS\\System32\\OpenSSH;
C:\\Program Files\\Git\\cmd;
C:\\Program Files\\Amazon\\AWSCLIV2;
C:\\Program Files\\dotnet;
C:\\GDAL;
C:\\Program Files\\GDAL;
C:\\Program Files\\TauDEM\\TauDEM5Exe;
C:\\Program Files\\WindowsPowerShell\\Scripts;
C:\\Users\\foo\\AppData\\Local\\Microsoft\\WindowsApps;
C:\\Users\\foo\\.dotnet\\tools;
C:\\Program Files\\RStudio\\resources\\app\\bin\\quarto\\bin;
C:\\Program Files\\RStudio\\resources\\app\\bin\\postback;
C:\\ArcGIS\\Pro\\bin\\Python\\envs\\ARCGIS~1;
C:\\ArcGIS\\Pro\\bin\\Python\\envs\\ARCGIS~1\\Scripts

when I reinstall reticulate, the outputs are different. Most of the files live within the ArcGIS Pro folder.

[1] ""
[2] "C:\ArcGIS\Pro\bin\Python\envs\arcgispro-py3"
[3] "C:\ArcGIS\Pro\Resources\ArcPy"
[4] "C:\ArcGIS\Pro\bin\Python\envs\ARCGIS1\python39.zip"
[5] "C:\ArcGIS\Pro\bin\Python\envs\ARCGIS
1\DLLs"
[6] "C:\ArcGIS\Pro\bin\Python\envs\ARCGIS1\lib"
[7] "C:\ArcGIS\Pro\bin\Python\envs\ARCGIS
1"
[8] "C:\ArcGIS\Pro\bin\Python\envs\ARCGIS1\lib\site-packages"
[9] "C:\ArcGIS\Pro\bin"
[10] "C:\ArcGIS\Pro\Resources\ArcToolbox\Scripts"
[11] "C:\ArcGIS\Pro\bin\Python\envs\ARCGIS
1\lib\site-packages\Babel-2.11.0-py3.9.egg"
[12] "C:\ArcGIS\Pro\bin\Python\envs\ARCGIS1\lib\site-packages\future-0.18.2-py3.9.egg"
[13] "C:\ArcGIS\Pro\bin\Python\envs\ARCGIS
1\lib\site-packages\pytz-2022.6-py3.9.egg"
[14] "C:\ArcGIS\Pro\bin\Python\envs\ARCGIS1\lib\site-packages\pywin32security"
[15] "C:\ArcGIS\Pro\bin\Python\envs\ARCGIS
1\lib\site-packages\sympy-1.9-py3.9.egg"
[16] "C:\Users\foo.foo\AppData\Local\R\win-library\4.3\reticulate\python"
[17] "C:\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\python39.zip"
[18] "C:\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\DLLs"
[19] "C:\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib"
[20] "C:\Program Files\RStudio\resources\app\bin"
[21] "C:\ArcGIS\Pro\bin\Python\envs\arcgispro-py3"
[22] "C:\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages"
[23] "C:\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\Babel-2.11.0-py3.9.egg"
[24] "C:\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\future-0.18.2-py3.9.egg"
[25] "C:\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\pytz-2022.6-py3.9.egg"
[26] "C:\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\pywin32security"
[27] "C:\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\sympy-1.9-py3.9.egg"

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants