error upgrading to 0.0.25 in windows 10 #214
Comments
|
Interesting. The error log says that -O3 is not recognized, so maybe We have two lines of attack:
Trying (1) first: can you try installing Microsoft Visual C++ 14.x with Visual Studio 2022 and see if doing so makes Incidentally, the link above does tell me that I should try to use |
|
I switched to |
|
I installed Microsoft Visual and got the same error. I then tried to "tell pip" where the c compiler is at. set CC=C:\path\to\gcc.exe or On Windows, if you are using Visual Studio, the easiest way is usually to open a "Developer Command Prompt for VS" They both failed the same way. I think you might have hard coded "CC" to "cl" so none of the options know what to do with "cl"... If there's no fix for this, maybe you can make the cython part optional so that other parts of the package can be upgraded and used...I think you use cython to speed up the distance calculations which is not that critical compared to other functions |
|
I presume you didn't literally type I'm willing to skip the fast distance calculations for Windows if necessary, but I want to try a bit more to see if we can get this working. If we can, I'll add documentation to the readme. |
|
yes of course I didn't type that...(you're forgiven for saying that). |
|
v.0.0.27 still has the same problem...any chance of adding the distance calculations bypass for windows. |
|
Sorry for the delay on this. I don't work in Windows, so I haven't gotten around to debugging this issue. If you're okay installing from source, as a temporary workaround I added a git clone --branch no-cython [email protected]:Infleqtion/qLDPC.git
pip install -e qLDPC |
|
Thanks, this installed without problems and the find_bbcode_layouts example had no issues. I don't think my sw skills are enough to debug the cython problem so hopefully you can also create a no-cython version for future releases. |
|
My intention is to make everything work in Windows, with cython. It may just take some time 🙂 In the meantime, I'll periodically merge Note, by the way, that there is a breaking change in a recent PR (#234) that was merged into |
|
Shot in the dark @vtomole have you ever had to get cython code to compile in Windows? |
|
I have not.
|
|
I don't use Linux subsystem. I do have Centos virtual machines but my main development is under windows unfortunately |
PR #211 added cython code to compute exact code distance by brute force. It's compute-heavy, so much better to dispatch to C (via cython) if possible. |
|
How slow is the pure python code compared to the cython? How much time was spent optimizing the pure python code until you threw your hands up and used cython? What were the bottlenecks of the pure python e.t.c? I'm trying to confirm that we haven't moved to C/C++ bindings too early. |
|
For "pure python", the biggest bottleneck is essentially that import time
import numpy as np
from qldpc import codes
from qldpc.objects import Pauli
num_trials = 1000
code = codes.ToricCode(6)
stabilizers = code.get_code(Pauli.X).canonicalized().matrix.view(np.ndarray).astype(np.uint8)
logical_ops = code.get_logical_ops(Pauli.X).view(np.ndarray).astype(np.uint8)
start = time.time()
for _ in range(num_trials):
codes.common.get_distance_quantum(logical_ops, stabilizers, homogeneous=True)
print((time.time() - start) / num_trials, "sec/trial")which for me prints |
|
Just a side note : GAP has a package (QDistRnd) that calculates the distance of a quantum code and from my experience it looks fast and accurate. Maybe you can consider it as an option. |
Having said that, I certainly think QDistRnd would be be nice to add as an alternative to |
|
Looking at This is what the AI is telling me. Let's try it?
|
|
I am generally bullish on numpy vectorization, but I am skeptical that it would help here. One immediate issue is memory footprint. To compute the distance of A second issue is that even if you have the required memory (in RAM), I am still skeptical that this would be faster. At the moment, each new |
|
Does the memory footprint issue apply to cython as well? |
|
No it does not. That memory footprint is the cost of doing everything in the |
|
I'm at a loss then. Looks like we need to dev with cython on Windows. A good first start is adding a Windows check to this repo. |
Let see how they do. |
|
Another side note : see this issue for another c code with deterministic distance calculation. |
|
Re: trying JAX or numba, I made a quick attempt in the Re: subsystem code support, this is something I am currently working on. Unfortunately it's not just an incremental addition, but it's in the works. The plan is to define codes by a matrix of gauge group generators (as opposed to stabilizer group generators). This turns out to require a bunch of minor changes in various places, but more importantly it requires (a) an added method to compute and separate out stabilizer generators from "gauge logicals", and (b) a calculation of logical Paulis from the gauge generator matrix. And this needs to be done for both the QuditCode and the CSSCode class. |
From @richrines1: Batching will reduce the memory footprint. |
|
@qodesign could I ask you to again try |
|
git clone --branch no-cython [email protected]:Infleqtion/qLDPC.git doesn't seem to include cython; I installed it and ran the command python build-cython.py |
|
Oh! It looks like the cython code was compiled! Can you now switch to import qldpc
code = qldpc.codes.RepetitionCode(3)
code._exact_distance = None
print(code.get_code_params())Regarding
That is correct behavior: I am still confused though why the log you shared suggests that the |
|
v.0.0.28 installs ok; so the problem seems to be fixed. |
|
I'm not sure of a good stress test. The runtime is exponential in block length and logical qubit number, so runtime goes from < 1 sec to > 1 hour pretty quickly. You could try the tests here I suppose. I would still like to close the loop on identifying the import sysconfig
print(sysconfig.get_config_var("CC")) |
|
I get "None" returned. as a check : echo %CC% returns the expected value...so the variable is defined |
|
Okay, so there are two issues:
For issue (1), what if you try the following? import os
print(os.environ.get("CC")) |
|
One more thing to try: import distutils.ccompiler
distutils.ccompiler.new_compiler().compiler_type |
|
(added print) it says msvc |
|
Which one says |
|
I made "CC" a global environment variable instead of being set by the batch file. This seems to make a difference; now I get : import os C:\mingw-w64\x86_64-7.2.0-posix-seh-rt_v5-rev1\mingw64\bin\gcc.exe |
|
Hmm... so you can set CC, but it does not seem to be used by What does Also, now that you can set the CC environment variable, what do you get from |
|
now python build-cython.py gives a new error : Traceback (most recent call last): v.0.0.28 also produced the same error. |
|
Change of plans. Rather than trying to meticulously detect the compiler to set its arguments, I will just add the option to override the C compiler arguments. As a workaround for possible issues, users can re-build with python build-cython.py --rebuild /O2See #245 |
|
As explained here, it is not strange (and is in fact expected) that you had to install cython. |
|
now build-cython doesn't run at all. |
|
I added this line, which I think should fix the last error you shared. If I could make one final request: can you try |
|
a new error running update build_cython (qldpc-v28) C:\Users\user\Desktop>python build_cython.py (qldpc-v28) C:\Users\user\Desktop> ...so I can't get to the compiler options |
|
I have confirmed that qLDPC works in Windows. Starting with a Windows machine with no software development history (read: clean slate, no development tools installed), I installed:
Starting from the conda create --name test python=3.12 -y
conda activate test
pip install .I then tried running the python script import qldpc
code = qldpc.codes.RepetitionCode(3)
code._exact_distance = None
code.get_code_params()which gave the expected output, My conclusion is that Windows is supported, and that any issues you are having @qodesign are likely issues with your development environment. I am happy to help you resolve those over time (we can continue talking on this issue page on github), but as far as |
|
Actually, maybe we should add a Windows CI test, as @vtomole I think suggested. Would somebody on the Infleqtion end be able to help with that? |
|
#246 adds a workflow to install |
|
On further reflection, it doesn't hurt to keep this issue open until @qodesign considers their problem solved, so I'll re-open. |
@perlinm are these numbers with the current (cython) implementation? if so here's a python/numpy-only implementation that seems to run in ~0.0016 and 23.0 sec/trial for those two experiments on my system: import numpy as np
import numpy.typing as npt
def rows_to_ints(array, dtype: npt.DTypeLike = np.uint):
array = np.asarray(array, dtype=dtype)
tsize = array.itemsize * 8
num_words = int(np.ceil(array.shape[-1] / tsize))
columns = [
np.bitwise_or.reduce(
[array[..., -1 - i - tsize * j] << i for i in range(array.shape[-1] - tsize * j)]
)
for j in range(num_words)
]
return np.vstack(columns[::-1]).T
def popcnt64(
arr: np.uint | npt.NDArray[np.uint],
buf: np.uint | npt.NDArray[np.uint] | None = None,
out: np.uint | npt.NDArray[np.uint] | None = None,
) -> npt.NDArray[np.uint]:
"""Somewhat efficient (vectorized) Hamming weight calculation. Assumes 64-bit (u)ints.
This is the weak point in the python/numpy implementation (unfortunately numpy<2.0.0 doesn't
expose processors' `popcnt` instruction).
"""
buf = np.right_shift(arr, 1, out=buf)
buf &= 0x5555555555555555
out = np.subtract(arr, buf, out=out)
buf = np.right_shift(out, 2, out=buf)
buf &= 0x3333333333333333
out &= 0x3333333333333333
out += buf
buf = np.right_shift(out, 4, out=buf)
out += buf
out &= 0x0F0F0F0F0F0F0F0F
out *= 0x0101010101010101
out >>= 56
return out
def ctz(num: int):
return int(np.log2(num ^ (num - 1)))
def get_distance_quantum(
logical_ops,
stabilizers,
block_size: int = 14,
dtype: npt.DTypeLike = np.uint,
):
"""Distance of a binary quantum code. Accepts any number of bits (though block_size may need to
be adjusted for multiples of 64 bits)."""
num_bits = np.shape(logical_ops)[-1]
int_logical_ops = rows_to_ints(logical_ops, dtype=dtype)
int_stabilizers = rows_to_ints(stabilizers, dtype=dtype)
assert int_logical_ops.itemsize == 8
block_size = min(block_size, len(int_logical_ops) + len(int_stabilizers))
# Precompute all combinations of first `block_size` operators
blocked = np.zeros((1, int_logical_ops.shape[-1]), dtype=dtype)
for op in int_logical_ops[:block_size]:
blocked = np.vstack([blocked, blocked ^ op])
# TODO: probably don't need to treat these as separate cases
if block_size >= len(int_logical_ops):
# drop the all-zero entry (so we don't consider products of stabilizers with no logical ops)
blocked = blocked[1:]
# fill out block with products of some stabilizers
for op in int_stabilizers[: block_size - len(int_logical_ops)]:
blocked = np.vstack([blocked, blocked ^ op])
int_stabilizers = int_stabilizers[block_size - len(int_logical_ops) :]
buf1, buf2 = np.empty((2, *blocked.shape), dtype=dtype)
weights = popcnt64(blocked, buf=buf1, out=buf2).sum(-1)
min_weight = weights.min(initial=num_bits)
# iterate over all products of remaining stabilizers
for ll in range(1, 2 ** len(int_stabilizers)):
blocked ^= int_stabilizers[ctz(ll)]
weights = popcnt64(blocked, buf=buf1, out=buf2).sum(-1, out=weights)
min_weight = min(min_weight, weights.min())
else:
blocked = np.zeros((1, int_logical_ops.shape[-1]), dtype=dtype)
for op in int_logical_ops[:block_size]:
blocked = np.vstack([blocked, blocked ^ op])
int_logical_ops = int_logical_ops[block_size:]
buf1, buf2 = np.empty((2, *blocked.shape), dtype=dtype)
# initially ignore blocked[0] to avoid products of stabilizers with no logical ops
weights = popcnt64(blocked[1:], buf=buf1[1:], out=buf2[1:]).sum(-1)
min_weight = weights.min(initial=num_bits)
for si in range(1, 2 ** len(int_stabilizers)):
blocked ^= int_stabilizers[ctz(si)]
weights = popcnt64(blocked[1:], buf=buf1[1:], out=buf2[1:]).sum(-1, out=weights)
min_weight = min(min_weight, weights.min())
for ll in range(1, 2 ** len(int_logical_ops)):
blocked ^= int_logical_ops[ctz(ll)]
weights = popcnt64(blocked, buf=buf1, out=buf2).sum(-1)
min_weight = min(min_weight, weights.min())
for si in range(1, 2 ** len(int_stabilizers)):
blocked ^= int_stabilizers[ctz(si)]
weights = popcnt64(blocked, buf=buf1, out=buf2).sum(-1, out=weights)
min_weight = min(min_weight, weights.min())
return min_weight |
|
@richrines1 Yes, those numbers are pretty much what I get on my laptop with the current cython implementation. I don't think I fully understand your implementation, but it certainly seems nice to have a python/numpy-only version! Questions:
Regarding numpy >= 2.0.0, at the moment this is blocked by (I think only) sinter. That should be fixed as soon as stim/sinter have a new release, at which point we can use |
|
This runs on my laptop in ~0.0015 sec/trial and ~24.5 sec/trial respectively on |
basically, though there's another factor of 3 due to the two buffers (we can probably drop these with numpy 2.* though)
every step in the loop is really doing 2**block_size iterations, so the python overhead is basically negligible - the "real" innermost loop is whatever's going on within the numpy functions, which are mostly compiled/well-optimized C code. python mostly just adds overhead to how fast we step between the numpy calls, which doesn't matter once each call is doing enough work
unfortunately i think we have a few other blockers internally, but it seems like it'd be relatively straightforward to have it use np.bitwise_count if available and otherwise fall back on this hacky implementation?
definitely curious to see but tbh i'm not sure how much of a difference i'd expect. this mostly works because every numpy call is doing a lot of work (and generally does so pretty efficiently), so if the only difference is that we step between them faster it might not make a huge difference (though it'd maybe decrease the ideal i think the biggest "easy" improvement would come from replacing the |
|
Hmm, I think that makes sense. I think we can just use the code inside the UPDATE: nvm I missed the UPDATE 2: oh but we could maybe initially do something like blocked = np.zeros((1, int_logical_ops.shape[-1]), dtype=dtype)
for op in int_logical_ops[:block_size]:
blocked = np.vstack([blocked, blocked ^ op])
for op in int_stabilizers[: block_size - len(int_logical_ops)]:
blocked = np.vstack([blocked, blocked ^ op])
stabilizers = stabilizers[min(block_size - len(int_logical_ops), 0) :]
int_logical_ops = int_logical_ops[block_size:]but I need to think more carefully about how to avoid the logical identity operator... In any case, let's move this discussion to #248 |
I tried to upgrade to 0.0.25 and got this error :
Building wheels for collected packages: qldpc
Building wheel for qldpc (pyproject.toml) ... error
error: subprocess-exited-with-error
× Building wheel for qldpc (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [69 lines of output]
Compiling qldpc\codes_distance.pyx because it changed.
[1/1] Cythonizing qldpc\codes_distance.pyx
cl : Command line warning D9002 : ignoring unknown option '-O3'
cl : Command line warning D9002 : ignoring unknown option '-march=native'
_distance.c
C:\Conda\envs\qldpc-env\include\pyconfig.h(59): fatal error C1083: Cannot open include file: 'io.h': No such file or directory
...
subprocess.CalledProcessError: Command '['C:/Conda/envs/qldpc-env/python.exe', 'build-cython.py']' returned non-zero exit status 1.
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for qldpc
Failed to build qldpc
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (qldpc)
I'm running on windows 10. Mingwin is my primary compiler and -O3 is a valid directive for it but maybe cython doesn't pick that up
(I get the same error if VisualC is in the path)
The text was updated successfully, but these errors were encountered: