Upstream v1.14.0 #106

SebMilardo · 2024-05-22T15:03:45Z

Updating the version of vroom included in pyvroom. I'm creating a draft PR to keep track of the work done:

Bump vroom to version v.1.14.0 using commit VROOM-Project/vroom@1fd711b
Fix compile time errors (when running make develop)
Check that all new functionalities are included
Update tests

I'm currently getting the following error when running make test:

ImportError while loading conftest '/pyvroom/conftest.py'.
conftest.py:3: in <module>
    import vroom
src/vroom/__init__.py:5: in <module>
    from ._vroom import _main, JOB_TYPE, STEP_TYPE  # type: ignore
E   ImportError: /pyvroom/src/vroom/_vroom.cpython-312-x86_64-linux-gnu.so: undefined symbol: _ZTVN5vroom4cvrp6TSPFixE

I'll try to find a solution...

jonathf · 2024-05-22T16:03:31Z

Errors like that means you are missing a an include to some resource in vroom upstream.

Use a c++ demangler (many online if you search) to find the name, and grep/search for the name in vroom.

In simple cases a simple include is enough. In more complicated cases we might need to update the bind layer code in some way.

SebMilardo · 2024-05-22T19:44:31Z

Thanks a lot! I fixed the imports. Now I have 5 tests failing and 39 passed. I'll try to solve them by tomorrow

=================================================================================== test session starts ===================================================================================
platform linux -- Python 3.12.3, pytest-8.2.1, pluggy-1.5.0
rootdir: /pyvroom
configfile: pyproject.toml
collected 44 items                                                                                                                                                                        

README.rst F                                                                                                                                                                        [  2%]
test/input/test_vehicle_step.py ...                                                                                                                                                 [  9%]
test/test_amount.py .F.                                                                                                                                                             [ 15%]
test/test_break.py ..                                                                                                                                                               [ 20%]
test/test_file_handle.py ..                                                                                                                                                         [ 25%]
test/test_job.py ..                                                                                                                                                                 [ 29%]
test/test_libvroom_examples.py F                                                                                                                                                    [ 31%]
test/test_location.py ...                                                                                                                                                           [ 38%]
test/test_time_window.py ......                                                                                                                                                     [ 52%]
test/test_vehicle.py F                                                                                                                                                              [ 54%]
src/vroom/amount.py .                                                                                                                                                               [ 56%]
src/vroom/break_.py .                                                                                                                                                               [ 59%]
src/vroom/input/forced_service.py .                                                                                                                                                 [ 61%]
src/vroom/input/input.py .                                                                                                                                                          [ 63%]
src/vroom/input/vehicle_step.py .......                                                                                                                                             [ 79%]
src/vroom/job.py ...                                                                                                                                                                [ 86%]
src/vroom/location.py ...                                                                                                                                                           [ 93%]
src/vroom/time_window.py .                                                                                                                                                          [ 95%]
src/vroom/vehicle.py F.                                                                                                                                                             [100%]

======================================================================================== FAILURES =========================================================================================
__________________________________________________________________________________ [doctest] README.rst ___________________________________________________________________________________
052   ...                           vroom.Job(1515, location=1),
053   ...                           vroom.Job(1616, location=2),
054   ...                           vroom.Job(1717, location=3)])
055 
056   >>> solution = problem_instance.solve(exploration_level=5, nb_threads=4)
057 
058   >>> solution.summary.cost
059   6411
060 
061   >>> solution.routes.columns
UNEXPECTED EXCEPTION: RuntimeError('bad optional access')
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/doctest.py", line 1361, in __run
    exec(compile(example.source, filename, "single",
  File "<doctest README.rst[7]>", line 1, in <module>
  File "/pyvroom/src/vroom/solution/solution.py", line 62, in routes
    array = numpy.asarray(self._routes_numpy())
                          ^^^^^^^^^^^^^^^^^^^^
RuntimeError: bad optional access
/pyvroom/README.rst:61: UnexpectedException
__________________________________________________________________________________ test_amount_operator ___________________________________________________________________________________

    def test_amount_operator():
        amo1 = vroom.Amount([1, 2, 3])
        amo2 = vroom.Amount([2, 2, 3])
    
>       assert amo1 << amo2

test/test_amount.py:31: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = vroom.Amount([1, 2, 3]), other = vroom.Amount([2, 2, 3])

    def __lshift__(self, other: Amount) -> bool:
        other = Amount(other)
        if len(self) != len(other):
            raise _vroom.VroomInternalException("Comparing two Amount of different length")
>       return self._lshift(other)
E       AttributeError: 'Amount' object has no attribute '_lshift'. Did you mean: '__lshift__'?

src/vroom/amount.py:98: AttributeError
_____________________________________________________________________________ test_example_with_custom_matrix _____________________________________________________________________________

    def test_example_with_custom_matrix():
        problem_instance = vroom.Input()
    
        problem_instance.set_durations_matrix(
            profile="car",
            matrix_input=[[0, 2104, 197, 1299],
                          [2103, 0, 2255, 3152],
                          [197, 2256, 0, 1102],
                          [1299, 3153, 1102, 0]],
        )
        problem_instance.add_vehicle([vroom.Vehicle(7, start=0, end=0),
                                      vroom.Vehicle(8, start=2, end=2)])
        problem_instance.add_job([vroom.Job(id=1414, location=0),
                                  vroom.Job(id=1515, location=1),
                                  vroom.Job(id=1616, location=2),
                                  vroom.Job(id=1717, location=3)])
        solution = problem_instance.solve(
            exploration_level=5, nb_threads=4)
    
        assert solution.summary.cost == 6411
        assert solution.summary.unassigned == 0
        assert solution.unassigned == []
    
>       routes = solution.routes

test/test_libvroom_examples.py:31: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <vroom.solution.solution.Solution object at 0x7f18ef75f7d0>

    @property
    def routes(self) -> pandas.DataFrame:
        """
        Frame outlining all routes for all vehicles.
    
        It includes the following columns.
    
        vehicle_id:
            Id of the vehicle assigned to this route.
        type:
            The activity the vehicle is performing. The available
            categories are `start`, `end`, `break`, `job`, `delivery`
            and `pickup`.
        arrival:
            Timepoint when the actvity ends.
        duration:
            The length of the activity.
        setup:
            Total setup time for this route.
        service:
            Total service time for this route.
        waiting_time:
            Total waiting time for this route.
        location_index:
            The index for the location of the destination.
        longitude:
            If available, the longitude of the destination.
        latitude:
            If available, the latitude of the destination.
        id:
            The identifier for the task that was performed.
        description:
            Text description provided to this step.
        """
>       array = numpy.asarray(self._routes_numpy())
E       RuntimeError: bad optional access

src/vroom/solution/solution.py:62: RuntimeError
________________________________________________________________________________________ test_repr ________________________________________________________________________________________

    def test_repr():
    
>       assert (repr(vroom.Vehicle(1, start=4, profile="bus"))
                == "vroom.Vehicle(1, start=4, profile='bus')")
E       assert 'vroom.Vehicl...036854775807)' == "vroom.Vehicl...rofile='bus')"
E         
E         - vroom.Vehicle(1, start=4, profile='bus')
E         + vroom.Vehicle(1, start=4, profile='bus', max_distance=9223372036854775807)

test/test_vehicle.py:6: AssertionError
_____________________________________________________________________________ [doctest] vroom.vehicle.Vehicle _____________________________________________________________________________
089         speed_factor:
090             The speed factor for which this vehicle runs faster or slower than
091             the default.
092         max_tasks:
093             The maximum number of tasks this vehicle can perform.
094         steps:
095             Set of custom steps this vehicle should take.
096 
097     Examples:
098         >>> vroom.Vehicle(1, end=1)
Expected:
    vroom.Vehicle(1, end=1)
Got:
    vroom.Vehicle(1, end=1, max_distance=9223372036854775807)

/pyvroom/src/vroom/vehicle.py:98: DocTestFailure
================================================================================= short test summary info =================================================================================
FAILED README.rst::README.rst
FAILED test/test_amount.py::test_amount_operator - AttributeError: 'Amount' object has no attribute '_lshift'. Did you mean: '__lshift__'?
FAILED test/test_libvroom_examples.py::test_example_with_custom_matrix - RuntimeError: bad optional access
FAILED test/test_vehicle.py::test_repr - assert 'vroom.Vehicl...036854775807)' == "vroom.Vehicl...rofile='bus')"
FAILED src/vroom/vehicle.py::vroom.vehicle.Vehicle
============================================================================== 5 failed, 39 passed in 4.00s ===============================================================================
make: *** [Makefile:14: test] Error 1

SebMilardo · 2024-05-23T10:37:08Z

Tests are now ok. I had to slightly change the result reported in the README file as sol.summary.duration is now 2702 instead of 2704. I might need to add some tests to check the new features included in 1.14.0.

codecov · 2024-05-25T19:04:49Z

Codecov Report

Attention: Patch coverage is 88.05970% with 8 lines in your changes missing coverage. Please review.

Project coverage is 79.1%. Comparing base (da2c48d) to head (4449941).

Additional details and impacted files

@@           Coverage Diff           @@
##            main    #106     +/-   ##
=======================================
- Coverage   80.0%   79.1%   -1.0%     
=======================================
  Files         29      29             
  Lines       1572    1628     +56     
  Branches     114     139     +25     
=======================================
+ Hits        1259    1288     +29     
- Misses       304     330     +26     
- Partials       9      10      +1

Flag	Coverage Δ
binding	`69.1% <87.2%> (-1.7%)`	⬇️
python	`91.4% <90.0%> (+0.2%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

jonathf · 2024-05-29T13:41:41Z

Thanks so much for doing this! This is much appriciated.

I took a quick look, and I want to test a few things before merging. I am also noticing that macos isn't building on release. I am investigating it now.

jonathf · 2024-06-04T13:21:46Z

I appologies that this has been just sitting there.

As you can see macos isn't building correctly, preventing the build to complete. It is missing an external dep which until now was installed through brew. I can't recreate the bug locally, so the solving this is just a matter of tinkering with deps, paths and/or includes in the CI until I get it right.

But until that issue is fixed, merging this will be moot, as we can not release it.

SebMilardo · 2024-06-07T15:31:52Z

I also have a mac, I'll try to find a solution

jonathf · 2024-06-10T06:33:58Z

That is above and beyond! Let me know if you need anything from our side.

SebMilardo · 2024-06-10T17:32:03Z

I found two major issues:

the compiler could not find asio so I had to add it to LDFLAGS and CPPFLAGS
jthread is not supported by the included llvm version so I had to update llvm to version 18 and add the "-fexperimental-library'' flag

jcoupey · 2024-06-11T12:36:59Z

FYI upstream related ticket VROOM-Project/vroom#1062

jonathf · 2024-06-26T15:50:42Z

Main now has fixes that resolves the issue of building on macos. Hopefully now it should be much easier to get to a 1.14 release.

Let me know if you have the time to do the last stretch, or if you want me to take over.

SebMilardo · 2024-07-01T08:59:54Z

I merged the changes It should be ok now

Use LLVM@18

jonathf · 2024-07-03T07:00:05Z

Getting there.

Do you mind moving Homebrew installs into pyproject.toml, flags into setup.py and constant evironment variable under env: directly in the build section?

SebMilardo · 2024-07-03T08:25:42Z

Sure, no problem

SebMilardo · 2024-07-03T12:08:27Z

I am trying to solve the issue on macos-intel. As far as I have understood:

the build fails because the minimum target for some libraries is OSX 13.6

  delocate.libsana.DelocationError: Library dependencies do not satisfy target MacOS version 13:
  /private/var/folders/qm/cb4xw0f50z50qsp98fg8rfqm0000gn/T/tmpd7jh26nx/wheel/vroom/.dylibs/libc++.1.0.dylib has a minimum target of 13.6
  /private/var/folders/qm/cb4xw0f50z50qsp98fg8rfqm0000gn/T/tmpd7jh26nx/wheel/vroom/.dylibs/libc++abi.1.0.dylib has a minimum target of 13.6
  /private/var/folders/qm/cb4xw0f50z50qsp98fg8rfqm0000gn/T/tmpd7jh26nx/wheel/vroom/.dylibs/libunwind.1.0.dylib has a minimum target of 13.6

But if I increase MACOSX_DEPLOYMENT_TARGET to 13.6 it fails during the "Testing wheel..." step because

ERROR: pyvroom-0.1.dev1+g2493307-cp39-cp39-macosx_13_6_x86_64.whl is not a supported wheel on this platform.

as the image is based on macos-13. I also tried adding extra_compile_args.append("-mmacosx-version-min=13.0") but I still get the same "...has a minimum target of 13.6" error.

If I add CIBW_ARCHS_MACOS: x86_64 arm64 to macos-arm to use macos-14 and cross compile for x86_64 I get a different error: ImportError: dlopen(/private/var/folders/sx/mf6nj_sn1ll2crpml02qjgbr0000gn/T/cibw-run-xxe41lvf/cp39-macosx_x86_64/venv-test-x86_64/lib/python3.9/site-packages/vroom/_vroom.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace '_BIO_ctrl'

Any idea? We could:

git revert 02eb40b4e and skip all the issues related to jthreads
set MACOSX_DEPLOYMENT_TARGET to 13.6 and skip the testing
investigate further on why cross-compiling is failing

jonathf · 2024-07-03T12:16:09Z

Lets try the easy solution and update deploy target to 13.6.

jonathf · 2024-07-04T09:34:27Z

I just tried updating th target to 13.6 (and then to 14) and made a release(s), but I get the same error unfortunatly.

I prefer not reverting as it will prevent pyvroom from having feature parity with upstream vroom. So for now I think further investigation is warrented.

Am I to understand that you believe jthreads is the likely culprit here?

* Move flags envs and install * Revert "Update release.yml" This reverts commit 1cd7911. * Update setup.py * Update release.yml * Update release.yml * Update release.yml * Test brew link * Cleaning * Revert "Cleaning" This reverts commit a131e0b. * Update release.yml * Cleaning * Update release.yml * Update release.yml * Fix error on macos-intel * Update release.yml * Cross compile * Update release.yml * Update setup.py * Update * Only build for OSX Arm * Ad universal2 * x86_64 * Update pyproject.toml * Update pyproject.toml * Use [email protected] * Remove [email protected] * Play with openssl * Test * Clean * Try gcc * Update pyproject.toml * Update release.yml * Update release.yml * Update release.yml * Update setup.py * Remove extra instructions * Update pyproject.toml

SebMilardo · 2024-07-07T10:17:03Z

Using gcc-13 instead of clang solves all the issues. Could you run the workflow now?

jonathf · 2024-07-08T11:59:16Z

That it does!

Looking through the changes I see that you have not only updated to 1.14, but also fixed a couple of bugs along the way. Excellent work Sebastiano!

SebMilardo · 2024-07-08T13:05:12Z

The main branch is still failing to build as I forgot to update the main_push.yml file. It should be a quick fix.

SebMilardo added 9 commits May 22, 2024 15:30

Update vroom version to 1.14.0

7260f37

Set cxx_std = 20

828c1d8

Remove lshift operator from amount

20bc7d4

Update Vehicle class

e26ecea

Update solution package classes

fd9ae44

Fix matrix.cpp import

fda87e9

Remove const from Violations init

3093846

Lint and format

d1784ac

Add set_distances_matrix

b8a5301

Add missing includes

347b5c4

SebMilardo added 4 commits May 23, 2024 12:28

Re-add _lshift

e728155

Fix error involving location and has_coordinates

da6ca51

Fix default value for max_distance

c9fbcd3

Update README.rst

1ea9dad

SebMilardo added 2 commits May 23, 2024 14:21

Show distance column in solution only when required

ee59f99

Add tests for max_distance and max_travel_time

6b1012b

SebMilardo marked this pull request as ready for review May 23, 2024 12:24

Update checkout, setup-python, and codecov actions

eaea1bf

Update LLVM@18 to support C++20

e7372f2

Update README.rst

6578859

Merge branch 'main' into upstream-v1.14.0

818a6dd

SebMilardo added 4 commits July 3, 2024 01:05

Update release.yml

cccfb15

Update release.yml

cf79781

Update release.yml

1cd7911

Merge pull request #4 from SebMilardo/mac-os-test

a9c09b6

Use LLVM@18

Move flags envs and install

16822ed

Update release.yml

4449941

jonathf merged commit 825f15e into VROOM-Project:main Jul 8, 2024
9 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upstream v1.14.0 #106

Upstream v1.14.0 #106

SebMilardo commented May 22, 2024 •

edited

Loading

jonathf commented May 22, 2024

SebMilardo commented May 22, 2024

SebMilardo commented May 23, 2024

codecov bot commented May 25, 2024 •

edited

Loading

jonathf commented May 29, 2024

jonathf commented Jun 4, 2024

SebMilardo commented Jun 7, 2024

jonathf commented Jun 10, 2024

SebMilardo commented Jun 10, 2024

jcoupey commented Jun 11, 2024

jonathf commented Jun 26, 2024

SebMilardo commented Jul 1, 2024

jonathf commented Jul 3, 2024

SebMilardo commented Jul 3, 2024

SebMilardo commented Jul 3, 2024

jonathf commented Jul 3, 2024 •

edited

Loading

jonathf commented Jul 4, 2024

SebMilardo commented Jul 7, 2024 •

edited

Loading

jonathf commented Jul 8, 2024

SebMilardo commented Jul 8, 2024

Upstream v1.14.0 #106

Upstream v1.14.0 #106

Conversation

SebMilardo commented May 22, 2024 • edited Loading

jonathf commented May 22, 2024

SebMilardo commented May 22, 2024

SebMilardo commented May 23, 2024

codecov bot commented May 25, 2024 • edited Loading

Codecov Report

jonathf commented May 29, 2024

jonathf commented Jun 4, 2024

SebMilardo commented Jun 7, 2024

jonathf commented Jun 10, 2024

SebMilardo commented Jun 10, 2024

jcoupey commented Jun 11, 2024

jonathf commented Jun 26, 2024

SebMilardo commented Jul 1, 2024

jonathf commented Jul 3, 2024

SebMilardo commented Jul 3, 2024

SebMilardo commented Jul 3, 2024

jonathf commented Jul 3, 2024 • edited Loading

jonathf commented Jul 4, 2024

SebMilardo commented Jul 7, 2024 • edited Loading

jonathf commented Jul 8, 2024

SebMilardo commented Jul 8, 2024

SebMilardo commented May 22, 2024 •

edited

Loading

codecov bot commented May 25, 2024 •

edited

Loading

jonathf commented Jul 3, 2024 •

edited

Loading

SebMilardo commented Jul 7, 2024 •

edited

Loading