Skip to content

Safely handle parallel/concurrent invocations #957

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
everythingfunctional opened this issue Jul 18, 2023 · 6 comments · May be fixed by #1116
Open

Safely handle parallel/concurrent invocations #957

everythingfunctional opened this issue Jul 18, 2023 · 6 comments · May be fixed by #1116
Labels
enhancement New feature or request

Comments

@everythingfunctional
Copy link
Member

Description

At present, if you invoke fpm on the same project multiple times in parallel, the compiler will trip over itself as it tries to compile the same files at the same time, and can leave the project in an inconsistent state such that subsequent invocations do not try to rebuild when they should. If possible, fpm should avoid trying to build a project multiple times in parallel.

Possible Solution

A quick and dirty solution would be to create a "lock file" on fpm startup and delete it on program shutdown. If the "lock file" is already present, just wait until it's removed. It's not flawless, but could be sufficient.

Additional Information

No response
@everythingfunctional everythingfunctional added the enhancement New feature or request label Jul 18, 2023
@milancurcic
Copy link
Member

milancurcic commented Jul 18, 2023 •
edited
Loading

I can reproduce. Lock file sounds reasonable to me. I'm only concerned about situations where fpm hits an unhandled exception and leaves the lock file behind. Is there a way to handle that?

@perazz
Copy link
Member

perazz commented Jul 18, 2023 •
edited
Loading

Interesting enhancement @everythingfunctional.

I think the proposed solution looks very reasonable. Could that be done on a per-dependency basis? (think something similar to cache.toml). We now have the option for a local registry, so, that has to be dealt with as well.

I think we also want to understand what causes this:

and can leave the project in an inconsistent state such that subsequent invocations do not try to rebuild

because we could figure out a fix and force a full rebuild if something went corrupt.

Edit: also see rust-lang/cargo#354

@everythingfunctional
Copy link
Member Author

I'm only concerned about situations where fpm hits an unhandled exception and leaves the lock file behind.

Yep, that's the main danger. If you wanted to get fancy the lock file could contain the PID of the process that created, and then check if that process is still executing, if not delete it and create a new one. Basically, there are solutions.

we also want to understand what causes this

The compiler stepping on its own toes, but updating the timestamp of files anyways. I.e. it looks like the object/mod file got produced after the source code was updated, but it didn't get produced correctly.

@everythingfunctional
Copy link
Member Author

FYI, my use case is trying to get ctest to run my tests in parallel, but there's no way to ensure they're actually compiled without invoking fpm test.

@emmabastas
Copy link

emmabastas commented Mar 1, 2025 •
edited
Loading

Hey, I started working on this issue as a way to get familiar with Fortran and fpm because I'm interested in working on fpm as part of GSoC2025.

My work thus far is in this commit emmabastas@301e17b
I've implemented locking of package directories (the lock file way) for fpm run (i.e. I acquire the lock in the cmd_run subroutine).
On my machine it makes it so that concurrent fpm run's wait for one another.

I just wanted to check-in about this issue, is it still relevant, and am I heading in the right direction with my implementation? I'm not asking for a proper code-review or anything like that, I just don't want to spend time on this only to find out that the issue isn't relevant any more, or that I have the complete wrong idea of how to fix it, etc :-)

@everythingfunctional
Copy link
Member Author

I'm no longer working on the project that needed this, but I imagine it will still be useful. It looks like you've made a promising start down the path I suggested.

emmabastas added a commit to emmabastas/fpm that referenced this issue Mar 25, 2025
@emmabastas
Mege concurrent-invocations
f9b119d 
fixes fortran-lang#957

Squashed commit of the following:

commit 998ebba0c55cdd681e8575c1cc8d951e699c1907
Author: Emma Bastås <[email protected]>
Date:   Tue Mar 25 13:17:38 2025 +0100

    Remove nix flake

commit b35852a94fb3cb2444ed973b7f9c28b46964e93d
Author: Emma Bastås <[email protected]>
Date:   Tue Mar 25 13:09:34 2025 +0100

    Use fpm_lock in fpm commands

commit 0f72adccb1c4bdf4c9b49b6336b550fb705fb2f0
Author: Emma Bastås <[email protected]>
Date:   Tue Mar 25 11:40:27 2025 +0100

    Better unitest fixtures

commit 49dbf5e2d4ace90b98dda959f903af41b42f94ed
Author: Emma Bastås <[email protected]>
Date:   Mon Mar 24 18:48:55 2025 +0100

    Fix memory issues

commit 45d8b1454d99277fad78fb26def34773d41d01b0
Author: Emma Bastås <[email protected]>
Date:   Mon Mar 24 16:39:59 2025 +0100

    Fix a typo

commit 0d86b10c76b6594d7363e6b2b697b08c9eccb92d
Author: Emma Bastås <[email protected]>
Date:   Mon Mar 24 16:39:31 2025 +0100

    Add implementation notes

commit b90dad7fdbd0c9c66700732c5af0f5919bb0ea64
Author: Emma Bastås <[email protected]>
Date:   Mon Mar 24 16:16:14 2025 +0100

    Remove lock-files with atexit

commit 4b6fea47f3771e280f89538f5435cb6e2b510b19
Author: Emma Bastås <[email protected]>
Date:   Sun Mar 23 16:20:13 2025 +0100

    Simplify logic

commit 66ff71676e3b8f5f744b146d9f6515e16c1b83fe
Author: Emma Bastås <[email protected]>
Date:   Fri Mar 21 15:49:02 2025 +0100

    All tests pass on Linux and Windows!!

commit 019ffc96ee4c9b0d388271543ff295a77eebd192
Author: Emma Bastås <[email protected]>
Date:   Fri Mar 21 15:42:57 2025 +0100

    Almost everthing works

commit b7e2c1af4589a427e925fb98068cecf598cb2366
Author: Emma Bastås <[email protected]>
Date:   Fri Mar 21 11:54:29 2025 +0100

    Implement for Windows.

commit 588fcf7f5674dfd92b9bfd485222f71ca1e49db0
Author: Emma Bastås <[email protected]>
Date:   Fri Mar 21 11:54:00 2025 +0100

    Kill processes without resorting to C.

commit 688a38d5e7076fb1f71cbd33fe828691058d3443
Author: Emma Bastås <[email protected]>
Date:   Wed Mar 19 20:08:17 2025 +0100

    More documentation and comments

commit f39ef3ef5ca498920212c1470808cef6ee385032
Author: Emma Bastås <[email protected]>
Date:   Wed Mar 19 18:03:10 2025 +0100

    Handle more errors properly

commit b831f652b26166e7f2c124ec9634214f0fb3b527
Author: Emma Bastås <[email protected]>
Date:   Wed Mar 19 17:13:05 2025 +0100

    Add a unit test

commit fd7ffb8f5e8d76bb60545c79ac3c0a14c8dd4abd
Author: Emma Bastås <[email protected]>
Date:   Wed Mar 19 17:00:27 2025 +0100

    Reap child process after kill

commit 6f12e87f7d9a92e10efc51922b3d6ba27646c85a
Author: Emma Bastås <[email protected]>
Date:   Wed Mar 19 17:00:11 2025 +0100

    Change some comments

commit 757300003dacc095724885ccbedb2a5f98006b9e
Author: Emma Bastås <[email protected]>
Date:   Wed Mar 19 16:59:54 2025 +0100

    Implement a process_alive function

commit 26eb598ab124c338e38751a61bf5a8c21a412a21
Author: Emma Bastås <[email protected]>
Date:   Wed Mar 19 15:02:36 2025 +0100

    Fail test early if package-locking fails

commit 107dcec984d3a34b7dc340e82f189b30acdf27f2
Author: Emma Bastås <[email protected]>
Date:   Wed Mar 19 14:56:32 2025 +0100

    Make it better

commit 82e9224c458ad899660ed94866c87161cde8111c
Author: Emma Bastås <[email protected]>
Date:   Sat Mar 15 17:59:40 2025 +0100

    Add tests

commit e1169ca897df690309941d83872447515656234d
Author: Emma Bastås <[email protected]>
Date:   Sat Mar 15 17:58:04 2025 +0100

    WIP: Pid in lock-file + error handling

commit 903ec6fc758d50944a95acdf10a7bc7237ed5834
Author: Emma Bastås <[email protected]>
Date:   Sat Mar 1 18:59:32 2025 +0100

    Start work on package locking

commit 65f0905a4ed6f043eaabdb2d08bfd4f9654e161e
Author: Emma Bastås <[email protected]>
Date:   Sat Mar 1 13:42:46 2025 +0100

    TODO: Remove. nix dev enviroment
@emmabastas emmabastas linked a pull request Mar 25, 2025 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Lock package directories with lock-files (fixes #957) emmabastas/fpm
4 participants
@everythingfunctional @milancurcic @perazz @emmabastas