Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sync scheduler overhaul #15

Merged
merged 130 commits into from
Mar 31, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
130 commits
Select commit Hold shift + click to select a range
2ab2c11
i may have forgotten to commit along the way
carykees98 Nov 20, 2024
28f08ad
updated doxygen comments in utils.hpp and updated `read_JSON_from_file`
carykees98 Nov 20, 2024
df365f8
updated cmakelists
carykees98 Feb 7, 2025
c50837f
source changes (I need committing to be more alluring else I'll conti…
carykees98 Feb 7, 2025
6ece08c
remove mirror logging
carykees98 Feb 7, 2025
d9dd75e
removed mirror-logging dependency
carykees98 Feb 13, 2025
0eea85a
add ability to load logging level from environment variable
carykees98 Feb 13, 2025
39ea32d
successfully built sync schedule
carykees98 Feb 13, 2025
2d7005c
renamed load_config
carykees98 Feb 13, 2025
0092eea
added error printing to load_mirrors_config when file is not good
carykees98 Feb 13, 2025
c34bc92
include json_fwd
carykees98 Feb 13, 2025
2ab2c7c
changed exception names to abbreviations of the exception type
carykees98 Feb 13, 2025
7fa7f2a
renamed Project to SyncDetails
carykees98 Feb 16, 2025
71e94f5
added project catalogue using statement
carykees98 Feb 16, 2025
0fe4677
removed thread pool files
carykees98 Feb 16, 2025
cfdcc6d
moved project list to SyncScheduler from Schedule
carykees98 Feb 16, 2025
365132b
renamed Project to SyncDetails
carykees98 Feb 16, 2025
3621bd7
initialize project catalogue in initizlizer list
carykees98 Feb 16, 2025
c006931
declaration order matters, not just initialization list order
carykees98 Feb 16, 2025
7d9e455
sync-scheduler, website: Implement API for manual syncing
brendanjconnelly Feb 16, 2025
15da7b6
renamed Project to SyncDetails
carykees98 Feb 16, 2025
d274564
added project catalogue using statement
carykees98 Feb 16, 2025
f0da05f
removed thread pool files
carykees98 Feb 16, 2025
158bb97
moved project list to SyncScheduler from Schedule
carykees98 Feb 16, 2025
121c248
renamed Project to SyncDetails
carykees98 Feb 16, 2025
8106062
initialize project catalogue in initizlizer list
carykees98 Feb 16, 2025
2989966
sync-scheduler, website: Implement API for manual syncing
brendanjconnelly Feb 16, 2025
1c97b91
Fix cmake
brendanjconnelly Feb 16, 2025
6db7551
renamed Project to SyncDetails
carykees98 Feb 16, 2025
e68ca29
declaration order matters, not just initialization list order
carykees98 Feb 16, 2025
85976fa
sync-scheduler, website: Implement API for manual syncing
brendanjconnelly Feb 16, 2025
8fce6c0
renamed Project to SyncDetails
carykees98 Feb 16, 2025
ac32359
Fix cmake again
brendanjconnelly Feb 16, 2025
c3d7cad
Merge branch 'sync-scheduler-overhaul' into manual-sync-endpoint
brendanjconnelly Feb 16, 2025
63875c8
frequent commits? never heard of them unfortunately
carykees98 Feb 18, 2025
406f1a4
added JobManager to CMakeLists
carykees98 Feb 18, 2025
0d9f656
remove storage directory
carykees98 Feb 18, 2025
f6880f0
remove src-old
carykees98 Feb 18, 2025
1665a3e
remove TODO.md
carykees98 Feb 18, 2025
990a22b
renamed sync scheduler config
carykees98 Feb 18, 2025
e05408a
Merge branch 'sync-scheduler-overhaul' into manual-sync-endpoint
carykees98 Feb 18, 2025
9073c2a
Merge pull request #38 from COSI-Lab/manual-sync-endpoint
carykees98 Feb 18, 2025
78d8455
integrate manual sync
carykees98 Feb 18, 2025
73fe166
Merge branch 'main' into sync-scheduler-overhaul
carykees98 Feb 18, 2025
9919994
remove scripts directory
carykees98 Feb 18, 2025
837a669
renamed is_job_running to job_is_running
carykees98 Feb 19, 2025
8cb1ae3
check if job is running before starting new sync
carykees98 Feb 19, 2025
cb686f7
added clang-tidy for sync scheduler to disable check causing issues w…
carykees98 Feb 19, 2025
836a961
add zmq as a cmake dependency
carykees98 Feb 19, 2025
fbedfba
resolved clang tidy variable naming issues
carykees98 Feb 19, 2025
a549200
updated clang-tidy CMake config
carykees98 Feb 19, 2025
ce8f980
turned off clang tidy
carykees98 Feb 27, 2025
da43e00
updated dockerfile
carykees98 Feb 27, 2025
c46098d
updated zmq responses for manual syncs in sync scheduler
carykees98 Feb 27, 2025
0a7eebb
added pyzmq to pip packages for mirror website
carykees98 Feb 27, 2025
88b95b3
updated manual sync endpoint
carykees98 Feb 27, 2025
cc64578
don't look for clang-tidy when building sync scheduler
carykees98 Feb 27, 2025
518e080
use send_string instead of send to send project name
carykees98 Feb 27, 2025
3b0cb9c
fixed log statement being in the wrong part of an if
carykees98 Feb 27, 2025
c0b8f9f
trace dump syncConfig when syncing a project
carykees98 Feb 27, 2025
29da2e7
I forgot how std::format works
carykees98 Feb 27, 2025
07d5fbd
changed name of secondary and tertiary rsync jobs
carykees98 Feb 27, 2025
1f345a6
add stdout pipe and remove memory leak caused by use of strdup withou…
carykees98 Feb 28, 2025
8c29f1b
update calls to free to match convention
carykees98 Feb 28, 2025
2dd8617
print pid when reaping processes
carykees98 Feb 28, 2025
6094926
updated "Catalogued project" log message
carykees98 Feb 28, 2025
4f3c0c8
added predicate to condition variable wait_for
carykees98 Feb 28, 2025
f8d3aeb
moved wait to the top of the process reaper lambda and check to see i…
carykees98 Feb 28, 2025
f4b33eb
updated wait logic
carykees98 Feb 28, 2025
a2f8fad
reap zombie processes
carykees98 Feb 28, 2025
7fa3910
added trace log for no jobs being available to deregister
carykees98 Feb 28, 2025
038935c
forgot to display the exit code for reaped zombie processes
carykees98 Feb 28, 2025
d6452ed
updated log messages related to killing child processes
carykees98 Feb 28, 2025
64fcac2
rewrote reaping logic to allow reaping of known and unknown children …
carykees98 Feb 28, 2025
cf17a6c
forgot to make pid_t non reference in definition of kill_job
carykees98 Feb 28, 2025
8592d77
update timeout to 6 hours
carykees98 Feb 28, 2025
587bc74
continue if wait doesn't return a pid
carykees98 Feb 28, 2025
d76624d
log pid when going through process reaping
carykees98 Feb 28, 2025
fbd39cf
check to make sure that the children file opened properly
carykees98 Feb 28, 2025
d971f25
use string() instead of generic_u8string()
carykees98 Feb 28, 2025
e65d9f7
create path from string not by building it with operators
carykees98 Feb 28, 2025
08e7345
didn't have placeholder for children file path
carykees98 Feb 28, 2025
d8146c6
task not tasks
carykees98 Feb 28, 2025
573cc64
gather all child processid from all items in the task directory
carykees98 Feb 28, 2025
e9deb31
make directory item a reference
carykees98 Feb 28, 2025
10a993f
made `kill_job()` static
carykees98 Feb 28, 2025
6079b63
moved collection of child process ids to a separate function
carykees98 Mar 1, 2025
b1c639f
only add job to completedJobs if it is a known job
carykees98 Mar 1, 2025
6d5c2b8
don't cache the sync endpoint
carykees98 Mar 2, 2025
4be69de
try sigterm before sigkill and use pgid instead of pid
carykees98 Mar 2, 2025
0d7cd2c
updated job manager log statements
carykees98 Mar 2, 2025
c5b76e2
removed extra qualifier on get_child_process_ids
carykees98 Mar 2, 2025
a4676ea
remove static from cpp file
carykees98 Mar 2, 2025
e7f6f0c
updated sync scheduler readme
carykees98 Mar 3, 2025
ed1632e
renamed readme to lowercase
carykees98 Mar 3, 2025
40e34d0
made errorMessage variables non-static and restructured reap_processes
carykees98 Mar 3, 2025
3b8ed40
updated syntax for calling private methods
carykees98 Mar 3, 2025
684a3a6
updated log at the end of schedule verification
carykees98 Mar 3, 2025
35a71b3
deleted extra constructors and assignment operators for JobManager an…
carykees98 Mar 3, 2025
88540d0
if exec fails, exit child process
carykees98 Mar 3, 2025
ea8ccea
setpgid with 0 0
carykees98 Mar 3, 2025
91e62bf
make sure that the rsync password environment var lives to when exec …
carykees98 Mar 3, 2025
3ca20cd
stop redirect of stderr and stdout in child process
carykees98 Mar 3, 2025
2274e43
assert that argv contains no nullptrs
carykees98 Mar 3, 2025
298bd40
copy the contents of the command into a new char array
carykees98 Mar 3, 2025
dfa7e57
end argv with a nullptr
carykees98 Mar 3, 2025
d3c4b16
remove assertions, overwrite stdout and stderr in child, and use sete…
carykees98 Mar 3, 2025
eb90f16
updated nolint comments
carykees98 Mar 3, 2025
6159a43
updated how sync duration is set when checking timeout
carykees98 Mar 3, 2025
8adce22
remove start-sync-scheduler.sh
carykees98 Mar 3, 2025
f709966
completedJobs is no longer static
carykees98 Mar 3, 2025
84564aa
static cast false in setenv
carykees98 Mar 3, 2025
11da336
update creation of argv for child process
carykees98 Mar 3, 2025
1ae8e4d
get_child_process_ids now takes an argument of the pid to gather info…
carykees98 Mar 16, 2025
9a9ca3b
formatting update
carykees98 Mar 16, 2025
d749315
converted process reaper to a member function
carykees98 Mar 16, 2025
d499ab7
updated argv comment block
carykees98 Mar 16, 2025
4f8f3dc
update interrupt job to recursively send signals to children rather t…
carykees98 Mar 16, 2025
4838aae
convert manual sync loop to function
carykees98 Mar 16, 2025
199d0a1
appease clang-tidy
carykees98 Mar 16, 2025
c43de97
load manual sync port from environment
carykees98 Mar 25, 2025
817a01b
updated gitignore
carykees98 Mar 25, 2025
f758bc3
change how sync config is processed to remove unnecessary use of json
carykees98 Mar 25, 2025
e78dc5a
update job name generation
carykees98 Mar 25, 2025
bacac0f
remove unnecessary comma
carykees98 Mar 25, 2025
13dd181
remove extra format spot
carykees98 Mar 25, 2025
bc75d67
updated variable names for consistency
carykees98 Mar 27, 2025
5d38e24
name change in last commt still didn't make sense
carykees98 Mar 27, 2025
8e191f5
convert config from json to env file
carykees98 Mar 31, 2025
77253fe
add env files to compose file
carykees98 Mar 31, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions configs/sync-scheduler.env.example
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
DRY_RUN=true
MANUAL_SYNC_PORT=9281
6 changes: 0 additions & 6 deletions configs/sync-scheduler.json.example

This file was deleted.

53 changes: 28 additions & 25 deletions docker-compose-dev.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -39,11 +39,14 @@ services:
depends_on:
- website


# --- Sync scheduler ---
sync-scheduler:
build: ./mirror-sync-scheduler
container_name: mirror-sync-scheduler
env_file:
- configs/sync-scheduler.env
expose:
- 9281
volumes:
- "./mirror-sync-scheduler/storage:/storage"
- "./mirror-sync-scheduler/configs:/mirror/configs:ro"
Expand All @@ -56,32 +59,32 @@ services:
tty: true

# --- Torrent handler ---
# torrent-handler:
# build: ./mirror-torrent-handler
# container_name: mirror-torrent-handler
# volumes:
# - "./mirror-torrent-handler/storage:/storage"
# - "./mirror-torrent-handler/configs:/mirror/configs:ro"
# networks:
# - mirror
# depends_on:
# - log-server
# torrent-handler:
# build: ./mirror-torrent-handler
# container_name: mirror-torrent-handler
# volumes:
# - "./mirror-torrent-handler/storage:/storage"
# - "./mirror-torrent-handler/configs:/mirror/configs:ro"
# networks:
# - mirror
# depends_on:
# - log-server

# --- Map ---
# map:
# build: ./mirror-map
# container_name: mirror-map
# restart: unless-stopped
# ports:
# - "30302:8080"
# networks:
# - mirror
# volumes:
# - "./mirror-map/configs:/mirror/configs:ro"
# - "./mirror-map/GeoLite2City:/mirror/GeoLite2City"
# depends_on:
# - log-server
# - metrics
# map:
# build: ./mirror-map
# container_name: mirror-map
# restart: unless-stopped
# ports:
# - "30302:8080"
# networks:
# - mirror
# volumes:
# - "./mirror-map/configs:/mirror/configs:ro"
# - "./mirror-map/GeoLite2City:/mirror/GeoLite2City"
# depends_on:
# - log-server
# - metrics

# --- Metrics engine ---
metrics:
Expand Down
10 changes: 6 additions & 4 deletions docker-compose-prod.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -57,12 +57,16 @@ services:
depends_on:
- log-server
- metrics

# --- Sync scheduler ---
sync-scheduler:
build: ./mirror-sync-scheduler
container_name: mirror-sync-scheduler
expose:
- 9281
restart: unless-stopped
env_file:
- configs/sync-scheduler.env
volumes:
- "/storage:/storage"
- "./configs:/mirror/configs:ro"
Expand Down Expand Up @@ -117,7 +121,7 @@ services:
- mirror
depends_on:
- metrics

# --- Grafana ---
grafana:
image: grafana/grafana:latest
Expand All @@ -132,12 +136,10 @@ services:
depends_on:
- prometheus


networks:
mirror:
external: true


volumes:
prometheus: {}
grafana: {}
Expand Down
3 changes: 3 additions & 0 deletions mirror-sync-scheduler/.clang-tidy
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
InheritParentConfig: true
Checks:
- -clang-analyzer-optin.cplusplus.UninitializedObject # spdlog causing issues
3 changes: 2 additions & 1 deletion mirror-sync-scheduler/.gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
.vscode
build
*.secret
*.secret
.idea
27 changes: 10 additions & 17 deletions mirror-sync-scheduler/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -1,23 +1,16 @@
cmake_minimum_required(VERSION 3.22)
project(syncScheduler)
cmake_minimum_required(VERSION 3.20...3.30)

#find_program(CLANG_TIDY_EXE NAMES "clang-tidy-20" "clang-tidy" REQUIRED)
#set(CLANG_TIDY_COMMAND ${CLANG_TIDY_EXE} --header-filter=${CMAKE_SOURCE_DIR}/include)

add_executable(${PROJECT_NAME}
src/main.cpp
src/schedule.cpp
src/queue.cpp
)

include(FetchContent)
set(CPPZMQ_BUILD_TESTS OFF CACHE INTERNAL "")

FetchContent_Declare(json
GIT_REPOSITORY https://github.com/nlohmann/json
GIT_TAG v3.11.3
project(syncScheduler
LANGUAGES CXX
)
FetchContent_MakeAvailable(json)

add_subdirectory(lib/mirror-logging)
include(cmake/fetch_deps.cmake)

add_subdirectory(src)

target_link_libraries(${PROJECT_NAME}
PUBLIC mirror-logging
PRIVATE nlohmann_json::nlohmann_json
)
21 changes: 10 additions & 11 deletions mirror-sync-scheduler/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -5,18 +5,17 @@
FROM ubuntu:24.04 as builder
RUN apt update && apt install -y g++ cmake libzmq3-dev ninja-build git
WORKDIR /mirror
COPY ./src /mirror/src
COPY ./lib /mirror/lib
COPY ./CMakeLists.txt /mirror/CMakeLists.txt
RUN cmake -S/mirror -B/mirror/build -G Ninja
RUN cmake --build /mirror/build --target clean
RUN cmake --build /mirror/build --target all
COPY src src
COPY include include
COPY cmake cmake
COPY CMakeLists.txt CMakeLists.txt
RUN cmake -B build -G Ninja
RUN cmake --build build --target all

# Run
FROM ubuntu:24.04
RUN apt update && apt install -y python3 curl wget
RUN apt install -y libzmq3-dev rsync
RUN apt update && apt install -y python3 curl wget libzmq3-dev rsync
WORKDIR /mirror
COPY --from=builder /mirror/build/syncScheduler .
RUN chmod 744 syncScheduler
ENTRYPOINT ["./syncScheduler"]
COPY --from=builder /mirror/build/src/sync_scheduler .
RUN chmod 744 sync_scheduler
ENTRYPOINT ["./sync_scheduler"]
17 changes: 0 additions & 17 deletions mirror-sync-scheduler/README.md

This file was deleted.

19 changes: 19 additions & 0 deletions mirror-sync-scheduler/cmake/fetch_deps.cmake
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
include(FetchContent)

FetchContent_Declare(json
GIT_REPOSITORY https://github.com/nlohmann/json
GIT_TAG v3.11.3
)
FetchContent_MakeAvailable(json)

FetchContent_Declare(spdlog
GIT_REPOSITORY https://github.com/gabime/spdlog
GIT_TAG v1.15.1
)
FetchContent_MakeAvailable(spdlog)

FetchContent_Declare(cppzmq
GIT_REPOSITORY https://github.com/zeromq/cppzmq
GIT_TAG v4.10.0
)
FetchContent_MakeAvailable(cppzmq)
Loading