-
Notifications
You must be signed in to change notification settings - Fork 290
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Worker Domain Refactor #5338
Open
hugoghx
wants to merge
10
commits into
main
Choose a base branch
from
oss/llb-worker-domain-refactor
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Worker Domain Refactor #5338
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
core/db
core/session
core
core/proto
core/sql
core/daemon
core/server
labels
Dec 6, 2024
This comment has been minimized.
This comment has been minimized.
hugoghx
force-pushed
the
oss/llb-worker-domain-refactor
branch
from
December 6, 2024 15:34
2de156f
to
9ada784
Compare
This comment has been minimized.
This comment has been minimized.
This commit moves all the logic to select a worker to handle a session to the internal/server package. Most functions were moved to this package with no changes, however, there are a couple of notable exceptions: - Given that we don't use GRPC errors in repos, all errors were adjusted to use the internal/errors package instead. - StorageBucketFilterCredIdFn is now used to prevent the internal/server package from importing internal/storage/plugin, which would cause a dep cycle. Callers of SelectSessionWorkers must implement this func which is responsible for returning a storage bucket's worker filter as well as its associated credential id. - authorizeSessionWithWorkerFilter now receives an internal/server.(*Repository) object when called. This is transparent to SelectSessionWorkers callers as it is handled internally.
hugoghx
force-pushed
the
oss/llb-worker-domain-refactor
branch
from
December 6, 2024 15:38
9ada784
to
2ccccb6
Compare
Database schema diff between To understand how these diffs are generated and some limitations see the FunctionsUnchanged Tablesdiff --git a/.schema-diff/tables_5896a3b44b650875b9bad08d5b0c7d8d8fe2ba4c/public server_worker_tag.sql b/.schema-diff/tables_c73574f3e9f1f1f18a3e09675f58b14a2d276ada/public server_worker_api_tag.sql
similarity index 100%
rename from .schema-diff/tables_5896a3b44b650875b9bad08d5b0c7d8d8fe2ba4c/public server_worker_tag.sql
rename to .schema-diff/tables_c73574f3e9f1f1f18a3e09675f58b14a2d276ada/public server_worker_api_tag.sql
diff --git a/.schema-diff/tables_5896a3b44b650875b9bad08d5b0c7d8d8fe2ba4c/public server_worker_tag_enm.sql b/.schema-diff/tables_c73574f3e9f1f1f18a3e09675f58b14a2d276ada/public server_worker_config_tag.sql
similarity index 100%
rename from .schema-diff/tables_5896a3b44b650875b9bad08d5b0c7d8d8fe2ba4c/public server_worker_tag_enm.sql
rename to .schema-diff/tables_c73574f3e9f1f1f18a3e09675f58b14a2d276ada/public server_worker_config_tag.sql
diff --git a/.schema-diff/tables_5896a3b44b650875b9bad08d5b0c7d8d8fe2ba4c/server_worker_tag.sql b/.schema-diff/tables_c73574f3e9f1f1f18a3e09675f58b14a2d276ada/server_worker_api_tag.sql
similarity index 64%
rename from .schema-diff/tables_5896a3b44b650875b9bad08d5b0c7d8d8fe2ba4c/server_worker_tag.sql
rename to .schema-diff/tables_c73574f3e9f1f1f18a3e09675f58b14a2d276ada/server_worker_api_tag.sql
index eaaf39ea2..600ea2f15 100644
--- a/.schema-diff/tables_5896a3b44b650875b9bad08d5b0c7d8d8fe2ba4c/server_worker_tag.sql
+++ b/.schema-diff/tables_c73574f3e9f1f1f18a3e09675f58b14a2d276ada/server_worker_api_tag.sql
@@ -21,17 +21,23 @@ set default_tablespace = '';
set default_table_access_method = heap;
--
--- name: server_worker_tag; type: table; schema: public; owner: -
+-- name: server_worker_api_tag; type: table; schema: public; owner: -
--
-create table public.server_worker_tag (
+create table public.server_worker_api_tag (
worker_id public.wt_public_id not null,
key public.wt_tagpair not null,
- value public.wt_tagpair not null,
- source text not null
+ value public.wt_tagpair not null
);
+--
+-- name: table server_worker_api_tag; type: comment; schema: public; owner: -
+--
+
+comment on table public.server_worker_api_tag is 'server_worker_api_tag is a table where each row represents a worker api tag.';
+
+
--
-- postgresql database dump complete
--
diff --git a/.schema-diff/tables_5896a3b44b650875b9bad08d5b0c7d8d8fe2ba4c/server_worker_tag_enm.sql b/.schema-diff/tables_c73574f3e9f1f1f18a3e09675f58b14a2d276ada/server_worker_config_tag.sql
similarity index 56%
rename from .schema-diff/tables_5896a3b44b650875b9bad08d5b0c7d8d8fe2ba4c/server_worker_tag_enm.sql
rename to .schema-diff/tables_c73574f3e9f1f1f18a3e09675f58b14a2d276ada/server_worker_config_tag.sql
index 7a28a8530..6395667ec 100644
--- a/.schema-diff/tables_5896a3b44b650875b9bad08d5b0c7d8d8fe2ba4c/server_worker_tag_enm.sql
+++ b/.schema-diff/tables_c73574f3e9f1f1f18a3e09675f58b14a2d276ada/server_worker_config_tag.sql
@@ -21,15 +21,23 @@ set default_tablespace = '';
set default_table_access_method = heap;
--
--- name: server_worker_tag_enm; type: table; schema: public; owner: -
+-- name: server_worker_config_tag; type: table; schema: public; owner: -
--
-create table public.server_worker_tag_enm (
- source text not null,
- constraint only_predefined_server_worker_tag_sources_allowed check ((source = any (array['configuration'::text, 'api'::text])))
+create table public.server_worker_config_tag (
+ worker_id public.wt_public_id not null,
+ key public.wt_tagpair not null,
+ value public.wt_tagpair not null
);
+--
+-- name: table server_worker_config_tag; type: comment; schema: public; owner: -
+--
+
+comment on table public.server_worker_config_tag is 'server_worker_config_tag is a table where each row represents a worker config tag.';
+
+
--
-- postgresql database dump complete
-- Viewsdiff --git a/.schema-diff/views_5896a3b44b650875b9bad08d5b0c7d8d8fe2ba4c/server_worker_aggregate.sql b/.schema-diff/views_5896a3b44b650875b9bad08d5b0c7d8d8fe2ba4c/server_worker_aggregate.sql
deleted file mode 100644
index 789833e78..000000000
--- a/.schema-diff/views_5896a3b44b650875b9bad08d5b0c7d8d8fe2ba4c/server_worker_aggregate.sql
+++ /dev/null
@@ -1,69 +0,0 @@
---
--- postgresql database dump
---
-
--- dumped from database version 13.18
--- dumped by pg_dump version 16.6 (ubuntu 16.6-1.pgdg24.04+1)
-
-set statement_timeout = 0;
-set lock_timeout = 0;
-set idle_in_transaction_session_timeout = 0;
-set client_encoding = 'utf8';
-set standard_conforming_strings = on;
-select pg_catalog.set_config('search_path', '', false);
-set check_function_bodies = false;
-set xmloption = content;
-set client_min_messages = warning;
-set row_security = off;
-
---
--- name: server_worker_aggregate; type: view; schema: public; owner: -
---
-
-create view public.server_worker_aggregate as
- with worker_config_tags(worker_id, source, tags) as (
- select ct_1.worker_id,
- ct_1.source,
- string_agg(distinct concat_ws('y'::text, ct_1.key, ct_1.value), 'z'::text) as tags
- from public.server_worker_tag ct_1
- group by ct_1.worker_id, ct_1.source
- ), connection_count(worker_id, count) as (
- select session_connection.worker_id,
- count(1) as count
- from public.session_connection
- where (session_connection.closed_reason is null)
- group by session_connection.worker_id
- )
- select w.public_id,
- w.scope_id,
- w.description,
- w.name,
- w.address,
- w.create_time,
- w.update_time,
- w.version,
- w.last_status_time,
- w.type,
- w.release_version,
- w.operational_state,
- w.local_storage_state,
- cc.count as active_connection_count,
- wt.tags as api_tags,
- ct.tags as worker_config_tags
- from (((public.server_worker w
- left join worker_config_tags wt on ((((w.public_id)::text = (wt.worker_id)::text) and (wt.source = 'api'::text))))
- left join worker_config_tags ct on ((((w.public_id)::text = (ct.worker_id)::text) and (ct.source = 'configuration'::text))))
- left join connection_count cc on (((w.public_id)::text = (cc.worker_id)::text)));
-
-
---
--- name: view server_worker_aggregate; type: comment; schema: public; owner: -
---
-
-comment on view public.server_worker_aggregate is 'server_worker_aggregate contains the worker resource with its worker provided config values and its configuration and api provided tags.';
-
-
---
--- postgresql database dump complete
---
- TriggersUnchanged IndexesUnchanged Constraintsdiff --git a/.schema-diff/constraints_c73574f3e9f1f1f18a3e09675f58b14a2d276ada/server_worker_api_tag_pkey.sql b/.schema-diff/constraints_c73574f3e9f1f1f18a3e09675f58b14a2d276ada/server_worker_api_tag_pkey.sql
new file mode 100644
index 000000000..079266134
--- /dev/null
+++ b/.schema-diff/constraints_c73574f3e9f1f1f18a3e09675f58b14a2d276ada/server_worker_api_tag_pkey.sql
@@ -0,0 +1,2 @@
+-- name: server_worker_api_tag server_worker_api_tag_pkey; type: constraint; schema: public; owner: -
+ add constraint server_worker_api_tag_pkey primary key (worker_id, key, value);
diff --git a/.schema-diff/constraints_c73574f3e9f1f1f18a3e09675f58b14a2d276ada/server_worker_config_tag_pkey.sql b/.schema-diff/constraints_c73574f3e9f1f1f18a3e09675f58b14a2d276ada/server_worker_config_tag_pkey.sql
new file mode 100644
index 000000000..50e2bb0bf
--- /dev/null
+++ b/.schema-diff/constraints_c73574f3e9f1f1f18a3e09675f58b14a2d276ada/server_worker_config_tag_pkey.sql
@@ -0,0 +1,2 @@
+-- name: server_worker_config_tag server_worker_config_tag_pkey; type: constraint; schema: public; owner: -
+ add constraint server_worker_config_tag_pkey primary key (worker_id, key, value);
diff --git a/.schema-diff/constraints_5896a3b44b650875b9bad08d5b0c7d8d8fe2ba4c/server_worker_tag_enm_pkey.sql b/.schema-diff/constraints_5896a3b44b650875b9bad08d5b0c7d8d8fe2ba4c/server_worker_tag_enm_pkey.sql
deleted file mode 100644
index 5663d8b58..000000000
--- a/.schema-diff/constraints_5896a3b44b650875b9bad08d5b0c7d8d8fe2ba4c/server_worker_tag_enm_pkey.sql
+++ /dev/null
@@ -1,2 +0,0 @@
--- name: server_worker_tag_enm server_worker_tag_enm_pkey; type: constraint; schema: public; owner: -
- add constraint server_worker_tag_enm_pkey primary key (source);
diff --git a/.schema-diff/constraints_5896a3b44b650875b9bad08d5b0c7d8d8fe2ba4c/server_worker_tag_pkey.sql b/.schema-diff/constraints_5896a3b44b650875b9bad08d5b0c7d8d8fe2ba4c/server_worker_tag_pkey.sql
deleted file mode 100644
index d8ece81fe..000000000
--- a/.schema-diff/constraints_5896a3b44b650875b9bad08d5b0c7d8d8fe2ba4c/server_worker_tag_pkey.sql
+++ /dev/null
@@ -1,2 +0,0 @@
--- name: server_worker_tag server_worker_tag_pkey; type: constraint; schema: public; owner: -
- add constraint server_worker_tag_pkey primary key (worker_id, key, value, source); Foreign Key Constraintsdiff --git a/.schema-diff/fk_constraints_5896a3b44b650875b9bad08d5b0c7d8d8fe2ba4c/server_worker_fkey.sql b/.schema-diff/fk_constraints_c73574f3e9f1f1f18a3e09675f58b14a2d276ada/server_worker_fkey.sql
index 7723af6cf..6fe8e49c9 100644
--- a/.schema-diff/fk_constraints_5896a3b44b650875b9bad08d5b0c7d8d8fe2ba4c/server_worker_fkey.sql
+++ b/.schema-diff/fk_constraints_c73574f3e9f1f1f18a3e09675f58b14a2d276ada/server_worker_fkey.sql
@@ -1,5 +1,3 @@
--- name: server_worker_tag server_worker_fkey; type: fk constraint; schema: public; owner: -
- add constraint server_worker_fkey foreign key (worker_id) references public.server_worker(public_id) on update cascade on delete cascade;
-- name: session_connection server_worker_fkey; type: fk constraint; schema: public; owner: -
add constraint server_worker_fkey foreign key (worker_id) references public.server_worker(public_id) on update cascade on delete set null;
-- name: worker_auth_authorized server_worker_fkey; type: fk constraint; schema: public; owner: -
@@ -10,3 +8,7 @@
add constraint server_worker_fkey foreign key (worker_id) references public.server_worker(public_id) on update cascade on delete cascade;
-- name: worker_storage_bucket_credential_state server_worker_fkey; type: fk constraint; schema: public; owner: -
add constraint server_worker_fkey foreign key (worker_id) references public.server_worker(public_id) on update cascade on delete cascade;
+-- name: server_worker_config_tag server_worker_fkey; type: fk constraint; schema: public; owner: -
+ add constraint server_worker_fkey foreign key (worker_id) references public.server_worker(public_id) on update cascade on delete cascade;
+-- name: server_worker_api_tag server_worker_fkey; type: fk constraint; schema: public; owner: -
+ add constraint server_worker_fkey foreign key (worker_id) references public.server_worker(public_id) on update cascade on delete cascade;
diff --git a/.schema-diff/fk_constraints_5896a3b44b650875b9bad08d5b0c7d8d8fe2ba4c/server_worker_tag_enm_fkey.sql b/.schema-diff/fk_constraints_5896a3b44b650875b9bad08d5b0c7d8d8fe2ba4c/server_worker_tag_enm_fkey.sql
deleted file mode 100644
index 65d35df9f..000000000
--- a/.schema-diff/fk_constraints_5896a3b44b650875b9bad08d5b0c7d8d8fe2ba4c/server_worker_tag_enm_fkey.sql
+++ /dev/null
@@ -1,2 +0,0 @@
--- name: server_worker_tag server_worker_tag_enm_fkey; type: fk constraint; schema: public; owner: -
- add constraint server_worker_tag_enm_fkey foreign key (source) references public.server_worker_tag_enm(source) on update cascade on delete restrict; |
johanbrandhorst
approved these changes
Dec 6, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.