Skip to content

Commit 5a25ea0

Browse files
tobiichi3227LifeAdventurer
tobiichi3227
and
LifeAdventurer
authored
Misc: E2E test, migration, upgrade script, testcase setup UI, custom problem-class (#94)
* fix: missing third-party js script * chore: add redis-db and service port install option * refactor: we don't need multi-threading to do cleanup work * refactor: there should be no distinction between kernels and users Follow TOJ Spec * feat(UI): feat: add password eye Copy From TNFSH-Scoreboard * feat: add db and redis migration * feat: add simple upgrade script * feat: problem add allow submit option * feat: user add custom motto * perf: incremental refresh challenge_state The challenge_state is used to cache all challenge results. When refreshing the challenge_state, the database will recalculate all data about the challenges, but many challenges are unnecessarily updated. So the incremental refresh only updates challenges that have changed. * refactor: remove unused problem expire field * refactor: move get problem state from `list_pro()` to `map_acct_rate()` * feat: redirect to sign-in page when user is a guest * test: add e2e test * feat: prevent unauthorized user from updating password * test: add user motto test * test: add allow_submit option test * fix: change progress bar text * fix: pdf file is displayed as garbled text * feat: add a hash to check file integrity during upload * fix: escape characters when content contain code block * fix: do not push empty url to history * feat: add testcase setup ui In this PR, we add the full file management UI. Now we can use the web UI to manage our test case files, attachments, and checkers. We no longer need to upload a problem package to overwrite files to achieve this goal. We also introduce multiple language limit settings, allowing us to set specific languages. This is important for Python 3 or Java because they are usually slower than compiled languages like C or C++. We will provide multiple test case file operations, which can reduce many single and duplicate operations for users. * feat: add a simple log viewer for log params * feat: new problem class system In this PR, we added a customizable proclass for users, which they can set to be publicly shared for others to use. We improved the original proclass selection menu and added a new interface for this purpose. We also introduced a feature to collect proclass, allowing users to gather their favorite proclass. * fix: follow the target behavior when a link has a target attribute * refactor: move `self.acct` to the default namespace to reduce argument passing * perf: use batch inserts to reduce SQL execution * fix: corrected wrong acct_id in a multi-threading environment * perf: use SQL to calculate user rank instead of Python In TOJ, response time decreased from 2000ms to about 300ms * fix: only update the code hash when the code is actually submitted * ci: bypass installation restrictions of the package manager * ci: remove unnecessary zip compression, as actions/upload-artifact will handle this * fix: module not found error * refactor: make ruff and pyright happy * Refactor: Improve migration logic and error handling in main function * fix: correct wrong log message grammar * chore: remove `f.close()`, as the with statement handles this --------- Co-authored-by: lifeadventurer <[email protected]>
1 parent f6cb976 commit 5a25ea0

File tree

158 files changed

+7924
-1148
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

158 files changed

+7924
-1148
lines changed

‎.github/workflows/tests.yml

+69
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
name: tests
2+
on:
3+
push:
4+
branches:
5+
- '**'
6+
7+
permissions:
8+
contents: read
9+
10+
jobs:
11+
e2etest:
12+
runs-on: ubuntu-latest
13+
steps:
14+
- uses: actions/checkout@v3
15+
- uses: actions/setup-python@v5
16+
with:
17+
python-version: 3.12
18+
19+
- name: Install Poetry
20+
run: |
21+
curl -sSL https://install.python-poetry.org | python3 -
22+
23+
- name: Install PostgreSQL, Redis, Dos2Unix
24+
run: |
25+
sudo curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | gpg --dearmor | sudo tee /usr/share/keyrings/postgresql.gpg
26+
echo deb [arch=amd64 signed-by=/usr/share/keyrings/postgresql.gpg] http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main | sudo tee /etc/apt/sources.list.d/postgresql.list
27+
sudo apt update -y
28+
sudo apt install -f -y postgresql-16 postgresql-client-16 redis dos2unix
29+
sudo sed -i 's/peer/trust/' /etc/postgresql/16/main/pg_hba.conf
30+
sudo service postgresql start
31+
sudo service redis-server start
32+
33+
- name: Install Coverage
34+
run: |
35+
$HOME/.local/bin/poetry add --dev coverage
36+
37+
- name: Install Project dependencies
38+
run: |
39+
sed -i '/^mkdocs-material/d' pyproject.toml # test don't need mkdocs-material
40+
rm poetry.lock # we need to remove it because we change the pyproject file
41+
$HOME/.local/bin/poetry install
42+
$HOME/.local/bin/poetry add beautifulsoup4
43+
$HOME/.local/bin/poetry add requests
44+
45+
- name: Deploy NTOJ-Judge
46+
run: |
47+
cd $HOME
48+
git clone https://github.com/tobiichi3227/NTOJ-Judge
49+
cd NTOJ-Judge/src
50+
sudo pip3 install tornado cffi --break-system-packages
51+
chmod +x ./runserver.sh
52+
sudo ./runserver.sh > output.log 2>&1 &
53+
54+
- name: Run e2e test
55+
run: |
56+
cd src
57+
chmod +x runtests.sh
58+
./runtests.sh
59+
60+
- name: Output judge log
61+
run: |
62+
cd $HOME/NTOJ-Judge/src
63+
cat output.log
64+
65+
- name: Upload Coverage Report
66+
uses: actions/upload-artifact@v3
67+
with:
68+
name: coverage-report
69+
path: src/htmlcov

‎migration/20240920182633_add_challenge_contest_id_index.py

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
async def dochange(db_conn, rs_conn):
2+
await db_conn.execute('CREATE INDEX challenge_idx_contest_id ON public.challenge USING btree (contest_id)')

‎migration/20240925015623_add_problem_allow_submit.py

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
async def dochange(db, rs):
2+
await db.execute('ALTER TABLE problem ADD allow_submit boolean DEFAULT true')

‎migration/20240925163422_user_add_motto.py

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
async def dochange(db, rs):
2+
await db.execute(
3+
'''
4+
ALTER TABLE account ADD motto character varying DEFAULT ''::character varying
5+
'''
6+
)
7+
8+
result = await db.fetch("SELECT last_value FROM account_acct_id_seq;")
9+
cur_acct_id = int(result[0]["last_value"])
10+
11+
for acct_id in range(1, cur_acct_id + 1):
12+
await rs.delete(f"account@{acct_id}")
13+
14+
await rs.delete("acctlist")

‎migration/20240925173122_add_incremental_challenge_state.py

+108
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
async def dochange(db, rs):
2+
await db.execute('DROP MATERIALIZED VIEW challenge_state;')
3+
4+
await db.execute(
5+
'''
6+
CREATE TABLE challenge_state (
7+
chal_id integer NOT NULL,
8+
state integer,
9+
runtime bigint DEFAULT 0,
10+
memory bigint DEFAULT 0,
11+
rate integer DEFAULT 0
12+
);
13+
''')
14+
await db.execute(
15+
'''
16+
ALTER TABLE ONLY public.challenge_state
17+
ADD CONSTRAINT challenge_state_forkey_chal_id FOREIGN KEY (chal_id) REFERENCES public.challenge(chal_id) ON DELETE CASCADE;
18+
''')
19+
20+
await db.execute("ALTER TABLE challenge_state ADD CONSTRAINT challenge_state_unique_chal_id UNIQUE(chal_id);")
21+
22+
23+
await db.execute(
24+
'''
25+
CREATE TABLE last_update_time (
26+
view_name TEXT PRIMARY KEY,
27+
last_update TIMESTAMP WITH TIME ZONE
28+
);
29+
'''
30+
)
31+
32+
await db.execute("INSERT INTO last_update_time (view_name, last_update) VALUES ('challenge_state', NOW());")
33+
34+
await db.execute("ALTER TABLE test ADD COLUMN last_modified TIMESTAMP WITH TIME ZONE DEFAULT NOW();")
35+
36+
await db.execute("CREATE INDEX idx_test_last_modified ON test (last_modified);")
37+
await db.execute("CREATE UNIQUE INDEX ON test_valid_rate (pro_id, test_idx);")
38+
39+
await db.execute(
40+
'''
41+
CREATE OR REPLACE FUNCTION update_test_last_modified()
42+
RETURNS TRIGGER AS $$
43+
BEGIN
44+
NEW.last_modified = NOW();
45+
RETURN NEW;
46+
END;
47+
$$ LANGUAGE plpgsql;
48+
''')
49+
50+
await db.execute(
51+
'''
52+
CREATE TRIGGER test_last_modified_trigger
53+
BEFORE UPDATE ON test
54+
FOR EACH ROW EXECUTE FUNCTION update_test_last_modified();
55+
''')
56+
57+
await db.execute(
58+
'''
59+
CREATE OR REPLACE FUNCTION refresh_challenge_state_incremental()
60+
RETURNS VOID AS $$
61+
DECLARE
62+
last_update_time TIMESTAMP WITH TIME ZONE;
63+
BEGIN
64+
SELECT last_update INTO last_update_time
65+
FROM last_update_time
66+
WHERE view_name = 'challenge_state';
67+
68+
WITH challenge_summary AS (
69+
SELECT
70+
t.chal_id,
71+
MAX(t.state) AS max_state,
72+
SUM(t.runtime) AS total_runtime,
73+
SUM(t.memory) AS total_memory,
74+
SUM(CASE WHEN t.state = 1 THEN tvr.rate ELSE 0 END) AS total_rate
75+
FROM test t
76+
LEFT JOIN test_valid_rate tvr ON t.pro_id = tvr.pro_id AND t.test_idx = tvr.test_idx
77+
WHERE t.last_modified > last_update_time
78+
GROUP BY t.chal_id
79+
),
80+
upsert_result AS (
81+
INSERT INTO challenge_state (chal_id, state, runtime, memory, rate)
82+
SELECT
83+
chal_id,
84+
max_state,
85+
total_runtime,
86+
total_memory,
87+
total_rate
88+
FROM challenge_summary
89+
ON CONFLICT (chal_id) DO UPDATE
90+
SET
91+
state = EXCLUDED.state,
92+
runtime = EXCLUDED.runtime,
93+
memory = EXCLUDED.memory,
94+
rate = EXCLUDED.rate
95+
WHERE
96+
challenge_state.state != EXCLUDED.state OR
97+
challenge_state.runtime != EXCLUDED.runtime OR
98+
challenge_state.memory != EXCLUDED.memory OR
99+
challenge_state.rate != EXCLUDED.rate
100+
)
101+
102+
UPDATE last_update_time
103+
SET last_update = NOW()
104+
WHERE view_name = 'challenge_state';
105+
END;
106+
$$ LANGUAGE plpgsql;
107+
''')
108+
await db.execute('SELECT refresh_challenge_state_incremental();')

‎migration/20240926201600_remove_problem_expire_field.py

+19
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
async def dochange(db, rs):
2+
await db.execute('DROP MATERIALIZED VIEW test_valid_rate;')
3+
await db.execute('ALTER TABLE problem DROP COLUMN expire;')
4+
await db.execute(
5+
'''
6+
CREATE MATERIALIZED VIEW public.test_valid_rate AS
7+
SELECT test_config.pro_id,
8+
test_config.test_idx,
9+
count(DISTINCT account.acct_id) AS count,
10+
test_config.weight AS rate
11+
FROM (((public.test
12+
JOIN public.account ON ((test.acct_id = account.acct_id)))
13+
JOIN public.problem ON (((((test.pro_id = problem.pro_id)) AND (test.state = 1)))))
14+
RIGHT JOIN public.test_config ON (((test.pro_id = test_config.pro_id) AND (test.test_idx = test_config.test_idx))))
15+
GROUP BY test_config.pro_id, test_config.test_idx, test_config.weight
16+
WITH NO DATA;
17+
''')
18+
await db.execute('REFRESH MATERIALIZED VIEW test_valid_rate;')
19+
await rs.delete('prolist')

‎migration/20240930220712_add_testcase.py

+12
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
import json
2+
3+
async def dochange(db, rs):
4+
test_configs = await db.fetch('SELECT pro_id, test_idx, metadata FROM test_config;')
5+
6+
for pro_id, test_group_idx, metadata in test_configs:
7+
metadata = json.loads(metadata)
8+
for i in range(len(metadata["data"])):
9+
metadata["data"][i] = str(metadata["data"][i])
10+
11+
await db.execute('UPDATE test_config SET metadata = $1 WHERE pro_id = $2 AND test_idx = $3',
12+
json.dumps(metadata), pro_id, test_group_idx)

‎migration/20241001153200_move_test_fields_to_problem.py

+49
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
import json
2+
3+
async def dochange(db, rs):
4+
CHECK_TYPES = {
5+
"diff": 0,
6+
"diff-strict": 1,
7+
"diff-float": 2,
8+
"ioredir": 3,
9+
"cms": 4
10+
}
11+
12+
await db.execute("ALTER TABLE problem ADD check_type integer DEFAULT 0")
13+
await db.execute("ALTER TABLE problem ADD is_makefile boolean DEFAULT false")
14+
await db.execute("""
15+
ALTER TABLE problem ADD "limit" jsonb DEFAULT '{"default": {"timelimit": 0, "memlimit":0}}'::jsonb
16+
""")
17+
await db.execute("ALTER TABLE problem ADD chalmeta jsonb DEFAULT '{}'::jsonb")
18+
19+
res = await db.fetch("SELECT pro_id FROM problem;")
20+
for pro in res:
21+
pro_id = pro['pro_id']
22+
limit = {
23+
'default': {
24+
'timelimit': 0,
25+
'memlimit': 0,
26+
}
27+
}
28+
f_check_type = 0
29+
f_is_makefile = False
30+
f_chalmeta = {}
31+
32+
res = await db.fetch('SELECT check_type, compile_type, chalmeta, timelimit, memlimit FROM test_config WHERE pro_id = $1', pro_id)
33+
for check_type, compile_type, chalmeta, timelimit, memlimit in res:
34+
f_check_type = CHECK_TYPES[check_type]
35+
f_is_makefile = compile_type == 'makefile'
36+
f_chalmeta = json.loads(chalmeta)
37+
limit['default']['timelimit'] = timelimit
38+
limit['default']['memlimit'] = memlimit
39+
40+
await db.execute("UPDATE problem SET check_type = $1, is_makefile = $2, \"limit\" = $3, chalmeta = $4 WHERE pro_id = $5",
41+
f_check_type, f_is_makefile, json.dumps(limit), json.dumps(f_chalmeta), pro_id)
42+
43+
44+
await db.execute('ALTER TABLE test_config DROP COLUMN check_type;')
45+
await db.execute('ALTER TABLE test_config DROP COLUMN score_type;')
46+
await db.execute('ALTER TABLE test_config DROP COLUMN compile_type;')
47+
await db.execute('ALTER TABLE test_config DROP COLUMN chalmeta;')
48+
await db.execute('ALTER TABLE test_config DROP COLUMN timelimit;')
49+
await db.execute('ALTER TABLE test_config DROP COLUMN memlimit;')

‎migration/20241005212800_add_trigger_for_test_deleted.py

+19
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
async def dochange(db, rs):
2+
await db.execute(
3+
'''
4+
CREATE OR REPLACE FUNCTION delete_challenge_state()
5+
RETURNS TRIGGER AS $$
6+
BEGIN
7+
DELETE FROM challenge_state WHERE chal_id = OLD.chal_id;
8+
RETURN OLD;
9+
END;
10+
$$ LANGUAGE plpgsql;
11+
''')
12+
13+
await db.execute(
14+
'''
15+
CREATE TRIGGER trigger_delete_challenge_state
16+
AFTER DELETE ON test
17+
FOR EACH ROW
18+
EXECUTE FUNCTION delete_challenge_state();
19+
''')

‎migration/20241006130410_change_log_param_to_jsonb.py

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
async def dochange(db, rs):
2+
await db.execute(
3+
"ALTER TABLE log ALTER COLUMN params TYPE jsonb USING params::jsonb"
4+
)
5+
await db.execute(
6+
"ALTER TABLE log ALTER COLUMN params SET DEFAULT '{}'::jsonb"
7+
)

‎migration/20241006151800_problem_class.py

+22
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
class ProClassConst:
2+
OFFICIAL_PUBLIC = 0
3+
OFFICIAL_HIDDEN = 1
4+
USER_PUBLIC = 2
5+
USER_HIDDEN = 3
6+
7+
8+
async def dochange(db, rs):
9+
# NOTE: rename
10+
await db.execute("ALTER TABLE pubclass RENAME TO proclass")
11+
await db.execute("ALTER SEQUENCE pubclass_pubclass_id_seq RENAME TO proclass_proclass_id_seq")
12+
await db.execute("ALTER TABLE proclass RENAME COLUMN pubclass_id TO proclass_id")
13+
await db.execute("ALTER TABLE proclass RENAME CONSTRAINT pubclass_pkey TO proclass_pkey")
14+
15+
await db.execute('''ALTER TABLE proclass ADD "desc" text DEFAULT \'\'''')
16+
await db.execute("ALTER TABLE proclass ADD acct_id integer")
17+
await db.execute('ALTER TABLE proclass ADD "type" integer')
18+
await db.execute(
19+
"ALTER TABLE proclass ADD CONSTRAINT proclass_forkey_acct_id FOREIGN KEY (acct_id) REFERENCES account(acct_id) ON DELETE CASCADE"
20+
)
21+
await db.execute('UPDATE proclass SET "type" = $1', ProClassConst.OFFICIAL_PUBLIC)
22+
await db.execute('ALTER TABLE proclass ALTER COLUMN "type" SET NOT NULL')

‎migration/20241007235900_add_user_proclass_collection.py

+9
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
async def dochange(db, rs):
2+
await db.execute("ALTER TABLE account ADD proclass_collection integer[] NOT NULL DEFAULT '{}'::integer[]")
3+
result = await db.fetch("SELECT last_value FROM account_acct_id_seq;")
4+
cur_acct_id = int(result[0]['last_value'])
5+
6+
for acct_id in range(1, cur_acct_id + 1):
7+
await rs.delete(f"account@{acct_id}")
8+
9+
await rs.delete('acctlist')

0 commit comments

Comments
 (0)