Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: Remove retries for testcafe tests but increase timeout. Wait for lag in parallel #1380

Merged
merged 54 commits into from
Aug 26, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
3315abc
Bump typescript version to 5.5 and also bump related libraries
robbie-c Aug 23, 2024
e0bbafd
Remove retries but increase timeout
robbie-c Aug 22, 2024
2b980a4
Wrap in Selector
robbie-c Aug 22, 2024
d0ecfa0
Add console log saying how long it took to get results
robbie-c Aug 22, 2024
37e5313
Add "after" query param
robbie-c Aug 22, 2024
0df6992
Add max allowed api errors
robbie-c Aug 22, 2024
2c0971a
Stop on first fail
robbie-c Aug 22, 2024
e978e53
Make testSessionId more readable, change ingestion endpoint
robbie-c Aug 22, 2024
20b7b22
Don't use uuidv7
robbie-c Aug 22, 2024
3d97db7
Pass env vars as dependencies
robbie-c Aug 22, 2024
7d11f80
Improve test namr
robbie-c Aug 22, 2024
7a3992c
Fix typo
robbie-c Aug 22, 2024
2cc7d44
Typo
robbie-c Aug 22, 2024
cb48da6
only assert that custom-event was seen
Phanatic Aug 22, 2024
f6baa6e
try using a different distinct_id
Phanatic Aug 22, 2024
5ba43b5
undo changes, look for all events
Phanatic Aug 22, 2024
1f41f5a
use successfunction to look for custom-event in response
Phanatic Aug 23, 2024
dec57aa
Move asserts to a later step in the CI so that ingestion lag can be i…
robbie-c Aug 23, 2024
4e64374
Tidy up deps
robbie-c Aug 23, 2024
7874f6a
Tidy up how results file are written and read
robbie-c Aug 23, 2024
f973ec2
Use jest override
robbie-c Aug 23, 2024
59d1e0d
Fix complaining about timeout and interval types
robbie-c Aug 23, 2024
9533e30
WIP
robbie-c Aug 23, 2024
2ea7b9d
Override testcafe's typescript compiler
robbie-c Aug 23, 2024
148c9e0
Revert check script to js
robbie-c Aug 23, 2024
7e6136b
Sanitise file names
robbie-c Aug 23, 2024
a27f8ed
Correctly named script
robbie-c Aug 23, 2024
9f21b0a
Some more debuggability
robbie-c Aug 24, 2024
6419efe
Add a script to make it easier to run locally
robbie-c Aug 24, 2024
21c9eed
Add some debugging
robbie-c Aug 24, 2024
87a351a
Fix check file
robbie-c Aug 24, 2024
2975313
Revert changes to wait duration
robbie-c Aug 24, 2024
5e47fb6
Move waits around
robbie-c Aug 24, 2024
8a2b7c4
Add assertion for whether the snippet has loaded
robbie-c Aug 24, 2024
b28630d
Revert the assertion event count
robbie-c Aug 24, 2024
246c7b6
Re-add missing asserts
robbie-c Aug 24, 2024
00851c9
Remove some unused stuff
robbie-c Aug 24, 2024
e632bd8
Be more specific about test files
robbie-c Aug 24, 2024
32db853
Fix url
robbie-c Aug 24, 2024
9f88517
Disable request batching
robbie-c Aug 24, 2024
ec008a8
Rewrite the retry loop to do an attempt immediately
robbie-c Aug 24, 2024
6dd2362
Assert on the number of captures
robbie-c Aug 24, 2024
7553f55
Fix event count assert
robbie-c Aug 24, 2024
6150e9e
Add captures map
robbie-c Aug 24, 2024
30e7485
Test
robbie-c Aug 24, 2024
0b75fb6
Fix captures
robbie-c Aug 24, 2024
9c605c2
Fix first assert
robbie-c Aug 24, 2024
f5a923c
Fix other asserts
robbie-c Aug 24, 2024
13630aa
Fix deadline
robbie-c Aug 24, 2024
0252a45
Increase timeout to 30 minutes
robbie-c Aug 24, 2024
984bb3f
Use optional catch binding syntax
robbie-c Aug 26, 2024
5c863d0
Add an explainer comment
robbie-c Aug 26, 2024
44d5130
Remove unnecessary hackiness
robbie-c Aug 26, 2024
6caf509
Slightly improve the hackiness even more
robbie-c Aug 26, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .eslintrc.js
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ const rules = {
'@typescript-eslint/no-this-alias': 'off',
'@typescript-eslint/no-explicit-any': 'off',
'@typescript-eslint/no-unused-vars': ['error'],
'@typescript-eslint/no-unused-expressions': 'off',
'no-prototype-builtins': 'off',
'no-empty': 'off',
'no-console': 'error',
Expand Down Expand Up @@ -101,6 +102,12 @@ module.exports = {
Cypress: true,
},
},
{
files: 'testcafe/**/*',
globals: {
__dirname: true,
},
},
],
root: true,
}
9 changes: 8 additions & 1 deletion .github/workflows/testcafe.yml
Original file line number Diff line number Diff line change
Expand Up @@ -54,4 +54,11 @@ jobs:
run: pnpm build-rollup

- name: Run ${{ matrix.name }} test
run: node scripts/run-testcafe-with-retries.mjs --browser "${{ matrix.browser }}" --attempts 3
env:
BRANCH_NAME: ${{ github.head_ref || github.ref_name }}
RUN_ID: ${{ github.run_id }}
BROWSER: ${{ matrix.browser }}
run: pnpm testcafe ${{ matrix.browser }} --stop-on-first-fail

- name: Check ${{ matrix.name }} events
run: pnpm check-testcafe-results
7 changes: 6 additions & 1 deletion .testcaferc.js
Original file line number Diff line number Diff line change
@@ -1,3 +1,8 @@
module.exports = {
src: './testcafe',
src: './testcafe/*.spec.js',
compilerOptions: {
typescript: {
customCompilerModulePath: require.resolve('typescript'),
},
},
}
21 changes: 13 additions & 8 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,9 @@
"typecheck": "tsc --noEmit --project tsconfig.json",
"cypress": "cypress open",
"prepare": "husky install",
"deprecate-old-versions": "node scripts/deprecate-old-versions.mjs"
"deprecate-old-versions": "node scripts/deprecate-old-versions.mjs",
"check-testcafe-results": "ts-node testcafe/check-testcafe-results.js",
"run-testcafe-localhost": "node scripts/run-testcafe-localhost.mjs"
},
"main": "dist/main.js",
"module": "dist/module.js",
Expand Down Expand Up @@ -59,37 +61,39 @@
"@testing-library/preact": "^3.2.4",
"@types/eslint": "^8.44.6",
"@types/jest": "^29.5.1",
"@types/node": "^22.5.0",
"@types/react-dom": "^18.0.10",
"@types/sinon": "^17.0.1",
"@types/uuid": "^9.0.1",
"@types/web": "^0.0.154",
"@typescript-eslint/eslint-plugin": "^6.19.0",
"@typescript-eslint/parser": "^6.19.0",
"@typescript-eslint/eslint-plugin": "^8.2.0",
"@typescript-eslint/parser": "^8.2.0",
"babel-eslint": "10.1.0",
"babel-jest": "^26.6.3",
"compare-versions": "^6.1.0",
"cypress": "13.6.3",
"cypress-localstorage-commands": "^2.2.6",
"date-fns": "^3.6.0",
"eslint": "8.56.0",
"eslint": "8.57.0",
"eslint-config-posthog-js": "link:eslint-rules",
"eslint-config-prettier": "^8.5.0",
"eslint-plugin-compat": "^4.1.4",
"eslint-plugin-jest": "^27.2.3",
"eslint-plugin-jest": "^28.8.0",
"eslint-plugin-no-only-tests": "^3.1.0",
"eslint-plugin-posthog-js": "link:eslint-rules",
"eslint-plugin-prettier": "^4.2.1",
"eslint-plugin-react": "^7.30.1",
"eslint-plugin-react-hooks": "^4.6.0",
"express": "^4.18.2",
"expect": "^29.7.0",
"express": "^4.19.2",
"fast-check": "^2.17.0",
"husky": "^8.0.1",
"jest": "^27.5.1",
"jsdom": "16.5.0",
"jsdom-global": "3.0.2",
"lint-staged": "^10.2.11",
"localStorage": "1.0.4",
"msw": "^1.2.1",
"msw": "^1.3.3",
"node-fetch": "^2.6.11",
"posthog-js": "link:",
"preact-render-to-string": "^6.3.1",
Expand All @@ -103,7 +107,8 @@
"testcafe": "1.19.0",
"testcafe-browser-provider-browserstack": "1.14.0",
"tslib": "^2.5.0",
"typescript": "^4.9.5",
"ts-node": "^10.9.2",
"typescript": "^5.5.4",
"yargs": "^17.7.2"
},
"lint-staged": {
Expand Down
Loading
Loading