Improve reliability by reverting to transaction mode #12
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…functions Modify postgres connection configuration to set prepare to false in: - Worker.ts - cpu_intensive/index.ts - max_concurrency/index.ts - serial_sleep/index.ts - tests/sql.ts This change updates the postgres connection settings to disable statement preparation, potentially addressing performance or compatibility issues across multiple edge worker functions and test configurations.
Enhance logging for task execution and archiving process with more descriptive debug messages. Updated log statements to provide clearer insights into task lifecycle and execution status.
Update import statements in deno.json to reorganize and add new dependencies, and increase max client connections in Supabase configuration from 200 to 250
Replace deprecated p-queue library with @henrygd/queue, updating import statements, queue initialization, and method calls in ExecutionController. Remove unnecessary dependencies and update project lock file accordingly. Breaking changes: - Switched from npm:p-queue to jsr:@henrygd/queue - Modified queue initialization and method usage - Commented out archiver flush method
…nction Remove random delay and replace with minimal delay to optimize function behavior
Add additional internal exports for Worker, Queries, Queue, and types to provide more comprehensive access to edge worker package internals
…tion Update connection pooling configuration and add note about transaction mode requirements for edge workers. Modify config.toml example to enable pooler and clarify connection settings.
Enable database pooler in config example and update code block formatting
Deleted documentation file for edge worker troubleshooting, which contained details about worker message processing issues and connection timeout errors
Incremented package version from 0.0.2 to 0.0.3 in deno.json configuration
Adds a check to the Worker.stop() that just returns if worker is stopping or stopped. This should never happen because `onbeforeunload` is called only once, but for some reason in high concurrency scenarios it is called again and triggers transition error on WorkerState (good thing!).
…ing metrics Add two new SQL query files to help diagnose system performance: - debug_connections.sql: Tracks connection states and usage for Supavisor and Postgres - debug_processing_gaps.sql: Provides detailed analysis of message processing times, retries, and performance bottlenecks
Adjust test configuration by increasing message count and remove redundant delay in sequence increment function
Extract common database and utility functions into a centralized utils module, simplifying imports and reducing code duplication across edge worker functions. Reduce boilerplate by moving shared configurations and utility methods to a single file. - Create new utils.ts with shared SQL connection, sleep function, and random integer generator - Remove redundant database URL logging and connection setup in individual functions - Simplify imports across multiple edge worker function files - Reduce test message count in performance test
Refine documentation for Edge Worker project status, including: - Restructure content for clarity - Update description of connection pool saturation issue - Revise section on high-concurrency challenges - Add planned next steps and architectural improvement notes Provides more focused and precise overview of current system limitations and future improvements
Remove warning emoji from project status title and navigation Add badge to highlight project status importance
Add dynamic scaling and breathing glow animations for hero image - Replace static hover transform with keyframe animations - Implement scale-up animation from 0.7 to 1 - Create breathing-glow effect with subtle shadow pulsing
Add theme-specific logo glow color variables and replace hardcoded color with dynamic variable reference
…nd percentile metrics Modify SQL query to: - Add retry count calculation - Adjust processing time calculations - Introduce percentile-based processing time metrics - Include min/max processing time measurements
Deploying pgflow with
|
| Latest commit: |
0079118
|
| Status: | ⚡️ Build in progress... |
|
View your CI Pipeline Execution ↗ for commit 0079118.
☁️ Nx Cloud last updated this comment at |
… worker output Enhance example logs with more specific worker execution details, including worker ID and task scheduling information
Improve example Edge Worker code to demonstrate fetching and logging website response details
Enhance documentation with more context about web scraping workers, including their use case and benefits. Improve introductory description to provide clearer explanation of the worker's purpose and setup process.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR includes various fixes and will be released as 0.0.3: