From ee72cf96b0dad65355d4843f3e058aa2a3ecb74e Mon Sep 17 00:00:00 2001 From: Michael Kofi Armah Date: Tue, 11 Feb 2025 13:49:07 +0000 Subject: [PATCH] [Integration][Gitlab] Fix Memory Leak Identified In Live Events Sync (#1371) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ##Description What Currently, when a live event is received, we pass a deep copy of the event object to each observer. This was originally implemented to ensure that every observer received its own, uniquely referenced copy of the event. However, in our current workflow, observers do not modify the event data—they only use it to fetch resources from GitLab. Why - Since the observers only perform read operations (i.e., fetching data) and do not modify the event object, creating deep copies is redundant. - It has been observed that the deep‑copied objects are not being garbage collected after use, which results in increased memory usage and could eventually lead to memory exhaustion. - The deep copy operation introduces additional processing time and memory overhead that is not needed given the current use case. How - Modify the event handling logic to pass the original event object (or a shallow copy if absolutely necessary for safety) to observers instead of performing a deep copy. - Since observers only read from the event data, they can safely operate on the same shared object without risking race conditions or data corruption. ## Type of change Please leave one option from the following and delete the rest: - [x] Bug fix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] New Integration (non-breaking change which adds a new integration) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Non-breaking change (fix of existing functionality that will not change current behavior) - [ ] Documentation (added/updated documentation)

All tests should be run against the port production environment(using a testing org).

### Core testing checklist - [ ] Integration able to create all default resources from scratch - [ ] Resync finishes successfully - [ ] Resync able to create entities - [ ] Resync able to update entities - [ ] Resync able to detect and delete entities - [ ] Scheduled resync able to abort existing resync and start a new one - [ ] Tested with at least 2 integrations from scratch - [ ] Tested with Kafka and Polling event listeners - [ ] Tested deletion of entities that don't pass the selector ### Integration testing checklist - [ ] Integration able to create all default resources from scratch - [ ] Resync able to create entities - [ ] Resync able to update entities - [ ] Resync able to detect and delete entities - [ ] Resync finishes successfully - [ ] If new resource kind is added or updated in the integration, add example raw data, mapping and expected result to the `examples` folder in the integration directory. - [ ] If resource kind is updated, run the integration with the example data and check if the expected result is achieved - [ ] If new resource kind is added or updated, validate that live-events for that resource are working as expected - [ ] Docs PR link [here](#) ### Preflight checklist - [ ] Handled rate limiting - [ ] Handled pagination - [ ] Implemented the code in async - [ ] Support Multi account ## Screenshots Include screenshots from your environment showing how the resources of the integration will look. ## API Documentation Provide links to the API documentation used for this integration. --- integrations/gitlab/CHANGELOG.md | 8 ++++++++ .../gitlab/gitlab_integration/events/event_handler.py | 4 ++-- integrations/gitlab/pyproject.toml | 2 +- 3 files changed, 11 insertions(+), 3 deletions(-) diff --git a/integrations/gitlab/CHANGELOG.md b/integrations/gitlab/CHANGELOG.md index d74cb5deb1..a2a38a34aa 100644 --- a/integrations/gitlab/CHANGELOG.md +++ b/integrations/gitlab/CHANGELOG.md @@ -7,6 +7,14 @@ this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm +0.2.34 (2025-02-11) +=================== + +### Improvements + +- Removed the need for sending a deepcopy of the event body to all listeners, which previously led to a memory leak + + 0.2.33 (2025-02-09) =================== diff --git a/integrations/gitlab/gitlab_integration/events/event_handler.py b/integrations/gitlab/gitlab_integration/events/event_handler.py index 004e86740a..611a566470 100644 --- a/integrations/gitlab/gitlab_integration/events/event_handler.py +++ b/integrations/gitlab/gitlab_integration/events/event_handler.py @@ -98,7 +98,7 @@ async def _notify(self, event: str, body: dict[str, Any]) -> None: event=event, handler=handler, ) - asyncio.create_task(observer(event, deepcopy(body))) # type: ignore + asyncio.create_task(observer(event, body)) # type: ignore class SystemEventHandler(BaseEventHandler): @@ -119,7 +119,7 @@ async def _notify(self, event: str, body: dict[str, Any]) -> None: # access the project results = await asyncio.gather( *( - hook_handler(client).on_hook(event, deepcopy(body)) + hook_handler(client).on_hook(event, body) for client in self._clients for hook_handler in self._hook_handlers.get(event, []) ), diff --git a/integrations/gitlab/pyproject.toml b/integrations/gitlab/pyproject.toml index f9a45d25e4..247d6a7966 100644 --- a/integrations/gitlab/pyproject.toml +++ b/integrations/gitlab/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "gitlab" -version = "0.2.33" +version = "0.2.34" description = "Gitlab integration for Port using Port-Ocean Framework" authors = ["Yair Siman-Tov "]