Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/queue connector #458

Closed
wants to merge 70 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
7c35d73
queue connector in progress
vishnuszipstack May 27, 2024
6d4bade
Merge branch 'main' into feat/queue-connector
vishnuszipstack May 28, 2024
7cfb9b7
Merge branch 'main' into feat/queue-connector
vishnuszipstack May 29, 2024
a393db3
queue connector base class added and typo fix in Actions.jsx
vishnuszipstack May 31, 2024
fe6a08d
Merge branch 'main' into feat/queue-connector
vishnuszipstack May 31, 2024
31efe7d
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] May 31, 2024
a06e378
Delete backend/workflow_data directory
vishnuszipstack May 31, 2024
fc3724a
addressed PR comments by changing Queue type to MANUAL_REVIEW and som…
vishnuszipstack Jun 5, 2024
6ceff1f
Merge branch 'main' into feat/queue-connector
vishnuszipstack Jun 5, 2024
f2b951e
Merge branch 'main' into feat/queue-connector
vishnuszipstack Jun 5, 2024
bdd3cc8
merged main
vishnuszipstack Jun 5, 2024
596e88c
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 5, 2024
8d87268
Merge branch 'main' into feat/queue-connector
vishnuszipstack Jun 6, 2024
8e2dfc8
Merge branch 'main' into feat/queue-connector
vishnuszipstack Jun 7, 2024
4b01494
Merge branch 'main' into feat/queue-connector
vishnuszipstack Jun 13, 2024
ca25fe1
Merge branch 'main' into feat/queue-connector
vishnuszipstack Jun 20, 2024
b3896f9
merged main
vishnuszipstack Jun 25, 2024
183a5ef
queue connector changes and role expose to frontend and role adding o…
vishnuszipstack Jun 25, 2024
8c2ba49
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 25, 2024
055dd8a
merged migrations
vishnuszipstack Jun 25, 2024
b68b05e
Merge remote-tracking branch 'origin/feat/queue-connector' into feat/…
vishnuszipstack Jun 25, 2024
ad7bc40
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 25, 2024
724b9ef
pr comment fix
vishnuszipstack Jun 25, 2024
5173e0b
Merge remote-tracking branch 'origin/feat/queue-connector' into feat/…
vishnuszipstack Jun 25, 2024
a2a4fc7
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 25, 2024
9cf506b
added abstract method for redis connector
vishnuszipstack Jun 26, 2024
53a886d
Merge remote-tracking branch 'origin/feat/queue-connector' into feat/…
vishnuszipstack Jun 26, 2024
81d4899
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jun 26, 2024
6e8bbab
merged main
vishnuszipstack Jul 1, 2024
5a652e5
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 1, 2024
6faea66
added feature flag
vishnuszipstack Jul 1, 2024
1c750f6
Merge remote-tracking branch 'origin/feat/queue-connector' into feat/…
vishnuszipstack Jul 1, 2024
67382d6
Merge branch 'main' into feat/queue-connector
vishnuszipstack Jul 1, 2024
700f3fb
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 1, 2024
38ef61e
Merge branch 'main' into feat/queue-connector
nehabagdia Jul 2, 2024
bcd8b10
Merge branch 'main' into feat/queue-connector
vishnuszipstack Jul 2, 2024
6f27c56
Merge remote-tracking branch 'origin/feat/queue-connector' into feat/…
vishnuszipstack Jul 2, 2024
19ccf1f
added feature flags for manual review queue and added whisper hash to…
vishnuszipstack Jul 3, 2024
e46fd59
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 3, 2024
12e63e9
added removed line
vishnuszipstack Jul 3, 2024
80c35b4
changed return empty to pass in abstract methods in connector base class
vishnuszipstack Jul 3, 2024
0229401
Merge remote-tracking branch 'origin/feat/queue-connector' into feat/…
vishnuszipstack Jul 3, 2024
6e814c3
Merge branch 'main' into feat/queue-connector
vishnuszipstack Jul 3, 2024
22ad735
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 3, 2024
edbe74e
reverted change return pass to previous values
vishnuszipstack Jul 3, 2024
42a1fde
Merge remote-tracking branch 'origin/feat/queue-connector' into feat/…
vishnuszipstack Jul 3, 2024
17d7d44
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 3, 2024
a848ffd
added data class for queue result
vishnuszipstack Jul 4, 2024
00368eb
Merge remote-tracking branch 'origin/feat/queue-connector' into feat/…
vishnuszipstack Jul 4, 2024
799cff0
added removed subscription import
vishnuszipstack Jul 4, 2024
c8d83d9
merged main and resolved conflicts
vishnuszipstack Jul 4, 2024
48d6a81
added new migrations for manual review queue
vishnuszipstack Jul 4, 2024
15c35e5
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 4, 2024
77efdc5
Merge branch 'main' into feat/queue-connector
vishnuszipstack Jul 4, 2024
0f59b28
Merge branch 'main' into feat/queue-connector
vishnuszipstack Jul 5, 2024
254b2eb
pr comment fix removed multiple initialization of auth controller
vishnuszipstack Jul 5, 2024
278de45
Update backend/utils/user_session.py
vishnuszipstack Jul 5, 2024
124b2a0
Update backend/workflow_manager/endpoint/queue_utils.py
vishnuszipstack Jul 5, 2024
2d3444d
Update unstract/connectors/src/unstract/connectors/queues/__init__.py
vishnuszipstack Jul 5, 2024
18c8e04
Update unstract/connectors/src/unstract/connectors/queues/register.py
vishnuszipstack Jul 5, 2024
885affa
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 5, 2024
be4de0d
pr comment fixes
vishnuszipstack Jul 5, 2024
27cdb7d
Merge branch 'main' into feat/queue-connector
vishnuszipstack Jul 5, 2024
fc29aee
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 5, 2024
a4844fe
removed unused code
vishnuszipstack Jul 5, 2024
4226bfb
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jul 5, 2024
b7f35f4
pre commit -issue fixes
vishnuszipstack Jul 5, 2024
08d4f02
Merge remote-tracking branch 'origin/feat/queue-connector' into feat/…
vishnuszipstack Jul 5, 2024
bb9927e
merged main and resolved conflicts
vishnuszipstack Jul 6, 2024
fda8b42
added dataclass and changes in destination queue result
vishnuszipstack Jul 8, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion backend/workflow_manager/endpoint/destination.py
Original file line number Diff line number Diff line change
Expand Up @@ -526,7 +526,7 @@ def _push_to_queue(
file_content=file_content_base64,
)
# Convert the result dictionary to a JSON string
queue_result_json = json.dumps(queue_result)
queue_result_json = json.dumps(queue_result.to_serializable_dict())
conn = QueueUtils.get_queue_inst()
# Enqueue the JSON string
conn.enqueue(queue_name=q_name, message=queue_result_json)
10 changes: 10 additions & 0 deletions backend/workflow_manager/endpoint/queue_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,3 +39,13 @@ class QueueResult:
result: Any
workflow_id: str
file_content: str

def to_serializable_dict(self):
return {
"file": self.file,
"whisper_hash": self.whisper_hash,
"status": self.status,
"result": self.result,
"workflow_id": self.workflow_id,
"file_content": self.file_content,
}
2 changes: 0 additions & 2 deletions frontend/src/routes/Router.jsx
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,6 @@ try {
require("../plugins/app-deployment/AppDeployments.jsx").AppDeployments;
ChatAppPage =
require("../plugins/app-deployment/chat-app/ChatAppPage.jsx").ChatAppPage;
ChatAppLayout =
require("../plugins/app-deployment/chat-app/ChatAppLayout.jsx").ChatAppLayout;
} catch (err) {
// Do nothing, Not-found Page will be triggered.
}
Expand Down