Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add indices migration #942

Merged
merged 14 commits into from
Dec 10, 2024
Merged

feat: Add indices migration #942

merged 14 commits into from
Dec 10, 2024

Conversation

whiterabbit1983
Copy link
Contributor

@whiterabbit1983 whiterabbit1983 commented Dec 9, 2024

Important

Adds migration for indices and updates model queries to utilize these indices for improved performance.

  • Migration:
    • Adds migrate_1733755642_transition_indices.py to create/drop indices for executions, tasks, agents, sessions, docs, users, and transitions tables.
  • Model Updates:
    • Updates queries in search_docs_by_embedding.py and search_docs_by_text.py to use docs:owner_id_metadata_doc_id_idx.
    • Updates count_executions.py to use executions:task_id_execution_id_idx.
    • Updates create_execution_transition.py, get_paused_execution_token.py, and list_execution_transitions.py to use transitions:execution_id_type_created_at_idx.
    • Updates get_execution.py and update_execution.py to use executions:execution_id_status_idx.
    • Updates get_task.py to use tasks:task_id_agent_id_idx.
    • Updates get_user.py, list_users.py, and patch_user.py to use users:developer_id_metadata_user_id_idx.

This description was created by Ellipsis for 2f88b8c. It will automatically update as commits are pushed.

@creatorrr
Copy link
Contributor

also - it needs two colons, ::index create developers:developer_id_idx { developer_id }

@creatorrr
Copy link
Contributor

creatorrr commented Dec 10, 2024

also these (and we have to be mindful of the order of fields):

::index create executions:execution_id_status_idx { execution_id, status }

::index create executions:execution_id_task_id_idx { execution_id, task_id }

::index create tasks:task_id_agent_id_idx { task_id, agent_id }

::index create agents:agent_id_developer_id_idx { agent_id, developer_id }

::index create sessions:session_id_developer_id_idx { session_id, developer_id }

::index create docs:owner_id_metadata_doc_id_idx { owner_id, metadata, doc_id }

::index create agents:developer_id_metadata_agent_id_idx { developer_id, metadata, agent_id }

::index create users:developer_id_metadata_user_id_idx { developer_id, metadata, user_id }

::index create transitions:execution_id_type_created_at_idx { execution_id, type, created_at }

@whiterabbit1983 whiterabbit1983 marked this pull request as ready for review December 10, 2024 12:52
Comment on lines +175 to +186
# Make sure the current/next targets are valid
match data.type:
case "finish_branch":
pass # TODO: Implement
case "finish" | "error" | "cancelled":
pass

last_transition_type[min_cost(type_created_at)] :=
*transitions {{
execution_id: to_uuid("{str(execution_id)}"),
type,
created_at,
}},
type_created_at = [type, -created_at]
### FIXME: HACK: Fix this and uncomment

matched[collect(last_type)] :=
last_transition_type[data],
last_type_data = first(data),
last_type = if(is_null(last_type_data), "init", last_type_data),
valid_transition[last_type, $next_type]
### assert (
### data.next is None
### ), "Next target must be None for finish/finish_branch/error/cancelled"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Make sure the current/next targets are valid
match data.type:
case "finish_branch":
pass # TODO: Implement
case "finish" | "error" | "cancelled":
pass
last_transition_type[min_cost(type_created_at)] :=
*transitions {{
execution_id: to_uuid("{str(execution_id)}"),
type,
created_at,
}},
type_created_at = [type, -created_at]
### FIXME: HACK: Fix this and uncomment
matched[collect(last_type)] :=
last_transition_type[data],
last_type_data = first(data),
last_type = if(is_null(last_type_data), "init", last_type_data),
valid_transition[last_type, $next_type]
### assert (
### data.next is None
### ), "Next target must be None for finish/finish_branch/error/cancelled"
"""Validates the transition targets based on the transition type.
Args:
data (CreateTransitionRequest): The transition request data to validate.
Raises:
ValueError: If the transition type is invalid.
AssertionError: If the transition targets are not valid for the given type.
"""
match data.type:
case "finish_branch":
pass # TODO: Implement
case "finish" | "error" | "cancelled":
assert (
data.next is None
), "Next target must be None for finish/finish_branch/error/cancelled"

add docstring to validate_transition_targets function

@whiterabbit1983 whiterabbit1983 merged commit 875943c into dev Dec 10, 2024
14 checks passed
@whiterabbit1983 whiterabbit1983 deleted the f/cozo-indices branch December 10, 2024 12:52
Copy link
Contributor

@ellipsis-dev ellipsis-dev bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Looks good to me! Reviewed everything up to 2f88b8c in 36 seconds

More details
  • Looked at 538 lines of code in 13 files
  • Skipped 0 files when reviewing.
  • Skipped posting 7 drafted comments based on config settings.
1. agents-api/agents_api/models/docs/search_docs_by_embedding.py:91
  • Draft comment:
    The PR lacks a description, which makes it difficult to understand the full context and intent of the changes. Please provide a description for better clarity.
  • Reason this comment was not posted:
    Confidence changes required: 50%
    The PR introduces new indices in the database queries, which is a good practice for performance optimization. However, the PR description is missing, which makes it difficult to understand the full context and intent of the changes. The code changes seem consistent with the migration script, but without a description, it's hard to verify if all necessary changes are included.
2. agents-api/agents_api/models/docs/search_docs_by_text.py:90
  • Draft comment:
    The PR lacks a description, which makes it difficult to understand the full context and intent of the changes. Please provide a description for better clarity.
  • Reason this comment was not posted:
    Confidence changes required: 50%
    The PR introduces new indices in the database queries, which is a good practice for performance optimization. However, the PR description is missing, which makes it difficult to understand the full context and intent of the changes. The code changes seem consistent with the migration script, but without a description, it's hard to verify if all necessary changes are included.
3. agents-api/agents_api/models/execution/count_executions.py:42
  • Draft comment:
    The PR lacks a description, which makes it difficult to understand the full context and intent of the changes. Please provide a description for better clarity.
  • Reason this comment was not posted:
    Confidence changes required: 50%
    The PR introduces new indices in the database queries, which is a good practice for performance optimization. However, the PR description is missing, which makes it difficult to understand the full context and intent of the changes. The code changes seem consistent with the migration script, but without a description, it's hard to verify if all necessary changes are included.
4. agents-api/agents_api/models/execution/create_execution_transition.py:90
  • Draft comment:
    The PR lacks a description, which makes it difficult to understand the full context and intent of the changes. Please provide a description for better clarity.
  • Reason this comment was not posted:
    Confidence changes required: 50%
    The PR introduces new indices in the database queries, which is a good practice for performance optimization. However, the PR description is missing, which makes it difficult to understand the full context and intent of the changes. The code changes seem consistent with the migration script, but without a description, it's hard to verify if all necessary changes are included.
5. agents-api/agents_api/models/execution/get_execution.py:56
  • Draft comment:
    The PR lacks a description, which makes it difficult to understand the full context and intent of the changes. Please provide a description for better clarity.
  • Reason this comment was not posted:
    Confidence changes required: 50%
    The PR introduces new indices in the database queries, which is a good practice for performance optimization. However, the PR description is missing, which makes it difficult to understand the full context and intent of the changes. The code changes seem consistent with the migration script, but without a description, it's hard to verify if all necessary changes are included.
6. agents-api/agents_api/models/execution/get_paused_execution_token.py:42
  • Draft comment:
    The PR lacks a description, which makes it difficult to understand the full context and intent of the changes. Please provide a description for better clarity.
  • Reason this comment was not posted:
    Confidence changes required: 50%
    The PR introduces new indices in the database queries, which is a good practice for performance optimization. However, the PR description is missing, which makes it difficult to understand the full context and intent of the changes. The code changes seem consistent with the migration script, but without a description, it's hard to verify if all necessary changes are included.
7. agents-api/agents_api/models/execution/list_execution_transitions.py:38
  • Draft comment:
    The PR lacks a description, which makes it difficult to understand the full context and intent of the changes. Please provide a description for better clarity.
  • Reason this comment was not posted:
    Confidence changes required: 50%
    The PR introduces new indices in the database queries, which is a good practice for performance optimization. However, the PR description is missing, which makes it difficult to understand the full context and intent of the changes. The code changes seem consistent with the migration script, but without a description, it's hard to verify if all necessary changes are included.

Workflow ID: wflow_4qmJnZjnq7t1BR32


You can customize Ellipsis with 👍 / 👎 feedback, review rules, user-specific overrides, quiet mode, and more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants