Skip to content

End to end local Integration test for parseImage function: See issue #1 #15

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 70 commits into
base: feat/tests
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
40eb170
autoshutdown working; discovery warning
abrichr Mar 25, 2025
d2361dd
just use boto3 for discovery
abrichr Mar 25, 2025
bdfda9a
INACTIVITY_TIMEOUT_MINUTES = 60
abrichr Mar 28, 2025
53844f3
feat(demo): Add MVP for LLM action planning from synthetic UI
abrichr Mar 28, 2025
e95203a
add demo_output/login_screen.png, login_screen_highlighted.png
abrichr Mar 28, 2025
ea652f1
End to end local Integration test for parseImage function: See issue #1
gabrielbugarija Mar 28, 2025
bbd4ecf
feat(demo): Add multi-step planning and simulation
abrichr Mar 29, 2025
4b01bb0
add demo_output_multistep
abrichr Mar 29, 2025
2a936e5
feat(demo): Add dimming and text annotation to highlights
abrichr Mar 29, 2025
aaf7887
Add tests; ci.yml
abrichr Mar 29, 2025
af59011
ruff
abrichr Mar 29, 2025
d8b7bbb
ci.yml: uv venv
abrichr Mar 29, 2025
5c8a004
pyproject.toml: Only include the main package source directory
abrichr Mar 29, 2025
36a1f87
uv add ruff
abrichr Mar 29, 2025
c7f195e
ruff
abrichr Mar 29, 2025
bc1acbc
feat(demo): Add MVP Demo for LLM UI Action Planning
abrichr Mar 29, 2025
41c9230
documentation
abrichr Mar 30, 2025
f15270c
Merge infrastructure: EC2 deploy + auto-shutdown
abrichr Mar 30, 2025
13f4461
Merge tests: Unit and E2E tests for deployment and core
abrichr Mar 30, 2025
e3daa78
wip
abrichr Mar 30, 2025
06b73df
wip
abrichr Mar 30, 2025
0e2c419
fix(omniparser): Correct deployment, auto-shutdown, and client init
abrichr Mar 30, 2025
18632ea
ruff
abrichr Mar 30, 2025
8e58552
ruff
abrichr Mar 30, 2025
ea81d41
chore(tests): Stabilize test collection and skip failing e2e/core tests
abrichr Mar 30, 2025
189ec97
ruff format, ruff check
abrichr Mar 30, 2025
0b77c78
feat: Add demo GIF, generation script, and update README
abrichr Mar 30, 2025
ccf84b6
feat: Add EC2 deployment, auto-shutdown, tests, and CI setup
abrichr Mar 30, 2025
1610d22
update README
abrichr Mar 30, 2025
7baeef5
fix(README): fix regressions from previous merge
abrichr Mar 30, 2025
b2aff85
fix(tests): Correct core tests and consolidate test structure
abrichr Mar 30, 2025
98e4b7b
ruff format
abrichr Mar 30, 2025
9d73757
fix(tests): Mock input controllers in core tests for CI compatibility
abrichr Mar 30, 2025
b75c022
fix(tests): Correct core tests and consolidate test structure
abrichr Mar 30, 2025
b19272f
feat: Add tests for response mapping and integrate into test script
abrichr Mar 31, 2025
ac8a9c2
feat: Add tests for response mapping and integrate into test script
abrichr Mar 31, 2025
70e1b83
feat: Integrate real parsing and mapping via VisualState
abrichr Mar 31, 2025
7692836
feat: Add detailed LLM prompt/completion logging
abrichr Mar 31, 2025
445e8f1
fix(omniparser): Finalize EC2 deployment and auto-shutdown logic
abrichr Mar 31, 2025
fab1119
feat: Add detailed logging for LLM prompts and responses
abrichr Mar 31, 2025
f20c412
refactor: Use OmniParserClient in core, fix tests & consolidate struc…
abrichr Mar 31, 2025
c2d4967
feat(agent): Implement working multi-step calculator demo loop
abrichr Apr 1, 2025
cdaffca
pytest.mark.skipif
abrichr Apr 1, 2025
0196974
conditionally import pynput in input.py
abrichr Apr 1, 2025
8771b47
fix test_omnimcp_core.py
abrichr Apr 1, 2025
1ce041a
remove test_deploy_screenshot.png
abrichr Apr 1, 2025
ecdc363
add demo_synthetic.py
abrichr Apr 1, 2025
edcb55e
replace make_gif.sh with make_gif.py; update README
abrichr Apr 1, 2025
35a2bdc
feat(agent): Implement working multi-step calculator demo loop
abrichr Apr 1, 2025
1776cdf
fix omnimcp_demo.gif
abrichr Apr 1, 2025
174b97b
refactor: Introduce AgentExecutor for core loop orchestration
abrichr Apr 5, 2025
a8c54db
fix(tests): Mock take_screenshot in AgentExecutor tests for CI
abrichr Apr 5, 2025
fd224b0
refactor: Introduce AgentExecutor for core loop orchestration
abrichr Apr 5, 2025
66ff2d9
fix: Resolve mcp dev connection issues and move VisualState
abrichr Apr 5, 2025
0a5a2a9
update README.md
abrichr Apr 5, 2025
8683e59
fix tests
abrichr Apr 5, 2025
e16496d
fix tests
abrichr Apr 5, 2025
65b8d57
fix: Resolve mcp dev connection issues and move VisualState
abrichr Apr 5, 2025
8a58cd6
fix cli.py; save log to run dir; config.LOG_DIR/RUN_OUTPUT_DIR/ANTHRO…
abrichr Apr 5, 2025
2d5b912
fix smoke test
abrichr Apr 5, 2025
1e745b5
delay imports in cli.py; add ci_mode arg
abrichr Apr 5, 2025
0763da3
fix(cli): add ci_mode to cli.py
abrichr Apr 5, 2025
87d3776
feat: Add screenshot downsampling and refine LLM config/calls
abrichr Apr 5, 2025
c716ea9
ruff
abrichr Apr 5, 2025
ed8c77f
add DEBUG_FULL_PROMPTS to .env.example
abrichr Apr 5, 2025
7a562a1
feat: Add screenshot downsampling and refine LLM config/calls
abrichr Apr 5, 2025
7ec6783
feat: Add initial tracking infrastructure and metrics logging
abrichr Apr 6, 2025
6246521
Merge branch 'OpenAdaptAI:main' into main
gabrielbugarija May 29, 2025
2bff659
Delete test_parseImage_local.py
gabrielbugarija Jun 4, 2025
95d6123
Updated end-end parse image testing
gabrielbugarija Jun 4, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 42 additions & 5 deletions .env.example
Original file line number Diff line number Diff line change
@@ -1,8 +1,45 @@
# .env.example

# Copy this file to .env and fill in your AWS credentials

# --- Project ---
PROJECT_NAME=omnimcp

# --- Core LLM Configuration ---
# Required for planning/reasoning steps (Only one provider needed usually)
ANTHROPIC_API_KEY=your_anthropic_api_key
# OPENAI_API_KEY=
# GOOGLE_API_KEY=
AWS_ACCESS_KEY_ID=your_access_key_id
AWS_SECRET_ACCESS_KEY=your_secret_access_key
AWS_REGION=us-east-1
# Not yet supported:
# OPENAI_API_KEY=your_openai_api_key
# GOOGLE_API_KEY=your_google_api_key

# Optional: Specify exact Anthropic model (defaults to "claude-3-7-sonnet-20250219" in config.py)
# ANTHROPIC_DEFAULT_MODEL=claude-3-sonnet-20240229
# ANTHROPIC_DEFAULT_MODEL=claude-3-haiku-20240307

# --- OmniParser Service Configuration ---
# Option 1: Leave blank/commented to use auto-deployment features (requires AWS keys below)
# OMNIPARSER_URL=

# Option 2: Specify URL if running OmniParser manually or elsewhere
# OMNIPARSER_URL=http://<your_parser_ip>:8000

# Optional: Factor (0.1-1.0) to resize screenshot before parsing (lower = faster, less accurate)
# Default is 1.0 (no downsampling). Set to e.g. 0.5 for 50% scaling.
# OMNIPARSER_DOWNSAMPLE_FACTOR=1.0

# --- AWS Credentials (Required ONLY for OmniParser auto-deployment) ---
# AWS_ACCESS_KEY_ID=YOUR_AWS_ACCESS_KEY
# AWS_SECRET_ACCESS_KEY=YOUR_SECRET_KEY
# AWS_REGION=us-east-1 # Optional: Defaults to us-east-1 if not set

# --- Optional AWS EC2 Configuration (Overrides defaults in config.py for auto-deploy) ---
# See config.py for default AMI/Instance Type (currently G6/DLAMI)
# AWS_EC2_INSTANCE_TYPE=g6.xlarge
# AWS_EC2_AMI=ami-xxxxxxxxxxxxxxxxx

# --- Logging ---
# Log level: DEBUG, INFO, WARNING, ERROR, CRITICAL (Default: INFO)
# LOG_LEVEL=INFO
#
# Optional: view full prompts in DEBUG mode
# DEBUG_FULL_PROMPTS=False
54 changes: 54 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
# .github/workflows/ci.yml
name: CI

on:
push:
branches: [ main ]
pull_request:
branches: [ main ]

jobs:
lint_and_test:
runs-on: ubuntu-latest

steps:
# --- 1. Checkout Repository ---
- name: Checkout repository
uses: actions/checkout@v4

# --- 2. Set up Python ---
- name: Set up Python 3.10
uses: actions/setup-python@v5
with:
python-version: '3.10'

# --- 3. Install uv ---
- name: Install uv
run: curl -LsSf https://astral.sh/uv/install.sh | sh
- name: Add uv to PATH
run: echo "$HOME/.cargo/bin" >> $GITHUB_PATH

# --- 4. Create Virtual Environment using uv --- ### ADDED STEP ###
- name: Create virtual environment
run: uv venv

# --- 5. Install Dependencies using uv --- ### Should now work ###
# uv pip install will now detect and use the .venv directory created above
- name: Install dependencies
run: uv pip install -e ".[test]"

# --- 6. Lint and Format Check with Ruff (via uv) ---
- name: Lint with Ruff
run: uv run ruff check .
- name: Check formatting with Ruff
run: uv run ruff format --check .

# --- 7. Run Tests with Pytest (via uv) ---
- name: Run tests
env:
ANTHROPIC_API_KEY: "ci_dummy_key"
run: uv run pytest tests/

# --- 8. Smoke test cli.lpy
- name: Run CLI Smoke Test (--help)
run: uv run python cli.py --help
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,3 +4,6 @@
omnimcp.egg-info/
omnimcp.log
__pycache__
runs/
logs/
images/*/
Loading