Handling P1P1 Premier Drafts

[unrealities]

One issue with the current app is that pack one pick one cards are not displayed. This issue is due to insufficient logging. You only get the pack data after the first selection is made.

One possible solution is taking a screenshot of the app and then using OCR to detect card names.

Pillow provides a solution for taking screenshots. However, OCR like pytesseract can require additional downloads or installations.

It isn't realistic to ask a user to install Tesseract to get p1p1 data. We could create an AWS Lambda / Google Cloud Function to handle requests. I would be fine spending a couple of dollars a month to support this, but we would need to ensure quotas and rate limiting to prevent abuse and high costs.

[FiYir]

Cool idea.

Are there any decent standalone OCR Python modules that could be bundled with the app? I did a brief search, and it seems like EasyOCR might be a possibility, although it hasn't received updates in 8 months and currently has 365 unresolved issues.

[unrealities]

I did some more digging.

Quentin Fortier did this two years ago using Azure computer vision and wrote up an article about it here

I used EasyOCR manually to do a trial, and it "worked." I got card names, and they were grouped by card. There were some mistaking commas for semi-colons. It didn't need to install a separate app, but the model was downloaded when it ran.

[FiYir]

I've thought a bit on how this might function and what would be necessary to integrate the feature with the existing code; here's what I've come up with.

If the event is detected and the current pack pick is 0,0, the app will capture a screenshot using PIL and pass it to your OCR method when the 'Refresh' button is pressed. The refresh button already invokes the ArenaScanner draft_data_search method, so you'd simply replace the existing p1p1 methods with your OCR methods. OCR might not be suitable for packs after P1P1, as it could mistake the card pool for cards from the pack.

The app would need to determine if the screenshot is from the P1P1 draft screen, disregarding loading screens and such, so perhaps it could search through the processed data for "Pack 1 / Pick 1"? If it doesn't find that text, it would discard the data.

The code handling the dataset should probably be relocated from log_scanner to a separate module. You could create a class that manages dataset interactions and include a method that accepts a list of card names as an argument and returns the dataset data for those cards. Maybe name the method get_cards_by_name and replace the existing log_scanner method retrieve_card_data with get_cards_by_id? Maybe it's better to create a convert_name_to_id method that takes in a list of names as an argument and returns a list of arena IDs? This would minimize the number of changes that would need to be made to ArenaScanner.

What are your thoughts?

[unrealities]

This makes sense. Having to refresh manually should limit the number of accidental/wasted calls. I'll start making a Google Cloud function that takes an image and returns a list of card names. I can then monitor the usage at the end of the month, and if it's a shocking number, we can disable it or start figuring out a contribution option.

[FiYir]

Also, you don't need to enter an Arena event to test this. You can create a log file with just the event entry, open that log file in the app, open up a video of a YouTuber playing, and just have the app appear over it. The lower image quality might make for a better test.
DraftLog_OTJ_PremierDraft_Example.log

[unrealities]

This is a great idea. I can quickly grab these for testing and not waste gems.

[unrealities]

This is mostly for my own documentation sake, but I just successfully ran a manual trial off the image you posted here and the data I received from Google Vision API was:

["Home\n144\nPack 1 | Pick 1\nVersion 3.18\nFile Data Cards Settings Help\nBlooming Marsh\nCanyon Crab\nRaucous Entertainer\nSpinewoods Armadillo (4\nPRASPERITY CO\nHUMILIATE\nZ\nMTGA Draft 17Lands\nCurrent Draft: OTJ PremierDraft\nData Source: PremierDraft (Top)\nDeck Filter: Auto (All Decks)\nPack 0 / Pick 0\nLand\nNAME\nGIHWR ALSA\nIWD\nMissing Cards\nBlooming Marsh enters the battlefield\ntapped unless you control two or\nfewer other lands.\nAdd\nor\nNAME\nGIHWR ALSA\nIWD\nThis application is not endorsed by 17Lands\nYEONG-HAD HAN\nCreature Crab\n: Canyon Crab gets +2/-2 until end\nof turn.\nAt the beginning of your end step, if\nyou haven't cast a spell from your hand\nthis turn, draw a card, then discard a\ncard.\nCreature Plant Bard\n.: Put a +1/+1 counter on each\ncreature you control that entered the\nbattlefield this turn.\nCreature -Armadillo\nReach, Ward\n1. Discard Spinewoods Armadillo:\nSearch your library for a basic land\ncard or a Desert card, reveal it, put it\nInto your hand, then shuffle. You gain 3\nlife.\nSORCERY\nTarget\nget opponent reveals their hand.\nYou choose a nonland card from it.\nThat player discards that card. Put al\n+1/+1 counter on a creature you\ncontrol.\nIGNATIUS BUDI\n0/5\nFORREST IMEL\n2/2\nIRIS COMPLET\n7/7\nMARK A. NELSON\n(Bridled Bighorn\n(Stagecoach Security\n4*\nDaring Thunder-Thief\n36\nRooftop Assassin\n3*\nMine Raider\nCreature - Sheep Mount\nCreature - Human Soldier\nCreature - Turtle Rogue\nVigilance\nWhenever Bridled Bighorn attacks\nwhile saddled, create a 1/1 white Sheep\ncreature token.\nWhen Stagecoach Security enters the\nbattlefield, creatures you control get\n+1/+1 and gain vigilance until end of\nCreature - Vampire Assassin\nFlash\nFlying, Lifelink\nFlash\nWhen Re\nturn.\nDaring Thunder-Thief enters the\nbattlefield tapped.\nn Rooftop Assassin enters the\nbattlefield, destroy target creature an\nopponent controls that was dealt\nSaddle 2\nPlot\ndamage this turn.\nEDGAR S\u00c1NCHEZ HIDALGO\n3/4\nNINO VECIA\n4/5\nINKOGNIT\n4/4\nJose HERNAIZ\nOutlaws' Fury\nSpinewoods Paladin\n4+\nOasis Gardener\n(3\nForest\nInstant\nCreatures you control get +2/+0 until\nend of turn. If you control an outlaw,\nexile the top card of your library. Until\nthe end of your next turn, you may\nplay that card.\nCreature Human Knight\nTrample\nWhen Spinewoods Paladin enters the\nbattlefield, you gain 3 life.\nPlot\nArtifact Creature\nScarecrow\nWhen Oasis Gardener enters the\nbattlefield, you gain 2 life.\nAdd one mana of any color.\nDIEGO GISBERT\n0:05 / 1:24:26\nKAI CARPENTER\n0:56\n5/4\nKRISTINA CARROLL\nBasic Land - Forest\n2/2\nSERGEY GLUSHARDY\nCreature-Human Rogue\nTrample\nWhen Mine Raider enters the\nbattlefield, if you control another\noutlaw, create a Treasure token.\n2/2\nWARREN MARY\n3/2\n56,850\n27,900\nConfirm Pick\nScroll for details\nDeck\n0/40 Cards", "Home", "144", "Pack", "1", "|", "Pick", "1", "Version", "3.18", "File", "Data", "Cards", "Settings", "Help", "Blooming", "Marsh", "Canyon", "Crab", "Raucous", "Entertainer", "Spinewoods", "Armadillo", "(", "4", "PRASPERITY", "CO", "HUMILIATE", "Z", "MTGA", "Draft", "17Lands", "Current", "Draft", ":", "OTJ", "PremierDraft", "Data", "Source", ":", "PremierDraft", "(", "Top", ")", "Deck", "Filter", ":", "Auto", "(", "All", "Decks", ")", "Pack", "0", "/", "Pick", "0", "Land", "NAME", "GIHWR", "ALSA", "IWD", "Missing", "Cards", "Blooming", "Marsh", "enters", "the", "battlefield", "tapped", "unless", "you", "control", "two", "or", "fewer", "other", "lands", ".", "Add", "or", "NAME", "GIHWR", "ALSA", "IWD", "This", "application", "is", "not", "endorsed", "by", "17Lands", "YEONG", "-", "HAD", "HAN", "Creature", "Crab", ":", "Canyon", "Crab", "gets", "+", "2", "/", "-2", "until", "end", "of", "turn", ".", "At", "the", "beginning", "of", "your", "end", "step", ",", "if", "you", "haven't", "cast", "a", "spell", "from", "your", "hand", "this", "turn", ",", "draw", "a", "card", ",", "then", "discard", "a", "card", ".", "Creature", "Plant", "Bard", ".", ":", "Put", "a", "+", "1", "/", "+", "1", "counter", "on", "each", "creature", "you", "control", "that", "entered", "the", "battlefield", "this", "turn", ".", "Creature", "-Armadillo", "Reach", ",", "Ward", "1.", "Discard", "Spinewoods", "Armadillo", ":", "Search", "your", "library", "for", "a", "basic", "land", "card", "or", "a", "Desert", "card", ",", "reveal", "it", ",", "put", "it", "Into", "your", "hand", ",", "then", "shuffle", ".", "You", "gain", "3", "life", ".", "SORCERY", "Target", "get", "opponent", "reveals", "their", "hand", ".", "You", "choose", "a", "nonland", "card", "from", "it", ".", "That", "player", "discards", "that", "card", ".", "Put", "al", "+", "1", "/", "+", "1", "counter", "on", "a", "creature", "you", "control", ".", "IGNATIUS", "BUDI", "0/5", "FORREST", "IMEL", "2/2", "IRIS", "COMPLET", "7/7", "MARK", "A.", "NELSON", "(", "Bridled", "Bighorn", "(", "Stagecoach", "Security", "4", "*", "Daring", "Thunder", "-", "Thief", "36", "Rooftop", "Assassin", "3", "*", "Mine", "Raider", "Creature", "-", "Sheep", "Mount", "Creature", "-", "Human", "Soldier", "Creature", "-", "Turtle", "Rogue", "Vigilance", "Whenever", "Bridled", "Bighorn", "attacks", "while", "saddled", ",", "create", "a", "1/1", "white", "Sheep", "creature", "token", ".", "When", "Stagecoach", "Security", "enters", "the", "battlefield", ",", "creatures", "you", "control", "get", "+", "1", "/", "+", "1", "and", "gain", "vigilance", "until", "end", "of", "Creature", "-", "Vampire", "Assassin", "Flash", "Flying", ",", "Lifelink", "Flash", "When", "Re", "turn", ".", "Daring", "Thunder", "-", "Thief", "enters", "the", "battlefield", "tapped", ".", "n", "Rooftop", "Assassin", "enters", "the", "battlefield", ",", "destroy", "target", "creature", "an", "opponent", "controls", "that", "was", "dealt", "Saddle", "2", "Plot", "damage", "this", "turn", ".", "EDGAR", "S\u00c1NCHEZ", "HIDALGO", "3/4", "NINO", "VECIA", "4/5", "INKOGNIT", "4/4", "Jose", "HERNAIZ", "Outlaws", "'", "Fury", "Spinewoods", "Paladin", "4+", "Oasis", "Gardener", "(", "3", "Forest", "Instant", "Creatures", "you", "control", "get", "+", "2", "/", "+", "0", "until", "end", "of", "turn", ".", "If", "you", "control", "an", "outlaw", ",", "exile", "the", "top", "card", "of", "your", "library", ".", "Until", "the", "end", "of", "your", "next", "turn", ",", "you", "may", "play", "that", "card", ".", "Creature", "Human", "Knight", "Trample", "When", "Spinewoods", "Paladin", "enters", "the", "battlefield", ",", "you", "gain", "3", "life", ".", "Plot", "Artifact", "Creature", "Scarecrow", "When", "Oasis", "Gardener", "enters", "the", "battlefield", ",", "you", "gain", "2", "life", ".", "Add", "one", "mana", "of", "any", "color", ".", "DIEGO", "GISBERT", "0:05", "/", "1:24:26", "KAI", "CARPENTER", "0:56", "5/4", "KRISTINA", "CARROLL", "Basic", "Land", "-", "Forest", "2/2", "SERGEY", "GLUSHARDY", "Creature", "-", "Human", "Rogue", "Trample", "When", "Mine", "Raider", "enters", "the", "battlefield", ",", "if", "you", "control", "another", "outlaw", ",", "create", "a", "Treasure", "token", ".", "2/2", "WARREN", "MARY", "3/2", "56,850", "27,900", "Confirm", "Pick", "Scroll", "for", "details", "Deck", "0/40", "Cards"]

The first entry looks like what we'd want to look at. It keeps newlines intact. By visually parsing the results, I see the following "cards."

• Blooming Marsh
• Canyon Crab
• Raucous Entertainer
• Spinewoods Armadillo (4
• HUMILIATE
• (Bridled Bighorn
• (Stagecoach Security
• Daring Thunder-Thief
• Rooftop Assassin
• Mine Raider
• Outlaws' Fury
• Spinewoods Paladin
• Oasis Gardener
• Forest

There are obvious issues.

1. Duplicate cards will probably be missed.
2. Cards that mention other cards in their body text may cause false positives.
3. Additional characters may be included at the beginning/ending of lines.
4. Cannot be case-sensitive
5. A lot of lines will need to be ignored. We should only return what we believe to actually be cards.

[unrealities]

The current solution:

The cloud function receives two objects:

1. card_names: List of all possible cards (this way we don't have to update the function and limits the possible results to a specific set)
2. image: base-64 string version of the screenshot

The cloud function then uses the Cloud Vision API to identify all text. It then uses thefuzz to get the best match and returns a list of matching cards.

I have tried a few screengrabs, and it seems to work well, but on a card like Kraum, where it mentions itself in its card body, we get duplicate entries. But this also supports the case where there are duplicates of the card.

It's a pretty simple script: https://github.com/unrealities/mtgalimited/blob/main/functions/main.py

I have it deployed to Google Cloud. Haven't thought about limits or restrictions.

[FiYir]

@unrealities I was thinking that you could replace the existing p1p1 methods in ArenaScanner with a method that calls your OCR API. This minimizes changes to the remaining code.

[FiYir]

I got it working! I combined the code from your 0319 branch with the code above.

[unrealities]

So cool!!!

Did you have to hit the refresh button, or was it auto-generated when the draft started?
Does the data get overwritten adequately after the selection? I know we will have non-perfect results, and I don't want to leave dirty data.

I was impressed at how fast the endpoint is. Under 1s end-to-end.

I also want to discuss what restrictions to impose before releasing it in the wild. Only the first 1000 requests are free each month. I have no idea how many users we have, but 100 users doing ten drafts a month puts us near that cap.

As long as the code is not repeatedly hit by the same draft or outside of the app, I think we're okay. It is $1.50/1000 requests, so I don't mind paying for something like ~ 5000/mo personally.

We can obviously add an API-Key, but that is just going to be exposed in the code, but at least it is some security.

We should also ensure the endpoint is only requested once per draft in the code and at least is not called again after the pack data has been loaded. I don't know if OAuth 2.0 is overkill.

Since the endpoint is separate from the main project, I can also turn it off if the costs are too high. I could also try Azure, which offers 5000 free.

[FiYir]

Did you have to hit the refresh button, or was it auto-generated when the draft started?
Does the data get overwritten adequately after the selection? I know we will have non-perfect results, and I don't want to leave dirty data.

It was automatically generated when I opened the log file. Currently, this new function gets called every time the Player.log is updated, and it continues to be called with each update until either this OCR function returns a pack or the P1P1 entry appears in the log. The data doesn't get overwritten when the P1P1 log entry appears.

We could add a 'source' argument to the draft_data_search method and have the app call the endpoint only when the source is the refresh button.

I also want to discuss what restrictions to impose before releasing it in the wild. Only the first 1000 requests are free each month. I have no idea how many users we have, but 100 users doing ten drafts a month puts us near that cap.

If you check the GitHub API results for your last release, you can see that the asset was downloaded 70+ times, meaning there are at least 70 Windows users using your program. This number will likely exceed 100 once MH3 is released. Additionally, you need to consider people using your endpoint for their own projects, like this person. While we can add restrictions to the program, it's crucial to implement rate-limiting or other server-side restrictions.

[FiYir]

Here are the steps to reproduce the results in this image.

1. Run the script in my branch
2. Open this video to full-screen and select 1080p60 quality
3. Open this log file in the app (File>Open)
4. Wait for the results

[unrealities]

FYI, I included your changes in my 0319 branch. I don't want to merge yet, but you are free to keep using the endpoint on your own however you like for real drafts or testing.

[unrealities]

I got it to run "successfully" locally on my 0319 branch. I had small cards, and it missed a few, but it ran! I don't know why, but it seems to love to mistake cards as "Fling."

[FiYir]

The fuzzy match logic is interpreting "Flying" as "Fling". A solution might be to increase the score threshold for single-word matches. Also, it is unable to recognize the split card "Crime /// Punishment". A solution for split cards might be to have the parser reformat the card name so that "Crime /// Punishment" becomes "Crime\nPunishment\n" and then perform a fuzzy match on that name.

[unrealities]

A solution to this issue has been released in v3.19