bayes_or_spam.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Bayes or Spam? – Spam Filtering with Bayesian Epistemology\n",
    "\n",
    "<img src=\"https://upload.wikimedia.org/wikipedia/commons/d/d0/Google_Colaboratory_SVG_Logo.svg\" height=\"16\"> [Run in Google Colab](https://colab.research.google.com/github/tbitai/bayes-or-spam/blob/main/bayes_or_spam.ipynb)\n",
    "&nbsp;\n",
    "<img src=\"https://huggingface.co/datasets/huggingface/brand-assets/resolve/main/hf-logo.svg\" height=\"16\"> [Demo on Hugging Face](https://huggingface.co/spaces/tbitai/bayes-or-spam)\n",
    "&nbsp;\n",
    "<img src=\"https://raw.githubusercontent.com/primer/octicons/refs/heads/main/icons/mark-github-16.svg\" height=\"16\"> [View source on GitHub](https://github.com/tbitai/bayes-or-spam)\n",
    "\n",
    "This is an analysis of Bayesian spam filtering from the viewpoint of Bayesian epistemology. We review the Bayesian spam filtering algorithm, point out where it deviates from the principles of Bayesian epistemology, and present test results showing that the \"principled\" Bayesian spam filtering is also more effective. Finally, we compare Bayesian filtering to neural networks, with mixed results: neural models perform better, but at a significantly higher computational cost. These technical results provide epistemological arguments for Bayesianism.\n",
    "\n",
    "### Table of contents\n",
    "\n",
    "[Introduction](#scrollTo=Introduction)\n",
    "\n",
    "[Preparing the test dataset](#scrollTo=Preparing_the_test_dataset)\n",
    "\n",
    "[Graham's algorithm](#scrollTo=Graham_s_algorithm)\n",
    "\n",
    "[Correcting Graham's bias](#scrollTo=Correcting_Graham_s_bias)\n",
    "\n",
    "[Tuning the interestingness threshold](#scrollTo=Tuning_the_interestingness_threshold)\n",
    "\n",
    "[Neural networks](#scrollTo=Neural_networks)\n",
    "\n",
    "[Large language models](#scrollTo=Large_language_models)\n",
    "\n",
    "[Epistemological conclusion](#scrollTo=Epistemological_conclusion)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "Bayesian spam filtering is one of the leading technologies of tackling spam. In his 2002 essay, [*A Plan for Spam*](http://paulgraham.com/spam.html), Paul Graham described the first effective Bayesian spam filtering algorithm. Today's Bayesian spam filters use essentially the same algorithm.\n",
    "\n",
    "Graham's motivation was that his earlier spam filters looking for individual spam features (e.g. does it contain `\"click\"`?) were not effective enough. As he writes, Bayesian spam filters recognized more features, e.g. `\"ff0000\"` (red's hexa code used in HTML)."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Preparing the test dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import google.generativeai as genai\n",
    "from google.generativeai.types import HarmCategory, HarmBlockThreshold\n",
    "import numpy as np\n",
    "import matplotlib as mpl\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.ticker as mtick\n",
    "import itertools\n",
    "import re\n",
    "import string\n",
    "import sys\n",
    "import io\n",
    "import os\n",
    "import time"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "We're going to use the [Enron spam dataset collection](http://www.aueb.gr/users/ion/data/enron-spam/index.html)'s `Enron1` dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "ds_url = 'file://' + os.path.sep.join([os.path.abspath(os.path.curdir), 'data', 'enron1.tar.gz'])\n",
    "ds_path = tf.keras.utils.get_file(origin=ds_url, untar=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 5172 files belonging to 2 classes.\n",
      "Using 4655 files for training.\n",
      "0: ham\n",
      "1: spam\n",
      "b'Subject: meter variances - ua 4 clean - up\\r\\ndaren / vance -\\r\\nthe two meters below are new and have unallocatable flow . . . . . . . i will need a\\r\\npurchase for each of them . please respond with a deal number , or further\\r\\nsuggestions for resolution so that i can clear this up as soon as possible .\\r\\nmary\\r\\nenron on 09 / 19 / 2000 08 : 44 am - - - - - - - - - - - - - - - - - - - - - - - - - - -\\r\\nkatherine benedict @ ect\\r\\n09 / 18 / 2000 04 : 53 pm\\r\\nto : mary poorman / na / enron @ enron , fred boas / hou / ect @ ect , aimee\\r\\nlannou / hou / ect @ ect\\r\\ncc :\\r\\nsubject : meter variances - ua 4 clean - up\\r\\nhi guys ,\\r\\nplease take a look at the following meters which are showing up with\\r\\nvariances on my ua 4 report . once the variance has been cleared , please send\\r\\nme an e - mail . please try to clear these within the next couple of days .\\r\\nthanks ,\\r\\nkathy benedict\\r\\nmeter number : 9851\\r\\nvariance : 11 , 204\\r\\nmeter type : daily swing\\r\\nproduction month : august 2000\\r\\nlogistics contact : none\\r\\nvolume mgmt contact : anita luong\\r\\nmeter number : 9852\\r\\nvariance 568\\r\\nmeter type : daily swing\\r\\nproduction month : august 2000\\r\\nlogistics contact : none\\r\\nvolume mgmt contact : anita luong'\n",
      "0\n",
      "b'Subject: revision # 1 - hpl noms for november 3 , 2000\\r\\n( see attached file : hplnl 103 . xls )\\r\\n- hplnl 103 . xls'\n",
      "0\n",
      "b'Subject: txu fuels / sds nomination for april 2001\\r\\nattached is the april 2001 nomination for our takes under the gas sales and\\r\\npurchase contract between txu fuel company ( previously tufco ) and sds . please\\r\\nadvise should you have any questions concerning the attached .\\r\\nthanks ,\\r\\nccs\\r\\n( see attached file : sdsnom . xls )\\r\\n- sdsnom . xls'\n",
      "0\n",
      "b\"Subject: cleburne\\r\\ndaren & john ,\\r\\nsee the attached document regarding gas deliveries / nominations for the\\r\\ncleburne site . right now , i ' m planning to be in the houston office on\\r\\nthursday , august 3 , 2000 so we may be able to meet then . if there are any\\r\\nquestions , please call me for additional info .\\r\\nmichael mazowita\\r\\nphone : ( 248 ) 442 - 4823\\r\\ncell : ( 248 ) 345 - 4002\"\n",
      "0\n",
      "b\"Subject: rescheduling of usecase analysis for enom mtg .\\r\\nplease let me know if you would be available for the above mtg on tuesday ,\\r\\njan . 25 th from 1 : 00 - 4 : 00 asap .\\r\\nthanks ,\\r\\nmary\\r\\n- - - - - - - - - - - - - - - - - - - - - - forwarded by mary weatherstone / hou / ect on 01 / 21 / 2000\\r\\n11 : 42 am - - - - - - - - - - - - - - - - - - - - - - - - - - -\\r\\nmary weatherstone on 01 / 21 / 2000 11 : 30 : 15 am\\r\\nto : inja chun / hou / ect @ ect , paul f poellinger / hou / ect @ ect , dave\\r\\nnommensen / hou / ect @ ect , kevin montagne / hou / ect @ ect , michael\\r\\nguadarrama / hou / ect @ ect , robert superty / hou / ect @ ect , randall l\\r\\ngay / hou / ect @ ect , george smith / hou / ect @ ect , katherine l kelly / hou / ect @ ect ,\\r\\nedward terry / hou / ect @ ect , daren j farmer / hou / ect @ ect\\r\\ncc : jay webb / hou / ect @ ect , heather choate / hou / ect @ ect , yvette g\\r\\nconnevey / corp / enron @ enron , cindy cicchetti / hou / ect @ ect\\r\\nsubject : usecase analysis for enom mtg . - cancelled\\r\\nthe above referenced meeting that was scheduled for today at 2 : 00 - 5 : 00 in\\r\\neb 2601 a has been cancelled . i will contact everyone for a time to reschedule\\r\\nnext week .\\r\\nthanks ,\\r\\nmary\\r\\n- - - - - - - - - - - - - - - - - - - - - - forwarded by mary weatherstone / hou / ect on 01 / 21 / 2000\\r\\n11 : 27 am - - - - - - - - - - - - - - - - - - - - - - - - - - -\\r\\nmary weatherstone on 01 / 18 / 2000 05 : 34 : 25 pm\\r\\nto : inja chun / hou / ect @ ect , paul f poellinger / hou / ect @ ect , dave\\r\\nnommensen / hou / ect @ ect , kevin montagne / hou / ect @ ect , michael\\r\\nguadarrama / hou / ect @ ect , robert superty / hou / ect @ ect , randall l\\r\\ngay / hou / ect @ ect , george smith / hou / ect @ ect , katherine l kelly / hou / ect @ ect ,\\r\\nedward terry / hou / ect @ ect , daren j farmer / hou / ect @ ect\\r\\ncc : jay webb / hou / ect @ ect , heather choate / hou / ect @ ect , yvette g\\r\\nconnevey / corp / enron @ enron , cindy cicchetti / hou / ect @ ect\\r\\nsubject : usecase analysis for enom mtg .\\r\\nper jay ' s request the above referenced meeting has been scheduled for friday ,\\r\\njanuary 21 st at 2 : 00 - 5 : 00 in eb 2601 a .\\r\\nplease plan on attending . if you cannot attend , please let me know asap .\\r\\nthanks ,\\r\\nmary\"\n",
      "0\n",
      "b\"Subject: 54 - cy . . . adrian hideout\\r\\nor ( der the best mens products today\\r\\ncialis\\r\\n-\\r\\n( super\\r\\nviagra )\\r\\nvia ( gra\\r\\nvia ( gra\\r\\nsoft\\r\\ntabs\\r\\nlevitra\\r\\nsa ( ve u ( p t ( o 70 %\\r\\nwe ship to all us states and worldwide\\r\\ngenuine lo ( w pri ( ces , secure or ( der fo ( rm\\r\\ndiscreet billing and packaging\\r\\nother products\\r\\navailable\\r\\nambien - xanax\\r\\nvalium\\r\\n- m e r i d i a\\r\\nlipi tor - x e n i c a\\r\\nl\\r\\nand many other meds . . .\\r\\nm o r e i n\\r\\nf o h e r e\\r\\nto say adios muchachos head on over to :\\r\\nwww . rainbowboatl . com / d / uout . php\\r\\ndeem prefab wy downey ann weatherproof cadillac paraffin chairmen raven agricola belate applicate assassinate duplex caveman bayport semitic sulfide ambrose paralinguistic megabit difficult agate dreary alliance\\r\\nmirfak ladle diagonal codfish khan osborne parimutuel colloquy fresnel iq kurt ascend act lyman explain mammoth chigger alabama meier siva historic bolshevism diverse circumscribe aarhus chivalry askew towhee soon betatron imprison whimper arlen mulish scholastic gnp piper mezzo\\r\\nsymbiotic slick maximal lynn soothsay selectmen female incorrect ascription checkbook shopkeep crinkle ian austin expressible vernier resistor folksy vassal matrices roar where ' d consist crosswort d gilmore fiftieth cowgirl jaeger monetarist spheroidal mallow afterthought climax cognizant gerard caucasian sensorimotor menelaus gunflint lumpish theft abstractor andover featherweight cab walls they ' ll claudio hypodermic jujube\\r\\ncrossover sixtieth contraception convince cryostat rex whomsoever chose belief panacea adsorption clothesline afraid elinor hershel jockstrap numerology gonzales clothier cooke alberta lessee\\r\\nexxon flout rhodium anchovy toggle granny anneal derisive curran failure broomcorn errand elijah authenticate giddap hazelnut vellum denominate counterfeit haystack fastidious cognate pertain repulsion swarthmore hostess cecropia commiserate greater airmass debbie auditorium reinforce chippendale droopy centric wyoming horsefly dialup allotting burst combat heroism asylum allot cardioid burglarproof cobra bauble embraceable uphill seminar spellbound helmsman mockup airy coachmen deprive disjunct lest dropout laurie\\r\\nnumeral beatitude throw touchstone ox degradation curl arithmetic curb bye plank aborning phycomycetes finland carbone indomitable ornament armata transmitter annuli waylaid courtier mirfak brasilia baron berman spoonful coddle adsorptive charlemagne vengeful hewlett bucolic scorpion bondage coexistent connors decker jealous cahoot mutual oblige pyroxenite arrack navajo vociferous decision hut welsh salaried choreograph acorn rod cowl airframe bookplate recession\\r\\nhendrickson impregnate activation contagious versus ross olefin mountainside swain skill distribution myocardial eyesight embroider probity cogent decontrol christen automorphism blinn cabbage bicentennial cairo tug hub mba chug towhee dying magnet dietrich aeronautic nasturtium anarchic dysentery cannibal diamond byte discrepant conquistador chokeberry divulge needlepoint arachnid foundry critic urchin beebe conformation\\r\\nbrooke sutton freehold cacophony anterior proclaim deficit banana sophie gush swordplay sierra chess cross character boast involution edith hyperbolic eyeball league millikan placebo gingham ackley earsplitting ichneumon prevent danish apposition blinn bulrush melancholy minnesota lunar gingham downfall definitive cavilling regimen are trod increase responsible coors intestinal aquarius incubi budapest albany testicular jeff impression steely etruscan\\r\\nmadame orthogonal alto sludge scriptural worthington charlottesville custodial bryant ballard conformance pyre cheshire subvert bianco lacerta baptiste sperm coastline conjunct special compagnie cajole sago merciful insatiable canberra bid cenozoic flashlight arclength vestige blanchard selma inculcate beneficiary bootlegged stain governance betsy e ' er perspire liqueur dossier coeditor choreography contractual regiment oscillatory thoroughgoing contrary correlate fantastic aureomycin binghamton venerate provocateur raven steelmake aniseikonic\\r\\nbeckon addison antiperspirant audubon liniment mutter memorable caribbean snarl durer hale gauche britannica discriminant cobalt mcallister artery wallboard bridge hedonism crock educable megavolt colt millard lehman midpoint strata mingle\\r\\nallegiant cypriot marrow yukon pearlstone incant apocalypse bulldoze proline thrust est hygiene acrimony lanthanide ascendant aides chiffon crossword slope beacon atlantes curious wheatstone beep minstrel stopwatch produce handwrite diane salvation fenton charity cartilage bimini buoyant febrile marseilles abram tyler hurley dialectic\\r\\nconvolve insufferable riotous theism zachary dried coates geoduck soy neutron purge flaunt ambient brevet stirrup marin telephone disputant intrigue cardinal harem collapsible twiddle salk pithy insurrection nothing gleason goofy minnesota phobic yaw marietta highball here wingmen astrophysicist lenore straggle fireman british grosbeak charlie coven graves cuddly gorse electrolytic victual moderate germanium audrey act\\r\\nstopover tapeworm bremsstrahlung baffle henchman grieve heavyweight photo scarlatti churchgoing candide terminate statesman registry tour atheist ferocity pear schooner slough whitlock doubleday cylinder emblazon ababa cliffhang alderman spartan piccolo squeal aniline hager mnemonic canterelle absent bible schumann papacy plutonium coot beguile creosote delphic ackley\\r\\nductile chortle ribonucleic expressive arcsine casework sale mantlepiece turnout cliff beachcomb sunspot crush transduction compensate tread draftsperson pivot mustachio consist alabama osmotic johnny pantomimic cartilage boom angeles conley stereography fat concrete discussion daylight foot barberry\\r\\ncolosseum albania cornfield shark stuck sock plagiarist crabmeat crispin mccallum oases chocolate prone aruba whop capitol grosbeak interpretation thai antietam gumshoe pathogen cavilling contradistinct proteolysis aegean antisemitic selenate toil slip chin lavender pegging bayonet fossiliferous nne khaki alone muncie bitterroot manometer dab radium imagery gentle lessee voltaire dreamt secretary ammo catchup adjoin\\r\\ndesperate bangor inhere corpse alias befitting employer augment judicature bedevil boo schuyler hexagonal epileptic hornet potable sham bawl erodible catharsis bouffant scion avenge mental inattention lumbermen technology beryl diety areawide avis avocado certificate inhibition nectarine amoral banbury blank facile smooch become wheat bittersweet hardtack oat alyssum rochester sanitarium law umbilical nanking characteristic effusive prosodic usage delinquent catenate spay inequivalent\\r\\ninfrequent barry beadle dominant tunis sleet hayward degassing beaver motivate bernstein emile baronet panty america accede cobweb dervish discretion forever cerulean heath pinehurst duopolist melbourne ambrose vase lehman paperbound lipton caveman yellowish booklet rna t centigrade muse debar selenium grimaldi cybernetics fledge tasteful airway apartheid die squeegee dixie mad trenton registrable billiard\\r\\nprotestation seahorse honoraria abstractor cousin calliope cutback freight crackle priam supra newsman polarogram m ' s thrush forbore dyspeptic mawr faithful topeka brothel cybernetic alveolus immortal coffman coworker significant stress announce bowfin monongahela colombo cookbook carnival redden bel bunkmate tiffany compline courthouse flanagan boastful\\r\\ncontradistinction cylinder kaskaskia dissociable ridden concession chuff chimique indispensable very west centum stump volkswagen blanchard yon delicious mete tighten opel chard marigold chortle bilinear drone garb jigging profane click euphemist corpsman biaxial counterpoise copernican airy shag preface morbid contributory diethylstilbestrol chart exhaustion possess dandy bodybuild versus aesthetic incarcerate wan sunglasses montpelier vortices\\r\\nallergy determinate tulip corinth acumen perfidious fabricate carolingian basketball bun fischer mr colander dodecahedra effluvia monterey econometric inclination dodecahedral dickens bobbie colette dieldrin kauffman bangladesh oedipal lay cession tactic lorelei circumspect directorate monastery bogy rubble johannes reprehensible vaduz bean adoption hatfield chemise affirmative clockwatcher dysprosium mcknight gigantic\\r\\nmainline echelon boast retiree forum oldster cue boule agenda conquer nomadic acerbic testes focussed egan iran buzzword faustian darwinian perforce animadvert familiar taskmaster consultation kalmia ferric bed hanlon contrition engine postorder clothesbrush connoisseur connive camelot artichoke contagious lola switch stephanie cyrus filth referent dublin photometry\\r\\nbette ocular spear anecdote judith monarchy drift ainu comet elijah pent competent ectopic clad damsel firebug contretemps cannonball noel bevy turtle bakery debarring maxwellian becloud knobby typescript confidante aires bilharziasis simpleminded edgy manfred snoop mischievous technic bateman veal jet intelligentsia eclogue conscious bravado collage knead jobholder antigorite abdominal\\r\\ncit un bradshaw melville troubador classify crystal festive dope celebes fee czerniak sourberry raster perry efface malton elgin arpa generous arkansan colossus drunk dashboard robbin cynic prognosis directory ytterbium al diary cram embroidery speedy inelastic rebelled gordian bagley dereference forrest ilyushin achromatic herbert beman vigil bramble madden zellerbach botulism invoice ridgepole bon produce lancashire scopic warn abject squabble colloquy schedule divorce cadenza dicotyledon\\r\\narturo ninetieth beetle summary divine java cramer geminate transpacific excisable footpath braid codeword axis demitting applicant champagne bedridden masochism audio heathenish mat shove shrove postprocess sleepy altair cocoon archbishop flaky doubleton xerox boris verbiage calculate courtesy novice catalina black dallas ursula ainu catapult pygmy alimony aubrey contagion hackney bang\\r\\ninjudicious quadrille aggressive mortgagor celtic quarryman abel icosahedral stroke drive ambiance boeing base colossus felice hedgehog aberdeen mobster senegal cursor blandish capacity betony gown awoke tuneful cringe brassy aerodynamic demarcate leigh kirkland pizzicato\\r\\nathlete intemperance cargill loam help liqueur sloop cationic gymnosperm controversial incite vaughan brunswick school hey advert headquarters rustic handwaving lawbreaker unary conceive predisposition clergyman horseback narrow hassle cossack indelicate tragedian hydrofluoric shuffle conclude vichy abrogate anna quantitative alveolar douce felicitous crankcase maine builtin bellman swarthmore doll\\r\\nextenuate salmonberry orographic transposition dec she honesty retentive desert foggy bolshevist credo concubine tarbell dogbane dairymen harrow busboy shrapnel yacht cloak acrimony obviate impropriety paramount patrick polypropylene inhibit lentil julep deflater barge hive intercom fee nobel incise ravine publication thallophyte automatic blush chairwomen orography situs alterate charon compulsory controversial macdonald centerpiece kowloon parsley sleek cloak beauty frigga commutate gershwin strategy colette prurient cassius mill emil wholly exorcise lounge pathetic yip\"\n",
      "1\n",
      "b\"Subject: submision result\\r\\nhave you heard ?\\r\\nour home l o an department has gone mad !\\r\\nthey ' ve just reduced our m o r tga ges to 4 . 95 % fixed !\\r\\nez cred i t , ba n kruptcys and c r e dit problems are\\r\\nour specialty , let us help you .\\r\\nhttp : / / www . curdom . com /\\r\\nwilda yeager\\r\\nthe tony on our physique departure\\r\\nso of nautoclave endurance\\r\\nany revere be of and deprave\\r\\nchromatography incontrovertible no and crystal\\r\\n\"\n",
      "1\n",
      "b'Subject: online pahrma . cy you want\\r\\nwelcome to health suite ! chek it out !\\r\\niagra vic\\r\\nalis val\\r\\nx\\r\\nv\\r\\nodin ci\\r\\nium xana\\r\\nwe are the only store which gives this great deal\\r\\nto you .\\r\\nsave your health and\\r\\nmoney !'\n",
      "1\n",
      "b'Subject: dell pentium 4 2 . 8 ghz system\\r\\n$ 585 . 00\\r\\nthis dell system features a powerful\\r\\ncombination with the intel pentium 4 2 . 8 ghz processor\\r\\nand 256 mb ram . a large 40 gb hard disk drive plays host\\r\\nto microsoft windows xp home edition . a cd - rw drive ,\\r\\ndvd - rom drive and a floppy disk drive are all\\r\\npre - installed . integrated graphics and integrated audio\\r\\nare included to get you up and running\\r\\n.\\r\\nprocessor\\r\\nram\\r\\nhdd\\r\\ndrive\\r\\no / s\\r\\nothers\\r\\np 4 2 . 8 ghz\\r\\n256 mb\\r\\n40 gb\\r\\ndvd / cdrw + fdd\\r\\nwinxp home\\r\\nvga + sound\\r\\nvisit : http : / / www . computron - me . com for deals !\\r\\nyour one stop\\r\\ndistributorjebel ali duty free zonewww . computron - me . com\\r\\nfor latest clearance sale listing contact our\\r\\nsales department .\\r\\nonly limited quantities available on selected\\r\\nspecials ! ! ! !\\r\\nif you have any\\r\\ncomplaints / suggestions contact customerservice @ computron - me . com\\r\\ncompaq\\r\\nhewlett packard\\r\\n3 com\\r\\ndell\\r\\nintel\\r\\niomega\\r\\nepson\\r\\naopen\\r\\ncreative\\r\\ntoshiba\\r\\napc\\r\\ncisco\\r\\nus\\r\\nrobotics\\r\\nmicrosoft\\r\\ncanon\\r\\nintellinet\\r\\ntargus\\r\\nviewsonic\\r\\nibm\\r\\nsony\\r\\n- - - - - - - and lots more\\r\\n! ! !\\r\\ntel + 971 4 8834464\\r\\nall prices in u . s . dollars , ex - works ,\\r\\nfax + 971 4 8834454\\r\\njebel ali duty free zone\\r\\nwww . computron - me . com\\r\\nprices and availability subject to change\\r\\nusa - canada u . a . e .\\r\\nwithout\\r\\nnotice .\\r\\nto receive our special offers\\r\\nin plain\\r\\ntext format reply to this\\r\\nmail with the request * for export only\\r\\n*\\r\\nthis\\r\\nemail can not be considered spam as long as we include : contact\\r\\ninformation remove instructions . this message is intended for dealer\\r\\nand resellers only . if you have somehow gotten on this list in error , or\\r\\nfor any other reason would like to be removed , please reply with \" remove\\r\\n\" in the subject line of your message . this message is being sent to you\\r\\nin compliance with the federal legislation for commercial e - mail\\r\\n( h . r . 4176 - section 101 paragraph ( e ) ( 1 ) ( a ) and bill s . 1618 title iii\\r\\npassed by the 105 th u . s . congress .\\r\\nall logos and\\r\\ntrademarks are the property of their respective ownerstoshiba for export\\r\\nonly\\r\\nproducts may not be exactly as shown\\r\\nabove\\r\\n- -\\r\\nto unsubscribe from : computron 8 , just follow this link :\\r\\nclick the link , or copy and paste the address into your browser .'\n",
      "1\n",
      "b'Subject: d - link airplus g 802 . 11 g wireless router $ 87 . 00\\r\\n$ 87 . 00\\r\\nhigh speed\\r\\n( 802 . 11 g ) , 4 port\\r\\nwireless router\\r\\n54 mbps\\r\\n( 802 . 11 g ) , 4 port wireless routerdi - 524\\r\\nairplus g 802 . 11 g wireless\\r\\nrouter\\r\\nvisit : http : / / www . computron - me . com for deals !\\r\\nd - link airplus g\\r\\n802 . 11 g wireless router\\r\\nd - link , the industry leader in wireless\\r\\nnetworking , introduces another breakthrough in wireless\\r\\nconnectivity . the d - link\\r\\nairplus g di - 524 wireless router which is capable\\r\\nof transferring data at rates of up to\\r\\n54 mbps in the 2 . 4 ghz frequency - the same\\r\\nwireless frequency as 802 . 11 b . the d - link di - 524\\r\\nwireless router also offers four ethernet ports to\\r\\nsupport multiple computers . the advanced wireless\\r\\ntechnology built into the di - 524 wireless router offers\\r\\ndata transfer speeds up to 54 mbps through its wireless\\r\\nchannels allowing streaming videos and other high\\r\\nbandwidth applications , such as online gaming events , to\\r\\noperate without the hassle of ethernet cables . the\\r\\nability to use high bandwidth applications also makes\\r\\nstreaming real - time programs more enjoyable and more\\r\\nefficient . with the di - 524 wireless router \\' s built - in\\r\\nadvanced firewall , threats of hackers penetrating your\\r\\nnetwork are minimized .\\r\\ngeneral features :\\r\\n- 4 - port wireless\\r\\nrouter - 54 mbps maximum data transfer rate - web\\r\\nbased device management - dhcp server and client\\r\\ndevice management - 64 / 128 - bit wep encryption - up\\r\\nto 328 - feet\\r\\nrange\\r\\nyour one stop\\r\\ndistributorjebel ali duty free zonedubai , uae . www . computron - me . com\\r\\nfor latest clearance sale listing contact our\\r\\nsales department .\\r\\nonly limited quantities available on selected\\r\\nspecials ! ! ! !\\r\\nfor further details please send\\r\\nyour enquiries to : dealers @ emirates . net . aeor contact via www . computron - me . com\\r\\ncompaq\\r\\nhewlett packard\\r\\n3 com\\r\\ndell\\r\\nintel\\r\\niomega\\r\\nepson\\r\\naopen\\r\\ncreative\\r\\ntoshiba\\r\\napc\\r\\ncisco\\r\\nus\\r\\nrobotics\\r\\nmicrosoft\\r\\ncanon\\r\\nintellinet\\r\\ntargus\\r\\nviewsonic\\r\\nibm\\r\\nsony\\r\\n- - - - - - - and lots more\\r\\n! ! !\\r\\nif you have any\\r\\ncomplaints / suggestions contact : customerservice @ computron - me . com\\r\\ntel + 971 4\\r\\n8834464\\r\\nall prices in u . s . dollars , ex - works ,\\r\\nfax + 971 4\\r\\n8834454\\r\\njebel ali duty free zone\\r\\nwww . computron - me . com\\r\\nprices and availability subject to change\\r\\nusa -\\r\\ncanada u . a . e .\\r\\nwithout\\r\\nnotice .\\r\\nto receive our special offers\\r\\nin plain\\r\\ntext format reply to this\\r\\nmail with the request * for\\r\\nexport only *\\r\\nthis\\r\\nemail can not be considered spam as long as we include : contact\\r\\ninformation remove instructions . this message is intended for dealer\\r\\nand resellers only . if you have somehow gotten on this list in error , or\\r\\nfor any other reason would like to be removed , please reply with \" remove\\r\\n\" in the subject line of your message . this message is being sent to you\\r\\nin compliance with the federal legislation for commercial e - mail\\r\\n( h . r . 4176 - section 101 paragraph ( e ) ( 1 ) ( a ) and bill s . 1618 title iii\\r\\npassed by the 105 th u . s . congress .\\r\\nall logos and\\r\\ntrademarks are the property of their respective\\r\\nowners\\r\\nproducts may not be exactly as shown\\r\\nabove\\r\\n- -\\r\\nto unsubscribe from : computron 3 , just follow this link :\\r\\nclick the link , or copy and paste the address into your browser .\\r\\nplease give it atleast 48 hours for unsubscription to be effective .'\n",
      "1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-06-17 16:18:08.585425: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA\n",
      "To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2022-06-17 16:18:08.586080: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: \n"
     ]
    }
   ],
   "source": [
    "seed = 42\n",
    "ds_base_kwargs = dict(directory=ds_path, seed=seed, validation_split=0.1)\n",
    "ds_train = tf.keras.utils.text_dataset_from_directory(**ds_base_kwargs, \n",
    "                                                      subset='training')\n",
    "\n",
    "for i, label in enumerate(ds_train.class_names):\n",
    "  print(f'{i}: {label}')\n",
    "for text, label in ds_train.unbatch().take(5).concatenate(\n",
    "                   ds_train.unbatch().filter(lambda _, l: l == 1).take(5)):\n",
    "  print(text.numpy())\n",
    "  print(label.numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "enron hello\n"
     ]
    }
   ],
   "source": [
    "len_subj = len('Subject: ')\n",
    "\n",
    "def clean(text):\n",
    "    return text[len_subj:]\n",
    "\n",
    "print(clean('Subject: enron hello'))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Graham's algorithm\n",
    "\n",
    "Here's an implementation of Graham's algorithm using Python and the TensorFlow machine learning package:\n",
    "\n",
    "Graham used a simple tokenization he came up with, but he admits that it can improved. Let's use TensorFlow's `tf.keras.preprocessing.text.text_to_word_sequence`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'Subject: eastrans nomination change effective 9 / 8 / 00\\r\\nplease decrease deliveries to eastrans to 0 mmbtu / dy for 9 / 8 / 00\\r\\nthe redeliveries will be 0 as well .\\r\\nthe over - delivery quantity for the fuels cotton valley gas will be valued per\\r\\nparagraph 3 of the contract price ( per mmbtu ) section in the transaction\\r\\nagreement .'\n",
      "['eastrans', 'nomination', 'change', 'effective', '9', '8', '00', 'please', 'decrease', 'deliveries', 'to', 'eastrans', 'to', '0', 'mmbtu', 'dy', 'for', '9', '8', '00', 'the', 'redeliveries', 'will', 'be', '0', 'as', 'well', 'the', 'over', 'delivery', 'quantity', 'for', 'the', 'fuels', 'cotton', 'valley', 'gas', 'will', 'be', 'valued', 'per', 'paragraph', '3', 'of', 'the', 'contract', 'price', 'per', 'mmbtu', 'section', 'in', 'the', 'transaction', 'agreement']\n",
      "b'Subject: eastrans nomination effective 5 / 1 / 01\\r\\neffective 5 / 1 / 01 , the deliveries and redeliveries will be 0 mmbtu / dy .\\r\\nthe transaction agreement will not be utilized .'\n",
      "['eastrans', 'nomination', 'effective', '5', '1', '01', 'effective', '5', '1', '01', 'the', 'deliveries', 'and', 'redeliveries', 'will', 'be', '0', 'mmbtu', 'dy', 'the', 'transaction', 'agreement', 'will', 'not', 'be', 'utilized']\n",
      "b'Subject: exxonmobil beaumont refinery fuel needs\\r\\nfyi - you will need to enter this into sitara - remember , first 10 , 000 / d is\\r\\neog transport .\\r\\n- - - - - - - - - - - - - - - - - - - - - - forwarded by lee l papayoti / hou / ect on 03 / 27 / 2000\\r\\n10 : 56 am - - - - - - - - - - - - - - - - - - - - - - - - - - -\\r\\n\" brian e nichols \" on 03 / 24 / 2000 02 : 20 : 27 pm\\r\\nto : lpapayo @ ect . enron . com\\r\\ncc : ( bcc : lee l papayoti / hou / ect )\\r\\nsubject : exxonmobil beaumont refinery fuel needs\\r\\nlee ,\\r\\ni did my fuel gas estimate and think that we will need 120 - 125 bbtu / day of\\r\\nfuel\\r\\ngas for the month of april . on a side note , we will bring down our gas\\r\\nturbine\\r\\ngenerator on sunday morning . this will bring our high pressure gas\\r\\nrequirements\\r\\ndown to essentially zero ( it will be zero if our hydrogen plant stays down )\\r\\nand\\r\\nthe demand will transfer to our low pressure gas . we will be in this\\r\\narrangement for approximately 20 days .\\r\\nbrian'\n",
      "['exxonmobil', 'beaumont', 'refinery', 'fuel', 'needs', 'fyi', 'you', 'will', 'need', 'to', 'enter', 'this', 'into', 'sitara', 'remember', 'first', '10', '000', 'd', 'is', 'eog', 'transport', 'forwarded', 'by', 'lee', 'l', 'papayoti', 'hou', 'ect', 'on', '03', '27', '2000', '10', '56', 'am', 'brian', 'e', 'nichols', 'on', '03', '24', '2000', '02', '20', '27', 'pm', 'to', 'lpapayo', 'ect', 'enron', 'com', 'cc', 'bcc', 'lee', 'l', 'papayoti', 'hou', 'ect', 'subject', 'exxonmobil', 'beaumont', 'refinery', 'fuel', 'needs', 'lee', 'i', 'did', 'my', 'fuel', 'gas', 'estimate', 'and', 'think', 'that', 'we', 'will', 'need', '120', '125', 'bbtu', 'day', 'of', 'fuel', 'gas', 'for', 'the', 'month', 'of', 'april', 'on', 'a', 'side', 'note', 'we', 'will', 'bring', 'down', 'our', 'gas', 'turbine', 'generator', 'on', 'sunday', 'morning', 'this', 'will', 'bring', 'our', 'high', 'pressure', 'gas', 'requirements', 'down', 'to', 'essentially', 'zero', 'it', 'will', 'be', 'zero', 'if', 'our', 'hydrogen', 'plant', 'stays', 'down', 'and', 'the', 'demand', 'will', 'transfer', 'to', 'our', 'low', 'pressure', 'gas', 'we', 'will', 'be', 'in', 'this', 'arrangement', 'for', 'approximately', '20', 'days', 'brian']\n",
      "b\"Subject: re : meter 9707\\r\\nfyi - there is flow for 10 / 2 as well .\\r\\nal\\r\\ndaren j farmer\\r\\n10 / 02 / 2000 02 : 23 pm\\r\\nto : aimee lannou / hou / ect @ ect\\r\\ncc : robert cotten / hou / ect @ ect\\r\\nsubject : re : meter 9707\\r\\ni rolled this deal for day one in october only .\\r\\nbob - please check with vance on this deal . has it terminated or should it\\r\\nroll for october . i don ' t think that vance nom ' d anything .\\r\\nd\\r\\naimee lannou 10 / 02 / 2000 10 : 49 am\\r\\nto : daren j farmer / hou / ect @ ect , robert cotten / hou / ect @ ect\\r\\ncc :\\r\\nsubject : meter 9707\\r\\ndaren - meter 9707 has flow for oct . 1 ( 1 . 459 mmbtu ) . deal # 125786 was used\\r\\nfor sept . 2000 . could you please extend the deal or have a new one\\r\\ncreated ? thanks .\\r\\nal\"\n",
      "['re', 'meter', '9707', 'fyi', 'there', 'is', 'flow', 'for', '10', '2', 'as', 'well', 'al', 'daren', 'j', 'farmer', '10', '02', '2000', '02', '23', 'pm', 'to', 'aimee', 'lannou', 'hou', 'ect', 'ect', 'cc', 'robert', 'cotten', 'hou', 'ect', 'ect', 'subject', 're', 'meter', '9707', 'i', 'rolled', 'this', 'deal', 'for', 'day', 'one', 'in', 'october', 'only', 'bob', 'please', 'check', 'with', 'vance', 'on', 'this', 'deal', 'has', 'it', 'terminated', 'or', 'should', 'it', 'roll', 'for', 'october', 'i', 'don', \"'\", 't', 'think', 'that', 'vance', 'nom', \"'\", 'd', 'anything', 'd', 'aimee', 'lannou', '10', '02', '2000', '10', '49', 'am', 'to', 'daren', 'j', 'farmer', 'hou', 'ect', 'ect', 'robert', 'cotten', 'hou', 'ect', 'ect', 'cc', 'subject', 'meter', '9707', 'daren', 'meter', '9707', 'has', 'flow', 'for', 'oct', '1', '1', '459', 'mmbtu', 'deal', '125786', 'was', 'used', 'for', 'sept', '2000', 'could', 'you', 'please', 'extend', 'the', 'deal', 'or', 'have', 'a', 'new', 'one', 'created', 'thanks', 'al']\n",
      "b'Subject: hpl nom for march 21 , 2001\\r\\n( see attached file : hplno 321 . xls )\\r\\n- hplno 321 . xls'\n",
      "['hpl', 'nom', 'for', 'march', '21', '2001', 'see', 'attached', 'file', 'hplno', '321', 'xls', 'hplno', '321', 'xls']\n"
     ]
    }
   ],
   "source": [
    "encoding = 'latin_1'\n",
    "\n",
    "def tokenize(text):\n",
    "    return tf.keras.preprocessing.text.text_to_word_sequence(\n",
    "        clean(text.decode(encoding)), \n",
    "        filters='!\"#$%&()*+,-./:;<=>?@[\\]^_`{|}~\\t\\n\\r'  # Default + '\\r'\n",
    "        )\n",
    "\n",
    "for text, _ in ds_train.unbatch().take(5):\n",
    "  text_np = text.numpy()\n",
    "  print(text_np)\n",
    "  print(tokenize(text_np))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Now let's count the spam and ham occurrences of each word:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.97 s, sys: 132 ms, total: 2.1 s\n",
      "Wall time: 1.92 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "good = {}\n",
    "bad = {}\n",
    "for text_batch, label_batch in ds_train:\n",
    "  for i in range(len(text_batch)):\n",
    "    text = text_batch.numpy()[i]\n",
    "    label = label_batch.numpy()[i]\n",
    "    for token in tokenize(text):\n",
    "      good_or_bad = bad if label == 1 else good\n",
    "      good_or_bad.setdefault(token, 0)\n",
    "      good_or_bad[token] += 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'the': 16460, 'to': 13750, 'ect': 12624, 'for': 7254, 'and': 7118, 'hou': 6627, 'enron': 6028, 'on': 5474, 'a': 5473, \"'\": 5357, 'of': 5057, 'you': 4874, 'i': 4773, 'this': 4444, 'is': 4409, 'in': 4060, '2000': 3981, 'be': 3407, 'will': 3125, 'that': 3064, 'from': 2914, 'have': 2895, 'we': 2863, 'at': 2677, 'gas': 2593}\n",
      "{'the': 6584, 'to': 4650, 'and': 4402, 'of': 4087, 'a': 3427, 'in': 2802, 'you': 2471, 'for': 2293, 'this': 2044, 'is': 2033, 'your': 1761, \"'\": 1579, 'with': 1339, 's': 1210, 'that': 1208, 'be': 1190, 'or': 1177, 'on': 1130, 'as': 1087, 'are': 1025, 'we': 994, 'i': 981, 'it': 948, 'not': 947, 'our': 945}\n"
     ]
    }
   ],
   "source": [
    "def dslice(dictionary, stop):\n",
    "  return dict(itertools.islice(dictionary.items(), stop))\n",
    "\n",
    "def dvsorted(dictionary, reverse):\n",
    "  return dict(sorted(dictionary.items(), key=lambda i: i[1], reverse=reverse))\n",
    "\n",
    "print(dslice(dvsorted(good, reverse=True), 25))\n",
    "print(dslice(dvsorted(bad, reverse=True), 25))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "The next step is to determine the word probabilities. Graham uses a complicated formula for that, which can be simplified using the fact that the number of spam and ham emails is the same for him. Additionally removing the edge-case handlings, we arrive at the standard relative frequency formula:\n",
    "\n",
    "$$\\frac{b / n_b}{g / n_g + b / n_b} \\underset{n_g = n_b}{=} \\frac{b}{g + b} = \\frac{\\mathbb{P}(\\textrm{spam} \\land \\textrm{word})}{\\mathbb{P}(\\textrm{word})} = \\mathbb{P}(\\textrm{spam} \\mid \\textrm{word}).$$\n",
    "\n",
    "Graham also added a bias: he double counted the non-spam occurrences. As he wrote, this reduced the number of false positive for him (he used his own mailbox for testing)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 42.6 ms, sys: 311 µs, total: 42.9 ms\n",
      "Wall time: 42.7 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "word_probs_graham = {}\n",
    "for word in {**good, **bad}.keys():\n",
    "  if word in word_probs_graham:\n",
    "    continue\n",
    "  g = 2 * good.get(word, 0)  # Here's the bias\n",
    "  b = bad.get(word, 0)\n",
    "  word_probs_graham[word] = b / (g + b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'txu': 0.0, 'fuels': 0.0029850746268656717, 'sds': 0.0, 'nomination': 0.0009319664492078285, 'for': 0.13647997143027202, 'january': 0.08254397834912043, '2001': 0.001091703056768559, 'attached': 0.006602336211274758, 'is': 0.1873560040549258, 'the': 0.16666666666666666}\n"
     ]
    }
   ],
   "source": [
    "print(dslice(word_probs_graham, 10))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "For a word which was never seen before, he sets $0.4$ probability. (\"\\[I\\]t is probably fairly innocent; spam words tend to be all too familiar.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.0\n",
      "0.44625407166123776\n",
      "True\n",
      "0.4\n"
     ]
    }
   ],
   "source": [
    "def get_word_prob(word, word_probs):\n",
    "  return word_probs.get(word, 0.4)\n",
    "\n",
    "print(get_word_prob('enron', word_probs_graham))\n",
    "print(get_word_prob('company', word_probs_graham))\n",
    "print('neverseen' not in word_probs_graham)\n",
    "print(get_word_prob('neverseen', word_probs_graham))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "The conditional probability on independent events' intersection can be calculated from the conditional probabilities on the individual events $p_1, \\dots , p_n$ as\n",
    "\n",
    "$$\\frac{\\prod_{i=1}^n p_i}{\\prod_{i=1}^n p_i + \\prod_{i=1}^n (1 - p_i)}$$\n",
    "\n",
    "if none of them is $0$ (that's the case for Graham), and $0$ otherwise:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7941176470588235\n",
      "0\n",
      "0.5\n"
     ]
    }
   ],
   "source": [
    "def combine(probs):\n",
    "    if any(p == 0 for p in probs):\n",
    "        return 0\n",
    "    prod = np.prod(probs)\n",
    "    neg_prod = np.prod([1 - p for p in probs])\n",
    "    if prod + neg_prod == 0:  # Still possible due to floating point arithmetic\n",
    "        return 0.5  # Assume that prod and neg_prod are equally small\n",
    "    return prod / (prod + neg_prod)\n",
    "\n",
    "print(combine([0.6, 0.72]))\n",
    "print(combine([0.5, 0]))\n",
    "print(combine([1 / sys.maxsize for _ in range(100)] + [1]))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "The spam probability of an email is calculated based on the most interesting fifteen words in it, where interestingness is defined by how far they are from $0.5$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.001, 0.002, 0.003, 0.004, 0.005, 0.006, 0.007, 0.008, 0.009, 0.01, 0.95, 0.94, 0.93, 0.92, 0.91]\n"
     ]
    }
   ],
   "source": [
    "def get_interesting_probs(probs, intr_threshold=None):\n",
    "    if intr_threshold is None:\n",
    "        intr_threshold = 15\n",
    "    return sorted(probs, \n",
    "                  key=lambda p: abs(p - 0.5),\n",
    "                  reverse=True)[:intr_threshold]\n",
    "\n",
    "print(get_interesting_probs([0.001, 0.002, 0.003, 0.004, 0.005,\n",
    "                             0.91, 0.92, 0.93, 0.94, 0.95,\n",
    "                             0.006, 0.007, 0.008, 0.009, 0.01,\n",
    "                             0.4, 0.5, 0.6]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.974025974025974\n",
      "0\n"
     ]
    }
   ],
   "source": [
    "def get_text_prob(text, word_probs, intr_threshold=None):\n",
    "  words = tokenize(text)\n",
    "  probs = [get_word_prob(w, word_probs) for w in words]\n",
    "  interesting_probs = get_interesting_probs(probs, intr_threshold)\n",
    "  return combine(interesting_probs)\n",
    "\n",
    "print(get_text_prob(b'Subject: sex', word_probs_graham))\n",
    "print(get_text_prob(b'Subject: Enron', word_probs_graham))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "An email is predicted to be spam above $0.9$ probability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "0\n"
     ]
    }
   ],
   "source": [
    "prob_threshold = 0.9\n",
    "\n",
    "def predict_label(text, word_probs, intr_threshold=None):\n",
    "  return 1 if get_text_prob(text, word_probs, intr_threshold) > prob_threshold else 0\n",
    "\n",
    "print(predict_label(b'Subject: sex', word_probs_graham))\n",
    "print(predict_label(b'Subject: Enron', word_probs_graham))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Now let's set up our test dataset, and evaluate Graham's spam filter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 5172 files belonging to 2 classes.\n",
      "Using 517 files for validation.\n",
      "b'Subject: cdnow shipment confirmation\\r\\ndear daren ,\\r\\nthank you for shopping at cdnow . please keep this email invoice for\\r\\nyour records .\\r\\non april 3 we shipped your order number 17817331 for the following item :\\r\\nrobert earl keen : walking distance\\r\\nformat : cd quantity : 1 price : 12 . 99\\r\\nto the following address via u . s . postal service :\\r\\ndaren farmer\\r\\n5519 clarkston ln\\r\\nspring , tx 77379\\r\\ntotal number of items : 1\\r\\nsubtotal : $ 12 . 99\\r\\nshipping : $ 2 . 99\\r\\nsales tax : $ 0 . 00\\r\\n- - - - - - - - - -\\r\\nshipment total : $ 15 . 98\\r\\nyour order was billed to daren farmer . this shipment completes your\\r\\norder and is paid in full .\\r\\nmost orders arrive within 4 - 8 business days . however , in rare instances\\r\\nit may take up to 2 weeks .\\r\\nfor complete information about your order ( number 17817331 ) or to\\r\\nconfirm the status , click or copy / paste this link into your web browser :\\r\\nhttp : / / cdnow . com / myorder / otid = 17817331\\r\\nyou can also access your order history directly from our home page .\\r\\nplease do not reply to this email . if you have questions about your order\\r\\nthat are not addressed in your online order history , please visit our contact\\r\\ncdnow page using this link :\\r\\nhttp : / / cdnow . com / service\\r\\nfast forward rewards ( tm ) program points for this order will\\r\\nbe credited in your membership summary as \" points earned . \"\\r\\nthanks again for your order .\\r\\nsincerely ,\\r\\ncustomer service\\r\\ncdnow , inc . your music . your store .\\r\\nhttp : / / cdnow . com\\r\\naol keyword : cdnow\\r\\n1757978'\n",
      "0\n",
      "b'Subject: bottle of beer bodice rippers beyond 3735\\r\\nreinforce checksum continuo bell finery refection headway cromwellian jiffy mephistopheles cougar fungoid violin carthage majestic monmouth squeeze eveready dispensary stateroom douglas floury deferring too catkin embassy blythe greenhouse ireland denote warden vertical demerit school ac emmanuel evenhanded shirk vixen grandparent rm 8\\r\\nstreamside betrayer al rhombic eyewitness tack baggage cowboy axisymmetric idiosyncrasy vwy 9\\r\\ndurableoperon'\n",
      "1\n",
      "b'Subject: over 160 fda approved meds\\r\\n% rnd _ alt % rnd _ alt % rnd _ alt\\r\\nover 160 meds to choose from with no prescription required\\r\\nnever mind !\\r\\n% rnd _ alt % rnd _ alt % rnd _ alt\\r\\n'\n",
      "1\n",
      "b\"Subject: equistar feb - 01\\r\\ndaren - here are the equistar # ' s . there should be a total of 65 . 000 . there\\r\\nis a new deal per janice at equistar at meter 1373 effective 2 / 01 . i have\\r\\nnot seen a deal for this . let me know if it is legitimate . no volumes were\\r\\nadded , just moved from meter to meter . ( deal # 378904 ) thanks .\\r\\nal\"\n",
      "0\n",
      "b'Subject: the foundation course - bottom - line project management ( july 25 - 26 )\\r\\nwe said it was coming and here it is . . . . .\\r\\nbottom - line project management\\r\\nthe foundation course\\r\\njuly 25 \\x01 ) 26 , 2000\\r\\ntarget audience\\r\\nthis class is for people new to project management , from all functions and at\\r\\nall leadership levels . in addition , it \\' s for shirt sleeve project managers ,\\r\\nmembers of large project teams and individual contributors . participants may\\r\\nhave business experience but little project management training . they do\\r\\npart of the project themselves and use borrowed resources to do the rest .\\r\\nthe \\x01 & basics \\x01 8\\r\\nteaches the basics \\x01 ) how to plan , schedule , resource , influence , track ,\\r\\nreport and complete any size project in any area . features pre - work , action\\r\\nlearning , individual and team exercises and a mini - simulation of a project on\\r\\nwhich participants practice skills and get immediate feedback on results .\\r\\nwhat you learn\\r\\nparticipants learn and take away an application package , the bottom - line tool\\r\\nkit , to successfully implement their next project assignment . there is an\\r\\noption to use project software in class to identify critical path , modify\\r\\ntasks , and control projects .\\r\\nkey learning points :\\r\\nspeak a common language \\x01 ) use basic terms , answer key questions\\r\\nidentify behaviors of effective project leaders and teams\\r\\n5 - step common sense methodology to plan , organize , resource , implement and\\r\\nclose - out any size project \\x01 ) in any area\\r\\norganizational terms & definitions : stakeholders , customers , sponsors , etc .\\r\\ndevelop and apply 3 key visual tools to planing and track performance\\r\\nresource , staff and complete projects , with limited , \\x01 billed to your rc and company\\r\\nnumber\\r\\nto enroll\\r\\nsign up on line through the organizational development & training intranet\\r\\nsite , which can be accessed through enron \\' s home page . click on \" sites \" ,\\r\\n\" intranet sites \" and then \" organizational development & training \" . then\\r\\nclick on \" training \" and then \" leadership training \" . or , go directly to the\\r\\nsite by entering the intranet address below .\\r\\nif you have any questions about this course or any other offerings\\r\\nsponsored by od & t , feel free to contact\\r\\nsusan m . lewis at 713 - 853 - 9712 or slewis 2 @ enron . com .'\n",
      "0\n",
      "b'Subject: strapon fucked cunt lickin dyke .\\r\\nshe especially loves black penis that can bang her wet pussy to give her a real good pounding .\\r\\nthe gorgeous gang bang floosie .\\r\\nnatural , hairy , dreadlocked , hemp wearing , barefoot , earth - loving goddesses .\\r\\nbut their determination to banish fools foundered ultimately in the installation of absolute idiots .\\r\\n'\n",
      "1\n",
      "b\"Subject: new product ! cialis soft tabs .\\r\\nhi !\\r\\nwe have a new product that we offer to you , c _ i _ a _ l _ i _ s soft tabs ,\\r\\ncialis soft tabs is the new impotence treatment drug that everyone is talking\\r\\nabout . soft tabs acts up to 36 hours , compare this to only two or three hours\\r\\nof viagra action ! the active ingredient is tadalafil , same as in brand cialis .\\r\\nsimply disolve half a pill under your tongue , 10 min before sex , for the best\\r\\nerections you ' ve ever had !\\r\\nsoft tabs also have less sidebacks ( you can drive or mix alcohol drinks with them ) .\\r\\nyou can get it at : http : / / go - medz . com / soft /\\r\\nno thanks : http : / / go - medz . com / rr . php\\r\\n\"\n",
      "1\n",
      "b'Subject: may activity survey\\r\\ndaren ,\\r\\nplease provide the number of producer services deals for the month of may\\r\\n( new vs . existing )\\r\\nshari\\r\\n3 - 3859\\r\\n- - - - - - - - - - - - - - - - - - - - - - forwarded by shari mao / hou / ect on 06 / 06 / 2000 01 : 02 pm\\r\\n- - - - - - - - - - - - - - - - - - - - - - - - - - -\\r\\njames scribner @ enron\\r\\n05 / 30 / 2000 11 : 43 am\\r\\nto : bob klein / hou / ect @ ect , bryce baxter / hou / ect @ ect , carrie\\r\\nhollomon / hou / ect @ ect , cheryl dudley / hou / ect @ ect , daren j farmer / hou / ect @ ect ,\\r\\nfaye ellis / hou / ect @ ect , gordon heaney / corp / enron @ enron , judy\\r\\nbarnes / hou / ect @ ect , kathryn cordes / hou / ect @ ect , kim s theriot / hou / ect @ ect ,\\r\\nlinda s bryan / hou / ect @ ect , lisa csikos / hou / ect @ ect , mary g\\r\\ngosnell / hou / ect @ ect , michael e moscoso / hou / ect @ ect , pat\\r\\nclynes / corp / enron @ enron , rita wynne / hou / ect @ ect , d todd hall / hou / ect @ ect ,\\r\\nirene flynn / hou / ect @ ect , donna consemiu / hou / ect @ ect , bill d hare / hou / ect @ ect ,\\r\\nconnie sutton / hou / ect @ ect , dan houston / corp / enron @ enron , michael\\r\\neiben / hou / ect @ ect\\r\\ncc : lisa b cousino / hou / ect @ ect , shari mao / hou / ect @ ect , jennifer\\r\\nreside / hou / ect @ ect , elmo gore / corp / enron @ enron\\r\\nsubject : may activity survey\\r\\nattached please find the activity survey for the month of may , 2000 .\\r\\ni am sending the survey out this time as an opportunity to introduce myself\\r\\nto those of you i have not yet met . i am being asked to take a direct roll\\r\\nin the further development and rollout of the operations pricing model ( fka\\r\\ntransaction cost model ) . in the past , i have directed the implementation of\\r\\nseveral pricing systems based on activity based costing similar to the\\r\\noperations pricing model ( opm ) . over the past couple of weeks my\\r\\nobservations of the model have lead me to be concerned with the quality of\\r\\ndata used in the model . it will take efforts by both you and the opm team to\\r\\nget to a satisfactory level of data quality . this process can start with\\r\\nthis months data collection efforts . please pay particular attention this\\r\\nmonth as the \" bar is being raised \" by the management of energy operations for\\r\\ndata quality and timeliness .\\r\\nthe attached survey drives your costs from your activities to the commercial\\r\\nteams . please select your rcs , and fill in the driver quantity for the\\r\\nrequested driver for the month of may . please fill out the attached activity\\r\\ndriver survey with may numbers for your rc and return to shari mao by end of\\r\\nday friday , june 2 .\\r\\nplease note the following updates :\\r\\nfor nat gas , sally beck is interested in physical ( as denoted as ph in the\\r\\nsurvey ) vs . financial ( as denoted as fi in the survey ) numbers , please\\r\\nprovide statistics accordingly\\r\\nif a driver has not been updated per your request , please call or email\\r\\njennifer reside\\r\\nif you have additional updates / suggestions or items that need correction ,\\r\\nplease call or email jennifer reside\\r\\nif your group currently generates a standard report that captures the\\r\\nsurveyed data , please send the report to jennifer reside . this will\\r\\neliminate your need to fill out this survey every month . if you would like\\r\\nus to collect this information from someone within your team beginning next\\r\\nmonth , please let jennifer know .\\r\\nplease contact jennifer reside x 3 - 5971 with questions ( shari is on vacation\\r\\nthis week ) , or if you are unable to complete the survey within the allotted\\r\\ntime .'\n",
      "0\n",
      "b'Subject: wacog for sale to cannon winter 2000 - 01\\r\\naccording to the contract the pricing of the gas we are selling back to\\r\\ncannon is a wacog of what we bought it for this summer . here is the\\r\\ncalculation of the wacog . let me know if there are any questions / comments .\\r\\ndave'\n",
      "0\n",
      "b\"Subject: dave n out until july 5 th\\r\\ni will be taking a sabbatical beginning any moment . i have no control over\\r\\nthe exact time ; it may have already happened . i plan to visit the mountains\\r\\nand the ocean . they miss me . i miss them .\\r\\nunfortunately , enron requires employees to use vacation time instead of inner\\r\\nself time , so i will have to return someday . looks like july 5 th . i ' m kind\\r\\nof bummed about it , but maybe by then i will have transcended to a new\\r\\nplateau of understanding .\\r\\nin my absence , contact milind patil , x 39247 for unify performance issues .\\r\\nthanks for understanding ,\\r\\nthe persona formerly known as dave ( the keyboard can no longer be used to\\r\\nspell my new name )\"\n",
      "0\n"
     ]
    }
   ],
   "source": [
    "ds_test = tf.keras.utils.text_dataset_from_directory(**ds_base_kwargs,\n",
    "                                                     subset='validation')\n",
    "\n",
    "for text, label in ds_test.unbatch().take(10):\n",
    "  print(text.numpy())\n",
    "  print(label.numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "true_positive\n",
      "true_negative\n",
      "false_positive\n",
      "false_negative\n"
     ]
    }
   ],
   "source": [
    "def evaluate(prediction, label):\n",
    "  if prediction == 1 and label == 1:\n",
    "    return 'true_positive'\n",
    "  elif prediction == 0 and label == 0:\n",
    "    return 'true_negative'\n",
    "  elif prediction == 1:\n",
    "    return 'false_positive'\n",
    "  else:\n",
    "    return 'false_negative'\n",
    "\n",
    "print(evaluate(1, 1))\n",
    "print(evaluate(0, 0))\n",
    "print(evaluate(1, 0))\n",
    "print(evaluate(0, 1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'true_positives': 80, 'true_negatives': 369, 'false_positives': 0, 'false_negatives': 68}\n"
     ]
    }
   ],
   "source": [
    "def predict_and_evaluate(tf_text, tf_label, word_probs, intr_threshold=None):\n",
    "    return evaluate(predict_label(tf_text.numpy(), word_probs, intr_threshold), tf_label.numpy())\n",
    "\n",
    "def evaluate_bayesian(word_probs, intr_threshold=None):\n",
    "    results = {'true_positives': 0,\n",
    "               'true_negatives': 0,\n",
    "               'false_positives': 0,\n",
    "               'false_negatives': 0}\n",
    "    for t, l in ds_test.unbatch():\n",
    "        results[f'{predict_and_evaluate(t, l, word_probs, intr_threshold)}s'] += 1\n",
    "    return results\n",
    "\n",
    "results_graham = evaluate_bayesian(word_probs_graham)\n",
    "\n",
    "print(results_graham)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Correcting Graham's bias\n",
    "\n",
    "The bias of double counting the non-spam occurrences doesn't seem to fit in to the [Bayesian epistemological](https://plato.stanford.edu/archives/sum2024/entries/epistemology-bayesian/) framework, which might refer to relative frequencies, but hardly to biased ones. Let's visualize the bias, and see if we can perhaps get rid of it.\n",
    "\n",
    "One can verify that the formula for the bias is $\\frac{p}{2 - p}$ by substituting $p = \\frac{b}{g + b}$:\n",
    "\n",
    "$$\n",
    "\\frac{\\frac{b}{g + b}}{2 - \\frac{b}{g + b}} = \\frac{b}{2g + b}.\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def bias(p):\n",
    "    return p / (2 - p)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Here's a plot comparing Graham's biased probablities to the original ones:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "black_or_white = 'lightgrey'\n",
    "mpl.rcParams['text.color'] = black_or_white\n",
    "mpl.rc('axes', facecolor='none', edgecolor=black_or_white)\n",
    "mpl.rcParams['xtick.color'] = black_or_white\n",
    "mpl.rcParams['ytick.color'] = black_or_white\n",
    "mpl.rcParams['axes.labelcolor'] = black_or_white"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP4AAAEICAYAAAB/KknhAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAjxklEQVR4nO2de7xUZdXHv3BARcXBV6wUEHwVM1xWCt7yRqkJYtBrXgAxCYRXk9QQlbzUFrMwy9DCC5qhlhqZGYaJlzR9TQpJ0IWakSKilnjb5gUVPO8fax8Zhpkzc87Zs/eemfX9fPh8zsx+Zj+/M5zffm7rWU+n5uZmHMdpLDqnLcBxnORx4ztOA+LGd5wGxI3vOA2IG99xGhA3vuM0IG78KqOqV6rqeQnXeb+qnpBQXctV9eB2frakTlXdTlXfUtWmwrKqeqyq3tXKffdX1b+3R1Oj0CVtAbWOqi4HPg6sBT4A/gycKCLPA4jIiempq11EZAWweYlrvwR+2fJaVZuB/iKyLLr+IPDJJHTWKt7ix8OXRGRzYBvg38BPUtYTC6rqDUOd4v+xMSIiq1X1FmBGy3uqOhtYKSLnquqWwA3AXth3/xDWO1gZlR0LfBvYGngFODdq3VDVccAZwCeAvwITReS56Noh2MNmm+j+nUppVNUAEKyHchjwD+BrIrIkur4cuAI4Fvikqm4Wlfs+0AtYDJwkIk/m3XYPVb0sqv+26Prqcr9vxA6q+ldgZ+C+SMtrqtoPeBboKiJrCn6HscAJIrKfqj4Qvb0kavnHYw/fX4hI76j8ttH3cwDwFvBjEbksurYncDmwE/Au8EsRmVzq+6sXvMWPEVXdFDgGWFCiSGfg50BfYDvsD+2n0Wc3Ay4DhopId+BzmMlQ1RHA2cAR2EPhQeCm6FpP4FbgXKAn8E9g3zJSRwC/Bv4LuBG4TVW75l0fBQwDegD/HdV1WlT3HcDtqrpRXvljgUOBHTADnVvu983jq8A47KGxJvoOKkZEDoh+/IyIbC4iv8q/rqqdgduBJdiD6yDgNFU9NCpyKXCpiGwR6Z/TlvprFW/x4+E2VV0DbAaswkywASLyKvCblteqeiHWyrXwISCqukJEXgJeit4/Efh+Syurqt8DzlbVvsCBwFIRuSW6NgM4vYzeRXnlL4nK7409UAAua5mjUNVjgHkicnf0+ofAqdiD6f6o/E/zyl+Ita7nVvD7AtwgIhpdPw9YrKrHl9HfFvYAthaRadHrZ1T1amAkMB+bl9lRVXuKyCuUfmjXFd7ix8OXRaQHsAkwCfiTqn6isJCqbqqqV6nqc6r6JvAA0ENVm0Tkbay3cCLwkqrOU9Wdo4/2BS5V1TdU9Q3gNaw73wvYFni+pQ4Rac5/XYL88h8CK6P7bHA9ev+5gvLPR3UXK/9cy71a+31b+WxXrOcSF32BbVu+u+j7OxubkAUbGuwEPKWqC1X18Bjrzize4seIiKwFblXVq4D9gFsKipyOzTbvJSL/UtXPAo8SjclFZD4wX1W7Ad8Frgb2x8xxYct4Px9V7Q/0yXvdKf91CfLLdwZ6Ay/mXc/fsvkisGuR+79Q7H5Yl77lXq3+viU++wE2v1Hud6iU54FnRaR/sYsi8g9gVPQ9HAHcoqpbRQ/iusWNHyORKYYDWwJPFinSHRvnvqGq/wV8J++zH8e62/dEZd7Cuv4AVwIXqOpiEVmqqjngiyLya2Ae8FNVPQKYC5yMTQC2xsC88qcA71G6izsHmKqqB2Et9qlR+T/nlTlZVX8PvAOcA7SMs0v+vnmMUdXrgeXANOAWEVmrqmV+hfX4NzYXsazItb8C/1HVs7D5g/eBTwHdRGShqo4B5ovIqqg3AOu+97rFu/rxcLuqvgW8CVwIHC8iS4uUmwF0w1q0BcCdedc6A5Ox1vI1bOx+EoCI/Ba4CLg56jIrMDS69gpwFDAdeBXoj82et8bvsGHF68BxwBEi8kGxgiLyd2AMNm5/BfgStnz5fl6xG4G7gGewycXvVvD7tnADMBv4FzZUOqWM9mIEwHVRV/7oAv1rgcOBz2KrBK8A1wC5qMgQYGn0/3cpMFJE3m2HhpqikyfiaCyi5bwdRWRM2lqc9PAW33EaEDe+4zQg3tV3nAbEW3zHaUBSW85T1VXkBYY4jlMV+orI1oVvprmO/5yIDEqxfsepe1T1kWLve1ffcRoQN77jNCBufMdpQDIVq79o0aKPAT/EkjL4Q6k4HwJPAVMGDhz4ctpinNokU8bHTH8fMH7gwIFFY8cbnUWLFnXF4ut/iCWxcJw2U9b4qnottsnhZRGRItc7YZsbDsN2Z40Vkb+1U8/OuOlbZeDAgR8sWrToBmzfvuO0i0q607OxHUylGIrtCOsPTMTytbVbj5u+PNF35EMhp92U/eMRkQewbaKlGAFcLyLNIrIAy7CyTVwCHcdZx5Rzztz47vM+/8jc84YcUL50aeIY4/di/fRJK6P3XiosqKoTsV4BxJteKVZUtTcwExiAPRx/D5xRsAe9JXvrZSJyZJn73QGMFpE32qElAN4SkR+29bNOfdFv6ryu45re+dMhTX8beN2aQ0ZhiVHaRaKTeyIyC5gFpSOK0iaas7gVuEJERkT54WZhCTbOyCvXRUReBFo1PYCIHFYtvU5j0G/qvK5NrL1xdNO9e73a3P1fx3e5e1JH7heH8V9g/fxovVk/H1ut8QVgtYj8HCyDi6p+E3hWVZ/F5js2B5qibLC/FxGJUmvPxnLW/x1LOHmyiDwS5aofFH3uD8D/YVlqXwBGiMi7qjoB6w1thKWQOk5E3knql3ayS7+p87oCNw7rvODIHTu/CHAKQbi2I/eMw/hzgUmqejN2cEIYpYbuEP2mzpuBpUuKk8XLpw87rUyZXYBF+W+IyJuqugL7vnYHPp136EMLXwdeF5EBqipEOfGL0B8YJSITVHUO8BXgF8CtInI1gKp+F8v+Whcn8jjtp8X0Taw98vyu163CUpT9pszHylLJct5NwGCgp6quxBImdgUQkSuxAxYOw1qpd4CvdVRUxrlbRIpNdu6HLWtiHQB9rMTnnxWRxdHPi4B+0c8SGb4H1jOYH5dgpzaJTH8T8JXzutxw45ad3hoNnEQQdjgZaFnji8ioMtebscyusVJBy1wtnqBg3K6qW2Cpn9cAHU27/F7ez2uxZJRgw4Qvi8iS6IiowR2sx6lh8k3fjdWnj+1y1yQsNflv47i/rwVvyL3Apqr6VYBocu9HmDFbG3M/BBwdfWYAebnoK6Q7dpBGV+xIKqdByTc9MPnJTca9BWwPnBdHaw9u/A2IejD/Axylqv8AngZWY6evtMblwNaq+gSWXnopELah6vOAv2APkKfaqtupDwpNv3yT0VdgfxsLsGF1LGQtVj8TROfAfanIpdnRv5Zyy7FZfLCHw5jolNgdsIMxnovK9YvKvJJXnvy1eRG5giJRjyIStPf3cGqLDUw/fdiPCTgNWykbSxDGliDTjR8fmwL3RV31TsDXCwN+HKcUxU2f6471NO8lCO+Nsz43fkyIyH+wtXrHaRNFTW+chh1Nfk7cdbrxHSdFCkz/zeXTh80AIMhtBUwBbiMI/xJ3vT655zgp0RKcQ6HpjbOxeI5zq1G3G99xUiDP9EdSaPogtx0wCbieICx2+GqHceM7TsK0anrjfKCZ4seKx4KP8QtQ1bXA49jM/Fpgkoj8udItuB2odzAwRUQOr8b9nWxQ1vRBToDjgUsIwhXV0uHG35B3ReSzAKp6KPB94MBKt+A6TilKTuStz0VY4Nf3qqnFjd86WwCvA0Q78Vq24PYDbgA2i8q19Aq2AX4Vfa4LcJKIPKiqX8S6bxsD/wS+JiJvqeoQYAYWCvx/if1WTuJUZPog9wVsw9uZBGFrWa86THaNH+RmUIVtuQThaWXKdFPVxcAmwDbY/vxCXgYOiaL0+mP/oYOA0cB8EbkwivHfVFV7YjOzB4vI26p6FjBZVX8AXB3dfxn2wHDqkCLr9DM2KBTkOgM/AFaQwHbs7Bo/PfK7+vsA10f76/PpCvxUVT+LzQPsFL2/ELg2it67TUQWq+qBWAqvh1QVLNHGw1hG4WdF5B9RXb9gXVoyp05oJTinkJHAQOB4gnB1tXVl1/jlW+aqIyIPRy124Wmj3wT+DXwGWxlZHZV/QFUPAIYBs1X1EmyocHfh9ubooeHUMRWbPsh1w+aSHsWSslQdX85rBVXdGWgCXi24lANeEpEPscMtmqLyfYF/R5l0rsGy9SwA9lXVHaMym6nqTtgOvH7Rhh6AVvMeOLVFG1p6gFOxfA+nx7Xtthxu/A3ppqqLo3H+r4DjRaQwv9nlwPGqugTrsrck5xgMLFHVR4FjgEtFZBUwFrgpysrzMLCziKzGuvbzVPVv2LyBUwe0yfRB7mNYlN7tBOF9ySjMclc/JUSkqcT7y4m21Ebj8k/nXT4rev864Loin/0jsEeR9+/EHhxOndDGlh5gGraz88xqa8vHW3zHiYk2mz7I7QpMAGYShIkmX/EW33FioMLgnHUEuU7AJViwzvnV1ldI1lr8D6PTYJ1WiL6jRCaBnPKU2WVXisOBg4Hzqx2sU4ysGf8p4Dg3f2nyjsn2vHwZoIINNxsS5DbGWvsnsYnixMlaV38Kdu77iYsWLcraQykrfIiZfkraQhqddpneOAXYERhCEKZyOnSmjD9w4MCXga+mrcNxytFu0we5T2BZc28nCFM7NCVTxnecWqDNE3nrMx3brHV6FaRVjHenHacNdMj0Qe5zrNtr/4+qCKwQN77jVEhFu+xKEeSasF13L2BHrqeKd/UdpwLaEZFXyAnY3o3RBOFbcetrK97iO04ZOmz6INcT2313P3Bz3PragxvfcVohhpYezPTdgUlxHoPVEdz4jlOCWEwf5PYGxgOXVitVdnuoaIwf5Ya7FNt3fo2ITC+4vh22K61HVGaqiMR2sqfjJE0Hl+yMINcFOwj1RVKIx2+Nsi1+lDtuJjAUSyE1Kjr/PZ9zgTkishuWQiiVMETHiYN2xt4X42Qsb+RpBOF/4lEXD5V09fcElonIM9HprzcDIwrKNGOZZcGy07wYn0THSY4OhOGuT5DbFrgAuBP4TVz64qKSrn4v4Pm81yuBvQrKBMBdqvoNLOX0wcVupKoTWZdQsmeblDpOlYnN9MalWFLWzEzo5RPX5N4oYLaI9Mbygt+gqhvcW0RmicggERkEvBJT3Y7TYWI1fZAbFt3nAoLwn7EIjJlKjP8C0Cfvde/ovXzGA3PAMtNiOem9RXdqgryJvDhMvxk2J/YEttM0k1TS1V8I9FfV7THDj8QOjshnBXAQllL6U5jxV8Up1HGqQSyz9+tzPtAXOIAgfL+D96oaZVt8EVmDHdk7H0scMEdElqrqNFUdHhU7HZgQZZ29CRgrIpkb1zhOPh2KvS9GkBuInblwFUH4YIcFVpGK1vGjNfk7Ct77dt7PTwD7xivNcapHTBF567A1+6uxg1amdlhglfFNOk7DEbvpjcnAbsBRBOEbMdyvqnjIrtNQVMX0Qa4/Nrb/LRlcsy+GG99pGKpk+k7ALOA9MrpmXwzv6jsNQZW692ABaYOBCQRhzUSseovv1D1VM32Q2w64GLgH+Fks90wIb/GduibGDTfrs66L3xlr7Wuii9+CG9+pW2KOvS9kHHAo8A2CcHmM900E7+o7dUlVTR/k+mAn4dxPjW5Bd+M7dUeVTd8JC9RpAsYThDV5hqF39Z26osrde7BsuYdiS3fPxHzvxHDjO3VDFTbcrE+Q2x7r4t+LpdSqWbyr79QFCZi+M/Bz7NDScbXaxW/BW3yn5ol9l11xTgMOBL5GEK6owv0TxVt8p6apYkTeOoKcYLnxb8OySdc8bnynZknI9BsDNwBvABNrLVCnFN7Vd2qSRExvTMNSZI8gCOsmq5S3+E7NkZjpg9xg4AxgFkE4typ1pIS3+E5NUbXY+0KC3JbA9cAyLMlGXeHGd2qGBIJzDIvOuxLYBvgcQfh2VepJETe+UxMkZnpjLHA08C2CcGEV60kNH+M7mSdR01sarZ9gG3Aurlo9KePGdzJNwqbfGDsb8j3gOIJwbdXqShnv6juZpephuBvyPWB34MsE4coq15Uq3uI7mSRx09t5d5OBmQTh76paVwZw4zuZI6HY+3UEud5YKO4SYEpV68oIbnwnUyQYkWfYCTg3Yuc9Hk0Qrq5qfRnBx/hOZkjc9Mb5wP7YZN7TCdSXCbzFdzJBKqYPckOAs4FrCMJfVL2+DOHGd1InJdP3wXbdPQ6cUvX6MoYb30mVlEy/EfArbFx/FEH4btXrzBgVjfFVdQhwKZZZ9BoRmV6kzNFAADQDS0RkdIw6nTokpTE9wEXAPsAxBOHfE6ozU5Rt8VW1CZgJDAUGAKNUdUBBmf7At4B9RWQXLE2R45SkyC67ZEwf5I7G/j5/QhDOSaTODFJJV39PYJmIPCMi72MhjSMKykwAZorI6wAi8nK8Mp16IuENN+sIcp8CrgUepkHW60tRSVe/F/B83uuVwF4FZXYCUNWHsOFAICJ3Ft5IVSdip4sC9GyzWqfmSdH03YFbgbexcf37idSbUeJax+8C9MeOC+4NPKCqu4rIG/mFRGQWdtAgqvpITHU7NUKKpu8EzMb+Rg8hCF9IpN4MU0lX/wWgT97r3tF7+awE5orIByLyLPA09iU7DpCi6Y2pwBHAmQThfQnWm1kqMf5CoL+qbq+qGwEjgcL8Y7dhrT2q2hPr+tfs8UJOvOTN3idvegvSuRCbm0pq1SDzlDW+iKwBJgHzgSeBOSKyVFWnqerwqNh84FVVfQK4DzhDRF6tlmindkhha+06gtyOUd2PAyfUS2rsOOjU3JzOd6Gqj4jIoFQqdxIhxXX6lsm8h7G8eYMIwmcTqztDlPKZb9JxqkLKpu+MhePuDAxpVNO3hhvfiZ1UTW8EWKzJaQThPQnXXRN4rL4TK6mb3iLzzsNOtr0s0bprCDe+ExsZMP0gLJPOQ8BJPplXGu/qO7GQ2Ak3pQhy22LLyi8DRxCE7yVaf43hxnc6TMrBORDkNsViS3LAvgSh7xUpgxvf6RAZMH1n7Iy73YHhBOFjidZfo7jxnXaTuumN7xHNKRCEv0+h/prEJ/ecdpEJ0we5E4CzsAMuk6+/hvEW32kzqYbhthDkDgGuwMLFv+Ez+G3DW3ynTWTE9LsCt2B7R44mCNckrqHGceM7FZP4CTfFCHK9gDuAt4BhBOGbiWuoA7yr71RE6sE5AEFuC8z0PYD9CcLnW/+AUwo3vlOWjJh+I+A3WMLXYQTh4sQ11BFufKdVMmL6zliSzIOBsQThXYlrqDN8jO+UJBOmN6YDxwLnEITXpaShrnDjO0XJjOmD3GTgDOBy4PupaKhD3PjOBmTI9GOAH2FLd6f4Wn18+BjfWY/Ud9m1EOQOw/bU/xEYQxCuTUVHneItvvMRmQjDBQhy+2Ez+EuA//EttvHjxneATJn+s8DvgRXAUA/QqQ5ufCdLpt8JuAt4EzvxZlUqOhoAN36DkyHT9wVaEmMeQhCuSEVHg+CTew1MJjbcAAS5bTDTbwF8vlHPrE8SN36DkiHT9wTuxg6+OIQgfDQVHQ2GG78BycQuO4AgtyU2pt8Bi79/OBUdDYgbv8HIUEu/BfAHQLBceX9MRUeD4sZvIDJk+s2x7bUDgaMIwjtT0dHAuPEbhAyZfjNsnX5vYCRBeFsqOhocN34DkKEx/abA7cD+WBjuLanocHwdv97J0IabFtMfCBxHEN6Uig4HqLDFV9UhwKVAE3CNiEwvUe4r2E6qPUTkkdhUOu0iQ6bfDDP9YOB4gvDGVHQ4H1G2xVfVJmAmMBRLezRKVQcUKdcdOBX4S9winbaTIdNvjo3pDwS+ShDekIoOZz0q6ervCSwTkWdE5H3gZuzs8UIuAC4CVseoz2kHRSby0jJ9d2zJ7gCse/+LVHQ4G1CJ8XsB+dlMV0bvfYSq7g70EZF5rd1IVSeq6iOq+gjQs61infJkaPa+Bxacsw82e+/d+wzR4Vl9Ve0MXAKMLVdWRGYBs6LP+RxAzGTI9Fthpt8VO/Di1lR0OCWppMV/AeiT97p39F4L3bHoq/tVdTm2PjtXVQfFJdIpT4ZM/3HgPmAXLImGmz6DVNLiLwT6q+r2mOFHAqNbLopISF63XVXvB6b4rH5yZMj0fbBddr2BwwnCe8p8wkmJsi2+iKwBJmGHEz4JzBGRpao6TVWHV1ug0zoZMv2OwIPAJ4BD3fTZpqIxvojcgcVW57/37RJlB3dcllMJGTL9rtjW2i7AQQSh9/Yyjkfu1SgZMv0+wJ+ANcABbvrawI1fg2Qo9v6L2Jj+VWA/gvCJVHQ4bcaNX2NkKCJvJBaR9zRm+uWp6HDahRu/hsiQ6U/GEnQ+jOXI+3cqOpx249tya4RMmD7IdcJCs88BfgeMIgjfTVyH02Hc+DVARkzfFbgSGAdcDXydIFyTuA4nFryrn3EyYvrNsRZ+HNbi/6+bvrbxFj/DZGLJzkJwb8fy451IEF6VuAYndtz4GSUjpv8ktq32E1jc/dzENThVwY2fQTJi+v2B24C1wGCC8K+Ja3Cqho/xM0ZGTD8aC8xZBeztpq8/vMXPEKlH5Nly3bnANOABrHv/WqIanERw42eE1Gfvg9zG2DLdccANwASC8L1ENTiJ4V39DJAB02+Nde2PA76DZcJ109cx3uKnTAZMvwu2XLcNlhvvV4nW76SCt/gpkgHTD8Pi7bthM/du+gbBW/yUSNX0Nok3BUuH/igwgiBcmVj9Tup4i58CkelvJB3TdwOuB34A/BrY303feLjxEybP9EeSvOl7Y8t0Y4DzsDH9O4nV72QG7+onSIHpkw3OCXL7Yecabop17T38toFx4ydEaqa38fxJ2KGnzwJf8BRZjnf1EyBF03cDrsUOPb0L2NNN74Abv+qkaPp+WJ77scD5wJcIwjcSqdvJPN7VryKpbbgJcodiD5smYDhBeHsi9To1gxu/SqRi+iDXGdtkEwAKHEEQLqt6vU7N4cavAimZvie2uWYItk5/ki/VOaVw48dMSqbfB5gDfAw4EZhFEDZXvV6nZnHjx0ji++ltqW4yMB1YAexDEP6tqnU6dYEbPyYSj70PclsBPwe+BNwKjPdZe6dSfDkvBlIw/b7Y5pohwKnAkW56py1U1OKr6hAs8qsJuEZEphdcnwycgJ2YugoYJyLPxaw1kxRsuKnumD7INQFnYamxngM+56fTOu2hbIuvqk1Y5NdQYAAwSlUHFBR7FBgkIp/G4sF/ELfQLJJocE6Q2xaYD1yIfce7u+md9lJJi78nsExEngFQ1ZuBEcBHoZ8icl9e+QXY7q+6JmHTHw7MxhJmTAB+5rP2TkeoxPi9gOfzXq8E9mql/HjsEIYNUNWJwMToZc9KBGaRxExvsfYXAycDS7BttE9VpS6noYh1Vl9VxwCDgAOLXReRWcCsqGxNdlMTNP1ngF8CuwA/Br7lCTCduKjE+C8AffJe947eWw9VPRg7PvlAEanLP9BEgnMs7PabwPeA14AhBOH82OtxGppKjL8Q6K+q22OGHwmMzi+gqrsBVwFDROTl2FVmgIRMvx02lv88djrtCQThK7HX4zQ8ZWf1RWQNMAmbUX4SmCMiS1V1mqoOj4pdDGwO/FpVF6tqXWV3qXpEXpDrRJAbAzyGTaaegJ1i46Z3qkJFY3wRuQO4o+C9b+f9fHDMujJD1YNzbHPNFdicwUPYYRb/jLUOxynAI/daIQHTDweWYsuj3wIOdNM7SeCx+iWoqumDXA8sEvKr2DLdwQTh47Hd33HK4C1+Eaps+sOwJBnHAhdgefDc9E6ieItfQNVi74PclsAlWA48Bb7sIbdOWniLn0fVgnPWjeWPw2LtB7npnTTxFj+iKqa346cvw2IfHgMO90QZThZw41MF01tmnNHADCAHfBu4iCB8v0P3dZyYaHjjV8H0/bB1+SHYTsUTCMKlHVPpOPHS0MaPNQw3yHUBvgF8F2jGMuPMJAjXdlyp48RLwxo/ZtPvDlwN7A7MA75OEK6IQabjVIWGNH5ssfdBrjuWBusU4GXgaOAWT5LhZJ2GM34swTk2eXcEFn23LXAlcLYnvHRqhYYyfkym/2/gJ8BhWLjtUQThw3HqdJxq0zDG77Dpg9wmwBnA2Vg24dOBywjCNTFLdZyq0xDGj8H0Q7FWfgfsqKrJBOEGWYgcp1aoe+N3yPRBbnssCGc48HfgEILwnirIdJxEqWvjt3vDTZDbFJgKnIl1688CZnjknVMv1K3x2xWRZ7P1R2OpxPpEnz/Tu/VOvVGXxm9XcE6Q2w3r1h8ALAaOJQgfrJpIx0mRujN+m4NzgtzHsTDb8cCr2Pny13iorVPP1JXx2zSRZ8tzp2JnAXTDWvtpHoTjNAJ1Y/yKTb9uHH8R0Be4HZhCED6dkFTHSZ26MH7B7H1rpv8c8CNgbyzqbjxBeG9SOh0nK9S88Qtm74ubPsjtCEzHHgwvAeOA630c7zQqNW38st37IPcx4Dxswu494DvAjwjCtxOW6jiZomaN36rpg9zm2MGTZ2ITd1cD5xOE/0pBquNkjpo0fknTB7muwAQsx93HgVuBc/xMecdZn5ozflHT29HSI7GkGDsAD2CHTvp2WccpQk3l1d/A9JuMnkGQOxx4FPgl8DYwDBjspnec0tRMi79BGO4mox/DTpfdB/gndiTVzQThh+mpdJzaoCaMn2/6Y5vuuezCrtcOB34MrAQmArMJwg/S1Og4tURFxlfVIVh+uSbgGhGZXnB9Y+B6YCAW736MiCyPQ2CL6T/TadlXZnS9/KntO/+rJbHlacBVBOHqOOpxnEairPFVtQmYCRyCtbALVXWuiDyRV2w88LqI7KiqI7Fw2GM6Kq7f1Hld9+/82J1jm+Z/4aCmRwG2xvbGz/S1eMdpP5W0+HsCy0TkGQBVvRkYAeQbfwQQRD/fAvxUVTuJSLvTTI88++Kmmzf6zbN7d36y1+rmru9gh03+hCD8T3vv6TiOUYnxewHP571eCexVqoyIrFHVENgKeCW/kKpOxMbkAD1bq3TBhwM+fKP5zufuXLvHkiFNC0cRhG9WoNVxnApIdHJPRGYBswBUtdVjopdPH9YMw/ZNRJjjNBiVrOO/gKWhaqF39F7RMqraBTsh9tU4BDqOEz+VtPgLgf6quj1m8JHYEdD5zAWOBx7Gdsn9sSPje8dxqkvZFl9E1gCTgPnAk8AcEVmqqtNUdXhU7GfAVqq6DJiMZah1HCejdGpuTqdhVtVHRGRQKpU7ToNQymc1FavvOE48uPEdpwFx4ztOA+LGd5wGJM3deX3LBfFE9KQgAjBDZFkbuL6OkGVtULm+vkXfbW5uzvS/xx9//JG0NdSiNtdXv9ri0OddfcdpQNz4jtOA1ILxZ6UtoBWyrA1cX0fIsjbooL7UIvccx0mPWmjxHceJGTe+4zQgmcmym2ZCzxi0TQZOANYAq4BxIvJcEtoq0ZdX7itYarQ9RKSSGIpEtKnq0VjqtmZgiYgUbvtOTZ+qbgdcB/SIykwVkTsS0nYtcDjwsohIkeudMO2HAe8AY0Xkb5XcOxMtfl5Cz6HAAGCUqg4oKPZRQk8stfZFGdL2KDBIRD6NGesHSWhrgz5UtTtwKvCXLGlT1f7At4B9RWQXLHtyZvQB52Jb0XfDclFcnpQ+YDYwpJXrQ4H+0b+JwBWV3jgTxicvoaeIvA+0JPTMZwT25AUz10HREy91bSJyn4i8E71cgGUpSopKvjuAC7CHZZLpyCvRNgGYKSKvA4jIyxnT1wxsEf2cA15MSpyIPAC81kqREcD1ItIsIguAHqq6TSX3zorxiyX07FWqTJQcpCWhZxa05TMe+ENVFa1PWX2qujvQR0TmJagLKvvudgJ2UtWHVHVB1PVOikr0BcAYVV0J3AF8IxlpFdHWv82PyIrx6wJVHQMMAi5OW0sLqtoZuAQ4PW0tJeiCdVUHA6OAq1W1R5qCChgFzBaR3thY+oboO61psvILZDmhZyXaUNWDgXOA4SLyXgK6WiinrzsgwP2quhzYG5irqklkP6rku1sJzBWRD0TkWeBp7EGQBJXoGw/MARCRh4FNKJMaPkEq+tssRlZm9bOc0LOsNlXdDbgKGJLwGLWsPhEJyftDVdX7gSkJzepX8v96G9aq/lxVe2Jd/2cS0FapvhXAQcBsVf0UZvxVCekrx1xgUnTIzV5AKCIvVfLBTLT4WU7oWaG2i4HNgV+r6mJVnZuEtjboS4UKtc0HXlXVJ4D7gDNEJJHU7BXqOx2YoKpLsINbxyaVQVpVb8Iauk+q6kpVHa+qJ6rqiVGRO7CH5DLgauDrld7bQ3YdpwHJRIvvOE6yuPEdpwFx4ztOA+LGd5wGxI3vOA2IG99xGhA3vuM0IP8POdblpqSz0qEAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "dark"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.linspace(0, 1, 100)\n",
    "plt.plot(x, x, label='Original')\n",
    "plt.plot(x, bias(x), label='Biased')\n",
    "plt.title('Biased probabilities')\n",
    "plt.legend()\n",
    "plt.gca().set_aspect('equal')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "That is, Graham slightly decreased all probability values. One's intuition could be that this, instead of decreasing the number of false positives, will in fact increase the number of false negatives: for example, if a spam email consists of high-probability words which combined gives it a probability slightly above $0.9$, then with the bias we might get slightly under $0.9$, causing the email to just slip through the spam filter. Let's see if the `Enron1` dataset confirms our intuition.\n",
    "\n",
    "By a substitution similar to the above one, the reverse formula is $\\frac{2p}{p + 1}$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def unbias(p):\n",
    "    return (2 * p) / (p + 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'true_positives': 81, 'true_negatives': 369, 'false_positives': 0, 'false_negatives': 67}\n"
     ]
    }
   ],
   "source": [
    "word_probs_bayes = {w: unbias(p) for (w, p) in word_probs_graham.items()}\n",
    "\n",
    "results_bayes = evaluate_bayesian(word_probs_bayes)\n",
    "\n",
    "print(results_bayes)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "So there's one single email on which the unbiased Bayesian filter performed better. Let's check it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Subject: stop the aging clock\r\n",
      "nerissa\r\n",
      "\n",
      "Label: 1\n",
      "Graham: 0.8514251061249243 [0.5950413223140496, 0.16666666666666666, 0.8181818181818182, 0.8666666666666667, 0.4]\n",
      "Bayes: 0.9892113259060286 [0.7461139896373057, 0.2857142857142857, 0.9, 0.9285714285714286, 0.4]\n"
     ]
    }
   ],
   "source": [
    "def check_bayesian_diff(word_probs, intr_threshold, display, true_label, include_graham_intr=False):\n",
    "    for email, label in ds_test.unbatch():\n",
    "        email_bytes = email.numpy()\n",
    "        label_int = label.numpy()\n",
    "        if label_int == true_label and predict_label(email_bytes, word_probs_graham) != predict_label(email_bytes, word_probs, intr_threshold):\n",
    "            email_str = email_bytes.decode(encoding)\n",
    "            print(email_str)\n",
    "            print(f'Label: {label_int}')\n",
    "            words = tokenize(email_bytes)\n",
    "            prob_graham = get_text_prob(email_bytes, word_probs_graham)\n",
    "            prob = get_text_prob(email_bytes, word_probs, intr_threshold)\n",
    "            probs_graham = [get_word_prob(w, word_probs_graham) for w in words]\n",
    "            probs = [get_word_prob(w, word_probs) for w in words]\n",
    "            print(f'Graham: {prob_graham} {probs_graham}')\n",
    "            if include_graham_intr:\n",
    "                print(f'Graham interesting: {get_interesting_probs(probs_graham)}')\n",
    "            print(f'{display}: {prob} {probs}')\n",
    "\n",
    "check_bayesian_diff(word_probs_bayes, None, 'Bayes', 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    }
   ],
   "source": [
    "print('nerissa' not in word_probs_graham)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "It really contains a relatively high number of spammy words (\"aging\" and \"clock\", 2 out of 5), which got overly reduced in probability by the grahamian bias, and the email eventually slipped through Graham's filter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEICAYAAACeSMncAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAnI0lEQVR4nO3de7zlY93/8dc2JkxCIj85jaKkd8hIKoUOkmMHkvOpJlJSpNSIFOmulEppHBJRQmowd7jJnbqRGSlv9LipFMb5MOUeMmb274/r2ma123vPmjWz1t5r7/fz8diPWev6Xt/vur571l6fdZ17ent7iYiIaMVSw12AiIjoXgkiERHRsgSRiIhoWYJIRES0LEEkIiJaliASEREtSxCJiIiWLT3cBYhohe27gdWAeQ3JZ0v6yPCUaGSyfRywnqS9h7ssMToliEQ320nSfy0sk+2lJT3bL22cpHmDnTPANRYp/6Kw3QP0SJrfjutHtFNPZqxHN6o1kQ8MFERs7w98EPgtsC/wXWBN4ClgHWArYBfgvnpsk/r4aEnT6jXO7p+//2vZvha4HngrsAHwS+AASY/V41sAJwMbAn8FPibp2oZzfwNsDWwKvBpYBvgGMAmYC5wi6UTbSwFH1XtaCbgaOFjSY7YnAn8B9ge+AEwAvi7pBNvbAdOAHuCfwJ8kbWz7gHq9NYGHgS9L+l7DfR0FfBzoBT4HnA6sL+ku28sAJwDvq+W9BPi4pKdsrwKcDWwJzAduA7ZKcBzd0icSo9XrgD9TmrxOqGl71scvAG4ELgWuBF4MfBQ4z/YrGq7RmP/Xg7zOvsCBwOrAs8A3AWyvAVwOfBFYGTgSuNj2qg3n7gNMrtd/EPgv4BfAS4D1KMGCWrZ3UYLZS4DHgVP7lWNL4BWUgPY526+U9AvgROACSctL2rjmfQjYEVgBOAD4uu1Na7m3Az4BvK2WYet+r3MS8HJK4F0PWIMSaACOAO4FVqX83j9DCUQxiqU5K7rZz2w3NlN9UtLp9fEsSd+qj5+1DfBzSb8BsL0JsDxwUv2mfI3ty4A9gOPqec/lB54epAznSnK95jHALbb3A/YGpkuaXvNdZXsGsD3wg5p2tqTb6rk7Ag9I+lrD691YHx8MfETSvTXvccDfbO/TUI7PS3oK+L3t3wMbA3cMVGBJlzc8/W/bVwJvAm6m1DC+31Cu44C96uMeStDbqKG2dSJwPnA0pfa0OrCOpLuA6wb5ncUokiAS3exdQ/SJ3LOQtJcA9/Rravkr5Zv1UNcY6pp/BcYDq1CawXazvVPD8fGUJq+Bzl0L+NMgr7EOcIntxrLOo3zb7/NAw+M5lAA5INvvBI6l1CiWojSB3VoPvwSYMUgZV615Z9agDKWpbFx9/BVKAL6yHp8q6aTByhGjQ4JIjFYDNaM0ps0C1rK9VEMgWRv434Vco7+1Gh6vTfk2/gjlw/dcSR9ssoz3AO8fJN89wIENtaLn1D6RofzLPdQ+jYspzXA/lzTX9s8owQDgfkpfSZ/G+3uE0k/0Kkn39X8hSf+gNGkdYVuU2t1Nkq7unzdGj/SJxFh1I+Ub+1G2x9veGtgJ+PEiXmdv2xvangAcD1xUR3H9ENjJ9jtsj7O9rO2tba85yHUuA1a3fbjtZWy/wPbr6rHTgBNsrwNge1XbuzRZvgeBibVzHuB5lA7xhynNfO8Etm3I/xPgANuvrPd0TN+BGmxPp/ShvLiWZQ3b76iPd7S9Xm32mk2pLaVTfZRLEIludqntJxt+Lmn2REnPUILGOynfsL8D7Cvpj4tYhnMpI5IeAJYFDqvXv4cyAuwzlA/se4BPMsjfXP0W//ZapgeAO4Ft6uFTKKOsrrT9D+AGysCBZlxY/33U9s31dQ6jBIvHKYMHpjWU4z8pgwN+CdxVXwvK6C6AT/Wl2/47ZTBA32CE9evzJymj1r4jqbH5LkahDPGNaFEdpvtDSWcMd1naxfYrAQPL9J9rEwHpE4mIfmy/G5hO6UT/MnBpAkgMJs1ZEdHfhyhzSf5E6dc4ZHiLEyNZmrMiIqJlac6KiFgI22dRZvk/JEk17QuUwRPzKTW3/SXNGuDcXwBbAL+WtGND+tmUVQhm16T9Jd1i+72UkX6PUeZCPWr7ZcCJknZv1z22aszVRGw/TJkUFhHRrOUpwWJdyppgULoD+oYwv5gyOu9vA5z7gpp3VcrItj4TKQHk8X75X0EZnbcS5Yv+Q/V1Z7FglNxwWEfSqv0Tx2JN5K+SNhvuQkREd6kTOy8b6PPD9tHA2pIG7D+q85COHKAmcpmki/rlvQZ4L7AbJfjcBrxH0seXzJ20pi7b82/GYhCJiBYM0qTzFcrclmcoHfEHSHpigHM/RlmFuAc4XdI3Go59FDiU0ol/uaSjbL+RssLyM8Aeku60vRJlfst2I2VlYNsnUGb/z2bBvJ5FcYLtz1EW2/y0pH8CX6LMt5lFWYPtQgZfzWDYZXRWRDTrbGC7fmlXAZK0EWXJmKP7n1SXQPkgsDllYcgdba9Xj21D6VfYWNKrgK/W046gLFZ5OGUBSoAplH6BERFAACR9VtJawHnAom6IdjRlC4HXUlZ6/lS95lWSJknaifK7mQ683PZFtk+vKwmMGAkiEdEUSb+idPY2pl3ZMIfkBv513a0+rwRulDSn5v1v4D312CGUlZT/Wa/3UE2fS5mnMgGYWzuW1+rbj2UEOo/SBNU0SfdL6q33/n1KkH1ODRb7U5b9/zywH2VLgr2WRIGXlDRnATNnznwx5RvQBiSwDmU+8EfgyEmTJj20sMwx5hwIXDBAuinNNi+iLOC4PQtWCn458KbaLPQ0pd/gJkqTzjk1/z6Uv88p7S3+orG9vqQ769NdKH8bi3L+6pLur2uNvYvye2r0SeCbdZHM5SiLac6nBNYRI0Gk+CplraCDJk2aNHe4CzNSzZw5czwL/qD3HebixAhi+7OUTbnO639M0h22v0zZAOz/gFso/R9QPoNWpgyBfS3wE9svlXRLTcP2mymrC/fYvoBSSzlC0oPtvKdGtn9E2aBrFdv3UpbS375uYjafMuLz4Jp3M8rOkx+oz6+jfEFdvp57kKQrKJugrUrpJ7qFBc122H4JsLmkz9ekbwE3AU9QAs6IkSBSbEACyEJNmjRp7syZM8+l4c0e4bId8Y7AWyUNOGdA0pnAmTX/iZQdEKn//rSe99u6Z8oqlEUr+zbCmkLpWP4WZVvfiZRFJD/bnjsasPx7DJB85iB5ZwAfaHj+pkHyvWWI15sF7NDw/EIWLKY5oqTpplgqAaQ59feU900Az22nexSws6Q5Q+TrWzp+bUp/yPn10M+oo5psv5yyVP0jDafuS9kh8jFKM858RmCTzliWmkhENGWQJp2jKfuTXFV3M7xB0sG1OeYMSdvX0y+ufSJzgUMbhgGfBZzlcvIzwH59tZmGjuW+/U5OpoxUeoayhH2MAAkiA5j46cuX6DT+u0/aoWdheWzPo2xR2kNpL/6IpP9ZkuWIWByL2KQzi9KB3vd8sCadZyhzIQY6NoeGuReSrgNevQhFjg5IEBk5npK0CUDdKe5LlHV1ImJJOm7FsbXWU5/jZi/0y2wrEkRGphWo6+nYXh74OfBCYDwwRdLPbR8PPNY387cOkXxI0im2Pwm8j9LMcImkY20/nzLbd01gHPAFSQMNx4yIaFqCyMixnO1bKIu4rQ70jdx4Gni3pL/bXoWyLek0SlvyT4Fv1P2z3w9sbntbyjalm1OaxqbVIZKrArMk7QBge8XO3VpEjFYJIiNHY3PW64Fz6nIRPcCJNRDMB9YAVpN0t+1Hbb8GWA34XV0yeltKR+Tv6nWXpwSV64Cv1fH6l9X25RijlnS/Xze5e9nhLsHokiAyAkm6vtY6VqV0Tq4KTKozV++m1FYAzqCMXvl/lJoJlKDzJUnf639d25vW633R9tWSjm/rjUTEqJfx/iOQ7Q0o/RaPAitS+jrm1sXq1mnIegllQbzXAlfUtCuAA2tfCrbXsP3iOuRyjqQfAl8BNu3M3UTEaJaayACaGZLbBn19IlBqE/tJmmf7POBS27dS1ht6bn0eSc/Y/iXwhKR5Ne1K268Erq/j9p+kDKFcD/hKnRE8l+ybHRFLQILICCFp3CDpjwCvH+hY7VDfgrJ5TeM5pwCn9Mv+JxbUViIilog0Z3Up2xtSttq8umEl0YiIjkpNpEtJuh146XCXIyLGttREIiKiZQkiERHRsgSRiIhoWYJIRES0LB3rA1nSq3w2sXqm7dWAr1OG7D5O2TPhPyRd0sxL2J5IWc5Ei1HSiIhF0rYgYvssypaZD/V9sNn+CrAT5QPyT8ABDZvTNJ67HWWewzjKxjYn1fTzKPsJXCbpMzVtCmBJP2vXvbRb3QL0Z8APJO1Z09YBdu6Xb2lJz3a+hBERA2tnTeRs4NvAOQ1pVwFHS3q2LgR4NPCpxpNsjwNOBd5O2X/5prpq7dKURQo3sn1VXYV2AvA6SV9s4310wluAZySd1pcg6a/At+r+1e+hLKQ4zvYODLA0fD1tnO3TgTcA9wG7SHrK9geByZStR+8C9pE0x/bZwFPAa4AXAwdStiN9PXCjpP3betcR0fXa1ici6VfAY/3Srmz4Jn0DZW+L/jYH7pL057rr2Y+BXShLdSxXZ2mPp+z+dzxli85u9yrg5iGObwrsKmkrFiwNvyll17ev1ZoMlNV6T5X0KuAJ4L01/aeSXitpY+AO4KCGa7+QEjQ+DkyjNKm9Cni17U2WwL1FxCg2nH0iBwIDbYq0BnBPw/N7KbWNO2w/TPmwPZeyFtRSkob68AXA9mTKN3GAVRar1B1g+1RgS0qz36nAVZL6AvKAS8PXY3+RdEt9PBOYWB/L9heBlSg1msblTy6V1FvX5npQ0q21DLfV828hImIQwxJEbH8WeBY4b1HOk3R4wzUuBT5Ur7Ux5YP29EHOmwpMrefNaLHY7XQbC2oNSDq0LgXfV9b/a8i7F4MvDf/PhnzzgOXq47OBd0n6fW0e27ohX9858/udP58MvIiIhej4EN/6IbYjsJekgUZB3Qes1fB8zZrWeI1dKN+0lwdeJul9wK62J7Sl0O13DbCs7caVdQe7l6GWhh/MC4D7bY+nBKGIiCWio98066iro4CtJM0ZJNtNwPq216UEj/cDezZcYzxwOLADpQ+gLxCNo3QcD3bd5rVpQ/vB1OakdwFft30U8DCl9vEpFtQm+gy6NPwQjgFurNe9kRJUIiIWW09vb3t2ybT9I0qzySrAg5QO8KOBZSibLQHcIOngumHSGZK2r+duD3yDEhjOknRCw3UPp+yfcXbtUD4fEDBd0r+M9BqkXDMkbdaYNnPmzBmTJk3abLBz4l/l99X9xvb2uHsuPNNotJhfjgf67IQ21kQk7TFA8pmD5J1F2ba17/l0YPogeb/R8LgXGOh1IiKiA7LsSUREtCxBpJg/c+bM8cNdiG5Qf0/zh7scETEyJIgUfwT2SSAZWv397ENznfkRMQZkHkBxJPBV4OCZM2cmsA5uPiWAHDncBYmIkSFBBJg0adJDlDWjIiJiEeRbd0REtCxBJCIiWpYgEhERLUsQiYiIliWIREREyxJEIiKiZQkiERHRsgSRiIhoWYJIRES0LEEkIiJaliASEREtSxCJiIiWJYhERETLEkQiIqJlCSIREdGyBJGIiGhZgkhERLSsbTsb2j4L2BF4SJJq2srABcBE4G7gfZIeH+Dc/YAp9ekXJf3A9jLAz4E1ge9I+k7NOxU4TdLN7bqXiIgYWDtrImcD2/VL+zRwtaT1gavr839RA82xwOuAzYFjbb8QeAfwa2AjYJ+ad2NgXAJIRMTwaFsQkfQr4LF+ybsAP6iPfwC8a4BT3wFcJemxWku5ihKM5gITgPFAT837BeCYJVvyiIhoVtuaswaxmqT76+MHgNUGyLMGcE/D83tr2oWUGsgNwFds7wzcLGnWwl7U9mRgcn26Sotlj4iIfjodRJ4jqdd27yLkfxbYE8D2eOAKYBfbJwNrA+dImjbIuVOBqfXcGYtb9oiIKDo9OutB26sD1H8fGiDPfcBaDc/XrGmNPgycA2wBzAZ2B45Y4qWNiIghdTqITAP2q4/3o4y26u8KYFvbL6wd6tvWNABq2o6UIDIBmA/0Asu1sdwRETGAtgUR2z8CrgdeYfte2wcBJwFvt30n8Lb6HNub2T4DQNJjlA7zm+rP8TWtz+eAEyTNpwSXNwG3Aue2614iImJgPb29TXdLjAq2Z0jabLjLETGcJn768rH1h9/g7mX3HO4iDI/jZvcsPNPgBvvszIz1iIhoWYJIRES0LEEkIiJaliASEREtSxCJiIiWJYhERETLEkQiIqJlCSIREdGyhQYR27vZfkF9PMX2T21v2v6iRUTESNdMTeQYSf+wvSVlqZIzge+2t1gREdENmgki8+q/OwBTJV0OPK99RYqIiG7RTBC5z/b3KMutT697nacvJSIimgoG76OslvsOSU8AKwOfbGehIiKiOwy6s6HtlRueXtuQ9k8guwNGRMSQ2+POpGz2NNDywb3AS9tSooiI6BqDBhFJ63ayIBER0X2Gas7aQNIfB5sTIunm9hUrIiK6wVDNWZ8AJgNfG+BYL/CWtpQoIiK6xlDNWZPrv9t0rjgREdFNhqqJPMe2gA2BZfvSJJ3TrkJFRER3WGgQsX0ssDUliEwH3gn8GkgQiYgY45qZbLgr8FbgAUkHABsDK7a1VBER0RWaCSJPSZoPPGt7BeAhYK32FisiIrpBM30iM2yvBJxOmYD4JHD94ryo7Y8DH6CM8roVOEDS0w3Hl6E0l00CHgV2l3S37TdSVhB+BthD0p21bD8BtqvBLiIiOmShNRFJH5b0hKTTgLcD+9VmrZbYXgM4DNhMkoBxwPv7ZTsIeFzSesDXgS/X9COA7YHDgYNr2hTgxASQiIjOa3Z01kbAxL78tteT9NPFfN3lbM8FJgCz+h3fBTiuPr4I+LbtHqAv/wRgru2XAWtJunYxyhIRES1qZnTWWcBGwG1A37f9XqClICLpPttfBf4GPAVcKenKftnWAO6p+Z+1PRt4EfAlSjPXU8A+wFcpNZGF3cNkysRJgFVaKXdERPy7ZmoiW0jacEm9oO0XUmoa6wJPABfa3lvSDxd2rqRbgC3qdd4M3A/02L6AUks5QtKDA5w3FZhaz8sKxBERS0gzo7Out73Egghli92/SHpY0lxKjeYN/fLcRx0BZntpypDiR/sO1qatKcAXgGOBoygd/4ctwXJGRMRCNFMTOYcSSB6g7CXSA/RK2qjF1/wbsIXtCZRmqbfy7/uTTAP2o4wC2xW4RlJvw/F9gemSHqvXmV9/JrRYpoiIaEEzQeRMSv/DrSzoE2mZpBttXwTcDDwL/A6Yavt4YIakafU1z7V9F/AYDaO3atDYH9i2Jp1MmUn/DLDn4pYvIiKa10wQebh+sC8xko6lNEM1+lzD8aeB3QY5dw6wTcPz64BXL8nyRUREc5oJIr+zfT5wKaU5C4DFHOIbERGjQDNBZDlK8Ni2Ia3lIb4RETF6LDSILM7s9IiIGN0WOsTX9sttX23b9flGthc6wS8iIka/ZuaJnA4cTZnMh6Q/8O9rXUVExBjUTBCZIOm3/dKebUdhIiKiuzQTRB6pCx32AtjelbLcSEREjHHNjM46lLLu1Aa27wP+AuzV1lJFRERXaGZ01p+Bt9l+PrCUpH+0v1gREdENmtpPBEDS/7WzIBER0X2a6ROJiIgY0KBBxPZu9d91O1eciIjoJkPVRI6u/17ciYJERET3GapP5FHbVwLr2v63VXwl7dy+YkVERDcYKojsAGwKnAt8rTPFiYiIbjJoEJH0DHCD7TdIetj28jX9yY6VLiIiRrRmRmetZvt3wG3A7bZn2labyxUREV2gmSAyFfiEpHUkrQ0cUdMiImKMayaIPF/SL/ueSLoWeH7bShQREV2jmRnrf7Z9DKWDHWBv4M/tK1JERHSLZmoiBwKrUrbDvRhYpaZFRMQY18wCjI8Dh3WgLBER0WWaXoBxSbK9EnAGIMo+JQdKur7heA9wCrA9MAfYX9LNtl8BnA+MBz4k6XrbSwO/AHaWNKezdxIRMbYN1wKMpwC/kLQBsDFwR7/j7wTWrz+Tge/W9A8BH6MElyNr2iHADxNAIiI6r+M1EdsrAm8G9ofnJjU+0y/bLsA5knopEx5Xsr06ZZ/3CfVnbq3R7ARs15nSR0REo4UGEdtrAt8CtqQ0PV0HfEzSvS2+5rrAw8D3bW8MzKzXa9yvZA3gnobn99a0U4FzgGUotZJjgBMlzV/IPUym1GigDAyIiIgloJnmrO8D04DVgZcAl9a0Vi1NWZPru5JeA/wf8OlmTpT0N0lbS3o9pa9kTeAO2+favsD2ywc5b6qkzSRtBjyyGGWPiIgGzTRnrSqpMWicbfvwxXjNe4F7Jd1Yn1/EvweR+4C1Gp6vWdManQBMoYwcOwO4GziR7P8eEdExzdREHrW9t+1x9Wdv4NFWX1DSA8A9daQVwFuB2/tlmwbsa7vH9hbAbEn39x20vRUwS9KdlP6R+fVnQqvlioiIRddMTeRASp/I1yl9Iv8DHLCYr/tR4Dzbz6PMfj/A9sEAkk4DplNGYN1FabZ67vXq8N8pwO41aSpwXr2XQxazXBERsQh6ent7h7sMHWV7Ru0biRizJn768rH1h9/g7mX3HO4iDI/jZvcszumDfXYOWhOxfZSk/7D9LUoN5F9Iyiz2iIgxbqjmrL4JgDM6UZCIiOg+Q+1seGl9OEfShY3HbO/W1lJFRERXaGZ01tFNpkVExBgzVJ/IOykjpNaw/c2GQysAz7a7YBERMfIN1Scyi9IfsjNlaZI+/wA+3s5CRUREdxiqT+T3wO9tny9pbgfLFBERXaKZyYYTbX8J2BBYti9R0kvbVqqIiOgKzS7A+F1KP8g2lFV0f9jOQkVERHdoJogsJ+lqoEfSXyUdB+zQ3mJFREQ3aKY565+2lwLutP0Rymq6y7e3WBER0Q2aqYl8jLI67mHAJGAfYN92FioiIrrDQmsikm6qD5+krLY7Dng/cOPgZ0VExFgw1GTDFYBDKdvSTgOuqs+PAP5AWX49IiLGsKFqIucCjwPXAx8APgP0AO+WdEv7ixYRESPdUEHkpZJeDWD7DOB+YG1JT3ekZBERMeIN1bH+3Cx1SfMo+6IngERExHOGqolsbPvv9XEPsFx93gP0Slqh7aWLiIgRbai1s8Z1siAREdF9mpknEhERMaAEkYiIaFmCSEREtCxBJCIiWtbMAoxtUZdPmQHcJ2nHfseWoSw5Pwl4FNhd0t2230hZlv4ZYA9Jd9peCfgJsJ2k+Z28h4iIsW44ayIfA+4Y5NhBwOOS1gO+Dny5ph9B2ff9cODgmjYFODEBJCKi84YliNhek7InyRmDZNkF+EF9fBHwVts9lAmQE+rPXNsvA9aSdG17SxwREQMZruasbwBHAS8Y5PgawD0Akp61PRt4EfAlSjPXU5Ql6b9KqYkMyfZkYHJ9usriFDwiIhboeBCxvSPwkKSZtrdelHPrwo9b1Ou8mbKeV4/tCyi1lCMkPTjAeVOBqfW8GYt1AxER8ZzhaM56I7Cz7buBHwNvsd1/z/b7gLUAbC8NrEjpYKem9VBqIF8AjqXUak6nbJwVEREd0vEgIuloSWtKmkjZ3OoaSXv3yzYN2K8+3rXm6W04vi8wXdJjlP6R+fVnQlsLHxER/2LYhvj2Z/t4YIakacCZwLm27wIeowSbvnwTgP2BbWvSycB0yrDfPTtZ5oiIsW5Yg0gdVXVtffy5hvSngd0GOWcOsE3D8+uAV7eznBERMbDMWI+IiJYliERERMsSRCIiomUJIhER0bIEkYiIaFmCSEREtCxBJCIiWpYgEhERLUsQiYiIliWIREREyxJEIiKiZQkiERHRsgSRiIhoWYJIRES0LEEkIiJaliASEREtSxCJiIiWJYhERETLEkQiIqJlCSIREdGyBJGIiGhZgkhERLRs6U6/oO21gHOA1YBeYKqkU/rl6QFOAbYH5gD7S7rZ9iuA84HxwIckXW97aeAXwM6S5nTwViIixrzhqIk8CxwhaUNgC+BQ2xv2y/NOYP36Mxn4bk3/EPAxSnA5sqYdAvwwASQiovM6XhORdD9wf338D9t3AGsAtzdk2wU4R1IvcIPtlWyvDswFJtSfubZXAnYCtuvgLURERNXxINLI9kTgNcCN/Q6tAdzT8PzemnYqpSlsGUqt5BjgREnzF/I6kyk1GoBVFrvgEREBDGMQsb08cDFwuKS/N3OOpL8BW9fz1wPWBO6wfS7wPOAYSf87wHlTgan1vBlL5AYiImJ4RmfZHk8JIOdJ+ukAWe4D1mp4vmZNa3QCMAU4DDgDOAo4dsmXNiIiBtPxIFJHXp0J3CHp5EGyTQP2td1jewtgdu1L6bvGVsAsSXdS+kfm158J7S19REQ0Go7mrDcC+wC32r6lpn0GWBtA0mnAdMoIrLsoQ3wP6Du5BqEpwO41aSpwHuVeDml/8SMios9wjM76NdCzkDy9wKFDHHt7w/M7gE2XZBkjIqI5mbEeEREtSxCJiIiWJYhERETLEkQiIqJlCSIREdGyBJGIiGhZgkhERLQsQSQiIlqWIBIRES1LEImIiJYliERERMsSRCIiomUJIhER0bIEkYiIaNmw7rE+FtneDjgFGAecIemkfseXoewjPwl4FNhd0t312NHAQcA84DBJV9heFbgEWAmYIulnNe/PgUMkzerAbTVlLN97xGiVmkgH2R4HnAq8E9gQ2MP2hv2yHQQ8Lmk94OvAl+u5GwLvB14FbAd8p15vD+A0YHPg8Jp3J+B3I+lDdCzfe8RoliDSWZsDd0n6s6RngB8Du/TLswvwg/r4IuCtdTfHXYAfS/qnpL9Qdn3cHJhL2RZ4GWCe7aUpH6j/0e6bWURj+d4jRq0Ekc5aA7in4fm9NW3APJKeBWYDLxri3PMpH7JXAScCHwbOlTSnDeVfHGP53iNGrfSJdDlJs4EdAGy/EPg08G7bpwMvBL4m6fphLGLbjOV7jxgpUhPprPuAtRqer1nTBsxTm2dWpHQyN3PuMcAJlL6CXwP7AcctmaIvtrF87xGjVoJIZ90ErG97XdvPo3QWT+uXZxrlAxBgV+AaSb01/f22l7G9LrA+8Nu+k2yvD6wp6VpKP8F8oBdYro33syjG8r1HjFoJIh1U2/k/AlwB3AH8RNJtto+3vXPNdibwItt3AZ+gNNEg6TbgJ8DtwC+AQyXNa7j8CcBn6+MfAYdQPrhPae9dNWcs33vEaNbT29s73GXoKNszJG023OWIGE4TP3352PrDb3D3snsOdxGGx3Gzexbn9ME+O4elY73VSWe23wh8F3gG2EPSnbZXonxL3U7S/A7eRkTEmNfx5qzFmXQGHAFsT5kLcHBNmwKcmAASEdF5w9EnsjiTzvoml00A5tp+GbBW7VCNiIgOG47mrIEmjr1usDySnrXdN+nsS5RmrqeAfYCvUmoiQ7I9GZhcn77C9ozFuYExahXgkeEuRCwZl+29zszhLsNwMb8Zm+/lxf/cW2egxK6abCjpFmALANtvBu4HemxfQKmlHCHpwQHOmwpM7WBRR50MSIjRIu/lJWs4mrMWZ9IZNa2HUgP5AnAscBRwOnBY20odERH/ZjiCyOJMOuuzLzBd0mMsmFw2vz6OiIgO6XhzVu3j6Jt0Ng44q2/SGTBD0jTKpLNz66SzxyiBBgDbE4D9gW1r0snAdMqw3zE6ALwj0hwYo0Xey0vQmJtsGBERS06WPYmIiJYliERERMu6aojvWGV7HnBrQ9K7+vYeHyDvk5KW70jBhmD7JcA3Je1qexPgJZKm12M7Axv2X+4mRq9ufA8vjO39gSv7tmK2fQZwsqTbh7VgHZYg0h2ekrTJcBdiUdQ/rF3r002AzSgDIKiDJ/qPyIvRrevew03YHzAwC0DSB4a1NMMkQaQL2V4e+Dll977xwBRJP++XZ3XgAmAFyv/zIZKus70t8HnKvuR/Ag6Q9GS/c68Ffg9sVc89UNJvba8MnAW8FJgDTJb0B9tbsWDZ9V7gzZQVBi4DNgWOB5azvSVl1YHlKEHls8AfgHUlzbf9fOCP9fprU9ZYW7W+1gcl/XFxf3cxMnToPXwjsA2wEnBQPXcccBKwdT3/VEnfs70U8G3gLZTVMuZSRo5eZPtzwE6U9+3/AB8C3kt5D59n+yng9cB/AkfW9JdJ+mQty/7AZpI+Yntvyny259XyfbjftgZdJ30i3WE527fUn0uAp4F3S9qU8kfytToBs9GewBX129/GwC22V6FM0nxbPXcGZd+OgUyo536YEjig/OH+TtJGwGcoS9BA+cM5tOZ/E2VZGgDq+mifAy6QtImkCxqOzQZuoQQrgB1rmedShmF+VNKkev3vNPWbipFqON7DS0vanLJg67E17SBgtqTXAq8FPlg3OnsPMJGyKOw+lKDQ59uSXitJlECyo6SL6mvvVd/XTzXkvxh4d8Pz3YEf235lffzGek/zgL2G/K11gdREusO/NAXYHg+cWJd+mU9Za2w14IGGc24Czqp5fybpllpj2BD4jW0o34YG24P8RwCSfmV7hbrk/paUb2BIusb2i2yvAPwGONn2ecBPJd1br9+MCyh/WL+kzAf6Tv2W+gbgwobrLNPsBWNEGo738E/rvzMpAQLK/LKNbPc1ta5I2SlzS+DCuhr4A7Z/2XCdbWwfRZnMvDJwG3DpYDcq6WHbf7a9BXAnsAHlb+RQyvYWN9WyLwc8NNh1ukWCSHfai9LMM0nSXNt3A8s2Zqgf/m8GdgDOtn0y8DhwlaQ9mniN/hOIBp1QJOkk25dTlun/je13UL5pNmMa5cNkZcof2DXA84EnRmEbeizQiffwP+u/81jwWddDqeFe0ZjR9vYDXcD2spRa8GaS7rF9XP9yDuLHwPsozbOXSOqtNa0fSDq6ifO7RpqzutOKwEP1j28bBlhd0/Y6wIOSTgfOoPRN3AC80fZ6Nc/zbb98kNfYvebZklL9nw1cR61+294aeETS322/TNKtkr5M+fa4Qb9r/QN4wUAvUtuy+7ayvUzSPEl/B/5ie7f6Wj22N27qNxPdohPv4YFcARxSazfYfnnti/sN8F7bS9lejdJnAgsCxiO1hrxrw7UGfV8Dl1C2tNiDElAArgZ2tf3i+tor13vsagki3ek8YDPbt1LWERuow3lr4Pe2f0cJCKdIepgyouRHtv9AaQbo/4Hf5+l67mmUdmSA44BJ9dyTWLC+2eEu/kDpkPzPftf6JbBhbQ/ffYDXugDYu/7bZy/gINu/pzQf9N9zJrpbJ97DAzkDuB242aVN6XuUWsrFlG0pbgd+CNxM+fL0BGVxV1MC0E0N1zobOK2+r5drfBFJjwN3AOtI+m1Nu53Sn3NlLftVwOqLUPYRKcuexL+pI1uOlJR9V2LMsL28pCdtvwj4LaUD/IGFnTfWpU8kIqK4rA4geR7whQSQ5qQmEhERLUufSEREtCxBJCIiWpYgEhERLUsQiYiIliWIREREy/4/QkGqWGOhQKsAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "dark"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "len_test = sum(results_graham.values())\n",
    "\n",
    "def plot_results(results, bar_labels):\n",
    "  labels = ['false_positives', 'false_negatives']\n",
    "  labels_display = ['False positive', 'False negative']\n",
    "\n",
    "  def get_y(res):\n",
    "    return [res[l] / len_test * 100 for l in labels]\n",
    "\n",
    "  x = np.arange(len(labels))\n",
    "  width_sum = 0.7\n",
    "  width = width_sum / len(results)  # 0.35\n",
    "  fig, ax = plt.subplots()\n",
    "  rects = [ax.bar(x - width_sum / 2 + width / 2 + i * width, get_y(res), width, label=lab) for i, (res, lab) in enumerate(zip(results, bar_labels))]\n",
    "      \n",
    "  ax.set_ylabel('Ratio of emails')\n",
    "  ax.set_title('Error percentages')\n",
    "  ax.set_xticks(x)\n",
    "  ax.set_xticklabels(labels_display)\n",
    "  ax.legend()\n",
    "  for rec in rects:\n",
    "    ax.bar_label(rec, fmt='%.2f%%', padding=3)\n",
    "  ax.yaxis.set_major_formatter(mtick.PercentFormatter())\n",
    "    \n",
    "plot_results([results_bayes, results_graham], ['Bayes', 'Graham'])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "As the above plot shows, the unbiased Bayesian filter retains the $0$ false positive rate, and decreases the false negative rate from $13.15\\%$ to $12.96\\%$.\n",
    "\n",
    "One might ask: is this about $0.2\\%$ gain in effectiveness significant? In order to assess this, we have to bear in mind that spam filters are already very effective. Graham wrote:\n",
    "\n",
    "> [R]ecognizing that *last few percent* [emphasis mine] of spams got very hard.\n",
    "\n",
    "Another quote from the [announcement](https://cloud.google.com/blog/products/g-suite/ridding-gmail-of-100-million-more-spam-messages-with-tensorflow) of machine learning spam filters in Gmail written in TensorFlow:\n",
    "\n",
    "> [Gmail's filters] help block *more than 99.9 percent* of spam.\n",
    "\n",
    "That is, in the field of spam filtering, split-percent results are considered good, and removing the bias from Bayesian spam filtering seems to be such a result.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This constitutes an argument for Bayesian epistemology that is similar to the \n",
    "[Dutch book argument](https://plato.stanford.edu/archives/sum2024/entries/epistemology-bayesian/#AtteFounDutcBookArgu): \n",
    "a successful application of the Bayesian principles indicates that one should adhere to them."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Tuning the interestingness threshold\n",
    "\n",
    "Another enhancement possibility that comes to mind is to tune the interestingness threshold, i.e. how many words of an email we consider in the probability calculation as being the most interesting ones according to their word probabilities (this is $15$ for Graham).\n",
    "\n",
    "First, let's see what we'll get if we remove the threshold altogether:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "threshold: 9223372036854775807\n",
      "{'true_positives': 71, 'true_negatives': 369, 'false_positives': 0, 'false_negatives': 77}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'true_positives': 71,\n",
       " 'true_negatives': 369,\n",
       " 'false_positives': 0,\n",
       " 'false_negatives': 77}"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEICAYAAACeSMncAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAuLElEQVR4nO3de7ylc93/8dc2JgyhHLoxGKGkt0NGUhRSzoe6I+RMRCRF7pTTTaSUcldoSI6VQ9TQ5HCLO7mRGdG80S8q5Zhj4kbGzPz++H63Wba19qxZs4+z38/HYz/2Wtf1va7re82svT7X99w1c+ZMIiIiOjHfYGcgIiKGrwSRiIjoWIJIRER0LEEkIiI6liASEREdSxCJiIiOJYhERETH5h/sDER0wvYDwFuA6Q2bz5V08ODkaGiyfRywiqTdBjsvMW9KEInhbFtJ/z27RLbnl/RKj22jJE1vdUyTc8xR+jlhuwvokjSjP84f0Z+6MmI9hqNaEvlksyBiey9gP+C3wB7AGcBY4EVgRWAjYHvg4bpv7fr6SEkT6znO7Zm+57Vs3wjcAmwKrAbcAOwt6em6f33gVGB14K/AZyXd2HDszcDGwDrAGsACwLeB8cA04DRJJ9meDzii3tPiwPXAAZKetj0O+AuwF3ACMAb4lqQTbW8BTAS6gH8Bf5K0lu296/nGAk8AX5P0/Yb7OgL4HDATOAY4C1hV0v22FwBOBD5e83sF8DlJL9peEjgX2BCYAdwNbJTgOG9Lm0jMq94D/JlS5XVi3faJ+vqNwG3AlcC1wNLAZ4CLbL+94RyN6X/T4jp7APsAywCvAP8FYHs54BfAV4A3A4cDP7W9VMOxuwP71/P/Hfhv4GpgWWAVSrCg5u0jlGC2LPAM8L0e+dgQeDsloB1j+x2SrgZOAi6WtIiktWrax4FtgEWBvYFv2V6n5nsL4PPAh2oeNu5xnZOBt1EC7yrAcpRAA3AY8BCwFOXf/UuUQBTzsFRnxXD2M9uN1VRfkHRWff2IpO/U16/YBvi5pJsBbK8NLAKcXJ+Uf2X7KmAX4Lh63KvpgZda5OECSa7nPBq40/aewG7AJEmTarrrbE8GtgLOq9vOlXR3PXYb4DFJ32y43m319QHAwZIeqmmPA/5me/eGfPynpBeBu2zfBawF3Nssw5J+0fD2f2xfC7wfuINSwvhhQ76OA3atr7soQW/NhtLWScCPgCMppadlgBUl3Q/c1OLfLOYhCSIxnH2klzaRB2ezbVngwR5VLX+lPFn3do7ezvlXYDSwJKUabEfb2zbsH02p8mp27PLAn1pcY0XgCtuNeZ1Oedrv9ljD6xcoAbIp21sCx1JKFPNRqsCm1t3LApNb5HGpmnZKDcpQqspG1denUALwtXX/BEknt8pHzBsSRGJe1awapXHbI8DytudrCCQrAH+czTl6Wr7h9QqUp/EnKV++F0jar808Pgjs3CLdg8A+DaWiV9U2kd685h5qm8ZPKdVwP5c0zfbPKMEA4FFKW0m3xvt7ktJO9E5JD/e8kKTnKFVah9kWpXR3u6Tre6aNeUfaRGKkuo3yxH6E7dG2Nwa2BX4yh+fZzfbqtscAxwOX1V5cFwLb2t7c9ijbC9re2PbYFue5CljG9qG2F7D9RtvvqfvOBE60vSKA7aVsb99m/v4OjKuN8wBvoDSIP0Gp5tsS2Kwh/SXA3rbfUe/p6O4dNdieRWlDWbrmZTnbm9fX29hepVZ7PUspLaVRfR6XIBLD2ZW2n2/4uaLdAyW9TAkaW1KesE8H9pD0hznMwwWUHkmPAQsCh9TzP0jpAfYlyhf2g8AXaPE3V5/iP1zz9BhwH7BJ3X0apZfVtbafA26ldBxox6X191O276jXOYQSLJ6hdB6Y2JCPX1I6B9wA3F+vBaV3F8B/dG+3/U9KZ4Duzgir1vfPU3qtnS6psfou5kHp4hvRodpN90JJZw92XvqL7XcABhboOdYmAtImEhE92P4oMInSiP414MoEkGgl1VkR0dOnKGNJ/kRp1zhwcLMTQ1mqsyIiomOpzoqI6EO2z6HMCPC4JPXYdxjwDWApSU82OfbrwNaUWqLrKFPlzLS9E/BlypicqyT9R03/GUrJ8W+UcVMv294Q+Jikz/XbTTYYcSUR209QBoVFRPSHRShdm1eizB/WbTQwjtKL717KNDmNFqaMy+nuIbgaZU63Fyjzr3UfMw54CniupvkDZaaAFyhdq1elTPnT1xOGrihpqZ4bR2JJ5K+S1h3sTETEvKsOAr2q8bvG9mXUQZ7Ah3qWRGy/F/guZQqaLuDXlPnVuqfn2bSm2x14r6RP274N+ABlBoJfU2YVWELSt/vhniY3256G9YiIflYHhz4s6a5WaSTdQhmf82j9uUbSvZRxOW+3Pc72/JTJOLtnEvguZSzPCpRZoffm9ZNz9quRWBKJiBgwdeT/l3jtzADN0q0CvINZ085cZ/v9km6yfSBwMaWa7H+BlQEkXUAZ8IrtYygDRbe0vQdlgOth/T0Vf0oiERH9a2VK+8hddR2cscAdtv+tR7qPArdKel7S88AvgfcCSLpS0nskvRf4f7x2jjdsLwusJ+lnlPnLdgL+QVkaoF+lJAJMmTJlaUqPidVIYI3WZlAaMQ8fP37844OdmRgeJE2lrFkDvLqg2rpNemf9DdjP9lcpbSIbURYpw/bSkh63/Sbg05Qp+xudwKx1XRaiTLw5gzJgtF8liBTfoNRF7jt+/Phpg52ZGJqmTJkymtLQ+Q1KA2nE69j+MWUxryVtPwQcK+kHLdKuS1ml8pPAZcAHKdPyzwSulnRlTXqa7e5FxY6X9MeGc7wLQNIdddOP6jkeBL7el/fWzEjs4ju5Z++sKVOm/BbYIAEkZqcGkpvHjx+/3mDnJWIgNfvuhFTddJsvASTaUT8n+buJqPLHEBERHUubSBPjvviLPq3je+Dkrbtml8b2dEo9ZhdlpOnBkv63L/MREdHXEkSGjhclrQ1QV4r7KqV3RkQMcWuct8aQb1yeuufU2T7MdiJBZGhalLLqHLYXoUyT8CbK3DtHSfq57eOBp7unN7B9ImXCt9Nsf4HSBXAB4ApJx9pemLKa3VjKJG4nSLp4gO8rIuYxCSJDx0K276RMzrYMpasfwEvARyX90/aSlGVJJwLnAJcD367rZ+8MrGd7M8oEbOtRqsYm2v4AZU6dRyRtDWB7sYG7tYiYV6Vhfeh4UdLaklYDtgDOt91FCQQn2f49Zf3q5YC3SHqAsm72uyjTKfxO0lP19WbA74A7KAMoV6W0t3zY9tfqVArPDvD9RcQ8KCWRIUjSLbXUsRSwVf09XtK0Otp1wZr0bGAv4N8oJRMoQeerkr7f87y216nn+4rt6yUd3683EhHzvJREhiDbq1HaLZ4CFqO0dUyzvQmwYkPSKyillncD19Rt1wD71LYUbC9ne+k6t84Lki4ETgHWGZi7iYh5WUoiTbTTJbcfdLeJQClN7Clpuu2LgCttTwUmM2vBGuoqZjcA/5A0vW671vY7gFtsAzwP7AasApxiewYwjaybHRF9IEFkiJA0qsX2J6kzefZUG9TXB3bsccxpwGk9kv+JWaWViIg+keqsYcr26pTFaq6XdN9g5yciRqZ+K4nM5WL1ewJH1bdfkXSe7QUo4yXGAqdLOr2mnQCc2TCD5Ygg6R7grYOdj4gY2fqzJHIupdH3NWwvT+mC+rdmB9l+M2W94PdQxjocW+fQ3xz4DbAmZTpu6tTIo0ZaAImIGCr6LYhI+jXwdJNd3wKOoMyX38zmwHWSnpb0DHAdJRhNoyywMprS8AxlIZaj+zLfERHRvgFtWG9crL72HGpmOcpiKt0eqtsupZRAbqX0MtoOuEPSI21cd39g//p2yQ6zHxGDqFkVue0TgO0pq/g9DuzV7DvB9tWUTii/kbRNw/ZzKXPUdQ++3UvSnbY/BhxPeRD+iKSnbK8MnCRpp/66x+FowIJIu4vVtyLpFeAT9VyjKT2Ntrd9KrACcL6kiS2OnQBMqMdO7uT6ETHozgW+C5zfsO0USUcD2D6EskTsAU2OPYVSk/GpJvu+IOmyHts+Qxl/9e+U753vAF9hVlttVANZEmlcrB5mLVa/nqTHGtI9TFlasttY4MYe5/o05YO0PuUJYifgV0DTIDLHjlusb2fkPO7ZdqaCfwulqm99yuSLLwNfl3RFO5ewPQ64qmcnhojZafGEfwqwLeVz+Cdgb0n/aHLsZ4H9KFXMZ3VPCFr3fQY4iLK0wS8kHWF7A+CMet5dJN1ne3HK5KBbSJrRKp+Sfl0/543b/tnwdmFaVJNLut72xq3/FV5nBmUC0zHANNvvBx5LT8jXG7AuvpKmSlpa0jhJ4yjVVOv0CCBQShib2X5TbVDfjIbxDXXbNpQgMobynz2Tsjj9sFTnyPoZ8GtJb5U0njKh4tge6TKuJ/rDuby+E8x1gCStCfwROLLnQbZFCSDrAWsB29hepe7bhFLNtJakd1J6YwIcRpl651BmlRiOolQTtQwgvbF9ou0HgV0pJZE5daLt39v+Vu0FCmUphv+mBNIfU9peT+gkf/O6/uzi29Fi9ZKervWct9fdx0tqbKA/BjhR0gzb11CedKYCZ/bXvQyADwIvS3r1HiT9FfiO7b0oRepFgFG2t6bJ1PD1sFG2zwLeRynRbS/pRdv7UdqE3kAZW7K7pBdqffCLwLuApYF9gD0ogxtvk7RXv951DAktnvCvbXh7K7BDk0PfQfmcvABg+38on9WvU2ZEOFnSv+r5Hq/HdHeQ6X7CXxlYXtKNc5H/LwNftn0kcDCld2e7jgQeo/xtTAD+g/Kdcx0lkGJ7D2AS8Dbbh1NqCj7bfd8jXb8FEUm7zGb/uIbXk4FPNrw/h1kTCvY87nMNr1+iwzaWIeadlBl3W1kHWLMG2PlpPjU8lNl6d5G0n+1LgI8BFwKXSzoLwPZXgH0pdbxQgtF7ge0o1YEbUP4vbre9tqQ7+/JGY1jaB2i29owpT/FLUB5GtqJMzQPwNuD9dZ2bl4DDJd1OecI/v6bfnVJC6at2hosoX/ZtBxFJj9aX/7L9Q+Dwxv21LXcvSq/RqyhBcgdKqeesuc/y8JcR60OQ7e/Zvst2d2nsuobSWNOp4eu+vzR86U8BxtXXsn1TnX9rV0rQ6nalpJmU0tzfa7XjDODuhuNjhLL9ZeAVyhf0a0i6F/gacC1wNXAnpf0DygPqmyltfF8ALrHdJelOSetL2oQyWPZRoMv2xbYvrG2Dc5K/VRvebk/D3HJtHr9M/d0FfIQSGBt9AfgvSdMoVeYzKVXoY+bkOvOy1LEPDXdTSg0ASDqoljK6n+r+ryHtrrSeGv5fDemmM6ud6FxKN8W7avXYxg3puo+Z0eP4GeTzMaLVz8o2wKb1QeN1ahX1D2r6kyhtndTfl9fjflsn/lwSeKKm7aKUQHamlIqPoDy0HAJ8uUV+XldFDmxl++2Uz+tfqe0sjVXk9f1NlLV1FqnH7ivpGuAi20tRHs7upKFnV535ej1J/1k3fYdSzf4PSsAJ8iUxVPyKUro4UNIZdVurJ53epoZv5Y3Ao7Vr9K6U9pKIlmxvQfli36i3un/bS0t63PYKlKqe9euunwGbADfYfhulzaFxiqM9gEm1ira7g0yvT/gtqsibtrM2qSJ/f4t0H2y2ve57BNi64f2llPFq0SBBpJk2uuT2JUkzbX8E+JbtIyhPa/9HaeTr2eus5dTwvTgauK2e9zZKUIkAWj7hH0np4npd7ZJ/q6QD6tP52ZK2qof/tLaJTAMOaugGfA5wjsvBL1OWNphZr9fdztDdnnkqpS3jZepYsBg+umbO7NshEUOd7cmS1m3cNmXKlMnjx49ft9UxEY3yeYme1jhvjSH/RTp1z6lz9XDc7LsT0rAeERFzIdVZETF09fXsEf1lpRUGOweDJiWRiIjoWIJIRER0LNVZESPQuC/+YlhUEz2w4OzTxOBKSSQiIjqWkkgTfd1dr52udbanU6Ye6aKMNj9Y0v/2ZT7ayMPxlJmE/7uXNBtTJouc67zZXhtYVtKk+n47YHVJJ8/tuSNiYCSIDB0vSlobwPbmlInqNhrIDEhqZxrtjYHngdcFEdvz18XD2rU2sC5loBl1UbG+WRMmIgZEgsjQtChlumlsL0KTqd9rqeHp7kWA6mypj0s6zfYXgI9TRhxfIelY2wtTFv4ZC4wCTpD0mplZ69TwV0m6rM7JdR5lPYXRwI6U2VgPAKbb3o2y+tu+dfu7gJttfw/4HmV+rxeA/ST9wfaOlJHQ0ykLiX2IsvzoQrY3pATNhYB1JR1c8/JPSpD5N+CImq/5KKvbfZCyjPI04JwmK9O9hu01gS2BJSQd0cb/QUS0IUFk6FjI9p2UyRSXoXxJQvmCbjb1+znA5cC36xfrzsB6tjejTAm/HqVqbKLtD1C+1B+RtDWA7cXayNOTktax/WnKVN6ftH0m8Lykb9Tz7EsJTO+TNN329ZSJ7+6z/R7g9HovxwCbS3rY9uKSXrZ9DDVo1HPt1eP6ywAbUibOmwhcRpmfaRywOmUNlHtpsmyA7TcCH6YEjg0p08NMokyN/zq2u9dSuYQSwD9Fmdvp6238O0WMWGlYHzpelLS2pNUoq8ydX2c6bTr1u6QHgKdsv4syB9HvJD1VX28G/I6yRslqlKAyFfiw7a/Zfr+kZ9vI0+X1d+O08s1cWgPIIpQFsS6tAfH7lEAAcDNwbl0ga1Qb1wb4maQZku5h1nT3G9brzairYt7Q4tj9KEFnFWBv4N8lnSWp1eST75S0MeXf64OUyQNH12AUES2kJDIESbqlljqWoiz002rq97MpE9n9G7OexruAr0r6fs/z2l6nnu8rtq+XdPxsstI9Nfx0ev+sdE9VPx/wj+62nR73dEAtmWwNTLE9fjbXbrw+lPtqm6RTbf+IEpA/D6xh+1bgJ3UK8FaeBt5erzegE3FGDEcpiQxBtlejPK0/Re9Tv19B+ZJ8N7PWob8G2KeWCrC9nO2l6+yrL0i6EDiFslpiJ56jxSzAkv4J/KW2f2C7y/Za9fXKkm6rjfdPAMv3dq5e3Ax8zPZ8dQGjjVsllPSYpHMlfRwQJdAu3SK5a1Xc85QS2I31HM/NYf4iRpSURJqY29kuO9TdJgLlCXjPWkXUcur32q5wA+Xpf3rddq3tdwC31Cm8nwd2o1TrnFIXB5pGWQO7E1cCl9nentKw3tOuwBm2j6I0yP8EuKtee9V6b9fXbX8Dvljv+6ttXv+nwKbAPZSG9TsoDfWvYfszlHaNnm4HLui5UdIPgR82bMq6ERFt6Lep4G2fQ1kV7XFJqttOofT2eRn4E7B3w/oDjcduAZxGeRo/u3vcQP1CXYPSg+hLddtRgCX9rM18zTNTwdcG9TuAHSXdN9j5GSi2F5H0fF3H4rfABrV9ZEAM189Lo+EzYn14LC+yxjCYgHE4TgV/LqWqpdF1gCStCfyRsvDNa9geRekiuiWlB84utlevXTRfrMe+2/ZiLusjv6fdADIvsb06cD9w/UgKINVVtfRyE6Wr8oAFkIh4rX6rzpL0a9vjemy7tuHtrcAOTQ5dD7hf0p8BbP8E2J6y3OZC9el7NKWx93jK2IMRp/ZYeutg52Mw1F5UETEEDGbD+j7AL5tsX45S193tIWA5SfdSGmTvoNTLrwLMJ+mO2V3I9v62J9ueDCzZJMmMKVOmjJ7TG4iRp35OZgx2PiKGikFpWLf9ZeAVynrhbZN0aMM5rgQ+Vc+1FnCdpLNaHDcBmFCPm9wkyR+A3adMmXLB+PHjp81JnmLkqAFkd9pb1z5iRBjwIFJHJW8DbCqpWePew5Tun93G1m2N59ieMgBuEWBlSR+3fY3tiyS90EG2Dge+ARwwZcqUdHuOVmZQAsjhg52RiKFiQINI7XV1BLBRL1/2twOr2l6JEjx2Bl7tomF7NHAoZdDaqkB3IBoFvIEyX9McGT9+/OOUKS8iImIO9NtTt+0fA7cAb7f9UJ1j6buUwWXX2b6zzsOE7WVtd8/k+gpwMGXQ3L3AJZLubjj1QcB5NQj9HhhTx1BMadZdOCIi+k9/9s7apcnmH7RI+whlOo7u95Oo04M3SfvthtczgWbXiYiIAZD6/4iI6FiCSEREdCxBJCIiOpYgEhERHUsQiYiIjiWIRERExxJEIiKiYwkiERHRsQSRiIjoWIJIRER0LEEkIiI6liASEREdSxCJiIiOJYhERETHEkQiIqJjCSIREdGxBJGIiOjYbIOI7R1tv7G+Psr25bbX6f+sRUTEUNdOSeRoSc/Z3hD4EGWJ2zP6N1sRETEctLPG+vT6e2tggqRf2P7K7A6yfQ6wDfC4JNVtbwYuBsYBDwAfl/RMk2P3BI6qb78i6TzbCwA/B8YCp0s6vaadAJwp6Y427iUiIvpQOyWRh21/H9gJmFS/zNs57lxgix7bvghcL2lV4Pr6/jVqoDkWeA+wHnCs7TcBmwO/AdYEdq9p1wJGJYBERAyOdoLBx4FrgM0l/QN4M/CF2R0k6dfA0z02bw+cV1+fB3ykyaGbA9dJerqWUq6jBKNpwBhgNNBV054AHN3GPURERD9oWZ1VSwTdbmzY9i9gcofXe4ukR+vrx4C3NEmzHPBgw/uH6rZLKSWQW4FTbG8H3CHpkdld1Pb+wP717ZId5j0iInrorU1kCjCTWU/9jWYCb52bC0uaaXvmHKR/BfgEgO3RlNLR9rZPBVYAzpc0scWxE4AJ9dhOA2BERPTQMohIWqkfrvd328tIetT2MsDjTdI8DGzc8H4stSTU4NPA+cD6wLOU9ppfAU2DSERE9I/eqrNWk/SHVmNCOmzMngjsCZxcf/+8SZprgJNqYzrAZsCRDfl6E6XX1+bAtsAMSslooQ7yExERc6G36qzPU9oRvtlk30zgg72d2PaPKSWKJW0/ROlxdTJwie19gb9SGu2xvS5wgKRPSnra9gnA7fVUx0tqbKA/BjhR0gzb1wAHAVOBM3u904iI6HNdM2e23SwxT7A9WdK6g52PiME07ou/GBZ/+A8s+InBzkJb1lhphcHOwmxN3XNqs/bttrX67mxnsCG2BawOLNi9TdL5c5OhiIgY/mYbRGwfS6mWWh2YBGxJGfSXIBIRMcK1M9hwB2BT4DFJewNrAYv1a64iImJYaCeIvChpBvCK7UUp3XKX799sRUTEcNBOm8hk24sDZ1EGID4P3NKfmYqIiOFhtkFE0qfryzNtXw0sKun3/ZutiIgYDtrtnbUmZfr2+ev7VSRd3o/5ioiIYaCd3lnnUKZfv5syOhzKYMMEkYiIEa6dksj6klbv95xERMSw007vrFtsJ4hERMTrtFMSOZ8SSB6jrCXSBcyUtGa/5iwiIoa8doLIDyiLQU1lVptIREREW0HkiVaLPUVExMjWThD5ne0fAVdSqrMASBffiIhoJ4gsRAkemzVsSxffiIhoa8T63gORkYiIGH5m28XX9ttsX2/b9f2ato/q/6xFRMRQ1844kbMoa5xPA6jzZu3cn5mKiIjhoZ02kTGSflsLIt1emZuL2v4c8ElK28pUYG9JLzXsX4AyPmU88BSwk6QHbG8AnAG8DOwi6b46w/AlwBZ1yvqIiBgg7ZREnrS9MuULH9s7AI92ekHbywGHAOtKEjCK15ds9gWekbQK8C3ga3X7YcBWwKHAAXXbUcBJCSAREQOvnZLIQcAEYDXbDwN/AXbtg+suZHsaMAZ4pMf+7YHj6uvLgO/a7qJUqY2pP9NqcFte0o1zmZ+IiOhAO72z/gx8yPbCwHySnpubC0p62PY3gL8BLwLXSrq2R7LlgAdr+ldsPwssAXyVUs31ImUU/TcoJZGIiBgEba0nAiDp//rigrbfRClprAT8A7jU9m6SLmwjD3cC69fzfIBSrdZl+2JKKeUwSX9vcs39gf3r2yX74DYiIoL22kT62oeAv0h6QtI0yqDF9/VI8zB1HXfb8wOLURrYqdu6KCWQE4BjgSMovcgOaXZBSRMkrStpXeDJvr2diIiRq2UQsb1j/b1SH1/zb8D6tsfUYLApcG+PNBOBPevrHYBfSZrZsH8PYJKkpyntIzPqz5g+zmtERPSit5LIkfX3T/vygpJuozSW30Hp3jsfMMH28ba3q8l+ACxh+37g88AXu4+3PQbYC/he3XQqMAn4NnBmX+Y1IiJ611ubyFO2rwVWsv26WXwlbdfkmLZIOpZSDdXomIb9LwE7tjj2BWCThvc3AWt0mpeIiOhcb0Fka2Ad4ALgmwOTnYiIGE5aBhFJLwO32n6fpCdsL1K3Pz9guYuIiCGtnd5Zb7H9O+Bu4B7bU2yrn/MVERHDQDtBZALweUkrSlqBMvXIhP7NVkREDAftBJGFJd3Q/aZOMbJwv+UoIiKGjXZGrP/Z9tGUBnaA3YA/91+WIiJiuGinJLIPsBRlZPlPKdOG7NOfmYqIiOGhnQkYn6HFdCIRETGyDcbcWRERMY9IEImIiI4liERERMdm2yZieyzwHWBDyhK5NwGflfRQP+ctIiKGuHZKIj+kTM2+DLAscGXdFhERI1w740SWktQYNM61fWg/5SciIoaRdoLIU7Z3A35c3+9CwyqDERExcrU72PDjwGOUNc13APbuz0xFRMTw0M5gw78CHS9AFRER866WQcT2EZK+bvs7lF5ZryEpo9gjIka43koi99bfkwciIxERMfz0trLhlfXlC5Iubdxnu+n65+2yvThwNiBKKWcfSbc07O8CTgO2Al4A9pJ0h+23Az8CRgOfknSL7fmBq4Ht6vrrERExQNppWD+yzW1z4jTgakmrAWsxq9TTbUtg1fqzP3BG3f4p4LOU4HJ43XYgcGECSETEwOutTWRLypf1crb/q2HXosArnV7Q9mLAB4C94NW13F/ukWx74HxJMynrvC9uexlgGjCm/kyrJZptgS06zU9ERHSutzaRRyjtIdsBUxq2Pwd8bi6uuRLwBPBD22vVc39W0v81pFkOeLDh/UN12/eA84EFKKWSo4GTJM3o7YK296eUaKCshxIREX2gtzaRu4C7bP9I0rQ+vuY6wGck3Wb7NOCLlIDQK0l/AzYGsL0KMBa41/YFwBuAoyX9sclxE6jrwttOR4GIiD7Szoj1cba/CqwOLNi9UdJbO7zmQ8BDkm6r7y+jBJFGDwPLN7wfW7c1OhE4irJg1tnAA8BJwK4d5isiIuZQuxMwnkFpB9mEUp10YacXlPQY8GDtaQWwKXBPj2QTgT1sd9leH3hW0qPdO21vBDwi6T5K+8iM+jOm03xFRMSca6ckspCk62131dHrx9meAhwzF9f9DHCR7TcAfwb2tn0AgKQzgUmURv37KV18X51mpXb/PQrYqW6aAFxU7+XAuchTRETMoXaCyL9szwfcZ/tgSrXSInNzUUl3Auv22Hxmw/6ZwEEtjp0JfLjh/b2UNpaIiBhg7VRnfZZSTXQIMB7YHdijPzMVERHDQzsTMN5eXz5PqXYaBewM3Nb6qIiIGAl6G2y4KKVKaTlKQ/d19f1hwO8p7RARETGC9VYSuQB4BrgF+CTwJaAL+Ght04iIiBGutyDyVklrANg+m7Ig1QqSXhqQnEVExJDXW8P6q6PUJU2nDBBMAImIiFf1VhJZy/Y/6+suYKH6vguYKWnRfs9dREQMab3NnTVqIDMSERHDTzvjRCIiIppKEImIiI4liERERMcSRCIiomMJIhER0bEEkYiI6FiCSEREdCxBJCIiOpYgEhERHUsQiYiIjiWIREREx9pZY71f1BUSJwMPS9qmx74FgPMpy/E+Bewk6QHbGwBnAC8Du0i6z/biwCXAFpJmDOQ9RESMdINZEvkscG+LffsCz0haBfgW8LW6/TBgK+BQ4IC67SjgpASQiIiBNyhBxPZYYGvg7BZJtgfOq68vAza13UVZ42RM/Zlme2VgeUk39m+OIyKimcGqzvo2cATwxhb7lwMeBJD0iu1ngSWAr1KquV4Edge+QSmJ9Mr2/sD+9e2Sc5PxiIiYZcCDiO1tgMclTbG98ZwcW9d2X7+e5wOUJXu7bF9MKaUcJunvTY6bAEyox02eqxuIiIhXDUZ11gbAdrYfAH4CfND2hT3SPAwsD2B7fmAxSgM7dVsXpQRyAnAspVRzFnBIf2c+IiJmGfAgIulISWMljQN2Bn4labceySYCe9bXO9Q0Mxv27wFMkvQ0pX1kRv0Z06+Zj4iI1xi0Lr492T4emCxpIvAD4ALb9wNPU4JNd7oxwF7AZnXTqcAkSrffTwxkniMiRrpBDSK1V9WN9fUxDdtfAnZsccwLwCYN728C1ujPfEZERHMZsR4RER1LEImIiI4liERERMcSRCIiomMJIhER0bEEkYiI6FiCSEREdCxBJCIiOpYgEhERHUsQiYiIjiWIRERExxJEIiKiYwkiERHRsQSRiIjoWIJIRER0LEEkIiI6liASEREdSxCJiIiOJYhERETHBnyNddvLA+cDbwFmAhMkndYjTRdwGrAV8AKwl6Q7bL8d+BEwGviUpFtszw9cDWxX11+PiIgBMhglkVeAwyStDqwPHGR79R5ptgRWrT/7A2fU7Z8CPksJLofXbQcCFyaAREQMvAEviUh6FHi0vn7O9r3AcsA9Dcm2B86XNBO41fbitpcBpgFj6s8024sD2wJbDOAtRERENeBBpJHtccC7gNt67FoOeLDh/UN12/coVWELUEolRwMnSZoxm+vsTynRACw51xmPiAhgEIOI7UWAnwKHSvpnO8dI+huwcT1+FWAscK/tC4A3AEdL+mOT4yYAE+pxk/vkBiIiYnB6Z9keTQkgF0m6vEmSh4HlG96PrdsanQgcBRwCnA0cARzb97mNiIhWBjyI1J5XPwDulXRqi2QTgT1sd9leH3i2tqV0n2Mj4BFJ91HaR2bUnzH9m/uIiGg0GNVZGwC7A1Nt31m3fQlYAUDSmcAkSg+s+yldfPfuPrgGoaOAneqmCcBFlHs5sP+zHxER3Qajd9ZvgK7ZpJkJHNTLvg83vL8XWKcv8xgREe3JiPWIiOhYgkhERHQsQSQiIjqWIBIRER1LEImIiI4liERERMcSRCIiomMJIhER0bEEkYiI6FiCSEREdGxQ1xOZl9jegrKk7yjgbEkn99i/AGUtlPHAU8BOkh6o+44E9gWmA4dIusb2UsAVwOLAUZJ+VtP+HDhQ0iPzaj6HQx4jokhJpA/YHkVZMGtLYHVglyZL/u4LPCNpFeBbwNfqsasDOwPvpKzQeHo93y7AmcB6wKE17bbA7+YigAz5fA6HPEbELAkifWM94H5Jf5b0MvATyhK/jbYHzquvLwM2rTMSbw/8RNK/JP2FMnPxesxaCngBYLrt+SlfgF+fx/M5HPIYEVWCSN9otZxv0zSSXgGeBZbo5dgfUb4UrwNOAj4NXCDphXk8n8MhjxFRpU1kiJL0LLA1gO03AV8EPmr7LOBNwDcl3TKIWQSGRz6HQx4jhquURPpGO8v5vpqmVqcsRmkUbufYoynLAe8C/AbYEzhuHs3ncMhjRFQJIn3jdmBV2yvZfgOlcXdijzQTKV9YADsAv6oLbE0Edra9gO2VgFWB33YfZHtVYKykG5m1FPBMYKF5NJ/DIY8RUSWI9IFaL38wcA1wL3CJpLttH297u5rsB8AStu8HPk+pUkHS3cAlwD3A1cBBkqY3nP5E4Mv19Y8pSwDfTukCO8/lczjkMSJm6Zo5c+Zg52FA2Z4sad3BzkfEYBr3xV8Miz/8Bxb8xGBnoS1rrLTCYGdhtqbuObXXZclnp9V356A0rHc6mMz2BsAZwMvALpLus7045elzC0kzBvA2IiJGvAGvzpqbwWTAYcBWlD7+B9RtRwEnJYBERAy8wWgTmZvBZN2DxsYA02yvDCxfG0ojImKADUZ1VrMBYe9plUbSK7a7B5N9lVLN9SKwO/ANSkmkV7b3B/avb99ue/Lc3MAItSTw5GBnIvrGVbutOGWw89AOc3N/nLbPP8s/7suT9ZM++N5bsdnGYTXYUNKdwPoAtj8APAp02b6YUko5TNLfmxw3AZgwgFmd56RDQswr8lnuW4NRnTU3g8mo27ooJZATgGOBI4CzgEP6LdcREfE6gxFE5mYwWbc9gEmSnmbWoLEZ9XVERAyQAa/Oqm0c3YPJRgHndA8mAyZLmkgZTHZBHUz2NCXQAGB7DLAXsFnddCowidLtd3h0Kh+eUh0Y84p8lvvQiBtsGBERfSfTnkRERMcSRCIiomPDqovvSGV7OjC1YdNHutcUb5L2eUmLDEjGemF7WeC/JO1ge21gWUmT6r7tgNV7TncT867h+BmeHdt7Add2L7Fs+2zgVEn3DGrGBliCyPDwoqS1BzsTc6L+Ye1Q364NrEvpAEHtPNGzR17M24bdZ7gNewEGHgGQ9MlBzc0gSRAZhmwvAvycsirfaOAoST/vkWYZ4GJgUcr/84GSbrK9GfCflPXG/wTsLen5HsfeCNwFbFSP3UfSb22/GTgHeCvwArC/pN/b3ohZ06nPBD5AmWHgKmAd4HhgIdsbUmYdWIgSVL4M/B5YSdIM2wsDf6jnX4Eyx9pS9Vr7SfrD3P7bxdAwQJ/h24BNgMWBfeuxo4CTgY3r8d+T9H3b8wHfBT5ImS1jGqXn6GW2jwG2pXxu/xf4FPAxymf4ItsvAu8FfgkcXrevLOkLNS97AetKOtj2bpTxbG+o+ft0j+UKhp20iQwPC9m+s/5cAbwEfFTSOpQ/km/WAZiNPgFcU5/+1gLutL0kZZDmh+qxkynrcTQzph77aUrggPKH+ztJawJfokxBA+UP56Ca/v2UaWkAqPOjHQNcLGltSRc37HsWuJMSrAC2qXmeRumG+RlJ4+v5T2/rXyqGqsH4DM8vaT3KhK3H1m37As9KejfwbmC/uoDZvwPjKJPC7k4JCt2+K+ndkkQJJNtIuqxee9f6uX6xIf1PgY82vN8J+Intd9TXG9R7mg7s2uu/2jCQksjw8JqqANujgZPq1C8zKHONvQV4rOGY24FzatqfSbqzlhhWB262DeVpqNXa4j8GkPRr24vWKfc3pDyBIelXtpewvShwM3Cq7YuAyyU9VM/fjospf1g3UMYDnV6fUt8HXNpwngXaPWEMSYPxGb68/p5CCRBQxpetabu7qnUxygqYGwKX1tnAH7N9Q8N5NrF9BGUw85uBu4ErW92opCds/9n2+sB9wGqUv5GDKMtb3F7zvhDweKvzDBcJIsPTrpRqnvGSptl+AFiwMUH98v8AsDVwru1TgWeA6yTt0sY1eg4gajmgSNLJtn9Bmab/ZtubU5402zGR8mXyZsof2K+AhYF/zIN16DHLQHyG/1V/T2fWd10XpYR7TWNC21s1O4HtBSml4HUlPWj7uJ75bOEnwMcp1bNXSJpZS1rnSTqyjeOHjVRnDU+LAY/XP75NaDK7pu0Vgb9LOgs4m9I2cSuwge1VapqFbb+txTV2qmk2pBT/nwVuoha/bW8MPCnpn7ZXljRV0tcoT4+r9TjXc8Abm12k1mV3L1F7laTpkv4J/MX2jvVaXbbXautfJoaLgfgMN3MNcGAt3WD7bbUt7mbgY7bns/0WSpsJzAoYT9YS8g4N52r5uQauoCxpsQsloABcD+xge+l67TfXexzWEkSGp4uAdW1Ppcwj1qzBeWPgLtu/owSE0yQ9QelR8mPbv6dUA/T8wu/2Uj32TEo9MsBxwPh67MnMmt/sUBe/pzRI/rLHuW4AVq/14Ts1udbFwG71d7ddgX1t30WpPui55kwMbwPxGW7mbOAe4A6XOqXvU0opP6UsS3EPcCFwB+Xh6R+UyV1NCUC3N5zrXODM+rleqPEikp4B7gVWlPTbuu0eSnvOtTXv1wHLzEHeh6RMexKvU3u2HC4p667EiGF7EUnP214C+C2lAfyx2R030qVNJCKiuKp2IHkDcEICSHtSEomIiI6lTSQiIjqWIBIRER1LEImIiI4liERERMcSRCIiomP/H5qRulLv//IjAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "dark"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "def tune_intr_threshold(t, t_display=None, plot=True):\n",
    "    print(f'threshold: {t}')\n",
    "    if t_display is None:\n",
    "        t_display = t\n",
    "    results = evaluate_bayesian(word_probs_bayes, intr_threshold=t)\n",
    "    print(results)\n",
    "    if plot:\n",
    "        plot_results([results_bayes, results_graham, results],\n",
    "                     ['Bayes', 'Graham', f'Bayes intresting ≤ {t_display}'])\n",
    "    return results\n",
    "\n",
    "tune_intr_threshold(sys.maxsize, '∞')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "That's worse than setting Graham's threshold.\n",
    "\n",
    "But what if we decrease it instead?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "threshold: 1\n",
      "{'true_positives': 131, 'true_negatives': 357, 'false_positives': 12, 'false_negatives': 17}\n",
      "threshold: 2\n",
      "{'true_positives': 122, 'true_negatives': 364, 'false_positives': 5, 'false_negatives': 26}\n",
      "threshold: 3\n",
      "{'true_positives': 114, 'true_negatives': 368, 'false_positives': 1, 'false_negatives': 34}\n",
      "threshold: 4\n",
      "{'true_positives': 109, 'true_negatives': 368, 'false_positives': 1, 'false_negatives': 39}\n",
      "threshold: 5\n",
      "{'true_positives': 107, 'true_negatives': 368, 'false_positives': 1, 'false_negatives': 41}\n",
      "threshold: 6\n",
      "{'true_positives': 103, 'true_negatives': 368, 'false_positives': 1, 'false_negatives': 45}\n",
      "threshold: 7\n",
      "{'true_positives': 98, 'true_negatives': 368, 'false_positives': 1, 'false_negatives': 50}\n",
      "threshold: 8\n",
      "{'true_positives': 96, 'true_negatives': 369, 'false_positives': 0, 'false_negatives': 52}\n",
      "threshold: 8\n",
      "{'true_positives': 96, 'true_negatives': 369, 'false_positives': 0, 'false_negatives': 52}\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEICAYAAACeSMncAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAswklEQVR4nO3dd5zcVb3/8dcSWiICUi8dFBTxTZEgoqCABelYQKQ3RRBFEETQUAQpNhAVwVAuXToYECmXckUEJIEgb8DHD1Sk96LcgIQkvz/OWTIsu5vJZEtm9/18PPaxM996Jpmdz5z2OR3Tpk0jIiKiFXMMdgEiIqJ9JYhERETLEkQiIqJlCSIREdGyBJGIiGhZgkhERLQsQSQiIlo252AXIKIVth8GFgemNGw+U9LXB6dEsyfbRwArStpxsMsSQ1OCSLSzLST9z4wOsj2npDe6bBshaUpP53RzjZk6fmbY7gA6JE3tj+tH9KeOzFiPdlRrIl/uLojY3hX4CvBnYGfgZGBp4FVgOWB9YCvg8bpvjfr4EEnj6jXO7Hp813vZvhm4DfgksDJwE7CbpBfq/nWA44FVgH8C35R0c8O5twIbAGsCqwLzAD8DRgOTgRMlHWN7DuCg+poWBG4A9pL0gu3lgX8AuwJHAaOAEyQdbXtjYBzQAfwH+Juk1W3vVq+3NPAs8ENJv254XQcB+wPTgMOAU4GVJD1kex7gaOCLtbyXA/tLetX2IsCZwHrAVOA+YP0Ex6EtfSIxVH0Y+Dulyevoum37+vidwB3AlcB1wGLAN4DzbL+v4RqNx/+xh/vsDOwOLAG8AfwcwPZSwO+AHwALAQcCl9petOHcnYA96/WfBv4HuAZYEliREiyoZfssJZgtCbwInNSlHOsB76MEtMNsv1/SNcAxwIWS5pO0ej32GWBzYH5gN+AE22vWcm8MfAv4VC3DBl3ucxzwXkrgXRFYihJoAA4AHgMWpfy7f5cSiGIIS3NWtLMrbDc2U31b0qn18ROSflEfv2Eb4LeSbgWwvQYwH3Bc/aZ8o+2rgO2AI+p5bx4PvNZDGc6R5HrNQ4GJtncBdgSulnR1Pe562+OBTYGz6rYzJd1Xz90ceErSTxvud0d9vBfwdUmP1WOPAB6xvVNDOb4v6VXgHtv3AKsDD3RXYEm/a3j6v7avAz4G3EWpYfx3Q7mOAHaojzsoQW+1htrWMcD5wCGU2tMSwHKSHgJu6eHfLIaQBJFoZ5/tpU/k0RlsWxJ4tEtTyz8p36x7u0Zv1/wnMBewCKUZbBvbWzTsn4vS5NXducsAf+vhHssBl9tuLOsUyrf9Tk81PJ5ECZDdsr0JcDilRjEHpQns3rp7SWB8D2VctB47oQZlKE1lI+rjH1MC8HV1/1hJx/VUjhgaEkRiqOquGaVx2xPAMrbnaAgkywL/bwbX6GqZhsfLUr6NP0f58D1H0leaLOOjwJd6OO5RYPeGWtGbap9Ib97yGmqfxqWUZrjfSpps+wpKMAB4ktJX0qnx9T1H6Sf6gKTHu95I0r8pTVoH2BaldnenpBu6HhtDR/pEYri6g/KN/SDbc9neANgCuGAmr7Oj7VVsjwKOBC6po7jOBbaw/RnbI2zPa3sD20v3cJ2rgCVs72d7HtvvtP3huu8U4GjbywHYXtT2Vk2W72lg+do5DzA3pUP8WUoz3ybARg3HXwTsZvv99TUd2rmjBttTKX0oi9WyLGX7M/Xx5rZXrM1eL1NqS+lUH+ISRKKdXWn7lYafy5s9UdLrlKCxCeUb9q+AnSX9dSbLcA5lRNJTwLzAvvX6j1JGgH2X8oH9KPBtevibq9/iP13L9BTwILBh3X0iZZTVdbb/DdxOGTjQjIvr7+dt31Xvsy8lWLxIGTwwrqEcv6cMDrgJeKjeC8roLoDvdG63/S/KYIDOwQgr1eevUEat/UpSY/NdDEEZ4hvRojpM91xJpw12WfqL7fcDBubpOtcmAtInEhFd2P4ccDWlE/2HwJUJINGTNGdFRFdfpcwl+RulX2PvwS1OzM7SnBURES1Lc1ZEtAXbZ1Bm2j8jSXXbUZQBDFMptaddJT3RzbnXAOsAf5S0ecP2MymZAF6um3aVNNH2Fyij7V6gzEd63vZ7gGMkbdtfr7EdDbuaiO1nKZPCIqK9zEcJFitQ8nJBaZLvHEa8GGWE3CPdnPvOeuyilNFlnZanBJAXuxz/PsoIuQUpX7afqfd9gukj1Yab5SQt2nXjcKyJ/FPSWoNdiIiYeXVy5VXd/Q3bPgRYVlK3fTh1LtCB3dRErpJ0SZdjbwS+AGxDCT73AZ+XtH/fvJL2U9P2vM1wDCIR0aCHZqIfU+asvE7pYN9N0kvdnPtNSnbhDuBUST9r2PcNYB9K5/zvJB1ke11K5uTXge0kPWh7Qcq8lY1byfhr+2jKDPyXmT63ZmYcbfswSsLLgyX9BziWMuflCUoetIvpOaPAsJbRWRFxJrBxl23XA5K0GiUVzCFdT6qpTb4CrE1J+Li57RXrvg0pfRWrS/oA8JN62gGUJJT7URJLAoyh9DW0NLtd0vckLQOcB8zsomSHUNL4f4iSbfk79ZrXSxotaYv6Oq4G3mv7Etun1tn8QYJIxLAn6Q+UDuTGbdc1zA25nbfm0+r0fuAOSZPqsf8LfL7u25uSIfk/9XrP1O2TKfNPRgGTa2f1Mp3rrMyi8yhNUE2T9KSkabWc/00JiG+qwWJXSur97wO7UJYF2KEPyjskpDkLmDBhwmKUb0ork8AabzcV+Ctw4OjRo5+Z0cFD0O7Ahd1sN6UpaGFKYsZNmZ4B+L3Ax2pT02uUvog7Kc1EZ9fjd6L83Y1ptWC2V5L0YH26FeX/aWbOX0LSkzXf12fra2r0beDnNVHlSEpCy6mUIBgkiHT6CSVX0B6jR4+ePNiFidnLhAkT5mL6B97Og1ycAWX7e5TFts7ruk/SA7Z/SFnY6/+AiUxf835OSvPQOpSmootsv1vSxLoN2x+nZA3usH0hpZZygKSneyjLbyiLZC1i+zFKOvtN60JiUymjLveqx65FWf3xy/X5LZQvifPVc/eQdC1lIbJFKX06E5nexIbtJYG1JX2/bvoFcCfwEiXgBAkinVYmASR6MHr06MkTJkw4h4YPmOHAZZnhzYFPSup2LoCk04HT6/HHUFY2pP6+rJ7357oWyiKUZJSdC1yNoXRW/4KyXO/ylOSQ3+vhXtt1s/n0Ho4dD3y54fnHejjuE91tr/ueADZreH4x0xNaRpWmm2KOBJDoTX1/DJu/l7pM7kHAlpIm9XJcZ0r4ZSn9IefXXVdQR0rZfi8lBf1zDafuTFn58QVK09BU0kzUllITiRjmemgmOoSy7sj1dZXC2yXtVZt4TpO0aT390tonMhnYp2EY8BnAGS4nvw7s0lmbaeis7lzH5HjK6KfXKanpo40kiHRj+YN/16fT+B8+brOOGR1jewplidIOSrvy1yX9qS/LEdGdmWwmeoLSgd75vKdmotcp8yu62zeJhvkckm4BVp2JIsdsJEFk9vGqpDUA6kpxx1Jy+kQMX0cs0B55mY54eYZfFIeqBJHZ0/zUXD625wN+C7wLmAsYI+m3to8EXuicIVyHUj4j6UTb3wa+SGmOuFzS4bbfQZkVvDQwAjhKUnfDNiMimpYgMvsYaXsiJYHcEkDnqJHXgM9J+pftRSjLko6jtDlfBvysrp/9JWBt2xtRlildm9I0Nq4OpVwUeELSZgC2Fxi4lxYRQ1WCyOyjsTnrI8DZNa1EB3BMDQRTgaWAxSU9bPt52x8EFgfurumqN6J0WN5drzsfJajcAvy0juu/qrZDxzDV1/1+/eXheQe7BDEjCSKzIUm31VrHopROzEWB0XXW7MOU2grAaZRRLv9FqZlACTrHSvp11+vaXrNe7we2b5B0ZL++kIgY8obNuPd2YntlSr/F88AClL6OyTWp3XINh15OSZz3IeDauu1aYPfal4LtpWwvVodmTpJ0LvBjYM2BeTURMZSlJtKNZobk9oPOPhEotYldJE2xfR5wpe17KXmJ3swNJOl12zcBL0maUrddZ/v9wG11fP8rlKGWKwI/rjOHJ5N1syOiDySIzCYkjehh+3PAR7rbVzvU16EsnNN4zonAiV0O/xvTaysREX0izVltyvYqlGU+b2jIYhoRMaBSE2lTku4H3j3Y5YiI4S01kYiIaFmCSEREtCxBJCIiWpYgEhERLUvHenf6OnNoExk+bS8OnEAZsvsiZW2FH0m6vJlb2F6eks5Es1DSiIiZ0m9BxPYZlKU1n+n8YLP9Y2ALygfk34DdGhaxaTx3Y8o8hxGUBXCOq9vPo6w7cJWk79ZtYwBLuqK/Xkt/q0uFXgGcJWn7um05YMsux80p6Y2BL2FERPf6syZyJvBL4OyGbdcDh0h6oyYCPAT4TuNJtkcAJwGfpqzTfGfNWjsnJUnharavr1loRwEflvSDfnwdA+ETwOuSTuncIOmfwC/qOtefpyRSHGF7M7pJDV9PG2H7VOCjwOPAVpJetf0VYE/KEqUPATtJmmT7TOBV4IPAYsDulGVLPwLcIWnXfn3VEdH2+q1PRNIfgBe6bLuu4Zv07ZS1LbpaG3hI0t/r6mgXAFtRUnWMrLO056Ks/nckZSnPdvcB4K5e9q8JbC1pfaanhl+TsjrcT2tNBkq23pMkfQB4CfhC3X6ZpA9JWh14ANij4drvogSN/YFxlCa1DwCr2l6jD15bRAxhg9knsjvQ3aJISwGPNjx/jFLbeMD2s5QP23MouaDmkNTbhy8AtvekfBMHWGSWSj0AbJ8ErEdp9jsJuF5SZ0DuNjV83fcPSRPr4wnA8vWxbP8AWJBSo2lMf3KlpGk1N9fTku6tZbivnj+RiIgeDEoQsf094A3gvJk5T9J+Dde4EvhqvdbqlA/aU3s4bywwtp43vsVi96f7mF5rQNI+NRV8Z1n/r+HYHeg5Nfx/Go6bAoysj88EPivpnto8tkHDcZ3nTO1y/lQy8CIiZmDAh/jWD7HNgR0kdTcK6nFgmYbnS9dtjdfYivJNez7gPZK+CGxte1S/FLr/3QjMa7sxs25Pr6W31PA9eSfwpO25KEEoIqJPDOg3zTrq6iBgfUmTejjsTmAl2ytQgseXgO0brjEXsB+wGaUPoDMQjaB0HPd03eY1MSS3L9XmpM8CJ9g+CHiWUvv4DtNrE516TA3fi0OBO+p176AElYiIWdYxbVr/rJJp+zeUZpNFgKcpHeCHAPNQFlsCuF3SXnXBpNMkbVrP3RT4GSUwnCHp6Ibr7kdZP+PM2qF8PiDgaklvGenVQ7nGS1qrcduECRPGjx49eq2ezomAofU+aZ/lcbef8UGzgwH+4jkYuvvshH6siUjarpvNp/dw7BOUZVs7n18NXN3DsT9reDwN6O4+ERExAJL2JCIiWpYgEhERLUsQiYiIliWIREREyxJEIiKiZZmR3I1Vz1q1T4c/3rvLvc2kgp8C3EtJazIF+LqkP/VlOZoow5HAHyT9Ty/HbEBJFjnLZau5uZaso/GwvSWwSmfW5oFk+0eUuUdzUBKFfrOHybAR0SBBZPbxqqQ1AGx/BjgWWH8gCyDpsCYO2wB4BXhbEGkhVf0awFrU4dySxlGSQA4o2x8F1gVWq5v+SPm3v3mgyxLRbhJEZk/zUxamwvZ8dJP6vdYaXuicN2P7aEo6lBNtfxv4ImVi5+WSDrf9DuAiShqZEcBRkt6SALOmhr9K0iU1J9dZlPVf5gK2oWQQ3guYYntH4BuUjMCvUdLJ31qTR55Eye81CfiKpL/a3oYy4XQK8DLwKUoW5pG216MEzZHAWpK+XsvyL0qQ+S/goFquOShLDHyCkqhzMmVC6iW9/YPaXg3YBFhY0kFddk+j5B+bm1ITnIsyQTYiZiB9IrOPkbYn2v4rcBpwVN3eU+r3Myhrf1A/WL8EnGt7I0o6mLUp3/RH14y/GwNPSFq9LhJ2TRNleq7e92TgQEkPA6cAJ0haQ9It9bilgY9K+hYl0eU3JI0GDgR+VY85DPhMTUe/ZU3zfxhwYb1Wdxmdl6BkM94c6Gzi+jwlu/AqwE6UNPZvY/udtj9v+1TbDwDfpyxNcGLXYyXdBtwEPFl/rpX0QBP/PhHDXmois4/G5qyPAGfbFj2kfpf0sO3nbX+Qkgr+bknP1yCyEXB3ve58lKByCyUA/ZBS27iFGbus/p5A+fDuycWSptRa00eBi2137pun/r4VONP2RQ3XnZErJE0F7q/LB0MJKhfX7U/ZvqmHc78C/AT4X2A3yiJb3fZx2F4ReD/T17e53vbHmvw3ihjWEkRmQ5Juq6ngF6Wkg+kp9ftpwK6U5p4z6rYO4FhJv+56Xdtr1uv9wPYNko6cQVE6U8NPoff3Smeq+jkoec3W6OY17WX7w5TO6wm2R8/g3o33h/K6mibpeNvnU2pg36IssnU7cIGka7sc/jlKHrdXAGz/nlLDSRCJmIE0Z82GbK9M6bd4nt5Tv19O+ZD8ENMXmroW2L3WCrC9lO3FapLLSZLOBX5MWS2xFf+mhyzAkv4F/KP2f2C7w/bq9fF7JN1RO++fpaT77/FavbgV+ILtOWrtZIOeDpT0lKQz61IBogTaxbo59BFgfdtz1izR61NWgIyIGUhNpBvNDMntByNtT6yPO4BdahNRj6nfJb1em3NekjSlbrvO9vuB22qT0ivAjpSVIH9seyqlM7px7ZKZcSVwSV3T5Rvd7N8BONn2GEoH9QXAPfXeK9XXdkPd9ghwcH3dxzZ5/0uBTwL3UzrW76J01L+F7W8AX+3m/DspK2M2uoTSUX8vpZP9GklXNlmeiLewfQalH++Z2v+I7YUoK7kuDzwMfFHSi92cuwswpj79gaSz6va5KQNKNqA0a39P0qUN7/NHKAvPvV4HqnxB0v799iIb9Fsq+NnVUEoFXzvU7wK2kfTgYJdnoNieT9IrthcG/gysK+mp/r5vu75PupNU8H2sIRV87b98BTi7IYj8iDKa8jjbBwPv6rp0RQ004ykjEqdR+iJHS3rR9veBEZLG1L/7hSQ9V5toPwp8l/LF7CrKoJntGpbU7hM9pYJPc1absr0K8BBww3AKINVVtfZyC2Wocr8HkIhmSfoDZSRgo60oQ+apvz/bzamfoSzz/UKtpVxPaa4G2J1aW5c0VdJzdXvnkPRRlBaGHYHf93UA6U2as9qUpPuBdw92OQaDpA0GuwwRM2lxSU/Wx09RRlR2tRSlibbTY8BSthesz4+qGSP+Rslo8TSliet24D5Kf+FvKcFowKQmUkydMGHCXINdiJh91ffH1MEuR7S/OtR8ZpoT56QMP/9Tnbd1G2X4OpLOkfRBSTsC+wM/BzaxfYntE2rTV79KECn+CuyUQBLdqe+LnWhuPfuI7jxtewmA+vuZbo55nDJqsdPSddvzlOwPnfOrLqbL6Mo6+nJtSVcABwDbAi9RBqH0qzRnFQdSIvteEyZMSGCNrqZSAsiBg12QaFvjgF0omRd2oTQ7dXUtZWLxu+rzjYBDJE2zfSVlZNaNTB+d2OgoSgYIKOmDplHet6P68DV0K0EEGD169DPUFCIREbPC9m8oH/iL2H6MkjPuOOAi23sA/6TktsP2WsBekr4s6QXbR1GGoQMc2dBB/h3gHNs/o8yz2q3hfh8EkHRX3XQ+Zbj6o8CP+ut1dsoQ34hhKEN8+1jDEN+hKkN8IyKiz/Vbc1Zfz9q0PQ+lHXFp4FeSflWPHQuc0lCVi4iIAdKfNZEzmT5RptPBlMlxK1FSXxzc9aQaaA4HPkxJZ3547Wj6DGWxoNUoI2WoeZlGJIBERAyOfquJSPqD7eW7bN6K6QnzzqKsHPedLse8OWsTwHbnrM2XKCMN5mJ6RtejKIskRUQMmr5eUrs/9FdOwIEendXyrE3K2OidKLMzf+yyHvddkp6Y0U1t7wnsWZ8u0mLZIyKii0Eb4lvHPjcdveva3dsD1HTd1wJb2T4eWJaS7Kzb9bkljaWsuIft8bNa9oiIKAZ6dNaszNps9DXgbGAdShrwbSmzNCMiYgANdBDpnLUJvc/a3Mj2u2qH+kZMX3CJum1zShAZRZmVOY0ySzMiIgZQvwWROmvzNuB9th+rMzWPAz5t+0HgU/U5tteyfRpA7VDvnLV5J2+dtQllav/RdY3ta4GPUWZndl1oKCIi+ll/js7aroddb0sIJmk88OWG52cwfc3wrsfu3/D4NUpNJSIiBkFmrEdERMsSRCIiomUJIhER0bIEkYiIaFmCSEREtCxBJCIiWpYgEhERLUsQiYiIls0wiNjexvY76+Mxti+zvWb/Fy0iImZ3zdREDpX0b9vrUVKVnA6c3L/FioiIdtBMEJlSf28GjJX0O2Du/itSRES0i2aCyOO2f01Jt351Xes8fSkREdFUMPgiJVvuZyS9BCwEfLs/CxUREe2hxyy+thdqeHpzw7b/AFkdMCIiek0FP4Gy2FN3i7tPA97dLyWKiIi20WMQkbTCQBYkIiLaT2/NWStL+mtPc0Ik3dV/xYqIiHbQW3PWt4A9gZ92s28a8Il+KVFERLSN3pqz9qy/Nxy44kRERDtpao112wJWAebt3Cbp7P4qVEREtIcZBhHbhwMbUILI1cAmwB+BBJGIiGGumcmGWwOfBJ6StBuwOrBAv5YqIiLaQjNB5FVJU4E3bM8PPAMs07/FioiIdtBMn8h42wsCp1ImIL4C3DYrN7W9P/Blyiive4HdJL3WsH8eSnPZaOB5YFtJD9tel5JB+HVgO0kP1rJdBGxcg11ERAyQGdZEJH1N0kuSTgE+DexSm7VaYnspYF9gLUkCRgBf6nLYHsCLklYETgB+WLcfAGwK7AfsVbeNAY5JAImIGHjNjs5aDVi+83jbK0q6bBbvO9L2ZGAU8ESX/VsBR9THlwC/tN0BdB4/Cphs+z3AMpJunoWyREREi5oZnXUGsBpwH9D5bX8a0FIQkfS47Z8AjwCvAtdJuq7LYUsBj9bj37D9MrAwcCylmetVYCfgJ5SayIxew56UiZMAi7RS7oiIeLtmaiLrSFqlr25o+12UmsYKwEvAxbZ3lHTujM6VNBFYp17n48CTQIftCym1lAMkPd3NeWOBsfW8ZCCOiOgjzYzOus12nwURyhK7/5D0rKTJlBrNR7sc8zh1BJjtOSlDip/v3FmbtsYARwGHAwdROv737cNyRkTEDDRTEzmbEkieoqwl0gFMk7Rai/d8BFjH9ihKs9Qnefv6JOOAXSijwLYGbpQ0rWH/zsDVkl6o15laf0a1WKaIiGhBM0HkdEr/w71M7xNpmaQ7bF8C3AW8AdwNjLV9JDBe0rh6z3NsPwS8QMPorRo0dgU2qpuOp8ykfx3YflbLFxERzWsmiDxbP9j7jKTDKc1QjQ5r2P8asE0P504CNmx4fguwal+WLyIimtNMELnb9vnAlZTmLABmcYhvREQMAc0EkZGU4LFRw7aWh/hGRMTQMcMgMiuz0yMiYmib4RBf2++1fYNt1+er2Z7hBL+IiBj6mpkncipwCGUyH5L+wttzXUVExDDUTBAZJenPXba90R+FiYiI9tJMEHmuJjqcBmB7a0q6kYiIGOaaGZ21DyXv1Mq2Hwf+AezQr6WKiIi20MzorL8Dn7L9DmAOSf/u/2JFREQ7aGo9EQBJ/9efBYmIiPbTTJ9IREREt3oMIra3qb9XGLjiREREO+mtJnJI/X3pQBQkIiLaT299Is/bvg5YwfbbsvhK2rL/ihUREe2gtyCyGbAmcA7w04EpTkREtJMeg4ik14HbbX9U0rO256vbXxmw0kVExGytmdFZi9u+G7gPuN/2BNvq53JFREQbaCaIjAW+JWk5ScsCB9RtERExzDUTRN4h6abOJ5JuBt7RbyWKiIi20cyM9b/bPpTSwQ6wI/D3/itSRES0i2ZqIrsDi1KWw70UWKRui4iIYa6ZBIwvAvsOQFkiIqLNNJ2AsS/ZXhA4DRBlnZLdJd3WsL8DOBHYFJgE7CrpLtvvA84H5gK+Kuk223MC1wBbSpo0sK8kImJ4G6wEjCcC10haGVgdeKDL/k2AlerPnsDJdftXgW9SgsuBddvewLkJIBERA2/AayK2FwA+DuwKb05qfL3LYVsBZ0uaRpnwuKDtJSjrvI+qP5NrjWYLYOOBKX1ERDSaYRCxvTTwC2A9StPTLcA3JT3W4j1XAJ4F/tv26sCEer3G9UqWAh5teP5Y3XYScDYwD6VWcihwjKSpM3gNe1JqNFAGBkRERB9opjnrv4FxwBLAksCVdVur5qTk5DpZ0geB/wMObuZESY9I2kDSRyh9JUsDD9g+x/aFtt/bw3ljJa0laS3guVkoe0RENGimOWtRSY1B40zb+83CPR8DHpN0R31+CW8PIo8DyzQ8X7pua3Q0MIYycuw04GHgGLL+e0TEgGmmJvK87R1tj6g/OwLPt3pDSU8Bj9aRVgCfBO7vctg4YGfbHbbXAV6W9GTnTtvrA09IepDSPzK1/oxqtVwRETHzmqmJ7E7pEzmB0ifyJ2C3WbzvN4DzbM9Nmf2+m+29ACSdAlxNGYH1EKXZ6s371eG/Y4Bt66axwHn1tew9i+WKiIiZ0Mxkw38CfboAlaSJwFpdNp/SsH8asE8P504DPt3w/AFKH0tERAywHoOI7YMk/cj2Lyg1kLeQlFnsERHDXG81kc4JgOMHoiAREdF+elvZ8Mr6cJKkixv32d6mX0sVERFtoZnRWYc0uS0iIoaZ3vpENqGMkFrK9s8bds0PvNHfBYuIiNlfb30iT1D6Q7akpCbp9G9g//4sVEREtIfe+kTuAe6xfb6kyQNYpoiIaBPNTDZc3vaxwCrAvJ0bJb2730oVERFtodkEjCdT+kE2pGTRPbc/CxUREe2hmSAyUtINQIekf0o6Atisf4sVERHtoJnmrP/YngN40PbXKdl05+vfYkVERDtopibyTUp23H2B0cBOwM79WaiIiGgPzSRgvLM+fIWSbXcE8CXgjp7PioiI4aC3yYbzUzLpLkVZ3+P6+vwA4C+U9OsRETGM9VYTOQd4EbgN+DLwXaAD+FxN5R4REcNcb0Hk3ZJWBbB9GvAksKyk1wakZBERMdvrrWP9zVnqkqZQ1kVPAImIiDf1VhNZ3fa/6uMOYGR93gFMkzR/v5cuIiJma73lzhoxkAWJiIj208w8kYiIiG4liERERMsSRCIiomUJIhER0bJmEjD2i5o+ZTzwuKTNu+ybh5JyfjTwPLCtpIdtr0tJS/86sJ2kB20vCFwEbCxp6kC+hoiI4W4wayLfBB7oYd8ewIuSVgROAH5Ytx9AWfd9P2Cvum0McEwCSETEwBuUIGJ7acqaJKf1cMhWwFn18SXAJ213UCZAjqo/k22/B1hG0s39W+KIiOjOYDVn/Qw4CHhnD/uXAh4FkPSG7ZeBhYFjKc1cr1JS0v+EUhPple09gT3r00VmpeARETHdgAcR25sDz0iaYHuDmTm3Jn5cp17n45R8Xh22L6TUUg6Q9HQ3540Fxtbzxs/SC4iIiDcNRnPWusCWth8GLgA+Ybvrmu2PA8sA2J4TWIDSwU7d1kGpgRwFHE6p1ZxKWTgrIiIGyIAHEUmHSFpa0vKUxa1ulLRjl8PGAbvUx1vXY6Y17N8ZuFrSC5T+kan1Z1S/Fj4iIt5i0Ib4dmX7SGC8pHHA6cA5th8CXqAEm87jRgG7AhvVTccDV1OG/W4/kGWOiBjuBjWI1FFVN9fHhzVsfw3YpodzJgEbNjy/BVi1P8sZERHdy4z1iIhoWYJIRES0LEEkIiJaliASEREtSxCJiIiWJYhERETLEkQiIqJlCSIREdGyBJGIiGhZgkhERLQsQSQiIlqWIBIRES1LEImIiJYliERERMsSRCIiomUJIhER0bIEkYiIaFmCSEREtCxBJCIiWpYgEhERLUsQiYiIliWIREREy+Yc6BvaXgY4G1gcmAaMlXRil2M6gBOBTYFJwK6S7rL9PuB8YC7gq5Jusz0ncA2wpaRJA/hSIiKGvcGoibwBHCBpFWAdYB/bq3Q5ZhNgpfqzJ3By3f5V4JuU4HJg3bY3cG4CSETEwBvwmoikJ4En6+N/234AWAq4v+GwrYCzJU0Dbre9oO0lgMnAqPoz2faCwBbAxgP4EiIiohrwINLI9vLAB4E7uuxaCni04fljddtJlKaweSi1kkOBYyRNncF99qTUaAAWmeWCR0QEMIhBxPZ8wKXAfpL+1cw5kh4BNqjnrwgsDTxg+xxgbuBQSf+vm/PGAmPreeP75AVERMTgjM6yPRclgJwn6bJuDnkcWKbh+dJ1W6OjgTHAvsBpwEHA4X1f2oiI6MmAB5E68up04AFJx/dw2DhgZ9sdttcBXq59KZ3XWB94QtKDlP6RqfVnVP+WPiIiGg1Gc9a6wE7AvbYn1m3fBZYFkHQKcDVlBNZDlCG+u3WeXIPQGGDbumkscB7ltezd/8WPiIhOgzE6649AxwyOmQbs08u+Tzc8fwBYsy/LGBERzcmM9YiIaFmCSEREtCxBJCIiWpYgEhERLUsQiYiIliWIREREyxJEIiKiZQkiERHRsgSRiIhoWYJIRES0LEEkIiJaliASEREtSxCJiIiWJYhERETLBnWN9aHE9sbAicAI4DRJx3XZPw9lffjRwPPAtpIervsOAfYApgD7SrrW9qLA5cCCwBhJV9RjfwvsLemJoVrOdihjRBSpifQB2yOAk4BNgFWA7Wyv0uWwPYAXJa0InAD8sJ67CvAl4APAxsCv6vW2A04B1gb2q8duAdw9CwFkti9nO5QxIqZLEOkbawMPSfq7pNeBC4CtuhyzFXBWfXwJ8Mm6SuNWwAWS/iPpH5TVHNcGJlOW+50HmGJ7TsoH4I+GeDnboYwRUSWI9I2lgEcbnj9Wt3V7jKQ3gJeBhXs593zKh+L1wDHA14BzJE0a4uVshzJGRJU+kdmUpJeBzQBsvws4GPic7VOBdwE/lXTbIBYRaI9ytkMZI9pVaiJ943FgmYbnS9dt3R5Tm1MWoHQKN3PuocDRlLb9PwK7AEcM0XK2QxkjokoQ6Rt3AivZXsH23JTO3XFdjhlH+cAC2Bq4UdK0uv1LtuexvQKwEvDnzpNsrwQsLelmSrv+VGAaMHKIlrMdyhgRVYJIH6jt8l8HrgUeAC6SdJ/tI21vWQ87HVjY9kPAtyhNKki6D7gIuB+4BthH0pSGyx8NfK8+/g2wN+WD9sShWM52KGNETNcxbdq0wS7DgLI9XtJag12OiMG0/MG/a4s//Ifn3X6wi9CUVVdYdrCLMEP37nJvx6yc39Nn56B0rLc6mcz2usDJwOvAdpIetL0g5dvnxpKmDuDLiIgY9ga8OWtWJpMBBwCbUsb471W3jQGOSQCJiBh4g9EnMiuTyTonjY0CJtt+D7BM7SiNiIgBNhjNWd1NCPtwT8dIesN252SyYynNXK8COwE/odREemV7T2DP+vR9tsfPygsYphYBnhvsQkTfuGrH5SYMdhmaYW7tj8v2+Xv5N315sX7SB597y3W3sa0mG0qaCKwDYPvjwJNAh+0LKbWUAyQ93c15Y4GxA1jUIScDEmKoyHu5bw1Gc9asTCajbuug1ECOAg4HDgJOBfbtt1JHRMTbDEYQmZXJZJ12Bq6W9ALTJ41NrY8jImKADHhzVu3j6JxMNgI4o3MyGTBe0jjKZLJz6mSyFyiBBgDbo4BdgY3qpuOBqynDfttjUHl7SnNgDBV5L/ehYTfZMCIi+k7SnkRERMsSRCIiomVtNcR3uLI9Bbi3YdNnO9cU7+bYVyTNNyAF64XtJYGfS9ra9hrAkpKurvu2BFbpmu4mhq52fA/PiO1dges6l1i2fRpwvKT7B7VgAyxBpD28KmmNwS7EzKh/WFvXp2sAa1EGQFAHT3QdkRdDW9u9h5uwK2DgCQBJXx7U0gySBJE2ZHs+4LeUVfnmAsZI+m2XY5YALgTmp/w/7y3pFtsbAd+nrDf+N2A3Sa90Ofdm4B5g/Xru7pL+bHsh4Azg3cAkYE9Jf7G9PtPTqU8DPk7JMHAVsCZwJDDS9nqUrAMjKUHle8BfgBUkTbX9DuCv9frLUnKsLVrv9RVJf53Vf7uYPQzQe/gOYENgQWCPeu4I4Dhgg3r+SZJ+bXsO4JfAJyjZMiZTRo5eYvswYAvK+/ZPwFeBL1Dew+fZfhX4CPB74MC6/T2Svl3LsiuwlqSv296RMp9t7lq+r3VZrqDtpE+kPYy0PbH+XA68BnxO0pqUP5Kf1gmYjbYHrq3f/lYHJtpehDJJ81P13PGU9Ti6M6qe+zVK4IDyh3u3pNWA71JS0ED5w9mnHv8xSloaAGp+tMOACyWtIenChn0vAxMpwQpg81rmyZRhmN+QNLpe/1dN/UvF7Gow3sNzSlqbkrD18LptD+BlSR8CPgR8pS5g9nlgeUpS2J0oQaHTLyV9SJIogWRzSZfUe+9Q39evNhx/KfC5hufbAhfYfn99vG59TVOAHXr9V2sDqYm0h7c0BdieCzimpn6ZSsk1tjjwVMM5dwJn1GOvkDSx1hhWAW61DeXbUE9ri/8GQNIfbM9fU+6vR/kGhqQbbS9se37gVuB42+cBl0l6rF6/GRdS/rBuoswH+lX9lvpR4OKG68zT7AVjtjQY7+HL6u8JlAABZX7ZarY7m1oXoKyAuR5wcc0G/pTtmxqus6HtgyiTmRcC7gOu7OmFSnrW9t9trwM8CKxM+RvZh7K8xZ217COBZ3q6TrtIEGlPO1CaeUZLmmz7YWDexgPqh//Hgc2AM20fD7wIXC9puybu0XUCUY8TiiQdZ/t3lDT9t9r+DOWbZjPGUT5MFqL8gd0IvAN4aQi2ocd0A/Ee/k/9PYXpn3UdlBrutY0H2t60uwvYnpdSC15L0qO2j+hazh5cAHyR0jx7uaRptaZ1lqRDmji/baQ5qz0tADxT//g2pJvsmraXA56WdCpwGqVv4nZgXdsr1mPeYfu9Pdxj23rMepTq/8vALdTqt+0NgOck/cv2eyTdK+mHlG+PK3e51r+Bd3Z3k9qW3blE7VWSpkj6F/AP29vUe3XYXr2pf5loFwPxHu7OtcDetXaD7ffWvrhbgS/YnsP24pQ+E5geMJ6rNeStG67V4/sauJyypMV2lIACcAOwte3F6r0Xqq+xrSWItKfzgLVs30vJI9Zdh/MGwD2276YEhBMlPUsZUfIb23+hNAN0/cDv9Fo99xRKOzLAEcDoeu5xTM9vtp+Lv1A6JH/f5Vo3AavU9vBtu7nXhcCO9XenHYA9bN9DaT7ouuZMtLeBeA935zTgfuAulzalX1NqKZdSlqW4HzgXuIvy5eklSnJXUwLQnQ3XOhM4pb6vRzbeRNKLwAPAcpL+XLfdT+nPua6W/XpgiZko+2wpaU/iberIlgMlZd2VGDZszyfpFdsLA3+mdIA/NaPzhrv0iUREFFfVASRzA0clgDQnNZGIiGhZ+kQiIqJlCSIREdGyBJGIiGhZgkhERLQsQSQiIlr2/wEsJQjqqc5O1AAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "dark"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "intr_threshold_opt = 1\n",
    "while True:\n",
    "    results = tune_intr_threshold(intr_threshold_opt, plot=False)\n",
    "    if results['false_positives'] == 0:\n",
    "        break\n",
    "    intr_threshold_opt += 1\n",
    "results_bayes_intr_threshold_tuned = tune_intr_threshold(intr_threshold_opt)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Let's check the false positive email for the just suboptimal threshold:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Subject: the expatriate zone - issue # 1 2 / 14 / 00\r\n",
      "the expatriate zone 1 / 15 / 00 issue # 1\r\n",
      "where every day is \" just another shitty day in paradise \"\r\n",
      "the zine for the tropically impaired\r\n",
      "- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - in this issue\r\n",
      "* featured expatriate of the month - james \" sunny jim \" white\r\n",
      "* travel into \" the zone \" - and beat the irs at the same time .\r\n",
      "* \" jimmydotcom \" - where ' s that parrot head book we ' ve been hearing about ?\r\n",
      "* club trini update\r\n",
      "* millennium change bubba style - see what all the hullabaloo online was\r\n",
      "about .\r\n",
      "* cabo 1999 - bad boys of the \" expatriated phan club \" spotted south of the\r\n",
      "border .\r\n",
      "* parrothead pilots association - new club in the making .\r\n",
      "- - - - - - - - - - - - - - - - - - - -\r\n",
      "for the web page edition go to :\r\n",
      "it includes some great pictures . we suggest printing it out and laying it out\r\n",
      "on your coffee table . that way , you have somewhere to set your \" red stripe \"\r\n",
      "so you don ' t leave rings on the table .\r\n",
      "- - - - - - - - - - - - - - - - - - - -\r\n",
      "welcome to the expatriate zone\r\n",
      "welcome to the \" first \" issue of \" the expatriate zone , \" the official\r\n",
      "newsletter of the new \" expatriate zone \" web site . after publishing \" the\r\n",
      "temperate zone \" for the past two years , i ' m taking the opportunity to join\r\n",
      "forces with bob segel who has been publishing \" the coconut modem \" with karen\r\n",
      "coyne since april of 1994 .\r\n",
      "last year , after operating in aol exclusively , bob created \" parrotkey . com \"\r\n",
      "out on the \" internet \" where it would join it ' s sister site on aol and be\r\n",
      "accessible to all parrot heads . at the time bob expanded \" pk , \" we had been\r\n",
      "working on my little area of \" club st . somewhere , \" and the new\r\n",
      "\" parrotkey . com \" seemed like the perfect place to open the doors of the club .\r\n",
      "while we were talking about the direction we wanted to see \" club st .\r\n",
      "somewhere \" and \" parrot key \" go , we could see that this was the perfect chance\r\n",
      "to , as emeril says , \" kick it up a notch . \"\r\n",
      "jimmy buffett ' s music was a big part of our lives , but even more so was the\r\n",
      "\" expatriate lifestyle \" he sang about . a lifestyle that includes tropical\r\n",
      "travel , sailing , boating , parasailing and surfing , and yes , the music and\r\n",
      "lyrics of mr . james w . buffett .\r\n",
      "so it is , that we created our new web site , \" the expatriate zone \"\r\n",
      ", along with this newsletter of that same\r\n",
      "name . in december , \" the expatriate zone \" opened its doors with its \" island\r\n",
      "trading post \" with its holiday shopping service at the \" island trading post . \"\r\n",
      "over 3 , 600 people visited the site the first day , and daily prizes were\r\n",
      "awarded for the first two weeks . the \" island trading post \" will open again\r\n",
      "soon , offering a complete line of tropical items , not just to those of us\r\n",
      "that live a few blocks from wal - mart , but also for those of us lucky enough\r\n",
      "to be on a sailboat or living on an island . for not , we ' ll start with the\r\n",
      "\" t - shirt of the month . \"\r\n",
      "we want \" the expatriate zone \" to not only keep you entertained and informed\r\n",
      "while you toil away at your everyday \" corporate gig , \" but it will also get\r\n",
      "you ready for that day you finally shuck it all and disappear down into \" the\r\n",
      "expatriate zone . \"\r\n",
      "we hope you all enjoy our new effort , and also continue to visit \" parrot key \"\r\n",
      "and \" club st . somewhere . \" both of which will be closely associated with \" the\r\n",
      "expatriate zone \" and \" parrotkey . com . \"\r\n",
      "jackson & bob\r\n",
      "- - - - - - - - - - - - - - - - - - - -\r\n",
      "\" expatriate of the month \"\r\n",
      "james \" sunny jim \" white\r\n",
      "when i first ventured on the internet in 1995 , i ran in to a guy named \" sunny\r\n",
      "jim \" who was living down in the cayman islands . we never really got a chance\r\n",
      "to connect , for which i am truly sorry . but , when something is supposed to\r\n",
      "happen , it usually does .\r\n",
      "on the first leg of my \" road dawg \" tour\r\n",
      "( click on \" road dawg \" ) i stayed with my good friend alex & janet leist and\r\n",
      "their wonderful children . over the five days i stayed at their house , alex\r\n",
      "and i listened to a lot of music , and one of the guys we both liked was\r\n",
      "\" sunny jim . \" i bought one of james ' cds from alex , and fell in love with it\r\n",
      "immediately . the cd was his second , called \" life in the laid back lane . \"\r\n",
      "little did i know that i would meet up with james and his roady \" shady floyd \"\r\n",
      "several times over the next six months . james was embarking on his first real\r\n",
      "tour of the us since he moved \" down island \" almost twelve years ago . we got\r\n",
      "several chances to talk over the summer , and we will have an in depth article\r\n",
      "about this \" expatriate \" soon on \" the expatriate zone \" web site . be sure to\r\n",
      "watch for it .\r\n",
      "in the meantime , here are a couple of quick things i learned about this great\r\n",
      "musician . when i asked him about his \" expatriate \" status , his reply was ,\r\n",
      "\" technically , an expatriate is someone who lives outside their own country ,\r\n",
      "for whatever reason . for some it ' s legal reasons , others just because they\r\n",
      "want to , or that ' s where the work is . i have been an expatriate for nearly\r\n",
      "twelve years now . i have pretty fully lived the whole caribbean experience ,\r\n",
      "from climbing the coconut trees and spear fishing , to sailing , scuba diving ,\r\n",
      "hot afternoons on the beach and romantic moonlit nights under the swaying\r\n",
      "palms . \"\r\n",
      "even as james is about to repatriate himself and his family , i ask if there\r\n",
      "were any other places he would consider living as an expatriate ? \" maybe\r\n",
      "somewhere on the coast of mexico or in a group of islands that are all within\r\n",
      "sailing distance of one another , \" was the advice he gave .\r\n",
      "until we can learn more of \" sunny jim ' s \" world , stop by his web site at\r\n",
      "and if you don ' t have one of his cds , including his\r\n",
      "latest release \" sandbar serenade , \" for goodness sake , pick one up . you can\r\n",
      "also read a great interview alex conducted with james at\r\n",
      "- - - - - - - - - - - - - - - - - - - -\r\n",
      "travel into \" the zone \"\r\n",
      "\" some are running from the irs \"\r\n",
      "we strongly believe that we all have to get away from our computer screens\r\n",
      "and out into the real world as often as we can . does this mean we give up one\r\n",
      "our most useful tools of communication , computers ? heck no ! with laptops ,\r\n",
      "cell phones and gps systems , we can go anywhere , and still stay in touch if\r\n",
      "we feel the need . but get real and throw that \" pager \" away as soon as you can .\r\n",
      "we hope to keep this section current and informative , hopefully hearing from\r\n",
      "those of you already living your lives in \" the zone , \" helping the rest of us\r\n",
      "prepare for our getaway from the cities . i ' m sure there are some of us that\r\n",
      "are preparing to \" disappear \" to places like costa rica , where the \" the\r\n",
      "expatriate zone \" will become an important tool .\r\n",
      "how would you like to check out a place to \" disappear \" to and write it off\r\n",
      "your taxes at the same time ? want to go down into \" the zone \" to places like\r\n",
      "the bahamas , belize , st . croix , puerto rico , bonaire , the dominica republic\r\n",
      "or saba ? want to do something for the planet , work with dolphins , save a\r\n",
      "coral reef , or study monkeys and the rain forest at the same time ?\r\n",
      "well we have the answer . check out these organization ' s web sites , and then\r\n",
      "get some advice from your financial advisor on 501 c non - profit organizations .\r\n",
      "depending on the tax bracket you ' re in , uncle sam could end up paying for up\r\n",
      "to half the trip .\r\n",
      "the oceanic society\r\n",
      "earthwatch institute\r\n",
      "reef , the reef environmental education foundation\r\n",
      "after that , all that ' s left is to make that plane reservation , and remember ,\r\n",
      "\" no plane on sunday . \"\r\n",
      "- - - - - - - - - - - - - - - - - - - -\r\n",
      "\" jimmydotcom \"\r\n",
      "after many delays , \" jimmydotcom \" has finally gone to the printer with a due\r\n",
      "date of the middle of april . i hit some \" bumps in the road \" on my way to\r\n",
      "publishing the first ever book about the parrot head phenomena , but it was\r\n",
      "all worth it .\r\n",
      "this book isn ' t about jimmy buffett , it ' s about the effect of the internet on\r\n",
      "parrot heads . for more information , visit . we\r\n",
      "have a complete outline of the book , reviews from other parrot heads , and\r\n",
      "samples of the pages on the site .\r\n",
      "financial considerations have limited the first printing to 1 , 000 copies at\r\n",
      "the present time , so if your even thinking of buying a book , i suggest you\r\n",
      "reserve your copy by putting your name on the \" reservation list . \" you will be\r\n",
      "under no financial obligation . it will simply indicate to my financial\r\n",
      "partners ( who aren ' t parrot heads ) , just how many books we need to print . do\r\n",
      "it right away , because pre - sales were brisk , and if enough of you reserve a\r\n",
      "copy , i can get the financing to expand the first printing .\r\n",
      "let ' s show the world what parrot heads stand for by making this book a big\r\n",
      "success . when i was on the bhotm tour this summer , i had many people tell me\r\n",
      "they wanted to buy an extra book to give to their boss . they wanted the boss\r\n",
      "to understand why getting the day off for a buffett concert is so important .\r\n",
      "maybe it will help the rest of us to explain ourselves to our friends \" that\r\n",
      "just don ' t get it . \"\r\n",
      "here ' s one of the latest reviews of the book from bob robinson , founder &\r\n",
      "president of the sandlapper phc in south carolina . i had a great time with\r\n",
      "this club on the \" road dawg \" tour , and shared an advance copy with his club :\r\n",
      "\" when we received jackson ' s plea for pictures , our members were blown away .\r\n",
      "we hustled up a package of pix , as did a majority of the parrot head clubs\r\n",
      "from around the globe , and the result is pure parrothedonistic pleasure . this\r\n",
      "collection is bound for glory in the annals of island time . if moderation is\r\n",
      "your key to life then stay away from jimmydotcom , ' cause this book is packed\r\n",
      "with those stories only we can tell . congratulations bubbafreak ! \"\r\n",
      "bobrob sandlapper phc\r\n",
      "- - - - - - - - - - - - - - - - - - - -\r\n",
      "club trini update\r\n",
      "speaking of expatriates , how about the guys who play the true music of \" the\r\n",
      "zone ? \" club trini ' s long awaited live cd will finally be coming out this\r\n",
      "april , one year after it ' s recording . jimmy has been keeping mike and robert\r\n",
      "pretty busy , so it ' s understandable that it took awhile . it will be the third\r\n",
      "in the \" margaritaville cafe - late nite \" series . i can ' t wait to hear it , as\r\n",
      "my wife and i were some of the lucky ones who were in new orleans when it was\r\n",
      "recorded .\r\n",
      "you can get a look at all the songs included on the new cd at club trini ' s\r\n",
      "revamped web site at . along with the great songs we ' ve\r\n",
      "become accustomed to , are a few i ' m pretty excited about . if you haven ' t\r\n",
      "heard ct ' s version of \" african friend \" you ' re in for a real treat . along with\r\n",
      "that , is a song that tina and nadirah have been treating audiences to this\r\n",
      "past summer , \" cario \" and bob marley ' s \" no woman , no cry . \"\r\n",
      "at the new web site , fans can \" now \" order their favorite club trini cds using\r\n",
      "their credit cards . so there ' s no excuse to not owning everything available .\r\n",
      "i think a true \" expatriate musical library \" is incomplete without club trini .\r\n",
      "this summer , helen hiatt , jimmy and the coral reefer ' s wardrobe designer ,\r\n",
      "came up with a shirt that club trini members lovingly call the \" ice cream\r\n",
      "vendor \" shirt , which they wear out front at the start of the concert . ok ,\r\n",
      "pete sometimes forgets his , and when he does have it on , it lost it ' s sleeves\r\n",
      "somewhere along the way . there ' s a great design that has been applied to a\r\n",
      "t - shirt that we all can own . check it out at their site .\r\n",
      "- - - - - - - - - - - - - - - - - - - -\r\n",
      "jesters and friends\r\n",
      "note card series\r\n",
      "tro - pics has just released their first note cards\r\n",
      "series from their \" jesters and friends \" original oils . friends , places and\r\n",
      "musicians from past years have inspired these cards and parrot heads and\r\n",
      "expatriates alike will want to have these great looking cards for those times\r\n",
      "when an email just isn ' t personal enough .\r\n",
      "the original artwork is also on sale at this site , so be sure to stop by and\r\n",
      "visit them . included in the \" jesters \" series are our good friends tim glancey\r\n",
      "and wally nickel .\r\n",
      "- - - - - - - - - - - - - - - - - - - -\r\n",
      "millennium change bubba style\r\n",
      "i ' ve got to tell you , i wasn ' t buying into this \" new millennium \" thing , one\r\n",
      "bit . in fact , i had almost decided not to go to the buffett new year ' s eve\r\n",
      "show . nothing to do with jimmy , but i just didn ' t get the whole hype . but at\r\n",
      "the last minute , i had a change of heart .\r\n",
      "and now after being a special guest at seagram ' s party honoring the new\r\n",
      "margaritaville tequila , i ' m sure glad i went . the shows kind of melded\r\n",
      "together for me , making the millennium change over very special .\r\n",
      "if you want to hear more about this or see some pictures of both , visit our\r\n",
      "new \" the expatriate zone \" web site at and go to\r\n",
      "the web version of this newsletter .\r\n",
      "- - - - - - - - - - - - - - - - - - - -\r\n",
      "flash ! ! ! \" expatriated phans \" spotted south of the border\r\n",
      "after making both of \" tropo travel ' s \" previous trips down to cobo san lucas ,\r\n",
      "mexico , i had to miss this year ' s trek due to my being with the \" beach house\r\n",
      "on the moon \" tour on it ' s florida , new orleans and houston swing .\r\n",
      "as the story goes , those bad boys and girls from \" the expatriated phan club \"\r\n",
      "showed up again , mainly for the t - shirts that\r\n",
      "said , \" too much tequila . . . or not quite enough \" on the back . who are those\r\n",
      "people anyway ? whenever i go to their web site , all of them have black bars\r\n",
      "covering their eyes . if you want to see pictures of the trip or need info for\r\n",
      "\" down to the banana republics - 2000 \" visit mike\r\n",
      "mclain also has some other great trips planned this year , check them out .\r\n",
      "speaking of \" tropo travel , \" if you need a hotel room for the vegas or irvine\r\n",
      "shows and want a great deal , visit there site for some great rates . i ' m\r\n",
      "joining some of my parrot head friends at the \" key largo , \" though i ' m kind of\r\n",
      "afraid to be in a town like vegas with sue and ponch aprea from the detroit\r\n",
      "phc . i ' m joining a lot of my other friends i met on the road there too . i\r\n",
      "hate seeing jimmy in vegas , but i ' ve got to keep the streak alive , and seeing\r\n",
      "all my parrot head friends will make it all worthwhile . i hope peter mayer\r\n",
      "ends up playing there again this year .\r\n",
      "and speaking of those great t - shirts , we have some neat shirts available at\r\n",
      "the \" island store . \" they ' re similar to the shirt you got if you went on this\r\n",
      "year ' s trip , except they have \" tropo travel ' s \" seaplane on the back and the\r\n",
      "\" no bad days \" logo on the front breast . you can see a picture at\r\n",
      ".\r\n",
      "- - - - - - - - - - - - - - - - - - - -\r\n",
      "parrothead pilots association\r\n",
      "here are a couple of tidbits for your \" airplane nuts \" and pilots out there .\r\n",
      "a new online friend who ' s a medivac chopper pilot is trying to start a parrot\r\n",
      "head ' s pilot ' s association . if you ' re interested , contact dan at\r\n",
      ".\r\n",
      "i also ran across a really cool page that features jimmy ' s \" hemisphere\r\n",
      "dancer \" including it ' s \" jamaican bullet hole . \" ed booth , jr . an attorney from\r\n",
      "jacksonville , florida , who belongs to \" lawyer - pilots bar association \" is the\r\n",
      "webmaster and creator .\r\n",
      "the web site is \" grumman n 928 j \" ( flashback - how many of you remember the tv\r\n",
      "show ' whirly birds ' ? wasn ' t their number n 975 b ? ) and is at\r\n",
      "it shows a lot of pictures of jimmy ' s 1954\r\n",
      "grumman i think you ' ll enjoy .\r\n",
      "now all we need is for these guys to team up with the parrot heads pilot ' s\r\n",
      "association .\r\n",
      "- - - - - - - - - - - - - - - - - - - -\r\n",
      "this document is copyrighted 2000 by st . somewhere press , publisher . \" the\r\n",
      "expatriate zone \" is a trademark of st . somewhere press . you are permitted\r\n",
      "and encouraged to copy this document in its entirety ( including this message )\r\n",
      "and email it to friends , etc .\r\n",
      "to subscribe or unsubscribe to this newsletter , simply go to :\r\n",
      "this document is copyrighted 2000 by st . somewhere press , publisher . the\r\n",
      "expatriate zone is a trademark of st . somewhere press . you are permitted\r\n",
      "and encouraged to copy this document in it ' s entirety ( including this\r\n",
      "message ) and email it to friends , etc .\r\n",
      "to subscribe or unsubscribe from this newsletter please visit :\r\n",
      "\n",
      "Label: 0\n",
      "Graham: 0 [0.16666666666666666, 1.0, 0.3068181818181818, 0.024096385542168676, 0.1504222780187172, 0.2151179312304632, 0.09490740740740741, 0.1362025316455696, 0.16666666666666666, 1.0, 0.3068181818181818, 0.1504222780187172, 0.18136439267886856, 0.1362025316455696, 0.024096385542168676, 0.1504222780187172, 0.18636363636363637, 0.3130699088145897, 0.06908665105386416, 0.1873560040549258, 0.21550946798917944, 0.11392405063291139, 0.4, 0.06908665105386416, 0.25654642006958434, 1.0, 0.16666666666666666, 1.0, 0.13647997143027202, 0.16666666666666666, 0.4, 1.0, 0.25654642006958434, 0.18697402122210025, 0.024096385542168676, 1.0, 1.0, 0.28779663403985634, 0.16666666666666666, 0.06127946127946128, 0.023622047244094488, 0.3333333333333333, 0.008333333333333333, 0.21212121212121213, 0.2523364485981308, 0.12373737373737374, 0.16666666666666666, 0.3068181818181818, 0.23618413992917694, 0.2, 0.16666666666666666, 0.0, 0.11107421550722231, 0.16666666666666666, 0.18130841121495328, 0.22031571722717913, 0.4, 0.18636363636363637, 0.12844708370617425, 0.24513776337115073, 0.16466739367502725, 0.4, 0.3274336283185841, 0.13157894736842105, 0.14791666666666667, 0.12844708370617425, 0.2520775623268698, 0.13382269904009034, 0.24324324324324326, 0.22103533278553822, 0.3333333333333333, 0.4, 0.08761329305135952, 0.625, 0.0652699435938759, 0.0, 0.9398496240601504, 0.07984031936127745, 0.13808049535603714, 0.29910714285714285, 0.16666666666666666, 0.4, 0.5735294117647058, 0.12389380530973451, 0.22103533278553822, 0.4, 0.0189873417721519, 0.4074074074074074, 0.6666666666666666, 0.28779663403985634, 0.16666666666666666, 0.4, 0.4, 0.3333333333333333, 0.4, 0.3116883116883117, 0.28779663403985634, 0.16666666666666666, 1.0, 0.4, 0.4, 0.35294117647058826, 0.1781818181818182, 0.3333333333333333, 0.25654642006958434, 0.16666666666666666, 0.23076923076923078, 0.13647997143027202, 0.16666666666666666, 0.2832764505119454, 0.15057915057915058, 0.9622641509433962, 0.279874213836478, 0.14463452566096424, 0.19190283400809716, 0.296875, 0.14182475158084915, 0.31313131313131315, 0.16049382716049382, 0.14791666666666667, 0.0967741935483871, 0.5, 0.19190283400809716, 0.17097862767154107, 0.23618413992917694, 0.1111111111111111, 0.19190283400809716, 0.17097862767154107, 0.093558536181487, 0.32187899835496253, 0.3333333333333333, 0.7011494252873564, 0.16466739367502725, 0.24157303370786518, 0.20222604141091743, 0.11952554744525548, 0.3333333333333333, 0.14463452566096424, 0.0773067331670823, 0.32187899835496253, 0.38666666666666666, 1.0, 0.1218026796589525, 0.20222604141091743, 0.17329910141206675, 0.12844708370617425, 0.21687344913151366, 0.15384615384615385, 0.0, 0.093558536181487, 0.16666666666666666, 0.7011494252873564, 0.14285714285714285, 0.14463452566096424, 0.16666666666666666, 1.0, 0.3068181818181818, 0.14285714285714285, 0.14463452566096424, 0.16666666666666666, 0.18451612903225806, 0.024096385542168676, 0.28779663403985634, 0.16666666666666666, 1.0, 0.3068181818181818, 0.16666666666666666, 0.6666666666666666, 0.8581560283687943, 0.28779663403985634, 0.16666666666666666, 0.1781818181818182, 1.0, 0.3068181818181818, 0.2832764505119454, 0.5378787878787878, 0.20388349514563106, 1.0, 0.16666666666666666, 0.4, 0.3068181818181818, 0.13647997143027202, 0.16666666666666666, 0.4489795918367347, 0.2007042253521127, 0.367816091954023, 0.09318894271872329, 0.12844708370617425, 0.15940224159402241, 0.21804511278195488, 0.16666666666666666, 0.3258426966292135, 0.14463452566096424, 0.21649484536082475, 0.7647058823529411, 0.2269106931028639, 0.0030627871362940277, 0.0, 0.17766497461928935, 0.17573385518590998, 0.13382269904009034, 1.0, 0.16666666666666666, 0.1111111111111111, 0.35294117647058826, 0.2269106931028639, 0.0037174721189591076, 0.0, 0.1415929203539823, 0.013422818791946308, 0.28779663403985634, 0.0, 0.15719063545150502, 0.22900763358778625, 0.20388349514563106, 0.1796875, 0.25654642006958434, 0.012552301255230125, 0.1111111111111111, 0.0030627871362940277, 0.028132992327365727, 0.4, 0.15648336727766463, 0.17097862767154107, 0.093558536181487, 0.16666666666666666, 0.4720812182741117, 0.18636363636363637, 0.19190283400809716, 0.10931174089068826, 0.21649484536082475, 0.19190283400809716, 0.12844708370617425, 0.24513776337115073, 0.6363636363636364, 0.5378787878787878, 0.093558536181487, 0.012552301255230125, 0.23618413992917694, 0.14867566216891553, 0.4, 0.14463452566096424, 0.29910714285714285, 0.4, 0.0, 0.11107421550722231, 0.16666666666666666, 0.22031571722717913, 0.0030627871362940277, 0.1875, 1.0, 0.14791666666666667, 0.15332428765264586, 0.13382269904009034, 0.11036789297658862, 0.093558536181487, 0.2460881934566145, 0.323943661971831, 0.32222222222222224, 0.28779663403985634, 0.3333333333333333, 0.19436619718309858, 0.3333333333333333, 0.23618413992917694, 0.16666666666666666, 0.1781818181818182, 0.4, 0.15648336727766463, 0.16666666666666666, 0.2288135593220339, 0.16666666666666666, 0.7352941176470589, 0.16883116883116883, 0.14463452566096424, 0.3254437869822485, 0.16666666666666666, 0.0, 0.28779663403985634, 0.16666666666666666, 0.3333333333333333, 0.2682926829268293, 0.14791666666666667, 0.12612612612612611, 0.2222222222222222, 0.22103533278553822, 0.16666666666666666, 0.14285714285714285, 0.14791666666666667, 0.088, 0.14463452566096424, 0.07984031936127745, 0.3333333333333333, 0.19436619718309858, 0.3333333333333333, 0.23618413992917694, 0.4, 0.2857142857142857, 0.279874213836478, 0.14791666666666667, 0.1905829596412556, 0.07984031936127745, 0.16466739367502725, 0.18697402122210025, 0.12389380530973451, 0.16666666666666666, 0.7352941176470589, 0.18181818181818182, 0.14463452566096424, 0.23391435334624489, 0.4, 0.25757575757575757, 0.05263157894736842, 0.19190283400809716, 0.15384615384615385, 0.23843317331106936, 0.4, 0.14285714285714285, 0.0, 0.12844708370617425, 0.24513776337115073, 0.54, 0.12389380530973451, 0.23843317331106936, 0.2, 0.30407523510971785, 0.28779663403985634, 0.3268765133171913, 0.2777777777777778, 0.15903614457831325, 0.4264705882352941, 0.46563573883161513, 0.1218026796589525, 0.12389380530973451, 0.16666666666666666, 1.0, 0.2, 0.1626848691695108, 0.4, 0.22103533278553822, 0.23843317331106936, 0.2, 0.16466739367502725, 0.296875, 1.0, 0.2523364485981308, 0.047619047619047616, 0.4, 0.0, 0.23618413992917694, 1.0, 0.23618413992917694, 0.15966386554621848, 0.16666666666666666, 0.54, 0.23618413992917694, 1.0, 0.28779663403985634, 0.5855855855855856, 0.023622047244094488, 0.15239477503628446, 0.0, 0.1218026796589525, 0.19190283400809716, 0.1873560040549258, 0.16466739367502725, 0.14791666666666667, 0.028132992327365727, 0.3268765133171913, 0.1781818181818182, 0.2832764505119454, 0.5378787878787878, 0.16666666666666666, 1.0, 0.3068181818181818, 0.19047619047619047, 0.2269106931028639, 0.18697402122210025, 0.8581560283687943, 0.28779663403985634, 0.16466739367502725, 0.18130841121495328, 0.16560509554140126, 0.25654642006958434, 0.026217228464419477, 0.16666666666666666, 1.0, 0.3068181818181818, 0.23076923076923078, 0.491679273827534, 0.0, 0.2269106931028639, 0.491679273827534, 0.15384615384615385, 0.12093023255813953, 0.3584905660377358, 0.2269106931028639, 0.491679273827534, 0.17857142857142858, 0.1566265060240964, 0.24358974358974358, 0.11107421550722231, 0.16666666666666666, 0.15384615384615385, 0.12093023255813953, 0.3584905660377358, 0.3188405797101449, 0.23285568065506654, 0.1717171717171717, 0.24880382775119617, 1.0, 0.16666666666666666, 0.5378787878787878, 0.16666666666666666, 0.18451612903225806, 0.06908665105386416, 0.23618413992917694, 0.021568627450980392, 0.07317073170731707, 0.12612612612612611, 0.4117647058823529, 0.13647997143027202, 0.16666666666666666, 0.18451612903225806, 0.2007042253521127, 0.2375, 0.16666666666666666, 0.15384615384615385, 0.12093023255813953, 0.3584905660377358, 0.08931953955995921, 0.3254437869822485, 0.14666666666666667, 0.11538461538461539, 0.6904761904761905, 0.23843317331106936, 0.2186046511627907, 0.2765957446808511, 0.28779663403985634, 1.0, 0.06467661691542288, 0.21439891328956306, 0.21550946798917944, 0.14463452566096424, 0.20898876404494382, 0.28779663403985634, 0.3074101247248716, 0.16466739367502725, 0.5277777777777778, 0.23843317331106936, 0.24770642201834864, 0.4, 0.13286713286713286, 0.0, 0.3333333333333333, 0.15903614457831325, 0.15474209650582363, 0.13647997143027202, 0.20898876404494382, 0.28779663403985634, 0.3074101247248716, 0.76, 0.25663716814159293, 0.14463452566096424, 0.14867566216891553, 0.093558536181487, 0.23843317331106936, 1.0, 0.27038823799678385, 0.4074074074074074, 0.093558536181487, 0.2473416551086454, 0.15384615384615385, 0.13647997143027202, 0.21439891328956306, 0.14791666666666667, 0.12844708370617425, 0.14124293785310735, 0.16666666666666666, 0.2269106931028639, 0.16666666666666666, 0.21687344913151366, 0.3333333333333333, 0.28779663403985634, 0.16666666666666666, 0.06127946127946128, 0.14791666666666667, 0.21939953810623555, 0.16666666666666666, 1.0, 0.3068181818181818, 0.14463452566096424, 0.21439891328956306, 0.355397951142632, 0.16112531969309463, 0.20222604141091743, 1.0, 0.23618413992917694, 0.0410958904109589, 0.2682926829268293, 0.20222604141091743, 1.0, 0.38317757009345793, 0.11107421550722231, 0.32187899835496253, 0.4117647058823529, 0.24271844660194175, 0.3333333333333333, 0.15903614457831325, 0.19190283400809716, 0.08931953955995921, 0.15474209650582363, 0.23794037940379403, 0.20222604141091743, 0.2413793103448276, 0.13647997143027202, 0.16466739367502725, 0.06908665105386416, 0.20222604141091743, 0.3898305084745763, 1.0, 0.19190283400809716, 0.29910714285714285, 0.23618413992917694, 1.0, 0.13814432989690723, 0.12373737373737374, 0.16666666666666666, 1.0, 0.3068181818181818, 0.14791666666666667, 0.1391304347826087, 0.20222604141091743, 0.29910714285714285, 0.3563218390804598, 0.3268765133171913, 0.1781818181818182, 0.36283185840707965, 0.23618413992917694, 0.15474209650582363, 0.12435233160621761, 0.14463452566096424, 0.47058823529411764, 0.4, 0.2857142857142857, 0.23618413992917694, 0.3333333333333333, 0.19436619718309858, 0.3333333333333333, 0.12582781456953643, 0.28779663403985634, 0.20985691573926868, 0.08931953955995921, 0.14867566216891553, 0.16666666666666666, 0.09565217391304348, 0.2269106931028639, 0.16666666666666666, 1.0, 0.3068181818181818, 0.23618413992917694, 0.4, 0.15648336727766463, 0.019230769230769232, 0.0030627871362940277, 1.0, 0.28779663403985634, 0.16666666666666666, 0.06127946127946128, 0.023622047244094488, 0.3333333333333333, 0.008333333333333333, 0.21212121212121213, 0.12749349522983522, 0.09318894271872329, 0.18451612903225806, 0.4, 0.093558536181487, 0.16666666666666666, 0.4720812182741117, 0.25654642006958434, 0.9, 0.09318894271872329, 0.3, 0.25654642006958434, 0.14463452566096424, 0.23843317331106936, 0.14285714285714285, 0.09090909090909091, 0.3333333333333333, 0.008333333333333333, 0.17766497461928935, 0.12389380530973451, 0.4074074074074074, 0.13814432989690723, 0.25654642006958434, 0.16666666666666666, 0.4, 0.3333333333333333, 0.14791666666666667, 0.39487179487179486, 0.2830188679245283, 0.22448979591836735, 0.23843317331106936, 0.18181818181818182, 0.14463452566096424, 0.15789473684210525, 0.13647997143027202, 0.20985691573926868, 0.09318894271872329, 0.02559576345984113, 0.3103448275862069, 0.04807692307692308, 0.15903614457831325, 0.12749349522983522, 0.10256410256410256, 0.1873560040549258, 0.03225806451612903, 0.14463452566096424, 0.1111111111111111, 0.19190283400809716, 0.25, 0.22105263157894736, 0.093558536181487, 0.16666666666666666, 0.18451612903225806, 0.3333333333333333, 0.28779663403985634, 0.2460881934566145, 0.3442622950819672, 0.4, 0.36363636363636365, 0.4144620811287478, 0.093558536181487, 0.3442622950819672, 0.4, 0.09318894271872329, 0.3333333333333333, 0.2269106931028639, 0.2460881934566145, 0.2565947242206235, 0.6226415094339622, 0.0, 0.016216216216216217, 0.4, 0.23618413992917694, 0.22508591065292097, 0.2, 0.5636363636363636, 0.3188405797101449, 0.16666666666666666, 0.3763440860215054, 0.1329479768786127, 0.09318894271872329, 0.3333333333333333, 0.11107421550722231, 0.22508591065292097, 0.43283582089552236, 0.0, 0.23618413992917694, 0.09318894271872329, 0.4, 0.14463452566096424, 0.23843317331106936, 0.2222222222222222, 0.28779663403985634, 0.54, 0.23618413992917694, 0.2696969696969697, 0.28779663403985634, 0.16666666666666666, 0.12056737588652482, 0.14791666666666667, 0.12582781456953643, 0.2, 0.12389380530973451, 0.3333333333333333, 0.008333333333333333, 0.09318894271872329, 0.3076923076923077, 0.2696969696969697, 0.28779663403985634, 0.023622047244094488, 0.12844708370617425, 0.8867924528301887, 0.13286713286713286, 0.0, 0.23618413992917694, 1.0, 0.25654642006958434, 0.3300970873786408, 0.2269106931028639, 0.19190283400809716, 0.3387096774193548, 0.16666666666666666, 0.717948717948718, 0.12389380530973451, 0.3202614379084967, 0.36904761904761907, 0.1, 0.6363636363636364, 0.25654642006958434, 0.16666666666666666, 0.6666666666666666, 0.15545755237045203, 0.047619047619047616, 0.323943661971831, 0.10683012259194395, 0.09318894271872329, 0.04779277635899307, 0.16466739367502725, 0.09318894271872329, 0.10931174089068826, 0.19, 0.15384615384615385, 0.2269106931028639, 0.023622047244094488, 0.23618413992917694, 0.3202614379084967, 0.4, 0.4, 0.0, 0.1024390243902439, 0.2919254658385093, 0.3188405797101449, 0.16666666666666666, 0.16850393700787403, 0.36507936507936506, 0.168141592920354, 0.023622047244094488, 0.12389380530973451, 0.4, 0.093558536181487, 0.3202614379084967, 0.18451612903225806, 0.5243243243243243, 0.36363636363636365, 0.28779663403985634, 0.16666666666666666, 0.3074101247248716, 0.1415929203539823, 0.1626848691695108, 0.0410958904109589, 0.13814432989690723, 0.15384615384615385, 0.5, 1.0, 0.367816091954023, 0.3333333333333333, 0.14791666666666667, 0.22448979591836735, 0.1024390243902439, 0.3333333333333333, 0.14463452566096424, 0.11627906976744186, 0.3188405797101449, 0.16666666666666666, 0.1111111111111111, 0.23618413992917694, 0.14791666666666667, 0.08931953955995921, 0.11952554744525548, 0.2473416551086454, 0.25654642006958434, 0.4666666666666667, 0.7333333333333333, 0.22103533278553822, 0.18697402122210025, 1.0, 0.11538461538461539, 0.093558536181487, 0.16666666666666666, 1.0, 0.3068181818181818, 0.2832764505119454, 0.5378787878787878, 0.14867566216891553, 0.05952380952380952, 0.14463452566096424, 0.6885245901639344, 0.13647997143027202, 0.19190283400809716, 0.25654642006958434, 0.16666666666666666, 0.07692307692307693, 0.44851063829787235, 0.20309094511591044, 0.23843317331106936, 0.078125, 0.28779663403985634, 0.4406779661016949, 0.1414141414141414, 0.09318894271872329, 0.16666666666666666, 0.22103533278553822, 0.18697402122210025, 0.31313131313131315, 1.0, 0.12749349522983522, 0.09318894271872329, 0.07878787878787878, 0.16058394160583941, 0.22103533278553822, 0.3202614379084967, 1.0, 0.04807692307692308, 0.3202614379084967, 0.5703703703703704, 0.12389380530973451, 0.4, 0.2473416551086454, 1.0, 0.1873560040549258, 0.09597523219814241, 0.17766497461928935, 0.2777777777777778, 0.10843373493975904, 0.22508591065292097, 0.5278969957081545, 0.603448275862069, 0.13647997143027202, 0.16363636363636364, 0.3469387755102041, 0.13647997143027202, 0.14182475158084915, 0.19190283400809716, 0.12844708370617425, 0.24513776337115073, 0.15675675675675677, 0.3333333333333333, 0.21875, 0.21550946798917944, 0.2374429223744292, 0.10961420698101654, 0.21939953810623555, 0.14463452566096424, 0.27038823799678385, 0.16466739367502725, 0.12844708370617425, 0.24513776337115073, 0.18636363636363637, 0.16666666666666666, 0.13821138211382114, 0.1873560040549258, 0.09318894271872329, 0.11952554744525548, 0.13382269904009034, 0.2473416551086454, 1.0, 0.13647997143027202, 0.68, 1.0, 0.367816091954023, 0.30566037735849055, 0.09318894271872329, 0.11952554744525548, 0.15384615384615385, 0.45098039215686275, 1.0, 0.16666666666666666, 0.1048951048951049, 0.9166666666666666, 0.38333333333333336, 0.13286713286713286, 1.0, 0.16666666666666666, 0.1111111111111111, 0.42857142857142855, 0.23618413992917694, 1.0, 0.14285714285714285, 0.14463452566096424, 0.047619047619047616, 0.6, 0.0, 0.8823529411764706, 0.4, 0.093558536181487, 0.16666666666666666, 0.35, 0.23618413992917694, 0.3333333333333333, 1.0, 0.2727272727272727, 0.1506849315068493, 0.16666666666666666, 0.4, 0.4, 0.4264705882352941, 0.23391435334624489, 0.023622047244094488, 0.1873560040549258, 0.22103533278553822, 0.14463452566096424, 0.4, 0.5555555555555556, 0.23618413992917694, 0.3202614379084967, 0.5784313725490197, 0.09318894271872329, 0.16022099447513813, 0.09420713867758923, 0.11088810837932765, 0.12612612612612611, 0.1610608020698577, 0.28884254431699685, 0.42857142857142855, 0.1626848691695108, 0.10931174089068826, 0.32, 0.4074074074074074, 0.23391435334624489, 0.2473416551086454, 1.0, 0.09090909090909091, 0.3333333333333333, 0.093558536181487, 0.16666666666666666, 0.10344827586206896, 0.28779663403985634, 0.4444444444444444, 0.27038823799678385, 0.25654642006958434, 0.23843317331106936, 0.15857605177993528, 0.28779663403985634, 0.3333333333333333, 0.16466739367502725, 0.20309094511591044, 0.29910714285714285, 0.5717171717171717, 0.047619047619047616, 0.5555555555555556, 0.28779663403985634, 0.2696969696969697, 0.11392405063291139, 0.12389380530973451, 0.16666666666666666, 0.764102564102564, 0.1626848691695108, 0.07692307692307693, 0.06748466257668712, 0.14791666666666667, 0.1767002928734136, 0.5294117647058824, 0.46563573883161513, 0.28779663403985634, 0.3333333333333333, 0.008333333333333333, 0.12844708370617425, 0.24513776337115073, 0.6731517509727627, 0.5950413223140496, 0.1270854788877446, 0.3202614379084967, 0.2832764505119454, 0.5378787878787878, 0.11107421550722231, 0.23618413992917694, 0.09420713867758923, 0.20222604141091743, 0.17329910141206675, 0.12844708370617425, 0.21687344913151366, 0.11952554744525548, 0.2696969696969697, 0.28779663403985634, 0.3202614379084967, 0.8867924528301887, 0.40336134453781514, 0.3202614379084967, 0.36619718309859156, 0.34806629834254144, 0.4, 0.4, 0.13647997143027202, 0.4, 0.5, 0.3563218390804598, 0.2696969696969697, 0.15384615384615385, 0.20222604141091743, 0.1767002928734136, 0.15474209650582363, 0.3383458646616541, 0.23843317331106936, 0.31313131313131315, 0.03225806451612903, 0.0, 0.42857142857142855, 0.2269106931028639, 0.023622047244094488, 0.11107421550722231, 0.2523364485981308, 0.12373737373737374, 0.16666666666666666, 0.3068181818181818, 0.14182475158084915, 0.20309094511591044, 0.22105263157894736, 0.13286713286713286, 0.16666666666666666, 0.0, 0.14791666666666667, 1.0, 0.2597402597402597, 0.16466739367502725, 0.14791666666666667, 0.29910714285714285, 0.11952554744525548, 0.14463452566096424, 0.23794037940379403, 0.38317757009345793, 0.13286713286713286, 0.3268765133171913, 0.5503875968992248, 0.8666666666666667, 0.23618413992917694, 0.17097862767154107, 0.12373737373737374, 0.16666666666666666, 0.5243243243243243, 0.6731517509727627, 0.23391435334624489, 0.3953488372093023, 0.23391435334624489, 0.14791666666666667, 0.1767002928734136, 0.22105263157894736, 0.18697402122210025, 0.16216216216216217, 0.14791666666666667, 0.15918367346938775, 0.15384615384615385, 0.2696969696969697, 0.3268765133171913, 0.49859154929577465, 0.14285714285714285, 0.45098039215686275, 0.28779663403985634, 0.34782608695652173, 0.5789473684210527, 0.3333333333333333, 0.3798103736754044, 0.2269106931028639, 0.4, 0.042735042735042736, 0.2857142857142857, 0.23618413992917694, 0.4, 0.2832369942196532, 0.14791666666666667, 0.1767002928734136, 0.279874213836478, 0.6, 0.23618413992917694, 0.11823647294589178, 0.11627906976744186, 0.25654642006958434, 0.14893617021276595, 0.09420713867758923, 0.14791666666666667, 0.28125, 0.16666666666666666, 0.08105906313645621, 0.15903614457831325, 0.23794037940379403, 0.5243243243243243, 0.23618413992917694, 0.5, 0.16466739367502725, 0.0, 0.38317757009345793, 0.23391435334624489, 0.11538461538461539, 0.23391435334624489, 0.20222604141091743, 0.1767002928734136, 0.14791666666666667, 0.1391304347826087, 0.14463452566096424, 0.16112531969309463, 0.18697402122210025, 0.5090252707581228, 0.21763085399449036, 0.23618413992917694, 0.0, 0.0, 0.24324324324324326, 0.13286713286713286, 0.20898876404494382, 0.28779663403985634, 0.20222604141091743, 0.16030534351145037, 0.4074074074074074, 0.32187899835496253, 0.2777777777777778, 0.25654642006958434, 0.16666666666666666, 0.3068181818181818, 0.12, 0.16666666666666666, 0.20408163265306123, 0.28779663403985634, 0.3074101247248716, 0.05714285714285714, 0.13647997143027202, 0.3268765133171913, 0.5, 0.13286713286713286, 0.16666666666666666, 0.21739130434782608, 0.09318894271872329, 0.12844708370617425, 0.15940224159402241, 0.05952380952380952, 0.11088810837932765, 0.20309094511591044, 0.14182475158084915, 0.28779663403985634, 0.3074101247248716, 0.16466739367502725, 0.20309094511591044, 0.08333333333333333, 0.14463452566096424, 1.0, 0.14463452566096424, 0.42857142857142855, 0.2288135593220339, 1.0, 1.0, 0.18636363636363637, 0.16666666666666666, 0.16666666666666666, 1.0, 0.3068181818181818, 0.08931953955995921, 0.47058823529411764, 0.2473416551086454, 0.2947976878612717, 0.28, 0.22905027932960895, 0.10931174089068826, 0.20222604141091743, 0.2288135593220339, 0.14463452566096424, 0.18485523385300667, 0.17097862767154107, 0.23843317331106936, 0.16883116883116883, 0.14463452566096424, 1.0, 0.14463452566096424, 0.23618413992917694, 0.2777777777777778, 0.19190283400809716, 0.21978021978021978, 0.32187899835496253, 0.19230769230769232, 0.11107421550722231, 0.16666666666666666, 0.18130841121495328, 0.22031571722717913, 0.21939953810623555, 0.14463452566096424, 0.279874213836478, 0.13814432989690723, 0.12373737373737374, 0.16666666666666666, 0.3068181818181818, 0.14463452566096424, 0.42857142857142855, 0.2288135593220339, 0.16666666666666666, 1.0, 1.0, 0.19436619718309858, 1.0, 0.0, 0.14285714285714285, 0.4, 0.16666666666666666, 1.0, 0.13636363636363635, 0.27038823799678385, 0.4, 0.21939953810623555, 0.14463452566096424, 0.1586139580283065, 0.10256410256410256, 0.13647997143027202, 0.16666666666666666, 1.0, 0.13821138211382114, 0.2269106931028639, 0.4, 0.6630824372759857, 0.23843317331106936, 0.04, 0.4, 0.27038823799678385, 0.4074074074074074, 0.4, 0.23618413992917694, 0.16666666666666666, 1.0, 0.1111111111111111, 0.11107421550722231, 0.16666666666666666, 0.18130841121495328, 0.22031571722717913, 0.1259259259259259, 0.14791666666666667, 0.11952554744525548, 0.16666666666666666, 0.14285714285714285, 0.18485523385300667, 0.17097862767154107, 0.15795090715048027, 0.19402985074626866, 0.12844708370617425, 0.24513776337115073, 0.2832764505119454, 0.4782608695652174, 0.23618413992917694, 0.18267419962335216, 0.23794037940379403, 0.14182475158084915, 0.764102564102564, 0.13286713286713286, 0.32187899835496253, 0.27147766323024053, 0.8333333333333334, 0.093558536181487, 0.1111111111111111, 0.2874859075535513, 0.1774193548387097, 0.45161290322580644, 0.4666666666666667, 0.0, 0.093558536181487, 0.16666666666666666, 0.13675213675213677, 1.0, 0.20222604141091743, 0.12844708370617425, 0.058549373725604426, 0.25654642006958434, 0.4, 0.02702702702702703, 0.1905829596412556, 0.1277533039647577, 0.15384615384615385, 0.23684210526315788, 0.13647997143027202, 0.15384615384615385, 0.14463452566096424, 0.4791666666666667, 0.16666666666666666, 0.16363636363636364, 0.16666666666666666, 0.4, 1.0, 0.4, 1.0, 0.4, 0.16666666666666666, 0.4, 0.6190476190476191, 0.32075471698113206, 0.6, 0.20388349514563106, 0.16466739367502725, 0.29910714285714285, 0.16466739367502725, 0.12844708370617425, 0.24513776337115073, 0.13744075829383887, 0.1873560040549258, 0.14463452566096424, 0.20329024676850763, 0.16466739367502725, 0.2727272727272727, 0.42857142857142855, 0.23618413992917694, 0.35, 0.3798103736754044, 0.2727272727272727, 0.093558536181487, 0.09774436090225563, 0.4, 0.20388349514563106, 0.5148247978436657, 0.6, 0.4, 0.17573385518590998, 0.3898305084745763, 0.5, 0.14463452566096424, 0.16666666666666666, 0.5714285714285714, 0.2269106931028639, 0.23843317331106936, 0.18134715025906736, 0.10427350427350428, 0.28779663403985634, 0.16666666666666666, 0.7171717171717171, 0.28779663403985634, 0.013422818791946308, 0.09318894271872329, 0.17857142857142858, 0.14182475158084915, 0.4, 0.25654642006958434, 0.16666666666666666, 0.3442622950819672, 0.093558536181487, 0.2460881934566145, 0.24157303370786518, 0.14463452566096424, 1.0, 0.16666666666666666, 0.18451612903225806, 0.43037974683544306, 0.13157894736842105, 0.22103533278553822, 0.16666666666666666, 0.4, 0.3274336283185841, 1.0, 0.15903614457831325, 0.19190283400809716, 0.12389380530973451, 0.29910714285714285, 0.5, 0.19190283400809716, 0.18697402122210025, 0.13157894736842105, 0.16666666666666666, 0.12844708370617425, 0.21687344913151366, 0.22103533278553822, 0.14285714285714285, 0.0, 0.19190283400809716, 0.12844708370617425, 0.24513776337115073, 0.22103533278553822, 0.16666666666666666, 0.22580645161290322, 0.28779663403985634, 0.16666666666666666, 0.4720812182741117, 0.093558536181487, 0.4, 0.0, 0.13647997143027202, 0.46563573883161513, 0.29793878825733916, 0.47058823529411764, 0.14791666666666667, 0.11952554744525548, 0.23843317331106936, 0.2186046511627907, 0.0, 0.28779663403985634, 0.16666666666666666, 0.13157894736842105, 0.3888888888888889, 0.13286713286713286, 0.28884254431699685, 0.4, 0.0, 0.23618413992917694, 0.1111111111111111, 0.28779663403985634, 0.16666666666666666, 0.28, 0.093558536181487, 0.16666666666666666, 0.5378787878787878, 0.27147766323024053, 1.0, 0.11952554744525548, 0.5520833333333334, 0.16666666666666666, 0.18451612903225806, 0.5, 0.14463452566096424, 0.1504222780187172, 0.05823902842947833, 0.058823529411764705, 0.11107421550722231, 0.16666666666666666, 0.16279069767441862, 0.22031571722717913, 0.1218026796589525, 0.09420713867758923, 0.32187899835496253, 0.4264705882352941, 0.22580645161290322, 0.28779663403985634, 0.09333333333333334, 0.23843317331106936, 0.13157894736842105, 0.09318894271872329, 0.0967741935483871, 0.20222604141091743, 0.0625, 0.32187899835496253, 0.2823920265780731, 0.1270854788877446, 0.05660377358490566, 0.32187899835496253, 0.16560509554140126, 0.093558536181487, 0.16666666666666666, 0.42857142857142855, 0.13225058004640372, 0.20222604141091743, 0.08931953955995921, 0.14867566216891553, 0.1506849315068493, 0.3798103736754044, 0.27147766323024053, 0.25, 0.19190283400809716, 0.08931953955995921, 0.38317757009345793, 0.10256410256410256, 0.14463452566096424, 0.2460881934566145, 0.27147766323024053, 0.19708029197080293, 0.17766497461928935, 0.23076923076923078, 0.12844708370617425, 0.21687344913151366, 0.4, 0.0, 0.21550946798917944, 0.22905027932960895, 0.5148247978436657, 0.2558139534883721, 0.14791666666666667, 0.08105906313645621, 0.14463452566096424, 0.05333333333333334, 0.1586139580283065, 0.19190283400809716, 0.2459016393442623, 0.38317757009345793, 0.2374429223744292, 0.38461538461538464, 0.15945945945945947, 0.12612612612612611, 1.0, 0.23618413992917694, 0.09420713867758923, 0.25663716814159293, 0.28779663403985634, 0.20222604141091743, 0.0625, 0.23843317331106936, 0.2823920265780731, 0.09318894271872329, 0.1767002928734136, 0.23794037940379403, 0.16666666666666666, 0.8461538461538461, 0.14463452566096424, 0.26666666666666666, 0.16666666666666666, 0.18451612903225806, 0.5, 0.0315893385982231, 0.12844708370617425, 0.24513776337115073, 0.15815085158150852, 0.16666666666666666, 0.6731517509727627, 0.13808049535603714, 0.4, 0.0, 0.22033898305084745, 0.13647997143027202, 0.1270854788877446, 0.23076923076923078, 0.18697402122210025, 0.13157894736842105, 0.23843317331106936, 0.2, 0.32558139534883723, 0.12749349522983522, 0.09318894271872329, 0.12389380530973451, 0.093558536181487, 0.16666666666666666, 0.4, 0.36363636363636365, 0.18697402122210025, 0.1111111111111111, 0.09318894271872329, 0.15332428765264586, 0.5148247978436657, 0.24880382775119617, 0.2, 0.11063627730294397, 0.10961420698101654, 0.088, 0.14463452566096424, 0.34502923976608185, 0.2473416551086454, 0.42857142857142855, 0.13157894736842105, 0.14463452566096424, 0.15918367346938775, 0.14463452566096424, 0.22508591065292097, 0.3333333333333333, 0.10961420698101654, 0.088, 0.16666666666666666, 0.3333333333333333, 0.14463452566096424, 0.07586206896551724, 0.24092409240924093, 0.18248175182481752, 0.16666666666666666, 0.06908665105386416, 0.21978021978021978, 0.13647997143027202, 0.23843317331106936, 0.0, 1.0, 0.1873560040549258, 0.1218026796589525, 0.2947976878612717, 0.09090909090909091, 0.19190283400809716, 0.08931953955995921, 0.10025062656641603, 0.16666666666666666, 0.20408163265306123, 0.28779663403985634, 0.3074101247248716, 0.14463452566096424, 0.14285714285714285, 0.23076923076923078, 0.14463452566096424, 0.3268765133171913, 0.2, 0.16466739367502725, 0.21550946798917944, 0.17329910141206675, 0.12844708370617425, 0.21687344913151366, 0.23794037940379403, 0.19190283400809716, 0.44851063829787235, 0.12844708370617425, 0.24513776337115073, 0.2696969696969697, 0.28779663403985634, 0.16666666666666666, 0.36619718309859156, 0.3888888888888889, 0.28779663403985634, 0.16666666666666666, 0.13157894736842105, 0.13286713286713286, 0.0030627871362940277, 0.06666666666666667, 0.5, 0.21311475409836064, 0.28779663403985634, 0.16666666666666666, 0.4, 1.0, 0.25654642006958434, 0.3116883116883117, 1.0, 0.09318894271872329, 0.15332428765264586, 0.23843317331106936, 0.31313131313131315, 0.22031571722717913, 0.2269106931028639, 0.18697402122210025, 0.3333333333333333, 0.093558536181487, 0.16666666666666666, 0.3442622950819672, 0.4, 0.36363636363636365, 0.23618413992917694, 0.4, 0.2473416551086454, 0.07692307692307693, 0.2823920265780731, 0.2269106931028639, 0.3202614379084967, 0.3333333333333333, 0.12749349522983522, 0.14791666666666667, 0.09839816933638444, 0.019230769230769232, 0.12844708370617425, 0.24513776337115073, 0.3333333333333333, 0.13647997143027202, 0.16049382716049382, 0.3268765133171913, 0.22, 0.12612612612612611, 1.0, 0.38317757009345793, 0.14791666666666667, 0.4, 0.15384615384615385, 0.23843317331106936, 0.17307692307692307, 0.28779663403985634, 0.4, 0.23391435334624489, 0.10683012259194395, 0.23843317331106936, 0.47368421052631576, 0.28779663403985634, 0.16666666666666666, 0.4, 0.3274336283185841, 1.0, 0.13286713286713286, 0.16379310344827586, 0.16666666666666666, 0.38461538461538464, 0.23618413992917694, 0.16666666666666666, 0.3333333333333333, 0.1873560040549258, 0.07692307692307693, 0.4, 0.6216216216216216, 0.18697402122210025, 0.5714285714285714, 0.1873560040549258, 0.42857142857142855, 0.13647997143027202, 0.75, 0.25654642006958434, 0.16666666666666666, 1.0, 0.28779663403985634, 0.15384615384615385, 0.22031571722717913, 0.09420713867758923, 0.4, 0.1873560040549258, 0.32187899835496253, 0.2857142857142857, 0.14463452566096424, 0.6363636363636364, 0.18267419962335216, 0.11627906976744186, 0.38317757009345793, 0.13286713286713286, 0.4, 0.12844708370617425, 0.5714285714285714, 0.18697402122210025, 0.13157894736842105, 0.1873560040549258, 0.6666666666666666, 0.2269106931028639, 0.20898876404494382, 0.52, 0.355397951142632, 0.14791666666666667, 0.1767002928734136, 0.2, 0.7037037037037037, 0.4, 0.4, 0.4, 1.0, 0.3333333333333333, 0.4, 0.08761329305135952, 0.3333333333333333, 0.28779663403985634, 1.0, 0.22905027932960895, 0.22103533278553822, 0.16666666666666666, 0.12056737588652482, 0.17766497461928935, 0.32231404958677684, 0.16666666666666666, 0.30158730158730157, 0.54, 0.28779663403985634, 0.16666666666666666, 0.3068181818181818, 0.3333333333333333, 0.4, 0.12844708370617425, 0.24513776337115073, 0.38613861386138615, 0.0, 0.5277777777777778, 0.717948717948718, 0.08931953955995921, 0.3898305084745763, 0.14867566216891553, 0.020618556701030927, 0.17097862767154107, 0.18697402122210025, 0.013422818791946308, 0.2696969696969697, 0.22900763358778625, 0.20388349514563106, 0.19190283400809716, 0.12844708370617425, 0.24513776337115073, 0.3333333333333333, 0.14285714285714285, 0.17573385518590998, 0.13382269904009034, 0.21052631578947367, 0.010810810810810811, 0.23618413992917694, 0.006150061500615006, 0.15384615384615385, 0.21212121212121213, 0.1218026796589525, 0.19190283400809716, 0.12844708370617425, 0.24513776337115073, 0.4, 0.16466739367502725, 0.19190283400809716, 0.203125, 0.8181818181818182, 0.19190283400809716, 0.08931953955995921, 0.14867566216891553, 0.16666666666666666, 0.4251497005988024, 0.25654642006958434, 0.16666666666666666, 0.4, 0.4, 0.2903225806451613, 1.0, 0.3103448275862069, 0.09318894271872329, 0.1767002928734136, 0.12844708370617425, 0.21687344913151366, 0.40540540540540543, 0.14463452566096424, 0.14960629921259844, 0.19190283400809716, 0.23391435334624489, 0.2460881934566145, 0.6140350877192983, 0.23618413992917694, 0.09318894271872329, 0.12612612612612611, 0.14182475158084915, 0.28779663403985634, 0.16666666666666666, 0.76, 0.16883116883116883, 0.17766497461928935, 0.12612612612612611, 0.25654642006958434, 0.1781818181818182, 0.06666666666666667, 0.12749349522983522, 0.19190283400809716, 0.12389380530973451, 0.0784313725490196, 0.20222604141091743, 0.1767002928734136, 0.23794037940379403, 0.23843317331106936, 0.17472118959107807, 0.11107421550722231, 0.29910714285714285, 0.16666666666666666, 0.3333333333333333, 0.13793103448275862, 0.093558536181487, 0.16666666666666666, 0.1781818181818182, 0.717948717948718, 0.11107421550722231, 0.3333333333333333, 0.4, 0.12844708370617425, 0.24513776337115073, 1.0, 0.2832764505119454, 0.5378787878787878, 0.11107421550722231, 0.19047619047619047, 0.2269106931028639, 0.16666666666666666, 0.31313131313131315, 0.3333333333333333, 0.14791666666666667, 0.12844708370617425, 0.2520775623268698, 0.47058823529411764, 0.4, 0.14463452566096424, 0.20309094511591044, 0.23843317331106936, 0.24770642201834864, 0.09318894271872329, 0.12844708370617425, 0.15940224159402241, 0.15384615384615385, 0.2631578947368421, 0.22103533278553822, 0.09420713867758923, 0.20222604141091743, 0.05405405405405406, 0.12844708370617425, 0.21687344913151366, 0.15463917525773196, 0.023809523809523808, 0.12844708370617425, 0.24513776337115073, 0.23595505617977527, 0.28779663403985634, 1.0, 0.6226415094339622, 0.20222604141091743, 0.12844708370617425, 0.058549373725604426, 0.25654642006958434, 0.13647997143027202, 0.23843317331106936, 0.5243243243243243, 0.7333333333333333, 0.19047619047619047, 0.2269106931028639, 0.16466739367502725, 0.1873560040549258, 0.23843317331106936, 0.391304347826087, 0.16466739367502725, 0.013333333333333334, 0.23618413992917694, 0.4, 0.11952554744525548, 0.13382269904009034, 0.25, 0.2, 0.14463452566096424, 0.18697402122210025, 0.4489795918367347, 0.1111111111111111, 0.4, 0.23618413992917694, 0.0030627871362940277, 0.4, 0.12844708370617425, 0.24513776337115073, 0.3798103736754044, 0.7058823529411765, 0.3798103736754044, 1.0, 0.11107421550722231, 0.16666666666666666, 0.1781818181818182, 0.2832764505119454, 0.5378787878787878, 0.14285714285714285, 0.1767002928734136, 0.30566037735849055, 0.21122994652406418, 0.22508591065292097, 0.36, 0.3333333333333333, 0.4, 0.8867924528301887, 0.2268370607028754, 0.22508591065292097, 0.3473684210526316, 0.5862068965517241, 0.1218026796589525, 0.11088810837932765, 0.12844708370617425, 0.24513776337115073, 0.3798103736754044, 0.5, 0.14463452566096424, 0.21439891328956306, 1.0, 0.25675675675675674, 0.31417624521072796, 0.09318894271872329, 0.06711409395973154, 0.23843317331106936, 0.30158730158730157, 1.0, 0.4, 1.0, 0.1873560040549258, 0.4, 0.5263157894736842, 0.3333333333333333, 0.4, 0.18697402122210025, 0.1111111111111111, 0.5555555555555556, 0.14285714285714285, 0.14285714285714285, 0.23618413992917694, 0.16666666666666666, 0.04, 0.4, 0.12844708370617425, 0.24513776337115073, 0.5, 1.0, 0.21739130434782608, 0.15384615384615385, 0.2269106931028639, 0.23843317331106936, 0.3333333333333333, 0.16466739367502725, 0.3333333333333333, 0.4, 0.22, 0.4, 0.05560165975103734, 0.16666666666666666, 0.5, 0.25, 0.7142857142857143, 0.3333333333333333, 0.20985691573926868, 0.10961420698101654, 0.23076923076923078, 0.17097862767154107, 0.673469387755102, 0.11107421550722231, 0.16666666666666666, 0.16666666666666666, 0.28779663403985634, 0.16666666666666666, 1.0, 0.15315315315315314, 0.058823529411764705, 0.45454545454545453, 1.0, 0.3202614379084967, 0.23618413992917694, 0.12749349522983522, 0.1626848691695108, 0.22105263157894736, 0.11952554744525548, 0.19190283400809716, 0.093558536181487, 0.19190283400809716, 0.3114754098360656, 0.19190283400809716, 0.12844708370617425, 0.24513776337115073, 0.4, 0.3333333333333333, 0.19047619047619047, 0.16666666666666666, 0.24157303370786518, 0.11088810837932765, 0.12844708370617425, 0.24513776337115073, 0.23843317331106936, 0.31313131313131315, 0.52, 0.16466739367502725, 0.17573385518590998, 0.13382269904009034, 0.030303030303030304, 0.14463452566096424, 0.23843317331106936, 0.21687344913151366, 0.3333333333333333, 0.16466739367502725, 0.14791666666666667, 0.29910714285714285, 0.1767002928734136, 0.5278969957081545, 0.18485523385300667, 0.19190283400809716, 0.17097862767154107, 0.11107421550722231, 0.22508591065292097, 0.5378787878787878, 0.4, 0.23618413992917694, 0.2, 0.10392609699769054, 0.4174757281553398, 0.3103448275862069, 0.4, 0.631578947368421, 0.17573385518590998, 0.21550946798917944, 0.25, 0.22508591065292097, 0.18451612903225806, 0.10392609699769054, 0.5862068965517241, 0.3103448275862069, 0.13286713286713286, 0.22508591065292097, 0.4, 0.23618413992917694, 0.2, 0.0454065469904963, 0.4, 0.2, 0.42857142857142855, 0.23618413992917694, 0.4, 0.13286713286713286, 0.4489795918367347, 0.367816091954023, 0.11952554744525548, 0.0, 0.15795090715048027, 0.5862068965517241, 0.23618413992917694, 0.4, 0.0, 0.23618413992917694, 1.0, 1.0, 0.08931953955995921, 0.21939953810623555, 0.14463452566096424, 0.11952554744525548, 0.15795090715048027, 0.31313131313131315, 0.5388235294117647, 0.5862068965517241, 0.13647997143027202, 0.20898876404494382, 0.2919254658385093, 0.12749349522983522, 0.2473416551086454, 0.39416745061147695, 0.21550946798917944, 0.16666666666666666, 0.12844708370617425, 0.21687344913151366, 0.2972972972972973, 0.25663716814159293, 0.16666666666666666, 0.0454065469904963, 1.0, 0.1873560040549258, 0.15474209650582363, 0.093558536181487, 0.11933174224343675, 0.11107421550722231, 0.18697402122210025, 0.5378787878787878, 0.1218026796589525, 0.14867566216891553, 0.05952380952380952, 0.14463452566096424, 0.5950413223140496, 0.1270854788877446, 0.23618413992917694, 0.47058823529411764, 0.1431693989071038, 0.13793103448275862, 0.25654642006958434, 0.16666666666666666, 0.4, 0.3103448275862069, 0.20309094511591044, 0.3268765133171913, 0.2565947242206235, 0.2, 0.10810810810810811, 0.4, 0.23618413992917694, 0.0, 1.0, 0.625, 0.0652699435938759, 0.0, 0.9398496240601504, 0.09318894271872329, 0.12844708370617425, 0.2520775623268698, 0.22448979591836735, 0.14463452566096424, 0.2, 0.20222604141091743, 0.09318894271872329, 0.01818181818181818, 0.12844708370617425, 0.21687344913151366, 0.09333333333333334, 0.12373737373737374, 0.18697402122210025, 0.1781818181818182, 0.625, 0.41509433962264153, 0.2696969696969697, 0.3333333333333333, 0.25654642006958434, 0.6521739130434783, 0.09318894271872329, 0.15332428765264586, 0.5, 0.14285714285714285, 0.21439891328956306, 0.14463452566096424, 0.279874213836478, 0.14463452566096424, 0.16666666666666666, 0.0, 0.1781818181818182, 0.22900763358778625, 0.12844708370617425, 0.24513776337115073, 0.6, 0.15815085158150852, 0.4, 0.14463452566096424, 0.1586139580283065, 0.2269106931028639, 0.14285714285714285, 0.15903614457831325, 0.09318894271872329, 0.21550946798917944, 0.08860759493670886, 0.12844708370617425, 0.21687344913151366, 0.23794037940379403, 0.16666666666666666, 0.1048951048951049, 1.0, 0.15903614457831325, 0.11107421550722231, 0.16666666666666666, 0.15719063545150502, 0.5789473684210527, 0.09318894271872329, 0.15332428765264586, 0.23843317331106936, 0.0652699435938759, 0.28779663403985634, 0.48936170212765956, 0.23618413992917694, 0.30566037735849055, 0.20388349514563106, 0.18578352180936994, 0.23843317331106936, 0.4461538461538462, 0.05555555555555555, 0.11107421550722231, 0.4, 0.12844708370617425, 0.24513776337115073, 0.3252032520325203, 1.0, 0.16666666666666666, 0.1781818181818182, 0.4, 0.4, 0.09318894271872329, 0.12844708370617425, 0.15940224159402241, 0.05952380952380952, 0.2, 0.09318894271872329, 0.13559322033898305, 0.16666666666666666, 0.023255813953488372, 0.48717948717948717, 0.28779663403985634, 0.4, 0.09782608695652174, 0.13647997143027202, 0.11063627730294397, 0.23076923076923078, 0.16666666666666666, 0.625, 0.0652699435938759, 0.3188405797101449, 0.3234567901234568, 0.4461538461538462, 0.09420713867758923, 0.20222604141091743, 0.21939953810623555, 0.14463452566096424, 0.14960629921259844, 0.46563573883161513, 0.22103533278553822, 0.18697402122210025, 0.27038823799678385, 0.07984031936127745, 0.14182475158084915, 0.16049382716049382, 0.28779663403985634, 0.12582781456953643, 0.47058823529411764, 0.3268765133171913, 0.1781818181818182, 0.16666666666666666, 1.0, 0.3068181818181818, 0.2832764505119454, 0.5378787878787878, 0.11107421550722231, 0.23618413992917694, 0.279874213836478, 0.14463452566096424, 0.16666666666666666, 0.2832764505119454, 0.23595505617977527, 0.28779663403985634, 0.18697402122210025, 0.8581560283687943, 0.22784810126582278, 0.4, 0.4, 0.4, 0.3116883116883117, 0.28779663403985634, 0.16666666666666666, 1.0, 0.20388349514563106, 0.23076923076923078, 0.12582781456953643, 0.28779663403985634, 0.4, 0.2523364485981308, 0.12844708370617425, 0.24513776337115073, 0.1, 0.5, 0.13814432989690723, 0.14463452566096424, 0.4, 0.09302325581395349, 0.3333333333333333, 0.4444444444444444, 0.09318894271872329, 0.15332428765264586, 0.14463452566096424, 0.22448979591836735, 0.18697402122210025, 0.22900763358778625, 0.12844708370617425, 0.24513776337115073, 0.4, 0.18134715025906736, 0.14463452566096424, 0.2460881934566145, 0.18578352180936994, 0.2269106931028639, 0.16666666666666666, 0.35, 0.43283582089552236, 0.093558536181487, 0.16666666666666666, 0.25, 0.36363636363636365, 0.093558536181487, 0.19190283400809716, 0.12844708370617425, 0.24513776337115073, 0.24324324324324326, 0.1781818181818182, 0.06666666666666667, 0.23618413992917694, 0.007228915662650603, 0.010526315789473684, 0.23391435334624489, 0.16666666666666666, 0.6610169491525424, 0.2619047619047619, 0.20898876404494382, 0.4074074074074074, 0.6666666666666666, 0.23618413992917694, 0.8245614035087719, 0.13286713286713286, 0.16666666666666666, 0.4, 0.4, 0.3333333333333333, 0.2, 0.15384615384615385, 0.14666666666666667, 0.23076923076923078, 0.13647997143027202, 0.16666666666666666, 0.21687344913151366, 0.0, 0.16466739367502725, 0.2649164677804296, 0.3020833333333333, 0.2857142857142857, 0.4, 0.27038823799678385, 0.21439891328956306, 0.4838709677419355, 0.25663716814159293, 0.093558536181487, 0.16666666666666666, 0.15545755237045203, 0.17766497461928935, 0.20309094511591044, 0.20898876404494382, 0.24880382775119617, 0.09433962264150944, 0.15384615384615385, 0.09318894271872329, 0.279874213836478, 0.14463452566096424, 0.22508591065292097, 0.2832764505119454, 0.5378787878787878, 0.29910714285714285, 0.28779663403985634, 0.1431693989071038, 0.11952554744525548, 0.22522522522522523, 0.0, 0.2631578947368421, 0.22508591065292097, 0.3888888888888889, 0.09420713867758923, 0.20222604141091743, 0.21939953810623555, 0.14463452566096424, 0.07984031936127745, 0.16049382716049382, 0.28779663403985634, 0.16666666666666666, 0.16363636363636364, 0.27038823799678385, 0.08105906313645621, 0.4759036144578313, 0.13647997143027202, 0.13814432989690723, 0.14463452566096424, 0.16666666666666666, 1.0, 0.4, 0.00797408422626464, 0.47058823529411764, 0.010810810810810811, 0.4, 0.15474209650582363, 0.17573385518590998, 0.14182475158084915, 0.28884254431699685, 0.31313131313131315, 0.5, 0.02127659574468085, 0.18697402122210025, 0.22900763358778625, 0.18485523385300667, 0.1431693989071038, 0.17097862767154107, 0.3333333333333333, 0.28779663403985634, 0.4, 0.2523364485981308, 0.09420713867758923, 0.20222604141091743, 0.08105906313645621, 0.23843317331106936, 0.10843373493975904, 0.056, 0.13647997143027202, 0.16666666666666666, 0.6216216216216216, 0.27038823799678385, 0.0, 0.023255813953488372, 0.23618413992917694, 0.21939953810623555, 0.23843317331106936, 0.31313131313131315, 0.006987577639751553, 0.47058823529411764, 0.11088810837932765, 0.5378787878787878, 0.13647997143027202, 0.14182475158084915, 0.31313131313131315, 0.19480519480519481, 0.09318894271872329, 0.12844708370617425, 0.15940224159402241, 0.0, 0.14182475158084915, 0.28779663403985634, 0.2460881934566145, 0.4, 0.3274336283185841, 0.2, 0.11107421550722231, 0.16666666666666666, 0.2857142857142857, 0.4, 0.1724137931034483, 0.09318894271872329, 0.12844708370617425, 0.15940224159402241, 0.48717948717948717, 0.28779663403985634, 0.375, 0.14463452566096424, 0.14867566216891553, 0.25654642006958434, 0.23843317331106936, 0.08, 0.2288135593220339, 0.6216216216216216, 0.2269106931028639, 0.2, 0.23618413992917694, 0.4, 0.4, 0.13286713286713286, 0.16666666666666666, 0.4, 1.0, 0.09318894271872329, 0.12844708370617425, 0.15940224159402241, 0.0, 0.23843317331106936, 0.2222222222222222, 0.28779663403985634, 0.2460881934566145, 0.28884254431699685, 0.2, 0.09318894271872329, 0.14606741573033707, 0.093558536181487, 0.16666666666666666, 0.3442622950819672, 0.11088810837932765, 0.3020833333333333, 0.09318894271872329, 0.68, 0.16666666666666666, 0.14285714285714285, 0.25654642006958434, 0.6216216216216216, 0.15903614457831325, 0.09318894271872329, 0.12844708370617425, 0.2520775623268698, 0.22448979591836735, 0.14463452566096424, 0.16112531969309463, 0.16666666666666666, 0.4, 0.5294117647058824, 0.23618413992917694, 0.16666666666666666, 0.29910714285714285, 0.2460881934566145, 0.4, 0.3274336283185841, 0.2, 0.08931953955995921, 0.20329024676850763, 0.19190283400809716, 0.29910714285714285, 1.0, 0.09318894271872329, 0.1391304347826087, 0.2, 1.0, 0.09090909090909091, 0.15384615384615385, 0.25, 0.11088810837932765, 0.14666666666666667, 0.18697402122210025, 0.22900763358778625, 0.23618413992917694, 0.3333333333333333, 0.28779663403985634, 0.20898876404494382, 0.31313131313131315, 0.21687344913151366, 0.0, 0.14791666666666667, 0.11952554744525548, 0.14182475158084915, 0.1111111111111111, 0.0, 0.31417624521072796, 0.11107421550722231, 0.16666666666666666, 0.15384615384615385, 0.5692307692307692, 0.10961420698101654, 0.12844708370617425, 0.058549373725604426, 0.2727272727272727, 0.14463452566096424, 0.16666666666666666, 0.3333333333333333, 0.20222604141091743, 0.22448979591836735, 0.09420713867758923, 0.20222604141091743, 0.13559322033898305, 0.093558536181487, 0.18697402122210025, 0.22900763358778625, 0.12844708370617425, 0.24513776337115073, 0.16363636363636364, 0.0684931506849315, 0.10961420698101654, 0.11952554744525548, 0.4, 0.2523364485981308, 0.12844708370617425, 0.24513776337115073, 0.4, 0.093558536181487, 0.16666666666666666, 0.15545755237045203, 0.23618413992917694, 0.16666666666666666, 0.3798103736754044, 0.4074074074074074, 0.1329479768786127, 0.6470588235294118, 0.093558536181487, 0.16666666666666666, 0.673469387755102, 0.7142857142857143, 0.20222604141091743, 0.1767002928734136, 0.07984031936127745, 0.23843317331106936, 0.5, 0.11107421550722231, 0.4, 0.4, 0.35294117647058826, 0.44851063829787235, 0.20309094511591044, 0.23843317331106936, 0.078125, 0.28779663403985634, 1.0, 0.13647997143027202, 0.32187899835496253, 0.5, 0.3333333333333333, 0.23618413992917694, 0.4, 0.17097862767154107, 0.11088810837932765, 0.23843317331106936, 0.1781818181818182, 0.5735294117647058, 0.6226415094339622, 0.17766497461928935, 0.12844708370617425, 0.24513776337115073, 0.23843317331106936, 0.4, 0.4, 0.1111111111111111, 0.1873560040549258, 0.04819277108433735, 0.14463452566096424, 0.16666666666666666, 0.23843317331106936, 0.4, 0.3274336283185841, 0.12844708370617425, 0.24513776337115073, 0.1111111111111111, 0.12844708370617425, 0.24513776337115073, 0.35294117647058826, 0.09420713867758923, 0.20222604141091743, 0.12844708370617425, 0.058549373725604426, 0.27388535031847133, 0.25655644241733183, 0.014084507042253521, 0.11107421550722231, 0.09318894271872329, 0.15474209650582363, 0.3, 0.09090909090909091, 0.23843317331106936, 0.2830188679245283, 0.5, 0.15057915057915058, 0.16466739367502725, 0.5619047619047619, 0.14285714285714285, 0.12844708370617425, 0.24513776337115073, 0.6, 0.4, 0.40336134453781514, 0.19190283400809716, 0.12844708370617425, 0.24513776337115073, 0.4, 1.0, 0.5, 0.3707865168539326, 0.0, 0.047619047619047616, 0.2473416551086454, 0.5384615384615384, 0.13286713286713286, 1.0, 0.24324324324324326, 0.17766497461928935, 0.14285714285714285, 0.14463452566096424, 0.45454545454545453, 0.4, 0.13636363636363635, 0.35294117647058826, 0.1873560040549258, 0.16666666666666666, 1.0, 0.23618413992917694, 1.0, 0.16666666666666666, 0.2832764505119454, 0.5378787878787878, 0.1873560040549258, 1.0, 0.4474327628361858, 0.0, 0.014827018121911038, 1.0, 0.22905027932960895, 0.5148247978436657, 0.28779663403985634, 0.20222604141091743, 0.35, 0.16666666666666666, 0.5416666666666666, 0.15815085158150852, 0.12844708370617425, 0.4, 1.0, 0.12844708370617425, 0.01818181818181818, 0.12844708370617425, 0.21687344913151366, 0.22508591065292097, 0.19174757281553398, 0.4474327628361858, 0.0, 0.2788971367974549, 0.23618413992917694, 0.1873560040549258, 0.11107421550722231, 0.19190283400809716, 0.023255813953488372, 0.23843317331106936, 0.2222222222222222, 0.28779663403985634, 0.16049382716049382, 0.28779663403985634, 0.14285714285714285, 0.12844708370617425, 0.24513776337115073, 0.0, 1.0, 0.09318894271872329, 0.06711409395973154, 0.20222604141091743, 0.12844708370617425, 0.14124293785310735, 0.3563218390804598, 0.30566037735849055, 0.29910714285714285, 0.14791666666666667, 0.08105906313645621, 0.1873560040549258, 0.13647997143027202, 0.15795090715048027, 0.12056737588652482, 0.14463452566096424, 0.07066381156316917, 0.15384615384615385, 0.2269106931028639, 0.16666666666666666, 0.4, 0.0, 0.1111111111111111, 0.12844708370617425, 0.24513776337115073, 0.35294117647058826, 0.18697402122210025, 0.03875968992248062, 0.1873560040549258, 0.0, 0.00797408422626464, 0.1270854788877446, 0.19436619718309858, 0.3333333333333333, 0.7684210526315789, 0.9759036144578314, 0.16666666666666666, 1.0, 0.3068181818181818, 0.1873560040549258, 0.23843317331106936, 0.6, 0.28779663403985634, 0.19436619718309858, 0.3333333333333333, 0.7684210526315789, 0.20222604141091743, 0.20309094511591044, 0.2, 0.23618413992917694, 0.07692307692307693, 0.14463452566096424, 0.2823920265780731, 0.18697402122210025, 0.03875968992248062, 0.25654642006958434, 0.491679273827534, 0.3333333333333333, 0.40336134453781514, 0.18697402122210025, 0.18593644354293443, 0.23618413992917694, 0.39416745061147695, 0.19190283400809716, 0.14463452566096424, 0.2, 0.21052631578947367, 0.14463452566096424, 0.14285714285714285, 0.27038823799678385, 0.3701657458563536, 0.14463452566096424, 0.18697402122210025, 0.8581560283687943, 0.38317757009345793, 0.279874213836478, 0.14463452566096424, 0.18697402122210025, 0.03875968992248062, 0.1873560040549258, 0.0, 0.00797408422626464, 0.1270854788877446, 0.19436619718309858, 0.3333333333333333, 0.7684210526315789, 0.9759036144578314, 0.16666666666666666, 1.0, 0.3068181818181818, 0.1873560040549258, 0.23843317331106936, 0.6, 0.28779663403985634, 0.19436619718309858, 0.3333333333333333, 0.7684210526315789, 0.20222604141091743, 0.20309094511591044, 0.2, 0.23618413992917694, 0.07692307692307693, 0.14463452566096424, 0.2823920265780731, 0.18697402122210025, 0.03875968992248062, 0.25654642006958434, 0.19190283400809716, 0.12844708370617425, 0.24513776337115073, 0.3333333333333333, 0.40336134453781514, 0.18697402122210025, 0.18593644354293443, 0.23618413992917694, 0.39416745061147695, 0.19190283400809716, 0.14463452566096424, 0.2, 0.21052631578947367, 0.14463452566096424, 0.14285714285714285, 0.27038823799678385, 0.3701657458563536, 0.13286713286713286, 0.18697402122210025, 0.8581560283687943, 0.08116102791643558, 0.47058823529411764]\n",
      "Graham interesting: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.0, 0.0, 1.0, 1.0, 0.0, 1.0, 1.0, 1.0]\n",
      "Bayes interesting ≤ 7: 1.0 [0.2857142857142857, 1.0, 0.46956521739130436, 0.047058823529411764, 0.2615079365079365, 0.3540692235734331, 0.1733615221987315, 0.23975044563279854, 0.2857142857142857, 1.0, 0.46956521739130436, 0.2615079365079365, 0.30704225352112674, 0.23975044563279854, 0.047058823529411764, 0.2615079365079365, 0.314176245210728, 0.4768518518518519, 0.12924424972617743, 0.3155852219807513, 0.35459940652818994, 0.20454545454545456, 0.4, 0.12924424972617743, 0.408335762168464, 1.0, 0.2857142857142857, 1.0, 0.24018016130721695, 0.2857142857142857, 0.4, 1.0, 0.408335762168464, 0.31504315659679405, 0.047058823529411764, 1.0, 1.0, 0.44695975503062113, 0.2857142857142857, 0.11548223350253806, 0.046153846153846156, 0.5, 0.01652892561983471, 0.35, 0.40298507462686567, 0.22022471910112362, 0.2857142857142857, 0.46956521739130436, 0.38211805555555556, 0.33333333333333337, 0.2857142857142857, 0.0, 0.19994022713687984, 0.2857142857142857, 0.3069620253164557, 0.36107986501687284, 0.4, 0.314176245210728, 0.22765282583621685, 0.39375203384315005, 0.2827715355805243, 0.4, 0.49333333333333335, 0.23255813953488372, 0.2577132486388385, 0.22765282583621685, 0.40265486725663713, 0.23605577689243026, 0.391304347826087, 0.36204576043068637, 0.5, 0.4, 0.16111111111111112, 0.7692307692307693, 0.12254160363086232, 0.0, 0.9689922480620156, 0.1478743068391867, 0.2426550598476605, 0.4604810996563574, 0.2857142857142857, 0.4, 0.7289719626168224, 0.2204724409448819, 0.36204576043068637, 0.4, 0.03726708074534161, 0.5789473684210525, 0.8, 0.44695975503062113, 0.2857142857142857, 0.4, 0.4, 0.5, 0.4, 0.4752475247524752, 0.44695975503062113, 0.2857142857142857, 1.0, 0.4, 0.4, 0.5217391304347826, 0.30246913580246915, 0.5, 0.408335762168464, 0.2857142857142857, 0.375, 0.24018016130721695, 0.2857142857142857, 0.44148936170212766, 0.26174496644295303, 0.9807692307692307, 0.4373464373464373, 0.25271739130434784, 0.3220108695652174, 0.4578313253012048, 0.24841772151898736, 0.47692307692307695, 0.276595744680851, 0.2577132486388385, 0.17647058823529413, 0.6666666666666666, 0.3220108695652174, 0.2920268972142171, 0.38211805555555556, 0.19999999999999998, 0.3220108695652174, 0.2920268972142171, 0.1711084191399152, 0.4870022123893805, 0.5, 0.8243243243243243, 0.2827715355805243, 0.3891402714932127, 0.3364193328795099, 0.21352893235533824, 0.5, 0.25271739130434784, 0.14351851851851852, 0.4870022123893805, 0.5576923076923077, 1.0, 0.21715526601520088, 0.3364193328795099, 0.29540481400437635, 0.22765282583621685, 0.35644371941272435, 0.2666666666666667, 0.0, 0.1711084191399152, 0.2857142857142857, 0.8243243243243243, 0.25, 0.25271739130434784, 0.2857142857142857, 1.0, 0.46956521739130436, 0.25, 0.25271739130434784, 0.2857142857142857, 0.31154684095860563, 0.047058823529411764, 0.44695975503062113, 0.2857142857142857, 1.0, 0.46956521739130436, 0.2857142857142857, 0.8, 0.9236641221374046, 0.44695975503062113, 0.2857142857142857, 0.30246913580246915, 1.0, 0.46956521739130436, 0.44148936170212766, 0.6995073891625615, 0.33870967741935487, 1.0, 0.2857142857142857, 0.4, 0.46956521739130436, 0.24018016130721695, 0.2857142857142857, 0.6197183098591549, 0.3343108504398827, 0.5378151260504201, 0.17049009384775807, 0.22765282583621685, 0.2749731471535983, 0.35802469135802467, 0.2857142857142857, 0.4915254237288135, 0.25271739130434784, 0.3559322033898305, 0.8666666666666666, 0.36988950276243093, 0.0061068702290076335, 0.0, 0.3017241379310345, 0.2989347536617843, 0.23605577689243026, 1.0, 0.2857142857142857, 0.19999999999999998, 0.5217391304347826, 0.36988950276243093, 0.007407407407407408, 0.0, 0.24806201550387597, 0.026490066225165563, 0.44695975503062113, 0.0, 0.27167630057803466, 0.37267080745341613, 0.33870967741935487, 0.304635761589404, 0.408335762168464, 0.024793388429752063, 0.19999999999999998, 0.0061068702290076335, 0.05472636815920398, 0.4, 0.2706193131787496, 0.2920268972142171, 0.1711084191399152, 0.2857142857142857, 0.6413793103448276, 0.314176245210728, 0.3220108695652174, 0.19708029197080293, 0.3559322033898305, 0.3220108695652174, 0.22765282583621685, 0.39375203384315005, 0.7777777777777778, 0.6995073891625615, 0.1711084191399152, 0.024793388429752063, 0.38211805555555556, 0.25886447683271696, 0.5714285714285715, 0.25271739130434784, 0.4604810996563574, 0.4, 0.0, 0.19994022713687984, 0.2857142857142857, 0.36107986501687284, 0.0061068702290076335, 0.3157894736842105, 1.0, 0.2577132486388385, 0.26588235294117646, 0.23605577689243026, 0.19879518072289157, 0.1711084191399152, 0.39497716894977164, 0.4893617021276596, 0.4873949579831933, 0.44695975503062113, 0.5, 0.32547169811320753, 0.5, 0.38211805555555556, 0.2857142857142857, 0.30246913580246915, 0.4, 0.2706193131787496, 0.2857142857142857, 0.37241379310344824, 0.2857142857142857, 0.8474576271186441, 0.2888888888888889, 0.25271739130434784, 0.49107142857142855, 0.2857142857142857, 0.0, 0.44695975503062113, 0.2857142857142857, 0.5, 0.4230769230769231, 0.2577132486388385, 0.22399999999999998, 0.3636363636363636, 0.36204576043068637, 0.2857142857142857, 0.25, 0.2577132486388385, 0.16176470588235292, 0.25271739130434784, 0.1478743068391867, 0.5, 0.32547169811320753, 0.5, 0.38211805555555556, 0.4, 0.4444444444444445, 0.4373464373464373, 0.2577132486388385, 0.32015065913371, 0.1478743068391867, 0.2827715355805243, 0.31504315659679405, 0.2204724409448819, 0.2857142857142857, 0.8474576271186441, 0.3076923076923077, 0.25271739130434784, 0.3791419602371817, 0.4, 0.40963855421686746, 0.1, 0.3220108695652174, 0.2666666666666667, 0.3850561797752809, 0.4, 0.25, 0.0, 0.22765282583621685, 0.39375203384315005, 0.7012987012987013, 0.2204724409448819, 0.3850561797752809, 0.33333333333333337, 0.46634615384615385, 0.44695975503062113, 0.4927007299270073, 0.4347826086956522, 0.2744282744282744, 0.5979381443298969, 0.6354044548651817, 0.21715526601520088, 0.2204724409448819, 0.2857142857142857, 1.0, 0.33333333333333337, 0.27984344422700586, 0.4, 0.36204576043068637, 0.3850561797752809, 0.33333333333333337, 0.2827715355805243, 0.4578313253012048, 1.0, 0.40298507462686567, 0.0909090909090909, 0.4, 0.0, 0.38211805555555556, 1.0, 0.38211805555555556, 0.2753623188405797, 0.2857142857142857, 0.7012987012987013, 0.38211805555555556, 1.0, 0.44695975503062113, 0.7386363636363636, 0.046153846153846156, 0.26448362720403024, 0.0, 0.21715526601520088, 0.3220108695652174, 0.3155852219807513, 0.2827715355805243, 0.2577132486388385, 0.05472636815920398, 0.4927007299270073, 0.30246913580246915, 0.44148936170212766, 0.6995073891625615, 0.2857142857142857, 1.0, 0.46956521739130436, 0.32, 0.36988950276243093, 0.31504315659679405, 0.9236641221374046, 0.44695975503062113, 0.2827715355805243, 0.3069620253164557, 0.28415300546448086, 0.408335762168464, 0.051094890510948905, 0.2857142857142857, 1.0, 0.46956521739130436, 0.375, 0.6592292089249493, 0.0, 0.36988950276243093, 0.6592292089249493, 0.2666666666666667, 0.21576763485477177, 0.5277777777777778, 0.36988950276243093, 0.6592292089249493, 0.30303030303030304, 0.27083333333333337, 0.3917525773195876, 0.19994022713687984, 0.2857142857142857, 0.2666666666666667, 0.21576763485477177, 0.5277777777777778, 0.48351648351648346, 0.37775010377750107, 0.29310344827586204, 0.39846743295019155, 1.0, 0.2857142857142857, 0.6995073891625615, 0.2857142857142857, 0.31154684095860563, 0.12924424972617743, 0.38211805555555556, 0.04222648752399232, 0.13636363636363635, 0.22399999999999998, 0.5833333333333334, 0.24018016130721695, 0.2857142857142857, 0.31154684095860563, 0.3343108504398827, 0.3838383838383838, 0.2857142857142857, 0.2666666666666667, 0.21576763485477177, 0.5277777777777778, 0.16399143927233814, 0.49107142857142855, 0.2558139534883721, 0.20689655172413793, 0.8169014084507042, 0.3850561797752809, 0.35877862595419846, 0.43333333333333335, 0.44695975503062113, 1.0, 0.12149532710280372, 0.3530947054436987, 0.35459940652818994, 0.25271739130434784, 0.345724907063197, 0.44695975503062113, 0.47025813692480356, 0.2827715355805243, 0.6909090909090909, 0.3850561797752809, 0.39705882352941174, 0.5714285714285715, 0.2345679012345679, 0.0, 0.5, 0.2744282744282744, 0.2680115273775216, 0.24018016130721695, 0.345724907063197, 0.44695975503062113, 0.47025813692480356, 0.8636363636363636, 0.4084507042253521, 0.25271739130434784, 0.25886447683271696, 0.1711084191399152, 0.3850561797752809, 1.0, 0.42567811934900546, 0.5789473684210525, 0.1711084191399152, 0.396590066716086, 0.2666666666666667, 0.24018016130721695, 0.3530947054436987, 0.2577132486388385, 0.22765282583621685, 0.24752475247524755, 0.2857142857142857, 0.36988950276243093, 0.2857142857142857, 0.35644371941272435, 0.5, 0.44695975503062113, 0.2857142857142857, 0.11548223350253806, 0.2577132486388385, 0.35984848484848486, 0.2857142857142857, 1.0, 0.46956521739130436, 0.25271739130434784, 0.3530947054436987, 0.5244186046511629, 0.2775330396475771, 0.3364193328795099, 1.0, 0.38211805555555556, 0.07894736842105263, 0.4230769230769231, 0.3364193328795099, 1.0, 0.5540540540540541, 0.19994022713687984, 0.4870022123893805, 0.5833333333333334, 0.39062500000000006, 0.5, 0.2744282744282744, 0.3220108695652174, 0.16399143927233814, 0.2680115273775216, 0.38441330998248685, 0.3364193328795099, 0.3888888888888889, 0.24018016130721695, 0.2827715355805243, 0.12924424972617743, 0.3364193328795099, 0.5609756097560975, 1.0, 0.3220108695652174, 0.4604810996563574, 0.38211805555555556, 1.0, 0.24275362318840582, 0.22022471910112362, 0.2857142857142857, 1.0, 0.46956521739130436, 0.2577132486388385, 0.24427480916030533, 0.3364193328795099, 0.4604810996563574, 0.5254237288135594, 0.4927007299270073, 0.30246913580246915, 0.5324675324675325, 0.38211805555555556, 0.2680115273775216, 0.22119815668202764, 0.25271739130434784, 0.6399999999999999, 0.4, 0.4444444444444445, 0.38211805555555556, 0.5, 0.32547169811320753, 0.5, 0.2235294117647059, 0.44695975503062113, 0.3469119579500657, 0.16399143927233814, 0.25886447683271696, 0.2857142857142857, 0.17460317460317462, 0.36988950276243093, 0.2857142857142857, 1.0, 0.46956521739130436, 0.38211805555555556, 0.4, 0.2706193131787496, 0.03773584905660378, 0.0061068702290076335, 1.0, 0.44695975503062113, 0.2857142857142857, 0.11548223350253806, 0.046153846153846156, 0.5, 0.01652892561983471, 0.35, 0.22615384615384615, 0.17049009384775807, 0.31154684095860563, 0.4, 0.1711084191399152, 0.2857142857142857, 0.6413793103448276, 0.408335762168464, 0.9473684210526316, 0.17049009384775807, 0.4615384615384615, 0.408335762168464, 0.25271739130434784, 0.3850561797752809, 0.25, 0.16666666666666669, 0.5, 0.01652892561983471, 0.3017241379310345, 0.2204724409448819, 0.5789473684210525, 0.24275362318840582, 0.408335762168464, 0.2857142857142857, 0.4, 0.5, 0.2577132486388385, 0.5661764705882353, 0.4411764705882353, 0.36666666666666664, 0.3850561797752809, 0.3076923076923077, 0.25271739130434784, 0.2727272727272727, 0.24018016130721695, 0.3469119579500657, 0.17049009384775807, 0.049913941480206545, 0.4736842105263158, 0.09174311926605505, 0.2744282744282744, 0.22615384615384615, 0.18604651162790695, 0.3155852219807513, 0.0625, 0.25271739130434784, 0.19999999999999998, 0.3220108695652174, 0.4, 0.3620689655172414, 0.1711084191399152, 0.2857142857142857, 0.31154684095860563, 0.5, 0.44695975503062113, 0.39497716894977164, 0.5121951219512195, 0.4, 0.5333333333333333, 0.5860349127182044, 0.1711084191399152, 0.5121951219512195, 0.4, 0.17049009384775807, 0.5, 0.36988950276243093, 0.39497716894977164, 0.4083969465648855, 0.7674418604651162, 0.0, 0.031914893617021274, 0.4, 0.38211805555555556, 0.36746143057503505, 0.33333333333333337, 0.7209302325581396, 0.48351648351648346, 0.2857142857142857, 0.546875, 0.23469387755102036, 0.17049009384775807, 0.5, 0.19994022713687984, 0.36746143057503505, 0.6041666666666666, 0.0, 0.38211805555555556, 0.17049009384775807, 0.4, 0.25271739130434784, 0.3850561797752809, 0.3636363636363636, 0.44695975503062113, 0.7012987012987013, 0.38211805555555556, 0.42482100238663484, 0.44695975503062113, 0.2857142857142857, 0.2151898734177215, 0.2577132486388385, 0.2235294117647059, 0.33333333333333337, 0.2204724409448819, 0.5, 0.01652892561983471, 0.17049009384775807, 0.47058823529411764, 0.42482100238663484, 0.44695975503062113, 0.046153846153846156, 0.22765282583621685, 0.9400000000000001, 0.2345679012345679, 0.0, 0.38211805555555556, 1.0, 0.408335762168464, 0.49635036496350365, 0.36988950276243093, 0.3220108695652174, 0.5060240963855421, 0.2857142857142857, 0.835820895522388, 0.2204724409448819, 0.48514851485148514, 0.5391304347826087, 0.18181818181818182, 0.7777777777777778, 0.408335762168464, 0.2857142857142857, 0.8, 0.26908396946564883, 0.0909090909090909, 0.4893617021276596, 0.1930379746835443, 0.17049009384775807, 0.09122562674094707, 0.2827715355805243, 0.17049009384775807, 0.19708029197080293, 0.319327731092437, 0.2666666666666667, 0.36988950276243093, 0.046153846153846156, 0.38211805555555556, 0.48514851485148514, 0.4, 0.4, 0.0, 0.18584070796460175, 0.4519230769230769, 0.48351648351648346, 0.2857142857142857, 0.28840970350404316, 0.5348837209302325, 0.2878787878787879, 0.046153846153846156, 0.2204724409448819, 0.4, 0.1711084191399152, 0.48514851485148514, 0.31154684095860563, 0.6879432624113475, 0.5333333333333333, 0.44695975503062113, 0.2857142857142857, 0.47025813692480356, 0.24806201550387597, 0.27984344422700586, 0.07894736842105263, 0.24275362318840582, 0.2666666666666667, 0.6666666666666666, 1.0, 0.5378151260504201, 0.5, 0.2577132486388385, 0.36666666666666664, 0.18584070796460175, 0.5, 0.25271739130434784, 0.20833333333333334, 0.48351648351648346, 0.2857142857142857, 0.19999999999999998, 0.38211805555555556, 0.2577132486388385, 0.16399143927233814, 0.21352893235533824, 0.396590066716086, 0.408335762168464, 0.6363636363636364, 0.846153846153846, 0.36204576043068637, 0.31504315659679405, 1.0, 0.20689655172413793, 0.1711084191399152, 0.2857142857142857, 1.0, 0.46956521739130436, 0.44148936170212766, 0.6995073891625615, 0.25886447683271696, 0.11235955056179775, 0.25271739130434784, 0.8155339805825244, 0.24018016130721695, 0.3220108695652174, 0.408335762168464, 0.2857142857142857, 0.14285714285714288, 0.619271445358402, 0.3376152832674572, 0.3850561797752809, 0.14492753623188406, 0.44695975503062113, 0.611764705882353, 0.24778761061946902, 0.17049009384775807, 0.2857142857142857, 0.36204576043068637, 0.31504315659679405, 0.47692307692307695, 1.0, 0.22615384615384615, 0.17049009384775807, 0.14606741573033705, 0.27672955974842767, 0.36204576043068637, 0.48514851485148514, 1.0, 0.09174311926605505, 0.48514851485148514, 0.7264150943396227, 0.2204724409448819, 0.4, 0.396590066716086, 1.0, 0.3155852219807513, 0.1751412429378531, 0.3017241379310345, 0.4347826086956522, 0.1956521739130435, 0.36746143057503505, 0.6910112359550562, 0.7526881720430108, 0.24018016130721695, 0.28125, 0.5151515151515151, 0.24018016130721695, 0.24841772151898736, 0.3220108695652174, 0.22765282583621685, 0.39375203384315005, 0.2710280373831776, 0.5, 0.358974358974359, 0.35459940652818994, 0.38376383763837635, 0.19757174392935986, 0.35984848484848486, 0.25271739130434784, 0.42567811934900546, 0.2827715355805243, 0.22765282583621685, 0.39375203384315005, 0.314176245210728, 0.2857142857142857, 0.24285714285714285, 0.3155852219807513, 0.17049009384775807, 0.21352893235533824, 0.23605577689243026, 0.396590066716086, 1.0, 0.24018016130721695, 0.8095238095238095, 1.0, 0.5378151260504201, 0.4682080924855491, 0.17049009384775807, 0.21352893235533824, 0.2666666666666667, 0.6216216216216216, 1.0, 0.2857142857142857, 0.18987341772151897, 0.9565217391304348, 0.5542168674698795, 0.2345679012345679, 1.0, 0.2857142857142857, 0.19999999999999998, 0.6, 0.38211805555555556, 1.0, 0.25, 0.25271739130434784, 0.0909090909090909, 0.7499999999999999, 0.0, 0.9375, 0.4, 0.1711084191399152, 0.2857142857142857, 0.5185185185185185, 0.38211805555555556, 0.5, 1.0, 0.42857142857142855, 0.2619047619047619, 0.2857142857142857, 0.4, 0.4, 0.5979381443298969, 0.3791419602371817, 0.046153846153846156, 0.3155852219807513, 0.36204576043068637, 0.25271739130434784, 0.4, 0.7142857142857143, 0.38211805555555556, 0.48514851485148514, 0.732919254658385, 0.17049009384775807, 0.2761904761904762, 0.17219251336898395, 0.19963866305329722, 0.22399999999999998, 0.2774373259052925, 0.44822006472491904, 0.6, 0.27984344422700586, 0.19708029197080293, 0.48484848484848486, 0.5789473684210525, 0.3791419602371817, 0.396590066716086, 1.0, 0.16666666666666669, 0.5, 0.1711084191399152, 0.2857142857142857, 0.1875, 0.44695975503062113, 0.6153846153846153, 0.42567811934900546, 0.408335762168464, 0.3850561797752809, 0.2737430167597765, 0.44695975503062113, 0.5, 0.2827715355805243, 0.3376152832674572, 0.4604810996563574, 0.7275064267352186, 0.0909090909090909, 0.7142857142857143, 0.44695975503062113, 0.42482100238663484, 0.20454545454545456, 0.2204724409448819, 0.2857142857142857, 0.8662790697674418, 0.27984344422700586, 0.14285714285714288, 0.1264367816091954, 0.2577132486388385, 0.30033185840707965, 0.6923076923076924, 0.6354044548651817, 0.44695975503062113, 0.5, 0.01652892561983471, 0.22765282583621685, 0.39375203384315005, 0.8046511627906977, 0.7461139896373057, 0.22551169590643275, 0.48514851485148514, 0.44148936170212766, 0.6995073891625615, 0.19994022713687984, 0.38211805555555556, 0.17219251336898395, 0.3364193328795099, 0.29540481400437635, 0.22765282583621685, 0.35644371941272435, 0.21352893235533824, 0.42482100238663484, 0.44695975503062113, 0.48514851485148514, 0.9400000000000001, 0.5748502994011977, 0.48514851485148514, 0.5360824742268041, 0.5163934426229507, 0.4, 0.4, 0.24018016130721695, 0.4, 0.6666666666666666, 0.5254237288135594, 0.42482100238663484, 0.2666666666666667, 0.3364193328795099, 0.30033185840707965, 0.2680115273775216, 0.5056179775280898, 0.3850561797752809, 0.47692307692307695, 0.0625, 0.0, 0.6, 0.36988950276243093, 0.046153846153846156, 0.19994022713687984, 0.40298507462686567, 0.22022471910112362, 0.2857142857142857, 0.46956521739130436, 0.24841772151898736, 0.3376152832674572, 0.3620689655172414, 0.2345679012345679, 0.2857142857142857, 0.0, 0.2577132486388385, 1.0, 0.4123711340206185, 0.2827715355805243, 0.2577132486388385, 0.4604810996563574, 0.21352893235533824, 0.25271739130434784, 0.38441330998248685, 0.5540540540540541, 0.2345679012345679, 0.4927007299270073, 0.71, 0.9285714285714286, 0.38211805555555556, 0.2920268972142171, 0.22022471910112362, 0.2857142857142857, 0.6879432624113475, 0.8046511627906977, 0.3791419602371817, 0.5666666666666667, 0.3791419602371817, 0.2577132486388385, 0.30033185840707965, 0.3620689655172414, 0.31504315659679405, 0.27906976744186046, 0.2577132486388385, 0.2746478873239437, 0.2666666666666667, 0.42482100238663484, 0.4927007299270073, 0.6654135338345865, 0.25, 0.6216216216216216, 0.44695975503062113, 0.5161290322580645, 0.7333333333333334, 0.5, 0.5505254648342766, 0.36988950276243093, 0.4, 0.08196721311475409, 0.4444444444444445, 0.38211805555555556, 0.4, 0.44144144144144143, 0.2577132486388385, 0.30033185840707965, 0.4373464373464373, 0.7499999999999999, 0.38211805555555556, 0.2114695340501792, 0.20833333333333334, 0.408335762168464, 0.25925925925925924, 0.17219251336898395, 0.2577132486388385, 0.43902439024390244, 0.2857142857142857, 0.14996232102486812, 0.2744282744282744, 0.38441330998248685, 0.6879432624113475, 0.38211805555555556, 0.6666666666666666, 0.2827715355805243, 0.0, 0.5540540540540541, 0.3791419602371817, 0.20689655172413793, 0.3791419602371817, 0.3364193328795099, 0.30033185840707965, 0.2577132486388385, 0.24427480916030533, 0.25271739130434784, 0.2775330396475771, 0.31504315659679405, 0.6746411483253588, 0.3574660633484163, 0.38211805555555556, 0.0, 0.0, 0.391304347826087, 0.2345679012345679, 0.345724907063197, 0.44695975503062113, 0.3364193328795099, 0.2763157894736842, 0.5789473684210525, 0.4870022123893805, 0.4347826086956522, 0.408335762168464, 0.2857142857142857, 0.46956521739130436, 0.21428571428571425, 0.2857142857142857, 0.3389830508474576, 0.44695975503062113, 0.47025813692480356, 0.1081081081081081, 0.24018016130721695, 0.4927007299270073, 0.6666666666666666, 0.2345679012345679, 0.2857142857142857, 0.3571428571428571, 0.17049009384775807, 0.22765282583621685, 0.2749731471535983, 0.11235955056179775, 0.19963866305329722, 0.3376152832674572, 0.24841772151898736, 0.44695975503062113, 0.47025813692480356, 0.2827715355805243, 0.3376152832674572, 0.15384615384615385, 0.25271739130434784, 1.0, 0.25271739130434784, 0.6, 0.37241379310344824, 1.0, 1.0, 0.314176245210728, 0.2857142857142857, 0.2857142857142857, 1.0, 0.46956521739130436, 0.16399143927233814, 0.6399999999999999, 0.396590066716086, 0.4553571428571429, 0.43750000000000006, 0.37272727272727274, 0.19708029197080293, 0.3364193328795099, 0.37241379310344824, 0.25271739130434784, 0.3120300751879699, 0.2920268972142171, 0.3850561797752809, 0.2888888888888889, 0.25271739130434784, 1.0, 0.25271739130434784, 0.38211805555555556, 0.4347826086956522, 0.3220108695652174, 0.36036036036036034, 0.4870022123893805, 0.32258064516129037, 0.19994022713687984, 0.2857142857142857, 0.3069620253164557, 0.36107986501687284, 0.35984848484848486, 0.25271739130434784, 0.4373464373464373, 0.24275362318840582, 0.22022471910112362, 0.2857142857142857, 0.46956521739130436, 0.25271739130434784, 0.6, 0.37241379310344824, 0.2857142857142857, 1.0, 1.0, 0.32547169811320753, 1.0, 0.0, 0.25, 0.4, 0.2857142857142857, 1.0, 0.24000000000000002, 0.42567811934900546, 0.4, 0.35984848484848486, 0.25271739130434784, 0.2737994945240101, 0.18604651162790695, 0.24018016130721695, 0.2857142857142857, 1.0, 0.24285714285714285, 0.36988950276243093, 0.4, 0.7974137931034483, 0.3850561797752809, 0.07692307692307693, 0.4, 0.42567811934900546, 0.5789473684210525, 0.4, 0.38211805555555556, 0.2857142857142857, 1.0, 0.19999999999999998, 0.19994022713687984, 0.2857142857142857, 0.3069620253164557, 0.36107986501687284, 0.22368421052631576, 0.2577132486388385, 0.21352893235533824, 0.2857142857142857, 0.25, 0.3120300751879699, 0.2920268972142171, 0.2728110599078341, 0.325, 0.22765282583621685, 0.39375203384315005, 0.44148936170212766, 0.6470588235294118, 0.38211805555555556, 0.3089171974522293, 0.38441330998248685, 0.24841772151898736, 0.8662790697674418, 0.2345679012345679, 0.4870022123893805, 0.427027027027027, 0.9090909090909091, 0.1711084191399152, 0.19999999999999998, 0.44658493870402804, 0.3013698630136986, 0.6222222222222222, 0.6363636363636364, 0.0, 0.1711084191399152, 0.2857142857142857, 0.24060150375939854, 1.0, 0.3364193328795099, 0.22765282583621685, 0.11062190423775453, 0.408335762168464, 0.4, 0.052631578947368425, 0.32015065913371, 0.2265625, 0.2666666666666667, 0.38297872340425526, 0.24018016130721695, 0.2666666666666667, 0.25271739130434784, 0.647887323943662, 0.2857142857142857, 0.28125, 0.2857142857142857, 0.4, 1.0, 0.4, 1.0, 0.4, 0.2857142857142857, 0.4, 0.7647058823529412, 0.4857142857142857, 0.7499999999999999, 0.33870967741935487, 0.2827715355805243, 0.4604810996563574, 0.2827715355805243, 0.22765282583621685, 0.39375203384315005, 0.24166666666666667, 0.3155852219807513, 0.25271739130434784, 0.337890625, 0.2827715355805243, 0.42857142857142855, 0.6, 0.38211805555555556, 0.5185185185185185, 0.5505254648342766, 0.42857142857142855, 0.1711084191399152, 0.1780821917808219, 0.4, 0.33870967741935487, 0.6797153024911032, 0.7499999999999999, 0.4, 0.2989347536617843, 0.5609756097560975, 0.6666666666666666, 0.25271739130434784, 0.2857142857142857, 0.7272727272727273, 0.36988950276243093, 0.3850561797752809, 0.30701754385964913, 0.18885448916408668, 0.44695975503062113, 0.2857142857142857, 0.8352941176470587, 0.44695975503062113, 0.026490066225165563, 0.17049009384775807, 0.30303030303030304, 0.24841772151898736, 0.4, 0.408335762168464, 0.2857142857142857, 0.5121951219512195, 0.1711084191399152, 0.39497716894977164, 0.3891402714932127, 0.25271739130434784, 1.0, 0.2857142857142857, 0.31154684095860563, 0.6017699115044248, 0.23255813953488372, 0.36204576043068637, 0.2857142857142857, 0.4, 0.49333333333333335, 1.0, 0.2744282744282744, 0.3220108695652174, 0.2204724409448819, 0.4604810996563574, 0.6666666666666666, 0.3220108695652174, 0.31504315659679405, 0.23255813953488372, 0.2857142857142857, 0.22765282583621685, 0.35644371941272435, 0.36204576043068637, 0.25, 0.0, 0.3220108695652174, 0.22765282583621685, 0.39375203384315005, 0.36204576043068637, 0.2857142857142857, 0.3684210526315789, 0.44695975503062113, 0.2857142857142857, 0.6413793103448276, 0.1711084191399152, 0.4, 0.0, 0.24018016130721695, 0.6354044548651817, 0.4590952839268527, 0.6399999999999999, 0.2577132486388385, 0.21352893235533824, 0.3850561797752809, 0.35877862595419846, 0.0, 0.44695975503062113, 0.2857142857142857, 0.23255813953488372, 0.56, 0.2345679012345679, 0.44822006472491904, 0.4, 0.0, 0.38211805555555556, 0.19999999999999998, 0.44695975503062113, 0.2857142857142857, 0.43750000000000006, 0.1711084191399152, 0.2857142857142857, 0.6995073891625615, 0.427027027027027, 1.0, 0.21352893235533824, 0.7114093959731543, 0.2857142857142857, 0.31154684095860563, 0.6666666666666666, 0.25271739130434784, 0.2615079365079365, 0.11006781429316641, 0.1111111111111111, 0.19994022713687984, 0.2857142857142857, 0.28, 0.36107986501687284, 0.21715526601520088, 0.17219251336898395, 0.4870022123893805, 0.5979381443298969, 0.3684210526315789, 0.44695975503062113, 0.17073170731707318, 0.3850561797752809, 0.23255813953488372, 0.17049009384775807, 0.17647058823529413, 0.3364193328795099, 0.11764705882352941, 0.4870022123893805, 0.4404145077720208, 0.22551169590643275, 0.10714285714285715, 0.4870022123893805, 0.28415300546448086, 0.1711084191399152, 0.2857142857142857, 0.6, 0.2336065573770492, 0.3364193328795099, 0.16399143927233814, 0.25886447683271696, 0.2619047619047619, 0.5505254648342766, 0.427027027027027, 0.4, 0.3220108695652174, 0.16399143927233814, 0.5540540540540541, 0.18604651162790695, 0.25271739130434784, 0.39497716894977164, 0.427027027027027, 0.32926829268292684, 0.3017241379310345, 0.375, 0.22765282583621685, 0.35644371941272435, 0.4, 0.0, 0.35459940652818994, 0.37272727272727274, 0.6797153024911032, 0.40740740740740744, 0.2577132486388385, 0.14996232102486812, 0.25271739130434784, 0.10126582278481014, 0.2737994945240101, 0.3220108695652174, 0.3947368421052631, 0.5540540540540541, 0.38376383763837635, 0.5555555555555556, 0.27505827505827507, 0.22399999999999998, 1.0, 0.38211805555555556, 0.17219251336898395, 0.4084507042253521, 0.44695975503062113, 0.3364193328795099, 0.11764705882352941, 0.3850561797752809, 0.4404145077720208, 0.17049009384775807, 0.30033185840707965, 0.38441330998248685, 0.2857142857142857, 0.9166666666666666, 0.25271739130434784, 0.4210526315789474, 0.2857142857142857, 0.31154684095860563, 0.6666666666666666, 0.061244019138755976, 0.22765282583621685, 0.39375203384315005, 0.273109243697479, 0.2857142857142857, 0.8046511627906977, 0.2426550598476605, 0.4, 0.0, 0.3611111111111111, 0.24018016130721695, 0.22551169590643275, 0.375, 0.31504315659679405, 0.23255813953488372, 0.3850561797752809, 0.33333333333333337, 0.49122807017543857, 0.22615384615384615, 0.17049009384775807, 0.2204724409448819, 0.1711084191399152, 0.2857142857142857, 0.4, 0.5333333333333333, 0.31504315659679405, 0.19999999999999998, 0.17049009384775807, 0.26588235294117646, 0.6797153024911032, 0.39846743295019155, 0.33333333333333337, 0.19923044035912785, 0.19757174392935986, 0.16176470588235292, 0.25271739130434784, 0.5130434782608695, 0.396590066716086, 0.6, 0.23255813953488372, 0.25271739130434784, 0.2746478873239437, 0.25271739130434784, 0.36746143057503505, 0.5, 0.19757174392935986, 0.16176470588235292, 0.2857142857142857, 0.5, 0.25271739130434784, 0.14102564102564102, 0.38829787234042556, 0.30864197530864196, 0.2857142857142857, 0.12924424972617743, 0.36036036036036034, 0.24018016130721695, 0.3850561797752809, 0.0, 1.0, 0.3155852219807513, 0.21715526601520088, 0.4553571428571429, 0.16666666666666669, 0.3220108695652174, 0.16399143927233814, 0.18223234624145784, 0.2857142857142857, 0.3389830508474576, 0.44695975503062113, 0.47025813692480356, 0.25271739130434784, 0.25, 0.375, 0.25271739130434784, 0.4927007299270073, 0.33333333333333337, 0.2827715355805243, 0.35459940652818994, 0.29540481400437635, 0.22765282583621685, 0.35644371941272435, 0.38441330998248685, 0.3220108695652174, 0.619271445358402, 0.22765282583621685, 0.39375203384315005, 0.42482100238663484, 0.44695975503062113, 0.2857142857142857, 0.5360824742268041, 0.56, 0.44695975503062113, 0.2857142857142857, 0.23255813953488372, 0.2345679012345679, 0.0061068702290076335, 0.125, 0.6666666666666666, 0.3513513513513513, 0.44695975503062113, 0.2857142857142857, 0.4, 1.0, 0.408335762168464, 0.4752475247524752, 1.0, 0.17049009384775807, 0.26588235294117646, 0.3850561797752809, 0.47692307692307695, 0.36107986501687284, 0.36988950276243093, 0.31504315659679405, 0.5, 0.1711084191399152, 0.2857142857142857, 0.5121951219512195, 0.4, 0.5333333333333333, 0.38211805555555556, 0.5714285714285715, 0.396590066716086, 0.14285714285714288, 0.4404145077720208, 0.36988950276243093, 0.48514851485148514, 0.5, 0.22615384615384615, 0.2577132486388385, 0.17916666666666667, 0.03773584905660378, 0.22765282583621685, 0.39375203384315005, 0.5, 0.24018016130721695, 0.276595744680851, 0.4927007299270073, 0.36065573770491804, 0.22399999999999998, 1.0, 0.5540540540540541, 0.2577132486388385, 0.4, 0.2666666666666667, 0.3850561797752809, 0.29508196721311475, 0.44695975503062113, 0.4, 0.3791419602371817, 0.1930379746835443, 0.3850561797752809, 0.6428571428571429, 0.44695975503062113, 0.2857142857142857, 0.4, 0.49333333333333335, 1.0, 0.2345679012345679, 0.2814814814814815, 0.2857142857142857, 0.5555555555555556, 0.38211805555555556, 0.2857142857142857, 0.5, 0.3155852219807513, 0.14285714285714288, 0.4, 0.7666666666666667, 0.31504315659679405, 0.7272727272727273, 0.3155852219807513, 0.6, 0.24018016130721695, 0.8571428571428571, 0.408335762168464, 0.2857142857142857, 1.0, 0.44695975503062113, 0.2666666666666667, 0.36107986501687284, 0.17219251336898395, 0.4, 0.3155852219807513, 0.4870022123893805, 0.4444444444444445, 0.25271739130434784, 0.7777777777777778, 0.3089171974522293, 0.20833333333333334, 0.5540540540540541, 0.2345679012345679, 0.4, 0.22765282583621685, 0.7272727272727273, 0.31504315659679405, 0.23255813953488372, 0.3155852219807513, 0.8, 0.36988950276243093, 0.345724907063197, 0.6842105263157895, 0.5244186046511629, 0.2577132486388385, 0.30033185840707965, 0.33333333333333337, 0.8260869565217391, 0.4, 0.4, 0.4, 1.0, 0.5, 0.4, 0.16111111111111112, 0.5, 0.44695975503062113, 1.0, 0.37272727272727274, 0.36204576043068637, 0.2857142857142857, 0.2151898734177215, 0.3017241379310345, 0.48749999999999993, 0.2857142857142857, 0.46341463414634143, 0.7012987012987013, 0.44695975503062113, 0.2857142857142857, 0.46956521739130436, 0.5, 0.4, 0.22765282583621685, 0.39375203384315005, 0.5571428571428572, 0.0, 0.6909090909090909, 0.835820895522388, 0.16399143927233814, 0.5609756097560975, 0.25886447683271696, 0.04040404040404041, 0.2920268972142171, 0.31504315659679405, 0.026490066225165563, 0.42482100238663484, 0.37267080745341613, 0.33870967741935487, 0.3220108695652174, 0.22765282583621685, 0.39375203384315005, 0.5, 0.25, 0.2989347536617843, 0.23605577689243026, 0.34782608695652173, 0.021390374331550804, 0.38211805555555556, 0.012224938875305624, 0.2666666666666667, 0.35, 0.21715526601520088, 0.3220108695652174, 0.22765282583621685, 0.39375203384315005, 0.4, 0.2827715355805243, 0.3220108695652174, 0.33766233766233766, 0.9, 0.3220108695652174, 0.16399143927233814, 0.25886447683271696, 0.2857142857142857, 0.596638655462185, 0.408335762168464, 0.2857142857142857, 0.4, 0.4, 0.45000000000000007, 1.0, 0.4736842105263158, 0.17049009384775807, 0.30033185840707965, 0.22765282583621685, 0.35644371941272435, 0.5769230769230769, 0.25271739130434784, 0.26027397260273977, 0.3220108695652174, 0.3791419602371817, 0.39497716894977164, 0.7608695652173912, 0.38211805555555556, 0.17049009384775807, 0.22399999999999998, 0.24841772151898736, 0.44695975503062113, 0.2857142857142857, 0.8636363636363636, 0.2888888888888889, 0.3017241379310345, 0.22399999999999998, 0.408335762168464, 0.30246913580246915, 0.125, 0.22615384615384615, 0.3220108695652174, 0.2204724409448819, 0.14545454545454545, 0.3364193328795099, 0.30033185840707965, 0.38441330998248685, 0.3850561797752809, 0.2974683544303798, 0.19994022713687984, 0.4604810996563574, 0.2857142857142857, 0.5, 0.2424242424242424, 0.1711084191399152, 0.2857142857142857, 0.30246913580246915, 0.835820895522388, 0.19994022713687984, 0.5, 0.4, 0.22765282583621685, 0.39375203384315005, 1.0, 0.44148936170212766, 0.6995073891625615, 0.19994022713687984, 0.32, 0.36988950276243093, 0.2857142857142857, 0.47692307692307695, 0.5, 0.2577132486388385, 0.22765282583621685, 0.40265486725663713, 0.6399999999999999, 0.4, 0.25271739130434784, 0.3376152832674572, 0.3850561797752809, 0.39705882352941174, 0.17049009384775807, 0.22765282583621685, 0.2749731471535983, 0.2666666666666667, 0.4166666666666667, 0.36204576043068637, 0.17219251336898395, 0.3364193328795099, 0.10256410256410257, 0.22765282583621685, 0.35644371941272435, 0.2678571428571429, 0.046511627906976744, 0.22765282583621685, 0.39375203384315005, 0.38181818181818183, 0.44695975503062113, 1.0, 0.7674418604651162, 0.3364193328795099, 0.22765282583621685, 0.11062190423775453, 0.408335762168464, 0.24018016130721695, 0.3850561797752809, 0.6879432624113475, 0.846153846153846, 0.32, 0.36988950276243093, 0.2827715355805243, 0.3155852219807513, 0.3850561797752809, 0.5625, 0.2827715355805243, 0.02631578947368421, 0.38211805555555556, 0.4, 0.21352893235533824, 0.23605577689243026, 0.4, 0.33333333333333337, 0.25271739130434784, 0.31504315659679405, 0.6197183098591549, 0.19999999999999998, 0.4, 0.38211805555555556, 0.0061068702290076335, 0.4, 0.22765282583621685, 0.39375203384315005, 0.5505254648342766, 0.8275862068965517, 0.5505254648342766, 1.0, 0.19994022713687984, 0.2857142857142857, 0.30246913580246915, 0.44148936170212766, 0.6995073891625615, 0.25, 0.30033185840707965, 0.4682080924855491, 0.3487858719646799, 0.36746143057503505, 0.5294117647058824, 0.5, 0.4, 0.9400000000000001, 0.3697916666666667, 0.36746143057503505, 0.515625, 0.7391304347826086, 0.21715526601520088, 0.19963866305329722, 0.22765282583621685, 0.39375203384315005, 0.5505254648342766, 0.6666666666666666, 0.25271739130434784, 0.3530947054436987, 1.0, 0.4086021505376344, 0.478134110787172, 0.17049009384775807, 0.12578616352201258, 0.3850561797752809, 0.46341463414634143, 1.0, 0.4, 1.0, 0.3155852219807513, 0.5714285714285715, 0.6896551724137931, 0.5, 0.4, 0.31504315659679405, 0.19999999999999998, 0.7142857142857143, 0.25, 0.25, 0.38211805555555556, 0.2857142857142857, 0.07692307692307693, 0.4, 0.22765282583621685, 0.39375203384315005, 0.6666666666666666, 1.0, 0.3571428571428571, 0.2666666666666667, 0.36988950276243093, 0.3850561797752809, 0.5, 0.2827715355805243, 0.5, 0.4, 0.36065573770491804, 0.4, 0.10534591194968553, 0.2857142857142857, 0.6666666666666666, 0.4, 0.8333333333333333, 0.5, 0.3469119579500657, 0.19757174392935986, 0.375, 0.2920268972142171, 0.8048780487804877, 0.19994022713687984, 0.2857142857142857, 0.2857142857142857, 0.44695975503062113, 0.2857142857142857, 1.0, 0.265625, 0.1111111111111111, 0.625, 1.0, 0.48514851485148514, 0.38211805555555556, 0.22615384615384615, 0.27984344422700586, 0.3620689655172414, 0.21352893235533824, 0.3220108695652174, 0.1711084191399152, 0.3220108695652174, 0.475, 0.3220108695652174, 0.22765282583621685, 0.39375203384315005, 0.4, 0.5, 0.32, 0.2857142857142857, 0.3891402714932127, 0.19963866305329722, 0.22765282583621685, 0.39375203384315005, 0.3850561797752809, 0.47692307692307695, 0.6842105263157895, 0.2827715355805243, 0.2989347536617843, 0.23605577689243026, 0.05882352941176471, 0.25271739130434784, 0.3850561797752809, 0.35644371941272435, 0.5, 0.2827715355805243, 0.2577132486388385, 0.4604810996563574, 0.30033185840707965, 0.6910112359550562, 0.3120300751879699, 0.3220108695652174, 0.2920268972142171, 0.19994022713687984, 0.36746143057503505, 0.6995073891625615, 0.4, 0.38211805555555556, 0.33333333333333337, 0.1882845188284519, 0.5890410958904109, 0.4736842105263158, 0.4, 0.7741935483870968, 0.2989347536617843, 0.35459940652818994, 0.4, 0.36746143057503505, 0.31154684095860563, 0.1882845188284519, 0.7391304347826086, 0.4736842105263158, 0.2345679012345679, 0.36746143057503505, 0.4, 0.38211805555555556, 0.33333333333333337, 0.08686868686868687, 0.4, 0.33333333333333337, 0.6, 0.38211805555555556, 0.4, 0.2345679012345679, 0.6197183098591549, 0.5378151260504201, 0.21352893235533824, 0.0, 0.2728110599078341, 0.7391304347826086, 0.38211805555555556, 0.4, 0.0, 0.38211805555555556, 1.0, 1.0, 0.16399143927233814, 0.35984848484848486, 0.25271739130434784, 0.21352893235533824, 0.2728110599078341, 0.47692307692307695, 0.7003058103975535, 0.7391304347826086, 0.24018016130721695, 0.345724907063197, 0.4519230769230769, 0.22615384615384615, 0.396590066716086, 0.5654520917678812, 0.35459940652818994, 0.2857142857142857, 0.22765282583621685, 0.35644371941272435, 0.4583333333333333, 0.4084507042253521, 0.2857142857142857, 0.08686868686868687, 1.0, 0.3155852219807513, 0.2680115273775216, 0.1711084191399152, 0.21321961620469082, 0.19994022713687984, 0.31504315659679405, 0.6995073891625615, 0.21715526601520088, 0.25886447683271696, 0.11235955056179775, 0.25271739130434784, 0.7461139896373057, 0.22551169590643275, 0.38211805555555556, 0.6399999999999999, 0.2504780114722753, 0.2424242424242424, 0.408335762168464, 0.2857142857142857, 0.4, 0.4736842105263158, 0.3376152832674572, 0.4927007299270073, 0.4083969465648855, 0.33333333333333337, 0.1951219512195122, 0.4, 0.38211805555555556, 0.0, 1.0, 0.7692307692307693, 0.12254160363086232, 0.0, 0.9689922480620156, 0.17049009384775807, 0.22765282583621685, 0.40265486725663713, 0.36666666666666664, 0.25271739130434784, 0.33333333333333337, 0.3364193328795099, 0.17049009384775807, 0.03571428571428572, 0.22765282583621685, 0.35644371941272435, 0.17073170731707318, 0.22022471910112362, 0.31504315659679405, 0.30246913580246915, 0.7692307692307693, 0.5866666666666668, 0.42482100238663484, 0.5, 0.408335762168464, 0.7894736842105263, 0.17049009384775807, 0.26588235294117646, 0.6666666666666666, 0.25, 0.3530947054436987, 0.25271739130434784, 0.4373464373464373, 0.25271739130434784, 0.2857142857142857, 0.0, 0.30246913580246915, 0.37267080745341613, 0.22765282583621685, 0.39375203384315005, 0.7499999999999999, 0.273109243697479, 0.5714285714285715, 0.25271739130434784, 0.2737994945240101, 0.36988950276243093, 0.25, 0.2744282744282744, 0.17049009384775807, 0.35459940652818994, 0.1627906976744186, 0.22765282583621685, 0.35644371941272435, 0.38441330998248685, 0.2857142857142857, 0.18987341772151897, 1.0, 0.2744282744282744, 0.19994022713687984, 0.2857142857142857, 0.27167630057803466, 0.7333333333333334, 0.17049009384775807, 0.26588235294117646, 0.3850561797752809, 0.12254160363086232, 0.44695975503062113, 0.6571428571428571, 0.38211805555555556, 0.4682080924855491, 0.33870967741935487, 0.31335149863760214, 0.3850561797752809, 0.6170212765957447, 0.10526315789473684, 0.19994022713687984, 0.4, 0.22765282583621685, 0.39375203384315005, 0.49079754601226994, 1.0, 0.2857142857142857, 0.30246913580246915, 0.4, 0.4, 0.17049009384775807, 0.22765282583621685, 0.2749731471535983, 0.11235955056179775, 0.33333333333333337, 0.17049009384775807, 0.23880597014925373, 0.2857142857142857, 0.04545454545454545, 0.6551724137931034, 0.44695975503062113, 0.4, 0.17821782178217824, 0.24018016130721695, 0.19923044035912785, 0.375, 0.2857142857142857, 0.7692307692307693, 0.12254160363086232, 0.48351648351648346, 0.4888059701492537, 0.6170212765957447, 0.17219251336898395, 0.3364193328795099, 0.35984848484848486, 0.25271739130434784, 0.26027397260273977, 0.6354044548651817, 0.36204576043068637, 0.31504315659679405, 0.42567811934900546, 0.1478743068391867, 0.24841772151898736, 0.276595744680851, 0.44695975503062113, 0.2235294117647059, 0.6399999999999999, 0.4927007299270073, 0.30246913580246915, 0.2857142857142857, 1.0, 0.46956521739130436, 0.44148936170212766, 0.6995073891625615, 0.19994022713687984, 0.38211805555555556, 0.4373464373464373, 0.25271739130434784, 0.2857142857142857, 0.44148936170212766, 0.38181818181818183, 0.44695975503062113, 0.31504315659679405, 0.9236641221374046, 0.3711340206185567, 0.4, 0.4, 0.4, 0.4752475247524752, 0.44695975503062113, 0.2857142857142857, 1.0, 0.33870967741935487, 0.375, 0.2235294117647059, 0.44695975503062113, 0.4, 0.40298507462686567, 0.22765282583621685, 0.39375203384315005, 0.18181818181818182, 0.6666666666666666, 0.24275362318840582, 0.25271739130434784, 0.4, 0.1702127659574468, 0.5, 0.6153846153846153, 0.17049009384775807, 0.26588235294117646, 0.25271739130434784, 0.36666666666666664, 0.31504315659679405, 0.37267080745341613, 0.22765282583621685, 0.39375203384315005, 0.4, 0.30701754385964913, 0.25271739130434784, 0.39497716894977164, 0.31335149863760214, 0.36988950276243093, 0.2857142857142857, 0.5185185185185185, 0.6041666666666666, 0.1711084191399152, 0.2857142857142857, 0.4, 0.5333333333333333, 0.1711084191399152, 0.3220108695652174, 0.22765282583621685, 0.39375203384315005, 0.391304347826087, 0.30246913580246915, 0.125, 0.38211805555555556, 0.014354066985645933, 0.020833333333333332, 0.3791419602371817, 0.2857142857142857, 0.7959183673469388, 0.41509433962264153, 0.345724907063197, 0.5789473684210525, 0.8, 0.38211805555555556, 0.9038461538461539, 0.2345679012345679, 0.2857142857142857, 0.4, 0.4, 0.5, 0.33333333333333337, 0.2666666666666667, 0.2558139534883721, 0.375, 0.24018016130721695, 0.2857142857142857, 0.35644371941272435, 0.0, 0.2827715355805243, 0.41886792452830185, 0.464, 0.4444444444444445, 0.4, 0.42567811934900546, 0.3530947054436987, 0.6521739130434783, 0.4084507042253521, 0.1711084191399152, 0.2857142857142857, 0.26908396946564883, 0.3017241379310345, 0.3376152832674572, 0.345724907063197, 0.39846743295019155, 0.1724137931034483, 0.2666666666666667, 0.17049009384775807, 0.4373464373464373, 0.25271739130434784, 0.36746143057503505, 0.44148936170212766, 0.6995073891625615, 0.4604810996563574, 0.44695975503062113, 0.2504780114722753, 0.21352893235533824, 0.36764705882352944, 0.0, 0.4166666666666667, 0.36746143057503505, 0.56, 0.17219251336898395, 0.3364193328795099, 0.35984848484848486, 0.25271739130434784, 0.1478743068391867, 0.276595744680851, 0.44695975503062113, 0.2857142857142857, 0.28125, 0.42567811934900546, 0.14996232102486812, 0.6448979591836734, 0.24018016130721695, 0.24275362318840582, 0.25271739130434784, 0.2857142857142857, 1.0, 0.4, 0.015822002472187888, 0.6399999999999999, 0.021390374331550804, 0.4, 0.2680115273775216, 0.2989347536617843, 0.24841772151898736, 0.44822006472491904, 0.47692307692307695, 0.6666666666666666, 0.04166666666666667, 0.31504315659679405, 0.37267080745341613, 0.3120300751879699, 0.2504780114722753, 0.2920268972142171, 0.5, 0.44695975503062113, 0.4, 0.40298507462686567, 0.17219251336898395, 0.3364193328795099, 0.14996232102486812, 0.3850561797752809, 0.1956521739130435, 0.10606060606060606, 0.24018016130721695, 0.2857142857142857, 0.7666666666666667, 0.42567811934900546, 0.0, 0.04545454545454545, 0.38211805555555556, 0.35984848484848486, 0.3850561797752809, 0.47692307692307695, 0.013878180416345412, 0.6399999999999999, 0.19963866305329722, 0.6995073891625615, 0.24018016130721695, 0.24841772151898736, 0.47692307692307695, 0.32608695652173914, 0.17049009384775807, 0.22765282583621685, 0.2749731471535983, 0.0, 0.24841772151898736, 0.44695975503062113, 0.39497716894977164, 0.4, 0.49333333333333335, 0.33333333333333337, 0.19994022713687984, 0.2857142857142857, 0.4444444444444445, 0.4, 0.29411764705882354, 0.17049009384775807, 0.22765282583621685, 0.2749731471535983, 0.6551724137931034, 0.44695975503062113, 0.5454545454545454, 0.25271739130434784, 0.25886447683271696, 0.408335762168464, 0.3850561797752809, 0.14814814814814814, 0.37241379310344824, 0.7666666666666667, 0.36988950276243093, 0.33333333333333337, 0.38211805555555556, 0.4, 0.4, 0.2345679012345679, 0.2857142857142857, 0.4, 1.0, 0.17049009384775807, 0.22765282583621685, 0.2749731471535983, 0.0, 0.3850561797752809, 0.3636363636363636, 0.44695975503062113, 0.39497716894977164, 0.44822006472491904, 0.33333333333333337, 0.17049009384775807, 0.25490196078431376, 0.1711084191399152, 0.2857142857142857, 0.5121951219512195, 0.19963866305329722, 0.464, 0.17049009384775807, 0.8095238095238095, 0.2857142857142857, 0.25, 0.408335762168464, 0.7666666666666667, 0.2744282744282744, 0.17049009384775807, 0.22765282583621685, 0.40265486725663713, 0.36666666666666664, 0.25271739130434784, 0.2775330396475771, 0.2857142857142857, 0.4, 0.6923076923076924, 0.38211805555555556, 0.2857142857142857, 0.4604810996563574, 0.39497716894977164, 0.4, 0.49333333333333335, 0.33333333333333337, 0.16399143927233814, 0.337890625, 0.3220108695652174, 0.4604810996563574, 1.0, 0.17049009384775807, 0.24427480916030533, 0.33333333333333337, 1.0, 0.16666666666666669, 0.2666666666666667, 0.4, 0.19963866305329722, 0.2558139534883721, 0.31504315659679405, 0.37267080745341613, 0.38211805555555556, 0.5, 0.44695975503062113, 0.345724907063197, 0.47692307692307695, 0.35644371941272435, 0.0, 0.2577132486388385, 0.21352893235533824, 0.24841772151898736, 0.19999999999999998, 0.0, 0.478134110787172, 0.19994022713687984, 0.2857142857142857, 0.2666666666666667, 0.7254901960784313, 0.19757174392935986, 0.22765282583621685, 0.11062190423775453, 0.42857142857142855, 0.25271739130434784, 0.2857142857142857, 0.5, 0.3364193328795099, 0.36666666666666664, 0.17219251336898395, 0.3364193328795099, 0.23880597014925373, 0.1711084191399152, 0.31504315659679405, 0.37267080745341613, 0.22765282583621685, 0.39375203384315005, 0.28125, 0.1282051282051282, 0.19757174392935986, 0.21352893235533824, 0.4, 0.40298507462686567, 0.22765282583621685, 0.39375203384315005, 0.4, 0.1711084191399152, 0.2857142857142857, 0.26908396946564883, 0.38211805555555556, 0.2857142857142857, 0.5505254648342766, 0.5789473684210525, 0.23469387755102036, 0.7857142857142858, 0.1711084191399152, 0.2857142857142857, 0.8048780487804877, 0.8333333333333333, 0.3364193328795099, 0.30033185840707965, 0.1478743068391867, 0.3850561797752809, 0.6666666666666666, 0.19994022713687984, 0.4, 0.4, 0.5217391304347826, 0.619271445358402, 0.3376152832674572, 0.3850561797752809, 0.14492753623188406, 0.44695975503062113, 1.0, 0.24018016130721695, 0.4870022123893805, 0.6666666666666666, 0.5, 0.38211805555555556, 0.4, 0.2920268972142171, 0.19963866305329722, 0.3850561797752809, 0.30246913580246915, 0.7289719626168224, 0.7674418604651162, 0.3017241379310345, 0.22765282583621685, 0.39375203384315005, 0.3850561797752809, 0.4, 0.4, 0.19999999999999998, 0.3155852219807513, 0.09195402298850576, 0.25271739130434784, 0.2857142857142857, 0.3850561797752809, 0.4, 0.49333333333333335, 0.22765282583621685, 0.39375203384315005, 0.19999999999999998, 0.22765282583621685, 0.39375203384315005, 0.5217391304347826, 0.17219251336898395, 0.3364193328795099, 0.22765282583621685, 0.11062190423775453, 0.43, 0.40834845735027225, 0.02777777777777778, 0.19994022713687984, 0.17049009384775807, 0.2680115273775216, 0.4615384615384615, 0.16666666666666669, 0.3850561797752809, 0.4411764705882353, 0.6666666666666666, 0.26174496644295303, 0.2827715355805243, 0.7195121951219512, 0.25, 0.22765282583621685, 0.39375203384315005, 0.7499999999999999, 0.4, 0.5748502994011977, 0.3220108695652174, 0.22765282583621685, 0.39375203384315005, 0.4, 1.0, 0.6666666666666666, 0.5409836065573771, 0.0, 0.0909090909090909, 0.396590066716086, 0.7000000000000001, 0.2345679012345679, 1.0, 0.391304347826087, 0.3017241379310345, 0.25, 0.25271739130434784, 0.625, 0.4, 0.24000000000000002, 0.5217391304347826, 0.3155852219807513, 0.2857142857142857, 1.0, 0.38211805555555556, 1.0, 0.2857142857142857, 0.44148936170212766, 0.6995073891625615, 0.3155852219807513, 1.0, 0.6182432432432432, 0.0, 0.02922077922077922, 1.0, 0.37272727272727274, 0.6797153024911032, 0.44695975503062113, 0.3364193328795099, 0.5185185185185185, 0.2857142857142857, 0.7027027027027027, 0.273109243697479, 0.22765282583621685, 0.4, 1.0, 0.22765282583621685, 0.03571428571428572, 0.22765282583621685, 0.35644371941272435, 0.36746143057503505, 0.32179226069246436, 0.6182432432432432, 0.0, 0.4361525704809287, 0.38211805555555556, 0.3155852219807513, 0.19994022713687984, 0.3220108695652174, 0.04545454545454545, 0.3850561797752809, 0.3636363636363636, 0.44695975503062113, 0.276595744680851, 0.44695975503062113, 0.25, 0.22765282583621685, 0.39375203384315005, 0.0, 1.0, 0.17049009384775807, 0.12578616352201258, 0.3364193328795099, 0.22765282583621685, 0.24752475247524755, 0.5254237288135594, 0.4682080924855491, 0.4604810996563574, 0.2577132486388385, 0.14996232102486812, 0.3155852219807513, 0.24018016130721695, 0.2728110599078341, 0.2151898734177215, 0.25271739130434784, 0.132, 0.2666666666666667, 0.36988950276243093, 0.2857142857142857, 0.4, 0.0, 0.19999999999999998, 0.22765282583621685, 0.39375203384315005, 0.5217391304347826, 0.31504315659679405, 0.07462686567164178, 0.3155852219807513, 0.0, 0.015822002472187888, 0.22551169590643275, 0.32547169811320753, 0.5, 0.869047619047619, 0.9878048780487805, 0.2857142857142857, 1.0, 0.46956521739130436, 0.3155852219807513, 0.3850561797752809, 0.7499999999999999, 0.44695975503062113, 0.32547169811320753, 0.5, 0.869047619047619, 0.3364193328795099, 0.3376152832674572, 0.33333333333333337, 0.38211805555555556, 0.14285714285714288, 0.25271739130434784, 0.4404145077720208, 0.31504315659679405, 0.07462686567164178, 0.408335762168464, 0.6592292089249493, 0.5, 0.5748502994011977, 0.31504315659679405, 0.31356898517673887, 0.38211805555555556, 0.5654520917678812, 0.3220108695652174, 0.25271739130434784, 0.33333333333333337, 0.34782608695652173, 0.25271739130434784, 0.25, 0.42567811934900546, 0.5403225806451613, 0.25271739130434784, 0.31504315659679405, 0.9236641221374046, 0.5540540540540541, 0.4373464373464373, 0.25271739130434784, 0.31504315659679405, 0.07462686567164178, 0.3155852219807513, 0.0, 0.015822002472187888, 0.22551169590643275, 0.32547169811320753, 0.5, 0.869047619047619, 0.9878048780487805, 0.2857142857142857, 1.0, 0.46956521739130436, 0.3155852219807513, 0.3850561797752809, 0.7499999999999999, 0.44695975503062113, 0.32547169811320753, 0.5, 0.869047619047619, 0.3364193328795099, 0.3376152832674572, 0.33333333333333337, 0.38211805555555556, 0.14285714285714288, 0.25271739130434784, 0.4404145077720208, 0.31504315659679405, 0.07462686567164178, 0.408335762168464, 0.3220108695652174, 0.22765282583621685, 0.39375203384315005, 0.5, 0.5748502994011977, 0.31504315659679405, 0.31356898517673887, 0.38211805555555556, 0.5654520917678812, 0.3220108695652174, 0.25271739130434784, 0.33333333333333337, 0.34782608695652173, 0.25271739130434784, 0.25, 0.42567811934900546, 0.5403225806451613, 0.2345679012345679, 0.31504315659679405, 0.9236641221374046, 0.15013679890560877, 0.6399999999999999]\n"
     ]
    }
   ],
   "source": [
    "intr_threshold_subopt = intr_threshold_opt - 1\n",
    "check_bayesian_diff(word_probs_bayes, \n",
    "                    intr_threshold=intr_threshold_subopt, \n",
    "                    display=f'Bayes interesting ≤ {intr_threshold_subopt}', \n",
    "                    true_label=0, \n",
    "                    include_graham_intr=True)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "The first $7$ interesting probabilities are $1$, accounting for the false positive result; then comes the $8$th value, a $0$, which makes it a true negative. So $8$ is legitimately the optimal threshold."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Neural networks\n",
    "\n",
    "Let's compare the Bayesian filters to neural networks! We'll start with TensorFlow's [basic text classification tutorial](https://www.tensorflow.org/tutorials/keras/text_classification).\n",
    "\n",
    "For a neural network, we need to further split our training dataset to training and validation subsets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Lengths from the output of loading\n",
    "len_all = 5172\n",
    "len_train = 4655\n",
    "\n",
    "assert len_test == len_all - len_train\n",
    "\n",
    "len_train_neural = len_train - len_test\n",
    "\n",
    "ds_train_neural = ds_train.take(len_train_neural)\n",
    "ds_val_neural = ds_train.skip(len_train_neural)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "The `Enron1` dataset is already lowercase, and to remain similar to the Bayesian filters, we don't want to strip HTML, so in the standardization, we just need to remove `\"Subject: \"` from the start of the emails and strip punctuation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def standardize(input_data):\n",
    "    cleaned = tf.strings.substr(input_data, len_subj, -1)\n",
    "    return tf.strings.regex_replace(cleaned,\n",
    "                                    '[%s]' % re.escape(string.punctuation),\n",
    "                                    '')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Here's our vectorization layer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "max_features = 10000\n",
    "sequence_length = 250\n",
    "\n",
    "vectorize_layer = tf.keras.layers.TextVectorization(\n",
    "    standardize=standardize,\n",
    "    max_tokens=max_features,\n",
    "    output_mode='int',\n",
    "    output_sequence_length=sequence_length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "ds_train_neural_text = ds_train_neural.map(lambda x, y: x)\n",
    "vectorize_layer.adapt(ds_train_neural_text)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Let's test the vectorization:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tf.Tensor(b'Subject: enron / hpl actuals for sept . 27 , 2000\\r\\nteco tap 67 . 500 / hpl gas daily ; 60 . 000 / enron', shape=(), dtype=string)\n",
      "tf.Tensor(0, shape=(), dtype=int32)\n",
      "tf.Tensor(\n",
      "[[  15   49  344    6  781  239   19  238  322 4397  421   49   36  185\n",
      "   345   52   15    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0    0    0]], shape=(1, 250), dtype=int64) tf.Tensor(0, shape=(), dtype=int32)\n"
     ]
    }
   ],
   "source": [
    "for email, label in ds_train_neural.unbatch():\n",
    "    if 'enron' in email.numpy().decode(encoding):\n",
    "        print(email)\n",
    "        print(label)\n",
    "        print(vectorize_layer(tf.expand_dims(email, -1)), label)\n",
    "        break"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "The first and last words of the above email are `\"enron\"`, which was encoded as `15`, so `\"Subject: \"` was correctly removed. `\"hpl\"` is the second and 12th word, excluding punctuation, and `49`, its encoding, appears at these places, so punctuation was also correctly removed."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Configure the datasets for performance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "AUTOTUNE = tf.data.AUTOTUNE\n",
    "\n",
    "ds_train_neural = ds_train_neural.cache().prefetch(buffer_size=AUTOTUNE)\n",
    "ds_val_neural = ds_val_neural.cache().prefetch(buffer_size=AUTOTUNE)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Set random seed for deterministic dropout layer and optimizer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "tf.random.set_seed(seed)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "Create, train and evaluate the model (for the metrics, we use false positives and false negatives for comparison to the Bayesian filters):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " text_vectorization (TextVec  (None, 250)              0         \n",
      " torization)                                                     \n",
      "                                                                 \n",
      " embedding (Embedding)       (None, 250, 16)           160016    \n",
      "                                                                 \n",
      " dropout (Dropout)           (None, 250, 16)           0         \n",
      "                                                                 \n",
      " global_average_pooling1d (G  (None, 16)               0         \n",
      " lobalAveragePooling1D)                                          \n",
      "                                                                 \n",
      " dropout_1 (Dropout)         (None, 16)                0         \n",
      "                                                                 \n",
      " dense (Dense)               (None, 1)                 17        \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 160,033\n",
      "Trainable params: 160,033\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "embedding_dim = 16\n",
    "\n",
    "model = tf.keras.Sequential([\n",
    "  vectorize_layer,\n",
    "  tf.keras.layers.Embedding(max_features + 1, embedding_dim),\n",
    "  tf.keras.layers.Dropout(0.2),\n",
    "  tf.keras.layers.GlobalAveragePooling1D(),\n",
    "  tf.keras.layers.Dropout(0.2),\n",
    "  tf.keras.layers.Dense(1, activation='sigmoid')])\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "model.compile(loss=tf.keras.losses.BinaryCrossentropy(),\n",
    "              optimizer='adam',\n",
    "              metrics=[tf.metrics.FalsePositives(thresholds=prob_threshold),\n",
    "                       tf.metrics.FalseNegatives(thresholds=prob_threshold)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "146/146 [==============================] - 2s 6ms/step - loss: 0.6345 - false_positives: 0.0000e+00 - false_negatives: 1352.0000\n",
      "Epoch 2/10\n",
      "146/146 [==============================] - 1s 6ms/step - loss: 0.5336 - false_positives: 0.0000e+00 - false_negatives: 1352.0000\n",
      "Epoch 3/10\n",
      "146/146 [==============================] - 1s 6ms/step - loss: 0.4657 - false_positives: 0.0000e+00 - false_negatives: 1333.0000\n",
      "Epoch 4/10\n",
      "146/146 [==============================] - 1s 6ms/step - loss: 0.4027 - false_positives: 0.0000e+00 - false_negatives: 1287.0000\n",
      "Epoch 5/10\n",
      "146/146 [==============================] - 1s 6ms/step - loss: 0.3474 - false_positives: 0.0000e+00 - false_negatives: 1266.0000\n",
      "Epoch 6/10\n",
      "146/146 [==============================] - 1s 6ms/step - loss: 0.3019 - false_positives: 1.0000 - false_negatives: 1229.0000\n",
      "Epoch 7/10\n",
      "146/146 [==============================] - 1s 6ms/step - loss: 0.2648 - false_positives: 2.0000 - false_negatives: 1166.0000\n",
      "Epoch 8/10\n",
      "146/146 [==============================] - 1s 6ms/step - loss: 0.2352 - false_positives: 4.0000 - false_negatives: 1113.0000\n",
      "Epoch 9/10\n",
      "146/146 [==============================] - 1s 6ms/step - loss: 0.2112 - false_positives: 2.0000 - false_negatives: 1048.0000\n",
      "Epoch 10/10\n",
      "146/146 [==============================] - 1s 6ms/step - loss: 0.1900 - false_positives: 3.0000 - false_negatives: 982.0000\n",
      "CPU times: user 12.1 s, sys: 676 ms, total: 12.8 s\n",
      "Wall time: 10.3 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f2f96ef4130>"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def fit_model(epochs=10, callbacks=None):\n",
    "    return model.fit(\n",
    "        ds_train_neural,\n",
    "        validation_data=ds_val_neural,\n",
    "        epochs=epochs,\n",
    "        callbacks=callbacks)\n",
    "    \n",
    "%time fit_model()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "17/17 [==============================] - 0s 4ms/step - loss: 0.2088 - false_positives: 1.0000 - false_negatives: 107.0000\n",
      "{'false_positives': 1.0, 'false_negatives': 107.0}\n"
     ]
    }
   ],
   "source": [
    "def evaluate_model():\n",
    "    tf_res = model.evaluate(ds_test)\n",
    "    tf_res_dict = {k: v for k, v in zip(model.metrics_names, tf_res)}\n",
    "    res = {'false_positives': None, 'false_negatives': None}\n",
    "    for tf_k in tf_res_dict:\n",
    "        for k in res:\n",
    "            if k in tf_k:\n",
    "                res[k] = tf_res_dict[tf_k]\n",
    "    return res\n",
    "\n",
    "results_neural = evaluate_model()\n",
    "\n",
    "print(results_neural)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEICAYAAACeSMncAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAtfUlEQVR4nO3dedzlc/3/8cdg7CFZvraoKE3PkLFVKlrs0mLNrpqI+hKVsk2ktBDfhIZ8LSGVbSyFn/iSJDOyPEWbyL6vUWa5fn+83xfHcS1nzlznWuZ63m+363ad81nfZ+Zc53Xe2+s9pquri4iIiHbMNdQFiIiIkStBJCIi2pYgEhERbUsQiYiItiWIRERE2xJEIiKibQkiERHRtnmGugAR7bB9D7A0MKNh82mS9hmaEg1PticCK0vaaajLEnOmBJEYybaU9P/6O8j2PJKmN22bW9KM3s7p4RqzdPyssD0GGCNpZieuH9FJYzJjPUaiWhP5TE9BxPZuwGeBPwC7ACcCywMvAisCHwC2Ah6o+9aoj78maXK9xmnNxzffy/Y1wA3Ah4BVgauB3SU9WfevBxwDjAPuBf5b0jUN514PbACsCbwTmA84FhgPTAOOk/Qt23MBX6mvaTHgKmBPSU/aXgn4B7AbcASwIPADSUfa3gSYDIwB/gP8XdLqtnev11seeAz4jqQfN7yurwD7AV3AocDJwCqS/mZ7PuBIYNta3guA/SS9aHsJ4DRgfWAmcAfwgQTHOVv6RGJOtS5wN6XJ68i67VP18euAG4GLgSuApYAvAGfZflvDNRqP/20v99kF2ANYBpgO/A+A7eWAS4FvAosDBwDn2V6y4dydgQn1+o8A/w/4NbAssDIlWFDL9jFKMFsWeAr4UVM51gfeRgloh9p+u6RfA98CzpW0sKTV67GPAlsAiwC7Az+wvWYt9ybAl4AP1zJs0HSfo4C3UgLvysBylEADsD9wP7Ak5d/965RAFHOwNGfFSHah7cZmqi9LOrk+flDSD+vj6bYBLpJ0PYDtNYCFgaPqN+Xf2L4E2AGYWM97+Xjg372U4UxJrtc8BLjF9q7ATsBlki6rx11pewqwGXB63XaapDvquVsAD0s6uuF+N9bHewL7SLq/HjsR+KftnRvK8Q1JLwK32r4VWB24s6cCS7q04en/2b4CeB9wM6WG8b8N5ZoI7Fgfj6EEvdUaalvfAs4GvkapPS0DrCjpb8B1vfybxRwkQSRGso/10SdyXz/blgXua2pquZfyzbqva/R1zXuBscASlGawbWxv2bB/LKXJq6dzVwD+3ss9VgQusN1Y1hmUb/vdHm54/AIlQPbI9qbAYZQaxVyUJrDb6+5lgSm9lHHJeuzUGpShNJXNXR9/jxKAr6j7J0k6qrdyxJwhQSTmVD01ozRuexBYwfZcDYHkjcBf+rlGsxUaHr+R8m38ccqH75mSPttiGe8Dtu/luPuAPRpqRS+rfSJ9edVrqH0a51Ga4S6SNM32hZRgAPAQpa+kW+Pre5zST/QOSQ8030jSc5Qmrf1ti1K7u0nSVc3HxpwjfSIxWt1I+cb+FdtjbW8AbAn8bBavs5PtcbYXBA4HfllHcf0U2NL2xrbntj2/7Q1sL9/LdS4BlrG9r+35bL/O9rp130nAkbZXBLC9pO2tWizfI8BKtXMeYF5Kh/hjlGa+TYGNGo7/ObC77bfX13RI944abE+m9KEsVcuynO2N6+MtbK9cm72eodSW0qk+h0sQiZHsYtvPN/xc0OqJkl6iBI1NKd+wTwB2kXTXLJbhTMqIpIeB+YEv1uvfRxkB9nXKB/Z9wJfp5W+ufov/SC3Tw8BfgQ3r7uMoo6yusP0c8HvKwIFW/KL+fsL2zfU+X6QEi6cogwcmN5TjV5TBAVcDf6v3gjK6C+Cr3dttP0sZDNA9GGGV+vx5yqi1EyQ1Nt/FHChDfCPaVIfp/lTSKUNdlk6x/XbAwHzNc20iIH0iEdHE9seByyid6N8BLk4Aid6kOSsimn2OMpfk75R+jb2GtjgxnKU5KyIi2pbmrIiIftheATiDMjenizIH5jjbiwPnAisB9wDbSnqq6dwNgR80bFoV2F7ShbbfRBkR+AZgKrCzpJdsf4FSI/wnZT7US7bXBz4pab8OvtRZNupqIrYfo0wKi4ho1dj68wKlG2AcZZTaEpR0Nw8D/0WZePmaOTQN5qbkSbuNMvz5zZRRck9R5hm9SBnNtypwFyUDwAuUIdOrUFL5dCQRaAtWlLRk88bRWBO5V9JaQ12IiBi5bF8EHF9/NpD0kO1lgGv6+nyxPYGSlHLHOp/mMeBtkqbbfjcwUdLGtm8E3k/JLHAtJVvAGyQd29lX1ruatuc1RmMQiYhoW80S8C7KhNWlJT1Udz3Mq1PR9GR7SmZnKE1YTzeMfLufV9LuHE+Zo3MHJdvzRcDGA1H+gZbRWRERLbK9MCVtzL6Snm3cJ6mLPlLl1JrKO4HL+7uPpDMlvasuJrYfZQLoprZ/afsHDRkIhtywKUhExHBmeywlgJwl6fy6+ZEaHLqDxKN9XGJb4AJJ0+rzJ4DFbHe3CC1PU3+K7WWBdSRdSMlLth3wNCXl/7CQ5ixg6tSpSwHfp3RmJbCOLDMpHZAHjB8/vq8/4Ii21f6LnwB3SjqmYddkYFfKOiu7UpqderMDJWU+UGoutq8GtqaM0Orp/CN4Zb2WBSg1nZmUiaDDQoJI8X1KrqBPjx8/flp/B8fwMXXq1LGUxZ2+T8lMG9EJ76W8z263fUvd9nVK8Pi57U9TRn1uC2B7Lcrqk5+pz1eiZET+v6brfhX4me1vAn+kBCrqOe8CkHRz3XQ2JWX/fcB3B/bltW80DvGd0jx6YurUqX8A3psAMjLVQHL9+PHj1xnqskTMqXr67IQ03XSbKwFk5Kr/d3kvRwyB/OFFRETb0ifSg5UOvHRA2/juOWrzMf0dY3sGpb1zDGVG6j6SfjeQ5ejlvqcAx0j602xeZzdgLUn7DMB1rpD04OxcJyIGR4LI8PGipDUA6kpx3wY+0Ombdnf8DSO7UdavSBCJzpi46OjqCG408Zl+v9DOqgSR4WkRSi6d7slNFwGvp+TuOVjSRbYPB57sToNg+0jg0ZoU7suUUSLzUcalH2Z7IcpqdstT8vccIencurDSAZKm2D4RWJsylPCXkg6r174HOJ2y6t5YYJteVgBcoV5vOcpiTd+o5+9EWU1vXsos38/X438CrEUZtngqZdTJWsBZtl8E3i3pxdn6l4yIjkoQGT4WqEMH56ckXftg3f5v4OOSnrW9BGVZ0smUD93zgWPr7NXtgXVsb0RJ1LYOpWlssu33U3LvPChpcwDbi/ZQhoMkPWl7buAq26tJuq3ue1zSmrY/DxwA9FSDWQcQJWHcTbYvBf5FmSD1XknTbJ8A7EhJ57CcJNXyLCbpadv7UINaO/+IETG4EkSGj8bmrHcDZ9gWJRB8qwaCmZRv+UtLusf2E3Us+dLAHyU9UYPIRpQx5wALU4LKdcDRtr8DXCLpuh7KsG1NEDcPJZCNo2QbhRKwoKSr/kQvr+FKSU/U13A+sD4lw+l4SlCBUst5FLgYeLPtHwKXAle0/k8VEcNFx4LI7OTfr+fvChxcn35T0um256M07SwPnCDphHrsJOCkhkk5I5qkG2qtY0lgs/p7fP0mfw+ltgJwCqUP4b8oNRMoQefbkn7cfF3ba9brfdP2VZIOb9j3JkoNY21JT9k+reE+AP+pv2fQ+/umua25q5bndElfaz7Y9uqUpHJ7Uprf9ujluhExTHVyiO90YH9J44D1gL1tjwMOBK6StApwVX3+KjXQHAasS2kiOcz26ykfOL8FVqPMHu3+IJp7TgkgALZXpfRbPAEsSunrmFYXt1mx4dALgE0o/RjdSd0uB/aofSnYXs72UjUHzwuSfgp8D1iz6baLUJqenrG9NLBpG0X/iO3FbS8AfIySffQqYGvbS9XyLG57xRok55J0HuXLQnd5ngNe18a9I2IIdKwmUtMjP1QfP2f7TkpTzFbABvWw04FrKFP/G21MaRp5EsD2lZQPy6cpOWPGUr7hQskts+dAlr2VIbkd0N0nAuW17Spphu2zgItt3w5MoeSJAqCudnY1JZ30jLrtCttvB26ozUfPAzsBKwPfsz0TmEbTutmSbrX9x3r9+ygBYFb9gZKgbnlKx/oUANsHA1fUvptpwN6UxXf+tyEbaXdN5TTgpHSsR4wMg9In0kb+/eUoH2TduvPs/4JSA/k95QPxo8DN/c0pqO38E+rTJdp8GR0lae5etj8OvLunffUDeD1gm6ZzjgOOazr87/SQglrSBg2Pd+ulDCs1PJ7CK18CGo85jRIAejr/XEoTZrPm2hC1ZnJeT9eJiOGn40GkOf9+/XYMvJzFsuUx23Xxlk/V646lfChuZfsYytKSZ0ia3MN5k4BJ9bw5YtRPbRq8hDKE969DXZ6IGJ06GkT6yr/fsJxkT+m7H+DV33aXpzR7Nfo8peN+Pcr6w9sBv6GkZp7j1Rnmbx7qckTE6NaxjvUW8u9D7/n3Lwc2sv362qG+EQ1NMXXbFpQgsiBl6GsXZfhoREQMkk6OzurOv/9B27fUn80o+fc/YvuvwIfrc2yvVfM4UTvUjwBuqj+Hd3eyV4cCR0qaSQku76PknTqzg68nIiKadHJ01m95ZQRVs9cs7Vg7bD/T8PxUXpn70Hzsfg2P/02pqURExCBLKviIiGhb0p70ZKCzfLaQObOOUjtG0v71+QHAwpImDmhZXnvfaxiAXFW21wCWlXTZQJSr4bobUMq3xUBeNyIGRmoiw8d/gE/UmdwDxvaYhgl9nbQGJaXKgLGdLzkRw1z+SIeP6ZS5LPsBBzXusL0kcBJlLgyUOTfX254IPC/p+/U4U0atQRlwcCMl+eFmtg+khzTvvekt/XtNKf9DSrbescBE4FfA4ZRZ9+tT1kI5mDLg4RngcWA/SWfYPoMyAOI64ERK6vfpwJckXV0XpfoEJXHk3JT0N91lWrv+G20t6e99/3NGxGBITWR4+RGwYw9p2o8DfiBpbeCTlMSL/VmFkqTyHZLupaR5X4uSd+wDtldr4RqPS1qT8mF/QN12EPAbSesAG1LycI2ljJg7V9IadYb69ZQReu8A7qYEFCiz739HSX3SJemdwA7A6ba7Ez6uSQkULy/KZfs9lEC6VQJIxPCRIDKMSHqWMvfli027PgwcX3NrTQYW6U6w2Id7Jf2+4fm2tm+mpIh/ByXNe38a07+vVB9vBBxYy3INJdPvG5tPpNQ03l9/TgTeaXs54ClJ/6Kkif8pQF3g6l7grfXcl/OmVW+n1EC2lPTPFsodEYMkzVnDz7HAzcD/NmybC1ivDmd+me3pvPqLQGPq9n81HNdfmvfe9JT+fQzwSUl/birLuk3nXkupbbyRUnv5OLA1Jbj0519Nzx+q5X0XWTY3YlhJTWSYqd/Afw58umHzFcAXup/UkVBQ1mNZs25bE3hTL5cdiDTv3S4HvlAzElAXxYKmFO6S7qMku1xF0t2UFP4HUIILlGCyY73GWynB5lWBqcHTwObAt+torYgYJlIT6UkHFrOfRUcD+zQ8/yLwI9u3Uf7PrqWkvz8P2MX2HZRO9L/0dLEBSvPe7QhKbem2OurrH5TO/Kt5pZnr27Vf5EZK5ziUoPFtSjABOAE4saa4nw7sJuk/jQk6m17DI7a3AH5lew9JN87Ga4iIATKmq2tgp0QMd7an1A7ml02dOnXK+PHj1+rtnBj+8n8YLRvoeWAjyWx8Qe7psxPSnBUREbMhQSQiItqWIFLMnDp16tihLkS0p/7fzRzqckSMRgkixV3AzgkkI0/9P9uZhrXnI2LwZHRWcQDwfWDPqVOnJrCOLDMpAeSA/g6MiIGXIAKMHz/+UWCXoS5HRMRI07EgYvtUyvyBRyWpbjsXeFs9ZDHgaUlr9HDuPZTJazOA6d3Dymx/hzJR7hZJu9RtOwFLSDq2U68lIiJ61smayGnA8ZRcUABI2q77se2jKRlee7OhpMcbjl8UWFPSarZPsf1O4G/A7sAmA1z2iIhoQcfa/yVdCzzZ076aMmNb4JxZuORMYGw9d0FgGqUd/IeSps1mcSMiog1D1Yn8PuARSX/tZX8XcIXtqbYnAEh6DriMkoX2IUotZl1JF/Z3M9sTbE+xPYWSzykiIgbAUHWs70DftZD1JT1geyngStt3SbpW0neB7wLYPgU41PZnKOnJb5P0zZ4uJmkSJZU4NZBERMQAGPSaSF3y9BPAub0dI+mB+vtR4AJgnaZrvIuSkvzPlBX3tgXeYnuVTpU7IiJeayiasz4M3CXp/p522l7I9uu6H1NqGc2pXY8ADqGsqNedJXYmpa8kIiIGSceCiO1zgBuAt9m+33b3+hjb09SUZXtZ25fVp0sDv7V9K/AH4FJJv2449mPAFEkPSnoauKWmE59f0q2dej0REfFaSQUfEaNLUsG3JangIyJiwCWIRERE2xJEIiKibQkiERHRtgSRiIhoW4JIRES0LUEkIiLaliASERFtSxCJiIi2JYhERETbEkQiIqJtCSIREdG2BJGIiGhbgkhERLQtQSQiItqWIBIREW2bp1MXtn0qsAXwqCTVbROBzwKP1cO+LumyHs7dBDiOsvTtKZKOqtvPAt4JXCLp63XbwYAlXdip1xIRET3rWBABTgOOB85o2v4DSd/v7STbcwM/Aj4C3A/cZHsypawvSlrN9pW2F6Wsqb6upG924gVERETfOtacJela4Mk2Tl0H+JukuyW9BPwM2AqYBixgey5gLDADOBw4bICKHBERs6iTNZHe7GN7F2AKsL+kp5r2Lwfc1/D8fkpt407bjwE3A2cCKwNzSbq5vxvangBMqE+XmN0XEBERxWAHkROBI4Cu+vtoYI9WT5a0b/dj2xcDn7N9ELA6cKWkk3s5bxIwqZ43pd3CR0TEqw3q6CxJj0iaIWkmcDKl6arZA8AKDc+Xr9teZnsrYCqwMPAWSdsCW9tesDMlj4iIngxqELG9TMPTjwPu4bCbgFVsv8n2vMD2wOSGa4wF9gW+CyxAqdVAGck1bweKHRERvejkEN9zgA2AJWzfT+kA38D2GpQP/nuAz9Vjl6UM5d1M0nTb+wCXUwLDqZLuaLj03sDpkl6wfRuwoO3bgcskPd2p1xMREa81pqurq/+j5iC2p0haa6jLERFDZOKio+tDr9HEZ8a0e2pvn539NmfZ3sb26+rjg22fb3vNdgsSERFzjlb6RA6R9Jzt9YEPAz+hjLKKiIhRrpUgMqP+3hyYJOlS0oEdERG0FkQesP1jYDvgMtvztXheRETM4VoJBttSRkptXEc/LQ58uZOFioiIkaHXIb62F294ek3Dtv9QUpZERMQo19c8kamU+Rw9DQnrAt7ckRJFRMSI0WsQkfSmwSxIRESMPH01Z60q6a7e5oS0kj03IiLmbH01Z32Jkj796B72dQEf7EiJIiJixOirOWtC/b3h4BUnIiJGkpYSMNoWMA6Yv3ubpOZlbyMiYpTpN4jYPoySjXcccBmwKfBbXrt2ekREjDKtTDbcGvgQ8LCk3SmrCC7a0VJFRMSI0EoQebGuRDjd9iLAo7x65cGIiBilWukTmWJ7McpytlOB54EbOlmoiIgYGfoNIpI+Xx+eZPvXwCKSbuvvPNunAlsAj0pS3fY9YEvgJeDvwO49rUZo+x7gOUoG4endC6HY/g6lT+YWSbvUbTsBS0g6tr8yRUTEwGopG6/t1Wx/FFgTWNn2J1o47TRgk6ZtVwKStBrwF+BrfZy/oaQ1GgLIosCa9dyXbL/T9gLA7sCPWnkdERExsFoZnXUqsBpwBzCzbu4Czu/rPEnX2l6padsVDU9/T+m0b9VMYKztMcCCwDTgAOCHkqbNwnUiImKAtNInsp6kcR249x7Aub3s6wKusN0F/FjSpLq64mXAH4GrgGeAdSUd0d+NbE+gzL4HWGL2ix4REdBaELnB9jhJfxqom9o+CJgOnNXLIetLesD2UsCVtu+SdK2k7wLfrdc4BTjU9meAjYDbJH2zp4tJmgRMqucljX1ExABppU/kDEog+bPt22zfbrvfjvXe2N6N0uG+o6Suno6R9ED9/ShwAbBO0zXeRUlR/2dgG0nbAm+xvUq75YqIiFnXSk3kJ8DOwO280ifSFtubAF8BPiDphV6OWQiYqzZfLUSpZRzedNgRlOapscDcddtMSl9JREQMklaCyGOSJs/qhW2fQ0mXsoTt+4HDKKOx5qM0UQH8XtKetpcFTpG0GbA0cEHdPw9wtqRfN1z3Y8AUSQ/W57fYvp3SnHXrrJYzIiLaN6arq8cWpZfZPgFYDLiYsjQuAJL6HJ01XNme0j1sOCJGoYmL9v2hNyeb+ExPK9W2pLfPzlZqIgtQgsdGDdv6HeIbERFzvlZmrO8+GAWJiIiRp9/RWbbfavsq106KOnv94M4XLSIihrtWhvieTOkQnwZQ82Zt38lCRUTEyNBKEFlQ0h+atk3vRGEiImJkaSWIPG77LZTOdGxvDTzU0VJFRMSI0MrorL0pKUNWtf0A8A9gx46WKiKGpV6WeFickgdvJeAeYFtJT/Vw7q5Ad3/qNyWdXrfPCxxPmVc2EzhI0nm2vwB8Dvgn8DFJL9leH/ikpP069iJjlvRbE5F0t6QPA0sCq0paX9K9nS9aRAxDp/HaJR4OBK6StAolOeqBzSfVQHMYsC4ljdFhtl9fdx9ECUpvBcYB/1e370jJIP47YOOawfsQSsaKGCZaqYkAIOlfnSxIRAx/PS3xAGxFqUUAnA5cA3y16ZiNgSslPQlg+0pKMDqHktF71Xr9mcDj9ZwxlNRG3Us/7AT8qvsaMTy0HEQiInqxtKTuftKHKamLmi0H3Nfw/H5gubr0NsARtjegrHi6j6RHKE1cv6esZXQ9cBElGMUw0mtzlu1t6u83DV5xImIkq5m5ZyWtyDzA8sDvJK0J3AB8v17rTEnvkrQTsB/wP8Cmtn9p+we2W1qZNTqrr/+E7qVrzxuMgkTEiPWI7WUA6u9HezjmAWCFhufL121PAC/wShqlX1CW4X5ZTdC6jqQLgf2B7YCngQ8N2CuItvXVnPWE7SuAN9l+TRZfSR/tXLEiYgSZDOwKHFV/X9TDMZcD32roTN8I+JqkLtsXU/pUfkMJDM0L4B0BHFofL0Cp6WTph2GiryCyOeUbwZnA0YNTnIgYznpZ4uEo4Oe2Pw3cC2xbj10L2FPSZyQ9afsI4KZ6qcMbOsi/Cpxp+1jgMWD3hvu9C0DSzXXT2ZS1je6jrnIaQ6uVVPBLSnrM9sIAkp4flJJ1SFLBR4xySQXflt4+O1vpmFra9h8pIyT+ZHuqbbVbkIiImHO0MsR3EvAlSVcD1GF4k4D39HfiQM9utT0fpb11eeAESSfUYycBJzVUeSMiYhC0UhNZqDuAAEi6BlioxeufxsDObt0Y+C1lFuvO9djVgbkTQCIiBl8rNZG7bR9C6WCHMmv07lYu3oHZrU9TRmSMpcxmhTJyY89WyhMRxUoHXjpq+wXumX+oSzBnaSWI7AF8gzKOuwu4rm5rV9uzWyljyHemzGL9nu2PAjdLerCvG9qeAEyoT5eYjbJHRESDVpbHfQr4YiduXseIt/yNSNJ04FMAtsdSxp5vZfsY4I3AGZJ6mtMyidKPg+0pA1H2iIhorU9koM3O7NZGnwfOANYDnqHMYt1/wEsbERG9Goog0j27Ffqe3bqR7dfXDvWN6jYA6rYtKEFkQcrs1S7KbNaIiBgkHQ0idXbrDcDbbN9fZ7QeBXzE9l+BD9fn2F7L9ikAtUO9e3brTbx6diuUFAhH1rTRlwPvo8xiPZOIiBg0rcxYXx74IbA+r3Ss/7ek+ztfvIGXGesRo3101qeGughDZ4hmrP8vpQlqGWBZ4OK6LSIiRrlWhvguKakxaJxme98OlSciIkaQVoLIE7Z3oixjCbADZQ2AiIgY5VppztqDktr5YeAhYGsaUjVHRMTo1cpkw3uBLEAVERGv0WsQsf0VSd+1/UN6WDNZUkdmsUdExMjRV03kzvo7aUIiIqJHvQYRSRfXhy9I+kXjPtvbdLRUERExIrTSsf61FrdFRMQo01efyKbAZsBytv+nYdciwPROFywiIoa/vvpEHqT0h3wUmNqw/Tlgv04WKiIiRoa++kRuBW61fbakaYNYpoiIGCFambG+ku1vA+OAlxeWlPTmjpUqIiJGhFYTMJ5I6QfZkLKGx087WaiIiBgZWgkiC0i6Chgj6V5JE4HNO1usiIgYCVppzvqP7bmAv9reh7JM7cKdLVZERIwErQSR/6YsQftFymqDHwR2afeGtt8GnNuw6c3AoZKObThmA8qyuf+om86XdLjtJYELgMWAgyVdWI+/CNhL0oPtlisiImZdKwkYb6oPnwd2tz03sD1wYzs3lPRnYA2Aeq0HKIGh2XWStmjatgNwEnA+cBlwoe0tgT8mgEREDL6+JhsuAuwNLEdZ2fDK+nx/4DbgrAG4/4eAv9dMwa2YRqkVzQfMsD0PsC+w5QCUJSIiZlFfNZEzgaeAG4DPAF8HxgAfl3TLAN1/e15Z7KrZu23fSpn0eICkO4Cz688E4KvA54EzJb3Q101sT6jnACwxEAWPiIi+g8ibJb0TwPYplAWp3ijp3wNxY9vzUmbD95SH62ZgRUnP294MuBBYRdIz1JFhtl8PHAh83PbJwOuBoyXd0HwxSZOASfW8ZCWOiBggfQ3xfXmWuqQZwP0DFUCqTYGbJT3SvEPSs5Ker48vA8babq5BHAIcSekn+S2wKzBxAMsXERH96KsmsrrtZ+vjMcAC9fkYoEvSIrN57x3opSnL9n8Bj0jqsr0OJdg90bB/FWB5SdfYXh34N2XhrAVms0wRETEL+sqdNXenbmp7IeAjwOcatu1Z73sSZR33vWxPB14EtpfUuLrikcBB9fE5lOauA4FDO1XmiIh4rVbmiQw4Sf8C3tC07aSGx8cDx/dx/rYNjx8F3tOBYkZERD9aSXsSERHRowSRiIhoW4JIRES0LUEkIiLaliASERFtSxCJiIi2JYhERETbEkQiIqJtCSIREdG2BJGIiGhbgkhERLQtQSQiItqWIBIREW1LEImIiLYliERERNsSRCIiom1DsigVgO17gOeAGcB0SWs17R8DHAdsBrwA7CbpZttvA84GxgKfk3SD7XmAXwMflfTCIL6MiIhRbciCSLWhpMd72bcpsEr9WRc4sf7+HPDfwD2UIPNJYC/gpwkgERGDa6iDSF+2As6oa6v/3vZitpcBpgEL1p9pthcDtgQ2GbKSRkSMUkMZRLqAK2x3AT+WNKlp/3LAfQ3P76/bfgScAcxHqZUcAnxL0szebmR7AjChPl1iYIofERFDGUTWl/SA7aWAK23fJena/k6S9E9gAwDbKwPLA3faPhOYFzhE0l+azpkETKrnTBnYlxERMXoN2egsSQ/U348CFwDrNB3yALBCw/Pl67ZGRwIHA18ETgG+AhzWifJGRMRrDUkQsb2Q7dd1PwY2Atx02GRgF9tjbK8HPCPpoYZrfAB4UNJfKf0jM+vPgoPxGiIiYuias5YGLrDdXYazJf3a9p4Akk4CLqMM7/0bZYjv7t0n1+G/BwPb1U2TgLPqtfYapNcQETHqDUkQkXQ3sHoP209qeNwF7N3L+V3ARxqe3wmsOfAljYiIvmTGekREtC1BJCIi2pYgEhERbUsQiYiItiWIRERE2xJEIiKibQkiERHRtgSRiIhoW4JIRES0LUEkIiLaliASERFtSxCJiIi2JYhERETbEkQiIqJtCSIREdG2BJGIiGjboC9KZXsF4AzK6oZdwCRJxzUdswFwEfCPuul8SYfbXpKyHvtiwMGSLqzHXwTsJenBwXgNERFRDMXKhtOB/SXdXNdZn2r7Skl/ajruOklbNG3bATgJOJ+yfO6FtrcE/pgAEhEx+AY9iEh6CHioPn7O9p3AckBzEOnJNGBBYD5ghu15gH2BLTtT2oiI6MuQrLHezfZKwLuAG3vY/W7btwIPAgdIugM4u/5MAL4KfB44U9IL/dxnQj0HYImBKX1ERAxZELG9MHAesK+kZ5t23wysKOl525sBFwKrSHoG2Lye/3rgQODjtk8GXg8cLemG5ntJmgRMqudN6dBLiogYdYZkdJbtsZQAcpak85v3S3pW0vP18WXAWNvNNYhDgCMp/SS/BXYFJnay3BER8WqDHkRsjwF+Atwp6Zhejvmvehy216GU84mG/asAy0u6htJHMpMy0muBzpY+IiIaDUVz1nuBnYHbbd9St30deCOApJOArYG9bE8HXgS2l9TVcI0jgYPq43MozV0HAod2uvAREfGKoRid9VtgTD/HHA8c38f+bRsePwq8Z8AKGBERLcuM9YiIaFuCSEREtC1BJCIi2pYgEhERbUsQiYiItg1p2pMYXWxvAhwHzA2cIumopv3vB44FVqMM6/5lw77vULMVAEdIOrduPwt4J3CJpK/XbQcD7s7yHBGdk5pIDArbcwM/AjYFxgE72B7XdNg/gd0o+dEaz90cWBNYA1gXOMD2IrZXA16UtBqwtu1FbS8DrJsAEjE4UhMZZC18G5+Pst7KeMos/e0k3VP3fQ34NDAD+KKky0fQGivrAH+TdDeA7Z8BW9GQvbnhdc5sOncccK2k6cB027cBmwC3AwvYngsYS/l3ORw4rLMvJSK6pSYyiFr8Nv5p4ClJKwM/AL5Tzx0HbA+8g/IBekK9XvcaK+tQ0uIzTNdYWQ64r+H5/XVbK24FNrG9YM2htiGwgqQ7gccoCTsvBlYG5pJ088AVOyL6kprI4Or323h9PrE+/iVwfM0jthXwM0n/Af5h+2/1enP8GiuSrrC9NvA7StC4gVLrQNK+3cfZvhj4nO2DgNWBKyWdPPgljhg9UhMZXK18G3/5mNp88wzwhj7OPZsSYK4EvkWLa6wMgQeAFRqeL1+3tUTSkZLWkPQRStqcvzTut70VMBVYGHhLTY2zte0FZ7vkEdGr1ERGuHbXWBkCNwGr2H4TJXhsD3yqlRNrs91ikp6onemrAVc07B9LqX1tDqxCyegMpd9pXmC4BdSIOUZqIoOrlW/jLx9Tm6YWpXSwt3LusF1jpdaq9gEuB+4Efi7pDtuH2/4ogO21bd8PbAP82PYd9fSxwHW2/0RZXGyner1uewOn19rXbcCCtm8Hpkp6ejBeX8RolZrI4Grl2/hkyof/DZSU+L+R1GV7MnC27WOAZSnfuP/QfVLjGiu2Vwf+zTBbY6UuMHZZ07ZDGx7fRAmOzef9mzIQobfrHtvwuIsSRCNiEKQmMoha+TZOWbDrDbXj/EuU5inqGvM/p3TC/xrYW9KMhss3r7GyFyVoHdfZVxURo9mYrq6u/o+ag9ieImmtoS5HxFBa6cBLR9cffoN75m+pK27ONPGZPtdy6ktvn51D0pzV7oQ72+8FTgReAnaQ9Ffbi1G+oW8iqXmSWkREdNCgB5GGCXcfoQxTvcn2ZEmNcyVennBne3vKhLvtgP2BzYCVgD3r84OBbyWAdNjERUfnN9fZ+OYWMRoMRZ/IyxPuJL0EdE+4a7QVcHp9/EvgQ3XCXffEugWBabbfQpm5fM2glDwiIl5lKJqzepo0t25vx0iabrt7wt23Kc1cLwI7A9+n1ET6ZHsCMKE+fZvtKbPzAkalra+/F3h8qIsx6ObQ98olO604dajLMFTM9UswGt/LMLvv5xV72jiihvhKugVYD15OG/4QMMb2uZRayv6SHunhvEmU+QXRpgxIiDlF3ssDayias2Znwh112xhKDeQISsbWrwAnA1/sWKkjIuI1hiKIvDzhzva8lAl3k5uO6Z5wBw0T7hr27wJcJulJSv/IzPqTPEkREYNo0Juzah9H94S7uYFTuyfcAVMkTaZMuDuzTrh7khJoAKgJ9XYDNqqbjqHMgn6JFnMxRVvSHBhziryXB9Com2wYEREDJ2lPIiKibQkiERHRthE1xHe0sj2Dsp54t491r0few7HPS1p4UArWB9vLAv8jaWvbawDL1iy+1GST45rT3cScayS+h/tjezfgiu5lqG2fAhzTlH1jjpcgMjK8KGmNoS7ErKh/WFvXp2sAa1HTwNfBE80j8mLONuLewy3YDTDwIICkzwxpaYZIgsgIZHth4CLKyoVjgYMlXdR0zDLAucAilP/nvSRdZ3sj4BuUNdn/Duwu6fmmc68BbgU+UM/dQ9IfbC8OnAq8mbJa4ARJt9n+AK+knO8C3k/JMHAJsCZwOLCA7fUpWQcWoASVgyiLSL1J0kzbCwF31eu/kZJjbcl6r89Kumt2/+1ieBik9/CNwIbAYsCn67lzA0cBG9TzfyTpx7bnAo4HPkjJljGNMnL0l7YPBbakvG9/B3wO+CTlPXyW7ReBdwO/Ag6o298i6cu1LLsBa0nax/ZOlPls89byfb5pSYcRJ30iI8MCtm+pPxdQFpz6uKQ1KX8kR9cJmI0+BVxev/2tDtxiewnKJM0P13OnUNYs6cmC9dzPUwIHlD/cP0paDfg6JQUNlD+cvevx76OkpQGg5kc7FDi3rpF+bsO+Z4BbKMEKYIta5mmUYZhfkDS+Xv+Elv6lYrgaivfwPJLWoSydfFjd9mngGUlrA2sDn62LxH2Ckth1HCWl0rsbrnO8pLUliRJItpD0y3rvHev7+sWG488DPt7wfDvgZ7bfXh+/t76mGcCOff6rjQCpiYwMr2oKqGuKf6umfplJyTW2NPBwwzk3AafWYy+UdEutMYwDrrcN5dtQb+uvnwMg6Vrbi9SU++tTvoEh6Te232B7EeB64BjbZwHnS7q/Xr8V51L+sK6mzAc6oX5LfQ/wi4brzNfqBWNYGor38Pn191RKgIAyv2w1291NrYtSVgldH/hFzQb+sO2rG66zoe2vUCYzLw7cAVzc2wuV9Jjtu22vB/wVWJXyN7I3ZXmLm2rZFwAe7e06I0WCyMi0I6WZZ7ykabbvAeZvPKB++L8f2Bw4rS6r+xRwpaRWlo9tnkDU64QiSUfZvpSSpv962xtTvmm2YjLlw2Rxyh/Yb4CFgKfnwDb0eMVgvIf/U3/P4JXPujGUGu7ljQfa3qynC9ien1ILXkvSfbYnNpezFz8DtqU0z15Ql7geA5wu6WstnD9ipDlrZFoUeLT+8W1ID9k1ba8IPCLpZOAUSt/E74H32l65HrOQ7bf2co/t6jHrU6r/zwDXUavftjcAHpf0rO23SLpd0nco3x5XbbrWc8DrerpJbcvuXsb3EkkzJD0L/MP2NvVeY+q68THnGIz3cE8uB/aqtRtsv7X2xV0PfNL2XLaXpvSZwCsB4/FaQ9664Vq9vq+BCyhLWuxACSgAVwFb216q3nvx+hpHtASRkeksYC3bt1PyiPXU4bwBcKvtP1ICwnGSHqOMKDnH9m2UZoDmD/xu/67nnkRpRwaYCIyv5x7FK/nN9nVxG6VD8ldN17oaGFfbw7fr4V7nAjvV3912BD5t+1ZK80HzmjMxsg3Ge7gnpwB/Am52aVP6MaWWch5lWYo/AT8FbqZ8eXqaktzVlAB0U8O1TgNOqu/rBRpvIukp4E5gRUl/qNv+ROnPuaKW/UpgmVko+7CUtCfxGnVkywGS5si1NCJ6YnthSc/bfgPwB0oH+MP9nTfapU8kIqK4pA4gmRc4IgGkNamJRERE29InEhERbUsQiYiItiWIRERE2xJEIiKibQkiERHRtv8PuHXWNmbGsSwAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "dark"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_results([results_bayes_intr_threshold_tuned, results_neural], ['Bayesian best', 'Neural network'])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "We got $20.7\\%$ false negatives, and also $0.19\\%$ false positives."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "But an enhancement suggests itself: we could allow more epochs. Let's set a maximum of 1000 epochs (following [Google's guide](https://developers.google.com/machine-learning/guides/text-classification/step-4)), and early stopping monitoring the convergence of false negatives (false positives are likely to converge faster, as we set a close-to-1 threshold):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/1000\n",
      "146/146 [==============================] - 1s 7ms/step - loss: 0.1721 - false_positives: 4.0000 - false_negatives: 910.0000\n",
      "Epoch 2/1000\n",
      "146/146 [==============================] - 1s 7ms/step - loss: 0.1570 - false_positives: 4.0000 - false_negatives: 844.0000\n",
      "Epoch 3/1000\n",
      "146/146 [==============================] - 1s 6ms/step - loss: 0.1431 - false_positives: 4.0000 - false_negatives: 781.0000\n",
      "Epoch 4/1000\n",
      "146/146 [==============================] - 1s 6ms/step - loss: 0.1331 - false_positives: 4.0000 - false_negatives: 755.0000\n",
      "Epoch 5/1000\n",
      "146/146 [==============================] - 1s 6ms/step - loss: 0.1221 - false_positives: 4.0000 - false_negatives: 692.0000\n",
      "Epoch 6/1000\n",
      "146/146 [==============================] - 1s 6ms/step - loss: 0.1136 - false_positives: 4.0000 - false_negatives: 657.0000\n",
      "Epoch 7/1000\n",
      "146/146 [==============================] - 1s 6ms/step - loss: 0.1056 - false_positives: 4.0000 - false_negatives: 627.0000\n",
      "Epoch 8/1000\n",
      "146/146 [==============================] - 1s 5ms/step - loss: 0.0986 - false_positives: 3.0000 - false_negatives: 590.0000\n",
      "Epoch 9/1000\n",
      "146/146 [==============================] - 1s 5ms/step - loss: 0.0918 - false_positives: 3.0000 - false_negatives: 558.0000\n",
      "Epoch 10/1000\n",
      "146/146 [==============================] - 1s 5ms/step - loss: 0.0864 - false_positives: 3.0000 - false_negatives: 529.0000\n",
      "Epoch 11/1000\n",
      "146/146 [==============================] - 1s 6ms/step - loss: 0.0817 - false_positives: 2.0000 - false_negatives: 485.0000\n",
      "Epoch 12/1000\n",
      "146/146 [==============================] - 1s 6ms/step - loss: 0.0763 - false_positives: 2.0000 - false_negatives: 467.0000\n",
      "Epoch 13/1000\n",
      "146/146 [==============================] - 1s 6ms/step - loss: 0.0715 - false_positives: 2.0000 - false_negatives: 427.0000\n",
      "Epoch 14/1000\n",
      "146/146 [==============================] - 1s 7ms/step - loss: 0.0677 - false_positives: 2.0000 - false_negatives: 423.0000\n",
      "Epoch 15/1000\n",
      "146/146 [==============================] - 1s 6ms/step - loss: 0.0642 - false_positives: 2.0000 - false_negatives: 395.0000\n",
      "Epoch 16/1000\n",
      "146/146 [==============================] - 1s 5ms/step - loss: 0.0609 - false_positives: 2.0000 - false_negatives: 365.0000\n",
      "Epoch 17/1000\n",
      "146/146 [==============================] - 1s 5ms/step - loss: 0.0582 - false_positives: 2.0000 - false_negatives: 360.0000\n",
      "Epoch 18/1000\n",
      "146/146 [==============================] - 1s 5ms/step - loss: 0.0546 - false_positives: 1.0000 - false_negatives: 341.0000\n",
      "Epoch 19/1000\n",
      "146/146 [==============================] - 1s 6ms/step - loss: 0.0517 - false_positives: 1.0000 - false_negatives: 309.0000\n",
      "Epoch 20/1000\n",
      "146/146 [==============================] - 1s 6ms/step - loss: 0.0499 - false_positives: 1.0000 - false_negatives: 313.0000\n",
      "Epoch 21/1000\n",
      "146/146 [==============================] - 1s 5ms/step - loss: 0.0477 - false_positives: 1.0000 - false_negatives: 291.0000\n",
      "Epoch 22/1000\n",
      "146/146 [==============================] - 1s 5ms/step - loss: 0.0453 - false_positives: 1.0000 - false_negatives: 283.0000\n",
      "Epoch 23/1000\n",
      "146/146 [==============================] - 1s 6ms/step - loss: 0.0429 - false_positives: 1.0000 - false_negatives: 266.0000\n",
      "Epoch 24/1000\n",
      "146/146 [==============================] - 1s 5ms/step - loss: 0.0408 - false_positives: 0.0000e+00 - false_negatives: 254.0000\n",
      "Epoch 25/1000\n",
      "146/146 [==============================] - 1s 6ms/step - loss: 0.0393 - false_positives: 0.0000e+00 - false_negatives: 244.0000\n",
      "Epoch 26/1000\n",
      "146/146 [==============================] - 1s 5ms/step - loss: 0.0383 - false_positives: 1.0000 - false_negatives: 250.0000\n",
      "Epoch 27/1000\n",
      "146/146 [==============================] - 1s 5ms/step - loss: 0.0364 - false_positives: 0.0000e+00 - false_negatives: 238.0000\n",
      "Epoch 28/1000\n",
      "146/146 [==============================] - 1s 5ms/step - loss: 0.0350 - false_positives: 0.0000e+00 - false_negatives: 227.0000\n",
      "Epoch 29/1000\n",
      "146/146 [==============================] - 1s 5ms/step - loss: 0.0330 - false_positives: 0.0000e+00 - false_negatives: 214.0000\n",
      "Epoch 30/1000\n",
      "146/146 [==============================] - 1s 5ms/step - loss: 0.0324 - false_positives: 0.0000e+00 - false_negatives: 205.0000\n",
      "Epoch 31/1000\n",
      "146/146 [==============================] - 1s 5ms/step - loss: 0.0311 - false_positives: 0.0000e+00 - false_negatives: 209.0000\n",
      "Epoch 32/1000\n",
      "146/146 [==============================] - 1s 5ms/step - loss: 0.0295 - false_positives: 0.0000e+00 - false_negatives: 198.0000\n",
      "Epoch 33/1000\n",
      "146/146 [==============================] - 1s 5ms/step - loss: 0.0288 - false_positives: 0.0000e+00 - false_negatives: 188.0000\n",
      "Epoch 34/1000\n",
      "146/146 [==============================] - 1s 5ms/step - loss: 0.0277 - false_positives: 0.0000e+00 - false_negatives: 177.0000\n",
      "Epoch 35/1000\n",
      "146/146 [==============================] - 1s 5ms/step - loss: 0.0265 - false_positives: 0.0000e+00 - false_negatives: 179.0000\n",
      "Epoch 36/1000\n",
      "146/146 [==============================] - 1s 5ms/step - loss: 0.0260 - false_positives: 0.0000e+00 - false_negatives: 179.0000\n",
      "CPU times: user 39.2 s, sys: 2.38 s, total: 41.6 s\n",
      "Wall time: 29.6 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7f2fb82b6730>"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "fit_model(epochs=1000, callbacks=[tf.keras.callbacks.EarlyStopping(monitor='false_negatives', patience=2)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "17/17 [==============================] - 0s 4ms/step - loss: 0.0604 - false_positives: 1.0000 - false_negatives: 22.0000\n",
      "{'false_positives': 1.0, 'false_negatives': 22.0}\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZEAAAEICAYAAACeSMncAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAqaklEQVR4nO3dd7xdVZ3+8c8FAhIREEEHQhVQZB5qaDMgxYJUGUeqFCkKIogoyIgEycBQnEGEGaUEVKqKNA2IAoPwE5GWhPYAjiCG3ksUCZD2+2OtC4fDvTcnJ7fkJM/79crrnr3P3vusfXPu+Z7VvqtrxowZREREtGO+oS5ARER0rgSRiIhoW4JIRES0LUEkIiLaliASERFtSxCJiIi2JYhERETbFhjqAkS0w/ZE4APAtIbd50o6eGhKNGeyPRpYRdIeQ12WmDsliEQn217S/87sINsLSJratG9+SdN6O6eHa8zS8bPCdhfQJWn6QFw/YiB1ZcZ6dKJaE/lCT0HE9t7AF4Hbgb2AM4BlgcnACsBmwA7AE/W5tevjIyWNrdc4t/n45teyfSNwC/BxYDXgBmAfSS/W5zcCTgFWBx4BvirpxoZzbwY2B9YF1gAWAk4FRgJTgNMknWB7PuCIek+LA9cDX5L0ou0Vgb8AewPHAcOB70k63vZWwFigC3gd+LOktWzvU6+3LPAc8B1JZzXc1xHA14AZwLeBs4FVJT1keyHgeGDnWt4rgK9Jmmx7SeBcYBNgOnAfsFmC49wtfSIxt9oQeJjS5HV83fe5+vg9wG3AlcC1wPuBrwAX2f5wwzUaj/99L6+zF7AvsDQwFfhvANsjgF8B/wEsARwOXGZ7qYZz9wT2r9d/Bvhf4DfAMsAqlGBBLdu/UILZMsBLwA+ayrEJ8GFKQPu27Y9I+g1wAnCxpEUkrVWPfRbYDlgU2Af4nu11a7m3Ar4OfKKWYfOm1zkJ+BAl8K4CjKAEGoDDgMeBpSi/929RAlHMxdKcFZ3sF7Ybm6m+Iens+vhJSf9TH0+1DfBLSTcD2F4bWAQ4qX5T/q3tq4DdgNH1vDePB17rpQwXSHK95tHAXbY/D+wBXC3p6nrcdbbHAdsA59V950q6r567HfC0pO82vN5t9fGXgIMlPV6PHQ08anvPhnL8u6TJwN227wbWAh7oqcCSftWw+f9sXwt8FJhAqWH8uKFco4Hd6+MuStBbs6G2dQLwE+BISu1paWAFSQ8BN/XyO4u5SIJIdLJ/6aNP5LGZ7FsGeKypqeURyjfrvq7R1zUfAYYBS1KawXayvX3D88MoTV49nbsc8OdeXmMF4ArbjWWdRvm23+3phsevUgJkj2xvDRxDqVHMR2kCu7c+vQwwrpcyLlWPHV+DMpSmsvnr4/+iBOBr6/NjJJ3UWzli7pAgEnOrnppRGvc9CSxne76GQLI88KeZXKPZcg2Pl6d8G3+e8uF7gaQvtljGx4BdeznuMWDfhlrRm2qfSF/edg+1T+MySjPcLyVNsf0LSjAAeIrSV9Kt8f6ep/QT/aOkJ5pfSNLfKE1ah9kWpXZ3h6Trm4+NuUf6RGJedRvlG/sRtofZ3hzYHvjZLF5nD9ur2x4OHAtcWkdxXQhsb/tTtue3/S7bm9tetpfrXAUsbftQ2wvZfo/tDetzZwLH214BwPZStndosXzPACvWznmABSkd4s9Rmvm2BrZsOP7nwD62P1Lv6ejuJ2qwPZvSh/L+WpYRtj9VH29ne5Xa7DWJUltKp/pcLkEkOtmVtl9p+HdFqydKeoMSNLamfMM+HdhL0h9nsQwXUEYkPQ28CzikXv8xygiwb1E+sB8DvkEvf3P1W/wna5meBh4EtqhPn0YZZXWt7b8Bt1IGDrTikvrzBdsT6uscQgkWL1EGD4xtKMevKYMDbgAeqq8FZXQXwL9177f9V8pggO7BCKvW7Vcoo9ZOl9TYfBdzoQzxjWhTHaZ7oaRzhrosA8X2RwADCzXPtYmA9IlERBPbnwGupnSifwe4MgEkepPmrIhodgBlLsmfKf0aBw5tcWJOluasiIhoW5qzIqIltn9Emen+rCTVfUsAFwMrAhOBnSW91MO5nwdG1c3/kHRe3b8g8H3KzPjpwFGSLrP9FUqN6FHKfKA3bG8CfFbS1wbsJmOWzXM1EdvPUSaFRcSsWYTyQb8SJS8WlDklUykjyv6BMvGweQ7J/JT8YffX7e7H0yiTG6HM24HyxXYqJRfZHykz4F+lDBlelZLKZkASYcZMrSBpqead82JN5BFJ6w11ISI6UZ3ceFX335Dt/wM2l/SU7aWBG5v/vmzvVo85oG6fVY/7qe3HgNUk/b3pnNuATSkz639HmS3/PkmnDugNRq9q2p53mBeDSET0nw9Ieqo+fpq3p2LpNoK3p095HBhhe/G6fVyd7PlnSo6wZyhNXLdSajw3A78EPtXvpY/ZltFZEdEvJM1g1rL2LkBpDvuDpHUpExRPrte6QNI6dTGtr1EmQG5t+1Lb32uYgR9DLP8RETE7nqnNWNSfz/ZwzBO8PQfXsnXfC5T+jsvr/ksoa6u8yfYywAaSfkHJy7UL8DIl5X3MAdKcBYwfP/79lG9Aq5HAGkNrOqVD+fCRI0f29IE8pxkLfJ6yzsjnKc1Oza4BTrD93rq9JWUBsBm2r6SMzPotJTDc33Tucby1XsnClJrOdMpEyJgDJIgUJ1NyBe03cuTIKUNdmJh3jR8/fhhlsaqTKZl25xi2f0r5wF/S9uOUTu+TgJ/b3o8y6nHneux6lNUXv1BXYDwOuKNe6tju9UgoubgusH0qJcfYPg2vtw6ApAl1108oKesfA/5zoO4zZs28OMR3XPPokfHjx98ObJwAEnOCGkhuHjly5AZDXZaIbj19dkKabrrNlwASc4r6XszfZnSEAWvO6u/ZrXUxnV9SOuVOl3R6PXYMcGZDlTciIgbJQPaJnEsZ631+w75vAtdLOsn2N+v2vzWeVAPNMcB6lE608bbHUtaA/j1wAmXc+Om21wLm7+8AsuI3f9WvbXwTT9q2a2bH2J5Gae/toszIPVjSH/qzHL287jnAKZKaOzRn9Tp7A+tJOrgfrnOtpCdndmxEDL0BCyKSftfD0p07UDrmAM4DbqQpiFAmFF3X3fFm+zpgK8qwvuGUdaq7P5SPA77UvyUfMpMlrQ1QV4o7EdhsoF9U0hcG+jVm0d6U9SsSRAZQf39R6iStfKmL1g326Ky2Z7dSxpDvSZnF+l+2Pw1MaOUbq+39gf3r5pJtln0wLUpZdQ7bi1Ca8d5LCaCjJP3S9rHAi91pIGwfT2k6PM32NyijZBYCrpB0jO13U1azW5aSy+g4SRfXhZUOlzTO9hnA+pShlJdKOqZeeyIl6G9fy7BTLysALlevN4KyWNO/1/P3oKymtyBlWdov1+N/yFs1zh9R/t/XAy6yPRn4J0mTZ+s3GREDasiG+NYx4i1/G6qL4nwOwPYwytjzHWyfAiwPnC9pbC/njgHG1HN7zP8yB1jY9l2UJVaXBj5W978GfEbSX20vSVmWdCzlQ/dy4NQ6e3dXYAPbW1IS1W1AqbGNtb0pJffQk5K2BbC9WA9lOKoOx5wfuN72mpLuqc89L2ld218GDgd6qsFsAIgygewO278C/k6ZILaxpCm2Twd2p6SzGNHQX7a4pJdtH0wNau38EiNicA32CJDZmd3a6MuUvpaNKNk9d6HMZu1kkyWtLWk1SvPd+ba7KIHgBNv3UNavHkGp0U2krJu9DmXy1p2SXqiPtwTuBCZQJlCuSulv+aTt79j+qKRJPZRhZ9sT6rn/SMm22q17VvF4ysCInlwn6YVae7gc2IQygWwkJajcVbc/SMnG+kHb/2N7K+Cvs/C7iog5xGDXRNqe3dr9ZN2/HaXvZHvK7NUZlCaYuYKkW2qtYylgm/pzZP0mP5FSWwE4h9KH8A+UmgmUoHOipLOar2t73Xq9/7B9vaRjG55biVLDWF/SS7bPbXgdgNfrz2n0/r5prlnOqOU5T9KRzQfXgRGfovRr7Qzs28t1I2IONWA1kTq79Rbgw7YfrzNaT6J8G34Q+ETdxvZ6dZQQtUO9e3brHbx9diuUFAjHS5pOCTgfpXzLvmCg7mWw2V6N0m/xArAYpa9jiu0tgBUaDr2CUmtZn/K7oP7ct/alYHuE7ffXHESvSroQ+C+achRR+mH+Dkyy/QFg6zaK/knbS9heGPgXyii664Edbb+/lmcJ2yvUIDmfpMsow7m7y/M34D1tvHZEDIGBHJ21Wy9PvSNxWm3//kLD9o9465t187Ffa3j8GqWm0q+GaPRGd58IlG/vn5c0zfZFwJW27wXGUfIqAVBXe7sBeFnStLrvWtsfAW6xDfAKsAewCmVAwnRgCk3rZku62/ad9fqPUQLArLoduIzSBHlhd7+G7VHAtbXvZgpwEDAZ+HFDNtbumsq5wJnpWI/oDEl7AowfP37cyJEjO26hqvoBPIEyWurBoS5P9J+Bfk9miG/MqqQ9mcvYXh14iDJ5MwEkIoZEsvh2qDrD/INDXY6ImLelJhIREW1LEImIiLYliERERNsSRCIiom3pWO/J6MX6d/jj6EmtpIKfQUnJfljdPhxYRNLofi3LO1/3RvohV5XttYFlJF3dH+VquO7mlPJt18/XnUhJXf98f163jXLcSHKFRQdLTWTO8Trwr3Umd7+x3dUwoW8grU1JqdJvbOdLTsQcLn+kc46plEzDXwOOanzC9lLAmZRsxQCHSrrZ9mjgFUkn1+NMySsGJf3JbZTkh9vURcDekea9N72lf68p5f+Hkq13GDAa+DVwLGXW/SaUtVBGUVLSTAKeB74m6Xzb51NS1NwEnEFJ/T4V+LqkG+qiVP8KLEJJ/fJmOW2vX39HO0r6c8P++SkpdDanpL//gaSzai1mdH19UZJH7iGpu6b5FdvN97cBcBolb9hkYB9J/1fL9WnKmjYrU1LsH1FffyvKYmnzU7Idf7yn31NN4b8w8GNgLUp2gLkm51vMm1ITmbP8ANi9hzTtpwHfk7Q+8FlK4sWZWZWyjPA/SnqEkuZ9PWBNYDPba7ZwjeclrUv5sD+87jsK+K2kDYAtKHm4hlFyml1cMxFfTEmbsjElG/DDlIAC8E/AHyipT2ZIWgPYDTjPdnfCx3UpgeLNRbls/zMlkO7QGECq/YBJ9fezPvDFmlASYB3gUEpG4g/WMvV1f38EPippnXpPJzQcvzYlY/QawC62l6sB/mzgs5LWAnbq7fdUA8uBlBxmH6EEyJFEdLAEkTmIpL9SUtwf0vTUJ4Dv19xaY4FFuxMs9uERSbc2bPeV5r03PaV/3xL4Zi3LjZRv7Ms3n0ipaWxa/50BrGF7BPCSpL9T0sRfCFAXuHoE+FA9982VLauPUGog20t6tIfX2hLYq5bpNuB9lCAKcLukx2vCzrt4exr7nu5vMeCSWqv7HuV31e16SZNqzrb7KckwNwJ+J+kv9V66y93b72nThvu+B7iHiA6W5qw5z6mUfFg/btg3H7BR/fB6k+2pvP2LQGPq9r83HDezNO+96Sn9exflW/f/NZVlw6Zzf0epbSxP+Vb+GWBHSnCZmb83bT9Vy7sOPS+b2wV8RdI1jTtrc9brDbua09j3dH/HATdI+ozL8s439nB8T9fqqUw9/Z76OCWi86QmMoep32R/Tmmi6XYt8JXujToSCmAiNYV6XStkJXrWH2neu11D6Uvoqq+7Tt3/thTukh6jLEW8qqSHgd9TAtnv6iE3UVY4xPaHKMHmbR+4DV4GtgVOrIGhpzId6LLiJbY/VJuO2rEYby2CtncLx98KbNrdfGZ7iYYy9fR7+h1vrdApSvNiRMdKTaQnLQzJHWDfBQ5u2D4E+EFd3XABygfRlyhp1/eyfR+lGedPPV2sn9K8dzuOUlu6p476+gulM/8G3mq+ObH2i9xG6WyGEjROpAQTgNOBM2qK+6nA3pJe7+2buqRnbG8H/Nr2vpJua3j6HEpz1IT6of0cZT2TdvwnpX9mFPCrmR0s6Tnb+wOX19/Hs8An6f33dAYlBf4DwAOUprSIjpVU8HRuKviYeyUV/MBJKvj2JBV8RET0uwSRiIhoW4JIMX38+PHDhroQEQD1vTh9qMsR0YoEkeKPwJ4JJDHU6ntwT8p7MmKOl9FZxeHAycCXxo8fn8AaQ2k6JYAcPrMDI+YECSLAyJEjnwX2GupyRER0mnzrjoiItiWIRERE2xJEIiKibQkiERHRtgSRiIhoW4JIRES0LUEkIiLaliASERFtSxCJiIi2JYhERETbhiTtie2vAV8AZgD3Avs0rh9ueyHgfGAk8AKwi6SJtjemrAz3BrCbpAdtL05ZTnYrScl8GhExiAa9JmJ7BGW51/UkibJ86q5Nh+0HvCRpFeB7wHfq/sOAbYBDKcvDAowCTkgAiYgYfEPVnLUAsLDtBYDhwJNNz+8AnFcfXwp8vK6dPaUePxyYYntlYDlJNw5KqSMi4m0GvTlL0hO2TwYeBSYD10q6tumwEcBj9fipticB7wNOpDRzTaasuXAypSbSJ9v7A/vXzSX74z4iImIIgojt91JqGisBLwOX2N5D0oUzO1fSXcBG9TqbAk8BXbYvptRSDpP0TA/njQHG1PPG9c+dRETEUDRnfQL4i6TnJE0BLgf+uemYJ4DlAGqT12KUDnbqvi5KDeQ44BjgCOBsSl9LREQMkqEIIo8CG9keXoPBx4EHmo4ZC3y+Pt4R+K2kGQ3P7wVcLelFSv/I9Ppv+ICWPCIi3mYo+kRus30pMAGYCtwJjLF9LDBO0ljgh8AFth8CXqRh9Jbt4cDewJZ11ynA1ZRhv58brPuIiIghmici6RhKM1Sjbzc8/xqwUy/nvgps0bB9E7DGABQzIiJmIjPWIyKibQkiERHRtgSRiIhoW4JIRES0LUEkIiLaliASERFtSxCJiIi2JYhERETbEkQiIqJtCSIREdG2BJGIiGhbgkhERLQtQSQiItqWIBIREW1LEImIiLbNNIjY3sn2e+rjUbYvt73uwBctIiLmdK3URI6W9Dfbm1DWR/8hcMbAFisiIjpBK0FkWv25LTBG0q+ABQeuSBER0SlaCSJP2D4L2AW42vZCLZ4XERFzuVaCwc7ANcCnJL0MLAF8YyALFRERnWGB3p6wvUTD5o0N+14Hxg1ssSIiohP0GkSA8cAMoKuH52YAHxyQEkVERMfoNYhIWmkwCxIREZ2nr+as1ST9sbc5IZImDFyxIiKiE/TVnPV1YH/guz08NwP42ICUKCIiOkZfzVn7159bDF5xIiKik/RVE3mTbQGrA+/q3ifp/IEqVEREdIaZBhHbxwCbU4LI1cDWwO+BBJGIiHlcK5MNdwQ+DjwtaR9gLWCxAS1VRER0hFaCyGRJ04GpthcFngWWG9hiRUREJ2ilT2Sc7cWBsykTEF8BbhnIQkVERGeYaRCR9OX68EzbvwEWlXTPwBYrIiI6Qaujs9YEVuw+3vYqki5v90VrzeYcQJQ5J/tKuqXh+S7gNGAb4FVgb0kTbH8Y+AkwDDhA0i22FwB+A3xa0qvtlikiImZdK6OzfgSsCdwHTK+7ZwBtBxFKgPiNpB1tLwgMb3p+a2DV+m9DyiJYGwIHAF8FJtZrfBY4ELgwASQiYvC1UhPZSNLq/fWCthcDNgX2BpD0BvBG02E7AOdLmgHcantx20sDUygBZzgwpdZotge26q/yRURE61oJIrfYXl3S/f30misBzwE/tr0WpbP+q5L+3nDMCOCxhu3H674fUOanLESplRwNnFBHj/XK9v6UFC4AS/bHTURERGtB5HxKIHmaspZIFzBD0pqz8ZrrAl+RdJvt04BvUgJCnyQ9Spn4iO1VgGWBB2xfQFmy92hJf+rhvDHAmHpe1kKJiOgnrQSRHwJ7AvfyVp/I7HgceFzSbXX7UkoQafQEb5+Lsmzd1+h4YBRwCKWTfiJwArB7P5QxIiJa0EoQeU7S2P56QUlP237M9ocl/R9lNnxzU9lY4GDbP6N0qE+S9FT3k7Y3A56U9KDt4ZTgNp13dtBHRMQAaiWI3Gn7J8CVlOYsAGZniC/wFeCiOjLrYWAf21+q1z2TkqNrG+AhyhDffbpPrMN/RwG71F1jgIvqvRw4G2WKiIhZ1EoQWZgSPLZs2DdbQ3wl3QWs17T7zIbnZwAH9XLuDOCTDdsPUPpYIiJikLUyY32fmR0TERHzppkmYLT9IdvX23bdXtP2qIEvWkREzOlayeJ7NnAkZaIfNW/WrgNZqIiI6AytBJHhkm5v2jd1IAoTERGdpZUg8rztlSmd6djeEXiq71MiImJe0MrorIMow2hXs/0E8BcyoS8iImhtdNbDwCdsvxuYT9LfBr5YERHRCVpaTwSgKUFiRERES30iERERPeo1iNjeqf5cafCKExERnaSvmsiR9edlg1GQiIjoPH31ibxg+1pgJdvvyOIr6dMDV6yIiOgEfQWRbSmJDS8Avjs4xYmIiE7SaxCpa5/favufJT1ne5G6/5VBK11ERMzRWhmd9QHbdwL3AffbHm9bA1yuiIjoAK0EkTHA1yWtIGl54LC6LyIi5nGtBJF3S7qhe0PSjcC7B6xEERHRMVqZsf6w7aMpHewAe1CWtI2IiHlcK0FkX+DfKcvhzgBuqvsiIuZZtucHxgFPSNqu6bmvA1+gLJvxHLCvpEfqc8sD5wDLUT5Tt5E00fZFwBrAVZK+VY8dBVjSLwbnrmZdKwkYXwIOGYSyRER0kq8CDwCL9vDcncB6kl61fSDwn8Au9bnzgeMlXVdHvU63vSYwWdKatq+zvRgwHNhQ0n8M/K20L7mzIiJmke1lKXPpzunpeUk3SHq1bt4KLFvPWx1YQNJ19bhX6nFTgIVtzwcMA6YBxwLHDOiN9IOWs/hGRMSbTgWOAN7TwrH7Ab+ujz8EvGz7cmAl4H+Bb0p6wPZzwARK//MqlKU3JvR3wftbgkhExCywvR3wrKTxtjefybF7AOsBm9VdCwAfBdYBHgUuBvYGfijp0IbzrgQOsH0UsBZwnaSz+/dO+sdMm7NsL2v7CtvP2X7W9mW1KhcRMS/aGPi07YnAz4CP2b6w+SDbnwCOAj4t6fW6+3HgLkkPS5oK/IKSXqrxvB2A8cAiwMqSdgZ2tD18gO5ntrTSJ/JjYCywNLAMcGXdFxExz5F0pKRlJa0I7Ar8VtIejcfYXgc4ixJAnm146g5gcdtL1e2PAfc3nDcMOJTSEb8wZfQWwPzAgv1/N7OvlSCylKQfS5pa/50LLDWzkyIi5iW2j7Xdnd38vyg1iUts39WdCV3SNOBw4Hrb9wJdQGMz1UHAebWz/R5geD1uvKSXB+lWZkkrfSIv1Ha9n9bt3YAXBq5IERGdoWbwuLE+/nbD/k/0cc51wJq9PHdqw+MZlM/bOVorNZF9gZ2Bp4GngB2BfQayUBER0RlamWz4CJAFqCIi4h16DSK2j5D0n7b/h7c6d94kKbPYIyLmcX3VRB6oP8cNRkEiIgbF6MXe8aV4njB6UtdAXLavlQ2vrA9flXRJ43O2dxqIwkRERGdpZXTWkcAlLeybJTPJgLkQJUnZSMpIsF1qlsuNgTOAN4DdJD1oe3Hg58BWkqbPTpkiImLW9NUnsjWwDTDC9n83PLUoJb3x7OorA+Z+wEuSVrG9K/AdSgbMw2qZVgS+VLdHASckgEREDL6+hvg+SakpvEaZgt/9byzwqdl50ZllwAR2AM6rjy8FPm67i5Lpcnj9N8X2ysBydax2REQMsr76RO4G7rb9E0lT+vl1T6XvDJgjgMdqOabangS8DziR0sw1GdgTOJlSE+mT7f2B/evmkrNT8IiIeEsrfSIr2j4RWB14V/dOSR9s5wVnJQNmM0l3ARvV62xKmfzYZftiSi3lMEnP9HDeGGBMPS+jzSIi+kmrCRjPoPSDbEGpCbwjY+UsaCUD5hOUpSOxvQCwGA2pVmrT1ijgOMqiLUdQ8s9k7kpExCBqJYgsLOl6oEvSI5JGU/oz2tJKBkxKv8vn6+Md6zGNY7v3Aq6W9CKlf2R6/TdHpkqOiJhbtdKc9XpdsvFB2wdTagmL9HdBbB8LjJM0FvghcIHth4AXKcGm+7jhlEVctqy7TgGupgz7/Vx/lysiInrXShD5KuUb/iGU5qOPUWoCs62PDJivAT1OaKwpkrdo2L4JWKM/yhMREbOmlQSMd9SHrwD71EmCuwK3DWTBIiJiztfXZMNFKQukjKD0UVxXtw+jLJZy0WAUMCIi5lx91UQuAF4CbgG+AHyLsgrXZ+pQ24iImMf1FUQ+KGkNANvnUOZkLF/7KyIiIvoc4vvmLPW6LvDjCSAREdGor5rIWrb/Wh93AQvX7S5ghqSeEidGRMQ8pK/cWfMPZkEiIqLztDJjPSIiokcJIhER0bYEkYiIaFuCSEREtC1BJCIi2pYgEhERbUsQiYiItiWIRERE2xJEIiKibQkiERHRtgSRiIhoW4JIRES0LUEkIiLaliASERFtSxCJiIi2JYhERETbEkQiIqJtCSIREdG2BJGIiGhbgkhERLQtQSQiItqWIBIREW1LEImIiLYliERERNsSRCIiom0LDPYL2l4OOB/4ADADGCPptKZjuoDTgG2AV4G9JU2w/WHgJ8Aw4ABJt9heAPgN8GlJrw7irUREzPOGoiYyFThM0urARsBBtldvOmZrYNX6b3/gjLr/AOCrlOByeN13IHBhAkhExOAb9JqIpKeAp+rjv9l+ABgB3N9w2A7A+ZJmALfaXtz20sAUYHj9N8X24sD2wFaDeAsREVENehBpZHtFYB3gtqanRgCPNWw/Xvf9gNIUthClVnI0cIKk6QNe2IiIeIchCyK2FwEuAw6V9NdWzpH0KLB5PX8VYFngAdsXAAsCR0v6Uw+vtT+lWQxgydkvfUREwBAFEdvDKAHkIkmX93DIE8ByDdvL1n2NjgdGAYcA5wATgROA3ZsvJmkMMKa+9rjZLH5ERFSD3rFeR179EHhA0im9HDYW2Mt2l+2NgEm1L6X7GpsBT0p6kNI/Mr3+Gz6wpY+IiEZDURPZGNgTuNf2XXXft4DlASSdCVxNGYH1EGWI7z7dJ9cgNArYpe4aA1xEuZcDB774ERHRbShGZ/0e6JrJMTOAg/p47pMN2w8A6/ZnGSMiojWZsR4REW1LEImIiLYliERERNsSRCIiom0JIhER0bYEkYiIaFuCSEREtC1BJCIi2pYgEhERbUsQiYiItiWIRERE2xJEIiKibQkiERHRtgSRiIhoW4JIRES0LUEkIiLaliASERFtSxCJiIi2JYhERETbEkQiIqJtCwx1AWLeYXsr4DRgfuAcSSc1Pb8pcCqwJrCrpEsbnvsOsG3dPE7SxXX/RcAawFWSvlX3jQIs6RcDekMRkZpIDA7b8wM/ALYGVgd2s71602GPAnsDP2k6d1tgXWBtYEPgcNuL2l4TmCxpTWB924vZXhrYMAEkYnAkiMRg2QB4SNLDkt4Afgbs0HiApImS7gGmN527OvA7SVMl/R24B9gKmAIsbHs+YBgwDTgWOGZgbyUiuiWIxGAZATzWsP143deKu4GtbA+3vSSwBbCcpAeA54AJwJXAKsB8kib0X7Ejoi/pExlkLfQLLAScD4wEXgB2kTSxPncksB/lG/chkq6xvRRwBbA4MKq7Gcf2L4EDJT05CLc1oCRda3t94A+UoHEL5XeApEO7j7N9JXCA7aOAtYDrJJ09+CWOmHekJjKIWuwX2A94SdIqwPeA79RzVwd2Bf6R0pRzer3ebsCZlOaiQ+ux2wN3zmEB5AlguYbtZeu+lkg6XtLakj4JdAF/anze9g7AeGARYGVJOwM72h4+2yWPiF6lJjK43uwXALDd3S9wf8MxOwCj6+NLge/b7qr7fybpdeAvth+q15sCDAcWAqbZXoASTLYf8LuZNXcAq9peiRI8dgU+18qJNVguLumF2pm+JnBtw/PDKPe8LbAqMKM+NT+wIPBqP91DRDRJTWRwtdIv8OYxkqYCk4D39XHuTygB5jrgBODLwAWS5qgPznovBwPXAA8AP5d0n+1jbX8awPb6th8HdgLOsn1fPX0YcJPt+4ExwB71et0OAs6r93wPMNz2vcB4SS8Pxv1FzKtSE+lwkiZR50/Yfi/wTeAzts8G3gt8V9ItQ1jEN0m6Gri6ad+3Gx7fQWnmaj7vNUrzX2/XPbXh8QxKE19EDILURAZXK/0Cbx5Tm6YWo3Swt3Lu0cDxlA/R3wOf562msYiIfpcgMrje7BewvSClX2Bs0zFjKR/+ADsCv63frscCu9peqPYrrArc3n2S7VWBZSXdSOkjmU7pG1h4AO8nIuZxCSKDqJV+AeCHwPtqx/nXKc1TSLoP+DmlE/43wEGSpjVc/njgqPr4p8CBlKB12sDeVUTMy7pmzJgx86PmIrbHSVpvqMvRcUYvNm+9UbqNntQ11EUYCCt+81fz5v8nMPFdLQ0KnPvM5nu5t8/OIelYb3fCne2NgTOAN4DdJD1oe3HKN/StJDWny4iIiAE06M1ZszPhDjgM2IYyJ+BLdd8o4IQEkIiIwTcUfSIzTcRXt8+rjy8FPl4n3HVPrBsOTLG9MiWH0o2DUvKIiHiboWjO6mnS3Ia9HSNpqu3uCXcnUpq5JgN7AidTaiJ9sr0/sH/d/LDtcbNzA/OkHW9+BHh+qIsx6ObS98pVe6wwfqjLMFTMzUuS93I7VuhpZ0dNNpR0F7ARvLmA0VNAl+2LKbWUwyQ908N5YygznaNNGZAQc4u8l/vXUDRnzc6EO+q+LkoN5DjK2hFHAGcDhwxYqSMi4h2GIojMzoS7bnsBV0t6kbcm1k2vjyMiYpAMenNW7ePonnA3P/Cj7gl3wDhJYykT7i6oE+5epAQaAGpq772BLeuuUyj5mN6gxayw0ZY0B8bcIu/lfjTPTTaMiIj+k7QnERHRtgSRiIhoW0cN8Z1X2Z4G3Nuw61+6113v4dhXJC0yKAXrg+1lgP+WtKPttYFl6noi1GSTqzenu4m5Vye+h2fG9t7Atd3LUNs+BzhF0v19njiXSRDpDJMlrT3UhZgV9Q9rx7q5NrAedUGqOniieURezN067j3cgr0BA08CSPrCkJZmiCSIdCDbiwC/pKxcOAwYJemXTccsDVwMLEr5fz5Q0k22twT+nbIm+5+BfSS90nTujcDdwGb13H0l3W57CeBHwAcp65bvL+ke25vxVsr5GcCmlAwDVwHrAscCC9vehJJ1YGFKUDmKspztSpKm23438Md6/eUpOdaWqq/1RUl/nN3fXcwZBuk9fBuwBbA4sF89d37gJGDzev4PJJ1lez7g+8DHKNkyplBGjl5q+9vA9pT37R+AA4DPUt7DF9meDPwT8Gvg8Lp/ZUnfqGXZG1hP0sG296DMZ1uwlu/LTUs6dJz0iXSGhW3fVf9dAbwGfEbSupQ/ku/WCZiNPgdcU7/9rQXcZXtJyiTNT9Rzx1HWLOnJ8HrulymBA8of7p2S1gS+RUlBA+UP56B6/EcpaWkAqPnRvg1cLGltSRc3PDcJuIsSrAC2q2WeQhmG+RVJI+v1T2/pNxVzqqF4Dy8gaQNKwtZj6r79gEmS1gfWB75YF3n7V2BFSlLYPSlBodv3Ja0vSZRAsp2kS+tr717f15Mbjr8M+EzD9i7Az2x/pD7euN7TNGD3Pn9rHSA1kc7wtqYA28OAE2rql+mUXGMfAJ5uOOcO4Ef12F9IuqvWGFYHbrYN5dtQb+uv/xRA0u9sL1pT7m9C+QaGpN/afp/tRYGbgVNsXwRcLunxev1WXEz5w7qBMh/o9Pot9Z+BSxqus1CrF4w50lC8hy+vP8dTAgSU+WVr2u5ual2MskroJsAlNRv407ZvaLjOFraPoExmXgK4D7iytxuV9Jzth21vBDwIrEb5GzmIsrzFHbXsCwPP9nadTpEg0pl2pzTzjJQ0xfZE4F2NB9QP/02BbYFzbZ8CvARcJ2m3Fl6jeQJRrxOKJJ1k+1eUNP032/4U5ZtmK8ZSPkyWoPyB/RZ4N/DyXNiGHm8ZjPfw6/XnNN76rOui1HCvaTzQ9jY9XcD2uyi14PUkPWZ7dHM5e/EzYGdK8+wVkmbUmtZ5ko5s4fyOkeaszrQY8Gz949uCHrJr2l4BeEbS2cA5lL6JW4GNba9Sj3m37Q/18hq71GM2oVT/JwE3UavftjcHnpf0V9srS7pX0nco3x5Xa7rW34D39PQitS27exnfqyRNk/RX4C+2d6qv1WV7rZZ+M9EpBuM93JNrgANr7QbbH6p9cTcDn7U9n+0PUPpM4K2A8XytIe/YcK1e39fAFZQlLXajBBSA64Edbb+/vvYS9R47WoJIZ7oIWM/2vZQ8Yj11OG8O3G37TkpAOE3Sc5QRJT+1fQ+lGaD5A7/ba/XcMyntyACjgZH13JN4K7/ZoS7uoXRI/rrpWjcAq9f28F16eK2LgT3qz267A/vZvpvSfNC85kx0tsF4D/fkHOB+YIJLm9JZlFrKZZRlKe4HLgQmUL48vUxJ7mpKALqj4VrnAmfW9/XCjS8i6SXgAWAFSbfXffdT+nOurWW/Dlh6Fso+R0rak3iHOrLlcElz5VoaET2xvYikV2y/D7id0gH+9MzOm9elTyQioriqDiBZEDguAaQ1qYlERETb0icSERFtSxCJiIi2JYhERETbEkQiIqJtCSIREdG2/w9fikUu6rzB/QAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "dark"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "results_neural_enhanced = evaluate_model()\n",
    "print(results_neural_enhanced)\n",
    "plot_results([results_bayes_intr_threshold_tuned, results_neural_enhanced], ['Bayesian best', 'Neural network enhanced'])"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "With the enhanced neural network, we achieved an impressively low $4.26\\%$ false negative rate! But there are still $0.19\\%$ false positives, that is, one single email. Let's have a look at that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "CODE"
    },
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/1 [==============================] - 0s 110ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 48ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 45ms/step\n",
      "1/1 [==============================] - 0s 98ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 49ms/step\n",
      "1/1 [==============================] - 0s 46ms/step\n",
      "1/1 [==============================] - 0s 46ms/step\n",
      "1/1 [==============================] - 0s 37ms/step\n",
      "1/1 [==============================] - 0s 39ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 38ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 36ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 47ms/step\n",
      "1/1 [==============================] - 0s 37ms/step\n",
      "1/1 [==============================] - 0s 44ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 40ms/step\n",
      "1/1 [==============================] - 0s 43ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 68ms/step\n",
      "1/1 [==============================] - 0s 37ms/step\n",
      "1/1 [==============================] - 0s 74ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 66ms/step\n",
      "1/1 [==============================] - 0s 62ms/step\n",
      "1/1 [==============================] - 0s 26ms/step\n",
      "1/1 [==============================] - 0s 29ms/step\n",
      "1/1 [==============================] - 0s 40ms/step\n",
      "1/1 [==============================] - 0s 75ms/step\n",
      "1/1 [==============================] - 0s 44ms/step\n",
      "1/1 [==============================] - 0s 28ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 26ms/step\n",
      "1/1 [==============================] - 0s 28ms/step\n",
      "1/1 [==============================] - 0s 45ms/step\n",
      "1/1 [==============================] - 0s 43ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 103ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 25ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 28ms/step\n",
      "1/1 [==============================] - 0s 27ms/step\n",
      "1/1 [==============================] - 0s 52ms/step\n",
      "1/1 [==============================] - 0s 87ms/step\n",
      "1/1 [==============================] - 0s 37ms/step\n",
      "1/1 [==============================] - 0s 47ms/step\n",
      "Subject: digitize your memories with compaq scanners\r\n",
      "tired of searching for your photos or important files ?\r\n",
      "quit looking . we have a solution . we \u000f ' re offering a compaq tested and\r\n",
      "approved agfa ( tm ) snapscan touch usb scanner for just $ 99 .\r\n",
      "or for sharper photo reproduction , an epson ( tm ) perfection 1200 u\r\n",
      "usb scanner is available for just $ 199 .\r\n",
      "picture having all your photos on your hard drive , scanned in and\r\n",
      "ready to e - mail to friends and family or to store in cd libraries .\r\n",
      "you ' ll also be able to unclutter that office , allowing you to store\r\n",
      "documents as easily as placing the original on the scanner bed\r\n",
      "and clicking .\r\n",
      "not only will these scanners simplify your life , but these great\r\n",
      "values come with easy - to - use software , are usb compatible and are ready\r\n",
      "to go a few moments out of the box .\r\n",
      "so stop your search . everything you need is just a click away .\r\n",
      "to order the agfa snapscan , go to\r\n",
      "http : / / greatideas . compaq . com / cgi - bin 6 / flo ? y = ecp 20 bdtnxori 0 crxv .\r\n",
      "to order the epson perfection , go to\r\n",
      "http : / / greatideas . compaq . com / cgi - bin 6 / flo ? y = ecp 20 bdtnxori 0 cryw .\r\n",
      "if you would rather not receive e - mails about future contests and\r\n",
      "promotions , service updates , product enhancements , cool activities or\r\n",
      "new products , please e - mail\r\n",
      "to remove your name from our mailing list .\r\n",
      "- attl . htm\n",
      "Label: 0\n",
      "Prediction: 0.9993051886558533\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 49ms/step\n",
      "1/1 [==============================] - 0s 73ms/step\n",
      "1/1 [==============================] - 0s 44ms/step\n",
      "1/1 [==============================] - 0s 27ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 64ms/step\n",
      "1/1 [==============================] - 0s 49ms/step\n",
      "1/1 [==============================] - 0s 88ms/step\n",
      "1/1 [==============================] - 0s 43ms/step\n",
      "1/1 [==============================] - 0s 28ms/step\n",
      "1/1 [==============================] - 0s 84ms/step\n",
      "1/1 [==============================] - 0s 36ms/step\n",
      "1/1 [==============================] - 0s 56ms/step\n",
      "1/1 [==============================] - 0s 38ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 44ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 55ms/step\n",
      "1/1 [==============================] - 0s 49ms/step\n",
      "1/1 [==============================] - 0s 36ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 42ms/step\n",
      "1/1 [==============================] - 0s 40ms/step\n",
      "1/1 [==============================] - 0s 36ms/step\n",
      "1/1 [==============================] - 0s 55ms/step\n",
      "1/1 [==============================] - 0s 43ms/step\n",
      "1/1 [==============================] - 0s 61ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 45ms/step\n",
      "1/1 [==============================] - 0s 58ms/step\n",
      "1/1 [==============================] - 0s 48ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 80ms/step\n",
      "1/1 [==============================] - 0s 83ms/step\n",
      "1/1 [==============================] - 0s 49ms/step\n",
      "1/1 [==============================] - 0s 56ms/step\n",
      "1/1 [==============================] - 0s 108ms/step\n",
      "1/1 [==============================] - 0s 54ms/step\n",
      "1/1 [==============================] - 0s 59ms/step\n",
      "1/1 [==============================] - 0s 50ms/step\n",
      "1/1 [==============================] - 0s 43ms/step\n",
      "1/1 [==============================] - 0s 54ms/step\n",
      "1/1 [==============================] - 0s 43ms/step\n",
      "1/1 [==============================] - 0s 37ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 66ms/step\n",
      "1/1 [==============================] - 0s 64ms/step\n",
      "1/1 [==============================] - 0s 28ms/step\n",
      "1/1 [==============================] - 0s 41ms/step\n",
      "1/1 [==============================] - 0s 65ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 29ms/step\n",
      "1/1 [==============================] - 0s 70ms/step\n",
      "1/1 [==============================] - 0s 37ms/step\n",
      "1/1 [==============================] - 0s 46ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 72ms/step\n",
      "1/1 [==============================] - 0s 75ms/step\n",
      "1/1 [==============================] - 0s 46ms/step\n",
      "1/1 [==============================] - 0s 76ms/step\n",
      "1/1 [==============================] - 0s 73ms/step\n",
      "1/1 [==============================] - 0s 44ms/step\n",
      "1/1 [==============================] - 0s 39ms/step\n",
      "1/1 [==============================] - 0s 41ms/step\n",
      "1/1 [==============================] - 0s 39ms/step\n",
      "1/1 [==============================] - 0s 37ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 96ms/step\n",
      "1/1 [==============================] - 0s 60ms/step\n",
      "1/1 [==============================] - 0s 29ms/step\n",
      "1/1 [==============================] - 0s 68ms/step\n",
      "1/1 [==============================] - 0s 39ms/step\n",
      "1/1 [==============================] - 0s 63ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 58ms/step\n",
      "1/1 [==============================] - 0s 27ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 48ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 40ms/step\n",
      "1/1 [==============================] - 0s 40ms/step\n",
      "1/1 [==============================] - 0s 40ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 73ms/step\n",
      "1/1 [==============================] - 0s 41ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 36ms/step\n",
      "1/1 [==============================] - 0s 38ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 44ms/step\n",
      "1/1 [==============================] - 0s 63ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 74ms/step\n",
      "1/1 [==============================] - 0s 50ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 37ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 72ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 59ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 36ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 47ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 28ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 47ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 36ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 41ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 26ms/step\n",
      "1/1 [==============================] - 0s 65ms/step\n",
      "1/1 [==============================] - 0s 37ms/step\n",
      "1/1 [==============================] - 0s 67ms/step\n",
      "1/1 [==============================] - 0s 27ms/step\n",
      "1/1 [==============================] - 0s 27ms/step\n",
      "1/1 [==============================] - 0s 63ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 49ms/step\n",
      "1/1 [==============================] - 0s 26ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 27ms/step\n",
      "1/1 [==============================] - 0s 66ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 27ms/step\n",
      "1/1 [==============================] - 0s 27ms/step\n",
      "1/1 [==============================] - 0s 29ms/step\n",
      "1/1 [==============================] - 0s 28ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 54ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 38ms/step\n",
      "1/1 [==============================] - 0s 37ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 58ms/step\n",
      "1/1 [==============================] - 0s 28ms/step\n",
      "1/1 [==============================] - 0s 29ms/step\n",
      "1/1 [==============================] - 0s 29ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 28ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 56ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 76ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 40ms/step\n",
      "1/1 [==============================] - 0s 71ms/step\n",
      "1/1 [==============================] - 0s 29ms/step\n",
      "1/1 [==============================] - 0s 26ms/step\n",
      "1/1 [==============================] - 0s 61ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 51ms/step\n",
      "1/1 [==============================] - 0s 61ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 26ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 28ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 29ms/step\n",
      "1/1 [==============================] - 0s 39ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 36ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 60ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 85ms/step\n",
      "1/1 [==============================] - 0s 27ms/step\n",
      "1/1 [==============================] - 0s 26ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 60ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 37ms/step\n",
      "1/1 [==============================] - 0s 60ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 36ms/step\n",
      "1/1 [==============================] - 0s 29ms/step\n",
      "1/1 [==============================] - 0s 28ms/step\n",
      "1/1 [==============================] - 0s 89ms/step\n",
      "1/1 [==============================] - 0s 39ms/step\n",
      "1/1 [==============================] - 0s 38ms/step\n",
      "1/1 [==============================] - 0s 60ms/step\n",
      "1/1 [==============================] - 0s 36ms/step\n",
      "1/1 [==============================] - 0s 54ms/step\n",
      "1/1 [==============================] - 0s 52ms/step\n",
      "1/1 [==============================] - 0s 38ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 36ms/step\n",
      "1/1 [==============================] - 0s 27ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 39ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 66ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 84ms/step\n",
      "1/1 [==============================] - 0s 43ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 61ms/step\n",
      "1/1 [==============================] - 0s 60ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 41ms/step\n",
      "1/1 [==============================] - 0s 44ms/step\n",
      "1/1 [==============================] - 0s 38ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 62ms/step\n",
      "1/1 [==============================] - 0s 38ms/step\n",
      "1/1 [==============================] - 0s 76ms/step\n",
      "1/1 [==============================] - 0s 26ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 61ms/step\n",
      "1/1 [==============================] - 0s 60ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 36ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 40ms/step\n",
      "1/1 [==============================] - 0s 27ms/step\n",
      "1/1 [==============================] - 0s 66ms/step\n",
      "1/1 [==============================] - 0s 36ms/step\n",
      "1/1 [==============================] - 0s 39ms/step\n",
      "1/1 [==============================] - 0s 71ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 64ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 37ms/step\n",
      "1/1 [==============================] - 0s 38ms/step\n",
      "1/1 [==============================] - 0s 36ms/step\n",
      "1/1 [==============================] - 0s 54ms/step\n",
      "1/1 [==============================] - 0s 28ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 62ms/step\n",
      "1/1 [==============================] - 0s 38ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 41ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 26ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 26ms/step\n",
      "1/1 [==============================] - 0s 29ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 27ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 42ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 37ms/step\n",
      "1/1 [==============================] - 0s 29ms/step\n",
      "1/1 [==============================] - 0s 28ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 73ms/step\n",
      "1/1 [==============================] - 0s 29ms/step\n",
      "1/1 [==============================] - 0s 29ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 28ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 61ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 66ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 39ms/step\n",
      "1/1 [==============================] - 0s 37ms/step\n",
      "1/1 [==============================] - 0s 28ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 66ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 27ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 61ms/step\n",
      "1/1 [==============================] - 0s 48ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 36ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 29ms/step\n",
      "1/1 [==============================] - 0s 43ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 37ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 36ms/step\n",
      "1/1 [==============================] - 0s 29ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 56ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 64ms/step\n",
      "1/1 [==============================] - 0s 73ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 70ms/step\n",
      "1/1 [==============================] - 0s 27ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 29ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 26ms/step\n",
      "1/1 [==============================] - 0s 72ms/step\n",
      "1/1 [==============================] - 0s 52ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 91ms/step\n",
      "1/1 [==============================] - 0s 29ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 61ms/step\n",
      "1/1 [==============================] - 0s 43ms/step\n",
      "1/1 [==============================] - 0s 60ms/step\n",
      "1/1 [==============================] - 0s 29ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 63ms/step\n",
      "1/1 [==============================] - 0s 27ms/step\n",
      "1/1 [==============================] - 0s 67ms/step\n",
      "1/1 [==============================] - 0s 39ms/step\n",
      "1/1 [==============================] - 0s 66ms/step\n",
      "1/1 [==============================] - 0s 28ms/step\n",
      "1/1 [==============================] - 0s 29ms/step\n",
      "1/1 [==============================] - 0s 28ms/step\n",
      "1/1 [==============================] - 0s 28ms/step\n",
      "1/1 [==============================] - 0s 32ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 63ms/step\n",
      "1/1 [==============================] - 0s 37ms/step\n",
      "1/1 [==============================] - 0s 37ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 66ms/step\n",
      "1/1 [==============================] - 0s 44ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 37ms/step\n",
      "1/1 [==============================] - 0s 39ms/step\n",
      "1/1 [==============================] - 0s 36ms/step\n",
      "1/1 [==============================] - 0s 38ms/step\n",
      "1/1 [==============================] - 0s 47ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 36ms/step\n",
      "1/1 [==============================] - 0s 36ms/step\n",
      "1/1 [==============================] - 0s 38ms/step\n",
      "1/1 [==============================] - 0s 51ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 57ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 35ms/step\n",
      "1/1 [==============================] - 0s 36ms/step\n",
      "1/1 [==============================] - 0s 37ms/step\n",
      "1/1 [==============================] - 0s 49ms/step\n",
      "1/1 [==============================] - 0s 38ms/step\n",
      "1/1 [==============================] - 0s 43ms/step\n",
      "1/1 [==============================] - 0s 36ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 33ms/step\n",
      "1/1 [==============================] - 0s 52ms/step\n",
      "1/1 [==============================] - 0s 36ms/step\n",
      "1/1 [==============================] - 0s 37ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 36ms/step\n",
      "1/1 [==============================] - 0s 36ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 40ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 59ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 60ms/step\n",
      "1/1 [==============================] - 0s 28ms/step\n",
      "1/1 [==============================] - 0s 28ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 31ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 30ms/step\n",
      "1/1 [==============================] - 0s 47ms/step\n",
      "1/1 [==============================] - 0s 69ms/step\n",
      "1/1 [==============================] - 0s 34ms/step\n",
      "1/1 [==============================] - 0s 71ms/step\n",
      "1/1 [==============================] - 0s 26ms/step\n",
      "1/1 [==============================] - 0s 67ms/step\n",
      "1/1 [==============================] - 0s 36ms/step\n",
      "1/1 [==============================] - 0s 63ms/step\n"
     ]
    }
   ],
   "source": [
    "for email, label in ds_test.unbatch():\n",
    "    email_str = email.numpy().decode(encoding)\n",
    "    label_int = label.numpy()\n",
    "    prediction_float = model.predict([email_str])[0][0]\n",
    "    if label_int == 0 and prediction_float > prob_threshold:\n",
    "        print(email_str)\n",
    "        print(f'Label: {label_int}')\n",
    "        print(f'Prediction: {prediction_float}')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "It's a Compaq advertisement, arguably spam. So the situation here seems more like AI has been smarter than human intelligence."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {
    "datalore": {
     "hide_input_from_viewers": false,
     "hide_output_from_viewers": false,
     "type": "MD"
    },
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "To sum up, Bayesian filters on the `Enron1` dataset proved to be better than a naive text classification neural network, but an enhanced neural network can exceed them. However, this comes at a significant computational cost: on an [AWS T2](https://aws.amazon.com/ec2/instance-types/t2/) medium instance with 2 CPUs and 4 GiB memory, even the naive neural network's training took about 10 seconds, the enhanced one's took about a half minute, whereas in the Bayesian filtering, calculating the word probabilities (the single computationally expensive part of the algorithm; comparable to training in neural networks) only required about 2 seconds. On a machine with an Intel Core i5-6440HQ processor and 24 GiB memory, similar times were measured."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This technical result provides an [application-driven argument](https://plato.stanford.edu/archives/sum2024/entries/epistemology-bayesian/#AlteFoun) \n",
    "for Bayesian epistemology: Bayesianism seems to strike the best balance between accuracy and simplicity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9eBiwQbud7dk"
   },
   "source": [
    "## Large language models\n",
    "\n",
    "Let's evaluate large language models (LLMs). We'll start with Google's Gemini. We will use Gemini 1.5 Flash, \n",
    "the default model in [Google AI Studio](https://aistudio.google.com) as of September 2024."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try one prompt first:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Prompt:\n",
    "\n",
    "> Is this a spam email? \"Subject: meter variances - ua 4 clean - up\\r\\ndaren / vance -\\r\\nthe two meters below are new and have unallocatable flow . . . . . . . i will need a\\r\\npurchase for each of them . please respond with a deal number , or further\\r\\nsuggestions for resolution so that i can clear this up as soon as possible .\\r\\nmary\\r\\nenron on 09 / 19 / 2000 08 : 44 am - - - - - - - - - - - - - - - - - - - - - - - - - - -\\r\\nkatherine benedict @ ect\\r\\n09 / 18 / 2000 04 : 53 pm\\r\\nto : mary poorman / na / enron @ enron , fred boas / hou / ect @ ect , aimee\\r\\nlannou / hou / ect @ ect\\r\\ncc :\\r\\nsubject : meter variances - ua 4 clean - up\\r\\nhi guys ,\\r\\nplease take a look at the following meters which are showing up with\\r\\nvariances on my ua 4 report . once the variance has been cleared , please send\\r\\nme an e - mail . please try to clear these within the next couple of days .\\r\\nthanks ,\\r\\nkathy benedict\\r\\nmeter number : 9851\\r\\nvariance : 11 , 204\\r\\nmeter type : daily swing\\r\\nproduction month : august 2000\\r\\nlogistics contact : none\\r\\nvolume mgmt contact : anita luong\\r\\nmeter number : 9852\\r\\nvariance 568\\r\\nmeter type : daily swing\\r\\nproduction month : august 2000\\r\\nlogistics contact : none\\r\\nvolume mgmt contact : anita luong\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Response:\n",
    "\n",
    "> This email is not spam. It's an internal communication from an Enron employee, likely related to their energy trading business.  \n",
    "> \n",
    "> Here's why:\n",
    "> \n",
    "> * **Context:** The email refers to \"meter variances,\" \"ua 4 report,\" and \"production month,\" indicating a business context related to energy production and billing.\n",
    "> * **Sender and Recipient:**  The email includes names like \"Mary Poorman,\" \"Fred Boas,\" and \"Aimee Lannou,\" all associated with Enron. \n",
    "> * **Specific Details:** The email includes detailed information about specific meters, variances, and contact persons. This level of detail is typical of internal business communications.\n",
    "> * **No Promotion or External Links:** There is no attempt to sell anything or direct the reader to external websites.\n",
    "> \n",
    "> **However, it's important to note:** \n",
    "> \n",
    "> * **Enron's History:**  Enron was a company involved in a massive financial scandal in the early 2000s. This email is dated 2000, so it could potentially be related to their unethical practices.\n",
    "> * **Privacy:** This email may contain sensitive company information, and it's not appropriate to share publicly.\n",
    "> \n",
    "> **If you found this email, you should be careful about sharing it or using the information it contains.** \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's add the system instruction\n",
    "\n",
    "> Answer in one term (spam or not spam)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Response:\n",
    "\n",
    "> not spam \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's try to evaluate Gemini on our full test dataset.\n",
    "\n",
    "For this, we'll make use of the Gemini API Python SDK:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "genai.configure(api_key=os.environ[\"GEMINI_API_KEY\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instantiate the Gemini model, with disabled safety settings (necessary for analyzing e.g. sexually explicit spam). \n",
    "Also, we set the temperature parameter to 0, in order to avoid more detailed responses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 208,
   "metadata": {
    "id": "fkBbj62heAmG"
   },
   "outputs": [],
   "source": [
    "model = genai.GenerativeModel(\n",
    "  model_name=\"gemini-1.5-flash\",\n",
    "  generation_config={\n",
    "    \"temperature\": 0,\n",
    "  },\n",
    "  safety_settings={\n",
    "      HarmCategory.HARM_CATEGORY_HATE_SPEECH: HarmBlockThreshold.BLOCK_NONE,\n",
    "      HarmCategory.HARM_CATEGORY_HARASSMENT: HarmBlockThreshold.BLOCK_NONE,\n",
    "      HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: HarmBlockThreshold.BLOCK_NONE,\n",
    "      HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT: HarmBlockThreshold.BLOCK_NONE,\n",
    "  },\n",
    "  system_instruction=\"Answer in one term (spam or not spam).\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 209,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 52
    },
    "id": "257zT8Z_fr-t",
    "outputId": "374f48dd-bcfe-4142-e19f-eb891287bb43"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "spam \n",
      "\n"
     ]
    }
   ],
   "source": [
    "response = model.generate_content(\"Is this a spam email? \\\"Subject: Viagra\\\"\")\n",
    "\n",
    "print(response.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next comes our evaluation code. The Gemini API quota is 2 requests per second and 240 requests per day, which unfortunately isn't enough for evaluating in one go, so we will go in chunks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {
    "id": "8RjiULbUhWwZ"
   },
   "outputs": [],
   "source": [
    "def evaluate_gemini(test_start, test_end):\n",
    "    def print_details(e, l, p):\n",
    "        print(e)\n",
    "        print(f'Label: {l}')\n",
    "        print(p)\n",
    "\n",
    "    for email, label in list(ds_test.unbatch())[test_start:test_end]:\n",
    "        email_str = email.numpy().decode(encoding)\n",
    "        label_int = label.numpy()\n",
    "        prediction = model.generate_content(f'Is this a spam email? \"{email_str}\"').text\n",
    "        if re.match(r'^spam\\s*$', prediction):\n",
    "          if label_int == 0:\n",
    "            print_details(email_str, label_int, prediction)\n",
    "        elif re.match(r'^not spam\\s*$', prediction):\n",
    "          if label_int == 1:\n",
    "            print_details(email_str, label_int, prediction)\n",
    "        else:\n",
    "          print_details(email_str, label_int, prediction)\n",
    "        time.sleep(.5)  # Gemini API quota is 2 requests per second"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 582
    },
    "id": "PKEEOG2WEWqC",
    "outputId": "f3fdf624-959b-4d69-a4b6-c07dbf91e0b8"
   },
   "outputs": [],
   "source": [
    "evaluate_gemini(0, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "Subject: re : noms / actual flow for 03 / 14\n",
    "we agree\n",
    "\" eileen ponton \" on 03 / 15 / 2001 09 : 47 : 26 am\n",
    "to : david avila / lsp / enserch / us @ tu , charlie stone / texas utilities @ tu , melissa\n",
    "jones / texas utilities @ tu , hpl . scheduling @ enron . com , liz . bellamy @ enron . com\n",
    "cc :\n",
    "subject : noms / actual flow for 03 / 14\n",
    "date nom flow - mcf flow - mmbtu\n",
    "3 / 14 23 , 750 23 , 112 23 , 736\n",
    "btu = 1 . 027\n",
    "Label: 0\n",
    "spam \n",
    "\n",
    "Subject: ces - duke en field services\n",
    "the purchase and sale is at felmont # 1 ( point sale ) . we will be terminating\n",
    "the agreement . feb should be the last month .\n",
    "Label: 0\n",
    "Spam \n",
    "\n",
    "Subject: hpl noms for april 27 , 2001\n",
    "( see attached file : hplno 427 . xls )\n",
    "- hplno 427 . xls\n",
    "Label: 0\n",
    "spam \n",
    "\n",
    "Subject: enron actuals for june 26 , 2000\n",
    "teco tap 20 . 000 / enron ; 5 . 833 / hpl iferc ; 94 . 167 / hpl gas daily\n",
    "ls hpl lsk ic 30 . 000 / enron\n",
    "Label: 0\n",
    "spam \n",
    "\n",
    "Subject: enron / hpl actuals for july 6 , 2000\n",
    "teco tap 10 . 000 / enron ; 90 . 000 / hpl iferc\n",
    "ls hpl lsk ic 30 . 000 / enron\n",
    "Label: 0\n",
    "spam \n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like Gemini on its own is not trained enough for this dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fine-tuning the [Gemini embeddings](https://ai.google.dev/gemini-api/tutorials/text_classifier_embeddings) \n",
    "promises a better, task-specific training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from google.api_core import retry\n",
    "\n",
    "@retry.Retry(timeout=300.0)\n",
    "def embed(text: str) -> list[float]:\n",
    "  embedding = genai.embed_content(model='models/embedding-001',\n",
    "                                  content=text[:1000],  # Prevent too long Gemini API requests.\n",
    "                                  task_type=\"classification\")\n",
    "  return embedding['embedding']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_embeddings_and_labels(ds, embed_fn=embed):\n",
    "    embeddings = []\n",
    "    labels = []\n",
    "    for email, label in ds.unbatch():\n",
    "        email_str = email.numpy().decode(encoding)\n",
    "        label_int = label.numpy()\n",
    "        embeddings.append(embed_fn(email_str))\n",
    "        labels.append(label_int)\n",
    "        print(f'{len(embeddings)} emails processed')\n",
    "    return np.stack(embeddings), np.array(labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time gemini_test, gemini_test_labels = create_embeddings_and_labels(ds_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_batch_train_gemini = 30  # Use about 1000 emails only, for a reasonable number of requests.\n",
    "\n",
    "ds_train_gemini = ds_train_neural.take(num_batch_train_gemini)\n",
    "ds_val_gemini = ds_train_neural.skip(num_batch_train_gemini).take(num_batch_train_gemini // 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time gemini_val, gemini_val_labels = create_embeddings_and_labels(ds_val_gemini)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time gemini_train, gemini_train_labels = create_embeddings_and_labels(ds_train_gemini)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = tf.keras.Input((768,))\n",
    "hidden = tf.keras.layers.Dense(768, activation='relu')(inputs)\n",
    "outputs = tf.keras.layers.Dense(1, activation='sigmoid')(hidden)\n",
    "gemini_embedding_model = tf.keras.Model(inputs, outputs)\n",
    "\n",
    "gemini_embedding_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gemini_embedding_model.compile(loss=tf.keras.losses.BinaryCrossentropy(),\n",
    "                               optimizer='adam',\n",
    "                               metrics=[tf.metrics.FalsePositives(thresholds=prob_threshold, name='gemini_false_positives'),\n",
    "                                        tf.metrics.FalseNegatives(thresholds=prob_threshold, name='gemini_false_negatives')])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gemini_embedding_model.fit(x=gemini_train,\n",
    "                           y=gemini_train_labels,\n",
    "                           validation_data=(\n",
    "                               gemini_val, \n",
    "                               gemini_val_labels),\n",
    "                           callbacks=[tf.keras.callbacks.EarlyStopping(\n",
    "                               monitor='val_gemini_false_negatives',\n",
    "                               mode='min', \n",
    "                               patience=5)],  # 5 has performed best.\n",
    "                           batch_size=32,\n",
    "                           epochs=20,)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "Epoch 1/20\n",
    "30/30 ━━━━━━━━━━━━━━━━━━━━ 2s 21ms/step - gemini_false_negatives: 141.4194 - gemini_false_positives: 0.0000e+00 - loss: 0.4598 - val_gemini_false_negatives: 12.0000 - val_gemini_false_positives: 0.0000e+00 - val_loss: 0.1497\n",
    "Epoch 2/20\n",
    "30/30 ━━━━━━━━━━━━━━━━━━━━ 1s 10ms/step - gemini_false_negatives: 59.0645 - gemini_false_positives: 0.0000e+00 - loss: 0.0979 - val_gemini_false_negatives: 4.0000 - val_gemini_false_positives: 0.0000e+00 - val_loss: 0.0933\n",
    "Epoch 3/20\n",
    "30/30 ━━━━━━━━━━━━━━━━━━━━ 1s 11ms/step - gemini_false_negatives: 24.2903 - gemini_false_positives: 1.6774 - loss: 0.0534 - val_gemini_false_negatives: 2.0000 - val_gemini_false_positives: 0.0000e+00 - val_loss: 0.0694\n",
    "Epoch 4/20\n",
    "30/30 ━━━━━━━━━━━━━━━━━━━━ 1s 11ms/step - gemini_false_negatives: 18.7742 - gemini_false_positives: 0.0000e+00 - loss: 0.0366 - val_gemini_false_negatives: 2.0000 - val_gemini_false_positives: 0.0000e+00 - val_loss: 0.0566\n",
    "Epoch 5/20\n",
    "30/30 ━━━━━━━━━━━━━━━━━━━━ 1s 10ms/step - gemini_false_negatives: 11.6452 - gemini_false_positives: 0.0000e+00 - loss: 0.0268 - val_gemini_false_negatives: 1.0000 - val_gemini_false_positives: 0.0000e+00 - val_loss: 0.0482\n",
    "Epoch 6/20\n",
    "30/30 ━━━━━━━━━━━━━━━━━━━━ 1s 11ms/step - gemini_false_negatives: 6.9032 - gemini_false_positives: 0.0000e+00 - loss: 0.0204 - val_gemini_false_negatives: 1.0000 - val_gemini_false_positives: 0.0000e+00 - val_loss: 0.0408\n",
    "Epoch 7/20\n",
    "30/30 ━━━━━━━━━━━━━━━━━━━━ 1s 14ms/step - gemini_false_negatives: 3.7097 - gemini_false_positives: 0.0000e+00 - loss: 0.0157 - val_gemini_false_negatives: 1.0000 - val_gemini_false_positives: 0.0000e+00 - val_loss: 0.0335\n",
    "Epoch 8/20\n",
    "30/30 ━━━━━━━━━━━━━━━━━━━━ 1s 16ms/step - gemini_false_negatives: 3.7097 - gemini_false_positives: 0.0000e+00 - loss: 0.0119 - val_gemini_false_negatives: 1.0000 - val_gemini_false_positives: 0.0000e+00 - val_loss: 0.0270\n",
    "Epoch 9/20\n",
    "30/30 ━━━━━━━━━━━━━━━━━━━━ 1s 22ms/step - gemini_false_negatives: 1.9677 - gemini_false_positives: 0.0000e+00 - loss: 0.0089 - val_gemini_false_negatives: 1.0000 - val_gemini_false_positives: 0.0000e+00 - val_loss: 0.0224\n",
    "Epoch 10/20\n",
    "30/30 ━━━━━━━━━━━━━━━━━━━━ 1s 22ms/step - gemini_false_negatives: 1.0000 - gemini_false_positives: 0.0000e+00 - loss: 0.0067 - val_gemini_false_negatives: 1.0000 - val_gemini_false_positives: 0.0000e+00 - val_loss: 0.0200\n",
    "\n",
    "<keras.src.callbacks.history.History at 0x784031c75660>\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gemini_embedding_model.evaluate(gemini_test, gemini_test_labels, return_dict=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "17/17 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step - gemini_false_negatives: 8.5000 - gemini_false_positives: 0.0000e+00 - loss: 0.0386\n",
    "\n",
    "{'gemini_false_negatives': 15.0,\n",
    " 'gemini_false_positives': 0.0,\n",
    " 'loss': 0.042361751198768616}\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the best performance we got so far, but the time of the Gemini API requests is very long. \n",
    "Can we speed it up with some open source embeddings model? After trying out some models from \n",
    "the [MTEB Leaderboard](https://huggingface.co/spaces/mteb/leaderboard) at Hugging Face, \n",
    "[GIST-large-Embedding-v0](https://huggingface.co/avsolatorio/GIST-large-Embedding-v0) proved to \n",
    "be the best:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install sentence-transformers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sentence_transformers import SentenceTransformer\n",
    "\n",
    "st_model = SentenceTransformer(\"avsolatorio/GIST-large-Embedding-v0\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "st_emb_dim = st_model.encode('Subject: enron / hpl actuals for july 6 , 2000').shape[0]\n",
    "print(st_emb_dim)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time embs_test_st, labels_test_st = create_embeddings_and_labels(ds_test, st_model.encode)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time embs_train_st, labels_train_st = create_embeddings_and_labels(ds_train_neural, st_model.encode)  # FIXME: batch division in current TF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "...\n",
    "4128 emails processed\n",
    "CPU times: user 3min 40s, sys: 1.19 s, total: 3min 41s\n",
    "Wall time: 3min 45s\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%time embs_val_st, labels_val_st = create_embeddings_and_labels(ds_val_neural, st_model.encode)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inputs = tf.keras.Input((st_emb_dim,))\n",
    "hidden = tf.keras.layers.Dense(st_emb_dim, activation='relu')(inputs)\n",
    "outputs = tf.keras.layers.Dense(1, activation='sigmoid')(hidden)\n",
    "model = tf.keras.Model(inputs, outputs)\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss=tf.keras.losses.BinaryCrossentropy(),\n",
    "              optimizer='adam',\n",
    "              metrics=[tf.metrics.FalsePositives(thresholds=prob_threshold, name='false_positives'),\n",
    "                       tf.metrics.FalseNegatives(thresholds=prob_threshold, name='false_negatives')])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.fit(x=embs_train_st,\n",
    "          y=labels_train_st,\n",
    "          validation_data=(embs_val_st, labels_val_st),\n",
    "          callbacks=[tf.keras.callbacks.EarlyStopping(\n",
    "              monitor='val_false_negatives',\n",
    "              mode='min',\n",
    "             patience=2)],\n",
    "          batch_size=32,\n",
    "          epochs=100,)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.evaluate(embs_test_st, labels_test_st, return_dict=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```\n",
    "17/17 ━━━━━━━━━━━━━━━━━━━━ 0s 11ms/step - loss: 0.0446 - st_false_negatives: 7.6111 - st_false_positives: 0.0000e+00\n",
    "\n",
    "{'loss': 0.06037163734436035,\n",
    " 'st_false_negatives': 15.0,\n",
    " 'st_false_positives': 0.0}\n",
    "\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the same performance we got with the Gemini embeddings, but faster. \n",
    "The main part of calculating the embeddings of the training dataset took \n",
    "3 minutes and 45 seconds on a T4 GPU machine provided by Google Colab."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So our evaluation shows that the performance of the text classification neural networks using LLM embeddings is even more impressive than \n",
    "that of the one using self-trained embeddings, but comes at an even higher computational cost. The following table \n",
    "summarizes our results:\n",
    "\n",
    "| Model | False positives | False negatives | Training time |\n",
    "| ----- | -------------- | --------------- | ------------- |\n",
    "| Bayesian | 0%  | 10% | 2s   |\n",
    "| Basic NN | 0%* | 4%  | 30s  |\n",
    "| LLM      | 0%  | 3%  | 225s |\n",
    "\n",
    "\\* After manual correction."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These results strengthen the epistemological implication of the previous section: arguably, the Bayesian model \n",
    "balances the best between accuracy and simplicity."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Epistemological conclusion\n",
    "\n",
    "We have seen that on the `Enron1` spam dataset, the Bayesian text classifier performs better than its originally specified biased version, and has a good enough performance compared to its more recently proposed machine learning alternatives, considering their complexity. This is in line with the conclusion of Earman's book [*Bayes or Bust?*](https://mitpress.mit.edu/9780262519007/bayes-or-bust/) – in our spam filtering examination, too, Bayesianism has provided the most epistemologically promising method."
   ]
  }
 ],
 "metadata": {
  "datalore": {
   "base_environment": "default",
   "computation_mode": "JUPYTER",
   "package_manager": "pip",
   "packages": [
    {
     "name": "tensorflow",
     "source": "PIP",
     "version": "2.7.0"
    },
    {
     "name": "mock",
     "source": "PIP",
     "version": "4.0.3"
    }
   ],
   "version": 1
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}