Merge pull request #19; use python-chess 1.5

Change python-chess API to v1.5

.gitignore

@@ -8,3 +8,5 @@ venv
 .idea/
 .venv
 .DS_Store
+*.priv
+*.priv.*

DEVELOPMENT.md

@@ -0,0 +1,50 @@
+## About 
+This document describes development-related topics of pgn-tactics-generator. 
+pgn-tactics-generator is a python application dedicated to creating chess puzzles/tactics from a pgn file.
+See [readme](./README.md) file for more details.
+
+
+# Dependencies
+
+This script requires the [Requests](https://docs.python-requests.org/) and [python-chess](https://python-chess.readthedocs.io/)
+libraries to run, as well as a copy of *Stockfish*.
+Is recommended that you use Python 3 and pip3. 
+It should work with Python 2.7 and pip (probably you will need to install futures `pip install futures` )
+
+To install the requirements use something, like:
+`pip3 install -r requirements.txt --user`
+
+It's recommended that the dependencies are isolated in a [virtual environment](https://docs.python.org/3/tutorial/venv.html)
+
+# Testing
+The project comes with a simple set of basic tests.
+
+## Running the tests
+The test can be run with:
+```bash
+python -m unittest
+```
+
+The recommended way to run the test is using `pytest`:
+```bash
+pytest
+```
+
+ATTOW, the tests cover:
+  * investigate() function (running investigate with know arguments and expected result)
+  * puzzle.is_complete
+  * .ambiguous() and .move_list() methods of position_list class
+
+The idea is that after introducing some changes to the app, you are able to check
+if puzzle generation logic stayed untouched (produces the same results as the previous version).
+
+## Test data
+The tests are using [example data](./test/data) obtained with [positions_for_investigation.py](./positions_for_investigation.py).
+
+The script works in similar fashion as `main.py`: It reads PGN file, goes through
+the games, finds "interesting" positions and generates tactics puzzles.  
+Some intermediate data (for example the puzzle definitions) are recorded and
+writen to json files.
+
+You don't need to re-run the script (as the project already includes required data), unless you want to
+modify/extend the tests.

README.md

@@ -10,9 +10,9 @@ It's based on the great  [https://github.com/clarkerubber/Python-Puzzle-Creator]
 Things that I changed:
 - Use a local pgn file with games as a source.
 - Write results to a file called tactics.pgn
-- Default engine depth to 8 so it's faster. Before it was nodes=3500000 this is a depth around 20. So it took several minutes to analyze a game. With depth 8 it takes seconds.
+- Default engine depth to 8, so it's faster. Before it was nodes=3500000 this is a depth around 20. So it took several minutes to analyze a game. With depth 8 it takes seconds.
 - You can use the `depth` argument to change the depth if you want more precision.
-- chess.pop_count to chess.popcount because it was failing
+- chess.pop_count to chess.popcount, because it was failing
 
 ### This is too complex, give something easy.
 There is another option if you don't want to install and manage python scripts
@@ -24,6 +24,8 @@ It uses a different approach to create tactics, so probably it will generate a d
 This script requires the *Requests* and *Python-Chess* libraries to run, as well as a copy of *Stockfish*
 Is recommended that you use Python 3 and pip3. But it could work with Python 2.7 and pip (probably you will need to install futures `pip install futures` )
 
+Please, take a look at [development doc](DEVELOPMENT.md) for details.
+
 ### Install requirements
 
 `pip3 install -r requirements.txt --user`
@@ -39,7 +41,7 @@ You can download games from a specific user using this command:
 `python3 download_games.py <lichess username>`
 
 
-By default it will download the last 60 games from blitz, rapid and classical.
+By default, it will download the last 60 games from blitz, rapid and classical.
 
 **Arguments**
 
@@ -73,14 +75,14 @@ To execute the generator execute this command. By default it will look for the `
 **Arguments**
 
 - `--quiet` to reduce the screen output.
-- `--depth=8` select the stockfish depth analysis. Default is `8` and will take some seconds to analyze a game, with `--depth=18` will take around 6 minutes.
+- `--depth=8` select the Stockfish depth analysis. Default is `8` and will take some seconds to analyze a game, with `--depth=18` will take around 6 minutes.
 - `--games=ruy_lopez.pgn` to select a specific pgn file. Default is `games.pgn`
 - `--strict=False` Use `False` to generate more tactics but a little more ambiguous. Default is `True`
 - `--threads=4` Stockfish argument, number of engine threads, default `4`
 - `--memory=2048` Stockfish argument, memory in MB to use for engine hashtables, default `2048`
 - `--includeBlunder=False` If False then generated puzzles won't include initial blunder move, default is `True`
 - `--stockfish=./stockfish-x86_64-bmi2` Path to Stockfish binary. 
-  Optional. If ommited, the program will try to locate Stockfish in current directory or download it from the net
+  Optional. If omitted, the program will try to locate Stockfish in current directory or download it from the net
 
 Example:
 `python3 main.py --quiet --depth=12 --games=ruy_lopez.pgn --strict=True --threads=2 --memory=1024`
@@ -93,7 +95,7 @@ The `result header` is the tactic result and not the game result. It can be load
 ## Problems?
 
 #### Stockfish errors
-- If you have problems building stockfish try downloading stockfish directly https://stockfishchess.org/download/
+- If you have problems building Stockfish try downloading Stockfish directly https://stockfishchess.org/download/
 
 ## Want to see all my chess related projects?
 Check [My projects](http://vitomd.com/blog/projects/) for a full detailed list.

main.py

@@ -4,10 +4,9 @@
 
 import argparse
 import logging
-import sys
 
+import chess.engine
 import chess.pgn
-import chess.uci
 
 from modules.api.api import post_puzzle
 from modules.bcolors.bcolors import bcolors
@@ -48,11 +47,8 @@ def prepare_settings():
 
 stockfish_command = get_stockfish_command(settings.stockfish)
 logging.debug(f'Using {stockfish_command} to run Stockfish.')
-engine = chess.uci.popen_engine(stockfish_command)
-engine.setoption({'Threads': settings.threads, 'Hash': settings.memory})
-engine.uci()
-info_handler = chess.uci.InfoHandler()
-engine.info_handlers.append(info_handler)
+engine = chess.engine.SimpleEngine.popen_uci(stockfish_command)
+engine.configure({'Threads': settings.threads, 'Hash': settings.memory})
 
 all_games = open(settings.games, "r")
 tactics_file = open("tactics.pgn", "w")
@@ -67,27 +63,25 @@ def prepare_settings():
     logging.debug(bcolors.WARNING + "Game ID: " + str(game_id) + bcolors.ENDC)
     logging.debug(bcolors.WARNING + "Game headers: " + str(game) + bcolors.ENDC)
 
-    prev_score = chess.uci.Score(None, None)
+    prev_score = chess.engine.Cp(0)
     puzzles = []
 
     logging.debug(bcolors.OKGREEN + "Game Length: " + str(game.end().board().fullmove_number))
     logging.debug("Analysing Game..." + bcolors.ENDC)
 
-    engine.ucinewgame()
-
     while not node.is_end():
         next_node = node.variation(0)
-        engine.position(next_node.board())
 
-        engine.go(depth=settings.depth)
-        cur_score = info_handler.info["score"][1]
+        info = engine.analyse(next_node.board(), chess.engine.Limit(depth=settings.depth))
+
+        cur_score = info["score"].relative
         logging.debug(bcolors.OKGREEN + node.board().san(next_node.move) + bcolors.ENDC)
-        logging.debug(bcolors.OKBLUE + "   CP: " + str(cur_score.cp))
-        logging.debug("   Mate: " + str(cur_score.mate) + bcolors.ENDC)
+        logging.debug(bcolors.OKBLUE + "   CP: " + str(cur_score.score()))
+        logging.debug("   Mate: " + str(cur_score.mate()) + bcolors.ENDC)
+
         if investigate(prev_score, cur_score, node.board()):
             logging.debug(bcolors.WARNING + "   Investigate!" + bcolors.ENDC)
-            puzzles.append(
-                puzzle(node.board(), next_node.move, str(game_id), engine, info_handler, game, settings.strict))
+            puzzles.append(puzzle(node.board(), next_node.move, str(game_id), engine, info, game, settings.strict))
 
         prev_score = cur_score
         node = next_node
@@ -101,3 +95,5 @@ def prepare_settings():
             tactics_file.write("\n\n")
 
 tactics_file.close()
+
+engine.quit()

modules/investigate/investigate.py

@@ -1,4 +1,6 @@
 import chess
+from chess import Board
+from chess.engine import Score
 
 
 def sign(a):
@@ -14,20 +16,25 @@ def material_count(board):
     return chess.popcount(board.occupied)
 
 
-def investigate(a, b, board):
-    # determine if the difference between position A and B 
-    # is worth investigating for a puzzle.
-    if a.cp is not None and b.cp is not None:
-        if (((-110 < a.cp < 850 and 200 < b.cp < 850)
-             or (-850 < a.cp < 110 and -200 > b.cp > -850))
+def investigate(a: Score, b: Score, board: Board):
+    """
+    determine if the difference between position A and B
+    is worth investigating for a puzzle.
+    """
+    a_cp, a_mate = a.score(), a.mate()
+    b_cp, b_mate = b.score(), b.mate()
+
+    if a_cp is not None and b_cp is not None:
+        if (((-110 < a_cp < 850 and 200 < b_cp < 850)
+             or (-850 < a_cp < 110 and -200 > b_cp > -850))
                 and material_value(board) > 3
                 and material_count(board) > 6):
             return True
-    elif a.cp is not None and b.mate is not None and material_value(board) > 3:
-        if (a.cp < 110 and sign(b.mate) == -1) or (a.cp > -110 and sign(b.mate) == 1):
+    elif a_cp is not None and b_mate is not None and material_value(board) > 3:
+        if (a_cp < 110 and sign(b_mate) == -1) or (a_cp > -110 and sign(b_mate) == 1):
+            # from an even position, walking int a checkmate
             return True
-    elif (a.mate is not None
-          and b.mate is not None):
-        if sign(a.mate) == sign(b.mate):  # actually means that they're opposite
+    elif a_mate is not None and b_mate is not None:
+        if sign(a_mate) == sign(b_mate):  # actually means that they're opposite
             return True
     return False

modules/puzzle/analysed.py

@@ -7,9 +7,9 @@ def sign(self, val):
         return -1 if val <= 0 else 1
 
     def sort_val(self):
-        if self.evaluation.cp is not None:
-            return self.evaluation.cp
-        elif self.evaluation.mate is not None:
-            return self.sign(self.evaluation.mate) * (abs(100 + self.evaluation.mate)) * 10000
+        if self.evaluation.score() is not None:
+            return self.evaluation.score()
+        elif self.evaluation.is_mate():
+            return self.sign(self.evaluation.mate()) * (abs(100 + self.evaluation.mate())) * 10000
         else:
             return 0

modules/puzzle/position_list.py

@@ -2,7 +2,7 @@
 from operator import methodcaller
 
 import chess
-import chess.uci
+import chess.engine
 
 from modules.bcolors.bcolors import bcolors
 from modules.puzzle.analysed import analysed
@@ -23,11 +23,11 @@ def __init__(self, position, engine, info_handler, player_turn=True, best_move=N
     def move_list(self):
         if self.next_position is None or self.next_position.ambiguous() or self.next_position.position.is_game_over():
             if self.best_move is not None:
-                return [self.best_move.bestmove.uci()]
+                return [self.best_move.move.uci()]
             else:
                 return []
         else:
-            return [self.best_move.bestmove.uci()] + self.next_position.move_list()
+            return [self.best_move.move.uci()] + self.next_position.move_list()
 
     def category(self):
         if self.next_position is None:
@@ -58,19 +58,20 @@ def generate(self, depth):
 
     def evaluate_best(self, depth):
         logging.debug(bcolors.OKGREEN + "Evaluating Best Move...")
-        self.engine.position(self.position)
-        self.best_move = self.engine.go(depth=depth)
-        if self.best_move.bestmove is not None:
-            self.evaluation = self.info_handler.info["score"][1]
+
+        self.best_move = self.engine.play(self.position, chess.engine.Limit(depth=depth), info=chess.engine.INFO_ALL)
+
+        if self.best_move.move is not None:
+            self.evaluation = self.best_move.info['score'].relative
             self.next_position = position_list(self.position.copy(),
                                                self.engine,
                                                self.info_handler,
                                                not self.player_turn,
                                                strict=self.strict)
-            self.next_position.position.push(self.best_move.bestmove)
-            logging.debug("Best Move: " + self.best_move.bestmove.uci() + bcolors.ENDC)
-            logging.debug(bcolors.OKBLUE + "   CP: " + str(self.evaluation.cp))
-            logging.debug("   Mate: " + str(self.evaluation.mate) + bcolors.ENDC)
+            self.next_position.position.push(self.best_move.move)
+            logging.debug("Best Move: " + self.best_move.move.uci() + bcolors.ENDC)
+            logging.debug(bcolors.OKBLUE + "   CP: " + str(self.evaluation.score()))
+            logging.debug("   Mate: " + str(self.evaluation.mate()) + bcolors.ENDC)
             return True
         else:
             logging.debug(bcolors.FAIL + "No best move!" + bcolors.ENDC)
@@ -81,14 +82,15 @@ def evaluate_legals(self, depth):
         for i in self.position.legal_moves:
             position_copy = self.position.copy()
             position_copy.push(i)
-            self.engine.position(position_copy)
-            self.engine.go(depth=depth)
-            self.analysed_legals.append(analysed(i, self.info_handler.info["score"][1]))
+
+            info = self.engine.analyse(position_copy, chess.engine.Limit(depth=depth))
+            self.analysed_legals.append(analysed(i, info["score"].relative))
+
         self.analysed_legals = sorted(self.analysed_legals, key=methodcaller('sort_val'))
         for i in self.analysed_legals[:3]:
             logging.debug(bcolors.OKGREEN + "Move: " + str(i.move.uci()) + bcolors.ENDC)
-            logging.debug(bcolors.OKBLUE + "   CP: " + str(i.evaluation.cp))
-            logging.debug("   Mate: " + str(i.evaluation.mate))
+            logging.debug(bcolors.OKBLUE + "   CP: " + str(i.evaluation.score()))
+            logging.debug("   Mate: " + str(i.evaluation.mate()))
         logging.debug("... and " + str(max(0, len(self.analysed_legals) - 3)) + " more moves" + bcolors.ENDC)
 
     def material_difference(self):
@@ -109,7 +111,7 @@ def is_complete(self, category, color, first_node, first_val):
                 if (self.material_difference() > 0.2
                         and abs(self.material_difference() - first_val) > 0.1
                         and first_val < 2
-                        and self.evaluation.mate is None
+                        and self.evaluation.mate() is None
                         and self.material_count() > 6):
                     return True
                 else:
@@ -118,7 +120,7 @@ def is_complete(self, category, color, first_node, first_val):
                 if (self.material_difference() < -0.2
                         and abs(self.material_difference() - first_val) > 0.1
                         and first_val > -2
-                        and self.evaluation.mate is None
+                        and self.evaluation.mate() is None
                         and self.material_count() > 6):
                     return True
                 else:
@@ -133,19 +135,18 @@ def ambiguous(self):
         # If strict == False then it will generate more tactics but  more ambiguous
         move_number = 1 if self.strict else 2
         if len(self.analysed_legals) > 1:
-            if (self.analysed_legals[0].evaluation.cp is not None
-                    and self.analysed_legals[1].evaluation.cp is not None):
-                if (self.analysed_legals[0].evaluation.cp > -210
-                        or self.analysed_legals[move_number].evaluation.cp < -90):
+            if (self.analysed_legals[0].evaluation.score() is not None
+                    and self.analysed_legals[1].evaluation.score() is not None):
+                if (self.analysed_legals[0].evaluation.score() > -210
+                        or self.analysed_legals[move_number].evaluation.score() < -90):
                     return True
-            if (self.analysed_legals[0].evaluation.mate is not None
-                    and self.analysed_legals[1].evaluation.mate is not None):
-                if (self.analysed_legals[0].evaluation.mate < 1
-                        and self.analysed_legals[1].evaluation.mate < 1):
+            if (self.analysed_legals[0].evaluation.mate() is not None
+                    and self.analysed_legals[1].evaluation.mate() is not None):
+                if (self.analysed_legals[0].evaluation.mate() < 1
+                        and self.analysed_legals[1].evaluation.mate() < 1):
                     return True
-            if (self.analysed_legals[0].evaluation.mate is not None
-                    and self.analysed_legals[1].evaluation.cp is not None):
-                if self.analysed_legals[1].evaluation.cp < -200:
+            if self.analysed_legals[0].evaluation.is_mate() and not self.analysed_legals[1].evaluation.is_mate():
+                if self.analysed_legals[1].evaluation.score() < -200:
                     return True
         return False