Everyone likes a strong chess engine (but you still love our relatively easy chess game, right?). Anyway, the best chess software does give the best game analysis. It is logical, Captain. If this analysis is performed by a FREE chess engine then so much the better. Well, fortunately, a near unbeatable chess engine is available for free – Stockfish 8. It has been a consistent top three contender for the title of worlds strongest chess engine for several years now. Stockfish is the most frequently chosen online chess engine to power a number of the best chess websites(!), as well as numerous online and offline chess apps. My first thought was to test Stockfish vs Houdini or perhaps Stockfish vs Komodo, but then an idea hit me. Why not have a battle of the best free chess engines?
IS STOCKFISH THE BEST FREE CHESS ENGINE?
According to the CCRL 40/40 list Stockfish is the second best chess engine of any type. It is the best open source, free chess engine by five places in this list.
In the future, I hope to play Fire 5, Andscacs 0.88, Fizbo 1.9 and Equinox 3.2 in chess tournaments of their own soon, but for now, this chess match is restricted to free and open source chess engines only. Open source is good for the development of these engines overall as it allows newcomers to learn from the best. I want to support this.
CHOICE OF CHESS GUI
For the “match” tournament interface I chose to use the excellent Aquarium 2017 chess software. I could also have used the free Arena Chess GUI, but as I have just bought the update of Aquarium 2015 to Aquarium 2017 it seemed like a good excuse to use it. The update, which is cheaper than the full version, was only £22 (about $25) – bargain! Aquarium also has a native setting which makes starting a chess engine match at FIDE Classic time controls really, really, easy. Anyway, we have a “venue” and two competitors, so what about the rules?
WHY CHOOSE LONG TIME CONTROLS?
A good reason for using long time controls is that absolutely everyone runs blitz chess tests. There seems little point in posting a blitz chess match that almost anyone can recreate in ten minutes. Be honest, if you have previously bought a computer chess program like Fritz, running an engine competition was one of the first things you did, wasn’t it?
Be honest! 🙂
Another issue is that the difference in playing strength between these two chess titans virtually guarantees the result. For this reason, a chess engine competition like this cannot truly be called a “test” as the result seems certain. Rather, it is a free chess engine match, between two world class chess opponents. If this was a F.I.D.E. rated world championship competition then, of course, it would take place at long time controls – e.g. 2 hours for 40 moves (FIDE Classic).
Playing chess at long time controls takes a lot of computer time. Fortunately, I have just bought a new MacBook Pro which means my old i7 PC is largely free these days. The old Windows PC is four core and so is compatible with normal computer chess ranking standards. These chess games took nearly a week of 24 hours/day computer time! Something that was not possible for me until recently as my PC was also my work computer.
Long time controls allow Stockfish to regularly evaluate 30-35 ply ahead. Gull manages roughly 25-30 ply with the same time available. I thought it might be interesting to see if the difference in ply depth analysis would be more telling, or whether a shallower depth and perhaps better overall analysis would win the day?
CHOICE OF CHESS OPENINGS AND OPENING BOOK
The chosen opening book was “NarrowBook” which is optimised for chess computers. Perhaps I will update it at some point as it is at least three years old and theory may have moved on, but for now it should be fine. Opening book knowledge was restricted to 12 moves, although the computers were regularly out of book before that point.
The match was configured to make the engines play the same opening as black and white. Although restricting the choice of opening moves is somewhat artificial, I thought it was a worthy sacrifice as it enables us to see if the victory in a given game was due to playing strength, or opening advantage. Any free chess engine will have a question over its quality of analysis for some. As both programs use open source code we know their evaluations will differ, but I hoped for some insight into the quality of evaluation given the extra time.
The chess openings chosen by ECO code and English description were:
- B43 – Sicilian, Kan, 5.Nc3
- B92 – Sicilian, Najdorf, 6.f4
- B97 – Sicilian, Najdorf, 7…Qb6 (poisoned pawn variation)
- D48 – Queens Gambit Declined (QGD); Meran, 8…a6
- B33 – Sicilian, Sveshnikov (Lasker–Pelikan) Variation, 2.Nf3 Nc6 3.d4 cxd4 4.Nxd4 Nf6 (without 5.Nc3 e6 (B45) 5…g6 (B34) 5…d6 (B56))
I am in two minds about the wisdom of letting the chess software randomly choose from a limited opening book. While Sicilians are a commonly played chess opening, my intention was not to have four of five games include them. Four semi-closed games and only one semi-open is hardly a wide variety. Chess computers excel at chess tactics, a few open games would have been interesting. At some point in the future I will repeat this match and in future, all chess engine matches will utilise these different settings.
Lesson learned!
STOCKFISH 8 vs GULL 3 – THE RESULTS
Stockfish managed to win 3 out of ten games, with seven draws.
First of all, let me say that Gull is a very good chess program. Unfortunately, here it is simply outclassed. This result is not a surprise.
Indeed, if Gull had managed to take a point off Stockfish then that would have been very interesting. Does this near unbeatable free chess engine have a weakness after all? If it does, these games did not find it.
Let us not be too picky though. One thing to remember is that either free chess engine is far better than any human chess player, even chess grandmasters. They would both be an excellent tool for your own chess improvement.
RESULTS TABLE
| Total | Diff | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | |
| Stockfish 8 64 | 6.5 | +108 | 1 | = | = | 1 | = | = | = | = | = | 1 |
| Gull 3 x64 | 3.5 | -108 | 0 | = | = | 0 | = | = | = | = | = | 0 |
FREE PGN DOWNLOAD OF STOCKFISH VS GULL
For anyone that keeps a chess database I have made the PGN download available. The file is zipped down to 30k. Any unzip program should be able to open it. While both the machine this file was created on and this server are both regularly virus checked, do please run a virus checker on any file you download.
Stockfish 8 vs Gull 3 – free pgn download (zip)
All ten games. More free chess engine tests coming soon!