So I thought it’d be interesting to determine how strong the first ChessUp 2 bot that is based on Stockfish, bot level 12 “Stockfish elo: 1500” is. I first started out by testing it against the chess.com bots and discovered that the ratings for the chess.com bots are much higher than they should be.
ChessUp 2 level 12 managed to mow through the chess.com bots up to the ones rated in the 2500 range. It had some difficulty with the Kosteniuk (“2561”) bot but managed to win a 6 game match against it, and after drawing the first game against the Naroditsky (“2650”) bot I decided to expedite things and moved it up to the Judit Polgar bot (“2735”). The Judit bot made short work of the ChessUp 2 level 12 bot easily winning the three games they played.
So after that I thought it’d be interesting if I use Fritz 20’s rated game feature to see what it thinks the ChessUp 2’s level 12 bot’s rating is. Fritz 20’s rated game feature has you playing against a weakened Fritz 20 which can make its moves within 1 second each and lets you choose its strength between 1040 and 2440. The 2440 setting brutally outplayed CU2 L12 so from then on I tried to pair it against lower opponents.
in 20 games the highest rating the CU2 L12 achieved was 1967 and with a performance of 10.5 points out of 20 games against opponents with an average rating of 1945 the CU2 L12 ended up with a rating of 1924. As I’m nowhere near that strength I cannot verify it but it sounds like a reasonable estimate, and if that estimate is correct it’d mean it’s over 400 elo stronger than what the “1500” label claims.
That’ll be on my to-do list but I think it might be a little bit trickier due to the current endgame bug I discovered for bots level 11 and below in which they don’t know how to mate with queen vs lone king, this could cause it to allow its opponent to escape with a draw. I discuss the bug in this thread: Major endgame flaw with the hardwired ChessUp bots (levels 1 to 11)?
Completely unscientific and anecdotal but I have always found Chess.com bots much easier than an equivalent CU2 offline bot. Im roughly in the 1200 range. Can beat chess.com bots up to around 1500 but on CU2 1100 is a good match.
Definitely feel that Chess.com bots are overrated make blunders and mistakes no human would ever make at those levels. CU2 feels more accurate.
I used the same Fritz 20 rated game feature to test ChessUp level 13 (Stockfish elo: 1600) and ended up with a rating of 2053 after 20 games against an average opponent rating of 2113. While this is very close to the +100 elo you’d expect from going between ChessUp level 12 and 13 they both seem to be much stronger than the “Stockfish elo: 1500 and 1600” descriptions.
However it is possible the Fritz 20 rated game feature is overestimating the rating?
I think it is more likely that the estimate from the Fritz 20 rating tool is closer to the truth. Here’s one of the games in which ChessUp level 13 managed to get 3 brilliancies according to the chess.com Game Review at max settings, and an 84% accuracy score vs black’s 78.8%:
I must say though that a few times during my testing ChessUp level 13 played strangely horrific in the opening, like in this game where it looks like it was tilted:
But more often than not I feel that ChessUp level 13 played solid chess. According to chess.com’s Game Review it scored an average of 85.56% as white and 82.02% as black.
Next up for me is ChessUp level 11 although I’m unsure how well that would go because it currently has no endgame knowledge and can allow draws even if it’s up by a queen against a lone king. I think my best option in this case is to put it against an opponent that is rated 200 elo stronger, but finding that out might be tricky and am not sure if it’s worth the effort since they plan on giving it some endgame knowledge.
After 4 games in my test of the ChessUp level 11 (“1400”) bot I have decided to terminate the test after this disappointing game where level 11 was completely winning and gifted white with a draw by repetition:
So I’ve been testing the ChessUp 2 level 13 bot at Lichess against the community bots and after 23 games it has a rapid rating of 2258 and my observation is that it is very close in strength with the SimpleEval bot which currently has a 2234 rapid rating and a 2167 blitz rating.
The question is how does the Lichess rating compare to the estimated rating given by the Fritz rating tool which was 2053? I wish I had done the Lichess testing with blitz time control as the Chessup 2 bots already play fast no matter what time control but even if we were to assume the Chessup 2 level 13 bot is only around 2160 in blitz (I’m assuming in blitz it’d still be around the same strength as the SimpleEval bot) that’s still a 100 elo difference between the two estimates.
But now there’s two estimates putting the ChessUp 2 level 13 bot at over 2000 elo which is significantly higher than the bot’s description of “1600” elo. Perhaps the description should be updated or preferably the bot be weakened without sacrificing its endgame knowledge.