----------------------------------------------------------------------------------------------------------------------------- 
Raubfisch is an engine derived from Stockfish and Torpedo 1.1 XJR (private engine by Frank Karl Werner and Max Fehler).
-----------------------------------------------------------------------------------------------------------------------------
Result compared to the previous version :.....
			  
-----------------------------------------------------------------------------------------------------------------------------
* The options ICC_Analysis and Clear_Search are dedicated to the chess players by correspondence.

  ICC means International Correspondence Chess.

>> For example, if you want to analyze a difficult position with Raubfisch X series, check "Analysis ICC", and uncheck "Null Move Heuristic". Then validate your settings and launch the analysis.

Always do the following order: 

- Stage 1: First uncheck "Null Move Heuristic", start your position analysis. If after a time you feel long, you have not find satisfaction in your search, stop the analysis.

- Stage 2 : Set in more "Analysis_ICC" and start the analysis of the position.
 
Gradually adjust the values of "Analysis_ICC" from 1 to 4. The value 3 or 4 appears to give satisfaction in the majority of positions to analyze. 
However some very difficult positions to analyse will need to switch in the range from 5 to 8. 
Reminder: the value 1 corresponds to multiPV = 2, the value 2 to multiPV = 4. 
Finally the value 8, corresponding to multiPV = 256, thus following the mathematical law 2 exponent n (2^n); With n taking the following discrete values n = {0, 1, 2, 3, 4, 5, 6, 7, 8}.


* "Vielfalt" means Diversity of lines (moves) , which may be useful during the analysis of a game played or other... (Correspondence chess games)

* "Dynamic Optimism":

Activate the "Dynamic opt" option when playing a game. However, when analyzing a game, please deactivate this option and simultaneously set the value of "Optimism" to 0. 
This last action is intended to avoid sudden variations in the evaluation score and, in the end, allows you to obtain a better evaluation of the position thus analyzed.

----------------------------------------------------------------------------------------------------------------------------------
*-Tune-- > Null move search/dynamic reduction based on depth and value ---

*NullSearchDepthFactor = 38 
*NullSearchValue = 249 
*NullReductionDepthValue = 1000 
*NullReductionDepthFactor = 66 
*NullReductionDepth = 256
*NullReductionPawnValue = 200

---------------------------------------------------------------------------------------------------------------------------------
** Tune-- > Extension search
 
ExtensionSearchDepthFactor1 = 19, 
ExtensionSearchValue1 = 177, 
ExtensionSearchDepthFactor2 = 80, 
ExtensionSearchValue2 = 1

According a 1200 games test, we see stable values in a range between 1.025 and 2, in the following ratio r: 

r = abs[(-ExtensionSearchDepthFactor1+ExtensionSearchValue1)/(-ExtensionSearchDepthFactor2+ExtensionSearchValue2)]
Either 1.025 <= r <= 2.
The above values give a ratio of the order of 2.0, which seems to offer Raubfisch, a rather interesting balance between aggressiveness and security.
---------------------------------------------------------------------------------------------------------------------------------
* Analysis Opt : The option Analysis optimism, called "Analysis Opt" is enabled by default on "Both". This means that white and black pieces will be simultaneously overvalued in an infinite analysis. 
This option can bring an advantage during a search of mate position. However it is recommended to disable this option by selecting "Off" during a deep analysis of position.

--------------------------------------------------------------------------------------------------------------------------------

* Adding Save/Load Hash File Capability of saving the full hash to file. This option code was written by Daniel Jose and improved by Max Fehler.

To save the hash, stop the analysis and click on the "SaveHashtoFile" button in the uci options screen of the GUI.

To load the hash file, load the game you saved and you want to continue Analysis, load the engine without starting it, and press the "LoadHashfromFile" button in the uci options screen of the GUI. 
Now you can start the analysis.

For example saving the full hash to file with Raubfisch X36d1: http://grabilla.com/07a17-1b1ae9d3-b1ca-4c4a-a55e-6b0542d36e03.html
--------------------------------------------------------------------------------------------------------------------------------
* Full Search Thread: If we have a lot of threads, the idea is to make a brute force full depth search.
For example, for a machine with a processor of 4 cores and 8 threads, if you set the value greater than or equal to 8, then the treads will be activated in depth search.
For better efficiency, it is advisable to use maxi a quarter of all threads. In the example above 2 threads are highly recommended
--------------------------------------------------------------------------------------------------------------------------------
* Opening Books

How to use classical book.bin option in Raubfisch engine ?

Raubfisch can use Polyglot opening books. When playing with a book, Raubfisch will move instantly if it finds a legal move leading to a position in the book.
Start by renaming your book by book1.bin . Then copy the book1.bin book to the same directory you put the your Raubfisch executable in. 
Also, make sure that Book1 UCI option is checked, and that "Book1File" is also set to book1.bin. 
The option “Best Book1 Line” when set will make Raubfisch play the best scoring book move, or randomly select between two or more book moves with the same score.
When off, it will tend to play the best scoring move more often, but still give the weaker moves some play.

Note that the instructions for use of the book1 is equally valid for book2 and book3.

http://grabilla.com/08c02-a6fe2baf-1d40-46f5-baa8-18521dd07dca.html

http://grabilla.com/08c02-377725b5-fa76-4ae8-8204-867eea93c9c0.html 

The simultaneous use of 5 books with Raubfisch!

* Raubfisch is an engine that can combine the use of 5 books at the same time; that is to say 3 books.bin (book1.bin, book2.bin and book3.bin), one LiveBook and one book.ctg at the same time! 
This gives you the possibility when the engine do not find the legal move in the 1st book and, if it is in the 2nd book or 3rd book or even the 4th book or 5th book, it will play automatically. 
It is a definite advantage in a tournament or during your workouts in the engine room.

To make the new versions of Raubfisch work properly, automatically generating the "Raubfisch_Files" file. First, start by putting the ".bin" file generated by Raubfisch in the 
same file created automatically by the executable "Raubfisch.exe" and which is called "Raubfisch_Files", thank you for seeing a copy of the screen below.

http://grabilla.com/09a06-cd984f9c-630f-4ac2-acef-c9723f58bf95.html

Then, you have to put "Raubfisch_Files", the Raubfisch engine and all the "books if you have any: book1.bin", "book2.bin" and "book3.bin" in a single directory (called here "Raubfisch_sl"), 
as I show you in the second copy of the screen below.

http://grabilla.com/09a06-90f04718-d4e1-47dd-87f6-f0b5de30f26b.html

If you follow my advice step by step, Raubfisch had to work well.

-------------------------------------------------------------------------------------------------------------------------------
* Self-learning process(²): (Monte Carlo Tree Search by Kelly Kynyama)

Notice that the user will have the choice between disabling Self_Learning or keep it activated as default.

If enabled (default), Self-learning option is a process of learning in the chess engine. During this process multiple files are generated that are experience.bin, pawngame.bin and openings.bin. 
Raubfisch can read and create files below according to the following conditions:

-  pawngame.bin when there is in all and for all 2 pieces for white and black.
 
- an experience.bin file that can be created during an opening or a position analysis (no low depth), satisfying the following conditions:

--> no more than 8 moves(16 half-moves) are played and when there are non more than 7 pieces (no pawns) on the chessboard for the white and black.
--> positions with no more than 6 pieces (no pawns) for white and black. The positions are stored and searched for by hash key.

- openings.bin i.e. Multiple files are also created. In the form .bin (>=1) at the initial stage of game with memorized the move played, the depth and the score. 
Every file is a single opening variation (random files).

Loading the engine, these files are also loaded in the memory in order to make statistical research such as the main variation, Transposition Table, ...

In these conditions, the engine will play better if the same stored position is met at a new game. It persists the following information on the Hard Disk:

The files thus created above therefore stored the following data: 

--> best move
--> board signature (hash key)
--> best move depth
--> best move score

If you have files from experience.bin to Raubfisch. You can merge them, by renaming the first by experience0.bin and the second by experience1.bin. 
Then put the two files in the same folder as the "Raubfisch.exe". Finally, double click Raubfisch.exe. The merger process will be done automatically and Raubfisch_Files will created with merged experience.bin file inside. 

You can thus merge more than two experience.bin files by renaming in turn the file obtained after merging the first two files by experience0.bin on the one hand, and the third file by experience1.bin on the other hand. 
Just click again on Raubfisch.exe. The merging process will be done automatically in the "Raubfisch_Files" file. The final file thus obtained will be named experience.bin, and is therefore ready for use.

Only conditions to guarantee the quality of the final file: be sure 100% that played games are of very good quality with a minimum time of 5 min

Note: If you are already a user of Raubfisch_Self-Learning before RaubfischX41b4 and Raubfisch_GTZ20b4, 
I advise you to copy and transfer all the files: experience.bin, pawngame.bin, as well as all the files openings.bin, 
which have been generated by previous versions of RaubfischX41b4 and Raubfisch_GTZ20b4 also Raubfisch_Self-Learning engines having preceded versions "X41b4" and "GTZ20b4" 
in the same directory titled "Raubfisch_Files". 
Note that there will be more ever to place new Raubfisch engines, from both "X41b4" and "GTZ20b4", in "Raubfisch_Files"

Important Note :
 
From now on the "Self Learning" option is disabled by default. Please check the appropriate box to enable it. 
If you don't want to use "Self Learning" and you already have a large learning database, you can just enable the "Experience Read_only" option. 
Your engine will then be able to tap into "Experience Read_only" to improve its playing style.
----------------------------------------------------------------------------------------------------------------------------------
* Neural-Network process(3)named NNUE: (Efficiently Updateable Neural Network by Yu Nasu, Motohiro Isozaki and Hisayori Noda)
Raubfisch nn-sl is derived from Stockfish NNUE  which is a port of a shogi neural network named NNUE (efficiently updateable neural network backwards) to Stockfish 12.

Important Note :

From now on the " USE NNUE" option is enabled by default. Please check the appropriate box to disabled it.

If you want to use the neural network provided with the Raubfisch engine, just copy and move it to the same directory as the engine. 
The engine will then be able to recognize the network easily and thus communicate with it. However, it is advised not to rename the network provided with the engine. 
However,if you want to use another network of your choice, please specify its path in the Uci and make sure its name is exactly the same as the one in the uci.
See screenshot below: 
http://grabilla.com/0a80e-d699efd1-12a4-4315-a0b7-c96ae4b89669.html

-----------------------------------------------------------------------------------------------------------------------------------

## Training Guide
### Generating Training Data
Use the "no-nnue.nnue-gen-sfen-from-original-eval" binary. The given example is generation in its simplest form. There are more commands. 
```
uci
setoption name Threads value x
setoption name Hash value y
setoption name SyzygyPath value path
isready
gensfen depth a loop b use_draw_in_training_data_generation 1 eval_limit 32000
```
Specify how many threads and how much memory you would like to use with the x and y values. The option SyzygyPath is not necessary, but if you would like to use it, you must first have Syzygy endgame tablebases on your computer, which you can find [here](http://oics.olympuschess.com/tracker/index.php). You will need to have a torrent client to download these tablebases, as that is probably the fastest way to obtain them. The path is the path to the folder containing those tablebases. It does not have to be surrounded in quotes.

This will save a file named "generated_kifu.bin" in the same folder as the binary. Once generation is done, rename the file to something like "1billiondepth12.bin" to remember the depth and quantity of the positions and move it to a folder named "trainingdata" in the same directory as the binaries.
#### Generation Parameters
- Depth is the searched depth per move, or how far the engine looks forward. This value is an integer.
- Loop is the amount of positions generated. This value is also an integer
### Generating Validation Data
The process is the same as the generation of training data, except for the fact that you need to set loop to 1 million, because you don't need a lot of validation data. The depth should be the same as before or slightly higher than the depth of the training data. After generation rename the validation data file to val.bin and drop it in a folder named "validationdata" in the same directory to make it easier. 
### Training a Completely New Network
Use the "avx2.halfkp_256x2-32-32.nnue-learn.2020-07-11" binary. Create an empty folder named "evalsave" in the same directory as the binaries.
```
uci
setoption name SkipLoadingEval value true
setoption name Threads value x
isready
learn targetdir trainingdata loop 100 batchsize 1000000 use_draw_in_training 1 use_draw_in_validation 1 eta 1 lambda 1 eval_limit 32000 nn_batch_size 1000 newbob_decay 0.5 eval_save_interval 250000000 loss_output_interval 1000000 mirror_percentage 50 validation_set_file_name validationdata\val.bin
```
Nets get saved in the "evalsave" folder. 

#### Training Parameters
- eta is the learning rate
- lambda is the amount of weight it puts to eval of learning data vs win/draw/loss results. 1 puts all weight on eval, lambda 0 puts all weight on WDL results.

### Reinforcement Learning
If you would like to do some reinforcement learning on your original network, you must first generate training data using the learn binaries. Make sure that your previously trained network is in the eval folder. Use the commands specified above. Make sure `SkipLoadingEval` is set to false so that the data generated is using the neural net's eval by typing the command `uci setoption name SkipLoadingEval value false` before typing the `isready` command. You should aim to generate less positions than the first run, around 1/10 of the number of positions generated in the first run. The depth should be higher as well. You should also do the same for validation data, with the depth being higher than the last run.

After you have generated the training data, you must move it into your training data folder and delete the older data so that the binary does not accidentally train on the same data again. Do the same for the validation data and name it to val-1.bin to make it less confusing. Make sure the evalsave folder is empty. Then, using the same binary, type in the training commands shown above. Do __NOT__ set `SkipLoadingEval` to true, it must be false or you will get a completely new network, instead of a network trained with reinforcement learning. You should also set eval_save_interval to a number that is lower than the amount of positions in your training data, perhaps also 1/10 of the original value. The validation file should be set to the new validation data, not the old data.

After training is finished, your new net should be located in the "final" folder under the "evalsave" directory. You should test this new network against the older network to see if there are any improvements.

## Using Your Trained Net
If you want to use your generated net, copy the net located in the "final" folder under the "evalsave" directory and move it into a new folder named "eval" under the directory with the binaries. You can then use the halfkp_256x2 binaries pertaining to your CPU with a standard chess GUI, such as Cutechess. Refer to the [releases page](https://github.com/nodchip/Stockfish/releases) to find out which binary is best for your CPU.

If the engine does not load any net file, or shows "Error! *** not found or wrong format", please try to sepcify the net with the full file path with the "EvalFile" option by typing the command `setoption name EvalFile value path` where path is the full file path.

## Resources
- [Stockfish NNUE Wiki](https://www.qhapaq.org/shogi/shogiwiki/stockfish-nnue/)
- [Training instructions](https://twitter.com/mktakizawa/status/1273042640280252416) from the creator of the Elmo shogi engine
- [Original Talkchess thread](http://talkchess.com/forum3/viewtopic.php?t=74059) discussing Stockfish NNUE
- [Guide to Stockfish NNUE](http://yaneuraou.yaneu.com/2020/06/19/stockfish-nnue-the-complete-guide/) 
- [Unofficial Stockfish Discord](https://discord.gg/nv8gDtt)


----------------------------------------------------------------------------------------------------------------------------------

* Live Book option (Windows builds by Khalid Omar and Andrea Manzo, Thanks a lot)

Live Book : When this option is enabled, 

The engine will choose Live_Book move after playing book1.bin, book2.bin and book3.bin book's moves.
Provided that book1.bin, book2.bin and book3.bin are activated! otherwise only the Live_Book will offer a move.

Live Book URL(wonderful project by noobpwnftw, Thanks a lot) :

Chess Cloud Database Query Interface online link : https://www.chessdb.cn/queryc_en/

https://github.com/noobpwnftw/chessdb

Talkchess Forum: http://talkchess.com/forum3/viewtopic.p ... it=chessdb


Live Book Timeout: Default 5000, min 0, max 10000

Live Book Diversity:  
If enabled, the engine will play various lines, reducing conversely its strength because already the live chessdb is very large.

Live Book Contribute: 
If activated, the engine sends a move, not in live chessdb, in its queue to be analysed. In this manner, we have a kind of learning cloud.

Live Book Depth:
Default 20, min 1, max (MAX_PLY-1)


Thanks a lot to Marco Zerbinati for your help about Curl_Library...

------------------------------------------------------------------------------------------------------------------------------------
(²) This topic is concerned that users of Raubfisch_nn_sl (Self-learning) only.
(3) This topic is concerned that users of Raubfisch_nn_sl (NNUE) only.
-------------------------------------------------------------------------------------------------------------------------------



*** Good luck and have the fun...
*** Enjoy... :-)

Frank Karl Werner and Max Felher