月星汐,极品少年花都燃情,9c8844
alphazero-gomoku-mpi
link
github : alphazero-gomoku-mpi
overview
this repo is based on junxiaosong/alphazero_gomoku, sincerely grateful for it.
i do these things:
- implement asynchronous self-play training pipeline in parallel like alphago zero's way
- write a root parallel mcts (vote a move using ensemble way)
- use resnet structure to train the model and set a transfer learning api to train a larger board model based on small board's model (like pre-training way in order to save time)
strength
- current model is on 11x11 board, and playout 400 times when test
- play with this model, can always win regardless of black or white
- play with , can rank around 20th-30th for some rough tests
- when i play white, i can't win ai. when i play black, end up with tie/lose for most of my time
references
blog
installation dependencies
- python3
- tensorflow>=1.8.0
- tensorlayer>=1.8.5
- mpi4py (parallel train and play)
- pygame (gui)
how to install
tensorflow/tensorlayer/pygame install :
conda install tensorflow
conda install tensorlayer
conda install pygame
mpi4py install
mpi4py on windows
how to run
python human_play.py
- play with parallel ai (-np : set number of processings, take care of oom !)
mpiexec -np 3 python -u human_play_mpi.py
python train.py
mpiexec -np 43 python -u train_mpi.py
algorithm
it's almost no difference between alphago zero except apv-mcts.
a ppt can be found in dir demo/slides
details
most settings are the same with alphago zero, details as follow :
- network structure
- current model uses 19 residual blocks, more blocks means more accurate prediction but also slower speed
- the number of filters in convolutional layer shows in the follow picture
- feature planes
- in alphago zero paper, there are 19 feature planes: 8 for current player's stones, 8 for opponent's stones, and the final feature plane represents the colour to play
- here i only use 4 for each player, it can be easily changed in
game_board.py
- dirichlet noise
- i add dirichlet noises in each node, it's different from paper that only add noises in root node. i guess alphago zero discard the whole tree after each move and rebuild a new tree, while here i keep the nodes under the chosen action, it's a little different
- weights between prior probabilities and noises are not changed here (0.75/0.25), though i think maybe 0.8/0.2 or even 0.9/0.1 is better because noises are added in every node
- parameters in detail
- training detials
- i train the model for about 100,000 games and takes 800 hours or so
- computer configuration : 2 cpu and 2 1080ti gpu
- we can easily find the computation gap with deepmind and rich people can do some future work
some tips
- network
- zeropadding with input : sometimes when play with ai, it's unaware of the risk at the edge of board even though i'm three/four in a row. zeropadding data input can mitigate the problem
- put the network on gpu : if the network is shallow, it's not matter cpu/gpu to use, otherwise it's faster to use gpu when self-play
- dirichlet noise
- add noise in node : in junxiaosong/alphazero_gomoku, noises are added outside the tree, seemingly like dqn's \(\epsilon-greedy\) way. it's ok when i test on 6x6 and 8x8 board, but when on 11x11 some problems occur. after a long time training on 11x11, black player will always play the first stone in the middle place with policy probability equal to 1. it's very rational for black to play here, however, the white player will never see other kifu that play in the other place at first stone. so, when i play black with ai and place somewhere not the middle place, ai will get very stupid because it has never seen this way at all. add noise in node can mitigate the problem
- smaller weight with noise : as i said before, i think maybe 0.8/0.2 or even 0.9/0.1 is a better choice between prior probabilities and noises' weights, because noises are added in every node
- randomness
- dihedral reflection or rotation : when use the network to output probabilities/value, it's better to do as paper said: the leaf node \(s_l\) is added to a queue for neural network evaluation, \((d_i(p),v)=f_{\theta}(d_i(s_l))\), where \(d_i\) is a dihedral reflection or rotation selected uniformly at random from \(i\) in \([1..8]\)
- add randomness when test : i add the dihedral reflection or rotation also when play with it, so as to avoid to play the same game all the time
- tradeoffs
- network depth : if the network is too shallow, loss will increase. if too deep, it's slow when train and test. (my network is still a little slow when play with it, i think maybe 9 blocks is all right)
- buffer size : if the size is small, it's easy to fit by network but can't guarantee it's performance for only learning from these few data. if it's too large, much longer time and deeper network structure should be taken
- playout number : if small, it's quick to finish a self-play game but can't guarantee kifu's quality. on the contrary with more playout times, better kifu will get but also take longer time
future work can try
- continue to train (a larger board) and increase the playout number
- try some other parameters for better performance
- alter network structure
- alter feature planes
- implement apv-mcts
- train on standard/renju rule
如对本文有疑问,请在下面进行留言讨论,广大热心网友会与你互动!!
点击进行留言回复
相关文章:
-
-
python如何查看网页代码
用python查看网页代码的方法:1、使用“import”导入requests包import requests2、使用requests包的get()函数通过网页...
[阅读全文]
-
-
python如何保存文本文件
python保存文本文件的方法:使用python内置的open()类可以打开文本文件,向文件里面写入数据可以用write()函数,写完之后,使用close()函...
[阅读全文]
-
python如何编写win程序
python可以编写win程序。win程序的格式是exe,下面我们就来看一下使用python编写exe程序的方法。编写好python程序后py2exe模块即可将...
[阅读全文]
-
-
-
-
-
-
网友评论