**Database Creatores:**

Robert Cattral (cattral@gmail.com)

Franz Oppacher (oppacher@scs.carleton.ca)

Carleton University, Department of Computer Science
Intelligent Systems Research Unit

1125 Colonel By Drive, Ottawa, Ontario, Canada, K1S5B6

https://archive.ics.uci.edu/ml/datasets/Poker+Hand

S1 - Suit of card 1

Â” Ordinal (1-4) representing: Hearts=1, Spades=2, Diamonds=3, Clubs=4

1 heart â™¥ 2 spade â™ 3 diamond â™¦ 4 club â™£

C1 - Rank of card 1Â”

Â” Numerical (1-13) representing: Ace=1, 2, 3, 4, 5, 6, 7, 8, 9, 10 , Jack=11, Queen=12, King=13S2 - Suit of card 2Â”

Â” Ordinal (1-4) representing: Hearts=1, Spades=2, Diamonds=3, Clubs=4C2 - Rank of card 2Â”

Â” Numerical (1-13) representing: Ace=1, 2, 3, 4, 5, 6, 7, 8, 9, 10 , Jack=11, Queen=12, King=13S3 - Suit of card 3Â”

Â” Ordinal (1-4) representing: Hearts=1, Spades=2, Diamonds=3, Clubs=4C3 - Rank of card 3Â”

Â” Numerical (1-13) representing: Ace=1, 2, 3, 4, 5, 6, 7, 8, 9, 10 , Jack=11, Queen=12, King=13S4 - Suit of card 4Â”

Â” Ordinal (1-4) representing: Hearts=1, Spades=2, Diamonds=3, Clubs=4C4 - Rank of card 4Â”

Â” Numerical (1-13) representing: Ace=1, 2, 3, 4, 5, 6, 7, 8, 9, 10 , Jack=11, Queen=12, King=13S5 - Suit of card 5Â”

Â” Ordinal (1-4) representing: Hearts=1, Spades=2, Diamonds=3, Clubs=4C5 - Rank of card 5Â”

Â” Numerical (1-13) representing: Ace=1, 2, 3, 4, 5, 6, 7, 8, 9, 10 , Jack=11, Queen=12, King=13CLASS Â“Poker HandÂ” Ordinal (0-9)

0 - Nothing in hand; not a recognized poker hand 1 - One pair; one pair of equal ranks within five cards 2 - Two pairs; two pairs of equal ranks within five cards 3 - Three of a kind; three equal ranks within five cards 4 - Straight; five cards, sequentially ranked with no gaps 5 - Flush; five cards with the same suit 6 - Full house; pair + different rank three of a kind 7 - Four of a kind; four equal ranks within five cards 8 - Straight flush; straight + flush 9 - Royal flush; {Ace, King, Queen, Jack, Ten} + flush

The following poker hand will be represented in the database by the following 10 features vector

** 1, 1, 4, 1, 2, 10, 3, 7, 1, 3**

To run the code below you'll need to download following private libraries which we use in several examples of this course:

- http://www.samyzaf.com/ML/style-notebook.css
- http://www.samyzaf.com/cgi-bin/view_file.py?file=ML/lib/kerutils.py
- http://www.samyzaf.com/cgi-bin/view_file.py?file=ML/lib/dlutils.py

You may download all course code libraries from this github repository:

https://github.com/samyzaf/kerutils

In [1]:

```
import pandas as pd
import matplotlib.pyplot as plt
from matplotlib import rcParams
rcParams['figure.figsize'] = 10,7
# Our deep learning library is Keras
from keras.models import Sequential
from keras.layers.core import Dense, Dropout
from keras.utils.np_utils import to_categorical
import numpy as np
# fixed random seed for reproducibility
np.random.seed(0)
# private libraies
from kerutils import *
%matplotlib inline
```

In [1]:

```
# These are css/html styles for good looking ipython notebooks
from IPython.core.display import HTML
css = open('style-notebook.css').read()
HTML('<style>{}</style>'.format(css))
```

Out[1]:

In [2]:

```
features = ['S1', 'C1', 'S2', 'C2', 'S3', 'C3', 'S4', 'C4', 'S5', 'C5', 'CLASS']
data = pd.read_csv('data/poker-hand-training.csv', names=features)
```

In [5]:

```
# Lets view the first 10 rows of the data set
# See bellow what these names mean
data.head(10)
```

Out[5]:

In [6]:

```
# How many rows do we have?
len(data.index)
```

Out[6]:

In [7]:

```
# How many columns do we have?
len(data.columns)
```

Out[7]:

In [8]:

```
# How many records do we have in our data set?
data.size # 11 * 25010
```

Out[8]:

In [9]:

```
# You can get everything in one line !
data.shape
```

Out[9]:

In [10]:

```
# View the last 10 records of our data set
data.tail(10)
```

Out[10]:

In [18]:

```
# Some statistics to get acquainted with the data
nb_classes = 10 # we have 10 classes of poker hands
cls = {}
for i in range(nb_classes):
cls[i] = len(data[data.CLASS==i])
print(cls)
```

In [23]:

```
# Let's keep a map of poker hand class id to class name
poker_hands = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
hand_name = {
0: 'Nothing in hand',
1: 'One pair',
2: 'Two pairs',
3: 'Three of a kind',
4: 'Straight',
5: 'Flush',
6: 'Full house',
7: 'Four of a kind',
8: 'Straight flush',
9: 'Royal flush',
}
```

In [20]:

```
for i in poker_hands:
print("%s: %d" % (hand_name[i], cls[i]))
```

The classes are highly imbalanced, which could hamper the training process. Our deep learning model will learn a lot about "One Pair" (10599 hands) but very little about "Royal Flash" (only 5 hands) !?

It is usually a good practice to keep this in mind and draw a class distribution bar chart before you start building and training deep learning models. The bar chart will give you a thick visual clue regarding imbalance.

In [25]:

```
plt.bar(poker_hands, [cls[i] for i in poker_hands], align='center')
plt.xlabel('Poker hand id')
plt.ylabel('Number of instances')
```

Out[25]:

We now proceed to build a neural network using Keras for detecting the hand poker class from the 10 feature of the 5 cards at hand. We will first split our data to the first 10 predictive features and the last CLASS feature will be converted to a catogorical form. As usuall, Pandas DataFrames must be converted to Numpy matrices in order to be recognized by Keras.

In [103]:

```
# Let's first extract the first 10 features from our data (from the 11 we have)
# We want to be able to predict the class (hand poker type)
X_train = data.iloc[:,0:10].as_matrix()
y_train = data.iloc[:,10].as_matrix()
```

In [104]:

```
# let's look at the first 20 records sample
X_train[0:20]
```

Out[104]:

In [28]:

```
y_train[0:20]
```

Out[28]:

Let's start with a simple neural network that consists of

- An input layer of 10 neurons
- A hidden layer of 16 neurons
- An ouput layer of 10 neurons (one for each poker hand class)

In [40]:

```
# Our first Keras Model
model1 = Sequential()
model1.add(Dense(16, input_shape=(10,), init='uniform', activation='relu'))
model1.add(Dense(10, init='uniform', activation='softmax'))
```

In [41]:

```
model1.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
```

Keras training method (**fit**) expects one-hot binary vectors as class output.
Hence Keras contains a special utility (**to_categorical**) for converting integer classes to
one-hot binary vectors. One-hot form conversion looks like this:

0 -> [1, 0, 0, 0, 0, 0, 0, 0, 0, 0] 1 -> [0, 1, 0, 0, 0, 0, 0, 0, 0, 0] 2 -> [0, 0, 1, 0, 0, 0, 0, 0, 0, 0] etc.

In [105]:

```
Y_train = to_categorical(y_train)
```

**FitMonitor** callback from our kerutils library (see above download links), which is a more compact progress monitor with intermediate summaries after each 10% period.
The **view_acc** utility is useful for viewing the model training accuracy.

In [45]:

```
# We use our FitMonitor callback from our kerutils library (see above for download)
fmon = FitMonitor()
h = model1.fit(X_train, Y_train, nb_epoch=300, batch_size=32, verbose=0, callbacks=[fmon])
# h is a history object that records the fitting process
```

In [46]:

```
view_acc(h)
```

In [47]:

```
# Validating the accuracy and loss of our training set
loss, accuracy = model1.evaluate(X_train, Y_train, verbose=0)
print("Train: accuracy=%f loss=%f" % (accuracy, loss))
```

In [48]:

```
features = ['S1', 'C1', 'S2', 'C2', 'S3', 'C3', 'S4', 'C4', 'S5', 'C5', 'CLASS']
tdata = pd.read_csv('data/poker-hand-testing.csv', names=features)
```

In [50]:

```
# Checking the success rate on our test set
X_test = tdata.iloc[:,0:10].as_matrix()
y_test = tdata.iloc[:,10].as_matrix()
Y_test = to_categorical(y_test)
loss, accuracy = model1.evaluate(X_test, Y_test, verbose=0)
print("Test: accuracy=%f loss=%f" % (accuracy, loss))
```

In [ ]:

```
```

Test accuracy is 57.6% which is pretty close to the training accuracy (58.1%). This is an encouraging indication that our deep learning model is doing its work as expected. It has been trained on 25,000 samples and gave an exact prediction on a totally different 1 milion samples! This is quite good, so far.

Let's save this model, and proceed to the next one ...

In [37]:

```
model1.save('model1.h5')
```

We can get the predictions vectors of our test set in the following way:

In [56]:

```
y_pred = model1.predict_classes(X_test, verbose=0)
```

And then count the number of failures

In [57]:

```
np.count_nonzero(y_pred - y_test)
```

Out[57]:

We have 423,973 false predictions. We can save them in a numpy array and analyze them

In [64]:

```
false_preds = [(x,y,p) for (x,y,p) in zip(X_test, y_test, y_pred) if y != p]
```

In [68]:

```
false_preds[0:20]
```

Out[68]:

So instead of trying to improve model1 by adding more epochs or tuning activity and optimization, lets build a new model with one extra hidden layer and with more neurons at each layer

In [70]:

```
model2 = Sequential()
model2.add(Dense(50, input_shape=(10,), init='uniform', activation='relu'))
model2.add(Dense(50, init='uniform', activation='relu'))
model2.add(Dense(nb_classes, init='uniform', activation='softmax'))
model2.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
fmon = FitMonitor(thresh=0.03, minacc=0.99, filename="model1.h5")
h = model2.fit(
X_train,
Y_train,
batch_size=32,
nb_epoch=200,
shuffle=True,
verbose=0,
callbacks = [fmon]
)
show_scores(model2, h, X_train, Y_train, X_test, Y_test)
```

This is progressing pretty well: 97.3% accuracy level on our second simple neural network. The model accuracy graph suggests that we get higher accuracy by adding more epochs. Worth trying as an exercise.

Lets see if we get a similar precision rate on our large test set

In [72]:

```
loss, accuracy = model2.evaluate(X_test, Y_test, verbose=0)
print("Test: accuracy=%f loss=%f" % (accuracy, loss))
```

In [76]:

```
y_pred = model2.predict_classes(X_test, verbose=1)
false_preds = [(x,y,p) for (x,y,p) in zip(X_test, y_test, y_pred) if y != p]
```

In [78]:

```
# How many false predictions do we have?
len(false_preds)
```

Out[78]:

In [83]:

```
# Lets take a look at 20 samples
[(a[1], a[2]) for a in false_preds[0:20]]
```

Out[83]:

In [ ]:

```
model2.save("model2.h5")
```

We will use two hidden layers, with 400 neurons each, and run 500 epochs.

In [96]:

```
model3 = Sequential()
model3.add(Dense(200, input_shape=(10,), init='uniform', activation='relu'))
model3.add(Dense(400, init='uniform', activation='relu'))
model3.add(Dense(200, init='uniform', activation='relu'))
model3.add(Dense(nb_classes, init='uniform', activation='softmax'))
model3.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
fmon = FitMonitor(thresh=0.03, minacc=0.996, filename="model3.h5")
h = model3.fit(
X_train,
Y_train,
batch_size=32,
nb_epoch=100,
shuffle=True,
verbose=0,
callbacks = [fmon]
)
show_scores(model3, h, X_train, Y_train, X_test, Y_test)
```

In [97]:

```
# Validating the accuracy and loss of our test set
loss, accuracy = model3.evaluate(X_test, Y_test, verbose=0)
print("Test: accuracy=%f loss=%f" % (accuracy, loss))
```

Adding more neurons and one more hidden layer did help to get 98.7% accuracy level without really hard work from our part. All we did is guess a few numbers and parameters (very easy to do) and Keras wonderfully produced an efficient clean working model.

It took less that 20 minutes to build this network, and it potentially saves us one month programmer work (are we approaching the end of human programmers? ;-).

From the model accuracy graph it does not look like we can do better by running more epochs. Let's save this model, and proceed to the next one ...

In [73]:

```
model3.save("model3.h5")
```

This time we'll throw a lot of neuron on each layer. This is however not wise or desired in most cases. Large neural networks are prone to be slow and consume much more memory than small ones. This is however a matter of trade-offs and we're still experimenting. Sometimes high precision is worth the extra weight.

In [106]:

```
model4 = Sequential()
model4.add(Dense(400, input_shape=(10,), init='uniform', activation='relu'))
model4.add(Dense(800, init='uniform', activation='relu'))
model4.add(Dense(400, init='uniform', activation='relu'))
model4.add(Dense(nb_classes, init='uniform', activation='softmax'))
model4.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
fmon = FitMonitor(thresh=0.03, minacc=0.996, filename="model3.h5")
h = model4.fit(
X_train,
Y_train,
batch_size=32,
nb_epoch=200,
shuffle=True,
verbose=0,
callbacks = [fmon]
)
show_scores(model3, h, X_train, Y_train, X_test, Y_test)
```

In [107]:

```
# Validating the accuracy and loss of our test set
loss, accuracy = model4.evaluate(X_test, Y_test, verbose=0)
print("Test: accuracy=%f loss=%f" % (accuracy, loss))
```

Looks like our best achievment so far: 99.8% training accuracy and 99.6% validation accuracy.

Find a minimal neural network which adhers to the following requirements:

- Produces at least 99.99% accuracy on the training and testing data sets
- Number of deep layers does not exceed 8
- Number of nuerons at each layer does not exceed 200
- Number of training epochs does not exceed 10000
- You're not allowed to use any test sample for training! (not even one instance ;-)
- You will have to submit a full model definition from the ground up: layers, compilation, and training methods

The solution with the smallest number of neurons will receive the highest grade, and the remaining solutions will receive smaller grades relative to their ranking compared to the best solution.