Basic Neural Network with two hidden layers
In Tensorflow¶
Extended code, courtesy of MorvanZhou
- https://github.com/MorvanZhou/tutorials/tree/master/tensorflowTUT/tf12_plot_result
- Youtube video tutorial: https://www.youtube.com/channel/UCdyjiB5H8Pu7aDTNVXTTpcg
- Youku video tutorial: http://i.youku.com/pythontutorial
LOAD PACKAGES¶
In [1]:
# These are css/html style for good looking ipython notebooks
from IPython.core.display import HTML
css = open('c:/ml/style-notebook.css').read()
HTML('<style>{}</style>'.format(css))
Out[1]:
In [2]:
from __future__ import print_function
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
print ("PACKAGES LOADED")
In [3]:
def add_layer(inputs, in_size, out_size, activation_function=None):
Weights = tf.Variable(tf.random_normal([in_size, out_size]))
biases = tf.Variable(tf.zeros([1, out_size]) + 0.1)
Wx_plus_b = tf.matmul(inputs, Weights) + biases
if activation_function is None:
outputs = Wx_plus_b
else:
outputs = activation_function(Wx_plus_b)
return outputs
Create some simple data¶
In [4]:
x_data = np.linspace(-1, 1, 100)
x_data = x_data.reshape(x_data.shape + (1,))
noise = np.random.normal(0, 0.04, x_data.shape)
y_data = np.square(x_data) - 0.5 + noise
In [5]:
plt.scatter(x_data, y_data)
Out[5]:
Define placeholders for network inputs¶
In [6]:
xs = tf.placeholder(tf.float32, [None, 1])
ys = tf.placeholder(tf.float32, [None, 1])
Add a first hidden layer¶
Input size is 1, output size is 10. xs is inputs vector.
In [7]:
lay_1 = add_layer(xs, 1, 10, activation_function=tf.nn.relu)
Add a second hidden layer¶
Input size is 10, output size is 10. The inputs come from lay_1:
In [8]:
lay_2 = add_layer(lay_1, 10, 10, activation_function=tf.nn.relu)
Add an output layer¶
This time let's experiment with the tanh activation function
In [9]:
prediction = add_layer(lay_2, 10, 1, activation_function=tf.nn.tanh)
Error between prediciton and real data¶
In [10]:
loss = tf.reduce_mean(tf.reduce_sum(tf.square(ys - prediction), reduction_indices=[1]))
train_step = tf.train.GradientDescentOptimizer(0.1).minimize(loss)
Session run¶
In [11]:
sess = tf.Session()
init = tf.global_variables_initializer()
sess.run(init)
Here is a simple model diagram
Putting it all together¶
Lets summarize the above code in one block to build our neural network from the groud up.
In [12]:
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
import time
x_data = np.linspace(-1, 1, 100)
x_data = x_data.reshape(x_data.shape + (1,))
noise = np.random.normal(0, 0.04, x_data.shape)
y_data = np.square(x_data) - 0.5 + noise
xs = tf.placeholder(tf.float32, [None, 1])
ys = tf.placeholder(tf.float32, [None, 1])
lay_1 = add_layer(xs, 1, 10, activation_function=tf.nn.relu)
lay_2 = add_layer(lay_1, 10, 10, activation_function=tf.nn.tanh)
prediction = add_layer(lay_2, 10, 1, activation_function=None)
loss = tf.reduce_mean(tf.reduce_sum(tf.square(ys - prediction), axis=[1]))
train_step = tf.train.GradientDescentOptimizer(0.1).minimize(loss)
sess = tf.Session()
init = tf.global_variables_initializer()
sess.run(init)
Training¶
We will now train our neural network, and draw the model output at certain time frames
In [13]:
n_steps = 501
checkpoint = 100
def draw_output():
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.scatter(x_data, y_data)
return ax
for i in range(n_steps):
# training
sess.run(train_step, feed_dict={xs: x_data, ys: y_data})
if i % checkpoint == 0:
# to visualize the result and improvement
print("train_step:", i)
ax = draw_output()
try:
ax.lines.remove(lines[0])
except Exception:
pass
prediction_value = sess.run(prediction, feed_dict={xs: x_data})
# plot the prediction
lines = ax.plot(x_data, prediction_value, 'r-', lw=5)
plt.show(block=False)
time.sleep(0.5)
