# Learning to Paint The Mona Lisa With Neural Networks

Can we recover an image by learning a deep regression map from pixels (x,y) to colors (r,g,b)?

Yes, we can.

The idea is to use a deep learning (DL) solution to do a deep regression to learn a mapping between pixel locations and RGB colors, with the goal of generating an image one pixel at a time. This means that if the dimensions of the target image are X-by-Y, then it is necessary to run the network X*Y. If the image is 100-by-100, then 10,000 training iterations are executed.

Keras with TensorFlow as a backend creates a working model for this project. Pythonistas rejoice: Keras functional API is a great abstraction for Theanos and Tensorflow (Keras could support new DL frameworks later) to help define complex models, such as multi-output models, directed acyclic graphs, or models with shared layers.

The Sequential model is probably a better choice to implement such a network, but it helps to start with something surprisingly simple.

Using the Model class:

- A layer instance is callable (on a tensor), and it returns a tensor.
- Input tensor(s) and output tensor(s) can then be used to define a Model.
- Such a model can be trained just like Keras Sequential models.

However, first, we need to be able to run Tensorflow on your computer.

#### Install Docker and run Tensorflow Notebook image on your machine

The best way to run the TensorFlow is to use a Docker container. There’s full documentation on installing Docker at docker.com, but in a few words, the steps are:

- Go to docs.docker.com in your browser.
- Step one of the instructions sends you to download Docker.
- Run that downloaded file to install Docker.
- At the end of the install process a whale in the top status bar indicates that Docker is running, and accessible from a terminal.
- Click the whale to get Preferences and other options.
- Open a command-line terminal, and run some Docker commands to verify that Docker is working as expected. Some useful commands to try are docker version to check that you have the latest release installed.
- Once Docker is installed, you can download the image which allows you to run Tensorflow on your computer.
- In a terminal run: docker pull 3blades/tensorflow-notebook
- MacOS & Linux: Run the deep learning image on your system:
`docker run -it -p 8888:8888 -p 6006:6006 -v /$(pwd):/notebooks 3blades/tensorflow-notebook`

- Windows: Run the deep learning image on your system:
`docker run -it -p 8888:8888 -p 6006:6006 -v C:/your/folder:/notebooks 3blades/tensorflow-notebook`

- Once you have completed these steps, you can check the installation by starting your web browser and introducing this URL:
`http://localhost:8888`

We are now ready to paint the Mona Lisa using Deep Regression from pixels to RGB.

Let’s get started!

The infamous Mona Lisa painting is are target image:

import matplotlib.image as mpimg

import matplotlib.pylab as plt

import numpy as np

%matplotlib inline

im = mpimg.imread("data/monalisa.jpg")

plt.imshow(im)

plt.show()

im.shape

**Our training dataset will be composed of pixels locations and input and pixel values as output:**

X_train = []

Y_train = []

for i in range(im.shape[0]):

for j in range(im.shape[1]):

X_train.append([float(i),float(j)])

Y_train.append(im[i][j])

X_train = np.array(X_train)

Y_train = np.array(Y_train)

print 'Samples:', X_train.shape[0]

print '(x,y):', X_train[0],'\n', '(r,g,b):',Y_train[0]

Samples: 30447

(x,y): [ 0. 0.]

(r,g,b): [ 85 105 116]

**Let’s now build our sequential model**

import keras

from keras.models import Sequential

from keras.layers.core import Dense, Activation, Dropout

from keras.optimizers import Adam, RMSprop, Nadam

# Model architecture

model = Sequential()

model.add(Dense(500, input_dim=2, init='uniform'))

model.add(Activation('relu'))

model.add(Dense(500, init='uniform'))

model.add(Activation('relu'))

model.add(Dense(500, init='uniform'))

model.add(Activation('relu'))

model.add(Dense(500, init='uniform'))

model.add(Activation('relu'))

model.add(Dense(500, init='uniform'))

model.add(Activation('relu'))

model.add(Dense(3, init='uniform'))

model.add(Activation('linear'))

model.summary()

# Compile model

model.compile(loss='mean_squared_error',

optimizer=Nadam(),

metrics=['accuracy'])

# Why use NAdam Optimizer?

# Much like Adam is essentially RMSprop with momentum, Nadam is Adam RMSprop with Nesterov momentum.

Our output:

____________________________________________________________________________________________________

Layer (type) Output Shape Param # Connected to

====================================================================================================

dense_1 (Dense) (None, 500) 1500 dense_input_1[0][0]

____________________________________________________________________________________________________

activation_1 (Activation) (None, 500) 0 dense_1[0][0]

____________________________________________________________________________________________________

dense_2 (Dense) (None, 500) 250500 activation_1[0][0]

____________________________________________________________________________________________________

activation_2 (Activation) (None, 500) 0 dense_2[0][0]

____________________________________________________________________________________________________

dense_3 (Dense) (None, 500) 250500 activation_2[0][0]

____________________________________________________________________________________________________

activation_3 (Activation) (None, 500) 0 dense_3[0][0]

____________________________________________________________________________________________________

dense_4 (Dense) (None, 500) 250500 activation_3[0][0]

____________________________________________________________________________________________________

activation_4 (Activation) (None, 500) 0 dense_4[0][0]

____________________________________________________________________________________________________

dense_5 (Dense) (None, 500) 250500 activation_4[0][0]

____________________________________________________________________________________________________

activation_5 (Activation) (None, 500) 0 dense_5[0][0]

____________________________________________________________________________________________________

dense_6 (Dense) (None, 3) 1503 activation_5[0][0]

____________________________________________________________________________________________________

activation_6 (Activation) (None, 3) 0 dense_6[0][0]

====================================================================================================

Total params: 1005003

____________________________________________________________________________________________________

Let’s now train our model with 1000 epochs and a 500 batch size.

# use this cell to find the best model architecture

history = model.fit(X_train, Y_train, nb_epoch=1000, shuffle=True, verbose=1, batch_size=500)

Y = model.predict(X_train, batch_size=10000)

k = 0

im_out = im[:]

for i in range(im.shape[0]):

for j in range(im.shape[1]):

im_out[i,j]= Y[k]

k += 1

print "Mona Lisa by DL"

plt.imshow(im_out)

plt.show()

Give it time to run, on a 4GB laptop, it will take you up to 3 hours to get a result.

Epoch 997/1000

30447/30447 [==============================] - 12s - loss: 231.1333 - acc: 0.9138

Epoch 998/1000

30447/30447 [==============================] - 12s - loss: 213.8869 - acc: 0.9170

Epoch 999/1000

30447/30447 [==============================] - 12s - loss: 215.9076 - acc: 0.9130

Epoch 1000/1000

30447/30447 [==============================] - 12s - loss: 217.6785 - acc: 0.9154

And here is our result by painting the Mona Lisa Keras using Tensorflow as backend.

plt.imshow(im_out)

plt.show()

Let’s now plot our model accuracy

# summarize history for accuracy

plt.plot(history.history['acc'], 'b')

plt.title('Model Accuracy')

plt.xlabel('epoch')

plt.show()

And what about our model loss?

# summarize history for loss

plt.plot(history.history['loss'], 'r')

plt.title('Model Loss')

plt.xlabel('epoch')

plt.show()

#### Conclusions

Several tests were made to achieve the presented result. Optimization methods play a significant role in the image quality. In this case, I decided to use Nadam optimizer. Nevertheless, the number of neurons and layers used on the model with a uniform init played a major role in the speed of the script as well as the quality of the output.

Training times significantly improve when using nvidia-docker to take advantage of your machine’s GPU processor. We will post a new article on how to install and use nvidia-docker, which in our experience improves training times by almost 20x!

Get the full code on github

Would you like a demo for 3Blades? Enroll for one by clicking on the button below!