Learning to Paint The Mona Lisa With Neural Networks

Can we recover an image by learning a deep regression map from pixels (x,y) to colors (r,g,b)?

Yes, we can.

The idea is to use a deep learning (DL) solution to do a deep regression to learn a mapping between pixel locations and RGB colors, with the goal of generating an image one pixel at a time. This means that if the dimensions of the target image are X-by-Y, then it is necessary to run the network X*Y. If the image is 100-by-100, then 10,000 training iterations are executed.

Keras with TensorFlow as a backend creates a working model for this project. Pythonistas rejoice: Keras functional API is a great abstraction for Theanos and Tensorflow (Keras could support new DL frameworks later) to help define complex models, such as multi-output models, directed acyclic graphs, or models with shared layers.

The Sequential model is probably a better choice to implement such a network, but it helps to start with something surprisingly simple.

Using the Model class:

  • A layer instance is callable (on a tensor), and it returns a tensor.
  • Input tensor(s) and output tensor(s) can then be used to define a Model.
  • Such a model can be trained just like Keras Sequential models.

However, first, we need to be able to run Tensorflow on your computer.

Install Docker and run Tensorflow Notebook image on your machine

The best way to run the TensorFlow is to use a Docker container. There’s full documentation on installing Docker at docker.com, but in a few words, the steps are:

  • Go to docs.docker.com in your browser.
  • Step one of the instructions sends you to download Docker.
  • Run that downloaded file to install Docker.
  • At the end of the install process a whale in the top status bar indicates that Docker is running, and accessible from a terminal.
  • Click the whale to get Preferences and other options.
  • Open a command-line terminal, and run some Docker commands to verify that Docker is working as expected. Some useful commands to try are docker version to check that you have the latest release installed.
  • Once Docker is installed, you can download the image which allows you to run Tensorflow on your computer.
  • In a terminal run: docker pull 3blades/tensorflow-notebook
  • MacOS & Linux: Run the deep learning image on your system: docker run -it -p 8888:8888 -p 6006:6006 -v /$(pwd):/notebooks 3blades/tensorflow-notebook
  • Windows: Run the deep learning image on your system: docker run -it -p 8888:8888 -p 6006:6006 -v C:/your/folder:/notebooks 3blades/tensorflow-notebook
  • Once you have completed these steps, you can check the installation by starting your web browser and introducing this URL: http://localhost:8888

We are now ready to paint the Mona Lisa using Deep Regression from pixels to RGB.

Let’s get started!

The infamous Mona Lisa painting is are target image:

import matplotlib.image as mpimg
import matplotlib.pylab as plt
import numpy as np
%matplotlib inline
im = mpimg.imread("data/monalisa.jpg")
plt.imshow(im)
plt.show()
im.shape
Mona Lisa Painting

Our training dataset will be composed of pixels locations and input and pixel values as output:

X_train = []
Y_train = []
for i in range(im.shape[0]):
for j in range(im.shape[1]):
X_train.append([float(i),float(j)])
Y_train.append(im[i][j])

X_train = np.array(X_train)
Y_train = np.array(Y_train)
print 'Samples:', X_train.shape[0]
print '(x,y):', X_train[0],'\n', '(r,g,b):',Y_train[0]
Samples: 30447
(x,y): [ 0. 0.]
(r,g,b): [ 85 105 116]

Let’s now build our sequential model

import keras
from keras.models import Sequential
from keras.layers.core import Dense, Activation, Dropout
from keras.optimizers import Adam, RMSprop, Nadam
# Model architecture
model = Sequential()
model.add(Dense(500, input_dim=2, init='uniform'))
model.add(Activation('relu'))
model.add(Dense(500, init='uniform'))
model.add(Activation('relu'))
model.add(Dense(500, init='uniform'))
model.add(Activation('relu'))
model.add(Dense(500, init='uniform'))
model.add(Activation('relu'))
model.add(Dense(500, init='uniform'))
model.add(Activation('relu'))
model.add(Dense(3, init='uniform'))
model.add(Activation('linear'))
model.summary()
# Compile model
model.compile(loss='mean_squared_error',
optimizer=Nadam(),
metrics=['accuracy'])
# Why use NAdam Optimizer?
# Much like Adam is essentially RMSprop with momentum, Nadam is Adam RMSprop with Nesterov momentum.

Our output:

____________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
====================================================================================================
dense_1 (Dense) (None, 500) 1500 dense_input_1[0][0]
____________________________________________________________________________________________________
activation_1 (Activation) (None, 500) 0 dense_1[0][0]
____________________________________________________________________________________________________
dense_2 (Dense) (None, 500) 250500 activation_1[0][0]
____________________________________________________________________________________________________
activation_2 (Activation) (None, 500) 0 dense_2[0][0]
____________________________________________________________________________________________________
dense_3 (Dense) (None, 500) 250500 activation_2[0][0]
____________________________________________________________________________________________________
activation_3 (Activation) (None, 500) 0 dense_3[0][0]
____________________________________________________________________________________________________
dense_4 (Dense) (None, 500) 250500 activation_3[0][0]
____________________________________________________________________________________________________
activation_4 (Activation) (None, 500) 0 dense_4[0][0]
____________________________________________________________________________________________________
dense_5 (Dense) (None, 500) 250500 activation_4[0][0]
____________________________________________________________________________________________________
activation_5 (Activation) (None, 500) 0 dense_5[0][0]
____________________________________________________________________________________________________
dense_6 (Dense) (None, 3) 1503 activation_5[0][0]
____________________________________________________________________________________________________
activation_6 (Activation) (None, 3) 0 dense_6[0][0]
====================================================================================================
Total params: 1005003
____________________________________________________________________________________________________

Let’s now train our model with 1000 epochs and a 500 batch size.

# use this cell to find the best model architecture
history = model.fit(X_train, Y_train, nb_epoch=1000, shuffle=True, verbose=1, batch_size=500)
Y = model.predict(X_train, batch_size=10000)
k = 0
im_out = im[:]
for i in range(im.shape[0]):
for j in range(im.shape[1]):
im_out[i,j]= Y[k]
k += 1

print "Mona Lisa by DL"
plt.imshow(im_out)
plt.show()

Give it time to run, on a 4GB laptop, it will take you up to 3 hours to get a result.

Epoch 997/1000
30447/30447 [==============================] - 12s - loss: 231.1333 - acc: 0.9138
Epoch 998/1000
30447/30447 [==============================] - 12s - loss: 213.8869 - acc: 0.9170
Epoch 999/1000
30447/30447 [==============================] - 12s - loss: 215.9076 - acc: 0.9130
Epoch 1000/1000
30447/30447 [==============================] - 12s - loss: 217.6785 - acc: 0.9154

And here is our result by painting the Mona Lisa Keras using Tensorflow as backend.

plt.imshow(im_out)
plt.show()
our model accuracy

Let’s now plot our model accuracy

# summarize history for accuracy
plt.plot(history.history['acc'], 'b')
plt.title('Model Accuracy')
plt.xlabel('epoch')
plt.show()
Model AccuracyModel Accuracy

And what about our model loss?

# summarize history for loss
plt.plot(history.history['loss'], 'r')
plt.title('Model Loss')
plt.xlabel('epoch')
plt.show()
Model LossModel Loss

Conclusions

Several tests were made to achieve the presented result. Optimization methods play a significant role in the image quality. In this case, I decided to use Nadam optimizer. Nevertheless, the number of neurons and layers used on the model with a uniform init played a major role in the speed of the script as well as the quality of the output.

Training times significantly improve when using nvidia-docker to take advantage of your machine’s GPU processor. We will post a new article on how to install and use nvidia-docker, which in our experience improves training times by almost 20x!

Get the full code on github

Would you like a demo for 3Blades? Enroll for one by clicking on the button below!

Request Demo

Deep Learning Data Science