Cool Stuff: Style Transfer

Merry Christmas ya’ll. Happy Hanukkah, Happy Kwanzaa, and a Fortuitous Festivus for the rest of us! In the spirit of the holiday, I wanted to give my Mom a little gift.

She’s not the biggest fan of data analytics. I remember one day, we were on the phone and she was telling me this extremely sad story about one of her friends. Her friend was a doctor from the University of Tehran in Iran, and from what my Mom has told me, seems to just be an extremely kind person altogether. For the past fifteen years, however, this woman has been in an emotionally abusive relationship with her husband, and my Mom was telling me about all the hardship she has gone through because of it.

I was a little confused what made my Mom suddenly dive into this story, but I was extremely moved by it, regardless. Then, however, as we reach the end, my Mom gets to her point:

(Thick Iranian Accent) “Kian …      do you know what her husband does? He’s in data analytics. That’s not what you do is it? ” 

At this point I realized, my mom had woven this entire tail of strength and sadness, of pain and struggle, just so she could take this shot at Data Analytics.

“I mean, it is just so boring, all the numbers and typing … ”  (She went on for a while)

 

So over the break, I decided I wanted to show my Mom that Data Analytics, while it is a lot of “numbers and typing,” is the furthest thing from boring, by bringing data to something she cares about.

My Mom has always loved art. And my Mom struggles with photography. So I decided I would try to implement my own style transfer network to show her what data science can actually do.

The hardest part of Deep Learning is correctly framing the problem: All you have to do is figure out what you want from your data, and what loss you want to minimize, and you generate a network that will minimize this loss. While this loss often comes from the difference between a Models predicted output and the correct output, style transfer is a little more complicated.

The data, rather than being given a single input and a single output, is rather two images that serve as both: a style image, and a content image, and the result is a new picture that, hopefully, contains the content of the content image and the style of the style image. The question then arises, how do we distinguish between style and content?

Fortunately, there are smarter people than me out there to answer this question. For my implementation of this goal, I turned to the 2014 paper A Neural Algorithm of Artistic Style.

The primary breakthrough this paper revealed was the difference in features between different convolution layers. The paper explains that more basic features, such as “style” are represented primarily in the early layers of a CNN while more complex features, such as “content” are represented primarily in the higher layers of a CNN.

style-transfer-diagram

Thus, we now have out optimization problem set up. We can minimize our loss between the style of our final image and the style of our style image by minimizing the loss between the lower level CNN representations of each image. Simultaneously, we can minimize our loss between the content of our final image and the content of our content image by minimizing the difference between the higher level CNN representations of each image. We then run back-propagation in order to transform our final image.

In order to run this optimization, I used scipy’s fmin_l_bfgs_b optimizer, which is an extremely fast optimizer that is very commonly used. Furthermore, rather than train a CNN from scratch, I used the VGG19 network available within Keras. After a long round of debugging and error handling, I eventually ran my style transfer and got my results.

 

Content Image (Beautiful Picture of USC’s campus at Twilight)

trojanLogo

 

Style Image: A Clipping from Starry Night

style2

My Transformed Output Image

tro-gogh

 

I put a relatively high weight on content loss rather than style loss, but depending on the weights and various layers you use, you can get a variety of different levels of style within you final paintings.

You can check out the code itself on my github: https://github.com/ghodouss/style-transfer

Have a Fantastic Holidays!