Batch Normalization with TensorFlow

mardi 13 février 2018

I was trying to use batch normalization in order to improve the accuracy of my CIFAR classifier with tf.layers.batch_normalization, and it seemed to have little to no effect. According to this StackOverflow post you need to do something extra, which is not mentioned in the documentation, in order to get the batch normalization to work.

extra_update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)[train_op, extra_update_ops], ...)

The batch norm update operations are added to UPDATE_OPS collection, so you need to create that operation and then feed it into the session along with the training op. Before I had added the extra_update_ops the batch normalization was definitely not running, now it is, whether it helps or not remains to be seen.

Also make sure to use a training=[BOOLEAN | TENSOR] in the call to batch_normalization() to prevent it from being applied during evaluation. I use a placeholder and pass whether it is training or not in via the feed_dict:

training = tf.placeholder(dtype=tf.bool)

And then use this in my batch norm and dropout layers:


There were a few other things I had to do to get batch normalization to work properly:

  1. I had been using local response normalization, which apparently doesn't help that much. I removed those layers and replaced them with batch normalization layers.
  2. Remove the activation from the conv2d layers. I run the output through the batch normalize layers and then apply the relu.

Before I made these changes the model with the batch normalization didn't seem to be training at all, the accuracy was just going up and down right around the baseline of .10. After these changes it seems to be training properly now.

Libellés: data_science, machine_learning, tensor_flow
Aucun commentaire

TensorFlow GPU Errors on Windows

lundi 12 février 2018

I have been loving TensorFlow lately and have installed tensorflow-gpu on my Windows 10 laptop. Given that the GPU on my laptop is not a really great one I have run into quite a few issues, most of which I have solved. My GPU is an Nvidia GeForce GT 750M with 2GB of RAM and I am running the latest release of tensorflow as of February 2018, with Python 3.6. 

If you are running into errors I would suggest you try these things in this order:

  1. Try reducing the batch size for training AND validation. I always use batches for training but would evaluate on the validation data all at once. By using batches for validation and averaging the results I am able to avoid most of the memory errors.
  2. If this doesn't work try to restrict the amount of GPU RAM available to tensorflow with config.gpu_options.per_process_gpu_memory_fraction = 0.7
    which restricts the amount  available to 70%. Note that I am unable to ever run the GPU with the memory fraction above 0.7
  3. If all else fails turn the GPU off and use the CPU: 
    config = tf.ConfigProto()
    config = tf.ConfigProto(device_count = {'GPU': 0})

The difference between using the CPU and the GPU is like night and day... With the CPU it takes all day to train through 20 epochs, with the GPU the same can be done in a few hours. I think the main roadblock with my GPU is the amount of RAM, which can easily be managed by controlling the batch size and the config settings above. Just remember to feed the config into the session.

Libellés: python, data_science, machine_learning, tensor_flow
Aucun commentaire

Why I Stopped Programming in PHP

lundi 12 février 2018

A few months ago I went to a university to interview for a job working on their website. Up until that day I had been programming in PHP with Laravel and Symfony. These MVC frameworks are object-oriented, and they make PHP seem like a "real" programming language. I love them for that. Everything is nicely organized and segmented and you can do almost all the programming in an object oriented fashion.

The university was using a CMS and they asked me to take a little test by looking at some of the code. The code was written in non-object-oriented PHP, which means that everything was done in the page. If you need data from the database you do the query right there in the page, do whatever manipulation you need, then loop through the results with everything embedded right in the HTML with <?php tags. 

I was shocked and dismayed looking at the code. It was about as elegant as the BASIC code I was writing when I was 10 years old, but with HTML mixed in for good measure. It was horrifying to realize that this was the language I was programming in, despite the fact that OOP frameworks make it into something that resembles nicely written code.

The other thing that happened that day was they asked me what my ideal job was. I said I would really like to be involved in some sort of research, but there was no need for PHP in that field. When I got home I thought about it for a few hours, then decided that if that was what I wanted to do I should learn whatever I needed to learn to do it. And that is what I did, and that is why I stopped programming in PHP.

Don't get me wrong - I'm not saying PHP is a horrible language, it can be used very well. But the majority of employers here in Switzerland looking for PHP programmers are not looking for people who do it very well, they are looking for people who have done a three month bootcamp and will work for very low pay. For me, it just wasn't worth the effort to do something I don't really even like that much.

Libellés: coding, php
Aucun commentaire

My Favorite Languages

mardi 23 janvier 2018

My favorite languages, in order:

  1. MatLab - MatLab is just a beautiful language. It is simple but very powerful and makes it easy to do very complex things. While the fact that is is designed solely for numeric computation is a drawback as far as using it for other things, that is one of the reasons I love it.
  2. Python - Python is by far my favorite scripting language. It is very powerful with a lot of features, but that also makes it a bit complex. It isn't as elegant as MatLab, but it is way more useful.
  3. R - I consider R to be somewhere in between MatLab and Python. It is optimized for numerical computing, but can also be used with text and character data. For statistics it stands alone - Python can do pretty much anything R can do, but R is simpler and easier. However, it is more a functional language than a programming language.
  4. SQL - I've worked with MySQL for almost 20 years and I know SQL very well. It is great at working with normalized data, however as storage costs have gone down and RAM has gone up, I'm not sure normalizing data really makes all that much sense these days. Having to join a bunch of tables can really impact performance, which when you just need to get a string out of a joined table doesn't really seem worth it. For web sites it makes sense, but for computational purposes I'm not sure it's really necessary, unless you have more data than you can fit in memory. However I will always remember SQL as my first love.
  5. C - I used C in university, but not much since then. I have forgotten most of what I once knew, but I plan on learning C again because of it's speed and efficiency. The fact that you can use C to write extensions for R, Python, MatLab etc makes it very useful.
  6. PHP - PHP is a reallly ugly language. It has a lot of features, but is inconsistent in syntax and not designed for manipulating data. It's main advantage is that it is easy to learn, but this also makes it very easy to do badly. PHP can be done very well, but good PHP programmers are few and far between and they seem to be getting crowded out by mediocre programmers.
  7. Javascript - Javascript has really been maturing recently. I started using Javascript back in 1996 or so, when all it could really do was alerts and confirmations, it can do a whole lot more than that these days. I have not yet worked with Node.js so I am not all that familiar with the full power of it. I don't really know what my problem with Javascript is. Maybe I still see it as the silly little language it was when I first learned it. Anyway the fact that it is on the bottom of my list should in no way be taken as a reflection of its value.


Libellés: coding, data_science
Aucun commentaire

Archives du Blogue