Machine Learning with OpenCV, Keras & ImageNet

Reference: http://www.pyimagesearch.com/2016/08/10/imagenet-classification-with-python-and-keras/
François Chollet pushed three Keras models (VGG16, VGG19, and ResNet50) online — these networks are pre-trained on the ImageNet dataset, meaning that they can recognize 1,000 common object classes out-of-the-box.
we can now easily apply VGG16, VGG19, and ResNet50 using Keras and Python to our own applications.
ImageNet is actually a project aimed at labeling and categorizing images into almost 22,000 categories based on a defined set of words and phrases. At the time of this writing, there are over 14 million images in the ImageNet project.

1. Install keras and tensorflow backend
   list the package needed
   
   (opencv_keras_tf) teddy@teddy-K43SJ:~$ pip freeze appdirs==1.4.0 h5py==2.6.0 Keras==1.2.2 numpy==1.12.0 olefile==0.44 packaging==16.8 Pillow==4.0.0 protobuf==3.0.0 pyparsing==2.1.10 PyYAML==3.12 scikit-learn==0.18.1 scipy==0.18.1 six==1.10.0 tensorflow==0.11.0 Theano==0.8.2

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

   (opencv_keras_tf) teddy@teddy-K43SJ:~$ pip freeze appdirs==1.4.0 h5py==2.6.0 Keras==1.2.2 numpy==1.12.0 olefile==0.44 packaging==16.8 Pillow==4.0.0 protobuf==3.0.0 pyparsing==2.1.10 PyYAML==3.12 scikit-learn==0.18.1 scipy==0.18.1 six==1.10.0 tensorflow==0.11.0 Theano==0.8.2

2. Clone the deep-learning-models repository
   Then, to gain access to VGG16, VGG19, and the ResNet50 architectures and pre-trained weights, you need to clone the deep-learning-models repository from GitHub:
   
   (opencv_keras_tf) teddy@teddy-K43SJ:~$ git clone https://github.com/fchollet/deep-learning-models Cloning into 'deep-learning-models'... remote: Counting objects: 47, done. remote: Total 47 (delta 0), reused 0 (delta 0), pack-reused 47 Unpacking objects: 100% (47/47), done. Checking connectivity... done.

   1 2 3 4 5 6

   (opencv_keras_tf) teddy@teddy-K43SJ:~$ git clone https://github.com/fchollet/deep-learning-models Cloning into 'deep-learning-models'... remote: Counting objects: 47, done. remote: Total 47 (delta 0), reused 0 (delta 0), pack-reused 47 Unpacking objects: 100% (47/47), done. Checking connectivity... done.

   
   Then go into the dir ‘deep-learning-models’
   
   (opencv_keras_tf) teddy@teddy-K43SJ:~$ cd deep-learning-models/ (opencv_keras_tf) teddy@teddy-K43SJ:~/deep-learning-models$

   1 2	(opencv_keras_tf) teddy@teddy-K43SJ:~$ cd deep-learning-models/ (opencv_keras_tf) teddy@teddy-K43SJ:~/deep-learning-models$

   
   Notice how we have four Python files. The resnet50.py , vgg16.py , and vgg19.py  files correspond to their respective network architecture definitions.
   The imagenet_utils  file, as the name suggests, contains a couple helper functions that allow us to prepare images for classification as well as obtain the final class label predictions from the network.
3. Write some Python code to classify image contents utilizing Convolutional Neural Networks (CNNs) pre-trained on the ImageNet dataset.
   To start, open up a new file, name it ‘test_imagenet.py’ , and insert the following code:
   
   # import the necessary packages from keras.preprocessing import image as image_utils from imagenet_utils import decode_predictions from imagenet_utils import preprocess_input from vgg16 import VGG16 import numpy as np import argparse import cv2 # construct the argument parse and parse the arguments ap = argparse.ArgumentParser() ap.add_argument("-i", "--image", required=True, help="path to the input image") args = vars(ap.parse_args()) # load the original image via OpenCV so we can draw on it and display # it to our screen later orig = cv2.imread(args["image"]) # load the input image using the Keras helper utility while ensuring # that the image is resized to 224x224 pxiels, the required input # dimensions for the network -- then convert the PIL image to a # NumPy array print("[INFO] loading and preprocessing image...") image = image_utils.load_img(args["image"], target_size=(224, 224)) image = image_utils.img_to_array(image) # our image is now represented by a NumPy array of shape (3, 224, 224), # but we need to expand the dimensions to be (1, 3, 224, 224) so we can # pass it through the network -- we'll also preprocess the image by # subtracting the mean RGB pixel intensity from the ImageNet dataset image = np.expand_dims(image, axis=0) image = preprocess_input(image) # load the VGG16 network print("[INFO] loading network...") model = VGG16(weights="imagenet") # classify the image print("[INFO] classifying image...") preds = model.predict(image) (inID, label) = decode_predictions(preds)[0] # display the predictions to our screen print("ImageNet ID: {}, Label: {}".format(inID, label)) cv2.putText(orig, "Label: {}".format(label), (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 0.9, (0, 255, 0), 2) cv2.imshow("Classification", orig) cv2.waitKey(0)

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49

   # import the necessary packages from keras.preprocessing import image as image_utils from imagenet_utils import decode_predictions from imagenet_utils import preprocess_input from vgg16 import VGG16 import numpy as np import argparse import cv2 # construct the argument parse and parse the arguments ap = argparse.ArgumentParser() ap.add_argument("-i", "--image", required=True,     help="path to the input image") args = vars(ap.parse_args()) # load the original image via OpenCV so we can draw on it and display # it to our screen later orig = cv2.imread(args["image"])   # load the input image using the Keras helper utility while ensuring # that the image is resized to 224x224 pxiels, the required input # dimensions for the network -- then convert the PIL image to a # NumPy array print("[INFO] loading and preprocessing image...") image = image_utils.load_img(args["image"], target_size=(224, 224)) image = image_utils.img_to_array(image)   # our image is now represented by a NumPy array of shape (3, 224, 224), # but we need to expand the dimensions to be (1, 3, 224, 224) so we can # pass it through the network -- we'll also preprocess the image by # subtracting the mean RGB pixel intensity from the ImageNet dataset image = np.expand_dims(image, axis=0) image = preprocess_input(image)   # load the VGG16 network print("[INFO] loading network...") model = VGG16(weights="imagenet") # classify the image print("[INFO] classifying image...") preds = model.predict(image) (inID, label) = decode_predictions(preds)[0] # display the predictions to our screen print("ImageNet ID: {}, Label: {}".format(inID, label)) cv2.putText(orig, "Label: {}".format(label), (10, 30),     cv2.FONT_HERSHEY_SIMPLEX, 0.9, (0, 255, 0), 2) cv2.imshow("Classification", orig) cv2.waitKey(0)

   
    
4. Run the python script ‘test_imagenet.py’.
   NOTE: FOR THE FIRST TIME IT’S VERY SLOW BECAUSE IT DOWNLOADED VGG16 WEIGHT FROM INTERNET (~500MB)
   
   (opencv_keras_tf) teddy@teddy-K43SJ:~/deep-learning-models$ python test_imagenet.py --image images/dog_beagle.png Using TensorFlow backend. I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so locally I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so locally I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so locally I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so locally [INFO] loading and preprocessing image... [INFO] loading network... K.image_dim_ordering: tf Downloading data from https://github.com/fchollet/deep-learning-models/releases/download/v0.1/vgg16_weights_tf_dim_ordering_tf_kernels.h5 I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 0 with properties: name: GeForce GT 520M major: 2 minor: 1 memoryClockRate (GHz) 1.48 pciBusID 0000:01:00.0 Total memory: 963.62MiB Free memory: 780.62MiB I tensorflow/core/common_runtime/gpu/gpu_device.cc:972] DMA: 0 I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 0: Y I tensorflow/core/common_runtime/gpu/gpu_device.cc:1014] Ignoring visible gpu device (device: 0, name: GeForce GT 520M, pci bus id: 0000:01:00.0) with Cuda compute capability 2.1. The minimum required Cuda capability is 3.0. [INFO] classifying image... Downloading data from https://s3.amazonaws.com/deep-learning-models/image-models/imagenet_class_index.json Traceback (most recent call last): File "test_imagenet.py", line 42, in <module> (inID, label) = decode_predictions(preds)[0] ValueError: too many values to unpack (expected 2)

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27

   (opencv_keras_tf) teddy@teddy-K43SJ:~/deep-learning-models$ python test_imagenet.py --image images/dog_beagle.png Using TensorFlow backend. I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcublas.so locally I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcudnn.so locally I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcufft.so locally I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcuda.so.1 locally I tensorflow/stream_executor/dso_loader.cc:111] successfully opened CUDA library libcurand.so locally [INFO] loading and preprocessing image... [INFO] loading network... K.image_dim_ordering: tf Downloading data from https://github.com/fchollet/deep-learning-models/releases/download/v0.1/vgg16_weights_tf_dim_ordering_tf_kernels.h5 I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 0 with properties: name: GeForce GT 520M major: 2 minor: 1 memoryClockRate (GHz) 1.48 pciBusID 0000:01:00.0 Total memory: 963.62MiB Free memory: 780.62MiB I tensorflow/core/common_runtime/gpu/gpu_device.cc:972] DMA: 0 I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 0:   Y I tensorflow/core/common_runtime/gpu/gpu_device.cc:1014] Ignoring visible gpu device (device: 0, name: GeForce GT 520M, pci bus id: 0000:01:00.0) with Cuda compute capability 2.1. The minimum required Cuda capability is 3.0. [INFO] classifying image... Downloading data from https://s3.amazonaws.com/deep-learning-models/image-models/imagenet_class_index.json Traceback (most recent call last):   File "test_imagenet.py", line 42, in <module>     (inID, label) = decode_predictions(preds)[0] ValueError: too many values to unpack (expected 2)

   
   I GOT AN ERROR AT ‘(inID, label) = decode_predictions(preds)[0]’
   SOLUTION: COMMENT LINE #42 in ‘test_imagenet.py’ AND ADD TWO NEW LINES LIKE THIS:
   
   ... preds = model.predict(image) #(inID, label) = decode_predictions(preds)[0] P = decode_predictions(preds) (inID, label, prob) = P[0][0] ...

   1 2 3 4 5 6

   ... preds = model.predict(image) #(inID, label) = decode_predictions(preds)[0] P = decode_predictions(preds) (inID, label, prob) = P[0][0] ...

   
   RUN AGAIN
   
   (opencv_keras_tf) teddy@teddy-K43SJ:~/deep-learning-models$ python test_imagenet.py --image images/dog_beagle.png Using TensorFlow backend. ... [INFO] loading and preprocessing image... [INFO] loading network... K.image_dim_ordering: tf I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 0 with properties: name: GeForce GT 520M major: 2 minor: 1 memoryClockRate (GHz) 1.48 pciBusID 0000:01:00.0 Total memory: 963.62MiB Free memory: 794.12MiB I tensorflow/core/common_runtime/gpu/gpu_device.cc:972] DMA: 0 I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 0: Y I tensorflow/core/common_runtime/gpu/gpu_device.cc:1014] Ignoring visible gpu device (device: 0, name: GeForce GT 520M, pci bus id: 0000:01:00.0) with Cuda compute capability 2.1. The minimum required Cuda capability is 3.0. [INFO] classifying image... ImageNet ID: n02088364, Label: beagle Exception ignored in: <bound method Session.__del__ of <tensorflow.python.client.session.Session object at 0x7f3732e2c400>> Traceback (most recent call last): File "/home/teddy/.virtualenvs/opencv_keras_tf/lib/python3.4/site-packages/tensorflow/python/client/session.py", line 532, in __del__ AttributeError: 'NoneType' object has no attribute 'TF_DeleteStatus'

   1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

   (opencv_keras_tf) teddy@teddy-K43SJ:~/deep-learning-models$ python test_imagenet.py --image images/dog_beagle.png Using TensorFlow backend. ... [INFO] loading and preprocessing image... [INFO] loading network... K.image_dim_ordering: tf I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:925] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero I tensorflow/core/common_runtime/gpu/gpu_device.cc:951] Found device 0 with properties: name: GeForce GT 520M major: 2 minor: 1 memoryClockRate (GHz) 1.48 pciBusID 0000:01:00.0 Total memory: 963.62MiB Free memory: 794.12MiB I tensorflow/core/common_runtime/gpu/gpu_device.cc:972] DMA: 0 I tensorflow/core/common_runtime/gpu/gpu_device.cc:982] 0:   Y I tensorflow/core/common_runtime/gpu/gpu_device.cc:1014] Ignoring visible gpu device (device: 0, name: GeForce GT 520M, pci bus id: 0000:01:00.0) with Cuda compute capability 2.1. The minimum required Cuda capability is 3.0. [INFO] classifying image... ImageNet ID: n02088364, Label: beagle Exception ignored in: <bound method Session.__del__ of <tensorflow.python.client.session.Session object at 0x7f3732e2c400>> Traceback (most recent call last):   File "/home/teddy/.virtualenvs/opencv_keras_tf/lib/python3.4/site-packages/tensorflow/python/client/session.py", line 532, in __del__ AttributeError: 'NoneType' object has no attribute 'TF_DeleteStatus'

   
   SUCCESS! IT CAN RECOGNIZED A DOG ‘BEAGLE’. HERE IS THE RESULTNOTE: The last error ‘NoneType’ not related with keras. TAKE CARE OF IT LATER (SOLUTION READ: http://www.pyimagesearch.com/2016/12/26/opencv-resolving-nonetype-errors/)
   TRY ANOTHER EXAMPLE:
   
   (opencv_keras_tf) teddy@teddy-K43SJ:~/deep-learning-models$ python test_imagenet.py --image images/space_shuttle.png

   1	(opencv_keras_tf) teddy@teddy-K43SJ:~/deep-learning-models$ python test_imagenet.py --image images/space_shuttle.png

   
   NOTE: The downloaded dataset would be stored in /.keras/models/ directory. the dataset filename is ‘vgg16_weights_tf_dim_ordering_tf_kernels.h5’ and the labeled index filename is ‘imagenet_class_index.json’.
5. MORE TEST
   I have some test for recognizing fish on some pictures. I put three samples in ‘images’ directory (/deep-learning-models/images). BUT NONE OF THEM SHOW THE CORRECT RESULT. HERE ARE THE RESULTRECOGNIZED AS ‘tench’ ???
   RECOGNIZED AS ‘electric_ray’ ???
   RECOGNIZED AS ‘hermit_crab’ ???
   I THINK I NEED TO TRAIN THESE SAMPLES FIRST BEFORE IT READY TO BE RECOGNIZED