Playing card detection using CNN

Mar 29, 2022by, Jinu John Veyccan

Uncategorized

In this blog post, we’ll build a playing card detector to determine which cards are present in an image (hearts of king, clubs of three etc). A pre-trained classifier with specific neural network architectures will be used. Before we get into the nitty gritty, let us look into object detection and classification.

Detection Vs Classification

                       Figure 1. 

When performing image classification, we provide the network with one input image and get one class label back. (See Figure 1). The system will recognise Fig 1 as King of Hearts (Kh).

                        Figure 2. 

However, when performing object detection, we can use a single input image to generate multiple bounding boxes and class labels. (See Figure 2)

Getting Started:

  1. Download the complete TensorFlow object detection repository from https://github.com/tensorflow/models by clicking the “Clone or Download” button and saving the zip file.This working directory will include the entire TensorFlow object detection framework, as well as training images, training data, the trained classifier, configuration files, and just about everything else required for the card detection classifier.

                                                          Figure 3.

2. The Faster-RCNN-Inception-V2-COCO model should be downloaded. TensorFlow’s model zoo contains a number of object detection models (pre-trained classifiers with precise neural network architectures).
Models like the SSD-MobileNet have an architecture that provides for faster detection but reduced accuracy, whereas models like the Faster-RCNN have a slower detection but higher accuracy.The Faster-RCNN-Inception-V2 model will be used.
Download the model here.
Open the faster_rcnn_inception_v2_coco_2018_01_28.tar.gz file you downloaded and extract the faster_rcnn_inception_v2_coco_2018_01_28 folder to the object_detection folder.

3. Install any required dependencies. Navigate to this link, and then, under the Table of Contents, Setup section, click on the Installation subsection. The Installation section consists primarily of a list of libraries on which the TensorFlow Object Detection API is dependent. So, before proceeding, make sure that all dependencies are installed.

4. Configure PYTHONPATH. We need to point PYTHONPATH to the \models, \models\research, and \models\research\slim directories. We can do this by issuing export command .

export PYTHONPATH=/home/user/card_detection/models:/home/user/card_detection/models/research:/home/user/card_detection/models/research/slim

5. Protobufs should be compiled. TensorFlow configures model and training parameters using protobufs. As a result, we’ll need to compile them next.

Download and install the protoc 3.0 release, then unzip the package.

Go to tensorflow/models/research/ and do the following:

wget -O protobuf.zip https://github.com/google/protobuf/releases/download/v3.0.0/protoc-3.0.0-linux-x86_64.zip
unzip protobuf.zip

Restart the compilation process, but this time use the downloaded version of protoc.

./bin/protoc object_detection/protos/*.proto –python_out=.

Every name.proto file in the \object detection\protos folder is transformed into a name pb2.py file using this command.

Now execute setup.py in the \models\research directory:

python setup.py build
python setup.py install

6. Collect and label photographs. After configuring the Tensorflow object detection API, the following step is to collect and categorise training photos. Tensorflow requires hundreds of photos with varying backgrounds, lighting conditions, and so on in order to create a competent detection classifier.

We have 53 classes (52+ joker) in this card detector, and I used 40–50 images from each class.

After gathering the images, we must move 20% of them to the \object_detection\images\test directory, and the remaining 80% of them to the \object_detection\images\train directory.
LabelImg can be used to label images. LabelImg creates an.xml file with the label data for each image. These.xml files will be used to generate TFRecords, which are among the TensorFlow trainer’s inputs. After you’ve labelled and saved each image, there will be one .xml file for each picture inside the \test and \train directories.

7. Preparing training data. When the labelling is finished, the next step is creating the TFRecords that will be used as input data for the TensorFlow training model. First, we must convert the.xml files holding all of the data for the train and test photos to CSVs.   In the Anaconda command prompt, run the following command from the \object_detection folder:

python xml_to_csv.py

This will generate the files train_labels.csv and test_labels.csv.

The classifiers must be added in generate_tfrecord.py. We’ll update
def class_text_to_int(row_label) 

def class_text_to_int(row_label):
  if row_label == ‘1d’:
      return 1
  elif row_label == ‘2d’:
      return 2
  elif row_label == ‘3d’:
      return 3
…//all other conditions
  else:
      return None

Then, from the \object_detection folder,  run the following commands to generate TFRecord files:

python generate_tfrecord.py –csv_input=images/train_labels.csv –image_dir=images/train –output_path=train.record
python generate_tfrecord.py –csv_input=images/test_labels.csv –image_dir=images/test –output_path=test.record

These result in the creation of a train.record and a test.record file in \object_detection and will then be utilised to train the card detection classifier.

8. Training configuration
Before training, we must generate a label map and modify the training configuration file.

By defining a mapping of class names to class ID numbers, the label map tells the trainer what each object is. Create a new file with a text editor and save it as labelmap.pbtxt in the  \object_detection\training folder. 

The content should be as described in the following:

item {
id: 1
name: ‘1d’
}
item {
id: 2
name: ‘2d’
}
// all 53 cases

Following that, we must specify which model and which parameters will be utilised for training. G to object_detection\samples\configs and copy the faster_rcnn_inception_v2_pets.config file to the \object_detection\training directory. Then, using a text editor, open the file. Several adjustments must be made to the.config file, including increasing the number of classes and examples and adding the file paths to the training data.

  • Replace num_classes with 53
  • Replace fine_tune_checkpoint with “object_detection/faster_rcnn_inception_v2_coco_2018_01_28/model.ckpt” path
  • Go to the train_input_reader section, and change the following keys:
    input_path : “/object_detection/train.record”
    label_map_path: “/object_detection/training/labelmap.pbtxt”
  • Replace num_examples with the number of images you have in the \images\test directory.
  • In the eval_input_reader section,  change the following keys :
    input_path :“/object_detection/test.record”,
    label_map_path: “/object_detection/training/labelmap.pbtxt”

After you’ve made your modifications, save the file.

9. Run training. Execute the following command from the \object_detection directory,

python train.py –logtostderr –train_dir=training/ –pipeline_config_path=training/faster_rcnn_inception_v2_pets.config

TensorFlow will start training (Fig 4), and each step of training will report the loss. It will begin high and gradually decrease as the training proceeds. Allow your model to train until the loss regularly falls below 0.05, which should take roughly 40,000 steps.

                                                                Figure 4. 

10. It is now time to put our card detector to the test.

After training is completed, we must generate the frozen inference graph (.pb file). Run the following command from the \object_detection folder : 

python export_inference_graph.py –input_type image_tensor –pipeline_config_path training/faster_rcnn_inception_v2_pets.config –trained_checkpoint_prefix training/model.ckpt-XXXX –output_directory inference_graph

 Note : XXXX should be the highest number in that folder.
This results in the creation of a frozen inference graph.pb file in the \object_detection\inference_graph folder. The object detection classifier is stored in the.pb file.

Before we execute the Python scripts, we need to change the NUM CLASSES to 53 and run the command.

python Object_detection_image.py

This will open the image we provided as input, and bounding boxes will appear around the card, as shown in Fig. 5.

                            Figure 5. 

Conclusion

We learned how to create a card detector with Tensorflow’s Object detection API and the pre-trained classifier Faster-RCNN-Inception-V2-COCO.