Tensorflow Object Detection API - Inference

After you get a machine learning model by training on Tensorflow, you can run that model and receive the detection result. The source code is here on Github. (https://github.com/koheikawata/objectdetectiontest). Below are examples of inputs and outputs of "object_detection_inference_test1.py"




Set up paths of label, model, and input image

In this example, THRESHOLD allows only inference results higher than 0.6 to be responded

``` THRESHOLD = 0.6 LABEL_PATH = 'object_detection/test1/pascal_label_map.pbtxt' MODEL_PATH = 'object_detection/test1/output/frozen_inference_graph.pb' IMAGE_PATH = 'object_detection/test1/JPEGImages/IMG_00000.jpg' ```

Load image file

``` image = Image.open(IMAGE_PATH) (im_width, im_height) = image.size image_np = np.array(image.getdata()).reshape((im_height, im_width, 3)).astype(np.uint8) image_np_expanded = np.expand_dims(image_np, axis=0) ```
image

``` type(image) ===> PIL.JpegImagePlugin.JpegImageFile image.size ===> (892, 595) ```

image_np

``` type(image_np) ===> numpy.ndarray image_np.size ===> 1592220 image_np.shape ===> (595, 892, 3) ```

image_np_expanded

``` type(image_np_expanded) ===> numpy.ndarray image_np_expanded.size ===> 1592220 image_np_expanded.shape ===> (1, 595, 892, 3) ```

Load model file
If you do not load a model file, ops is null.

``` ops = tf.get_default_graph().get_operations() ```

``` ops ===> [] type(ops) ===> list ```

You have to load a model file and then get tensor names.

``` with tf.gfile.GFile(MODEL_PATH, "rb") as f: graph_def = tf.GraphDef() graph_def.ParseFromString(f.read()) tf.import_graph_def(graph_def, name='') ops = tf.get_default_graph().get_operations() ```

``` for op in ops: print(ops.index(op)) print(op.name) print(op.outputs) print('\n') ```

out:

``` 0 image_tensor [<tf.Tensor 'image_tensor:0' shape=(?, ?, ?, 3) dtype=uint8>] 1 ToFloat [<tf.Tensor 'ToFloat:0' shape=(?, ?, ?, 3) dtype=float32>] 2 Preprocessor/map/Shape [<tf.Tensor 'Preprocessor/map/Shape:0' shape=(4,) dtype=int32>] 3 Preprocessor/map/strided_slice/stack [<tf.Tensor 'Preprocessor/map/strided_slice/stack:0' shape=(1,) dtype=int32>] . . . 4610 detection_boxes [<tf.Tensor 'detection_boxes:0' shape=<unknown> dtype=float32>] 4611 detection_scores [<tf.Tensor 'detection_scores:0' shape=<unknown> dtype=float32>] 4612 detection_classes [<tf.Tensor 'detection_classes:0' shape=<unknown> dtype=float32>] 4613 num_detections [<tf.Tensor 'num_detections:0' shape=<unknown> dtype=float32>] . . . ```

Then you are going to create dictionary of Tensor to input into tf.Session().run.

``` all_tensor_names = {output.name for op in ops for output in op.outputs} tensor_dict = {} for key in ['num_detections', 'detection_boxes', 'detection_scores', 'detection_classes']: tensor_name = key + ':0' if tensor_name in all_tensor_names: tensor_dict[key] = tf.get_default_graph().get_tensor_by_name(tensor_name) image_tensor = tf.get_default_graph().get_tensor_by_name('image_tensor:0') ```

``` type(all_tensor_names) ===> set type(tensor_dict) ===> dict tensor_dict ===> {'num_detections': <tf.Tensor 'num_detections:0' shape=<unknown> dtype=float32>, 'detection_boxes': <tf.Tensor 'detection_boxes:0' shape=<unknown> dtype=float32>, 'detection_scores': <tf.Tensor 'detection_scores:0' shape=<unknown> dtype=float32>, 'detection_classes': <tf.Tensor 'detection_classes:0' shape=<unknown> dtype=float32>} type(image_tensor) ===> tensorflow.python.framework.ops.Tensor image_tensor ===> <tf.Tensor 'image_tensor:0' shape=(?, ?, ?, 3) dtype=uint8> ```

Run inference

Run inference. feed_dict allows to input values. In this case, feed_dict inputs tensor_image made from the test image file.

``` with tf.Session() as sess: output_dict = sess.run(tensor_dict, feed_dict={image_tensor: image_np_expanded}) ```

``` print(output_dict) out: { 'num_detections': array([3.], dtype=float32), 'detection_boxes': array([[ [0.64777863, 0.00554798, 0.84373343, 0.13514924], [0.51240706, 0.5870129 , 0.6917817 , 0.70111954], [0.5978985 , 0.5720269 , 0.6826457 , 0.6266761 ], [0. , 0. , 0. , 0. ], . . . [0. , 0. , 0. , 0. ]]], dtype=float32), 'detection_scores': array([[0.98967135, 0.95399624, 0.737287 , ..., 0. , 0. ]], dtype=float32), 'detection_classes': array([[1., 2., 3., 1., 1., ..., 1., 1.]], dtype=float32) } ```

Change data type of the inference result

``` output_dict['num_detections'] = int(output_dict['num_detections'][0]) output_dict['detection_classes'] = output_dict['detection_classes'][0].astype(np.uint8).tolist() output_dict['detection_boxes'] = output_dict['detection_boxes'][0].tolist() output_dict['detection_scores'] = output_dict['detection_scores'][0].tolist() ```

``` print(output_dict) out: { 'num_detections': 3, 'detection_boxes': [ [0.6477786302566528, 0.005547978449612856, 0.8437334299087524, 0.13514924049377441], [0.5124070644378662, 0.5870128870010376, 0.6917816996574402, 0.7011195421218872], [0.5978984832763672, 0.5720269083976746, 0.6826456785202026, 0.626676082611084], [0.0, 0.0, 0.0, 0.0], . . . [0.0, 0.0, 0.0, 0.0]], 'detection_scores': [0.9896713495254517, 0.9539962410926819, 0.7372869849205017, 0.0, ..., 0.0], 'detection_classes': [1, 2, 3, 1, 1,..., 1, 1] } ```

Get labels and create a json file
If the example of pascal_label_map.pbtxt is like below,

``` item { id: 1 name: 'Ignition coil' } item { id: 2 name: 'Oil level gauge' } item { id: 3 name: 'Air Cleaner' } ```

category_index becomes like this

``` category_index = label_map_util.create_category_index_from_labelmap(LABEL_PATH, use_display_name=True) ```

``` type(category_index) ===> dict category_index ===> { 1: {'id': 1, 'name': 'Ignition coil'}, 2: {'id': 2, 'name': 'Oil level gauge'}, 3: {'id': 3, 'name': 'Air Cleaner'} } ```

And then this creates a json file for response

``` result = [] for idx, score in enumerate(output_dict['detection_scores']): if score > THRESHOLD: result.append({ 'class': output_dict['detection_classes'][idx], 'label': category_index[output_dict['detection_classes'][idx]]['name'], 'confidence': output_dict['detection_scores'][idx], 'bounding_box': output_dict['detection_boxes'][idx] }) json.dumps(result) ```

``` print(json.dumps(result) out: [ { "class": 1, "label": "Ignition Coil", "confidence": 0.9896713495254517, "bounding_box": [0.6477786302566528, 0.005547978449612856, 0.8437334299087524, 0.13514924049377441] }, { "class": 2, "label": "Oil level gauge", "confidence": 0.9539962410926819, "bounding_box": [0.5124070644378662, 0.5870128870010376, 0.6917816996574402, 0.7011195421218872] }, { "class": 3, "label": "Air Cleaner", "confidence": 0.7372869849205017, "bounding_box": [0.5978984832763672, 0.5720269083976746, 0.6826456785202026, 0.626676082611084] } ] ```