Multiobject Tracking in Videos Based on LSTM and Deep Reinforcement Learning

Abstract EN

Multiple-object tracking is a challenging issue in the computer vision community.

In this paper, we propose a multiobject tracking algorithm in videos based on long short-term memory (LSTM) and deep reinforcement learning.

Firstly, the multiple objects are detected by the object detector YOLO V2.

Secondly, the problem of single-object tracking is considered as a Markov decision process (MDP) since this setting provides a formal strategy to model an agent that makes sequence decisions.

The single-object tracker is composed of a network that includes a CNN followed by an LSTM unit.

Each tracker, regarded as an agent, is trained by utilizing deep reinforcement learning.

Finally, we conduct a data association using LSTM for each frame between the results of the object detector and the results of single-object trackers.

From the experimental results, we can see that our tracker achieves better performance than the other state-of-the-art methods.

Multiple targets can be steadily tracked even when frequent occlusions, similar appearances, and scale changes happened.