Skip to content
  • What we do

      Technology

      Application development
      Cloud and infrastructure
      Data migration
      Data science and analytics
      IoT enablement
      Modern data estate
      Modern workplace
      Video analytics

      Services

      Support engineering
      Localization

      Products

      Lightweight data virtualization tool

      Advanced video analytics solution

      Strategy

      Data Science
      Maturity Assessment

      Assess your data science maturity (or readiness) to receive a custom report with industry best practices

      Take the assessment
  • Our approach
  • Our work
  • Insights
  • Careers
Connect
Back to insights

Marry object tracking and neural networks to extract more insights from videos

By Alexandru Mureșan

As Blueprint Technologies’ data science team began to work on the video processing capabilities of NASH, our video analytics product, we faced a philosophical question: Should we take the tried-and-tested approach and put our computer vision skills to good use, or are we better off experimenting with newer concepts and technologies, getting our hands dirty with some spicy deep neural network magic?

At Blueprint, we prioritize innovation and experimentation as avenues to get the best results for our clients, and we thrive in the unknown. After some consideration, we decided this was one of those situations where our data scientists could get the best of both worlds. We set out to couple our computer vision skills with neural networks, to enable more precision and better flexibility when analyzing videos.

NASH allows users to extract information from videos – the lengths of which would make human analysis unfeasible. The hypothesis during NASH’s development was that marrying these two techniques would improve the video analytics capabilities included in the product, allowing users to extract more complex information from the videos they want to process. For example, instead of simply counting how many cars appear in a video, we wanted to also analyze the most common paths the cars take – and which cars deviate from those paths, making them anomalies.

Cars being detected and assigned IDs

Cars being detected and assigned unique IDs within NASH

When developing NASH, an early step for Blueprint’s team of data scientists was identifying a reliable way to detect certain objects (i.e., cars and trucks) and track them throughout a video with a unique ID applied to each object that would remain consistent despite:

  • Occlusions (other cars passing in front of it)
  • Illumination changes (passing from light to shadow)
  • Hiccups in the detection algorithm
  • Issues with the tracking algorithm

Neural networks

The main function of an object detector is, as the name implies, to detect objects. After detecting whichever object is needed, the detector draws a rectangle around that object and passes the coordinates forward. We went with an EfficientDet neural network as our object detector because it outshines classic computer vision techniques in two areas: adaptability and robustness.

Adaptability – If the neural network is properly trained it can be used over a multitude of videos with different viewing angles and resolutions without needing to tweak a long list of parameters.

Robustness – Because it is less finicky than a classic computer vision algorithm, you can be more confident that it will miss fewer detections.

The table below explains some of the reasons we chose EfficientDet over other neural networks.

High Level Detail
It was the best solution available at the time State of the art for Real-Time Object Detection task over the COCO dataset, with 55.1 mAP (mean average precision)
Close to real-time processing capability Smallest version could reach up to 36 FPS and maintaining 23 FPS for the 3d version of the neural network (D3)
Flexibility to choose between multiple options with emphasis on either accuracy or speed Various options between:
- quick processing, low resource usage - less accurate results (D0 version)
- slower processing, greater resource usage - more accurate results (D7 version)

For Blueprint’s project, the optimal solution (when optimizing between accuracy and resource consumption) was D3 as our default model. It had 45.6 mAP at 23 FPS and used 1.6 GB of our VRAM. While it is the best model for most applications, it can be easily adjusted should a potential client need a more lightweight model or a heavier, more performant model.

Object detection graph showing the flow of cars as it changes in time

Multi-object tracking

The basic function of a tracker is to assign an ID to a detected object and to keep that ID consistent throughout the entire video. It can, however, also work in unison with our neural network to help us mitigate any shortfalls in image detecting. We created our tracker to use a mix of multiple techniques that enable us to identify and monitor object activity. These include:

  1. Kalman filtering
    • for predicting the future position of detection boxes

  2. Joint detection embedding (JDE) tracking
    • assigns a tracklet to each bounding box
    • associates the detection with corresponding tracklets
    • identifies lost, removed, found and active tracklets

The tracker also has a sliding window search, which uses template matching and custom feature extractors to locate objects that were lost track of.

The basic function of a tracker is to assign an ID to a detected object and to keep that ID consistent throughout the entire video. It can, however, also work in unison with our neural network to help us mitigate any shortfalls in image detecting. We created our tracker to use a mix of multiple techniques that enable us to identify and monitor object activity. These include:


Kalman filtering

  • for predicting the future position of detection boxes

Joint detection embedding (JDE) tracking

  • assigns a tracklet to each bounding box
  • associates the detection with corresponding tracklets
  • identifies lost, removed, found and active tracklets

The tracker also has a sliding window search, which uses template matching and custom feature extractors to locate objects that were lost track of.

Cars being detected and assigned unique IDs

Cars being detected and assigned unique IDs within NASH

Marriage

The concept of using our object tracking algorithm in tandem with the neural network is not complicated. Essentially, what we do is feed our video to the detector, the detector processes each frame, detects the required objects then draws squares around those objects. The tracker then takes the coordinates of these squares and assigns an ID to each one of them. After every frame, the detector tries to see if any object in the current frame has been present in the previous frame. If this is true, the object gets assigned the same ID. If not, the object gets a new ID.

Object detection and tracking pipeline within NASH

Neural networks and object tracking are both great when used alone, but they each have their own weaknesses. The neural network can recognize a type of object, but it can be difficult for it to get more specific. For example, it can tell the difference between a vehicle and a giraffe, but it cannot identify a specific type of vehicle. The object tracker can track something once it knows what it is supposed to track but needs to be told precisely what the user wants it to follow.

Averaged tracked paths in NASH

Common paths identified and clustered

Because NASH combines neural networks with objecting tracking, instead of simply counting vehicles, it can follow specific vehicles to see if any do anything out of the ordinary. This marriage of technology means NASH has fewer false positives. Furthermore, NASH independently figures out what to track thanks to the neural network, alleviating any need for constant human intervention.

This novel and efficient combination of two data science techniques has made NASH a superior video analytics product. But it’s also a great example of what makes Blueprint so special. We’re into everything data, and our data science team works on the bleeding edge of technology and innovation. That experience and expertise makes for some exciting results. Let’s start a conversation about how Blueprint can help you get the most out of your data.

Want to know more about video analytics and how Blueprint can move you forward into this new technology? Introducing NASH, Blueprint’s Advanced Video Analytics tool.

Nash Video Analytics
Learn more about NASH

Let's build your future.

Contact us

Share with your network

You may also enjoy

Article

Connecting your Point-of-Sale data to enhance your customer loyalty program

From stranger to super-fan: 5 ways to give customers what they want, when they want it

Traditionally, customer loyalty programs focus strictly on signups and discounts. That’s not enough anymore. The solution lies in connecting your data to your customer experience.

Article

Customer personalization is a must in the dining industry

 What the top QSR chains are doing that others aren’t 

Since the peak in sales during the Covid-19 shutdown, fast food traffic has continued to see its post-pandemic decline. Consumers still want fast food, but they don’t want to go back to pre-pandemic times. They have new demands.

What we do

  • Cloud and infrastructure
  • Data migration
  • Modern data estate
  • Modern workplace
  • Data science and analytics
  • Application development
  • IoT enablement
  • Video analytics
  • Support engineering
  • Localization
Menu
  • Cloud and infrastructure
  • Data migration
  • Modern data estate
  • Modern workplace
  • Data science and analytics
  • Application development
  • IoT enablement
  • Video analytics
  • Support engineering
  • Localization

Our approach

  • Business strategy
  • Facilitated innovation
  • Project Definition Workshop
  • Course of Action Assessment
  • Proof of Concept
  • Product development
  • Solution development
  • Managed services
Menu
  • Business strategy
  • Facilitated innovation
  • Project Definition Workshop
  • Course of Action Assessment
  • Proof of Concept
  • Product development
  • Solution development
  • Managed services

Our work

Insights

Careers

Contact us

Nash Video Analytics
Linkedin Youtube Twitter Facebook Instagram
© 2022 Blueprint Technologies, LLC. 2600 116th Avenue Northeast, First Floor
Bellevue, WA 98004

All rights reserved.

Privacy Policy

  • What we do
  • Our approach
  • Our work
  • Insights
  • Careers
  • Connect
Menu
  • What we do
  • Our approach
  • Our work
  • Insights
  • Careers
  • Connect
Follow
  • LinkedIn
  • Youtube
  • Twitter
  • Facebook
  • Instagram
Menu
  • LinkedIn
  • Youtube
  • Twitter
  • Facebook
  • Instagram