What was once exclusively the tool of the military is quickly permeating all of US culture. What civilian industry can learn from how the military extracts data from drones.
(Note: this article has been adapted from a presentation given at the NAB 2015 Military and Government Workshop by MotionDSP CEO, Sean Varah)
Drones are fast exploding onto the business scene. Previously drones were almost exclusively for military use, but based on advances in technology, decreasing prices, and new waivers from the Federal Aviation Administration (FAA), they are clearly destined to be a part of everyday life. We’re seeing a myriad of news stories about their ability to change industry, do good in the world, and even save lives. The FAA has recently begun issuing more flight restriction waivers and an increasing number of companies are testing drones for use in markets as far reaching as agriculture, insurance and energy. The curiosity and viability for transformation is here, but what now? What do we do with these drones? And, what do we do with the vast amount of data they collect?
If we look to the military, which have used drones since the 1990s, drones are simply the first step in Intelligence, Surveillance and Reconnaissance (ISR) operations. The same will most likely be true for commercial applications. The military and others depend on drones because they can get to places to get “eyes on” without putting humans in harm’s way. They can journey into denied or restricted areas, and they can be deployed over an area to collect information for long periods of time.
But the story doesn’t end there. Drones are great data collectors, but they don’t provide answers. It’s the processing of the data (by humans and computers) that provides answers to critical questions. Drones help identify who, what, when and where for humans to calculate the why and make informed decisions on how to take action.
MotionDSP has worked with the military for the past decade, and we’ve learned a lot about best-practice processes for extracting critical data from video. The job entails tasking, collection, processing, exploitation, and dissemination (or TC-PED) to share, analyze, and deliver a coherent product, or answer. You might be amazed at how much human effort this process still takes and how labor-intensive it is, even in today’s technologically advanced world.
The military deploys thousands of man-hours to do the processing and analysis work. Of course, they have the manpower, and the work is sensitive. If you are providing oversight for an infantry platoon entering a hostile area, and you need to identify threats, make sure the soldiers don’t meet any surprises, and alert them of unforeseen factors; you’re not going to rely on a computer algorithm alone to do that.
Computers can process massive amounts of data quickly, but they cannot substitute for the experience, insight, and instinct of a trained human analyst. But, even the military is strained by the manpower required as it scales up drone missions – deploying as many as 57 Combat Air Patrols (CAPs) today. For military and commercial applications, the most powerful tool is the combination of man plus machine.
ISR Challenge: It’s Manual
A typical military drone video analysis cell consists of one human analyst watching a live video while typing the activity they see into a database. Another set of analysts manages the database information, generates reports and creates image products or video clips. Other humans combine this information with additional intelligence to create comprehensive reports. They send these reports to command to make informed decisions.
A non-military example of this same human-intensive process was the Boston Bombing. The questions authorities needed to answer quickly were: who were the bombers and where were they? Authorities arrived at an answer by assigning hundreds of detectives and analysts to manually scan hundreds of hours of security video (CCTV) from local businesses around the bombing site, identifying the individuals, locating them in the security footage, and, based on the location of the security cameras, mapping their movements. There were no computer algorithms used: it was old-school detective work.
Government video and satellite images are processed in the same manner. Roomfuls of people at agencies and private contractors load video and satellite data into computers, run manual workflows, and generate reports and products. As stated in the RAND study of the Air Force Distributed Common Ground System (the Air Force’s system for analyzing satellite imagery and manned/unmanned ISR video), hundreds (or thousands) of analysts watch video from dozens of missions each day. Of the 57 Combat Air Patrols per day, RAND estimates that each mission requires at least 75 humans: flyers, maintainers, analysts and more. Do the math.
Hollywood has the same challenge. For example, reality TV producers log hours of video into databases marking where the interesting events are, and then those marked moments are selected to string together into an episode.
Commercial organizations and city governments (like police forces) will face the same challenge. Obviously, the manual process of a human watching every second of a video doesn’t scale. And commercial businesses don’t have the luxury of unlimited manpower – the cost is too high. How can we scale the automated data processing and analysis of video information for commercial markets at a reasonable cost?
Image Processing and Computer Vision to Augment Humans
Image processing software can increase image fidelity by re-constructing video on-the-fly to extract detail and ‘normalize’ the information, making the images clearer and enabling analyst to see things better and faster, like in this video example. This also gives computer vision algorithms a consistent data source to extract information.
Computer vision can rapidly scan and analyze video pixel information, extracting features and patterns that are significant to the questions analysts seek to answer. One example is detection and tracking, where vehicles are automatically detected and tracked from aerial video. Before, human analysts spent time identifying and counting each vehicle separately. With algorithms robust enough to work with real world data (you can’t always collect perfect video), software can track cars over a whole city, accurately and consistently. Computer vision can also turn pixels into data to provide photogrammetry on any source, and reconstruct the 3D scene from video. All these capabilities save a massive amount of manpower.
These are just a few examples of how computer algorithms can help humans create better intelligence that is automated, faster, clearer, more efficient, so questions can be answered and more data analysis can be accomplished with a lot less manpower.
Man + Machine = Efficiency, Accuracy
The combination of people and innovative computer vision software enable video data extraction that is faster, better, accurate, and more efficient than just a computer, or just a human.
The next stage of extracting data from video, for both military and commercial applications looks like this: Humans and computers watch live video (taken from drones or other video collection devices), humans and computers enter observations and activity into a database. Reports and image products are automatically generated, and multi-intelligence data is automatically fused together and visualized.
Humans won’t sit and watch thousands of hours of footage to identify the few nuggets that will solve the case or keep watch on a dangerous situation. Analysis will be performed faster with less manpower, reducing the data to decisions timeline. If we let software help with the searching and reporting, then trained human experts can make more informed decisions faster and more efficiently.
As the US broadens its use of ‘drones for good,’ namely to collect valuable and novel information about all sorts of things, we will see amazing and innovative uses for actionable video in many fields, existing and new. Applying its decade long experience in military special operations, MotionDSP is currently working within a number of commercial industries to help analysts do their jobs better by using computer vision software to find answers within video information faster and more efficiently.