As the race for live video streaming supremacy heats up with tech companies like Facebook, YouTube and Periscope, it will be interesting to see what they do to differentiate their services from one another. One area we see a lot of opportunity is with live image processing. We don’t mean drawing mustaches on people or swapping faces. We think there are more sophisticated things you can do with real-time, high-performance computer vision and image processing to significantly increase the level of viewer engagement on live video streams.
Live streaming offers anyone the opportunity to capture and share events at a moment’s notice. This often means that there’s no professional lighting, high-end camera or even a tripod to capture a steady shot. People simply hold up the camera they have, typically a cell phone, and point it at the action. This can result in a number of video quality issues that can be resolved in the cloud, on live streams, with existing technology from companies like MotionDSP.
1. Light and Color
Smartphone cameras have become pretty advanced over the years, but they face physical challenges. To fit in your pocket, lenses have to be small, limiting the amount of light they can capture. In low-light situations, video gets dark and noisy. The example below shows video filmed in a dark bar during a concert. On the left is the original where most of the detail is lost due to the camera’s inability to take in light. On the right, we’ve used our light, color and contrast algorithms to bring out some of these lost details.
Even professionally shot videos like the one below of Lady Gaga backstage before a concert can have lighting issues.
Haze, metering and the time of day a video is recorded can lead to color issues such as oversaturation or undersaturation. In the underwater video below, the footage is heavily oversaturated with green. Using our color adjustment filter, we were able to bring down the green levels to give a clearer view of the diver and ship wreck.
Cell phones and drones have built-in stabilization, but both can still suffer from stability issues. Stabilization requires smoothing the motion between jittery frames, but small processors don’t have a lot of memory or compute so they can only use a few frames of video to stabilize. High-performance computing has lots of memory and compute so you can stabilize video over many more frames. Our stabilization algorithm, for example, uses up to 51 video frames.
Unless you’re a surgeon, your hands are likely not as sturdy as a tripod. Filming with a handheld camera while walking or running leads to even shakier video. If you decide to zoom in on something, that camera shake is amplified even more. An example of these issues can be seen in the video below taken at a Beyonce concert. The person filming zooms in on the stage, but the video is terribly unstable as a result. Aside from adjusting the light, color and contrast, we used our stabilization algorithm to remove some of the camera shake.
Drone cameras experience the same zooming issues as cell phones, but they also can be affected by wind. No matter how experienced the pilot is, camera shake will be an issue in windy conditions as seen in the following video.
4. Environmental Issues
Environmental conditions such as smoke, haze and dust can obstruct views. Take, for example, the visibility issues found in the drone footage of Burning Man below. Using a combination of our dehazing algorithm and some light and contrast adjustments, the viewer is able to see details such as people in the shadows beneath the large structure, that otherwise would not be visible due to the dust storm.
5. Limited Field of View
With a narrow field of view, it’s often difficult to understand the entirety of a scene. An example of this took place at a political rally earlier this year. After Trump argued that the cameras never showed the crowds at his rallies, a long panning shot was filmed by one of the cameramen. Since the camera is only able to zoom out so far, we used our mosaicking capabilities to stitch video frames together to create a panorama of the room.
Other Applications of Computer Vision & Image Processing on Live Video
There are some pretty incredible things that computer vision and image processing can accomplish in real time with live video streams outside of video enhancement and panoramas including mapping, and video analytics, but we’ll dig further into those topics in a future blog post.
Aside from accurate mustache placement, what other real-time image processing challenges do you face with live video? Feel free to comment below or send us an email.