Videos have always been an important part of our lives and as smartphones are getting faster and smarter, these handheld all-purpose devices can potentially replace your dedicated DSLRs, point and shoot cameras, and even action cameras for your videography needs.

As much as we have advanced in the smartphone imaging department, videos have often been a neglected feature on Android smartphones, even on flagships. However, in recent years, brands have eventually figured out the importance of smartphones as video cameras and are focusing on improving the overall videography experience.

OPPO has been at the forefront of optimizing smartphone cameras and we have already seen some interesting technologies from the brand like the RFB-ERSGAN neural network which upscales original low-resolution images into high-resolution ones without losing details or even the Hybrid Zoom technology with a stacked periscope zoom lens structure.

In the upcoming Reno5Pro 5G, OPPO will be using its brand new Full Dimension Fusion (FDF) Portrait Video System. This is a set of imaging algorithms that work behind the scenes to optimize the video on the new device. And in a recent briefing, OPPO explained how the system works and how it optimizes the videos according to a number of parameters for the best possible output.

So here’s a quick look at how OPPO’s FDF Portrait Video System on the OPPO Reno5Pro 5G works. This article isn’t meant to be a deep dive into the system. Instead, it’s a quick overview of what happens behind the scenes when you take a video using the phone.

As mentioned above, OPPO’s camera system is a collection of different algorithms that work in tandem to optimize the videos recorded on the OPPO Reno5Pro 5G. The system is comprised of two main engines:

1. Quality Enhancement Engine

As the name suggests, the Quality Enhancement Engine improves the quality of the video in a number of different parameters like brightness, dynamic range, and noise reduction. This engine works in the background to improve the overall quality of the video being recorded.

2. Portrait Perception Engine

The Portrait Perception Engine on the other hand focuses on scene recognition. It works in the background to identify human figures in the scene and effectively optimize them with respect to its environment. This particular engine is also responsible for enabling creative filters inside the video like the AI Color Portrait mode.

To understand it better, this flowchart will give you a quick overview of how the system works.

HOW THE SYSTEM WORKS 

The two engines work together to optimize the video on the Reno5Pro 5G. Each engine has its own set of optimizations being processed in the background. Now let’s take a quick look at these processes.

Quality Enhancement Engine

The Quality Enhancement Engine is comprised of two AI algorithm architectures – AI Highlight Video and Ultra Steady 3.0.

• • AI Highlight Video
    OPPO reveals that AI Highlight Video is its proprietary algorithm architecture that combines Ultra Night Video Algorithms and Live HDR Algorithms to optimize the brightness, color, and clarity in videos, irrespective of its light levels. OPPO also notes that this is the first time that they have been able to use both its Night and Live HDR algorithms simultaneously. This means that AI Highlight Video works in daylight as well as low-light situations and it’s available on both front and rear cameras. You just have to press a button in the camera UI to turn on this feature.
    AI Highlight Video optimizes the video in three different parameters.
    • • AI Adaptive Brightness Enhancement: The algorithm uses adaptive gain to control the exposure time to ensure the images in the video are bright yet not overexposed.

     

    • • AI Adaptive Dynamic Range: The Adaptive Dynamic Range adjusts the dynamic range to each frame in a video using Dynamic Range Conversion and Filmic Tone Mapping (FTM) technologies. OPPO claims that this process restores lost colors, brightness, and darkness to the image’s natural levels. This process also includes Multi-Frame HDR for better dynamic range across the entire video.

     

    • • AI Adaptive Noise Reduction: This process adds inter-frame fusion for temporal noise reduction (between two frames) and intra-frame fusion (inside the frame) for spatial noise reduction. OPPO claims that this process removes an image’s noise without removing essential details, unlike other regular de-noising algorithms.

        

        

     

    It’s clear that OPPO is giving a lot of importance to its AI Highlight Video algorithm. In fact, the company shared a few key stats on how it improves the overall quality and performance of videography on smartphones.

    OPPO reveals that AI Highlight video enhances the brightness of night-time videos by 74.4% with 33.7% clarity as compared to the regular video mode. It takes just 20ms to complete the entire process on the Reno5Pro 5G while the power consumption also drops by 40% as compared to its predecessor.

  • Ultra Steady 3.0
    OPPO’s Quality Enhancement Engine also depends upon its new generation of Ultra Steady 3.0 technology which provides stabilization technology to a broader range of scenarios like running, jumping, and even extreme sports.
    

Next up is the portrait perception engine. A lot of you might already be familiar with this engine. This set of algorithms is responsible for enabling portrait centric features like the cool AI Color Portrait videos, irrespective of the background or the state of the person.

Portrait Perception Engine

Portrait Perception Engine undergoes three typical processes when you shoot a video.

• • • • Scene Design Manager: The first stage is defining the portrait scene and processing the requirements needed for the same. Using OPPO’s own database with over 10 million data, the Scene Design manager can analyze real-life portrait scenes in different scenarios and provide required adjustments to the video against a set of fixed parameters. This is the first stage where the engine prepares the video for further processing.
        

     

    • • AI Modelling: Once the scene is assessed, AI models and deep neural networks kick into action, essentially separating out the portrait from the background. This is a very important part of OPPO’s portrait video processing. To increase the speed and accuracy of its AI model that identifies portraits, the company is using its own database to train the SOTA AI model. It helps to further compress these models making them suitable for smaller devices like smartphones. OPPO claims that the entire portrait recognition process can be performed in less than a millisecond, which is extremely important for a seamless and instantaneous videography experience.

     

    • Portrait Processing: When you look at its real-life applications, the Portrait Perception Engine forms the base of OPPO’s FDF Portrait Video System. The final portrait processing algorithm optimizes the video in three aspects:
      1. Face Optimization
      2. Motion Tracking
      3. Instance Segmentation
      

      

      

      

      Thanks to the Portrait Perception Engine which includes portrait identification and portrait processing, OPPO’s smartphones with the FDF Portrait Video system can:

      • Accurately identify features of every person and their faces even if the subjects are side-on to the camera or are partially obscured.
      • Apply AI Highlight Video optimizations like color, brightness, or contrast adjustments to portraits.
      • Add personalized optimization to each subject in the image (instance segmentation)
      • Add special features to portraits like AI Color Portrait

It’s clear that OPPO believes that videos are going to be a very important part of smartphones in the coming years. The company’s FDF Portrait Video System is just another effort to deliver more to its users on the video front.

The company has revealed that as of September 2020, it had applied for 7,300 imaging-related patents worldwide, out of which 2,350 have already been granted.

The OPPO Reno5 Pro 5G is the latest model to feature its new FDF Portrait Video System. The phone launches in India on January 18, so we’ll know soon how much of a difference this new Engine brings to the video experience.