Will Computational Photography replace Camera Photography?

What is computational photography?

Computational photography is a method of generating and editing digital images using computer technology and image processing techniques. It helps us to better understand and control all aspects of an image, such as brightness, contrast, color, sharpness, and more. Through computational photography, we can restore, improve, enhance, and composite images to make them more beautiful, clear, and useful. Computational photography has a wide range of applications, such as digital art, medical imaging, security surveillance, virtual reality, and more.

Over the years, computational photography has gained more and more attention as smartphones have improved their ability to take pictures. Because the size of DSLR cameras is relatively large, the competition of cameras is focused on the design of telephoto lens modules, sensor size, etc. Traditional professional cameras are actually less concerned about image processing.

Smartphones are limited by the size of the design, and the image quality formed by CMOS is not as good as DSLR cameras. In order to compensate for the inherent lack of hardware, the image-processing algorithms of smartphones need to be continuously improved, thus promoting the development of the interdisciplinary discipline of computational photography.

Techniques of computational photography

Most smartphone manufacturers now use technologies related to computational photography, such as super scoring, noise reduction, HDR, demosaic, white balance, depth estimation, portrait bokeh, etc., which are all related to computational photography.

Below are representative techniques of computational photography:

High Dynamic Range (HDR)Takes multiple photos of the same scene with different exposure settings and combines them to produce an image with greater dynamic range and detail.
Super ResolutionUses multiple low-resolution images of the same scene to create a high-resolution image with greater detail and sharpness.
Night SightEnhances low-light photography by taking multiple shots and combining them to reduce noise and increase brightness and detail.
Portrait ModeUses machine learning and depth sensors to isolate the subject from the background, creating a blurred background effect.
Live PhotosCaptures a few seconds of video before and after taking a photo to create a short, looping animation.
Slow-motionRecords video at a higher frame rate and plays it back at a slower speed, creating a dramatic effect of slowing down time.
Bokeh EffectBlurs the background of a photo to create an aesthetic, professional look by adjusting the depth of field.

The principle of computational photography can be described in more detail in the following image:

Principles of computational photography
  • Acquisition of images: The camera acquires images of the scene through the lens. These images can be individual images or sequences of consecutive images, depending on the computational photography technique used.
  • Image pre-processing: This step usually includes processing of the images such as denoising, color correction, and white balance adjustment, as well as transformations of the images such as rotation, scaling and cropping.
  • Special effects: In this step, computational photography techniques apply special algorithms and techniques to achieve different effects, such as super-resolution, HDR, and portrait bokeh. These effects usually require more advanced processing of the image, for example by merging multiple images, using image depth information, applying machine learning algorithms, etc.
  • Compositing images: In this step, computational photography techniques composite the processed images into the final output. For example, in portrait bokeh techniques, the foreground and background can be processed separately and composited into a single image at the end.
  • Output image: The final processed image is output to a display device, such as a cell phone, TV, or computer screen. The output image can be a standard image format, such as JPEG or PNG, or a special format, such as GIF or Dynamic WebP.

Representative phone brands that use computational photography

Apple iPhone

Apple is a pioneer in the field of computational photography, and its iPhone series phones have been equipped with a variety of computational photography technologies, some of which are represented below.

iPhone night mode
  • Smart HDR: Smart HDR is an Apple-exclusive technology that automatically detects and merges multiple photos to reveal a wider dynamic range and more detail. Smart HDR technology enables photos to look more realistic and natural and results in better photos in different lighting conditions.
  • Depth Detection Camera: Apple’s Depth Detection Camera is a camera that uses infrared light and lasers to detect depth. It can accurately detect the distance of objects in a photo and provide users with the ability to bokeh the background and improved portrait mode.
  • Portrait Mode: Portrait Mode is a computational photography technology from Apple that automatically defocuses the background when taking pictures of people, thereby making them stand out more. Portrait mode is also able to provide a variety of bokeh effects, allowing users to choose the bokeh effect that best suits their needs in different scenarios.
  • Night Mode: Night Mode is a computational photography technology from Apple that allows for better photos in low-light conditions. Night Mode uses the technique of composing multiple photos to achieve higher exposure and less noise, making photos taken at night clearer, more natural, and more realistic.

Google Pixel Series

Google’s computational photography technology, known as “Google Computational Photography,” uses a variety of computer vision algorithms to improve the quality of photos taken by cell phone cameras.

Pixel 6 new HDR algorithm

The following are some of the typical computational photography techniques used by Google.

  • HDR+: HDR+ is a high dynamic range photo technology developed by Google that enables cell phones to take photos with more accurate details in light and dark and more natural colors. It captures more detail and a wider dynamic range by using multiple photos to combine them into a single image.
  • Night Sight: Night Sight technology is a nighttime technology developed by Google to capture high-quality photos in low-light conditions. It reduces noise and blur by using multiple photos and uses machine learning techniques to enhance color and detail.
  • Portrait Mode: Portrait Mode technology is a portrait mode developed by Google that makes people stand out by defocusing the background. It does this by using depth information to determine the position of the person and machine learning algorithms to segment the image to achieve a bokeh background.
  • Super Res Zoom: Super Res Zoom technology is a high-definition zoom technology developed by Google that allows photos taken by cell phone cameras to maintain higher detail and clarity when zoomed in. It creates high-resolution images by combining multiple photos and using machine learning algorithms to improve sharpness and detail.

Huawei P series

Huawei is the first company to use NPUs (embedded neural network processors) in cell phones and integrate them into the CPU. the Mate 30, which already uses Huawei’s self-developed Da Vinci NPUs, incorporates AI capabilities for computational photography, and Huawei’s cell phones have made a qualitative leap forward in photography.

XD Fusion

Huawei’s flagship Mate 40 Pro last year, for example, creatively used the ISP+NPU fusion architecture to achieve stunning photo performance in low light, with highlights not exposed and dark areas retaining detail, and real-time 4K HDR for video, thanks to the Kirin 9000’s impressive computing power.

Huawei’s NPU strength is further demonstrated by the XD Fusion image engine, the first in the P40 series, which enhances image detail and improves overall image quality through multi-shot high-fidelity information fusion, semantic understanding and segmentation, and pixel-level image quality processing. XD Fusion, the “Ultra HD Image Engine”, enhances image detail and improves overall image quality through multi-lens high-fidelity information fusion, semantic understanding and segmentation, and pixel-level image processing.

Will smartphone photography replace camera photography?

Computational photography will be the next direction of cell phone photography. In the past, limited by size and relying on hardware alone, it was very difficult for smartphones to challenge professional cameras in terms of photography. But with the advent of computational photography, does it mean that it is possible for smartphones to replace, or at least rival, cameras?

While computational photography lowers the threshold for taking pictures, making it an easy and simple task, professional cameras are moving toward a more professional and segmented market, pursuing to record photo information as richly as possible.

Therefore, we can say that computational photography and cameras belong to different tracks. The former allows ordinary people to take a photo above the pass line without mastering professional photography knowledge, while the latter gives professional photographers or photography enthusiasts enough free space to play.

However, it is undeniable that the development of computational photography has been able to bridge the gap between cell phone camera sensor size and professional cameras to some extent, and the dynamic range of many cell phones’ HDR+ mode even exceeds that of professional cameras. We can be sure that under the leadership of Apple, Huawei, and other large companies, computational photography has become a new trend in the field of imaging and represents the new development direction of cell phone photography.

With the rapid evolution of AI and algorithms, computational photography may overturn people’s perceptions. In the coming years, smartphones equipped with computational photography capabilities will bring the world stunning photo and photography effects. It is not impossible that these photos and videos, with the addition of computational photography, will be comparable to professional cameras and even surpass them. Just like Open AI’s Chat GPT, it makes us feel that the development of technology is coming so fast, but in line with the development of technology.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top