Canon Improves Accuracy For Recognising Moving Subjects

Canon has announced it has developed a 1.0-inch, back-illuminated stacked CMOS sensor for monitoring applications that achieves an effective pixel count of approximately 12.6 million pixels (4,152 x 3,024) and provides an industry-leading dynamic range of 148 decibels (dB).

The new sensor divides the image into 736 areas and automatically determines the best exposure settings for each area. This eliminates the need for synthesizing images, which is often necessary when performing high-dynamic-range photography in environments with significant differences in brightness, thereby reducing the amount of data processed and improving the recognition accuracy of moving subjects.

With the increasingly widespread use of monitoring cameras in recent years, there has been a corresponding growth in demand for image sensors that can capture high-quality images in environments with significant differences in brightness, such as stadium entrances and nighttime roads. Canon has developed a new sensor for such applications, and will continue to pursue development of sensors for use in a variety of fields.

The new sensor realizes a dynamic range of 148 dB—the highest-level performance in the industry among image sensors for monitoring applications. It is capable of image capture at light levels ranging from approximately 0.1 lux to approximately 2,700,000 lux. The sensor’s performance holds the potential for use in such applications as recognizing both vehicle license plates and the driver’s face at underground parking entrances during daytime, as well as combining facial recognition and background monitoring at stadium entrances.

• 1Among market for CMOS sensors used in monitoring applications. As of January 11, 2023. Based on Canon research.
• 2Dynamic range at 30 fps is 148 dB. Dynamic range at approx. 60 fps is 142 dB.

In order to produce a natural-looking image when capturing images in environments with both bright and dark areas, conventional high-dynamic-range image capture requires taking multiple separate photos under different exposure conditions and then synthesizing them into a single image. Because exposure times vary in length, this synthesis processing often results in a problem called “motion artifacts,” in which images of moving subjects are merged but do not overlap completely, resulting in a final image that is blurry. Canon’s new sensor divides the image into 736 distinct areas, each of which can automatically be set to the optimal exposure time based on brightness level. This prevents the occurrence of motion artifacts and makes possible facial recognition with greater accuracy even when scanning moving subjects. What’s more, image synthesizing is not required, thereby reducing the amount of data to be processed and enabling high-speed image capture at speeds of approximately 60 frames-per-second³ (fps) and a high pixel count of approximately 12.6 million pixels.

• 3Dynamic range at 30 fps is 148 dB. Dynamic range at approx. 60 fps is 142 dB.

Video is comprised of a series of individual still images (single frames). However, if exposure conditions for each frame is not specified within the required time for that frame, it becomes difficult to track and capture images of subjects in environments with subject to significant changes in brightness, or in scenarios where the subject is moving at high speeds. Canon’s new image sensor is equipped with multiple CPUs and dedicated processing circuitry, enabling it to quickly and simultaneously specify exposure conditions for all 736 areas within the allotted time per frame. In addition, image capture conditions can be specified according to environment and use case. Thanks to these capabilities, the sensor is expected to serve a wide variety of purposes including fast and highly accurate subject detection on roads or in train stations, as well as stadium entrances and other areas where there are commonly significant changes in brightness levels.

Example use case for new sensor

• Parking garage entrance, afternoon: With conventional cameras, vehicle’s license plate is not legible due to whiteout, while driver’s face is not visible due to crushed blacks. However, the new sensor enables recognition of both the license plate and driver’s face.
• The new sensor realizes an industry-leading high dynamic range of 148 dB, enabling image capture in environments with brightness levels ranging from approx. 0.1 lux to approx. 2,700,000 lux. For reference, 0.1 lux is equivalent to the brightness of a full moon at night, while 500,000 lux is equivalent to filaments in lightbulbs and vehicle headlights.

New sensor

Conventional high-image-quality sensor

Conventional high-image-quality sensor

Technology behind the sensor’s wide dynamic range

• With conventional sensors, in order to produce a natural-looking image when capturing images in environments with both bright and dark areas, high-dynamic-range image capture requires taking multiple separate photos under different exposure conditions and then synthesizing them into a single image. (In the diagram below, four exposure types are utilized per single frame).
• With Canon’s new sensor, optimal exposure conditions are automatically specified for each of the 736 areas, thus eliminating the need for image synthesis.

Technology behind per-area exposure

• Portion in which subject moves is detected based on discrepancies between first image (one frame prior) and second image (two frames prior). ((1) Generate movement map).
• In first image (one frame prior) brightness of subject is recognized for each area⁴ and luminance map is generated (2). After ensuring difference in brightness levels between adjacent areas are not excessive ((3) Reduce adjacent exposure discrepancy), exposure conditions are corrected based on information from movement map, and final exposure conditions are specified (4).
• Final exposure conditions (4) are applied to images for corresponding frames.
• 4Diagram below is a simplified visualization. Actual sensor is divided into 736 areas.