{"id":357043,"date":"2024-05-20T23:42:21","date_gmt":"2024-05-20T23:42:21","guid":{"rendered":"http:\/\/savepearlharbor.com\/?p=357043"},"modified":"-0001-11-30T00:00:00","modified_gmt":"-0001-11-29T21:00:00","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=357043","title":{"rendered":"A (more) accurate camera sensor dynamic range measurement<\/span>"},"content":{"rendered":"
<\/div>\n
\n
\n
\n
\n

Introduction<\/h2>\n

Hello, everyone! In this post, let’s talk about how to (more) accurately measure the dynamic range of a camera sensor and what can be done with these measurements.<\/p>\n

Of course, I am not an expert in computer vision, a programmer or a statistician, so please feel free to correct me in the comments if I make mistakes in this post. Here my interest was primarily focused on everyday and practical tasks, such as photography, but I believe the results may also be useful to computer vision professionals.<\/p>\n\n

Update: mistakes were found here after publishing this post on the DPReview forum. I think it is important to note that here, and here is the link to the thread itself<\/a><\/p>\n\n

Existing Problem<\/h2>\n

The dynamic range of modern image sensors is limited; we cannot capture detail in very bright highlights and very dark shadows simultaneously without degrading the image quality due to noise. In general, the noise, in my opinion, is the most limiting factor for image quality today. The darker the scene, the more noise there is in the shadows, which when the image becomes darker increases in amplitude more and more, progressively \u201ceating away\u201d detail, especially that of low\u2011contrast objects.<\/p>\n

Why Measure Dynamic Range<\/h2>\n

In general, accurate dynamic range measurements can be used by both photographers (not only enthusiasts but also professionals) and computer vision specialists to evaluate some equipment.<\/p>\n

Firstly, measurements help to answer the question of which camera or sensor to choose for the future tasks or whether an upgrade is worth it\u00a0\u2014 to what extent the advantage will be noticeable. Secondly, through measurements, one can analyze an existing camera or sensor to find an optimal exposure and post\u2011processing strategy and also use the data for further research.<\/p>\n

How Sensor Dynamic Range is Currently Measured<\/h2>\n

In this chapter, I am using the method and data from Bill Claff’s website, <\/em>Photons to Photos<\/em><\/a>.<\/em><\/p>\n

The current method of measuring dynamic range<\/a> begins with the plotting of a graph of the signal-to-noise ratio for a specific ISO at different exposure values, as illustrated below:<\/p>\n

<\/figure>\n

In general, there are engineering and so-called photographic dynamic ranges. Here is what Mr. Claff writes regarding the first:<\/p>\n

\n

The low endpoint for Engineering Dynamic Range is determined by where the SNR curve crosses the value of 1.<\/p>\n

Here is an extreme close-up of that area of the Photon Transfer Curve:<\/p>\n<\/blockquote>\n

<\/figure>\n
\n

The ISO 100\u00a0line crosses 0\u00a0(the log2<\/sub>(1)) at 1.63\u00a0EV.<\/p>\n

So Engineering Dynamic Range is 14.00\u00a0EV\u00a0\u2014 1.63\u00a0EV = 12.37\u00a0EV.<\/p>\n

Or, if we use the White Level (from White Level<\/u><\/a>), (14.00\u00a0EV\u00a0\u2014 0.11\u00a0EV)\u00a0\u2014 1.63\u00a0EV = 12.26\u00a0EV.<\/p>\n<\/blockquote>\n

For comparison, here is the definition of the so-called photographic dynamic range:<\/p>\n

\n

My definition of Photographic Dynamic Range is a low endpoint with an SNR of 20\u00a0when adjusted for the appropriate Circle Of Confusion (COC) for the sensor.<\/p>\n

For the D300\u00a0the SNR values on this curve is for a 5.5\u00a0micron square photosite.<\/p>\n

To correct for a COC of.022mm we are looking for a log2<\/sub> SNR value of 2.49.<\/p>\n

Here is an close\u2011up of the area of the curve:<\/p>\n<\/blockquote>\n

<\/figure>\n
\n

Note that the ISO 100\u00a0crosses 2.49\u00a0at 5.00\u00a0EV.<\/p>\n

So Photographic Dynamic Range at ISO 100\u00a0is 14.00\u00a0EV\u00a0\u2014 5.00\u00a0EV = 9.00\u00a0EV.<\/p>\n

Or, if we use the White Level (from White Level<\/u><\/a>), (14.00\u00a0EV\u00a0\u2014 0.11\u00a0EV)\u00a0\u2014 5.00\u00a0EV = 8.89\u00a0EV.<\/p>\n<\/blockquote>\n

The graphs obtained after such measurements can be found here\u00a0\u2014 Photographic Dynamic Range versus ISO Setting<\/a>.<\/p>\n

For example, here is what the data looks like for the Pentax K-1:<\/p>\n

<\/figure>\n

However, choosing an exposure strategy based solely on this graph is not the best idea because some cameras are so\u2011called ISO\u2011less. This means that increasing the ISO in the camera before capturing the image will not improve the noisy shadow detail but will only destroy highlight one.<\/p>\n

Therefore, another graph is needed as well\u00a0\u2014 such as this one\u00a0\u2014 that shows shadow improvement versus ISO setting<\/a>:<\/p>\n

<\/figure>\n

Indeed, the chosen camera turned out to be ISO\u2011less, meaning that increasing ISO beyond 100\u00a0will not improve the shadow detail.<\/p>\n

For clarity, here are two frames from the same camera by DPReview<\/a>\u00a0\u2014 one at ISO 3200\u00a0with exposure increased by one stop, and another at ISO 100\u00a0with exposure increased by 6\u00a0stops:<\/p>\n

<\/figure>\n

But this doesn’t apply to all cameras. For example, here is the Canon 6D, increasing ISO indeed improves the shadows:<\/p>\n

<\/figure>\n

One cannot say that the camera is necessarily better or worse solely because of this characteristic, but it does influence the choice of an optimal exposure strategy during shooting.<\/p>\n

Shortcoming of Such Measurement Method<\/h2>\n

The problem with such a classical approach arises when a camera uses noise reduction. Yes, these are still RAW files, it’s just that noise reduction can be applied even before demosaicing. As seen in the graph below, the dynamic range sharply increases at ISO 640:<\/p>\n

<\/figure>\n

This does not necessarily mean that the image quality has actually improved. On the contrary, noise reduction could have destroyed some fine detail.<\/p>\n

On the Shadow Improvement chart, the advantage is even more apparent\u00a0\u2014 reaching almost 2\u00a0stops:<\/p>\n

<\/figure>\n

In reality, as you understand, things won’t be as great. This is the problem with the classical measurement approach\u00a0\u2014 DSP algorithms, such as noise reduction and detail recovery, can only be detected (which is what the triangles pointing down on the graph represent), but their actual impact on image quality cannot be accurately assessed.<\/p>\n

A more advanced method of dynamic range measurement<\/h2>\n

You are likely familiar with the libraw<\/a> library, and perhaps even use it. The developer of this library, Alexey Tutubalin (\u0410\u043b\u0435\u043a\u0441\u0435\u0439 \u0422\u0443\u0442\u0443\u0431\u0430\u043b\u0438\u043d), proposed<\/a> his way of measuring dynamic range back in 2017. He pondered the question of how much real resolution remains in an image due to the influence of noise. For this, Alexey used a low\u2011contrast rotated target to test the resolution. The rotation was introduced to challenge the internal DSP algorithms in the camera, in case such algorithms were present. It looked like this:<\/p>\n

<\/figure>\n
<\/figure>\n

After a series of shots and a semi-automatic analysis of results, this interesting graph was obtained:<\/p>\n

<\/figure>\n

In my opinion, the data turned out to be incredibly useful and illustrative, but I wanted to replicate the results using fully automated measurements and metrics, ensuring that the developed method is accessible to everyone in home conditions.<\/p>\n

Proposed Solution<\/h2>\n

I believe that testing resolution using special targets might lead to potential cheating attempts by the camera. The target should have resolution, i.e., detail, higher than that of a sensor. Therefore, we need a target with a large number of small details that cannot be tampered with\u00a0\u2014 essentially just noise. However, we cannot use ordinary noise, as it would decrease as the camera moves away from the target. So, we need a special, scale\u2011invariant pink noise.<\/p>\n

Imatest sells such targets for $330:<\/p>\n

<\/figure>\n

I just printed such a square on a regular printer on an A4 sheet.<\/p>\n

Unfortunately, I don’t have a standalone camera, so the testing was done on a regular phone with a SONY IMX686\u00a0sensor. I used the FreeDCam<\/a> app, controlled it from a PC using scrcpy<\/a> to avoid moving the phone between frames (in the case of a real camera, it would need to be placed on a tripod and remotely controlled from the PC). After that, I took series of shots of the resulting target at different ISOs with different exposures, from 1\u00a0to 12\u00a0stops below sensor saturation (careful readers will notice that, in fact, the lines on the graph are cut off before 12\u00a0stops at ISO 3200\u00a0and 6400\u00a0\u2014 FreeDCam simply does not have such short exposure time settings). The captured frames were converted into linear 16-bit TIFF without demosaicing but with white balance applied through channel amplification. Then they were analyzed relative to the reference frame\u00a0\u2014 10\u00a0averaged frames of the target taken at one stop below sensor saturation\u00a0\u2014 equivalent to a frame at ISO 5\u00a0with ideal exposure.<\/p>\n

The resulting graph shows the signal\u2011to\u2011noise ratio:<\/p>\n

<\/figure>\n

Unlike Mr. Claff’s graph, this graph, if not entirely, at least partially, is resistant to DSP\u00a0\u2014 please pay attention to the lower values at ISO 3200\u00a0and 6400, which are likely signs of noise reduction. Although the graph is not entirely free from its influence, there is no improvement relative to the upper values. Therefore, the obtained data has higher quality than the one that would be obtained by the standard method. In fact, I tested my phone without actual pink noise, so using real pink noise (which can be generated, for example, using functions from here<\/a>) will likely only improve the result.<\/p>\n

Instead of plotting graphs the classical way, I decided to go further and plot a graph similar to Alexey Tutubalin’s, again using data obtained fully automatically and, as I hope, with greater accuracy. For this, I wrote and used this numpy function:<\/p>\n

def find_real_resolution(image_resolution, ground_truth, noisy_image):      stock_contrast = np.max(ground_truth) \/ np.min(ground_truth)      noise = np.mean(np.abs(noisy_image - ground_truth))      real_contrast = (np.max(ground_truth) - noise) \/ (np.min(ground_truth) + noise)      contrast_change_coefficient = stock_contrast \/ real_contrast      real_resolution = image_resolution \/ contrast_change_coefficient      return real_resolution<\/code><\/pre>\n
\nMore about it:<\/summary>\n
\n
image_resolution<\/code> — the resolution of the entire frame (as in this case only a cropped part of 100×100 pixels was tested), ground_truth<\/code> — the reference image, noisy_image<\/code> — the tested image from the series. As you understand, all frames in the series just need to be passed through this function. But what does it do?<\/p>\n
Let’s take another look at the resolution test target:<\/p>\n
<\/figure>\n
In the worst case, noise will decrease the value of the white background and increase the value of the black lines, leading to a reduction in contrast, and consequently a drop in the resolution for the given target. Therefore, I first find the \u201caverage noise\u201d of each frame (line 2\u00a0inside the function), then subtract it from the white point of the reference image and add it to the black point of the same image (line 3). Then, I compare (line 4) the original (line 1) and the new (line 3) contrast, after which I simply divide (line 5) the entire frame’s resolution by this coefficient to obtain the actual resolution.<\/p>\n<\/div>\n<\/details>\n
The resulting graph looks like this:<\/p>\n
<\/figure>\n
Values below 0 are shown for illustrative purposes only.<\/em><\/p>\n
As mentioned earlier, such a graph can be used to compare two sensors to understand, for example, which one to choose for specific lighting conditions. Alternatively, one can investigate the graph of a single sensor to determine the optimal exposure strategy and to develop the best post\u2011processing algorithms, such as noise reduction.<\/p>\n
Further Research<\/h2>\n
For a further work I would like to note the following opportunities:<\/p>\n