Color balance

This article is about the process applied to still images. For the equivalent process applied to video, see Color grading.

In photography and image processing, color balance is the global adjustment of the intensities of the colors (typically red, green, and blue primary colors). An important goal of this adjustment is to render specific colors – particularly neutral colors like white or grey – correctly. Hence, the general method is sometimes called gray balance, neutral balance, or white balance. Color balance changes the overall mixture of colors in an image and is used for color correction. Generalized versions of color balance are used to correct colors other than neutrals or to deliberately change them for effect. White balance is one of the most common kinds of balancing, and is when colors are adjusted to make a white object (such as a piece of paper or a wall) appear white and not a shade of any other colour.

The left half shows the photo as it came from the digital camera. The right half shows the photo adjusted to make a gray surface neutral in the same light.

Image data acquired by sensors – either film or electronic image sensors – must be transformed from the acquired values to new values that are appropriate for color reproduction or display. Several aspects of the acquisition and display process make such color correction essential – including that the acquisition sensors do not match the sensors in the human eye, that the properties of the display medium must be accounted for, and that the ambient viewing conditions of the acquisition differ from the display viewing conditions.

The color balance operations in popular image editing applications usually operate directly on the red, green, and blue channel pixel values,^[1][2] without respect to any color sensing or reproduction model. In film photography, color balance is typically achieved by using color correction filters over the lights or on the camera lens.^[3]

Generalized color balance

 

Example of color balancing

Sometimes the adjustment to keep neutrals neutral is called white balance, and the phrase color balance refers to the adjustment that in addition makes other colors in a displayed image appear to have the same general appearance as the colors in an original scene.^[4] It is particularly important that neutral (gray, neutral, white) colors in a scene appear neutral in the reproduction.^[5]

Psychological color balance

Humans relate to flesh tones more critically than other colors. Trees, grass and sky can all be off without concern, but if human flesh tones are 'off' then the human subject can look sick or dead. To address this critical color balance issue, the tri-color primaries themselves are formulated to not balance as a true neutral color. The purpose of this color primary imbalance is to more faithfully reproduce the flesh tones through the entire brightness range.

Illuminant estimation and adaptation

 

A seascape photograph at Clifton Beach, South Arm, Tasmania, Australia. The white balance has been adjusted towards the warm side for creative effect.

 

Photograph of a ColorChecker as a reference shot for color balance adjustments.

 

Two photos of a high-rise building shot within a minute of each other with an entry-level point-and-shoot camera. Left photo shows a "normal", more accurate color balance, while the right side shows a "vivid" color balance, in-camera effects and no post-production besides black background.

 

Comparison of color versions (raw, natural, white balance) of Mount Sharp (Aeolis Mons) on Mars

 

A white-balanced image of Mount Sharp (Aeolis Mons) on Mars

Most digital cameras have means to select color correction based on the type of scene lighting, using either manual lighting selection, automatic white balance, or custom white balance.^[6] The algorithms for these processes perform generalized chromatic adaptation.

Many methods exist for color balancing. Setting a button on a camera is a way for the user to indicate to the processor the nature of the scene lighting. Another option on some cameras is a button which one may press when the camera is pointed at a gray card or other neutral colored object. This captures an image of the ambient light, which enables a digital camera to set the correct color balance for that light.

There is a large literature on how one might estimate the ambient lighting from the camera data and then use this information to transform the image data. A variety of algorithms have been proposed, and the quality of these has been debated. A few examples and examination of the references therein will lead the reader to many others. Examples are Retinex, an artificial neural network^[7] or a Bayesian method.^[8]

Chromatic colors

Color balancing an image affects not only the neutrals, but other colors as well. An image that is not color balanced is said to have a color cast, as everything in the image appears to have been shifted towards one color.^{[9][page needed]} Color balancing may be thought in terms of removing this color cast.

Color balance is also related to color constancy. Algorithms and techniques used to attain color constancy are frequently used for color balancing, as well. Color constancy is, in turn, related to chromatic adaptation. Conceptually, color balancing consists of two steps: first, determining the illuminant under which an image was captured; and second, scaling the components (e.g., R, G, and B) of the image or otherwise transforming the components so they conform to the viewing illuminant.

Viggiano found that white balancing in the camera's native RGB color model tended to produce less color inconstancy (i.e., less distortion of the colors) than in monitor RGB for over 4000 hypothetical sets of camera sensitivities.^[10] This difference typically amounted to a factor of more than two in favor of camera RGB. This means that it is advantageous to get color balance right at the time an image is captured, rather than edit later on a monitor. If one must color balance later, balancing the raw image data will tend to produce less distortion of chromatic colors than balancing in monitor RGB.

Mathematics of color balance

Color balancing is sometimes performed on a three-component image (e.g., RGB) using a 3x3 matrix. This type of transformation is appropriate if the image was captured using the wrong white balance setting on a digital camera, or through a color filter.

Scaling monitor R, G, and B

In principle, one wants to scale all relative luminances in an image so that objects which are believed to be neutral appear so. If, say, a surface with ${\displaystyle R=240}$  was believed to be a white object, and if 255 is the count which corresponds to white, one could multiply all red values by 255/240. Doing analogously for green and blue would result, at least in theory, in a color balanced image. In this type of transformation the 3x3 matrix is a diagonal matrix.

${\displaystyle \left[{\begin{array}{c}R\\G\\B\end{array}}\right]=\left[{\begin{array}{ccc}255/R'_{w}&0&0\\0&255/G'_{w}&0\\0&0&255/B'_{w}\end{array}}\right]\left[{\begin{array}{c}R'\\G'\\B'\end{array}}\right]}$ 

where ${\displaystyle R}$ , ${\displaystyle G}$ , and ${\displaystyle B}$  are the color balanced red, green, and blue components of a pixel in the image; ${\displaystyle R'}$ , ${\displaystyle G'}$ , and ${\displaystyle B'}$  are the red, green, and blue components of the image before color balancing, and ${\displaystyle R'_{w}}$ , ${\displaystyle G'_{w}}$ , and ${\displaystyle B'_{w}}$  are the red, green, and blue components of a pixel which is believed to be a white surface in the image before color balancing. This is a simple scaling of the red, green, and blue channels, and is why color balance tools in Photoshop have a white eyedropper tool. It has been demonstrated that performing the white balancing in the phosphor set assumed by sRGB tends to produce large errors in chromatic colors, even though it can render the neutral surfaces perfectly neutral.^[10]

Scaling X, Y, Z

If the image may be transformed into CIE XYZ tristimulus values, the color balancing may be performed there. This has been termed a “wrong von Kries” transformation.^[11][12] Although it has been demonstrated to offer usually poorer results than balancing in monitor RGB, it is mentioned here as a bridge to other things. Mathematically, one computes:

${\displaystyle \left[{\begin{array}{c}X\\Y\\Z\end{array}}\right]=\left[{\begin{array}{ccc}X_{w}/X'_{w}&0&0\\0&Y_{w}/Y'_{w}&0\\0&0&Z_{w}/Z'_{w}\end{array}}\right]\left[{\begin{array}{c}X'\\Y'\\Z'\end{array}}\right]}$ 

where ${\displaystyle X}$ , ${\displaystyle Y}$ , and ${\displaystyle Z}$  are the color-balanced tristimulus values; ${\displaystyle X_{w}}$ , ${\displaystyle Y_{w}}$ , and ${\displaystyle Z_{w}}$  are the tristimulus values of the viewing illuminant (the white point to which the image is being transformed to conform to); ${\displaystyle X'_{w}}$ , ${\displaystyle Y'_{w}}$ , and ${\displaystyle Z'_{w}}$  are the tristimulus values of an object believed to be white in the un-color-balanced image, and ${\displaystyle X'}$ , ${\displaystyle Y'}$ , and ${\displaystyle Z'}$  are the tristimulus values of a pixel in the un-color-balanced image. If the tristimulus values of the monitor primaries are in a matrix ${\displaystyle \mathbf {P} }$  so that:

${\displaystyle \left[{\begin{array}{c}X\\Y\\Z\end{array}}\right]=\mathbf {P} \left[{\begin{array}{c}L_{R}\\L_{G}\\L_{B}\end{array}}\right]}$ 

where ${\displaystyle L_{R}}$ , ${\displaystyle L_{G}}$ , and ${\displaystyle L_{B}}$  are the un-gamma corrected monitor RGB, one may use:

${\displaystyle \left[{\begin{array}{c}L_{R}\\L_{G}\\L_{B}\end{array}}\right]=\mathbf {P^{-1}} \left[{\begin{array}{ccc}X_{w}/X'_{w}&0&0\\0&Y_{w}/Y'_{w}&0\\0&0&Z_{w}/Z'_{w}\end{array}}\right]\mathbf {P} \left[{\begin{array}{c}L_{R'}\\L_{G'}\\L_{B'}\end{array}}\right]}$ 

Von Kries's method

Johannes von Kries, whose theory of rods and three color-sensitive cone types in the retina has survived as the dominant explanation of color sensation for over 100 years, motivated the method of converting color to the LMS color space, representing the effective stimuli for the Long-, Medium-, and Short-wavelength cone types that are modeled as adapting independently. A 3x3 matrix converts RGB or XYZ to LMS, and then the three LMS primary values are scaled to balance the neutral; the color can then be converted back to the desired final color space:^[13]

${\displaystyle \left[{\begin{array}{c}L\\M\\S\end{array}}\right]=\left[{\begin{array}{ccc}1/L'_{w}&0&0\\0&1/M'_{w}&0\\0&0&1/S'_{w}\end{array}}\right]\left[{\begin{array}{c}L'\\M'\\S'\end{array}}\right]}$ 

where ${\displaystyle L}$ , ${\displaystyle M}$ , and ${\displaystyle S}$  are the color-balanced LMS cone tristimulus values; ${\displaystyle L'_{w}}$ , ${\displaystyle M'_{w}}$ , and ${\displaystyle S'_{w}}$  are the tristimulus values of an object believed to be white in the un-color-balanced image, and ${\displaystyle L'}$ , ${\displaystyle M'}$ , and ${\displaystyle S'}$  are the tristimulus values of a pixel in the un-color-balanced image.

Matrices to convert to LMS space were not specified by von Kries, but can be derived from CIE color matching functions and LMS color matching functions when the latter are specified; matrices can also be found in reference books.^[13]

Scaling camera RGB

By Viggiano's measure, and using his model of gaussian camera spectral sensitivities, most camera RGB spaces performed better than either monitor RGB or XYZ.^[10] If the camera's raw RGB values are known, one may use the 3x3 diagonal matrix:

${\displaystyle \left[{\begin{array}{c}R\\G\\B\end{array}}\right]=\left[{\begin{array}{ccc}255/R'_{w}&0&0\\0&255/G'_{w}&0\\0&0&255/B'_{w}\end{array}}\right]\left[{\begin{array}{c}R'\\G'\\B'\end{array}}\right]}$ 

and then convert to a working RGB space such as sRGB or Adobe RGB after balancing.

Preferred chromatic adaptation spaces

Comparisons of images balanced by diagonal transforms in a number of different RGB spaces have identified several such spaces that work better than others, and better than camera or monitor spaces, for chromatic adaptation, as measured by several color appearance models; the systems that performed statistically as well as the best on the majority of the image test sets used were the "Sharp", "Bradford", "CMCCAT", and "ROMM" spaces.^[14]

General illuminant adaptation

The best color matrix for adapting to a change in illuminant is not necessarily a diagonal matrix in a fixed color space. It has long been known that if the space of illuminants can be described as a linear model with N basis terms, the proper color transformation will be the weighted sum of N fixed linear transformations, not necessarily consistently diagonalizable.^[15]

Examples

 

Neutral light

 

Warm light

 

Cold light

Comparison of resulted colors as shot by the digital camera for different light qualities (color temperature): Neutral, Warm and Cold.^[16]

 

Setting: As shot

 

Setting: Cloudy

 

Setting: Tungsten

Example of different white balance settings on digital camera for Neutral light.^[16]

See also

• Color cast
• Color temperature
• Gamma correction
• White point

References

1. Phyllis Davis (2000). The Gimp for Linux and Unix. Peachpit Press. p. 134. ISBN 978-0-201-70253-8.
2. Adobe Creative Team (2000). Adobe Photoshop 6.0. Adobe Press. p. 278. ISBN 978-0-201-71016-8.^{[need quotation to verify]}
3. Blain Brown (2002). Cinematography: Theory and Practice : Imagemaking for Cinematographers, Directors, and Videographers. Focal Press. p. 170. ISBN 978-0-240-80500-9.
4. Hsien-Che Lee (2005). Introduction to Color Imaging Science. Cambridge University Press. p. 450. ISBN 978-0-521-84388-1.
5. White Balance. Nikon Digital. Retrieved October 12, 2016.
6. Afifi, Mahmoud; Price, Brian; Cohen, Scott; Brown, Michael S (2019). "When Color Constancy Goes Wrong: Correcting Improperly White-Balanced Images" (PDF). Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition: 1535–1544. doi:10.1109/cvpr.2019.00163. ISBN 978-1-7281-3293-8. S2CID 196195956.
7. Brian Funt, Vlad Cardei, and Kobus Barnard, "Learning color constancy," in Proceedings of the Fourth IS&T/SID Color Imaging Conference, pp. 58–60 (1996).
8. Graham Finlayson; Paul M. Hubel; Steven Hordley (November 2001). "Color by correlation: a simple, unifying framework for color constancy" (PDF). IEEE Transactions on Pattern Analysis and Machine Intelligence. 23 (11): 1209–21. CiteSeerX 10.1.1.133.2101. doi:10.1109/34.969113.
9. John A C Yule, Principles of Color Reproduction. New York: Wiley, 1967.
10. Viggiano, J A Stephen (2004). "Comparison of the accuracy of different white-balancing options as quantified by their color constancy". In Blouke, Morley M; Sampat, Nitin; Motta, Ricardo J (eds.). Sensors and Camera Systems for Scientific, Industrial, and Digital Photography Applications V. Vol. 5301. pp. 323–333. doi:10.1117/12.524922. S2CID 8971750.
11. Heinz Terstiege (1972). "Chromatic adaptation: a state-of-the-art report". Journal of Color Appearance. 1 (4): 19–23 (cont. 40).
12. Mark D Fairchild, Color Appearance Models. Reading, MA: Addison-Wesley, 1998.
13. Gaurav Sharma (2003). Digital Color Imaging Handbook. CRC Press. p. 153. ISBN 978-0-8493-0900-7.
14. Sabine Süsstrunk; Jack Holm; Graham D. Finlayson (January 2001). . IS&T/SPIE Electronic Imaging. 4300. doi:10.1117/12.410788. S2CID 8140548. Archived from the original on 2006-10-18. Retrieved 2009-03-20.
15. Laurence T. Maloney; Brain A. Wandell (1987). "Color constancy: a method for recovering surface spectral reflectance". In Martin A. Fischler; Oscar Firschein (eds.). Readings in Computer Vision. Morgan-Kaufmann. ISBN 978-0-934613-33-0.
16. "photoskop: Interactive Photography Lessons". April 25, 2015.

External links

• White Balance - Intro at nikondigital.org
• photoskop: Interactive Photography Lessons - Interactive White Balance
• - Tutorial
• Affine color balance with saturation, with code and on-line demonstration
• Getting the White Balance Right for Neutral Colors - Photography Tutorial