Light field camera

A light field camera, also known as a plenoptic camera, is a camera that captures information about the light field emanating from a scene; that is, the intensity of light in a scene, and also the precise direction that the light rays are traveling in space. This contrasts with conventional cameras, which record only light intensity at various wavelengths.

Lytro Illum 2nd generation light field camera

Front and back of a Lytro, the first consumer light field camera, showing the front lens and LCD touchscreen

One type uses an array of micro-lenses placed in front of an otherwise conventional image sensor to sense intensity, color, and directional information. Multi-camera arrays are another type. Holograms are a type of film-based light field image.

History

Early research

The first light field camera was proposed by Gabriel Lippmann in 1908. He called his concept "integral photography". Lippmann's experimental results included crude integral photographs made by using a plastic sheet embossed with a regular array of microlenses, or by partially embedding small glass beads, closely packed in a random pattern, into the surface of the photographic emulsion.

In 1992, Adelson and Wang proposed a design that reduced the correspondence problem in stereo matching.^[1] To achieve this, an array of microlenses is placed at the focal plane of the camera main lens. The image sensor is positioned slightly behind the microlenses. Using such images, the displacement of image parts that are not in focus can be analyzed and depth information can be extracted.

Standard plenoptic camera

 

This demonstrates the capability of changing the focal distance and depth of field after a photo is taken - Near focus (top), Far focus (middle), Full depth of field (bottom) - using the Lytro Illum light field camera software

The "standard plenoptic camera" is a mathematical model used by researchers to compare designs. By definition it has microlenses placed one focal length away from the image plane of a sensor.^[2][3][4] In 2004, a team at Stanford University Computer Graphics Laboratory used a 16-megapixel camera to demonstrate that pictures can be refocused after they are taken. The system used a 90,000-microlens array, yielding a resolution of 90 kilopixels.^[2] Research has shown that its maximum baseline is confined to the main lens entrance pupil size which is small relative to stereoscopic setups.^[1][5] This implies that the "standard plenoptic camera" may be intended for close-range applications as it exhibits increased depth resolution at distances that can be metrically predicted based on the camera's parameters.^[6]

Focused plenoptic camera

Lumsdaine and Georgiev described a design in which the microlens array can be positioned before or behind the focal plane of the main lens. This modification samples the light field in a way that trades angular resolution for higher spatial resolution. With this design, images can be refocused with a much higher spatial resolution than images from a standard plenoptic camera. However, the lower angular resolution can introduce aliasing artifacts.

Coded aperture camera

A design that used a low-cost printed film mask instead of a microlens array was proposed in 2007.^[7] This design reduces the chromatic aberrations and loss of boundary pixels seen in microlens arrays, and allows greater spatial resolution. However, the mask-based design reduces the amount of light that reaches the image sensor, reducing brightness.

Features

Features include:

• Variable depth of field and "refocusing": Lytro's "Focus Spread" feature allows the depth of field (depth of focus) of a 2 dimensional representation of a Lytro image to be adjusted after a picture has been taken.^[8] Instead of setting the focus at a particular distance, "Focus Spread" allows more of a 2D image to be in focus. In some cases this may be the entire 2D image field. Users also are able to "refocus" 2D images at particular distances for artistic effects. The Illum allows the "refocus-able" and "Focus Spreadable" range to be selected using the optical focus and zoom rings on the lens. The Illum also features "focus bracketing" to extend the refocusable range by capturing 3 or 5 consecutive images at different depths.^[9]
• Speed: Because there is less need to focus the lens before taking a picture, a light field camera can capture images more quickly than conventional point-and-shoot digital cameras.^[10] This is an advantage in sports photography, for example, where many pictures are lost because the camera’s auto-focus system cannot precisely track a fast moving subject.
• Low-light sensitivity: The ability to adjust focus in post-processing allows the use of larger apertures than are feasible on conventional cameras, thus enabling photography in low-light environments.^[10][11]
• 3D views: Since a plenoptic camera records depth information, 3D views can be constructed in software from a single plenoptic image capture. 3D views are different from solely stereo images in this case. Stereo images may also be constructed.^[12][13]

Metalens array

In 2022, NIST announced a device with a focal range of 3 cm (1.2 in) to 1.7 km (1.1 mi). The device employed a 39x39-element titanium dioxide metalens array. Each metalens is either right- or left-circle polarized to create a different focal length. Each metalens was rectangular in shape. The light is routed separately through the shorter and longer sides of the rectangle, producing two focal points in the image. Differences among the metalenses were corrected algorithmically.^[14][15]

Manufacturers

Products

Lytro was founded by Stanford University Computer Graphics Laboratory alumnus Ren Ng to commercialize the light field camera he developed as a graduate student.^[16] Lytro's light field sensor uses an array of micro-lenses placed in front of an otherwise conventional image sensor; to sense intensity, color, and directional information.^[17] Software then uses this data to create displayable 2D or 3D images.^[18] Lytro trades maximum 2D resolution, at a given distance, for enhanced resolution at other distances. Users can convert the Lytro camera's proprietary image into a regular 2D image file, at any desired focal distance. The maximum Illum 2D resolution is 2450 × 1634 (4.0 megapixels), The 3D light field resolution is 40 "megarays".^[19] It has a maximum 2D resolution of 1080 × 1080 pixels (roughly 1.2 megapixels),^[20] Lytro ceased operations in March 2018.^[21]

Raytrix has offered several models of plenoptic cameras for industrial and scientific applications since 2010, with field of view starting from 1 megapixel.^[22][23]

d'Optron and Rebellion Photonics offer plenoptic cameras, specializing in microscopy and gas leak detection, respectively.^{[citation needed]}

Prototypes

Stanford University Computer Graphics Laboratory developed a prototype light field microscope using a microlens array similar to the one used in their light field camera. The prototype is built around a Nikon Eclipse transmitted light microscope/wide-field fluorescence microscope and standard CCD cameras. Light field capture is obtained by a module containing a microlens array and other optical components placed in the light path between the objective lens and camera, with the final multifocused image rendered using deconvolution.^[24][25][26]

A later prototype added a light field illumination system consisting of a video projector (allowing computational control of illumination) and a second microlens array in the illumination light path of the microscope. The addition of a light field illumination system both allowed for additional types of illumination (such as oblique illumination and quasi-dark-field) and correction for optical aberrations.^[25]

The Adobe light field camera is a prototype 100-megapixel camera that takes a three-dimensional photo of the scene in focus using 19 uniquely configured lenses. Each lens takes a 5.2-megapixel photo of the scene. Each image can be focused later in any way.^[27]

CAFADIS is a plenoptic camera developed by University of La Laguna (Spain).^[28] CAFADIS stands (in Spanish) for phase-distance camera, since it can be used for distance and optical wavefront estimation. From a single shot it can produce images focused at different distances, depth maps, all-in-focus images and stereo pairs. A similar optical design can be used in adaptive optics in astrophysics.

Mitsubishi Electric Research Laboratories's (MERL) light field camera^[7] is based on the principle of optical heterodyning and uses a printed film (mask) placed close to the sensor. Any hand-held camera can be converted into a light field camera using this technology by simply inserting a low-cost film on top of the sensor.^[29] A mask-based design avoids the problem of loss of resolution, since a high-resolution photo can be generated for the focused parts of the scene.

Pelican Imaging has thin multi-camera array systems intended for consumer electronics. Pelican's systems use from 4 to 16 closely spaced micro-cameras instead of a micro-lens array image sensor.^[30] Nokia invested in Pelican Imaging to produce a plenoptic camera system with 16-lens array that was expected to be implemented in Nokia smartphones in 2014.^[31] Pelican moved to designing supplementary cameras that add depth-sensing capabilities to a device's main camera, rather than stand-alone array cameras.^[32]

A collaboration between University of Bedfordshire and ARRI resulted in a custom-made plenoptic camera with a ray model for the validation of light-field geometries and real object distances.^[4][5]

In November 2021 the German-based company K|Lens^[33] announced the first light field lens available for any standard lens mount on Kickstarter. The project was canceled in January of 2022.

The modification of standard digital cameras requires little more than suitable sheets of micro-lens material, hence a number of hobbyists have produced cameras whose images can be processed to give either selective depth of field or direction information.^[34]

Applications

In a 2017 study, researchers observed that incorporation of light field photographed images into an online anatomy module did not result in better learning outcomes compared to an identical module with traditional photographs of dissected cadavers.^[35]

Plenoptic cameras are good for imaging fast-moving objects that outstrip autofocus capabilities, and for imaging objects where autofocus is not practical such as with security cameras.^[36] A recording from a security camera based upon plenoptic technology could be used to produce an accurate 3D model of a subject.^[37]

Software

Lytro Desktop is a cross-platform application to render light field photographs taken by Lytro cameras. It remains closed source and is not maintained since Google’s acquisition of Lytro.^[21] Several open-source tools have been released meanwhile. A Matlab tool for Lytro-type camera processing can be found.^[38] PlenoptiCam is a GUI-based application considering Lytro's and custom-built plenoptic cameras with cross-platform compatibility and the source code being made available online.^[39]

See also

• Angle-sensitive pixel
• Bokeh
• Compound eye
• Femto-photography
• Integral imaging
• Light-in-flight imaging
• Photo finish
• Streak camera
• Strip photography

References

1. Adelson, E. H.; Wang, J. Y. A. (1992). "Single Lens Stereo with Plenoptic Camera". IEEE Transactions on Pattern Analysis and Machine Intelligence. 14 (2): 99–106. CiteSeerX 10.1.1.53.7845. doi:10.1109/34.121783.
2. "Light Field Photography with a Hand-Held Plenoptic Camera". graphics.stanford.edu.
3. Lumsdaine, A., Georgiev, T., The Focused Plenoptic Camera, ICCP, April 2009.
4. Hahne, C.; Aggoun, A.; Velisavljevic, V.; Fiebig, S.; Pesch, M. (2016). "Refocusing distance of a standard plenoptic camera". Optics Express. 24 (19): 21521–21540. Bibcode:2016OExpr..2421521H. doi:10.1364/oe.24.021521. hdl:10547/622011. PMID 27661891.
5. Hahne, C.; Aggoun, A.; Velisavljevic, V.; Fiebig, S.; Pesch, M. (2017). "Baseline and Triangulation Geometry in a Standard Plenoptic Camera". Int. J. Comput. Vis. 126: 21–35. arXiv:2010.04638. doi:10.1007/s11263-017-1036-4. S2CID 255109335.
6. . Archived from the original on 2019-09-11. Retrieved 2018-03-27.
7. Ashok Veeraraghavan, Ramesh Raskar, Amit Agrawal, Ankit Mohan and Jack Tumblin. Dappled Photography: Mask Enhanced Cameras for Heterodyned Light Fields and Coded Aperture Refocusing. ACM Transactions on Graphics, Vol. 26, Issue 3, July 2007.
8. "Lytro software update introduces Focus Spread feature". DPREVIEW. Retrieved 25 March 2015.
9. . Lytro Illum manual. Lytro. Archived from the original on September 28, 2014. Retrieved October 19, 2014.
10. Fried, Ina. "Meet the Stealthy Start-Up That Aims to Sharpen Focus of Entire Camera Industry". All Things Digital. Retrieved 24 June 2011.
11. Geron, Tomio (21 June 2011). "Shoot First, Focus Later With Lytro's New Camera Tech". Forbes. Retrieved 19 August 2011.
12. José Manuel Rodríguez-Ramos (1 April 2011). "3D imaging and wavefront sensing with a plenoptic objective". SPIE.
13. . TVB Europe. 23 September 2011. Archived from the original on 18 December 2012.{{cite web}}: CS1 maint: bot: original URL status unknown (link)
14. Blain, Loz (2022-04-29). "Record-breaking camera keeps everything between 3 cm and 1.7 km in focus". New Atlas. Retrieved 2022-04-30.
15. Fan, Qingbin; Xu, Weizhu; Hu, Xuemei; Zhu, Wenqi; Yue, Tao; Zhang, Cheng; Yan, Feng; Chen, Lu; Lezec, Henri J.; Lu, Yanqing; Agrawal, Amit (2022-04-19). "Trilobite-inspired neural nanophotonic light-field camera with extreme depth-of-field". Nature Communications. 13 (1): 2130. doi:10.1038/s41467-022-29568-y. ISSN 2041-1723. PMC 9019092. PMID 35440101.
16. . Archived from the original on 2011-11-04. Retrieved 2011-10-30.
17. Coldewey, Devin (23 July 2011). "Doubts About Lytro's "Focus Later" Camera". TechCrunch. Retrieved 19 August 2011.
18. Lars Rehm, DP Review. "CES 2012: Lytro Photowalk." Jan 13, 2012. Retrieved Apr 20, 2012
19. "Lytro Illum 40 Megaray Light Field Camera". Digital Photography Review. Retrieved 2014-10-19.
20. Goldman, Joshua (26 October 2011). "Lytro camera: 5 things to know before you buy". CNET Editor. CNET. Retrieved 21 November 2018.
21. "Google is buying Lytro for about $40M". techcrunch.com. 21 March 2018.
22. "One Camera With 40,000 Lenses Helps Prevent Blurry Images". March 18, 2019.
23. "The First Plenoptic Camera on the Market | PetaPixel". petapixel.com. 23 September 2010.
24. Levoy, M; Ng, R; Adams, A; Footer, M; Horowitz, M (2006). "Light Field Microscopy". ACM Transactions on Graphics. 25 (3): 924–93. doi:10.1145/1141911.1141976.
25. Levoy, M; Zhang, Z; McDowall, I (2009). "Recording and controlling the 4D light field in a microscope". Journal of Microscopy. 235 (2): 144–162. CiteSeerX 10.1.1.163.269. doi:10.1111/j.1365-2818.2009.03195.x. PMID 19659909. S2CID 13194109.
26. "Stanford Light Field Microscope Project". www.graphics.stanford.edu.
27. Keats, Jonathon; Holland, Kris; McLeod, Gary. . PopSci.com. Popular Science. Archived from the original (Adobe Flash) on 2008-01-17. Retrieved 26 July 2009.
28. "CAFADIS - University of la Laguna". from the original on 26 November 2019.
29. Amit Agrawal (2013-12-31). . Archived from the original on 2013-12-31.
30. . www77.pelicanimaging.com. Archived from the original on 2020-10-20. Retrieved 2021-08-24.
31. "Pelican Imaging's 16-lens array camera coming to smartphones next year". May 2, 2013.
32. Koifman, Vladimir (2015-07-25). "Pelican Imaging Layoffs?". Image Sensors World. from the original on 26 November 2019. Retrieved 2015-11-17.
33. "KLens company".
34. "Lightfield Camera". cameramaker.se.
35. Pascoe, Michael A.; Lee, Lisa M.J. (September 2017). "Incorporation of Light Field Photography into an Online Anatomy Resource Does Not Influence Student Quiz Performance or Perceptions of Usability". Medical Science Educator. 27 (3): 465–474. doi:10.1007/s40670-017-0410-8. ISSN 2156-8650. S2CID 148803076.
36. . sites.google.com. Archived from the original on 2022-10-09. Retrieved 2021-07-12.
37. Strehlow, Anne (November 3, 2005). . Stanford University. Archived from the original on July 23, 2010. Retrieved April 14, 2008.
38. "LightFieldToolbox".
39. "PlenoptiCam Software". GitHub.

External links

• Article by Ren Ng of Stanford (now at Lytro)
• Say Sayonara to Blurry Pics. Wired.
• Fourier slice photography
• Light Field Microscopy video by Stanford Computer Graphics Laboratory.
• IEEE Spectrum article May 2012 Lightfield photography revolutionizes imaging, with sample images and diagrams of operation, retrieved 2012 May 11
• www.plenoptic.info Website explaining the plenoptic camera with animations.