Computational imaging

Computational imaging is the process of indirectly forming images from measurements using algorithms that rely on a significant amount of computing. In contrast to traditional imaging, computational imaging systems involve a tight integration of the sensing system and the computation in order to form the images of interest. The ubiquitous availability of fast computing platforms (such as multi-core CPUs and GPUs), the advances in algorithms and modern sensing hardware is resulting in imaging systems with significantly enhanced capabilities. Computational Imaging systems cover a broad range of applications include computational microscopy,^[1] tomographic imaging, MRI, ultrasound imaging, computational photography, Synthetic Aperture Radar (SAR), seismic imaging etc. The integration of the sensing and the computation in computational imaging systems allows for accessing information which was otherwise not possible. For example:

• A single X-ray image does not reveal the precise location of fracture, but a CT scan which works by combining multiple X-ray images can determine the precise location of one in 3D
• A typical camera image cannot image around corners. However, by designing a set-up that involves sending fast pulses of light, recording the received signal and using an algorithm, researchers have demonstrated the first steps in building such a system.^[2]

Computational imaging systems also enable system designers to overcome some hardware limitations of optics and sensors (resolution, noise etc.) by overcoming challenges in the computing domain. Some examples of such systems include coherent diffractive imaging, coded-aperture imaging and image super-resolution.

Computational imaging differs from image processing in a sense that the primary goal of the former is to reconstruct human-recognizable images from measured data via algorithms while the latter is to process already-recognizable images (that may be not sufficient in quality) to improve the quality or derive some information from them.

History

Computational imaging systems span a broad range of applications. While applications such as SAR, computed tomography, seismic inversion are well known, they have undergone significant improvements (faster, higher-resolution, lower dose exposures^[3]) driven by advances in signal and image processing algorithms (including compressed sensing techniques), and faster computing platforms. Photography has evolved from purely chemical processing to now being able to capture and computationally fuse multiple digital images (computational photography)^[4] making techniques such as HDR and panoramic imaging available to most cell-phone users. Computational imaging has also seen an emergence of techniques that modify the light source incident on an object using known structure/patterns and then reconstructing an image from what is received (For example: coded-aperture imaging, super-resolution microscopy, Fourier ptychography). Advances in the development of powerful parallel computing platforms has played a vital role in being able to make advances in computational imaging.

Techniques

Coded aperture imaging

Further information: Coded aperture

See also: Range imaging § Coded aperture

Imaging is usually made at optical wavelengths by lenses and mirrors. However, for X-rays and Gamma-rays, lenses and mirrors are impractical, therefore modulating apertures are often used instead. The pinhole camera is the most basic form of such a modulation imager, but its disadvantage is low throughput, as its small aperture allows through little radiation. Since only a tiny fraction of the light passes through the pinhole, which causes a low signal-to-noise ratio, imaging through pinholes involves unacceptable long exposures. This problem can be overcome to some degree by making the hole larger, which unfortunately leads to a decrease in resolution. Pinhole cameras have a couple of advantages over lenses - they have infinite depth of field, and they don't suffer from chromatic aberration, which can be cured in a refractive system only by using a multiple element lens. The smallest feature which can be resolved by a pinhole is approximately the same size as the pinhole itself. The larger the hole, the more blurred the image becomes. Using multiple, small pinholes might seem to offer a way around this problem, but this gives rise to a confusing montage of overlapping images. Nonetheless, if the pattern of holes is carefully chosen, it is possible to reconstruct the original image with a resolution equal to that of a single hole.

In recent years much work has been done using patterns of holes of clear and opaque regions, constituting what is called a coded aperture. The motivation for using coded aperture imaging techniques is to increase the photon collection efficiency whilst maintaining the high angular resolution of a single pinhole. Coded aperture imaging (CAI) is a two-stage imaging process. The coded image is obtained by the convolution of the object with the intensity point spread function (PSF) of the coded aperture. Once the coded picture is formed it has to be decoded to yield the image. This decoding can be performed in three ways, namely correlation, Fresnel diffraction or deconvolution. An estimation of the original image is attained by convolving the coded image with the original coded aperture. In general, the recovered image will be the convolution of the object with the autocorrelation of the coded aperture and will contain artifacts unless its autocorrelation is a delta function.

Some examples of coded apertures include the Fresnel zone plate (FZP), random arrays (RA), non-redundant arrays (NRA), uniformly redundant arrays (URA), modified uniformly redundant arrays (MURA), among others. Fresnel zone plates, called after Augustin-Jean Fresnel, may not be considered coded apertures at all since they consist of a set of radially symmetric rings, known as Fresnel zones, which alternate between opaque and transparent. They use diffraction instead of refraction or reflection to focus the light. Light hitting the FZP will diffract around the opaque zones, therefore an image will be created when constructive interference occurs. The opaque and transparent zones can be spaced so that imaging occurs at different focuses.

In the early work on coded-apertures, pinholes were randomly distributed on the mask and placed in front of a source to be analyzed. Random patterns, however, pose difficulties with image reconstruction due to a lack of uniformity in pinholes distribution. An inherent noise appears as a result of small terms present in the Fourier transform of large size random binary arrays. This problem was addressed by the development of uniformly redundant arrays (URAs). If the distribution of the transparent and opaque elements of the aperture can be represented as a binary encoding array A and the decoding array as G, then A and G can be chosen such that the reconstructed image (correlation of A and G with an addition of some noise signal N) approximates a delta function. It has experimentally been shown that URAs offer significant improvements to SNR in comparison with randomly distributed arrays, however, the algorithm used for the construction of URAs restricts the shape of the aperture to a rectangle. Therefore, Modified Uniformly Redundant Array (MURA), was introduced with a change to URA's encoding algorithm, enabling new arrays to be created in linear, hexagonal and square configurations. The design method for URAs was modified so that the new arrays were based on quadratic residues rather than pseudo-noise (PN) sequences.

Compressive spectral imaging

Conventional spectral imaging techniques typically scan adjacent zones of the underlying spectral scene and then merge the results to construct a spectral data cube. In contrast, compressive spectral imaging (CSI), which naturally embodies the principles of compressed sensing (CS), involves the acquisition of the spatial-spectral information in 2-dimensional sets of multiplexed projections. The remarkable advantage of compressive spectral imaging is that the entire data cube is sensed with just a few measurements and in some cases with as little as a single FPA snapshot such that the entire data set can be obtained during a single detector integration period.

In general, compressive spectral imaging systems exploit different optical phenomena such as spatial, spectral, or spatial-spectral coding and dispersion, to acquire the compressive measurements. The significant advantage behind CSI is that it is possible to design sensing protocols that capture the essential information from sparse signals with a reduced amount of measurements. Because the amount of captured projections is less than the number of voxels in the spectral data cube, the reconstruction process is performed by numerical optimization algorithms. This is the step where computational imaging plays a key role because the power of computational algorithms and mathematics is exploited to recover the underlying data cube.

In the CSI literature, different strategies can be encountered to attain the coded projections.^[5][6][7] The coded aperture snapshot spectral imager (CASSI) was the first spectral imager designed to take advantage of compressive sensing theory.^[8] CASSI employs binary coded apertures that create a transmission pattern at each column, such that these patterns are orthogonal with respect to all other columns. The spatial-spectral projection at the detector array is modulated by the binary mask in such a way that each wavelength of the data cube is affected by a shifted modulation code. More recent CSI systems include the CASSI using colored coded apertures (C-CASSI) instead of the black and white masks; a compact version of the colored CASSI, called snapshot colored compressive spectral imager (SCCSI), and a variation of the latter that uses a black-and-white coded aperture in the convolutional plane, known as the spatial–spectral encoded hyperspectral imager (SSCSI). Common characteristics of this kind of CSI systems include the use of a dispersive element to decouple the spectral information, and a coding element to encode the incoming data.

Algorithms

While computational imaging covers a broad range of applications, the algorithms used in computational imaging systems are often related to solving a mathematical inverse problem. The algorithms are generally divided into direct inversion techniques which are often "fast" and iterative reconstruction techniques that are computationally expensive but are able to model more complex physical processes. The typical steps to design algorithms for computational imaging systems are:

1. Formulating a relationship between the measurements and the quantity to be estimated. This process requires a mathematical model for how the measurements are related to the unknown. For example: In high-dynamic range imaging, the measurements are a sequence of known exposures of the underlying area to be imaged. In an X-ray CT scan, the measurements are X-ray images of the patient obtained from several known positions of the X-ray source and detector camera with a well-established relationship for X-ray propagation.
2. Choosing a metric to "invert" the measurements and reconstruct the quantity of interest. This could be a simple metric such as a least-squares difference between the measurements and the model or a more sophisticated metric based on precisely modeling the noise statistics of the detector and a model for the object of interest. This choice can be related to choosing a statistical estimator for the quantity to be reconstructed.
3. Designing fast and robust algorithms that compute the solution to Step 2. These algorithms often use techniques from mathematical optimization and mapping such methods to fast computing platforms to build practical systems.

References

1. CITRIS (2017-03-08), Computational Microscopy, retrieved 2017-09-04
2. March, G. (20 March 2012). "How to see around corners". Nature News. doi:10.1038/nature.2012.10258. S2CID 171886549.
3. Ju, Yun Hye; Lee, Geewon; Lee, Ji Won; Hong, Seung Baek; Suh, Young Ju; Jeong, Yeon Joo (8 August 2017). "Ultra-low-dose lung screening CT with model-based iterative reconstruction: an assessment of image quality and lesion conspicuity". Acta Radiologica. 59 (5): 553–559. doi:10.1177/0284185117726099. PMID 28786301. S2CID 4428506.
4. "Plenary talk (Peyman Milanfar)" (PDF).
5. Hagen, Nathan (13 June 2012). "Snapshot advantage: a review of the light collection improvement for parallel high-dimensional measurement systems" (PDF). Optical Engineering. 51 (11): 111702. Bibcode:2012OptEn..51k1702H. doi:10.1117/1.OE.51.11.111702. PMC 3393130. PMID 22791926.
6. Hagen, Nathan; Kudenov, Michael W. (23 September 2013). "Review of snapshot spectral imaging technologies" (PDF). Optical Engineering. 52 (9): 090901. Bibcode:2013OptEn..52i0901H. doi:10.1117/1.OE.52.9.090901.
7. Arce, Gonzalo R.; Rueda, Hoover; Correa, Claudia V.; Ramirez, Ana; Arguello, Henry (2017-02-15). Snapshot Compressive Multispectral Cameras. Wiley Encyclopedia of Electrical and Electronics Engineering. pp. 1–22. doi:10.1002/047134608X.W8345. ISBN 9780471346081.
8. Wagadarikar, Ashwin; John, Renu; Willett, Rebecca; Brady, David (8 February 2008). "Single disperser design for coded aperture snapshot spectral imaging". Applied Optics. 47 (10): B44-51. Bibcode:2008ApOpt..47B..44W. doi:10.1364/AO.47.000B44. PMID 18382550.

Further reading

Advances in the field of computational imaging research is presented in several venues including publications of SIGGRAPH and the IEEE Transactions on Computational Imaging.