Home › English › Research › Spectral imaging and display › Natural Vision

Japanese / English

Introduction of Natural Vision

１．Introduction

The issue of color is now receiving considerable attention in the applications of image and video technology. The quality of color becomes significantly improved in the recent displays and prints; better color tone, higher color contrast, and wider color gamut. Then it is desired to reproduce more and more realistic image on the advanced high-quality displays. For the realistic image display, the reproduction of the original color, i.e., natural color reproduction, is quite important [Fig.1]. The natural color reproduction is also one of the key issues in the telemedicine, electronic commerce, and electronic museum.
　

(a)Reproducing the color as if the observer were at the site.

(b)Reproducing the color as if the object were placed at the site of the observer

Fig.1

However, the conventional image and video systems do not reproduce the original color of subject, and it obstructs the applications of visual communication systems. The color management technology is greatly progressed especially in color printing, but there still remain limitations imposed by RGB 3-primary color systems.

To breakthrough the limitation, there have been reported that the use of multispectral imaging significantly improves the color accuracy. Moreover, “Natural Vision” (referred as NV hereafter) system [1-3] has been developed aiming at an innovative video and still-image communication technology, which enables high-fidelity color reproduction, based on spectral information.

NV is an industry-government-academy joint project [4] performed at Akasaka Natural Vision Research Center (NVRC) from 1999 through 2006, conducted by NICT (National Institute of Information and Communications Technology, formally TAO) under the support of the Ministry of Internal Affairs and Communication. Researchers from academic institutes (Tokyo Institute of Technology and Chiba University) and industry (NTT Data, Olympus, NTT, etc.) were participated in the project. The activity of NV is currently being continued. In NV project, the experimental systems for both still-image and video have been elaborated and shown the effectiveness of spectrum-based color reproduction scheme. This paper introduces the motivation, methodology, systems, and applications of spectrum-based color reproduction technology, mainly referring the results of NV project.

2. Limitation of RGB-based systems

There are limitations in conventional RGB-based systems, as follows;
(1) The RGB values obtained in conventional systems often have different meanings, depending on the device characteristics or color processing. For example, many of conventional color imaging systems are designed for user preference, RGB values does not represent the objective color information.
(2) The spectral sensitivity characteristics of color camera are different from the human vision; RGB signal does not have one-by-one correspondence to the tristimulus values perceived by human vision. If the spectral sensitivity is closer to human vision, the color fidelity is improved, but noise behavior is known to become worse.
(3) The RGB or any other 3-primary color signals that comply with the color space such as sRGB or YCbCr, which are defined under the white point such as CIE D65 or D50 standard illuminant. To reproduce the color as if the object were placed in the presence of the observer [fig.1(b)], the color under different illuminant is required, but the standard color space does not support the color under arbitrary illuminant.
(4) For the color reproduction shown in fig.1(b), the spectral reflectance of object and the illuminant spectrum are essentially required for calculating the color under different illuminant. White balance adjustment in RGB space does not give high colorimetric accuracy.
(5) The color gamut of a normal RGB display is limited, and some of high-saturation colors cannot be reproduced. Even if the display device allows the display of wider color gamut, conventional color signals such as sRGB or ITU-R BT.709 do not support wider gamut color signal.
(6) The observer metamerism effect, due to the observer dependence of color matching functions, cannot be ignored in high-accuracy color reproduction, ex., in the color proofing of printed materials with color monitors.
(7) In the image archive, database, or analysis, the utilization of color information is limited, since the RGB signal depends on the devices, illuminants, and preprocesses involved in the imaging systems. For example in the image retrieval using color information, target object cannot found if the illumination condition is different.
The target of spectrum-based color reproduction is to overcome these problems in the image and video communication systems.

3. Spectrum-based color reproduction

The concept of the spectrum-based color reproduction system [2,3] is depicted in fig.2. To obtain sufficient spectral information, multispectral camera (MSC) is desired as an input device. The profile of the input device including the spectral sensitivity, tone curve, and dark current level of the camera, and the spectral energy distribution of illuminant are attached to multispectral image (MSI) data. In image processing, transmission, and storage, the image is accompanied with the profile data, so that the spectral radiance, reflectance or transmittance can be retrieved. In image display, the image of tristimulus values or spectral radiance is reproduced on 3-primary or multiprimary color display with device calibration.

Fig.2 The concept of spectrum-based color reproduction system

The architecture of spectrum-based color reproduction system is similar to the ICC (International Color Consortium) color management system, but the profile connection space (PCS) is based on physical model, i.e., spectrum-based PCS (SPCS) instead of color appearance model. This can be any of CIEXYZ under arbitrary illumination, spectral radiance, or spectral reflectance. The information required for the transform to SPCS is held in the profile data. The profile data format was defined as NV image data format, which was proposed to CIE TC8-07. The XML version NV format was also developed for easier handling.

In this system, the numbers of channels in the image capture and the display are independent, and the input and output devices with the arbitrary numbers of channels can be employed. Three-channel devices can also be used in this system with proper device characterization, with certain restrictions in the color reproduction capability or accuracy.

Spectrum-based system offers the solution to the problems presented in the section 2, as follows:
(a) For the natural color reproduction of real scene as if the observer were at the site as shown in fig.1(a), the colorimetric or spectral information is accurately captured and reproduced on a screen.
(b) In the situation shown in fig.1(b), the spectral reflectance of the object is captured and the color under the illumination of observing environment can be reproduced on a screen. Fig.3 depicts the example of color accuracy estimated from the 16-band camera shown in fig.5 and commercial 3-band digital still camera (DSC) with spectral characterization. It shows the advantage of using spectral information for accurate color reproduction.
(c) The wider color gamut becomes available by multiprimary color display, i.e., using more than three-primary colors. Fig.4 and Table 1 show the color gamut obtained by the 6-primary color displays developed by NV project [5,6]. Also, the image and video data obtained from spectrum-based system can represent the wider gamut signal.
(d) The original attribute of object that generates color, ex., spectral radiance, reflectance or transmittance information of the object, is captured and preserved by means of multispectral imaging. Such information is useful for the analysis or the recognition of object, in the digital archives of cultural heritage, artworks, and clinical cases in medicine.

Fig. 3: The color estimation accuracy of 16-band MSC and 3-band DSC,evaluated using GretagMacbeth Color Checker, under daylight (DL), CIE A and F2 illuminants, where DL was used in the image capturing.

Fig. 4: The color gamut obtained by 6-primary color display shown in fig.7 in comparison with conventional RGB display.

4. Multispectral and Multiprimary Technology

Multispectral Image Capture
Fig.5(a) shows the 16-band MSC developed in NVRC. As in fig.3, the accuracy by MSC is high and better than the discriminable level of human vision, while visually apparent error is observed in 3-band DSC. Fig.5(b) shows a 6-band HDTV camera for the acquisition of motion picture [7]. In image capturing, the illumination light spectrum is also measured, and then the spectral reflectance of the object can be calculated using the multispectral pixel value, illumination spectrum, and the spectral sensitivity of the camera.

Multiprimary color displays and wide gamut systems

Although there were few previous works on multiprimary color displays for larger color gamut, the system development and evaluation of muliprimary display had been originally started by NV project [5,6,8]. The multiprimary projection displays were developed in NV as shown in fig.6 to confirm the advantage of wide gamut natural color reproduction system and to evaluate its component such as multiprimary color conversion techniques. As in fig.4 and table 1, 6-primary DLP display almost covers the gamut of natural objects and the gamut volume in CIELAB space is about 1.8times larger than the normal RGB projector. The gamut is enlarged in the dark red, cyan, purple regions and bright orange regions. There have been reports of multiprimary displays recently for commercial products from various companies.
 
Since the conventional color space for images and video does not cover the whole color gamut, it is required to use expanded color space for wide gamut display. Conventional color spaces are defined according to the narrower gamut color display, but it is not required to depend on the display characteristics in the definition of color space from the aspect of spectrum-based color reproduction. The wide gamut video signal can be directly encoded with CIEXYZ, CIELAB, though the efficiency is not optimized if considering the quantization. Recently, standards for wider gamut video signal become a reality, de facto or de jure, such as ITU-R BT.1361, xvYCC, and adobeRGB. It is expected to establish the content creation environment for wider gamut images and video, and to realize the broadcasting of wider gamut video for the widespread use of the benefit of latest display technology.

(a)　(a) 16-band MSC with rotating filter wheel

(b)　6-band HDTV camera for capturing motion picture

Fig. 5 Multispectral cameras

(a)6-primary rear-projection display

(b) 6-primary DLP projector

Fig. 6 Muliprimary displays developed in NV project

Table 1 Comparison of color gamut with conventional RGB display and natural objects (Pointer+SOCS gamut).

Spectral color display

There exists variability in color matching functions of human observers; originated from the macular pigments, lens absorption, and cone sensitivity. Due to the individual difference of color matching functions, the color difference may appear to a certain observer when two color stimuli are shown, even if they are perceived as the same color by another observer, or CIE standard observer, called observer metamerism. It causes the color mismatch between different media, such as color printed materials and displays, even though the colorimetric match is achieved for CIE XYZ values. Based on the multispectral and multiprimary technology, spectral color display becomes possible, and the observer metamerism effect can be disappeared [9]. The advantage of spectral color display was experimentally demonstrated using 6-primary display.

Multiprimary color conversion

The multiprimary color signal is generated from the image of tristimulus values or multispectral data, called multiprimary color conversion. For colorimetric color reproduction, tristimulus values are transformed to M-dimensional multiprimary color values (3toM), where a degree of freedom is involved. Several methods have been developed for 3toM color conversion [10,11]. For the spectral color reproduction explained in the previous paragraph, it becomes NtoM conversion, where N is the number of channels of MSI. In the proposed NtoM conversion method, the spectral error is minimized under the constraint that the colorimetric match is attained for the standard observer [9].

Multispectral video system

The prototype of the multispectral video transmission was also developed in NV project. By using multiprimary color conversion hardware, real-time colorimetric reproduction is realized with 6-band MSC and 6-primary display in HDTV resolution [2].

The compression and encoding are the other issues on MSI transmission. As an MSI compression technique considering the colorimetric accuracy, modified KL transform called weighted KL transform was proposed and combined with JPEG2000 scheme [12]. For video compression, several methods that support multichannel images were tested, including a coding method in which MSI signal is converted into visible and invisible components, considering the compatibility to the conventional video signal. JPEG2000 is one of the suitable formats for high-quality encoding of both video and still MSI’s [13]. The profile data for NV format described above can be easily implemented in JPEG2000 as a extended ICC profile or metadata with using XML.

MSI can be considered as a value-added color image, as every pixel has quantitative spectral data. MSI database system has been developed, which supports image processing, analysis, and retrieval that utilize spectral data, and applied to the various fields as explained in the next section.

5. Applications

It is expected that realistic images and video display with high-fidelity natural color reproduction technology will be applied in various fields. There have been experimentally demonstrated the effectiveness of spectrum-based system;

In medicine, the use of digital color images is extremely valuable for teleconference, teleconsultation, education, training, image analysis, and reference database, in pathology, endoscopy, and dermatology. It is also required to reproduce the complexion of a patient through visual communication system for the telediagnosis and homecare. If the reproduced color is not accurate, it may cause misdiagnosis, thus the reliability of reproduced color is critical. In dermatology, MSI systems of both still-image and video were tested and the color reproducibility of multispectral system was rated sufficiently high from the visual evaluation by dermatologists [13]. In the pathology application, the spectrum-based system enables not only the accurate color reproduction of stained specimen, but also the correction of variation in staining condition, with adjusting the amount of stain through digital process[14]. It was also shown that spectral information is useful for the image analysis that supports diagnosis in both pathology and dermatology; the grading of disorders or the quantitative evaluation of treatments.

In the field of electronic commerce (EC), high-fidelity color reproduction is very important. In the online-shopping and trading of textiles, apparels, cosmetics, vehicles, furniture, etc., the color difference causes the return of purchased items. Also in the color matching of samples between designers, factory, buyers, and sales, the quantitative color information is quite beneficial. High-fidelity natural color reproduction system will contribute the further progress in EC. A web-based server-client prototype of electronic catalog system was developed in NV, and its effectiveness was demonstrated.

In the digital archive and electronic museum for art and cultural heritage, it is undesirable that the color of the archive depends on the imaging device and illumination condition with the use of conventional imaging devices. There have been many reports on the multispectral imaging for digital archive. The result of color reproduction on a display using 16-band MSI was shown to be satisfactory for the art management staffs in NV project [15]. The image data can be also employed for the high-fidelity color prints. Moreover, the spectral information in the image will be useful for the analysis of pigments or dyes, or the selection of material for restoration.

As for the application to color prints, it has been also proved that multispectral imaging greatly improves the color fidelity. In the test assuming the printing of catalogs for product promotion, a 16-band MSC and professional RGB DSC’s were compared. From the printed images obtained from the RGB DSC, a professional print director put 4-5 instructions for color corrections on average per printed sample. On the other hand, no instruction for color correction was given to any of resultant samples from 16-band MSC. The print director also commented that the printed results from MSI is even better than the proof produced from RGB DSC after the color correction [16]. Furthermore, the spectral color display can be useful for the color proofing applications using softcopy monitors, since the color matching between display and print is considerably improved thanks to the spectral approximation.

Digital prototyping is an important technology for expedition and efficiency of product development, and computer graphics (CG) enables the realistic rendering of virtual products, using Bidirectional Reflectance Distribution Function (BRDF) or Bidirectional Texture Function (BTF), but the color disagrees with real products. Multispectral BRDF/BTF measurement system and multispectral rendering technique were developed for high-fidelity color digital prototyping [17].

The expanded color gamut provides a new tool or new color for graphics expression. A CG system for coloring with multiprimary colors, named “IRODORI” [18] in NV project. Through the experiences of graphic expression using 6-primary display, the significance of wide-gamut graphics; (a) rendering of reality in vivid or splendid colors, (b) the enhanced reality based on “memory color,” which often shifts to higher saturation with time, (c) strong impact or fantastic impression, and (d) rich color tone thanks to the expansion of color range, providing the faint color change with dark colors, or the expression of gloss, metallic colors, or emissive colors.

The application of high-fidelity color reproduction is also beneficial in multimedia education such as instructional materials and encyclopedia, theater or electronic museum for realistic color reproduction of nature, artworks, and historic subjects. It can be also applied to the high-quality color management in the contents production from the material of recorded picture, editing, distribution, and playback of CG, animation, and other visual programs. In NV project, video contents were produced using 6-band HDTV camera and multiprimary CG, and the feasibility of spectrum-based systems was demonstrated. The application to digital broadcasting is strongly expected to facilitate the widespread use of innovative color imaging technology.

6. Conclusion

The concept, technology, and applications of spectrum-based color reproduction system are introduced in this paper. It enables not only the high-fidelity color reproduction but also the application of image analysis based on the quantitative spectral information. Moreover, multispectral information will be also of great utility in the image editing for preferable color, and other various image processing such as object extraction or image synthesis, though those were not the main topic of NV project that basically aims to natural color reproduction.
 
Going beyond RGB, great benefit emerges in advanced imaging applications. Multiprimary printing is already available in commercial product, but to make better use of devices that support innovative color reproducibility, a platform for spectrum-based system is expected, i.e., multispectral and wide-gamut video contents creation, management, distribution, and utilization.
 
Along with the introduction of information technology into our life, the quality of information that flows through the system becomes very important. For example, the image is used for diagnostic decision in the telemedicine system, and in electronic commerce, purchase or order items based on the digital information including images. If inaccurate images were presented, it might mislead one’s decision. Exchanging objective and accurate information will be more important in our society.
 
The author acknowledges the former research members of Natural Vision project; Nagaaki Ohyama, leader, Hideaki Haneishi, sub-leader, Toshio Uchiyama, Kenro Ohsawa, Hiroyuki Fukuda, Junko Kishimoto, Hiroshi Kanazawa, Tomohito Fujikawa, Hideto Motomura, Tatsuki Inuzuka and Satoshi Nambu, other fellow members from Olympus Co., NTT DATA Co., NTT Co., Toppan Printing Co., Dai-Nippon Printing Co., Matsushita Electric Industrial Co., Hitachi Co., NHK and others, joint researchers of NV project, those co-operating institutes and companies, National Institute of Information and Communications Technology, Ministry of Internal Affairs and Communication, and Natural Vision Promotion Council for the great support in NV project.

Selected Publications

  1. M. Yamaguchi, R. Iwama, Y. Ohya, T. Obi, N. Ohyama and Y. Komiya, and T. Wada, “Natural color reproduction in the television system for telemedicine,” Proc. SPIE, 3031, 482-489 (1997)
  2. M. Yamaguchi, T. Teraji, et. al., “Color image reproduction based on the multispectral and multiprimary imaging: Experimental evaluation,” Proc. SPIE, 4663, 15-26 (2002)
  3. M. Yamaguchi, H. Haneishi, H. Fukuda, J. Kishimoto, H. Kanazawa, M. Tsuchida, R. Iwama, and Nagaaki Ohyama, “High-fidelity video and still-image communication based on spectral information: Natural Vision system and its applications,” Spectral Imaging: Eighth International Symposium on Multispectral Color Science Proc. of SPIE-IS&T Electronic Imaging, SPIE Vol. 6062, (2006) 60620G
  4. Annual reports of Natural Vision Project, FY1999-2003 (TAO), FY2004-2006 (NICT), (2000-2007)
  5. T. Ajito, T. Obi, M. Yamaguchi, and N. Ohyama, “Expanded color gamut reproduced by six-primary projection display,” Proc. SPIE, 3954, 130-137 (2000)
  6. H. Motomura, N. Ohyama, M. Yamaguchi, H. Haneishi, K. Kanamori, S. Sannohe, “Development of Six-Primary HDTV display system,” Proc. Int. Display Research Conference, 563-566 (2002)
  7. K. Ohsawa, T. Ajito, H. Fukuda, Y. Komiya, H. Haneishi, M. Yamaguchi, and N. Ohyama, “Six-band HDTV camera system for spectrum-based color reproduction,” J. Imag. Sci. and Tech., 48, 2, 85-92 (2004)
  8. S. Komura, I. Hiyama, and N. Ohyama, “Four-Primary-Color LCD for Natural Vision,” Information Display, 8/03 (2003) 18-21
  9. Y. Murakami, J. Ishii, T. Obi, M. Yamaguchi, and N. Ohyama, “Color conversion method for multi-primary display for spectral color reproduction,” J. Electronic Imaging, 13, 4, 701-708 (2004)
  10. T. Ajito, K. Ohsawa, T. Obi, M. Yamaguchi, and N. Ohyama, “Color conversion method for multiprimary display using matrix switching,” Opt. Rev., 8, 3, 191-197 (2001)
  11. H. Kanazawa, “Color conversion for multiprimary displays using a spherical average method,” Proc. 12th Color Imag. Conf., 65-69 (2004)
  12. S. Yu, Y. Murakami, T. Obi, M. Yamaguchi, and N. Ohyama, “Multispectral image compression for high fidelity colorimetric and spectral reproduction,” J. Imag. Sci. and Tech., 50, (1): 64-72 (2006).
  13. S. Takamura and N. Kobayashi, “Constructing a uniform color space for visually lossless color representation and image coding,” IEEE International Conference on Image Processing, 918-921 (2001)
  14. M. Yamaguchi, M. Mitsui, Y. Murakami, H. Fukuda, N. Ohyama, and Y. Kubota, ” Multispectral color imaging for dermatology: application in inflammatory and immunologic diseases,” 13th Color Imag. Conf. (2005) 52-58
  15. T. Abe, Y. Murakami, M. Yamaguchi, N. Ohyama, and Y. Yagi, “Color correction of pathological images based on dye amount quantification,” Opt. Rev., 12, 4, 293-300 (2005)
  16. H. Fukuda, T. Uchiyama, H. Haneishi, M. Yamaguchi, and N. Ohyama, “Development of 16-bands multispectral image archiving system,” Proc. SPIE, 5667, 136-145 (2005)
  17. A. Uchida, et al., “Product photography for catalogs with multispectral camera,” Proc. of Spring Conf. of the Japanese Society of Printing Science and Technology, 131-134 (2003) (in Japanese)
  18. M. Tsuchida, et. al., “Development of BRDF and BTF Measurement and Computer-aided Design Systems Based on Multispectral Imaging,” Proc. AIC Colour ’05 129-132 (2005)
  19. J. Kishimoto, M. Yamaguchi , H. Haneishi, and N. Ohyama, “IRODORI – A Color-rich Palette Based on Natural Vision Technology-,” ACM SIGGRAPH 2004 Emerging Technology (2004) Link to SIGGRAPH 04 site 

Other publications related to NV project

Link

Last update 2007/01/10
2007/01/10　Masahiro Yamaguchi