Fujitsu Limited
Fujitsu and Carnegie Mellon University develop technology that dynamically and accurately transforms people and objects into 3D images from 2D images captured by fixed monocular cameras.
……
Fujitsu Limited (Note 1) (hereinafter referred to as Fujitsu) and Carnegie Mellon University (Note 2) in the United States (hereinafter referred to as Carnegie Mellon University) have announced that as part of a joint research project on social digital twins that began in February 2022, By 2024, we will develop technology that dynamically reconstructs the 3D shape and position of people and objects with high precision by converting objects in images obtained from a single monocular camera into 3D images using AI and digitizing them. Developed in May. On February 22, 2024, the two companies began a demonstration experiment to verify the effectiveness of this technology using image data taken of intersections around Carnegie Mellon University.
This technology uses AI that has learned the shape of people and objects through deep learning to estimate the 3D shape of each 2D object seen on camera, as well as 3D shape estimation technology that includes buildings, terrain, etc. It is composed of two core technologies: 3D projection technology that accurately estimates and reconstructs the position of people and objects on 3D models. By utilizing these technologies, for example, images taken in scenes where people and cars are crowded, such as at intersections, can be anonymized and dynamically reconstructed into a three-dimensional image, allowing traffic accidents that could not be captured by surveillance cameras to be detected. Visualize potential issues such as the causes of problems.
In order to expand the scope of application, both parties will verify the usefulness not only in transportation but also in smart cities, etc., with the aim of commercializing this technology by fiscal 2025. 【background】
In February 2022, Fujitsu and Carnegie Mellon University began joint research on social digital twins that dynamically reproduce complex interactions between people, things, economy, and society in three dimensions. We have been working on the development of high-speed 3D scene restoration technology that generates high-speed,
high-definition images from videos shot from different angles. However, as we proceeded with our joint research, we found that in order to dynamically restore captured images to 3D, we found that the accuracy of video analysis was technically insufficient, and that we were unable to accurately accurately determine the position and shape of objects in 3D. Multiple cameras were required to reproduce this, and issues such as workload and cost were a barrier to social implementation.
In order to solve the above-mentioned problem, both companies reconstructed a dynamic 3D scene model even when an object is photographed from a fixed position with a single monocular camera, without combining images shot simultaneously by multiple cameras. We have developed a technology to do this.
[About the development technology]
This technology mainly consists of the following two core technologies. (1) 3D shape estimation technology: This technology uses deep learning to identify the types of objects such as buildings and people reflected in multiple images of the city taken from various angles. Utilize the model that was created. This makes it possible to represent even a single city image taken with a monocular camera as a collection of Voxels (Note 3) in 3D space, including categories such as buildings and people. In addition, advance machine learning enables accurate 3D shape estimation of areas that are hidden and not visible in images, such as the back side of buildings.
(2) 3D projection technology: This technology incorporates already learned social and humanistic human behavior analysis know-how on a 3D digital twin based on the output results of 3D shape estimation technology. This allows for high-precision placement in three dimensions along with information such as direction of travel and speed, while excluding human movements that cannot occur in the real world, such as when a person passes through an object. This not only makes it possible to reconstruct the movements of people and vehicles in a way that more closely matches the real world, but also enables accurate position estimation even when specific parts of objects are hidden by obstacles.
[Image: https://prtimes.jp/i/93942/260/resize/d93942-260-9cca867bfb6b61c609e1-0.png&s3=93942-260-97fab51309348b92245b06cc53d60671-980×482.png ]
Figure 1. Overview of development technology initiatives
[About demonstration experiment]
Period: From Thursday, February 22, 2024 to Friday, May 31, 2024 Location: Pittsburgh, Pennsylvania, USA
Content: Monocular cameras are installed inside Carnegie Mellon University and other locations to recognize objects that appear in image data taken at intersections around the university, anonymize people’s faces and car license plates, and protect privacy. Conducted a demonstration experiment to reproduce on social digital twin. Analyzing people’s traffic and traffic conditions around the university, using the analysis results to discover potential accidents such as blind spots caused by buildings and temporary crowds, and aiming to formulate preventive measures. , verified the effectiveness of the developed technology.
[Comment from Assistant Professor Laszlo A. Jeni, Carnegie Mellon University] We are pleased to announce this result, which is the result of joint research between the Fujitsu team and the Carnegie Mellon University team, including Professor Sean Qian and Professor Srinivasa
Narasimhan. Carnegie Mellon University will continue to advance research into cutting-edge technology through collaboration with Fujitsu.
[Comment from Daiki Masumoto, Fellow at Fujitsu Laboratories and Director of Converging Technology Research Institute, Fujitsu Limited] Our purpose is “to bring trust to society and make the world more sustainable through innovation.” The social digital twin we are developing aims to solve various social issues in line with this purpose. This milestone, achieved in collaboration with Carnegie Mellon University, is an important step toward our goal.
[About trademark]
Proper nouns such as product names listed are trademarks or registered trademarks of each company.
[Note]
Note 1
Fujitsu Limited: Headquarters Minato-ku, Tokyo, Representative Director and President Takahito Tokita
(Note: Depending on the publication media and viewing environment specifications, the character “Takashi” may not be displayed correctly. Correctly, “ichi” should be placed above the “raw” in “Takashi.”)
Note 2
Carnegie Mellon University: Location: Pennsylvania, USA, President Farnam Jahanian
Note 3
Voxel: A term that combines volume and pixel. Just as a pixel represents a single pixel in a two-dimensional image, a voxel represents a single point in three-dimensional space, so it is a three-dimensional representation method.
【Related Links】
・Fujitsu and Carnegie Mellon University begin joint research on social digital twins to solve social and economic issues (Press release February 8, 2022)
(https://pr.fujitsu.com/jp/news/2022/02/8.html)
・Fujitsu and Carnegie Mellon University joint research results develop dynamic 3D shape restoration technology using neural networks (July 6, 2023 Research and Development Technology Topics)
(https://www.fujitsu.com/jp/about/research/article/202307-cvpr2023.html) ・Infrastructure transformation using digital twins (February 19, 2024 Carnegie Mellon University)
(https://www.cmu.edu/cee/news/news-archive/2024/02-2024-transforming-infrastructure-through-digital-twins.html) [Inquiries regarding this matter]
Fujitsu contact line (general counter) 0120-933-200 (toll-free) Reception hours: 9:00 to 12:00 and 13:00 to 17:30 (excluding Saturdays, Sundays, holidays, and Fujitsu designated holidays) Contact form (https://contactline.jp.fujitsu.com/customform/csque04802/873532/) Product prices, specifications, service content, etc. stated in the press release are as of the date of announcement. It is subject to change without prior notice. Please note.
More details about this release:
https://prtimes.jp/main/html/rd/p/000000260.000093942.html