WO2024190373A1 - Image processing device, image processing method, program, and image processing system - Google Patents

Abstract

The present technology relates to an image processing device, an image processing method, a program, and an image processing system that make it possible to perform an image quality enhancement process specialized for a surgical environment. This image processing device selects an image quality enhancement process to be performed on captured image data, which has been obtained by imaging a surgical setting by means of an imaging device, on the basis of surgical imaging environmental information, which is information relating to the captured image data, and performs the selected image quality enhancement process on the captured image data. The present technology can be applied to a captured image processing system.

Description

IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, PROGRAM, AND IMAGE PROCESSING SYSTEM

This technology relates to an image processing device, an image processing method, a program, and an image processing system, and in particular to an image processing device, an image processing method, a program, and an image processing system that are capable of performing high-image-quality processing specialized for a surgical environment.

Technology for performing super-resolution processing using pre-generated training data has been proposed in Patent Document 1 and elsewhere.

JP 2008-140012 A

However, in surgical settings where surgery can have a significant impact on human lives, low-noise, high-resolution, high-quality images are required to perform precise and delicate procedures. Furthermore, in the case of ophthalmic surgery, for example, strong lighting can be invasive to the eye, so low-noise, bright surgical images are required even in environments with low lighting intensity.

This technology was developed in light of these circumstances, and makes it possible to perform high-image-quality processing specialized for surgical environments.

An image processing device according to one aspect of the present technology includes a high-image-quality processing selection unit that selects a high-image-quality processing to be performed on the captured image data based on surgical imaging environment information, which is information related to the captured image data obtained by imaging the surgical site with an imaging device, and a high-image-quality processing unit that performs the high-image-quality processing selected by the high-image-quality processing selection unit on the captured image data.

In one aspect of this technology, based on surgical imaging environment information, which is information about the captured image data obtained by imaging the surgical site with an imaging device, a high image quality processing to be performed on the captured image data is selected, and the selected high image quality processing is performed on the captured image data.

1 is a diagram illustrating an example of a configuration of a captured video processing system according to an embodiment of the present technology. 2 is a flowchart illustrating a process of the sink device in FIG. 1 . 3 is a flowchart illustrating the selection process of the image quality improvement process in step S12 of FIG. 2. 13 is a diagram showing a method for acquiring surgical imaging environment information using a resolution chart. FIG. 13A and 13B are diagrams illustrating a method for acquiring surgical imaging environment information using a color chart. FIG. 11 is a diagram illustrating an example of a learning process for image quality improvement processing. FIG. 13 is a diagram illustrating a second example of the configuration of the captured video processing system. FIG. 13 is a diagram illustrating a third example of a configuration of a captured video processing system. FIG. 13 is a diagram illustrating a fourth example of a configuration of a captured video processing system. FIG. 5 is a diagram illustrating a fifth example of a configuration of a captured video processing system. FIG. 1 is a block diagram illustrating an example of the configuration of a computer.

Hereinafter, an embodiment of the present technology will be described in the following order.
1. System configuration and operation 2. Modifications 3. Others

1. System configuration and operation
<First Configuration of Captured Video Processing System>
FIG. 1 is a block diagram showing a first example configuration of a captured video processing system according to an embodiment of the present technology.

The captured image processing system 1 in FIG. 1 is configured to include a source device 11 and a sink device 12.

The source device 11 is composed of an imaging device such as a surgical microscope or endoscope. The source device 11 transmits the imaging video data generated by imaging to the sink device 12. The source device 11 transmits surgical imaging environment information including, for example, the frequency characteristics (MTF) and noise characteristics of the imaging equipment to the sink device 12.

The surgical imaging environment information is information related to the captured image of the surgical site captured by the source device 11. The surgical imaging environment information will be described in detail later, but includes at least one of the following: imaging device information, imaging environment information (lighting intensity, color temperature of lighting), imaging subject information (organs and medical instruments), surgical procedure information, and signal processing information for the captured image in the source device 11.

The sink device 12 is composed of a video processing device that controls the processing of video data to be output to a medical monitor, etc.

The sink device 12 acquires the surgical imaging environment information transmitted from the source device 11, and selects an image quality improvement process for the imaging video data transmitted from the source device 11 based on the acquired surgical imaging environment information, for example, from a plurality of image quality improvement processes A to D.

Each of the high image quality processes A to D is made up of a combination of one or more types of image processing, and the optimal combination of image processing and the parameters of each image processing differ depending on the source device 11. The image processing includes, for example, high resolution processing and noise reduction processing.

The sink device 12 performs the selected image quality improvement processing on the captured video data sent from the source device 11 and outputs it.

The sink device 12 is configured to include, for example, a surgical imaging environment information receiving unit 21, a high image quality processing selection unit 22, a high image quality processing unit 23, a display control unit 24, and a display unit 25.

The surgical imaging environment information receiving unit 21 receives (acquires) the surgical imaging environment information transmitted from the source device 11. The surgical imaging environment information receiving unit 21 outputs the received surgical imaging environment information to the high image quality processing selection unit 22.

The high image quality processing selection unit 22 stores a plurality of high image quality processing (e.g., high image quality processing A to D) in a high image quality processing DB in association with the surgical imaging environment information. The high image quality processing selection unit 22 selects the most suitable high image quality processing from the high image quality processing DB based on the surgical imaging environment information supplied from the surgical imaging environment information receiving unit 21. Note that the high image quality processing selection unit 22 can also select general image processing that is not high image quality.

The high-image-quality processing unit 23 performs the high-image-quality processing selected by the high-image-quality processing selection unit 22 on the captured video data transmitted from the source device 11, and outputs the high-image-quality video data after the high-image-quality processing to the display control unit 24.

The display control unit 24 causes the display unit 25 to display high-quality images corresponding to the high-quality image data after the high-quality processing by the high-quality processing unit 23. The display control unit 24 causes the display unit 25 to display information to be presented to the user in the selection process of the high-quality processing by the high-quality processing selection unit 22.

The display unit 25 is configured with a medical monitor or the like. The display unit 25 does not have to be configured in the sink device 12, and may be an external monitor or the like.

The surgical imaging environment information may be transmitted from the source device 11 to the sink device 12 via the same connection path as the captured video data, using SDI Ancillary Data Formats or HDMI (registered trademark) Vendor Specific Commands, etc. Alternatively, the surgical imaging environment information may be transmitted from the source device 11 to the sink device 12 via a path separate from the captured video data, using a USB, Ethernet cable, WI-FI, etc.

The source of the surgical imaging environment information transmitted to the sink device 12 may be a device other than the source device 11. For example, imaging subject information may be obtained by receiving surgical procedure information transmitted from a surgical management system in a hospital or the like.

Next, we will explain surgical imaging environment information in detail.

As described above, the surgical imaging environment information includes at least one of the following: imaging device information, imaging environment information (lighting intensity, lighting color temperature), information about the surgery (imaged object information (organs and medical instruments), surgical procedure information), and signal processing information in the source device 11.

The imaging device information consists of frequency characteristic information of the imaging device (source device 11), noise characteristic information of the imaging device, RAW image format information of the imaging device (3-chip, Bayer), setting information of the imaging device (optical zoom information), and ID (identification information) such as the endoscope, camera head, or CCU.

The signal processing information in the source device 11 is signal processing information that has been performed on the captured image in the source device 11. The signal processing information in the source device 11 includes at least one of gamma correction processing information, gain processing information, demosaic processing information, noise removal information, edge enhancement processing information, defective pixel correction processing information, color matrix correction processing information, black level correction processing information, shading correction processing information, white balance correction processing information, tone curve correction information, and super-resolution processing information.

For example, imaging device information (endoscopic scope, camera head, CCU ID) and imaging device setting information (optical zoom information) may be received, and frequency characteristic information of the imaging device may be obtained by searching a frequency characteristic information DB prepared in advance. Also, surgical procedure information may be received, and imaging subject information may be obtained by searching a DB of surgical procedure information and imaging subject information (organs and medical instruments) prepared in advance.

Note that for any missing signal processing information, generic settings will be used. However, the effect of optimizing image quality improvement processing using the acquired signal processing information will be realized, and as a result, it is possible to provide image quality improvement processing that is more effective than before.

<Processing of sink device>
FIG. 2 is a flowchart illustrating the captured image processing of the sink device of FIG.

In step S11, the surgical imaging environment information receiving unit 21 of the sink device 12 receives the surgical imaging environment information transmitted from the source device 11. The surgical imaging environment information receiving unit 21 outputs the received surgical imaging environment information to the high image quality processing selecting unit 22.

In step S12, the image quality improvement process selection unit 22 performs image quality improvement process selection processing. This image quality improvement process selection processing will be described later with reference to FIG. 3. By the processing of step S12, the image quality improvement process that is optimal for the source device 11 is selected.

In step S13, the image quality improvement processing unit 23 receives the captured video data transmitted from the source device 11.

In step S14, the image quality improvement processing unit 23 performs the image quality improvement processing selected by the image quality improvement processing selection unit 22 on the received captured image data, and outputs the high image quality image data after the image quality improvement processing to the display control unit 24.

In step S15, the display control unit 24 causes the display unit 25 to display a high-definition image corresponding to the high-definition video data.

FIG. 3 is a flowchart explaining the image quality improvement process selection process in step S12 of FIG. 2.

In step S31, the image quality improvement process selection unit 22 determines whether or not there is an image quality improvement process in the image quality improvement process DB that has a completely matching surgical imaging environment information. If it is determined in step S31 that there is no image quality improvement process that has a completely matching surgical imaging environment information, the process proceeds to step S32.

In step S32, the display control unit 24 causes the display unit 25 to present the estimated time required for creating high image quality processing that completely matches the surgical imaging environment information.

In step S33, the high image quality processing selection unit 22 determines whether or not to create high image quality processing that completely matches the surgical imaging environment information. If it is determined in step S33 that high image quality processing that completely matches the surgical imaging environment information is to be created in response to a user instruction input from an operation unit (not shown), the process proceeds to step S34.

In step S34, the high image quality processing selection unit 22 creates a high image quality processing that completely matches the surgical imaging environment information. For example, the high image quality processing and its parameters are learned by a learning process described below. Note that since bias occurs in the surgical imaging environment information for each piece of equipment used, it is possible to prepare in advance combinations of surgical imaging environment information that are highly biased. Then, the process proceeds to step S35.

Also, if it is determined in step S31 that there is a high image quality process that completely matches the surgical imaging environment information, processing proceeds to step S35.

In step S35, the display control unit 24 causes the display unit 25 to present the contents of the high image quality processing that completely matches the surgical imaging environment information.

In step S36, the image quality improvement process selection unit 22 selects an image quality improvement process that completely matches the surgical imaging environment information. After that, the image quality improvement process selection process ends.

On the other hand, if it is determined in step S33 that no high image quality processing is to be created, processing proceeds to step S37.

In step S37, the image quality improvement process selection unit 22 determines whether or not there is an image quality improvement process in the image quality improvement process DB that has at least a portion of the surgical imaging environment information that matches. If it is determined in step S37 that there is an image quality improvement process that has at least a portion of the surgical imaging environment information that matches, the process proceeds to step S38.

In addition, when determining whether there is a partial match of the image quality improvement parameters, priority is given to matches of surgical imaging environment information that contributes greatly to the image quality improvement parameters (such as frequency characteristic information of the endoscope, noise characteristic information of the endoscope and CCU, and gain processing information in the source device 11).

In step S38, the display control unit 24 causes the display unit 25 to present the contents of the image quality improvement processing that at least partially matches the surgical imaging environment information.

In step S39, the image quality improvement process selection unit 22 determines whether to select an image quality improvement process that matches at least a portion of the surgical imaging environment information. If it is determined in step S39 that an image quality improvement process that matches at least a portion of the surgical imaging environment information is to be selected, the process proceeds to step S40.

In step S40, the image quality improvement process selection unit 22 selects an image quality improvement process that matches at least a portion of the surgical imaging environment information. After that, the image quality improvement process selection process ends.

If it is determined in step S37 that there is no high image quality process that matches at least a portion of the surgical imaging environment information, or if it is determined in step S39 that no high image quality process that matches at least a portion of the surgical imaging environment information is to be selected, processing proceeds to step S41.

In step S41, the image quality improvement process selection unit 22 selects a general-purpose image quality improvement process. After that, the image quality improvement process selection process ends.

2. Modified Examples
<Method of acquiring surgical imaging environment information>
In the above description, an example has been described in which the surgical imaging environment information transmitted from the source device 11 is received. Below, another method for acquiring the surgical imaging environment information will be described.

Figure 4 shows a method for acquiring surgical imaging environment information using a resolution chart.

In the captured image processing system 1 in FIG. 4, the sink device 12 is replaced with a sink device 51. The sink device 51 differs from the sink device 12 in FIG. 2 in that a frequency characteristic measuring unit 61 is added.

In other words, the source device 11 captures an image of a pre-set resolution (ISO) chart 62 and transmits the captured image data of the resolution chart 62 that has been captured and generated.

The frequency characteristic measurement unit 61 of the sink device 51 receives the captured image data of the resolution chart 62, analyzes the captured image data, and acquires the frequency characteristic information of the source device 11. The frequency characteristic measurement unit 61 outputs the acquired frequency characteristic information of the source device 11 to the surgical imaging environment information receiving unit 21.

In this way, the surgical imaging environment information receiving unit 21 can acquire frequency characteristic information of the source device 11 as one piece of surgical imaging environment information.

Figure 5 shows a method for acquiring surgical imaging environment information using a color chart.

In the captured image processing system 1 in FIG. 5, the sink device 12 is replaced with a sink device 81. The sink device 81 differs from the sink device 12 in FIG. 2 in that a noise characteristic measuring unit 91 is added.

In other words, the source device 11 captures a pre-set resolution chart 82 and transmits the captured image of the resolution chart 82 that has been generated.

The noise characteristic measurement unit 91 of the sink device 81 receives the captured image data of the resolution chart 62, analyzes the captured image data, and acquires the noise characteristic information of the source device 11. The noise characteristic measurement unit 91 outputs the acquired noise characteristic information to the surgical imaging environment information receiving unit 21.

As a result, the surgical imaging environment information receiving unit 21 can acquire noise characteristic information of the source device 11 as one piece of surgical imaging environment information.

<Learning high image quality processing>
FIG. 6 is a diagram showing an example of a learning process for image quality improvement processing.

The image quality improvement processing performed by the image quality improvement processor 23 can also learn using surgical imaging environment information. AI, machine learning, deep neural networks, etc. are used to learn the image quality improvement processing.

FIG. 6 shows an example of the configuration of a sink device 101 that has a learning function unit that performs learning processing for image quality improvement processing.

The sink device 101 in FIG. 6 has the same configuration as the sink device 12, except that a learning function unit 111 has been added. Note that in FIG. 6, the illustration of each unit in the sink device 101 other than the image quality improvement processing unit 23 and the learning function unit 111 has been omitted.

The learning function unit 111 in FIG. 6 shows a case where parameters for image quality improvement processing (e.g., image quality improvement processing A) consisting of high resolution processing and noise removal processing are learned.

The learning function unit 111 is configured to include a learning dataset image DB 121, an image data preparation unit 122, and a learning unit 123.

The learning dataset image DB121 stores RAW image data.

The image data preparation unit 122 uses the raw image data from the learning dataset image DB 151 based on the surgical imaging environment information to prepare teacher data and student data that the learning unit 123 uses for learning.

The teacher data is close to ideal image data that simulates the results of image quality improvement processing. The student data is image data that simulates images captured by an imaging device.

In other words, the image data preparation unit 122 prepares teacher data by simulating the results of the high image quality processing based on the surgical imaging environment information, and prepares student data by simulating the images captured by the imaging device (source device 11).

The image data preparation unit 122 is configured to include, for example, a learning data set extraction unit 151, a level adjustment unit 152, a color temperature adjustment unit 153, development processing units 154-1 and 154-2, an image degradation processing unit 155, and a noise addition unit 156.

Note that for ease of explanation, the development processing units 154-1 and 154-2 are given subnumbers, but they are actually the same development processing unit 154, and only the input and output data is different.

The learning dataset extraction unit 151 extracts RAW image data from the learning dataset image DB 121 based on the imaging subject information and RAW image format information from the surgical imaging environment information, and outputs the extracted RAW image data to the level adjustment unit 152.

The level adjustment unit 152 adjusts the luminance level of the RAW image data supplied from the learning data set extraction unit 151 based on the illumination intensity information in the surgical imaging environment information. The level adjustment unit 152 outputs the image data after the level adjustment to the color temperature adjustment unit 153.

The color temperature adjustment unit 153 adjusts the color temperature of the image data supplied from the level adjustment unit 152 based on the lighting color temperature information in the surgical imaging environment information. The color temperature adjustment unit 153 outputs the image data after color temperature adjustment to the development processing unit 154-1 and the image degradation processing unit 155.

The development processing unit 154-1 performs development processing on the image data supplied from the color temperature adjustment unit 153 using the signal processing information in the imaging device that is part of the surgical imaging environment information, and outputs the image data after development processing to the learning unit 123 as teacher data.

The image degradation processing unit 155 performs image degradation processing using the frequency characteristics (MTF) of the imaging device on the image data supplied from the color temperature adjustment unit 153. The image degradation processing unit 155 outputs the image data after the image degradation processing to the noise addition unit 156.

The noise addition unit 156 performs noise addition processing on the image data supplied from the image degradation processing unit 155. The noise addition unit 156 outputs the image data after noise addition to the development processing unit 154-2.

The image data supplied from the development processing unit 154-2 and noise addition unit 156 is subjected to development processing using the signal processing information in the imaging device from the surgical imaging environment information, and the image data after development processing is output to the learning unit 123 as student data.

The learning unit 123 learns using the teacher data supplied from the development processing unit 154-1 and the student data supplied from the development processing unit 154-2 as input, and obtains image quality improvement parameters for image quality improvement process A (high resolution processing and noise removal processing).

In other words, the learning function unit 111 learns the high resolution processing parameters and the noise removal processing parameters that constitute the image quality improvement process A using teacher data simulated using the surgical imaging environment information and student data simulated using the surgical imaging environment information. This makes it possible to obtain image quality improvement parameters with higher accuracy for the image quality improvement process A.

The image quality improvement processing unit 23 can perform the image quality improvement process A that is optimal for the source device 11 by using more accurate image quality improvement parameters obtained by learning. This allows the image quality improvement process to be more optimal for the source device 11.

<Second Configuration of Imaging Video System>
The above-mentioned learning may be performed in an external device other than the sink device, and FIG. 7 shows an example in which learning of image quality improvement processing is performed in an external device other than the sink device.

FIG. 7 shows a second example of the configuration of a captured image processing system.

The captured image processing system 201 in FIG. 7 differs from the captured image processing system 1 in FIG. 1 in that a learning server 211 on a cloud 212, which is an external device other than the sink device 12, is added.

The sink device 12 transmits the surgical imaging environment information acquired from the source device 11 to the learning server 211 on the cloud 212. The sink device 12 receives the image quality improvement parameters obtained by learning transmitted from the learning server 211, and uses the received image quality improvement parameters for image quality improvement processing of the captured video data transmitted from the source device 11.

The sink device 12 displays high-definition video corresponding to the high-definition video data after image quality enhancement processing, for viewing by, for example, a surgeon.

The learning server 211 is configured to include the learning function unit 111 of FIG. 6. The learning server 211 performs learning using the surgical imaging environment information transmitted from the sink device 12, and obtains image quality improvement parameters for the image quality improvement process of the source device 11. The learning server 211 transmits the obtained image quality improvement parameters to the sink device 12.

<Third Configuration of Imaging Video System>
FIG. 8 is a diagram showing a third example of the configuration of the captured video processing system.

The captured image processing system 221 in FIG. 8 differs from the captured image processing system 1 in FIG. 1 in that a relay device 231 is added between the source device 11 and the sink device 12.

The source device 11 transmits the captured image data generated through imaging to the sink device 12 via the relay device 231. The source device 11 transmits the surgical imaging environment information to the sink device 12 via the relay device 231.

The relay device 231 transmits the captured video data transmitted from the source device 11 to the sink device 12. The relay device 231 transmits the surgical imaging environment information transmitted from the source device 11 to the sink device 12.

The sink device 12 acquires the surgical imaging environment information transmitted from the relay device 231, and selects, for example, from among the image quality improvement processes A to D, the image quality improvement process that is most suitable for the source device 11 based on the acquired surgical imaging environment information.

The sink device 12 performs the selected image quality improvement processing on the captured video data sent from the relay device 231 and outputs it.

As described above, the connection between the source device 11 and the sink device 12 may be an indirect connection with the relay device 231 intervening.

<Fourth Configuration of Imaging Video System>
FIG. 9 is a diagram showing a fourth example of the configuration of the captured video processing system.

The captured image processing system 251 in FIG. 9 differs from the captured image processing system 1 in FIG. 1 in that the sink device 12 is replaced with a sink device 260, and a medical monitor 261, a medical recorder 262, and an operating room video system 263 are added.

The sink device 260 differs from the sink device 12 in that a transmission unit 271 is added. For ease of explanation, the display control unit 24 and the display unit 25 are not shown in the sink device 260.

The high-image-quality processing unit 23 outputs the high-image-quality video data after the high-image-quality processing to the transmission unit 271. The transmission unit 271 transmits the high-image-quality video data after the high-image-quality processing to the medical monitor 261, the medical recorder 262, and the operating room video system 263.

The medical monitor 261, medical recorder 262, and operating room video system 263 record the high-definition video data transmitted from the sink device 260 and display high-definition video corresponding to the high-definition video data.

As described above, the video data after high-resolution processing may be transmitted to another device.

<Fifth Configuration of Imaging Video System>
FIG. 10 is a diagram showing a fifth example of the configuration of the captured video processing system.

The captured image processing system 281 in FIG. 10 differs from the captured image processing system 1 in FIG. 1 in that the source device 11 is replaced with source devices 11-1 and 11-2.

In other words, source devices 11-1 and 11-2 each transmit captured video data generated to sink device 12. Source devices 11-1 and 11-2 each transmit surgical imaging environment information to sink device 12.

The sink device 12 receives captured video data transmitted from the multiple source devices 11-1 and 11-2, and receives surgical imaging environment information transmitted from the multiple source devices 11-1 and 11-2.

The high image quality processing selection unit 22 selects the best image quality processing for the source device 11-1 from the high image quality processing DB based on the surgical imaging environment information from the source device 11-1 supplied by the surgical imaging environment information receiving unit 21. The high image quality processing unit 23 performs the high image quality processing selected by the high image quality processing selection unit 22 on the imaging video data sent from the source device 11-1, and outputs the high image quality video data after the high image quality processing to the display control unit 24.

The high image quality processing selection unit 22 selects the best image quality processing for the source device 11-2 from the high image quality processing DB based on the surgical imaging environment information from the source device 11-2 supplied from the surgical imaging environment information receiving unit 21. The high image quality processing unit 23 performs the high image quality processing selected by the high image quality processing selection unit 22 on the captured image data transmitted from the source device 11-2, and outputs the high image quality image data after the high image quality processing to the display control unit 24.

As described above, it is possible to perform high-quality image processing optimized for each of the source devices 11-1 and 11-2. As a result, even when multiple images are displayed simultaneously, such as in PIP (Picture In Picture), it is possible to achieve optimal high-quality image processing for each image.

<3. Other>
<Effects of this technology>
As described above, in the present technology, based on surgical imaging environment information, which is information relating to the imaging video data obtained by imaging the surgical site using an imaging device, a high image quality processing to be performed on the imaging video data is selected, and the selected high image quality processing is performed on the imaging video data.

This technology makes it possible to perform high-quality image processing suited to the imaging device equipment and surgical environment, and to obtain high-quality images. This makes it possible to obtain the high-quality images required at the surgical site, enabling precise and delicate procedures to be performed.

Furthermore, in the surgical imaging environment, the imaging environment during surgery (medical equipment used for imaging, lighting environment (lighting equipment and settings), imaging subject (organs, surgical tools)) is limited depending on the patient's case and the procedure being performed, and it is assumed that the environment will be the same, so the effect of using surgical imaging environment information to improve the image quality of images will be high.

<Example of computer configuration>
The above-mentioned series of processes can be executed by hardware or software. When the series of processes is executed by software, the program constituting the software is installed from a program recording medium into a computer incorporated in dedicated hardware or a general-purpose personal computer.

FIG. 11 is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes using a program.

CPU (Central Processing Unit) 301, ROM (Read Only Memory) 302, and RAM (Random Access Memory) 303 are interconnected by bus 304.

Further connected to the bus 304 is an input/output interface 305. Connected to the input/output interface 305 are an input unit 306 consisting of a keyboard, mouse, etc., and an output unit 307 consisting of a display, speakers, etc. Also connected to the input/output interface 305 are a storage unit 308 consisting of a hard disk or non-volatile memory, a communication unit 309 consisting of a network interface, etc., and a drive 310 that drives removable media 311.

In a computer configured as described above, the CPU 301, for example, loads a program stored in the storage unit 308 into the RAM 303 via the input/output interface 305 and the bus 304 and executes the program, thereby performing the above-mentioned series of processes.

The programs executed by the CPU 301 are provided, for example, by being recorded on removable media 311, or via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting, and are installed in the storage unit 308.

The program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or a program in which processing is performed in parallel or at the required timing, such as when called.

In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.

Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also exist.

The embodiment of this technology is not limited to the above-mentioned embodiment, and various modifications are possible without departing from the gist of this technology.

For example, this technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices over a network.

In addition, each step described in the above flowchart can be executed by a single device, or can be shared and executed by multiple devices.

Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device, or can be shared and executed by multiple devices.

<Examples of configuration combinations>
The present technology can also be configured as follows.
(1)
an image quality improvement processing selection unit that selects an image quality improvement processing to be performed on the captured image data based on surgical imaging environment information, which is information about the captured image data obtained by capturing an image of the surgical site by an imaging device;
an image quality improvement processing unit that performs an image quality improvement process selected by the image quality improvement processing selection unit on the captured video data.
(2)
The image processing device according to (1), wherein the surgical imaging environment information is at least one of information related to surgery, information related to the imaging device, signal processing information in the imaging device, and imaging environment information.
(3)
The image processing device according to (1) or (2), wherein the image quality improvement processing selection unit selects an image quality improvement processing associated with the surgical imaging environment information that at least partially coincides with the surgical imaging environment information.
(4)
The image processing device according to (3), wherein the image quality improvement processing selection unit selects a general-purpose image quality improvement processing when there is no image quality improvement processing associated with the surgical imaging environment information that at least partially matches the image quality improvement processing.
(5)
The image processing device described in (3), wherein, when there is no high image quality processing associated with the surgical imaging environment information that matches all of the surgical imaging environment information, the high image quality processing selection unit selects a high image quality processing associated with the surgical imaging environment information that partially matches the surgical imaging environment information.
(6)
The image processing device described in (3), wherein, when there is no high image quality processing associated with the surgical imaging environment information that matches all of the surgical imaging environment information, the high image quality processing selection unit creates a high image quality processing associated with the surgical imaging environment information that matches all of the surgical imaging environment information, and selects the created high image quality processing.
(7)
The image processing device according to any one of (1) to (6), wherein the image quality improvement processing is learned based on the surgical imaging environment information.
(8)
The image processing device described in (7) above, wherein the high image quality processing is learned using student data generated using the surgical imaging environment information and teacher data generated using the surgical imaging environment information.
(9)
The image processing device according to (7), further comprising a learning unit configured to learn pre-image quality improvement processing based on the surgical imaging environment information.
(10)
The image processing device described in (9), wherein the learning unit learns high image quality processing associated with the surgical imaging environment information using student data generated using the surgical imaging environment information and teacher data generated using the surgical imaging environment information.
(11)
The image processing device according to any one of (1) to (10), further comprising a display control unit that controls display of an image corresponding to the image data that has been subjected to image quality improvement processing.
(12)
The image processing device according to any one of (1) to (11), further comprising a transmission unit that transmits the image data that has been subjected to the image quality improvement processing to another device.
(13)
The image processing device according to any one of (1) to (12), wherein the image quality improvement processing includes at least one of high resolution processing and noise removal processing.
(14)
The image processing device according to any one of (1) to (13), further comprising a surgical imaging environment information acquisition unit that acquires the surgical imaging environment information.
(15)
The image processing device according to (14), wherein the surgical imaging environment information acquisition unit acquires the surgical imaging environment information by receiving the surgical imaging environment information transmitted by the imaging device.
(16)
The image processing device according to (14), wherein the surgical imaging environment information acquisition unit acquires frequency characteristic information of the surgical imaging environment information based on chart image data generated by capturing an image of a chart with a predetermined resolution by the imaging device.
(17)
The image processing device according to (14), wherein the surgical imaging environment information acquisition unit acquires noise characteristic information of the surgical imaging environment information based on chart image data generated by capturing an image of a predetermined color chart by the imaging device.
(18)
The image processing device
Selecting an image quality improvement process to be performed on the captured image data based on surgical imaging environment information, which is information about captured image data obtained by capturing an image of the surgical site with an imaging device;
An image processing method for performing selected image quality improvement processing on the captured video data.
(19)
an image quality improvement processing selection unit that selects an image quality improvement processing to be performed on the captured image data based on surgical imaging environment information, which is information about the captured image data obtained by capturing an image of the surgical site by an imaging device;
an image quality improvement processing unit that performs image quality improvement processing selected by the image quality improvement processing selection unit on the captured image data,
The programs that make a computer function.
(20)
An imaging device for obtaining imaging video data by imaging the surgical site;
an image quality improvement processing selection unit that selects an image quality improvement processing to be performed on the captured image data based on surgical imaging environment information that is information about the captured image data;
and an image processing device including an image quality improvement processing unit that performs an image quality improvement process selected by the image quality improvement processing selection unit on the captured video data.

1. Image capture and processing system, 11, 11-1, 11-2. Source device, 12. Sink device, 21. Surgical imaging environment information receiving unit, 22. High image quality processing selection unit, 23. High image quality processing unit, 24. Display control unit, 25. Display unit, 51. Sink device, 61. Frequency characteristic measurement unit, 62. Resolution chart, 81. Sink device, 82. Color chart, 91. Noise characteristic measurement unit, 101. Sink device, 111. Learning function unit, 121. Learning dataset image DB, 122. Image data preparation unit, 123. Learning unit, 15 1 Image data set extraction unit, 152 Level adjustment unit, 153 Color temperature adjustment unit, 154, 154-1, 154-2 Development processing unit, 155 Image degradation processing unit, 156 Noise addition unit, 201 Image capture processing system, 211 Learning server, 212 Cloud, 221 Image capture processing system, 231 Relay device, 251 Image capture processing system, 260 Sink device, 261 Medical monitor, 262 Medical recorder, 263 Operating room video system, 271 Transmission unit, 281 Image capture processing system

Claims (20)

an image quality improvement processing selection unit that selects an image quality improvement processing to be performed on the captured image data based on surgical imaging environment information, which is information about the captured image data obtained by capturing an image of the surgical site by an imaging device;
an image quality improvement processing unit that performs an image quality improvement process selected by the image quality improvement processing selection unit on the captured video data. The image processing device according to claim 1 , wherein the surgical imaging environment information is at least one of information related to surgery, information about the imaging device, signal processing information in the imaging device, and imaging environment information. The image processing device according to claim 1 , wherein the image quality improvement processing selection unit selects an image quality improvement processing associated with the surgical imaging environment information that at least partially coincides with the surgical imaging environment information. The image processing device according to claim 3 , wherein the image quality improvement process selection unit selects a general-purpose image quality improvement process when there is no image quality improvement process associated with the surgical imaging environment information that at least partially matches the image quality improvement process. The image processing device according to claim 3, wherein when there is no high image quality processing associated with the surgical imaging environment information that entirely matches the surgical imaging environment information, the high image quality processing selection unit selects a high image quality processing associated with the surgical imaging environment information that partially matches the surgical imaging environment information. 4. The image processing device according to claim 3, wherein, when there is no high image quality process associated with the surgical imaging environment information that completely matches the surgical imaging environment information, the high image quality process selection unit creates a high image quality process associated with the surgical imaging environment information that completely matches the surgical imaging environment information, and selects the created high image quality process. The image processing device according to claim 1 , wherein the image quality improvement processing is learned based on the surgical imaging environment information. The image processing device according to claim 7 , wherein the image quality improvement processing is performed by learning using student data generated using the surgical imaging environment information and teacher data generated using the surgical imaging environment information. The image processing device according to claim 7 , further comprising a learning unit configured to learn pre-image quality improvement processing based on the surgical imaging environment information. The image processing device according to claim 9 , wherein the learning unit learns image quality improvement processing associated with the surgical imaging environment information by using student data generated using the surgical imaging environment information and teacher data generated using the surgical imaging environment information. The image processing device according to claim 1 , further comprising a display control unit that controls display of an image corresponding to the image data that has been subjected to image quality improvement processing. The image processing device according to claim 1 , further comprising a transmission unit that transmits the image data that has been subjected to the high image quality processing to another device. The image processing device according to claim 1 , wherein the image quality improvement processing includes at least one of a high resolution processing and a noise removal processing. The image processing device according to claim 1 , further comprising a surgical imaging environment information acquisition unit for acquiring the surgical imaging environment information. The image processing device according to claim 14 , wherein the surgical imaging environment information acquisition unit acquires the surgical imaging environment information by receiving the surgical imaging environment information transmitted by the imaging device. The image processing device according to claim 14 , wherein the surgical imaging environment information acquisition unit acquires frequency characteristic information of the surgical imaging environment information based on chart image data generated by capturing an image of a chart with a predetermined resolution by the imaging device. The image processing device according to claim 14 , wherein the surgical imaging environment information acquisition unit acquires noise characteristic information from the surgical imaging environment information based on chart image data generated by capturing an image of a predetermined color chart by the imaging device. The image processing device
Selecting an image quality improvement process to be performed on the captured image data based on surgical imaging environment information, which is information about captured image data obtained by capturing an image of the surgical site with an imaging device;
An image processing method for performing selected image quality improvement processing on the captured video data. an image quality improvement processing selection unit that selects an image quality improvement processing to be performed on the captured image data based on surgical imaging environment information, which is information about the captured image data obtained by capturing an image of the surgical site by an imaging device;
an image quality improvement processing unit that performs image quality improvement processing selected by the image quality improvement processing selection unit on the captured image data,
The programs that make a computer function. An imaging device for obtaining imaging video data by imaging the surgical site;
an image quality improvement processing selection unit that selects an image quality improvement processing to be performed on the captured image data based on surgical imaging environment information that is information about the captured image data;
and an image processing device including an image quality improvement processing unit that performs an image quality improvement process selected by the image quality improvement processing selection unit on the captured video data.

Images