JP7779444B1 - Color conversion device, color conversion method, program, training device, image judgment device, and color conversion system - Google Patents

Abstract

The color conversion device disclosed herein includes a conversion rule storage unit that stores an inverse conversion rule of a first conversion rule that converts an image expressed in a first color space into an image expressed in a reference color space, and a second conversion rule that converts an image expressed in a second color space into an image expressed in the reference color space, and a conversion unit that converts a captured image that reflects color characteristics specific to the second imaging device into a reference image expressed in the reference color space according to the second conversion rule, and converts the reference image into a converted image expressed in the first color space according to the inverse conversion rule.

Description

The present disclosure relates to a color conversion device, a color conversion method, a program, a training device, an image assessment device, a machine learning model, and a color conversion system.

It is common to capture a subject using an imaging device and generate an image. Even when images are generated using the same imaging device, the color tone of the generated image may change if the conditions at the time of shooting, such as lighting, change. For this reason, there is a demand for technology to align the color tones of multiple images.

Patent Document 1 discloses a color correction device that performs color correction on image data of biological tissues obtained by imaging with a medical imaging device, to make it suitable for display on a color monitor. Furthermore, Patent Document 2 discloses an endoscopic device that records a color chart during an examination and corrects the color image during the next examination based on the color difference between a temporary color chart generated for the next examination and the color chart used during the previous examination.

International Publication No. 2020/166697 JP 2016-54758 A

When a subject is photographed using an imaging device and an image is generated, the generated image generally reflects the color characteristics specific to the imaging device. As a result, even if images of the same subject are photographed using different imaging devices, the color tones may differ.

The present disclosure aims to provide a color conversion device, color conversion method, program, training device, image assessment device, and machine learning model that can correct images generated by different imaging devices and bring color tones closer together.

A color conversion device according to one aspect of the present disclosure includes a conversion rule storage unit that stores an inverse conversion rule of a first conversion rule that converts an image expressed in a first color space into an image expressed in a reference color space, and a second conversion rule that converts an image expressed in a second color space into an image expressed in the reference color space, and a conversion unit that converts a captured image that reflects color characteristics specific to the second imaging device into a reference image expressed in the reference color space according to the second conversion rule, and converts the reference image into a converted image expressed in the first color space according to the inverse conversion rule.

A color conversion method according to one aspect of the present disclosure is a color conversion method executed by a color conversion device that stores an inverse conversion rule of a first conversion rule that converts an image expressed in a first color space into an image expressed in a reference color space, and a second conversion rule that converts an image expressed in a second color space into an image expressed in the reference color space. The color conversion method acquires a captured image expressed in the second color space, converts the captured image into a reference image expressed in the reference color space according to the second conversion rule, and converts the reference image into a converted image expressed in the first color space according to the inverse conversion rule.

A program according to one aspect of the present disclosure is stored on a non-transitory storage medium. The program is executed by a computer of a color conversion device that stores an inverse conversion rule of a first conversion rule for converting an image expressed in a first color space into an image expressed in a reference color space, and a second conversion rule for converting an image expressed in a second color space into an image expressed in the reference color space. The program causes the computer to perform the following steps: acquiring a captured image expressed in the second color space; converting the captured image into a reference image expressed in the reference color space in accordance with the second conversion rule; and converting the reference image into a converted image expressed in the first color space in accordance with the inverse conversion rule.

A training device according to one aspect of the present disclosure includes an acquisition unit that acquires a training dataset including training images expressed in a first color space and annotations to be judged, and a training unit that uses the training dataset to train a machine learning model that judges images expressed in the first color space, wherein the training images are generated by converting a captured image expressed in the second color space into a reference image expressed in the reference color space in accordance with a second transformation rule that converts an image expressed in the second color space into an image expressed in the reference color space, and converting the reference image into a transformed image expressed in the first color space in accordance with an inverse transformation rule of the first transformation rule that converts an image expressed in the first color space into an image expressed in the reference color space.

An image assessment device according to one aspect of the present disclosure includes an acquisition unit that acquires images expressed in a first color space, and a assessment unit that assesses images expressed in the first color space using a trained machine learning model that assesses images expressed in the first color space, wherein the trained machine learning model has been trained using a training dataset that includes training images expressed in the first color space and annotations to be assessed. The training images are generated by converting a captured image expressed in the second color space into a reference image expressed in the reference color space in accordance with a second transformation rule that converts an image expressed in a second color space into an image expressed in a reference color space, and converting the reference image into a transformed image expressed in the first color space in accordance with an inverse transformation rule of the first transformation rule that converts an image expressed in the first color space into an image expressed in the reference color space.

A machine learning model according to one aspect of the present disclosure is a machine learning model that causes a computer to acquire images expressed in a first color space and judge the images expressed in the first color space, wherein the machine learning model is trained using a training dataset that includes training images expressed in the first color space and annotations to be judged, and the training images are generated by converting a captured image expressed in the second color space into a reference image expressed in the reference color space in accordance with a second conversion rule that converts an image expressed in a second color space into an image expressed in a reference color space, and converting the reference image into a converted image expressed in the first color space in accordance with an inverse conversion rule of the first conversion rule that converts an image expressed in the first color space into an image expressed in the reference color space.

A color conversion system according to one aspect of the present disclosure is a color conversion system comprising the above-described color conversion device and a conversion rule generation device, wherein the conversion rule generation device generates the inverse conversion rule and the second conversion rule, and the color conversion device acquires the generated inverse conversion rule and the second conversion rule.

A color transformation system according to one aspect of the present disclosure is a color transformation system comprising the above-described color transformation device, a standard transformation rule generation device, and an inverse transformation rule generation device, wherein the transformation rule generation device generates the first transformation rule and the second transformation rule, the inverse transformation rule generation device generates the inverse transformation rule of the first transformation rule, and the color transformation device acquires the generated inverse transformation rule and the second transformation rule.

A color conversion device according to one aspect of the present disclosure includes a conversion rule storage unit that stores an inverse conversion rule of a first conversion rule that converts an image reflecting color characteristics specific to a first imaging device into a reference image that is not dependent on the color characteristics of a specific imaging device, and a second conversion rule that converts an image reflecting color characteristics specific to a second imaging device into the reference image; and a conversion unit that converts an image reflecting color characteristics specific to the second imaging device into the reference image according to the second conversion rule, and converts the reference image into a converted image having color characteristics specific to the first imaging device according to the inverse conversion rule.

Images produced by different imaging devices can be corrected to bring the color tones closer together.

FIG. 1 is a diagram showing an example of the configuration of a first example of a color conversion system. FIG. 1 is a diagram showing an example of the functional configuration of a color conversion device according to a first example; FIG. 1 is a diagram showing an example of a functional configuration of an imaging device; 1 is a flowchart illustrating an example of the operation of a color conversion device during a process for generating a conversion rule. 1 is a flowchart illustrating an example of the operation of a color conversion device when performing color conversion processing. FIG. 10 is a diagram showing an example of the configuration of a color conversion system according to a fourth modified example. FIG. 13 is a diagram showing an example of the functional configuration of a color conversion device according to Modification 4. FIG. 1 is a diagram showing an example of a functional configuration of a conversion rule generation device. FIG. 13 is a diagram showing an example of the configuration of a color conversion system according to a fifth modified example. FIG. 1 is a diagram showing an example of the functional configuration of a standard conversion rule generation device. FIG. 1 is a diagram showing an example of the functional configuration of an inverse conversion rule generation device. FIG. 10 is a diagram showing an example of the configuration of a second example of a color conversion system. FIG. 1 is a diagram showing an example of the functional configuration of a training device. Flowchart for explaining an example of the operation of the training device A flowchart for explaining an example of the operation of a color conversion system including a training device. FIG. 10 is a diagram showing an example of the configuration of a third example of a color conversion system. FIG. 1 is a diagram showing an example of a functional configuration of an image assessment device. Flowchart for explaining an example of operation of the image assessment device A flowchart for explaining an example of the operation of a color conversion system including an image determination device. FIG. 10 is a diagram showing an example of the hardware configuration of each device.

<Overview>
Multiple images may be generated using multiple image capture devices. Each image capture device has its own unique color characteristics, and images of the same subject may be generated with different color tones.

When an observer views multiple images like this, differences in color tone between the images can affect their observations. For example, if an observer is accustomed to the color tones of images generated by a particular imaging device, they may feel uncomfortable when viewing images generated by a different imaging device due to the differences in color tone.

The present disclosure relates to a color conversion device that performs color conversion processing to bring the color tones of images generated by multiple imaging devices other than a specific imaging device closer to the color tones of images generated by the specific imaging device.

The color conversion device of the present disclosure can be applied to, for example, the following cases. Pathological diagnoses are commonly performed in which an observer, such as a pathologist, examines a pathology image created by extracting tissue from a subject to determine the presence and type of a lesion. A pathology image is, for example, an image obtained by taking a magnified photograph of a specimen obtained by extracting, fixing, staining, and thinly slicing a portion of the subject's tissue. Pathology images are generated by photographing the specimen using an imaging device with a magnification function, such as a microscope, placed in a hospital.

When an observer observes a pathology image and makes a diagnosis, the color or intensity of the stained tissue can be an important clue. When observers make a diagnosis using pathology images generated by various imaging devices, the color tone of the pathology image can change due to the color characteristics specific to the imaging device, which can have a negative impact on the diagnosis.

One way to resolve this situation is to correct the image generated by the imaging device, for example, to an image with the original color tone of the stained specimen, so that the color tone specific to the imaging device is not reflected in the image observed by the observer.

However, observers who have observed many pathological images with color tones specific to a particular imaging device may find the corrected color tones unnatural. Here, a specific imaging device refers to an imaging device that generates images that the observer has observed many times in the past. In such cases, it becomes difficult for the observer to make a diagnosis based on their past experience. It is desirable for the annotation work, in which observers add diagnostic information to images, to be performed using color tones that the observer is familiar with. There is a concern that annotation work performed on images with unfamiliar color tones due to correction will significantly reduce work efficiency.

In addition, pathology image analysis AI (Artificial Intelligence) has been developed that uses a trained machine learning model trained using a large number of pathology images and label data corresponding to each pathology image to assist observers in diagnosing new pathology images.

In such pathology image analysis AI, there are two ways to train a machine learning model: using pathology images of various color tones, or training by standardizing on pathology images of one color tone. These methods are used separately or in combination depending on the purpose of the pathology image analysis AI and the images used. For example, training by using pathology images of various color tones is thought to improve the robustness of the trained machine learning model. On the other hand, training by standardizing on pathology images of one color tone is thought to improve learning efficiency.

In light of such cases, the color conversion device disclosed herein converts images of various color tones generated by various imaging devices into images of a specific color tone based on the color characteristics unique to a specific imaging device. Here, the specific color tone may be a color tone dependent on a specific imaging device that generates images that an observer has observed many times in the past. Alternatively, the specific color tone may be a color tone dependent on a specific imaging device that generates images that an image analysis AI machine learning model has learned many times in the past. This allows observers who have observed many images of specific color tones dependent on a specific imaging device to make observations and diagnoses using their past knowledge. Furthermore, by correcting images of various color tones into images of specific color tones dependent on a specific imaging device and using them as training data for the image analysis AI machine learning model, accurate diagnostic assistance can be provided for new pathological images generated by a specific imaging device.

Note that the following description will discuss devices and methods for handling pathology images, but the present disclosure is not limited to this. The devices and methods of the present disclosure can handle various images other than pathology images, as long as they are digitized images.

Below, the color conversion device disclosed herein and a color conversion system including the color conversion device are described in detail, using specific examples.

<First Example>
First, a first example of a color conversion system including a color conversion device 10 according to an embodiment of the present disclosure will be described.

[Configuration example]
1 is a diagram showing an example of the configuration of a first example of a color conversion system. The color conversion system 100 according to the first example includes a color conversion device 10, a plurality of image capture devices 20, and an observation device 30.

The multiple imaging devices 20 each have a camera that captures an image of the subject and generates a digital image. More specifically, the multiple imaging devices 20 are devices that generate pathological images. This disclosure does not particularly limit the number of multiple imaging devices, but the following description will describe a case where the multiple imaging devices 20 include a first imaging device 20_1 and a second imaging device 20_2, as shown in FIG. 1.

In the following description, the first imaging device 20_1 and the second imaging device 20_2 may be collectively referred to as imaging device 20, or may refer to either the first imaging device 20_1 or the second imaging device 20_2.

The first imaging device 20_1 generates an image expressed in a first color space. The second imaging device 20_2 generates an image expressed in a second color space. In this embodiment, the first color space or the second color space may each be an RGB color space. In the following description, the image generated by the first imaging device 20_1 will be referred to as the first image, and the image generated by the second imaging device 20_2 will be referred to as the second image.

The first image reflects color characteristics specific to the first imaging device. The second image reflects color characteristics specific to the second imaging device. The first image and the second image each have a different color tone. In this way, each imaging device included in the multiple imaging devices 20 generates an image that reflects different color characteristics.

In the following description, the first imaging device 20_1 is assumed to be a specific imaging device, i.e., an imaging device that generates images that the observer has observed many times in the past.

The color conversion device 10 does not perform color conversion processing on the first image acquired from the first imaging device 20_1 of the multiple imaging devices 20, but performs color conversion processing on the second image acquired from the second imaging device 20_2. In this case, the color conversion processing is a process of converting an image that reflects color characteristics specific to the second imaging device 20_2 into an image that reflects color characteristics specific to the first imaging device 20_1. In the following description, an image that has undergone color conversion processing by the color conversion device 10 will be referred to as a converted image.

The observation device 30 is a device that allows an observer to observe the pathological images generated by the first imaging device 20_1 or the second imaging device 20_2 and perform diagnosis, etc. The observation device 30 is, for example, a type of computer equipped with a display device, such as a PC (Personal Computer) or a tablet terminal.

The observation device 30 acquires and displays a first image from the first imaging device 20_1. The observation device 30 also acquires and displays a converted image, which is a color conversion process performed on the second image generated by the second imaging device 20_2, from the color conversion device 10. This allows the observer to observe an image that reflects the color characteristics unique to the first imaging device 20_1, regardless of which of the multiple imaging devices 20 generated the image.

Figure 2 is a diagram showing an example of the functional configuration of the color conversion device 10 in the first example. As shown in Figure 2, the color conversion device 10 includes a communication unit 11, a memory unit 12, and a control unit 13. The color conversion device 10 is, for example, a type of computer such as a PC, a workstation, or a tablet terminal.

The communication unit 11 sends and receives various information between the imaging device 20 or the observation device 30 via wired or wireless communication.

The memory unit 12 stores various information necessary for the color conversion process performed by the color conversion device 10. The information stored in the memory unit 12 includes, for example, information about the conversion rules used in the color conversion process, and images before and after the color conversion process is performed. The memory unit 12 has a conversion rule storage unit 121 that stores various conversion rules.

The control unit 13 controls the operation of the color conversion device 10. The control unit 13 has an acquisition unit 131, a conversion rule generation unit 132, and a conversion unit 133.

The acquisition unit 131 acquires the image on which color conversion processing is to be performed. In the example shown in Figure 1, the acquisition unit 131 acquires the second image received from the second imaging device 20_2 by the communication unit 11.

The conversion rule generation unit 132 generates conversion rules used in the color conversion process. The conversion rules are, for example, ICC (International Color Consortium) profiles, which indicate the correspondence between the color values of the image before conversion and the color values of the image after conversion.

More specifically, the conversion rule generation unit 132 first generates an ICC profile that converts the image generated by the imaging device 20 into a reference image that is not dependent on the color characteristics of the imaging device 20.

In the present embodiment, the reference image is an image expressed in a reference color space and is independent of the color characteristics of the imaging device 20. The reference color space may be, for example, the well-known L ^* a ^* b ^* color space or XYZ color space. The reference color space may be referred to as a profile connection space. In the example shown in FIG. 1 , the transformation rule generation unit 132 generates a first transformation rule for transforming an image generated by the first imaging device 20_1, which reflects color characteristics specific to the first imaging device 20_1, into the reference image, and a second transformation rule for transforming an image generated by the second imaging device 20_2, which reflects color characteristics specific to the second imaging device 20_2, into the reference image.

The first conversion rule is a rule for converting an image expressed in a first color space that reflects color characteristics specific to the first imaging device 20_1 into a reference image expressed in a reference color space that is independent of the color characteristics of the imaging device 20. The conversion rule generation unit 132 determines the first conversion rule based on the correspondence between the color values of each pixel in the first chart image and the colorimetric values, using the first chart image acquired by the first imaging device 20_1 photographing the first color chart and the colorimetric values of the first color chart acquired in advance. Alternatively, if the original color of an object appearing in the image reflecting the color characteristics specific to the first imaging device 20_1 is known, the conversion rule generation unit 132 may generate the first conversion rule so as to match or approximate the color values of each pixel in the reference image to the original color of the object. Alternatively, the conversion rule generation unit 132 may generate the first conversion rule using the color values of each color patch in the first chart image and the colorimetric values of each color patch of the first color chart acquired in advance.

The first conversion rule is generated, for example, in the form of a look-up table (LUT). Alternatively, the first conversion rule may be generated, for example, in the form of a matrix. Table 1 shows an example of the first conversion rule expressed in LUT format. Table 1 shows that the values ( _r1 , _g1 , _b1 ) indicated by RGB on the left side of Table 1 correspond to the values (x1 _{, y1} , _z1 ) indicated by XYZ on the same row.

The second conversion rule is a rule for converting an image expressed in the second color space and reflecting color characteristics specific to the second imaging device 20_2 into a reference image expressed in the reference color space and independent of the color characteristics of the imaging device 20. The conversion rule generation unit 132 uses the second chart image acquired by the second imaging device 20_2 photographing the second color chart and the colorimetric values of the second color chart acquired in advance, and determines the second conversion rule based on the correspondence between the color values of each pixel in the second chart image and the colorimetric values. Alternatively, if the original color of an object appearing in the image reflecting color characteristics specific to the second imaging device 20_2 is known, the conversion rule generation unit 132 may generate the second conversion rule so as to match the color value of each pixel in the reference image to the original color of the object.

Like the first transformation rule, the second transformation rule may be generated, for example, in the form of an LUT or in the form of a matrix.

The first chart image and the second chart image may be prepared in advance by, for example, an administrator of the color conversion system 100, or may be generated by the first imaging device 20_1 and the second imaging device 20_2 as needed. The colorimetric values of each color patch of the first color chart included in the first chart image and the colorimetric values of each color patch of the second color chart included in the second chart image are known. Prior to the conversion rule generation process by the conversion rule generation unit 132, the color values of each color patch in the first chart image and the colorimetric values of each color patch of the first color chart included in the first chart image are input to the color conversion device 10.

The first chart image and the second chart image may be different images. More specifically, the first color chart and the second color chart may each include different color patches, for example, color patches of different colors.

The first color chart and the second color chart may each include at least red, green, blue, and white color patches.

More specifically, the red color patches included in the first and second color charts have a transmission spectrum with a peak wavelength in the range of 620 nm to 680 nm and a half-width of 100 nm or less. The green color patches included in the first and second color charts have a transmission spectrum with a peak wavelength in the range of 510 nm to 540 nm and a half-width of 80 nm or less. The blue color patches included in the first and second color charts have a transmission spectrum with a peak wavelength in the range of 450 nm to 472 nm and a half-width of 80 nm or less. The white color patches included in the first and second color charts can be made of a base material that is not colored with dyes, pigments, or other colorants. Examples of base material materials include glass, film, acrylic resin, polycarbonate, gelatin, and paper.

By including such color patches in the first color chart and the second color chart, the coordinates of the color patches of each color on the xy chromaticity diagram can be made to match or approximate the coordinates of each color specified in the BT.2020 standard. This makes it possible to perform suitable color calibration when the BT.2020 standard is applied to the imaging device 20.

Note that the red, green, and blue colors contained in the first color patch are not limited to the examples above, and may be, for example, within the following ranges on the xy chromaticity diagram.

The color of the red color patch included in the first color patch may be within the area bounded by coordinates on the xy chromaticity diagram (0.735, 0.265), (0.627, 0.372), the curve portion of the blackbody locus BL where the color deviation between color temperatures of 1200 K and 6000 K is -0.01, and (0.520, 0.174). The blackbody locus refers to the locus on the chromaticity diagram that depicts the state of a completely black object as it changes with absolute temperature.

The color of the green color patch included in the first color patch may be within the area surrounded by the coordinates on the xy chromaticity diagram: (0.005, 0.587), (0.014, 0.750), (0.074, 0.834), (0.155, 0.806), (0.345, 0.652), and the curve of the blackbody locus BL with a color deviation of +0.02 between color temperatures of 6000 K and above and 7000 K and below.

The color of the blue color patch included in the first color patch may be within the area surrounded by the coordinates on the xy chromaticity diagram: (0.174, 0.005), (0.131, 0.046), (0.073, 0.185), the straight line connecting the points on the blackbody locus BL where the color deviation is +0.02 and -0.01 at a color temperature of 15000 K, the curve portion on the blackbody locus BL where the color deviation is -0.01 at color temperatures from 15000 K to 7000 K, and (0.280, 0.054).

By using a chart image containing such color patches, it is possible to generate conversion rules that can support a wide color gamut.

Furthermore, as an example of a color chart other than those described above, the following color chart may be adopted to make the pathology images handled by the color conversion system 100 easier to see. Such a first color chart and second color chart each include at least a first color patch, and the coordinate point of the first color patch is within the triangular area formed by connecting the three points (0.200, 0.070), (0.450, 0.280), and (0.310, 0.340) on the xy chromaticity diagram with straight lines.

Only one of the first color chart and the second color chart may include such a first color patch. With this configuration, if the pathology image reflecting the color characteristics of the first image capture device 20_1 deviates significantly (has a strong quirk) from the actual stained specimen, and the pathology image reflecting the color characteristics of the second image capture device 20_2 does not deviate significantly (has a weak quirk) from the actual specimen, using the first color patch in the first color chart can produce highly accurate conversion coefficients for the first image capture device with a strong quirk, and efficiently produce conversion coefficients for the second image capture device with a weak quirk.

If the pathology image is an image of cells stained with hematoxylin and eosin (HE), the coordinate points of the first color patch can be set within the above range to bring the color of the color patch closer to the characteristic color of the pathology image. This makes it possible to generate a first or second conversion rule that enables conversion to bring the color closer to the characteristic color of the pathology image. Therefore, the first or second conversion rule can be used to perform color correction so that the converted image is closer to the color of the actual object (stained cells).

The coordinate points of the first color patch may be within a pentagonal area on the xy chromaticity diagram formed by connecting the following five points with straight lines in that order: (0.310, 0.340), (0.350, 0.300), (0.350, 0.200), (0.260, 0.150), and (0.280, 0.250). Having the coordinate points of the first color patch within the above pentagonal area allows for more accurate color correction.

The coordinate points of the first color patch may also be within a rectangular area on the xy chromaticity diagram formed by connecting four points (0.311, 0.334), (0.347, 0.210), (0.270, 0.162), and (0.292, 0.216) with straight lines. Having the coordinate points of the first color patch within the rectangular area allows for even more accurate color correction.

The first color chart and the second color chart may further include a second color patch that is a different color from the first color patch. In this case, the coordinate points of both the first color patch and the second color patch may be within any of the above-mentioned regions on the xy chromaticity diagram. By having the coordinate points of multiple color patches within any of the above-mentioned regions, it is possible to generate a first conversion rule or a second conversion rule that can perform color correction on images that have color tones characteristic of pathology images.

Furthermore, the conversion rule generation unit 132 generates an inverse conversion rule for performing the inverse conversion of the first conversion rule based on the first conversion rule generated as described above. The inverse conversion rule is a rule for converting a reference image that is expressed in a reference color space and does not depend on the color characteristics of the imaging device into an image that reflects the color characteristics specific to the first imaging device 20_1. The inverse conversion rule can be created, for example, using the following method. Note that the following describes the case where the reference color space is the XYZ color space and the conversion rule is expressed in LUT format.

LUT-format inverse transformation rules can be created, for example, by at least the following three steps. The three steps include a first step of preparing a set of grid points in a reference color space, a second step of determining the relationship between each grid point in the set prepared in the first step and the grid points in the reference color space in the LUT, which is the first transformation rule for transforming from the first color space to the reference color space, and a third step of determining, from the relationship determined in the second step, grid points in the first color space that correspond to each grid point in the set prepared in the first step.

An example of a method for creating an inverse transformation rule in LUT format will be described in detail below. First, the transformation rule generation unit 132 creates a set of lattice points in the XYZ color space. The following describes an example in which the definition of ISO 15076-1:2010 is used as the LUT format. In this example, a three-dimensional LUT format is used as the LUT format. The three-dimensional LUT is defined as (X _i , Y _j , Z _k ). When i, j, and k are specified, the converted RGB values corresponding to each specified value are defined. Table 2 shows an example of a set of lattice points in the XYZ color space. Table 2 shows a case in which the ranges available for XYZ are equally divided for each of X, Y, and Z. If each range available for XYZ is equally divided into eight, the number of lattice points will be 9 cubed. min and max respectively indicate the minimum and maximum lattice point numbers in the range available for XYZ. If it is divided into eight equal parts, min is 0 and max is 8.

The values of X _min , Y _min , and Z _min may be preset to a predetermined value, for example, 0. The values of X _max , Y _max , and Z _max may be preset to a predetermined value, for example, 128. Alternatively, the values of X _min , Y _min , and Z _min may be the minimum values in the XYZ color space that can be obtained when each color value in the RGB color space is changed from the minimum value (for example, 0) to the maximum value (for example, 255). The values of X _max , Y _max , and Z _max may be the maximum values in the XYZ space that can be obtained when each color value in the RGB color space is changed from the minimum value (for example, 0) to the maximum value (for example, 255).

Next, the conversion rule generation unit 132 plots the color values of all grid points in the XYZ color space corresponding to all grid points in the RGB color space in the LUT of the first conversion rule in the XYZ color space, and creates a solid in the XYZ space corresponding to the RGB color space.

Next, the conversion rule generation unit 132 checks the position of each grid point in the XYZ color space shown in Table 1 within the solid. The conversion rule generation unit 132 performs interpolation as necessary based on the correspondence between each grid point in the XYZ color space and a position on the solid. The interpolation can be performed using known methods, such as a spline function or linear interpolation. If a grid point in the XYZ color space shown in Table 1 is located outside the solid, the conversion rule generation unit 132 may replace the grid point so that it is located on the outermost or innermost hull of the solid. Examples of replacement methods include replacing the grid point with a point that minimizes the color difference or a point with the same hue. Alternatively, the grid point may be replaced with the outermost grid point in the solid that minimizes the color difference with a grid point in the XYZ color space shown in Table 1. Alternatively, the grid point may be replaced with the outermost grid point in the solid that has the same hue as a grid point in the XYZ color space shown in Table 1.

This allows the conversion rule generation unit 132 to generate an LUT, i.e., an inverse conversion rule, that converts the grid points in the XYZ color space shown in Table 1 into grid points in the RGB color space.

The method for creating inverse transformation rules by the transformation rule generation unit 132 has been specifically described above, but the method for creating inverse transformation rules in this disclosure is not limited to the above-described method. For example, if the first transformation rule is expressed as a determinant (e.g., 3 x 3), the transformation rule generation unit 132 can create an inverse transformation rule by calculating its inverse matrix.

The conversion rule generation unit 132 may generate the first conversion rule, the second conversion rule, and the inverse conversion rule and store them in the conversion rule storage unit 121 of the memory unit 12 before the color conversion process is performed by the conversion unit 133. Alternatively, the conversion rule generation unit 132 may perform the color conversion process and the process of generating new conversion rules in parallel or consecutively.

The conversion unit 133 performs color conversion processing on the second image acquired from the second imaging device 20_2 using the first conversion rule, the second conversion rule, and the inverse conversion rule.

As described above, the color conversion process converts the second image, which is expressed in the second color space and reflects color characteristics specific to the second imaging device 20_2, into an image expressed in the first color space and reflects color characteristics specific to the first imaging device 20_1. The conversion unit 133 first uses the second conversion rules to convert the second image acquired from the second imaging device 20_2 into a reference image, which is expressed in the reference color space and is independent of the color characteristics of the imaging device 20. Then, the conversion unit 133 uses the inverse conversion rules to convert the reference image into a converted image, which is expressed in the first color space and has color characteristics specific to the first imaging device 20_1.

The converted image converted by the color conversion process is output to the observation device 30 via the communication unit 11. As a result, as described above, the observation device 30 acquires the first image generated by the first imaging device 20_1 and the converted image. The observer can observe the first image or the converted image displayed by the observation device 30 and make a diagnosis. Here, both the first image and the converted image have color characteristics unique to the first imaging device 20_1. Therefore, for an observer who has observed many first images captured by the first imaging device 20_1 in the past, the first image or the converted image displayed on the observation device 30 will have a familiar color tone because they both have the color characteristics of the first imaging device 20_1. In this way, the color conversion system 100 can convert an image reflecting color characteristics different from those unique to a specific imaging device 20 into an image reflecting the color characteristics unique to a specific imaging device 20, so that an observer accustomed to seeing images reflecting the color characteristics unique to a specific imaging device 20 can easily observe them.

[Example of operation]
Next, a description will be given of an example of operation in the first configuration example of the color conversion system 100. Fig. 4 is a flowchart for explaining an example of operation of the conversion rule generation process in the color conversion system 100.

In step S1, the first imaging device 20_1 and the second imaging device 20_2 capture images of a first color chart and a second color chart, respectively, which have been prepared in advance, and generate a first chart image and a second chart image, respectively. The generated first chart image and second chart image are acquired by the color conversion device 10.

In step S2, the conversion rule generation unit 132 of the color conversion device 10 generates a first conversion rule based on the first chart image and a second conversion rule based on the second chart image. The above-described methods may be used as appropriate to generate each conversion rule.

In step S3, the conversion rule generation unit 132 generates an inverse conversion rule based on the first conversion rule. The method for generating the inverse conversion rule may be any of the methods described above. The conversion rules and inverse conversion rules generated in steps S2 and S3 are stored in the conversion rule storage unit 121 of the color conversion device 10 and used in the color conversion process described below.

Next, we will explain an example of the operation of the color conversion process in the color conversion system 100. Figure 5 is a flowchart for explaining an example of the operation of the color conversion process in the color conversion system 100.

In step S11, the acquisition unit 131 of the color conversion device 10 acquires a second image from the second imaging device 20_2.

In step S12, the conversion unit 133 of the color conversion device 10 converts the second image into a reference image using the second conversion rule.

In step S13, the conversion unit 133 of the color conversion device 10 converts the reference image into a converted image using the inverse conversion rules. The converted image generated in step S13 is transmitted to the observation device 30.

In step S14, the observation device 30 displays the converted image. This allows the observer to observe the converted image, even if the object to be observed was photographed by the second imaging device 20_2, as an image that has been converted into an image having color characteristics specific to the first imaging device 20_1. This makes it much easier for an observer who is accustomed to observing images that reflect color characteristics specific to the first imaging device 20_1 to observe the converted image than if they simply observed an image generated by the second imaging device 20_2 that reflects color characteristics specific to the second imaging device 20_2.

<Modification of the First Example> (Modification 1)
In the first example described above, the multiple imaging devices 20 include the first imaging device 20_1 and the second imaging device 20_2. In the present disclosure, the number of imaging devices 20 is not particularly limited, and the multiple imaging devices 20 may include imaging devices other than the first imaging device 20_1 and the second imaging device 20_2.

When the multiple imaging devices in the color conversion system include three or more imaging devices, the color conversion device acquires images from imaging devices other than a specific imaging device (first imaging device 20_1 in the example shown in Figure 1) and performs color conversion processing on each image. In this case, before performing the color conversion processing, the color conversion device generates conversion rules for converting images generated by each imaging device into a reference image and inverse conversion rules for converting the reference image into an image reflecting color characteristics specific to the specific imaging device. The color conversion device first uses the conversion rules to convert images generated by imaging devices other than the specific imaging device into the reference image, and then uses inverse conversion rules to convert the reference image into an image reflecting color characteristics specific to the specific imaging device, thereby generating a converted image with color characteristics specific to the specific imaging device.

If the multiple imaging devices in the color conversion system include three or more imaging devices, an imaging device other than the specific imaging device is an example of a second imaging device of the present disclosure, and a color space representing an image generated by an imaging device other than the specific imaging device is an example of a second color space of the present disclosure.

(Variation 2)
In the first example described above, the color conversion device 10 performs color conversion processing on the second image generated by the second imaging device 20_2. However, in the present disclosure, for example, an observation device may have the functionality of a color conversion device and perform color conversion processing. In this case, the observation device acquires the first image and the second image from the first imaging device and the second imaging device, respectively, and does not perform color conversion processing on the first image, but performs color conversion processing on the second image before displaying it. This allows an observer to perform diagnosis, etc., by observing the first image or the converted image displayed on the observation device.

(Variation 3)
In addition, in the present disclosure, among the multiple imaging devices, an imaging device other than a specific imaging device, for example, a second imaging device, may have the function of a color conversion device and perform color conversion processing. In this case, the first imaging device directly transmits the generated first image to the observation device, and the second imaging device performs color conversion processing on the generated second image and then transmits the converted image to the observation device. This allows an observer to perform diagnosis, etc. by observing the first image or the converted image displayed on the observation device. In this case, the imaging device is a type of computer having a processor for performing color conversion processing.

The second imaging device may obtain the inverse transformation rules for transforming the reference image into the transformed image from an external device. Alternatively, the second imaging device may obtain the first chart image and information regarding the colorimetric values of the first color chart, and create the inverse transformation rules based on these.

Pathological images tend to have large data volumes, so color conversion processing can require relatively large processing power or time. By having an imaging device other than the specific imaging device perform the color conversion processing, the processing power or processing time required for the color conversion processing can be distributed among multiple imaging devices. For this reason, having imaging devices other than the specific imaging device each perform the color conversion processing is particularly effective when, for example, there are a relatively large number of imaging devices.

(Variation 4)
In the first example described above, the color conversion device 10 has the conversion rule generation unit 132, which generates various conversion rules, but in the present disclosure, the color conversion device 10 may acquire conversion rules from an external device. In this case, the color conversion device 10 does not need to have the conversion rule generation unit 132.

Figure 6 is a diagram showing an example of the configuration of a color conversion system in variant example 4. The color conversion system 100A in variant example 4 comprises a color conversion device 10A, multiple imaging devices 20, an observation device 30, and a conversion rule generation device 60.

Figure 7 is a diagram showing an example of the functional configuration of a color conversion device 10A in variant example 4. As shown in Figure 7, the color conversion device 10A in variant example 4 differs from the color conversion device 10 of the first example shown in Figure 2 in that it does not have a conversion rule generation unit. The functions and operations of each component of the color conversion device 10A shown in Figure 7 are similar to the functions and operations of each component described in the first example above.

Figure 8 is a diagram showing an example of the functional configuration of the conversion rule generation device 60. As shown in Figure 8, the conversion rule generation device 60 includes a communication unit 61, an acquisition unit 62, and a conversion rule generation unit 63.

The communication unit 61 transmits and receives various information between the color conversion device 10A and multiple imaging devices 20 via wired or wireless communication.

The acquisition unit 62 acquires a first chart image and a second chart image for generating each conversion rule.

The conversion rule generation unit 63 generates a first conversion rule and a second conversion rule, and also generates an inverse conversion rule of the first conversion rule. The method used by the conversion rule generation unit 63 to generate each conversion rule may be the same as the method used by the conversion rule generation unit 132 of the color conversion device 10 in the first example described above.

The second transformation rule and inverse transformation rule generated by the transformation rule generation device 60 are transmitted to the color transformation device 10A. The color transformation device 10A performs the color transformation process described in the first example above using the second transformation rule and inverse transformation rule obtained from the transformation rule generation device 60.

As explained above, in variant example 4, a transformation rule generation device 60 external to color transformation device 10A generates the second transformation rule and the inverse transformation rule, and color transformation device 10A acquires the second transformation rule and the inverse transformation rule from the transformation rule generation device 60 and performs the color transformation process. According to variant example 4, color transformation device 10A does not need to have the function of generating each transformation rule. This allows for reduction in manufacturing and operating costs of color transformation device 10A.

(Variation 5)
In the above-described modification 4, the conversion rule generation device 60 generates the first conversion rule and the second conversion rule, and generates the inverse conversion rule of the first conversion rule. In the present modification 5, an example will be described in which a device for generating the first conversion rule and the second conversion rule and a device for generating the inverse conversion rule of the first conversion rule are provided separately.

Figure 9 is a diagram showing an example of the configuration of a color conversion system in variant example 5. The color conversion system 100B in variant example 5 comprises a color conversion device 10A, multiple imaging devices 20, an observation device 30, a standard conversion rule generation device 70, and an inverse conversion rule generation device 80.

The configuration and operation of the color conversion device 10A are the same as those of variant example 4.

Figure 10 is a diagram showing an example of the functional configuration of the standard conversion rule generation device 70. As shown in Figure 10, the standard conversion rule generation device 70 includes a communication unit 71, an acquisition unit 72, and a standard conversion rule generation unit 73.

The communication unit 71 transmits and receives various information between multiple imaging devices 20 and the inverse conversion rule generation device 80 via wired or wireless communication.

The acquisition unit 72 acquires a first chart image and a second chart image for generating each conversion rule.

The standard conversion rule generation unit 73 generates the first conversion rule and the second conversion rule. The method by which the conversion rule generation unit 63 generates the first conversion rule and the second conversion rule may be the same as the generation method by the conversion rule generation unit 132 of the color conversion device 10 in the first example described above.

The first transformation rule generated by the standard transformation rule generation device 70 is transmitted to the inverse transformation rule generation device 80. The second transformation rule generated by the standard transformation rule generation device 70 may be transmitted to the color transformation device 10A via the inverse transformation rule generation device 80, or may be transmitted to the color transformation device 10A without passing through the inverse transformation rule generation device 80.

Figure 11 is a diagram showing an example of the functional configuration of the inverse conversion rule generation device 80. As shown in Figure 11, the inverse conversion rule generation device 80 includes a communication unit 81, an acquisition unit 82, and an inverse conversion rule generation unit 83.

The communication unit 81 sends and receives various information between the color conversion device 10A and the standard conversion rule generation device 70 via wired or wireless communication.

The acquisition unit 72 acquires at least the first conversion rule from the standard conversion rule generation device 70.

The inverse conversion rule generation unit 83 generates an inverse conversion rule of the first conversion rule based on the first conversion rule. The method for generating the inverse conversion rule by the inverse conversion rule generation unit 83 may be the same as the generation method by the conversion rule generation unit 132 of the color conversion device 10 in the first example described above.

The inverse conversion rules generated by the inverse conversion rule generation device 80 are transmitted to the color conversion device 10A. The color conversion device 10A then performs the color conversion process described in the first example above using the acquired second conversion rule and inverse conversion rule.

As explained above, in variant example 5, a standard conversion rule generation device 70 external to color conversion device 10A generates the first conversion rule and the second conversion rule, an inverse conversion rule generation device 80 external to color conversion device 10A generates the inverse conversion rule of the first conversion rule, and color conversion device 10A acquires the second conversion rule and the inverse conversion rule and performs the color conversion process. According to variant example 5, color conversion device 10A does not need to have the function of generating each conversion rule. This allows for reduction in manufacturing and operating costs of color conversion device 10A.

(Variation 6)
In the present disclosure, the multiple imaging devices may belong to the same facility, such as the same medical institution (such as a hospital or clinic) or the same research institute, or may belong to different facilities. Furthermore, the facilities to which the imaging devices belong may be established in different countries.

Because the number of pathologists who can accurately diagnose pathology images is limited, such pathologists often receive a large number of requests for pathology image diagnoses from multiple hospitals. Without the color conversion device of the present disclosure, the requested pathologist would need to make a diagnosis using pathology images in unfamiliar color tones. However, with a color conversion system including the color conversion device of the present disclosure, even in such cases, the pathologist can make a diagnosis using pathology images in familiar color tones.

(Variation 7)
In the first example described above, an example of operation was described in which the color conversion device 10 performs the process of generating a conversion rule before the color conversion process. As described above, in the present disclosure, the color conversion device may perform the color conversion process and the process of generating a new conversion rule in parallel or consecutively.

For example, a color conversion device acquires, from a new imaging device other than a specific imaging device, a chart image generated by the imaging device by capturing an image of a color chart, information regarding the colorimetric values of the color chart, and an image generated by the imaging device by capturing an object. Based on the chart image and the information regarding the colorimetric values, the color conversion device generates a conversion rule for converting an image that reflects color characteristics specific to the new imaging device into a reference image, and generates the reference image using the acquired image and the conversion rule. The color conversion device then converts the reference image using a previously generated inverse conversion rule for converting the reference image into an image that reflects color characteristics specific to the specific imaging device, to generate a converted image. In this way, when acquiring an image from a new imaging device that generates an image for which a conversion rule for converting to a reference image has not been generated, the conversion rule generation process and the color conversion process may be performed in parallel or sequentially.

For example, consider a pathologist who collects images generated by multiple imaging devices from multiple hospitals and performs diagnoses using these images. In this case, the pathologist may receive a request from a new hospital to perform a diagnosis using images. In such a case, by having the color conversion device acquire the chart image and observation image generated by a new imaging device from the new hospital at approximately the same time, a converted image with a color tone familiar to the observer can be generated without any problems, even if conversion rules for converting an image that reflects the color characteristics unique to the imaging device into a reference image have not been generated in advance.

<Second Example>
Next, a second example of a color conversion system including the color conversion device 10 according to an embodiment of the present disclosure will be described.

[Configuration example]
12 is a diagram showing an example of the configuration of a second example of a color conversion system. A color conversion system 200 according to the second example includes a color conversion device 10, multiple image capture devices 20, and a training device 40. The color conversion device 10 and the multiple image capture devices 20 are the same as those described in the first example above.

The training device 40 uses images (pathological images) acquired directly from multiple imaging devices 20 or via the color conversion device 10 as training images to train the machine learning model to be trained. The machine learning model to be trained is a machine learning model that causes the image assessment device 50 described below to acquire images that are expressed in a first color space and reflect color characteristics specific to the first imaging device 20_1, and to assess the images that are expressed in the first color space and reflect color characteristics specific to the first imaging device 20_1. The training device 40 is a type of computer such as a PC, workstation, or tablet terminal.

Assessing an image expressed in the first color space and reflecting color characteristics specific to the first image capture device 20_1 means performing a judgment process that assists an observer in making a diagnosis using a pathology image, such as determining the presence or absence of a lesion in the pathology image or the location or type of the lesion. To train a machine learning model that can perform such a judgment process, the training device 40 trains the machine learning model to be trained using a training dataset that includes training images expressed in the first color space and reflecting color characteristics specific to the first image capture device 20_1, and label data for the training images.

The label data for the training images includes, for example, annotations of the target to be determined that are included in the training images. The target to be determined is, for example, whether or not the pathology image contains a lesion. In this case, the label data includes, for example, mask data that obscures the lesion in the training image. Alternatively, the target to be determined may be, for example, the type of tissue to which a site of interest included in the pathology image is classified. Alternatively, the target to be determined may be, for example, if a pathology image contains a lesion, the type of disease that the lesion is caused by. The annotation is, for example, correct data indicating whether or not the pathology image contains a lesion (i.e., segmentation of the lesion), the type of tissue to which the site of interest included in the training image is classified, or the type of disease that the lesion included in the training image is caused by.

Figure 13 is a diagram showing an example of the functional configuration of the training device 40. The training device 40 includes a communication unit 41, an operation unit 42, an acquisition unit 43, a training unit 44, and a memory unit 45.

The communication unit 41 sends and receives various information between the color conversion device 10 or the imaging device 20 via wired or wireless communication.

The operation unit 42 accepts an operation to add annotations to the first image and the converted image. The operation to add annotations to the first image and the converted image is performed, for example, by a pathologist.

The acquisition unit 43 acquires a training dataset including training images expressed in a first color space and reflecting color characteristics specific to the first imaging device 20_1, and annotations to be determined. The training images include a first image received from the first imaging device 20_1 and a converted image received from the color conversion device 10.

The memory unit 45 stores the machine learning model 46 to be trained.

The training unit 44 uses the training dataset to train a machine learning model 46 to be trained, which judges images that are expressed in the first color space and reflect color characteristics specific to the first imaging device 20_1. By performing the training process using a sufficient number of images, a trained machine learning model that can perform highly accurate judgment processing can be generated.

In this way, in a second example of a color conversion system including a color conversion device according to the present disclosure, the training device conducts training using images that reflect color characteristics specific to a specific imaging device, so that the color conversion device converts images generated by an imaging device other than the specific imaging device into converted images that have color characteristics specific to the specific imaging device. This increases the number of images with specific color tones included in the training data. A trained machine learning model trained using such converted images can effectively assist pathologists in making accurate diagnoses of pathological images.

In the second example, as in the first example, the number of imaging devices is not limited to two. Furthermore, if sufficient processing power can be ensured, the training device may, for example, have the functionality of a color conversion device and be able to perform conversion rule generation processing and color conversion processing.

[Example of operation of training device 40]
FIG. 14 is a flowchart for explaining an example of the operation of the training process in the training device 40.

In step S21, the acquisition unit 43 of the training device 40 acquires the first image transmitted from the first imaging device 20_1. The training device 40 may acquire multiple first images.

In step S22, the acquisition unit 43 acquires the converted image transmitted from the color conversion device 10. The training device 40 may acquire multiple converted images. The order in which steps S21 and S22 are performed may be reversed. Furthermore, steps S21 and S22 may be performed in parallel.

In step S23, the operation unit 42 accepts annotation input operations by a pathologist or the like.

In step S24, the training unit 44 trains the machine learning model 46 to be trained based on the first image, the converted image, and the annotations.

The above steps S21 to S24 may be performed repeatedly.

By performing this operation, even if multiple imaging devices 20 generate images that reflect their own unique color characteristics, the training device 40 can train the machine learning model 46 to be trained using images with specific color tones. This eliminates the need for correction processing or setting training parameters related to the color characteristics unique to each of the multiple imaging devices 20, making it possible to generate a trained machine learning model that can make accurate judgments with little effort.

In the example shown in FIG. 14, the training device 40 accepts annotation input operations via the operation unit 42, but the present disclosure is not limited to this. For example, a device other than the training device may generate annotations. In this case, the acquisition unit of the training device may acquire the annotations from an external device along with the first image and the converted image.

In addition, in the present disclosure, the training device may perform a certain amount of training on the machine learning model to be trained using supervised learning that utilizes annotations, and then perform training using unsupervised learning using first images and converted images that do not have annotations. In this case, the effort required to prepare annotations can be reduced.

Furthermore, in the present disclosure, the training images may include a second image received from the second imaging device 20_2 and a converted image received from the color conversion device. The converted image is an image having color characteristics specific to the first imaging device 20_1. By including images that reflect the color characteristics of the different imaging devices, i.e., images with different color tones, in the training images, a highly robust trained machine learning model can be generated.

[Example of operation of color conversion system 200]
FIG. 15 is a flowchart for explaining an example of the operation of the color conversion system 200 including the training device 40.

In step S31, the color conversion device 10 acquires a second image from the second imaging device 20_2.

In step S32, the color conversion device 10 converts the second image into a reference image using the second conversion rule.

In step S33, the conversion unit 133 of the color conversion device 10 uses the inverse conversion rules to convert the reference image into a converted image that reflects the color characteristics specific to the first imaging device 20_1. The converted image generated in step S33 is transmitted to the training device 40.

In step S34, the training device 40 executes the training process described in FIG. 14.

In this way, in the color conversion system 200, the color conversion device 10 converts the image generated by the second imaging device 20_2 into a converted image having color characteristics specific to the first imaging device 20_1, and the training device 40 performs training using the converted image. This allows the training device 40 to easily increase the number of images with specific color tones included in the training data. Furthermore, a trained machine learning model trained using such converted images can effectively assist pathologists in making accurate diagnoses when diagnosing pathological images.

<Third Example>
Next, a third example of a color conversion system including the color conversion device 10 according to an embodiment of the present disclosure will be described.

[Configuration example]
16 is a diagram showing an example of the configuration of a third example of a color conversion system. The color conversion system 300 according to the third example includes a color conversion device 10, a plurality of image capture devices 20, and an image assessment device 50. The color conversion device 10 and the plurality of image capture devices 20 are the same as those described in the first example above.

The image assessment device 50 performs a judgment process in which images (pathological images) acquired directly from multiple imaging devices 20 or via the color conversion device 10 are input into a trained machine learning model, which outputs a judgment result. The trained machine learning model is a machine learning model generated by the training device 40 performing the training process in the second example described above. In other words, the trained machine learning model is a machine learning model trained using images expressed in the first color space and reflecting color characteristics unique to the first imaging device 20_1. The image assessment device 50 is a type of computer, such as a PC, workstation, or tablet terminal.

Figure 17 is a diagram showing an example of the functional configuration of the image assessment device 50. The image assessment device 50 comprises a communication unit 51, an acquisition unit 52, a assessment unit 53, and a memory unit 54.

The communication unit 51 sends and receives various information between the color conversion device 10 or the imaging device 20 via wired or wireless communication.

The acquisition unit 52 acquires the first image from the first imaging device 20_1 and the converted image from the color conversion device 10. Both the first image and the converted image are images that reflect the color characteristics specific to the first imaging device 20_1.

The memory unit 54 stores a trained machine learning model 55.

The determination unit 53 uses the trained machine learning model 55 to determine an image that is expressed in the first color space and reflects color characteristics specific to the first imaging device 20_1. More specifically, the determination unit 53 inputs at least one of the newly acquired first image and the converted image into the trained machine learning model 55, and generates a determination result based on the output result. The determination result is output to a device used by pathologists and others to assist in diagnosis, such as a display device not shown in FIG. 16.

In this way, in a third example of a color conversion system having a color conversion device according to the present disclosure, the image assessment device can accurately assess new images that reflect color characteristics specific to a specific imaging device using a trained machine learning model trained using images that reflect color characteristics specific to a specific imaging device.

In the third example, as in the first or second example, the number of imaging devices is not limited to two. Furthermore, if sufficient processing power can be ensured, for example, the image assessment device may have the functionality of a color conversion device and be able to perform conversion rule generation processing and color conversion processing.

[Example of operation of image assessment device 50]
FIG. 18 is a flowchart for explaining an example of the operation of the image determination process in the image determination device 50.

In step S41, the acquisition unit 52 of the image assessment device 50 acquires the first image transmitted from the first imaging device 20_1. The image assessment device 50 may acquire multiple first images.

In step S42, the acquisition unit 52 acquires the converted image transmitted from the color conversion device 10. The image assessment device 50 may acquire multiple converted images. The order in which steps S41 and S42 are performed may be reversed. Furthermore, steps S41 and S42 may be performed in parallel.

In step S43, the judgment unit 53 inputs at least one of the first image obtained in step S41 and the converted image obtained in step S42 into the trained machine learning model 55.

In step S44, the determination unit 53 generates a determination result based on the output result from the trained machine learning model 55. The determination result may be the output result itself, or may have been processed in some way. An example of processing is when the output result is text information, and the determination result is displayed in the form of a table or diagram that makes it easy for a pathologist to understand the content at a glance.

This operation enables highly accurate judgment processing to be performed on input images that reflect the color characteristics unique to a particular imaging device 20.

[Example of operation of color conversion system 300]
FIG. 19 is a flowchart for explaining an example of the operation of the color conversion system 300 including the image assessment device 50.

In step S51, the color conversion device 10 acquires a second image from the second imaging device 20_2.

In step S52, the color conversion device 10 converts the second image into a reference image using the second conversion rule.

In step S53, the conversion unit 133 of the color conversion device 10 converts the reference image into a converted image having color characteristics specific to the first imaging device 20_1 using the inverse conversion rules. The converted image generated in step S53 is transmitted to the training device 40.

In step S54, the image assessment device 50 performs the image assessment process described in Figure 18.

In this way, in the color conversion system 300, the image assessment device 50 can accurately assess new images that reflect the color characteristics unique to the first imaging device 20_1 using a trained machine learning model trained using images that reflect the color characteristics unique to the first imaging device 20_1.

<Example of hardware configuration>
Each of the components in the above-described embodiments has been described as a type of computer. Below, an example of a computer hardware configuration for realizing the functions of each of these components will be described.

Figure 20 shows an example of the hardware configuration of a computer for realizing the functions of each component of the color conversion system.

The computer 1000 comprises an input device 1001, an output device 1002, a CPU 1003, a ROM (Read Only Memory) 1004, a RAM (Random Access Memory) 1005, a storage device 1006, a reading device 1007 that reads information from various recording media, and a transmission/reception device 1008, and each part is connected by a bus 1009.

The reading device 1007 reads a program for realizing the functions of each component of the color conversion system from a non-transitory storage medium on which the program is recorded, and stores the program in the storage device 1006. Alternatively, the transmitting/receiving device 1008 communicates with a system device connected to the network, and stores the program downloaded from the system device in the storage device 1006.

Then, the CPU 1003 copies the program stored in the storage device 1006 to the RAM 1005, and sequentially reads and executes the instructions contained in the program from the RAM 1005, thereby realizing the functions of each component.

The disclosures of the specification, drawings and abstract contained in the Japanese application No. 2024-049772, filed on March 26, 2024, are incorporated herein by reference in their entirety.

This disclosure is useful for color conversion devices that perform color conversion processing on images.

100, 100A, 100B, 200, 300 Color conversion system 10, 10A Color conversion device 11 Communication unit 12 Memory unit 121 Conversion rule storage unit 13 Control unit 131 Acquisition unit 132 Conversion rule generation unit 133 Conversion unit 20 Imaging device 20_1 First imaging device 20_2 Second imaging device 30 Observation device 40 Training device 41 Communication unit 42 Operation unit 43 Acquisition unit 44 Training unit 45 Memory unit 46 Machine learning model to be trained 50 Image judgment device 51 Communication unit 52 Acquisition unit 53 Judgment unit 54 Memory unit 55 Trained machine learning model 60 Conversion rule generation device 61 Communication unit 62 Acquisition unit 63 Conversion rule generation unit 70 Standard conversion rule generation device 71 Communication unit 72 Acquisition unit 73 Standard conversion rule generation unit 80 Inverse conversion rule generation device 81 Communication unit 82 Acquisition unit 83 Inverse conversion rule generation unit

Claims (18)

a conversion rule storage unit that stores an inverse conversion rule of a first conversion rule for converting an image expressed in a first color space into an image expressed in a reference color space, and a second conversion rule for converting an image expressed in a second color space into an image expressed in the reference color space;
a conversion unit that converts the captured image expressed in the second color space into a reference image expressed in the reference color space in accordance with the second conversion rule, and converts the reference image into a converted image expressed in the first color space in accordance with the inverse conversion rule;
and
the first conversion rule is determined based on a first chart image acquired by photographing a first color chart with a first imaging device and colorimetric values of the first color chart;
the second conversion rule is determined based on a second chart image acquired by photographing a second color chart with a second imaging device and colorimetric values of the second color chart;
Color conversion device. the image expressed in the first color space and the converted image expressed in the first color space are images that reflect color characteristics of a first imaging device;
The color conversion device according to claim 1 . the first color chart and the second color chart each include a different color patch;
The color conversion device according to claim 1 . the first color chart and the second color chart each include at least red, green, blue, and white color patches;
The color conversion device according to claim 1 . The reference color space is an L ^* a ^* b ^* color space or an XYZ color space.
The color conversion device according to claim 1 . the first color space and the second color space are RGB color spaces;
The color conversion device according to claim 1 . the first conversion rule includes a first look-up table indicating a correspondence relationship between coordinate values in the first color space and coordinate values in the reference color space;
the second conversion rule includes a second lookup table indicating a correspondence relationship between coordinate values in the second color space and coordinate values in the reference color space;
The color conversion device according to claim 1 . The captured image is a pathological image.
The color conversion device according to claim 1 . the first color chart for determining the first conversion rule and the second color chart for determining the second conversion rule each include at least a first color patch;
The coordinate point of the first color patch is within a triangular area formed by connecting three points (0.200, 0.070), (0.450, 0.280), and (0.310, 0.340) with straight lines on the xy chromaticity diagram.
9. The color conversion device according to claim 8. 1. A color conversion method executed by a color conversion device that stores an inverse conversion rule of a first conversion rule that converts an image expressed in a first color space into an image expressed in a reference color space, and a second conversion rule that converts an image expressed in a second color space into an image expressed in the reference color space,
A process of acquiring a captured image expressed in the second color space;
a process of converting the captured image into a reference image expressed in the reference color space in accordance with the second conversion rule;
and converting the reference image into a converted image expressed in the first color space in accordance with the inverse conversion rule;
the first conversion rule is determined based on a first chart image acquired by photographing a first color chart with a first imaging device and colorimetric values of the first color chart;
the second conversion rule is determined based on a second chart image acquired by photographing a second color chart with a second imaging device and colorimetric values of the second color chart;
Color conversion method. A program executed by a computer of a color conversion device that stores an inverse conversion rule of a first conversion rule for converting an image expressed in a first color space into an image expressed in a reference color space, and a second conversion rule for converting an image expressed in a second color space into an image expressed in the reference color space,
acquiring a captured image expressed in the second color space;
a step of converting the captured image into a reference image expressed in the reference color space in accordance with the second conversion rule;
transforming the reference image into a transformed image expressed in the first color space according to the inverse transformation rule;
causing the computer to execute
the first conversion rule is determined based on a first chart image acquired by photographing a first color chart with a first imaging device and colorimetric values of the first color chart;
the second conversion rule is determined based on a second chart image acquired by photographing a second color chart with a second imaging device and colorimetric values of the second color chart;
program. an acquisition unit that acquires a training dataset including training images expressed in a first color space and annotations to be determined;
a training unit that uses the training dataset to train a machine learning model that determines images expressed in the first color space;
and
The training images are
converting the captured image expressed in the second color space into a reference image expressed in the reference color space according to a second conversion rule for converting an image expressed in the second color space into an image expressed in the reference color space;
converting the reference image into a converted image expressed in the first color space according to an inverse conversion rule of a first conversion rule for converting an image expressed in the first color space into an image expressed in the reference color space;
This training device is generated by the captured image is a pathological image,
The machine learning model outputs a determination result in response to an input of the pathological image, the determination target of which is whether or not the pathological image includes a lesion.
13. The training device of claim 12. the captured image is a pathological image,
The machine learning model outputs a determination result in response to an input of the pathological image, the determination target being a tissue into which a site of interest included in the pathological image is classified.
13. The training device of claim 12. a determination unit that determines an image expressed in a first color space using a trained machine learning model that determines an image expressed in the first color space;
and
the trained machine learning model is trained using a training dataset including training images represented in the first color space and annotations to be determined;
The training images are
converting the captured image expressed in the second color space into a reference image expressed in the reference color space according to a second conversion rule for converting an image expressed in the second color space into an image expressed in the reference color space;
converting the reference image into a converted image expressed in the first color space according to an inverse conversion rule of a first conversion rule for converting an image expressed in the first color space into an image expressed in the reference color space;
An image assessment device that is generated by the above. A color conversion system comprising the color conversion device according to claim 1 and a conversion rule generation device,
the conversion rule generation device generates the inverse conversion rule and the second conversion rule;
the color transformation device acquires the generated inverse transformation rule and the second transformation rule;
Color conversion system. A color transformation system comprising the color transformation device according to claim 1, a standard transformation rule generation device, and an inverse transformation rule generation device,
the standard conversion rule generation device generates the first conversion rule and the second conversion rule;
the inverse conversion rule generation device generates the inverse conversion rule of the first conversion rule;
the color transformation device acquires the generated inverse transformation rule and the second transformation rule;
Color conversion system. a conversion rule storage unit that stores an inverse conversion rule of a first conversion rule that converts an image that reflects color characteristics specific to a first imaging device into a reference image that does not depend on color characteristics of a specific imaging device, and a second conversion rule that converts an image that reflects color characteristics specific to a second imaging device into the reference image;
a conversion unit that converts a captured image in which color characteristics specific to the second imaging device are reflected into the reference image in accordance with the second conversion rule, and converts the reference image into a converted image having color characteristics specific to the first imaging device in accordance with the inverse conversion rule;
A color conversion device comprising: