CN106483338A - Image output device, picture transmitter device, image received device, image output method - Google Patents

Abstract

The present disclosure provides an image output device, an image sending device, an image receiving device, and an image output method. The image output device (1001) includes: an image acquisition unit (1101) that acquires a first-resolution image; and a high-resolution image acquisition unit (1102) that acquires a second-resolution image that is an image higher in resolution than the first resolution ); an enlargement accepting unit (1201) that accepts a magnification; a determination unit (1202) that determines whether an evaluation value based on the accepted magnification is higher than a predetermined value; If it is high, the second resolution image is transmitted, and when it is determined that the evaluation value is not higher than the predetermined value, the second resolution image is not transmitted.

Description

Image output device, image sending device, image receiving device, image output method

technical field

The present disclosure relates to an image output device, an image transmission device, an image reception device, and the like.

Background technique

Conventionally, optical microscopes have been used to observe microstructures in living tissue and the like. An optical microscope uses light transmitted through an object to be observed or reflected light. The observer observes the image enlarged by the lens. There is also known a digital microscope that captures an image enlarged by a lens of the microscope and displays it on a monitor. By using a digital microscope, simultaneous observation by multiple people, remote observation, etc. can be realized.

In recent years, a technique of observing microstructures using a CIS (Contact Image Sensing; Contact Image Sensing) method has attracted attention. In the case of the CIS method, the observation target is placed close to the imaging surface of the image sensor. As an image sensor, a two-dimensional image sensor in which a large number of photoelectric conversion parts are arranged in rows and columns within an imaging surface is generally used. The photoelectric conversion unit is typically a photodiode formed on a semiconductor layer or a semiconductor substrate, and receives incident light to generate charges.

The image acquired by an image sensor is defined by a large number of pixels. Each pixel is divided by a unit area including one photoelectric conversion portion. Therefore, the resolving power (resolution) of a two-dimensional image sensor generally depends on the arrangement pitch (pitch) of the photoelectric conversion parts on the imaging surface. In this specification, the resolution determined by the arrangement pitch of the photoelectric conversion parts may be referred to as the "intrinsic resolution" of the image sensor. Since the arrangement pitch of each photoelectric conversion part is as short as about the wavelength of visible light, it is difficult to further improve the intrinsic resolution.

Techniques for realizing resolving power (ie, high resolution) exceeding the inherent resolving power of image sensors have been proposed. Patent Document 1 discloses a technique of forming an image of a subject (ie, a high-resolution image) using a plurality of images obtained by shifting the imaging position of the subject.

prior art literature

Patent Document 1: Japanese Patent Laid-Open No. 62-137037

Contents of the invention

However, there is a problem that a high-resolution image has a large amount of data, and processing loads such as transmission, reception, and storage of the image are heavy.

A non-limiting exemplary aspect of the present disclosure is an image output device, an image transmitting device, and an image receiving device capable of reducing the burden of processes such as transmission, reception, and storage of high-resolution images. Additional advantages and advantages of one embodiment of the present disclosure will be clarified from this specification and the accompanying drawings. The benefits and/or advantages can be provided independently by the various modes and features disclosed in the specification and drawings, and not all of them are necessary to obtain more than one benefit and/or advantages.

An image output device according to one aspect of the present disclosure includes: at least one processor; and a non-transitory recording medium holding a command for causing the at least one processor to execute: acquiring a first resolution image, the The first resolution image is an image of the object to be photographed based on the shooting of the digital microscope; the second resolution image is obtained, and the second resolution image is higher in resolution than the first resolution image based on the obtained the image of the object to be photographed obtained by the digital microscope; receiving the magnification of the first resolution image displayed on the display device; determining whether the evaluation value based on the received magnification is larger than a predetermined value high; when it is determined that the evaluation value is higher than the predetermined value, the second resolution image is sent to the external device; when it is determined that the evaluation value is not higher than the predetermined value, the The external device sends the second resolution image.

In addition, these comprehensive or specific methods can be implemented by a system, method, integrated circuit, computer program or computer-readable recording medium, or can be implemented by any combination of systems, methods, integrated circuits, computer programs or recording media. accomplish. The computer-readable recording medium includes, for example, nonvolatile recording media such as CD-ROM (Compact Disc-ReadOnly Memory).

According to the present disclosure, it is possible to reduce the load of processing such as transmission, reception and storage of high-resolution images.

Description of drawings

FIG. 1 is a configuration diagram showing an example of an image processing system including an image output device in Embodiment 1. As shown in FIG.

FIG. 2 is a diagram showing an example of a display screen displayed on a display device in Embodiment 1. FIG.

FIG. 3 is a flowchart showing an example of processing operations of the image output device in Embodiment 1. FIG.

FIG. 4 is a flowchart showing another example of processing operations of the image output device in Embodiment 1. FIG.

5 is a configuration diagram showing an example of an image processing system including an image output device according to a modified example of Embodiment 1. FIG.

FIG. 6 is a configuration diagram showing an example of an image processing system including an image output device in Embodiment 2. FIG.

FIG. 7 is a flowchart showing the processing operations of the video transmission device and the video reception device in Embodiment 2. FIG.

8 is a configuration diagram showing an example of an image processing system including an image output device according to Modification 1 of Embodiment 2. FIG.

9 is a flowchart showing the processing operations of the video transmission device and the video reception device according to Modification 1 of Embodiment 2. FIG.

10 is a configuration diagram showing an example of an image processing system including an image output device according to Modification 2 of Embodiment 2. FIG.

11 is a flowchart showing the processing operation of the image transmission device and the image reception device including Modification 2 of Embodiment 2.

FIG. 12 is a configuration diagram showing an example of an image processing system including an image output device in Embodiment 3. FIG.

13 is a configuration diagram showing an example of an image processing system including an image output device according to Modification 1 of Embodiment 3. FIG.

14 is a configuration diagram showing another example of an image processing system including an image output device according to Modification 2 of Embodiment 3. FIG.

FIG. 15A is a plan view schematically showing a part of the subject 2 .

FIG. 15B is a plan view schematically showing extracted and schematically shown photodiodes related to imaging of the region shown in FIG. 15A .

FIG. 16A is a cross-sectional view schematically showing the direction of light rays that pass through the subject 2 and enter the photodiode 4p.

FIG. 16B is a cross-sectional view schematically showing the direction of light rays that pass through the subject 2 and enter the photodiodes 4p.

FIG. 16C is a diagram schematically showing 6 pixels Pa captured by 6 photodiodes 4p.

FIG. 17A is a cross-sectional view schematically showing a state where light is made incident from an irradiation direction different from the irradiation directions shown in FIGS. 16A and 16B .

FIG. 17B is a cross-sectional view schematically showing a state where light is made incident from an irradiation direction different from the irradiation direction shown in FIGS. 16A and 16B .

FIG. 17C is a diagram schematically showing six pixels Pb acquired in the irradiation directions shown in FIGS. 17A and 17B .

18A is a cross-sectional view schematically showing a state where light is incident from an irradiation direction different from the irradiation direction shown in FIGS. 16A and 16B and the irradiation direction shown in FIGS. 17A and 17B .

18B is a cross-sectional view schematically showing a state where light is incident from an irradiation direction different from the irradiation direction shown in FIGS. 16A and 16B and the irradiation direction shown in FIGS. 17A and 17B .

FIG. 18C is a diagram schematically showing six pixels Pc acquired in the irradiation directions shown in FIGS. 18A and 18B .

FIG. 19A schematically shows that light is incident from an irradiation direction different from the irradiation direction shown in FIGS. 16A and 16B , the irradiation direction shown in FIGS. 17A and 17B , and the irradiation direction shown in FIGS. 18A and 18B . cutaway view of the state.

FIG. 19B is a diagram schematically showing six pixels Pd acquired in the irradiation direction shown in FIG. 19A .

FIG. 20 is a diagram showing a high-resolution image HR synthesized from four sub-images Sa, Sb, Sc, and Sd.

FIG. 21 is a cross-sectional view schematically showing irradiation directions adjusted so that light beams passing through two adjacent areas of the subject 2 are respectively incident on different photodiodes.

FIG. 22A is a diagram schematically showing an example of a cross-sectional structure of a module.

FIG. 22B is a plan view showing an example of the appearance of the module 10 shown in FIG. 22A viewed from the image sensor 4 side.

FIG. 23 is a diagram for explaining an example of a method of creating a module.

FIG. 24A is a cross-sectional view showing an example of an irradiation angle when sub-images are acquired.

24B is a cross-sectional view showing an example of a method of irradiating a subject at an irradiation angle different from that shown in FIG. 24A .

FIG. 25 is a schematic diagram illustrating an example of the configuration of an image acquisition device according to an embodiment of the present disclosure.

Fig. 26A is a perspective view showing an exemplary appearance of the image acquisition device 100a.

FIG. 26B is a perspective view showing a state in which the cover portion 120 is closed in the image pickup device 100 a shown in FIG. 26A .

FIG. 26C is a diagram schematically showing an example of a method of loading the socket 130 on the stage 32 of the image acquisition device 100a.

FIG. 27A is a diagram schematically showing an example of a method of changing the irradiation direction.

FIG. 27B is a diagram schematically showing changes in the direction of light rays incident on a subject when the stage 32 is tilted by an angle θ with respect to the reference plane.

FIG. 28 is a diagram showing an example of a cross-sectional structure of a CCD image sensor and a distribution of relative transmittance Td of a subject.

29A is a diagram illustrating an example of a cross-sectional structure of a backside illuminated CMOS image sensor and a distribution of relative transmittance Td of a subject.

29B is a diagram illustrating an example of a cross-sectional structure of a back-illuminated CMOS image sensor and a distribution of relative transmittance Td of a subject.

FIG. 30 is a diagram illustrating an example of a cross-sectional structure of a photoelectric conversion film stacked image sensor and a distribution of relative transmittance Td of a subject.

Label description

100S, 100Sa, 200S, 200Sa, 200Sb image processing system; 1001, 1001a, 2001, 2001a image output device; 1100, 1100a, 1100b image sending device; 1101 image acquisition unit; 1102 high resolution image acquisition unit; 1103 second output 1200, 1200a, 1200b image receiving device; 1201 enlargement acceptance unit; 1202 judgment unit; 1203 display output unit; 1204, 1204a transmission unit; 1205 first output unit; Device; 1502, 1503 recording medium

detailed description

(Knowledge that became the basis of the present invention)

The inventors of the present invention have found the following problems with regard to the high-resolution technology described in the "Background Art" section.

In a pathological examination using a microscope, the magnification of a lens used varies depending on the pathology of each specimen (also referred to as a pathological specimen) to be observed. For example, in the examination of cancer, first, observation is performed with a 10-fold objective lens. For a sample that does not have an area with a high possibility of being suspected of being cancer, the observation is completed with only the observation of the objective lens at 10 magnification, but if there is an area with a high possibility of being suspected of being cancer, the objective lens is switched to 40 magnification lens for inspection. In this way, the magnification of the lens is changed according to the specimen. Since it is not possible to determine whether a high magnification is required before the examination, the specimen is imaged at each predetermined magnification from high magnification to low magnification.

When an image of a specimen is captured by a digital microscope, the higher the resolution of the image, the longer the imaging time and image size (size). If the size of an image equivalent to 10 times the object obtained by a digital microscope is 100 Mbytes, the size of an image equivalent to 40 times the object is 1.6 Gbytes.

In addition, the number of pathological specimens that must be examined increases, while the increase in pathologists cannot keep up with the increase in the number of pathological specimens. Therefore, in the future, it is assumed that pathological images, which are images obtained by a digital microscope, will be transmitted remotely, and pathological diagnosis will be performed by a pathologist at a remote location. In addition, with the development of overall medical care, it is difficult for a single pathologist to diagnose all organs or symptoms. Therefore, it is necessary to perform remote diagnosis, that is, to transmit images of specimens that are difficult to diagnose by on-site pathologists to remote pathologists with specialized knowledge for diagnosis. However, there is a problem that it takes time to transmit high-resolution pathological images.

In order to solve such a problem, an image output device according to an aspect of the present invention includes: an image acquisition unit that acquires a first-resolution image, the first-resolution image being an image of an object captured by a digital microscope; a high-resolution image acquiring unit, configured to acquire a second-resolution image of the object to be photographed based on photographing by the digital microscope with a resolution higher than that of the first-resolution image an image of an enlarged image; an enlargement receiving unit accepts a magnification factor for the first resolution image displayed on the display device; a determination unit determines whether an evaluation value based on the magnification rate accepted by the magnification receiving unit is higher than a predetermined value and a transmission unit that transmits the second resolution image to an external device when the evaluation value is determined by the determination unit to be higher than the predetermined value, and when the determination unit determines that the evaluation value is not higher than the predetermined value; When the predetermined value is high, the second resolution image is not sent to the external device.

Thus, the second resolution image is transmitted only when the evaluation value is high. That is, it is possible to transmit, for example, a high-resolution image of a sample (subject) that is sufficiently necessary for diagnosis to a remote location. Therefore, it is possible to suppress the delay in sending the high-resolution image of a sample that is sufficient for diagnosis in order to send, for example, a high-resolution image of a low-importance sample that can be diagnosed in a short time. In addition, since high-resolution images of specimens with low evaluation values are not transmitted, it is possible to reduce the processing load of high-resolution images.

In addition, the transmission unit may transmit the first resolution image to the external device when the determination unit determines that the evaluation value is not higher than the predetermined value.

Thus, for example, a pathologist at a remote location can also diagnose the specimen using the transmitted low-resolution image, that is, the first resolution image.

In addition, the image output device may further include a first output unit configured to output the second resolution image when it is determined by the determination unit that the evaluation value is higher than the predetermined value. and stored in a recording medium, and when the determination unit determines that the evaluation value is not higher than the predetermined value, the second resolution image is not output to the recording medium.

A high-resolution image (second resolution image) has a large image size, and images of specimens used for inspection are stored in units of 10 years in pathological examinations. In addition, as the technology for obtaining specimens from the human body continues to improve, the number of specimens to be photographed is expected to continue to increase in the future. Therefore, preservation of images captured by a digital microscope, especially high-resolution images, has become an important issue.

Therefore, in one aspect of the present invention, the second resolution image is stored only when the evaluation value is high. That is, it is possible to save images of only specimens (subjects) that are sufficiently necessary for diagnosis at high resolution. Therefore, it is possible to prevent the free capacity of the recording medium from being limited in order to store high-resolution images of low-importance specimens that can be diagnosed in a short time, for example. In addition, since high-resolution images of specimens with low evaluation values are not stored, it is possible to reduce the processing load of high-resolution images.

Also, for example, the determination unit derives, as the evaluation value, a maximum amplification factor among the one or more amplification factors accepted by the amplification receiving unit. Alternatively, the higher the magnification ratio that is higher than the threshold value is accepted by the magnification accepting unit, or the image of the subject is displayed on the display device at the high magnification ratio. The longer the time, the higher the evaluation value derived by the determination unit. Alternatively, the determination unit may calculate a higher evaluation value as the area of the subject displayed on the display device at a magnification higher than a threshold is larger.

Accordingly, it is possible to derive an appropriate evaluation value according to the application of the image of the subject, and it is possible to transmit only the most suitable high-resolution image (second resolution image).

In addition, an image output device according to another aspect of the present invention is an image output device including an image transmitting device and an image receiving device connected to the image transmitting device via a communication line, and the image transmitting device includes: an image acquisition unit that acquires a first A resolution image, the first resolution image is an image of the object to be photographed based on digital microscope shooting; a high resolution image acquisition part acquires a second resolution image, and the second resolution image is a resolution image an image of the subject captured by the digital microscope that has a higher resolution than the first image; whether it is higher than a predetermined value; and the transmitting unit, when it is determined by the determination unit that the evaluation value is higher than the predetermined value, transmits the second resolution image to the image receiving device, and the When the determination unit determines that the evaluation value is not higher than the predetermined value, the first resolution image is transmitted to the image receiving device, and the image receiving device includes: a magnification ratio of the first-resolution image; and an output unit for display, which acquires the first-resolution image or the second-resolution image from the image transmission device, and causes the display device to display The first resolution image or the second resolution image enlarged by the magnification factor received by the magnification receiving unit, the display output unit further transmits to the image transmission device The evaluation value associated information of the magnification.

Alternatively, an image output device according to another aspect of the present invention is an image output device including an image sending device and an image receiving device connected to the image sending device via a communication line, and the image sending device includes: an image acquisition unit that acquires a first A resolution image, the first resolution image is an image of the object to be photographed based on digital microscope shooting; a high resolution image acquisition part acquires a second resolution image, and the second resolution image is a resolution image an image of the subject captured by the digital microscope that is higher than the first resolution image; and a sending unit that sends the first resolution image or the For a second resolution image, the image receiving device includes: an enlargement accepting unit that accepts a magnification factor for the first resolution image displayed on a display device; and a display output unit that acquires the second resolution image from the image transmitting device. a first resolution image or the second resolution image, causing the display device to display the first resolution image or the second resolution image enlarged based on the magnification accepted by the enlargement accepting unit and a judging section that judges whether an evaluation value based on the magnification factor accepted by the magnification accepting section is higher than a predetermined value, and notifies the image transmission device of a judgment result, wherein the transmission section expresses the evaluation value In the case of the determination result higher than the predetermined value, the second resolution image is transmitted to the image receiving device, and in the case of the determination result indicating that the evaluation value is not higher than the predetermined value Next, the image with the first resolution is sent to the image receiving device. Furthermore, the evaluation value-related information is information including the magnification accepted by the magnification accepting unit, the number of times the magnification was accepted, the time when the image of the subject was displayed, or the area of the displayed subject.

Thus, even if there is no pathologist at the place where the specimen to be photographed is placed, the pathologist at a remote place can use the image receiving device to acquire the image of the specimen transmitted from the image transmitting device installed at the place, and Diagnosis of the specimen based on the image is performed. At this time, unnecessary high-resolution images are not transmitted to the image receiving device, so the burden of processing high-resolution images can be reduced.

In addition, the image transmission device may further include a second output unit configured to output the second resolution image to Output and save to a recording medium.

Thus, even when the high-resolution image (second resolution image) is not transmitted to the remote pathologist due to the low evaluation value, the high-resolution image is stored. Therefore, when the pathologist considers that a high-resolution image is necessary, the high-resolution image can be quickly read out from the recording medium and sent to the pathologist.

In addition, with respect to the image output device, the first resolution image is a first first resolution image, and the command causes the at least one processor to execute: acquiring a second first resolution image, the second The first resolution image is an image of the object to be photographed based on the shooting of the digital microscope, and the second resolution image is based on the first first resolution image and the second first resolution image The digital microscope emits a first light at a first angle to the object to be photographed and a second light at a second angle to the object to be photographed, and emits the first light after emitting the first light In the second light, the digital microscope generates the first first resolution image based on the first plurality of pixel values, generates the second first resolution image based on the second plurality of pixel values, and the second resolution The number of pixel values of the image is greater than the number of the first plurality of pixel values, and the number of pixel values of the second resolution image is greater than the number of the second plurality of pixel values, the captured The object includes a first part and a second part adjacent to the first part, and the plurality of photoelectric conversion units output a first converted light including a first pixel value based on the first converted light obtained by passing the first light through the subject. The first plurality of pixel values, based on the second converted light obtained by the second light passing through the subject, output the second plurality of pixel values including the second pixel value, the plurality of The photoelectric conversion part includes a first photoelectric conversion part and a second photoelectric conversion part adjacent to the first photoelectric conversion part, and the first photoelectric conversion part passes through the first part when a part of the first light passes through the first photoelectric conversion part. A part of the first converted light is obtained to output the first pixel value, and the second photoelectric conversion unit is based on the second part of the second light obtained by passing through the second part. Convert a part of the light and output the 2nd pixel value.

In addition, each of the image output device, the image transmission device, and the image reception device according to one aspect of the present invention may be configured as a non-transitory recording medium including at least one processor and holding commands. In this case, each component included in each of the above-mentioned devices is realized by using the at least one processor to execute the command stored in the recording medium.

Hereinafter, embodiments will be specifically described with reference to the drawings.

In addition, the embodiment described below is all general or the embodiment which showed a specific example. Numerical values, shapes, materials, components, arrangement positions and connections of components, steps, order of steps, etc. shown in the following embodiments are examples, and are not intended to limit the present invention. In addition, among the constituent elements in the following embodiments, the constituent elements not described in the independent claims representing the highest concept are described as arbitrary constituent elements.

(Embodiment 1)

FIG. 1 is a configuration diagram showing an example of an image processing system including an image output device according to Embodiment 1. As shown in FIG.

The image processing system 100S includes: a digital microscope 1500 , an image output device 1001 and a display device 1501 .

The digital microscope 1500 is an image acquisition device described below, which takes a pathological specimen as an object to be photographed. Here, in this specification, a "digital microscope" is a device for digitally imaging at least a part of a pathological tissue examination slide specimen. For example, a digital microscope is an image acquisition device described below that captures the specimen by a contact image sensing (CIS: Contact Image Sensing) method, or a virtual slide scanner (virtual slide scanner). Here, an image acquisition device that performs shooting by the CIS method is a device that directly arranges a subject on an image sensor to capture a first-resolution image and a second-resolution image as images of the subject. . Specifically, the image acquisition device irradiates the subject with light from a plurality of different irradiation directions to photograph the subject, thereby generating a plurality of first-resolution images. In addition, the image acquisition device rearranges the pixels of the plurality of first-resolution images to generate a second-resolution image having a higher resolution than the first-resolution images. In addition, as the digital microscope 1500 , a digital microscope such as a virtual slide scanner that generates enlarged images of multiple resolutions may be used instead of an image acquisition device described below. The virtual slide scanner is a device for photographing specimens through a microscope. The operation of shifting the relative position of the slide to the microscope and the operation of imaging are repeated, and the obtained images are stitched together to generate an image of the specimen in a wide range. In the virtual slide scanner, the resolution of the image to be acquired is selected by changing the objective lens mounted on the microscope. When a low-magnification objective lens is used, the resolution becomes low, but the range of the specimen captured in one image becomes larger, so there are fewer imaging parts and the imaging time is shortened. The image acquisition device will be described in detail in Embodiment 4. FIG.

The display device 1501 is a device for displaying images captured by a digital microscope to an operator (for example, a pathologist), and is, for example, a liquid crystal display or a projector.

The image output device 1001 acquires an image captured by the digital microscope 1500 and causes the display device 1501 to display the image. In addition, the image output device 1001 transmits the image to a device operated by a remote pathologist, for example, via a communication line.

The image output device 1001 includes an image acquisition unit 1101 , a high-resolution image acquisition unit 1102 , an enlargement reception unit 1201 , a determination unit 1202 , a display output unit 1203 , and a transmission unit 1204 .

The image acquisition unit 1101 acquires a first-resolution image, which is an image of a subject obtained based on imaging by the digital microscope 1500 . In addition, the image of the object to be photographed by the digital microscope 1500 may be an image obtained directly by the photographing, or may be an image obtained by processing the directly obtained image. Here, the image acquisition unit 1101 acquires, as a first resolution image, a sub-image obtained by imaging by, for example, a contact image sensor in the digital microscope 1500 . In addition, the resolution of the first resolution image is referred to as the first resolution.

The high-resolution image acquiring unit 1102 acquires a second-resolution image. The second-resolution image is an image (high-resolution image) with a higher resolution than the first-resolution image, and is obtained by shooting with the digital microscope 1500. Image of the subject being photographed. The high-resolution image acquisition unit 1102 may acquire the second-resolution image by generating the second-resolution image from a plurality of sub-images, or may acquire the second-resolution image from the digital microscope 1500 . In addition, the resolution of the second resolution image is referred to as a second resolution. Here, the pixel pitch of the first resolution is 0.9 μm, and the pixel pitch of the second resolution is 0.3 μm. In addition, when the ratio of the pixel pitch of the first resolution to the pixel pitch of the second resolution is taken as the resolution ratio, the resolution ratio is 3.

The magnification accepting unit 1201 accepts the magnification of the first-resolution image displayed on the display device 1501 . Specifically, the magnification accepting unit 1201 acquires the magnification K as a numerical value through keyboard operations, for example. In addition, the magnification accepting unit 1201 may update the magnification K using, for example, the number of rotations of a mouse having a scroll wheel as a change value of the magnification.

The display output unit 1203 outputs the image with the first resolution or the image with the second resolution enlarged based on the magnification accepted by the magnification receiving unit 1201 to the display device 1501 . Thus, the enlarged image with the first resolution or the image with the second resolution is displayed on the display device 1501 . Furthermore, although the image output device 1001 in this embodiment includes the display output unit 1203 , it does not need to include the display output unit 1203 .

The determination unit 1202 determines whether or not the evaluation value based on the magnification received by the magnification accepting unit 1201 is higher than a predetermined value. Here, the predetermined value may be arbitrarily set by, for example, a pathologist or an operator, or may be a predetermined value. Specifically, when the evaluation value is derived as the largest magnification factor among one or more magnification factors accepted by the magnification accepting unit 1201, the predetermined value may be, for example, a ratio to the resolution (that is, , in the case of the above example, 3) equal values.

Also, the predetermined value is a value larger than 1. However, if the predetermined value is set to a low value, the following output will be frequently performed. In order to avoid such a situation, the predetermined value may also be as high as possible. The output described below is the transmission performed by the transmission unit 1204 or the output performed by the first output unit 1205 . Therefore, the predetermined value may be switched according to the thickness of the communication line and/or the congestion state of the line. For example, when using a mobile phone line with a thin line, the predetermined value is set to a high value of 10, and when a thick optical line is used, the predetermined value is set to a low value of 3. In addition, when the line is congested, the predetermined value is set to a high value of 10, and when the line is free, the predetermined value is set to a low value of 3. In addition, the predetermined value may be switched according to the size of the memory used for saving (for example, any one of the recording media 1502 in FIG. 8 and FIGS. 12 to 14 described below). For example, when the memory is a large-capacity HDD (Hard Disk Drive: Hard Disk Drive), the predetermined value can be set to 3, and when the memory is a small-capacity SSD (Solid State Drive: Solid State Drive), the predetermined value can be set to 3. The value is 10.

The transmission unit 1204 transmits the second-resolution image when it is determined by the determination unit 1202 that the evaluation value is higher than a predetermined value. On the other hand, when the determination unit 1202 determines that the evaluation value is not higher than the predetermined value, the transmission unit 1204 does not transmit the second resolution image. At this time, that is, when the determination unit 1202 determines that the evaluation value is not high, the transmission unit 1204 transmits the first-resolution image. The first-resolution image or the second-resolution image is sent to, for example, a device operated by a remote pathologist via a communication line.

FIG. 2 is a diagram showing an example of a display screen displayed on the display device 1501 .

The display device 1501 displays a display screen 601 including an end key 602 , an overhead image display unit 603 , an observation result input field 604 , a read key 605 , and an enlarged image display unit 606 .

When the end key 602 is selected, the display device 1501 transmits a signal notifying the image output device 1001 that the display of the image should be terminated. The whole of the first resolution image or the second resolution image is displayed on the bird's-eye view image display unit 603 . In the observation result input field 604, for example, the observation result of the pathologist is written. When the read key 605 is selected, the display device 1501 sends to the image output device 1001 a signal to end the display of the first-resolution image and instruct to read the second-resolution image, for example. In the enlarged image display unit 606 , the first resolution image or the second resolution image is displayed enlarged.

FIG. 3 is a flowchart showing an example of processing operations of the image output device 1001 .

First, the image acquisition unit 1101 of the image output device 1001 acquires a first-resolution image from the digital microscope 1500 (step S11 ). Then, the magnification accepting unit 1201 accepts the magnification ratio of the image in accordance with, for example, the operation of the keyboard by the operator (step S12 ). Then, the display output unit 1203 enlarges the first-resolution image at the accepted enlargement factor and outputs it to the display device 1501 . Based on this output, the display output unit 1203 causes the display device 1501 to display the first-resolution image enlarged to the accepted magnification factor (step S13 ).

Next, the display output unit 1203 determines whether the display of the first-resolution image should be terminated based on a signal output from the display device 1501 when the end key 602 or the read key 605 is selected (step S14 ). Here, when it is determined that the display should not be terminated (step S14: No), the enlargement accepting unit 1201 further accepts a new enlargement factor. That is, the operator who observes the image displayed on the display device 1501 observes by changing the magnification according to the specimen.

On the other hand, when it is determined that the display should be terminated (step S14: Yes), the determination unit 1202 derives an evaluation value based on the magnification received by the magnification receiving unit 1201, and determines whether the evaluation value is higher than a predetermined value (step S15).

The evaluation value is, for example, the last accepted magnification (final magnification) or the largest magnification (maximum magnification) among the one or more magnifications accepted in step S12. That is, the determination unit 1202 derives the final magnification factor or the maximum magnification factor as an evaluation value. Alternatively, the determination unit 1202 may derive the evaluation value based on other requirements. For example, other requirements are at least one of the number of times the magnification is accepted in step S12 (accepted number), the time when the display device 1501 displays the image (display time), and the area of the image displayed by the display device 1501 (display area).

In addition, the determination unit 1202 may multiply each of the maximum magnification, the number of receptions, the display time, and the display area by a coefficient and integrate them to derive an evaluation value.

Here, when it is determined that the evaluation value is not higher than the predetermined value (step S15: No), the transmission unit 1204 transmits the first resolution image via the communication line, but does not transmit the second resolution image (step S16). On the other hand, when it is determined that the evaluation value is higher than the predetermined value (step S15: Yes), the high-resolution image acquisition unit 1102 acquires a second-resolution image (step S17). Furthermore, the transmitting unit 1204 transmits the second-resolution image acquired by the high-resolution image acquiring unit 1102 via the communication line (step S18 ).

In the flow chart of FIG. 3 , the image enlarged and displayed at the acceptable magnification ratio is only the first resolution image, but as shown in the flow chart of FIG. 4 , the second resolution image can also be enlarged at the magnification ratio. to display.

FIG. 4 is a flowchart showing another example of the processing operation of the image output device 1001 .

First, the image acquisition unit 1101 of the image output device 1001 acquires a first-resolution image from the digital microscope 1500 (step S21 ). Then, the display output unit 1203 outputs the first resolution image to the display device 1501, thereby causing the display device 1501 to display the first resolution image (step S22).

Next, the magnification accepting unit 1201 accepts the magnification of the image in accordance with, for example, the operation of the keyboard by the operator (step S23 ). The determination unit 1202 derives an evaluation value based on the magnification received by the magnification receiving unit 1201, and determines whether the evaluation value is higher than a predetermined value (step S24).

Here, when it is determined that the evaluation value is not higher than the predetermined value (step S24 : No), the display output unit 1203 enlarges the first resolution image to the magnification ratio accepted in step S23 and outputs it to the display device 1501 . Based on this output, the display output unit 1203 causes the display device 1501 to display the enlarged first resolution image (step S25 ). At this time, the second resolution image is not sent. In addition, the display output unit 1203 determines whether or not the display of the first-resolution image should be terminated based on a signal output from the display device 1501 when the end key 602 or the read key 605 is selected (step S26 ). Here, when it is determined that the display should be terminated (step S26: Yes), the image output device 1001 terminates the process. On the other hand, when it is determined that the display should not be terminated (step S26: No), the image output device 1001 repeatedly executes the processing from step S23.

In addition, in step S24, when it is determined that the evaluation value is higher than the predetermined value (step S24: Yes), the determination unit 1202 determines whether the determination is the first determination of the sample (step S27). Here, when it is determined that it is the first time (step S27: Yes), the high-resolution image acquisition unit 1102 acquires a second-resolution image (step S28). Furthermore, the transmitting unit 1204 transmits the second-resolution image acquired by the high-resolution image acquiring unit 1102 via the communication line (step S29 ).

After the second-resolution image is transmitted in step S29, or when it is determined in step S27 that it is not the first time (step S27: No), the display output unit 1203 enlarges the second-resolution image to the magnification factor accepted in step S23. And output to the display device 1501 . Based on this output, the display device 1501 causes the display device 1501 to display the enlarged second resolution image (step S30). At this time, instead of enlarging the second resolution image at the enlarging factor accepted in step S23, it may be enlarging at a value obtained by dividing the enlarging factor by the resolution ratio. For example, when the accepted magnification factor is 3 and the resolution ratio is 3, the second resolution image is magnified by 1.

Next, the display output unit 1203 determines whether or not the display of the second resolution image should be terminated based on the signal output from the display device 1501 when the end key 602 is selected (step S31 ). Here, when it is determined that the display should be terminated (step S31: Yes), the image output device 1001 terminates the process. On the other hand, when it is determined that the display should not be terminated (step S31: No), the image output device 1001 repeatedly executes the processing from step S23.

(Effect)

Thus, in this embodiment, the second resolution image is transmitted only when the evaluation value is high. That is, it is possible to transmit, for example, a high-resolution image of a sample (subject) that is sufficiently necessary for diagnosis to a remote location. Therefore, it is possible to suppress, for example, delay in transmission of high-resolution images of specimens necessary for diagnosis in order to transmit high-resolution images of low-importance specimens that can be diagnosed in a short time. In addition, since high-resolution images of specimens with low evaluation values are not transmitted, it is possible to reduce the processing load of high-resolution images.

Specifically, it is considered that the image output device 1001 in this embodiment is used in a local hospital with only inexperienced pathologists or only surgeons. In this case, the transmission unit 1204 can remotely transmit the second resolution image of the specimen that is difficult to diagnose to an experienced pathologist. First, a low-resolution first resolution image is sent to the pathologist. Then, when a high evaluation value is obtained, that is, only when a more detailed diagnosis of the specimen is required, the high-resolution second resolution image can be transmitted. Assuming that the first resolution image is 100 Mbytes (= 800 Mbits), the second resolution image is 1.6 Gbytes, and the communication speed is 50 Mbps, the first resolution image can be transmitted in 16 seconds, but the high The resolution image takes 256 seconds to send. Sending all images at high resolution is quite time consuming. However, in biological examinations such as medical examinations, it is often possible to determine that there is no abnormality with a low-resolution image. Therefore, it is possible to improve diagnosis efficiency by transmitting high-resolution images only for necessary specimens. In addition, the image output device 1001 can be used in a small hospital with only one pathologist, when an important image is selected for review by a pathologist located at a remote location.

(details of evaluation value)

Here, the above-mentioned evaluation values will be described in detail.

As described above, the determination unit 1202 may derive an evaluation value based on at least one of the reception count, display time, and display area.

Evaluation values based on the number of receptions, display time, or display area are effective in judging whether specimen observation at high magnification is not necessary for diagnosis, or whether specimen observation at high magnification is necessary for diagnosis of. That is, in ordinary diagnosis using a microscope, since it takes time and effort to replace lenses, high-magnification observation is not performed if it is known that the specimen is normal during low-magnification observation. In addition, in determining whether a sample is normal, if there is no tumor in the sample or no inspection items necessary for diagnosis remain, the sample is judged to be normal. On the other hand, high-magnification magnification of an image acquired by an electron microscope such as the digital microscope 1500 of this embodiment can be performed by simple operations of a mouse or a keyboard. Therefore, in the digital microscope 1500 , observation at a high magnification is often performed on a specimen that has only been observed at a low magnification by a normal microscope.

This is considered to be that a diagnostician such as a pathologist, physician, or laboratory technician observes an image enlarged at a high magnification to confirm whether imaging of the specimen is properly performed. In addition, it is considered to be due to the request of the diagnostician to observe a small region included in the sample as an image enlarged at a high magnification even for a normal sample.

Parameters such as the number of acceptances, display time, and display area for observation at a high magnification that is not necessary for such a diagnosis vary according to the preference or experience of the diagnoser. However, the numerical values of these parameters are extremely small compared with those observed at high magnifications necessary for diagnosis. Therefore, in this embodiment, by using evaluation values based on the number of receptions, display time, or display area, it is possible to determine whether observation at a high magnification is necessary for diagnosis. In order to effectively utilize the limited communication bandwidth, it is not necessary to transmit high-resolution images when observing at a high magnification that is not necessary for such a diagnosis.

Also, when transmitting a high-resolution image by deriving an evaluation value based on at least one of the reception count, display time, and display area, the transmission unit 1204 may attach a flag to the high-resolution image. That is, the transmitting unit 1204 attaches a mark indicating the type of the parameter (at least one of the number of receptions, display time, and display area) used for the evaluation value and the evaluation value to the high-resolution image, and transmits the marked image. high resolution images. Accordingly, the image receiving device that receives the high-resolution image can extract a high-resolution image to which a flag indicating that, for example, the number of acceptance times is used as a type of parameter is added from among a plurality of high-resolution images. In addition, if there are multiple extracted high-resolution images, the image receiving device may arrange these high-resolution images in ascending order or descending order of evaluation values based on the evaluation values indicated by these marks. Accordingly, the diagnostician can easily select a high-resolution image to be observed.

(the details of the evaluation value: the reception number of times)

Specifically, the number of acceptances is the number of times that the amplification receiving unit 1201 accepts a high amplification factor that is a higher amplification factor than the threshold value. The threshold is, for example, the aforementioned resolution ratio (specifically, 3). The higher the number of acceptances, the higher the evaluation value derived by the determination unit 1202 . For example, the evaluation value may be the number of receptions itself. In this case, the determination unit 1202 determines whether or not the number of acceptances as the evaluation value is more than one (predetermined value=1). That is, if the number of receptions is one, there is a high possibility that the display of the first resolution image enlarged to a high magnification ratio is not required for diagnosis. In other words, it is highly likely that the specimen is determined to be normal from the first resolution image with a lower magnification than the high magnification. Conversely, if the number of receptions is two or more, the display of the first resolution image enlarged to a high magnification ratio is highly likely to be necessary for diagnosis. That is, there is a high possibility that the determination of whether the specimen is normal is doubtful. Therefore, in this way, it is determined whether the number of receptions is more than one, and when it is determined that it is more, transmission of a high-resolution image is performed, thereby effectively utilizing a limited communication bandwidth.

Alternatively, the determination unit 1202 may determine whether or not the number of acceptances as the evaluation value is more than two (predetermined value=2). Specifically, in the case of diagnosing a malignant tumor, diagnosis is often performed three or more times in order to confirm the shape and size of the tumor. For example, the center, the left end, and the right end of the tumor are often diagnosed separately. Therefore, by setting the predetermined value to "2" as described above, it is possible to transmit only high-resolution images of specimens suspected of being malignant tumors.

In this way, when the evaluation value is derived based on the number of receptions, it is possible to suppress transmission of high-resolution images regardless of whether the specimen observation at high magnification is performed only once or twice or less. That is, it is possible to suppress transmission of a high-resolution image due to specimen observation at a high magnification unnecessary for diagnosis.

(Details of evaluation value: display time)

Specifically, the display time is the time when the image of the subject is displayed by the display device 1501 at a high magnification. The longer the display time is, the higher the evaluation value derived by the determination unit 1202 is. For example, the evaluation value may be the display time during which the image of the subject is displayed on the display device 1501 at a high magnification ratio equal to or higher than the above-mentioned resolution ratio (specifically, 3). In this case, the determination unit 1202 determines whether or not the display time as the evaluation value is longer than, for example, 10 seconds (predetermined value=10 seconds). That is, if the display time is 10 seconds or less, the display of an image enlarged to a high magnification ratio is highly likely to be unnecessary for diagnosis. Thereby, it is possible to suppress transmission of a high-resolution image due to observation of the specimen at a high magnification unnecessary for diagnosis.

In addition, the time required for the diagnosis of a malignant tumor varies depending on the diagnoser or the site to be observed. Therefore, the predetermined value (10 seconds in the above-mentioned example) may be changed according to the diagnoser, the site to be observed, or the method of obtaining and preparing the sample. For example, the predetermined value may be set to 10 seconds for a sample prepared by a normal method, and may be set to 30 seconds for a sample prepared in a short time for rapid diagnosis during surgery.

In addition, in the above-mentioned example, the predetermined value is set to 10 seconds or 30 seconds, but the time m times (m is a real number greater than 1) of the average display time required for normal specimen observation may be set to etc. as predetermined values.

In addition, the determination unit 1202 may multiply the display time at the magnification factor by a weight for each accepted magnification factor (that is, a high magnification factor of 3 or higher), and use the sum of the weighted display times as the evaluation result. value to export. The higher the magnification, the larger the weight. Specifically, the determination unit 1202 treats the magnification itself as a weight, and derives an evaluation value using evaluation value=Σ[magnification (k)×display time (k)]. In addition, the magnification (k) is the magnification of 3 or more accepted at the kth magnification, and the display time (k) is the display time for displaying the image with the first resolution enlarged at the kth magnification. That is, when the magnification ratio equal to or higher than the resolution ratio is accepted n times, the evaluation value is derived as the sum of magnification ratio (k)×display time (k) where k is 1 to n. For example, the accepted magnification factor Q and the display time T (seconds) at this time are (Q, T)=(3, 7), (4, 6), (5, 5), and (6, 4). In such a case, the determination unit 1202 derives the evaluation value by evaluation value=(3×7)+(4×6)+(5×5)+(6×4). In addition, in calculating the evaluation value, instead of the magnification (k), "magnification (k)-resolution ratio (specifically, 3)+1" may be used. In this case, the determination unit 1202 derives the evaluation value by evaluation value=(1×7)+(2×6)+(3×5)+(4×4). If such an evaluation value is used, the longer the observation time or the higher the magnification (magnification) during observation, the higher the evaluation value will be derived. In addition, in this case, the determination unit 1202 may determine whether or not the evaluation value exceeds, for example, 30 (predetermined value=30).

In addition, the determination unit 1202 derives the evaluation value after calculating the display time at each magnification, but the evaluation value at the time of observation may be updated at any time. That is, the determination unit 1202 may acquire the magnification at the time of observation every time a certain time elapses, and add the result of the product of the acquired magnification and the certain time to the evaluation value derived just before, whereby the evaluation value to update. As a result, a new evaluation value is derived every time a predetermined period of time elapses.

In addition, instead of multiplying the display time by the magnification, the determination unit 1202 may multiply the display time by a value that monotonically increases with respect to the magnification. A value that increases monotonically with respect to the magnification is a value obtained by a function, for example, is the logarithm of the magnification. That is, the determination unit 1202 derives an evaluation value by evaluation value=Σ[log magnification (k)×display time (k)].

In addition, instead of deriving the evaluation value from the sum as described above, the determination unit 1202 may derive a value obtained by integrating the magnification or the logarithm of the magnification with the display time as the evaluation value.

(Details of evaluation value: display area)

Specifically, the display area is the area of the subject displayed on the display device 1501 at the high magnification. For example, the determination unit 1202 uses edge detection to specify a region in which the subject is reflected in one frame, and calculates the area of the specified region as the area of the image of the subject. In addition, when a plurality of areas are detected in one frame by edge detection, the determination unit 1202 may set the area of the area including the position (coordinates) indicated by the user with the mouse, among the plurality of areas, as the area of the area to be selected. The area of the subject's image. Here, if the displayed image is moved away, the display area will expand the area of the newly displayed image. That is, when the area including the position (ie, coordinates) pointed by the user with the mouse is continuous (ie, there is no edge), when the user moves away from the image, a new area continuous with the area Appear. Then, the determination unit 1202 may add the area of the newly appeared area and the area continuous to the area including the position indicated by the user with the mouse to the area of the image of the subject. In addition, the area of the subject as the display area is a value (quotient) obtained by dividing the area of the image of the subject calculated as described above by the actual magnification. The larger the display area, the higher the evaluation value derived by the determination unit 1202 .

For example, the evaluation value is a display area of a subject displayed at a high magnification ratio (specifically, 3) or higher. In this case, the determination unit 1202 determines whether or not the display area as the evaluation value is larger than 10,000 μm ² . When the resolution ratio is 3, if the interval between the positions of the specimen corresponding to two adjacent pixels is 1/3 μm (= 0.333 μm), the area of the specimen imaged by one pixel becomes 1/333 μm. 9 μm ² , the sample size corresponding to 10,000 μm ² is 10,000 μm ² /(1/9 μm ² )=9×10 ⁴ pixels. That is, the determination unit 1202 determines whether or not it is larger than 9×10 ⁴ pixels (predetermined value=9×10 ⁴ pixels). That is, if the display area is 9×10 ⁴ pixels or less, there is a high possibility that the display of the first resolution image enlarged to a high magnification is not required for diagnosis. Accordingly, it is possible to suppress transmission of a high-resolution image due to specimen observation at a high magnification unnecessary for diagnosis.

In addition, in the above-mentioned example, the predetermined value is set to 10,000 μm ² (that is, 9×10 ⁴ pixels), but it is also possible to set m times the average display area required for observation of a normal specimen (m is the ratio 1 large real number) or the like as predetermined values.

In addition, the determination unit 1202 may multiply the display area at the magnification factor by a weight for each accepted magnification factor (that is, a high magnification factor greater than or equal to 3), and use the total of the weighted display areas as derived from evaluation values. The higher the magnification ratio, the larger the value of this weight. Specifically, the determination unit 1202 treats the square of the magnification factor as a weight, and the determination unit 1202 derives the evaluation value by evaluation value=Σ[magnification factor (k)^2×display area (k)]. In addition, the magnification (k) is a magnification of 3 or more accepted at the k th magnification similarly to the above, and the display area (k) is the display area of the subject magnified at the k th magnification. In addition, A^B means the calculation of A raised to the power of B. That is, when the magnification of the resolution ratio or higher is accepted n times, the evaluation value is derived as the sum of magnification (k)^2×display area (k) where k is 1 to n. For example, let the magnification Q and the display area S (1×10 ⁴ pixels) at this time be (Q, S) = (3, 2), (4, 0.5), (5, 1.5), (6, 1) . In such a case, the judgment unit 1202 judges the evaluation value=[(3^2)×2×10 ⁴ ]+[(4^2)×0.5×10 ⁴ ]+[(5^2)×1.5×10 ⁴ ]+[(6^2)×1×10 ⁴ ] to derive the evaluation value. In addition, in calculating the evaluation value, instead of the square of the magnification factor (k), the square of “magnification factor (k)−resolution ratio (specifically, 3)+1” may be used as a weight. In this case, the determination unit 1202 judges that evaluation value=[(1^2)×2×10 ⁴ ]+[(2^2)×0.5×10 ⁴ ]+[(3^2)×1.5×10 ⁴ ] +[(4^2)×1×10 ⁴ ] to derive the evaluation value. If such evaluation values are used, the larger the area of the subject to be observed or the higher the magnification (magnification) at the time of observation, the higher the evaluation value is derived. In addition, in this case, the determination unit 1202 may determine whether or not the evaluation value exceeds, for example, 8.1×10 ⁵ (predetermined value=8.1×10 ⁵ ).

In addition, the weight may be a larger value as the magnification is higher, and any function may be used without being limited to the square of the magnification as long as it is a function that can calculate such a value. For example, the determination unit 1202 can derive the evaluation value by evaluation value=Σ[magnification (k)^3×display area (k)].

In addition, the determination unit 1202 may derive, as an evaluation value, a value obtained by integrating the power of the magnification by the display area, instead of deriving the evaluation value from the sum as described above.

(Modification)

Here, a modified example of the image output device in Embodiment 1 will be described. The image output device of this modification transmits and stores the second resolution image when the evaluation value is high.

FIG. 5 is a configuration diagram showing an example of an image processing system including an image output device according to this modification.

An image processing system 100Sa of this modified example includes a digital microscope 1500 , an image output device 1001 a , a display device 1501 , and a recording medium 1502 .

The recording medium 1502 holds images output from the image output device 1001a. The recording medium 1502 is realized by, for example, a storage device such as a hard disk, BD (Blu-ray (registered trademark) Disc), DVD, SD card (registered trademark), RAM (Random Access Memory: random access memory), or cache.

The image output device 1001 a includes a first output unit 1205 in addition to the components included in the image output device 1001 according to Embodiment 1 described above.

When the determination unit 1202 determines that the evaluation value is higher than the predetermined value, the first output unit 1205 outputs and stores the second resolution image in the recording medium 1502 . On the other hand, when the determination unit 1202 determines that the evaluation value is not higher than the predetermined value, the first output unit 1205 does not output the second resolution image to the recording medium 1502 .

In other words, the first output unit 1205 outputs the image transmitted from the transmission unit 1204 to the recording medium 1502 , thereby storing the image in the recording medium 1502 . That is, the first output unit 1205 saves the first resolution image or the second resolution image sent in steps S16 and S18 in FIG. 3 or step S29 in FIG. 4 in the recording medium 1502 .

(Effect)

In such an image output device 1001a of this modified example, it is possible to automatically store a high-resolution image (second resolution image) for a specimen requiring high-resolution diagnosis. It is also conceivable to set a button for selecting whether to save high-resolution images. However, considering that pathologists sometimes diagnose more than 60 specimens in one hour, they may forget to press the button. However, if the image output device 1001a of this modified example is used, forgetting to operate can be prevented, and important high-resolution images can be reliably saved.

In addition, if the image output device 1001a of this modified example is used, it is possible to save and transmit a high-resolution image for a specimen for diagnosis with a high-resolution image, and to perform a diagnosis on a low-resolution image (first resolution image). The specimen for diagnosis is stored and transmitted with this low-resolution image. Thus, by setting the size of the stored and transmitted image to a size corresponding to the specimen, it is possible to reduce the size of the stored image, the size of the transmitted (transferred) image, and the transfer time.

(Embodiment 2)

The image output device in this embodiment is composed of an image transmitting device and an image receiving device connected to each other via a communication line. In addition, among the devices and components in this embodiment, the same devices and components as in Embodiment 1 are assigned the same reference numerals as in Embodiment 1, and their detailed descriptions are omitted.

FIG. 6 is a configuration diagram showing an example of an image processing system including an image output device in Embodiment 2. FIG.

The image processing system 200S includes: a digital microscope 1500 , an image output device 2001 , a display device 1501 , a recording medium 1502 , and a recording medium 1503 .

The recording medium 1502 holds images output from the image receiving device 1200 of the image output device 2001 .

The recording medium 1503 holds images output from the image transmission device 1100 of the image output device 2001 . Like the recording medium 1502, the recording medium 1503 is realized by a storage device such as a hard disk, BD (Blu-ray (registered trademark) Disc), DVD, SD card (registered trademark), RAM, or cache, for example.

The image output device 2001 includes an image transmitting device 1100 and an image receiving device 1200 connected to each other via a communication line. In addition, the image output device 2001 has the same function as the image output device 1001 of Embodiment 1 as a whole. In addition, for example, a group including the digital microscope 1500 , the image transmission device 1100 , and the recording medium 1503 is placed in a facility such as a hospital without a pathologist. A group including the image receiving device 1200 , the display device 1501 , and the recording medium 1502 is placed, for example, in a facility with a pathologist located far from the hospital.

The image transmission device 1100 includes an image acquisition unit 1101 , a high-resolution image acquisition unit 1102 , a transmission unit 1204 , and a second output unit 1103 .

The second output unit 1103 outputs the second resolution image acquired by the high resolution image acquisition unit 1102 to the recording medium 1503 , thereby storing the second resolution image in the recording medium 1503 . That is, the second output unit 1103 outputs and stores the second resolution image in the recording medium 1503 when the determination unit 1202 determines that the evaluation value is not higher than the predetermined value.

The image receiving device 1200 includes an enlargement receiving unit 1201 , a determination unit 1202 , a display output unit 1203 , and a first output unit 1205 .

The display output unit 1203 acquires the first resolution image or the second resolution image from the image transmission device 1100 via the communication line. Then, similarly to Embodiment 1, the display output unit 1203 outputs to the display device 1501 the image with the first resolution or the image with the second resolution enlarged based on the magnification accepted by the magnification receiving unit 1201 . As a result, the display output unit 1203 causes the display device 1501 to display the enlarged first-resolution image or the second-resolution image.

As in the first embodiment, the determination unit 1202 determines whether or not the evaluation value is higher than a predetermined value. Then, the determination unit 1202 notifies the image transmission device 1100 of the determination result via the communication line.

The transmitting unit 1204 of the image transmitting device 1100 transmits the second resolution image to the image receiving device 1200 via the communication line when the determination result indicates that the evaluation value is high. On the other hand, when the determination result indicates that the evaluation value is not high, the transmission unit 1204 transmits the first resolution image instead of the second resolution image via the communication line.

FIG. 7 is a flowchart showing the processing operations of the video transmission device 1100 and the video reception device 1200 .

The image acquisition unit 1101 of the image transmission device 1100 acquires a first-resolution image from the digital microscope 1500 (step S41 ). The transmitting unit 1204 transmits the first-resolution image to the image receiving device 1200 via the communication line (step S42).

The display output unit 1203 of the image receiving device 1200 receives the first-resolution image transmitted from the image transmitting device 1100 via the communication line (step S51 ). Next, the magnification accepting unit 1201 of the image receiving device 1200 accepts the magnification ratio of the image in accordance with, for example, the operation of the keyboard by the operator (step S52 ). Then, the display output unit 1203 enlarges the first resolution image at the accepted enlargement factor and outputs it to the display device 1501 . Based on this output, the display output unit 1203 causes the display device 1501 to display the first resolution image enlarged to the accepted magnification factor (step S53 ).

Next, the display output unit 1203 of the image receiving device 1200 determines whether to end the display of the first-resolution image based on the signal output from the display device 1501 when the end key 602 or the read key 605 is selected (step S54). Here, when it is determined that the display should not be terminated (step S54: No), the enlargement accepting unit 1201 further accepts a new enlargement factor. That is, the operator (pathologist) who observes the image displayed on the display device 1501 performs observation by changing the magnification according to the specimen.

On the other hand, when it is determined that the display should be terminated (step S54: Yes), the determination unit 1202 derives an evaluation value based on the magnification received by the magnification receiving unit 1201, and determines whether the evaluation value is higher than a predetermined value (step S55).

Here, when it is determined that the evaluation value is not higher than the predetermined value (step S55: No), the first output unit 1205 of the image receiving device 1200 outputs and stores the first resolution image to the recording medium 1502 (step S59). On the other hand, when it is determined that the evaluation value is higher than the predetermined value (step S55: Yes), the first output unit 1205 requests the image transmission device 1100 for an image of the second resolution via the communication line (step S56).

The transmitting unit 1204 of the image transmitting device 1100 determines whether or not an image of the second resolution is requested from the image receiving device 1200 (step S43). Here, when it is determined that there is no request (step S43: No), the high-resolution image acquisition unit 1102 acquires a second-resolution image (step S44). In addition, the second output unit 1103 outputs and stores the second resolution image acquired by the high resolution image acquisition unit 1102 in the recording medium 1503 (step S45). Therefore, the second output unit 1103 outputs and stores the second resolution image in the recording medium 1503 when the determination unit 1202 determines that the evaluation value is not high.

On the other hand, when it is determined that a second-resolution image is requested (step S43: Yes), the high-resolution image acquisition unit 1102 acquires a second-resolution image (step S46). In addition, the transmitting unit 1204 transmits the second-resolution image acquired by the high-resolution image acquiring unit 1102 to the image receiving device 1200 via the communication line (step S47). Therefore, similarly to Embodiment 1, the transmitting unit 1204 transmits the second resolution image when the evaluation value is determined by the determination unit 1202 to be high, and does not transmit the second resolution image when the determination unit 1202 determines that the evaluation value is not high. two-resolution image.

In addition, in the above example, the processing of step S44 and step S46 is performed after the determination of step S43, but it may be performed before the determination of step S43.

When the second resolution image is transmitted in step S47, the first output unit 1205 of the image receiving device 1200 receives the second resolution image (step S57). Then, the first output unit 1205 outputs and stores the second resolution image to the recording medium 1502 (step S58).

(Effect)

In such an image output device 2001 of the present embodiment, even if the facility that acquires the image of the subject is separated from the facility that observes the image, the same effect as that of the first embodiment can be achieved.

(Modification 1)

Here, a first modified example of the image output device in Embodiment 2 will be described. In this modified example, when it is determined that the evaluation value is high for a plurality of samples, a plurality of second-resolution images are sequentially transmitted from the second-resolution image corresponding to the sample with a high evaluation value.

FIG. 8 is a configuration diagram showing an example of an image processing system including an image output device according to this modification.

An image processing system 200Sa of this modified example includes a digital microscope 1500 , an image output device 2001 a , a display device 1501 , a recording medium 1502 , and a recording medium 1503 .

The image output device 2001a includes an image transmitting device 1100a and an image receiving device 1200a connected to each other via a communication line.

The image receiving device 1200 a includes an evaluation value storage unit 1206 in addition to the components included in the image receiving device 1200 according to the second embodiment. The evaluation value storage unit 1206 stores evaluation values derived for each specimen (subject). Also, the first output unit 1205 requests the image transmission device 1100 a for an image of the second resolution corresponding to each evaluation value stored in the evaluation value storage unit 1206 .

The image transmission device 1100a includes a transmission unit 1204a instead of the transmission unit 1204 in the second embodiment. This sending unit 1204a not only has the same function as the sending unit 1204 in Embodiment 2, but also receives a request from the first output unit 1205, from the second resolution image corresponding to the sample with a high evaluation value to send a plurality of second resolution images in sequence.

FIG. 9 is a flowchart showing processing operations of the image transmitting device 1100a and the image receiving device 1200a.

The image acquisition unit 1101 and the high-resolution image acquisition unit 1102 of the image transmission device 1100a acquire the first resolution image and the second resolution image of the subject (step S71 ). The transmitting unit 1204a transmits the first resolution image to the image receiving device 1200a via the communication line (step S72). Then, the image acquisition unit 1101 and the high-resolution image acquisition unit 1102 determine whether or not there is a next subject (step S73 ). When it is determined that the next subject exists (step S73: Yes), the image transmission device 1100a repeatedly executes the process from step S71.

The display output unit 1203 of the image receiving device 1200a receives the first resolution image transmitted from the image transmitting device 1100a via the communication line (step S81). Next, the magnification accepting unit 1201 of the image receiving device 1200a accepts the magnification ratio of the image in accordance with, for example, the operation of the keyboard by the operator (step S82). Then, the display output unit 1203 enlarges the first resolution image at the accepted enlargement factor and outputs it to the display device 1501 . Based on the output, the display output unit 1203 causes the display device 1501 to display the first-resolution image enlarged to the accepted magnification factor (step S83 ).

Next, the display output unit 1203 of the image receiving device 1200a determines whether the display of the first-resolution image should be terminated based on the signal output from the display device 1501 when the end key 602 or the read key 605 is selected (step S84). Here, when it is determined that the display should not be terminated (step S84: No), the enlargement accepting unit 1201 further accepts a new enlargement factor. That is, the operator (pathologist) who observes the image displayed on the display device 1501 performs observation by changing the magnification according to the specimen.

On the other hand, when it is determined that the display should be terminated (step S84: Yes), the determination unit 1202 derives an evaluation value based on the magnification received by the magnification receiving unit 1201, and determines whether the evaluation value is higher than a predetermined value (step S85).

Here, when it is determined that the evaluation value is not higher than the predetermined value (step S85: No), the first output unit 1205 of the image receiving device 1200a outputs and stores the first resolution image to the recording medium 1502 (step S87). On the other hand, when it is determined that the evaluation value is higher than the predetermined value (step S85: Yes), the determination unit 1202 associates the evaluation value higher than the predetermined value with the subject and stores it in the evaluation value storage unit 1206 (step S86 ).

Next, the display output unit 1203 of the image receiving device 1200a determines whether or not there is a next subject (step S88). Here, when it is determined that there is a next subject (step S88: Yes), the image receiving device 1200a repeatedly executes the process from step S81. On the other hand, when it is determined that there is no next subject (step S88 : No), the first output unit 1205 of the image receiving device 1200 a refers to the evaluation value stored in the evaluation value storage unit 1206 . In addition, the first output unit 1205 requests the image transmission device 1100 to transmit the second-resolution images of the subjects associated with the evaluation values in order from the highest evaluation value via the communication line (step S89 ).

The transmitting unit 1204a of the image transmitting device 1100a determines whether or not an image of the second resolution is requested from the image receiving device 1200a (step S74). Here, when it is determined that the request is not requested (step S74: No), the second output unit 1103 outputs and saves the second-resolution image of each subject acquired in step S71 to the recording medium 1503 (step S75). On the other hand, when the transmitting unit 1204a determines that a second resolution image is requested (step S74: Yes), the sum of the second resolution images of each subject acquired in step S71 is higher than a predetermined value. The second resolution image corresponding to the evaluation value of is sent to the image receiving device 1200a. At this time, the transmitting unit 1204a transmits the second-resolution images in order from the second-resolution images with higher evaluation values to the image receiving device 1200a via the communication line (step S76).

The first output unit 1205 of the image receiving device 1200a receives and outputs to the recording medium 1502 the second resolution images corresponding to the evaluation values higher than the predetermined value in order from the second resolution image with the higher evaluation value (step S90) . As a result, each second-resolution image having an evaluation value higher than a predetermined value is stored in the recording medium 1502 .

(Effect)

In this way, in this modified example, when the second resolution images of a plurality of subjects determined to have high evaluation values are aggregated and transmitted, the second resolution images with high evaluation values are sequentially transmitted. Therefore, it is possible to quickly transmit the second resolution image with high importance. As a result, the pathologist can diagnose the specimen using the second resolution images sequentially from the second resolution images with high importance.

That is, it is considered that the image transmitting apparatus 1100a is used in a local hospital where only pathologists with little experience or only surgeons are available. In this case, the second-resolution image of the difficult-to-diagnose specimen is sent to a large hospital with experienced pathologists through the sending unit 1204a. Although the communication line is limited, if the image output device 2001a in this modified example is used, it is possible to preferentially transmit second-resolution images of specimens that are difficult to diagnose from a local hospital to a large hospital. In addition, the image transmission device 1100a can be used in a small hospital with only one pathologist to select an image to be transmitted when an important image is transmitted to a pathologist at a remote location for review.

In addition, in pathological examinations, multiple samples may be photographed from the same patient. For example, in the examination of cancer, a part of an organ is sometimes obtained, and slices of different cross-sections of the part are observed. That is, a plurality of two-dimensional images are obtained by imaging specimen slices at different positions (for convenience, different heights) in the vertical direction, and three-dimensional cancer ranges are confirmed using these images. Here, when cancer is suspected in an image at an arbitrary height, it is desirable to transmit or hold that image and images at other heights at high resolution.

Therefore, the high-resolution image acquisition unit 1102 may add the same image group identifier to a plurality of second-resolution images obtained from the same organ of the same patient. In this case, when transmitting the second resolution image, the transmitting unit 1204a also transmits to the image receiving device 1200a other second resolution images associated with the image group identifier added to the second resolution image. As a result, the pathologist who operates the image receiving device 1200a can acquire multiple second-resolution images obtained for the same organ of the same patient without having to individually select them, and can easily confirm the range of cancer and the like three-dimensionally.

(Modification 2)

In Embodiment 2 and Modification 1 described above, the determination unit 1202 is provided in the image receiving device, but may also be provided in the image transmitting device. An image transmission device according to the second modification includes a determination unit 1202 .

FIG. 10 is a configuration diagram showing an example of an image processing system including an image output device according to this modification.

An image processing system 200Sb of this modification includes a digital microscope 1500 , an image output device 2001 b , a display device 1501 , a recording medium 1502 , and a recording medium 1503 .

The image output device 2001b includes an image transmitting device 1100b and an image receiving device 1200b connected to each other via a communication line.

The image transmission device 1100b includes a determination unit 1202 in addition to the constituent elements included in the image transmission device 1100a of the second embodiment and its first modification.

The image receiving device 1200 b does not include the determination unit 1202 , but includes an enlargement receiving unit 1201 , a display output unit 1203 , and a first output unit 1205 .

FIG. 11 is a flowchart showing processing operations of the image transmitting device 1100b and the image receiving device 1200b.

The image acquisition unit 1101 of the image transmission device 1100b acquires a first resolution image from the digital microscope 1500 (step S101). The transmitting unit 1204 transmits the first-resolution image to the image receiving device 1200 via the communication line (step S102 ).

The display output unit 1203 of the image receiving device 1200b receives the first-resolution image transmitted from the image transmitting device 1100b via the communication line (step S121). Next, the magnification accepting unit 1201 of the image receiving device 1200b accepts the magnification ratio of the image based on, for example, the operation of the keyboard by the operator (step S122). Then, the display output unit 1203 enlarges the first resolution image at the accepted enlargement factor and outputs it to the display device 1501 . Based on the output, the display output unit 1203 causes the display device 1501 to display the image with the first resolution enlarged to the accepted magnification factor (step S123 ).

Next, the display output unit 1203 of the image receiving device 1200b determines whether to end the display of the first resolution image based on the signal output from the display device 1501 by selecting the end key 602 or the read key 605 (step S124). Here, when it is determined that the display should not be terminated (step S124: No), the enlargement accepting unit 1201 further accepts a new enlargement factor. That is, the operator (pathologist) who observes the image displayed on the display device 1501 performs observation by changing the magnification according to the specimen. On the other hand, when it is determined that the display should be terminated (step S124: Yes), the display output unit 1203 transmits the evaluation value-related information necessary for deriving the evaluation value to the image transmission device 1100b via the communication line (step S126). The evaluation value-related information is, for example, at least one of the above-mentioned last magnification, maximum magnification, display count, display time, and display area.

The determination unit 1202 of the image transmission device 1100b receives the evaluation value-related information (step S104). Then, the determination unit 1202 derives an evaluation value based on the evaluation value-related information. Furthermore, the determination unit 1202 determines whether the derived evaluation value is higher than a predetermined value, and notifies the image receiving device 1200b of the determination result via the communication line (step S105).

When the first output unit 120 of the image receiving device 1200b receives the notified determination result from the image transmitting device 1100b via the communication line (step S127), it checks whether the determination result is affirmative (Yes) (step S128). Here, when it is confirmed that the determination result is negative, that is, when the evaluation value is not higher than the predetermined value (step S128: No), the first output unit 1205 of the image receiving device 1200b outputs and saves the image with the first resolution in the record. medium 1502 (step S132). On the other hand, when it is confirmed that the determination result is affirmative, that is, when the evaluation value is higher than the predetermined value (step S128: Yes), the first output unit 1205 of the image receiving device 1200b requests the image transmitting device 1100 via the communication line for the first Two-resolution image (step S129).

The transmitting unit 1204 of the image transmitting device 1100b determines whether or not an image of the second resolution has been requested from the image receiving device 1200b (step S106). Here, when it is determined that there is no request (step S106: No), the high-resolution image acquisition unit 1102 acquires a second-resolution image (step S107). Furthermore, the second output unit 1103 outputs and saves the second resolution image acquired by the high resolution image acquisition unit 1102 to the recording medium 1503 (step S108 ).

On the other hand, when it is determined that a second-resolution image is requested (step S106: Yes), the high-resolution image acquisition unit 1102 acquires a second-resolution image (step S109). Then, the transmitting unit 1204 transmits the second-resolution image acquired by the high-resolution image acquiring unit 1102 to the image receiving device 1200b via the communication line (step S110).

In addition, in the above example, the processing of step S107 and step S109 is performed after the determination of step S106, but the processing of step S107 and step S109 may be performed before the determination of step S106.

When the second resolution image is transmitted in step S110, the first output unit 1205 of the image receiving device 1200b receives the second resolution image (step S130). Then, the first output unit 1205 outputs and stores the second resolution image to the recording medium 1502 (step S131 ).

(Effect)

In this way, in this modified example, even if the determination unit 1202 is provided in the image transmission device 1100b, the same effects as those of the first and second embodiments described above can be achieved.

Also in this modified example, as in Modified Example 1, each second-resolution image of a plurality of subjects associated with an evaluation value higher than a predetermined value may be selected from the second resolution image with a higher evaluation value. Images are sent and received sequentially.

(Embodiment 3)

In Embodiment 1, Embodiment 2, and their modifications, the transmission of the second-resolution image is switched or not transmitted, depending on whether the evaluation value is higher than a predetermined value. In this embodiment, the above-mentioned switching between transmission and non-transmission is not performed, but according to whether the evaluation value is higher than a predetermined value, switching between storing the second resolution image or not storing the second resolution image is performed. In addition, among the devices and components in this embodiment, those that are the same as those in Embodiment 1 or 2 are given the same reference numerals as in Embodiment 1 or 2, and their detailed descriptions are omitted.

FIG. 12 is a configuration diagram showing an example of an image processing system including an image output device according to Embodiment 3. FIG.

The image processing system 300S includes a digital microscope 1500 , an image output device 3001 , a display device 1501 , and a recording medium 1502 .

The image output device 3001 includes an image acquisition unit 1101 , a high-resolution image acquisition unit 1102 , an enlargement reception unit 1201 , a determination unit 1202 , a display output unit 1203 , and a first output unit 1205 . That is, the image output device 3001 in the present embodiment includes the first output unit 1205 instead of the transmission unit 1204 among the components included in the image output device 1001 in the first embodiment. In addition, the image output device 3001 in this embodiment includes the display output unit 1203 , but similarly to the first embodiment, the display output unit 1203 may not be provided.

Like the modification of Embodiment 1, the first output unit 1205 outputs and stores the second resolution image in the recording medium 1502 when the determination unit 1202 determines that the evaluation value is higher than the predetermined value. On the other hand, the first output unit 1205 does not output the second resolution image to the recording medium 1502 when the determination unit 1202 determines that the evaluation value is not higher than the predetermined value.

(Effect)

Thus, in this embodiment, as in the modified example of Embodiment 1, it is possible to automatically save a high-resolution image (second resolution image) for a specimen requiring diagnosis at a high resolution. In addition, since high-resolution images are not stored for specimens that do not require high-resolution diagnosis, it is possible to suppress the limitation of the free capacity of the recording medium 1502 .

(Modification 1)

The image output device 3001 in Embodiment 3 described above is configured as one device, but may be configured as two devices as in Embodiment 2. The image output device in this modified example is composed of an image transmitting device and an image receiving device connected to each other via a communication line.

FIG. 13 is a configuration diagram showing an example of an image processing system including an image output device according to a first modified example of Embodiment 3. FIG.

The image processing system 300Sa is configured to include a digital microscope 1500 , an image output device 3001 a , a display device 1501 , and a recording medium 1502 .

The image output device 3001a includes an image transmitting device 3100a and an image receiving device 3200a connected to each other via a communication line. In addition, the image output device 3001a has the same function as the image output device 3001 of Embodiment 3 as a whole. In addition, for example, a group including the digital microscope 1500 and the image transmission device 3100a is arranged in a facility such as a hospital without a pathologist. A group including the image receiving device 3200a, the display device 1501, and the recording medium 1502 is arranged, for example, in a facility with a pathologist located far from the hospital.

The image transmission device 3100 a includes an image acquisition unit 1101 . The image acquiring unit 1101 acquires a first-resolution image from the digital microscope 1500, and transmits the first-resolution image to the image receiving device 3200a via a communication line.

The image receiving device 3200 a includes a high-resolution image acquisition unit 1102 , an enlargement receiving unit 1201 , a determination unit 1202 , a display output unit 1203 , and a first output unit 1205 . The high-resolution image acquisition unit 1102 of this modified example acquires a plurality of first-resolution images from the image transmission device 3100 a via a communication line in response to a request from the first output unit 1205 . Then, the high-resolution image acquisition unit 1102 generates a second-resolution image based on these first-resolution images, thereby acquiring the second-resolution image.

When the judgment unit 1202 determines that the evaluation value is higher than the predetermined value, the first output unit 1205 requests the high-resolution image acquisition unit 1102 for a second-resolution image, and acquires the second-resolution image from the high-resolution image acquisition unit 1102. image. In addition, the first output unit 1205 outputs and stores the second resolution image in the recording medium 1502 . On the other hand, when the determination unit 1202 determines that the evaluation value is not higher than the predetermined value, the first output unit 1205 does not output the second resolution image to the recording medium 1502 .

(Effect)

In such an image output device 3001a in this modified example, even if the facility that acquires the image of the subject and the facility that observes the image are separated, the same effect as that of Embodiment 3 can be achieved.

(Modification 2)

In the first modified example of the third embodiment described above, the determination unit 1202 is provided in the image receiving device, but it may also be provided in the image transmitting device. The image transmission device in the second modification includes a determination unit 1202 .

14 is a configuration diagram showing another example of an image processing system including an image output device according to a second modified example of Embodiment 3. FIG.

The image processing system 300Sb includes: a digital microscope 1500 , an image output device 3001 b , a display device 1501 , and a recording medium 1502 .

The image output device 3001b includes an image transmitting device 3100b and an image receiving device 3200b connected to each other via a communication line. In addition, the image output device 3001b has the same function as the image output device 3001 of Embodiment 3 as a whole. In addition, for example, a group including the digital microscope 1500 and the image transmission device 3100b is arranged in a facility such as a hospital without a pathologist. A group including the image receiving device 3200b, the display device 1501, and the recording medium 1502 is arranged, for example, in a facility with a pathologist located far from the hospital.

The image transmission device 3100 b includes an image acquisition unit 1101 and a determination unit 1202 . The image acquiring unit 1101 acquires a first-resolution image from the digital microscope 1500, and transmits the first-resolution image to the image receiving device 3200b via a communication line. The determination unit 1202 acquires the above-described evaluation value-related information transmitted from the image receiving device 3200b via the communication line, and derives an evaluation value based on the evaluation value-related information. In addition, the determination unit 1202 determines whether the evaluation value is higher than a predetermined value, and notifies the image receiving device 3200b of the determination result via the communication line.

The image receiving device 3200 b includes a high-resolution image acquisition unit 1102 , an enlargement receiving unit 1201 , a display output unit 1203 , and a first output unit 1205 .

The high-resolution image acquisition unit 1102 acquires a plurality of first-resolution images from the image transmission device 3100b via a communication line in response to a request from the first output unit 1205 . In addition, the high-resolution image acquisition unit 1102 generates a second-resolution image based on these first-resolution images, thereby acquiring the second-resolution image.

The display output unit 1203 transmits the evaluation value-related information to the image transmission device 3100b via the communication line. As a result, the determination by the determination unit 1202 of the image transmission device 3100b is performed.

When the first output unit 1205 confirms that the determination result notified by the image transmission device 3100b is affirmative, that is, when it confirms that the evaluation value is higher than a predetermined value, it requests the high-resolution image acquisition unit 1102 for a second-resolution image. In addition, the first output unit 1205 acquires the second resolution image from the high resolution image acquisition unit 1102 , and outputs and stores the second resolution image in the recording medium 1502 . On the other hand, when the first output unit 1205 confirms that the judgment result notified from the image transmission device 3100b is negative, that is, when it confirms that the evaluation value is not higher than the predetermined value, it does not send the second resolution image to the recording medium 1502. output.

(Effect)

In this manner, in this modified example, even if the determination unit 1202 is provided in the image transmission device 3100b, the same effects as those of the third embodiment and its first modified example can be achieved.

(Embodiment 4)

Here, the digital microscope 1500 of each of the above-mentioned embodiments and modifications thereof will be described in detail below. In addition, the digital microscope 1500 will be referred to as an image acquisition device (digitizer) below.

<Principle of High Resolution Image Formation>

In the present disclosure, using a plurality of images obtained by performing photographing a plurality of times by changing the irradiation direction of illumination light, an image having a higher resolution (resolving power) than each of these plurality of images (hereinafter referred to as "high-resolution image") is formed. high-resolution image” or “high-resolution image”). In addition, the high-resolution image corresponds to the above-mentioned second resolution image, and the plurality of images (sub-images) used to form the high-resolution image correspond to the above-mentioned first resolution image, respectively. First, the principle of high-resolution image formation will be described with reference to FIGS. 15A to 20 . Here, a CCD (Charge Coupled Device: Charge Coupled Device) image sensor will be described as an example. In addition, in the following description, components having substantially the same function may be denoted by common reference numerals, and description thereof may be omitted.

Refer to FIG. 15A and FIG. 15B. Fig. 15A is a plan view schematically showing a part of a subject. The subject 2 shown in FIG. 15A is, for example, a thin slice of biological tissue (typically, having a thickness of several tens of μm or less). When an image of the subject 2 is acquired, the subject 2 is placed close to the imaging surface of the image sensor. The distance from the imaging surface of the image sensor to the subject 2 is typically 1 mm or less, and can be set to, for example, about 1 μm.

15B is a plan view schematically showing photodiodes related to imaging of the region shown in FIG. 15A among the photodiodes of the image sensor. In the example described here, six photodiodes are shown among the photodiodes 4 p of the image sensor 4 . In addition, for reference, in FIG. 15B , arrows indicating the x direction, the y direction, and the z direction orthogonal to each other are illustrated. The z direction represents the normal direction of the imaging surface. FIG. 15B also illustrates an arrow indicating the u direction, which is a direction rotated by 45° from the x-axis toward the y-axis in the xy plane. In other drawings, arrows indicating the x direction, the y direction, the z direction, or the u direction may be shown in figures.

Components other than the photodiode 4 p in the image sensor 4 are covered with a light-shielding layer. In FIG. 15B , the hatched area indicates the area covered by the light-shielding layer. The area (S2) of the light-receiving surface of one photodiode on the imaging surface of the CCD image sensor is smaller than the area (S1) of the unit region including the photodiode. The ratio (S2/S1) of the light-receiving area S2 to the area S1 of the pixel is referred to as "aperture ratio". Here, description will be made with an aperture ratio of 25%.

16A and 16B schematically show the direction of light rays that pass through the subject 2 and enter the photodiode 4p. 16A and 16B show a state where light is incident from a direction perpendicular to the imaging surface. As shown schematically in FIGS. 16A and 16B , no imaging lens is arranged between the subject 2 and the image sensor 4 , and the image of the subject 2 is transmitted through the subject 2 . to obtain substantially parallel rays of light.

FIG. 16C schematically shows an image Sa (first sub-image Sa) acquired in the irradiation directions shown in FIGS. 16A and 16B . As shown in FIG. 16C , the first sub-image Sa is composed of six pixels Pa captured by six photodiodes 4p. Each pixel Pa has a value (pixel value) indicating the amount of light incident on each photodiode 4p.

As shown in FIG. 16A and FIG. 16B, when the subject 2 is irradiated from a direction perpendicular to the photographing surface, the light transmitted through the area directly above the photodiode 4p in the entire subject 2 enters the photodiode 4p. Diode 4p. In this example, the first sub image Sa has information on areas A1 , A2 , A3 , A4 , A5 , and A6 (see FIG. 15A ) in the entire subject 2 . In addition, light transmitted through a region not directly above the photodiode 4p does not enter the photodiode 4p. Therefore, in the first sub-image Sa, information on regions other than the regions A1 , A2 , A3 , A4 , A5 , and A6 in the entire subject 2 is missing.

17A and 17B show a state where light is made to enter from an irradiation direction different from the irradiation direction shown in FIGS. 16A and 16B . The light rays shown in FIGS. 17A and 17B are inclined in the x direction with respect to the z direction. At this time, light that has passed through a region of the entire subject 2 that is different from the region immediately above the photodiode 4p enters the photodiode 4p.

FIG. 17C schematically shows an image Sb (second sub-image Sb) acquired in the irradiation directions shown in FIGS. 17A and 17B . As shown in FIG. 17C , the second sub-image Sb is also composed of six pixels captured by six photodiodes 4p. However, the pixels Pb constituting the second sub-image Sb have areas B1, B2, B3, B4, B5, and B6 different from areas A1, A2, A3, A4, A5, and A6 ( See Figure 15A) for the associated pixel values. In other words, the second sub-image Sb does not have information on areas A1, A2, A3, A4, A5, and A6 of the entire subject 2, but instead has information on areas B1, B2, B3, B4, B5, and B6. information. Here, for example, the area B1 is an area adjacent to the right side of the area A1 in the subject 2 (see FIG. 15A ).

As can be understood by comparing FIGS. 16A and 16B with FIGS. 17A and 17B , by appropriately changing the irradiation direction, light rays transmitted through different regions of the subject 2 can be incident on the photodiode 4p. As a result, the first sub-image Sa and the second sub-image Sb can include pixel information corresponding to different positions in the subject 2 .

18A and 18B show a state where light is made incident from an irradiation direction different from the irradiation direction shown in FIGS. 16A and 16B and the irradiation direction shown in FIGS. 17A and 17B . The light rays shown in FIGS. 18A and 18B are inclined in the y direction with respect to the z direction.

FIG. 18C schematically shows an image Sc (third sub-image Sc) acquired in the irradiation directions shown in FIGS. 18A and 18B . As shown in FIG. 18C, the third sub-image Sc is composed of six pixels Pc captured by six photodiodes 4p. As shown in the figure, the third sub-image Sc has information on areas C1 , C2 , C3 , C4 , C5 , and C6 shown in FIG. 15A in the entire subject 2 . Here, the area C1 is, for example, an area adjacent to the upper side of the area A1 in the subject 2 (see FIG. 15A ).

FIG. 19A shows a state where light is incident from an irradiation direction different from the irradiation direction shown in FIGS. 16A and 16B , the irradiation direction shown in FIGS. 17A and 17B , and the irradiation direction shown in FIGS. 18A and 18B . The light beam shown in FIG. 19A is inclined in a direction forming an angle of 45° with respect to the x-axis in the xy plane with respect to the z-direction.

FIG. 19B schematically shows an image Sd (fourth sub-image Sd) acquired in the irradiation direction shown in FIG. 19A . As shown in FIG. 19B , the fourth sub-image Sd is composed of six pixels Pd captured by six photodiodes 4p. The fourth sub-image Sd has information on areas D1 , D2 , D3 , D4 , D5 , and D6 shown in FIG. 15A in the entire subject 2 . Here, for example, the area D1 is an area adjacent to the right side of the area C1 (see FIG. 15A ). In this way, each of the sub-images Sa, Sb, Sc, and Sd includes images composed of different parts of the subject 2 .

FIG. 20 shows a high-resolution image HR synthesized from four sub-images Sa, Sb, Sc, and Sd. As shown in FIG. 20 , the number of pixels or pixel density of the high-resolution image HR is four times that of each of the four sub-images Sa, Sb, Sc, and Sd.

For example, attention is paid to blocks in the areas A1, B1, C1, and D1 shown in FIG. 15A in the subject 2. FIG. As can be seen from the description so far, the pixel Pa1 of the sub-image Sa shown in FIG. 20 does not have information on the entire block described above, but only information on the area A1. Therefore, the sub-image Sa can be said to be an image in which the information of the regions B1, C1, and D1 is missing.

However, by using sub-images Sb, Sc, and Sd having pixel information corresponding to different positions in the subject 2, as shown in FIG. The information of the high-resolution image HR. While the resolution of each sub-image is equal to the intrinsic resolution of the image sensor 4 , in this example, a resolution four times the intrinsic resolution of the image sensor 4 can be obtained. The degree of high resolution (super resolution) depends on the aperture ratio of the image sensor. In this example, since the aperture ratio of the image sensor 4 is 25%, it is possible to increase the resolution by a maximum of four times by irradiating light from four different directions. When N is an integer equal to or greater than 2, if the aperture ratio of the image sensor 4 is approximately equal to 1/N, a maximum of N times higher resolution can be realized.

In this way, by sequentially irradiating parallel light from a plurality of different irradiation directions based on the subject to capture the subject, it is possible to increase the pixel information "spatially" sampled from the subject. By combining the obtained sub-images, it is possible to form a high-resolution image higher than the resolution of each of the sub-images. Of course, the irradiation direction is not limited to the irradiation direction described with reference to FIGS. 16A to 19B .

In addition, in the above example, the sub-images Sa, Sb, Sc, and Sd shown in FIG. 20 have pixel information of mutually different regions in the subject 2 and do not overlap. However, it is also possible to have overlaps between different sub-images. In addition, in the above-mentioned example, light rays passing through two adjacent areas of the subject 2 are both incident on the same photodiode. However, the setting of the irradiation direction is not limited to this example. For example, as shown in FIG. 21 , the irradiation direction may also be adjusted so that light rays passing through two adjacent regions of the subject 2 are respectively incident on different photodiodes.

<module>

In forming a high-resolution image based on the principles described with reference to FIGS. 15A to 20 , sub-images are acquired while the subject 2 and the imaging surface of the image sensor 4 are arranged close to each other. In the embodiment of the present disclosure, sub-images are acquired using a module having a structure in which the subject 2 and the image sensor 4 are integrated. Hereinafter, an example of a configuration of a module and an example of a method of manufacturing the module will be described with reference to the drawings.

FIG. 22A schematically shows an example of a cross-sectional structure of a module. In module 10 shown in FIG. 22A , subject 2 covered with mounting medium 6 is arranged on imaging surface 4A of image sensor 4 . In the illustrated example, a transparent plate (typically a glass plate) 8 is arranged on the subject 2 . That is, in the configuration illustrated in FIG. 22A , subject 2 is sandwiched between image sensor 4 and transparent plate 8 . It is advantageous if the module 10 has the transparent plate 8 because the operability is improved. As the transparent plate 8, for example, a general slide glass can be used. In addition, in the drawing, each element is schematically shown, and the actual size and shape of each element do not necessarily correspond to the size and shape shown in the drawing. The same applies to other drawings to be referred to below.

In the configuration illustrated in FIG. 22A , image sensor 4 is fixed to package 5 . FIG. 22B shows an example of the appearance of the module 10 shown in FIG. 22A viewed from the image sensor 4 side. As shown in FIGS. 22A and 22B , package 5 has back electrode 5B on the surface opposite to transparent plate 8 . The back electrode 5B is electrically connected to the image sensor 4 via a wiring pattern (not shown) formed in the package 5 . That is, the output of the image sensor 4 can be taken out via the back electrode 5B. In this specification, a structure in which a package and an image sensor are integrated is called an "imaging element".

Referring to FIG. 23 , an example of a method of manufacturing the module 10 will be described. Here, as the subject 2, a thin slice of biological tissue (tissue section) is exemplified. The module 10 having a slice of biological tissue as the subject 2 can be used for pathological diagnosis.

First, as shown in FIG. 23 , the tissue section A02 is placed on the transparent plate 8 . The transparent plate 8 may be a slide glass used for observation of a sample by an optical microscope. Hereinafter, a slide glass is illustrated as the transparent plate 8 . Next, the tissue sections A02 are stained by immersing each transparent plate 8 in the staining solution Ss. Next, the mounting medium 6 is supplied onto the transparent plate 8 , whereby the subject 2 obtained by staining the tissue section A02 is covered with the mounting medium 6 . The mounting medium 6 has the function of protecting the object 2 to be photographed. Next, the imaging element 7 is arranged on the subject 2 such that the imaging surface of the image sensor 4 faces the subject 2 . In this way, module 10 is obtained.

The module 10 is created for each subject to be photographed. For example, in the case of pathological diagnosis, multiple (for example, 5 to 20) tissue sections are prepared from one specimen. Therefore, a plurality of modules 10 including tissue slices obtained from the same specimen as the subject 2 can be produced. If a plurality of sub-images are acquired for each of the plurality of modules 10, a high-resolution image corresponding to each of the plurality of modules 10 can be formed.

As shown in FIG. 22A , the module 10 includes an image sensor 4 that acquires an image of a subject 2 , unlike a test slide used for observation by an optical microscope. Such a module may also be referred to as an "electronic test slide". As shown in FIG. 22A , by using the module 10 having the structure in which the subject 2 and the imaging element 7 are integrated, it is possible to obtain an advantage that the arrangement between the subject 2 and the image sensor 4 can be fixed.

When the acquisition of an image of the subject 2 is performed using the module 10 , the subject 2 is irradiated with illumination light via the transparent plate 8 . The illumination light transmitted through the subject 2 enters the image sensor 4 . Thus, an image of the subject 2 is obtained. By sequentially performing imaging while changing the relative arrangement of the light source and the subject, it is possible to obtain a plurality of different images by changing the angle during irradiation. For example, as shown in FIG. 24A , a light source 310 is arranged directly above the image sensor 4 . In addition, if the subject 2 is captured with the collimated light CL irradiated from the normal direction of the imaging surface 4A of the image sensor 4 to the subject 2, the same sub-image Sa as shown in FIG. 16C can be obtained. image. In addition, as shown in FIG. 24B , if the subject 2 is irradiated with collimated light CL and photographed while the module 10 is tilted, the sub-image Sb shown in FIG. 17C (or the sub-image Sb shown in FIG. 18C The sub-image Sc) shown is the same sub-image. In this manner, by sequentially performing imaging while changing the posture of the module 10 relative to the light source, it is possible to obtain a high-resolution image by applying the principle described with reference to FIGS. 15A to 20 .

<Image acquisition device>

FIG. 25 schematically shows an example of the configuration of an image acquisition device according to an embodiment of the present disclosure. The image acquisition device 100 a shown in FIG. 25 has an illumination system 30 . In the configuration illustrated in FIG. 25 , the illumination system 30 includes: a light source 31 that generates illumination light; a stage 32 configured to detachably load the module 10; Table drive mechanism 33. FIG. 25 schematically shows a state where the modules 10 are loaded on the stage 32 . However, the illustration of the sealing agent 6 and the transparent plate 8 in the module 10 is omitted. The module 10 is not an essential component of the image acquisition device 100a.

The module 10 has an arrangement such that the illumination light transmitted through the subject 2 enters the imaging element 7 in a state connected to the stage 32 . The illumination system 30 changes the irradiation direction based on the subject 2 by, for example, changing the attitude of the stage 32 . A change in "posture" in this specification broadly includes a change in inclination with respect to a reference plane, a change in a rotation angle with respect to a reference orientation, a change in a position with respect to a reference point, and the like. The subject 2 is irradiated sequentially with illumination light generated by the light source 31 from a plurality of different irradiation directions based on the subject 2 . The details of the configuration of the lighting system 30 and an example of its operation will be described below. The subject 2 is irradiated by changing the irradiation direction, and a plurality of different images (sub-images) are acquired corresponding to the plurality of different irradiation directions by the imaging element 7 . Using the obtained plurality of images, a high-resolution image can be formed.

The image acquisition device 100a shown in FIG. 25 has an irradiation direction determination unit 40a. The irradiation direction determination unit 40 a determines a plurality of different irradiation directions when the imaging element 7 acquires a plurality of sub-images. In an embodiment of the present disclosure, the acquisition of the sub-image is performed under a plurality of different irradiation directions determined by the irradiation direction determination unit. In other words, the sub-images in the embodiment of the present disclosure are a plurality of different images corresponding to a plurality of different irradiation directions determined by the irradiation direction determination unit. A specific example of the configuration and operation of the irradiation direction determining unit 40a will be described below.

Next, an example of a method of changing the irradiation direction of illumination light based on a subject will be described with reference to FIGS. 26A to 27B .

26A and 26B show an exemplary appearance of the image acquisition device 100a. In the configuration illustrated in FIG. 26A , an image acquisition device 100 a includes a main body 110 including a light source 31 and a stage 32 , and a cover 120 connected to the main body 110 so as to be openable and closable. By closing the lid portion 120, a dark room can be formed inside the image acquisition device 100a (see FIG. 26B ).

In the illustrated example, a socket 130 for holding the module 10 is connected to the stage 32 . The socket 130 may be fixed to the stage 32 or may be configured to be detachable from the stage 32 . Here, a configuration in which the socket 130 is detachably attached to the stage 32 is exemplified. The socket 130 includes, for example, a lower base 132 configured so that the module 10 can be attached and detached, and an upper base 134 in which the opening Ap is formed. In the structure illustrated in FIG. 26A , socket 130 holds module 10 by sandwiching module 10 between lower base material 132 and upper base material 134 .

The lower substrate 132 may have an electrical connection portion having electrical contacts for electrical connection with the imaging element 7 of the module 10 . When capturing an image of a subject, the module 10 is placed on the lower substrate 132 such that the imaging surface of the imaging element 7 faces the light source 31 . At this time, the electrical contact of the electrical connection portion is in contact with the rear surface electrode 5B (see FIGS. 22A and 22B ) of the imaging element 7 , thereby electrically connecting the imaging element 7 of the module 10 and the electrical connection portion of the lower substrate 132 .

FIG. 26C shows an example of how the socket 130 is loaded on the stage 32 of the image acquisition device 100a. In the configuration illustrated in FIG. 26C , socket 130 has electrodes 136 protruding from the bottom surface. The electrode 136 may be a part of the electrical connection portion of the lower substrate 132 . In addition, in the example shown in FIG. 26C , the stage 32 of the image acquisition device 100 a has a mounting portion 34 provided with an insertion hole 36 . As shown in FIG. 26C , for example, the socket 130 holding the module 10 is loaded on the stage 32 so that the electrodes 136 of the socket 130 are inserted into the socket 36 . Thereby, the electrical connection between the imaging element 7 in the module 10 held by the socket 130 and the image acquisition device 100 a is established. The stage 32 may have a circuit that receives the output of the image sensor 4 in a state where the socket 130 of the holding module 10 is loaded. In the embodiment of the present disclosure, the image acquisition device 100 a acquires information (image signal or image data) representing an image of the subject 2 by using the electrical connection part of the socket 130 as a medium.

In addition, when shooting a plurality of subjects using a plurality of modules 10 , the same number of sockets 130 as the modules 10 may be prepared, and the objects to be photographed may be changed by replacing the sockets 130 that hold the modules 10 . Alternatively, the object to be photographed may be changed by replacing the mounting module 10 while maintaining the state that one socket 130 is mounted on the stage 32 .

As shown in FIG. 26C , by loading the socket 130 on the stage 32 , the bottom surface of the socket 130 can be brought into close contact with the top surface of the mounting portion 34 . Thereby, the arrangement|positioning of the socket 130 with respect to the stage 32 is fixed. Therefore, the arrangement of the stage 32 and the modules 10 held by the socket 130 can be kept constant before and after the change in the posture of the stage 32 . Typically, when the socket 130 is mounted on the stage 32 , the main surface of the transparent plate 8 of the module 10 is substantially parallel to the stage 32 .

FIG. 27A shows an example of a method of changing the irradiation direction. As shown in the figure, the module 10 held by the socket 130 is irradiated with the illumination light CL emitted from the light source 31 . The illumination light CL enters the subject of the module 10 through the opening Ap provided in the socket 130 . The light transmitted through the subject is incident on the imaging surface of the imaging element 7 of the module 10 .

The light emitted from the light source 31 is typically collimated light. However, the light emitted from the light source 31 may not be collimated light when it can be considered that the light incident on the subject is substantially parallel light.

The light source 31 includes, for example, an LED chip. The light source 31 may also contain a plurality of LED chips each having a peak in a different wavelength range. For example, the light source 31 may include an LED chip that emits blue light, an LED chip that emits red light, and an LED chip that emits green light. When a plurality of light emitting elements are arranged close to each other (eg, about 100 μm), they can be regarded as point light sources.

Using a plurality of light-emitting elements that emit light of different colors, for example, by sequentially irradiating light of different colors in each irradiation direction, a plurality of sub-images for each color can be obtained. For example, a group of blue sub-images, a group of red sub-images, and a group of green sub-images may be acquired. Using the acquired set of sub-images, a color high-resolution image can be formed. For example, in the case of pathological diagnosis, more useful information on the presence or absence of lesions can be obtained by using color high-resolution images. As the light source 31, a white LED chip is used, and a color filter is arranged on the optical path, whereby illumination lights of mutually different colors can be obtained in a time-sequential manner. In addition, an image sensor for color imaging may be used as the image sensor 4 . However, from the viewpoint of suppressing a decrease in the amount of light incident on the photoelectric conversion portion of the image sensor 4 , the configuration without disposing a color filter is advantageous.

The light source 31 is not limited to an LED, but may also be an incandescent lamp, a laser element, a fiber laser, a discharge tube, or the like. The light emitted from the light source 31 is not limited to visible light, and may be ultraviolet rays, infrared rays, or the like. The number and arrangement of light emitting elements included in the light source 31 can also be set arbitrarily.

As shown in FIGS. 25 and 27A , the image acquisition device 100 a has a stage driving mechanism 33 . The stage drive mechanism 33 includes a gonio mechanism, a rotation mechanism, etc., and changes the inclination of the stage 32 relative to the main body 110 and/or the rotation angle about an axis passing through the center of the stage 32 . The stage drive mechanism 33 may also include a slide mechanism capable of parallelly moving the stage 32 within a reference plane (typically a horizontal plane).

The posture of the stage 32 can be changed by operating the stage driving mechanism 33 . Here, since the socket 130 holding the state of the module 10 is attached to the stage 32 , the posture of the module 10 can be changed by changing the posture of the stage 32 . For example, let the incident direction of the illumination light when the stage 32 is not inclined with respect to the reference plane be the normal direction of the imaging surface of the image sensor. Here, the relationship (for example, parallelism) between the inclination of the stage 32 relative to the reference plane and the inclination of the module 10 relative to the reference plane (which may also be referred to as the inclination of the transparent plate 8) is determined by the attitude of the stage 32. It remains constant before and after the change. Therefore, as shown in FIG. 27B , when the stage 32 is tilted by the angle θ with respect to the reference plane, the direction of the light rays incident on the subject is also tilted by the angle θ. In addition, in FIG. 27B , a dotted line N indicates a normal line to the imaging surface of the image sensor.

In this way, by changing the posture of the module 10 together with the stage 32 , it is possible to sequentially irradiate the subject with illumination light from a plurality of different irradiation directions with the subject 2 as a reference. Therefore, a plurality of images corresponding to a plurality of different irradiation directions based on the subject 2 can be acquired by the imaging element 7 of the module 10 . The irradiation direction based on the subject 2 can be set, for example, by the angle (zenith angle θ shown in FIG. The angle (azimuth angle) formed by the reference azimuth on the imaging surface and the projection of the incident light on the imaging surface is expressed as a set.

In addition, by moving the light source 31 in the image acquisition device 100a or sequentially turning on a plurality of light sources arranged at different positions, the subject 2 can be irradiated from a plurality of different irradiation directions. For example, the irradiation direction may be changed by moving the light source 31 in a direction connecting the light source 31 and the subject 2 . The irradiation direction can also be changed by combining the change of the attitude of the stage 32 with the movement of the light source 31 .

<Image sensor for module>

In addition, in the embodiment of the present disclosure, the image sensor 4 is not limited to a CCD image sensor, and may be a CMOS (Complementary Metal-Oxide Semiconductor: Complementary Metal-Oxide Semiconductor) image sensor or other image sensors (as an example, the following photoelectric conversion film stacked image sensor). The CCD image sensor and the CMOS image sensor may be either a surface-illuminated type or a back-illuminated type. Hereinafter, the relationship between the element structure of the image sensor and the light incident on the photodiode of the image sensor will be described.

FIG. 28 shows an example of the cross-sectional structure of the CCD image sensor and the distribution of the relative transmittance Td of the subject. As shown in FIG. 28 , the CCD image sensor roughly includes a substrate 80 , an insulating layer 82 on the substrate 80 , and wiring 84 arranged in the insulating layer 82 . A plurality of photodiodes 88 are formed on the substrate 80 . On the wiring 84, a light shielding layer (not shown in FIG. 28) is formed. Here, illustration of transistors and the like is omitted. Illustration of transistors and the like is omitted in the following drawings. In addition, the cross-sectional structure near the photodiode in the surface-illuminated CMOS image sensor is roughly the same as the cross-sectional structure near the photodiode in the CCD image sensor. Therefore, illustration and description of the cross-sectional structure of the surface-illuminated CMOS image sensor are omitted here.

As shown in FIG. 28 , when the illumination light enters from the normal direction of the imaging surface, the illumination light transmitted through the region R1 directly above the photodiode 88 in the subject enters the photodiode 88 . On the other hand, the irradiated light transmitted through the region R2 of the subject directly above the light-shielding layer on the wiring 84 enters the light-shielding region (region where the light-shielding film is formed) of the image sensor. Therefore, when irradiation is performed from the normal direction of the imaging surface, an image showing the region R1 located directly above the photodiode 88 in the subject is obtained.

In order to obtain an image showing the region directly above the light-shielding film, it is only necessary to irradiate from a direction oblique to the normal direction of the imaging surface so that the light transmitted through the region R2 enters the photodiode 88 . At this time, depending on the irradiation direction, part of the light transmitted through the region R2 may be blocked by the wiring 84 . In the illustrated example, the light passing through the hatched portion does not reach the photodiode 88 . Therefore, in oblique incidence, sometimes pixel values are a bit lower. However, not all transmitted light is blocked, and therefore, a high-resolution image can be formed using sub-images obtained at this time.

29A and 29B show examples of the cross-sectional structure of the back-illuminated CMOS image sensor and the distribution of the relative transmittance Td of the subject. As shown in FIG. 29A , in the backside illuminated CMOS image sensor, even in the case of oblique incidence, the transmitted light is not blocked by the wiring 84 . However, light that has passed through an area of the subject different from the area to be imaged (light schematically indicated by a thick arrow BA in FIG. 29A and in FIG. 29B described below) enters the substrate. 80, resulting in noise, and the quality of the sub-image may be degraded. As shown in FIG. 29B, this degradation can be reduced by forming a light-shielding layer 90 on a region other than the region where the photodiode is formed on the substrate.

30 shows the cross-sectional structure of an image sensor equipped with a photoelectric conversion film formed of an organic material or an inorganic material (hereinafter referred to as a "photoelectric conversion film laminated image sensor") and the distribution of the relative transmittance Td of the subject. example.

As shown in FIG. 30 , the photoelectric conversion film stacked image sensor roughly includes a substrate 80 , an insulating layer 82 provided with a plurality of pixel electrodes, a photoelectric conversion film 94 on the insulating layer 82 , and a transparent electrode 96 on the photoelectric conversion film 94 . As shown in the figure, in the photoelectric conversion film stack type image sensor, instead of photodiodes formed on the semiconductor substrate, a photoelectric conversion film 94 for performing photoelectric conversion is formed on a substrate 80 (for example, a semiconductor substrate). The photoelectric conversion film 94 and the transparent electrode 96 are typically formed over the entire imaging surface. Here, the illustration of the protective film that protects the photoelectric conversion film 94 is omitted.

In the photoelectric conversion film stacked image sensor, charges (electrons or holes) generated by photoelectric conversion of incident light in the photoelectric conversion film 94 collect on the pixel electrode 92 . Thus, a value indicating the amount of light incident on the photoelectric conversion film 94 is obtained. Therefore, in the photoelectric conversion film stacked image sensor, it can be said that a unit area including one pixel electrode 92 corresponds to one pixel on the imaging surface. In the photoelectric conversion film multilayer image sensor, even in the case of oblique incidence as in the back-illuminated CMOS image sensor, transmitted light is not blocked by wiring.

As described with reference to FIGS. 15A to 20 , in forming a high-resolution image, a plurality of sub-images representing an image composed of different parts of a subject are used. However, in a typical photoelectric conversion film stacked image sensor, the photoelectric conversion film 94 is formed over the entire imaging surface. Therefore, for example, even in the case of normal incidence, it is possible to pass through images other than the desired area of the subject. The light in the area is photoelectrically converted by the photoelectric conversion film 94 . If excess electrons or holes generated at this time are introduced into the pixel electrode 92, there is a possibility that an appropriate sub-image cannot be obtained. Therefore, it is beneficial to selectively introduce into the pixel electrode 92 the charge generated in the region where the pixel electrode 92 overlaps with the transparent electrode 96 (the hatched region in FIG. 30 ).

In the configuration illustrated in FIG. 30 , dummy electrodes 98 are provided in the pixels corresponding to the respective pixel electrodes 92 . When acquiring an image of a subject, an appropriate potential difference is supplied between the pixel electrode 92 and the dummy electrode 98 . Thereby, the charge generated in the region other than the region where the pixel electrode 92 and the transparent electrode 96 overlap can be introduced into the dummy electrode 98, and the charge generated in the region where the pixel electrode 92 and the transparent electrode 96 overlap can be selectively introduced. pixel electrode 92 . In addition, the same effect can also be obtained by patterning the transparent electrode 96 or the photoelectric conversion film 94 . In such a configuration, it can be said that the ratio (S3/S1) of the area S3 of the pixel electrode 92 to the area S1 of the pixel corresponds to the "aperture ratio".

As already explained, when N is an integer equal to or greater than 2, if the aperture ratio of the image sensor 4 is approximately equal to 1/N, a maximum of N times higher resolution can be realized. In other words, a small aperture ratio is advantageous for high resolution. In the photoelectric conversion film stacked image sensor, by adjusting the area S3 of the pixel electrode 92, the ratio (S3/S1) corresponding to the aperture ratio can be adjusted. This ratio (S3/S1) is set, for example, within a range of 10% to 50%. A photoelectric conversion film multilayer image sensor having a ratio (S3/S1) within the above-mentioned range can be used for super-resolution.

In addition, as can be seen from FIGS. 28 and 29B , in the CCD image sensor and the surface-illuminated CMOS image sensor, the surface facing the subject is not flat. For example, in a CCD image sensor, there is a difference in height on its surface. In addition, in back-illuminated CMOS image sensors, in order to obtain sub-images for forming high-resolution images, it is necessary to provide a patterned light-shielding layer on the imaging surface, and the surface facing the subject is not flat.

On the other hand, as can be seen from FIG. 30 , the imaging surface of the photoelectric conversion film-stacked image sensor is almost a flat surface. Therefore, even when a subject is arranged on the imaging surface, deformation of the subject due to the shape of the imaging surface hardly occurs. In other words, by acquiring sub-images using a photoelectric conversion film stacked image sensor, it is possible to observe a more detailed structure of a subject.

Above, the image output device according to one or more aspects has been described based on several embodiments, but the present invention is not limited to these embodiments. As long as they do not depart from the gist of the present invention, forms obtained by implementing various modifications conceivable by those skilled in the art to the present embodiment and/or forms constructed by combining components in different embodiments may also be included in this document. public.

In addition, in each of the above-described embodiments, each constituent element may be configured using dedicated hardware, or may be realized by executing a software program suitable for each constituent element. Each constituent element can also be realized by a program execution unit such as a CPU or a processor reading and executing a software program (that is, a command) recorded in a recording medium such as a hard disk or a semiconductor memory. Here, the software for realizing the image output device or the like in each of the above-described embodiments is a program that causes a computer to execute each step in the flowchart of FIG. 3 , FIG. 4 , FIG. 7 , FIG. 9 , or FIG. 11 . In addition, the program execution unit may be constituted by one processor, or may be constituted by a plurality of processors.

In addition, in the present disclosure, all or a part of a unit or a device, or all or a part of the functional blocks in the block diagrams shown in FIG. 1 , FIG. 5 , FIG. 6 , FIG. 8 , FIG. 10 , and FIG. It is implemented by one or more electronic circuits including a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large scale integration). An LSI or an IC may be integrated into one chip, or may be formed by combining a plurality of chips. For example, functional blocks other than memory elements may be integrated into one chip. Here, it is called LSI or IC, but depending on the degree of integration, the name will change, and it may be called system LSI, VLSI (very large scale integration; very large scale integration), or ULSI (ultra large scale integration ; VLSI). A Field Programmable Gate Array (FPGA: Field Programmable Gate Array) programmed after LSI manufacturing, or a reconfigurable logic device capable of reconfiguring the bonding relationship inside the LSI or setting the circuit division inside the LSI can also be used for the same purpose .

Furthermore, the functions or operations of a unit, a device, or a part, all, or a part of a device may be performed by software processing. In this case, the software is recorded in one or more non-transitory recording media such as ROMs, optical disks, and hard disk drives. When the software is executed by a processor, the software enables the processor and the peripheral A device performs a specific function within software. A system or an apparatus may include one or more non-transitory recording media on which software is recorded, a processor, and necessary hardware devices such as interfaces.

The present disclosure has an effect of reducing the processing load of high-resolution images, and is applicable, for example, to image output devices and the like that process high-resolution images of pathological specimens.

Claims (15)

1. An image output device, comprising: at least one processor; and a non-transitory recording medium holding the order, The command causes the at least one processor to perform: Acquiring a first resolution image, where the first resolution image is an image of the object to be photographed based on digital microscope photography; Acquiring a second resolution image, where the second resolution image is an image of the object to be photographed based on shooting by the digital microscope with a higher resolution than the first resolution image; receiving a magnification for the first resolution image displayed by the display device; determining whether or not the evaluation value based on the accepted magnification ratio is higher than a predetermined value; When it is determined that the evaluation value is higher than the predetermined value, the second resolution image is transmitted to the external device, and when it is determined that the evaluation value is not higher than the predetermined value, not to the external device The second resolution image is sent. 2. The image output device according to claim 1, The command also causes the at least one processor to perform: When it is determined that the evaluation value is not higher than the predetermined value, the first resolution image is transmitted to the external device. 3. The image output device according to claim 1 or 2, The command also causes the at least one processor to perform: When it is determined that the evaluation value is higher than the predetermined value, the second resolution image is output and stored in a recording medium, and when it is determined that the evaluation value is not higher than the predetermined value, the outputting the second resolution image from the recording medium. 4. The image output device according to claim 1 or 2, The command also causes the at least one processor to perform: The largest magnification factor among the accepted one or more magnification factors is derived as the evaluation value. 5. The image output device according to claim 1 or 2, The command also causes the at least one processor to perform: The higher the number of times a high magnification ratio is accepted as a magnification ratio higher than a threshold value in the acceptance of the magnification ratio, or the longer the time for which the image of the subject is displayed on the display device at the high magnification ratio is, A higher evaluation value is then derived. 6. The image output device according to claim 1 or 2, The command also causes the at least one processor to perform: The higher the evaluation value is calculated, the larger the area of the subject is displayed on the display device at the magnification ratio higher than the threshold value. 7. The image output device according to claim 1, the first resolution image is a first first resolution image, The command causes the at least one processor to execute: acquiring a second first-resolution image, where the second first-resolution image is an image of the subject obtained based on shooting by the digital microscope, the second resolution image is generated based on the first first resolution image and the second first resolution image, The digital microscope emits first light at a first angle to the object to be photographed and second light at a second angle to the object to be photographed, and emits the second light after emitting the first light, The digital microscope generates the first first resolution image based on a first plurality of pixel values, generates the second first resolution image based on a second plurality of pixel values, The number of pixel values of the second resolution image is greater than the number of the first plurality of pixel values, and the number of pixel values of the second resolution image is greater than the number of the second plurality of pixel values many, The subject includes a first part and a second part adjacent to the first part, The plurality of photoelectric conversion units output the plurality of first pixel values including the first pixel value based on the first converted light obtained by the first light passing through the subject, and output the plurality of first pixel values including the first pixel value based on the second light passing through the outputting the second plurality of pixel values including the second pixel value from the second converted light obtained by photographing the subject, The plurality of photoelectric conversion parts includes a first photoelectric conversion part and a second photoelectric conversion part adjacent to the first photoelectric conversion part, the first photoelectric conversion unit outputs the first pixel value based on a part of the first converted light obtained by passing a part of the first light through the first part, The second photoelectric conversion unit outputs the second pixel value based on a portion of the second converted light obtained by passing a portion of the second light through the second portion. 8. An image output device having an image sending device and an image receiving device connected to the image sending device via a communication line, The image sending device has: at least one sending processor; and a non-transitory recording medium for transmission holding a command for transmission, The sending command causes the at least one sending processor to perform: Acquiring a first resolution image and sending it to the image receiving device, the first resolution image is an image of the object to be photographed based on a digital microscope; Acquiring a second resolution image, where the second resolution image is an image of the object to be photographed based on shooting by the digital microscope with a higher resolution than the first resolution image; determining whether the evaluation value indicated by the evaluation value-related information notified by the image receiving device is higher than a predetermined value; when it is determined that the evaluation value is higher than the predetermined value, sending the second resolution image to the image receiving device, The image receiving device has: at least one receiving processor; and a recording medium for non-transitory reception holding a command for reception, The receiving command causes the at least one receiving processor to perform: acquiring the first resolution image from the image sending device and displaying it on a display device; receiving a magnification for the first resolution image displayed by the display device; Enlarging the first resolution image displayed by the display device based on the accepted magnification ratio; The evaluation value-related information based on the received magnification factor is transmitted to the image transmission device. 9. An image output device having an image sending device and an image receiving device connected to the image sending device via a communication line, The image sending device has: at least one sending processor; and a non-transitory recording medium for transmission holding a command for transmission, The sending command causes the at least one sending processor to perform: Acquiring a first resolution image and sending it to the image receiving device, the first resolution image is an image of the object to be photographed based on a digital microscope; Acquiring a second resolution image, the second resolution image is an image of the object to be photographed based on the shooting of the digital microscope with a higher resolution than the first resolution image, and the image receives The device has: at least one receiving processor; and a recording medium for non-transitory reception holding a command for reception, The receiving command causes the at least one receiving processor to perform: acquiring the first resolution image from the image sending device and displaying it on a display device; receiving a magnification for the first resolution image displayed by the display device; Enlarging the first resolution image displayed by the display device based on the accepted magnification ratio; determining whether or not the received evaluation value based on the magnification factor is higher than a predetermined value, and notifying the image transmission device of a determination result, The sending command of the image sending device also causes the at least one sending processor to execute: When the determination result notified from the image receiving device indicates that the evaluation value is higher than the predetermined value, the second resolution image is transmitted to the image receiving device. 10. The image output device according to claim 8 or 9, The sending command further causes the at least one sending processor to perform: In the determination of the evaluation value, when it is determined that the evaluation value is not higher than the predetermined value, the second resolution image is output and stored in a recording medium. 11. An image output method, comprising: Acquiring a first resolution image, where the first resolution image is an image of the object to be photographed based on digital microscope photography; Acquiring a second resolution image, where the second resolution image is an image of the object to be photographed based on shooting by the digital microscope with a higher resolution than the first resolution image; receiving a magnification for the first resolution image displayed by the display device; determining whether or not the evaluation value based on the accepted magnification ratio is higher than a predetermined value; When it is determined that the evaluation value is higher than the predetermined value, the second resolution image is transmitted to the external device, and when it is determined that the evaluation value is not higher than the predetermined value, not to the external device The second resolution image is sent. 12. An image receiving device connected to an image sending device via a communication line, The image sending device has: at least one sending processor; and a non-transitory recording medium for transmission holding a command for transmission, The sending command causes the at least one sending processor to perform: Acquiring a first resolution image and sending it to the image receiving device, the first resolution image is an image of the object to be photographed based on a digital microscope; Acquiring a second resolution image, where the second resolution image is an image of the object to be photographed based on shooting by the digital microscope with a higher resolution than the first resolution image; determining whether the evaluation value indicated by the evaluation value-related information notified by the image receiving device is higher than a predetermined value; when it is determined that the evaluation value is higher than the predetermined value, sending the second resolution image to the image receiving device, The image receiving device has: at least one receiving processor; and a recording medium for non-transitory reception holding a command for reception, The receiving command causes the at least one receiving processor to perform: acquiring the first resolution image from the image sending device and displaying it on a display device; receiving a magnification for the first resolution image displayed by the display device; Enlarging the first resolution image displayed by the display device based on the accepted magnification ratio; The evaluation value-related information based on the received magnification factor is transmitted to the image transmission device. 13. An image receiving device connected to an image sending device via a communication line, The image sending device has: at least one sending processor; and a non-transitory recording medium for transmission holding a command for transmission, The sending command causes the at least one sending processor to perform: Acquiring a first resolution image and sending it to the image receiving device, the first resolution image is an image of the object to be photographed based on a digital microscope; acquiring a second resolution image, where the second resolution image is an image of the object to be photographed based on shooting by the digital microscope with a resolution higher than that of the first resolution image, The image receiving device has: at least one receiving processor; and a recording medium for non-transitory reception holding a command for reception, The receiving command causes the at least one receiving processor to perform: acquiring the first resolution image from the image sending device and displaying it on a display device; receiving a magnification for the first resolution image displayed by the display device; Enlarging the first resolution image displayed by the display device based on the accepted magnification ratio; determining whether or not the received evaluation value based on the magnification factor is higher than a predetermined value, and notifying the image transmission device of a determination result, The sending command of the image sending device also causes the at least one sending processor to execute: When the determination result notified from the image receiving device indicates that the evaluation value is higher than the predetermined value, the second resolution image is transmitted to the image receiving device. 14. An image sending device connected to an image receiving device via a communication line, The image receiving device has: at least one receiving processor; and a recording medium for non-transitory reception holding a command for reception, The receiving command causes the at least one receiving processor to perform: Obtaining a first resolution image from the image sending device and displaying it on a display device, the first resolution image is an image of the object to be photographed based on a digital microscope; receiving a magnification for the first resolution image displayed by the display device; Enlarging the first resolution image displayed by the display device based on the accepted magnification ratio; transmitting evaluation value-related information based on the accepted magnification factor to the image transmission device, The image sending device has: at least one sending processor; and a non-transitory recording medium for transmission holding a command for transmission, The sending command causes the at least one sending processor to perform: acquiring the first resolution image and sending it to the image receiving device; Acquiring a second resolution image, where the second resolution image is an image of the object to be photographed based on shooting by the digital microscope with a higher resolution than the first resolution image; determining whether an evaluation value indicated by the evaluation value-related information notified by the image receiving device is higher than a predetermined value; When it is determined that the evaluation value is higher than the predetermined value, the second resolution image is transmitted to the image receiving device. 15. An image sending device connected to an image receiving device via a communication line, The image receiving device has: at least one receiving processor; and a recording medium for non-transitory reception holding a command for reception, The receiving command causes the at least one receiving processor to perform: Obtaining a first resolution image from the image sending device and displaying it on a display device, the first resolution image is an image of the object to be photographed based on a digital microscope; receiving a magnification for the first resolution image displayed by the display device; Enlarging the first resolution image displayed by the display device based on the accepted magnification ratio; determining whether or not the received evaluation value based on the magnification factor is higher than a predetermined value, and notifying the image transmission device of a determination result, The image sending device has: at least one sending processor; and a non-transitory recording medium for transmission holding a command for transmission, The sending command causes the at least one sending processor to perform: acquiring the first resolution image and sending it to the image receiving device; Acquiring a second resolution image, where the second resolution image is an image of the object to be photographed based on shooting by the digital microscope with a higher resolution than the first resolution image; When the determination result notified from the image receiving device indicates that the evaluation value is higher than the predetermined value, the second resolution image is transmitted to the image receiving device.