CN121265297A

CN121265297A - Oral scanning method, apparatus, device, readable storage medium and program product

Info

Publication number: CN121265297A
Application number: CN202511442644.0A
Authority: CN
Inventors: 赵俊; 刘懿; 齐世霏
Original assignee: Shanghai Jiao Tong University
Current assignee: Shanghai Jiao Tong University
Priority date: 2025-10-10
Filing date: 2025-10-10
Publication date: 2026-01-06

Abstract

This application relates to an oral cavity scanning method, apparatus, device, computer-readable storage medium, and computer program product. The method includes: acquiring a target image, which is obtained by reflection from multiple mirrors, the mirrors being used to acquire images of biological tissue from different orientations; segmenting the target image to obtain sub-images corresponding to each mirror; determining a reference image from the multiple sub-images based on image information corresponding to each sub-image; converting each sub-image into a three-dimensional space matching the reference image based on the pose relationship between the mirrors, obtaining point data corresponding to each sub-image; and obtaining a three-dimensional image of the biological tissue based on the point data corresponding to each sub-image. This method can compress the acquisition time of complete oral cavity data, shorten the scanning trajectory, and improve the efficiency and accuracy of oral cavity data acquisition.

Description

Oral scanning method, apparatus, device, readable storage medium and program product

Technical Field

The present application relates to the field of image processing technology, and in particular, to an oral scanning method, apparatus, device, computer readable storage medium and computer program product.

Background

With the development of image processing technology, oral cavity dental model data can be obtained through a digital impression technology. However, in the conventional art, it is common to collect single frame data of soft and hard tissues in the oral cavity and perform multi-stitching to construct a complete oral dental model. The scanning track is complex, the time consumption of complete data acquisition is long, higher requirements are put forward on operators, and diagnosis and treatment comfort of patients is affected. And the scanning precision is insufficient under special conditions such as dentition deficiency, and the method cannot be suitable for clinical applications such as toothless jaw planting.

Disclosure of Invention

In view of the foregoing, there is a need for an oral scanning method, apparatus, oral scanning device, computer-readable storage medium, and computer program product that can compress the duration of acquisition of oral integrity data, shorten the scanning trajectory, and improve the efficiency and accuracy of oral data acquisition.

In a first aspect, the present application provides an oral scanning method comprising:

Acquiring a target image, wherein the target image is obtained by reflecting through a plurality of reflectors, and the reflectors are used for acquiring images of different orientations of the biological tissue;

Dividing the target image to obtain sub-images corresponding to the reflectors;

determining a reference image from a plurality of sub-images based on the corresponding directions of the sub-images, and converting each sub-image into a three-dimensional space matched with the reference image based on the pose relation among the reflectors to obtain point data corresponding to each sub-image;

And obtaining a three-dimensional structure diagram of the biological tissue based on the point data corresponding to each sub-image.

In one embodiment, the converting each sub-image into the three-dimensional space matched with the reference image based on the pose relation between the reflectors to obtain the point data corresponding to each sub-image includes obtaining a mirror action matrix corresponding to each reflector based on the pose relation between the reflectors, and processing the matched sub-images based on each mirror action matrix to obtain the point data of each sub-image in the three-dimensional space.

In one embodiment, before the mirror action matrix corresponding to each mirror is obtained based on the pose relation between each mirror, the method comprises the steps of determining object images of a calibration object in each direction from a plurality of sub-images, processing the object images to obtain each center point corresponding to object point clouds of the calibration object in each direction, processing other center points based on the center points corresponding to the reference images to obtain mirror normal vectors corresponding to each mirror, and obtaining the pose relation between the matched mirror and the mirror corresponding to the reference image based on the mirror normal vectors corresponding to each mirror.

In one embodiment, the obtaining the three-dimensional image of the biological tissue based on the point data corresponding to each sub-image includes obtaining a current scanning rough pose corresponding to the point data, obtaining differences between each saved scanning pose and the current scanning rough pose, determining target differences meeting target conditions from the differences, obtaining saved scanning poses corresponding to the target differences, obtaining target point data corresponding to the saved scanning poses, obtaining a current scanning refined pose corresponding to the point data based on a distance between the target point data and the current point data, wherein the precision of the current scanning refined pose is greater than that of the current scanning rough pose, and obtaining the three-dimensional image of the biological tissue based on the current scanning refined pose.

In one embodiment, the dividing the target image to obtain sub-images corresponding to the reflectors comprises obtaining posture information corresponding to a reflector group, wherein the reflector group comprises at least two reflectors, identifying edge features in the target image based on the posture information corresponding to the reflector group, and dividing the target image based on the edge features to obtain sub-images corresponding to the reflectors.

In a second aspect, the present application also provides an oral scanning device comprising:

The image scanning device comprises a plurality of reflectors, a plurality of imaging devices and a display device, wherein the reflectors are used for acquiring images of different orientations of the biological tissue;

The image processing device is communicated with the image scanning device to obtain a target image, the target image is obtained after being reflected by a plurality of reflectors included in the image scanning device, the target image is divided to obtain sub-images corresponding to the reflectors, a reference image is determined from the sub-images based on the directions corresponding to the sub-images, the sub-images are converted into a three-dimensional space matched with the reference image based on the pose relation between the reflectors, point data corresponding to the sub-images are obtained, and a three-dimensional image corresponding to the biological tissue is obtained based on the point data corresponding to the sub-images.

In one embodiment, the image scanning device comprises a scanning head and a scanning rod, the scanning rod is provided with a light source, the scanning head is connected with the scanning rod through a connecting piece, the scanning head comprises a first reflecting mirror, a second reflecting mirror and a third reflecting mirror, the first reflecting mirror and the second reflecting mirror are distributed at an obtuse angle, the first reflecting mirror and the third reflecting mirror are distributed at an obtuse angle, and the first reflecting mirror and an optical axis emitted by the light source are distributed at an acute angle.

In a third aspect, the present application also provides an oral scanning device comprising:

The acquisition module is used for acquiring target images, wherein the target images are obtained after being reflected by a plurality of reflectors, and the reflectors are used for acquiring images of different orientations of the biological tissue;

the segmentation module is used for carrying out segmentation processing on the target image to obtain sub-images corresponding to the reflectors;

The conversion module is used for determining a reference image from a plurality of sub-images based on the image information corresponding to each sub-image, and converting each sub-image into a three-dimensional space matched with the reference image based on the pose relation among the reflectors to obtain point data corresponding to each sub-image;

and the generation module is used for obtaining a three-dimensional image corresponding to the biological tissue based on the point data corresponding to each sub-image.

In a fourth aspect, the present application also provides a computer device comprising a memory and a processor, the memory storing a computer program, the processor implementing the following steps when executing the computer program:

Dividing the target image to obtain sub-images corresponding to the reflectors;

Determining a reference image from a plurality of sub-images based on image information corresponding to each sub-image, and converting each sub-image into a three-dimensional space matched with the reference image based on the pose relation among the reflectors to obtain point data corresponding to each sub-image;

and obtaining a three-dimensional image corresponding to the biological tissue based on the point data corresponding to each sub-image.

In a fifth aspect, the present application also provides a computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:

Dividing the target image to obtain sub-images corresponding to the reflectors;

In a sixth aspect, the application also provides a computer program product comprising a computer program which, when executed by a processor, performs the steps of:

Dividing the target image to obtain sub-images corresponding to the reflectors;

According to the oral cavity scanning method, the device, the oral cavity scanning equipment, the computer readable storage medium and the computer program product, the target image is obtained after being reflected by the reflectors, the reflectors are used for obtaining images of different directions of biological tissues, so that information of multiple surfaces of the biological tissues can be collected simultaneously, the scanning track and the scanning time length are shortened, the obtained target image comprises the information of the multiple surfaces of the biological tissues, the target image is divided to form a three-dimensional image of the biological tissues, sub-images corresponding to the reflectors are obtained, a reference image is determined from the sub-images based on the directions corresponding to the sub-images, the sub-images are converted into point data in a three-dimensional space matched with the reference image based on the pose relation among the sub-images, the point data corresponding to the sub-images are obtained, and the three-dimensional image corresponding to the biological tissues can be obtained based on the point data corresponding to the sub-images. Thus, the accuracy of the obtained three-dimensional image and the efficiency of scanning to obtain the three-dimensional image can be improved.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the related art, the drawings that are needed in the description of the embodiments of the present application or the related technologies will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and other related drawings may be obtained according to these drawings without inventive effort to those of ordinary skill in the art.

FIG. 1 is a diagram of an application environment of an oral scanning method in one embodiment;

FIG. 2 is a schematic diagram of an oral scanning device in one embodiment;

FIG. 3 is a flow chart of an oral scanning method in one embodiment;

FIG. 4 is a schematic diagram of the structure of a scanning head in one embodiment;

FIG. 5 is a schematic diagram of an image of a target in one embodiment;

FIG. 6 is a schematic diagram of a first optical path in one embodiment;

FIG. 7 is a schematic diagram of a second optical path and a third optical path in one embodiment;

FIG. 8 is a block diagram of an oral scanning device in one embodiment;

Fig. 9 is an internal structural diagram of the image processing apparatus in one embodiment.

Detailed Description

The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.

The oral cavity scanning method provided by the embodiment of the application can be applied to the oral cavity scanning equipment shown in figure 1. Wherein the oral scanning device comprises an image scanning device 10 and an image processing device 20. The image scanning device 10 comprises a plurality of mirrors for acquiring images of different orientations of the biological tissue. The biological tissue is, for example, oral tissue such as teeth and gums. The image processing apparatus 20 communicates with the image scanning apparatus 10 to acquire a target image. The target image is obtained after being reflected by a plurality of mirrors included in the image scanning apparatus 10. The image processing device 20 divides the target image to obtain sub-images corresponding to the reflectors, determines a reference image from the sub-images based on the directions corresponding to the sub-images, converts the sub-images into a three-dimensional space matched with the reference image based on the pose relation among the reflectors to obtain point data corresponding to the sub-images, and obtains a three-dimensional image corresponding to the biological tissue based on the point data corresponding to the sub-images. Illustratively, as shown in FIG. 2, the image scanning device 10 may include a reflector 110 and a handpiece 120. The reflecting member 110 may be constituted by a plurality of reflecting mirrors. The hand piece 120 is connected to the reflecting piece 110 through a connecting piece 121 for controlling the scanning track of the image scanning device 10. The handpiece 120 may be provided with an image acquisition device. The image acquisition device can be used for acquiring the optical signals so as to process the optical signals to obtain the target image. The image acquisition means may for example comprise a projector, a camera, a lens element, etc. The image scanning apparatus 10 may communicate with the image processing apparatus 20 through an image capturing device to acquire a target image. And then processing the target image to obtain a three-dimensional image corresponding to the target image obtained by current scanning. Along with the change of the scanning position, the acquired target image also changes correspondingly, so that an integral three-dimensional image corresponding to the biological tissue can be obtained.

Among them, the image processing device 20 may be, for example, but not limited to, various personal computers, notebook computers, smartphones, tablet computers, and portable wearable devices. The portable wearable device may be a smart watch, smart bracelet, headset, or the like. The head-mounted device may be a Virtual Reality (VR) device, an augmented Reality (Augmented Reality, AR) device, smart glasses, or the like. Alternatively, the image processing apparatus 20 may also communicate with a server through a network to process a target image through the server, and then transmit the resulting three-dimensional image to a terminal to display the generated three-dimensional image on an output device (e.g., a display screen) of the terminal. The server may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud computing service.

In an exemplary embodiment, as shown in fig. 3, an oral scanning method is provided, which is exemplified as the method applied to the image processing apparatus in fig. 1, and includes the following steps 302 to 308. Wherein:

step 302, a target image is acquired.

The target image is obtained after being reflected by a plurality of reflectors, and the reflectors are used for acquiring images of biological tissues in different directions.

The image acquisition device is illustratively provided with a light source. When biological tissue is scanned by an image scanning device, probe light is generated by a light source to propagate to a plurality of mirrors in the image scanning device. The plurality of mirrors then reflect the probe light to different surfaces of the biological tissue. For example, in the oral field, the different surfaces include occlusal, lingual, buccal surfaces of the teeth. And then the detection light is reflected by different surfaces and transmitted back to the reflector, and then the detection light is transmitted to imaging equipment (such as a camera) in the image acquisition device through the reflector, so that a target image is obtained. The image acquisition device may further comprise a communication module. The target image is transmitted to the image processing apparatus through the communication module so that the image processing apparatus obtains the target image.

Alternatively, the image processing device may also be connected to the image acquisition device. After the image capturing apparatus obtains the target image, a digital signal corresponding to the target image may be transmitted to the image processing apparatus through a connection apparatus (e.g., a data line) so that the image processing apparatus obtains the target image.

In one implementation, referring to FIG. 4, an image scanning device includes three planar mirrors. A first mirror 111 corresponding to the head window, a first mirror 112 and a third mirror 113 symmetrical to each other on both sides of the first mirror 111. The first mirror 111, the first mirror 112, and the third mirror 113 are arranged in a U-shape. The normal of the first mirror 111 is at an angle to the optical axis. And the first mirror 111 directs the probe light to be projected to the first and third mirrors 112, 113 in different directions. Illustratively, the normal of the first mirror 111 is 45 degrees from the optical axis. The probe light reflected by the first mirror 111 includes a first optical path. The first light path is perpendicular to the optical axis. That is, the first light path is projected to a first surface (e.g., occlusal surface) of the biological tissue. The normal directions of the first reflecting mirror 112 and the third reflecting mirror 113 are perpendicular to the optical axis, and the included angle between the normal directions and the central line of the U-shaped mirror is between 70 degrees and 80 degrees. Such as 70 degrees, 72 degrees, 75 degrees, 77 degrees, 80 degrees. The probe light reflected by the first reflecting mirror 111 further includes a second optical path and a third optical path. The first and third mirrors 112 and 113 project the second and third optical paths to a second surface (e.g., lingual side) and a third surface (e.g., buccal side) of the biological tissue, respectively.

Optionally, the imaging device comprises a projector, a camera, a lens element. The projector projects coded blue light or uncoded white light and other multicolor light sources to biological tissues through the head reflector group (namely the first reflector). The light reflected back from the surface of the biological tissue is captured by the camera head of the camera after being acted by the head reflecting mirror group, so that a target image comprising geometric shape and color information of the surface of the biological tissue is obtained.

The first mirror, the second mirror, and the third mirror may each be formed of a plurality of mirrors, and are not limited to one mirror.

And 304, dividing the target image to obtain sub-images corresponding to the reflectors.

Wherein the target image includes information of a plurality of surfaces of the biological tissue. It will be appreciated that each time a scan, a target image is acquired. Since the three-dimensional image to be finally generated is a three-dimensional structure requiring reduction of biological tissue, it is necessary to divide the target image to obtain sub-images corresponding to the respective mirrors. That is, by performing the segmentation process on the target image, information on each surface of the biological tissue is obtained. A stereoscopic three-dimensional image of the biological tissue may then be constructed based on the information of the various sides of the biological tissue.

Alternatively, the target image may also be subjected to the segmentation processing by acquiring an operation instruction of the user. For example, referring to a shown in fig. 5, after a target image is obtained, the target image is transmitted to a terminal corresponding to an image processing apparatus to display the target image on a display interface of the terminal. And responding to the clicking operation of the user on the target image, and obtaining the clicking position. And dividing the target image based on the click position. The number of click positions is 1 less than the number of mirrors. Optionally, the clicking position is taken as a starting point, and the dividing line is drawn towards two ends along the preset direction of the target image. The preset direction may be a direction parallel to the width of the image in some embodiments, and may be a direction parallel to the length of the image in some embodiments. The preset direction is related to the pose of the imaging device. The target image is thus divided into a plurality of sub-images by the dividing line.

And 306, determining a reference image from the plurality of sub-images based on the image information corresponding to each sub-image, and converting each sub-image into a three-dimensional space matched with the reference image based on the pose relation among the reflectors to obtain the point data corresponding to each sub-image.

And step 308, obtaining a three-dimensional image corresponding to the biological tissue based on the point data corresponding to each sub-image.

The image information corresponding to each sub-image may include azimuth information, area information, and the like. The azimuth information corresponding to each sub-image may refer to the azimuth of the surface of the biological tissue represented in the sub-image. For example, the front, left, right, etc. orientation of biological tissue. The area information corresponding to each sub-image may refer to the occupation area of the sub-image in the target image.

Optionally, if the reference image is determined by the azimuth information, determining the sub-image corresponding to the preset azimuth from the azimuth information corresponding to each sub-image as the reference image. The preset orientation is, for example, a frontal orientation (e.g., occlusal surface of teeth).

Alternatively, if the reference image is determined with area information, the sub-image corresponding to the maximum area may be determined as the reference image from the area information corresponding to each sub-image.

In this embodiment, after the reference image is determined, a reference coordinate system may be established based on the reference image, and then pixels in other sub-images may be converted to a reference coordinate system (i.e., in three-dimensional space) in which the reference image matches, based on the pose relationship between the mirrors. In this way, dot data corresponding to pixels in other sub-images in the three-dimensional space can be obtained, and a three-dimensional image corresponding to the biological tissue can be generated based on the dot data corresponding to each sub-image.

Wherein the point data may comprise spatial coordinates. In the process of establishing the reference coordinate system based on the reference image, the reference coordinate system may be established based on, for example, the center point of the reference image as the origin. Then, each pixel in the reference image can be firstly converted into a three-dimensional point in the three-dimensional space corresponding to the reference coordinate system, and then the pixel points of other sub-images are converted into the three-dimensional points in the three-dimensional space based on the pose relation among the reflectors. It will be appreciated that since the target image carries depth information, conversion of two-dimensional points in the image to three-dimensional points in three-dimensional space may be achieved.

Illustratively, the pose relationship between the mirrors may be obtained by measurement, the angles between the mirrors may be obtained by measurement, and the corresponding mirror effect matrix may be obtained based on the angles between the mirrors. Pixels in other sub-images are converted into corresponding three-dimensional spaces based on the specular interaction matrix.

Alternatively, the pose relationship between the mirrors may also be obtained by analyzing the target image. For example, a ball target is provided in the middle of the U-shaped mirror group. And determining the pose relation of each reflecting mirror based on the imaging information of the ball target in the target image.

It will be appreciated that the above procedure is a procedure corresponding to one scan. Thus, multiple scans are required to obtain a three-dimensional image of the entire biological tissue (e.g., the entire oral cavity, including multiple teeth). Therefore, after one target image is obtained after each scanning, the current target image and the target image obtained by the last scanning can be fused, so that an integral three-dimensional image with high accuracy is obtained, and the obtained integral three-dimensional image is also beneficial to interaction with a user.

In the oral cavity scanning method, the target image is obtained after being reflected by the plurality of reflectors, and the plurality of reflectors are images used for obtaining different directions of biological tissues, so that information of a plurality of surfaces of the biological tissues can be collected simultaneously, a scanning track and scanning time length are shortened, the obtained target image comprises the information of the plurality of surfaces of the biological tissues, therefore, in order to construct a three-dimensional image of the biological tissues, the target image is divided to obtain sub-images corresponding to the reflectors, a reference image is determined from the plurality of sub-images based on the directions corresponding to the sub-images, the sub-images are converted into a three-dimensional space matched with the reference image based on the pose relation among the reflectors, point data corresponding to the sub-images are obtained, and then the three-dimensional image corresponding to the biological tissues can be obtained based on the point data corresponding to the sub-images. Thus, the accuracy of the obtained three-dimensional image and the efficiency of scanning to obtain the three-dimensional image can be improved.

In an exemplary embodiment, since the imaging in the target image is based on the probe light projected by each mirror, in order to be able to restore the true spatial structure of the biological tissue, processing can be performed based on the specular matrix of each mirror. The method for obtaining the point data corresponding to each sub-image comprises the steps of obtaining a mirror action matrix corresponding to each mirror based on the pose relation among the mirrors, and processing the matched sub-images based on the mirror action matrices to obtain the point data of each sub-image in the three-dimensional space.

Wherein the specular interaction matrix (Reflection Matrix) is a matrix representation describing the "specular reflection" transformation in space. Specular reflection is a transformation that maps an object symmetrically about a certain mirror plane (straight in two dimensions, planar in three dimensions).

For example, the pose relationship of the mirror may be represented based on the mirror surface position and the normal vector corresponding to the mirror. Since the data of each sub-image is to be converted into the three-dimensional space corresponding to the reference image, the specular action matrix corresponding to the second mirror can be determined based on the pose relationship between the first mirror and the second mirror. And determining a mirror action matrix corresponding to the third mirror based on the pose relationship between the first mirror and the third mirror. The matched sub-images are processed based on the corresponding specular function matrix, so that the pixels in the sub-images can be correctly converted into the three-dimensional space corresponding to the reference image.

Wherein the mirror effect matrixCan be obtained from the following formula:

Wherein, the Representing a matrix of specular reflection,Can be measured from specular normal vectorWhen the calculated vector is applied to the space, the default vector start point is the mirror position. To obtainTranslation can be performed first, so that the vector starting point is positioned at the mirror surface position, and the vector is reversely translated and reset after the action of the reflection matrix.AndThe expression of (2) is as follows:

For the mirror surface position and normal vector corresponding to the reflecting mirror, besides the measurement by manual work, the determination can be performed by the ball target. Illustratively, a calibration object is provided at a designated location of the plurality of mirrors. Correspondingly, an object image corresponding to the calibration object is also included in the target image. In each sub-image, the sub-object images of the calibration object are not consistent, refer to a and b in fig. 5, and b in fig. 5 shows mask images corresponding to different sub-object images. In practice, however, the position of the calibration object is determined. Therefore, the pose information of the corresponding reflecting mirrors can be obtained by analyzing the images of the sub-objects, so that the pose relation among the reflecting mirrors can be obtained. Specifically, pixels in each sub-object image may be converted into a three-dimensional space, resulting in a corresponding point cloud. And respectively carrying out point cloud processing on each point cloud to obtain a center point corresponding to each point cloud. The step of performing the point cloud processing on each point cloud may include performing spherical fitting on each point cloud to obtain a center of sphere corresponding to each point cloud, so as to obtain a center point corresponding to each point cloud. If the calibration object is a sphere (e.g., a sphere target), the center point may be the center of the sphere. And then, based on the relation between the center points corresponding to the point clouds, the pose relation between the reflectors is obtained. For example, to obtain a specular action matrix corresponding to the second mirror, a first center point corresponding to the first mirror is processed with a second center point corresponding to the second mirror. Specifically, the first center point and the second center point are connected to obtain a second connecting line. And determining the mirror surface position corresponding to the second reflecting mirror based on the midpoint of the second connecting line. And determining a second normal corresponding to the second reflector based on the second connecting line. Wherein the direction of the second normal is directed from the second center point to the first center point. Correspondingly, in order to obtain a mirror action matrix corresponding to the third mirror, the first center point corresponding to the first mirror and the third center point corresponding to the third mirror are processed. And connecting the first center point with the third center point to obtain a third connecting line. And determining the mirror surface position corresponding to the third mirror based on the midpoint of the third connecting line. And determining a third normal corresponding to the third reflector based on the third connecting line. Wherein the direction of the third normal line is directed from the third center point to the first center point.

And then, processing the matched sub-images based on the mirror action matrix to obtain point data of each sub-image in the three-dimensional space. The step of processing the matched sub-image based on the specular interaction matrix may include fusing the specular interaction matrix with each pixel in the matched sub-image (e.g., multiplying the specular interaction matrix by a pixel value of each pixel) to obtain three-dimensional data corresponding to each pixel, thereby obtaining point data of each sub-image in three-dimensional space.

In this embodiment, the mirror action matrix corresponding to each mirror is obtained based on the pose relationship between each mirror, and the matched sub-images are processed based on each mirror action matrix to obtain the point data of each sub-image in the three-dimensional space, so that each point in the target image can be accurately converted into a three-dimensional point in the three-dimensional space, which is beneficial to follow-up three-dimensional construction and more intuitively understanding of the three-dimensional structure of the biological tissue.

As previously mentioned, the pose relationship between the mirrors may be determined based on the object images corresponding to the calibration objects in the respective sub-images. The method comprises the steps of determining object images of the calibration object in all directions from a plurality of sub-images, processing the object images to obtain all center points corresponding to object point clouds in all directions of the calibration object, processing other center points based on the center points corresponding to the reference image to obtain mirror normal vectors corresponding to all the mirrors, and obtaining the pose relationship between the matched mirrors and the mirrors corresponding to the reference image based on the mirror normal vectors corresponding to all the mirrors.

Wherein the calibration object may be referred to as a and c in fig. 5. The object image is part of a sub-image. Illustratively, the calibration object in the sub-images may be identified by a target detection algorithm to obtain an object image corresponding to the calibration object in each sub-image. And processing the object image, wherein the object image carries depth information, so that the space coordinates corresponding to each pixel point in the object image can be obtained.

It will be appreciated that in real space, the position of the calibration object corresponding to each mirror should be the same. Therefore, after the spatial points corresponding to the pixel points in each object image are obtained, the spatial points may be processed (for example, spherical fitting is performed on the spatial points) to obtain the center points corresponding to each object image. Through the center points corresponding to the object images, the pose relation among the reflectors can be obtained.

For example, if the number of the reflectors is three, the pose relationship between the second reflector and the first reflector may be obtained based on the first center point corresponding to the reflector and the second center point corresponding to the second reflector. And obtaining the pose relationship between the third reflector and the first reflection channel based on the first center point corresponding to the first reflector and the third center point corresponding to the third reflector. Wherein, the pose relation between the first mirror and the second mirror, the pose relation between the first mirror and the third mirror can be represented based on the mirror normal vector and the mirror position of the second mirror, the third mirror, respectively. In particular, for the second mirror, the specular position may be determined based on a midpoint of a line between the first center point and the second center point. The specular normal vector may be determined based on a vector in which the second center point points to the first center point. For the third mirror, the specular position may be determined based on a midpoint of a line between the first center point and the third center point. The specular normal vector may be determined based on a vector in which the third center point points to the first center point.

In this embodiment, the pose relationship between the reflectors is determined by calibrating the object, so that the sub-images corresponding to the reflectors can be registered into a unified three-dimensional space, which is beneficial to the three-dimensional imaging of the subsequent biological tissues. The method can also omit the step of manually calibrating, improve the calibration efficiency and reduce the adverse effect caused by manual errors. And the calibrated result (such as a mirror action matrix) can be reused after one calibration, so that the efficiency of subsequent modeling is improved.

Through the above steps, a corresponding three-dimensional image can be obtained once per scan. However, in some cases, it may be impossible to scan the entire tissue of the biological tissue through one scan, and multiple scans of the entire tissue are required to obtain an entire three-dimensional image. Therefore, the result after each scan needs to be processed to accurately obtain the final three-dimensional image. The three-dimensional image of the biological tissue is obtained by obtaining a current scanning rough pose corresponding to the current point data, obtaining differences between each saved scanning pose and the current scanning pose, determining target differences meeting target conditions from the differences, obtaining saved scanning poses corresponding to the target differences, obtaining target point data corresponding to the saved scanning poses, obtaining a current scanning refined pose corresponding to the current point data based on the distance between the target point data and the current point data, wherein the precision of the current scanning refined pose is greater than that of the current scanning rough pose, and obtaining the three-dimensional image of the biological tissue based on the current scanning refined pose.

For example, the current scanning coarse pose may be measured by a motion sensor on the image scanning device. In the moving process of the image scanning device, the biological tissue can be scanned for a plurality of times, and a plurality of scanning positions of the biological tissue can be obtained. After a three-dimensional image of the biological tissue is obtained, the corresponding scan pose is saved to a storage space. It can be understood that the scanning pose stored in the storage space is a scanning refined pose. The precision of scanning the fine pose is greater than the precision of scanning the coarse pose.

The current scanning accurate pose corresponding to the current scanning coarse pose can be obtained based on each saved scanning pose saved in the storage space. And comparing the current scanning coarse pose with each stored scanning pose in consideration of the proportional relation between the similarity of the scanning poses corresponding to the image scanning equipment and the coincidence degree of the scanned biological tissues, and optimizing the current scanning coarse pose based on the point data corresponding to the stored coarse pose with the highest similarity of the current scanning coarse pose. That is, each saved scanning pose is traversed, each saved scanning pose is compared with the current scanning coarse pose, the difference between each saved scanning pose and the current scanning coarse pose is obtained, and the target difference meeting the target condition is determined from a plurality of differences. Wherein, the target gap is smaller than other gaps. Thus, the saved scanning pose corresponding to the target gap can be obtained, and the target point data corresponding to the saved scanning pose can be obtained.

And then, the current scanning precision pose corresponding to the current point data can be obtained based on the distance between the target point data and the current point data corresponding to the current scanning coarse pose. For each current point in the current point data, the closest target point to the current point is determined from the target point data, and the distance between the target point and the current point is calculated, so that the distance between each current point and the matched target point can be obtained. And then optimizing the current scanning coarse pose based on the distance between each current point and the matched target point to obtain the current scanning fine pose.

Finally, a three-dimensional image of the biological tissue can be obtained based on the current scanning refined pose. For example, the current point data may be integrated into a three-dimensional model represented by a truncated symbol distance function or the like based on the current scan refined pose. The three-dimensional model may also be a novel representation based on neural radiation fields or gaussian splatter, etc.

It can be appreciated that in addition to the method of optimizing the current scanning coarse pose based on the differences between each saved scanning pose and the current scanning coarse pose, deep learning, artificial intelligence models can be trained to obtain the current corresponding scanning fine pose.

In this embodiment, the current pose is optimized by the scanning pose obtained in the past scanning process, so that the reconstructed three-dimensional point data can be combined to optimize the current point data, and further, the accuracy of the current point data is improved, and the accuracy of the finally obtained three-dimensional image are also improved.

In some embodiments, the step of dividing the target image to obtain sub-images corresponding to the respective mirrors includes obtaining pose information corresponding to a mirror group including at least two mirrors, identifying edge features in the target image based on the pose information corresponding to the mirror group, and dividing the target image based on the edge features to obtain sub-images corresponding to the respective mirrors.

Wherein the attitude information is the position and orientation parameters of each mirror in space, such as the tilt angle, deflection direction of the mirror.

Illustratively, edge features correspond to areas in the image where grey scale, brightness or texture is abrupt (typically to the boundary of an object, such as the physical edge of a mirror, the interface of imaging areas of different mirrors, etc.). The pose information of the mirror provides geometric constraints for edge recognition. Since the orientation and position of the mirror are known, the boundary morphology (e.g., approximate position, direction, curvature, etc. of the edge) of its corresponding projection region in the target image can be predicted by the principle of optical projection. For example, if a mirror is in an inclined position, its projected edge in the target image may exhibit a straight line or a broken line of a certain angle. The edge searching range can be reduced through the gesture information, or the edges conforming to the prediction direction (excluding irrelevant noise edges) can be preferentially detected, so that the edge characteristics (rather than the background or interference edges) related to the reflecting mirror can be more accurately identified.

And then the identified edge is taken as a boundary, and the connected area surrounded by the edge is divided into independent sub-images through algorithms such as area growth, threshold segmentation or morphological operation. For example, the imaging areas of two adjacent mirrors are separated by an edge along which the image is cut to obtain sub-images corresponding to the two mirrors, respectively.

In this embodiment, by automatically identifying edge features in the target image, automatic segmentation of the target image and image segmentation intellectualization can be achieved.

Based on the same inventive concept, the embodiment of the application also provides an oral cavity scanning device for realizing the above-mentioned oral cavity scanning method. Wherein the oral scanning device comprises an image scanning device and an image processing device. Wherein the image scanning device comprises a plurality of mirrors. A plurality of mirrors are used to acquire images of different orientations of biological tissue. The image processing device is communicated with the image scanning device to obtain a target image, the target image is obtained after being reflected by a plurality of reflectors included in the image scanning device, the target image is divided to obtain sub-images corresponding to the reflectors, a reference image is determined from the sub-images based on the directions corresponding to the sub-images, the sub-images are converted into a three-dimensional space matched with the reference image based on the pose relation among the reflectors to obtain point data corresponding to the sub-images, and the three-dimensional image corresponding to the biological tissue is obtained based on the point data corresponding to the sub-images.

In this embodiment, the oral treatment device acquires the target image, because the target image is obtained after being reflected by a plurality of reflectors, and the plurality of reflectors are images for acquiring different orientations of the biological tissue, the information of a plurality of surfaces of the biological tissue can be acquired simultaneously, the scanning track and the scanning time period are shortened, and because the acquired target image comprises the information of the plurality of surfaces of the biological tissue, in order to construct a three-dimensional image of the biological tissue, dividing the target image to obtain sub-images corresponding to the reflectors, determining a reference image from the sub-images based on the directions corresponding to the sub-images, converting the sub-images into a three-dimensional space matched with the reference image based on the pose relation among the reflectors to obtain point data corresponding to the sub-images, and obtaining a three-dimensional image corresponding to the biological tissue based on the point data corresponding to the sub-images. Thus, the accuracy of the obtained three-dimensional image and the efficiency of scanning to obtain the three-dimensional image can be improved.

In some exemplary embodiments, referring to fig. 2, an image scanning device includes a scanning head and a scanning shaft. Wherein the scanning rod part is provided with a light source which is used for emitting detection light to the reflecting mirror. The scanning rod part is also provided with imaging equipment. The imaging device is used to collect the reflected light to form a target image. The scanning head and the scanning rod are connected by a connecting member 121. It will be appreciated that the connector 121 may be part of the scanning head, part of the scanning rod, or a separate element. Therefore, the scanning rod part and the scanning head part can be detachable, and the cleaning and the disinfection are convenient. Illustratively, the image scanning device includes three mirrors. The three reflectors may be arranged in a U-shape. The angle between the normal line of the first reflector and the optical axis is an acute angle, and the second reflector and the third reflector are distributed on two sides of the first reflector. The first reflecting mirror and the second reflecting mirror are distributed at an obtuse angle, and the first reflecting mirror and the third reflecting mirror are distributed at an obtuse angle.

Specifically, fig. 6 shows a schematic view of the first mirror 211, the stem imaging system 220, and the first virtual imaging system 221 at a third viewing angle. The Z-axis and Y-axis directions of the defined world coordinate system are marked in the figure. The normal line of the first reflecting mirror 211 is on the ZY plane, and the included angle between the normal line and the optical axis of the imaging system is 40-50 degrees. Optionally, the included angle is 45 degrees. The probe light of the imaging system is reflected in a direction perpendicular to the original optical axis, wherein the optical path from the imaging system, after having been acted on by the first mirror 211, is directly projected onto the first surface of the biological tissue is referred to as a first optical path 301. The first virtual imaging system 221 corresponds to the first optical path 301, and the data acquired through the first optical path 301 may be understood as being directly acquired by the first virtual imaging system 221.

Fig. 7 shows the distribution of the first mirror, the virtual imaging system and the biological tissue at a fourth viewing angle. The X-axis and Z-axis of world coordinates are marked in the figure. A portion of the light projected to the first mirror 211 is projected to the second mirror 212 and the third mirror 213, and is projected to the second and third surfaces of the biological tissue 300 after being reflected a second time. The two optical paths are called a second optical path 302 and a third optical path 303, and correspond to the second virtual imaging system 222 and the third virtual imaging system 223, respectively. The imaging obtained with the three mirrors can be seen with reference to fig. 5.

Wherein the normals of the second and third mirrors are distributed in the XZ plane. The normal direction of the second reflecting mirror and the third reflecting mirror is at an angle of 70-80 degrees with the Z axis. This makes it possible to maximize the use of the structural space of the image scanning apparatus.

Optionally, the length of the first reflecting mirror is 21-26 mm. For example, the length of the first mirror may be 21 millimeters, 23 millimeters, 26 millimeters, and so on. The lengths of the second reflecting mirror and the third reflecting mirror are 17-21 mm. For example, 17 mm, 19 mm, 21 mm, etc. The width of the three reflectors is 14-17 mm. For example, the three mirrors have a width of 15 mm or 16 mm. The width of the image scanning device is between 23 mm and 32 mm. For example the width of the image scanning device is 24 mm or 26 mm or 30 mm. The image scanning device can be less than three times the thickness of the thickest molars of the human body, and the width of the scanning device can be detected in the oral cavity, so that the scanning device can repeatedly move to acquire three-way data.

In this embodiment, the scanning head adopts detachable design, is convenient for clean disinfection, and need not to integrate a plurality of cameras and projecting apparatus at the head, has effectively reduced scanning head size, has improved the implementation feasibility and the clinical suitability of system.

In some other embodiments, the scanning head of the oral scanning device may include two mirrors. Illustratively, the normal direction of the first mirror is at a 45 degree angle to the positive X-axis direction, the positive Y-axis direction, and the negative Z-axis direction. The second reflecting mirror is symmetrically distributed with respect to the ZY plane. Thus, the oral cavity scanning device can simultaneously capture information of two surfaces of an object, so that the scanning track can be reduced.

It should be understood that, although the steps in the flowcharts related to the above embodiments are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.

Based on the same inventive concept, the embodiment of the application also provides an oral cavity scanning device for realizing the above-mentioned oral cavity scanning method. The implementation of the solution provided by the device is similar to that described in the above method, so the specific limitations of one or more embodiments of the oral scanning device provided below may be referred to above for limitations of the oral scanning method, and will not be repeated here.

In one exemplary embodiment, as shown in FIG. 8, an oral scanning device 800 is provided, comprising an acquisition module 801, a segmentation module 802, a conversion module 803, and a generation module 804, wherein:

The acquiring module 801 is configured to acquire a target image, where the target image is obtained by reflecting the target image by a plurality of mirrors, and the plurality of mirrors are configured to acquire images of different orientations of the biological tissue.

The segmentation module 802 is configured to perform segmentation processing on the target image, so as to obtain sub-images corresponding to the respective mirrors.

And a conversion module 803, configured to determine a reference image from the multiple sub-images based on the image information corresponding to the sub-images, and convert each sub-image into a three-dimensional space matched with the reference image based on the pose relationship between the mirrors, so as to obtain point data corresponding to each sub-image.

The generating module 804 is configured to obtain a three-dimensional image corresponding to the biological tissue based on the point data corresponding to each sub-image.

In some embodiments, in converting each sub-image into a three-dimensional space matched with the reference image based on the pose relationship between the mirrors to obtain point data corresponding to each sub-image, the conversion module 803 is further configured to obtain a specular action matrix corresponding to each mirror based on the pose relationship between the mirrors, and process the matched sub-image based on each specular action matrix to obtain point data of each sub-image in the three-dimensional space.

In some embodiments, before obtaining the mirror effect matrix corresponding to each mirror based on the pose relationship between each mirror, the oral scanning device 800 is further configured to determine an object image of the calibration object in each azimuth from the plurality of sub-images, process the object image to obtain each center point corresponding to the object point cloud of the calibration object in each azimuth, process other center points based on the center points corresponding to the reference image to obtain the mirror normal vector corresponding to each mirror, and obtain the pose relationship between the matched mirror and the mirror corresponding to the reference image based on the mirror normal vector corresponding to each mirror.

In some embodiments, in terms of obtaining a three-dimensional image of the biological tissue based on the point data corresponding to each sub-image, the generating module 804 is further configured to obtain a current scanning coarse pose corresponding to the current point data, obtain differences between each saved scanning pose and the current scanning coarse pose, determine target differences meeting target conditions from the differences, obtain saved scanning poses corresponding to the target differences, obtain target point data corresponding to the saved scanning poses, obtain a current scanning refined pose corresponding to the current point data based on a distance between the target point data and the current point data, and obtain a three-dimensional image of the biological tissue based on the current scanning refined pose, wherein the precision of the current scanning refined pose is greater than that of the current scanning coarse pose.

In some embodiments, in terms of segmenting the target image to obtain sub-images corresponding to the respective mirrors, the segmentation module 802 is further configured to obtain pose information corresponding to a mirror group, where the mirror group includes at least two mirrors, identify edge features in the target image based on the pose information corresponding to the mirror group, and segment the target image based on the edge features to obtain sub-images corresponding to the respective mirrors.

The various modules in the oral scanning device described above may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.

In one exemplary embodiment, an oral scanning device is provided that includes an image processing device. The image processing apparatus may be a terminal, and an internal structure thereof may be as shown in fig. 9. The computer device includes a processor, a memory, an input/output interface, a communication interface, a display unit, and an input means. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The input/output interface of the computer device is used to exchange information between the processor and the external device. The Communication interface of the computer device is used for conducting wired or wireless Communication with an external terminal, and the wireless Communication can be realized through WIFI, a mobile cellular network, near field Communication (NEAR FIELD Communication) or other technologies. The computer program is executed by a processor to implement an oral scanning method. The display unit of the computer device is used for forming a visual picture, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, can also be a key, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.

It will be appreciated by persons skilled in the art that the architecture shown in fig. 9 is merely a block diagram of some of the architecture relevant to the present inventive arrangements and is not limiting as to the computer device to which the present inventive arrangements are applicable, and that a particular computer device may include more or fewer components than shown, or may combine some of the components, or have a different arrangement of components.

In an exemplary embodiment, a computer device is provided, comprising a memory and a processor, the memory having stored therein a computer program, the processor performing the steps of the method embodiments described above when the computer program is executed.

In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when executed by a processor, implements the steps of the method embodiments described above.

In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the method embodiments described above.

It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are both information and data authorized by the user or sufficiently authorized by each party, and the collection, use and processing of the related data are required to meet the related regulations.

Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in embodiments provided herein may include at least one of non-volatile memory and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (RESISTIVE RANDOM ACCESS MEMORY, reRAM), magneto-resistive Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (PHASE CHANGE Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in various forms such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The databases referred to in the embodiments provided herein may include at least one of a relational database and a non-relational database. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processor referred to in the embodiments provided in the present application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computation, an artificial intelligence (ARTIFICIAL INTELLIGENCE, AI) processor, or the like, but is not limited thereto.

The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the present application.

The foregoing examples illustrate only a few embodiments of the application and are described in detail herein without thereby limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of the application should be assessed as that of the appended claims.

Claims

1. A method of oral scanning, the method comprising:

Acquiring a target image, wherein the target image is obtained by reflecting through a plurality of reflectors, and the reflectors are used for acquiring images of biological tissues in different directions;

Dividing the target image to obtain sub-images corresponding to the reflectors;

2. The method according to claim 1, wherein the converting each sub-image into the three-dimensional space matched with the reference image based on the pose relationship between the mirrors to obtain the point data corresponding to each sub-image includes:

Obtaining a mirror surface action matrix corresponding to each reflecting mirror based on the pose relation among the reflecting mirrors;

And processing the matched sub-images based on the mirror effect matrixes to obtain point data of the sub-images in the three-dimensional space.

3. The method according to claim 2, wherein before the obtaining the mirror effect matrix corresponding to each of the mirrors based on the pose relationship between each of the mirrors, the method comprises:

Determining object images of the calibration object in all directions from a plurality of sub-images;

Processing the object image to obtain each center point corresponding to the object point cloud of the calibration object in each azimuth;

processing other center points based on the center points corresponding to the reference images to obtain mirror normal vectors corresponding to the reflectors;

And obtaining the pose relation between the matched reflecting mirror and the reflecting mirror corresponding to the reference image based on the mirror normal vector corresponding to each reflecting mirror.

4. The method according to claim 1, wherein the obtaining a three-dimensional image of the biological tissue based on the point data corresponding to each of the sub-images includes:

Acquiring a current scanning rough pose corresponding to the point data;

obtaining differences between each saved scanning pose and the current scanning coarse pose, and determining a target difference meeting target conditions from a plurality of differences;

Obtaining a saved scanning pose corresponding to the target gap, and obtaining target point data corresponding to the saved scanning pose;

obtaining a current scanning fine pose corresponding to the current point data based on the distance between the target point data and the current point data, wherein the precision of the current scanning fine pose is greater than that of the current scanning coarse pose;

And obtaining a three-dimensional image of the biological tissue based on the current scanning precision pose.

5. The method according to claim 1, wherein the dividing the target image to obtain sub-images corresponding to the respective mirrors includes:

Acquiring posture information corresponding to a reflector group, wherein the reflector group comprises at least two reflectors;

Identifying edge features in the target image based on the gesture information corresponding to the reflector group;

And dividing the target image based on the edge features to obtain sub-images corresponding to the reflectors.

6. An oral scanning device, comprising:

The image scanning device comprises a plurality of reflectors, a plurality of imaging devices and a display device, wherein the reflectors are used for acquiring images of biological tissues in different directions;

7. The apparatus of claim 6, wherein the image scanning apparatus comprises a scanning head and a scanning rod, the scanning rod is provided with a light source, the scanning head and the scanning rod are connected by a connecting piece, the scanning head comprises a first reflector, a second reflector and a third reflector, the first reflector and the second reflector are distributed at an obtuse angle, the first reflector and the third reflector are distributed at an obtuse angle, and the first reflector and an optical axis emitted by the light source are distributed at an acute angle.

8. An oral scanning device, the device comprising:

The acquisition module is used for acquiring target images, wherein the target images are obtained after being reflected by a plurality of reflectors, and the reflectors are used for acquiring images of biological tissues in different directions;

9. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 6.