CN110659637A - Electric energy meter number and label automatic identification method combining deep neural network and SIFT features - Google Patents

Abstract

The invention discloses an automatic identification method of electric energy representation number and label combined with deep neural network and SIFT features, comprising the following steps: step 1, selecting templates representing different types of electric energy meters; Carry out SIFT feature point matching to determine the type of electric energy meter of the picture to be detected; Step 3, locate the display area and Chinese character label area on the picture to be detected, and segment the display area to obtain a single digital area; Step 4, Filter out a single number area with digital display and a Chinese character label area with Chinese character display, and perform Chinese character label recognition and single number recognition. The method for automatic identification of electric energy indication numbers and labels combined with deep neural network and SIFT features provided by the invention can accurately locate and extract the screen positioning of electric energy meters, and at the same time improve the accuracy of indication numbers and label identification.

Description

An electrical energy representation number and label automation combining deep neural network and SIFT features recognition methods

technical field

The invention belongs to the technical field of automatic identification of electric energy representation numbers and labels, in particular to a method for automatic identification of electric energy representation numbers and labels combining deep neural network and SIFT features.

Background technique

With the rapid development of my country's power industry, fast and accurate acquisition of meter information has become an inevitable requirement for intelligent power management. However, the current electric energy representation and its label acquisition method are still dominated by manual recording, which cannot monitor the power consumption in time, and there is a lot of human waste and wrong transcription. Using image recognition technology to process and identify the image of the electric energy meter can realize the regular monitoring of electric energy meter data, reduce labor costs and improve the accuracy of meter reading, which is of great significance for the automation of electric energy meter management.

The automatic identification of electric energy representation number and label can be divided into two sub-tasks: localization and identification of the target area. The most common methods for accomplishing the above two tasks are mainly divided into two categories: neural network algorithms and non-neural network algorithms.

In the localization of the target area of the electric energy meter, in recent years, the neural network algorithm has continuously made breakthroughs in the target detection task of the image. However, the target detection algorithm based on neural network often cannot locate the position where the label appears. The reason is that the label area is too small relative to the entire image, and it is difficult for the deep neural network algorithm to extract the accurate features of small target objects, so the detection of small target objects is often missed, and it is not competent for the label positioning task in the electric energy meter.

In addition, the labels contained in the screen of different types of electric energy meters are not the same, among which the labels of "active", "reactive", "combined", "level", "valley" and "peak" are included in the high-voltage electric energy meter. , "Sharp", "Total", eight kinds of labels, the low-voltage electric energy meter contains five kinds of labels: "Flat", "Valley", "Peak", "Sharp", "Total". And in different models of electric energy meters, the same label appears in different positions, which greatly increases the difficulty of label positioning. The overall area of the electric energy meter is large, but the area of the numbers and labels on the screen accounts for a small proportion. If you directly use deep learning for prediction, it is difficult to ensure the detection accuracy. In addition, if the native data of the electric energy meter is directly used as the training set to fit the neural network, the source of the training set is also a problem that needs to be solved.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide an automatic identification method of electric energy representation number and label combining deep neural network and SIFT features, to perform electric energy meter type matching and to locate the screen area through SIFT features, and to use deep neural network to display the data in the electric energy meter screen. It can accurately locate the screen positioning of the electric energy meter, and at the same time improve the accuracy of the display and label identification.

The technical scheme provided by the present invention is:

An automatic identification method of electric energy representation number and label combining deep neural network and SIFT features, comprising the following steps:

Step 1. Select templates representing different types of electric energy meters;

Step 2: Perform SIFT feature point matching on the picture to be detected and the template to determine the type of electric energy meter of the picture to be detected;

Step 3, positioning the indication area and the Chinese character label area on the picture to be detected, and segmenting the indication area to obtain a single digital area;

Step 4: Screen out a single number area with digital display and a Chinese character label area with Chinese character display, and perform Chinese character label recognition and single number recognition.

Preferably, in the second step, after the SIFT feature point matching is performed between the to-be-detected picture and the template, the following steps are further included:

Step 1, calculate the SIFT feature vector of the picture to be detected, and calculate the Euclidean distance between the matching feature vector and the template, and filter out the matching pair of feature points that exceed the set threshold to obtain a preliminary screening The feature point matching pairs of ;

Step 2, inputting the initially screened feature point matching pairs into the surrogate model, and performing further screening to obtain the final feature point matching pairs; and determining the number of matches with the SIFT feature points of the picture to be detected according to the final feature point matching pairs the most templates;

Among them, the electric energy meter type of the template with the largest number of matching SIFT feature points of the picture to be detected is determined as the electric energy meter type of the picture to be detected.

Preferably, in the step 2, the surrogate model is a RANSAC model, and the final feature point matching pair is obtained, including the following steps:

Step a. Randomly extract 4 sample data from the data set composed of the final feature point matching pairs, and calculate the transformation matrix H, which is recorded as the model M;

Step b, calculate the projection error of all data in the described data set and the model M, if the error is less than the threshold, add the inner point set I;

Wherein, if the number of elements in the current interior point set I is greater than the optimal interior point set I_best, then update I_best=I, and update the number of iterations at the same time;

If the number of iterations is greater than k, exit; otherwise, increase the number of iterations by 1, and repeat the above steps;

Among them, the number of iterations k is:

Among them, p is the confidence; w is the ratio of the number of inliers to the number of data points; m is the minimum number of samples required to calculate the model.

Preferably, in the third step, positioning the indication area and the Chinese character label area on the picture to be detected, including the following steps:

Step A, according to the mapping relationship between the picture to be detected and the template of the corresponding electric energy meter type, obtain the pre-judgment screen area;

Step B, performing edge detection on the pre-judgment screen area, obtaining four edge straight lines of the liquid crystal screen, calculating the positions of the four vertices of the screen area, and using Hough transform for correction to obtain a standard liquid crystal screen area;

Step C: According to the type of the electric energy meter, determine the position where the display area and the Chinese character label area on the picture to be detected appear on the screen, and intercept the display area and the Chinese character label area.

Preferably, in the third step, a single digital area is obtained by dividing the display area in equal proportions.

Preferably, in the step 4, whether a single number area and a Chinese character label area are displayed with numbers or Chinese characters is determined by a two-class perceptron neural network, and a single number area with a number display and a Chinese character label with a Chinese character display are screened out. area.

Preferably, in the fourth step, the single digital area with digital display is identified through the lenet-5 neural network to obtain the electrical energy representation number.

The beneficial effects of the present invention are:

(1) The automatic identification method of electric energy representation number and label combining deep neural network and SIFT features provided by the present invention can realize the monitoring of electric energy meter data, and solve the problems of wrong copying and high cost in the manual meter reading process of electric energy meters.

(2) The method for automatic identification of electric energy representation number and label combining deep neural network and SIFT features provided by the present invention, through SIFT feature to match electric energy meter type and locate the screen area, and use deep neural network to compare the indications and labels on the electric energy meter screen. Label identification; it can accurately locate the screen positioning of the electric energy meter, and at the same time improve the accuracy of indication and label identification.

Description of drawings

FIG. 1 is a flow chart of the method for automatic identification of electric energy indication numbers and labels according to the present invention.

FIG. 2 is a schematic structural diagram of a two-class perceptron network structure according to the present invention.

FIG. 3 is a schematic diagram of the structure of the Lenet-5 neural network according to the present invention.

Detailed ways

The present invention will be further described in detail below with reference to the accompanying drawings, so that those skilled in the art can implement it with reference to the description.

As shown in Fig. 1, the present invention provides an automatic identification method of electric energy representation number and label combining deep neural network and SIFT feature, using SIFT feature to match electric energy meter type, and according to the projection relationship between the feature points, the approximate and use edge detection and Hough transform to correct the screen area of the electric energy meter, and finally use the deep neural network to identify the indications and labels in the electric energy meter screen; it specifically includes three functional modules: electric energy meter Category matching module, localization module and identification module.

(1) Electric energy meter category matching module

The electric energy meter type matching module is used to identify the type of the electric energy meter (such as a high-voltage or low-voltage electric energy meter). The module uses SIFT feature as the basic matching feature, and uses the RANSAC algorithm to complete the matching and identification process of the electric energy meter category.

The first step is to calculate the SIFT characteristics of each type of template of the electric energy meter;

Take the manually selected electric energy meter with good lighting, clean dial and complete shape as the template of the corresponding electric energy meter type, and then convert it into a 960*720 standard electric energy meter template through perspective transformation according to the four top corner positions of the electric energy meter pictures; calculate and store the SIFT feature points of these templates for matching with the features of the pictures to be detected.

The second step is to perform SIFT feature point matching between the image to be detected and each template image;

Calculate the SIFT features of the image to be detected and match with the SIFT features in each template image. In the SIFT feature variable, the position, scale and direction information of the feature points have been obtained, and a feature descriptor that does not change with the change of factors such as illumination and viewing angle is established for each feature point. In the matching process, according to the features and feature descriptors in the SIFT feature points, the pixel gradient is calculated for the 4*4 matrix grid near each feature point, so as to construct a 128-dimensional (4*4 matrix grid*8 gradient in each direction) feature vector. Calculate the Euclidean distance between the feature points of the image to be detected and the feature points of the template, and filter out the matching pairs that exceed a certain threshold.

The third step is to use the RANSAC algorithm to determine the final matching feature points;

The feature point pair filtered in the previous step is used as the input of the RANSAC algorithm, and the final feature point pair screening is performed. The RANSAC algorithm is an algorithm for estimating the parameters of a mathematical model in an iterative manner, which can estimate the parameters with high accuracy in a dataset containing a lot of noise. The main steps are as follows:

a. Randomly extract 4 sample data from the data set (the 4 samples cannot be collinear), calculate the transformation matrix H according to equation (1), and denote it as model M.

Among them, (x, y) represents the position of the corner of the target image, (x', y') is the position of the corner of the scene image, and s is the scale parameter. Usually h ₃₃ =1 is used to normalize the matrix, and the transformation matrix has 8 unknown parameters, so at least 8 linear equations are required to solve. Corresponding to the point position information, a set of point pairs can list two equations, and at least 4 sets of matching point pairs are included.

b. Calculate the projection error of all data in the dataset and the model M, if the error is less than the threshold, add the interior point set I;

c. If the number of elements in the current interior point set I is greater than the optimal interior point set I_best, then update I_best=I, and update the number of iterations at the same time;

d. If the number of iterations is greater than k, exit; otherwise, increase the number of iterations by 1, and repeat the above steps;

Among them, the number of iterations k is calculated according to the following formula:

Among them, p is the confidence level, generally 0.995; w is the ratio of the number of interior points to the number of data points; m is the minimum number of samples required to calculate the model.

After RANSAC screening, the matching accuracy of SIFT features can be effectively improved. However, because the RANSAC method is a random sampling process, its screening speed is relatively slow. Therefore, before using the RANSAC method, the Euclidean distance is used in advance to perform preliminary filtering on the features to improve the matching speed.

The fourth step is to compare the feature matching degree between the image to be detected and the template image;

After the third step is completed, the electric energy meter type of the template picture with the largest number of matching SIFT feature points of the picture to be detected is selected from the template pictures as the electric energy meter type of the picture to be detected. Among them, if there are multiple template pictures and the same picture to be detected with the same number of feature points successfully matched, the template electric energy meter type with the largest number of feature points matched without screening is used as the electric energy meter type of the picture to be detected.

(2) Electric energy representation number and Chinese character label positioning module

When obtaining the category of the electric energy meter in the picture to be detected, it is also necessary to know the position of the label and the indication in the electric energy meter. Therefore, this module uses the feature point mapping relationship of SIFT to project the approximate area of the electric energy meter screen. In the corrected screen area, the indication and label area of the image to be detected are obtained according to the positional relationship of the label and indication contained in the template image.

The first step is to obtain a rough screen area by using the mapping relationship between the image to be detected and the template image.

While completing the power meter type matching module, the corresponding template image of the picture to be detected has established a corresponding relationship about the SIFT feature points: y=F(x), where x is the picture to be detected, and y is the corresponding The template image of , F is the mapping function between x and y. According to the corresponding relationship between the two, the screen area of the electric energy meter in the picture to be detected corresponding to the screen area of the electric energy meter in the template picture can be calculated.

The second step is to use Canny edge detection algorithm and Hough transform algorithm to correct the obtained screen area.

After the first step, there may be some blank areas in the obtained image (the approximate area of the LCD screen), so it is necessary to fine-tune the image output in the first step. By observing the image of the electric energy meter, it is easy to find that the color of the LCD screen is significantly different from the surrounding area, and there is a relatively clear edge outline. The Canny edge detection method can effectively extract these edges. After using Canny to detect the edge line of the LCD screen, the position of the four corners of the LCD screen can be calculated according to the geometric relationship of the edge line of the screen. The screen is corrected to obtain the target liquid crystal area.

The third step is to complete the positioning of the indication and label area according to the geometric position relationship;

On the basis of obtaining the display area of the liquid crystal screen and the type of electric energy meter, using the a priori relationship of "the displayed number and label of the same type of electric energy meter are determined on the screen", according to the template image display number and the position of the label area are separated. Display the number and label area on the target screen.

The fourth step is to use the equal division method to divide the display area to obtain a single digital area. If there is a certain deviation between the determined detection frame and the real text position, the effect of equal division will be greatly affected. In the present invention, the standard screen image obtained by the Hough transform correction has laid a good foundation for the equal division method in this step. Therefore, the overall indication display area can be effectively located according to the geometric relationship, and the negative influence caused by this factor can be suppressed, and a better segmentation effect can be achieved.

(3) Electric energy representation number and label identification module

According to the a priori condition of "the content display area of the same type of electric energy meter is fixed", this module can use the binary classification method to judge whether the specified content area is displayed, so as to complete the filtering task of a single digital area and the identification task of Chinese character labels.

Therefore, this module designs a two-class perceptron neural network that discriminates whether there is display in the input area, and classifies the single digit area or Chinese character label area extracted by the previous module. And on the basis of single digit area filtering, the Le-Net recognition network is used to identify the filtered single digit image, and the task of identifying the number area is completed.

In the first step, the single-character area obtained in the positioning module in the previous stage needs to be filtered out of the area with no indication. At the same time, the Chinese character label area obtained by the positioning module in the previous stage needs to be judged whether it is displayed or not, so as to realize the identification of the Chinese character label. This embodiment accomplishes the above two tasks through a two-class perceptron neural network.

In the second step, the displayed single-character numerical area obtained in the previous step is sent to the trained Le-Net network for identification, and the task of identifying the number of electric energy representations is completed.

The method for automatic identification of electric energy representation numbers and labels combining deep neural network and SIFT features provided by the present invention, the specific implementation process includes the following steps:

Step 1: Manually screen out the general categories of electric energy meters, and select some electric energy meter images with good lighting, clean dials, and complete shapes for each category of electric energy meters as templates for matching electric energy meter categories.

Step 2: Calculate the SIFT features of each template image and save them temporarily.

Step 3: Calculate the SIFT feature of the image to be detected and match it with the SIFT feature of the template image.

Step 4: Use the Euclidean distance screening algorithm and the RANSAC algorithm to screen the successfully matched SIFT variables, and select the energy meter template category with the largest number of SIFT feature points matching as the energy meter category of the image to be detected.

Step 5: According to the SIFT mapping relationship between the picture to be detected and the template picture, the approximate area of the screen of the electric energy meter in the picture to be detected is obtained.

Step 6: Perform edge detection on the approximate area of the LCD screen obtained in Step 5, find out the four edge straight lines of the LCD screen, and then calculate the positions of the four vertices of the screen area, and use the Hough transform to obtain the standard LCD screen area to complete the LCD screen. Corrective tasks in the area.

Step 7: According to the type of the electric energy meter, determine the position where the display area and the Chinese character label area appear on the screen. Based on this, the presentation number area and the Chinese character label area are intercepted.

Step 8: Divide the display area in equal proportions to obtain the picture area of a single number.

Step 9: Use the two-category perceptron neural network to determine whether a single digit area and a Chinese character label area have numbers or Chinese characters displayed, screen out a single digit area with a digit display and a Chinese character label area with a Chinese character display, and complete the identification task of the Chinese character label.

As shown in Figure 2, the two-class perceptron network structure designed in this embodiment includes:

Input layer: The size of the input matrix is 784*1, so the image needs to be normalized to 28*28 size first, and then converted to 784*1 format;

Relu layer: activation function, breaking the linear relationship;

Softmax layer: output whether or not to display the predicted result and predicted probability.

First normalize the image to 28*28 size, then convert the image to 784 input sources for input, that is, convert the 28*28 image to 784*1 image, and use 784 pixels as the input of the perceptron network . In the perceptron network, only one relu layer and one softmax layer are designed, and structures such as convolution and fully connected layers are abandoned, and a very good recognition effect is achieved.

Step 10: Use the lenet-5 neural network to recognize the alphanumeric area that is judged to be displayed in the perceptron network, and complete the number recognition task.

As shown in Figure 3, it is the Lenet-5 network hierarchy. There are two convolution layers in the Lenet-5 network structure, and the size of the convolution kernel is 5*5. For input images with smaller scales, using larger convolution kernels is more conducive to obtaining higher-order feature structures of images, which meets the needs of electronic digital recognition. In the algorithm, a single-character image with a display of 28*28 pixels is used as input, and the output contains 10 categories, representing 0-9 digital categories.

Test Example 1

From the 1400 data set, 100 real electric energy meter pictures were randomly selected, and the comparison test of the extraction effect of the target area was carried out. The MSER algorithm, the algorithm proposed by Yang Juan et al. in "Research and Implementation of Electric Energy Meter Image Recognition Technology Based on Digital Image Processing" (hereinafter referred to as Yang Juan's algorithm) and the method of the present invention are used to compare the effects. For these 100 test set images, the detection accuracy rates of the three algorithms are shown in Table 1.

Table 1 compares the target region detection accuracy of different algorithms

From the test results in Table 1, it can be seen that the proposed algorithm has absolute advantages in the accuracy of target area detection (location extraction). Not only that, an additional angle correction step will be added to the extracted target area image, no matter how the angle of the target area in the original image changes, you can get a display area facing yourself, which provides a basis for the later identification of the target area content. Great convenience.

Test Example 2

From the 1400 data set, 100 real-shot electric energy meter pictures were randomly selected, and the target area identification and comparison test was carried out. The identification method in the present invention (using the perceptron network and the lenet-5 network to complete the identification task) and the algorithm proposed by Lin Jianping et al. The algorithm (referred to as the binary recognition algorithm) proposed in "Digital Multimeter Digital Recognition Method Based on OPENCV" is compared. The accuracy of the perceptron network has reached 96%, and the processing time of a single image is also controlled within 0.1s. The accuracy of the lenet-5 digital recognition network is also very high, reaching an accuracy of 98%.

However, since the LSSVM algorithm and the binarization recognition algorithm can only recognize the numbers on the electric energy meter, the correct rate of the digital recognition is used as the evaluation standard of the model effect. The processing effect and accuracy of the three methods are shown in Table 2.

Table 2 Recognition effect of target area

The main reason why the recognition method proposed in the present invention has such excellent recognition performance is that the proposed algorithm uses the method of equal division under the premise of ensuring the accurate acquisition of the target area, which effectively reduces the impact on shooting noise, illumination, etc. The influence of factors on the experimental results improves the robustness of the algorithm. And these problems happen to be not solved by the LSSVM algorithm and the binarization recognition algorithm.

The automatic identification method of electric energy representation number and label combining deep neural network and SIFT features provided by the present invention can effectively solve the problem that the deep neural network fails to detect subtle objects, and the traditional non-deep learning algorithm is relatively resistant to factors such as illumination and angle. The problem of poor performance and the long processing time of the existing automatic meter identification algorithm in the target identification process. It can effectively improve meter reading efficiency and recognition accuracy.

Although the embodiment of the present invention has been disclosed as above, it is not limited to the application listed in the description and the embodiment, and it can be applied to various fields suitable for the present invention. For those skilled in the art, it can be easily Therefore, the invention is not limited to the specific details and illustrations shown and described herein without departing from the general concept defined by the appended claims and the scope of equivalents.

Claims (7)

1. a kind of electric energy representation number and label automatic identification method in conjunction with deep neural network and SIFT feature, is characterized in that, comprises the steps: Step 1. Select templates representing different types of electric energy meters; Step 2: Perform SIFT feature point matching on the picture to be detected and the template to determine the type of electric energy meter of the picture to be detected; Step 3, positioning the indication area and the Chinese character label area on the picture to be detected, and segmenting the indication area to obtain a single digital area; Step 4: Screen out a single number area with digital display and a Chinese character label area with Chinese character display, and perform Chinese character label recognition and single number recognition. 2. the electric energy representation number and label automatic identification method combining deep neural network and SIFT feature according to claim 1, is characterized in that, in described step 2, the picture to be detected and described template are carried out SIFT feature point matching After that, it also includes the following steps: Step 1, calculate the SIFT feature vector of the picture to be detected, and calculate the Euclidean distance between the matching feature vector and the template, and filter out the matching pair of feature points that exceed the set threshold to obtain a preliminary screening The feature point matching pairs of ; Step 2, inputting the initially screened feature point matching pairs into the surrogate model, and performing further screening to obtain the final feature point matching pairs; and determining the number of matches with the SIFT feature points of the picture to be detected according to the final feature point matching pairs the most templates; Among them, the electric energy meter type of the template with the largest number of matching SIFT feature points of the picture to be detected is determined as the electric energy meter type of the picture to be detected. 3. the electric energy representation number and label automatic identification method combining deep neural network and SIFT feature according to claim 2, is characterized in that, in described step 2, described surrogate model is RANSAC model, obtains final characteristic Point matching pairs, including the following steps: Step a. Randomly extract 4 sample data from the data set composed of the final feature point matching pairs, and calculate the transformation matrix H, which is recorded as the model M; Step b, calculate the projection error of all data in the described data set and the model M, if the error is less than the threshold, add the inner point set I; Wherein, if the number of elements in the current interior point set I is greater than the optimal interior point set I_best, then update I_best=I, and update the number of iterations at the same time; If the number of iterations is greater than k, exit; otherwise, increase the number of iterations by 1, and repeat the above steps; Among them, the number of iterations k is:

Among them, p is the confidence; w is the ratio of the number of inliers to the number of data points; m is the minimum number of samples required to calculate the model. 4. according to claim 1 and 3 described in conjunction with deep neural network and SIFT characteristic electric energy representation number and label automatic identification method, it is characterized in that, in described step 3, treat the indication area and Chinese character label on the picture to be detected Area positioning, including the following steps: Step A, according to the mapping relationship between the picture to be detected and the template of the corresponding electric energy meter type, obtain the pre-judgment screen area; Step B, performing edge detection on the pre-judgment screen area, obtaining four edge straight lines of the liquid crystal screen, calculating the positions of the four vertices of the screen area, and using Hough transform for correction to obtain a standard liquid crystal screen area; Step C: According to the type of the electric energy meter, determine the position where the display area and the Chinese character label area on the picture to be detected appear on the screen, and intercept the display area and the Chinese character label area. 5. the electric energy representation number and label automatic identification method combining deep neural network and SIFT feature according to claim 4, is characterized in that, in described step 3, by carrying out equal ratio division to described indication region, Get a single number field. 6. the electric energy representation number and the label automatic identification method combining deep neural network and SIFT feature according to claim 5, it is characterized in that, in described step 4, distinguish single digit region and Chinese character by two-class perceptron neural network Whether there are numbers or Chinese characters displayed in the label area, and filter out a single number area with numbers displayed and a Chinese character label area with Chinese characters displayed. 7. The electric energy representation number and label automatic identification method combining deep neural network and SIFT feature according to claim 6, it is characterized in that, in described step 4, by lenet-5 neural network to have the single number of digital display The area is identified, and the electric energy representation number is obtained.

Images