CN108804421B - Text similarity analysis method and device, electronic equipment and computer storage medium - Google Patents
Text similarity analysis method and device, electronic equipment and computer storage medium Download PDFInfo
- Publication number
- CN108804421B CN108804421B CN201810522854.4A CN201810522854A CN108804421B CN 108804421 B CN108804421 B CN 108804421B CN 201810522854 A CN201810522854 A CN 201810522854A CN 108804421 B CN108804421 B CN 108804421B
- Authority
- CN
- China
- Prior art keywords
- words
- text
- word
- basic feature
- expansion
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/205—Parsing
- G06F40/216—Parsing using statistical methods
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/10—Services
- G06Q50/18—Legal services
- G06Q50/184—Intellectual property management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Business, Economics & Management (AREA)
- Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- General Physics & Mathematics (AREA)
- Audiology, Speech & Language Pathology (AREA)
- Technology Law (AREA)
- Computational Linguistics (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Tourism & Hospitality (AREA)
- Probability & Statistics with Applications (AREA)
- Operations Research (AREA)
- Entrepreneurship & Innovation (AREA)
- Economics (AREA)
- Human Resources & Organizations (AREA)
- Marketing (AREA)
- Primary Health Care (AREA)
- Strategic Management (AREA)
- General Business, Economics & Management (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application relates to the field of text processing, and discloses a text similarity analysis method, a text similarity analysis device, electronic equipment and a computer-readable storage medium, wherein the text similarity analysis method comprises the following steps: determining a first preset number of basic feature words of a target text; then, based on the trained text word vector library, expanding the basic feature words with a first preset number respectively to obtain expanded words with a second preset number corresponding to each basic feature word; and then determining similar texts of the target text from a preset text database based on the basic feature words, the expansion words and the weighted values of the words. The method of the embodiment of the application greatly expands the number of the extracted professional vocabularies capable of representing the target text, effectively improves the statistical characteristic of the word frequency of the text characteristic of the representation target text, can quickly and accurately select the similar patents of the target text from the preset text database, and greatly improves the accuracy of the patent similarity analysis.
Description
Technical Field
The present application relates to the field of identity recognition technologies, and in particular, to a text similarity analysis method and apparatus, an electronic device, and a computer storage medium.
Background
Text (e.g., paper text, patent text) as a carrier of natural language usually exists in an unstructured or semi-structured form. With the rapid development of the internet technology, the text similarity analysis has been widely applied in many fields, for example, in the fields of information retrieval, text classification, text clustering, automatic question answering, and the like, the text similarity analysis is a more fundamental and important task.
Taking a patent text as an example, in the process of performing the patent similarity analysis, it is necessary to convert an unstructured patent text into structured information which is convenient for computer recognition processing, perform feature extraction on the structured information, and perform the patent similarity analysis according to the extracted features. Common patent similarity analysis methods include patent semantic analysis methods, patent trees, text mining and the like, and although these methods have improved the analysis quality to some extent, the problem of low similarity analysis accuracy still exists in the patent similarity analysis process.
Disclosure of Invention
The object of the present application is to solve at least one of the above-mentioned technical drawbacks, in particular the low accuracy of the similarity analysis.
In a first aspect, a text similarity analysis method is provided, including:
determining a first preset number of basic feature words of a target text;
based on the trained text word vector library, respectively expanding the basic feature words with a first preset number to obtain expanded words with a second preset number corresponding to each basic feature word;
and determining similar texts of the target texts from a preset text database based on the basic feature words, the expansion words and the weighted values of the words.
In a second aspect, there is provided a text similarity analysis apparatus, including:
the first determining module is used for determining a first preset number of basic feature words of the target text;
the expansion module is used for expanding the basic feature words with the first preset number respectively based on the trained text word vector library to obtain expansion words with a second preset number corresponding to each basic feature word;
and the second determining module is used for determining the similar text of the target text from the preset text database based on the basic characteristic words, the expansion words and the weight values of the words.
In a third aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the computer program, the text similarity analysis method is implemented.
In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, which when executed by a processor implements the text similarity analysis method described above.
The text similarity analysis method provided by the application determines a first preset number of basic feature words of a target text, so that the text feature words capable of representing the target text are extracted, and a precondition guarantee is provided for respectively expanding the first preset number of basic feature words based on a trained text word vector library; based on the trained text word vector library, respectively expanding the basic feature words with the first preset number to obtain expanded words with a second preset number corresponding to each basic feature word, greatly expanding the number of extracted professional words capable of representing the target text, effectively improving the statistical characteristic of the text feature word frequency representing the target text, and laying a foundation for quickly and accurately determining the similar text of the target text subsequently; and determining similar texts of the target texts from a preset text database based on the basic characteristic words, the expansion words and the weighted values of the words, so as to quickly and accurately select similar patents of the target texts from the preset text database, and further identify technical competitors of enterprises or institutions to which the target texts belong according to the similar patents, thereby greatly improving the accuracy of patent similarity analysis and the accuracy of identification of the patent competitors.
Additional aspects and advantages of the present application will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the present application.
Drawings
The foregoing and/or additional aspects and advantages of the present application will become apparent and more readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings of which:
fig. 1 is a schematic flowchart of a text similarity analysis method according to an embodiment of the present application;
FIG. 2 is a diagram illustrating a weight distribution of text feature words according to an embodiment of the present application;
FIG. 3 is a diagram illustrating a text similarity analysis process according to an embodiment of the present application;
fig. 4 is a schematic diagram of a basic structure of a text similarity analysis apparatus according to an embodiment of the present application;
FIG. 5 is a detailed structural diagram of a text similarity analysis apparatus according to an embodiment of the present application;
fig. 6 is a schematic structural diagram of an electronic device according to an embodiment of the present application.
Detailed Description
Reference will now be made in detail to embodiments of the present application, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the drawings are exemplary only for the purpose of explaining the present application and are not to be construed as limiting the present application.
As used herein, the singular forms "a", "an", "the" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and/or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be understood that when an element is referred to as being "connected" or "coupled" to another element, it can be directly connected or coupled to the other element or intervening elements may also be present. Further, "connected" or "coupled" as used herein may include wirelessly connected or wirelessly coupled. As used herein, the term "and/or" includes all or any element and all combinations of one or more of the associated listed items.
To make the objects, technical solutions and advantages of the present application more clear, embodiments of the present application will be described in further detail below with reference to the accompanying drawings.
The patent text is an important carrier for recording scientific research activities and research methods, and is an important document for scientific research personnel to obtain scientific experience and know advanced technology in the industry. In the face of massive patent resources, an automatic method is adopted to quickly select similar patents of a certain enterprise or organization, and then a technical competitor of the enterprise or organization is identified. At present, in a method for identifying an enterprise competitor by using a data mining patent, feature words are extracted on the basis of data such as titles and abstracts of the patent, and on the basis of the extracted feature words, a VSM (Vector Space Model) Model is used for carrying out Vector expression on a patent text, and then similarity analysis of the patent is carried out. However, the title and abstract of the patent are short in space, statistical characteristics of word frequency of text features used for representing patent technology are not obvious, and the number of extracted professional vocabularies capable of representing the patent is insufficient, so that the information quantity of VSM vectors of patent texts obtained based on the method is insufficient, the representation capability of the patent texts is limited, the accuracy of the correlation analysis result of the patent is low, and the accuracy of identification of patent competitors is further influenced.
The text similarity analysis method, the text similarity analysis device, the electronic equipment and the computer readable storage medium aim to solve the technical problems in the prior art.
The following describes the technical solution of the present application and how to solve the above technical problems with specific embodiments. The following several specific embodiments may be combined with each other, and details of the same or similar concepts or processes may not be repeated in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.
Example one
The embodiment of the application provides a text similarity analysis method, as shown in fig. 1, including:
step S100, determining a first preset number of basic feature words of the target text.
Specifically, a first preset number of feature words of the target text are extracted from text information such as a title and an abstract of the target text (for example, a patent text), where the first preset number may be set according to actual needs in an extraction process, for example, the first preset number may be set to 5, 10, and 15, and the like, that is, 5 or 10 or 15 or other numerical feature words are extracted from the title and the abstract of the target text, and the extracted feature words are used as basic feature words of the target text, that is, vocabularies representing specialties of the target text.
And S200, expanding the basic characteristic words with the first preset number respectively based on the trained text word vector library to obtain expanded words with the second preset number corresponding to each basic characteristic word.
Specifically, the title and the abstract of the target text are short, the number of extracted basic feature words capable of representing the profession of the target text is very limited, the statistical property of the text feature word frequency of the target text is not enough represented, the extracted basic feature words with the first preset number are respectively expanded to obtain the expanded words with the second preset number corresponding to each basic feature word, the number of the extracted professional words capable of representing the target text can be greatly expanded, the statistical property of the text feature word frequency representing the target text is effectively improved, and a foundation is laid for quickly and accurately determining the similar text of the target text subsequently.
Further, the second preset number may be set according to actual needs in the expansion process, and the second preset number may be the same as the first preset number, or may be different from the first preset number, for example, the second preset number may be set to 5, 15, or 30, and the like, that is, each basic feature word is expanded to obtain 5 or 15 or 30 or other numerical expansion words of each basic feature word.
For example, when the basic feature word is "installer" and the second preset number is 6, the extension words may be "driver", "installation file", "the software", "installation package", "configuration file", and "client program".
Step S300, determining similar texts of the target texts from a preset text database based on the basic feature words, the expansion words and the weight values of the words.
Specifically, based on each basic feature word, each expansion word and the weight value of each word of the target file, similar texts of the target text are quickly and accurately selected from a large number of text resources in a preset text database.
For example, when the target text is a patent text and the name of the patent is "air purifier", similar patents of the patent, such as "electronic air purifier", "electric bag composite dust collector", etc., can be quickly and accurately selected from patent resources in a preset text database.
Further, after the similar text of the target text is determined, by clicking and checking the related information of the similar text, information of an enterprise or an organization to which the similar patent belongs can be obtained, and according to the information of the enterprise or the organization to which the similar patent belongs, a technical competitor of the enterprise or the organization to which the target text belongs can be further known, for example, the competitor is the enterprise or the organization to which the similar patent belongs.
Compared with the prior art, the text similarity analysis method provided by the embodiment of the application determines the first preset number of basic feature words of the target text, so that the text feature words capable of representing the target text are extracted, and a precondition guarantee is provided for respectively expanding the first preset number of basic feature words based on a trained text word vector library in the following process; based on the trained text word vector library, respectively expanding the basic feature words with the first preset number to obtain expanded words with a second preset number corresponding to each basic feature word, so that the quantity of extracted professional words capable of representing the target text is greatly expanded, the statistical characteristic of the text feature word frequency representing the target text is effectively improved, and a foundation is laid for quickly and accurately determining the similar text of the target text subsequently; based on the weight values of the basic characteristic words, the expansion words and the words, determining similar texts of the target texts from a preset text database, quickly and accurately selecting similar patents of the target texts from the preset text database, and further identifying technical competitors of enterprises or institutions to which the target texts belong according to the similar patents, so that the accuracy of patent similarity analysis and the accuracy of identification of the patent competitors are greatly improved.
Example two
The embodiment of the present application provides another possible implementation manner, and on the basis of the first embodiment, the method shown in the second embodiment is further included, wherein,
in step S100, a first preset number of basic feature words of the target text is determined by the TextRank algorithm.
Specifically, in the embodiment of the present application, taking the target text as an example of a patent text, the following description is made for the step S100 described above:
the existing method usually determines patent feature words according to word frequency on the basis of common methods such as word segmentation, part of speech tagging and the like, and because some words with high word frequency but poor speciality are extracted when the words are extracted by the methods, the words extracted by the methods do not have good patent representation capability. In order to solve the problem, the textRank algorithm is adopted to extract the basic feature words of the patent, the basic feature words extracted by the method have stronger speciality, and a foundation is laid for building a patent text VSM model.
The TextRank algorithm is a graph-based ranking algorithm for texts, basically thinks of a Pagerank algorithm derived from Google, and by dividing a text into a plurality of constituent units (such as words and sentences), establishing a graph model, ranking important components in the text by using a voting mechanism, and extracting keywords only by using the information of a single document. Different from models such as LDA (text Dirichlet Allocation, document theme generation Model), HMM (Hidden Markov Model), and the like, TextRank does not need to learn and train a plurality of documents in advance, and is widely applied because it is simple and effective. the textRank algorithm is to sort the subsequent keywords by using the relation (co-occurrence window) between local vocabularies and directly extract the keywords from the text.
Further, determining a first preset number of basic feature words of the target text through a TextRank algorithm, comprising the following steps:
1) segmenting a given target text according to a complete sentence;
2) for each sentence, performing word segmentation and part-of-speech tagging, filtering stop words, and only reserving words with specified part-of-speech, such as nouns, verbs, adjectives and the like, namely the reserved candidate key words;
3) and constructing a candidate keyword graph G ═ (V, E), wherein V is a node set, and E is an edge set. The candidate keywords generated in the step 2) are combined, then an edge between any two points is constructed by adopting a co-occurrence relation, the condition that the edge exists between the two nodes means that the vocabularies corresponding to the two nodes co-occur in a window with the length of K, and the K represents the size of the window, namely, the maximum number of K words co-occur;
4) iteratively propagating the weight of each node according to the formula G ═ V, E above until convergence;
5) the node weights are sorted in a reverse order, so that the most important T words are obtained and serve as candidate keywords, namely basic feature words in the embodiment of the application;
6) marking the most important T words obtained in the step 5) in the original text, and if adjacent phrases are formed, combining the words into a multi-word keyword.
For the embodiment of the application, the textRank algorithm is adopted to extract the basic feature words of the target text, so that the method has stronger speciality, and a plurality of documents do not need to be learned and trained in advance, so that the method is simpler, more convenient and quicker.
EXAMPLE III
The embodiment of the present application provides another possible implementation manner, and on the basis of the second embodiment, the method shown in the third embodiment is further included, wherein,
before step S200, step S101 (not labeled in the figure) is further included: and training the text in a preset database through a continuous bag-of-words neural network model to obtain a trained text word vector library.
Step S200 includes step S2001 (not shown), step S2002 (not shown), and step S2003 (not shown), wherein,
step S2001: and acquiring a first word vector of any basic characteristic word by querying the trained text word vector library.
Step S2002: and calculating a cosine similarity value between the first word vector and a second word vector, wherein the second word vector is a word vector except the first word vector in the trained text word vector library.
Step S2003: and determining words respectively corresponding to second word vectors of a second preset number with cosine similarity values larger than the first preset threshold value, and taking the words as the expansion words of any basic characteristic word.
Specifically, the embodiment of the application adopts a deep learning technology to expand the basic feature words, and the method comprises the following steps:
1) training text Word vector library by using Word2Vec (Word vector) method
Expressing words in text with word vectors is a core technique for introducing deep learning algorithms into natural language processing. Word2vec is an excellent model tool for acquiring Word vectors, which is sourced by google in 2013, and mainly adopts CBOW (Continuous Bag-Of-Words) and Skip-gram (Continuous Skip grammar) models. The method and the device for text word vector training adopt a CBOW neural network model with higher efficiency to train the text in the preset database to obtain a trained text word vector library.
For example, when the text is a patent text, the embodiment of the present application performs training on about 2000 ten thousand patent texts of about 10G to obtain a trained patent word vector library, where the patent text includes text fields such as patent titles and abstracts, the dimension of the generated word vector is 100, and the trained patent word vector library has about 100 ten thousand words and a size of about 990M.
2) Expanding basic characteristic words based on trained text word vector library
Specifically, when the target text is a patent text, the method for expanding the basic feature words extracted from each patent text is to query the patent word vector library one by one for the basic feature words with the first preset number obtained by the TextRank algorithm to obtain the word vector of each basic feature word (i.e., the first word vector in step S2001), and then perform a cosine similarity calculation process, where the cosine similarity calculation process is: calculating cosine similarity values between the word vector of any basic feature word and other word vectors (namely, the second word vector in step S2002) in the patent word vector library except the word vector of the basic feature word, and determining an expansion word of the basic feature word according to comparison between the cosine similarity values and a first preset threshold value and a second preset number.
Further, for each determined basic feature word, the above-mentioned cosine similarity value calculation process is performed, so as to determine the extension word of each basic feature word.
For example, when the basic feature words are "installation procedure", "low price", "water reuse", "decontamination", "high speed railway", and "bias", and the second preset number is 6, the expansion words of each basic feature word can be obtained as shown in table 1:
TABLE 1 basic characteristic words and corresponding expansion words
For the embodiment of the application, specific processes and operation steps for determining the extension words of each basic feature word based on the trained text word vector library are provided, so that a person skilled in the art can rapidly and accurately complete the extension of the basic feature words according to the steps in the embodiment of the application, greatly expand the number of the extracted professional words capable of representing the target text, effectively improve the statistical property of the text feature word frequency representing the target text, and lay a foundation for rapidly and accurately determining the similar text of the target text subsequently.
Example four
The embodiment of the present application provides another possible implementation manner, and on the basis of the third embodiment, the method shown in the fourth embodiment is further included, wherein,
before step S300, step S201 (not labeled in the figure) is further included: filtering stop words in the expansion words of any basic characteristic word; and/or filtering out words of which the reverse file frequency is smaller than a second preset threshold value in the expansion words of any basic characteristic word.
Before step S300, step S202 (not labeled in the figure) is further included: determining the weight value of each word. Determining the weight value of each word comprises the following steps:
determining the weight value of any word by the following formula:
wi＝idfi*(p_tfi+c_tfi)
wherein, wiRepresents the weight value idfiThe reverse file frequency, p _ tf, representing the wordiIndicating the frequency of the word in the text title and text abstract of the target text, c _ tfiIndicating the frequency of the word in text other than the target text.
Specifically, after a second preset number of expanded words corresponding to each basic feature word obtained in the above steps S2001, S2002, and S2003, the obtained expanded words need to be further filtered, wherein only stop words in the expanded words may be filtered according to needs, only words whose reverse text frequency is smaller than a second preset threshold may be filtered, and also stop words and words whose reverse text frequency is smaller than the second preset threshold may be simultaneously filtered, so that the expanded words may better represent the target text by filtering the obtained expanded words.
For example, when the second preset threshold is 4.0, in the process of filtering the obtained expanded word, only stop words may be filtered, only words in which the reverse text frequency is less than 4.0 may be filtered, and also stop words and words in which the reverse text frequency is less than 4.0 may be simultaneously filtered, so that a word set, that is, the expanded word of the basic feature word in the embodiment of the present application, is finally obtained.
Further, let us assume a logicalThe basic characteristic words and the expansion words of the basic characteristic words obtained by the steps are w1,w2,…,wNAnd the target text in the above steps is a patent text, and then the weighted value of each word (including each basic feature word and the expanded word of each basic feature word) can be calculated and determined by formula (1):
wi＝idfi*(p_tfi+c_tfi) (1)
wherein, wiDenotes the weight value, idf, of any wordiThe reverse file frequency, p _ tf, representing the wordiIndicating the frequency of any such term in the patent title and abstract; c _ tfiIndicating the frequency of occurrence of any of the words in text other than patent text, such as paper text. In addition, p _ tfiThe calculation method of (d) may be: (the number of occurrences of this word in the patent title and patent abstract + 1)/(the total number of words of each basic feature word and each expansion word of the basic feature words +1), for words that do not occur in the patent title and patent abstract, adding 1 may play a smoothing role.
Further, the weight value w of each word is obtainediThen, the obtained weight value w is further processediAnd (5) carrying out normalization processing to obtain the weight distribution of each word of the patent, as shown in figure 2.
For the embodiment of the application, the stop words and the words with the reverse file frequency smaller than the second preset threshold value in the expansion words are filtered, so that the expansion words can better represent the target text, and the influence of the stop words and the words with the reverse file frequency smaller than the second preset threshold value on the accuracy of the text similarity analysis is effectively avoided. In addition, the implementation method for determining the weight value of each word is convenient for technicians in the field to rapidly determine the weight value of each word, and provides a precondition guarantee for subsequently determining similar texts of the target text from the preset text database.
EXAMPLE five
The embodiment of the present application provides another possible implementation manner, and on the basis of the fourth embodiment, the method shown in the fifth embodiment is further included, wherein,
step S300 includes step S3001 (not shown), step S3002 (not shown), step S3003 (not shown) and step S3004 (not shown), wherein,
step S3001: respectively determining a first preset number of basic feature words of a plurality of texts to be screened in a preset text database, respectively expanding the first preset number of basic feature words based on a trained text word vector library to obtain a second preset number of expansion words corresponding to each basic feature word, and determining the weight value of each word, so as to obtain the basic feature words, the weight value of the basic feature words, the expansion words of the basic feature words and the weight value of the expansion words corresponding to each text to be screened.
Step S3002: and detecting whether the basic characteristic words and the expansion words of any text to be screened have the same words as the basic characteristic words and the expansion words of the target text.
Step S3003: and aiming at any text to be screened, if the text exists, calculating the product of the weight value of any same word in the text to be screened and the weight value of any same word in the target text, and calculating the sum of the products of all same words.
Step S3004: and selecting the texts to be screened, of which the sum of the products obtained by calculation is greater than a third preset threshold value, as the similar texts of the target texts, from the texts to be screened.
Specifically, when a large number of texts such as patents and articles are stored in the preset text database, and similar texts of a target text are screened from the preset text database, a plurality of texts to be screened in the preset text database are all subjected to the steps S100 (determining a first preset number of basic feature words), S200 (respectively expanding the first preset number of basic feature words based on the trained text word vector library to obtain a second preset number of expanded words corresponding to each basic feature word), S201 (filtering stop words in the expanded words of any basic feature word, and/or filtering words of which reverse file frequency is less than a second preset threshold value in the expanded words of any basic feature word), S202 (determining weight value of each word), and the like in the first to fourth embodiments, so as to obtain basic feature words corresponding to each text to be screened, The weight value of the basic characteristic word, the expansion word of the basic characteristic word and the weight value of the expansion word.
Further, in the process of searching for a similar text of a target text from each text to be screened in a preset text database, the text to be screened can be traversed according to the basic feature words and the expansion words of the target text, specifically, each text to be screened can be sequentially traversed by detecting whether the basic feature words and the expansion words of any text to be screened have words same as the basic feature words and the expansion words of the target text, the text to be screened which does not have the words same as the basic feature words and the expansion words of the target text is filtered, and only the text to be screened which has the words same as the basic feature words and the expansion words of the target text is retained for further processing.
Further, when the text to be screened has the same words as the basic feature words and the expansion words of the target text, calculating the product of the weight value of any same word in the text to be screened and the weight value of any same word in the target text, wherein when the same words are multiple, the products corresponding to the multiple words are accumulated respectively, namely, the sum of the products of all the same words is calculated, and when the same word is only one, the product is directly used as the final sum of the products.
Further, from the texts to be screened, which have words same as the basic feature words and the expansion words of the target text, the texts closest to the target text are screened out as the similar texts of the target text, wherein the texts to be screened, of which the sum of products is greater than a third preset threshold value, can be selected as the similar texts of the target text, and the value of the third preset threshold value can be dynamically set according to actual needs. Table 2 shows an example of relevant information about the target text and its corresponding similar text.
TABLE 2 target text and corresponding similar text information
Further, with reference to the methods in the first to fifth embodiments of the present application, the target text in fig. 3 is exemplified by patent texts, and a basic process for finding similar patents of a target patent is provided, where in fig. 3, step S1 (extraction of patent basic feature words based on TextRank) is performed first, step S2 (determination of deep learning algorithm) is performed, step S3 (training of a patent word library) is performed, step S4 (expansion of basic feature words based on a patent word vector library) is performed, step S5 (filtering of patent feature expansion words) is performed, step S6 (calculation of patent feature word weights) is performed, and step S7 (outputting similar patents and corresponding patent rights) is performed.
For the embodiment of the application, a specific process and operation steps for determining the similar text of the target text from the preset text database based on each basic feature word, each extension word and the weighted value of each word are provided, so that a person skilled in the art can quickly and accurately select the similar text of the target text from the preset text database according to the steps in the embodiment of the application, and further identify a technical competitor of an enterprise or an organization to which the target text belongs according to the similar specialization.
EXAMPLE six
Fig. 4 is a schematic structural diagram of a text information translation apparatus according to an embodiment of the present application, and as shown in fig. 4, the text information translation apparatus 40 may include: a first determination module 41, an expansion module 42 and a second determination module 43, wherein:
the first determining module 41 is configured to determine a first preset number of basic feature words of the target text;
the expansion module 42 is configured to expand the basic feature words of the first preset number based on the trained text word vector library, to obtain expanded words of a second preset number corresponding to each basic feature word;
the second determining module 43 is configured to determine a similar text of the target text from a preset text database based on the basic feature words, the expansion words, and the weight values of the words.
Specifically, the first determining module 41 is specifically configured to determine, through the TextRank algorithm, a first preset number of basic feature words of the target text.
Further, the apparatus further includes a training module 44, as shown in fig. 5, where the training module 44 is configured to train the text in the preset database through the continuous bag-of-words neural network model to obtain a trained text word vector library.
Further, the expansion module 42 includes an obtaining sub-module 421, a calculating sub-module 422, and an expanded word determining sub-module 423, as shown in fig. 5, where the obtaining sub-module 421 is configured to obtain a first word vector of any basic feature word by querying a trained text word vector library;
the calculating submodule 422 is configured to calculate a cosine similarity value between the first word vector and a second word vector, where the second word vector is a word vector in the trained text word vector library except the first word vector;
the expanded word determining submodule 423 is configured to determine words corresponding to second word vectors of a second preset number, of which the cosine similarity values are greater than the first preset threshold, and use the words as expanded words of any basic feature word.
Further, the apparatus further includes a filtering module 45, as shown in fig. 5, where the filtering module 45 is configured to filter stop words in the expanded words of any basic feature word; and/or filtering out words of which the reverse file frequency is smaller than a second preset threshold value in the expansion words of any basic characteristic word.
Further, the apparatus further includes a weight determining module 46, as shown in fig. 5, wherein the weight determining module 46 is configured to determine a weight value of each word; the method is specifically used for determining the weight value of any word through the following formula:
wi＝idfi*(p_tfi+c_tfi)
wherein, wiRepresents the weight value idfiMeans of any one of theReverse file frequency of words, p _ tfiIndicating the frequency of the word in the text title and text abstract of the target text, c _ tfiIndicating the frequency of the word in text other than the target text.
Further, the second determination module 43 comprises a pre-processing submodule 431, a detection submodule 432, a product calculation submodule 433 and a screening submodule 434, wherein,
the preprocessing submodule 431 is used for respectively acquiring a first preset number of basic feature words from a plurality of texts to be screened in a preset text database, respectively expanding the first preset number of basic feature words based on a trained text word vector library to obtain a second preset number of expansion words corresponding to each basic feature word, and determining a weight value of each word to obtain the basic feature words, the weight value of the basic feature words, the expansion words of the basic feature words and the weight values of the expansion words corresponding to each text to be screened;
the detection submodule 432 is configured to detect whether a word identical to the basic feature word and the expansion word of the target text exists in the basic feature word and the expansion word of any text to be screened;
the product calculation submodule 433 is configured to calculate, for any text to be screened, if the text exists, a product of a weight value of any same word in the text to be screened and a weight value of any same word in the target text, and calculate a sum of products of all the same words;
the screening submodule 434 is configured to select, as a similar text of the target text, a text to be screened whose sum of products obtained by calculation is greater than a third preset threshold value, from the plurality of texts to be screened.
Compared with the prior art, the device provided by the embodiment of the application determines the first preset number of basic feature words of the target text, so that the text feature words capable of representing the target text are extracted, and a precondition guarantee is provided for respectively expanding the first preset number of basic feature words based on a trained text word vector library; based on the trained text word vector library, expanding the basic feature words with the first preset number respectively to obtain expanded words with a second preset number corresponding to each basic feature word respectively, greatly expanding the number of extracted professional words capable of representing the target text, effectively improving the statistical characteristic of the text feature word frequency of the represented target text, and laying a foundation for quickly and accurately determining the similar text of the target text subsequently; and determining similar texts of the target texts from a preset text database based on the basic characteristic words, the expansion words and the weight values of the words, so as to quickly and accurately select similar patents of the target texts from the preset text database, further identify technical competitors of enterprises or institutions to which the target texts belong according to the similar patents, and greatly improve the accuracy of patent similarity analysis and the accuracy of identification of the patent competitors.
EXAMPLE seven
An embodiment of the present application provides an electronic device, as shown in fig. 6, an electronic device 600 shown in fig. 6 includes: a processor 601 and a memory 603. The processor 601 is coupled to the memory 603, such as via a bus 602. Further, the electronic device 600 may also include a transceiver 604. It should be noted that the transceiver 604 is not limited to one in practical applications, and the structure of the electronic device 600 is not limited to the embodiment of the present application.
The processor 601 is applied to the embodiment of the present application, and is configured to implement the functions of the first determining module, the extending module, and the second determining module shown in fig. 4. The transceiver 604 includes a receiver and a transmitter, and the transceiver 604 is applied in the embodiment of the present application to realize the functions of the acquisition sub-module shown in fig. 5.
The processor 601 may be a CPU, general purpose processor, DSP, ASIC, FPGA or other programmable logic device, transistor logic device, hardware component, or any combination thereof. Which may implement or perform the various illustrative logical blocks, modules, and circuits described in connection with the disclosure. The processor 601 may also be a combination of processors implementing computing functionality, e.g., a combination comprising one or more microprocessors, a combination of DSPs and microprocessors, or the like.
The memory 603 is used for storing application program codes for executing the scheme of the application, and the processor 601 controls the execution. The processor 601 is configured to execute application program codes stored in the memory 603 to implement the actions of the text similarity analysis apparatus provided in the embodiment shown in fig. 4.
The present application provides a computer-readable storage medium, on which a computer program is stored, and the computer program, when executed by a processor, implements the method shown in the first embodiment. Compared with the prior art, the method has the advantages that the first preset number of basic feature words of the target text is determined, so that the text feature words capable of representing the target text are extracted, and a precondition guarantee is provided for respectively expanding the first preset number of basic feature words based on a trained text word vector library in the follow-up process; based on the trained text word vector library, respectively expanding the basic feature words with the first preset number to obtain expanded words with a second preset number corresponding to each basic feature word, so that the number of extracted professional words capable of representing the target text is greatly expanded, the statistical characteristic of the text feature word frequency representing the target text is effectively improved, and a foundation is laid for quickly and accurately determining the similar text of the target text subsequently; and determining similar texts of the target texts from a preset text database based on the basic characteristic words, the expansion words and the weighted values of the words, so as to quickly and accurately select similar patents of the target texts from the preset text database, and further identify technical competitors of enterprises or institutions to which the target texts belong according to the similar patents, thereby greatly improving the accuracy of patent similarity analysis and the accuracy of patent competitor identification.
The computer-readable storage medium provided by the embodiment of the application is suitable for any embodiment of the method. And will not be described in detail herein.
It should be understood that, although the steps in the flowcharts of the figures are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and may be performed in other orders unless explicitly stated herein. Moreover, at least a portion of the steps in the flow chart of the figures may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, in different orders, and may be performed alternately or in turns with other steps or at least a portion of the sub-steps or stages of other steps.
The foregoing is only a partial embodiment of the present application, and it should be noted that, for those skilled in the art, several modifications and decorations can be made without departing from the principle of the present application, and these modifications and decorations should also be regarded as the protection scope of the present application.
Claims (12)
1. A text similarity analysis method is characterized by comprising the following steps:
determining a first preset number of basic feature words of a target text;
based on the trained text word vector library, respectively expanding the basic feature words with a first preset number to obtain expanded words with a second preset number corresponding to each basic feature word;
determining similar texts of the target texts from a preset text database based on the basic feature words, the expansion words and the weight values of the words;
before determining a similar text of a target text from a preset text database based on each basic feature word, each expansion word and the weight value of each word, the method comprises the following steps: determining the weight value of each word;
determining the weight value of each word comprises the following steps:
determining the weight value of any word by the following formula:
w＝idf*(p_tf+c_tf)
w represents a weight value, idf represents the reverse file frequency of any word, p _ tf represents the frequency of any word in the text title and the text abstract of the target text, and c _ tf represents the frequency of any word in other texts except the target text; the determining of the similar text of the target text from the preset text database based on the basic feature words, the expansion words and the weighted values of the words comprises the following steps:
respectively expanding the basic feature words with the first preset number on the basis of a trained text word vector library to obtain a second preset number of expansion words corresponding to the basic feature words, and determining the weight value of each word to obtain the basic feature words, the weight value of the basic feature words, the expansion words of the basic feature words and the weight value of the expansion words corresponding to the text to be screened;
detecting whether basic characteristic words and expansion words of any text to be screened have the same words as basic characteristic words and expansion words of a target text;
aiming at any text to be screened, if the text exists, calculating the product of the weight value of any same word in the text to be screened and the weight value of any same word in the target text, and calculating the sum of the products of all same words;
and selecting the texts to be screened, of which the sum of the products obtained by calculation is greater than a third preset threshold value, as the similar texts of the target texts.
2. The method of claim 1, wherein determining a first predetermined number of base feature words of the target text comprises:
and determining a first preset number of basic feature words of the target text through a TextRank algorithm.
3. The method of claim 1, further comprising, before expanding the first preset number of basic feature words based on the trained text word vector library, respectively:
and training the text in the preset database through the continuous bag-of-words neural network model to obtain a trained text word vector library.
4. The method according to any one of claims 1 to 3, wherein the expanding the basic feature words of a first preset number based on the trained text word vector library to obtain a second preset number of expanded words corresponding to each basic feature word respectively comprises:
acquiring a first word vector of any basic characteristic word by querying a trained text word vector library;
calculating a cosine similarity value between the first word vector and a second word vector, wherein the second word vector is a word vector except the first word vector in the trained text word vector library;
and determining words respectively corresponding to second word vectors of a second preset number with cosine similarity values larger than the first preset threshold value, and taking the words as the expansion words of any basic characteristic word.
5. The method according to claim 1, before determining the similar text of the target text from the preset text database based on the weight values of the basic feature words, the expansion words and the words, further comprising:
filtering stop words in the expansion words of any basic characteristic word; and/or
And filtering out words of which the reverse file frequency is less than a second preset threshold value in the expansion words of any basic characteristic word.
6. A text similarity analysis apparatus, comprising:
the first determining module is used for determining a first preset number of basic feature words of the target text;
the expansion module is used for expanding the basic feature words with the first preset number respectively based on the trained text word vector library to obtain expansion words with a second preset number corresponding to each basic feature word;
the second determining module is used for determining similar texts of the target texts from a preset text database based on the basic feature words, the expansion words and the weight values of the words;
the weight determining module is used for determining the weight value of each word; the method is specifically used for determining the weight value of any word through the following formula:
w＝idf*(p_f+c_tf)
w represents a weight value, idf represents the reverse file frequency of any word, p _ tf represents the frequency of any word in the text title and the text abstract of the target text, and c _ tf represents the frequency of any word in other texts except the target text;
the second determination module comprises a preprocessing submodule, a detection submodule, a product calculation submodule and a screening submodule;
the preprocessing submodule is used for respectively acquiring a first preset number of basic feature words from a plurality of texts to be screened in a preset text database, respectively expanding the first preset number of basic feature words based on a trained text word vector library to obtain a second preset number of expansion words corresponding to each basic feature word, and determining the weight value of each word to obtain the basic feature words, the weight value of the basic feature words, the expansion words of the basic feature words and the weight values of the expansion words corresponding to each text to be screened;
the detection submodule is used for detecting whether the basic characteristic words and the expansion words of any text to be screened have the same words as the basic characteristic words and the expansion words of the target text;
the product calculation submodule is used for calculating the product of the weight value of any same word in the text to be screened and the weight value of any same word in the target text if any text to be screened exists, and calculating the sum of the products of all the same words;
and the screening submodule is used for selecting the texts to be screened, of which the sum of the products obtained by calculation is greater than a third preset threshold value, from the plurality of texts to be screened as the similar texts of the target texts.
7. The apparatus according to claim 6, wherein the first determining module is specifically configured to determine a first preset number of basic feature words of the target text by a TextRank algorithm.
8. The apparatus of claim 6, further comprising a training module;
and the training module is used for training the text in the preset database through the continuous bag-of-words neural network model to obtain a trained text word vector library.
9. The device according to any one of claims 6 to 8, wherein the expansion module comprises an acquisition sub-module, a calculation sub-module and an expansion word determination sub-module;
the acquisition sub-module is used for acquiring a first word vector of any basic characteristic word by inquiring the trained text word vector library;
the calculation submodule is used for calculating a cosine similarity value between the first word vector and a second word vector, and the second word vector is a word vector except the first word vector in the trained text word vector library;
and the expansion word determining submodule is used for determining words corresponding to second word vectors of a second preset number, of which the cosine similarity value is greater than the first preset threshold value, and taking the words as expansion words of any basic characteristic word.
10. The apparatus of claim 6, further comprising a filtering module;
the filtering module is used for filtering stop words in the expansion words of any basic characteristic word; and/or filtering out words of which the reverse file frequency is smaller than a second preset threshold value in the expansion words of any basic feature word.
11. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the text similarity analysis method of any one of claims 1-5 when executing the program.
12. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored thereon a computer program which, when being executed by a processor, implements the text similarity analysis method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810522854.4A CN108804421B (en) | 2018-05-28 | 2018-05-28 | Text similarity analysis method and device, electronic equipment and computer storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810522854.4A CN108804421B (en) | 2018-05-28 | 2018-05-28 | Text similarity analysis method and device, electronic equipment and computer storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN108804421A CN108804421A (en) | 2018-11-13 |
| CN108804421B true CN108804421B (en) | 2022-04-15 |
Family
ID=64090466
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810522854.4A Active CN108804421B (en) | 2018-05-28 | 2018-05-28 | Text similarity analysis method and device, electronic equipment and computer storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108804421B (en) |
Families Citing this family (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109558481B (en) * | 2018-12-03 | 2022-05-24 | 中国科学技术信息研究所 | Method, device and equipment for measuring correlation between patent and enterprise and readable storage medium |
| CN109614478B (en) * | 2018-12-18 | 2020-12-08 | 北京中科闻歌科技股份有限公司 | Word vector model construction method, keyword matching method and device |
| CN109885813B (en) * | 2019-02-18 | 2023-04-28 | 武汉瓯越网视有限公司 | A computing method and system for text similarity based on word coverage |
| CN110427330B (en) * | 2019-08-13 | 2023-09-26 | 腾讯科技（深圳）有限公司 | Code analysis method and related device |
| CN111199148B (en) * | 2019-12-26 | 2023-01-20 | 东软集团股份有限公司 | Text similarity determination method and device, storage medium and electronic equipment |
| CN111414753A (en) * | 2020-03-09 | 2020-07-14 | 中国美术学院 | Method and system for vocabulary extraction of product perceptual imagery |
| CN112215008B (en) * | 2020-10-23 | 2024-04-16 | 中国平安人寿保险股份有限公司 | Entity recognition method, device, computer equipment and medium based on semantic understanding |
| CN113064979A (en) * | 2021-03-10 | 2021-07-02 | 国网河北省电力有限公司 | Keyword retrieval-based method for judging construction period and price reasonability |
| CN113033197A (en) * | 2021-03-24 | 2021-06-25 | 中新国际联合研究院 | Building construction contract rule query method and device |
| CN114331766B (en) * | 2022-01-05 | 2022-07-08 | 中国科学技术信息研究所 | Method and device for determining patent technology core degree, electronic equipment and storage medium |
| CN115358221A (en) * | 2022-08-12 | 2022-11-18 | 维正知识产权科技有限公司 | Enterprise patent data comparison method and device, electronic equipment and medium |
| CN116719999B (en) * | 2023-06-16 | 2025-08-19 | 平安科技（深圳）有限公司 | Text similarity detection method and device, electronic equipment and storage medium |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103377226A (en) * | 2012-04-25 | 2013-10-30 | 中国移动通信集团公司 | Intelligent search method and system thereof |
| CN105320772A (en) * | 2015-11-02 | 2016-02-10 | 武汉大学 | Associated paper query method for patent duplicate checking |
| CN107247780A (en) * | 2017-06-12 | 2017-10-13 | 北京理工大学 | A kind of patent document method for measuring similarity of knowledge based body |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7472121B2 (en) * | 2005-12-15 | 2008-12-30 | International Business Machines Corporation | Document comparison using multiple similarity measures |
-
2018
- 2018-05-28 CN CN201810522854.4A patent/CN108804421B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103377226A (en) * | 2012-04-25 | 2013-10-30 | 中国移动通信集团公司 | Intelligent search method and system thereof |
| CN105320772A (en) * | 2015-11-02 | 2016-02-10 | 武汉大学 | Associated paper query method for patent duplicate checking |
| CN107247780A (en) * | 2017-06-12 | 2017-10-13 | 北京理工大学 | A kind of patent document method for measuring similarity of knowledge based body |
Non-Patent Citations (1)
| Title |
|---|
| 论文和专利相结合的研究前沿识别方法研究;许晓阳、郑彦宁、刘志辉;《图书情报工作》;20161231;第60卷(第24期);第1-10页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN108804421A (en) | 2018-11-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108804421B (en) | Text similarity analysis method and device, electronic equipment and computer storage medium | |
| KR102123974B1 (en) | Similar patent search service system and method | |
| CN112256939B (en) | Text entity relation extraction method for chemical field | |
| CN118278365B (en) | Automatic generation method and device for scientific literature review | |
| CN108280114B (en) | Deep learning-based user literature reading interest analysis method | |
| CN114462392B (en) | Short text feature expansion method based on association degree of subject and association of keywords | |
| CN111191022A (en) | Method and device for generating short title of commodity | |
| CN111190997A (en) | Question-answering system implementation method using neural network and machine learning sequencing algorithm | |
| CN112307364B (en) | Character representation-oriented news text place extraction method | |
| CN107015961A (en) | A kind of text similarity comparison method | |
| CN112800226A (en) | Method for obtaining text classification model, method, apparatus and device for text classification | |
| CN114138979A (en) | Cultural relic safety knowledge map creation method based on word expansion unsupervised text classification | |
| Gopan et al. | Comparative study on different approaches in keyword extraction | |
| CN113656429A (en) | Keyword extraction method and device, computer equipment and storage medium | |
| CN110728135B (en) | Text theme indexing method and device, electronic equipment and computer storage medium | |
| CN118170899B (en) | AIGC-based media news manuscript generation method and related device | |
| CN112926340A (en) | Semantic matching model for knowledge point positioning | |
| CN108228612B (en) | Method and device for extracting network event keywords and emotional tendency | |
| Mouratidis et al. | Domain-specific term extraction: a case study on Greek Maritime legal texts | |
| Peleshchak et al. | Text Tonality Classification Using a Hybrid Convolutional Neural Network with Parallel and Sequential Connections Between Layers. | |
| CN109558481B (en) | Method, device and equipment for measuring correlation between patent and enterprise and readable storage medium | |
| Ramachandran et al. | Document clustering using keyword extraction | |
| CN114970467A (en) | Composition initial draft generation method, device, equipment and medium based on artificial intelligence | |
| CN112308453B (en) | Risk identification model training method, user risk identification method and related devices | |
| CN120805881A (en) | Article duplicate checking method, apparatus, device, readable storage medium and program product |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |