CN113360689B - Image retrieval system, method, related device and computer program product - Google Patents

Abstract

The disclosure provides image retrieval systems, methods, related devices and products, which relate to the field of artificial intelligence, specifically computer vision and deep learning technologies, and can be used in image search scenarios. The specific implementation plan is: the image retrieval system includes a storage device, a management device and a retrieval engine, the storage device is used to provide a database for each user in the user set, and each database stores the user corresponding to the database and the images belonging to the user The index information between the feature information of the target user; the management device is used to manage the index information in the storage device; the search engine is used to determine the target user corresponding to the received image retrieval request, and according to the target index information corresponding to the target user, use the target The user's customized retrieval strategy, processing image retrieval requests, and generating image retrieval results. The disclosure realizes sub-database storage of user data, provides an image retrieval system for multiple users, and improves the flexibility of image retrieval.

Description

Image retrieval system, method, related device and computer program product

technical field

The present disclosure relates to the field of artificial intelligence, in particular to computer vision and deep learning technologies, and in particular to image retrieval systems, methods, devices, electronic equipment, storage media, and computer program products, which can be used in image search scenarios.

Background technique

Image retrieval technology is a technology of image search, and many application platforms need to provide image retrieval functions based on image retrieval technology. At present, each application platform generally develops its own image retrieval system to provide image retrieval services, which requires a lot of human and material resources and a long development cycle.

Contents of the invention

The present disclosure provides an image retrieval system, method, device, electronic equipment, storage medium and computer program product.

According to the first aspect, an image retrieval system is provided, including a storage device, a management device and a retrieval engine, wherein: the storage device is used to provide a database for each user in the user set, and store the database corresponding The index information between the user and the feature information of the image belonging to the user; the management device is used to manage the index information in the storage device; the retrieval engine is used to determine the target user corresponding to the received image retrieval request, and according to The target index information corresponding to the target user uses the target user's customized retrieval strategy to process image retrieval requests and generate image retrieval results.

According to the second aspect, an image retrieval method is provided, including: obtaining an image retrieval request; determining the target user corresponding to the image retrieval request from a user set, wherein each user in the user set has a database, and each database Store the index information between the user corresponding to the database and the feature information of the image belonging to the user; according to the target index information corresponding to the target user, use the target user's customized retrieval strategy to process the image retrieval request and generate the image retrieval result.

According to a third aspect, an image retrieval device is provided, including: an acquisition unit configured to acquire an image retrieval request; a determination unit configured to determine a target user corresponding to an image retrieval request from a user set, wherein the user set Each user in has a database, and each database stores the index information between the user corresponding to the database and the feature information of the image belonging to the user; the retrieval unit is configured to use the target index information corresponding to the target user. The user's customized retrieval strategy, processing image retrieval requests, and generating image retrieval results.

According to a fourth aspect, there is provided an electronic device, comprising: at least one processor; and a memory communicatively connected to at least one processor; wherein, the memory stores instructions executable by at least one processor, and the instructions are processed by at least one processor executed by a processor, so that at least one processor can execute the method described in any implementation manner of the second aspect.

According to a fifth aspect, there is provided a non-transitory computer-readable storage medium storing computer instructions, the computer instructions are used to make a computer execute the method described in any implementation manner of the second aspect.

According to a sixth aspect, a computer program product is provided, including: a computer program, and when the computer program is executed by a processor, the method described in any implementation manner of the second aspect is implemented.

According to the technology of the present disclosure, each user is provided with a database for storing the index information between the user and the feature information of the images belonging to the user, and the user can set a customized retrieval strategy according to his own needs, so as to base the customized retrieval The strategy is used for image retrieval, thereby realizing the sub-database storage of user data, providing an image retrieval system for multiple users, and improving the flexibility of image retrieval.

It should be understood that what is described in this section is not intended to identify key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily understood through the following description.

Description of drawings

The accompanying drawings are used for better understanding of the present disclosure, and do not constitute a limitation to the present disclosure. in:

FIG. 1 is an exemplary system architecture diagram to which an embodiment of the present disclosure can be applied;

FIG. 2 is a schematic structural diagram of an embodiment of an image retrieval system according to the present disclosure;

Fig. 3 is a schematic structural diagram of another embodiment of an image retrieval system according to the present disclosure;

FIG. 4 is a flowchart of an embodiment of an image retrieval method according to the present disclosure;

FIG. 5 is a schematic diagram of an application scenario of the image retrieval method according to this embodiment;

FIG. 6 is a structural diagram of an embodiment of an image retrieval device according to the present disclosure;

FIG. 7 is a schematic structural diagram of a computer system suitable for implementing the embodiments of the present disclosure.

Detailed ways

Exemplary embodiments of the present disclosure are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding, and they should be regarded as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

In the technical solution of the present disclosure, the acquisition, storage and application of the user's personal information involved are in compliance with relevant laws and regulations, and do not violate public order and good customs.

FIG. 1 shows an exemplary architecture 100 to which the image retrieval method and apparatus of the present disclosure can be applied.

As shown in FIG. 1 , a system architecture 100 may include terminal devices 101 , 102 , 103 , a network 104 and a server 105 . The communication connections between the terminal devices 101 , 102 , and 103 constitute a topological network, and the network 104 is used to provide a communication link medium between the terminal devices 101 , 102 , 103 and the server 105 . Network 104 may include various connection types, such as wires, wireless communication links, or fiber optic cables, among others.

The terminal devices 101, 102, and 103 may be hardware devices or software that support network connections for data interaction and data processing. When the terminal devices 101, 102, 103 are hardware, they can be various electronic devices that support network connection, information acquisition, interaction, display, processing and other functions, including but not limited to smartphones, tablet computers, e-book readers, Laptops and desktop computers and more. When the terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above. It can be implemented, for example, as a plurality of software or software modules for providing distributed services, or as a single software or software module. No specific limitation is made here.

The server 105 may be a server that provides various services, for example, a background server that receives an image retrieval request sent by a terminal device and performs image retrieval. An image retrieval system is deployed in the server, and the image retrieval system provides each user with a database for storing index information between the user and the feature information of images belonging to the user, and users can set customized retrieval strategies according to their own needs, Perform image retrieval based on customized retrieval strategies. As an example, server 105 may be a cloud server.

It should be noted that the server may be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When the server is software, it can be implemented as multiple software or software modules (such as software or software modules for providing distributed services), or as a single software or software module. No specific limitation is made here.

It should also be noted that the image retrieval method provided by the embodiments of the present disclosure may be executed by a server, and correspondingly, each part (for example, each unit) included in the image retrieval device may be all set in the server.

It should be understood that the numbers of terminal devices, networks and servers in Fig. 1 are only illustrative. According to the implementation needs, there can be any number of terminal devices, networks and servers. When the electronic device on which the image retrieval method runs does not need to perform data transmission with other electronic devices, the system architecture may only include the electronic device (such as a server or terminal device) on which the image retrieval method runs.

Please refer to FIG. 2. FIG. 2 is a schematic structural diagram of an image retrieval system provided by an embodiment of the present disclosure, wherein the schematic structural diagram includes a storage device 201, a management device 202 and a retrieval engine 203, wherein:

The storage device 201 is configured to provide a database for each user in the user set, and use each database to store index information between the user corresponding to the database and the feature information of images belonging to the user. The users in the user set may be users who provide image retrieval services. As an example, the users in the user set are B (Business, merchant) end users who provide image retrieval services for C (Consumer, consumer) end users.

B-end users may have a large amount of image data. For the image data of B-end users, when acquiring image data, the feature information of the image data can be extracted through the feature extraction service, and the relationship between the user to which the image belongs and the feature information of the image data can be established. Index information between. Wherein, the characteristic information of the image may be multi-type and multi-faceted characteristic information involved in the image data. As an example, the feature information of the image may include feature information representing attributes such as age, category, expression, and body movement of the target object in the image, and feature information representing environment information corresponding to the image.

It can be understood that, for each user in the user set, the user information of the user is stored in the storage device 201, including but not limited to user ID (Identity document, identity identification number), database information corresponding to the user, and user-set self Define retrieval strategies.

The management device 202 is configured to manage index information in the storage device 201 . Index information is closely related to changes in the number of users in the user collection and changes in the number of image data for each user. In this embodiment, the management device 202 can manage the index information in the storage device 201 according to the processing request of the B-end user. As an example, the processing request of the B-end user may be a user's request to create a database, delete a database, or adjust the index information in the database.

The retrieval engine 203 is configured to determine the target user corresponding to the received image retrieval request, process the image retrieval request by using the target user's customized retrieval strategy according to the target index information corresponding to the target user, and generate an image retrieval result.

The image retrieval request may be an image retrieval request sent by the C-end user to the B-end user's image retrieval system, or an image retrieval request sent by the B-end user to the image retrieval system. As an example, the user of the B-end user's application platform (C-end user) sends an image retrieval request to the application platform, and the B-end user's application platform processes the image retrieval request through the image retrieval system. As another example, the user of the application platform of the B-end user publishes an image on the application platform, and the application platform sends an image retrieval request to the search engine to retrieve the legitimacy of the image posted by the C-end user through the preset sensitive image library .

According to the identifier representing the target user carried in the image retrieval request, the target user corresponding to the image retrieval request can be determined from the user set. Furthermore, the target index information corresponding to the target user may be determined from the index information in the storage device 201 . A customized retrieval strategy represents a customized retrieval strategy of a user in the user set. After the user sets a custom retrieval strategy, for the user's image retrieval request, the image retrieval is performed through the user's customized retrieval strategy.

Each user in the user set can customize the feature information of the image to be retrieved, the retrieval logic information, and the feature comparison algorithm to obtain the user's customized retrieval strategy. Specifically, in terms of the feature information concerned with the image to be retrieved, the user can customize the unilateral or multiple feature information involved in the image to be retrieved and concerned with image retrieval; in terms of retrieval logic information , the user can customize specific retrieval logic such as retrieval of the same image, retrieval of similar images, retrieval of the same target object involved in the image, retrieval of similar target objects, etc.; in terms of feature comparison algorithms, users can use well-known feature comparison algorithm, or a feature comparison algorithm independently developed by the user. In this embodiment, each user in the user set can customize multiple image retrieval strategies to meet the retrieval requirements of multiple C-end users served.

In this embodiment, each user is provided with a database for storing the index information between the user and the feature information of images belonging to the user, and the user can set a customized retrieval strategy according to his or her own needs, so that based on the customized retrieval strategy Image retrieval is carried out, thereby realizing the sub-database storage of user data, providing an image retrieval system for multiple users, and improving the flexibility of image retrieval.

In some optional implementation manners of this embodiment, the search engine 203 includes a first search service cluster and a second search service cluster. The first retrieval service cluster is used to distribute image retrieval requests to the second retrieval service cluster corresponding to the target based on the distribution operation determined by the management device according to the index information, and the cluster topology information of the first retrieval service cluster and the second retrieval service cluster At least one target retrieval service for the user; the second retrieval service cluster is used to process the image retrieval request and obtain the retrieval result by using the target user's customized retrieval strategy according to the target index information corresponding to the target user through the at least one target retrieval service; The first retrieval service cluster is further configured to post-process the retrieval results returned by at least one target retrieval service to obtain image retrieval results. Wherein, the storage device is also used to store cluster topology information.

In this implementation, the image retrieval system can provide image retrieval services for one user in the user set through multiple second retrieval services in the second retrieval service cluster; and, each second retrieval service in the second retrieval service cluster The image retrieval service may be provided for multiple users in the user collection. Specifically, the corresponding relationship between the user and the second retrieval service may be determined according to the size of the user's image data and the number of image retrieval requests corresponding to the user. For example, when the size of the user's image data is large and the number of image retrieval requests received by the user's application platform is large, more second retrieval services may be set for the user.

The management device may predetermine the corresponding relationship between the users in the user set and the second retrieval service in the second retrieval service cluster.

After determining the target user corresponding to the image retrieval request, according to the corresponding relationship between the users in the user set and the second retrieval service in the second retrieval service cluster, the above-mentioned executive body can determine at least one second retrieval service corresponding to the target user Service (target retrieval service); according to the cluster topology information of the first retrieval service cluster and the second retrieval service cluster, the route from the first retrieval service in the first retrieval service cluster to the target retrieval service in the second retrieval service cluster can be determined information. In this way, the image retrieval request is distributed to at least one target retrieval service of the target user, so as to provide the target user with the image retrieval service through the at least one target retrieval service.

After obtaining the respective retrieval results of at least one target retrieval service, the first retrieval service in the first retrieval service cluster may perform post-processing such as filtering and fusion on the retrieval results to obtain the final image retrieval result.

In this implementation, the first retrieval service cluster can also be used to perform preprocessing operations such as image verification, user legality verification, and image storage for the received image retrieval request.

In this implementation, the search engine is deployed in a distributed cluster mode, and the first search service cluster and the second search service cluster of the search engine perform their respective duties, cooperate to execute the received image search request, and improve image search efficiency.

In some optional implementation manners of this embodiment, the retrieval engine further includes a feature extraction service, which is used to extract feature information of the image.

In this implementation, the first retrieval service cluster is also used to determine whether to perform feature extraction on the image to be retrieved indicated by the image retrieval request according to the image retrieval request; The service extracts feature information of the image to be retrieved. Wherein, the image retrieval request may include an indication of whether feature extraction is performed on the image to be retrieved indicated by the image retrieval request. When the received image retrieval request indicates the feature information on which the image retrieval is based on the image to be retrieved, feature extraction is no longer performed on the point search image through the feature extraction service. The feature extraction service can be implemented based on a network model with a special diagnosis extraction function, and is used to extract various types and aspects of feature information involved in the image to be retrieved.

In this implementation, the second retrieval service cluster is further used to load the target index information corresponding to the target user through at least one target retrieval service, adopt the customized retrieval strategy of the target user, and combine the feature information in the target index information with the image to be retrieved Match the feature information to get the retrieval result.

In this implementation, the user who sends the image retrieval request can flexibly determine whether to extract the features of the image to be detected based on the feature extraction service provided by the system, or independently select the feature information of the image to be detected, and then the image retrieval system can target the image based on the user's choice The image retrieval request is processed efficiently, which improves the practicability of the image retrieval system.

Please refer to FIG. 3 , which shows a schematic structural diagram of a specific embodiment of an image retrieval system. The image retrieval system includes a storage device 301 , a management device 302 and a retrieval engine 303 including a first retrieval service cluster 3031 , a second retrieval service cluster 3032 and a feature extraction service 3033 . The user may send request information such as creating a database or deleting a database to the management device 302 through the application, and may also send an image retrieval request to the first retrieval service cluster. Moreover, the management device 302 may also receive request information such as adding index information and deleting index information from the administrator and perform corresponding operations.

In some optional implementation manners of this embodiment, the management device 202 is further configured to balance and adjust the quantity of index information loaded by each second retrieval service in at least one target retrieval service.

As an example, B-end user A has a total of 3000 index information, and the second retrieval service corresponding to B-end user b includes three. The 3,000 index information of B-end user b can be evenly loaded to three second retrieval services.

In this implementation, the information processing capability of each second retrieval service in at least one target retrieval service is considered, and the information processing capability of each second retrieval service in at least one target retrieval service is adjusted in a balanced manner based on the manner that the information processing capability is positively correlated with the amount of index information loaded. The number of index information loaded by the second retrieval service.

In this implementation, by balancing and adjusting the amount of index information loaded by the second retrieval service, the image retrieval system can achieve the effect of balancing the load of the second retrieval service in the second retrieval service cluster, and improve the retrieval efficiency of the image retrieval system .

In some optional implementations of this embodiment, the management device 202 is further configured to adjust the number of image retrieval requests processed by the second retrieval service in the second retrieval service cluster. Retrieve the correspondence between the service and the users in the users collection.

As an example, in a unit time, the number of image retrieval requests corresponding to user C at end B is huge, and the management device can increase the number of retrieval services provided for user C at end B on the basis of the original number of second retrieval services for user C at end B. The number of secondary retrieval services.

As yet another example, there is a second retrieval service in the second retrieval service cluster, and the number of image requests of multiple users served by the second retrieval service is relatively large, resulting in a heavy load on the second retrieval service. The management device may migrate part of the index data in the second retrieval service to the second retrieval service with a lighter load.

In this implementation, by adjusting the corresponding relationship between the second retrieval service in the second retrieval service cluster and the users in the user set, the image retrieval system can balance the load of the second retrieval service in the second retrieval service cluster The effect of improving the retrieval efficiency of the image retrieval system.

In some optional implementation manners of this embodiment, the management device 202 may also restart the second retrieval service in the second retrieval service cluster. In response to determining that there is a restarted second retrieval service in the second retrieval service cluster, the management device 202 determines whether the index information loaded in the restarted second retrieval service is invalid; in response to determining that the loaded index information is invalid, updates the restarted second retrieval service Index information in the service; in response to the update of the index information in the restarted second retrieval service, update the status information of the restarted second retrieval service in the cluster topology information to be available.

Specifically, first, the second retrieval service in the second retrieval service cluster can determine the version number of the locally loaded index information and the synchronized messages in the message queue. Wherein, the second retrieval service in the second retrieval service cluster obtains various messages to be executed based on the message queue. Then, the management device determines whether the loaded index information is invalid according to the version number of the locally loaded index information of the second retrieval service. In response to determining the failure, create snapshot information about the synchronized messages in the message queue, fully update the index information in the second retrieval service, and perform subsequent information to be executed based on the snapshot information of the synchronized messages in the message queue Incremental data synchronization; in response to determining that it is not invalid, start incremental data synchronization from the synchronized messages in the message queue. When the management device determines that the data in the restarted second retrieval service has been updated, the status of the second retrieval service in the cluster topology information is updated as enabled.

In this implementation manner, a method for restarting the second retrieval service in the second retrieval service cluster is provided, which can realize rapid restart of the second retrieval service, and further realize rapid response to image retrieval requests.

In some optional implementations of this embodiment, the management device 202 is further configured to generate snapshot information of index information based on a preset interval, and update the index loaded by the second retrieval service in the second retrieval service cluster according to the snapshot information information.

As an example, the management device regularly generates snapshot information for the index information in the storage device, and updates the data version of the index information. Then, the management device broadcasts the snapshot information to all the second retrieval services in the second retrieval service cluster, so as to update the data version of the index information in the second retrieval service.

In this embodiment, based on the snapshot function provided by the management device, the update of the index information loaded in the second retrieval service is implemented. Each second retrieval service processes the image retrieval request based on the loaded index information, and timely updates the index information loaded by each second retrieval service through the snapshot information, thereby improving the accuracy of image retrieval results.

In some optional implementation manners of this embodiment, the management device 202 is further configured to process the index information corresponding to the index processing request according to the received index processing request, so as to process the index information in the storage device; determine the The second retrieval service corresponding to the user corresponding to the processed index information; sending the index processing operation represented by the index processing request to the corresponding second retrieval service through the message queue, so that the second retrieval service executes the index processing operation.

Wherein, the index processing request includes, but is not limited to, an addition request, a deletion request, and a modification request of index information. The index processing request may be a processing request for a single piece of index information, or a processing request for batches of index information.

In this implementation, the image retrieval system provides a real-time processing function of index information to provide users with real-time addition, deletion, and modification operations for index data, and provides real-time and flexibility for the image retrieval system to process information.

In some optional implementations of this embodiment, the management device 202 is also configured to execute a library processing request for the database, adjust index information according to the library processing result, and load index information.

Wherein, the library processing request includes a registration request, a library building request, a library deletion request, a partition creation request, and a partition deletion request.

Specifically, in response to determining that a registration request of a new user is received, the user information of the new user is stored in a storage device. In response to determining that the user's database building request is received, a new user's database is created in the storage device, and the correspondence between the new user and the corresponding database is stored in the user's user information.

In response to determining that the user’s database deletion request has been received, the traffic information corresponding to the database to be deleted is shielded; the partition information and index information of the database to be deleted in the storage device are deleted, and the database deletion request is sent to the second database corresponding to the database to be deleted. The search service, so that the corresponding second search service deletes the index information of the database to be deleted.

In response to determining that the partition creation request from the user is received, determine the attribute information of the partition to be created, including logical partition combination information, index type information, retrieval logic information, and feature matching strategy information; if the created partition contains a new private part, then Create a new segment (segment) for the partition, and automatically assign the corresponding second retrieval service to the partition; when the partition is successfully created, save the index information of the created partition. In this implementation, when the user's database is large in scale, different partitions can be created in the database, and a corresponding second retrieval service is assigned to each partition, so that the index information of the partition can be loaded through the corresponding second retrieval service Perform image retrieval.

In response to determining that the partition deletion request from the user is received, the corresponding partition in the database is deleted, and the corresponding index information is deleted in the corresponding second retrieval service.

In this embodiment, the image retrieval system can process various database processing requests of users, which further improves the practicability of the image retrieval system.

In some optional implementation manners of this embodiment, the management device is further configured to monitor and display status information of the first retrieval service cluster and the second retrieval service cluster. As an example, the management device may determine the state information of each first retrieval service and each second retrieval service from the cluster topology information of the first retrieval service cluster and the second retrieval service cluster.

In this implementation manner, based on the monitoring and display function of the management device for the state information of the first retrieval service cluster and the second retrieval service cluster, the practicability of the image retrieval system is further improved.

In some optional implementations of this embodiment, the storage device 201, the management device 202, and the search engine 203 are all implemented based on open source components, so as to realize privatized deployment.

As an example, the storage device is implemented based on a TiDB (an open source distributed database system) storage system, specifically based on TiKV (an open source distributed transactional key-value pair database) + PD (Placement Driver) in TiDB. The message queue component is implemented based on kafka. Kafka is a distributed message subscription system. Its own status and elections depend on zookeeper. The kafka message system supports features such as partitioning, multiple subscribers, and message re-consumption. The service registration and discovery functions of the management device are implemented based on Etcd. Etcd is essentially a distributed key-value pair storage system, which is usually used for configuration sharing and service discovery. The monitoring and alarm functions of the management device are implemented based on Prometheus (an open source service monitoring system and time series database), clickhouse (column database management system), and grafana (a source data visualization tool). Realizing the above-mentioned devices through open-source components can realize privatized deployment for each user.

The functions of the management device in the image retrieval system can be summarized as: service discovery and registration functions; determine the retrieval route from the first retrieval service in the first retrieval service cluster to the second retrieval service in the second retrieval service cluster, so as to follow the user The function of distributing image retrieval requests to different second retrieval services; the function of storing and managing index information, sending it to the corresponding second retrieval service, and balancing and adjusting the index information loaded by the second retrieval service; the management function of user information; Collection and visualization of retrieval system status information.

Please refer to FIG. 4, which is a flow chart of an image retrieval method provided by an embodiment of the present disclosure, wherein the process 400 includes the following steps:

Step 401, acquire an image retrieval request.

In this embodiment, the execution subject of the image retrieval method (for example, the server in FIG. 1 ) can obtain the image retrieval request.

Wherein, the image retrieval request includes information such as the image to be retrieved and the corresponding B-end user.

Step 402, determine the target user corresponding to the image retrieval request from the user set.

In this embodiment, the execution subject may determine the target user corresponding to the image retrieval request from the user set. Each user in the user set has a database, and each database stores index information between the user corresponding to the database and the feature information of images belonging to the user.

Step 403, according to the target index information corresponding to the target user, adopt the target user's customized retrieval strategy, process the image retrieval request, and generate the image retrieval result.

In this embodiment, the execution subject may process the image retrieval request by using the target user's customized retrieval strategy according to the target index information corresponding to the target user, and generate an image retrieval result.

In some optional implementation manners of this embodiment, the retrieval engine processing the image retrieval request includes a first retrieval service cluster and a second retrieval service cluster. In this implementation, the execution subject executes the above step 203 in the following manner: First, determine the distribution from the first retrieval service in the first retrieval service cluster to the second retrieval service in the second retrieval service cluster according to the target index information Operation; second, based on the split operation and the cluster topology information of the first retrieval service cluster and the second retrieval service cluster, distribute the image retrieval request to at least one target retrieval service corresponding to the target user in the second retrieval service cluster; third , through at least one target retrieval service, according to the target index information, adopt the target user's customized retrieval strategy, process the image retrieval request, and obtain the retrieval result; fourth, post-process the retrieval result returned by at least one target retrieval service, and obtain the image Search Results.

In some optional implementation manners of this embodiment, the retrieval engine further includes a feature extraction service. In this implementation manner, the execution subject may determine whether to perform feature extraction on the image to be retrieved indicated by the image retrieval request according to the image retrieval request. The above-mentioned executive body can perform the above-mentioned third step in the following manner: load the target index information corresponding to the target user through at least one target retrieval service, adopt the customized retrieval strategy of the target user, and combine the feature information in the target index information with the image to be retrieved The feature information is matched to obtain the retrieval result.

In some optional implementation manners of this embodiment, the execution subject may balance and adjust the quantity of index information loaded by each second retrieval service in at least one target retrieval service.

In some optional implementations of this embodiment, the execution subject may also adjust the second retrieval service in the second retrieval service cluster according to the number of image retrieval requests processed by the second retrieval service in the second retrieval service cluster. The corresponding relationship with the users in the user collection.

In some optional implementations of this embodiment, the execution subject may also determine whether the index information loaded in the restarted second retrieval service is invalid in response to determining that there is a restarted second retrieval service in the second retrieval service cluster; In response to determining that the loaded index information is invalid, update the index information in the restarted second retrieval service; in response to the update of the index information in the restarted second retrieval service, update the status information of the restarted second retrieval service in the cluster topology information updated as available.

In some optional implementations of this embodiment, the execution subject may also generate snapshot information of index information based on a preset interval, and update the index information loaded by the second retrieval service in the second retrieval service cluster according to the snapshot information .

In some optional implementations of this embodiment, the execution subject may also process the index information corresponding to the index processing request according to the received index processing request; determine the second search function corresponding to the user corresponding to the processed index information Service: sending the index processing operation represented by the index processing request to the corresponding second retrieval service through the message queue, so that the second retrieval service executes the index processing operation.

In some optional implementations of this embodiment, the execution subject may also execute the library processing request for the database, and adjust the index information according to the library processing result, and the index information loaded in the second search service corresponding to the library processing result. Index information.

In some optional implementations of this embodiment, the execution subject may also monitor and display status information of the first retrieval service cluster and the second retrieval service cluster.

In this embodiment, each user is provided with a database for storing the index information between the user and the feature information of images belonging to the user, and the user can set a customized retrieval strategy according to his or her own needs, so that based on the customized retrieval strategy Image retrieval is carried out, thereby realizing the sub-database storage of user data, providing an image retrieval system for multiple users, and improving the flexibility of image retrieval.

Continue to refer to FIG. 5 , which is a schematic diagram 500 of an application scenario of the image retrieval method according to this embodiment. In the application scenario of FIG. 5 , an image retrieval system is deployed in the server 501 . The image retrieval system provides one-to-one corresponding databases 505, 506, 507 for the three B-end users 502, 503, 504. Each database stores index information between a user corresponding to the database and feature information of images belonging to the user. The C-end user 508 sends an image retrieval request to the server 501 through the terminal device 509 . After acquiring the image retrieval request, the server 501 determines from the user set that the target user corresponding to the image retrieval request is 503 . Then, according to the target index information corresponding to the target user 503, the target user's customized retrieval strategy is adopted to process the image retrieval request and generate an image retrieval result.

Continuing to refer to FIG. 6 , as an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of an image retrieval device. This device embodiment corresponds to the method embodiment shown in FIG. 5 , and the device can specifically Used in various electronic equipment.

As shown in FIG. 6 , the image retrieval device includes: an acquisition unit 601 configured to acquire an image retrieval request; a determination unit 602 configured to determine the target user corresponding to the image retrieval request from a user set, wherein the user in the user set Each user has a database, and each database stores the index information between the user corresponding to the database and the feature information of the image belonging to the user; the retrieval unit 603 is configured to use the target user according to the target index information corresponding to the target user. Customized retrieval strategies, process image retrieval requests, and generate image retrieval results.

In some optional implementations of this embodiment, the retrieval engine processing the image retrieval request includes a first retrieval service cluster and a second retrieval service cluster; and the retrieval unit is further configured to: A split operation from the first retrieval service in the service cluster to the second retrieval service in the second retrieval service cluster; based on the split operation and the cluster topology information of the first retrieval service cluster and the second retrieval service cluster, the image retrieval request is distributed to At least one target retrieval service corresponding to the target user in the second retrieval service cluster; through at least one target retrieval service, according to the target index information, using the target user's customized retrieval strategy, processing the image retrieval request, and obtaining the retrieval result; for at least one The retrieval results returned by the target retrieval service are post-processed to obtain image retrieval results.

In some optional implementations of this embodiment, the retrieval engine further includes a feature extraction service; and the image retrieval device further includes: a feature extraction unit (not shown in the figure), configured to determine whether to performing feature extraction on the image to be retrieved indicated by the image retrieval request; and the retrieval unit is further configured to: in response to determining that the retrieval request represents feature extraction on the image to be retrieved, extract feature information of the image to be retrieved through a feature extraction service; through at least one target The retrieval service loads the target index information corresponding to the target user, adopts the target user's customized retrieval strategy, matches the characteristic information in the target index information with the characteristic information of the image to be retrieved, and obtains the retrieval result.

In some optional implementations of this embodiment, the above-mentioned device further includes: a first adjustment unit (not shown in the figure), configured to evenly adjust at least one target retrieval service loaded by each second retrieval service The number of index information.

In some optional implementations of this embodiment, the above-mentioned device further includes: a second adjustment unit (not shown in the figure), configured to retrieve images processed by the second retrieval service in the second retrieval service cluster The number of requests adjusts the correspondence between the second retrieval service in the second retrieval service cluster and the users in the user collection.

In some optional implementations of this embodiment, the above device further includes: a restart unit (not shown in the figure), configured to determine to restart the second retrieval service in response to determining that there is a restarted second retrieval service in the second retrieval service cluster. Whether the index information loaded in the second retrieval service is invalid; in response to determining that the loaded index information is invalid, update the index information in the restarted second retrieval service; in response to completing the update of the index information in the restarted second retrieval service, update The status information of the restarted second retrieval service in the cluster topology information is updated to be available.

In some optional implementations of this embodiment, the above device further includes: an update unit (not shown in the figure), configured to generate snapshot information of index information based on a preset interval duration, and update the second index information according to the snapshot information. The index information loaded by the second retrieval service in the retrieval service cluster.

In some optional implementation manners of this embodiment, the foregoing apparatus further includes: a first processing unit (not shown in the figure), configured to process index information corresponding to the index processing request according to the received index processing request; Determine the second retrieval service corresponding to the user corresponding to the processed index information; send the index processing operation represented by the index processing request to the corresponding second retrieval service through the message queue, so that the second retrieval service executes the index processing operation .

In some optional implementations of this embodiment, the above-mentioned apparatus further includes: a second processing unit (not shown in the figure), configured to execute a library processing request for the database, and adjust index information according to the library processing result , and the index information loaded in the second search service corresponding to the library processing result.

In some optional implementations of this embodiment, the above apparatus further includes: a monitoring and display unit (not shown in the figure), configured to monitor and display the status information of the first retrieval service cluster and the second retrieval service cluster.

In this embodiment, each user is provided with a database for storing the index information between the user and the feature information of images belonging to the user, and the user can set a customized retrieval strategy according to his or her own needs, so that based on the customized retrieval strategy Image retrieval is carried out, thereby realizing the sub-database storage of user data, providing an image retrieval system for multiple users, and improving the flexibility of image retrieval.

According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, the electronic device includes: at least one processor; and a memory connected in communication with the at least one processor; wherein, the memory stores information executable by the at least one processor. An instruction, the instruction is executed by at least one processor, so that the at least one processor can implement the image retrieval method described in any of the above embodiments when executed.

According to an embodiment of the present disclosure, the present disclosure also provides a readable storage medium, the readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to implement the image retrieval method described in any of the above-mentioned embodiments. .

An embodiment of the present disclosure provides a computer program product, when the computer program is executed by a processor, the image retrieval method described in any of the foregoing embodiments can be implemented.

FIG. 7 shows a schematic block diagram of an example electronic device 700 that may be used to implement embodiments of the present disclosure. Electronic device is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are by way of example only, and are not intended to limit implementations of the disclosure described and/or claimed herein.

As shown in FIG. 7, the device 700 includes a computing unit 701 that can execute according to a computer program stored in a read-only memory (ROM) 702 or loaded from a storage unit 708 into a random-access memory (RAM) 703. Various appropriate actions and treatments. In the RAM 703, various programs and data necessary for the operation of the device 700 can also be stored. The computing unit 701 , ROM 702 , and RAM 703 are connected to each other through a bus 704 . An input/output (I/O) interface 705 is also connected to the bus 704 .

Multiple components in the device 700 are connected to the I/O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a magnetic disk, an optical disk, etc. ; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, and the like. The communication unit 709 allows the device 700 to exchange information/data with other devices over a computer network such as the Internet and/or various telecommunication networks.

The computing unit 701 may be various general-purpose and/or special-purpose processing components having processing and computing capabilities. Some examples of computing units 701 include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processing processor (DSP), and any suitable processor, controller, microcontroller, etc. The calculation unit 701 executes various methods and processes described above, such as an image retrieval method. For example, in some embodiments, the image retrieval method may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 708 . In some embodiments, part or all of the computer program may be loaded and/or installed on the device 700 via the ROM 702 and/or the communication unit 709 . When the computer program is loaded into RAM 703 and executed by computing unit 701, one or more steps of the image retrieval method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the image retrieval method in any other suitable manner (for example, by means of firmware).

Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips Implemented in a system of systems (SOC), load programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpreted on a programmable system including at least one programmable processor, the programmable processor Can be special-purpose or general-purpose programmable processor, can receive data and instruction from storage system, at least one input device, and at least one output device, and transmit data and instruction to this storage system, this at least one input device, and this at least one output device an output device.

Program codes for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special purpose computer, or other programmable data processing devices, so that the program codes, when executed by the processor or controller, make the functions/functions specified in the flow diagrams and/or block diagrams Action is implemented. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer discs, hard drives, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), optical fiber, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

To provide for interaction with the user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user. ); and a keyboard and pointing device (eg, a mouse or a trackball) through which a user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and can be in any form (including Acoustic input, speech input or, tactile input) to receive input from the user.

The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., as a a user computer having a graphical user interface or web browser through which a user can interact with embodiments of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system can be interconnected by any form or medium of digital data communication, eg, a communication network. Examples of communication networks include: Local Area Network (LAN), Wide Area Network (WAN) and the Internet.

A computer system may include clients and servers. Clients and servers are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the management difficulties existing in traditional physical host and virtual private server (VPS, Virtual Private Server) services large and weak business scalability; it can also be a server of a distributed system, or a server combined with a blockchain.

According to the technical solutions of the embodiments of the present disclosure, each user is provided with a database for storing the index information between the user and the feature information of images belonging to the user, and the user can set a customized retrieval strategy according to his own needs, based on Customized retrieval strategies are used for image retrieval, thereby realizing the sub-database storage of user data, providing an image retrieval system for multiple users, and improving the flexibility of image retrieval.

It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, the steps described in the present disclosure may be executed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution provided by the present disclosure can be achieved, no limitation is imposed herein.

The specific implementation manners described above do not limit the protection scope of the present disclosure. It should be apparent to those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made depending on design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present disclosure shall be included within the protection scope of the present disclosure.

Claims (24)

1. An image retrieval system comprising storage means, management means, and a retrieval engine, wherein:

the storage device is used for providing a database for each user in the user set, and storing index information between the user corresponding to the database and the characteristic information of the image belonging to the user through each database, wherein the users in the user set are merchant users providing image retrieval service for consumer users;

the management device is used for managing the index information in the storage device;

and the retrieval engine is used for determining a target user corresponding to an image retrieval request sent by a consumer user from the user set, and processing the image retrieval request by adopting a customized retrieval strategy of the target user according to target index information corresponding to the target user to generate an image retrieval result.

2. The system of claim 1, wherein the search engine comprises a first search service cluster and a second search service cluster; and

the first retrieval service cluster is configured to distribute the image retrieval request to at least one target retrieval service corresponding to the target user in the second retrieval service cluster based on the offloading operation determined by the management device according to the index information and the cluster topology information of the first retrieval service cluster and the second retrieval service cluster, and perform post-processing on a retrieval result returned by the at least one target retrieval service to obtain an image retrieval result;

the second retrieval service cluster is used for processing the image retrieval request by the at least one target retrieval service according to the target index information corresponding to the target user and by adopting a customized retrieval strategy of the target user to obtain a retrieval result; and

the storage device is further configured to store the cluster topology information.

3. The system of claim 2, wherein the search engine further comprises a feature extraction service; and

the first retrieval service cluster is further used for determining whether to perform feature extraction on the image to be retrieved indicated by the image retrieval request according to the image retrieval request;

performing feature extraction on the image to be retrieved in response to the retrieval request characterization, and extracting feature information of the image to be retrieved through the feature extraction service; and

the second retrieval service cluster is further configured to load target index information corresponding to the target user through the at least one target retrieval service, and match feature information in the target index information with feature information of the image to be retrieved by using a customized retrieval policy of the target user to obtain a retrieval result.

4. The system of claim 3, wherein the management device is further configured to balance the amount of index information loaded by each second search service of the at least one target search service.

5. The system of claim 2, wherein the management device is further configured to adjust a correspondence between a second search service in the second search service cluster and a user in the user set according to the number of image search requests processed by the second search service in the second search service cluster.

6. The system of claim 2, wherein the management apparatus is further configured to determine whether index information loaded in a restarted second search service is invalid in response to determining that the restarted second search service exists in the second search service cluster; updating the index information in the restarted second retrieval service in response to determining that the loaded index information is invalid; and updating the state information of the restarting second retrieval service in the cluster topology information to be available in response to the completion of updating of the index information in the restarting second retrieval service.

7. The system according to claim 2, wherein the management device is further configured to generate snapshot information of the index information based on a preset interval duration, and update the index information loaded by the second search service in the second search service cluster according to the snapshot information.

8. The system according to claim 2, wherein the management device is further configured to process, according to the received index processing request, index information corresponding to the index processing request; determining a second retrieval service corresponding to the user corresponding to the processed index information; and sending the index processing operation represented by the index processing request to a corresponding second retrieval service through a message queue, so that the second retrieval service executes the index processing operation.

9. The system according to claim 2, wherein the management device is further configured to execute a library processing request for the database, and adjust the index information and the index information loaded in the second search service corresponding to the library processing result according to the library processing result.

10. The system of claim 2, wherein the management device is further configured to monitor and display status information of the first search service cluster and the second search service cluster.

11. The system of any of claims 1-10, wherein the storage device, the management device, and the search engine are each implemented based on an open source component to enable privatized deployment.

12. An image retrieval method comprising:

acquiring an image retrieval request sent by a consumer user;

determining a target user corresponding to the image retrieval request from a user set, wherein each user in the user set is provided with a database, each database stores index information between the user corresponding to the database and the characteristic information of the image belonging to the user, and the user in the user set is a merchant user providing image retrieval service for a consumer user;

and processing the image retrieval request by adopting a customized retrieval strategy of the target user according to the target index information corresponding to the target user to generate an image retrieval result.

13. The method of claim 12, wherein a search engine processing the image search request comprises a first search service cluster and a second search service cluster; and

the processing the image retrieval request according to the target index information corresponding to the target user and the customized retrieval strategy of the target user to generate an image retrieval result comprises the following steps:

determining the distribution operation from a first retrieval service in a first retrieval service cluster to a second retrieval service in a second retrieval service cluster according to the target index information;

distributing the image retrieval request to at least one target retrieval service corresponding to the target user in the second retrieval service cluster based on the offloading operation and cluster topology information of the first retrieval service cluster and the second retrieval service cluster;

processing the image retrieval request by the at least one target retrieval service according to the target index information and by adopting a customized retrieval strategy of the target user to obtain a retrieval result;

and carrying out post-processing on the retrieval result returned by the at least one target retrieval service to obtain an image retrieval result.

14. The method of claim 13, wherein the search engine further comprises a feature extraction service; and

further comprising: determining whether to perform feature extraction on the image to be retrieved indicated by the image retrieval request according to the image retrieval request; and

the processing the image retrieval request by the at least one target retrieval service according to the target index information and by adopting the customized retrieval strategy of the target user to obtain a retrieval result comprises:

performing feature extraction on the image to be retrieved in response to the retrieval request characterization, and extracting feature information of the image to be retrieved through the feature extraction service;

and loading target index information corresponding to the target user through the at least one target retrieval service, and matching the characteristic information in the target index information with the characteristic information of the image to be retrieved by adopting a customized retrieval strategy of the target user to obtain a retrieval result.

15. The method of claim 14, further comprising:

and balancing and adjusting the quantity of the index information loaded by each second retrieval service in the at least one target retrieval service.

16. The method of claim 13, further comprising:

and adjusting the corresponding relation between the second retrieval service in the second retrieval service cluster and the users in the user set according to the number of the image retrieval requests processed by the second retrieval service in the second retrieval service cluster.

17. The method of claim 13, further comprising:

in response to determining that there is a restarted second retrieval service in the second retrieval service cluster, determining whether index information loaded in the restarted second retrieval service is invalid;

updating the index information in the restarted second retrieval service in response to determining that the loaded index information is invalid;

and updating the state information of the restarted second retrieval service in the cluster topology information to be available in response to the completion of updating of the index information in the restarted second retrieval service.

18. The method of claim 13, further comprising:

and generating snapshot information of the index information based on a preset interval duration, and updating the index information loaded by the second retrieval service in the second retrieval service cluster according to the snapshot information.

19. The method of claim 13, further comprising:

processing index information corresponding to the index processing request according to the received index processing request; determining a second retrieval service corresponding to the user corresponding to the processed index information; and sending the index processing operation represented by the index processing request to a corresponding second retrieval service through a message queue, so that the second retrieval service executes the index processing operation.

20. The method of claim 13, further comprising:

and executing a library processing request aiming at the database, and adjusting the index information and the index information loaded in the second retrieval service corresponding to the library processing result according to the library processing result.

21. The method of claim 13, further comprising:

and monitoring and displaying the state information of the first retrieval service cluster and the second retrieval service cluster.

22. An image retrieval apparatus comprising:

an acquisition unit configured to acquire an image retrieval request issued by a consumer user;

the determining unit is configured to determine a target user corresponding to the image retrieval request from a user set, wherein each user in the user set is provided with a database, each database stores index information between the user corresponding to the database and the feature information of the image belonging to the user, and the users in the user set are merchant users providing image retrieval services for consumer users;

and the retrieval unit is configured to process the image retrieval request by adopting a customized retrieval strategy of the target user according to the target index information corresponding to the target user, and generate an image retrieval result.

23. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein,

the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 12-21.

24. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 12-21.

Images