CN121433777A - Data processing methods, apparatuses, electronic devices, storage media, and software products - Google Patents

Abstract

The present application relates to a data processing method, apparatus, electronic device, computer readable storage medium and computer program product. The method comprises the steps of displaying an interface of a target application, responding to triggering operation of the target application, obtaining a corresponding target data acquisition mode according to the target application, wherein the target data acquisition mode is used for indicating a range for data acquisition of the interface of the target application, and carrying out data acquisition on interface data of the target application according to the target data acquisition mode to obtain target data. The target data acquisition mode corresponding to the target application can be adopted to acquire the required target data, so that the data corresponding to the data acquisition range is acquired for different applications, the unnecessary data is prevented from being acquired by the data, dirty data is prevented from being introduced, and the flexible acquisition of the required data is realized.

Description

Data processing method, apparatus, electronic device, storage medium, and program product

Technical Field

The present application relates to the field of computer technology, and in particular, to a data processing method, apparatus, electronic device, computer readable storage medium, and computer program product.

Background

With the popularity of electronic devices, more and more users use electronic devices. Applications of various functions are provided on the electronic device, and users want to acquire data corresponding to each application for different applications. In the conventional technology, the desired data cannot be flexibly obtained for different applications.

Disclosure of Invention

The embodiment of the application provides a data processing method, a data processing device, computer equipment, a computer readable storage medium and a computer program product, which can flexibly acquire required data according to different applications.

In a first aspect, the present application provides a data processing method, the method comprising:

Displaying an interface of a target application;

Responding to the triggering operation aiming at the target application, and acquiring a corresponding target data acquisition mode according to the target application, wherein the target data acquisition mode is used for indicating the range of data acquisition on an interface of the target application;

and carrying out data acquisition on the interface data of the target application according to the target data acquisition mode to obtain target data.

In a second aspect, the present application also provides a data processing apparatus, the apparatus comprising:

the display module is used for displaying the interface of the target application;

the system comprises a mode acquisition module, a mode control module and a mode control module, wherein the mode acquisition module is used for responding to the triggering operation aiming at the target application and acquiring a corresponding target data acquisition mode according to the target application;

And the data acquisition module is used for carrying out data acquisition on the interface data of the target application according to the target data acquisition mode to obtain target data.

In a third aspect, the application also provides an electronic device comprising a memory storing a computer program and a processor implementing the method steps of the first aspect when executing the computer program.

In a fourth aspect, the present application also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method steps of the first or second aspects.

In a fifth aspect, the application also provides a computer program product comprising a computer program which, when executed by a processor, implements the method steps of the first aspect.

The data processing method, the device, the electronic equipment, the computer readable storage medium and the computer program product display an interface of a target application, respond to triggering operation aiming at the target application, acquire a corresponding target data acquisition mode according to the target application, and the target data acquisition mode is used for indicating a range for data acquisition of the interface of the target application, and acquire the target data by acquiring the interface data of the target application according to the data acquisition range defined by the target data acquisition mode. The target data acquisition mode corresponding to the target application can be adopted to acquire the required target data, so that the data corresponding to the data acquisition range is acquired for different applications, the unnecessary data is prevented from being acquired by the data, dirty data is prevented from being introduced, and the flexible acquisition of the required data is realized.

Drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the related art, the drawings that are needed in the description of the embodiments of the present application or the related technologies will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and other related drawings may be obtained according to these drawings without inventive effort to those of ordinary skill in the art.

FIG. 1 is a schematic diagram of a computing system in one embodiment;

FIG. 2 is a schematic diagram of a data processing system in one embodiment;

FIG. 3 is a schematic diagram of a data processing system in another embodiment;

FIG. 4 is a flow diagram of a data processing method in one embodiment;

FIG. 5A is a schematic diagram of a three finger swipe trigger data acquisition function in one embodiment;

FIG. 5B is a schematic diagram of a voice assistant triggering a data acquisition function in one embodiment;

FIG. 6 is a schematic diagram of configuration parameter fields of a data acquisition mode configuration interface in one embodiment;

FIG. 7 is a schematic diagram of a page view of a news application in one embodiment;

FIG. 8A is a schematic diagram of a data acquisition mode configuration interface of a camera application in one embodiment;

FIG. 8B is a schematic diagram of a data acquisition range of a camera application in one embodiment;

FIG. 8C is a diagram illustrating the overall interface screenshot recognition results of a camera application in one embodiment;

FIG. 8D is a diagram illustrating a captured interface screenshot recognition result of a camera application in one embodiment;

FIG. 9 is a schematic diagram of a data collection scope of a media application in one embodiment;

FIG. 10 is a schematic diagram of a data acquisition range of a shooting application in one embodiment;

FIG. 11 is a schematic diagram of a data collection scope of a social application in one embodiment;

FIG. 12 is a schematic diagram of data acquisition ranges for an e-commerce application in one embodiment;

FIG. 13 is a schematic diagram of an application framework of a data processing method in one embodiment;

FIG. 14 is a diagram of an application framework for gathering text data in one embodiment;

FIG. 15 is a schematic diagram of an application framework for capturing images in one embodiment;

fig. 16 is a block diagram showing the structure of a data processing apparatus in one embodiment.

Detailed Description

The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.

FIG. 1 depicts a block diagram of an example computing system 100 executing a visual or multimodal search service, according to an example embodiment of the application. Computing system 100 includes a user computing system 110, a server computing system 130, and/or a third party computing system 150 communicatively coupled by a network 160.

The user computing system 110 may include any type of computing device, such as, for example, a personal computing device (e.g., a laptop computer or desktop computer), a mobile computing device (e.g., a smart phone or tablet computer), a game console or controller, a wearable computing device (e.g., a smart watch or smart glasses, etc.), an embedded computing device, or any other type of computing device.

The user computing system 110 includes one or more processors 112 and memory 114. The one or more processors 112 may be any suitable processing device (e.g., a processor core, microprocessor, ASIC (Application SPECIFIC INTEGRATED Circuit), FPGA (Field Programmable GATE ARRAY ), controller, microcontroller, etc.) and may be one processor or multiple processors operatively connected. Memory 114 may include one or more non-transitory computer-readable storage media such as RAM (Random Access Memory ), ROM (Read-Only Memory), EEPROM (ELECTRICALLY ERASABLE PROGRAMMABLE READ-Only Memory ), EPROM (Erasable Programmable Read-Only Memory), flash Memory devices, magnetic disks, and the like, as well as combinations thereof. Memory 114 may store data 116 and instructions 118 that are executed by processor 112 to cause user computing system 110 to perform operations.

In some implementations, the user computing system 110 may store or include one or more machine learning models 120. For example, the machine learning model 120 may be or otherwise include various machine learning models, such as a neural network (e.g., a deep neural network) or other types of machine learning models, including nonlinear models and/or linear models. The neural network may include a feed forward neural network, a recurrent neural network (e.g., a long and short term memory recurrent neural network), a convolutional neural network, or other form of neural network.

In some implementations, one or more machine learning models 120 may be received from the server computing system 130 over the network 160, stored in the memory 114 of the user computing system 110, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing system 110 may implement multiple parallel instances of a single machine learning model 120 (e.g., to perform parallel machine learning model processing across multiple instances of input data and/or detected features).

Additionally or alternatively, one or more machine learning models 140 may be included in the server computing system 130 or otherwise stored and implemented by the server computing system 130, the server computing system 130 in communication with the user computing system 110 according to a client/server relationship. For example, the machine learning model 140 may be implemented by the server computing system 130 as part of a web service (e.g., viewfinder service, visual search service, image processing service, environmental computing service, and/or overlay application service). Accordingly, one or more machine learning models 120 may be stored and implemented at the user computing system 110 and/or one or more machine learning models 140 may be stored and implemented at the server computing system 130.

The machine learning model 120 or 140 may include one or more generative models, one or more object detection models, one or more segmentation models, one or more classification models, one or more embedding models, one or more semantic analysis models, and/or one or more search engines, among others.

The one or more generation models may be used to process the display data and/or the one or more processing outputs to generate natural language output (e.g., natural language output including additional information about the display data and/or entities associated with data depicted in the displayed content), generate images, and/or other model-generated media content items. For example, one or more web resources may be accessed and processed to generate a summary of a particular topic. One or more object detection models may be used to perform object detection in the display data. One or more segmentation models may be used to segment objects and/or text segments from the displayed content. One or more classification models may be used to perform object classification, image classification, entity classification, format classification, emotion classification, and/or other classification tasks. One or more embedding models may be used to embed portions and/or all of the display data. The embedding may then be utilized to search for similar objects and/or text, classification, grouping, and/or compression. The semantic analysis model may be used to process the display data to generate semantic outputs that describe understanding of the display data with respect to topic understanding, scene understanding, focus, pattern recognition, application understanding, and/or one or more other semantic outputs.

One or more search engines may process the display data, portions of the display data, and/or one or more machine learning model outputs to determine one or more search results. The one or more search results may include web pages, images, text, video, and/or other data. The search results may be determined based on feature mapping, feature matching, embedded searching, metadata searching, tag searching, clustering, and/or other search techniques. The search results may be determined based on query intent classification, search result classification, and/or entity classification. The output of the model and/or the search results may be sent back to the user computing device for provision to the user through one or more user interface elements generated and provided by the visual search interface.

The user computing system 110 may also include one or more user input components 122 that receive user input. For example, the user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that is sensitive to touch by a user input object (e.g., a finger or stylus). The touch sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a conventional keyboard, or other device by which a user may provide user input.

In some implementations, the user computing system 110 can store and/or provide one or more user interfaces 124, which user interfaces 124 can be associated with one or more applications. The one or more user interfaces 124 may be configured to receive input and/or provide data for display (e.g., image data, text data, audio data, one or more user interface elements, an augmented reality experience, a virtual reality experience, and/or other data for display). The user interface 124 may be associated with one or more other computing systems (e.g., the server computing system 130 and/or the third party computing system 150). User interface 124 may include a viewfinder interface, a search interface, a generative model interface, a social media interface, a media content gallery interface, and the like.

The user computing system 110 may include one or more sensors 126 and/or receive data from one or more sensors 126. The one or more sensors 126 may be housed in a housing component that houses the one or more processors 112, the memory 114, and/or one or more hardware components that may store and/or cause execution of one or more software packages. The one or more sensors 126 may include one or more image sensors (e.g., cameras), one or more radar sensors, one or more audio sensors (e.g., microphones), one or more inertial sensors (e.g., inertial measurement units), one or more biological sensors (e.g., heart rate sensors, pulse sensors, retinal sensors, and/or fingerprint sensors), one or more infrared sensors, one or more location sensors (e.g., global positioning system, GPS), one or more touch sensors (e.g., conductive touch sensors and/or mechanical touch sensors), and/or one or more other sensors. One or more sensors may be utilized to obtain data associated with the user environment (e.g., an image of the user environment, a record of the environment, and/or a location of the user).

The user computing system 110 may include a user computing device 111 and/or be part of the user computing device 111. User computing device 111 may include a mobile computing device (e.g., a smart phone or tablet computer), a desktop computer, a laptop computer, a smart wearable device, and/or a smart appliance or aircraft or vehicle-mounted device, among others. Additionally and/or alternatively, the user computing system 110 may obtain data from one or more user computing devices 111 and/or generate data using one or more user computing devices 111. For example, image data depicting the environment may be captured with a camera of a smartphone, and/or data provided to the user may be tracked and/or processed with an overlay application of the user computing device 111. Similarly, data about the user and/or about the user's environment may be obtained using one or more sensors associated with the smart wearable device (e.g., image data may be obtained using a camera housed in the user's smart glasses). Additionally and/or alternatively, data may be obtained and uploaded from other user devices that may be dedicated to data acquisition or generation.

The server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 may be any suitable processing device (e.g., a processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be one processor or operatively connected multiple processors. Memory 134 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, and the like, as well as combinations thereof. Memory 134 may store data 136 and instructions 138 that are executed by processor 132 to cause server computing system 130 to perform operations.

In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. Where the server computing system 130 includes multiple server computing devices, such server computing devices may operate in accordance with a sequential computing architecture, a parallel computing architecture, or some combination thereof.

As described above, the server computing system 130 may store or otherwise include one or more machine learning models 140. For example, the machine learning model 140 may be or may otherwise include various machine learning models. Example machine learning models include neural networks or other multi-layer nonlinear models. Example neural networks include feed forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks.

Additionally and/or alternatively, the server computing system 130 can include a search engine 142 and/or be communicatively connected with the search engine 142, which can be used to crawl one or more databases (and/or resources). Search engine 142 may process data from user computing system 110, server computing system 130, and/or third party computing system 150 to determine one or more search results associated with the input data. Search engine 142 may perform term-based searches, tag-based searches, boolean searches, image searches, embedded-based searches (e.g., nearest neighbor searches), multimodal searches, and/or one or more other search techniques.

The server computing system 130 may store and/or provide one or more user interfaces 144 for obtaining input data and/or providing output data to one or more users. The one or more user interfaces 144 may include one or more user interface elements that may include input fields, navigation tools, content tiles, selectable tiles, widgets, data display carousel, dynamic animations, information popup windows, image enhancements, text-to-speech, speech-to-text, augmented reality, virtual reality, feedback loops, and/or other interface elements.

The user computing system 110 and/or the server computing system 130 may train the machine learning model 120 and/or 140 via interaction with a third party computing system 150 communicatively coupled via a network 160. The third party computing system 150 may be separate from the server computing system 130 or may be part of the server computing system 130. Alternatively and/or additionally, the third party computing system 150 may be associated with one or more network resources, one or more network platforms, one or more other users, and/or one or more contexts.

The third party computing system 150 may include one or more processors 152 and memory 154. The one or more processors 152 may be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, micro-processor)

Controller, etc.) and may be one processor or a plurality of processors operatively connected. Memory 154 may include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, and the like, as well as combinations thereof. The memory 154 may store data 156 and instructions 158 that are executed by the processor 152 to cause the third party computing system 150 to perform operations. In some implementations, the third party computing system 150 includes or is otherwise implemented by one or more server computing devices.

The network 160 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the internet), or some combination thereof, and may include any number of wired or wireless links. In general, communication over the network 180 may be performed via any type of wired and/or wireless connection using a variety of communication protocols (e.g., TCP/IP, HTTP, SMTP, FTP), coding or formats (e.g., HTML, XML), and/or protection schemes (e.g., VPN, secure HTTP, SSL).

The machine learning model described in the present disclosure may be used for various tasks, applications, and/or use cases.

In some implementations, the input to the machine learning model of the present application can be image data. The machine learning model may process the image data to generate an output. As one example, the machine learning model may process the image data to generate an image recognition output (e.g., recognition of the image data, potential embedding of the image data, encoded representation of the image data, hash of the image data, etc.). As another example, the machine learning model may process the image data to generate an image segmentation output. As another example, the machine learning model may process image data to generate an image classification output. As another example, the machine learning model may process the image data to generate an image data modification output (e.g., a change in the image data, etc.). As another example, the machine learning model may process the image data to generate an encoded image data output (e.g., an encoded and/or compressed representation of the image data, etc.). As another example, the machine learning model may process image data to generate an upgraded image data output. As another example, the machine learning model may process image data to generate a prediction output.

In some implementations, the input of the machine learning model of the present disclosure can be text or natural language data. The machine learning model may process text or natural language data to generate an output. As one example, the machine learning model may process natural language data to generate a language encoded output. As another example, the machine learning model may process text or natural language data to generate a potential text-embedded output. As another example, the machine learning model may process text or natural language data to generate a translation output. As another example, the machine learning model may process text or natural language data to generate a classification output. As another example, the machine learning model may process text or natural language data to generate a text segmentation output. As another example, the machine learning model may process text or natural language data to generate semantic intent output. As another example, the machine learning model may process text or natural language data to generate an upgraded text or natural language output (e.g., text or natural language data of higher quality than the input text or natural language, etc.). As another example, the machine learning model may process text or natural language data to generate a predictive output.

In some implementations, the input to the machine learning model of the present disclosure can be speech data. The machine learning model may process the speech data to generate an output. As one example, the machine learning model may process speech data to generate a speech recognition output. As another example, the machine learning model may process speech data to generate speech translation output. As another example, the machine learning model may process speech data to generate potential embedded outputs. As another example, the machine learning model may process the speech data to generate an encoded speech output (e.g., an encoded and/or compressed representation of the speech data, etc.). As another example, the machine learning model may process the speech data to generate an upgraded speech output (e.g., speech data of higher quality than the input speech data, etc.). As another example, the machine learning model may process the speech data to generate a textual representation output (e.g., a textual representation of the input speech data, etc.). As another example, the machine learning model may process speech data to generate a prediction output.

In some implementations, the input to the machine learning model of the present disclosure can be sensor data. The machine learning model may process the sensor data to generate an output. As one example, the machine learning model may process the sensor data to generate an identification output. As another example, the machine learning model may process the sensor data to generate a prediction output. As another example, the machine learning model may process the sensor data to generate a classification output. As another example, the machine learning model may process the sensor data to generate a segmented output. As another example, the machine learning model may process the sensor data to generate a segmented output. As another example, the machine learning model may process the sensor data to generate a visual output. As another example, the machine learning model may process the sensor data to generate a diagnostic output. As another example, the machine learning model may process the sensor data to generate a detection output.

In some cases, the input includes visual data and the task is a computer visual task. In some cases, pixel data including one or more images is input, and the task is an image processing task. For example, an image processing task may be an image classification, wherein the output is a set of scores, each score corresponding to a different object class and representing a likelihood that one or more images depict an object belonging to the object class.

The user computing system 110 may include a plurality of applications (e.g., applications 1 through N). Each application may include its own respective machine learning library and machine learning model. For example, each application may include a machine learning model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, browser applications, and the like. In some implementations, each application may use an API (e.g., a public API across all applications) to communicate with the central intelligence layer (and the models stored therein).

The central intelligence layer may include a plurality of machine learning models. For example, a respective machine learning model (e.g., model) may be provided for each application, and managed by a central intelligent layer. In other implementations, two or more applications may share a single machine learning model. For example, in some implementations, the central intelligence layer can provide a single model (e.g., a single model) for all applications. In some implementations, the central intelligence layer is included within or otherwise implemented by an operating system of the computing system 1100.

The central intelligence layer may communicate with the central device data layer. The central device data layer may be a centralized data store of the computing system 100. The central device data layer may communicate with many other components of the computing device, such as, for example, one or more sensors, a context manager, a device status component, and/or additional components. In some implementations, the central device data layer may communicate with each device component using an API (e.g., a proprietary API).

The user computing device 111 may be an electronic device. FIG. 2 is a schematic diagram of a data processing system of an electronic device 200 in some embodiments. The electronic device 200 includes a data acquisition module 210, a large model 220, and a database 230. The electronic device 200 can implement a series of functions of data collection, preprocessing, storage, mining, retrieval, recommendation, question-answering, and the like for various applications.

In the process of displaying the application page content or the web page content, the electronic device 200 may detect that the user starts the data recording function through a preset triggering manner, where the preset triggering manner may be triggering a target function key, gesture triggering (such as sliding upwards in three directions), or triggering of the data recording application program.

The data collection module 210 is configured to record, according to the opened data recording function, screen display content of the electronic device 200 or page content of an application running in the background. The screen display content or page content of the background running application may include data content such as text, pictures, video, audio, and the like. If the text is the text, the text data can be directly acquired. If the picture is the picture, the picture can be downloaded according to the address of the picture. If the video is the video, the URL address of the video can be obtained, and all or fragment contents of the video can be downloaded according to the URL address. If the audio is the audio, the download address of the audio can be obtained, and all or part of the data of the audio can be downloaded. The data acquisition module 210 may also record the screen display content in a screenshot mode, or acquire voice data input by a user through a microphone for preprocessing.

The data collection module 210 is further configured to invoke the large model 220 or some algorithms to preprocess the recorded screen display content, so as to obtain data in a target format. For example, if the screen display content is text, entity extraction and summary can be performed, if the screen display content is a picture, the picture can be identified to obtain picture content and classify the picture, if the screen display content is video, key frame extraction and key frame content identification can be performed on the video, and summary information can be generated on the video. The processed data may be stored in database 230. Wherein the large model 220 may be the machine learning model of fig. 1.

Database 230 is a carrier or system for storing, managing and retrieving information in a form that may vary depending on the object of storage, the context of use and the architecture of technology. Database 230 may store data and may have at least one of classification, retrieval, synchronization, rights control, and the like. The database 230 may be an application, applet, API interface, personal specific storage hardware, embedded hardware logging module, cloud service form database, browser plug-in, bookmark management tool, enterprise knowledge base system, cache type database, personal information base, etc. The personal special storage hardware can be a mobile hard disk with a special management system, an encrypted U disk, a personal cloud storage hard disk and the like. The embedded hardware recording module can be a special storage module in the intelligent equipment and is used for recording various data. The database of the cloud service form can be a cloud database providing interface calling service functions or a database of personal cloud data management service.

Database 230 includes a data management module 240 and a data interaction module 250. The data management module 240 may invoke the large model 220 to mine the stored data and store the mined data in the database 230, may provide data retrieval functionality, and the like. Mining modes can comprise data desensitization processing, label classification, entity extraction, schedule extraction, to-do item extraction, abstract generation, generation of a collection, recommendation of a collection, data association and the like. The data management module 240 may further perform induction and sorting according to various data recorded by the user, so as to obtain user portrait information. The user portrait information may include user personal information, personal preference information, and the like. The user personal information may include information of identity information, occupation, academy, family, etc. of the user, and the personal preference information may include favorite people, things, places, scenery, games, etc., without being limited thereto.

The data interaction module 250 may provide an interaction function with an application program, and may acquire data required for interaction through the data management module 240. The interactive functions may include, but are not limited to, recommendation collection, data search, data detail viewing, data sharing, AI questions and answers, data association, and the like. For example, the AI question and answer function, the data interaction module 250 acquires a question provided by the user in the AI question and answer application and transmits the question to the data management module 240, and the data management module 240 searches for a corresponding answer and provides the answer to the data interaction module 250 for feedback to the AI question and answer related application.

Further, the data collected by the data collection module 210 and the mined user portrait information in the electronic device 200 can be synchronized to the cloud end according to the needs, the collected data can be processed by means of a large cloud end model, and the data synchronized to the cloud end can be synchronized to other electronic devices through the cloud end, so that sharing of the data among different devices is realized.

FIG. 3 is a schematic diagram of the internal architecture of a data processing system of an electronic device in some embodiments. The data acquisition module 210 may include an acquisition entry sub-module 211, an extraction sub-module 212, a screenshot acquisition sub-module 213, a preprocessing sub-module 214, a writing sub-module 215, and the like.

The collection entry sub-module 211 is configured to provide an entry for data collection, for example, a voice assistant entry, through which a data recording instruction of a user voice input is received, or a gesture trigger entry, such as a long entry of a hardware key, or a three-way sliding entry, or a trigger entry of a user for a specific control, such as a praise control, a collection control, a sharing control, or the like. The acquisition inlet sub-module 21 can automatically acquire the page information according to the set rule, if the acquisition triggering condition is the preset page content, and if the preset page content exists in the page information displayed by the electronic equipment, the page information is automatically acquired. The collection inlet sub-module 21 may also receive donation data of some application programs (such as weather application, clock application, step counting application, etc.), such as weather data, time data, walking data, running mileage, etc., but is not limited thereto.

The extraction sub-module 212 may perform image-text extraction on the page content of the application to be recorded.

The screenshot obtaining sub-module 213 may obtain text content by screenshot the screen content and then invoking the screenshot content of the OCR recognition algorithm to perform text recognition.

The preprocessing sub-module 214 may segment or denoise or process the collected data into data in a preset format.

The writing sub-module 215 is used to write the collected data or the preprocessed data into the database 230.

The data management module 240 may include a data mining sub-module 241, a question and answer sub-module 242, a retrieval sub-module 243, a source data processing sub-module 244, and the like.

The data mining sub-module 241 is used for mining data. Mining modes can comprise data desensitization processing, label classification, entity extraction, schedule extraction, to-do item extraction, abstract generation, generation of a collection, recommendation of a collection, data association and the like.

The question and answer sub-module 242 is used for searching corresponding data according to the user questions and generating corresponding answers according to the searched data.

The retrieving sub-module 243 is configured to retrieve corresponding data from the database 230 according to the search request, and sort or directly feed back the retrieved data to the user.

The source data processing sub-module 244 is configured to provide add, modify, delete, update operations on the source data. The source data may refer to data recorded in the database 230 after collection or data after preprocessing the collected data.

The data interaction module 250 includes a data addition/editing/viewing sub-module 251, a data recommendation sub-module 252, a data search sub-module 254, an associated data sub-module 254, and the like.

The data adding/editing/viewing sub-module 251 is used to provide functions of adding, editing, viewing, etc. data.

The data recommendation sub-module 252 is used to recommend aggregate data to a user or data stored according to user image recommendations.

The data searching sub-module 253 is configured to obtain a search requirement of a user, and transmit the search requirement to the searching sub-module 243 in the data management module 240. The search sub-module 243 retrieves corresponding data from the database 230 according to the search requirement, and feeds back the retrieved data to the data search sub-module 253.

The association data sub-module 254 is used to associate or recommend the recorded data to other applications.

The data processing method provided by the embodiment of the application can be applied to the electronic equipment shown in fig. 1 to 3. The electronic device may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, unmanned aerial vehicles, low-altitude aircrafts, internet of things devices and portable wearable devices, and the internet of things devices may be smart speakers, smart televisions, smart air conditioners, smart vehicle devices, projection devices and the like. The portable wearable device may be a smart watch, smart bracelet, headset, or the like. The head-mounted device may be a Virtual Reality (VR) device, an augmented Reality (Augmented Reality, AR) device, smart glasses, or the like.

In some exemplary embodiments, as shown in fig. 4, a data processing method is provided, and the method is applied to an electronic device for illustration, and the method includes the following steps 402 to 406. Wherein:

step 402, displaying an interface of the target application.

The target application may be any application. For example, the target application may be a system application or a third party application or an applet application or a quick application, etc. The system application may be an application developed by an operating system vendor or electronic device manufacturer that is preloaded in the electronic device system, owning system permissions. The third party application may be an application developed by an independent developer, software company or non-official organization, focusing on specific field requirements, such as social, video, navigation, etc. applications, with limited general rights. The applet is a child application that runs on the parent application's internal container, requiring no download installation, or a lightweight application that can be used without download installation. The fast application is an installation-free application directly running in an operating system of the electronic equipment, the front-end technology stack development is used, the native rendering is realized through a system-level engine, the performance is close to that of the native application, the fast application is compatible across platforms, and all equipment supporting the fast application can be covered by one development.

The interface of the target application may be any page when the target application runs, for example, the target application is a social application, and the interface of the target application may be a setting page of the social application, various functional pages, and the like, but is not limited thereto. The interface of the target application may also be any interface when the target application runs, for example, the target application is a camera application, and the interface of the target application may be a shooting interface, a recording interface or a parameter setting interface of the camera application, which is not limited thereto.

The target application is run on the electronic device, and an interface of the target application is displayed.

Step 404, responding to the triggering operation aiming at the target application, and acquiring a corresponding target data acquisition mode according to the target application, wherein the target data acquisition mode is used for indicating the range of data acquisition of an interface of the target application.

The trigger operation may refer to a trigger operation for performing data collection on an interface of the target application. The data acquisition refers to acquiring, analyzing and storing data of a target application. The trigger operation may be a gesture trigger, or a voice assistant trigger, or a specified key trigger. The gesture trigger may be a preset swipe trigger, such as a three-finger swipe trigger, as shown in fig. 5A. The voice assistant trigger may be that the voice assistant receives voice data input by the user to identify and obtain a data acquisition instruction, as shown in fig. 5B. The designated key trigger may be that a configured designated key is triggered to generate a data acquisition instruction.

The range for acquiring data may be the range for acquiring data by the application, and may be the whole content of the interface or a part of the content of the interface. For example, the range of the application A for acquiring data is that the data in the window does not contain control information, the range of the application B for acquiring data is that of the control information, the range of the application C for acquiring data is that of the designated control information, and the like.

Different applications can configure different data acquisition modes, and can also configure the same data acquisition mode. Different pages of the same application can be configured with different data acquisition modes, and the same data acquisition mode can also be configured. The data collection means may be a collection strategy. For example, different data acquisition modes, i.e. different acquisition strategies, may be configured for different page types or different application types.

The electronic device is configured to trigger the data acquisition function in response to a triggering operation of the data acquisition function for the target application, and acquire a corresponding target data acquisition mode from the local or cloud of the electronic device according to the target application. The local acquisition target data acquisition mode can be adapted to the function in a network-free environment without using a network, and the acquisition is convenient and quick. The corresponding target data acquisition mode is acquired from the cloud, so that the local storage space can be saved, the latest data acquisition mode can be acquired, and the data acquisition mode can be acquired by crossing devices (the same account is logged in different devices and acquired from the cloud).

And step 406, data acquisition is carried out on the interface data of the target application according to the target data acquisition mode, so as to obtain target data.

Interface data may refer to various types of information presented on an interface. The information may include one or more of text, pictures, uniform resource locators, and the like.

The target data acquisition mode can be to acquire all the interface data, or to acquire part of the interface data. And if the data acquisition is carried out on the interface data, carrying out data acquisition according to the designated range.

The electronic device traverses interface data of the target application to acquire data according to the target data acquisition mode, and obtains target data.

In this embodiment, an interface of a target application is displayed, a corresponding target data acquisition mode is acquired according to the target application in response to a trigger operation for the target application, the target data acquisition mode is used for indicating a range of data acquisition for the interface of the target application, and data acquisition is performed for interface data of the target application according to the data acquisition range defined by the target data acquisition mode, so as to obtain target data. The target data acquisition mode corresponding to the target application can be adopted to acquire the required target data, so that the data corresponding to the data acquisition range is acquired for different applications, the unnecessary data is prevented from being acquired by the data, dirty data is prevented from being introduced, and the flexible acquisition of the required data is realized.

In some exemplary embodiments, acquiring the corresponding target data acquisition mode according to the target application includes acquiring a page type corresponding to an interface of the target application, and acquiring the corresponding target data acquisition mode from a corresponding relationship between the page type and the data acquisition mode according to the page type of the interface.

By way of example, the page types may include one or more of a native page, a browser page, or a hybrid page. Native pages, i.e., native pages, are developed using platform native languages, such as camera calls, complex animations, and the like. The device hardware application interface is directly invoked. Browser pages, i.e., web pages, developed based on HTML (Hyper Text Markup Language )/CSS (CASCADING STYLE SHEETS, cascading style sheets)/JS (JavaScript), can be cross-platform, such as news presentation pages, by embedding native applications through Web View. The Hybrid page is a Hybrid page, and combines Native and Web technologies, the core function is realized by adopting Native technology, and the universal module is realized by adopting H5 (namely HTML 5), such as an e-commerce commodity page.

The page types may also include, for example, one or more of an aggregate class page, a list class page, a content class page, or a function class page. The aggregate page is generally used as an application home page or a navigation center, and aggregates a plurality of function entries, such as an e-commerce home page, a tool application main interface and the like. List class pages are typically used to present the same class of information, such as a list of goods, news catalogs, etc. Content class pages are typically used to present detailed information such as item details, article reading pages, and the like. The function class page is generally used to perform specific operations such as searching, form filling, setting, etc., such as registering a login page, a search page, an opinion feedback page, etc.

Different data acquisition modes can be configured in advance according to different page types, and the corresponding relation between the page types and the data acquisition modes is stored. The corresponding relation between the page type and the data acquisition mode can be stored in the electronic equipment or the cloud.

The electronic device obtains an interface of the target application, then determines a page type of the interface of the target application, and searches for a corresponding target data acquisition mode from a corresponding relation between the page type and the data acquisition mode according to the page type corresponding to the interface.

Through the corresponding relation between the pre-constructed page type and the data acquisition mode, the corresponding target data acquisition mode can be quickly found according to the page type corresponding to the interface of the target application, the interface data is acquired by adopting the target data acquisition mode, the target data is obtained, the accuracy of the acquired data can be ensured, the acquired data can be ensured to be cleaner, and the processing efficiency is high.

In some exemplary embodiments, if the page type corresponding to the interface of the target application is a native page, according to the native page of the interface, a corresponding first data acquisition rule is obtained from a correspondence between the page type and the data acquisition mode, where the first data acquisition rule is used to indicate a range for data acquisition for the native page.

In some exemplary embodiments, if the page type corresponding to the interface of the target application is a web page, according to the web page of the interface, a corresponding second data acquisition rule is obtained from the corresponding relationship between the page type and the data acquisition mode, where the second data acquisition rule is used to indicate a range of data acquisition for the web page.

In some exemplary embodiments, if the page type corresponding to the interface of the target application is a mixed page, according to the mixed page of the interface, a corresponding third data acquisition rule is obtained from the corresponding relationship between the page type and the data acquisition mode, where the third data acquisition rule is used to indicate a range of data acquisition for the mixed page.

In some exemplary embodiments, different pages of the same application may be configured with different data collection manners, and different pages of different applications may be configured with different data collection manners. The method comprises the steps of obtaining page identifiers corresponding to interfaces of target applications, and obtaining corresponding target data acquisition modes from corresponding relations between the page identifiers and the data acquisition modes according to the page identifiers of the interfaces.

The page identity is used to uniquely identify the page of the application. The page identification may be a page name or a page ID.

The corresponding relation between the preconfigured page identification and the data acquisition mode can be stored in the local or cloud of the electronic equipment.

The electronic device obtains a page identifier corresponding to the interface of the target application, and then obtains a corresponding target data acquisition mode from a corresponding relation between the page identifier and the data acquisition mode stored locally or in the cloud of the electronic device according to the page identifier of the interface.

The corresponding data acquisition mode can be rapidly acquired through the page identifier corresponding to the interface of the target application, so that the interface data are acquired, and the target data are obtained.

In some exemplary embodiments, the method further comprises displaying a data acquisition mode configuration interface of the target application, acquiring configuration parameter information input in a configuration parameter field of the configuration interface, and generating a target data acquisition mode of the target application according to the configuration parameter information.

The configuration parameter fields may include a trigger mode field and an application attribute field. The application attribute field may include one or more of an application identification field, an application type field, a page identification field, a page type field, and an extension parameter field. The application attribute field may also include a page control name field. The configuration parameter information may include trigger mode information and application attribute information. The trigger mode information may include voice assistant triggers, gesture triggers, designated key triggers, and the like. The values of the extension parameter fields may be configured as desired for various desired parameters, such as list names, control names, etc. As shown in fig. 6, the values of the trigger mode fields in the configuration parameter fields of the data acquisition mode configuration interface are all triggers, the values of the scene fields are flashNotes, the values of the application identification fields, i.e., the application name fields, are the top-of-today bars, the values of the page identification fields, i.e., the page name fields, are the recommended news pages, the values of the page control name fields are abc, the displayable card fields are coupons, the values of the card rules, viewid and the page type fields are active, the values of the extension parameter fields are "listType": 1, "textType": "NATIVESPECIAL", "SPECIALLIST": "PreLayoutTextView, androidText", and the like.

The data acquisition mode, namely the acquisition rule, required by the application can be configured according to the requirement through the data acquisition configuration interface, and the method is flexible and convenient and does not need to change codes.

The data acquisition configuration interface of the application can be provided by a cloud end, configured by a manager or a developer, and used for realizing data acquisition processing of different service scenes, the processing is more flexible, the data acquisition configuration interface of the application can also be provided by an electronic device, configured by a user according to the needs, and the configuration of the user is more flexible.

In an exemplary embodiment, the method further comprises the steps of obtaining the corresponding relation between the application configured and generated after the user identifier logs in the first electronic device and the data acquisition mode, uploading the corresponding relation to the cloud end, obtaining the corresponding relation between the application and the data acquisition mode from the cloud end according to the user identifier, and storing the corresponding relation on the second electronic device. The user can share the corresponding relation between the application configured by the first electronic equipment and the data acquisition mode to the second electronic equipment through the cloud, so that the user can conveniently acquire the data of the interface data of the application by adopting the same corresponding relation on the second electronic equipment, the corresponding relation between the application and the data acquisition mode is not required to be reconfigured when the user uses different electronic equipment, and the interaction efficiency is improved.

In order to more clearly understand the configuration process of the configuration parameter information in the data acquisition configuration interface, taking the target application as the present headline, the page name as the recommended news page as an example, the present headline application has the self-rendering control, the data acquisition mode of configuring the present headline in the data acquisition configuration interface comprises the text information acquired from the rendering control, the corresponding data acquisition mode is shown in fig. 6, such as textType: NATIVESPECIAL, the acquisition file type is acquired from the rendering control, SPECIALIST: preLayoutTextView, androidText, the list of the self-rendering control class names is used, and the text information is extracted for the controls in the list during acquisition. The page view of the target application is shown in fig. 7, and text information is extracted for the controls 702, 704, 706 in the list. The self-rendering control is a control which does not add text information to the control according to an operating system native method, and corresponding analysis logic can be independently configured for the self-rendering control.

Taking the target application as a present headline and the page name as a recommended news page as an example, the present headline application has a self-rendering control, the data acquisition mode of the present headline is configured in a data acquisition configuration interface to contain text information acquired from the rendering control, and the corresponding data acquisition mode is textType: NATIVESPECIAL, the acquisition file type is acquired from the rendering control, SPECIALIST: androidText, text information is extracted for the control of the present type, and the other text information is discarded. The page view of the target application is shown in FIG. 7, with controls 702, 704, and 706 in the list, extracting text information for the specified control 702.

In some exemplary embodiments, data acquisition is performed on interface data of a target application according to a target data acquisition mode to obtain target data, and the method comprises the steps of acquiring an image corresponding to the interface data of the target application according to the target data acquisition mode, and identifying the image to obtain the target data.

The image corresponding to the interface data of the target application can be obtained from the android bottom layer according to the target data acquisition mode. If the target data acquisition mode includes the image content of the designated control, only the image corresponding to the designated control in the interface data can be acquired. If the image content of the designated control is the shot picture content, unprocessed preview data can be captured directly through the camera application interface, so that interface layers of camera applications are avoided, and images of shot pictures which do not contain data such as zoom magnification, shooting modes and the like are obtained.

The image corresponding to the interface data of the target application can be acquired through the target data acquisition mode, then the image is identified, the target data can be obtained, the target data acquisition mode indicates the range of the interface data acquisition, namely the range of the image acquisition is limited, the whole image or part of the image of the interface data can be acquired, the flexibility of the image acquisition is ensured, and the cleanliness of the data acquisition content is ensured.

In some exemplary embodiments, the image is identified to obtain the target data, which may include performing text recognition on the image corresponding to the interface data to obtain a file recognition result, performing semantic recognition on the image to obtain a semantic recognition result, and obtaining the target data according to the text recognition result and the semantic recognition result.

The word recognition process may employ OCR (Optical Character Recognition ). And carrying out semantic recognition on the image by adopting the large model to obtain a semantic recognition result, and generating abstract information according to the text recognition result and the semantic recognition result.

As shown in fig. 8A, taking the target application as the camera application as an example, the triggering manner of the data acquisition configuration interface of the camera application may be all triggers. The scene is flashNotes, the application name is a camera, the page name is a camera shooting page, and the page control name is com. The value of the extension parameter field may be list type 0, text type OCR, window identification and content of window name control. The data acquisition mode corresponding to the camera application refers to that the content of the window identifier and window name control is specified to perform screenshot for OCR extraction.

As shown in fig. 8B, the acquisition range of the camera application may be a camera photographing window 802, and shooting mode control information such as a master, a video, a photo, a portrait, and the like, and flash mark information and the like displayed on the camera application interface are not acquired.

As shown in fig. 8C, the whole interface of the camera application is captured and then identified, and the obtained result is a mobile phone camera interface and a cap picture, and the AI screen abstract is that the picture shows the mobile phone camera interface and a shooting scene. 1. Camera interface, with EV, one-key flash, etc. function buttons, can select master, video, photo, portrait, more modes, with 0.6X, 1X, 2X, 3X, 6X zoom options.

As shown in fig. 8D, the window identifier and window name control of the camera application are captured and then identified, and the obtained result is a computer screen and desktop object display, and the AI screen abstract is that the picture displays the computer screen and desktop object. 1. The computer screen displays the character of 'one-key flash memory'. 2. The tabletop articles are white caps, black patterns are arranged on the caps, and articles such as paper towels and the like are arranged beside the caps. Compared with fig. 8C, in fig. 8D, the information such as the shooting mode and the zoom option is not required to be acquired, and the acquisition of dirty data is avoided.

In some exemplary embodiments, the acquiring the corresponding target data acquisition mode according to the target application includes determining an application type of the target application, and acquiring the corresponding target data acquisition mode from a corresponding relationship between the application type and the data acquisition mode according to the application type of the target application.

The application types may be categorized as desired, such as media applications, photography applications, social applications, e-commerce applications, and the like. The corresponding relation between the application type and the data acquisition mode can be preset and stored in the local or cloud of the electronic equipment.

The electronic device determines an application type of the target application, and may obtain, according to the application type, a corresponding target data acquisition mode from a correspondence between the application type and the data acquisition mode stored locally or in the cloud of the electronic device.

In some exemplary embodiments, according to the application type of the target application, acquiring the corresponding target data acquisition mode from the corresponding relation between the application type and the data acquisition mode includes acquiring a corresponding first data acquisition mode from the corresponding relation between the application type and the data acquisition mode if the application type of the target application is a media application, where the first data acquisition mode is used to indicate that the data acquisition range of the media application includes one or more of a play window and a control.

The media application may be a news media application, a play media application, a short video application, etc. The news media application may be a newwave news application, a today's headline application, or the like. The media applications may be Tencel video, aiqi video, etc. The data collection range, i.e., collection range, may be one or more of a play window and a control. As shown in fig. 9, the data collection scope of the media application may be a play window 902, a control 904, or the like.

In some exemplary embodiments, according to the application type of the target application, acquiring the corresponding target data acquisition mode from the corresponding relation between the application type and the data acquisition mode includes acquiring a corresponding second data acquisition mode from the corresponding relation between the application type and the data acquisition mode if the application type of the target application is a shooting application, where the second data acquisition mode is used for indicating that the data acquisition range of the shooting application includes a shooting picture.

The photographing application may be a camera application. As shown in fig. 10, the data acquisition range may include a photographed screen 1002 (a dashed frame range). The method can only collect the content in the shooting picture, avoid collecting additional information such as controls and the like, and ensure the accuracy and the cleanliness of the collected data content.

In some exemplary embodiments, according to the application type of the target application, the corresponding target data acquisition mode is obtained from the corresponding relation between the application type and the data acquisition mode, including a third data acquisition mode obtained from the corresponding relation between the application type and the data acquisition mode if the application type of the target application is a social application, where the third data acquisition mode is used to indicate that the data acquisition range of the social application includes a session window and a session title.

The social application may be an instant messaging application such as WeChat, QQ, spike, etc. As shown in FIG. 11, the data collection scope of the social application includes a session window 1102 and a session title 1104. The conversation title can be a group ID or a friend identification, the conversation window is a group chat conversation window or a double conversation window, and the conversation abstract information can be automatically analyzed and generated by collecting chat information in the conversation window. The session summary information may be a summary description or a summary description of chat information. Session summary information may be stored with the session title in a data collection database. Further, the calendar information is included in the session summary information, and the calendar information in the session summary information may be added to the calendar application.

In some exemplary embodiments, according to the application type of the target application, a corresponding target data acquisition mode is obtained from a corresponding relation between the application type and the data acquisition mode, including a fourth data acquisition mode obtained from a corresponding relation between the application type and the data acquisition mode if the application type of the target application is an e-commerce application, where the fourth data acquisition mode is used for indicating that a data acquisition range of the e-commerce application includes a shop identifier, a commodity picture, an area where a commodity price is located, and a link address.

The e-commerce application can be an application provided by various e-commerce platforms or an application corresponding to a selling platform provided by a manufacturer or a merchant. As shown in fig. 12, the data collection range of the e-commerce application includes a store identification 1202, a commodity picture 1204, a commodity price 1206, and a link address 1208. By collecting the relevant information of the commodity in the commodity application, the user can conveniently inquire.

In some exemplary embodiments, the method further comprises identifying the target data to obtain a target tag of the target data, and storing the target data to a storage area in the data acquisition database corresponding to the same tag as the target tag.

And carrying out category identification on the target data through the artificial intelligent model to obtain a target label of the target data. A data collection database refers to a database or storage space for storing data. The data acquisition database may be divided into a plurality of storage areas. One tag corresponds to one storage area, and data with the same tag is stored in the corresponding storage area.

And the data with the same label is stored in the same storage area, so that the data management and the subsequent query can be facilitated.

In some exemplary embodiments, the method further comprises identifying the target data to obtain a target tag of the target data, and adding the target data to an associated application corresponding to the target tag.

The associated application refers to an application associated with a tag, such as a calendar application, and a codebook application, which is associated with user information.

By identifying the target label of the target data, the target data is automatically added into the management application corresponding to the target label, so that the manual adding operation of a user is saved, and the data adding efficiency is improved.

In some exemplary embodiments, the target tag is a calendar class and the associated application is a calendar application, and adding the target data to the associated application corresponding to the target tag includes adding the target data to the calendar application.

The target label of the target data is a calendar class, which indicates that the target data contains corresponding calendar content, and the target data can be directly added into the calendar application, or entity information about the calendar content can be extracted from the target data and added into the calendar application. The entity information refers to specific contents of a schedule, such as including time, place, event, person, etc. For example, the target data is a meeting notice that a meeting related to an AI model is available at 2 pm on the Tuesday and the research and development group personnel participate in the meeting, and the extracted entity information can comprise the meeting at 2 pm on the Tuesday, the AI model meeting and the research and development group personnel, and the storage space of calendar application can be saved by discarding some description information.

In some exemplary embodiments, the target tag is a user information class, the associated application is a codebook application, and the adding the target data to the associated application corresponding to the target tag includes adding the target data to the codebook application.

The target tag of the target data is illustratively a user information class, which indicates that the target data contains user personal information or friend personal information or relative personal information, etc., and the target data can be directly added into the calendar application, or key content about the user information can be extracted from the target data and added into the codebook application. The key content refers to the main content in personal information of the user, such as identification card number, mobile phone number, passport, driving license and the like. For example, the target data is "high-speed railway ticket buying, and the identification numbers of users A and B are provided", the extracted key content can comprise "the identification number of A and the identification number of B", and the extracted key content is added into the application of the codebook. Discarding some description information can save the storage space of the codebook application.

The above-described data processing method can be applied to the application framework shown in fig. 13. The following describes a data processing method in connection with an application framework, specifically including:

(1) The data acquisition stage comprises the steps that a user triggers a data acquisition function through three-finger sliding triggering or magic cube key triggering or voice assistant triggering on the electronic equipment, the screen recognition application receives the triggering data acquisition function, the screen recognition application acquires a corresponding target data acquisition mode according to a target application displayed on the electronic equipment, extracts interface data of the target application according to the target data acquisition mode, preprocesses the extracted data by adopting an artificial intelligent unit, and stores the preprocessed data in a database, wherein the target label of the target data is identified as a schedule type in the preprocessing process of the extracted data by the artificial intelligent unit, the target data is added into the calendar application, and the target label of the target data is identified as user information type, and the target data is added into the codebook application. The artificial intelligence unit can comprise NER entity extraction, OCR capability, multi-modal large model understanding capability, multi-modal large model cloud side deployment and multi-modal large model end side deployment. The extracted images and texts can be identified through various models in the artificial intelligence unit, and target data are obtained. Multimodal refers to multiple modalities such as text, images, and speech.

(2) And a data use stage, wherein a user can query the stored data from the database or delete the stored data.

The above data processing method can be applied to the application framework shown in fig. 14 when the data processing method is applied to collecting text data. As shown in fig. 14, after receiving a trigger operation on a target application, an acquisition manager in the screen recognition application acquires an acquisition strategy (i.e., a target data acquisition mode) corresponding to the target application from a flash cloud (i.e., a cloud), acquires interface data of the target application by using the acquisition strategy to obtain target data, and stores the target data in a data storage space of a database.

The above-described data processing method can be applied to an application framework shown in fig. 15 when an image is acquired. As shown in fig. 15, after receiving a trigger operation on a target application, an acquisition manager in a screen recognition application acquires an acquisition strategy (i.e., a target data acquisition mode) corresponding to the target application from a flash cloud, then performs image acquisition on interface data of the target application by using the acquisition strategy to obtain an image, performs character recognition on the image by using OCR to obtain a character recognition result, performs semantic recognition on the image to obtain a semantic recognition result, analyzes and generates summary information according to the character recognition result and the semantic recognition result, and stores the summary information in a data storage space of a database.

It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.

Based on the same inventive concept, the embodiment of the application also provides a data acquisition device for realizing the above related data acquisition method. The implementation of the solution provided by the apparatus is similar to that described in the above method, so the specific limitation of one or more embodiments of the apparatus provided below may be referred to as the limitation of the method hereinabove, and will not be repeated herein.

In some exemplary embodiments, as shown in fig. 16, a data processing apparatus 1600 includes a display module 1602, a mode acquisition module 1604, and a data acquisition module 1606.

A display module 1602 for displaying an interface of the target application.

The mode acquisition module 1604 is configured to acquire, according to the target application, a corresponding target data acquisition mode in response to a trigger operation for the target application, where the target data acquisition mode is used to indicate a range of data acquisition for an interface of the target application.

The data collection module 1606 is configured to perform data collection on the interface data of the target application according to the target data collection manner, so as to obtain target data.

In some exemplary embodiments, the mode obtaining module 1604 is further configured to obtain a page type corresponding to the interface of the target application, and obtain, according to the page type, a corresponding target data collection mode from a correspondence between the page type and the data collection mode.

In some exemplary embodiments, the data collection module 1606 is further configured to obtain an image corresponding to the interface data of the target application according to the target data collection manner, and identify the image to obtain target data.

In some exemplary embodiments, the data collection module 1606 is further configured to perform text recognition on an image corresponding to the interface data to obtain a file recognition result, perform semantic recognition on the image to obtain a semantic recognition result, and obtain the target data according to the text recognition result and the semantic recognition result.

In some exemplary embodiments, the mode obtaining module 1604 is further configured to determine an application type of the target application, and obtain, according to the application type of the target application, a corresponding target data acquisition mode from a correspondence between the application type and the data acquisition mode.

In some exemplary embodiments, the mode obtaining module 1604 is further configured to obtain, if the application type of the target application is a media application, a corresponding first data collection mode from a correspondence between the application type and the data collection mode, where the first data collection mode is used to indicate that a data collection range of the media application includes one or more of a play window and a control.

In some exemplary embodiments, the mode obtaining module 1604 is further configured to obtain a corresponding second data acquisition mode from the corresponding relationship between the application type and the data acquisition mode if the application type of the target application is a shooting application, where the second data acquisition mode is used to indicate that a data acquisition range of the shooting application includes a shooting picture.

In some exemplary embodiments, the manner obtaining module 1604 is further configured to obtain a third data collection manner from the correspondence between the application type and the data collection manner if the application type of the target application is a social application, where the third data collection manner is used to indicate that a data collection range of the social application includes a session window and a session title.

In some exemplary embodiments, the apparatus further comprises a tag identification module and a storage module.

The tag identification module is used for identifying the target data to obtain a target tag of the target data.

And the storage module is used for storing the target data into a storage area corresponding to the label which is the same as the target label in the digital database.

In some exemplary embodiments, the apparatus further comprises a tag identification module and an addition module.

The tag identification module is used for identifying the target data to obtain a target tag of the target data.

And the adding module is used for adding the target data into the associated application corresponding to the target tag.

In some exemplary embodiments, the target tag is a calendar class, the associated application is a calendar application, and the adding module is further configured to add the target data to the calendar application.

In some exemplary embodiments, the target tag is a user information class, the associated application is a codebook application, and the adding module is further configured to add the target data to the codebook application.

Each of the modules in the above-described apparatus may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules can be freely combined as required. The above modules may be embedded in hardware or independent of a processor in the electronic device, or may be stored in software in a memory in the electronic device, so that the processor may call and execute operations corresponding to the above modules.

In an exemplary embodiment, a computer device is provided comprising a memory and a processor, the memory having stored therein a computer program, the processor implementing the steps of the above method when executing the computer program.

In one embodiment, a computer readable storage medium is provided having a computer program stored thereon, which when executed by a processor, implements the steps of the above method.

In an embodiment, a computer program product is provided comprising a computer program which, when executed by a processor, implements the steps of the above method.

It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are both information and data authorized by the user or sufficiently authorized by each party, and the collection, use and processing of the related data are required to meet the related regulations.

Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in embodiments provided herein may include at least one of non-volatile memory and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (RESISTIVE RANDOM ACCESS MEMORY, reRAM), magneto-resistive Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (PHASE CHANGE Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in various forms such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The databases referred to in the embodiments provided herein may include at least one of a relational database and a non-relational database. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processor referred to in the embodiments provided in the present application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computation, an artificial intelligence (ARTIFICIAL INTELLIGENCE, AI) processor, or the like, but is not limited thereto.

The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the present application.

The foregoing examples illustrate only a few embodiments of the application and are described in detail herein without thereby limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of the application should be assessed as that of the appended claims.

Claims (15)

1. A method of data processing, the method comprising:

Displaying an interface of a target application;

Responding to the triggering operation aiming at the target application, and acquiring a corresponding target data acquisition mode according to the target application, wherein the target data acquisition mode is used for indicating the range of data acquisition on an interface of the target application;

and carrying out data acquisition on the interface data of the target application according to the target data acquisition mode to obtain target data.

2. The method according to claim 1, wherein the obtaining, according to the target application, the corresponding target data acquisition mode includes:

And acquiring a page type corresponding to the interface of the target application, and acquiring a corresponding target data acquisition mode from the corresponding relation between the page type and the data acquisition mode according to the page type.

3. The method according to claim 2, wherein the data acquisition of the interface data of the target application according to the target data acquisition mode, to obtain target data, includes:

Acquiring an image corresponding to the interface data of the target application according to the target data acquisition mode;

And identifying the image to obtain target data.

4. The method according to claim 1, wherein the obtaining, according to the target application, the corresponding target data acquisition mode includes:

Determining the application type of the target application;

And acquiring a corresponding target data acquisition mode from the corresponding relation between the application type and the data acquisition mode according to the application type of the target application.

5. The method according to claim 4, wherein the obtaining, according to the application type of the target application, the corresponding target data acquisition mode from the correspondence between the application type and the data acquisition mode includes:

And if the application type of the target application is a media application, acquiring a corresponding first data acquisition mode from the corresponding relation between the application type and the data acquisition mode, wherein the first data acquisition mode is used for indicating that the data acquisition range of the media application comprises one or more of a playing window and a control.

6. The method of claim 4, wherein the obtaining, according to the application type, the corresponding target data acquisition mode from the correspondence between the application type and the data acquisition mode includes:

And if the application type of the target application is a shooting application, acquiring a corresponding second data acquisition mode from the corresponding relation between the application type and the data acquisition mode, wherein the second data acquisition mode is used for indicating that the data acquisition range of the shooting application contains shooting pictures.

7. The method of claim 4, wherein the obtaining, according to the application type, the corresponding target data acquisition mode from the correspondence between the application type and the data acquisition mode includes:

And if the application type of the target application is a social application, acquiring a third data acquisition mode from the corresponding relation between the application type and the data acquisition mode, wherein the third data acquisition mode is used for indicating that the data acquisition range of the social application comprises a session window and a session title.

8. The method according to any one of claims 1 to 7, further comprising:

identifying the target data to obtain a target label of the target data;

And storing the target data into a storage area corresponding to the label which is the same as the target label in a database.

9. The method according to any one of claims 1 to 7, further comprising:

identifying the target data to obtain a target label of the target data;

and adding the target data into the associated application corresponding to the target label.

10. The method of claim 9, wherein the target tag is a calendar class and the associated application is a calendar application, and wherein the adding the target data to the associated application corresponding to the target tag comprises:

The target data is added to the calendar application.

11. The method of claim 9, wherein the target tag is a user information class and the associated application is a codebook application, and wherein the adding the target data to the associated application corresponding to the target tag comprises:

The target data is added to the codebook application.

12. A data processing apparatus, the apparatus comprising:

the display module is used for displaying the interface of the target application;

the system comprises a mode acquisition module, a mode control module and a mode control module, wherein the mode acquisition module is used for responding to the triggering operation aiming at the target application and acquiring a corresponding target data acquisition mode according to the target application;

And the data acquisition module is used for carrying out data acquisition on the interface data of the target application according to the target data acquisition mode to obtain target data.

13. An electronic device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any one of claims 1 to 11 when the computer program is executed.

14. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 11.

15. A computer program product comprising a computer program, characterized in that the computer program, when executed by a processor, implements the steps of the method of any one of claims 1 to 11.