CN112100284A - Interacting with real world objects and corresponding databases through virtual twin reality - Google Patents

Abstract

A system includes at least one cloud server of a cloud server computer system, the at least one cloud server including at least one processor and memory storing a persistent virtual world system including one or more virtual objects including virtual data and a model. The virtual objects include one or more of virtual twins, pure virtual objects or applications, wherein at least one of the virtual objects represents a store of real-world items connected to a periodically updated database associated with products of at least one store. A user may access the store through the persistent virtual world system via a user device that enables interaction with and between elements within the store.

Description

Interact with real world objects and corresponding databases through virtual twin reality

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of Provisional Application No. 62/863171, filed June 18, 2019, the entire disclosure of which is incorporated herein by reference for all purposes.

Background technique

Databases are often used to keep track of physical objects. In the first case, the user goes to the location of the physical object and brings it to another location. A barcode scanner, camera or other device can be used to determine that the object has been removed from its original location. In the second case, the remote user visits the website to request that the object be shipped to another location. In either case, the database can be updated to reflect that the physical object has moved from its original location or to indicate that a transaction has occurred.

However, the current technology in these cases has several disadvantages. For example, in the first case, the user must be physically present to interact with the object or complete a transaction, which can be a burden on the user. As another example, in the second case, the user cannot inspect the product or a realistic replica of the product, so if the product is not satisfactory, the user will not know until after the product is shipped.

SUMMARY OF THE INVENTION

This summary is provided to introduce some concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

The present invention relates generally to computer systems, and more particularly, to a system and method capable of updating a database of real world items via a virtual world system. The present disclosure includes several technical advantages. As just one example, consider a situation where a database tracks the presence or status of physical objects in a particular location, such as a warehouse or storage facility. Using previous techniques, in order to update the database, a user would need to be present at the facility to inspect real world items (eg, using a barcode scanner). In the described embodiment, a user can inspect an object or manipulate a virtual representation of the object and make or trigger a corresponding database update without appearing in the location. Additional technical advantages are described below.

The system of the present disclosure includes at least one cloud server of a cloud server computer system, the at least one cloud server including at least one processor and memory storing a persistent virtual world system including one or more including Dummy objects for dummy data and models. The virtual objects include one or more of virtual twins, pure virtual objects, or applications, wherein at least one of the virtual objects represents a store of real-world items connected to a periodically updated database of real-world items of the store, and wherein persistent The sex virtual world system provides access to the database via the user interface of the user device.

In some embodiments, a store is a building, group of buildings, or other physical location (eg, a warehouse or storage facility) in which real-world items are kept that a user can access in virtual reality from a remote physical location. In such embodiments, the storage may include a connected device capturing data from the user that is communicated to the avatar through the persistent virtual world system. The at least one virtual object representing the store of real-world items may be a virtual twin stored in a persistent virtual-world system and in communication with a plurality of connected devices configured to recognize real-world items or user interactions with real-world items , and send corresponding status updates to the database, where users can access the store in augmented reality or remotely in virtual reality. The plurality of connected devices may include one or more cameras, including RGBD cameras, depth cameras, LIDAR cameras, or CCTV cameras, or a combination thereof, wherein the cameras employ artificial intelligence-based machine vision to identify real-world objects or users Interaction with real world objects. Alternatively, the connected device may include sensors including temperature sensors, humidity sensors, moisture sensors, mass sensors, proximity sensors, inertial sensors, light sensors, gas and chemical sensors, pressure sensors, touch sensors, electrical sensors, or combinations thereof one or more. The system may be configured to determine whether the user or the user's avatar has entered or exited a predetermined real or virtual area, which may be signaled to at least one server to complete transactions involving real world objects.

In some embodiments, the store is a virtual store represented by a corresponding virtual object. In these embodiments, the virtual store has no real world counterpart. A user can access a virtual store in virtual reality from a remote physical location to remotely interact with items (eg, purchase products) stored in a corresponding supplier's periodically updated inventory. In yet another embodiment, the remote physical location is one of a virtual reality-dedicated cabin, a room within a building, or an outdoor area that includes connected devices that capture data from the user that is communicated to the persistent virtual world system to User avatar.

In other embodiments, the store is a building or group of buildings, such as a "physical" store or warehouse, that has a corresponding virtual twin stored in a persistent virtual world system. In such an embodiment, a store may include multiple connected devices configured to identify products and selections of products, and send corresponding status updates to a database (eg, a centralized inventory database in a multi-store, multi-database arrangement, or a store-specific database). Users can visit the store in person in augmented reality or remotely in virtual reality. The store may be mapped from the real world, and each or most items within the store may include one or more virtual twins of real world elements, the one or more virtual twins including virtual data and models. A store may include items in specific physical locations within the store (eg, products located on one or more shelves) as well as multiple connected devices that feed the store's real world data to a persistent virtual world system. However, in cases where user activities (eg, shopping activities) occur remotely in virtual reality, inactive elements (eg, walls, shelves) may be less active when compared to active elements (eg, items that may need to be inspected and interacted with by the user). and other elements that do not require a high level of interaction with the user) the level of detail (LOD) may not need to be very high, e.g. when the product may require close inspection by the user, the active element may require a higher LOD because the user needs a higher horizontal interaction to make purchase transaction decisions.

In some embodiments, the plurality of connected devices in the store are one or more cameras including RGBD cameras, depth cameras, LIDAR cameras, or CCTV cameras, wherein the cameras use artificial intelligence-based machine vision to identify products and from one or more Shelves to select products. In some embodiments, the camera may be further configured to record events, scan and generate 3D images of the store, detect state changes in one or more elements within the store, detect expressions and movements from users, or a combination thereof.

In another embodiment, interaction data including selection data captured by one or more connected devices is sent to the at least one server to update one or more corresponding databases. As an example of a multi-database scenario, a company may include several warehouses, and transactions or selections may trigger updates to respective individual warehouse inventory databases and a centralized inventory database containing data from individual store-specific databases. In yet another embodiment, selection data and state changes in one or more physical locations within the store (eg, on one or more shelves) are further captured by other sensing mechanisms on the connected device, the sensing mechanisms Configured to detect a set of attributes associated with items in those locations, the sensing mechanism includes one or more of a temperature sensor, a humidity sensor, a moisture sensor, a mass sensor, a proximity sensor, an inertial sensor. Light sensors, gas and chemical sensors, pressure sensors, touch sensors, electrical sensors or a combination thereof. By providing multiple connected devices with a sensing mechanism that continuously captures data from the real world, the virtual world and each synchronized virtual twin are kept updated with real-time, multi-source data reflecting real-world conditions. In some embodiments, the multi-source data is processed by artificial intelligence algorithms that further provide the virtual twin with autonomous behaviors that can augment the real world. AI processing of data may include performing data, image and object recognition; data and object classification; data, image or object segmentation; data, image or object masking; data, image or object classification; multidimensional trajectory generation of data and objects; or a combination thereof.

In some embodiments, multiple connected virtual twins form a virtual twin network that enables peer-to-peer (P2P) connections to be established through direct physical communication or through indirect virtual twin-based connections. Furthermore, multiple virtual twin networks in the interconnected world region together with pure virtual objects and applications form a persistent virtual world system that can be accessed through appropriate interfaces of user devices. In some embodiments, the virtual twin includes logic, virtual data and/or models that can be input through multiple software platforms, software engines and/or sensors connected to connected devices. In this disclosure, logic involves rules defining operational instructions and behaviors embedded in computer code that may be necessary to program virtual twins or applications available in persistent virtual world systems. Virtual data are properties of the virtual twin, and models are graphical, mathematical and logical representations of any aspect corresponding to real-world elements that can be defined and managed by the logic contained therein.

In some embodiments, suitable models include one or more of 3D models, dynamic models, geometric models, and machine learning models. Models and data can be input via a number of external platform or engine services included in the persistent virtual world system. The multiple external platforms may include, for example, an Internet of Things (IoT) platform, a machine learning (ML) platform, a big data platform, or a simulation platform. External platforms can connect to persistent virtual world systems through application programming interfaces (APIs) and software development tools (SDKs) in order to provide and manipulate the virtual twin's models and data. Engine services included in persistent virtual world systems can include artificial intelligence, simulation, 3D and haptic features, and more. The persistent virtual world system is also connected to a spatial data streaming platform configured to receive and manage spatial data from real world elements and from their virtual twins. The system may also include digital reality interfaces, such as virtual reality, augmented reality, and fused reality interfaces, to interact with fused or virtual reality. These interactive interfaces and mechanisms are defined through the use of computer code included in computer scripts and programs, and may be affected by applications (eg, distributed applications) smart contracts, etc., which intelligently Contracts etc. are available in persistent virtual world systems where interactions occur, digitally encoding possible and desired interactions between all or most elements of the real world in predetermined areas and the virtual world, resulting in a fusion of interactive realities view.

Geometric models include mathematical models that define the shape of the virtual twin based on real-world elements. The 3D model, together with the geometric model, displays the data included in each geometry of the virtual twin, such as textures, colors, shadows, reflections, lighting, collision effects. 3D models include 3D data structures used to visually represent virtual twins and other virtual elements in persistent virtual world systems, such as applications, advertisements, virtual robots, and the like. The 3D data structure may include, for example, one or more octrees, such as sparse voxel octrees or dense voxel trees, which may be suitable for virtualizing the world into a persistent virtual world system through 3D scanning techniques. However, other data structures such as quadtrees, BSP trees, 3D arrays, kD trees, point clouds, wireframes, boundary representations (B-Reps), constructed solid geometry trees (CSG trees), binary trees, and hexagons can also be included. shape structure. The 3D data structures serve to accurately and efficiently represent data for each geometry of virtual objects in the persistent virtual world system.

Dynamic models represent mathematical models that describe the time-varying behavior of real-world objects in a virtual world, can include a set of states that occur in a defined sequence, and can include continuous (e.g., algebraic or differential equations) and discrete ( For example, as a state machine or stochastic model) dynamic model.

Machine learning models are mathematical representations of real-world objects and processes, typically generated by machine learning algorithms based on actual or simulated data that has been used as training data to learn. The model can implement artificial intelligence techniques that can be used to optimize the operation and/or performance of real-world elements through the virtual twin. Machine learning models can employ machine learning algorithms that allow virtual twins to be taught about the behavior of real-world elements in order to simulate the behavior of real-world elements.

In some embodiments, the model used in the virtual object takes into account the level of detail (LOD) required for a particular scene computation. LOD involves reducing the complexity of the model representation as virtual objects move away from the viewer, or according to other metrics such as object importance, viewpoint relative velocity or position. In some embodiments, the individual viewer's LOD is further adjusted based on the classification of individual viewers including artificial intelligence viewers (eg, AI shopping assistants) or human viewers (eg, human shoppers). In some embodiments, the LOD is also adjusted according to the sub-category of artificial intelligence viewers or human viewers. LOD is a feature commonly used in game engines to optimize real-time rendering, using more detailed models only where the user's point of view is closer to the object. LOD management increases the efficiency of computational processes (e.g. rendering processes) by reducing the amount of work used by the graphics pipeline (usually vertex transformations) or by enhancing physics simulation, as different physics models can be associated with virtual twins, from low fidelity models to high-fidelity models, enabling different simulations to be performed depending on the situation and situation.

In embodiments of the present disclosure, the system may use cloud-to-edge infrastructure that can implement distributed computing capabilities, including the use of public or private clouds, distributed ledger-based infrastructure, cloud applets, and edge systems, such as enterprise systems , mobile platforms and user devices. A distributed ledger-based infrastructure may be a decentralized and immutable blockchain or distributed ledger network that facilitates the transfer and storage of data necessary to widely distribute persistent virtual world systems that This includes pure virtual objects, virtual twins, applications, and any events or interactions within persistent virtual world systems. Through a cloud-to-edge infrastructure, resources including physical servers and network devices enable shared storage and computing that can allow for shared storage and computing based on factors such as the user's distance from the resource and network, and the relationship from the user to each other and to where they are located. The computing requirements at the location are dynamically allocated by factoring in the order to be more or less computed. In some embodiments, the dynamic allocation of storage and computation is encoded into smart contracts using a distributed ledger infrastructure, enabling transaction logic (eg, computer logic after each allocation of storage and computation) and the result of such allocation or Results are automated.

In some embodiments, to reduce hardware and network requirements, help reduce network latency, and improve the actual experience of general consolidation, systems may be connected via a network including millimeter wave (mmW) or a combination of mmW and sub 6GHz communication systems , such as communication via fifth-generation wireless systems (5G). In other embodiments, the systems may be connected via a wireless local area network (Wi-Fi). The provided communication system can allow low latency end-to-end (E2E) latency and high downlink speeds to endpoints in the field, which is necessary to perform typical highly interactive digital reality applications or other demanding applications parameter. This results in high quality, low latency, real-time streaming of digital application content. In other embodiments, the systems may be connected by fourth generation wireless system communications (4G) communications, may be supported by a 4G communications system, or may include other wired or wireless communications systems.

According to an embodiment, the sensing mechanism installed on the user equipment comprises a combination of an inertial tracking sensing mechanism and a transceiver, wherein the data tracked by this combination can be used as embedded tracking data or tracking elements of the sensed dataset data. Inertial tracking sensing mechanisms can utilize devices such as accelerometers, gyroscopes, magnetometers, etc., which can be integrated in an inertial measurement unit (IMU), or can be separate from the IMU. In one embodiment, the transceiver is a mmW transceiver. In an embodiment using a mmW antenna, the mmW transceiver is configured to receive mmW signals from the antenna and transmit data back to the antenna. Inertial sensors and position tracking provided by mmW transceivers and precise tracking provided by mmW-based antennas, low latency and high QOS capabilities enable sub-centimeter or sub-millimeter position and orientation tracking, which can be used to track the real-time position of user elements and improved accuracy when orienting. In some embodiments, tracking may be accomplished by employing several techniques known in the art, such as time of arrival (TOA), angle of arrival (AOA), or other tracking techniques known in the art (eg, visual imaging, radar technology, etc.). In alternative embodiments, the sensing mechanism and transceiver may be coupled together in a single tracking module arrangement.

In some embodiments, the persistent virtual world system also includes an application that the user can interact with through the user device. For example, the application may be a distributed application using the distributed cloud-to-edge infrastructure of the present disclosure. Applications can help define the types of interactions a user can have with elements of the persistent virtual world system, such as enabling interactive game-like mechanics with virtual twins or avatars of other users in the persistent virtual world system. The application may be obtained at an application repository stored in the server's memory or from an external source, such as by connecting to the server's external application repository and providing the application to the persistent virtual world system.

In some embodiments, at least one server may store a virtual world layer, which may be divided into an augmented reality layer and a virtual reality layer. A separate layer may enable access to persistent virtual world systems in any augmented or virtual reality through fused reality, and may be activated by a user device connected to at least one server whenever one or the other type of reality is accessed. Each layer can include enhancements to the reality and virtual twins specific to each layer. These layers may also include one or more applications from one or more application repositories stored in the memory of at least one cloud server, from an external platform, or through a blockchain or distributed ledger-based distributed database to obtain. An application can be one or more traditional applications, distributed applications or decentralized applications.

In some embodiments, selection of products in virtual or augmented reality is performed by virtually selecting one or more products, or optionally by physical interaction (eg, physically moving real-world objects, such as physically grabbing Picking and dropping one or more products into a real shopping basket, or gesture-based interactions such as physically touching a product and making a gesture corresponding to placing the product in a virtual shopping basket) to perform an augmented reality experience Product selection. Selection data for each is identified by one or more cameras, other sensing mechanisms, or a combination thereof, and sent to at least one server to update store inventory and shopping baskets. In further embodiments, a user may use a search engine connected to the inventory to enter (eg, by typing, by voice command, or by selecting from a menu) the product name through the augmented reality interface, thereby being able to access the inventory to find the desired product. Product information may include, for example, product location in the store, available brands, price, fragrance, size, and color.

In some embodiments, the store has an associated digital identifier stored in the at least one server, the digital identifier being retrieved by the user device from the at least one server for use through the persistent virtual world The system accesses the associated store. In some embodiments, the digital identifier is one or more of a QR code, a URL, an IP address, a MAC address, a cryptographic hash, a universally unique identifier, and an organizationally unique identifier. In some embodiments, the digital identifier is shared openly as part of the store's contact details. In other embodiments, the user receives a digital identifier from the store via an invitation.

In some embodiments, the store also includes a personal assistant dedicated to displaying viewing events to the user through the persistent virtual world system. In further embodiments, the personal assistant is an artificial intelligence agent represented by an avatar visible in a real-world location, or a real person.

In some embodiments, each user device has a personal identifier that enables at least one server to identify the user in order to provide a personalized experience. In further embodiments, each interaction performed by the user, including, for example, transactions and engagement actions, is recorded in the memory of at least one server under the user device personal identifier. In further embodiments, each interaction performed by the user is recorded in a smart contract on the distributed ledger under the user device personal identifier. In some embodiments, the recorded interaction data may be used in the future to create targeted advertisements or personalized experiences for the user.

In some embodiments, personalizing the experience includes virtually customizing a storefront and/or product counter to display features that match the user's preferences, or to provide the user's favorite products and brands, or to provide personalized advertising or interactive video games , or a combination thereof. In some embodiments, preference data linked to a personal identifier is determined based on transaction data, interaction data, and other preference data associated with the user's personal profile. In some embodiments, other preference data is obtained through user input or from a connected social media personal profile.

In some embodiments, the memory of the at least one server also stores payment card data from the user linked to the personal identifier. In a further embodiment, physically or virtually passing through or entering or leaving a predetermined area of the store, a signal is sent to the server to automatically pay for the product using the stored payment card data. In other embodiments, the transaction may be enabled by using a smart contract that records the binding purchase agreement on a blockchain or distributed ledger and initiates the transaction of the relevant physical or digital currency to be transferred. By tracking the user's device, the checkout area can precisely detect the geographic and spatial location of the user's avatar in order to detect when the checkout has passed. Passing through or entering or leaving a predetermined checkout area can trigger the cloud server to retrieve user payment information associated with the user's personal identifier linked to the user's account in the persistent virtual world system, connected to a payment institution, and automatically paid Purchased items and trigger automated retrieval and delivery systems.

According to one embodiment, a method includes providing at least one server of a cloud server computer system, the at least one server including at least one processor and memory, the memory storing a persistent virtual world system, the persistent virtual world system including a or more virtual objects including virtual data and models, the virtual objects including one or more of virtual twins, pure virtual objects, or applications; and providing at least one virtual object representing a physical location, and continuing, to be associated with the store's A database associated with real world items is connected to at least one virtual object representing the store. The method then continues by receiving interaction or transaction instructions from the user device related to one or more real world items, wherein the interaction or transaction associated with the received instructions is performed in augmented or virtual reality through the persistent virtual world system . Finally, the method updates the database based on the received instructions.

The above summary is not intended to be an exhaustive list of all aspects of the present disclosure. It is intended that the present disclosure include all systems and methods that can be practiced from all suitable combinations of the various aspects outlined above, as well as those disclosed in the following detailed description and particularly pointed out in the claims filed with this application. Such a combination has advantages not specifically stated in the above summary. Other features and advantages of the present disclosure will become apparent from the accompanying drawings and the following detailed description.

Description of drawings

Particular features, aspects and advantages of the present disclosure will be better understood with reference to the following description and accompanying drawings, in which:

1 depicts a diagram depicting a cloud server that may be used to implement a system according to an embodiment.

2A-2C illustrate various embodiments of the system.

3 depicts a diagram describing data and models used to generate a virtual twin network for use in the system, according to an embodiment.

4 shows a diagram describing a platform and interface for generating a virtual twin network for use in a system, according to an embodiment.

5 illustrates an example store using a store fused reality system according to an embodiment of the present disclosure.

6 illustrates an operational component diagram of a connection device that may be used in the system, according to an embodiment.

Figure 7 illustrates a method according to an embodiment.

Figure 8 depicts further steps of a method according to an embodiment.

Detailed ways

In the following description, reference is made to the accompanying drawings, which illustrate various embodiments by way of example. Furthermore, various embodiments will be described below by referring to several examples. It is to be understood that the embodiments may incorporate changes in design and construction without departing from the scope of the claimed subject matter.

In the present disclosure, an accurate and flexible system and method is provided that can update a database of real world items via a virtual world system. The systems and methods can provide different mixes of reality that enable reality to be replaced in virtual reality by persistent virtual worlds, or to have only some parts of reality replaced or augmented by pure virtual objects, as in augmented or mixed in reality. Users can interact with the storage of real-world items represented by virtual objects. Stores represented by virtual objects may or may not exist in the real world, and may allow users to access or interact with real-world items or their virtual twins. In one embodiment, the system allows buyers to remotely purchase products stored in corresponding periodically updated store inventory connected to the persistent virtual world system. A store may be a building, group of buildings, or other physical location, such as a warehouse or "brick and mortar" store, including a corresponding virtual twin in which real-world elements are replicated in virtual form to include in addition to In addition to the data, there are multiple data representing not only the visual aspects of the element but also the physical and behavioral aspects. A store may be a store or business that includes products and enables users to make purchases or other types of transactions, such as restaurants (eg, fast food or traditional restaurants), supermarkets, hardware stores, clothing stores, accessory stores, and the like.

Where the store is a physical entity in the real world, by virtualizing the real world, the systems and methods of the present disclosure also enable real-time interaction between real and virtual objects. In some embodiments, the system is also capable of machine-to-machine, human-to-machine, machine-to-human, and human-to-human interactions. Thus, the system enables users to virtually visit a store, check places of interest, interact with sellers, personal assistants, or even other real or virtual users, and conduct automated transactions, such as making purchases during store visits. In augmented reality, the user has the option to add items or even some parts of the store to suit the situation, such as viewing, inspecting, and when possible modifying things such as the color, size, price, taste, brand, promotion of one or more products of interest and other properties. Other virtual elements, such as pure virtual objects or graphical representations of apps and games, can also be configured in the store to further enhance the experience. In virtual reality, a user can view a store and select a desired product for a transaction.

1 shows a diagram depicting a cloud server 100 that may be used to implement a server computer system of the system, according to an embodiment.

Cloud server 100 includes one or more processors 102 and memory 104 including data storage with virtual objects 106, pure virtual objects 110 not available in the real world, and applications 112 that can be located in various parts of the world, virtual Object 106 includes a virtual twin 108 of real world elements. The plurality of connected virtual twins 108 form a virtual twin network 114 that enables peer-to-peer (P2P) connections to be established through direct physical communication or through indirect virtual twin-based connections. Multiple virtual twin networks 114 in interconnected regions of the world together with pure virtual objects 110 and applications 112 form a persistent virtual world system 116, which can be accessed through an appropriate interface of the user device. The user equipment may be one of a mobile device, a personal computer, a game console, a media center, a smart contact lens, and a head mounted display, among others. At least one virtual object 106 represents a store connected to a periodically updated database 118 associated with products of at least one store. One or more users may shop in the store through the persistent virtual world system 116 via the user interface of the user device, interacting in augmented or virtual reality. The periodically updated database 118 receives transaction orders from users and sends the orders to the corresponding stores.

In this disclosure, the term "persistence" is used to characterize a state of a system that can continue to exist without continuously executing processes or network connections. For example, the term "persistence" may be used to characterize a virtual world system in which the virtual world system and all virtual twins, pure virtual objects and applications contained therein continue to exist after the process used to create the virtual twins, pure virtual objects and applications ceases, And does not depend on the user to connect to the virtual world system. Thus, the virtual world system is saved in a non-volatile storage location (eg, in the cloud server 100). In this way, even if the user is not connected to the server, virtual twins, pure virtual objects and applications can interact and collaborate with each other when configured to achieve specific goals.

The virtual twin 108 of the present disclosure refers to a virtual version of a real world element, also referred to herein as a reality twin, which may be drawn or modeled by computer aided drafting (CAD) or computer aided engineering (CAE) methods, by A computer model of the real object (eg, building an information model BIMS) or other suitable method is input, and it may reflect not only the appearance of the reality twin, but also the behavior of the reality twin. Real-world objects may also include sensors, which may provide multi-source input data to the virtual twin 108 for enriching and synchronizing the virtual twin 108 with its corresponding real-world counterparts. Accordingly, the virtual twin 108 may obtain data from one or more sources (eg, from one or more real-world objects, environmental sensors, computing devices, etc.). As used herein, the term "multi-source data" refers to data that can be obtained from multiple sources.

As described above, the virtual twin can be updated based on multi-source data, eg, by updating known parameters or features, by enriching the virtual twin with additional parameters or features, and the like. In this disclosure, the term "enrichment" is used to describe the act of providing further features to the virtual twin based on multi-source data. Enriching the virtual twin can be thought of as a special form of updating the virtual twin with one or more new forms of data that may not have previously existed in the virtual twin. For example, enriching a virtual twin may refer to providing real-world data captured from sensing mechanisms on multiple devices, where additional real-world data includes video data, temperature data, real-time energy consumption data, real-time water consumption data, speed or acceleration data Wait.

The virtual twin network 114 of the system enables machine-to-machine, human-to-machine, machine-to-human, and human-to-human communication and interaction. Interaction mechanisms are defined through the use of computer code included in computer scripts and computer programs, and can be implemented through applications, smart contracts available in distributed databases based on blockchain or distributed ledgers, or when interactions and interactions occur The application's persistent virtual world system provides a combination of them. Thus, applications, smart contracts, or a combination thereof enable the digital encoding and representation of possible and desired interactions and interaction interfaces between all or most elements of the real world in a predetermined area by a persistent virtual world system. Interaction can be experienced by the user in the form of an interactive game-like application or through a game-like interaction mechanism. For example, interactions may be enabled during a virtual shopping visit to a physical location through a virtual twin of the physical location, whereby a user visiting such a location can virtually interact with elements configured therein.

2A-2C illustrate various embodiments of systems 200a-c. Some of the elements of Figures 2A-2C may refer to similar or identical elements to the system shown in Figure 1, and thus the same reference numerals may be used. The embodiments of Figures 2A-2B describe a store that a user can visit remotely in virtual reality or personally in augmented reality. More specifically, Figure 2A illustrates an embodiment in which a user remotely accesses a store in virtual reality; Figure 2B depicts an embodiment in which a user physically visits a store in augmented reality; Figure 2C depicts displaying products from multiple stores of virtual stores that users can access remotely in virtual reality.

The system 200a of FIG. 2A includes at least one cloud server 100 configured to store and process input data, the at least one server including at least one processor and memory including a data store that stores a persistent virtual world system 116 mapped from the real world , and includes one or more virtual twins of real-world elements that include virtual data and models. In some embodiments, virtual data and models provide the virtual twin with self-computing capabilities and autonomous behavior.

The system 200a also includes a store 202 containing real world items. In an embodiment, the real-time items include products (not shown) located in one or more shelves 204 . The store 202 includes a plurality of connected devices 206a that feed the real world data of the store 202 to the persistent virtual world system 116 of the at least one cloud server 100 . The cloud server 100 also includes a store virtual twin 208 in the persistent virtual world system 116, which receives real-world data from the connected device 206a and is updated in real-time by the real-world data. In addition, cloud server 100 includes a centralized database 118 associated with real-world items in store 202, which in turn is connected to one or more store-specific databases 210 that receive updates from store 202 (eg, product inventory updates). The store virtual twin 208 is accessible from the user's physical location 214 through the persistent virtual world system 116 in virtual reality, where the user's physical location 214 includes the connected device 206b that captures an image that includes transmission to the user's avatar through the persistent virtual world system 116 220 of the user's movement or expression.

Each of the store virtual twins 208 and the synchronized virtual twins of products or other elements within the store 202 use real-time information reflecting the conditions of the store 202 by providing a sensing mechanism to the plurality of connected devices 206a that continuously captures data from the store 202 . , multi-source data to keep up-to-date. In some embodiments, the multi-source data is processed by artificial intelligence algorithms that further provide the virtual twin with autonomous behaviors that can augment the real world. AI processing of data may include performing data, image and object recognition; data and object classification; data, image or object segmentation; data, image or object masking; data, image or object classification; multidimensional trajectory generation of data and objects; or a combination thereof.

In some embodiments, the plurality of connected devices 206a in the store 202 include one or more cameras, including RGBD cameras, depth cameras, LIDAR cameras, or CCTV cameras, wherein the cameras employ artificial intelligence-based machine vision to identify items and interact with Interaction of items (eg, products and selection of products from one or more shelves 206a). In some embodiments, the camera may be further configured to record events in the store, scan and generate 3D images of the store, detect changes in state of one or more elements within the store, detect expressions and movements from the user 212, or the like combination.

Users 212 located in user physical locations 214 may access store 202 via user devices 216 accessing persistent virtual world system 116 over network 218 and may interact with elements from store 202 via user avatars 220 . For example, user 212 may log into persistent virtual world system 116 via user device 216 and select a desired store 202, thereby virtually transporting user 212 to store 202. In some embodiments, the user 212 may be hosted by a personal assistant 222 visible at the store 202 . User 212 may interact virtually with personal assistant 222 through user avatar 220, or with any element included within store 202, such as with products and with other real or virtual users. Additionally, elements within store 202 may also interact with each other through their respective virtual twins operating within store virtual twin 208 of persistent virtual world system 116 .

In some embodiments, personal assistant 222 may be an artificial intelligence agent represented by an avatar viewable in store 202 through persistent virtual world system 116 , or a real person physically present in store 202 . For example, a real person could physically grab and hold a virtual or real shopping basket, select products virtually by the user, tell the user where each item is, and after payment, schedule delivery, or make the delivery for the user himself , as determined by the delivery system. The AI agent could for example inform the virtual user of the location of each product and augment each product's data, for example by providing further details such as price, brand, manufacturer, expiration date, available scents, etc. AI agents can also add products to virtual baskets. However, the user may perform some of these tasks individually, such as product expansion or placing selected products into the virtual basket. In any event, the digital basket can be automatically updated based on the total price the user 212 may have to pay so that the user 212 maintains control over his or her budget. This can be achieved, for example, through object recognition by cameras and a connection between them and an inventory that maintains an updated price list.

In some embodiments, data for elements in the store 202 includes real-space data entered directly during the development phase of the storage virtual twin 208 , or captured and transmitted to the storage virtual twin 208 . When real-space data is captured and sent to the store virtual twin 208, the locations and orientations of objects and people within the store 202 are mapped or sensed by the plurality of connected devices 206a and updated in real-time in the persistent virtual world system 116, The user 212 is enabled to view any changes in the store 202 in real time.

FIG. 2B shows an embodiment of system 200b in which user 212 visits store 202 in person without a user avatar. Such access to store 202 may be in augmented reality, enabling user 212 to physically walk in store 202, check desired products and interact with a personal assistant or other real or virtual user in store 202. Thus, system 200b is similar to system 200a, except that the user is physically present in store 202 and therefore does not need to have a different physical location of the user, utilizing a connected device that captures user data.

In these embodiments, product selection is performed by virtually selecting one or more products (eg, selecting items from a distance without touching them) and tossing the products into a virtual shopping basket, or by physically grabbing One or more products and tossing one or more products into a physical basket is performed. Selection and/or transaction data associated with each product is identified by the connected device 206a including one or more cameras or other sensing mechanisms and sent to the store-specific database 210 to update the centralized database 118 in the at least one cloud server 100 .

FIG. 2C illustrates an embodiment of system 200c in which a user from physical location 212 accesses a virtual store represented by a pure virtual object, referred to herein as virtual store object 226 . The virtual store object 226 represents a virtual store without a real world counterpart that a user can access in virtual reality to remotely purchase products stored in the corresponding supplier's periodically updated store inventory database 1-N 210a-c, such as a store 1-N 202a-c. Thus, in the exemplary embodiment shown in FIG. 2C, the products available in the virtual store object 226 may vary according to the preferences of the user 212, wherein each preference selection may trigger product display adjustments in the virtual store object 226 and information about the corresponding Updates to Store Inventory Database 1-N 210a-c.

In some embodiments of Figures 2A and 2C, the user physical location 214 may be, for example, one of a virtual reality dedicated cabin, a room within a building, or an outdoor area, all of which include a connected device 206b captured from the user Data communicated to the user avatar by the persistent virtual world system 116 . In further embodiments, during remote access by physically moving within the physical room or by manipulating a controller interface on the user device (eg, a controller pad, buttons, joystick, air movement on an air interface, etc.) of user movement and interaction.

In some embodiments of the systems 200a-c of Figures 2A-2C, the store 202 or one or more of the stores 1-N 202a-c have an associated numerical identifier stored in the at least one cloud server 100, the number The identifier is retrieved by the user device 216 from the at least one cloud server 100 for accessing the associated store 202 or stores 1-N 202a-c through the persistent virtual world system 116 . In some embodiments, the digital identifier is one or more of a QR code, a URL, an IP address, a MAC address, a cryptographic hash, a universally unique identifier, and an organizationally unique identifier. In some embodiments, the digital identifier is shared openly as part of the store's 202 contact details. In other embodiments, the user receives a digital identifier from the store 202 via an invitation.

In some embodiments of the systems 200a-c of FIGS. 2A-2C, each user device 216 has a personal identifier that enables at least one server to identify the user 212 in order to provide the user 212 with a personalized experience. In further embodiments, each interaction by user 212 , including, for example, transaction and appointment actions (eg, selecting or checking a product) is recorded under a user device personal identifier in the memory of at least one cloud server 100 . In further embodiments, each interaction performed by the user 212 is recorded in a smart contract on the distributed ledger under the user device personal identifier. In some embodiments, the recorded interaction data may be used in the future to create targeted advertisements or personalized experiences for the user. In some embodiments, personalizing the experience includes customizing the storefront virtually to display features that match user 212's preferences, or provide user 212 with favorite products and brands, or provide personalized advertising or interactive video games, or its combination. In some embodiments, preference data linked to a personal identifier is determined based on transaction data, interaction data, and other preference data associated with the user's personal profile. Other preference data associated with the user's personal profile may be obtained by the user entering personal preferences when filling out the personal profile through the persistent virtual world system 116, or may be obtained from the social media profile if previously authorized by the user 212 The data. User 212 may, for example, fill out one or more forms or questionnaires that give details about the user's preferences.

In some embodiments of the systems 200a-c of Figures 2A-2C, the memory of the at least one cloud server 100 also stores payment card data from the user 212 linked to the personal identifier. In a further embodiment, the at least one cloud server 100 is signaled to automatically pay for the product by retrieving stored payment card data via a predetermined area of the store 202 that represents the store checkout area 224 . In other embodiments, transactions may be enabled by using a smart contract that records the binding purchase agreement on a blockchain or distributed ledger and initiates the transaction of the relevant physical or digital currency to be transferred. By tracking the user device 216, the checkout area 224 can accurately detect the geographic and spatial location of the user avatar 220 in order to detect when the checkout area 224 is passed. Passing through the predetermined checkout area 224 may trigger at least one cloud server 100 to retrieve user payment information associated with the user's personal identifier linked to the user's account in the persistent virtual world system, linked to a payment institution, and automatically paid Purchased items and trigger retrieval and delivery systems. For example, the retrieval system may be an autonomous retrieval system that includes computer circuitry and electromechanical devices programmed to search and retrieve selected items. In another example, the retrieval system may be executed directly by the personal assistant 222 of one or more persons responsible for retrieving products. The delivery system can organize the user's order and optimize the route for delivering the selected product to the user 212 .

In some embodiments of the systems 200a-c of Figures 2A-2C, interaction with the store virtual twin 202 or virtual storage object 226 is provided by computer code included in computer scripts and computer programs, and by including traditional, distributed application of decentralized or decentralized applications; smart contracts; etc., or a combination thereof to enable interaction. Thus, applications, smart contracts, or a combination thereof, enable the digital encoding of various virtual interactions, such as virtual purchases, virtual investments, virtual auctions, virtual plays, virtual tests, virtual inspections, virtual touches, virtual gazes, virtual physical speaking, virtual listening , virtual physical interaction, virtual voting, virtual signing, virtual drawing, virtual creation, virtual creation, virtual collaboration. Virtual movement, virtual driving, virtual flying, virtual design, virtual training, virtual learning, virtual education, virtual remote control, virtual feeling, virtual emotion, virtual shaking, virtual waving, virtual payment, virtual sharing and virtual collecting, etc. These interactions can be implemented individually, or with other real or virtual clients, or with other real or virtual clients through the live store virtual twin 208, to create corresponding real world effects and interactions.

In some embodiments, the systems 200a-c of Figures 2A-2C can use a cloud-to-edge infrastructure that can be implemented using public or dedicated cloud servers, fog servers, and services such as enterprise systems, mobile platforms, machines, Distributed computing power of other systems such as robots, vehicles, sensors, or user equipment, all of which may be connected via network 218 . Using cloud-to-edge computing networks, access to computing power, computer infrastructure (eg, through so-called infrastructure as a service, or IaaS), applications and business processes can be delivered as a service to users via client devices as needed. In some embodiments, the cloud-to-edge infrastructure also includes a distributed ledger-based infrastructure that facilitates the transfer and storage of data necessary to widely distribute the persistent virtual world system 116, which is 116 includes virtual twins, pure virtual objects, applications, and any events or interactions within persistent virtual world system 116 . In this way, resources including physical servers and network devices enable shared storage and computation that can be based on factors such as the user's distance to the resource and the network and computations from users relative to each other and to their location Demand factors are dynamically allocated, so that commands are calculated more or less accordingly. In some embodiments, the dynamic allocation of storage and computation is encoded into smart contracts using a distributed ledger infrastructure, automating the transaction logic (eg, computer logic after each allocation of storage and computation) and the outcome or outcome of the allocation .

In some embodiments of the systems 200a-c of Figures 2A-2C, tracking of the user equipment 216 is performed by one or more of time of arrival (TOA), angle of arrival (AOA), and visual imaging techniques. Tracking can be performed by one or more of radar technology, antennas, Wi-Fi, inertial measurement units, gyroscopes, and accelerometers.

In some embodiments of the systems 200a-c of FIGS. 2A-2C, to reduce hardware and network requirements, help reduce network latency, and improve the actual experience of general consolidation, the systems may be connected through a network 218, including mmWave ( mmW) or a combination of mmW and sub 6GHz communication systems such as communication via fifth generation wireless systems (5G). In other embodiments, the system may be connected via a wireless local area network (Wi-Fi) that provides 60GHz data. The provided communication system can allow low latency and high Gbps downlink speeds to endpoints in the field, which are within the parameters necessary to perform typical highly interactive digital reality applications or other demanding applications. This results in high quality, low latency, real-time streaming of digital application content. In other embodiments, the systems may be connected by fourth generation wireless system communications (4G) communications, may be supported by a 4G communications system, or may include other wired or wireless communications systems.

In other embodiments of the systems 200a-c of Figures 2A-2C, a global navigation satellite system (GNSS), collectively referred to as a plurality of satellite-based navigation systems, may be used to enable positioning of the device, such as GPS, BDS, Glonass, QZSS, Galileo and IRNSS. Using signals from a sufficient number of satellites and techniques such as triangulation and trilateration, GNSS can calculate a device's position, velocity, altitude and time. In one embodiment, the external positioning system is augmented by Assisted GNSS (AGNSS) through the architecture of existing cellular communication networks, where the existing architecture includes 5G. In other embodiments, the AGNSS tracking system is also supported by a 4G cellular communication network. In indoor embodiments, GNSS is further enhanced by a radio wireless local area network such as Wi-Fi, preferably but not limited to, providing 60GHz data. In alternative embodiments, the GNSS is augmented by other techniques known in the art, such as by Differential GPS (DGPS), Satellite Based Augmentation Systems (SBAS), Real Time Kinematics (RTK) systems. In some embodiments, tracking of the device is achieved through a combination of AGNSS and inertial sensors in the device.

3 depicts a diagram of a system 300 describing the design of a virtual twin network and a derived virtual twin network, according to an embodiment. Some elements of Figure 3 may refer to similar or identical elements to Figures 1-2C, and thus the same reference numerals may be used.

As shown in FIG. 3 , elements in the real world 302 including the connecting devices 206a-b and the user device 216 are connected to the persistent virtual world system through a data flow 304. These data flows 304 may be unidirectional or bidirectional, depending on the capabilities of the connecting devices 206a-b. As explained with reference to Figures 1-2C, persistent virtual world systems may be stored in one or more cloud servers 100 and shared across cloud-to-edge computing and networking infrastructure.

The data stream 304 sent by the connection device 206a may be obtained by sensors installed on the connection device 206a-b. In addition to sensor data, the data stream 304 of the user device 216 may include user input data generated by interacting with the application via the user device 216 . The virtual world and each synchronized virtual twin 108 are maintained with real-time, multi-source data reflecting the conditions of the real world 302 by providing a sensing mechanism to the plurality of connected devices 206a and user devices 216 that continuously captures data from the real world 302 to update.

Interaction with the virtual twins 108 in the persistent virtual world system is accomplished through data exchange using a publish/subscribe service connected to the data store 306 of each virtual twin 108 . Data types can include periodic and aperiodic, synchronous and asynchronous data. Each virtual twin 108 holds a data store 306 where data can be updated by the real twin or microservices (not shown). The data in the persistent virtual world system may be directly associated with a particular virtual twin 108 , or may be processed as anonymous data, which may include an aggregation of multiple streaming sources of related virtual twins 108 . For example, data from all products of a particular brand or type can be processed and aggregated to stream data that can be used for predictive maintenance services.

Microservices are individual services that can be independently developed, deployed, and maintained, each configured to perform discrete tasks and communicate with other services through APIs. Each microservice may use the virtual twin model and relationship to the environment to update data in the virtual twin's 108 data store 306 to modify the value of a particular property of the virtual twin 108 . Microservices can use specific core services of persistent virtual world systems, such as multiple engines, or connect to external platforms, as shown in Figure 4.

Data flow 304 may be stored in data store 306 via connector 308 . Connector 308 may include software and hardware for reading data from data stream 304 and writing data to data store 306 . The connector 308 may use a publish/subscribe application programming interface (API) to connect to the data store 306 and help feed the data stream 304 from the connected devices 206a-b and user devices 216 to the virtual twin 108. The data stream 304 of the connecting devices 206a-b is also fed through the connector 308 to the cyber-physical system 310, while the data stream 304 of the user device 216 is fed through the connector 308 to the user's avatar 312. The system 300 also includes an embodiment of a robot 314, which may include hardware and software configured to respond as an automated agent with human or human-like behavior by employing machine learning algorithms. Human avatar 312 may be configured to display physical characteristics of a human user, or may be configured to have different visual aspects and characteristics.

In this disclosure, a virtual-reality twin or twin pair may be considered a cyber-physical system 310, which is the integration of computing with physical processes whose behavior is defined by the network and physical parts of the system. Thus, virtual twin 108 is a networked portion of cyber-physical system 310 . The virtual twin 108 can then be thought of as an extension of the real twin that allows connecting physical parts with artificial intelligence and simulations to improve the capabilities and performance of objects. In some embodiments, virtual twin 108 may be a replacement for a portion of physical components and processes. For example, in the case of a sensor failure in the real counterpart, the sensing input of the real twin is provided by the interaction of the virtual twin in the virtual world. In another example, if the battery is running low in the real twin, part of the computation of the real twin can be done in the virtual world.

The virtual twin 108 may also include a model 316 that involves any graphical, mathematical or logical representation of a realistic aspect of reality that can be used to replicate reality in a persistent virtual world system. In some embodiments, suitable models 316 include one or more of 3D models 318 , geometric models 320 , dynamic models 322 , and machine learning models 324 . Although only four models are disclosed here, those skilled in the art will appreciate that the system may be adapted to implement fewer or more models than those presented, as desired.

The 3D model 318, along with the geometric model 320, displays the data included in each geometry of the virtual twin 108, such as textures, colors, shadows, reflections, collision effects, and the like. The 3D model 318 includes a 3D data structure used to visually represent the virtual twin 108 and other virtual elements in the persistent virtual world system, such as applications, advertisements, virtual robots, and the like. In one embodiment, the 3D data structure is an octree data structure, such as a sparse voxel octree or a dense voxel octree. In other embodiments, other suitable data structures may be used, such as quadtrees, BSP trees, 3D arrays, kD trees, point clouds, wireframes, boundary representations (B-Reps), constructed solid geometry trees (CSG trees) , binary trees and hexagonal structures. The 3D data structure serves to accurately and efficiently represent data for each geometry of the virtual twin in the persistent virtual world system. The correct choice of 3D data structure depends on the origin of the data, the precision of the geometry sought during rendering; whether the rendering is done in real-time or pre-rendered; whether rendering is performed via a cloud server, via a user device, a fog device, or a combination; Specific applications of persistent virtual world systems, e.g., sessions that need to examine specific textures and details of a product may require higher-level definitions; memory capacity from the server and from user devices, and thus required memory consumption; and others .

Geometric model 320 includes a mathematical model that defines the shape of virtual twin 108 based on real-world elements, and may be supplemented by 3D model 318 .

Dynamic model 322 represents a mathematical model that describes the time-varying behavior of real-world objects in a virtual world, can include a set of states that occur in a defined sequence, and can include continuous (eg, algebraic or differential equations) and discrete (eg as a state machine or stochastic model) a dynamic model.

Machine learning models 324 are mathematical representations of real-world objects and processes, typically generated by machine learning algorithms based on actual or simulated data that has been used as training data to learn from. The model can implement artificial intelligence techniques that can be used to optimize the operation and/or performance of the real twin through the virtual twin. The machine learning model may employ machine learning algorithms that allow the virtual twin 108 to be taught about the behavior of the real twin in order to simulate the behavior of the real twin.

In some embodiments, the model 316 used by the virtual twin 108 calculates the required level of detail (LOD) for a particular scene. LOD involves reducing the complexity of the model representation as the virtual twin 108 moves away from the observer, or according to other metrics such as object importance, viewpoint relative velocity, observer classification or location. LOD is a feature commonly used in game engines to optimize real-time rendering, using more detailed models only where the user's point of view is closer to the object. LOD management increases the efficiency of computational processes (such as rendering processes) by reducing the workload on graphics pipeline usage (often vertex transformation) or by enhancing physics simulation, as different physics models can be associated with the virtual twin 108, from low The fidelity model to high fidelity model enables different simulations to be performed depending on the situation, and situational LOD management increases the efficiency of computational processes (eg rendering processes) by reducing the workload on graphics pipeline usage. Typically, vertex transformations, or through augmented physics simulations, can be associated with the virtual twin 108 because different 3D models 318 or dynamic models 322 can be associated with the virtual twin 108 from a low-fidelity model to a high-fidelity model, so that different behaviors can be performed depending on the situation and situation. simulation. In general, LOD management can improve frame rates and reduce memory and computational requirements. LOD management also allows for an optimized user experience based on specific needs and situations. In some embodiments, the LOD is further adjusted to individual viewers, wherein the adjustment includes considering rendering based on whether the viewer is an artificial intelligence viewer (eg, an AI host or sales assistant) or a human viewer (eg, a buyer) LODs for specific scenarios. In some embodiments, the LOD of inactive elements (eg, walls, shelves, and other elements that do not require high-level user interaction) compared to active elements (eg, store products that may require inspection and user interaction) It may not need to be very high, and active elements may require higher LODs because the user needs a higher level of interaction to make a purchase transaction decision.

A plurality of connected virtual twins 108 form a virtual twin network 114 . Each virtual twin 108 in the virtual twin network 114 may display social connections 330 with each other, ie, interactions with each other. In some embodiments, these social connections 330 may be managed through a social media network.

In some embodiments, virtual twin 108 includes one or more of 3D world and building data, such as data based on SLAM or derived mapping; 3D geometric data; 3D point cloud data; or geographic information representing real-world structural properties System data, which can be used to model 3D structures for digital reality applications.

In some embodiments, each of the virtual twins 108 may be geolocated using a reference coordinate system suitable for use with current geolocation techniques. For example, the virtual twin may use a World Geodetic System standard such as WGS84, the current reference coordinate system used by GPS.

4 shows a diagram of a system 400 describing a platform and interfaces for generating a virtual twin network for use in the system, according to an embodiment. Some elements of FIG. 4 may refer to elements similar or identical to those of FIG. 1 . 1-3 can therefore use the same reference numerals. Elements located within the dashed lines represent the virtual twin and persistent virtual world system 116 in which the virtual twin network 114 resides.

As shown in FIG. 4 , the virtual twin network 114 may connect to multiple external platforms or to engine services included in the persistent virtual world system 116 . The multiple external platforms can include, for example, one or more Internet of Things (IoT) platforms 402, machine learning (ML) platforms 404, big data platforms 406, and simulation platforms 408, which can be developed through application programming interfaces (APIs) and software Tools (SDK) 410 connect to persistent virtual world system 116 to provide and manipulate models and consume or publish data to the virtual twin.

IoT platform 402 refers to software and/or hardware capable of managing multi-source input data received from sensors in connected devices and user devices. ML platform 404 refers to software and/or hardware that provides the virtual twin with the ability to use machine learning models and algorithms for artificial intelligence applications. Big data platform 406 refers to software and/or hardware that enables an organization to develop, deploy, operate, and manage big data associated with virtual twin network 114 . Simulation platform 408 involves software and/or hardware that enables the use of virtual twins and their data and models to virtually reconstruct the real-world behavior of real-world entities.

Engine services included in persistent virtual world system 116 may include artificial intelligence engine 412, simulation engine 414, 3D engine 416, and haptic engine 418, among others. The artificial intelligence engine 412 refers to software and/or hardware that enables the management and application of machine learning models and algorithms for artificial intelligence applications. Simulation engine 414 involves software and/or hardware that enables the use of the virtual twin and its data and models to virtually recreate the real-world behavior of real-world entities. 3D engine 416 refers to software and/or hardware that can be used to create and process 3D graphics of the virtual twin. Haptic engine 418 refers to software and/or hardware that enables haptic features to applications and virtual twins to provide touch-based interactions to users. The persistent virtual world system is also connected to a spatial data streaming platform 420 configured for real and virtual spatial data in the persistent virtual world system and between the persistent virtual world system and the fused reality 422 optimized exchange and management.

The 3D engine 416 and the haptic engine 418 may connect to the fused reality 422 via an appropriate digital reality interface 424 in the user device, enabling access to the fused reality 422 in any virtual or augmented reality. The fused reality 422 provides the user with an extended reality in which elements of reality are overlaid or augmented by persistent virtual twins affixed to specific geographic locations or reality elements in reality, and include AI and simulation of the virtual twin connected to reality. Users can interact with this fused reality 422 through their avatars without limit.

The virtual twin network 114 is an integral part of the persistent virtual world system 116 and enables a virtual twin reality 426 in which all or most real world elements are fully virtualized and can be virtually augmented (eg, to a virtual twin added feature). In the current disclosure, virtual twin reality 426 differs from the typical concept of virtual reality in that virtual reality can represent an immersive realization of a world in which all or most elements are virtual, while virtual twin reality 426 takes context into account, based on reality Precise geographic locations of world objects, and interactions and connections between virtual twins that are kept continuously updated through data and models imported and manipulated via multiple platforms and/or engines. Thus, virtual twin reality 426 refers to an actual virtual twin of a world within a persistent virtual world system, where the persistent virtual world system provides data, models, interactions, connections, and infrastructure, and in some embodiments, each virtual twin can be implemented of self-calculation and autonomous behavior.

In further embodiments, the system 400 may store separate layers for augmented reality and virtual reality in memory of at least one server. A separate layer may allow virtual twin reality 426 to access any augmented or virtual reality through fused reality 422, and access to one or other types of reality may be activated by a user device connected to at least one server. Each layer can include enhancements to the reality and virtual twins specific to each layer. For example, when accessing fused reality 422 in augmented reality, a user may view the real objects located in the current fused reality scene, the current augmentation of each real object via the corresponding virtual twin, and those configured to be visible only in augmented reality Pure virtual objects. In another example, when viewing fused reality 422 in virtual reality, a user may only view a version of virtual twin reality 426 configured for virtual reality, including augmentations configured for virtual reality views only. However, when in virtual reality, the user can activate the augmented reality layer in order to view augmented and virtual objects that originally went to augmented reality. Likewise, when in augmented reality, the user can activate the virtual reality layer for full transfer to virtual reality, while still being able to view the augmentation in augmented reality

"Self-computing capability," also referred to herein as "self-managing capability," refers to the ability to apply artificial intelligence algorithms in order to autonomously manage computing resources (eg, distributed computing resources). In one embodiment, a virtual twin with self-computing capabilities in a persistent virtual world system can autonomously manage computing resources to adapt to changes in the environment of the corresponding real-world element or the real-world element itself. Self-governing rules and conditions can also be managed through the use of smart contracts running on blockchain or distributed ledger technology to further encode rules and conditions in a distributed and transparent manner. Thus, in one embodiment, each virtual twin can exhibit autonomous behavior, autonomously sending and executing commands and/or Generate events as needed for each environment) act autonomously. Achieving this type of behavior may require training the virtual twin with artificial intelligence algorithms during its modeling and development. Thus, the role of the virtual dual developer may be limited to defining general policies and rules that guide the self-management process.

5A-5B depict a sample store according to embodiments of the present disclosure. Some elements of FIG. 5 may refer to similar or identical elements to FIG. 1 . 1-4 can therefore use the same reference numerals.

In Figure 5A, a shopper avatar 220 has accessed store 202 from a remote physical location (eg, user physical location 214 of Figure 2A) through a persistent virtual world system. The shopper avatar 220 can view and interact with the shopping assistant 222, which can be a real human or an artificial intelligence assistant; interact with a third-party real-life shopper 502 authorized for such interaction; with a third-party shopper authorized for such interaction The avatar 504 interacts; Shelf 204 also includes connected devices 206a (eg, cameras and/or IoT devices) configured to identify and select products from one or more shelves 204 and/or detect a product 506 associated with the shelf 204. group attributes, and/or use the data to update store virtual twins and store inventory available in at least one cloud server.

Each or most objects within the store 202 include a corresponding virtual twin included in a persistent virtual world system stored and computed in at least one cloud server, each virtual twin including logic that provides self-computing capabilities and autonomous behavior, virtual data and model. Virtual twins of objects in the store 202 are included in a virtual world layer 508 that is mapped and overlaid on top of the real world, in which one or more applications 510 (eg, applications 1-3) are configured that can be downloaded from the store. Available from one or more application repositories in the memory of at least one cloud server, available from external platforms, or available through a distributed database based on a blockchain or a distributed ledger. The virtual world layer 508 may include or be connected to one or more smart contracts that record interactions in a blockchain or distributed ledger.

Application 510 may be one or more traditional applications, distributed applications or decentralized applications. Traditional applications are usually based on the traditional client-server model and run on dedicated servers in a static infrastructure. Distributed applications are applications that are mainly stored on cloud computing platforms (such as the cloud server of the present disclosure), and can run on multiple systems and devices on the same network at the same time, or can be run on blockchain or distributed ledger-based systems. run on a distributed database. Decentralized applications mainly run on decentralized infrastructure, such as in distributed databases based on blockchain or distributed ledgers.

Virtual world layer 508 may also include separate layers for augmented and virtual reality objects and experiences, depicted in FIG. 5 as AR layer 512 and VR layer 514, respectively. Each of the AR or VR layers 512-514 may be activated by user equipment 216 connected to at least one cloud server, and may include augmentation of reality through virtual twins, applications, or a combination thereof dedicated to each layer . Thus, for example, a shopper can visit any augmented or virtual reality store 202 and can view the corresponding augmentation of each reality by activating the corresponding AR or VR layer 512-514. In another example, a shopper may activate AR layer 512 and be able to view enhancements corresponding to product 506 through a virtual twin of product 506, such as being able to view, examine, and if possible modify one or more attributes about product 506 ( For example, available brands, colors, fragrances, promotions, prices, etc.). In another example, a shopper can activate the AR layer 512 and view other third-party real-world shoppers 502 within the store 202 . In another example, a shopper can only remotely activate VR 514 and thus cannot view any additions to products 506, but only the store virtual twin and, if authorized, other third-party shopper avatars 504.

In another embodiment, the augmented reality layer 512 may provide shoppers with the store's interactive interface and information system stored in a cloud server, where all logical transactions and data tracking are stored within the information system for further analysis, personal Quotation, trajectory analysis, etc. Thus, through the augmented reality layer 512 of the store 202, the persistent virtual world system can further provide the shopper with personalized offer displays and shelf space information, or spatial contextual recommendations based on specific gesture activities or behaviors, or with the shopper's Preferences associated with the personal identifier (eg, entered by the user or obtained from other sources previously authorized by the shopper (eg, social media)), or a combination thereof. For example, if a shopper picks up and then returns to the chocolate in store 202, that action may trigger the cloud server's information system to provide the user with other relevant offers in augmented reality.

In some embodiments, selection of products 506 is performed by virtually selecting one or more products 506 (eg, selecting items from a distance without touching them) and dividing the products 506 into virtual baskets 516, or by physically Grabbing one or more products 506 and dividing one or more products 506 into physical baskets 518 is performed. Each selection 506 is identified by one or more connected devices 206a and transmitted to at least one server to update store inventory and corresponding baskets.

In FIG. 5B , corresponding shoppers visit virtual store 520 through respective shopper avatar A 522 and shopper avatar B 524 , who may be served by a human or AI shopping assistant 522 . The virtual store 520 includes a product counter 526 that can be customized for each user. For example, a user associated with shopper avatar A 522 may select red or a specific texture and/or design for product counter 526, while a user associated with shopper avatar B 524 may select blue and different textures and/or designs for product counter 526 / or design. Other areas of the store can also be adjusted according to user preferences, such as store background, shelves, etc. Additionally, the user may select different products 506 to be displayed on the product counter 526, which may be retrieved from each store's respective inventory database. The user may also view object enhancements, such as product descriptions 528 , promotional advertisements 530 , or games 532 . As a result, although shopper avatar A 522 and shopper avatar B 524 can view and interact with each other, the experience of each can be different because each can view the store in a different way.

As described with reference to Figures 2A-2C, to adjust product offerings, product counter 526 and/or preference data for general store design may be linked to user personal identifiers, and may be based on transaction data, interaction data, and other preference data (eg, When the user authorizes, it is determined by filling out a personal shopper profile, etc. from a social media profile). Preference data may be entered directly by the user prior to initiating a session on a store, such as virtual store 520, or may be automatically determined by the persistent virtual world system from preference data associated with the user's personal identifier.

FIG. 6 shows an operational component diagram 600 of a device that may be used in the system, according to an embodiment.

Device 602 may include operational components such as input/output (I/O) module 604; power supply 606; memory 608; sensing mechanism 610 and transceiver 612 forming tracking module 614; and network interface 616, all operably connected to processor 618.

I/O module 604 is implemented as computing hardware and software configured to interact with a user and provide user input data to one or more other system components. For example, I/O module 604 may be configured to interact with a user, generate user input data based on the interaction, and provide the user input data to processor 618 before being communicated to other processing systems, such as servers, via a network. In another example, I/O module 604 is implemented as an external computing pointing device (eg, touch screen, mouse, 3D controls, joystick, game pad, etc.) and/or text input device (eg, keyboard, dictation tool, etc.) , which is configured to interact with other connected elements. In other embodiments, the I/O module 604 may provide more, less, or different functionality than those described above.

Power supply 606 is implemented as computing hardware and software configured to power device 602 . In one embodiment, the power source 606 may be a battery. The power source 606 may be built into the device or removable from the device, and may or may not be rechargeable. In one embodiment, the device may be re-powered by replacing one power supply 606 with another power supply 606 . In another embodiment, the power supply 606 may be recharged via a cable connected to a charging source, such as a Universal Serial Bus ("USB") FireWire, Ethernet, Thunderbolt or headphone cable connected to a personal computer. In yet another embodiment, the power source 606 may be recharged by inductive charging, wherein an electromagnetic field is used to transfer energy from the inductive charger to the power source 606 when the inductive charger and the power source 606 are in close proximity, but not through The cables are plugged into each other. In another embodiment, a docking station may be used to facilitate charging.

Memory 608 may be implemented as computing hardware and software suitable for storing application instructions and for storing multi-source data captured by sensing mechanism 610 . Memory 608 may be of any suitable type capable of storing information accessible to processor 618, including computer-readable media, or other media that store data that can be read by means of electronic devices, such as hard drives, memory cards, etc. , Flash drives, ROM, RAM, DVD or other optical disks, and other writable and read-only memories. In addition to persistent storage, storage 608 may also include temporary storage.

Sensing mechanism 610 may be implemented as computing hardware and software suitable for obtaining multi-source data from the real world and determining/tracking the position and orientation of device 602, and thus determining/tracking one or more devices to which device 602 may be linked The location and orientation of real-world elements. Sensing mechanism 610 may include, but is not limited to, one or more temperature sensors, proximity sensors, inertial sensors, infrared sensors, pollution sensors (eg, gas sensors), pressure sensors, light sensors, ultrasonic sensors, smoke sensors, touch sensors, color sensors sensor, humidity sensor, water sensor, electrical sensor or a combination thereof. In particular, the sensing mechanism 610 includes one or more inertial measurement units (IMUs), accelerometers and gyroscopes. The IMU is configured to measure and report the velocity, acceleration, angular momentum, translational velocity, rotational velocity, and other telemetry metadata of the device 602 using a combination of accelerometers and gyroscopes. An accelerometer within the IMU and/or an accelerometer configured separately from the IMU may be configured to measure acceleration of the interacting device, including acceleration due to the Earth's gravitational field. In one embodiment, the accelerometer includes a triaxial accelerometer capable of measuring acceleration in three orthogonal directions.

Transceiver 612 may be implemented as computing hardware and software configured to enable a device to receive radio waves from an antenna and send data back to the antenna. In some embodiments, mmW transceiver 612 may be used, which may be configured to receive mmW wave signals from the antenna, and transmit data back to the antenna when interacting with immersive content. The transceiver 612 may be a two-way communication transceiver 612 .

According to an embodiment, transceiver 612 enables direct communication between computing devices via a distributed ledger-based communication pipeline connected to a network. A distributed ledger-based communication pipeline may enable direct communication between devices 602 over a decentralized network by using cryptography to allow the use of cryptographic "keys" and cryptographic signatures to store information in a secure and accurate manner. In other embodiments, a distributed ledger may also be used between the server and the device 602, where the server may be responsible for validating the data before distributing it to each device 602.

In an embodiment, the tracking module 614 may be implemented by combining the capabilities of the IMU, accelerometer and gyroscope with the position tracking provided by the transceiver 612, and the precise tracking, low latency and low latency provided by the mmW based antenna The high QOS capability enables sub-centimeter or sub-millimeter position and orientation tracking, which can improve accuracy when tracking the real-time position and orientation of the device 602 . In alternative embodiments, the sensing mechanism 610 and transceiver 612 may be coupled together in a single tracking module device.

The network interface 616 may be implemented as computing software and hardware to communicatively connect to a network, receive computer-readable program instructions from the network transmitted by one or more servers or by other devices, and forward the instructions for storage in the memory 608 for storage. For processor 618 to execute.

The processor 618 may be implemented as computing hardware and software configured to receive and process instructions. For example, processor 618 may be configured to provide imaging requests, receive imaging data, process imaging data into environmental or other data, process user input data and/or imaging data to generate user interaction data, execute edge-based (on-device) ), serving server requests, receiving server responses, and/or providing user interaction data, context data, and content object data to one or more other system components. For example, processor 618 may receive user input data from I/O module 604 and may implement applications stored in memory 608, respectively. In other examples, the processor 618 may receive multi-source data captured from the real world from the sensing mechanism 610, or may receive the exact location and orientation of the device 602 through the tracking module 614, and may send the data to a server for use in Prepare some data before further processing.

FIG. 7 shows a method 700 according to an embodiment.

The method 700 begins at step 702 by providing at least one cloud server of a cloud server computer system including at least one processor and memory, the memory storing a persistent virtual world system. The persistent virtual world system includes one or more virtual objects, the virtual objects include virtual data and models, and the virtual objects include one or more of virtual twins, pure virtual objects, or applications. In some embodiments, virtual data and models provide one or more virtual twins with self-computing capabilities and autonomous behavior. The method 700 proceeds to step 704 to provide a virtual object representing the store. Then, in step 706, the method continues by connecting the virtual object representing the store to a database associated with the store's real-world items. The method then continues in step 708 by receiving an interaction or transaction instruction from the user device related to one or more real world items, wherein the interaction or transaction is performed in augmented or virtual reality through the persistent virtual world system . Finally, the method continues to step 710, where the database is updated based on the received transaction orders.

In some embodiments, the store is a physical store that includes corresponding virtual objects stored in a persistent virtual world system. In other embodiments, the store is a virtual store that does not have a corresponding real world counterpart.

In some embodiments, method 700 further includes receiving data associated with the user from a connected device installed in the user's physical location, the data including user movement, location and orientation; and updating the user avatar within the persistent virtual world system. The user's physical location can be a VR-dedicated cabin, a room within a building, or an outdoor area.

In some embodiments, method 700 may also include creating an account by the user with the persistent virtual world system, which may be accomplished by accessing the persistent virtual world system via a cloud server, or may require downloading and installing in the user's device one or more Multiple programs in order to access persistent virtual world systems. Creating an account may include entering user preference data by filling out one or more questionnaires or forms. However, preference data may also be obtained from connected social media profiles if the user authorizes it. Method 700 may also include selecting, by the user, a desired store to virtually ship to.

According to an embodiment, the method 700 also includes accessing the virtual twin of the store from the user's physical location through the persistent virtual world system in augmented or virtual reality. The user physical location includes a connected device that captures images from the user that are delivered to the user avatar through a persistent virtual world system, where the shopper physical location is one of a virtual reality-dedicated cabin, a room within a building, or an outdoor area .

According to an embodiment, the method 700 further includes providing a separate layer for each of the augmented and virtual reality activated by the user device connected to the at least one cloud server, each layer including augmentation to each layer-specific reality and virtual twin .

According to an embodiment, the method 700 further includes providing, in the at least one cloud server, a digital identifier associated with each store retrieved by the user device for accessing the associated store through the persistent virtual world system. In some embodiments, the digital identifier includes one or more of QR codes, URLs, IP addresses, MAC addresses, cryptographic hashes, universally unique identifiers, and organizationally unique identifiers.

According to an embodiment, the method 700 further includes storing a user device personal identifier in the memory of the at least one cloud server, the user device personal identifier enabling the user to be identified in order to provide the user with a personalized experience; in the at least one server or distributed ledger Each transaction made by the user is recorded under the user device personal identifier in the memory of the user device.

According to an embodiment, method 700 further includes storing payment card data from the user linked to the personal identifier; detecting entry or exit or passing (human or virtual) by one or more sensors configured in one or more predetermined areas a user device personal identifier of a user in one of the one or more predetermined areas; the server retrieves the stored payment card data; uses the retrieved payment card data to pay for the product; and initiates an automatic retrieval and delivery system.

8 illustrates a method 800 where the store is a physical store and the corresponding virtual object is a virtual twin of the physical store, the method 800 begins at step 802 by providing one or more products on one or more shelves. The method continues in step 804, providing a corresponding virtual twin for each product. Then, as shown in step 806, the method provides a plurality of connected devices configured to identify the product and the user's selection of the product and send a corresponding status update to the database. Then, in step 808, the method receives, by the persistent virtual world system, the store's real world data from the connected device. Finally, at step 810, the method receives from the user interaction or transaction instructions with one or more elements of the store.

While certain embodiments have been described and illustrated in the drawings, it is to be understood that such embodiments are illustrative only and not restrictive of the broad disclosure, and that the disclosure is not limited to what is shown and described. specific structures and arrangements, as various other modifications will occur to those of ordinary skill in the art. Accordingly, this specification is to be regarded as illustrative rather than restrictive.

Claims (20)

1. A system capable of updating a database of real world items via a virtual world system, the system comprising:

at least one server of a cloud server computer system, the at least one server comprising at least one processor and memory, the memory storing a persistent virtual world system, the persistent virtual world system comprising one or more virtual objects comprising virtual data and a model, the virtual objects comprising one or more of virtual twins, pure virtual objects, or applications, wherein at least one of the virtual objects represents a store of real world items connected to a periodically updated database associated with the real world items of the store, and wherein the persistent virtual world system provides access to the database via a user interface of a user device.

2. The system of claim 1, wherein the store is a virtual store represented by a corresponding virtual object, wherein a user can access the virtual store in virtual reality from a remote physical location.

3. The system of claim 2, wherein the remote physical location comprises a connected device that captures data from the user, the data being transmitted to an avatar through the persistent virtual world system.

4. The system of claim 1, wherein the at least one virtual object representing the store comprises a virtual twin stored in the persistent virtual world system and is in communication with a plurality of connected devices configured to recognize a plurality of real world items or user interactions with the plurality of real world items and send corresponding status updates to the database, wherein the user can access the store in augmented reality or remotely in virtual reality.

5. The system of claim 4, wherein the plurality of connected devices comprise one or more cameras comprising RGBD cameras, depth cameras, LIDAR cameras, or CCTV cameras, or a combination thereof, wherein the cameras employ artificial intelligence based machine vision to identify the plurality of real-world items or the user's interactions with the real-world items.

6. The system of claim 4, wherein the plurality of connected devices comprise sensors comprising one or more of temperature sensors, humidity sensors, moisture sensors, mass sensors, proximity sensors, inertial sensors, optical sensors, gas and chemical sensors, pressure sensors, touch sensors, electrical sensors, or combinations thereof.

7. The system of claim 1, wherein the at least one server stores separate augmented reality and virtual reality layers that are activated by the user device connected to the at least one server, each layer including a virtual twin dedicated to each layer.

8. The system of claim 1, wherein the store has an associated digital identifier stored in the at least one server, the digital identifier being retrieved by the user device from the at least one server for accessing the associated store through the persistent virtual world system.

9. The system of claim 1, wherein the at least one virtual object representing the store further comprises a personal assistant, and wherein the personal assistant is an artificial intelligence agent.

10. The system of claim 1, wherein the user device has a personal identifier associated with a personal profile, and wherein transactions or interactions conducted by the user are recorded under the personal identifier.

11. The system of claim 1, wherein entering or leaving a predetermined area of the store sends a signal to the at least one server to complete a transaction.

12. A method of enabling updating of a database of real world items via a virtual world system, the method comprising:

providing at least one server of a cloud server computer system, the at least one server comprising at least one processor and memory storing a persistent virtual world system, the persistent virtual world system comprising one or more virtual objects, the one or more virtual objects comprising virtual data and a model, the virtual objects comprising one or more of a virtual twin, a pure virtual object, or an application;

providing at least one virtual object representing a store of real-world items;

connecting a database associated with the store to the at least one virtual object representing the store;

receiving interaction or transaction instructions from a user device relating to one or more of the real-world items, wherein the interaction or transaction associated with the received instructions is conducted in augmented or virtual reality by the persistent virtual world system; and

updating the database based on the received instruction.

13. The method of claim 12, further comprising:

receiving, from a connected device installed at a remote physical location, user data associated with a user of the user device, the user data including user movement, location, and orientation in the remote physical location; and

updating a user avatar within the persistent virtual world system.

14. The method of claim 12, wherein the at least one virtual object is a virtual twin of the store, the method further comprising:

providing a corresponding virtual twin for each of the plurality of real-world items;

providing a plurality of connected devices configured to recognize interactions of the plurality of real-world items or users with the plurality of real-world items and send corresponding status updates to the database;

the persistent virtual world system receiving real world data of the store from the connected device; and

an interaction or transaction instruction is received from a user device.

15. The method of claim 12, further comprising providing, in the at least one server, a numeric identifier associated with each store retrieved by the user device for accessing the associated store through the persistent virtual world system.

16. The method of claim 12, further comprising:

storing in the at least one server a user equipment personal identifier capable of identifying a user; and

recording, in the memory of the server, transactions conducted by the user under the user device personal identifier.

17. The method of claim 16, further comprising:

detecting, by one or more sensors configured in one or more predetermined areas, the user device personal identifier of the user entering or leaving one of the one or more predetermined areas;

completing a transaction based on the detection; and

the automatic retrieval and delivery system is initiated.

18. A computer-readable medium having executable instructions stored thereon, the executable instructions configured to cause at least one server of a cloud server computer system comprising at least one processor and memory to perform steps comprising:

storing a persistent virtual world system comprising one or more virtual objects comprising virtual data and a model, the virtual objects comprising one or more of a virtual twin, a pure virtual object, or an application;

providing at least one virtual object representing a store of real-world items;

connecting a database associated with the store to the at least one virtual object representing the store;

receiving interaction or transaction instructions from a user device relating to one or more of the real-world items, wherein the interaction or transaction associated with the received instructions is conducted in augmented or virtual reality by the persistent virtual world system; and

updating the database based on the received instruction.

19. The computer-readable medium of claim 18, wherein the steps further comprise:

receiving, from a connected device installed at a remote physical location, user data associated with a user of the user device, the user data including user movement, location, and orientation in the remote physical location; and

updating a user avatar within the persistent virtual world system.

20. The computer-readable medium of claim 18, wherein the at least one virtual object is a virtual twin of the store, the steps further comprising:

providing a corresponding virtual twin for each of the plurality of real-world items; and

receiving, by the persistent virtual world system, real world data of the store from a plurality of connected devices configured to recognize interactions with the plurality of real world items or users and to send corresponding status updates to the database.

Images