DE102023210263A1 - Method for generating a texture map - Google Patents

Abstract

The invention relates to a method for generating a texture map (232) of a floor surface in an environment in which a mobile device, in particular an at least partially automated vehicle or robot, in particular a cleaning robot, is moving or is intended to move, comprising: providing (200) device and/or environmental information (202, 204, 206) that has been acquired from the mobile device and/or the environment by means of at least one sensor of the mobile device, wherein the at least one sensor comprises a depth sensor; providing (208) images (210) that have been acquired from the environment by means of a camera of the mobile device; for a selected image or each of a plurality of selected images of the provided images: synchronizing (212) the image with at least a portion of the device and/or environmental information acquired by the depth sensor in spatial and/or temporal terms to obtain synchronized depth information (214), and extracting (216) from the image, based on the synchronized depth information, a ground image (218) of the environment; generating (230) a texture map (232) of the ground surface of the environment based on the one or more ground images; and providing (240) the texture map.

Description

The present invention relates to a method for generating a texture map of a ground surface in an environment in which environment a mobile device, in particular an at least partially automated vehicle or robot, is moving or is intended to move, as well as a system for data processing, a mobile device and a computer program for carrying out the method.

Background of the invention

Mobile devices, such as at least partially automated vehicles or robots, typically move within an environment, particularly a working area or workspace, such as a home, a garden, a factory hall, on the street, in the air, or in water. One of the fundamental problems of such or other mobile devices is to orient themselves, i.e., to know what the environment looks like, in particular where obstacles or other objects are, and where it is (absolutely) located. For this purpose, the mobile device can be equipped with various sensors, such as cameras, lidar sensors, or inertial sensors, which are used to capture the environment and the movement of the mobile device, for example, in two or three dimensions. This enables the mobile device to move locally, detect obstacles in a timely manner, and avoid them.

Disclosure of the invention

According to the invention, a method for generating a texture map, as well as a data processing system, a mobile device, and a computer program for implementing the method are proposed, having the features of the independent patent claims. Advantageous embodiments are the subject of the subclaims and the following description.

The invention generally relates to mobile devices that move or are intended to move in an environment or there, e.g., in a workspace, and in particular to the generation of a texture map of the floor surface in such an environment. A texture map is understood to be a map in which a texture or type of floor surface (e.g., a carpet, a flat floor, etc.) is recognizable or visible, e.g., in the form of a photo or image.

Examples of such mobile devices (or mobile work equipment) include robots and/or drones and/or partially or fully automated vehicles (on land, water, or in the air). Robots include, for example, household robots such as vacuum and/or mop robots, floor or street cleaning devices, or lawnmower robots, as well as other so-called service robots, as well as at least partially automated vehicles, such as passenger vehicles or goods transport vehicles (including industrial trucks, e.g., in warehouses), as well as aircraft such as drones or watercraft.

Such a mobile device comprises, in particular, a control and/or regulating unit and a drive unit for moving the mobile device, so that the mobile device can be moved in the environment, e.g., along a movement path or trajectory. Furthermore, a mobile device can comprise one or more sensors by means of which the environment or information in the environment can be detected.

In the following, the invention will be explained in particular using the example of a cleaning robot as a mobile device, although the principle can also be applied to other types of mobile devices.

Mobile devices can create and/or use a map as part of their navigation within an environment. Household robots, such as cleaning robots, can use a 2D map view to interact with a user. This can include showing the structure of the environment on an artificially colored map, with obstacles and walls representing boundaries or other objects. Maps can also be enhanced with pictograms, showing, for example, the position of the docking station, the current position of the robot, the path traveled, and other information. The user can also interactively place object placeholders, such as sofa or chair models, to enhance the map.

However, as it turns out, one particular problem for a user is that it is difficult to annotate different floor types in such a map (e.g., a carpet, a border between tiles in an open kitchen and living room, and the like). However, such information about a floor surface can be important if the user wants to define specific go- or no-go zones, i.e., specific areas in the environment into which the robot is allowed or not allowed to move.

Against this background, a method for generating a texture map of a ground surface in an environment, in particular an indoor space. The mobile device is intended to move in this environment. For this purpose, device and/or environmental information is provided which has been detected by at least one sensor of the mobile device from the mobile device and/or the environment. The at least one sensor comprises (at least) one depth sensor. A lidar sensor (or laser scanner) is particularly suitable as a depth sensor, and more specifically a 2D lidar sensor, i.e. a lidar sensor which scans the environment only in one plane. This plane is generally parallel to the ground surface, at least when the mobile device is on level ground.

In principle, other types of depth sensors are also possible, e.g., time-of-flight cameras and the like. Irrespective of this, the mobile device may also include one or more sensors other than a depth sensor, e.g., an inertial measurement unit (IMU), a camera, or odometry. Such other sensors can be used, in particular, for localization and/or navigation of the mobile device, e.g., in the context of so-called SLAM.

In SLAM, there are various approaches to representing maps and positions. Conventional SLAM methods generally rely on geometric information such as nodes and edges. Nodes and edges are typically components of the SLAM graph. The nodes and edges in the SLAM graph can take various forms; traditionally, the nodes correspond, for example, to the pose (position and orientation) of the mobile device or certain environmental features at specific times, while the edges represent relative measurements between the mobile device and environmental features, or between different poses of the mobile device at different times. Based on such a SLAM graph, a map of the environment in which the mobile device moves can be determined or will be determined. With each new data set containing information about the environment, which information is obtained from or is based on a sensor of the mobile device, the map (or SLAM graph) can be expanded or updated. The aforementioned map or 2D map for the user can also be generated based on the SLAM map or SLAM graph.

In addition to the device and/or environmental information, images captured by the mobile device's camera are also provided. In addition to the depth sensor, the mobile device therefore has at least one other camera; this is again independent of any other sensors. The camera or the images captured by it can also be used for localization and/or navigation of the mobile device, e.g., also within the framework of SLAM. In this context, this is also referred to as visual SLAM.

The following steps are then performed for a selected image or each of several selected images from the provided images. The image is spatially and/or temporally synchronized with at least a portion of the device and/or environmental information captured by the depth sensor—i.e., depth information—to obtain synchronized depth information.

This portion of the device and/or environmental information is, for example, data from a lidar scan. A lidar scan typically corresponds to a set of points, a so-called lidar point cloud. Each point corresponds to a location on an object or obstacle in the environment where the laser beam from the lidar sensor was reflected. Thus, each point also indicates the distance of an obstacle from the mobile device or lidar sensor. The point cloud as a whole also provides a type of depth map. A time-of-flight camera provides a similar type of depth information.

Synchronizing an image with depth information primarily involves temporal synchronization (so-called pairing) and, if necessary, subsequent transformation of the data so that they are available in a common reference coordinate system for further processing. Typically, the camera images and the depth sensor data are not captured at the same time or not always at the same time or synchronously in time. However, since the mobile device can move between two such times, spatial synchronization may be necessary. The synchronized depth information is therefore, in particular, depth information that corresponds to the camera image in terms of time and pose of the mobile device.

Based on the synchronized depth information, a ground image of the surroundings is then extracted from the image. It should be noted that a camera on a mobile device is typically directed forward (or possibly backward or to the side), which is usually necessary for navigation and obstacle detection. A ground image is now defined as an image that at least approximately shows (only) the ground surface, but not areas of the surroundings that are significantly elevated above the ground.

Based on the one or more ground images, the texture map of the ground surface of the environment is then generated, which is then also made available, e.g. to the user.

Compared to a non-textured representation, a texture map with, for example, a photorealistic view is easier for the user to understand and interact with. It facilitates, for example, manual spatial segmentation and zone annotation (e.g., no-go zones). Furthermore, the generated texture map provides a basis for automatic segmentation methods, which could, for example, use machine learning techniques to learn go-to or no-go zones from previous user input.

Compared to other techniques for generating a texture map, the use of a depth sensor, such as lidar scanning, offers an effective and efficient way to determine which points or areas belong to the ground surface or ground plane. This effectively establishes the boundaries of the image and prevents distorted objects from being depicted in the texture map. This combination of sensors makes it possible to create a more consistent image of the ground surface or ground plane than if only a top-down camera image transformation were used. Furthermore, this texture map can be created online (e.g., during runtime of the mobile device or even in real time; it is also conceivable to execute at least part of the process, e.g., in a cloud or on the internet). Furthermore, unlike live views from above, the known position from the SLAM system can be used to create a consistent texture map of the ground surface and save it, for example, for later reference.

In one embodiment, extracting the ground image comprises the following steps: A clear area is determined based on the synchronized depth information. The clear area covers at least part of the surrounding ground surface. This can, for example, be a certain angular range in front of the mobile device. The depth information or the synchronized depth information specifically allows only an area to be selected in which there is no obstacle or other object, such as a wall. The clear area therefore lies in particular in a plane, e.g., a plane parallel to the ground surface, in particular at a vertical distance that corresponds to a height of the depth sensor above the ground.

In the specific case of a lidar sensor or 2D lidar sensor, the free area can be defined, for example, as a set of points on straight lines, which correspond to a detection direction of the lidar sensor—i.e., lines along which the lidar sensor emits laser beams. These points should not be confused with the points in the lidar point cloud; rather, the points on the straight lines can be multiple points per straight line, which are then distributed at least approximately evenly across the free area and represent it.

This free area is then projected into the camera image, and image information from the image is assigned to at least part of the free area. The free area with the assigned image information is then used as the ground image for generating the texture map. In this way, depth information can be used particularly efficiently to generate or determine ground images for the texture map.

Alternatively, extracting the ground image can also include the following steps. First, a perspective transformation of the image (from the camera) is performed to obtain a transformed image. Based on the synchronized depth information, a clear area is determined, which clear area covers at least part of the surrounding ground surface. This can generally be done as described above. A portion of the transformed image corresponding to the clear area is then used as the ground image.

In one embodiment, generating the texture map comprises determining one or more image regions of the texture map for which image regions contain image information from multiple ground images. Such image regions can be, for example, individual pixels and/or groups of pixels. For one or at least one, preferably all, of the multiple image regions, a value for at least a portion of the image information is then determined based on the multiple ground images. This value is then used for the texture map.

This can be referred to as a so-called multi-view combination. One goal of multi-view combination is to create a consistent global image of the ground surface from individual images (the multiple ground images). Each ground image is assigned to a pose of the mobile device in the global map. To combine the individual ground images, for example, an averaging can be performed on a global pixel grid. For each pixel in this grid, all pixels of ground images can be collected or made available. that fall within this pixel of the grid. The color value of all these pixels can then be averaged, for example, taking gamma correction into account in order to obtain a natural mix of the observed colors. Individual pixels can be weighted differently depending on how far they were from the mobile device when captured by the camera. Pixels or image information that depict areas less distant from the camera generally exhibit less distortion caused by inverted perspective mapping and are also less affected by noise in the pose estimation. In this way, a (final) value (e.g. color value) can be found as a kind of averaged or otherwise derived value, which is used for the texture map.

Another approach to determining such a value, e.g., by averaging the different colors, is to average the colors, e.g., in the so-called LAB color space. For example, only the "best" color in each pixel could be retained or used after a quality criterion (for selecting a value from one of the several pixels as the final value) has been defined and applied (e.g., distance from the camera during observation or moderate brightness).

In one embodiment, the one or more selected images are selected from the provided images according to at least one criterion. Thus, not all images need to be used; rather, specific images can be selected or omitted to make the method faster and more efficient, or even to improve the result itself.

The following criteria are taken into account for each image: If the position of the mobile device has changed by less than a specified distance compared to the last captured image when the image was captured. Or if the orientation of the mobile device has changed by less than a specified angle compared to the last captured image when the image was captured. Position and orientation can also be referred to as pose. For example, only images in which the mobile device performs a limited number of translational and angular movements are included. In this way, errors caused by insufficient extrinsic calibration and motion blur can be limited. It is also conceivable that a position or orientation has changed by more than a (then different) specified distance or specified angle.

It can also be considered whether the orientation of the mobile device lies within a predefined range when the image is captured, or whether a (possibly global) pose of the mobile device when the image is captured was not present in any of a predetermined number of recently captured images. This means, for example, that only images in which the mobile device or its camera is pointed in a specific direction are taken into account in order to limit the effects of specular reflections from light sources on the final result. In practice, this is particularly possible for mobile devices that perform coverage movements to perform their tasks (e.g., cleaning robots, vacuum cleaners, and lawnmowers). It can also be determined when a pose has already been visited and a decision can be made whether a new image should be used there or the last one should be retained.

It is also considered whether one or more image parameters meet specified requirements. This allows image quality to be taken into account. For example, some basic parameters of the image itself can be evaluated to exclude images that are most likely extremely under- or overexposed and thus do not provide useful information in the region of interest.

It is also considered whether the distance of an obstacle in the camera's detection direction is greater than a predetermined value, with the distance being determined, in particular, based on the device and/or environmental information acquired by the depth sensor. For example, a decision can be made whether or not to use an image depending on how much free space in front of the mobile device can be detected with the depth sensor. In other words, images should not be used in which the mobile device is close to objects and therefore only a small part of the ground is visible.

In one embodiment, a segmented map of the environment's subsurface is also generated based on the texture map. This allows, for example, the floor texture to be post-processed using a CNN (Convolutional Neural Network) or another machine learning algorithm to derive different types of floor coverings. This can, for example, help with the correct segmentation of rooms, as different rooms often have different floor coverings; it can also be used to make suggestions for cleaning or cleaning methods based on the floor type (e.g., no mopping on carpet).

In one embodiment, active detection may also be considered. The mobile device can be directly instructed to move to specific locations or positions in order to capture new information or images for the texture map.

As mentioned, other depth sensors can be used instead of a lidar sensor, e.g. stereo cameras, depth cameras (based on time-of-flight or structured light) and ultrasonic sensors.

In one embodiment, so-called stitching can be used. The aforementioned multi-view combination can be improved, for example, by using stitching methods such as those used for stitching panoramic images. This can, for example, result in a more convincing and consistent image of the ground or ground surface.

In one embodiment, reflection modeling can be performed. The aforementioned multi-view combination can, for example, also be designed to capture reflections on the floor. One possibility is, for example, to create multiple global pixel grids, each filled with views captured from different global directions or orientations of the mobile device. For example, one grid would be filled only with single-image views in which the robot is pointing in a direction between 0° and 30°, another grid with views between 30° and 60°, and so on. This would result in multiple floor maps where the reflections are consistent in each map. These maps could then be used in a user interface, displaying a different map depending on the direction the user rotates the view in an app, resulting in a more realistic and immersive user experience. Another way to achieve this would be to train a so-called NeRF-based model on the individual scene, which is limited to the 2D use case and is therefore simpler (NeRF stands for Neural Radiance Fields and is a method for synthesizing novel views of complex scenes).

In one embodiment, panoptic mapping can be performed. In addition to directly mapping color data from the images, semantic or panoptic segmentation of the color images can also be performed, and the results of these methods can then be assigned. Panoptic segmentation in this context describes the assignment of each pixel to a unique object class and, if necessary, a differentiation between different instances of an object class. This would lead to a categorical assignment of the floor, which is even easier for the user to interpret by simply showing them in which areas, for example, carpets, tiles, wooden floors, and so on are present. For this to work, a different approach to averaging than the one discussed in more detail above can be used, possibly taking into account confidences predicted by the neural network used for segmentation.

Preferably, localization and/or navigation information for the mobile device is further determined based on the device and/or environment information and based on SLAM, as already mentioned above. The texture map can then be generated based on the localization and/or navigation information.

A data processing system according to the invention comprises means for executing the method according to the invention or its method steps. The system can be a computer or server, e.g., in a so-called cloud or cloud environment. There, the images and environmental and/or device information can be received from the mobile device, and the texture map can be transmitted back to the mobile device (e.g., via a wireless data connection). However, it is also conceivable that such a data processing system is a computer or a control unit in such a mobile device. It is also conceivable that some steps of the method are carried out on the computer or server in the cloud environment, while other steps are carried out on the computer or control unit in the mobile device.

The invention also relates to a mobile device having a depth sensor and a camera, as well as the aforementioned data processing system (then in the sense of a computer or control unit in the mobile device). Preferably, the mobile device also has a control and/or regulating unit and a drive unit for moving the mobile device.

The mobile device is preferably designed as an at least partially automated vehicle, in particular as a passenger transport vehicle or as a goods transport vehicle, and/or as a robot, in particular as a household robot, e.g. a vacuum and/or mop robot, a floor or street cleaning device or a lawnmower robot, and/or as a drone, as already explained above.

The implementation of a method according to the invention in the form of a computer program or computer program product with pro Program code for carrying out all method steps is advantageous because it results in particularly low costs, particularly if an executing control unit is also used for other tasks and is therefore already present. Finally, a machine-readable storage medium is provided with a computer program stored thereon as described above. Suitable storage media or data carriers for providing the computer program are, in particular, magnetic, optical and electrical memories, such as hard disks, flash memories, EEPROMs, DVDs, etc. Downloading a program via computer networks (Internet, intranet, etc.) is also possible. Such a download can be wired or cable-based or wireless (e.g. via a WLAN network, a 3G, 4G, 5G or 6G connection, etc.).

Further advantages and embodiments of the invention will become apparent from the description and the accompanying drawings.

The invention is illustrated schematically in the drawing using an embodiment and is described below with reference to the drawing.

Short description of the drawings

• 1 schematically shows a mobile device in an environment for explaining the invention in a preferred embodiment.
• 2 shows schematically a sequence of a method according to the invention in a preferred embodiment.
• 3 shows schematically a mobile device to explain the invention.

Embodiment(s) of the invention

In 1 A mobile device 100 is shown schematically in an environment 120 to explain the invention in a preferred embodiment. The mobile device 100 is, for example, a cleaning or vacuuming robot with a control or regulating unit 102 and a drive unit 104 (with wheels) for moving the vacuuming robot 100 in the environment 120, e.g., an apartment or a room. The vacuuming robot 100 moves on the, generally at least substantially flat, floor surface 122 of the environment 120.

Furthermore, the robot vacuum cleaner 100 has, for example, a sensor 106 embodied as a 2D lidar sensor with a detection field. For clarity, the detection field is chosen to be relatively small; in practice, however, the field of view can also be up to 360° (e.g., but at least 180° or at least 270°). Using the 2D lidar sensor 106, distances between the mobile device 100 (or the sensor 106) and objects in the environment can be detected or determined. The 2D lidar sensor 106 is arranged in particular such that it measures the distances from the objects in a plane parallel to the floor surface 122.

The 2D lidar sensor 106 is therefore an example of a depth sensor by means of which depth information can be captured, for example, as part of device and/or environmental information.

Furthermore, the robot vacuum cleaner 100 has a camera 110, which can be used to capture images of the surroundings 120. The camera is directed forward, i.e., in the normal direction of travel, and can thus capture images of the surroundings. The floor surface 122 is therefore only peripherally and distortedly visible in the camera images.

Furthermore, the robot vacuum cleaner 100 has a system 108 for data processing, e.g., a control unit, by means of which data can be exchanged, e.g., via an indicated radio connection, with a higher-level data processing system 150. In the system 150 (e.g., a server; it can also represent a so-called cloud), navigation information can be determined, for example, which is then transmitted to the system 108 in the robot vacuum cleaner 100, according to which the latter is then to be operated. However, it can also be provided that navigation information is determined in the system 108 itself or is otherwise received there. Instead of navigation information, the system 108 can, for example, also receive control information determined based on navigation information, and according to which the control or regulating unit 102 can move the robot vacuum cleaner 100 via the drive unit 104. The vacuum cleaner robot 100 can move according to a movement path or trajectory 130.

Furthermore, two obstacles 124, 128, e.g., furniture, are shown in the environment 120 by way of example. A carpet 126 is also shown by way of example, on which the robot vacuum cleaner 100 can move, for example. However, the nature or texture of the surface of the carpet 126—generally also a floor surface—is different than in other areas of the floor surface 122, where, for example, a wooden floor is present.

Furthermore, a mobile input device 140, e.g., a smartphone, is shown, which can be connected to the robot vacuum cleaner 100 or its system 108 and the computing system 150 for data transmission, e.g., via a wireless communication connection. An application (so-called app) can be provided on the mobile input device 140, for example, by means of which the robot vacuum cleaner 100 can be controlled, e.g., its movement can be started and stopped. The movement of the robot vacuum cleaner 100 itself, however, is generally automated or autonomous, in particular using SLAM, as explained above.

In addition, the mobile input device 140 or the application may be provided with the option of displaying a map of the environment 120 on which a user can then define, for example, certain areas as go-to or no-go zones or otherwise.

Within the scope of the present invention, a possibility for generating a texture map as such a map is proposed. A texture map 142 is shown in 1 shown schematically. There, in particular, the different characteristics or textures of the soil surface in the surrounding area can be seen.

2 A method according to the invention for generating a texture map is shown schematically in a preferred embodiment as a type of flow chart or sequence diagram. This is to be illustrated by way of example using a vacuum cleaner robot as a mobile device, such as in 1 shown and explained.

In a step 200, device and/or environmental information is provided, which has been acquired from the mobile device and/or the environment by means of at least one sensor of the mobile device. The device and/or environmental information includes, for example, information 202 from an IMU and information 204 from an odometry as sensors of the mobile device. In addition, the device and/or environmental information includes depth information 206, which has been acquired, for example, from a lidar sensor of the mobile device, as in 1 shown. The depth information can be presented, for example, as a point cloud.

In a step 208, images 210 are provided which have been captured by a camera of the mobile device from the environment, e.g. the 1 shown camera.

The device and/or environmental information as well as the images are usually recorded and provided repeatedly, but usually not synchronously in time or space.

In a step 212, a selected one of the images 210 is then synchronized with the depth information 206 in spatial and/or temporal terms to obtain synchronized depth information 214. In particular, a temporal synchronization is performed, followed, if necessary, by a transformation of the data so that they are present in a common reference coordinate system for further processing.

In a step 216, a ground image 218 of the environment is then extracted from the image based on the synchronized depth information 214.

These steps 212, 216 can be performed for all images 210 or only for selected ones. As explained in more detail above, various criteria 220 are considered for which images can be selected for this purpose. The soil images 218 thus obtained can then be temporarily stored, for example, in step 222.

In addition to the steps of synchronizing the images and extracting the ground images, localization and/or navigation information 224 can also be generated quite regularly for the operation of the mobile device, e.g., within the framework of the aforementioned SLAM. For this purpose, the device and/or environmental information, in particular the information 202 from the IMU, the information 204 from the odometry, and the depth information 206, are used. This information 202, 204, 206 is processed, in step 226, as part of, for example, continuous or quasi-continuous trajectory planning, based on which, possibly with intermediate steps, a pose is optimized using a SLAM graph in step 228.

Here, too, step 226 can be performed for all received device and/or environmental information, while the optimization in step 228 can be performed only for so-called keyframes, for example. Based on the trajectory planning in step 226, the criteria 220 can also be determined.

Based on the possibly cached ground images 218, a texture map 232 is then generated in step 230. For this purpose, the aforementioned multi-view combination is used, in particular, in step 234. In particular, image regions of the texture map for which image information from multiple ground images is available can be determined; moreover, values for these image regions can then be determined based on the ground images.

As already mentioned, one goal of multi-view combination is, in particular, to create a consistent global image of the ground surface from the ground images. Each ground image is assigned to a pose of the mobile device in the global map. For this purpose, the aforementioned averaging can be performed on a global pixel grid, as well as another type of multi-view combination, as already explained in detail. For this purpose, the results from the optimization in step 228 can be used, in particular.

The texture map 232, which is e.g. 1 The texture map 142 shown then comprises or is based in particular on a 2D map 236 obtained as part of the SLAM (localization and/or navigation information 224 and, if applicable, trajectory of the mobile device), as well as the ground images extracted from the camera images and then combined in a suitable manner. In step 240, the texture map can then be provided, for example, for further use.

It should be noted here that such a texture map can generally be created for environments of varying sizes and complexity, not just for a single room. For example, the texture map can be created for an apartment with numerous rooms, each with several different types of floor surfaces. Likewise, a texture map can also be created for a factory hall or similar environment, and in principle, for outdoor environments as well. Furthermore, the different types of floor surfaces do not necessarily have to have different roughnesses, such as carpet and wooden floors; different floor colors can also be distinguished as different types of floor surfaces.

In 3 A mobile device 300 is shown schematically to explain the invention, in particular with regard to the aspect of extracting a ground image. The mobile device 300 has, for example, a 2D lidar sensor 306 and a camera 310. This can, for example, be the 1 shown mobile device 100, ie the robot vacuum cleaner.

360 refers to the lidar or laser beams emitted by the lidar sensor. When the environment is detected using the lidar sensor, such 360 laser beams are typically emitted at a certain angular distance Δθ from each other. This angular distance can, but does not have to, be identical between any two 360 laser beams.

The laser beams 360 are then reflected off an obstacle (provided the obstacle is sufficiently close), and the reflected beams are detected. In this way, a point, designated here as 362, can be obtained along each laser beam. This point lies on a surface of the obstacle and thus indicates the distance to the obstacle. The set of points 362 can also be referred to as a lidar point cloud.

As mentioned, extracting the ground image can include determining a clear area based on the synchronized depth information, e.g., a lidar point cloud synchronized with an image. This clear area covers at least part of the surrounding ground surface. In particular, the clear area can be defined as a set of points on straight lines, which correspond to a detection direction of the lidar sensor.

Such points are in 3 designated 364. These points 364 can lie on straight lines defined by the laser beams. For example, within a range between a minimum radius r _min and a maximum radius r _max in a coordinate system of the lidar sensor (represented here with x, y, z), a set of such points 364 can be assigned to each of these laser beams. The points 364 can, for example, have a specific distance Δr (sampling distance) along the laser beams or straight lines.

In general, this is about "how far" along an imaginary line points can or should be assigned to the free area. The points 364 do not necessarily have to be sampled on straight lines that are Δθ apart, but could also be sampled more densely (e.g., 1/2 Δθ). Then there wouldn't be a point 362 for each ray that delimits it. In this case, the free area can be delimited, for example, via linear interpolation between two points 362, or the point with a smaller radius can be chosen (e.g., to play it safe).

The maximum radius r _max is a previously specified constant that states that points further away than r _max should not be considered under any circumstances (e.g. because it is assumed that the image would then be too distorted or because the depth sensor measurement is very noisy at great distances, etc.).

Analogous to r _max , r _min is also a previously defined parameter that limits the sampling range, e.g. to exclude areas in the image where parts of the robot are visible or where the depth measurements of the lidar are inaccurate (lidars usually have a minimum distance from which they provide reliable depth values).

In this way, the set of points 364 can be used to represent a clear area 366, particularly in a coordinate system, which clear area 366 is free of obstacles—at least at the level of the lidar sensor. A portion of the ground surface will therefore lie within the clear area 366 and will also be visible in the associated camera image.

This free area 366 or this set of points 364 can then be projected into the image, for which purpose, for example, extrinsic and/or intrinsic camera parameters are used. This allows image information, such as a color or a set of pixels with specific colors, to be assigned from the image to at least a portion of the free area, but in particular to each point 364.

Alternatively, as mentioned, a perspective transformation of the image can also be carried out.

Claims (15)

A method for generating a texture map (142, 232) of a floor surface (122) in an environment (120) in which a mobile device (100), in particular an at least partially automated vehicle or robot, in particular a cleaning robot, is moving or is intended to move, comprising: Providing (200) device and/or environmental information (202, 204, 206) that has been acquired from the mobile device and/or the environment by means of at least one sensor of the mobile device, wherein the at least one sensor comprises a depth sensor (106, 306); Providing (208) images (210) that have been acquired from the environment by means of a camera (110, 310) of the mobile device; For a selected image or each of several selected images of the provided images: - Synchronizing (212) the image with at least a portion of the device and/or environmental information acquired by the depth sensor, spatially and/or temporally, to obtain synchronized depth information (214), and - Extracting (216) from the image, based on the synchronized depth information, a ground image (218) of the environment; Generating (230) a texture map (232) of the ground surface of the environment based on the one or more ground images; and Providing (240) the texture map. Procedure according to Claim 1 , wherein extracting the ground image comprises: determining a clear area (366) based on the synchronized depth information, which clear area covers at least a portion of the ground surface of the environment, projecting the clear area into the image, and assigning image information from the image to at least a portion of the clear area, and using the clear area with the assigned image information as a ground image. Procedure according to Claim 2 , wherein the free area is determined as a set of points (364) on straight lines, which straight lines correspond to a detection direction of a depth sensor designed as a lidar sensor. Procedure according to Claim 1 wherein extracting the ground image comprises: performing a perspective transformation of the image to obtain a transformed image, determining a clear area based on the synchronized depth information, which clear area covers at least a part of the ground surface of the environment, and using a part of the transformed image corresponding to the clear area as a ground image. Method according to one of the Claims 2 until 4 , where the free area lies in one plane. The method of any preceding claim, wherein generating the texture map comprises: determining (234) one or more image regions of the texture map for which image regions have image information from multiple ground images; determining (234) for the one or at least one of the multiple image regions, a value for at least a portion of the image information based on the multiple ground images; and using the value for the texture map. The method according to any one of the preceding claims, further comprising: determining (226) localization and/or navigation information (224) for the mobile device based on the device and/or environmental information and based on SLAM; wherein the texture map is further generated based on the localization and/or navigation information (224). Method according to one of the preceding claims, wherein the one or more selected images are selected according to at least one criterion rium (220) are selected from the provided images, wherein the at least one criterion comprises at least one of the following criteria for a respective image: - a position of the mobile device when capturing the image has changed by less than a predetermined distance compared to the last captured image, - an orientation of the mobile device when capturing the image has changed by less than a predetermined angle compared to the last captured image, - an orientation of the mobile device when capturing the image is within a predetermined range, - a pose of the mobile device when capturing the image was not present in any of a predetermined number of last captured images, - one or more image parameters correspond to predetermined requirements, - a distance of an obstacle in the capture direction of the camera is greater than a predetermined value, wherein the distance has been determined in particular based on the device and/or environmental information captured by means of the depth sensor. The method of any preceding claim, further comprising: generating, based on the texture map, a segmented map of the subsurface of the environment. Method according to one of the preceding claims, wherein the depth sensor is designed as a lidar sensor, in particular as a 2D lidar sensor. A data processing system (108) comprising means for carrying out the method according to any one of the preceding claims. Mobile device (100, 300) comprising a depth sensor (106, 306), a camera (110, 310), and a system (108) according to Claim 11 preferably with a control or regulating unit (102) and a drive unit (104) for moving the mobile device according to the navigation information. Mobile device (100, 300) to Claim 12 which is designed as an at least partially automated vehicle, in particular as a passenger transport vehicle or as a goods transport vehicle, and/or as a robot, in particular as a household robot, e.g. a cleaning robot, floor or street cleaning device or lawnmower robot, and/or as a drone. Computer program, comprising instructions which, when executed by a computer, cause the computer to carry out the method steps of a method according to one of the Claims 1 until 10 to perform when it is running on the computer. Computer-readable storage medium on which the computer program is Claim 14 is stored.

Images