DE102023207369A1 - Universal platform for converting information into audio signals - Google Patents

Abstract

A device for converting information into audio signals has a plurality of inputs, wherein at least one input is configured to receive information from one or more sensors. At least one output of the device is configured to output an audio signal at the at least one output. An audio path of the device is configurable to be coupled between the plurality of inputs and the at least one output to generate the audio signal at the at least one output based on the information from the one or more sensors.

Description

The present invention relates to devices, methods and systems for converting information into audio signals, in particular from sensor signals and/or sensor data into audio signals.

State of the art

Psychological research shows that auditory feedback can play an important role in motor learning ( Shams et al., Trends Cognit. Sci., 12(11), 411, 2008 ), be it during sports ( Schaffert et al., Front. Psychol. 10(244), 2019 ) or in more general tasks such as learning gestures and other movements ( Danna et al., Hum. Mov. Sci., 43, 216, 2015 Of particular interest is the way in which acoustic feedback can be used to optimize performance, actions and/or movements.

The conversion of information into audio signals is generally referred to as sonification ( D. Worrall, Sonification Design, Springer Nature, 2019 ). State-of-the-art methods for sonification, particularly of non-acoustic sensor signals and/or sensor data, into audio signals are known. In recent years, for example, sports science has been researching whether artificially amplified and/or generated acoustic feedback can improve a training process, particularly for beginners. The results of the research indicate improvements in the training process when training is accompanied by suitable sonification.

The publication DE 102 97 168 T5 discloses the sonification of respiratory parameters to monitor human respiration and gas exchange during anesthesia and intensive care treatment.

The publication DE 20 2006 000 757 U1 discloses a sonifier that converts a data stream into sound pressure waves that are perceived as audible music, at least by a person exercising.

The publication DE 10 2020 001 182 A1 reveals feedback to a driver to signal through sonification whether or not to reduce the speed of a vehicle.

DE 10 2021 204 036 A1 discloses the use of sonification in hearing aids to reduce the risk of falls and/or as a rehabilitation measure to support and improve a user's self-movement.

However, the systems described in the literature and/or commercially available are designed for only one specific application or can only be used for a single project or a single sport, such as skiing (cf. Hasegawa et al. 2012, Realtime Sonification of the Center of Gravity for Skiing, in Proceedings of the 3rd Augmented Human International Conference, Megeve, France, 2012 ) or for rowing (cf. https://rowinginmotion.com [Accessed July 11, 2023 ]).

disclosure of the invention

The invention provides an apparatus and a method for converting information into audio signals having the features of the independent claims.

Preferred embodiments are the subject of the respective subclaims.

Devices and a method according to the invention have the advantage that they combine a large number of possible applications in one device and/or in one method. They also have the advantage that they enable different processing strategies for different tasks and/or sports and can be adapted to changing user preferences and new research findings, thus enabling flexible, application-specific and easily reconfigurable sonification of sensor data, in particular data from a sensor about movements of a user who wears one or more sensors directly or indirectly.

In other words, a universal platform for converting information into audio signals is provided that can be integrated into a device such as a body-worn device such as a smartwatch to support a variety of sports or other activities using different sonification strategies.

According to a first aspect, the invention relates to a device for converting information into audio signals with a plurality of inputs, wherein at least one input is configured to receive information from one or more sensors. At least one output of the device is configured to output an audio signal at the at least one output. An audio path of the device is configurable to be coupled between the plurality of inputs and the at least one output in order to generate the audio signal at the at least one output based on the information from the one or more sensors.

According to a further development, the at least one audio path has at least one submodule that can be configurably coupled between the plurality of inputs and the at least one output in order to generate the audio signal at the at least one output based on the information from the one or more sensors.

According to a development, the at least one submodule has an audio signal input and/or an audio data input; a control signal input and/or control data input; and an audio signal output and/or audio data output, wherein the audio signal input and/or audio data input and the audio signal output and/or audio data output can be configurably coupled between the plurality of inputs and the at least one output by means of signals or data that can be received via the control signal input and/or control data input.

According to a further development, the at least one submodule is configurable to provide a function for generating signals or data that are output from the submodule via the audio signal and/or the audio data output, or to provide a function for mapping signals or data that are received from the submodule via the audio signal and/or the audio data input to signals or data that are output from the submodule via the audio signal and/or the audio data output.

According to a further development, the function provided is configurable by the signals or data received via the control signal input and/or control data input.

According to a further development, the at least one audio path has a plurality of identical submodules which can be configurably coupled between the plurality of inputs and the at least one output in order to generate the audio signal at the at least one output based on the information from the one or more sensors.

According to a second aspect, the invention relates to a system for converting information into audio signals. The system comprises a device for converting information into audio signals as described above and at least one inertial measuring unit as a sensor of the one or more sensors coupled to the device described above. The system further comprises a sonification library for selectively configuring the device for different applications of sonification.

According to a further development, the at least one inertial measuring unit is integrated into a device worn on the body and/or into a sports device.

According to a further development, the inertial measuring unit integrated in a sports device can be wirelessly coupled to the device.

According to a third aspect, the invention relates to a method for configuring a device for converting information into audio signals as described above. The method comprises initializing one of a plurality of inputs to receive information from one or more sensors. The method further comprises initializing at least one output to output an audio signal. The method further comprises configuring an audio path between the plurality of inputs and the at least one output to generate the audio signal at the at least one output based on the information from the one or more sensors.

Short description of the drawings

• 1 eine Vorrichtung zum Umwandeln von Information in Audiosignale mit N Eingängen und M Ausgängen gemäß einem Ausführungsbeispiel;
• 2 eine Vorrichtung zum Umwandeln von Information in Audiosignale mit einer Vielzahl von Audiopfaden mit einem oder mit mehreren Submodulen zum Verarbeiten von Daten gemäß einem Ausführungsbeispiel; und
• 3 eine generische Struktur eines Submoduls zum Verarbeiten von Daten gemäß einem Ausführungsbeispiel.

They show:

• 1 an apparatus for converting information into audio signals with N inputs and M outputs according to an embodiment;
• 2 a device for converting information into audio signals with a plurality of audio paths with one or more submodules for processing data according to an embodiment; and
• 3 a generic structure of a submodule for processing data according to an embodiment.

In all figures, identical or functionally identical elements and devices are provided with the same reference symbols. The numbering of process steps serves the purpose of clarity and is generally not intended to imply a specific chronological order. In particular, several process steps can also be carried out simultaneously.

Description of the embodiments

1 shows a device 1000 for converting information into audio signals with N inputs and M outputs. The device 1000 can comprise four or fewer inputs. Via the N inputs, or a subset thereof, the device 1000 can receive information from sensors or other information sources. The receiving can be done wirelessly and/or wired. gen. Receiving may only be possible wirelessly. The information may include, for example, raw sensor signals and/or sensor data, processed sensor signals and/or sensor data, or derived sensor signals and/or sensor data derived from raw and/or processed sensor signals and/or sensor data. Each input n ∈ N may be configured to receive specific information from a single sensor or other specific information source.

The device 1000 for converting information into audio signals may comprise at most two outputs m ∈ M to enable either a mono or a stereo output. The frequencies of the output signals may be in the frequency range between 0 and 30 kHz, preferably in a frequency range between 20 and 20 kHz. The frequencies of input signals may be lower or higher than the frequencies of the corresponding output signals.

The device 1000 for converting information into audio signals can be configured to continuously receive information via one or more inputs. For example, the device 1000 can be configured to receive sensor signals and/or sensor data from at least one inertial measurement unit (IMU). An inertial measurement unit can be integrated, for example, in a device that is worn by a user, such as a golf club, another piece of sports equipment, or a device used to perform the sporting activity, such as a golf ball. In particular, the inertial measurement unit can be integrated in a smartwatch or a smartphone. The reception of the sensor signals and/or sensor data by the device 1000 can be wireless and/or wired.

One or more inertial measurement units may be configured to derive position data, a path such as a trajectory, a curvature, a torsion, or other data in the time domain, or enable deriving one or more of these based on the sensor signals.

The device 1000 may also be configured to receive other sensor signals and/or sensor data from other sensors, such as sensors for determining a relative position, e.g. by means of satellite technology and/or terrestrial mobile technology, inductive sensors, ultrasonic sensors, optical sensors, capacitive sensors, or magnetic sensors, etc.

The device 1000 may comprise several interconnectable submodules, as shown in the 2 shown as an example and described in detail below with reference to the 2 and 3 described.

2 shows a device 2000 for converting information into audio signals with a plurality of audio paths, for example the audio path 2100 and the audio path 2200. The plurality of audio paths are linked to an input n EN. The input can also be linked to only one audio path

An audio path may include one or more submodules. For example, an audio path 2100 may include at least one of submodule 2110, submodule 2120, submodule 2130, and/or submodule 2140. Submodule 2110 may be configured to generate a signal or data, such as sound data. In particular, submodule 2110 may be configured to generate white noise. White noise may be noise with a constant power density spectrum in a certain frequency range. Submodule 2120 may be a low-pass filter, a high-pass filter, or a combination thereof. Submodule 2130 may be a ring modulator. Submodules in an audio path, or a subset thereof, may be linked in parallel or in series.

An audio path 2200 may include a submodule 2210 and a summation node 2220, which is also summation node 2140 in audio path 2100. The summation node may be linked to an output m E M of device 2000. Device 2000 may have only two outputs or only one output.

A submodule may be configured to process one or more sensor signals and/or sensor data in different ways. In particular, a submodule may be configured to generate a sound, to filter amplitudes and/or frequencies of a signal in continuous form or in the form of discrete pulses.

The device 2000 can be configurable. In other words, the links of the inputs n ∈ N of the device 2000 with one or more signal paths can be configurable. Alternatively or additionally, the links of the one or more submodules in a signal path can be configurable. For example, the links between the submodules, their order or the use of a specific submodule can be configurable. A user can thus specify how the inputs and/or the signal paths and/or the submodules of the respective signal paths are linked to one another. A user may also configure a function of a submodule, e.g., a function of how an input signal and/or input data of the submodule is processed and/or controlled to generate an output signal and/or output data of the submodule. Configuring the device 2000 may be possible during a runtime of the device 2000.

The configurability of the connections of the submodules and/or their function allows the device 2000 for sonification to be used for a variety of different applications and in particular sports.

In conjunction with a system or components thereof, such as a body-worn mobile device, a wearer of the system or component may be directly sounded based on the received sensor signals and/or sensor data and the functions of the device 2000.

Sounding is also possible based on signals and/or data derived from prior processing or otherwise from sensor signals and/or sensor data. Examples of such signals and/or data are one-dimensional or multi-dimensional shape parameters such as curvature, torsion, etc. of, for example, a path, a trajectory, or other information such as position, inclination, etc. of, for example, an object.

Furthermore, sound is also based on signals and/or data about a similarity with respect to a reference, for example signals and/or data from which it can be directly determined or at least deduced how far a movement of a user or an object deviates from or is away from an ideal movement.

Alternatively or additionally, sound may also be emitted based on a determination that a user is performing a desired or undesired action or movement, or that the action or movement is correct or incorrect in a context.

3 shows a generic structure of a submodule 3120 for processing data as used in the device 2000. Without loss of generality (oBdA), the generic structure with respect to the submodule 2120 of the 2 , embedded between submodules 2110 and 2130. The generic structure of submodule 3120 also corresponds to the other submodules in the 2 , as described in detail below.

The submodules all have a common structure to facilitate their connection in an audio path (cf. 2 ). Each submodule 3120 has an audio signal input and/or audio data input 3121, a control signal input and/or control data input 3122, and an audio signal output and/or audio data output 3123. The signals or data received via the control signal input and/or control data input 3122 can define control parameters 3124 of a function 3125 in the submodule, which maps the signals or data received from the submodule 3120 via the audio signal input and/or audio data input 3121 to signals or data output from the submodule 3120 via the audio signal output and/or audio data output 3123.

Although each submodule of, for example, the device 2000 of 2 has the same generic structure, it is not necessary for each submodule to use all elements of the generic structure. A submodule 2110 for generating a signal or data such as sound data can, for example, allow no signals or data to be received by the submodule 2110 and the sound data to be generated by the function of the submodule itself, using the signals or data received via the control signal input and/or control data input from, for example, an inertial measuring unit. Furthermore, a submodule, for example a summation node 2140 or 2220 as in 2 shown, do not have or allow controllable parameters to provide its function.

This allows, using generic programming techniques, a function to be designed for each submodule in each audio path of a device to specify how to convert information into audio signals. The device can therefore be used for a variety of applications and can be easily programmed by a user of the device for different applications.

The device can also be configured and/or continuously updated for a variety of applications, for example using a sonification library. A connection to a sonification library can be wireless and/or wired.

By means of an interface, a user can be enabled to develop new applications. In particular, the user can initialize the input or inputs of the device, ie cause the input or inputs to be used to receive information, add or remove submodules to audio paths, and set parameters to define the function of a submodule (parameterize). Furthermore, the user can define audio paths for add or remove the final output of the audio signal.

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions.

Cited patent literature

• DE 102 97 168 T5 [0004]
• DE 20 2006 000 757 U1 [0005]
• DE 10 2020 001 182 A1 [0006]
• DE 10 2021 204 036 A1 [0007]

Cited non-patent literature

• Shams et al., Trends Cognit. Sci., 12(11), 411, 2008 [0002]
• Schaffert et al., Front. Psychol. 10(244), 2019 [0002]
• Danna et al., Hum. Mov. Sci., 43, 216, 2015 [0002]
• D. Worrall, Sonification Design, Springer Nature, 2019 [0003]
• Hasegawa et al. 2012, Realtime Sonification of the Center of Gravity for Skiing, in Proceedings of the ^3rd Augmented Human International Conference, Megeve, France, 2012 [0008]
• https://rowinginmotion.com [Accessed on July 11, 2023 [0008]

Claims (10)

Apparatus (1000; 2000) for converting information into audio signals, comprising: a plurality of inputs, wherein at least one input is configured to receive information from one or more sensors; at least one output, wherein the at least one output is configured to output an audio signal at the at least one output; and at least one audio path (2100, 2200), wherein the at least one audio path is configurable to be coupled between the plurality of inputs and the at least one output to generate the audio signal at the at least one output based on the information from the one or more sensors. Device (1000; 2000) according to claim 1 , wherein the at least one audio path (2100, 2200) comprises at least one submodule (3120) configurably coupled between the plurality of inputs and the at least one output to generate the audio signal at the at least one output based on the information from the one or more sensors. Device (1000; 2000) according to claim 2 , wherein the at least one submodule (3120) has an audio signal and/or an audio data input (3121); a control signal input and/or control data input (3122); and an audio signal output and/or audio data output (3123), wherein the audio signal input and/or audio data input (3121) and the audio signal output and/or audio data output (3123) can be configurably coupled between the plurality of inputs and the at least one output by means of signals or data that can be received via the control signal input and/or control data input (3122). Device (1000; 2000) according to claim 3 , wherein the at least one submodule (3120) is configurable to provide a function (3125) for generating signals or data that are output from the submodule (3120) via the audio signal and/or the audio data output (3123), or to provide a function (3125) for mapping signals or data received from the submodule (3120) via the audio signal and/or the audio data input (3121) to signals or data that are output from the submodule (3120) via the audio signal and/or the audio data output (3123). Device (1000; 2000) according to claim 4 , wherein the provided function (3125) is configurable by the signals or data received via the control signal input and/or control data input (3122). Device (1000; 2000) according to claim 1 , wherein the at least one audio path (2100, 2200) comprises a plurality of identical submodules (3120) configurably coupled between the plurality of inputs and the at least one output to generate the audio signal at the at least one output based on the information from the one or more sensors. A system for converting information into audio signals comprising: a device (1000; 2000) according to any one of the preceding claims; at least one inertial measurement unit as a sensor of the one or more sensors coupled to the device (1000; 2000); a sonification library for selectively configuring the device (1000; 2000) for different applications of sonification. system according to claim 7 wherein the at least one inertial measuring unit is integrated into a device worn on the body and/or into a sports device. system according to claim 8 wherein the inertial measuring unit integrated in a sports device can be wirelessly coupled to the device (1000; 2000). Method for configuring a device (1000; 2000) according to any of the preceding Claims 1 until 6 comprising the steps of: initializing one of a plurality of inputs to receive information from one or more sensors; initializing at least one output to output an audio signal; configuring an audio path (2100, 2200) between the plurality of inputs and the at least one output to generate the audio signal at the at least one output based on the information from the one or more sensors.

Images