WO2025260957A1 - Data processing apparatus, processor, board, and data processing method - Google Patents

Abstract

The present application discloses a data processing apparatus, a processor, a board, and a data processing method. The data processing apparatus can be comprised in a computing apparatus of a combined processing apparatus, and the combined processing apparatus can further comprise an interface apparatus and another processing apparatus. The data processing apparatus interacts with the other processing apparatus to jointly complete a computing operation specified by a user. The combined processing apparatus can also comprise a storage apparatus. The storage apparatus is separately connected to the computing apparatus and the other processing apparatus, and is used for storing data of the computing apparatus and the other processing apparatus. The present application provides a convolution operation scheme compatible with various fine-grained quantization formats, which can simplify processing and make full use of the advantages of low bit width operation and small memory occupation of the fine-grained quantization formats.

Description

Data processing device, processor, circuit board and data processing method

Cross-reference to related applications

This application claims priority to Chinese patent applications filed on June 21, 2024, with application number 202410813862.X, entitled "Data Processing Apparatus, Processor, Board and Data Processing Method" and filed on September 25, 2024, with application number 202411347002.8, entitled "Data Processing Apparatus, Processor, Board and Data Processing Method".

Technical Field

This disclosure generally relates to the field of computer processors. More specifically, this disclosure relates to a data processing apparatus, a processor, a circuit board, and a data processing method implemented by the data processing apparatus.

Background Technology

With the rapid development of artificial intelligence, the computational load and number of parameters in neural network models are continuously increasing. Research shows that during training and inference, reducing the precision of floating-point operations can effectively improve operating speed, reduce power consumption, and decrease chip area.

To address this, AI vendors have designed various floating-point formats, such as BF16 and TF32, to achieve a balance between AI system performance and accuracy. These floating-point formats each have their own application scenarios and advantages; however, the diverse or specialized nature of these formats also presents migration challenges. For example, a model trained by vendor A using FP6 cannot run on vendor B's inference hardware.

To establish a unified industry standard, the Open Compute Project (OCP) introduced the MX (MicroXcaling) quantization format. The MX quantization format is an interoperable quantization data format with shared, fine-grained block sizes. MX-compliant data formats support AI training and inference, featuring lower bit-width arithmetic operations and smaller memory footprints, thus driving improvements in hardware performance and efficiency, thereby reducing overhead and operating costs.

Therefore, it is necessary to develop an operation mode and corresponding hardware that are compatible with the MX quantization format, especially for convolution operations, which account for a large proportion of neural network operations, so as to give full play to the advantages of the MX quantization format.

Summary of the Invention

To address at least one or more of the technical problems mentioned above, this disclosure proposes a convolution operation scheme compatible with the MX quantization format in several aspects. Furthermore, an improved MX quantization scheme is also provided.

In a first aspect, this disclosure provides a data processing apparatus, comprising: a control circuit configured to parse a convolution instruction, the convolution instruction instructing a convolution operation on an input neuron and weights to obtain an output neuron, wherein either or both of the input neuron and weights are particle-quantized; and a computation circuit configured to perform a convolution operation on data of the input neuron and data of the weights according to the convolution instruction, and to fuse scaling factors of the input neuron and/or scaling factors of the weights in the convolution operation.

In a second aspect, this disclosure provides a processor that includes the data processing apparatus described in the first aspect.

In a third aspect, this disclosure provides a board that includes the processor described in the second aspect above.

In a fourth aspect, this disclosure provides a data processing method implemented by a data processing apparatus, the data processing apparatus including a control circuit and a computation circuit, the method comprising: the control circuit parsing a convolution instruction, the convolution instruction instructing a convolution operation on an input neuron and weights to obtain an output neuron, wherein either or both of the input neuron and weights are particle-quantized; and the computation circuit performing a convolution operation on the data of the input neuron and the data of the weights according to the convolution instruction, and fusing the scaling factor of the input neuron and/or the scaling factor of the weights in the convolution operation.

In the convolution operation scheme provided in this disclosure embodiment, when either or both of the input neurons and weights are in a particle quantization format, the data dequantization process can be integrated into the convolution operation without prior dequantization. This avoids separate dequantization processing and fully utilizes the advantages of low bit width and small memory footprint of the particle quantization format in the convolution operation. Here, the particle quantization format may include the MX quantization format proposed by OCP and the extended MX quantization format proposed in this disclosure embodiment.

Furthermore, in some embodiments, when the operand bit widths are asymmetrical, only the data portion of the operand can be processed, without needing to process the operand scaling factor, thereby simplifying the computation. Moreover, this processing allows for the mixed execution of operations with operands of various data types.

Optionally or additionally, in some embodiments, particle quantization of the results of convolution operations, i.e., the output neurons, is also supported. This particle quantization facilitates online hardware quantization and supports multiple data types of the quantized scaling factors, thereby improving algorithm accuracy.

Optionally or additionally, in some embodiments, the scaling factor obtained by extended MX quantization can also be compressed, thereby reducing the waste of storage space and bandwidth.

Attached Figure Description

The above and other objects, features, and advantages of exemplary embodiments of this disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

Figure 1 shows a structural diagram of the board card according to an embodiment of this disclosure;

Figure 2 shows a structural diagram of the combined processing apparatus according to an embodiment of this disclosure;

Figure 3 shows a schematic diagram of the internal structure of the processing core when the computing device is a single-core or multi-core device;

Figure 4 shows a schematic diagram of the MX quantization data format;

Figure 5 shows a schematic block diagram of a data quantization apparatus according to some embodiments of this disclosure;

Figure 6 shows a schematic block diagram of the comparison circuit of some embodiments of this disclosure;

Figure 7 illustrates a schematic structural diagram of a data processing apparatus that can implement embodiments of the present disclosure;

Figure 8 shows an exemplary flowchart of a data processing method implemented by a data processing circuit according to an embodiment of this disclosure.

Detailed Implementation

The technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

It should be understood that the terms "first," "second," "third," and "fourth," etc., that may appear in the claims, specification, and drawings of this disclosure are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this disclosure indicate the presence of the described features, integrals, steps, operations, elements, and/or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and/or collections thereof.

It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and/or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

The specific embodiments disclosed herein will now be described in detail with reference to the accompanying drawings.

First, we provide explanations of the technical terms that may be used in this disclosure.

Floating-point numbers: The IEEE floating-point standard represents a number in the form V = (-1) ^sign * mantissa * 2 ^E. Here, sign is the sign bit, 0 for positive and 1 for negative; E represents the exponent (or index), which is a weighted sum of 2 raised to the power of E (possibly a negative power); mantissa represents the mantissa, a binary fraction. Floating-point numbers are represented in computers using three fields, each encoded separately:

(1) A single sign bit directly encodes the symbol s;

(2) The exponent field of the k-bit is encoded as exp＝e(k-1)......e(1)e(0);

(3) The n-digit mantissa is used to encode the mantissa. However, the encoding result depends on whether the exponent field is all 0.

Fixed-point numbers are a way of representing numbers in computers, characterized by a fixed decimal point. Fixed-point numbers can represent integers or decimals, depending on the implicit position of the decimal point. A fixed-point number consists of three parts: a shared exponent, a sign bit, and a mantissa. The shared exponent means that the exponent is shared within a set of real numbers to be quantized; the sign bit indicates whether the fixed-point number is positive or negative; and the mantissa determines the number of significant digits, i.e., the precision.

Bit width: How many bits are used to represent data.

Quantization: The process of converting high-precision numbers into lower-precision data that takes up less memory.

Dequantization: The process of converting low-precision numbers into higher-precision data.

FP*: Represents a floating-point data type, where * can be an integer such as 2, 4, 6, 8, 16, 32, 64, etc. The value of * indicates the bit width of the data represented by this data type.

BF16 and TF32 are data types suitable for deep learning, which improve computational performance by sacrificing precision and range.

int*: Represents an integer data type, where * can be an integer such as 2, 4, 6, 8, 16, 32, 64, etc. The value of * indicates the bit width of the data represented by this data type.

ExMy: A data format where E represents the exponent, M represents the mantissa, x represents the exponent width, and y represents the mantissa width.

MX: refers to the Microscaling specification (see “OCP Microscaling Formats (MX) Specification Version 1.0”).

Extended MX: refers to a quantization scheme proposed in the embodiments disclosed herein.

Granular quantization: This includes MX and extended MX, as well as other quantizations with similar data structures. The quantization granularity is determined based on the amount of data that the arithmetic unit can process at one time.

It should be noted that in this disclosed embodiment, the data to be quantized or dequantized can be various data in the neural network, including but not limited to input neurons, weights, output neurons, gradients, etc. The neural network provided in this disclosed embodiment can be applied to various fields and can process various types of data. For example, in image recognition and processing, facial recognition, object detection, image classification, and medical image analysis can include data types such as pixel data and image features; in natural language processing (NLP), language translation, sentiment analysis, text summarization, and speech recognition can include data types such as text data, word embeddings, and sentence structure; in speech recognition and processing, intelligent assistants, automatic caption generation, and speech-to-text conversion can include data types such as audio signals and spectrograms; in recommendation systems, personalized content recommendation, product recommendation, and advertising can include data types such as user behavior data, item features, and rating data; in healthcare, disease diagnosis, drug discovery, and gene sequence analysis can include data types such as medical records, biomarker data, and genomic data; in the financial field, risk assessment, fraud detection, and stock market prediction can include data types such as transaction data, user credit scores, and market data; and in autonomous driving, vehicle perception, decision-making, and path planning can include data types such as sensor data, environmental features, and traffic signals. In gaming and entertainment, AI applications such as game AI, virtual reality, and animation generation can include data types such as game state data and user interaction data. In scientific research, applications such as physics simulation, chemical compound prediction, and astrophysical analysis can include data types such as experimental data, simulation results, and observational data. Generative AI applications, on the other hand, utilize complex algorithms, models, and rules to learn from large-scale datasets to create new, original content. This includes, but is not limited to, creating various types of content such as text, images, audio, video, and code. Correspondingly, the input neuron data processed can be unquantized or quantized image data, audio data, video data, speech data, text data, document data, etc., and the corresponding output neuron data can also be unquantized or quantized image data, audio data, video data, speech data, text data, document data, etc. The output of the neural network can include a probability score of an image belonging to a specific object category, a probability score of a document relating to a specific topic, a probability score of the correct translation of a text fragment in the target language from a text fragment in the source language, or a probability score of the correct transcription of a text fragment from spoken language, etc. These output data can also be quantized or unquantized. By quantizing/dequantizing the data, the computational load and resources can be reduced while ensuring the processing effectiveness of these tasks. The data quantization scheme of this disclosure embodiment can also be used in the inference and training of neural networks. By performing the quantization process described in this disclosure embodiment on the data of the neural network, the accuracy of the task can be guaranteed when the neural network performs one or more of the aforementioned processing tasks, while improving the speed and performance of the neural network model's inference and training.

Exemplary hardware architecture

Figure 1 shows a schematic diagram of the structure of a board 10 according to an embodiment of this disclosure. As shown in Figure 1, the board 10 includes one or more chips 101, which are system-on-chips (SoCs) that integrate one or more combined processing devices. The combined processing device is an artificial intelligence computing unit used to support various deep learning and machine learning algorithms, meeting the intelligent processing needs of complex scenarios in fields such as computer vision, speech, natural language processing, and data mining. In particular, deep learning technology is widely used in the field of cloud intelligence. A significant characteristic of cloud intelligence applications is the large amount of input data, which places high demands on the platform's storage and computing capabilities. The board 10 of this embodiment is suitable for cloud intelligence applications, possessing massive off-chip storage, on-chip storage, and powerful computing capabilities.

Chip 101 is connected to external device 103 via external interface device 102. External device 103 may be, for example, a server, computer, camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. Data to be processed can be transmitted from external device 103 to chip 101 via external interface device 102. The calculation results from chip 101 can be transmitted back to external device 103 via external interface device 102. Depending on the application scenario, external interface device 102 may have different interface forms, such as PCIe (Peripheral Component Interconnect Express) interface.

The board 10 also includes a storage device 104 for storing data, which includes one or more memory cells 105. The storage device 104 is connected to and transmits data with the controller 106 and the chip 101 via a bus. The controller 106 in the board 10 is configured to regulate the state of the chip 101. Therefore, in one application scenario, the controller 106 may include a microcontroller, also known as a microcontroller unit (MCU).

Figure 2 is a structural diagram illustrating the combined processing device in chip 101 of this embodiment. As shown in Figure 2, the combined processing device 20 includes a computing device 201, an interface device 202, a processing device 203, and a storage device 204.

The computing device 201 is configured to perform user-specified operations. It is mainly implemented as a single-core intelligent processor or a multi-core intelligent processor to perform deep learning or machine learning calculations. It can interact with the processing device 203 through the interface device 202 to jointly complete the user-specified operations.

Interface device 202 is used to transmit data and control commands between computing device 201 and processing device 203. For example, computing device 201 can obtain input data from processing device 203 via interface device 202 and write it to on-chip storage device of computing device 201. Further, computing device 201 can obtain control commands from processing device 203 via interface device 202 and write them to on-chip control cache of computing device 201. Alternatively or optionally, interface device 202 can also read data from storage device of computing device 201 and transmit it to processing device 203.

Processing device 203, as a general-purpose processing device, performs basic control including but not limited to data transfer, and starting and/or stopping computing device 201. Depending on the implementation, processing device 203 may be one or more types of processors, including but not limited to digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, computing device 201 disclosed herein can be considered as having a single-core structure or a homogeneous multi-core structure. However, when computing device 201 and processing device 203 are considered together, they are considered to form a heterogeneous multi-core structure.

Storage device 204 is used to store data to be processed. It may be DRAM (Dynamic Random Access Memory) or DDR (Double Data Rate) memory, typically 16G or larger, and is used to store data of computing device 201 and/or processing device 203.

Figure 3 shows a schematic diagram of the internal structure of the processing core when the computing device 201 in Figure 2 is a single-core or multi-core device. The computing device 301 is used to process input data such as computer vision, speech, natural language, and data mining. The computing device 301 includes three main modules: a control module 31 (also called a controller), an arithmetic module 32 (also called an arithmetic unit), and a storage module 33 (also called a memory).

The control module 31 coordinates and controls the operation of the computation module 32 and the storage module 33 to complete the deep learning task. It includes an instruction fetch unit (IFU) 311 and an instruction decode unit (IDU) 312. The instruction fetch unit 311 fetches instructions from the processing device 203, and the instruction decode unit 312 decodes the fetched instructions and sends the decoding result as control information to the computation module 32 and the storage module 33.

The computation module 32 includes a vector operation unit 321 and a matrix operation unit 322. The vector operation unit 321 is used to perform vector operations and can support complex operations such as vector multiplication, addition, and nonlinear transformations; the matrix operation unit 322 is responsible for the core computations of deep learning algorithms, namely matrix multiplication and convolution.

Storage module 33 is used to store or move relevant data, including Neuron RAM (NRAM) 331, Weight RAM (WRAM) 332, and Direct Memory Access (DMA) 333. NRAM 331 stores input neurons, output neurons, and intermediate results after computation; WRAM 332 stores the convolution kernels of the deep learning network, i.e., the weights; DMA 333 is connected to DRAM 204 via bus 34 and is responsible for data transfer between computing device 301 and DRAM 204. It should be noted that NRAM and WRAM here can be two storage areas formed by dividing the same memory in logical storage space, or they can be two independent memories; no specific limitation is made here.

Exemplary MX quantization data

Figure 4 illustrates the MX quantized data format, which represents the components of MX quantized data. As shown, the MX quantized data format is characterized by three parts: the scaling factor (X, shared scale), the scaling block size (K, scaling block size), and the private element data within the block (P _{_i} ).

The scaling factor X relates to the data type or encoding of the scaling factor. The private element (P _{_i} ) also relates to its data type or encoding. The scaling block size (K) represents the number of private elements P _{_i} within the block. All K elements P_{_i} within the block have the same data type and therefore the same bit width. The scaling factor X is shared among these K elements. The data type of the elements and the data type of the scaling factor can be chosen independently. In a sense, MX can be viewed as a mechanism for constructing vector data types based on scalar data types. The values V _₁ , ..., V_{_k} represented in an MX block can be determined as follows: _Vi = XP_{_i} .

As shown in the figure, W represents the number of bits used to encode the shared scaling factor X, which is also the bit width of scaling factor X; d represents the number of bits used to represent each element _Pi , which is also the bit width of _Pi . Therefore, for an MX block containing K elements, it can be encoded using (W+Kd) bits.

Table 1 shows some common MX formats and related parameters.

Table 1

In Table 1 above, ExMy represents a scalar format notation with x exponent bits and y mantissa bits. For example, FP8 (E4M3) represents an FP8 format with 1 sign bit, 4 indicator bits, and 3 mantissa bits. When y is 0 (e.g., E8M0), the format does not include a sign bit.

Furthermore, in MX quantization, the scaling factor X is calculated as follows:

(1) Determine the largest power of the absolute value of the K elements Pi that are less than or equal to the K elements _Pi ;

(2) Divide the maximum power of 2 by the power of the maximum value that the data type of element P _i can represent, and set the result as the scaling factor Scale X.

The inventors have discovered that, for MX quantization, processing the original data into ultra-low precision data types such as F4 (representing a 4-bit data format) leads to a sharp decrease in the accuracy of large models, significantly impacting the accuracy of the computational results. Taking image recognition tasks as an example, task accuracy or processing performance is reflected in the accuracy of the image recognition results, i.e., whether the image recognition computation is accurate. The quantized image data obtained after quantization using the aforementioned quantization method affects the accuracy of the image recognition results.

The inventors also discovered that if the data type of the scaling factor used in the quantization method is the same as the original data type, although the above-mentioned problem of decreased accuracy can be solved to some extent, this method will lead to greater resource overhead (such as storage space, area overhead required for hardware support, etc.).

Exemplary Extended MX Quantization Method

Based on the above findings, this disclosure provides a novel data quantization scheme. This scheme determines a scaling factor based on the maximum absolute value of the data to be quantized, and obtains a scaling coefficient by converting the scaling factor to a different data type. Since the converted scaling coefficient has a non-zero mantissa, the accuracy of the quantized data is improved, resulting in processing tasks performed on the quantized data according to the embodiments of this disclosure exhibiting better accuracy. Furthermore, setting the quantization granularity based on the hardware's single-processing capacity facilitates the quantization of a block of granularity in a single hardware operation. This makes online quantization/dequantization more convenient, facilitating pipelined design and enabling neural network models using this hardware as a computing platform to perform quantization operations at various stages, including inference and training, thereby improving the computational efficiency of the neural network model.

In the extended MX quantization scheme, the scaling factor is calculated as follows:

(1) Determine the maximum absolute value in a set of data to be quantified;

(2) Obtain the maximum value that the target data type can represent after the data to be quantized is quantized;

(3) Determine the scaling factor corresponding to this set of data to be quantized based on the ratio between the maximum absolute value and the maximum value;

(4) Convert the scaling factor from the first data type to the second data type to obtain the scaling coefficient, wherein the bit width of the second data type can be smaller than the bit width of the first data type.

The target data type is the data type of the quantized data obtained after quantizing the data to be quantized. Before quantization, the target data type can be set as needed, and its precision is generally lower than that of the original data type of the data to be quantized. In some embodiments, the original data type of the data to be quantized may include at least one of FP16, BF16, FP32, FP64, TF32, etc.; the target data type may include at least one of FP8, FP6, FP4, FP2, int8, int4, int2, int6, FP12, etc. Among these, floating-point data types such as FP8, FP6, and FP4 can support multiple data formats. Taking FP4 as an example, it can be divided in several ways according to the rule of "sign bit + exponent bit + mantissa bit," with the sign bit typically occupying 1 bit. Common combinations of exponent and mantissa bits include E3M0 and E2M1. Similarly, FP8 can support data formats such as E4M3 and E5M2, and FP6 can support data formats such as E3M2 and E2M3.

Due to the bit width limitations of each data type, each data type has a maximum representable value. For example, the maximum representable value when the target data type is FP4 is 6; the maximum representable value when the target data type is int8 is 127. Typically, there is a one-to-one correspondence between the maximum value and the target data type, which can be obtained through methods such as table lookup.

Since the data type of the absolute maximum value is the same as the original data type of the data to be quantized, the data type of the scaling factor obtained by the ratio (i.e., the first data type) is the same as the original data type of the corresponding data to be quantized, or the bit width of the first data type can be greater than the bit width of the original data type.

The first data type is the scaling factor's data type, and the second data type is the scaling coefficient's data type. Data type conversion makes the bit width of the second data type smaller than that of the first data type, thus the precision of the scaling coefficient is less than the precision of the scaling factor. In some embodiments, the conversion from scaling factor to scaling coefficient can be achieved using a conversion instruction, such as `cvt_to_scale_dtype()`. In other embodiments, the conversion supports multiple round modes, such as `rn(round to nearest, ties to even)`.

In other embodiments, the second data type can be represented as ExMy, where E represents the exponent bits, M represents the mantissa bits, x represents the exponent bit width, y represents the mantissa bit width, and x + y < the bit width of the original data type. For example, assuming the original data type is FP16, then x + y < 16. The specific values of x and y can be set according to needs or the hardware environment, where x is a positive integer and y ≥ 0. In still other embodiments, 3 ≤ x + y < the bit width of the original data type. In some embodiments, 8 ≤ x + y < the bit width of the original data type. In some embodiments, the second data type may include at least one of, for example, E8M0, E8M1, E8M2, E8M3, E8M4, E8M5, E8M6, E8M7, E5M0, E5M1, E5M2, E5M3, E5M4, E5M5, E5M6, E3M0, E3M1, E3M2, E3M3, E3M4, E3M5, E3M6, E3M7, E3M8, E3M9, etc. In some preferred embodiments, y > 0. In a further preferred embodiment, the second data type may be E8M2 or E8M3.

For ease of understanding, taking the original data type FP16 as an example, when 8 < x + y < the bit width of the original data type, the bit width (i.e., x + y) represented by the second data type ExMy of the scaling factor can be between 8 bits and 16 bits. Using such a scaling factor to quantize the computational data of the neural network, and then using the quantized data for neural network inference and training, the resulting neural network precision is between that of an 8-bit representation (e.g., E8M0 in MX) and the original 16-bit representation (e.g., E5M10, E8M7, i.e., FP16, BF16). Compared to the MX format, this scaling factor in the disclosed embodiment is beneficial for improving the precision of the quantized data; compared to the case where the data type of the scaling factor is the same as the original data type, this scaling factor in the disclosed embodiment is beneficial for reducing hardware resource overhead or storage overhead.

Furthermore, since the mantissa bit width affects the quantization accuracy, a larger mantissa bit width can provide a scaling factor with higher accuracy, thereby preserving more original data details during the quantization process. Therefore, compared with the scaling factor of E8M0 format in MX quantization, when the mantissa bit width y of the scaling factor in this disclosed embodiment is greater than 0, it is beneficial to improve the accuracy of quantized data and the accuracy of processing tasks.

Furthermore, this disclosed embodiment enables the scaling factor to support multiple data types by performing data type conversion, thereby allowing for more flexible adaptation to different data characteristics and precision requirements. Regarding the selection of the exponent and mantissa bit widths of the scaling factor, the distribution characteristics of the data to be quantized can be considered. A larger exponent bit width allows for the representation of more values. If the range of values in a set of data to be quantized varies significantly, the exponent bit width of the scaling factor can be set larger; conversely, if the range of values in a set of data to be quantized varies only slightly, the exponent bit width of the scaling factor can be set smaller, while the mantissa bit width can be set larger.

For example, assuming the second data type needs to be set to 8 bits, if the absolute value range of a set of data to be quantized is between 2 ^{^-127} and 2 ^¹²⁷ , then the second data type can be set to E8M0; if the absolute value range of a set of data to be quantized is between 2 ^{^-4} and 2^⁴ , then the second data type can be set to E3M5. Of course, it's understandable that the bit width of the second data type is not limited to 8 bits; it can also be set to 5 bits, 4 bits, etc., as needed.

After determining the scaling factor in the above manner, the scaling factor can be used to quantize the corresponding data to be quantized, or to dequantize the quantized data.

In some embodiments, quantization processing may include: dividing the data to be quantized by a scaling factor to obtain an intermediate quantization result; and performing data type conversion on the intermediate quantization result to obtain quantized data with the target data type. This data type conversion can also be implemented using a conversion instruction, such as `cvt_to_scale_dtype()`. This conversion instruction can be a hardware or software instruction at various levels, such as processor microinstructions, ISA instructions, or function instructions implemented in high-level languages at various software levels.

Dequantization is generally the process of restoring low-precision data to high-precision data. In some embodiments, dequantizing quantized data using a scaling factor may include multiplying the quantized data with the scaling factor. In other embodiments, the quantized data may be converted from a target data type to a desired data type before being multiplied with the scaling factor, wherein the precision of the desired data type is higher than that of the target data type. In still other embodiments, dequantization can restore the quantized data to its original state before the quantization operation.

As can be seen from the second data type of the scaling factor obtained from the previous conversion, the bit width of a single scaling factor is 3 to 15 bits. For example, E8M4 requires 12 bits, which typically requires 2 bytes (2B) of storage space, so it will definitely not be fully utilized. This inevitably leads to a waste of storage space and transmission bandwidth. To solve this problem, in some embodiments, two or more scaling factors can be compressed, thereby saving storage space and bandwidth.

In some embodiments, after quantizing two or more sets of data to be quantized using the aforementioned method to obtain two or more scaling factors, these two or more scaling factors can be compressed. Then, the compressed scaling factors are used to quantize the corresponding sets of data to be quantized, respectively, to output quantized data with the target data type.

Specifically, after obtaining the scaling factors as described above, a compression process can be included, which specifically includes: determining the shared exponent of the Z scaling factors to be compressed, where Z ≥ 2; determining the Z new mantissas corresponding to these Z scaling factors; and bundling the shared exponent with the Z new mantissas for output for association. That is, after compression, the exponents of the Z scaling factors are reused, and their mantissas are different, thereby saving at least Z-1 exponent bits.

In some implementations, determining the shared exponent for the Z scaling factors to be compressed may involve setting the largest exponent among these Z scaling factors as the shared exponent. Choosing the largest exponent as the shared exponent ensures the precision of larger numbers as much as possible, since larger numbers typically have a greater impact on the calculation results.

In some implementations, determining the Z new mantissas corresponding to the Z scaling factors can include: setting the new mantissa of scaling factors whose exponent is less than the maximum exponent to 0; and setting the new mantissa of scaling factors whose exponent is equal to the maximum exponent to the original mantissa. It can be understood that for scaling factors whose exponent is equal to the maximum exponent, since the exponent remains unchanged, its mantissa remains unchanged; however, for scaling factors whose exponent is less than the maximum exponent, after the exponent is scaled up to the maximum exponent, to maintain equivalence, the mantissa would be proportionally reduced, exceeding the range of values that the mantissa can represent. Therefore, its new mantissa is set to 0 here. This setting will result in some precision loss, but compared to the storage and bandwidth gains, this precision loss is acceptable in some cases.

In some implementations, the shared exponent and the Z new mantissas can be bundled together in sequence for association. For example, first the shared exponent, then the new mantissa of the first scaling factor, the new mantissa of the second scaling factor, and so on up to the new mantissa of the Z scaling factor.

Next, the compressed scaling factors are used to quantize the corresponding groups of data to be quantized, so as to output quantized data with the target data type. Since the exponent and mantissa of the scaling factors with an exponent smaller than the maximum exponent will change after compression, the compressed scaling factors are used to quantize the corresponding groups of data to be quantized separately in order to maintain the consistency of quantization and dequantization.

In some implementations, the second data type of the Z scaling factors to be compressed is the same. For example, all Z scaling factors are of type E8M4. In this way, the shared exponent bit width after compression is 8 bits, and the Z new mantissas are 4Z bits.

In some implementations, the Z scaling factors belong to Z adjacent groups of data to be quantized. This allows the data to be processed sequentially without requiring additional information to indicate the scaling factors that are compressed together.

In some implementations, Z can be 2, 4, 8, or other values. Those skilled in the art can choose an appropriate Z value based on the number K of data to be quantized in a set of data. For example, when K is small, the value of Z can be larger; when K is large, the value of Z needs to be smaller, thereby ensuring that the benefit of using the compression scheme is greater than the benefit of amplifying K by Z times.

Similarly, when dequantizing extended MX quantized data using a compressed format, the scaling factors need to be decompressed first. Specifically, during dequantization, in response to the scaling factors being in a compressed format, the compressed scaling factors are decompressed; and the decompressed scaling factors are used to dequantize the quantized data.

Based on the compression method and bundling approach described above, the bundled Z scaling factors can be decoded into Z independent scaling factors. Specifically, the shared exponent can be combined with each of the Z new mantissas to obtain the Z decompressed scaling factors.

It is understood that in some embodiments, in addition to calculating and storing the scaling factor, the reciprocal of the scaling factor is also calculated and stored. This facilitates converting the division operation between the data to be quantized and the scaling factor into a multiplication operation between the data to be quantized and the reciprocal of the scaling factor during the quantization process. The reciprocal of the scaling factor has the same data type as the scaling factor. Therefore, the compression and decompression processes described above can also be applied to the reciprocal of the scaling factor, which will not be described in detail here.

In the extended MX quantization scheme described above, the scaling factor is determined based on the maximum absolute value of the data to be quantized. This ensures the accuracy of the data after quantization based on the maximum absolute value. Furthermore, since the larger the value of the data to be quantized during the quantization process, the greater the loss of accuracy and the greater the impact on the task calculation results, ensuring the accuracy of the data based on the maximum absolute value helps to reduce the loss of task accuracy. Therefore, compared to methods such as calculating the scaling factor in MX, the extended MX quantization scheme is better at ensuring task accuracy.

Furthermore, compared to the quantization method where the data type of the scaling factor is the same as the original data type, the scaling factor with a second data type obtained by the number of revolutions in this disclosed embodiment can reduce the storage space required to store the scaling factor and the hardware overhead required to execute the task while ensuring the accuracy of the task.

In some embodiments, the number of data to be quantized in a set of data to be quantized can be determined based on the single processing volume and/or the number of input ports of the processing circuit performing the quantization process; or the number of data to be quantized in a set of data to be quantized can be determined based on the single operation volume and/or the number of input ports of the arithmetic unit that performs the operation using the quantized data.

For example, the number of data points to be quantized in a set of data can be determined based on the processing capacity of the processing circuit (i.e., the amount of data that can be processed in a single run), or based on the processing capacity of the arithmetic unit that performs operations such as multiply-accumulate or vector inner product using quantized data (i.e., the amount of data that can be processed in a single run), and grouped according to this granularity. With this setup, a set of data points sharing a single scaling factor can be processed in one computation cycle, without needing to span multiple computation cycles, thus improving hardware implementation and fully utilizing hardware computing power.

For example, the number of data points to be quantized in a set of data can be determined based on the number of input ports of the processing circuit performing quantization or the number of input ports of the arithmetic unit that uses quantized data for calculation, and grouped accordingly. With this setup, a set of data to be quantized can be input at once using the number of hardware input ports, allowing a set of data sharing a single scaling factor to be processed in a single input, eliminating the need for multiple inputs. This improves hardware implementation, reduces hardware processing latency, and lowers I/O overhead.

For example, assuming the aforementioned processing circuit or arithmetic unit can process 32 data points in a single operation, and its corresponding number of input ports is also 32, then the data points to be quantized can be grouped into sets of 32. If the aforementioned processing circuit or arithmetic unit can process 64 data points in a single operation, and its corresponding number of input ports is less than 64, for example, 32, then the data points to be quantized can be grouped according to the number of input ports, ensuring that the number of data points to be quantized in each group is equal to the number of input ports. In this case, 32 data points are input at once, and the two groups of data can be processed in parallel; alternatively, the 64 data points to be quantized can still be grouped together and input twice to complete the processing of one group of data points at once. If the aforementioned processing circuit or arithmetic unit can process 32 data points in a single operation, and its corresponding number of input ports is more than 32, then the 32 data points to be quantized can be grouped together to achieve one-time input and processing of one group of data points.

In some embodiments, the data to be quantized can be elements of tensor data. Dividing multiple data to be quantized into multiple groups can include: dividing the multiple data to be quantized into multiple groups along at least one dimension of the tensor data. For example, suppose tensor data A(B, S), where B and S represent the dimensions of the first and second dimensions, respectively. If the division is performed along the second dimension, taking a grouping granularity of 32 as an example (i.e., each group contains 32 data to be quantized), dividing 32 consecutive data to be quantized along dimension S into a group results in (B×S/32) groups. Taking each group as a data block, the shape of the tensor composed of each data block after grouping (i.e., the dimensions and dimensions of the tensor) can be represented as (B, S/32), that is, the dimension of the second dimension after grouping is S/32.

For example, suppose tensor data A’(B, S, D), where B, S, and D represent the dimensions of the first, second, and third dimensions, respectively. If we divide the data along the third dimension, taking a grouping granularity of 64 as an example, we can divide 64 consecutive data points to be quantized along the third dimension into a group, resulting in (B×S×D/64) groups. Each group can be considered as a data block. The shape of the tensor composed of these data blocks after grouping can be represented as (B, S, D/64), which means the size of the third dimension after grouping is D/64.

It is understandable that dividing multiple data sets to be quantized into multiple groups is not limited to dividing them only along one dimension of the tensor data; it can also be done along multiple dimensions as needed. The grouping granularity is not limited to the aforementioned 32 or 64; it can be set as needed, for example, 16, 128, etc. The grouping granularity is related to the computing power or input port limitations of the processing circuit/arithmetic unit mentioned earlier, and will not be elaborated upon here.

It should be further explained that as the number of groups (i.e., the number of data blocks) increases, the number of scaling factors also increases accordingly. The scaling factors can represent a larger number of data points, which is beneficial for further improving data quantization accuracy. In particular, as the number of scaling factors increases, the sum of the exponent bit width and mantissa bit width increases, thus allowing for the representation of more data points than the E8M0 data format in MX quantization. Consequently, compared to MX quantization, the data accuracy represented by the scaling factors in the extended MX quantization of this disclosed embodiment is higher.

Therefore, based on the technical solution of obtaining scaling factors through data type conversion and the maximum absolute value, group quantization is used to increase the quantization granularity. A corresponding scaling factor is provided for each group of data to be quantized, which is conducive to further improving the accuracy of the quantized data and the accuracy of the executed task.

Exemplary device for extended MX quantization

Figure 5 shows a schematic block diagram of a data quantization apparatus according to some embodiments of this disclosure. As shown, the apparatus 500 may include a first processing circuit 510 and a second processing circuit 520, wherein the first processing circuit 510 is used to output a scaling factor, and the second processing circuit 520 is used to perform quantization processing. Specifically, the first processing circuit 510 may be used to: determine the maximum absolute value in a set of data to be quantized; obtain the maximum value that the target data type can represent after quantization of the data to be quantized; determine a scaling factor corresponding to a set of data to be quantized based on the ratio between the maximum absolute value and the maximum value; convert the scaling factor from a first data type to a second data type to output a scaling factor, wherein the bit width of the second data type is smaller than the bit width of the first data type. The second processing circuit 520 may be used to quantize a set of data to be quantized using the scaling factor to output quantized data with the target data type.

In some embodiments, the first processing circuit 510 may be further configured to: determine the number of data to be quantized in a set of data to be quantized based on the single processing volume and/or the number of input ports of the second processing circuit 520; or determine the number of data to be quantized in a set of data to be quantized based on the single operation volume and/or the number of input ports of the arithmetic unit that performs operations using quantized data.

In some embodiments, the apparatus 500 may further include a compression circuit 540. The compression circuit 540 is configured to compress two or more scaling factors after quantization of two or more sets of data to be quantized. In this case, the second processing circuit 520 is configured to quantize the corresponding sets of data to be quantized using the compressed scaling factors, respectively, to output quantized data with the target data type.

Correspondingly, when device 500 needs to dequantize extended MX quantized data using a compressed format, the scaling factors must first be decompressed. In these embodiments, device 500 may also include a decompression circuit (not shown in the figure). The decompression circuit is configured to decompress the compressed scaling factors in response to the scaling factors being in a compressed format. At this time, the circuit in device 500 that performs the dequantization process (e.g., the second processing circuit 520 or the third processing circuit 530) can be configured to perform dequantization processing on the quantized data using the decompressed scaling factors.

The operations performed by the first processing circuit 510, the second processing circuit 520, the compression circuit 540, and the decompression circuit have been described in detail above in conjunction with the extended MX quantization method, and will not be repeated here.

In some embodiments, the first processing circuit 510 may include: a comparison circuit 511, which can be used to determine the maximum absolute value in a set of data to be quantized; a first division circuit 512, which can be used to obtain the maximum value that the target data type after quantization of the data to be quantized can represent, and determine the scaling factor corresponding to a set of data to be quantized based on the ratio between the maximum absolute value and the maximum value; and a first rotation circuit 513, which can be used to convert the scaling factor from a first data type to a second data type to output the scaling coefficient. In some embodiments, the first division circuit 512 can obtain the maximum value by means of table lookup or other methods.

In some embodiments, the second processing circuit 520 may include a second division circuit 521 and a second rotation circuit 522, wherein the second division circuit 521 can be used to perform a division operation on the data to be quantized and the scaling factor, and the second rotation circuit 522 can be used to perform a rotation operation on the result of the division operation (i.e., the intermediate quantization result) to output the result of quantization of the data to be quantized (i.e., the quantized data).

In other embodiments, the second processing circuit 520 can also be used to perform dequantization processing on the quantized data using scaling factors. For example, the second processing circuit 520 may also include a multiplication circuit for multiplying the quantized data and the scaling factors. In still other embodiments, the second rotation circuit 522 can also be used to perform rotation operations on the quantized data to convert the quantized data from the target data type to the data type required for dequantization processing; the multiplication circuit is used to multiply the rotated quantized data and the scaling factors to obtain the dequantization result. When the required data type is the original data type of the aforementioned data to be quantized, dequantization processing can restore the quantized data to the data to be quantized before the quantization operation.

It is understood that the dequantization process is not limited to being performed by the second processing circuit 520 as described above, but can also be performed by other circuits. For example, in some embodiments, the apparatus 500 may further include a third processing circuit 530 (shown in a dashed box in the figure) for dequantizing the quantized data using a scaling factor. The third processing circuit 530 may receive, for example, quantized data output by the second processing circuit 520, and may also receive scaling factors output by the first processing circuit 510. In some embodiments, a storage circuit may be further included between the third processing circuit 530 and the second processing circuit 520, or the second processing circuit 520 may include a storage circuit for storing quantized data so that the third processing circuit 530 can read the quantized data from the storage circuit for dequantization processing.

In some embodiments, when a piece of data to be quantized is input to the comparison circuit 511, the comparison circuit 511 can output the absolute value of that piece of data as the maximum absolute value. In some embodiments, when multiple pieces of data to be quantized are input to the comparison circuit 511, the comparison circuit 511 can output the maximum absolute value among the multiple pieces of data to be quantized. In still some embodiments, the comparison circuit 511 can also be used to divide the multiple pieces of data to be quantized into multiple groups and output the maximum absolute value of each group of data to be quantized; the first division circuit 512 can be further used to determine a scaling factor corresponding to each group of data to be quantized; and the first revolution circuit 513 can be further used to determine a scaling coefficient corresponding to each group of data to be quantized based on each scaling factor. In some embodiments, the data to be quantized is an element of tensor data, and the comparison circuit 511 can be further used to divide the multiple pieces of data to be quantized of the tensor data into multiple groups in at least one dimension of the tensor data.

It is understood that the above description is exemplary and not restrictive. For example, the first processing circuit for obtaining the scaling factor may not be limited to the comparison circuit, the first division circuit, and the first rotation circuit described above, but may also be implemented using, for example, a lookup table (circuit). Since the dividend (the maximum value that the target data type can represent) is a constant, a lookup table or similar method can be used to directly determine the value based on the input data, thereby implementing the division operation and data type conversion functions described above to obtain the scaling factor. Furthermore, to facilitate understanding of the grouping function implemented by the comparison circuit, an exemplary description will be provided below with reference to Figure 6.

Figure 6 shows a schematic block diagram of a comparison circuit according to some embodiments of this disclosure. As shown, the comparison circuit 511 may include a comparator tree composed of multiple levels of comparators, for example, it may include layers a, b, c, ..., z, each level including one or more comparators, with the number of comparators decreasing sequentially at each level. The outputs of multiple comparators in the upper level of an adjacent level can be input to the comparators in the lower level. In practical applications, the outputs of the required levels can be selected as needed by means of software or hardware to achieve grouping of multiple data to be quantized and to obtain the maximum absolute value of each group of data to be quantized. For ease of understanding, further examples will be provided below with reference to Figure 6.

As further illustrated in Figure 6, layer a includes comparators a1, a2, a3, ..., an, where n represents the number of comparators in layer a; layer b includes comparators b1, b2, etc.; layer c includes comparator c1, etc.; and layer z includes comparator z1. It is understood that the comparator tree shown in the figure is exemplary, and the number of levels in the comparator tree can be set more or less as needed, and the number of comparators included in each level can also be set as required.

In some embodiments, the number of comparators *n* in layer *a* can be the same as the number of input data to be quantized, so that each data point to be quantized is input into a corresponding comparator in layer *a*. As the number of comparators in subsequent layers decreases sequentially, the output can be gradually reduced to obtain the desired grouping result and the maximum absolute value of each group. For example, assuming the input layer receives 128 data points to be quantized, and layer *a* has 128 comparators, then each data point to be quantized is input into a corresponding comparator in layer *a*. The output of layer *a* can then divide the 128 data points to be quantized into 128 groups. Assuming layer *b* includes 64 comparators, then the output of every two comparators in layer *a* can be input into a comparator in the next layer, layer *b*. For example, the outputs of comparators *a1* and *a2* in layer *a* can be input into comparator *b1* in layer *b*. Layer *b* then outputs 64 maximum absolute values, thus dividing the 128 data points to be quantized into 64 groups. Assuming layer c contains 32 comparators, the outputs of every two comparators in layer b can be input into a comparator in the next layer c. For example, the outputs of comparators b1 and b2 in layer b can be input into comparator c1 in layer c. At this point, layer c outputs 32 maximum absolute values, effectively dividing the 128 data points to be quantized into 32 groups. This process continues until the last layer z outputs a single maximum absolute value, thus dividing the 128 data points to be quantized into one group.

Taking this embodiment as an example, in practical applications, assuming that 128 data points to be quantized need to be divided into groups of four, resulting in 32 groups, the output of layer C can be selected as the output of comparator circuit 411 through methods such as programming, logic control, or selectors. This allows the first division circuit to determine the scaling factor corresponding to each of the 32 groups of data to be quantized based on the 32 outputs of layer C (i.e., the 32 maximum absolute values). Therefore, the comparator tree provides multiple granularity grouping methods, allowing users to flexibly select the output of a certain level in the comparator tree according to the required quantization granularity.

Furthermore, it should be noted that the comparator tree structure is not limited to the two comparators in each level shown in the diagram whose outputs correspond to one comparator in the next level. It can also be configured to have, for example, three or more comparators whose outputs correspond to one comparator in the next level. For instance, assuming level a has 128 comparators, level b can have 4 comparators. That is, the outputs of the 32 comparators in level a correspond to one comparator in level b. Therefore, the output of level b can divide the 128 data points to be quantized into 4 groups and output the maximum absolute value of the 32 data points in each group.

In other embodiments, the number of comparators n in layer a can be different from the number of input data to be quantized, as illustrated below. Assuming the number of comparators in layer a is less than the number of data to be quantized, multiple data to be quantized can be input into a single comparator in layer a. This achieves the division of multiple data to be quantized into fewer groups for output in layer a. In this embodiment, the data to be quantized input to each comparator in layer a can also be controlled before layer a (e.g., in the input layer) by using methods such as address decoders, multiplexers (MUX), or other logic circuits.

For example, suppose the input layer receives 128 data points to be quantized, and layer 'a' has 64 comparators. Then, every two data points to be quantized are input into one of the corresponding comparators in layer 'a'. At this point, the output of layer 'a' divides the 128 data points into 64 groups. The number of comparators in subsequent layers is halved sequentially until the last layer outputs the maximum absolute value, at which point the 128 data points are divided into one group.

The foregoing description, in conjunction with Figure 6, exemplarily illustrates the scheme for implementing grouping and outputting the maximum absolute value using a comparison circuit according to an embodiment of this disclosure. It is understood that the structure of the comparison circuit shown in the figure and the preceding description are exemplary and not restrictive. For example, the last level of the comparison circuit 511 may not be limited to including only one comparator as shown in the figure, but may also include multiple comparators as needed, such as using level c in the figure as the last level. Furthermore, the comparator circuit 511 may not be limited to including only one comparator tree, but may also include multiple comparator trees as needed, thereby enabling, for example, parallel grouping. The comparison circuit of this embodiment of the disclosure may not be limited to being implemented using the aforementioned comparator tree, but may also be implemented based on radix sorting or other methods.

It is also understood that the grouping method in this disclosed embodiment is not limited to the comparison circuit, but can also be implemented by combining software. For example, by setting grouping parameters (including information such as the number of data to be quantized in each group K) through software, the comparison circuit can read K consecutive data to be quantized as a group according to these grouping parameters, and directly output the maximum absolute value in each group of data to be quantized.

In this disclosed embodiment, when referring to "particle quantization format," "particle size format," or "particle quantization," it refers to the three-part data format shown in Figure 4. However, the specific element data types, scaling block sizes, scaling factor data types, etc., are not limited to those listed in Table 1, but can include many other combinations. In other words, the particle quantization format can include the MX quantization format proposed by OCP, the extended MX quantization format proposed in this disclosed embodiment, or any other data format conforming to the structure of Figure 4.

In this disclosed embodiment, the term "instruction" can include software instructions, hardware instructions, firmware instructions, or any combination thereof. Software instructions typically refer to statements or commands in a programming language; they are high-level, abstract instructions, such as function calls, machine code, and bytecode. Hardware instructions, also known as machine instructions or instruction set architecture (ISA) instructions, are low-level commands that computer hardware can directly recognize and execute, such as processor instructions and CPU instructions. Firmware instructions include, for example, opcodes and microcode. Furthermore, different CPU architectures and different instruction sets can have different hardware instruction sets. Based on the complexity of the computer instruction set, hardware design, execution speed, compiler complexity, and instruction format, they can be further classified as: CISC (Complex Instruction Set Computer), RISC (Reduced Instruction Set Computer), and VLIW (Very Long Instruction Word).

Exemplary Convolution Operation Principle

Convolutional layers in neural network models perform convolution operations, extracting features by applying convolution kernels (also called weights, filters, etc.) to the input neurons (also called input data or input feature maps). A convolutional layer can contain multiple convolution kernels, with each element of the kernel corresponding to a weight coefficient and a bias.

In conventional 3D convolution operations, assuming the tensor shape of the input neurons or input feature maps in a convolutional layer is represented as X[N Hi Wi Ci], indicating N sets of input feature maps, each containing Hi×Wi×Ci pieces of information, where Hi and Wi are the height and width of the input feature map, respectively, and Ci is the number of input feature maps, also known as the number of input channels. The tensor shape of the convolutional kernel is represented as K[Co Kh Kw Ci], meaning the convolutional layer has Ci×Co kernels of size Kh×Kw, where Ci is the number of input channels, Co is the number of output feature maps (or output channels), and Kh and Kw are the height and width of the convolutional kernel, respectively. The output result is Y[N Ho Wo Co], meaning there are N sets of output feature maps, each containing Ho×Wo×Co pieces of information, where Ho and Wo are the height and width of the output feature map, respectively, and Co is the number of output channels. Therefore, the simplified mathematical formula for convolution operations can be expressed as follows:
Y _in,jc,jh,jw ＝∑ _{0≤ic≤ci,0≤ih≤kh,0≤iw≤kw} X _{in,ic,jh×sh+ih,jw×sw+iw} ×K _jc,ic,ih,iw (1)

In the above formula, sh and sw are the convolution stride in the height and width directions, respectively. The size of the convolution stride affects the size of the output feature map. The formula ignores the bias, padding, and dilation, and assumes that the input data X has already been padded and the convolution kernel has already been dilated. The formula also ignores the N-dimensional dimension; the forward computation of the neural network model is independent in the N-dimensional dimension. When the convolution kernel is working, it scans the input features with a certain stride, performs element-wise matrix multiplication and summation on the input features within the convolution window, and adds the bias.

Note that in this paper, the terms input feature map, input data, neuron, or input neuron are used interchangeably; convolution kernel, filter, or weights are used interchangeably. Furthermore, the H (height) and Y dimensions are used interchangeably, as are the W (width) and X dimensions. Accordingly, the H dimension of the input feature map can be represented as Hi or Yi, the H dimension of the output feature map can be represented as Ho or Yo, and the W dimension is represented similarly. In this disclosed embodiment, each convolution output point has a corresponding convolution window, the shape of which is equal to the shape of the convolution kernel. The value of each convolution output point corresponds to the positional multiplication and accumulation result of the input feature map and weights within its convolution window.

Exemplary data processing device

This disclosure presents a convolution operation scheme compatible with MX quantization format. In this convolution operation, any one or more of the input neurons, weights, and output neurons can be in a particle-quantized data format. When any one or both of the input neurons and weights are in a particle-quantized format, the data dequantization process can be integrated into the convolution operation without prior dequantization. This avoids separate dequantization processing and fully utilizes the advantages of low bit width and small memory footprint of the particle-quantized format in the convolution operation. Furthermore, when it is required that the output neurons or input neurons and weights that are not in a particle-quantized format be converted to a particle-quantized format, they can be quantized online via hardware circuitry.

Figure 7 illustrates a schematic structural diagram of a data processing apparatus that can implement embodiments of the present disclosure. It can be understood that this structure can be viewed as a refinement of the internal structure of a single processing core in Figure 3, or as a functional partitioning diagram based on multiple processing cores shown in Figure 3. As shown in Figure 7, the data processing apparatus 700 may include a control circuit 710, a storage circuit 720, and an arithmetic circuit 730, which are interconnected to transmit various data and instructions.

The control circuit 710 is responsible for handling various functions on the data processing device 700, including but not limited to control, instruction fetching, decoding, and calculation. The control circuit 710 may include, for example, the control module 31 shown in FIG3.

The storage circuit 720 can be used to store information, including at least pre-processing and/or post-processing information, and may also include intermediate information that needs to be cached during processing, such as various RAMs shown in FIG3, or on-chip caches. In some embodiments, the storage circuit 720 can be configured to store input neurons, weights, convolution operation results (output neurons), and/or cache intermediate results.

The arithmetic circuit 730 can be configured to perform various arithmetic operations according to relevant instructions. For example, the arithmetic circuit 730 can be configured to perform convolution operations on the input neurons and weights under the control of the control circuit 710. In this case, the arithmetic circuit 730 may include a convolution circuit 731.

In some embodiments, the control circuit 710 may be configured to parse convolution instructions and control the storage circuit 720 and the arithmetic circuit 730 to perform convolution operations on the input neurons and weights.

The convolution instruction directs the convolution operation on the input neuron and weights to obtain the output neuron. In some embodiments, the convolution instruction may support the granular quantization of any one or more of the input neuron, weights, and output neuron. For example, the data formats of the input neuron, weights, and output neuron may differ in different application scenarios. In the first convolutional layer of a neural network, the input neuron may be unquantized, while the weights may have been quantized, for example, offline, and the output neuron requires quantization before being transmitted to the next convolutional layer. In this case, in the convolution operation of the next convolutional layer, the output neuron received from the previous layer is now quantized as the input neuron, the weights are also quantized, and the output neuron obtained from the convolution operation also requires quantization before being input to the next convolutional layer. Those skilled in the art will understand that the above scenario is merely exemplary, and various other application scenarios may exist where any one or more of the input neuron, weights, and output neuron are granularly quantized.

As described above, a granular quantization format includes three parts: a scaling factor (X, scale), a granular block size (K, scaling block size), and the private element data within the block (P _{_i} , data). Typically, the granular block size is pre-defined, for example, 32. Therefore, the convolution operation requires two parts for the granular quantization parameters: the scaling factor X and the element data P_{_i} . Compared to convolution operations performed on non-quantized data, this adds the memory access requirement for the scaling factor X.

In some embodiments, the computation circuit 730 is configured to perform a convolution operation on the data of the input neuron and the data of the weights according to the convolution instructions described above, and to fuse the scaling factors of the input neuron and/or the scaling factors of the weights in the convolution operation. More specifically, the scaling factors can be fused in the convolution operation in units of particle-level granular block size.

In some embodiments, the convolution circuit 731 can be configured to perform the convolution operation as follows: iteratively calculate the partial sum of each granular block in units of granularity block size T (quantified by particle size); and iteratively accumulate the partial sums of each granular block within a single convolution window to obtain the output neuron value of the corresponding convolution window. It is understood that the granularity block size T typically does not exceed the amount of hardware computation per operation, and the computational data involved in a convolution window usually includes multiple granular blocks. Therefore, partial sums can be calculated in units of granularity blocks, and then the partial sums can be accumulated.

Furthermore, in some embodiments, the convolution circuit 731 can be configured to calculate the partial sum of each granularity block as follows: within each granularity block, the sum of the product of the data of the input neuron and the data of the weights is calculated; the product of this sum of products with the scaling factor of the input neuron and/or the scaling factor of the weights is calculated to obtain the partial sum of the granularity block. Since the element data shares the same scaling factor within the same granularity block, the sum of the element-wise products can be calculated first, and then multiplied by the common scaling factor, thereby incorporating dequantization into the convolution operation.

Equation (1) above gives the convolution operation for non-quantized data. The order of the convolution operation can be adjusted depending on which dimension(s) of the tensor data the granularity blocks are divided on. In some embodiments, when the input neurons and weights are in a granular quantization format and have been quantized according to the granularity block size T in the input channel Ci dimension, Equation (1) can be adjusted to Equation (2) as follows:

In formula (2), Y is the output neuron with dimensions Ho, Wo, and Co; A is the input neuron with dimensions Ci, Hi, and Wi; W is the weight with dimensions Ci, Kh, Kw, and Co, where _Kh and _Kw are the sizes of the Kh and Kw dimensions, respectively. The input neuron and/or weights have been quantized in the Ci dimension, and the quantized granularity block size is T. are the scaling factors for the nth granularity block of the input neuron A and the weight W, respectively, and s _{_h} and s_{_w} are the convolution strides of the weights in the dimensions K_h and K_w, respectively.

As can be seen from formula (2), in the above embodiment, since the input neurons and weights share a scaling factor for quantization every T data points in the Ci dimension, and the convolution operation is cumulative in the Ci dimension, the input neuron data and weight data can be multiplied and accumulated to obtain a partial sum in units of granularity block size T. Then, multiply by their respective scaling factors to achieve dequantization and obtain the dequantized partial sum. The dequantized partial sum can be accumulated with the dequantized partial sums of other granularity blocks within the same convolution window to obtain the output point corresponding to that convolution window, i.e., the value of the output neuron.

Specifically, in some embodiments, the convolution circuit 731 may be configured to perform the above convolution operation as follows: in the input channel Ci dimension, the partial sum of each granularity block is calculated iteratively in units of granularity block size T; in the input channel Ci dimension, the partial sum of each granularity block is accumulated to obtain a first accumulation result; and in the height H and width W dimensions, the first accumulation results within a single convolution window are accumulated iteratively to obtain the output neuron value of the corresponding convolution window.

In the above embodiment, particle quantization is the quantization of shared scaling factors on the Ci dimension according to T data. When the number of data on the Ci dimension is not sufficient to align to T, when it is split into less than T data, the remaining data share the same scaling factor.

It is understandable that although formula (2) above assumes that both the input neuron and the weights have scaling factors, it is also suitable for cases where either or both of the input neuron and the weights are non-quantized data. For example, when the input neuron is non-quantized data and the weights are quantized data, there is no scaling factor for the input neuron; the partial sum of the granularity blocks can be obtained by multiplying the sum of the products by the scaling factor of the weights. In other words, the scaling factor of the input neuron is 1 in this case, and vice versa. It is understandable that formula (2) is also compatible with the case where both the input neuron and the weights are non-quantized data, in which case the scaling factors of both the input neuron and the weights are 1.

In some applications, the bit widths of the input neuron data and the weight data may differ. Typically, computing hardware supports operands with the same bit width, i.e., symmetrical input. To better adapt to the hardware, a preprocessing circuit can be added to the computing circuit 730.

As shown in Figure 7, in some embodiments, the computation circuit 730 may further include a first rotation circuit 732, which can be configured to rotate the data of the input neuron and/or the data of the weights in response to a convolution instruction indicating that the data bit width of the input neuron and the weights are different, so that the data bit width of the input neuron and the weights are the same. Then, the convolution circuit 731 can perform the aforementioned convolution operation based on the data of the input neuron and the data of the weights having the same data bit width, as well as the scaling factor of the input neuron and/or the scaling factor of the weights.

The convolution instructions disclosed in this embodiment are compatible with various input conditions. The following describes possible processing solutions for each condition.

(1) Non-quantized weights and non-quantized input neurons have the same bit width. No revolutions are required in this case.

(2) Non-quantized weights and non-quantized input neurons have different bit widths. In this case, the first rotation circuit 732 can rotate the weights and/or input neurons to make the data bit widths of the weights and input neurons the same. For example, data with shorter bit widths in the weights and input neurons can be converted to data with longer bit widths to unify the bit widths. Of course, both the weights and input neurons can also be converted to data with bit widths longer than their original widths, thereby improving accuracy.

(3) The particle-quantized weights or input neurons have the same bit width as the non-quantized input neurons or weights. There are two processing schemes at this time. One is to not perform a number of revolutions, and to dequantize the convolution circuit 731 during the convolution operation, for example, according to the operation method described above in conjunction with formula (2). The other is to perform dequantization on the particle-quantized weights or input neurons in the first number of revolutions circuit 732, so that the subsequent convolution operation can be performed together with the non-quantized input neurons or weights according to the conventional convolution operation method (for example, formula (1)).

(4) The bit width of the quantized weights or input neurons differs from that of the unquantized input neurons or weights. In this case, a rotation is required, but similar to (3), there are two possible solutions. One is that the first rotation circuit 732 rotates the weights and/or input neurons to make their data bit widths the same. Typically, the bit width of the quantized data is shorter, so the quantized weights or input neurons can be converted to the same bit width as the unquantized input neurons or weights (longer bit width data). The scaling factor of the quantization is then processed by the convolution circuit during the convolution operation. Similarly, both the weights and input neurons can be converted to data with a longer bit width than their original values. Another solution is that the first rotation circuit performs dequantization while rotating the quantized weights or input neurons, allowing subsequent convolution operations to be performed together with the unquantized input neurons or weights using the conventional convolution operation method.

(5) The particle quantization weights and input neurons have the same bit width. At this time, no revolutions are needed. The convolution circuit 731 performs inverse quantization by fusing the scaling factor during the convolution operation, for example, according to the operation method described above in conjunction with formula (2).

(6) The weights and input neurons are quantized into particles with different bit widths. At this point, the first rotation circuit 732 can rotate the weights and/or input neurons to make their data bit widths the same. For example, data with shorter bit widths in the weights and input neurons can be converted to data with longer bit widths, thus unifying the bit width. The scaling factor of the particle quantization is then processed by the convolution circuit during the convolution operation. Similarly, both the weights and input neurons can be converted to data with a bit width longer than their original values. For example, if the data type of the input neuron after particle quantization is FP8 and the data type of the weight after particle quantization is FP4, both data types can be converted to FP16, and convolution operations can be performed based on the FP16 data type. Note that the converted FP16 is still particle quantization data with scaling factors.

Among the various solutions described above, compared to the approach of performing dequantization before convolution, the advantage of using the scaling factor of particle quantization (i.e., convolution fused dequantization) in the convolution operation of this disclosed embodiment is that it can improve the performance speedup. For example, when the input neurons and weights are both particle quantized data, such as neuron data and weights being FP8, if dequantization is performed first, converting them to quantized FP16, the computation bandwidth is N bytes, and the number of data that can be processed in one cycle is N/2. However, if the quantized data format of FP8 is directly input into the computation circuit, and dequantization is performed using the scaling factor X during the operation, the computation bandwidth N can be fully utilized, and the computing power is twice that of the previous approach.

It is understandable that when revolutions count is required, the actual memory access on the revolutions channel needs to be based on the channel bandwidth/revolutions ratio to access the memory circuit (e.g., memory circuit 720) to avoid the amount of data after revolutions exceeding the channel bandwidth. For example, if the bandwidth is 128B before revolutions and FP4 after revolutions, then the amount of data accessed in a single instance is 128B/(8/4) = 64B.

As mentioned earlier, compared to convolution operations performed on non-quantized data, the convolution instructions suitable for particle quantization in this disclosure embodiment add memory access requirements for the scaling factor X.

In some implementations, the scaling factor X can be stored contiguously with its associated k element data P _{_i}. In this case, the scaling factor X and the element data P_{_i} can be accessed together, for example, through a single memory address. In this scenario, the convolution instruction can provide only one memory address for a single parameter.

In other implementations, the scaling factor X can be stored independently of its associated k element data P _{_i}. In this case, the scaling factor X and the element data P_{_i} can be accessed independently, for example, through different addresses. In this scenario, the convolution instruction requires two memory addresses for a single parameter.

In some embodiments, the scaling factor X can adopt the aforementioned compressed format to save storage space and transmission bandwidth. In this case, the convolution instruction can add corresponding indication information to indicate this compressed format of the scaling factor X. The computation circuit 730 may further include a decompression circuit 734 configured to decompress the compressed scaling factor in response to the convolution instruction indicating that the scaling factor is in a compressed format, to obtain the decompressed scaling factor. In some implementations, the compressed format includes a shared exponent of Z compressed scaling factors and corresponding Z new mantissas, where Z ≥ 2. In this case, the decompression circuit 734 is further configured to combine the shared exponent with the Z new mantissas respectively to obtain Z decompressed scaling factors for use by the convolution circuit 731.

In current AI chips, access to on-chip RAM is typically a bottleneck, as existing data already consumes a significant portion of the on-chip memory bandwidth. To address the memory access requirements of newly added scaling factors, two approaches can be taken. One approach is to increase the memory access bandwidth, for example, by adding several new ports to the on-chip RAM for accessing these new scaling factors. The other approach is to reuse existing ports for scaling factor access. For example, a selector can be added to the existing neuron read port to select whether to read neuron data or neuron scaling factors; a selector can be added to the existing weight read port to select whether to read weight data or weight scaling factors. Similarly, a selector can be added to the existing output neuron (result) write-back port to select whether to write back result data or result scaling factors.

When multiplexing ports, data cannot be continuously supplied because time needs to be allocated for reading scaling factors, which inevitably leads to some performance degradation. Therefore, in some embodiments, the property that scaling factors are shared within granular blocks can be utilized to fully utilize the bandwidth when reading scaling factors in each cycle, and buffer them for use in subsequent cycles.

Specifically, in some embodiments, the convolution instruction also instructs the scaling factor and its corresponding data to reuse the same read port, and for each frame of scaling factor read, multiple frames of corresponding data are read proportionally. It can be understood that even without port reuse, the scaling factor read port and the data read port can still be controlled to perform read operations proportionally. For example, for each frame of scaling factor read by the scaling factor read port, the data read port reads the corresponding data for n frames. In this way, the scaling factor read port will be idle for n-1 frames, which can then be used to read other data.

In these embodiments, as shown in FIG7, the data processing apparatus 700 further includes a first buffer 741 configured to buffer scaling factors read from off-chip to be supplied in proportion to the corresponding data to the convolution circuit 731 in the arithmetic circuit 730 for performing convolution operations.

The above ratio is determined at least based on the granularity block size T, the data bit width of the scaling factor, and the data bit width of the corresponding data. Since a scaling factor is shared among K (i.e., T = K, for example, 32) data elements, the usage ratio is 1 scaling factor to T corresponding data elements. Based on the data bit width _Xbw of the scaling factor and the data bit width _Pbw of the corresponding data, the usage ratio (in bytes) can be determined as: 1 × _Xbw : T × _Pbw .

For example, when the scaling factor is in E8M3 format with a bit width of 2 bytes, and the element data is in FP8 format with a bit width of 1 byte, and T = 32, the ratio is 2:32 = 1:16. When reading data in each frame, whether reading the scaling factor or the corresponding data, the data volume should be read to fully utilize the transmission bandwidth, thus making full use of transmission performance. For example, assuming the reading/transmission bandwidth is 128 bytes, then reading one frame of scaling factor (128 bytes/2 bytes = 64 scaling factors) corresponds to reading 16 frames of element data (128 bytes × 16 / 1 byte = 64 × 32). The read scaling factor is buffered in the first buffer. When the arithmetic circuit performs convolution operations, for every 128 bytes of element data supplied (e.g., to fully utilize the arithmetic bandwidth), 128 bytes/16 = 8 bytes of scaling factor are supplied from the first buffer. Therefore, the scaling factor read in one frame can remain in the first buffer for multiple frames, thus alleviating bandwidth constraints.

It is understandable that when the scaling factor adopts a compressed format, the above ratio can be adjusted accordingly based on the compression ratio. For example, when the scaling factor format is E8M3, every two adjacent scaling factors are compressed, occupying 2 bytes after compression, and the element data is in FP8 format with a bit width occupying 1 byte, and T=32, the above ratio is 2:2×32=1:32. It can be seen that reading 2 bytes of scaling factor corresponds to 64 elements of data, requiring the reading of 64 bytes of element data. The decompression circuit 734 can decompress the scaling factors cached in the first buffer 741 and provide them to the convolution circuit 731.

It can also be understood that separate cache spaces can be set up for the scaling coefficients of the input neurons and the scaling coefficients of the weights. For example, two first caches can be set up, one for caching the scaling coefficients of the input neurons and the other for caching the scaling coefficients of the weights, and the data can be shifted and retrieved in each first cache according to the usage in each operation cycle. Of course, they can also be set up in the same cache, and the embodiments disclosed herein are not limited in this respect.

It can also be understood that when the data has a number of revolutions, the above ratio is calculated based on the data bit width after the number of revolutions.

Similarly, when the output neuron (the result of the convolution operation) needs to be granularized, the write-back storage circuit will also include the result data and the result scaling factor. Likewise, since a result scaling factor is shared among K (e.g., 32) result data, there may not be enough result scaling factors when the result data can fill the write-back transmission bandwidth. To control the efficient use of the write-back path's transmission bandwidth and avoid write port bottlenecks, a second buffer (second buffer 742 in Figure 7) can be set up to temporarily store the result scaling factors. When the result scaling factors accumulate to a certain amount, for example, when the transmission bandwidth is full, the write-back task is executed, writing these result scaling factors back to the storage circuit in one cycle. Therefore, the output order can be controlled to output n cycles of result data first, followed by one cycle of result scaling factors. Similarly, n is determined based on the granularity block size T, the bit width of the result data, and the bit width of the result scaling factor.

Because the scaling factor output of the arithmetic circuit is written back to the storage circuit in the order of Co-Wo-Ho, and the granular quantization is split and quantized in the Co dimension, there may be discontinuities and jumps in the Co dimension. Therefore, when caching the scaling factor in the second buffer, it is cached in stride order according to the continuous Co. For output points with discontinuous Co, a certain stride is skipped. The stride is related to the instruction Co, and its calculation method is: stride = Co/T*X _bw , where Co is the total Co, T is the granularity block size, and X _bw is the bit width of the scaling factor. For example, assuming 64 Co are calculated per cycle, and the next cycle calculates 64 Co for the next Wo, with a quantization granularity block size T = 32, then 2 scaling factors X will be quantized per cycle. If the total Co is 256, then the storage stride for the scaling factor in two consecutive cycles is stride = 256/32*X _bw , which is calculated based on the difference between the starting storage positions of the two scaling factors.

When writing the result scaling factor from the second buffer back to the storage circuit, a masked write-back can be used. The mask indicates the valid data that needs to be written to the storage circuit in the current write-back cycle. Since the result scaling factor is cached by stride, there will be empty portions. By using the mask, only the necessary data (i.e., valid data) can be written back to the storage circuit.

The particle quantization of the output neuron can be performed in the quantization circuit 733 in Figure 7. In some embodiments, the quantization circuit 733 can be configured to particle quantize the result of the convolution operation according to the convolution instruction to generate result data and result scaling factors. Optionally or additionally, in some embodiments, the quantization circuit 733 can also be configured to particle quantize the input neurons and weights that have not been particle quantized according to the convolution instruction. Through particle quantization, the computation of high-bit-width data can be converted into the computation of low-bit-width data, reducing computational power consumption.

In some embodiments, the quantization circuit 733 may further include a compression circuit (not shown), configured to: compress two or more scaling factors after quantization of two or more sets of data to be quantized, in response to a convolution instruction indicating that the scaling factors are in a compressed format. In this case, the quantization circuit uses the compressed scaling factors to quantize the corresponding sets of data to be quantized, respectively, to output quantized data with the target data type.

The quantization circuit 733 can be implemented in various ways. In some embodiments, the specific implementation of the quantization circuit 733 can be referred to the previous description of extended MX quantization, which will not be repeated here.

The convolution operation scheme compatible with particle quantization format has been described above with reference to Figure 7. Those skilled in the art will understand that although the various buffers and operation circuits are shown as discrete modules in Figure 7, depending on the configuration, the buffers and operation circuits can also be combined into a single module. For example, the first buffer can be combined with the convolution circuit, and the second buffer can be combined with the quantization circuit. The embodiments disclosed herein are not limited in this respect.

Figure 8 illustrates an exemplary flowchart of a data processing method implemented by a data processing circuit according to an embodiment of this disclosure. In this data processing method, a control circuit controls a computation circuit to perform a convolution operation on the input neurons and weights.

More specifically, in step 810, the control circuit parses the convolution instruction, which instructs to perform a convolution operation on the input neuron and weights to obtain the output neuron, and either or both of the input neuron and weights are quantized.

Next, in step 820, under the control of the control circuit, the computation circuit performs convolution operation on the data of the input neuron and the data of the weights according to the convolution instruction, and incorporates the scaling factor of the input neuron and/or the scaling factor of the weights into the convolution operation.

In some embodiments, step 820 may further include: sub-step 821, in response to a convolution instruction indicating that the data bit width of the input neuron and the weights are different, performing a rotation on the data of the input neuron and/or the data of the weights to make the data bit width of the input neuron and the weights the same; and sub-step 822, performing a convolution operation based on the data of the input neuron and the data of the weights having the same data bit width, and the scaling factor of the input neuron and/or the scaling factor of the weights. Sub-step 821 may be executed by a first rotation circuit, and sub-step 822 may be executed by a convolution circuit.

In some embodiments, the number of revolutions in sub-step 821 may be to convert the shorter bit-width data in the input neuron and weights into longer bit-width data.

In some embodiments, substep 822 may be performed as follows: cyclically calculating the partial sum of each granularity block in units of granularity block size T measured by microparticles; and cyclically accumulating the partial sum of each granularity block within a single convolution window to obtain the output neuron value of the corresponding convolution window.

Specifically, calculating the partial sum of each granularity block may include: within each granularity block, calculating the sum of the product of the data of the input neuron and the data of the weights; calculating the product of this sum of products with the scaling factor of the input neuron and/or the scaling factor of the weights to obtain the partial sum of that granularity block.

In some embodiments, step 820 may further include: sub-step 823, which involves quantizing the result of the convolution operation according to the convolution instruction to generate result data and result scaling factors. Sub-step 823 may be performed by a quantization circuit. Specific quantization methods can be found in the preceding description.

Optionally or additionally, in some embodiments, the method may further include a preprocessing step, such as quantizing the input neurons and weights that are not quantized by a quantization circuit according to convolution instructions. Quantization can convert computation of high-bit-width data into computation of low-bit-width data, reducing computational power consumption.

Those skilled in the art will understand that the steps described in the method flowchart correspond to the various circuits of the data processing device described above in conjunction with Figure 7, and therefore the features described above also apply to the method steps, and will not be repeated here.

This disclosure also provides a processor, including the aforementioned data processing apparatus. This disclosure further provides a chip, which may include the processor of any of the embodiments described above in conjunction with the accompanying drawings. Furthermore, this disclosure also provides a board that may include the aforementioned chip.

Depending on the application scenario, the electronic devices or apparatus disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablets, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and/or medical devices. The vehicles include airplanes, ships, and/or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and/or electrocardiographs. The electronic devices or apparatus disclosed herein can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus disclosed herein can also be used in application scenarios related to artificial intelligence, big data, and/or cloud computing, such as cloud computing, edge computing, and terminal applications. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present disclosure can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and/or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and/or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and/or edge devices based on the hardware information of the terminal devices and/or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

It should be noted that, for the sake of brevity, this disclosure describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solutions disclosed herein are not limited by the order of the described actions. Therefore, based on the disclosure or teachings of this document, those skilled in the art will understand that some steps can be performed in a different order or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this disclosure can be considered optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions disclosed herein. In addition, depending on the solution, the description of some embodiments in this disclosure may have different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this disclosure can also be referred to the relevant descriptions of other embodiments.

In terms of specific implementation, based on the disclosure and teachings of this document, those skilled in the art will understand that several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or apparatus embodiments described above, this document has divided them based on logical functions, but in actual implementation, there may be other ways of division. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationships between different units or components, the connections discussed above in conjunction with the accompanying drawings can be direct or indirect couplings between units or components. In some scenarios, the aforementioned direct or indirect couplings involve communication connections utilizing interfaces, where the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this disclosure. Additionally, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically independently. For example, in some embodiments, the quantization circuit, as a post-processing circuit, may be integrated into the convolution circuit or independent of the convolution circuit. In other embodiments, the quantization circuit, as a pre-processing circuit, may be integrated into the revolutions circuit or independent of the revolutions circuit. The embodiments of this disclosure are not limited in this respect.

In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and/or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as central processing units, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage unit or storage device can be any suitable storage medium (including magnetic storage medium or magneto-optical storage medium, etc.), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), ROM (read-only memory), and RAM (random access memory), etc.

The embodiments of this disclosure have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this disclosure. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this disclosure. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this disclosure. Therefore, the content of this specification should not be construed as a limitation of this disclosure.

Claims (30)

A data processing apparatus, comprising: Control circuitry configured to parse convolution instructions, the convolution instructions directing a convolution operation on an input neuron and weights to obtain an output neuron, wherein either or both of the input neuron and weights are particle-quantized; and The computation circuit is configured to perform a convolution operation on the data of the input neuron and the data of the weights according to the convolution instruction, and to fuse the scaling factor of the input neuron and/or the scaling factor of the weights in the convolution operation. According to claim 1, the data processing apparatus, wherein the arithmetic circuit comprises: A rotation circuit is configured to rotate the data of the input neuron and/or the data of the weights in response to a convolution instruction indicating that the data bit widths of the input neuron and the weights are different, so that the data bit widths of the input neuron and the weights are the same; and A convolution circuit configured to perform a convolution operation based on data and weights of input neurons having the same data bit width, and the scaling factor of the input neurons and/or the scaling factor of the weights. According to claim 2, the data processing apparatus further includes: The decompression circuit is configured to decompress the scaling factor in the compressed format in response to the convolution instruction indicating that the scaling factor is in a compressed format, so as to obtain the decompressed scaling factor. According to the data processing apparatus of claim 3, the compression format includes a shared exponent of Z compressed scaling factors and corresponding Z new mantissas, where Z ≥ 2, and the decompression circuit is further configured to: The shared index is combined with the Z new mantissas to obtain Z decompressed scaling factors for use by the convolution circuit. The data processing apparatus according to any one of claims 2-4, wherein the convolution circuit is further configured to convert the shorter bit-width data in the input neuron and the weights into longer bit-width data based on the convolution instruction. The data processing apparatus according to any one of claims 1-5, wherein the arithmetic circuit is configured to perform convolution operations as follows: Using the particle size T as the unit, the partial sum of each particle size block is calculated iteratively; and By iteratively summing the partial sums of each granularity block within a single convolution window, the output neuron value of the corresponding convolution window is obtained. The data processing apparatus according to claim 6, wherein the arithmetic circuitry is configured to calculate the partial sum of each granularity block as follows: Within each granularity block, calculate the sum of the product of the input neuron's data and the weight data; and The product of the sum and the scaling factor of the input neuron and/or the scaling factor of the weights is calculated to obtain a partial sum of the granularity block. The data processing apparatus according to any one of claims 1-7, wherein the arithmetic circuit is configured to perform a convolution operation according to the following formula (1):
Where Y is the output neuron with dimensions Ho, Wo, and Co; A is the input neuron with dimensions Ci, Hi, and Wi; W is the weight with dimensions Ci, Kh, Kw, and Co, where _Kh and _Kw are the sizes of the Kh and Kw dimensions, respectively. The input neuron and/or the weight have been quantized in the Ci dimension, and the block size is T. are the scaling factors for the nth granularity block of the input neuron A and the weight W, respectively, and s _{_h} and s_{_w} are the convolution strides of the weights in the dimensions K_h and K_w, respectively. The data processing apparatus according to any one of claims 6-8, wherein the convolution instruction further indicates a scaling factor for each read frame, and reads data corresponding to multiple frames proportionally, the data processing apparatus further includes: A first buffer is configured to cache scaling factors read from off-chip to be supplied to the computing circuitry in proportion to the corresponding data for performing the convolution operation, wherein the proportion is determined at least based on the granularity block size, the data bit width of the scaling factor, and the data bit width of the corresponding data. According to the data processing apparatus of claim 9, the amount of data read in each frame fully utilizes the transmission bandwidth. The data processing apparatus according to any one of claims 1-10, wherein the arithmetic circuit further comprises: A quantization circuit is configured to perform particle quantization on the result of the convolution operation according to the convolution instruction to generate result data and result scaling factor; and/or is configured to perform particle quantization on the input neurons and weights that have not been particle quantized according to the convolution instruction. The data processing apparatus according to claim 11, wherein the quantization circuit comprises: A first processing circuit is configured to: determine the maximum absolute value of a set of data to be quantized in the result of the convolution operation; obtain the maximum value that the target data type after quantization can represent; determine a scaling factor corresponding to the set of data to be quantized based on the ratio between the maximum absolute value and the maximum value; convert the scaling factor from a first data type to a second data type to output a scaling coefficient, wherein the bit width of the second data type is smaller than the bit width of the first data type; and The second processing circuit is configured to quantize the set of data to be quantized using the scaling factor to output quantized data having the target data type. The data processing apparatus according to claim 12, wherein the first processing circuit comprises: A comparator circuit is used to determine the maximum absolute value in a set of data to be quantized; The first division circuit is used to obtain the maximum value that the target data type after quantization of the data to be quantized can represent, and to determine the scaling factor corresponding to the set of data to be quantized based on the ratio between the maximum absolute value and the maximum value; and The first rotation circuit is used to convert the scaling factor from a first data type to a second data type to output the scaling coefficient. According to the data processing apparatus of claim 13, wherein the first processing circuit is further configured to: determine the number of data to be quantized in the set of data to be quantized based on the single processing volume and/or the number of input ports of the second processing circuit; or The number of data points to be quantized in the set of data points is determined based on the single operation quantity and/or the number of input ports of the arithmetic unit that performs the operation using the quantized data. The data processing apparatus according to any one of claims 12-14, wherein the result of the convolution operation is divided into multiple groups at least in the Co dimension for particle quantization. The data processing apparatus according to any one of claims 12-15, wherein The first data type is the same as the original data type of the data to be quantized, or the bit width of the first data type is greater than the bit width of the original data type; and The second data type is represented as ExMy, where x + y < bit width of the original data type, where E represents the exponent bits, M represents the mantissa bits, x represents the exponent bit width, and y represents the mantissa bit width. The data processing apparatus according to claim 16, wherein y > 0. The data processing apparatus according to any one of claims 12-17, wherein The original data type of the data to be quantized includes at least one of FP16, BF16, FP32, TF32, and FP64; the second data type includes at least one of E8M0, E8M1, E8M2, E8M3, E8M4, E8M5, E8M6, E8M7, E5M0, E5M1, E5M2, E5M3, E5M4, E5M5, E5M6, E3M0, E3M1, E3M2, E3M3, E3M4, E3M5, E3M6, E3M7, E3M8, and E3M9; and/or The target data type includes at least one of FP12, FP8, FP6, FP4, FP2, int8, int4, int2, and int6, and FP12, FP8, FP6, and FP4 support multiple data formats. The data processing apparatus according to any one of claims 12-18, wherein the quantization circuit further comprises a compression circuit configured for: In response to the convolution instruction indicating that the scaling factors are in a compressed format, two or more scaling factors after quantization of two or more sets of data to be quantized are compressed; and The second processing circuit is configured to quantize the corresponding group of data to be quantized using the compressed scaling factor, so as to output quantized data with the target data type. The data processing apparatus of claim 19, wherein the compression circuit is further configured to: Determine the shared exponent of the Z scaling factors to be compressed, Z≥2; Determine the Z new mantissas corresponding to the Z scaling factors; and The shared index is bundled with the Z new tail numbers and output for association. The data processing apparatus of claim 20, wherein the compression circuit is further configured to: Set the largest index among the Z scaling factors as the shared index. The data processing apparatus of claim 21, wherein the compression circuit is further configured to: Set the new mantissa of the scaling factor whose exponent is less than the maximum exponent to 0; and Set the new mantissa, whose exponent is equal to the scaling factor of the maximum exponent, as the original mantissa. The data processing apparatus according to any one of claims 20-22 further comprises one or more of the following: The second data type of the Z scaling factors to be compressed is the same; The Z scaling factors belong to Z adjacent groups of data to be quantized; Z can be 2, 4, or 8. The data processing apparatus according to any one of claims 11-23 further includes: The second buffer is configured to cache the scaling factor of the result so that it is written back to the storage circuit in proportion to the corresponding result data. According to the data processing apparatus of claim 24, the result scaling factor is cached in the second cache in a step-wise manner according to the Co dimension, and the result scaling factor is written back to the storage circuit as a masked write, wherein the mask indicates the valid data that needs to be written to the storage circuit in the content of the current write-back cycle. The data processing apparatus according to any one of claims 1-25, wherein the quantized data and scaling factor of each of the input neuron, the weights and/or the output neuron support independent or joint memory access. The data processing apparatus according to any one of claims 1-26, wherein when one of the input neuron and the weight is non-quantized data, the computation circuit sets the corresponding scaling factor to 1. A processor comprising the data processing apparatus as described in any one of claims 1-27. A board comprising the processor according to claim 28. A data processing method implemented by a data processing device, the data processing device including a control circuit and a calculation circuit, the method comprising: The control circuit parses convolution instructions, which instruct the convolution operation to be performed on the input neuron and weights to obtain the output neuron, wherein either or both of the input neuron and weights are particle-quantized; and The computational circuit performs a convolution operation on the data of the input neuron and the data of the weights according to the convolution instruction, and integrates the scaling factor of the input neuron and/or the scaling factor of the weights into the convolution operation.