JP2018093323A - Image processing system - Google Patents

Abstract

PURPOSE: To provide an imager processing system capable of reducing the code amount of an image signal, while preventing recognition accuracy for an area of interest in the image signal.CONSTITUTION: An imager processing system includes an area of interest determination unit for determining whether or not an area of interest, specified as a desired area, in an image based on an input image signal is encoded with a prescribed quality, generating area of interest information for urging encoding with a prescribed quality, when a determination is made that the area of interest is encoded with a prescribed quality, and generating area of interest information for urging encoding with a quality lower than the prescribed quality, when a determination is made that the area of interest is not encoded with a prescribed quality, and an image coding unit outputting a bit stream signal including a signal encoded the area of interest in the input image signal with the prescribed quality or lower quality according to the area of interest information.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing system for compressing and encoding an image signal.

  In recent years, surveillance cameras have become popular, and further higher resolution, higher frame rate, and multi-viewpoint are desired. However, increasing the resolution, increasing the frame rate, and increasing the number of viewpoints cause a significant increase in the amount of moving image data, resulting in increased communication costs and storage costs.

  In order to alleviate this problem, for example, a specific area where “face” is displayed is detected as a ROI (Region of Interest) from a moving image, and a large number of bits are assigned to this ROI to perform image compression processing. An image processing apparatus has been proposed (see, for example, Patent Document 1).

  In the image processing apparatus, the information amount of the area other than the ROI, that is, the background area is reduced from the information amount of the ROI by using the function of the compression unit that sets the parameter for controlling the code amount and the video quality for each block. As described above, the data amount is reduced by controlling the parameters given to the compression unit.

  However, when a compressor that does not have such a parameter control function is used, the amount of data cannot be reduced by the above-described method.

  Therefore, in order to reduce the amount of information in the background area without depending on the compression unit, the amount of information in the background area is reduced by filtering the background area with a low-pass filter to remove high-frequency component information. An image coding apparatus has been proposed (see, for example, Patent Document 2). In this image encoding device, the generation of artifacts occurring at block boundaries is suppressed while reducing the amount of image signal information by filtering with a low-pass filter.

JP 2009-49979 A JP-A-4-219089

  However, if a scene in which one picture is filled with ROI, for example, a scene in which an up face is projected or a scene that is filled with many faces continues, the entire screen continues to be encoded with high quality. In other words, the above-described method cannot obtain a code amount reduction effect. Therefore, there is a risk of exceeding the assumed communication capacity and storage capacity.

Therefore, it is conceivable to perform picture thinning out as a countermeasure against the excess of capacity, but in this case, there is a possibility that a definitive scene may be missed.
In addition, it is possible to prevent the capacity from being exceeded by lowering the quality of the ROI. However, since the purpose is to monitor, reducing the quality of the ROI area leads to a decrease in monitoring capability. Further, when some post-processing such as authentication processing is performed on the decryption side, the missed shooting of a definitive scene or the deterioration of image quality may result in a loss of post-processing accuracy. For example, in the case where ROI is used as a face area and individual authentication is performed for each face area on the decoding side, decisive missing scenes (for example, missing scenes facing the front) and image quality degradation are recognized. There is a possibility of degrading accuracy. Further, when the ROI is a license plate, the recognition accuracy of the license plate is deteriorated.

  Accordingly, an object of the present invention is to provide an image processing system capable of reducing the code amount of an image signal while preventing a reduction in ROI (region of interest) recognition accuracy on the decoding side.

  An image processing system according to the present invention is an image processing system that encodes an input image signal and outputs it as a bit stream signal, wherein an attention area designated as a desired area in an image based on the input image signal has a predetermined quality. In the case where it is determined whether to encode with the predetermined quality, the information that prompts the encoding with the predetermined quality is determined. An attention area determination unit that generates attention area information indicating information urging encoding with low quality lower than quality in association with the attention area; and the attention area in the input image signal according to the attention area information An image encoding unit that outputs the bit stream signal including a signal encoded with the predetermined quality or the low quality.

  In the present invention, when the ROI (region of interest) included in the input image signal is encoded, it is selectively performed whether the encoding is performed at a predetermined quality or at a lower quality lower than the predetermined quality. I have to. Therefore, first, the attention area is encoded with a predetermined quality, and thereafter, the attention area is encoded with low quality. As a result, since encoding with a predetermined quality using a relatively large number of bits is performed at least once for the attention area, a reduction in recognition accuracy for the attention area on the decoding side can be suppressed. In addition, after the region of interest is encoded with a predetermined quality, the encoding is performed with a lower quality that uses fewer bits than the encoding with the predetermined quality, so that the code amount of the image signal is reduced. .

  Therefore, according to the present invention, it is possible to reduce the code amount of the image signal while preventing the recognition accuracy of the attention area on the decoding side from being lowered.

It is a block diagram which shows the structure of the 1st Example of image processing system S1. It is a flowchart showing operation | movement of ROI detection part U1. It is a flowchart showing operation | movement of ROI detection part U1. It is a block diagram which shows the structure of the 2nd Example of image processing system S1. It is a flowchart showing operation | movement of ROI detection part U1a. It is a flowchart showing operation | movement of ROI detection part U1a. It is a figure showing an example of the function f (L) which calculates | requires the usefulness degree U. It is a figure showing another example of the function f (L) which calculates | requires the usefulness U. It is a figure showing the modification of other examples of function f (L) which calculates usefulness U. FIG. 11 is an operation concept diagram of the prefetch buffer 203. It is a block diagram which shows the structure of the 3rd Example of image processing system S1. It is a flowchart showing the operation | movement implemented by the ROI detection part U1b and the code amount monitoring part 310. FIG. It is a flowchart showing the operation | movement implemented by the ROI detection part U1b and the code amount monitoring part 310. FIG.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram showing an image processing system S1 according to the present invention. Note that the image processing system S1 is used, for example, to transmit a bit stream signal in which a monitoring image captured by a monitoring camera or the like is compressed to suppress a data capacity to another device. That is, the image processing system S1 receives the monitoring image as an input image signal D12 including a series of pixel values representing the luminance level for each pixel, and a bit stream signal obtained by compressing the input image signal D12. D08 is output.

  As shown in FIG. 1, the image processing system S1 includes a ROI (Region of Interest) detection unit U1 and an image encoding unit U2.

  The ROI detection unit U1 detects a region where a specific object such as “face” is displayed as an ROI, that is, a region of interest, by applying a face detection algorithm, for example, from the input image signal D12. Then, the ROI detection unit U1 determines, for each detected ROI, information indicating the position of the ROI in the image and the ROI specific quality (hereinafter referred to as ROI quality) when the ROI is encoded by the image encoding unit U2. And ROI information D04 including information indicating whether to encode at low quality lower than the ROI quality, and supply the image encoding unit U2 with the ROI information D04.

  Based on the ROI information D04, the image encoding unit U2 is obtained by compressing the ROI that is encoded with the ROI quality in the input image signal D12 and the ROI that is encoded with a quality lower than the ROI quality with different quality. The bit stream signal D08 is output. Specifically, the image encoding unit U2 performs encoding by assigning more bits than the predetermined number of encoding bits to the ROI to be encoded with ROI quality. On the other hand, for an ROI that is encoded with a lower quality than the ROI quality, the image encoding unit U2 performs the encoding by assigning a smaller number of bits than the predetermined number of encoded bits.

  As shown in FIG. 1, the ROI detection unit U1 includes an ROI tracking unit 101, an ROI determination unit 104, and a determination record information storage unit 105.

  The ROI tracking unit 101 detects and tracks the ROI from the input image signal D12 by applying, for example, a face detection algorithm, and updates the information representing the position and size of the detected ROI, the ROI identifier, etc. by tracking. Initial information D01 is generated. The ROI tracking unit 101 supplies the ROI initial information D01 to the ROI determination unit 104. The ROI identifier is assigned by the ROI tracking unit 101 to identify one or more ROIs detected in the picture. The same ROI is used for the same ROI across pictures having different capture times. An identifier is attached. For example, when the ROI identifier “I” is assigned to the ROI indicating the face area of a certain person, the ROI tracking unit 101 indicates the face area of the person even if the person has moved in the next time picture. The ROI identifier “I” is continuously assigned to the ROI. In this proposal, the term “tracking” is used in the sense that the ROI to which the ROI identifier is assigned is tracked over time. In short, every time the ROI tracking unit 101 detects a new undetected ROI from the input image signal D12, the ROI identifier for identifying this ROI is assigned in association with the ROI. In the above example, the ROI tracking unit 101 is described to track the ROI using the input image signal D12. However, the ROI may be tracked by a different means. For example, in addition to using the face detection algorithm described above, a method for tracking ROI using a user interface for manually specifying an ROI region or using an image taken with an infrared camera or an image taken with a depth sensor It doesn't matter. In this case, the input image signal D12 is not input to the ROI tracking unit 101, but a signal related to an image captured by an infrared camera or an image captured by a depth sensor is input.

  The ROI determination unit 104 determines, based on the determination recording information D05, whether or not to encode the ROI with ROI quality in the subsequent image encoding unit U2 for each ROI indicated by the ROI initial information D01. Here, when it is determined that the encoding is performed with the ROI quality, the ROI determination unit 104 generates information D04 that represents the information that prompts the encoding with the ROI quality in association with the ROI. On the other hand, when it is determined that the encoding is not performed with the ROI quality, the ROI determination unit 104 generates the ROI information D04 that represents the information that prompts the encoding with the low quality lower than the ROI quality in association with the ROI. . The ROI determination unit 104 supplies this ROI information D04 to the determination record information storage unit 105 and the image encoding unit U2.

  The determination record information accumulation unit 105 accumulates the ROI information D04 and supplies the accumulated information to the ROI determination unit 104 as the determination record information D05.

  As illustrated in FIG. 1, the image encoding unit U2 includes a filter unit 406, a compression unit 407, and a bit stream transmission unit 408.

  The filter unit 406 receives the input image signal D12 and the ROI information D04, and among the pixel value series represented by the input image signal D12, for the pixel values classified into the background area by the ROI information D04. For example, a convolution process is performed using a Gaussian filter or the like, thereby obtaining a filter image signal D06 in which high frequency components are suppressed.

  The compression unit 407 converts the filter image signal D06 into JPEG (Joint Photographic Experts Group), H.264 / MPEG (Moving Picture Experts Group) -4 AVC (Advanced Video Coding), or H.264 H.265 / MPEG-H HEVC (High Efficiency Video Coding) is used to generate compressed video data D07 that is compressed using an image encoding method, and supplies this to the bit stream transmission unit 408.

  The bit stream transmission unit 408 generates a bit stream signal D08 obtained by multiplexing the ROI information D04 and the compressed video data D07.

  Hereinafter, the internal operation of the ROI detection unit U1 shown in FIG. 1 will be specifically described along the flowcharts shown in FIGS.

  First, the ROI tracking unit 101 detects and tracks the ROI from the input image signal D12, thereby acquiring information representing the position and size of the ROI, the ROI identifier, etc., and generating this as ROI initial information D01 (step P10). ). Note that the ROI includes, for example, a face area, and detection of the face area can be realized by performing, for example, identification using Haar-like feature values for each area.

  Next, the ROI determination unit 104 selects one ROI identifier to be determined from the ROI initial information D01, and stores the selected ROI identifier as a ROI identifier X in the built-in register (step P11).

  Next, the ROI determination unit 104 determines whether or not to encode the ROI indicated by the ROI identifier X with the ROI quality based on the determination record information D05 (step P12). Here, the ROI determination unit 104 determines whether or not the ROI indicated by the ROI identifier X has been determined to be encoded with ROI quality before the current time based on the determination record information D05, and has already been determined. For ROIs that have been determined to be encoded with quality, it is determined that they are not encoded with ROI quality, and otherwise they are determined to be encoded with ROI quality.

  Next, when it is determined that the ROI indicated by the ROI identifier X is encoded with the ROI quality, the ROI determination unit 104 proceeds to the execution of the following step P14, while the ROI indicated by the ROI identifier X is encoded with the ROI quality. When it is determined not to be converted, the process proceeds to execution of the following step P15 (step P13).

  In step P14, the ROI determination unit 104 generates ROI information D04 for encouraging encoding at a lower quality than the ROI quality for the ROI indicated by the ROI identifier X. On the other hand, in step P15, the ROI determination unit 104 generates ROI information D04 for encouraging encoding with ROI quality for the ROI indicated by the ROI identifier X.

  After execution of step P14 or P15, the ROI determination unit 104 determines whether or not there is an ROI identifier that has not yet been selected as a target of the above-described determination processing from the ROI initial information D01 (step P16).

  When it is determined in step P16 that there is an unselected ROI identifier, the ROI determination unit 104 selects one ROI identifier to be subjected to determination processing next from the unselected ROI identifiers, The selected ROI identifier is newly set as the ROI identifier X (step P17). After execution of step P17, the ROI determination unit 104 returns to the execution of step P12 described above, and performs the operations of steps P12 to P16 described above again.

  On the other hand, when it is determined in step P16 that there is no unselected ROI identifier, the ROI determination unit 104 supplies the ROI information D04 to the image encoding unit U2 (step P18).

  Then, the determination record information accumulation unit 105 accumulates the ROI information D04 as a determination record (step P19).

  Therefore, thereafter, based on the past determination record recorded in the determination record information storage unit 105, the ROI determination unit 104 performs the determination process in step P12 described above. Note that the number of recording data (recording period) in the determination recording information storage unit 105 may be limited to N pictures (N is an integer of 1 or more). In this case, when the recording period has passed, the ROI determination unit 104 again determines to encode with the ROI quality even if it has been encoded with the ROI quality in the past.

  As described above in detail, in the first embodiment shown in FIG. 1, for the ROI once encoded with high quality, the number of bits is replaced with the ROI quality using a large number of bits thereafter. Control is performed so as to switch to encoding with low quality with less. As a result, the data capacity of the bit stream signal D08 can be suppressed even if image scenes in which the screen is filled with a large number of ROIs or large-sized ROIs are temporally continuous.

  FIG. 4 is a block diagram showing the configuration of the second embodiment of the image processing system S1. The image processing system S1 shown in FIG. 4 adopts the ROI detection unit U1a in place of the ROI detection unit U1 shown in FIG. 1, and other configurations except that a synchronization buffer 206 is newly provided are shown in FIG. Identical to that shown. That is, the internal configuration and operation of the image encoding unit U2 shown in FIG. 4 are the same as those of the image encoding unit U2 shown in FIG.

  In FIG. 4, after receiving and buffering the input image signal D12, the synchronization buffer 206 supplies a signal obtained by delaying the input image signal D12 by a predetermined time to the filter unit 406 of the image encoding unit U2 as a synchronization image signal D06. The predetermined time is a time corresponding to the delay time in the prefetch buffer 203 included in the ROI detection unit U1a. That is, the synchronization buffer 206 is provided in order to synchronize the image between the ROI information D04 supplied to the image encoding unit U2 by the ROI detection unit U1a and the input image signal D12.

  The ROI detection unit U1a employs an ROI determination unit 104a instead of the ROI determination unit 104 shown in FIG. 1, and newly includes an ROI usefulness assigning unit 202 and a prefetch buffer 203. Since the ROI tracking unit 101 and the determination record information storage unit 105 shown in FIG. 4 are the same as those shown in FIG. In the image processing system S1 shown in FIG. 4, together with the input image signal D12 described above, necessary size information D13 for designating an upper limit size LQ indicating the upper limit of the ROI size required for calculating the usefulness of the ROI. Receive.

  The ROI usefulness assigning unit 202 obtains the usefulness for each ROI while referring to the required size information D13, generates ROI secondary information D02 assigned with the usefulness in association with each ROI, and pre-reading buffer 203.

  The prefetch buffer 203 buffers the ROI secondary information D02 for each predetermined picture, generates ROI determination target information D03A and ROI prefetch information D03B based on the buffered ROI secondary information D02, and returns the ROI. It supplies to the determination part 104a.

  The ROI determination unit 104a encodes the ROI with ROI quality in the subsequent image encoding unit U2 for each ROI included in the ROI determination target information D03A based on the determination record information D05 and the ROI prefetch information D03B. It is determined whether or not to make it. If the ROI determination unit 104a determines that encoding is performed with ROI quality, the ROI determination unit 104a generates information D04 in which information that prompts encoding with ROI quality is associated with the ROI. On the other hand, when it is determined that the encoding is not performed with the ROI quality, the ROI determination unit 104a generates information D04 that represents the information that prompts the encoding with the low quality lower than the ROI quality in association with the ROI.

  In this embodiment, a configuration example in which the ROI usefulness assigning unit 202 uses the necessary size information D13 is shown. However, the ROI usefulness assigning unit 202 does not consider the necessary size information D13, When calculating the usefulness, the necessary size information D13 is not necessary.

  Hereinafter, the operation of the ROI detection unit U1a shown in FIG. 4 will be specifically described along the flowcharts shown in FIGS.

  First, the ROI tracking unit 101 detects and tracks the ROI from the input image signal D12, thereby acquiring information representing the position, size, orientation, ROI identifier, and the like of the ROI, and outputs this as ROI initial information D01. (Step P10). The ROI initial information D01 includes information necessary for calculation of usefulness performed by the ROI usefulness assigning unit 202 described later.

  Next, the ROI usefulness assigning unit 202 calculates the usefulness of the ROI for each ROI identifier based on the required size information D13 and the information indicating the size and orientation of the ROI included in the ROI initial information D01. Information indicating the degree added to the ROI initial information D01 is generated as the ROI secondary information D02 (step P21).

As the calculation of the usefulness U, for example, the following calculation methods 1 to 3 are used.
[Calculation method 1]
The usefulness U is a function f of ROI size L (L is a positive real number) as shown below.

U = f (L)
As the ROI size L, for example, one of the following six types is used.

L = H
L = W
L = min (H, W)
L = max (H, W)
L = H × W
L = H + W
H: Number of pixels in the vertical direction (vertical direction) of the screen
W: Number of pixels in the horizontal direction (horizontal direction) of the screen Here, the function f is expressed by the following mathematical expression, for example.

f (L) = α ₁ × L + β ₁
α ₁ , β ₁ : Predetermined constants As the function f, any nonlinear function may be adopted. For example, the function f having an arbitrary shape can be defined by using a look-up table that represents the correspondence between the size L and the usefulness U.

  At this time, when the image processing system S1 uses a monitoring image captured by the monitoring camera as a processing target, it is considered that the larger the ROI, the more useful. Therefore, as the function f, a function is used in which the usefulness U increases as the ROI size L increases.

  Also, assuming that the decoding side that decodes the bit stream signal D08 output from the image processing system S1 performs post-processing such as face authentication and recognition, the image size required for the post-processing is determined in advance. There may be. For example, assuming that an authentication program that can authenticate a face image of the upper limit size LQ is executed as post-processing, an ROI whose size is equal to or larger than the upper limit size LQ only causes an increase in data amount. Low.

  In such a case, as the function f, for example, as shown in FIG. 7, when the ROI size L is less than the upper limit size LQ, the usefulness U increases as the size L increases, and when the ROI size L is greater than or equal to the upper limit size LQ, A usefulness U is constant Q. Also, as the function f, as shown in FIG. 8 or FIG. 9, when the ROI size L is less than the upper limit size LQ, the usefulness U increases as the size L increases, and when the ROI size L is the upper limit size LQ, When the usefulness U is a QM larger than the constant Q, and the ROI size L is larger than the upper limit size LQ, it is possible to adopt a configuration in which the usefulness U decreases as the size L increases. When the function f shown in FIG. 8 is adopted, the usefulness U decreases as the ROI size L is larger than the upper limit size LQ, but the value does not fall below the constant Q. On the other hand, when the function f shown in FIG. 9 is adopted, the usefulness U decreases as the ROI size L is larger than the upper limit size LQ, and when the size L exceeds LQm, the usefulness U becomes a constant Q. Below.

  In addition, although the linear function is employ | adopted in the function f shown in FIG. 7 or FIG. 8, you may employ | adopt the arbitrary nonlinear functions which can achieve the same objective. In short, for example, any function that changes from an increasing function to a decreasing function or a constant function at the upper limit size LQ may be used.

Here, the upper limit size LQ is specified by the required size information D13 as described above.
[Calculation method 2]
As shown below, the usefulness U is a function g of a posture parameter Q that represents the posture of the ROI as a numerical value. The posture parameter Q represents, for example, a deviation angle of the actual ROI in the three axis directions when the front of the ROI (for example, a human face) is set to the minimum angle 0, that is, pitch, yaw or roll.

U = g (Q)
As the ROI attitude parameter Q, for example, one of the following seven types is used.

Q = P
Q = Y
Q = R
Q = min (P, Y, R)
Q = max (P, Y, R)
Q = P × Y × R
Q = P + Y + R
P: Pitch (degrees)
Y: Yaw (degree)
R: Roll (degree)
Here, the function g is expressed by the following mathematical formula, for example.

g (Q) = (− α ₂ ) × Q + β ₂
α ₂ , β ₂ : Predetermined constants As the function g, any nonlinear function may be adopted. For example, a function g having an arbitrary shape can be defined by using a look-up table representing the correspondence between the posture parameter Q and the usefulness U.

At this time, when the image processing system S1 uses a monitoring image captured by the monitoring camera as a processing target, the usefulness is considered to decrease as the pitch, yaw, or roll of the ROI is larger, that is, the deviation from the front is larger. A decreasing function that decreases as the posture parameter Q increases is adopted as the function g.
[Calculation method 3]
The usefulness U is obtained based on the following equation by combining the function f and the function g described above.

U = h [f (L), g (Q)]
As the function h, for example, h [f (L), g (Q)] = α ₃ × f (L) + β ₃ × g (Q)
α ₃ , β ₃ : predetermined constant or
h [f (L), g (Q)] = α ₃ × f (L) × β ₃ × g (Q)
Is adopted.

  Further, as the function h, an arbitrary nonlinear function may be adopted.

  At this time, when the image processing system S1 sets a monitoring image captured by the monitoring camera as a processing target, the larger the f (L) and g (Q), the higher the usefulness. Therefore, the value obtained by f (L) Are the same, a function is adopted in which the usefulness U increases as g (Q) increases. Further, if the value obtained by g (Q) is the same, a function is adopted in which the usefulness U increases as f (L) increases.

  When the calculation of the usefulness U is completed in step P21, the prefetch buffer 203 stores a predetermined amount of ROI secondary information D02, and the ROI determination target information D03A and ROI are stored from the stored ROI secondary information D02. Pre-read information D03B is generated and output (step P22).

  Hereinafter, the generation operation of the ROI determination target information D03A and the ROI prefetch information D03B performed by the prefetch buffer 203 in Step P22 will be described with reference to the operation conceptual diagram shown in FIG.

  The prefetch buffer 203 operates as a queue as, for example, a FIFO (First In First Out) queue. When new data is enqueued, the oldest data is dequeued. The prefetch buffer 203 accumulates the ROI secondary information D02 for data pieces for predetermined pictures (for example, data 1 to 5 for 5 pictures in FIG. 10), and the oldest data piece to be dequeued is subject to ROI determination. Information D03A is generated, and other data pieces in the buffer are generated as ROI prefetch information D03B. At this time, when viewed from the ROI determination target information D03A, the ROI prefetch information D03B corresponds to future information. Note that information of pictures belonging to the same group as the picture corresponding to the ROI determination target information D03A is used as information to be grouped in units of a plurality of pictures and output as ROI prefetch information D03B in advance. You may make it limit to.

  When the ROI determination target information D03A and the ROI prefetch information D03B described above are generated in step P22, the ROI determination unit 104a next selects one ROI identifier to be processed from the ROI determination target information D03A. The selected ROI identifier is set as the ROI identifier X (step P11a).

  Next, the ROI determination unit 104a determines whether or not the ROI indicated by the ROI identifier X is to be encoded with the ROI quality based on the ROI prefetch information D03B and the determination record information D05 (step P12a).

Examples of the determination method performed in step P12a include the following determination methods 1 to 4. [Determination method 1]
When the usefulness of the ROI identifier X included in the ROI prefetch information D03B is higher than the usefulness of the ROI identifier X included in the ROI determination target information D03A, the ROI indicated by the ROI identifier X is encoded with ROI quality. Judge that not. Otherwise, it is determined to encode with ROI quality. When the determination method 1 is adopted, the determination record information accumulation unit 105 and the determination record information accumulation unit 105 illustrated in FIG. 4 are not necessary.
[Judgment method 2]
When the usefulness of the ROI identifier X included in the ROI prefetch information D03B is higher than the usefulness of the ROI identifier X included in the ROI determination target information D03A, the ROI indicated by the ROI identifier X must be encoded with ROI quality. judge. However, even in other cases, based on the determination record information D05, it is determined whether the ROI indicated by the ROI identifier X has been determined to be encoded with ROI quality in the past, and is encoded with ROI quality in the past. A ROI that has been determined is determined not to be encoded with ROI quality. Otherwise, it is determined to encode with ROI quality.
[Judgment method 3]
When the usefulness of the ROI identifier X included in the ROI prefetch information D03B is higher than the usefulness of the ROI identifier X included in the ROI determination target information D03A, the ROI indicated by the ROI identifier X must be encoded with ROI quality. judge. However, even in other cases, based on the determination record information D05, it is determined whether the ROI indicated by the ROI identifier X has been determined to be encoded with ROI quality in the past, and is encoded with ROI quality in the past. If the usefulness of the ROI identifier X that is determined and included in the ROI determination target information D03A is lower than the usefulness that is determined to be encoded in the ROI quality in the past, the ROI is encoded in the ROI quality Judge that not. In other cases, it is determined to encode with ROI quality. When the determination method 3 is adopted, the determination recording unit 105 also accumulates the usefulness at the time of performing the determination as a part of the determination record, and outputs it as determination record information D05.
[Judgment Method 4]
It combines with any of the above-mentioned determination methods 1-3. For example, even if the conditions for determining that encoding is not performed with ROI quality in the determination methods 1 to 3 are satisfied, when the ROI indicated by the ROI identifier X first appears in the picture, or after determining that encoding is performed with ROI quality When the time corresponding to a predetermined number of consecutive pictures has elapsed, it is determined that the encoding is performed with ROI quality. When the determination method 4 is employed, the determination record information storage unit 105 illustrated in FIG. 4 also stores time information indicating how many pictures have elapsed since the ROI determination unit 104a determined to encode with ROI quality. To do.

  When the determination as to whether or not to encode the ROI with the ROI quality is completed in Step P12a, the ROI determination unit 104a executes Step P14 when determining that the ROI indicated by the ROI identifier X is to be encoded with the ROI quality. On the other hand, if it is determined that the ROI indicated by the ROI identifier X is not encoded with the ROI quality, the process proceeds to execution of step P15 (step P13).

  The operations performed in steps P14 to P18 shown in FIG. 6 are the same as those shown in FIG. 3 except that this is executed by the ROI determination unit 104a instead of the ROI determination unit 104. Further, the operation performed in Step P19 shown in FIG. 6 is the same as Step P19 shown in FIG. Therefore, the description of the operations of Steps P14 to P19 shown in FIG. 6 is omitted.

  As described above in detail, in the second embodiment shown in FIG. 4, the bitstream is calculated by calculating the usefulness of the ROI and selecting whether to encode with high quality based on the usefulness. It is possible to improve the accuracy of monitoring and post-processing authentication and recognition processing while suppressing the data capacity of the signal D08.

  FIG. 11 is a block diagram showing the configuration of the third embodiment of the image processing system S1. The image processing system S1 shown in FIG. 11 adopts the ROI detection unit U1b in place of the ROI detection unit U1a shown in FIG. 4 and other configurations except that a code amount monitoring unit 310 is newly provided. This is the same as shown in FIG. The ROI detection unit U1b adopts the prefetch buffer 203a and the RIO determination unit 104b instead of the prefetch buffer 203 and the RIO determination unit 104a, and other configurations except that the mode determination unit 311 is newly added. It is the same as that shown in FIG. Therefore, the image processing system S1 shown in FIG. 11 will be described below with a focus on the difference from the configuration shown in FIG.

  The code amount monitoring unit 310 receives the bit stream signal D08, monitors the actual code amount by detecting the length of the bit stream, and determines the measured code amount D10 indicating the code amount in the mode determination of the ROI detection unit U1b. To the unit 311.

  The mode determination unit 311 receives the actually measured code amount D10 and the set code amount information D14 indicating the code amount for setting the code amount. The mode determination unit 311 determines the normal mode or the suppression mode as the operation mode for operating the ROI detection unit U1b based on the actually measured code amount D10 and the set code amount information D14, and sets the mode information D11 indicating the operation mode to the ROI. It supplies to the determination part 104b.

  The prefetch buffer 203a accumulates the ROI secondary information D02 only for data pieces for predetermined pictures (for example, data 1 to 5 for 5 pictures in FIG. 10), and the oldest data piece to be dequeued is subject to ROI determination. Information D03A is generated and supplied to the ROI determination unit 104b.

  When the mode information D11 indicates the normal mode, the ROI determination unit 104b generates ROI information D04 indicating that all ROIs indicated by the ROI determination target information D03A are encoded with ROI quality, and this is recorded as determination record information. The data is supplied to the storage unit 105 and the image encoding unit U2. On the other hand, when the mode information D11 indicates the suppression mode, the ROI determination unit 104b uses the same method as the ROI determination unit 104a to convert each ROI included in the ROI determination target information D03A to the ROI quality based on the determination record information D05. It is determined whether or not to encode. Here, when it is determined that the encoding is performed with the ROI quality, the ROI determination unit 104b generates information D04 that represents the information that prompts the encoding with the ROI quality in association with the ROI. On the other hand, when it is determined that the encoding is not performed with the ROI quality, the ROI determination unit 104b generates information D04 that represents the information that prompts the encoding with the low quality lower than the ROI quality in association with the ROI. Note that only the configuration (104b, 300, 311) newly added in the third embodiment shown in FIG. 11 is the same as that of the first embodiment shown in FIG. You may make it add to a structure.

  Hereinafter, operations performed by the ROI detection unit U1b and the code amount monitoring unit 310 illustrated in FIG. 11 will be specifically described along the flowcharts illustrated in FIGS. 12 and 13.

  First, the code amount monitoring unit 310 monitors the code amount of the bit stream based on the bit stream signal D08 output from the image encoding unit U2, generates an actually measured code amount D10 indicating the code amount, and generates an ROI detection unit U1b. To the mode determination unit 311 (step P31).

The code amount monitoring unit 310 obtains the actually measured code amount D10 by the following code amount calculation method 1 or 2, for example.
[Code amount calculation method 1]
As shown in the following equation, a value obtained by adding the lengths of bitstreams corresponding to M (M is an integer of 2 or more) consecutive pictures in the bitstream signal D08, or corresponding to M consecutive pictures The code length A is calculated as an average value of the lengths of the bitstreams to be performed or a value obtained by multiplying the average value by a constant using the constant C.

Ｖ_i：Ｍ個のピクチャのうちのｉ番目のピクチャのビットストリームの長さ
Ｃ：定数
すなわち、符号量算出方法１では、部分ピクチャベースで符号量を算出する。
［符号量算出方法２］
以下の数式に示されるように、符号化した全ピクチャ数をＫ（Ｋは２以上の整数）個とした場合に、Ｋ個のピクチャに夫々対応するビットストリームの長さを加算した値、又はＫ個のピクチャに夫々対応するビットストリームの長さの平均値、および平均値を、定数Ｃを使って定数倍した値を符号量Ｂとして算出する。

V _i : Length of the bit stream of the i-th picture among M pictures
C: Constant That is, in the code amount calculation method 1, the code amount is calculated on a partial picture basis.
[Code amount calculation method 2]
As shown in the following equation, when the total number of encoded pictures is K (K is an integer of 2 or more), a value obtained by adding the lengths of bitstreams corresponding to K pictures, or An average value of the lengths of bitstreams corresponding to K pictures and a value obtained by multiplying the average value by a constant using a constant C are calculated as a code amount B.

Ｖ_i：Ｋ個のピクチャのうちのｉ番目のピクチャのビットストリームの長さ
Ｃ：定数
すなわち、符号量算出方法２では、総ピクチャベースで符号量を算出する。

V _i : Length of bit stream of i-th picture among K pictures
C: Constant That is, in the code amount calculation method 2, the code amount is calculated on a total picture basis.

  The code amount monitoring unit 310 supplies the mode determination unit 311 with the actually measured code amount D10 representing one or both of the code amounts A and B described above. Here, how the measured code amount D10 is calculated depends on the mode determination method in the mode determination unit 311 as described later.

  When the calculation of the code amount is completed in step P31, steps P10, P21, and P22 are successively performed. Note that the exemplary operations in steps P10, P21, and P22 are the same as those shown in FIG. 5, and therefore, the description of the operations in these steps P10, P21, and P22 is omitted.

  Here, when step P22 ends, the mode determination unit 311 determines one of the normal mode and the suppression mode as the operation mode by comparing the actually measured code amount D10 and the set code amount information D14. Mode information D11 representing the operation mode is generated (step P32).

The operation mode is determined by the following operation mode determination methods 1 to 3, for example.
[Operation mode determination method 1]
If the actually measured code amount D10 is larger than the set code amount information D14, it is determined as the suppression mode, and otherwise it is determined as the normal mode.
[Operation mode determination method 2]
One of the normal mode and the suppression mode is set as an initial mode, and the measured code amount D10 is larger than the set code amount information D14, or the measured code amount D10 includes zero from the set code amount information D14. When the value is smaller than a certain amount subtracted, the mode shifts to the suppression mode, otherwise the mode is left unchanged.
[Operation mode determination method 3]
One of the normal mode and the suppression mode is set as an initial operation mode, and the code amount A calculated on the partial picture base in the actually measured code amount D10 is set code amount information corresponding to the partial picture base. If it is larger than D14, it shifts to the suppression mode. In addition, the partial picture base code amount A in the actually measured code amount D10 is smaller than a value obtained by subtracting a predetermined amount including zero from the set code amount information D14 corresponding to the partial picture base, and in the actually measured code amount D10. When the total picture base code amount B is smaller than a value obtained by subtracting a predetermined amount including zero from the set code amount information D14 corresponding to the total picture base, the normal mode is entered. Otherwise, leave the mode unchanged.

  For example, the transition to the suppression mode is controlled in conjunction with other mode control mechanisms such as rate control that monitors the bit rate and controls the video quality parameter to keep the bit rate constant. Also good. For example, H.M. In the case of interlocking with the rate control algorithm of an encoder compliant with H.264 / MPEG-4 AVC, the maximum value (value with the smallest code amount) of the quantization parameter defined in the standard is “51”. Therefore, the transition to the suppression mode is controlled by adding the condition for implementing the suppression mode to the rate control algorithm as the quantization parameter “52”. In this case, the normal mode is when the quantization parameter is from “0” to “51”. Note that the mode determination by the mode determination unit 311 may be performed for each predetermined number of pictures instead of for each picture.

  When the generation of the mode information D11 ends in step P32, the ROI determination unit 104b determines whether or not the mode information D11 indicates the suppression mode. If it is determined that the mode information D11 indicates the suppression mode, the process proceeds to the execution of step S12. On the other hand, when it is determined that the suppression mode is not indicated, the process proceeds to execution of the following step P34 (step P33). In addition, since each operation | movement of step P12-P19 implemented after step S12 and it is the same as what is shown in FIG. 3, the operation | movement description in each of these steps P12-P19 is abbreviate | omitted.

  In Step P34, the ROI determination unit 104b generates ROI information D04 that urges all ROIs indicated by the ROI determination target information D03A to be encoded with ROI quality. After execution of step P34, the ROI determination unit 104b proceeds to execution of step P18.

  In the flowcharts shown in FIGS. 12 and 13, step P31 is executed first, but need not always be executed first if it is before step P32.

  As described in detail, in the third embodiment shown in FIG. 11, since the mode is controlled using the set code amount information D14 as an index, normal bit allocation is performed when there is a margin in the code amount ( In the normal mode), the code amount is suppressed (suppression mode) by using the method shown in the first and second embodiments only when there is no margin in the code amount. As a result, it is possible to improve the accuracy of monitoring and post-processing authentication and recognition processing while preventing the code amount from being exceeded, and to transmit and accumulate a lot of ROI information as long as the code amount is sufficient. This is effective in preventing a decrease in monitoring ability.

  In all of the embodiments, the ROI determination unit (104, 104a, 104b) determines whether to encode with ROI quality for each ROI included in the input image signal (D12), and generates ROI information D04. However, the determination that encoding is not performed with ROI quality does not necessarily mean that encoding is performed with bit allocation similar to that of the background region. Therefore, the image encoding unit U2 includes an ROI region determined to be encoded with ROI quality, an ROI region determined not to be encoded with ROI quality, and a background region (region other than the ROI), respectively. Encoding may be performed with different quality, and in this case, a certain effect can be expected.

  In short, an image processing system (S1) that encodes an input image signal and outputs it as a bit stream signal may be any one that includes the following attention area determination unit and image encoding unit. The attention area determination unit (104, 104a, 104b) determines whether or not the attention area attention area designated as a desired area is encoded with a predetermined quality (ROI quality) (P13), and is encoded with a predetermined quality. In this case, information that prompts encoding at a predetermined quality is associated with the attention area, and information that prompts encoding at a lower quality lower than the predetermined quality is associated with the attention area. The attention area information (D04) expressed as follows is generated. The image encoding unit (U2) outputs a bit stream signal (D08) including a signal obtained by encoding the attention area in the input image signal with a predetermined quality or low quality according to the attention area information.

101 ROI tracking unit 104 ROI determination unit 406 Filter unit 407 Compression unit 408 Bit stream transmission unit S1 Image processing system U1 ROI detection unit U2 Image encoding unit

Claims (28)

An image processing system that encodes an input image signal and outputs it as a bit stream signal,
In the image based on the input image signal, it is determined whether or not the region of interest designated as a desired region is to be encoded with a predetermined quality. When it is determined that the information that prompts encoding is not to be encoded with the predetermined quality, attention area information that indicates information that prompts encoding with low quality lower than the predetermined quality is associated with the attention area. An attention area determination unit to be generated;
An image encoding unit that outputs the bitstream signal including a signal obtained by encoding the region of interest in the input image signal with the predetermined quality or the low quality according to the region of interest information. Processing system. An attention area in which an area where a predetermined object is displayed in an image based on the input image signal is detected and tracked as the attention area, and information on the detected attention area is updated by the tracking to generate attention area initial information. Including the tracking part,
The image processing system according to claim 1, wherein the attention area determination unit determines whether or not to encode the attention area indicated by the attention area initial information with the predetermined quality. A determination record information storage unit that accumulates the attention region information generated by the attention region determination unit and supplies the accumulated information as determination record information to the attention region determination unit;
The image processing system according to claim 2, wherein the attention area determination unit determines whether or not to encode the attention area with the predetermined quality based on the determination record information. The attention area tracking unit, every time the undetected attention area is detected from the input image signal, assigns an attention area identifier indicating the detected attention area in association with the attention area,
The attention area determination unit determines whether or not the attention area indicated by the attention area identifier has been determined to be encoded with the predetermined quality in the past by referring to the determination record information. It is determined that the attention area that has been determined to be encoded with the predetermined quality is not encoded with the predetermined quality, and the other attention areas are determined to be encoded with the predetermined quality. The image processing system according to claim 3.   5. The image processing system according to claim 3, wherein the determination record information storage unit limits the number of recorded data of the determination record information to a predetermined finite number of pictures. A synchronization buffer that buffers the input image signal and delays it for a predetermined period as a synchronized image signal, and supplies the synchronized image signal to the image encoding unit instead of the input image signal;
An attention area usefulness assigning unit that calculates usefulness of the attention area for each of the attention areas indicated by the attention area initial information, and generates attention area secondary information in which the usefulness is included in the attention area initial information. When,
The attention area secondary information is buffered for each predetermined picture, and the portion corresponding to the first picture in the buffered attention area secondary information is output as attention area determination target information, and the first picture A prefetching buffer that outputs a portion corresponding to a group of pictures following to the target region prefetching information,
Whether the attention area determination unit encodes each of the attention areas indicated in the attention area determination target information with the predetermined quality by referring to the attention area prefetch information. The image processing system according to claim 1, wherein the image processing system is determined. The usefulness assigning unit, when the usefulness is U and the size of the region of interest is L,
U = f (L)
The image processing system according to claim 6, wherein the usefulness is obtained by a function f (L) represented by:   The image processing system according to claim 7, wherein the function f (L) is an increasing function in which the usefulness increases as the size of the attention area increases. Receiving necessary size information for specifying an upper limit size indicating an upper limit of the size of the attention area that is required when calculating the usefulness of the attention area;
The image processing system according to claim 7, wherein the function f (L) has a maximum value when the size of the attention area is the upper limit size.   The function f (L) is an increasing function in which the usefulness increases according to the size of the attention area when the size of the attention area is smaller than the upper limit size, and the size of the attention area is the upper limit size. The image processing system according to claim 7, wherein the function is a function in which the usefulness is constant when the value is greater than.   The function f (L) is an increasing function in which the usefulness increases according to the size of the attention area when the size of the attention area is smaller than the upper limit size, and the size of the attention area is the upper limit. The image processing system according to claim 7, wherein the image processing system is a decreasing function that reduces the usefulness in accordance with the size of the region of interest when the size is larger than the size. The usefulness assigning unit, when the usefulness level is U, and the posture parameter including the pitch, yaw or roll representing the posture of the attention area is Q,
U = g (Q)
The image processing system according to claim 6, wherein the usefulness is obtained by a function g (Q) represented by: The posture parameter is minimized when the region of interest is facing the front, and increases as the deviation angle from the front increases.
The image processing system according to claim 12, wherein the function g (Q) is a decreasing function that decreases the usefulness as the posture parameter increases. The usefulness grant unit is:
A function f (L) having the size of the region of interest as a variable L and a function g (Q) having a posture parameter including a pitch, yaw or roll representing the posture of the region of interest as a variable Q,
U = h [f (L), g (Q)]
The image processing system according to claim 6, wherein the usefulness is obtained by a function h [f (L), g (Q)] represented by U: usefulness.   If the function h [f (L), g (Q)] has the same value of the function f (L), the larger the value of the function g (Q), the higher the usefulness becomes. 15. The image processing system according to claim 14, wherein if the function g (Q) is the same, the usefulness increases as the value of the function f (L) increases. The prefetch buffer operates as a FIFO (First In First Out) queue, and when the last data piece in the series of data pieces representing the attention area secondary information is enqueued, the first data piece is dequeued. Buffer,
16. The head data piece to be dequeued is output as the attention area determination target information, and the other data pieces are output as the attention area prefetch information. The image processing system described in 1.   The prefetch buffer groups and buffers a sequence of pictures constituting a moving image for each of a plurality of pictures, and information to be output as the attention area prefetch information includes a picture corresponding to the attention area determination target information The image processing system according to claim 16, wherein the image processing system belongs to the same group. The attention area determination unit represents the attention area to be determined as to whether or not to encode with the predetermined quality by an attention area identifier X,
The attention that is included in the attention area pre-read information and indicated by the attention area identifier X is greater than the usefulness of the attention area that is included in the attention area determination target information and indicated by the attention area identifier X. When the usefulness of the area is higher, it is determined that the attention area indicated by the attention area identifier X is not encoded with the predetermined quality. Otherwise, the attention area is encoded with the predetermined quality. The image processing system according to claim 6, wherein the determination is made. The attention area determination unit represents the attention area to be determined as to whether or not to encode with the predetermined quality by an attention area identifier X,
The attention area that is included in the attention area look-ahead information and indicated by the attention area identifier X than the usefulness of the attention area that is included in the attention area determination target information and indicated by the attention area identifier X If the usefulness is higher, it is determined that the attention area indicated by the attention area identifier X is not encoded with the predetermined quality, and in other cases, based on the determination recording information. It is determined whether or not the region of interest indicated by the region of interest identifier X has been determined to be encoded with the predetermined quality in the past, and for the region of interest that has been determined to be encoded with the predetermined quality in the past The image processing system according to any one of claims 6 to 17, wherein it is determined that encoding is not performed with the predetermined quality, and encoding is performed with the predetermined quality in other cases. . The attention area determination unit represents the attention area to be determined as to whether or not to encode with the predetermined quality by an attention area identifier X,
The attention area that is included in the attention area look-ahead information and indicated by the attention area identifier X than the usefulness of the attention area that is included in the attention area determination target information and indicated by the attention area identifier X If the usefulness is higher, it is determined that the attention area indicated by the attention area identifier X is not encoded with the predetermined quality, and in other cases, based on the determination recording information. It is determined whether the attention area indicated by the attention area identifier X has been determined to be encoded with the predetermined quality in the past, and when it is determined that it has been determined to encode with the predetermined quality in the past, When the usefulness of the attention area identifier X included in the attention area determination target information is lower than the usefulness when the target area identifier X was encoded with the predetermined quality in the past, Is determined not to encode quality, it determines that encoded in the given quality otherwise,
18. The determination record information accumulating unit correspondingly accumulates the usefulness when the determination is performed as a part of the determination record information, and outputs it as the determination record information. The image processing system according to any one of the above. Even when the attention area determination unit determines that the attention area indicated by the attention area identifier X is not encoded with the predetermined quality, when the attention area indicated by the attention area identifier X first appears as a picture, Alternatively, after a time corresponding to a predetermined number of pictures has elapsed from the time when it is determined to encode with the predetermined quality, it is determined that the encoding with the predetermined quality is performed,
The image processing system according to any one of claims 18 to 20, wherein the determination record information storage unit also stores time information indicating an elapsed time from a time point when it is determined that the encoding is performed with the predetermined quality. A code amount monitoring unit that monitors a code amount based on a bit stream length in the bit stream signal and obtains the monitored code amount as an actually measured code amount;
Receives the set code amount information indicating the code amount at the time of encoding as the set code amount, and determines one of the normal mode and the suppression mode based on the magnitude comparison result between the set code amount and the actually measured code amount A mode determination unit that performs,
When the operation mode indicates the normal mode, the attention area determination unit generates the attention area information that prompts encoding with low quality lower than the predetermined quality, while the operation mode indicates the suppression mode. The image processing according to any one of claims 1 to 21, wherein the attention area information that urges encoding at the predetermined quality or the low quality is generated based on a result of the determination. system.   The image according to claim 22, wherein the mode determination unit determines the suppression mode if the measured code amount is larger than the set code amount, and determines the normal mode otherwise. Processing system.   The mode determination unit sets one of the normal mode and the suppression mode as an initial state of the operation mode, and the measured code amount is larger than the set code amount, or the measured code amount is the set code amount. 23. The operation mode is shifted to the suppression mode if the value is smaller than a value obtained by subtracting a predetermined amount including zero, and otherwise the current state of the operation mode is deferred. Image processing system. The code amount corresponding to the bit stream length corresponding to a predetermined number of pictures is a partial picture base code amount, and the code amount corresponding to the bit stream length corresponding to each picture of all the encoded pictures is the total picture Base code amount,
The mode determination unit shifts to the suppression mode if the partial picture base code amount of the measured code amount is larger than a code amount corresponding to the partial picture base code amount of the set code amount. And
The partial picture base code amount of the actually measured code amount is smaller than a value obtained by subtracting a predetermined amount including zero from a code amount corresponding to the partial picture base code amount of the set code amount, and the If the total picture base code amount of the actually measured code amount is smaller than the value obtained by subtracting the predetermined amount from the code amount corresponding to the total picture base code amount of the set code amount, the normal Switch to mode,
23. The image processing system according to claim 22, wherein in other cases, the current state of the operation mode is deferred.   The image processing system according to claim 22, wherein the mode determination unit cooperates with a mode control mechanism that controls a video quality parameter, and sets the suppression mode as one mode of the mode control mechanism.   27. The image processing system according to claim 22, wherein the mode determination unit performs the determination for each predetermined number of pictures.   The image encoding unit, the attention region determined by the attention region determination unit to be encoded with the predetermined quality, the attention region determined not to be encoded with the predetermined quality, and regions other than the attention region, The image processing system according to any one of claims 1 to 27, wherein the images are encoded with different qualities.

Images