JP2019012477A - Diagnostic program, diagnostic method, and diagnostic apparatus - Google Patents

Abstract

[PROBLEMS] To determine whether the performance degradation of an application is caused by the cloud platform or the application. A delay determination unit (41) acquires delay degree information from a delay information creation device (2), determines whether a delay has occurred in an application, and a first correlation analysis unit (42) and a second correlation analysis unit (44) Analyzes the correlation between resource data related to the app where the problem occurred and the degree of delay. Then, the delay cause diagnosis unit 46 identifies the cause of the delay based on the correlation analysis results of the first correlation analysis unit 42 and the second correlation analysis unit 44. [Selection] Figure 2

Description

  The present invention relates to a diagnostic program, a diagnostic method, and a diagnostic apparatus.

  In a system that provides a resource to a customer, it is important to monitor the status of the provided resource and check whether there is a problem with the resource. For example, in a cloud system that provides a virtual machine, it is important to monitor the response time and load of an application operating on the virtual machine to check whether there is a problem in the performance of the application.

  Here, the virtual machine is a virtual computer that operates on a physical machine (computer). A cloud system is a system that provides computer hardware and software to a user via a network.

  Agents are used to collect data about application performance. FIG. 38 is a diagram for explaining monitoring by an agent. As shown in FIG. 38, a virtual machine 9a operates on the physical machine 9, and an application and an agent are executed by the virtual machine 9a. Here, the application is an application. The agent collects performance data from the application and monitors the performance of the application.

  In addition to detecting performance degradation of resources allocated to multiple instances that are logical server configurations, extract instances that share resources that exhibit performance degradation, and compare performance trends and performance patterns of the extracted instances There is technology. Here, the performance pattern indicates a feature amount extracted from the resource performance information in association with the bottleneck factor. According to this technology, the bottleneck factor can be estimated from the performance pattern that maximizes the similarity with the performance tendency of the instance.

  Also, a plurality of pieces of information related to each of a plurality of resources providing services in cooperation are acquired from different management devices, and an abnormality is generated based on a difference between a correlation between the acquired pieces of information and a predetermined correlation. There is a technology for identifying the cause of anomaly resources. In this technology, a resource in which a difference between a correlation between a plurality of pieces of information and a predetermined correlation is larger than an allowable value is extracted as an abnormal resource candidate, and based on configuration information indicating a plurality of resource configurations in the service, Identify the cause-of-failure resource from among the candidate abnormal resources. According to this technique, it is possible to appropriately specify the abnormality cause resource.

International Publication No. 2015/145664 JP 2013-161305 A

  In the performance monitoring shown in FIG. 38, the performance of each application can be monitored, but there is a problem that it is impossible to specify whether the performance degradation of the application is caused by the cloud infrastructure or the application. . Here, the cloud platform is a platform that provides information and communication technology (ICT) infrastructure such as servers, networks, and storages using virtualization technology. The cloud platform has functions such as virtual machine management, storage management, and network management.

  In one aspect, the present invention has an object of identifying whether a performance degradation of an application is caused by a cloud platform or an application.

  In one aspect, the diagnostic program causes the computer to execute a process of acquiring a delay degree indicating an execution delay for each application and determining whether the delay degree is equal to or greater than a predetermined threshold. Then, when the diagnosis program determines that the delay level is equal to or greater than a predetermined threshold, the diagnosis program executes a process of identifying the cause of the delay based on the correlation between the information related to the resource related to the application and the delay level. Let

  In one aspect, the present invention can identify whether application performance degradation is caused by the cloud infrastructure or the application.

FIG. 1 is a diagram for explaining delay cause diagnosis by the diagnosis apparatus according to the first embodiment. FIG. 2 is a diagram illustrating the configuration of the cloud system according to the first embodiment. FIG. 3 is a diagram illustrating the configuration of the delay information creation apparatus. FIG. 4 is a diagram illustrating an example of the type determination data storage unit. FIG. 5 is a diagram illustrating an example of the type information storage unit. FIG. 6 is a diagram for explaining a method of calculating the response time. FIG. 7 is a diagram illustrating an example of a response time information storage unit. FIG. 8 is a diagram for explaining normalization of the average response time. FIG. 9 is a diagram illustrating an example of the representative information storage unit. FIG. 10 is a diagram for explaining the ex-Gaussian distribution. FIG. 11 is a diagram for explaining outliers. FIG. 12 is a diagram illustrating an example of the delay degree information storage unit. FIG. 13 is a diagram for explaining the delay determination of the application. FIG. 14 is a diagram illustrating an example of the VM information storage unit. FIG. 15 is a diagram illustrating an example of the resource data storage unit. FIG. 16 is a diagram illustrating an example of a relationship between resource data and a delay degree. FIG. 17 is a diagram for explaining the correlation analysis based on whether or not there is a difference in the tendency of the resource data between when it is delayed and when it is not delayed. FIG. 18A is a diagram illustrating a case where the correlation is weak and a case where the correlation is strong. FIG. 18B is a diagram for describing extraction of resource data that is well correlated with the degree of delay. FIG. 19 is a diagram illustrating an example of the first analysis result storage unit. FIG. 20 is a diagram illustrating an example of the configuration information storage unit. FIG. 21 is a diagram for explaining correlation analysis by the second correlation analysis unit. FIG. 22 is a diagram illustrating an example of the second analysis result storage unit. FIG. 23 is a diagram illustrating an example of a diagnosis result storage unit. FIG. 24 is a diagram illustrating an example of notification to the cloud operation manager. FIG. 25 is a flowchart showing a flow of processing by the diagnostic apparatus. FIG. 26 is a flowchart showing the flow of the first correlation analysis process. FIG. 27 is a flowchart showing a flow of resource data extraction processing. FIG. 28 is a flowchart showing the flow of the second correlation analysis process. FIG. 29 is a flowchart showing a flow of delay cause diagnosis processing. FIG. 30 is a flowchart showing the flow of the first correlation analysis process using the two-group comparison test. FIG. 31 is a flowchart showing the flow of the second correlation analysis process using the two-group comparison test. FIG. 32 is a diagram for explaining the removal of data with an unstable delay degree. FIG. 33 is a diagram illustrating a configuration of a cloud system according to the second embodiment. FIG. 34 is a diagram for explaining a method of removing data with an unstable delay degree. FIG. 35 is a flowchart showing a flow of processing by the diagnostic apparatus. FIG. 36 is a flowchart showing a flow of threshold value calculation processing. FIG. 37 is a diagram illustrating a hardware configuration of a computer that executes the diagnostic program according to the first and second embodiments. FIG. 38 is a diagram for explaining monitoring by an agent.

  Hereinafter, embodiments of a diagnostic program, a diagnostic method, and a diagnostic device disclosed in the present application will be described in detail with reference to the drawings. The embodiments do not limit the disclosed technology.

  First, delay cause diagnosis by the diagnostic apparatus according to the first embodiment will be described. FIG. 1 is a diagram for explaining delay cause diagnosis by the diagnosis apparatus according to the first embodiment. As illustrated in FIG. 1, the diagnostic apparatus according to the first embodiment performs correlation analysis between resource data of VM (Virtual Machine) related to an application and a delay degree (t1).

  Here, the VM related to the application is a VM used by the application, and there may be a plurality of VMs. For example, in a three-tier system composed of a Web server, an application server, and a DB (Database) server, there are three VMs related to one application. The degree of delay is the degree of delay of the application.

  The resource data includes two types of data: data representing a resource load amount for each VM resource and data affected by resource performance degradation. For example, when the resource is a CPU (Central Processing Unit), the data indicating the resource load amount has a CPU usage rate, and the data affected by the resource performance degradation has a CPU allocation waiting time ratio. FIG. 1 shows a case where the resource is a CPU.

  If the resource is a disk device, the data indicating the resource load amount includes the number of read issues, the amount of read, the number of write issues, the amount of write, etc. There is an average write time. When the resource is a network, the data indicating the resource load amount includes the number of packets, the communication amount, and the like, and the data affected by the resource performance degradation includes the round trip time, the packet drop rate, and the like.

  The diagnosis apparatus according to the first embodiment performs, for each VM resource related to an application, a correlation analysis between data representing a resource load amount and a delay degree, and a correlation analysis between data affected by resource performance degradation and a delay degree. .

  When the degree of delay is not correlated with the resource data of the CPU of a certain VM, the diagnosis apparatus according to the first embodiment diagnoses that the delay is not related to the CPU of this VM (t2). When the degree of delay has a correlation with the CPU usage rate, the diagnosis apparatus according to the first embodiment diagnoses a delay due to an application factor (t3).

  When the degree of delay correlates with the CPU allocation waiting time ratio, the diagnosis apparatus according to the first embodiment shares the VM and the CPU whose delay degree correlates with the CPU allocation waiting time ratio. Correlation analysis between the CPU usage rate and the delay degree is performed for all the VMs that are present (t4). Here, all VMs sharing a CPU with a VM whose delay degree is correlated with the CPU allocation waiting time ratio include VMs whose delay degree is correlated with the CPU allocation waiting time ratio. .

  When the degree of delay correlates with the CPU usage rate of other VMs, the CPU load increases due to the use of a large amount of CPUs by other VMs, and a delay occurs. The apparatus diagnoses a delay due to a CPU factor, that is, a cloud infrastructure factor (t5). Further, when the delay degree has no correlation with the CPU usage rate of any VM, the diagnosis device according to the first embodiment diagnoses that the cause of the CPU load increase is unknown (t6). In addition, if the degree of delay correlates with the CPU usage rate of the VM itself that is affected by the CPU load, the VM of the delayed app itself increases the CPU load, causing a delay. The diagnostic device according to Example 1 diagnoses a delay due to an application factor (t3).

  As described above, the diagnosis apparatus according to the first embodiment analyzes the correlation between the data indicating the resource load amount and the delay degree for each VM resource related to the application, and the data and the delay degree affected by the resource performance degradation. Perform correlation analysis. When there is a correlation between the delay degree and the data affected by the resource performance decrease, the diagnosis apparatus according to the first embodiment uses the VM and the resource whose delay degree is correlated with the data affected by the resource performance decrease. The correlation analysis with the data representing the resource load amount is performed for all VMs sharing the. Then, the diagnosis apparatus according to the first embodiment diagnoses the cause of the delay of the application based on the result of the correlation analysis. Therefore, the diagnosis apparatus according to the first embodiment can identify whether the performance degradation of the application is caused by the cloud platform or the application.

  Next, the configuration of the cloud system according to the first embodiment will be described. FIG. 2 is a diagram illustrating the configuration of the cloud system according to the first embodiment. As illustrated in FIG. 2, the cloud system 1 according to the first embodiment includes a delay information creation device 2, a diagnosis device 4, a VM information storage unit 31, a resource data storage unit 32, a configuration information storage unit 33, Resource 34. The resource 34 includes a CPU, a disk device, and a network switch.

  The delay information creation device 2 creates delay degree information of the application, and stores the created delay degree information in the delay degree information storage unit 30. FIG. 3 is a diagram illustrating a configuration of the delay information creation device 2. As shown in FIG. 3, the delay information creation device 2 includes a capture unit 21, a packet information storage unit 22, a type determination data storage unit 23, a type determination unit 24, a type information storage unit 25, and a response time. A calculation unit 26 and a response time information storage unit 27 are included. The delay information creation device 2 includes a normalization unit 28, a representative information storage unit 29, and a delay degree information storage unit 30.

  The capture unit 21 captures a communication packet passing through the network switch 3 c by port mirroring, and stores the captured communication packet information in the packet information storage unit 22. The packet information storage unit 22 stores information on communication packets that pass through the network switch 3c.

  The type determination data storage unit 23 stores data for determining the type of application. There are two types of apps: an app whose response time is important in terms of performance and another app. The delay information creation device 2 sets an application whose response time is important in terms of performance as a target for creating the delay degree information.

  The type determination unit 24 uses the data stored in the type determination data storage unit 23 to determine the type of application for each communication connection. FIG. 4 is a diagram illustrating an example of the type determination data storage unit 23. FIG. 4A shows a case where a port list that is a list of port numbers is stored as data for determining the type of application. In FIG. 4A, the port number stored in the type determination data storage unit 23 is a port number used by an application whose response time is important in terms of performance. For example, the type determination data storage unit 23 stores “80”, “443”, and the like as port numbers used by applications whose response time is important in terms of performance.

  The type determination unit 24 analyzes the information of the communication packet stored in the packet information storage unit 22 and extracts the server-side port number. Here, the server is a virtual machine. When the extracted port number is included in the port list stored in the type determination data storage unit 23, the type determination unit 24 determines that the response time is important in terms of performance for an application to be transmitted or received in the analyzed communication packet. It is determined as an app. Then, the type determination unit 24 stores the determination result in the type information storage unit 25.

  Further, the type determination unit 24 determines the type of application by machine learning with an input of a communication pattern for an application for which the type of application cannot be determined from the port number.

  Specifically, the type determination unit 24 collects communication packets in advance for apps and other apps whose response time is important in terms of performance. Then, the type determination unit 24 analyzes the collected communication packets, average response time at a fixed time interval (for example, 1 minute), server average communication amount, server average communication count, client device average communication amount, and client Calculate the average number of communications of the device.

  Then, the type determination unit 24 constructs a learning device using the calculated value as learning data. As the learning device, SVM (Support Vector Machine), random forest, or the like can be used. FIG. 4B illustrates a case where the type determination data storage unit 23 stores learning data as data for determining the type of application. As shown in FIG. 4B, the type determination data storage unit 23 includes an application type, an average response time, a server average communication amount, a server average communication count, a client device average communication amount, and a client device's average communication amount. The average number of communications is stored as one learning data. The unit of the average response time is microseconds, and the unit of the average communication amount of the server and the average communication amount of the client device is byte.

  In FIG. 4B, two types of learning data are shown for the application type “application whose response time is important in terms of performance”, and one piece of learning data is shown for the application type “other apps”. Yes. In one of the learning data of the application type “application whose response time is important in terms of performance”, the average response time is “600”, the average traffic of the server is “100”, and the average communication of the server The number of times is “1”. In the learning data, the average communication amount of the client device is “100”, and the average communication count of the client device is “1”.

  Then, the type determination unit 24 determines the average response time of the same time interval as the learning data for each communication connection from the captured communication packet, the average communication amount of the server, the average number of communication times of the server, the average communication amount of the client device, and the client Calculate the average number of communications of the device. Then, the type determination unit 24 determines the type of the application corresponding to the communication connection using the learning device from the calculated value. Then, the type determination unit 24 stores the determination result in the type information storage unit 25.

  The type information storage unit 25 stores the determination result of the application type. FIG. 5 is a diagram illustrating an example of the type information storage unit 25. As illustrated in FIG. 5, the type information storage unit 25 stores an IP address, a port number, and a type for each application. The IP address is the IP address of the VM on which the application is operating. The port number is a port number used by the application. The type is the type of application. For example, an application running on a VM whose IP address is “10.20.30.40” has a port number of “80” and “an application whose response time is important in terms of performance”.

  The response time calculation unit 26 analyzes a communication packet for an application whose response time is important in terms of performance, calculates a response time, and stores the calculated response time in the response time information storage unit 27. When the communication packet is not encrypted, the response time calculation unit 26 reconstructs the protocol message and calculates the response time from the request time and the response time.

  That is, the response time calculation unit 26 reconstructs the protocol message from the communication packet, and determines which communication packet is the request message and which communication packet is the response message. Then, the response time calculation unit 26 calculates the time from when the request message is transmitted until the response message is returned as the response time.

  FIG. 6 is a diagram for explaining a method of calculating the response time. As illustrated in FIG. 6, the request message transmitted from the client device is processed by the application operating in the VM in the cloud system 1, and the response message is transmitted from the application to the client device. The response time calculation unit 26 sets the time between the time when the request message is captured and the time when the response message is captured as the response time.

  When the communication packet is encrypted, the response time calculation unit 26 estimates the application response time by analyzing the flow of transmission and reception of the communication packet. When the communication packet is encrypted, the response time calculation unit 26 cannot analyze the protocol because the inside of the communication packet is not seen, and therefore cannot reconstruct the request message or the response message. Therefore, the response time calculation unit 26 considers “long polling”, which is a technique for unilaterally transmitting data in real time from the application side, and determines from the communication packet time interval between the client device and the application on the cloud system 1. Estimate response time.

  The response time information storage unit 27 stores the response time calculated for each application by the response time calculation unit 26. FIG. 7 is a diagram illustrating an example of the response time information storage unit 27. As shown in FIG. 7, the response time information storage unit 27 stores time, an IP address, a port number, and a response time in association with each other.

  The time is the time when the response time is calculated. The IP address is the IP address of the VM on which the application is operating. The port number is a port number used by the application. The response time is a response time calculated by the response time calculation unit 26. The unit of response time is microseconds. For example, the response time of “2016/06/24 09:00:00” of an application that operates on a VM whose IP address is “10.20.30.40” and whose port number to be used is “80” is: “600”.

  The normalization unit 28 reads the response time calculated by the response time calculation unit 26 from the response time information storage unit 27, and calculates an average response time for each application. Then, the normalization unit 28 normalizes the average response time using the information stored in the representative information storage unit 29, and stores the normalized average response time in the delay degree information storage unit 30 as delay degree information. .

  FIG. 8 is a diagram for explaining normalization of the average response time. The response time varies depending on the application, the value that can be taken at normal times, and the standard for considering it as a delay. For this reason, it is difficult to determine whether there is a delay in the application if the response time of the application is used as it is. Therefore, the delay information creation device 2 normalizes the average response time and converts it into a scale that can be compared between applications. In FIG. 8, the response times can be compared by normalizing the average response time for the apps # 1 and # 2 having different values that can be taken during normal operation.

Normalizing unit 28, as a representative response time basic statistics t _r of the mean response time t, the normalized mean response time t _n is calculated by t _n = t / t _r. Basic statistics include the mean, median, mode, etc.

  The representative information storage unit 29 stores a representative response time for each application. FIG. 9 is a diagram illustrating an example of the representative information storage unit 29. As shown in FIG. 9, the representative information storage unit 29 stores the time, IP address, port number, and response time for each application.

  The time is the time when the representative response time is calculated. The IP address is the IP address of the VM on which the application is operating. The port number is a port number used by the application. The response time is a representative response time. The unit of typical response time is microseconds. For example, a representative that is calculated as “2016/06/23 00:00:00” of an application that operates on a VM with an IP address “10.20.30.40” and a port number to be used “80”. The response time is “600”.

Normalizing unit 28 calculates the average response time t for each predetermined time interval (e.g., 1 minute) for each application, to calculate the basic statistics t _r of average response time t. As data for calculating the basic statistic, data for the previous day is used. When the fixed time interval is 1 minute, 60 * 24 = 1440 sample data is obtained from the data for one day.

Further, the normalization unit 28, instead of basic statistics t _r, may be used as a representative response time μ when the distribution of the average response time was fitted by ex-Gaussian distribution. FIG. 10 is a diagram for explaining the ex-Gaussian distribution. The ex-Gaussian distribution is one of probability distributions, and is a convolution integral of a Gaussian distribution (normal distribution) and an exponential distribution, as shown in FIG. The ex-Gaussian distribution is determined by three parameters: an average μ and standard deviation σ of a normal distribution, and an average and standard deviation τ of an exponential distribution. In the ex-Gaussian distribution, μ is a value of a peak portion of the distribution as shown in FIG.

  The normalization unit 28 calculates an average response time t for each fixed time interval (for example, 1 minute) for each application, and fits the average response time distribution with an ex-Gaussian distribution. As data for fitting to the ex-Gaussian distribution, the data for the previous day is used. When the fixed time interval is 1 minute, 60 * 24 = 1440 sample data is obtained from the data for one day.

  Then, the normalizing unit 28 determines the probability of fitting by a one-sample Kolmogorov-Smirnov test. There are two inputs for the one-sample Kolmogorov-Smirnov test: the distribution of the average response time and the distribution curve of the fitting result. For example, if the average response time distribution is an ex-Gaussian distribution, the normalization unit 28 sets the significance level μ to the representative average response time.

  Note that the normalization unit 28 may perform outlier removal before fitting the average response time distribution with the ex-Gaussian distribution. FIG. 11 is a diagram for explaining outliers. As shown in FIG. 11, the outlier is a value greatly deviating from other values. If there is an outlier in the average response time, the ex-Gaussian distribution may not fit well. For this reason, the normalization unit 28 performs outlier removal before fitting. The outlier removal method includes Tukey outlier removal.

  The delay degree information storage unit 30 stores the average response time normalized by the normalization unit 28 for each application as delay degree information. FIG. 12 is a diagram illustrating an example of the delay degree information storage unit 30. As illustrated in FIG. 12, the delay degree information storage unit 30 stores time, an IP address, a port number, a normalized average response time, and the number of requests for each application.

  The time is the time when the response time is calculated. The IP address is the IP address of the VM on which the application is operating. The port number is a port number used by the application. The normalized average response time is the normalized average response time. The number of requests is the number of requests used to calculate the normalized average response time.

  For example, the response time calculated as “2016/06/24 09:00:00” for an application that operates on a VM with an IP address of “10.20.30.40” and a port number to be used is “80” The average response time normalized with respect to is 1.0. The number of requests used to calculate the normalized average response time is “2”.

  Returning to FIG. 2, the diagnostic device 4 includes a delay determination unit 41, a first correlation analysis unit 42, a first analysis result storage unit 43, a second correlation analysis unit 44, and a second analysis result storage unit 45. The delay cause diagnosis unit 46 and the diagnosis result storage unit 47 are included.

The delay determination unit 41 acquires the normalized response time from the delay information creation device 2 as delay degree information, and determines whether or not a delay has occurred in the application. FIG. 13 is a diagram for explaining the delay determination of the application. As illustrated in FIG. 13, the delay determination unit 41 periodically counts the number of times that the delay degree is equal to or greater than the threshold T _rt in the data of the diagnosis target period for each application, and the number of times is equal to or greater than the threshold T _dr. It is determined that the application is delayed.

The interval for executing the determination is, for example, one day. Alternatively, the delay determination unit 41 may perform the delay determination based on an instruction from the cloud operation manager. In FIG. 13, assuming that T _rt = 10 and T _dr = 5, the number of times that the delay degree of the application # 2 is _{equal to} or greater than T _rt is “5”, so it is determined that a delay has occurred in the application # 2.

  The first correlation analysis unit 42 performs a correlation analysis between the resource data of the VM related to the application and the degree of delay for each application determined to have a delay by the delay determination unit 41, thereby relating to the delay. Filter resource data to be used. The first correlation analysis unit 42 refers to the VM information storage unit 31 and identifies a VM related to the application.

  FIG. 14 is a diagram illustrating an example of the VM information storage unit 31. As illustrated in FIG. 14, the VM information storage unit 31 stores the change date and time, the VM IP address, the port number, and the related VM list in association with each other for each application. The change date is the date when the corresponding information was changed. The VM IP address is the IP address of the VM on which the application operates. The port number is a port number used by the application. One application is specified by a combination of the VM IP address and the port number.

  The related VM list is a list of VMs related to the application specified by the IP address and port number of the VM. For example, the VMs related to the application specified by the IP address “10.20.30.40” and the port number “80” are “VM # 1”, “VM # 2”, and “VM # 3”.

  The first correlation analysis unit 42 acquires VM resource data from the resource data storage unit 32. FIG. 15 is a diagram illustrating an example of the resource data storage unit 32. As illustrated in FIG. 15, the resource data storage unit 32 stores time, VM name, resource data name, and value in association with each other for each resource data.

  The time is the time when the value was acquired. The VM name indicates to which VM the resource data belongs. The resource data name is a name for identifying resource data. The value is a value of resource data. For example, in “2016/06/24 09:00:00”, the “CPU usage rate” of “VM # 1” is “10.0”%.

  The resource data storage unit 32 stores data acquired from the resource 34. The first correlation analysis unit 42 performs a correlation analysis with the resource data of all VMs used by the user of the application when the information on the VM related to the application cannot be obtained.

  The first correlation analysis unit 42 performs correlation analysis using the correlation coefficient. That is, the first correlation analysis unit 42 calculates a correlation coefficient between the delay degree and the resource data, and tests whether there is a positive correlation between the delay degree and the resource data. FIG. 16 is a diagram illustrating an example of a relationship between resource data and a delay degree. In FIG. 16, a set of resource data value and delay value at one time is represented by a point. The first correlation analysis unit 42 calculates a correlation coefficient using the values of a plurality of points shown in FIG.

  The first correlation analysis unit 42 uses, for example, Pearson's correlation coefficient and Spearman's rank correlation coefficient as correlation coefficients. Then, the first correlation analysis unit 42 calculates the p-value of the decorrelation test.

The first correlation analysis unit 42 performs a test of the number of VM resource data related to the application. As described above, when a plurality of tests are performed, the number of cases in which there is a correlation even though there is originally no correlation increases. For example, when the significance level is 0.05, the probability that the correlation is erroneously determined is 5% even though there is no correlation in one test. If this test is repeated three times, the chance of mistakes even once increases to 1- (1-0.05) ³ = 14%.

  Therefore, the first correlation analysis unit 42 corrects the calculated p-value by multiple test correction in order to reduce erroneous determinations. As a correction method, for example, there is a Benjamini-Hochberg method. The first correlation analysis unit 42 determines that the resource data whose corrected p-value is equal to or less than the threshold has a correlation with the delay degree. For example, 0.05 is used as the threshold value.

  The first correlation analysis unit 42 determines whether there is a difference in the tendency of the resource data when it is delayed and when it is not delayed, instead of the correlation analysis using the correlation coefficient. An analysis may be performed. When the correlation coefficient is used, all data are handled equivalently. For this reason, whether or not there is a correlation may be determined by the larger number of data, that is, the data that is not delayed. As a result, there may be a case where a large value of the resource data is missed when there is a delay. Thus, by separately handling resource data at the time of delay, the first correlation analysis unit 42 can determine whether or not there is a correlation while suppressing the influence of the number of data.

FIG. 17 is a diagram for explaining the correlation analysis based on whether or not there is a difference in the tendency of the resource data between when it is delayed and when it is not delayed. As illustrated in FIG. 17, the first correlation analysis unit 42 divides resource data depending on whether the delay is greater than or equal to a threshold T _{rt, depending on} whether it is delayed or not. Then, the first correlation analysis unit 42 calculates a p-value of a test (two-group comparison test) as to whether the resource data when delayed is larger than when the resource data is not delayed. Test methods include t-test, Wilcoxon rank sum test, Mann-Whitney U test, and the like.

  Then, the first correlation analysis unit 42 corrects the calculated p-value by multiple test correction. As a correction method, for example, there is a Benjamini-Hochberg method. The first correlation analysis unit 42 determines that the resource data whose corrected p-value is equal to or less than the threshold has a correlation with the delay degree. For example, 0.05 is used as the threshold value.

  Even if the correlation analysis tests that there is a correlation, the strength of the correlation varies from data to data. FIG. 18A is a diagram illustrating a case where the correlation is weak and a case where the correlation is strong. As shown in FIG. 18A, resource data # 1 has a weak correlation with the delay degree, but resource data # 2 has a strong correlation with the delay degree. Therefore, the first correlation analysis unit 42 extracts only resource data that is well correlated with the degree of delay from the correlated resource data in order to reduce the resource candidates that cause the delay of the application.

  FIG. 18B is a diagram for describing extraction of resource data that is well correlated with the degree of delay. As illustrated in FIG. 18B, the first correlation analysis unit 42 uses the data at the time of delay and estimates the value of the resource data at the time of delay. Then, the first correlation analysis unit 42 compares the distribution of the degree of delay when the resource data is larger than the estimated value for the two resource data, and identifies resource data that is often delayed when the resource data is large. In FIG. 18B, resource data # 2 is extracted as resource data that is well correlated with the degree of delay.

Specifically, the first correlation analysis unit 42 executes the following procedures (1) and (2) for each resource data correlated with the delay degree.
(1) The first correlation analysis unit 42 calculates the median value of the resource data having a delay degree equal to or greater than the threshold value T _{rt and} sets it as T _r (uses the data hatched with left diagonal lines).
(2) Then, the first correlation analysis unit 42 extracts data on the degree of delay when the resource data is _{equal to} or greater than T _r (extracts data hatched with right diagonal lines).

  Then, the first correlation analysis unit 42 tests the extracted delay degree data by the Steel Dwass method, and specifies the resource data having the highest number of times when there is a difference in the distribution of the delay degree between the two resource data. . For example, 0.05 is used as the significance level of the test. Then, the first correlation analysis unit 42 sets the resource data having the largest number of times that there is a difference and the resource data that has no difference between the resource data as resource data having a good correlation with the delay degree.

  The first correlation analysis unit 42 stores the correlation analysis result in the first analysis result storage unit 43. FIG. 19 is a diagram illustrating an example of the first analysis result storage unit 43. As illustrated in FIG. 19, the first analysis result storage unit 43 stores diagnosis date / time, start time, end time, VM IP address, port number, VM name, resource type, and resource data name in association with each other.

  The diagnosis date and time is the date and time when diagnosis was performed. The start time is the start time of the diagnosis target period. The end time is the end time of the diagnosis target period. The VM IP address is the IP address of the VM on which the application operates. The port number is a port number used by the application. The VM name indicates to which VM the resource data belongs. The resource type is a type of the resource 34 related to delay, and includes “CPU”, “disk device”, “network”, and the like. The resource data name is a name for identifying resource data correlated with the delay degree.

  For example, in a diagnosis performed from “2016/06/24 00:00:00” to “2016/06/24 00:00:00” as a diagnosis target period at “2016/06/24 00:00:00” There was resource data correlated with the degree of delay of the app. The application is specified by the IP address “10.20.30.40” of the VM and the port number “80”, and between the “ratio of CPU allocation waiting time” of “VM # 3” and the degree of delay of the application. There was a correlation.

  When there is a correlation between the data affected by the resource performance degradation and the delay degree, the second correlation analysis unit 44 shares all the resources 34 and the VMs that are correlated with the data affected by the resource performance degradation. A correlation analysis is performed between the data indicating the resource load amount and the delay degree for the VM.

  The second correlation analysis unit 44 performs correlation analysis with reference to the resource data storage unit 32 and the configuration information storage unit 33 that stores configuration information of the cloud system 1. FIG. 20 is a diagram illustrating an example of the configuration information storage unit 33. As illustrated in FIG. 20, the configuration information storage unit 33 stores the change date, physical machine, user, VM name, VM IP address, and disk name in association with each VM.

  The change date is the date when the corresponding information was changed. A physical machine is a physical machine on which a VM operates. The user is a user who uses the VM. The VM name is a name for identifying the VM. The VM IP address is an IP address assigned to the VM. The used disk name is a name of a disk device used by the VM. For example, “VM # 1” operates on “PM # 1” and uses “disk # 1”, the user of “VM # 1” is “user # 1”, and the IP address of “VM # 1” Is “10.20.30.40”. The information of “VM # 1” has been changed to “2016/06/24 00:00:00”.

  FIG. 21 is a diagram for explaining correlation analysis by the second correlation analysis unit 44. If there is a correlation between the data affected by the resource performance degradation and the delay degree, it is considered that the application is delayed due to the performance degradation of the resource 34. For this reason, the second correlation analysis unit 44 identifies all VMs that share resources 34 and VMs that have been correlated with data affected by resource performance degradation (the VMs that have been correlated with data affected by resource performance degradations). Are included) with reference to the configuration information storage unit 33. In FIG. 21, the VM in which the VM of the user # 1 is correlated with the data affected by the resource performance deterioration is the delay of the app # 1. Further, the VM of user # 2 is identified as the VM that has been correlated with the data affected by the resource performance degradation and the other VM sharing the resource 34.

  Then, the second correlation analysis unit 44 performs a correlation analysis between the data indicating the resource load amount of each specified VM and the delay degree with reference to the resource data storage unit 32, thereby calculating the load from the specified VM. Specify the VM you are running. A VM having a correlation between data representing a resource load amount and a delay degree is a VM to which a load is applied. In FIG. 21, the correlation between the CPU usage rate of the VM of user # 2 and the degree of delay of application # 1 is analyzed.

  Then, the second correlation analysis unit 44 extracts resource data having a good correlation with the delay degree from the resource data correlated by the correlation analysis. Then, the second correlation analysis unit 44 stores the correlation analysis result in the second analysis result storage unit 45 for the extracted resource data.

  FIG. 22 is a diagram illustrating an example of the second analysis result storage unit 45. As illustrated in FIG. 22, the second analysis result storage unit 45 associates diagnosis date and time, start time, end time, VM IP address, port number, application VM, resource type, correlation VM, and resource load data. Remember.

  The diagnosis date and time is the date and time when diagnosis was performed. The start time is the start time of the diagnosis target period. The end time is the end time of the diagnosis target period. The VM IP address is the IP address of the VM on which the application operates. The port number is a port number used by the application. The application VM is a VM on which an application having a correlation between the degree of delay and the data affected by the resource performance degradation operates. The resource type is a type of the resource 34 related to delay. The correlation VM is a VM in which there is a correlation between the data indicating the resource load amount and the delay degree. The resource load amount data is data representing the resource load amount correlated with the delay degree.

  For example, “2016/06/24 00:00:00” to “2016/06/24 00:00:00” was diagnosed at “2016/06/24 00:00:00” as a diagnosis target period. . The application in which the delay has occurred is specified by the VM IP address “10.20.30.40” and the port number “80”, and operates with “VM # 3”. The type of the resource 34 that is affected is “CPU”, and the “CPU usage rate” of “VM # 4” sharing “CPU” with “VM # 3” correlates with the degree of delay.

  The delay cause diagnosis unit 46 refers to the first analysis result storage unit 43 and the second analysis result storage unit 45 to determine the cause of the application delay for each type of VM resource data. Specifically, when there is a correlation with data representing the resource load amount of the VM related to the application, the delay cause diagnosis unit 46 increases the average response time by processing a request that always takes time. Since it is delayed, it is determined as “delay due to application factor”.

  In addition, when there is a correlation with the data affected by the VM resource performance degradation related to the application, the delay cause diagnosis unit 46 determines in the following three cases. First, when there is a correlation with the data representing the resource load amount of the same VM, the VM itself of the delayed application increases the load on the resource 34 and a delay occurs, so the delay cause diagnosis unit No. 46 is determined as “delay due to application factors”. Second, if there is a correlation with the data representing the resource load amount of another VM sharing the resource 34, the resource load increases due to the large amount of the resource 34 used by the other VM, and a delay occurs. Therefore, the delay cause diagnosis unit 46 determines “delay due to cloud base factor”. Third, if there is no correlation with the data representing the resource load amount of the VM sharing the resource 34, the delay cause diagnosis unit 46 determines that “the cause of the resource load increase is unknown”.

When there are a plurality of correlated resources 34, the delay cause diagnosis unit 46 outputs a plurality of delay causes. Further, the delay cause diagnosis unit 46 determines that the cause could not be specified when there is no correlated resource data in all VMs or when there is a large amount of data (when the number of delay causes is equal to or greater than the threshold _Td ). .

  Then, the delay cause diagnosis unit 46 stores the determination result in the diagnosis result storage unit 47. FIG. 23 is a diagram illustrating an example of the diagnosis result storage unit 47. As illustrated in FIG. 23, the diagnosis result storage unit 47 includes a diagnosis date and time, a start time, an end time, a VM IP address, a port number, a delay ratio, an application VM, a resource type, resource data, a load VM, and resource load amount data. The diagnostic results are stored in association with each other.

  The diagnosis date and time is the date and time when diagnosis was performed. The start time is the start time of the diagnosis target period. The end time is the end time of the diagnosis target period. The VM IP address is the IP address of the VM on which the application operates. The port number is a port number used by the application. The delay ratio is a ratio of the total time in which the delay occurs. The application VM is a VM on which a correlated application operates. The resource type is a type of the resource 34 related to delay. The resource data is the VM resource data of the correlated application. The load VM is a VM sharing the resource 34 and applying a load. The resource load amount data is resource load amount data having a correlation between VMs sharing the resource 34 and applying a load. The diagnosis result is a determination result by the delay cause diagnosis unit 46.

  For example, “2016/06/24 00:00:00” to “2016/06/24 00:00:00” was diagnosed at “2016/06/24 00:00:00” as a diagnosis target period. . The application in which the delay has occurred at a rate of “0.10” in the total time is specified by the VM IP address “10.20.30.40” and the port number “80”, and operates with “VM # 3”. The type of the resource 34 related to the delay is “CPU”. The “CPU allocation wait time ratio” correlates with the delay degree, and the “CPU usage rate” of “VM # 4” sharing “VM # 3” and “CPU” correlates with the delay degree. It is volume data, and the cause of delay is “cloud infrastructure factor”.

  In addition, when the “cloud base factor” is included as the delay factor of the application, the delay cause diagnosis unit 46 manages the operation of the delayed application, the delay factor of the application, the VM resource 34 related to the delay, and the like. The person in charge. FIG. 24 is a diagram illustrating an example of notification to the cloud operation manager. FIG. 24 shows a case where the DB server of the three-tier system including the Web server, the application server, and the DB server is affected by the VM CPU load of another application.

  As shown in FIG. 24, the application, the diagnosis target period, the ratio of the total time during which the delay occurs, the cause of the delay, the VM and the resource 34 of the application affected by the resource load, and the VM giving the resource load The resource 34 is notified to the cloud operation manager.

  Next, the flow of processing by the diagnostic device 4 will be described with reference to FIGS. FIG. 25 is a flowchart showing the flow of processing by the diagnostic device 4. As illustrated in FIG. 25, the diagnosis device 4 repeats the processes in steps S1 to S5 for the number of applications. That is, the diagnostic device 4 acquires delay degree information of the diagnosis target period of the app (step S1).

The diagnostic device 4, the number of delay degree is greater than or equal to the threshold T _rt it is determined whether or T _dr (step S2), and if not more T _dr is to process the next application. On the other hand, if it is equal to or greater than T _dr , the diagnostic device 4 performs a first correlation analysis process (step S3). Here, the first correlation analysis process is a process performed by the first correlation analysis unit 42.

  And the diagnostic apparatus 4 performs a 2nd correlation analysis process (step S4). Here, the second correlation analysis process is a process performed by the second correlation analysis unit 44. The process of step S4 is performed when there is a correlation between the data affected by the resource performance degradation and the delay degree. And the diagnostic apparatus 4 performs a delay cause diagnostic process (step S5). Here, the delay cause diagnosis process is a process performed by the delay cause diagnosis unit 46.

  Then, when the processes of step S1 to step S5 are repeated for the number of applications, the diagnosis device 4 determines whether there is an application diagnosed as having a response delay due to a cloud-based factor (step S6). Notifies the cloud operations manager (step S7).

  As described above, when there is an application diagnosed as having a response delay due to a cloud platform factor, the diagnostic device 4 notifies the cloud operations manager so that the cloud operations manager can consider measures for the cloud platform. .

  FIG. 26 is a flowchart showing the flow of the first correlation analysis process. As illustrated in FIG. 26, the first correlation analysis unit 42 acquires VM information related to the application from the VM information storage unit 31 (step S11), and the resources of all VMs related to the application from the resource data storage unit 32. Data is acquired (step S12).

  Then, the first correlation analysis unit 42 repeats the process of calculating the p-value of the decorrelation test between the degree of delay of the application and the resource data (step S13) by the number of resource data. Then, the first correlation analysis unit 42 executes multiple test correction (step S14), and extracts only resource data whose p-value corrected by the multiple test correction is equal to or less than a threshold value (step S15).

  Then, the first correlation analysis unit 42 performs resource data extraction processing for extracting resource data that is well correlated with the degree of delay from the extracted resource data (step S16). And the 1st correlation analysis part 42 preserve | saves a correlation analysis result in the 1st analysis result memory | storage part 43 (step S17).

  In this way, the first correlation analysis unit 42 calculates the p-value of the non-correlation test between the delay degree of the application and the resource data, corrects the p-value by multiple test correction, and the corrected p-value is the threshold value. Extract only the following resource data. Then, the first correlation analysis unit 42 further extracts resource data that correlates well with the degree of delay from the extracted resource data. Therefore, it is possible to accurately extract resource data having a correlation with the delay degree.

FIG. 27 is a flowchart showing a flow of resource data extraction processing. As shown in FIG. 27, the first correlation analysis unit 42 repeats the processing from step S21 to step S22 for the number of correlated resource data. That is, the first correlation analysis unit 42 calculates the median value of the resource data when the delay degree is equal to or greater than the threshold value T _{rt and} sets it as the threshold value _Tr (step S21), and determines the delay degree when the resource data is equal to or greater than the threshold value _Tr . Data is extracted (step S22).

  Then, the first correlation analysis unit 42 performs a multiple comparison test with the delay degree extracted for each resource data as an input (step S23). Then, the first correlation analysis unit 42 identifies the resource data having the largest number of times that tends to become larger than the distribution of the delay degree of the other resource data as a result of the test (step S24).

  Then, the first correlation analysis unit 42 correlates the resource data determined not to be different from that of the specified resource data in the extracted distribution of delay degrees and the specified resource data with a good degree of delay. Output as certain resource data (step S25).

  As described above, by extracting resource data having a good correlation with the delay degree by the resource data extraction process, the first correlation analysis unit 42 can extract only resource data having a strong correlation with the delay degree. The resource data extraction process is performed by the second correlation analysis unit 44 when called by the second correlation analysis unit 44 as described later.

  FIG. 28 is a flowchart showing the flow of the second correlation analysis process. As illustrated in FIG. 28, the second correlation analysis unit 44 repeats the processes in steps S <b> 31 to S <b> 37 as many times as there are resources correlated with the data affected by the resource performance degradation.

  That is, the second correlation analysis unit 44 acquires information on VMs that share the resource of interest (step S31). Here, the resource of interest is the resource 34 that is the target of one process from step S31 to step S37. Then, the second correlation analysis unit 44 acquires data representing the resource load amounts of all VMs sharing the target resource (step S32). All VMs include the VM of the resource of interest.

  Then, the second correlation analysis unit 44 performs a process (step S33) of calculating the p-value of the decorrelation test between the delay degree of the application and the data indicating the resource load amount in all VM resources sharing the resource 34. Repeat for the number of data representing the load. Then, the second correlation analysis unit 44 performs multiple test correction (step S34), and extracts only resource data whose p-value corrected by the multiple test correction is equal to or less than a threshold value (step S35). Then, the second correlation analysis unit 44 performs resource data extraction processing (step S36), and stores the correlation analysis result in the second analysis result storage unit 45 (step S37).

  As described above, the second correlation analysis unit 44 performs an uncorrelation test on each resource 34 correlated with data affected by resource performance degradation with data representing the resource load amount of all VMs sharing the resource 34. p-value is calculated. Then, the second correlation analysis unit 44 corrects the p-value by multiple test correction, and extracts only the resource data whose corrected p-value is equal to or less than the threshold value. Then, the second correlation analysis unit 44 further extracts resource data that correlates well with the degree of delay from the extracted resource data. Therefore, the second correlation analysis unit 44 performs a correlation analysis with respect to the delay degree of the data representing the resource load amount of all VMs sharing the resource 34 for each resource 34 having the correlation of the data affected by the resource performance degradation. Can be done accurately.

FIG. 29 is a flowchart showing a flow of delay cause diagnosis processing. As illustrated in FIG. 29, the delay cause diagnosis unit 46 calculates the ratio of the total time that has been delayed from the number of times that the degree of delay is equal to or greater than the threshold value _Trt (step S41). Then, the delay cause diagnosis unit 46 repeats the processing from step S42 to step S49 for the number of VMs of the application having correlated resource data.

  That is, the delay cause diagnosis unit 46 determines whether or not there is a correlation with data representing the resource load amount of the target VM (step S42). Here, the attention VM is a VM that is a target of one processing of Step S42 to Step S49. If there is a correlation with the data representing the resource load amount of the target VM, the delay cause diagnosis unit 46 determines that the delay is due to the application factor (step S43), and proceeds to the next VM process.

  On the other hand, if there is no correlation with the data representing the resource load amount of the target VM, the delay cause diagnosis unit 46 determines whether there is a correlation with the data affected by the resource performance degradation of the target VM ( In step S44), if not, the process proceeds to the next VM. On the other hand, when there is a correlation with the data affected by the resource performance degradation of the target VM, the delay cause diagnosis unit 46 has a correlation with the data representing the resource load amount for all the VMs sharing the resource 34. Whether or not (step S45). The delay cause diagnosis unit 46 determines that the cause of the increase in resource load is unknown when there is no VM that correlates the data indicating the resource load amount and the delay degree (step S46). Proceed to processing.

  In addition, the delay cause diagnosis unit 46 performs the processing from step S47 to step S49 for each VM having a correlation between the data indicating the resource load amount and the delay degree. That is, the delay cause diagnosis unit 46 determines whether the data affected by the correlated resource performance degradation and the data representing the resource load amount are the same VM (step S47).

  If the data affected by the correlated resource performance degradation and the data representing the resource load amount are the same VM, the delay cause diagnosis unit 46 determines that the delay is caused by the application factor (step S48). The process proceeds to the next VM. On the other hand, if the data affected by the correlated resource performance degradation and the data representing the resource load amount are not the same VM, the delay cause diagnosis unit 46 determines that the delay is caused by the cloud base factor (step S49). The process proceeds to the next VM.

When the repetition processing of step S42 to step S49 is completed, the delay cause diagnosis unit 46 determines whether the number of determination results is 1 or more and the threshold _Td or less (step S50). If the number of determination results is 1 or more and the threshold value _Td or less, the delay cause diagnosis unit 46 outputs the determination result assuming that the resource 34 related to the delay has been identified (step S51). On the other hand, when the number of determination results is not less than 1 and not less than the threshold value _Td, it is assumed that the delay cause diagnosis unit 46 cannot identify the resource 34 related to the delay (step S52).

  Then, the delay cause diagnosis unit 46 stores the delay diagnosis result (step S53). As described above, the delay cause diagnosis unit 46 identifies the cause of the delay based on the analysis results obtained by the first correlation analysis unit 42 and the second correlation analysis unit 44, so that the diagnosis device 4 has a problem in the cloud base. If so, the problematic resource 34 can be identified.

  Next, the flow of the first correlation analysis process and the second correlation analysis process when performing correlation analysis using the two-group comparison test will be described with reference to FIGS. 30 and 31. FIG. FIG. 30 is a flowchart showing the flow of the first correlation analysis process using the two-group comparison test.

  As illustrated in FIG. 30, the first correlation analysis unit 42 acquires VM information related to the application from the VM information storage unit 31 (step S61), and the resources of all VMs related to the application from the resource data storage unit 32. Data is acquired (step S62).

The first correlation analysis unit 42, the delay degree divided into resource data and the threshold T _rt or more resources data when less than the threshold value T _rt (step S63), calculates the p-value by two groups comparison test (step S64) The process is repeated for the number of resource data. Then, the first correlation analysis unit 42 performs multiple test correction (step S65), and extracts only resource data whose p-value corrected by the multiple test correction is equal to or less than a threshold value (step S66).

  Then, the first correlation analysis unit 42 performs resource data extraction processing (step S67), and stores the correlation analysis result in the first analysis result storage unit 43 (step S68).

  FIG. 31 is a flowchart showing the flow of the second correlation analysis process using the two-group comparison test. As illustrated in FIG. 31, the second correlation analysis unit 44 repeats the processes in steps S <b> 71 to S <b> 78 for the number of resources that have been correlated with the data affected by the resource performance degradation.

  That is, the second correlation analysis unit 44 acquires information on VMs sharing the target resource (step S71), and acquires data representing the resource load amount of all VMs sharing the target resource (step S72). ).

Then, the second correlation analysis unit 44 repeats the processes of step S73 and step S74 for the number of data representing the resource load amount of all VMs sharing the resource 34. That is, the second correlation analysis unit 44, the delay degree divided into resource data and the threshold T _rt or more resources data when less than the threshold value T _rt (step S73), calculates the p-value by two groups comparison test (step S74).

  Then, the second correlation analysis unit 44 executes multiple test correction (step S75), and extracts only resource data whose p-value corrected by the multiple test correction is equal to or less than a threshold value (step S76). Then, the second correlation analysis unit 44 performs resource data extraction processing (step S77), and stores the correlation analysis result in the second analysis result storage unit 45 (step S78).

Thus, the first correlation analyzer 42 and the second correlation analysis unit 44 divides the resource data and the threshold T _rt over resource data when the delay degree is less than the threshold value T _rt, p-value by two groups comparison test By calculating, correlation analysis can be performed while suppressing the influence of undelayed data.

  As described above, in the first embodiment, the delay determination unit 41 acquires the delay degree information from the delay information creation device 2 to determine whether or not a delay has occurred in the application, and the first correlation analysis unit 42 performs the delay. Correlation analysis is performed between the VM resource data related to the application in which the error occurs and the degree of delay. When there is a correlation between the data affected by the resource performance degradation and the delay degree, the second correlation analysis unit 44 uses the data indicating the resource load amount and the delay degree for all VMs sharing the resource 34. Analyze the correlation. Then, the delay cause diagnosis unit 46 identifies the cause of the delay based on the correlation analysis results of the first correlation analysis unit 42 and the second correlation analysis unit 44.

  Therefore, the diagnostic apparatus 4 can specify whether the performance degradation of the application is caused by the cloud platform or the application. Further, the diagnosis device 4 can specify which resource 34 has the cause when the cause is in the cloud infrastructure.

  Further, in the embodiment, when the first correlation analysis unit 42 performs the correlation analysis by determining whether there is a difference in the tendency of the resource data when it is delayed and when it is not delayed, Correlation analysis can be performed while suppressing the influence of non-delayed data.

  In the embodiment, the first correlation analysis unit 42 calculates a correlation coefficient with the degree of delay, performs a non-correlation test, calculates a p-value, and corrects the calculated p-value by multiple test correction. Whether or not there is a correlation with the delay degree is determined based on the corrected p-value. Therefore, the first correlation analysis unit 42 can accurately perform the correlation analysis between the delay degree and the resource data.

  By the way, when the number of requests is small, the average response time becomes unstable, and the delay degree becomes unstable. If there is an unstable part of the delay degree, the resource 34 related to the delay may not be found. Therefore, in the second embodiment, a diagnostic apparatus that performs a delay cause diagnosis both when the data of the portion where the delay degree is unstable is removed and when all the data is used will be described.

  FIG. 32 is a diagram for explaining the removal of data with an unstable delay degree. The horizontal axis in FIG. 32 indicates the number of requests, and the vertical axis indicates the degree of delay. As shown in FIG. 32, when the number of requests is small, the delay degree becomes unstable. For this reason, the diagnostic apparatus according to the second embodiment performs the delay cause diagnosis both in the case of using all the data enclosed by the solid square and in the case of using only the data enclosed by the dashed square.

  FIG. 33 is a diagram illustrating a configuration of a cloud system according to the second embodiment. Here, for convenience of explanation, functional units that play the same functions as the respective units shown in FIG.

  As illustrated in FIG. 33, the cloud system 1a according to the second embodiment includes a diagnostic device 4a instead of the diagnostic device 4 as compared to the cloud system 1 illustrated in FIG. The diagnostic device 4 a includes a data removal unit 40 that is not included in the diagnostic device 4, and includes a delay determination unit 41 a instead of the delay determination unit 41.

  The data removal unit 40 removes data whose delay degree is unstable. The delay determination unit 41a determines whether or not there is a delay in the application in both the data from which the data with the unstable degree of delay is removed by the data removal unit 40 and all the data created by the delay information creation device 2. Determine.

  Since the average response time used for calculating the delay degree is stable as the number of requests increases, the dispersion of the delay degree decreases as the number of requests increases. Therefore, the data removing unit 40 divides the data based on the number of requests, uses the number of requests whose dispersion of delay degrees changes from decrease to increase as a threshold, and removes data below the threshold as data with unstable delay.

  FIG. 34 is a diagram for explaining a method of removing data with an unstable delay degree. FIG. 34A shows data division. The horizontal axis is the number of requests, and the vertical axis is the degree of delay. As shown in FIG. 34A, the data is divided according to the number of requests. IDs of 0, 1, 2, and 3 are added to the divided data in order from the smallest number of requests.

  FIG. 34B shows the dispersion of the delay degree of the divided data. The horizontal axis represents the data ID, and the vertical axis represents the dispersion of the delay degree. As shown in FIG. 34 (b), the dispersion of the delay degree for the data whose ID is 0 is about 3.3, and the dispersion of the delay degree for the data whose ID is 1 is about 2.0. The dispersion of the delay degree for the data whose ID is 2 is about 4.9. That is, the dispersion of the delay degree decreases until ID = 1, and increases when ID = 2.

  Therefore, the data removal unit 40 removes data with the number of requests equal to or less than the threshold as data with an unstable delay degree, using the number of requests at the boundary between ID = 1 and ID = 2 as a threshold. In FIG. 34B, the threshold is about 1600, and data with ID = 0 and ID = 1 is removed.

Specifically, the data removal unit 40 removes data by the following procedure.
(1) The data removal unit 40 sets C _min and C _max as the minimum and maximum requests, respectively, among the data having a delay degree equal to or greater than the threshold value T _rt .
(2) Then, the data removal unit 40 calculates the number of divisions k as k = 1 + log ₂ n using the Sturges formula, where n is the number of requests with the number of requests being C _min or more and C _max or less.
(3) Then, the data removal unit 40 divides the data having the number of requests of C _min or more and C _max or less into k pieces in the order of the number of requests, and calculates the dispersion of the degree of delay for each divided data.
(4) Then, the data removal unit 40 looks at the distribution in order from the data with the smallest number of requests, and sets the minimum number of requests for division at which the distribution changes from decreasing to increasing as T _c .
(5) Then, the data removal unit 40 removes data whose number of requests is equal to or less than the threshold value T _c .

FIG. 35 is a flowchart showing the flow of processing by the diagnostic device 4a. As illustrated in FIG. 35, the diagnosis device 4a repeats the processing from step S81 to step S91 for the number of applications. That is, the diagnostic device 4a acquires the delay degree information of the diagnosis target period of the app (step S81). Then, the diagnostic device 4a performs a threshold value calculation process for calculating the request number threshold value _Tc (step S82).

The diagnostic device 4a is the number of times larger and the delay degree number of requests than the threshold value T _c is greater than or equal to the threshold T _rt it is determined whether or T _dr (step S83), if the decision result in the step S83 is No Advances to step S88. On the other hand, when the determination result of step S83 is Yes, the diagnostic device 4a extracts data whose number of requests is larger than the threshold value _Tc (step S84), and performs the first correlation analysis process (step S85).

  Then, the diagnostic device 4a performs the second correlation analysis process (step S86). Note that the process of step S86 is performed when there is a correlation of data affected by resource performance degradation. Then, the diagnostic device 4a performs a delay cause diagnosis process (step S87).

The diagnostic device 4a, when the number of delay degree is greater than or equal to the threshold T _rt it is determined whether or T _dr (step S88), the number of times the delay degree is equal to or more than the threshold T _rt not more T _dr is the following Process the app. On the other hand, when the number of times that the delay degree is equal to or greater than the threshold _Trt is equal to or greater than _Tdr , the diagnostic device 4a performs the first correlation analysis process (step S89).

  Then, the diagnostic device 4a performs the second correlation analysis process (step S90), and the process of step S90 is performed when there is a correlation of data that is affected by resource performance degradation. Then, the diagnostic device 4a performs a delay cause diagnosis process (step S91).

  Then, when the processing of step S81 to step S91 is repeated for the number of applications, the diagnosis device 4a determines whether there is an application diagnosed as having a response delay due to cloud-based factors (step S92). The cloud operation manager is notified (step S93).

FIG. 36 is a flowchart showing a flow of threshold value calculation processing. As shown in FIG. 36, the data removal unit 40 sets C _min and C _max as the minimum request number and the maximum request number, respectively, of the data having a delay degree equal to or greater than the threshold T _rt (step S101). Then, the data removal unit 40 calculates the division number k, where n is the number of requests with the number of requests equal to or greater than C _min and equal to or less than C _max (step S102).

Then, the data removal unit 40 rearranges the data having the number of requests from C _min to C _{max in the} order of the number of requests and divides the data into k pieces (step S103). Then, the data removal unit 40 repeats the process of calculating the variance of the delay degree of the divided data (step S104) for the number of divisions. Then, the data removal unit 40 sets T _c as the minimum number of requests for the divided data whose variance changes from decrease to increase (step S105).

As described above, in the second embodiment, the diagnosis device 4a determines whether or not there is a delay in the application in both the data from which the data with the number of requests equal to or less than _Tc is removed and all the data. To diagnose the cause of the application delay. Therefore, the diagnostic device 4a can diagnose the cause of the application delay without being affected by data whose delay degree is unstable.

  In the first and second embodiments, the diagnostic apparatuses 4 and 4a have been described. However, a diagnostic program having the same function can be obtained by realizing the configuration of the diagnostic apparatuses 4 and 4a with software. Therefore, a computer that executes a diagnostic program will be described.

  FIG. 37 is a diagram illustrating a hardware configuration of a computer that executes the diagnostic program according to the first and second embodiments. As shown in FIG. 37, the computer 50 includes a main memory 51, a CPU 52, a LAN (Local Area Network) interface 53, and an HDD (Hard Disk Drive) 54. The computer 50 includes a super IO (Input Output) 55, a DVI (Digital Visual Interface) 56, and an ODD (Optical Disk Drive) 57.

  The main memory 51 is a memory for storing a program and a program execution result. The CPU 52 is a central processing unit that reads a program from the main memory 51 and executes it. The CPU 52 includes a chip set having a memory controller.

  The LAN interface 53 is an interface for connecting the computer 50 to another computer via a LAN. The HDD 54 is a disk device that stores programs and data, and the super IO 55 is an interface for connecting an input device such as a mouse or a keyboard. The DVI 56 is an interface for connecting a liquid crystal display device, and the ODD 57 is a device for reading / writing a DVD.

  The LAN interface 53 is connected to the CPU 52 by PCI Express (PCIe), and the HDD 54 and ODD 57 are connected to the CPU 52 by SATA (Serial Advanced Technology Attachment). The super IO 55 is connected to the CPU 52 by LPC (Low Pin Count).

  The diagnostic program executed in the computer 50 is stored in a DVD that is an example of a recording medium readable by the computer 50, read from the DVD by the ODD 57, and installed in the computer 50. Alternatively, the diagnostic program is stored in a database or the like of another computer system connected via the LAN interface 53, read from these databases, and installed in the computer 50. The installed diagnostic program is stored in the HDD 54, read into the main memory 51, and executed by the CPU 52.

  In the first and second embodiments, the case where the normalized average response time is used as the delay degree has been described. However, the present invention is not limited to this, and the same applies when other values are used as the delay degree. Can be applied.

  In the first and second embodiments, the case where the delay information creation device 2 is a device different from the diagnosis devices 4 and 4a has been described. However, the present invention is not limited to this, and the delay information creation device 2 The same can be applied when the function is included in the diagnostic apparatuses 4 and 4a.

DESCRIPTION OF SYMBOLS 1,1a Cloud system 2 Delay information creation apparatus 3c Network switch 4,4a Diagnostic apparatus 9 Physical machine 9a Virtual machine 21 Capture part 22 Packet information storage part 23 Type determination data storage part 24 Type determination part 25 Type information storage part 26 Response Time calculation unit 27 Response time information storage unit 28 Normalization unit 29 Representative information storage unit 30 Delay degree information storage unit 31 VM information storage unit 32 Resource data storage unit 33 Configuration information storage unit 40 Data removal unit 41, 41a Delay determination unit 42 First correlation analysis unit 43 First analysis result storage unit 44 Second correlation analysis unit 45 Second analysis result storage unit 46 Delay cause diagnosis unit 47 Diagnosis result storage unit 50 Computer 51 Main memory 52 CPU
53 LAN interface 54 HDD
55 Super IO
56 DVI
57 ODD

Claims (10)

On the computer,
Obtaining a delay degree indicating an execution delay for each application and determining whether the delay degree is equal to or greater than a predetermined threshold;
A diagnostic program for identifying a cause of the delay based on a correlation between information on a resource related to the application and the delay degree when it is determined that the delay degree is equal to or greater than a predetermined threshold.   The specified process includes a first correlation that is a correlation between information on a resource used by a virtual machine on which the application is executed and the degree of delay, and a resource load of all virtual machines that share the resource The diagnostic program according to claim 1, wherein the cause of the delay is specified based on a second correlation that is a correlation between data representing a quantity and the degree of delay.   The first correlation is a correlation between the data indicating the resource load amount of the virtual machine and the delay degree, and a correlation between the data affected by a decrease in resource performance of the virtual machine and the delay degree. The diagnostic program according to claim 2.   The first correlation is a correlation between a CPU usage rate of the virtual machine and the delay degree, and a correlation between a ratio of CPU allocation waiting time of the virtual machine and the delay degree. The diagnostic program according to claim 3.   The specified process has a correlation between the CPU allocation waiting time ratio of the virtual machine and the delay degree, and the CPU usage of other virtual machines sharing the CPU used by the virtual machine. The diagnosis program according to claim 4, wherein when there is a correlation between a rate and the degree of delay, a cloud base is specified as the cause of the delay.   The process of identifying the cause of the delay is performed when there is a difference between the information about the resource when the delay degree is equal to or greater than a predetermined threshold and the information about the resource when the delay degree is not equal to or greater than the predetermined threshold. The diagnostic program according to claim 1, wherein it is determined that the correlation exists. The degree of delay is a normalized average response time obtained by normalizing an average value of response times for a request every fixed time,
Among the plurality of normalized average response times, the computer further executes a process of removing the normalized average response time when the number of requests used for calculating the average value is small as an unstable normalized average response time,
The specifying process is performed when the number of the normalized response times excluding the unstable normalized average response time is equal to or greater than a second threshold value during a certain period, or 7. The cause of the delay is specified when the number of normalized response times including the normalized average response time is equal to or greater than a second threshold value during a certain period of time. The diagnostic program as described in any one.   The specifying process calculates a correlation coefficient with the delay degree, performs a non-correlation test to calculate a p-value, corrects the calculated p-value by a multiple test correction, and converts the corrected p-value into a corrected p-value. 6. The diagnostic program according to claim 2, wherein the first correlation is analyzed by determining whether there is a correlation with the degree of delay based on the diagnosis. Computer
Obtaining a delay degree indicating an execution delay for each application and determining whether the delay degree is equal to or greater than a predetermined threshold;
When it is determined that the delay degree is equal to or greater than a predetermined threshold value, a process of identifying the cause of the delay based on a correlation between information related to resources related to the application and the delay degree is performed. Diagnostic method to do. A delay determination unit that acquires a delay degree indicating an execution delay for each application and determines whether the delay degree is equal to or greater than a predetermined threshold;
A specifying unit that identifies the cause of the delay based on a correlation between the information related to the resource related to the application and the delay degree when the delay determination part determines that the delay degree is equal to or greater than a predetermined threshold; A diagnostic apparatus comprising:

Images