613:
host whose performance is being tested. Usually, a separate PC acts as a test conductor, coordinating and gathering metrics from each of the injectors and collating performance data for reporting purposes. The usual sequence is to ramp up the load: to start with a few virtual users and increase the number over time to a predetermined maximum. The test result shows how the performance varies with the load, given as number of users vs. response time. Various tools are available to perform such tests. Tools in this category usually execute a suite of tests which emulate real users against the system. Sometimes the results can reveal oddities, e.g., that while the average response time might be acceptable, there are outliers of a few key transactions that take considerably longer to complete – something that might be caused by inefficient database queries, pictures, etc.
638:), weighted by the transaction-mix (business transactions per hour). The weighted transaction resource demands are added up to obtain the hourly resource demands and divided by the hourly resource capacity to obtain the resource loads. Using the response time formula (R=S/(1-U), R=response time, S=service time, U=load), response times can be calculated and calibrated with the results of the performance tests. Analytical performance modeling allows evaluation of design options and system sizing based on actual or anticipated business use. It is therefore much faster and cheaper than performance testing, though it requires thorough understanding of the hardware platforms.
449:
is to identify the "weakest link" – there is inevitably a part of the system which, if it is made to respond faster, will result in the overall system running faster. It is sometimes a difficult task to identify which part of the system represents this critical path, and some test tools include (or can have add-ons that provide) instrumentation that runs on the server (agents) and reports transaction times, database access times, network overhead, and other server monitors, which can be analyzed together with the raw performance statistics. Without such instrumentation one might have to have someone crouched over
312:, also known as endurance testing, is usually done to determine if the system can sustain the continuous expected load. During soak tests, memory utilization is monitored to detect potential leaks. Also important, but often overlooked is performance degradation, i.e. to ensure that the throughput and/or response times after some long period of sustained activity are as good as or better than at the beginning of the test. It essentially involves applying a significant load to a system for an extended, significant period of time. The goal is to discover how the system behaves under sustained use.
68:
170:
27:
791:
Analyze, consolidate, and share results data. Make a tuning change and retest. Compare the results of both tests. Each improvement made will return smaller improvement than the previous improvement. When do you stop? When you reach a CPU bottleneck, the choices then are either improve the code or add
747:
Identify the response time, throughput, and resource-use goals and constraints. In general, response time is a user concern, throughput is a business concern, and resource use is a system concern. Additionally, identify project success criteria that may not be captured by those goals and constraints;
521:
In performance testing, it is often crucial for the test conditions to be similar to the expected actual use. However, in practice this is hard to arrange and not wholly possible, since production systems are subjected to unpredictable workloads. Test workloads may mimic occurrences in the production
448:
However, performance testing is frequently not performed against a specification; e.g., no one will have expressed what the maximum acceptable response time for a given population of users should be. Performance testing is frequently used as part of the process of performance profile tuning. The idea
329:
Breakpoint testing is similar to stress testing. An incremental load is applied over time while the system is monitored for predetermined failure conditions. Breakpoint testing is sometimes referred to as
Capacity Testing because it can be said to determine the maximum capacity below which the system
612:
Performance testing technology employs one or more PCs or Unix servers to act as injectors, each emulating the presence of numbers of users and each running an automated sequence of interactions (recorded as a script, or as a series of scripts to emulate different types of user interaction) with the
566:
Each of the tools mentioned in the above list (which is not exhaustive nor complete) either employs a scripting language (C, Java, JS) or some form of visual representation (drag and drop) to create and simulate end user work flows. Most of the tools allow for something called "Record & Replay",
546:
It is critical to the cost performance of a new system that performance test efforts begin at the inception of the development project and extend through to deployment. The later a performance defect is detected, the higher the cost of remediation. This is true in the case of functional testing, but
469:
It is always helpful to have a statement of the likely peak number of users that might be expected to use the system at peak times. If there can also be a statement of what constitutes the maximum allowable 95 percentile response time, then an injector configuration could be used to test whether the
330:
will perform to its required specifications or
Service Level Agreements. The results of breakpoint analysis applied to a fixed environment can be used to determine the optimal scaling strategy in terms of required hardware or conditions that should trigger scaling-out events in a cloud environment.
359:
This is a relatively new form of performance testing when global applications such as
Facebook, Google and Knowledge, are performance tested from load generators that are placed on the actual target continent whether physical machines or cloud VMs. These tests usually requires an immense amount of
300:
is normally used to understand the upper limits of capacity within the system. This kind of test is done to determine the system's robustness in terms of extreme load and helps application administrators to determine if the system will perform sufficiently if the current load goes well above the
248:
is in general a testing practice performed to determine how a system performs in terms of responsiveness and stability under a particular workload. It can also serve to investigate, measure, validate or verify other quality attributes of the system, such as scalability, reliability and resource
395:
If a system identifies end-users by some form of log-in procedure then a concurrency goal is highly desirable. By definition this is the largest number of concurrent system users that the system is expected to support at any given moment. The work-flow of a scripted transaction may impact true
461:
would then need to be configured to introduce the lag that would typically occur on public networks. Loads should be introduced to the system from realistic points. For example, if 50% of a system's user base will be accessing the system via a 56K modem connection and the other half over a
533:
or platform require coordinated performance testing, with all consumers creating production-like transaction volumes and load on shared infrastructures or platforms. Because this activity is so complex and costly in money and time, some organizations now use tools to monitor and simulate
567:
where in the performance tester will launch the testing tool, hook it on a browser or thick client and capture all the network transactions which happen between the client and server. In doing so a script is developed which can be enhanced/modified to emulate various business scenarios.
320:
Spike testing is done by suddenly increasing or decreasing the load generated by a very large number of users, and observing the behavior of the system. The goal is to determine whether performance will suffer, the system will fail, or it will be able to handle dramatic changes in load.
732:
and the production environment as well as the tools and resources available to the test team. The physical environment includes hardware, software, and network configurations. Having a thorough understanding of the entire test environment at the outset enables more efficient
547:
even more so with performance testing, due to the end-to-end nature of its scope. It is crucial for a performance test team to be involved as early as possible, because it is time-consuming to acquire and prepare the testing environment and other key performance requisites.
338:
Rather than testing for performance from a load perspective, tests are created to determine the effects of configuration changes to the system's components on the system's performance and behavior. A common example would be experimenting with different methods of
575:
This forms the other face of performance testing. With performance monitoring, the behavior and response characteristics of the application under test are observed. The below parameters are usually monitored during the a performance test execution
382:
Many performance tests are undertaken without setting sufficiently realistic, goal-oriented performance goals. The first question from a business perspective should always be, "why are we performance-testing?". These considerations are part of the
440:
It is critical to detail performance specifications (requirements) and document them in any performance test plan. Ideally, this is done during the requirements development phase of any system development project, prior to any design effort. See
697:
Configure the test environment (ideally identical hardware to the production platform), router configuration, quiet network (we don't want results upset by other users), deployment of server instrumentation, database test sets developed,
456:
Performance testing can be performed across the web, and even done in different parts of the country, since it is known that the response times of the internet itself vary regionally. It can also be done in-house, although
768:
Prepare the test environment, tools, and resources necessary to execute each strategy, as features and components become available for test. Ensure that the test environment is instrumented for resource monitoring as
466:, then the load injectors (computers that simulate real users) should either inject load over the same mix of connections (ideal) or simulate the network latency of such connections, following the same user profile.
268:
is the simplest form of performance testing. A load test is usually conducted to understand the behavior of the system under a specific expected load. This load can be the expected concurrent number of users on the
411:
This refers to the time taken for one system node to respond to the request of another. A simple example would be a HTTP 'GET' request from browser client to web server. In terms of response time this is what all
252:
Performance testing, a subset of performance engineering, is a computer science practice which strives to build performance standards into the implementation, design and architecture of a system.
620:, in order to see what happens when an acceptable load is exceeded. Does the system crash? How long does it take to recover if a large load is reduced? Does its failure cause collateral damage?
351:
Isolation testing is not unique to performance testing but involves repeating a test execution that resulted in a system problem. Such testing can often isolate and confirm the fault domain.
762:
that variability, define test data, and establish metrics to be collected. Consolidate this information into one or more models of system usage to implemented, executed, and analyzed.
604:
to measure what parts of a device or software contribute most to the poor performance, or to establish throughput levels (and thresholds) for maintained acceptable response time.
513:(UAT) or development. As a best practice it is always advisable to have a separate performance testing environment resembling the production environment as much as possible.
563:
This part of performance testing mainly deals with creating/scripting the work flows of key identified business processes. This can be done using a wide variety of tools.
737:
and planning and helps you identify testing challenges early in the project. In some situations, this process must be revisited periodically throughout the project's
596:
As a first step, the patterns generated by these 4 parameters provide a good indication on where the bottleneck lies. To determine the exact root cause of the issue,
428:
apart from recognizing a period of time where there is no activity 'on the wire'. To measure render response time, it is generally necessary to include functional
529:) have created additional complexities with performance testing. To truly replicate production-like states, enterprise services or assets that share a common
701:
Dry run the tests - before actually executing the load test with predefined users, a dry run is carried out in order to check the correctness of the script.
782:
748:
for example, using performance tests to evaluate which combination of configuration settings will result in the most desirable performance characteristics.
1154:
387:
of the testing. Performance goals will differ depending on the system's technology and purpose, but should always include some of the following:
85:
40:
132:
104:
842:
626:
is a method to model the behavior of a system in a spreadsheet. The model is fed with measurements of transaction resource demands (
491:
What is the
Application Workload Mix of each system component? (for example: 20% log-in, 40% search, 30% item select, 10% checkout).
403:
If the system has no concept of end-users, then performance goal is likely to be based on a maximum throughput or transaction rate.
111:
482:
In detail, what is the performance test scope? What subsystems, interfaces, components, etc. are in and out of scope for this test?
424:
Load-testing tools have difficulty measuring render-response time, since they generally have no concept of what happens within a
704:
Execute tests – probably repeatedly (iteratively) in order to see whether any unaccounted-for factor might affect the results.
118:
277:
within the set duration. This test will give out the response times of all the important business critical transactions. The
650:
Decide whether to use internal or external resources to perform the tests, depending on inhouse expertise (or lack of it).
100:
721:
According to the
Microsoft Developer Network the Performance Testing Methodology consists of the following activities:
453:
at the server to see how much CPU load the performance tests are generating (assuming a
Windows system is under test).
522:
environment as far as possible, but only in the simplest systems can one exactly replicate this workload variability.
509:
To ensure consistent results, the performance testing environment should be isolated from other environments, such as
231:
213:
151:
54:
46:
195:
1092:
814:
681:
Specify test data needed and charter effort (often overlooked, but vital to carrying out a valid performance test).
485:
For the user interfaces (UIs) involved, how many concurrent users are expected for each (specify peak vs. nominal)?
707:
Analyze the results - either pass/fail, or investigation of critical path and recommendation of corrective action.
623:
1193:
835:
738:
397:
180:
89:
416:
tools actually measure. It may be relevant to set server response time goals between all nodes of the system.
535:
526:
1117:
488:
What does the target system (hardware) look like (specify all server and network appliance configurations)?
125:
601:
506:
A stable build of the system which must resemble the production environment as closely as is possible.
340:
187:
1188:
1061:
828:
617:
297:
691:
Develop detailed performance test project plan, including all dependencies and associated timelines.
1102:
1051:
1021:
963:
808:
534:
production-like conditions (also referred as "noise") in their performance testing environments (
510:
442:
191:
78:
494:
What is the System
Workload Mix? (for example: 30% Workload A, 20% Workload B, 50% Workload C).
653:
Gather or elicit performance requirements (specifications) from users and/or business analysts.
497:
What are the time requirements for any/all back-end batch processes (specify peak vs. nominal)?
274:
981:
976:
802:
538:) to understand capacity and resource requirements and verify / validate quality attributes.
432:
as part of the performance test scenario. Many load testing tools do not offer this feature.
286:
265:
1026:
1016:
450:
270:
785:. Execute validated tests for analysis while monitoring the test and the test environment.
688:
scripts for each application/component under test, using chosen test tools and strategies.
8:
1011:
1006:
996:
940:
885:
875:
1066:
1041:
1001:
991:
935:
458:
378:
It can measure which parts of the system or workload cause the system to perform badly.
282:
1076:
902:
870:
865:
635:
597:
425:
1046:
1031:
986:
912:
897:
890:
851:
729:
685:
660:(or project charter), including requirements, resources, timelines and milestones.
1071:
1122:
1097:
1036:
945:
917:
530:
289:
in the application software and the hardware that the software is installed on
1182:
478:
Performance specifications should ask the following questions, at a minimum:
384:
1112:
1107:
950:
880:
413:
400:
especially if the iterative part contains the log-in and log-out activity.
309:
285:, etc. are also monitored during the test, this will assist in identifying
1127:
971:
907:
734:
429:
675:
668:
664:
463:
198:. Statements consisting only of original research should be removed.
67:
820:
759:
755:
278:
775:
Develop the performance tests in accordance with the test design.
555:
Performance testing is mainly divided into two main categories:
781:
Run and monitor your tests. Validate the tests, test data, and
758:, determine variability among representative users and how to
372:
It can demonstrate that the system meets performance criteria.
657:
631:
627:
375:
It can compare two systems to find which performs better.
360:
preparation and monitoring to be executed successfully.
716:
525:
Loosely-coupled architectural implementations (e.g.:
92:. Unsourced material may be challenged and removed.
368:Performance testing can serve different purposes:
1180:
1155:"Rational Performance Tester: Tips & Tricks"
836:
435:
390:
811: – Estimation of web server performance
646:Tasks to perform such a test would include:
363:
805: – Standardized performance evaluation
694:Install and configure injectors/controller.
55:Learn how and when to remove these messages
843:
829:
16:Testing performance under a given workload
817: – Standard for managing bottlenecks
745:Identify Performance Acceptance Criteria.
616:Performance testing can be combined with
570:
232:Learn how and when to remove this message
214:Learn how and when to remove this message
152:Learn how and when to remove this message
558:
470:proposed system met that specification.
333:
419:
406:
1181:
1152:
824:
671:, workloads, environment info, etc.).
641:
324:
1146:
850:
717:Performance testing web applications
346:
163:
90:adding citations to reliable sources
61:
20:
473:
354:
13:
789:Analyze Results, Tune, and Retest.
667:(including detailed scenarios and
516:
14:
1205:
292:
36:This article has multiple issues.
1093:Graphical user interface testing
815:Application Response Measurement
501:
315:
273:performing a specific number of
255:
168:
66:
25:
766:Configure the Test Environment.
663:Develop a detailed performance
624:Analytical Performance Modeling
304:
260:
244:In software quality assurance,
77:needs additional citations for
44:or discuss these issues on the
726:Identify the Test Environment.
711:
101:"Software performance testing"
1:
1139:
607:
1118:Software reliability testing
1057:Software performance testing
7:
796:
579:Server hardware Parameters
194:the claims made and adding
10:
1210:
960:Testing types, techniques,
773:Implement the Test Design.
436:Performance specifications
391:Concurrency and throughput
1085:
959:
928:
858:
541:
364:Setting performance goals
1103:Orthogonal array testing
1052:Smoke testing (software)
1022:Dynamic program analysis
550:
1153:Thakur, Nitish (2012).
809:Web server benchmarking
511:user acceptance testing
443:Performance Engineering
752:Plan and Design Tests.
728:Identify the physical
571:Performance monitoring
1194:Software optimization
982:Compatibility testing
803:Benchmark (computing)
656:Develop a high-level
559:Performance scripting
334:Configuration testing
1027:Installation testing
1017:Differential testing
451:Windows Task Manager
420:Render response time
407:Server response time
86:improve this article
1012:Development testing
1007:Destructive testing
997:Conformance testing
941:Integration testing
886:Model-based testing
876:Exploratory testing
592:Network utilization
246:performance testing
1067:Symbolic execution
1042:Regression testing
1002:Continuous testing
992:Concurrent testing
936:Acceptance testing
859:The "box" approach
783:results collection
642:Tasks to undertake
600:use tools such as
598:software engineers
586:Memory Utilization
445:for more details.
325:Breakpoint testing
301:expected maximum.
283:application server
179:possibly contains
1136:
1135:
1077:Usability testing
903:White-box testing
871:All-pairs testing
866:Black-box testing
779:Execute the Test.
347:Isolation testing
242:
241:
234:
224:
223:
216:
181:original research
162:
161:
154:
136:
59:
1201:
1189:Software testing
1173:
1172:
1170:
1168:
1159:
1150:
1047:Security testing
1032:Negative testing
987:Concolic testing
913:Mutation testing
898:Grey-box testing
891:Scenario testing
852:Software testing
845:
838:
831:
822:
821:
730:test environment
686:proof-of-concept
589:Disk utilization
474:Questions to ask
355:Internet testing
237:
230:
219:
212:
208:
205:
199:
196:inline citations
172:
171:
164:
157:
150:
146:
143:
137:
135:
94:
70:
62:
51:
29:
28:
21:
1209:
1208:
1204:
1203:
1202:
1200:
1199:
1198:
1179:
1178:
1177:
1176:
1166:
1164:
1157:
1151:
1147:
1142:
1137:
1132:
1081:
1072:Test automation
961:
955:
924:
854:
849:
799:
719:
714:
644:
610:
583:CPU Utilization
573:
561:
553:
544:
519:
517:Test conditions
504:
476:
438:
422:
409:
393:
366:
357:
349:
336:
327:
318:
307:
295:
263:
258:
238:
227:
226:
225:
220:
209:
203:
200:
185:
173:
169:
158:
147:
141:
138:
95:
93:
83:
71:
30:
26:
17:
12:
11:
5:
1207:
1197:
1196:
1191:
1175:
1174:
1144:
1143:
1141:
1138:
1134:
1133:
1131:
1130:
1125:
1123:Stress testing
1120:
1115:
1110:
1105:
1100:
1098:Manual testing
1095:
1089:
1087:
1083:
1082:
1080:
1079:
1074:
1069:
1064:
1062:Stress testing
1059:
1054:
1049:
1044:
1039:
1037:Random testing
1034:
1029:
1024:
1019:
1014:
1009:
1004:
999:
994:
989:
984:
979:
974:
968:
966:
957:
956:
954:
953:
948:
946:System testing
943:
938:
932:
930:
929:Testing levels
926:
925:
923:
922:
921:
920:
918:Static testing
915:
910:
900:
895:
894:
893:
888:
883:
878:
873:
862:
860:
856:
855:
848:
847:
840:
833:
825:
819:
818:
812:
806:
798:
795:
794:
793:
786:
776:
770:
763:
749:
742:
718:
715:
713:
710:
709:
708:
705:
702:
699:
695:
692:
689:
682:
679:
672:
661:
654:
651:
643:
640:
618:stress testing
609:
606:
594:
593:
590:
587:
584:
572:
569:
560:
557:
552:
549:
543:
540:
531:infrastructure
518:
515:
503:
500:
499:
498:
495:
492:
489:
486:
483:
475:
472:
437:
434:
421:
418:
408:
405:
392:
389:
380:
379:
376:
373:
365:
362:
356:
353:
348:
345:
341:load-balancing
335:
332:
326:
323:
317:
314:
306:
303:
298:Stress testing
294:
293:Stress testing
291:
262:
259:
257:
254:
240:
239:
222:
221:
176:
174:
167:
160:
159:
74:
72:
65:
60:
34:
33:
31:
24:
15:
9:
6:
4:
3:
2:
1206:
1195:
1192:
1190:
1187:
1186:
1184:
1163:
1156:
1149:
1145:
1129:
1126:
1124:
1121:
1119:
1116:
1114:
1111:
1109:
1106:
1104:
1101:
1099:
1096:
1094:
1091:
1090:
1088:
1084:
1078:
1075:
1073:
1070:
1068:
1065:
1063:
1060:
1058:
1055:
1053:
1050:
1048:
1045:
1043:
1040:
1038:
1035:
1033:
1030:
1028:
1025:
1023:
1020:
1018:
1015:
1013:
1010:
1008:
1005:
1003:
1000:
998:
995:
993:
990:
988:
985:
983:
980:
978:
975:
973:
970:
969:
967:
965:
958:
952:
949:
947:
944:
942:
939:
937:
934:
933:
931:
927:
919:
916:
914:
911:
909:
906:
905:
904:
901:
899:
896:
892:
889:
887:
884:
882:
879:
877:
874:
872:
869:
868:
867:
864:
863:
861:
857:
853:
846:
841:
839:
834:
832:
827:
826:
823:
816:
813:
810:
807:
804:
801:
800:
790:
787:
784:
780:
777:
774:
771:
767:
764:
761:
757:
754:Identify key
753:
750:
746:
743:
740:
736:
731:
727:
724:
723:
722:
706:
703:
700:
696:
693:
690:
687:
683:
680:
677:
673:
670:
666:
662:
659:
655:
652:
649:
648:
647:
639:
637:
633:
629:
625:
621:
619:
614:
605:
603:
599:
591:
588:
585:
582:
581:
580:
577:
568:
564:
556:
548:
539:
537:
532:
528:
523:
514:
512:
507:
502:Prerequisites
496:
493:
490:
487:
484:
481:
480:
479:
471:
467:
465:
460:
454:
452:
446:
444:
433:
431:
427:
417:
415:
404:
401:
399:
388:
386:
385:business case
377:
374:
371:
370:
369:
361:
352:
344:
342:
331:
322:
316:Spike testing
313:
311:
302:
299:
290:
288:
284:
280:
276:
272:
267:
256:Testing types
253:
250:
247:
236:
233:
218:
215:
207:
197:
193:
189:
183:
182:
177:This article
175:
166:
165:
156:
153:
145:
142:February 2024
134:
131:
127:
124:
120:
117:
113:
110:
106:
103: –
102:
98:
97:Find sources:
91:
87:
81:
80:
75:This article
73:
69:
64:
63:
58:
56:
49:
48:
43:
42:
37:
32:
23:
22:
19:
1165:. Retrieved
1161:
1148:
1113:Soak testing
1108:Pair testing
1056:
951:Unit testing
881:Fuzz testing
788:
778:
772:
765:
751:
744:
725:
720:
645:
630:, disk I/O,
622:
615:
611:
595:
578:
574:
565:
562:
554:
545:
524:
520:
508:
505:
477:
468:
455:
447:
439:
430:test scripts
423:
414:load testing
410:
402:
394:
381:
367:
358:
350:
337:
328:
319:
310:Soak testing
308:
305:Soak testing
296:
275:transactions
266:Load testing
264:
261:Load testing
251:
245:
243:
228:
210:
201:
178:
148:
139:
129:
122:
115:
108:
96:
84:Please help
79:verification
76:
52:
45:
39:
38:Please help
35:
18:
1167:February 3,
1128:Web testing
972:A/B testing
908:API testing
735:test design
712:Methodology
398:concurrency
287:bottlenecks
271:application
204:August 2021
1183:Categories
1140:References
769:necessary.
739:life cycle
669:test cases
608:Technology
188:improve it
112:newspapers
41:improve it
977:Benchmark
792:more CPU.
756:scenarios
676:test tool
665:test plan
602:profilers
192:verifying
47:talk page
1086:See also
797:See also
760:simulate
684:Develop
279:database
964:tactics
674:Choose
459:routers
249:usage.
186:Please
126:scholar
542:Timing
128:
121:
114:
107:
99:
1158:(PDF)
551:Tools
133:JSTOR
119:books
1169:2024
962:and
698:etc.
678:(s).
658:plan
426:node
105:news
1162:IBM
636:WAN
632:LAN
628:CPU
536:PTE
527:SOA
190:by
88:by
1185::
1160:.
634:,
464:T1
343:.
281:,
50:.
1171:.
844:e
837:t
830:v
741:.
235:)
229:(
217:)
211:(
206:)
202:(
184:.
155:)
149:(
144:)
140:(
130:·
123:·
116:·
109:·
82:.
57:)
53:(
Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.
