523:. Some selected inputs might consist of a negative number, zero, and a positive number. When using these numbers to test software in this way, the developer generalizes the set of all reals into three numbers. This is a more efficient and manageable method, but more prone to failure. Generalizing test cases is an example of just one technique to deal with failure—specifically, failure due to invalid user input. Systems generally may also fail due to other reasons as well, such as disconnecting from a network.
1272:
1279:
555:
generally only need one kidney, but having a second kidney allows room for failure. This same principle may be taken to apply to software, but there are some challenges. When applying the principle of redundancy to computer science, blindly adding code is not suggested. Blindly adding code introduces
621:
The programmer assumes users will try incorrect, bogus and malformed inputs. As a consequence, the programmer returns to the user an unambiguous, intuitive error message that does not require looking up error codes. The error message should try to be as accurate as possible without being misleading
667:
Robust machine learning typically refers to the robustness of machine learning algorithms. For a machine learning algorithm to be considered robust, either the testing error has to be consistent with the training error, or the performance is stable after adding some noise to the dataset. Recently,
518:
In general, building robust systems that encompass every point of possible failure is difficult because of the vast quantity of possible inputs and input combinations. Since all inputs and input combinations would require too much time to test, developers cannot run through all cases exhaustively.
657:
Very often, code is modified and may introduce a possibility that an "impossible" case occurs. Impossible cases are therefore assumed to be highly unlikely instead. The developer thinks about how to handle the case that is highly unlikely, and implements the handling
676:
Robust network design is the study of network design in the face of variable or uncertain demands. In a sense, robustness in network design is broad just like robustness in software design because of the vast possibilities of changes or inputs.
646:
are correctly built, users use them without finding loopholes to modify the interface. The interface should already be correctly implemented, so the user does not need to make modifications. The user therefore focuses solely on their own
753:
1990. IEEE Standard
Glossary of Software Engineering Terminology, IEEE Std 610.12-1990 defines robustness as "The degree to which a system or component can function correctly in the presence of invalid inputs or stressful environmental
526:
Regardless, complex systems should still handle any errors encountered gracefully. There are many examples of such successful systems. Some of the most robust systems are evolvable and can be easily adapted to new situations.
596:
Robust programming is a style of programming that focuses on handling unexpected termination and unexpected actions. It requires code to handle these terminations and actions gracefully by displaying accurate and unambiguous
556:
more errors, makes the system more complex, and renders it harder to understand. Code that does not provide any reinforcement to the already existing code is unwanted. The new code must instead possess equivalent
572:, and increases in size, it becomes more complex. Thus, when making a more redundant system, the system also becomes more complex and developers must consider balancing redundancy with complexity.
668:
consistently with their rise in popularity, there has been an increasing interest in the robustness of neural networks. This is particularly due their vulnerability to adverserial attacks.
613:
When building software, the programmer assumes users are out to break their code. The programmer also assumes that their own written code may fail or work incorrectly.
642:
to data structures. This information should be hidden from the user so that the user does not accidentally modify them and introduce a bug in the code. When such
535:
Programs and software are tools focused on a very specific task, and thus are not generalized and flexible. However, observations in systems such as the
1315:
765:
931:
1065:
461:
643:
543:
demonstrate adaptation to their environments. One of the ways biological systems adapt to environments is through the use of
1308:
1471:
506:, are essential to showing robustness since this type of testing involves invalid or unexpected inputs. Alternatively,
208:
91:
1120:
1282:
1231:
1023:
1787:
652:
583:. One of the main reasons why there is no focus on robustness today is because it is hard to do in a general way.
1605:
1301:
560:, so that if a function is broken, another providing the same function can replace it, using manual or automated
1226:
1663:
1176:
1058:
639:
375:
818:
1196:
575:
Currently, computer science practices do not focus on building robust systems. Rather, they tend to focus on
510:
can be used to test robustness. Various commercial products perform robustness testing of software analysis.
454:
246:
242:
847:"Importance of Making Generalized Testcases - Software Testing Club - An Online Software Testing Community"
569:
1413:
1247:
1145:
846:
95:
1658:
1383:
1257:
704:
520:
71:
1100:
951:
Li, Linyi; Xie, Tao; Li, Bo (9 September 2022). "SoK: Certified
Robustness for Deep Neural Networks".
1792:
1441:
1105:
1051:
183:
1577:
1519:
1446:
1378:
1252:
544:
447:
434:
324:
188:
158:
104:
564:. To do so, the new code must know how and when to accommodate the failure point. This means more
1582:
1537:
1476:
1408:
1368:
1166:
499:
212:
163:
126:
63:
490:
and cope with erroneous input. Robustness can encompass many areas of computer science, such as
1668:
1509:
1201:
1186:
1181:
724:
580:
557:
425:
519:
Instead, the developer will try to generalize such cases. For example, imagine inputting some
1504:
1481:
1398:
699:
491:
487:
420:
353:
348:
216:
1008:
Proceedings of the 19th international symposium on
Software testing and analysis - ISSTA '10
1640:
1600:
1542:
1130:
1110:
8:
1620:
1514:
1456:
1353:
631:
429:
404:
387:
296:
196:
130:
53:
1766:
1761:
1716:
1673:
1635:
1592:
1451:
1029:
1000:
952:
906:
561:
399:
320:
316:
258:
250:
121:
868:
1678:
1650:
1373:
1343:
1135:
1019:
540:
232:
192:
83:
67:
36:
1688:
1466:
1393:
1074:
1033:
1011:
784:
780:
475:
365:
340:
308:
266:
204:
79:
1736:
1683:
1572:
1529:
1461:
1418:
1403:
1363:
1095:
694:
507:
336:
332:
262:
200:
154:
116:
99:
87:
58:
1726:
1703:
1630:
1388:
1324:
1140:
635:
395:
328:
282:
75:
22:
1781:
1731:
1693:
1552:
598:
483:
271:
138:
1015:
972:
1625:
1557:
1486:
1221:
1171:
1001:"Automatically identifying critical input regions and code in applications"
932:"What is the definition of the robustness of a machine learning algorithm?"
503:
391:
221:
142:
1756:
1751:
1741:
1711:
1610:
1562:
1496:
1125:
1115:
576:
495:
415:
379:
370:
344:
312:
291:
286:
254:
146:
601:. These error messages allow the user to more easily debug the program.
1547:
1433:
178:
134:
1615:
1428:
1338:
1161:
383:
174:
41:
1721:
1043:
957:
536:
304:
1293:
568:
needs to be added to the system. But as a system adds more logic,
1746:
300:
237:
16:
Ability of a computer system to cope with errors during execution
866:
548:
764:
Baker, Jack W.; Schubert, Matthias; Faber, Michael H. (2008).
565:
552:
685:
There exist algorithms that tolerate errors in the input.
622:
to the user, so that the problem can be fixed with ease.
901:
899:
897:
895:
893:
891:
999:Carbin, Michael; Rinard, Martin C. (12 July 2010).
816:
763:
888:
482:is the ability of a computer system to cope with
1779:
725:"A Model-Based Approach for Robustness Testing"
1309:
1059:
455:
998:
929:
547:. Many organs are redundant in humans. The
1316:
1302:
1066:
1052:
867:Agents on the wEb : Robust Software.
662:
462:
448:
956:
671:
950:
817:Gerald Jay Sussman (January 13, 2007).
1780:
844:
749:
747:
1297:
1047:
812:
810:
808:
806:
804:
802:
800:
591:
1278:
1073:
680:
1323:
744:
13:
1472:Evolutionary developmental biology
869:"Building Robust Systems an essay"
819:"Building Robust Systems an essay"
797:
209:Evolutionary developmental biology
14:
1804:
1232:List of system quality attributes
766:"On the assessment of robustness"
1277:
1271:
1270:
630:Users should not gain access to
1606:Ordinary differential equations
992:
965:
513:
376:Ordinary differential equations
1664:Partial differential equations
944:
923:
860:
838:
785:10.1016/j.strusafe.2006.11.004
757:
717:
247:Partial differential equations
1:
710:
604:
530:
502:. Formal techniques, such as
7:
1414:Particle swarm optimization
1248:Software quality management
1227:Non-functional requirements
845:Joseph, Joby (2009-09-21).
688:
96:Particle swarm optimization
10:
1809:
1659:Reaction-diffusion systems
1384:Self-organized criticality
1258:Software quality assurance
705:Non-functional requirement
243:Reaction–diffusion systems
72:Self-organized criticality
1702:
1649:
1591:
1528:
1495:
1442:Artificial neural network
1427:
1352:
1331:
1266:
1240:
1214:
1154:
1088:
1081:
184:Artificial neural network
1520:Evolutionary game theory
1447:Evolutionary computation
1379:Collective consciousness
1253:Software quality control
586:
435:Evolutionary game theory
325:Second-order cybernetics
189:Evolutionary computation
105:Collective consciousness
1788:Reliability engineering
1538:Social network analysis
1477:Artificial intelligence
1409:Ant colony optimization
1369:Collective intelligence
1016:10.1145/1831708.1831713
1010:. ACM. pp. 37–48.
973:"Robust Network Design"
849:. Software Testing Club
663:Robust machine learning
500:Robust Security Network
496:robust machine learning
213:Artificial intelligence
127:Social network analysis
92:Ant colony optimization
64:Collective intelligence
1669:Dissipative structures
1510:Rational choice theory
426:Rational choice theory
251:Dissipative structures
1482:Evolutionary robotics
1399:Agent-based modelling
700:Defensive programming
672:Robust network design
551:is one such example.
354:Theory of computation
217:Evolutionary robotics
84:Agent-based modelling
1641:Coupled map lattices
1601:Time series analysis
1543:Small-world networks
907:"Robust Programming"
826:Groups.csail.mit.edu
627:Dangerous implements
405:Coupled map lattices
371:Time series analysis
131:Small-world networks
1621:Population dynamics
1515:Bounded rationality
1457:Genetic programming
1215:Standards and lists
430:Bounded rationality
388:Population dynamics
297:Conversation theory
197:Genetic programming
122:Scale-free networks
54:Collective behavior
1767:Computation theory
1762:Information theory
1717:Operationalization
1593:Nonlinear dynamics
1505:Prisoner's dilemma
1452:Genetic algorithms
930:El Sayed Mahmoud.
911:Nob.cs.ucdavis.edu
592:Robust programming
562:software diversity
541:biological systems
492:robust programming
421:Prisoner's dilemma
366:Nonlinear dynamics
321:Operationalization
317:Information theory
193:Genetic algorithms
1775:
1774:
1679:Cellular automata
1651:Pattern formation
1583:Adaptive networks
1374:Collective action
1344:Self-organization
1291:
1290:
1210:
1209:
1136:Understandability
773:Structural Safety
681:Robust algorithms
472:
471:
259:Cellular automata
233:Pattern formation
164:Adaptive networks
68:Collective action
37:Self-organization
1800:
1793:Software quality
1689:Self-replication
1578:Dynamic networks
1467:Machine learning
1394:Phase transition
1318:
1311:
1304:
1295:
1294:
1281:
1280:
1274:
1273:
1086:
1085:
1075:Software quality
1068:
1061:
1054:
1045:
1044:
1038:
1037:
1005:
996:
990:
989:
987:
986:
977:
969:
963:
962:
960:
948:
942:
941:
939:
938:
927:
921:
920:
918:
917:
903:
886:
885:
883:
882:
873:
864:
858:
857:
855:
854:
842:
836:
835:
833:
832:
823:
814:
795:
794:
792:
791:
770:
761:
755:
751:
742:
741:
739:
738:
729:
721:
476:computer science
464:
457:
450:
341:Systems thinking
267:Self-replication
205:Machine learning
159:Dynamic networks
80:Phase transition
19:
18:
1808:
1807:
1803:
1802:
1801:
1799:
1798:
1797:
1778:
1777:
1776:
1771:
1737:System dynamics
1698:
1684:Spatial ecology
1645:
1587:
1573:Systems biology
1524:
1491:
1462:Artificial life
1432:
1423:
1419:Swarm behaviour
1404:Synchronization
1364:Social dynamics
1355:
1348:
1327:
1325:Complex systems
1322:
1292:
1287:
1262:
1236:
1206:
1150:
1101:Maintainability
1077:
1072:
1042:
1041:
1026:
1003:
997:
993:
984:
982:
975:
971:
970:
966:
949:
945:
936:
934:
928:
924:
915:
913:
905:
904:
889:
880:
878:
871:
865:
861:
852:
850:
843:
839:
830:
828:
821:
815:
798:
789:
787:
768:
762:
758:
752:
745:
736:
734:
727:
723:
722:
718:
713:
695:Fault tolerance
691:
683:
674:
665:
636:data structures
607:
594:
589:
533:
516:
508:fault injection
468:
439:
438:
437:
432:
428:
423:
418:
408:
407:
402:
398:
394:
390:
386:
382:
378:
373:
368:
358:
357:
356:
351:
347:
343:
339:
337:Systems science
335:
333:System dynamics
331:
327:
323:
319:
315:
311:
307:
303:
299:
294:
289:
275:
274:
269:
265:
263:Spatial ecology
261:
257:
253:
249:
245:
240:
235:
225:
224:
219:
215:
211:
207:
203:
201:Artificial life
199:
195:
191:
186:
181:
167:
166:
161:
157:
155:Systems biology
153:
149:
145:
141:
137:
133:
129:
124:
119:
109:
108:
107:
102:
100:Swarm behaviour
98:
94:
90:
88:Synchronization
86:
82:
78:
74:
70:
66:
61:
59:Social dynamics
56:
46:
45:
44:
39:
23:Complex systems
17:
12:
11:
5:
1806:
1796:
1795:
1790:
1773:
1772:
1770:
1769:
1764:
1759:
1754:
1749:
1744:
1739:
1734:
1729:
1727:Self-reference
1724:
1719:
1714:
1708:
1706:
1704:Systems theory
1700:
1699:
1697:
1696:
1691:
1686:
1681:
1676:
1671:
1666:
1661:
1655:
1653:
1647:
1646:
1644:
1643:
1638:
1633:
1631:Multistability
1628:
1623:
1618:
1613:
1608:
1603:
1597:
1595:
1589:
1588:
1586:
1585:
1580:
1575:
1570:
1565:
1560:
1555:
1550:
1545:
1540:
1534:
1532:
1526:
1525:
1523:
1522:
1517:
1512:
1507:
1501:
1499:
1493:
1492:
1490:
1489:
1484:
1479:
1474:
1469:
1464:
1459:
1454:
1449:
1444:
1438:
1436:
1425:
1424:
1422:
1421:
1416:
1411:
1406:
1401:
1396:
1391:
1389:Herd mentality
1386:
1381:
1376:
1371:
1366:
1360:
1358:
1350:
1349:
1347:
1346:
1341:
1335:
1333:
1329:
1328:
1321:
1320:
1313:
1306:
1298:
1289:
1288:
1286:
1285:
1275:
1267:
1264:
1263:
1261:
1260:
1255:
1250:
1244:
1242:
1238:
1237:
1235:
1234:
1229:
1224:
1218:
1216:
1212:
1211:
1208:
1207:
1205:
1204:
1199:
1194:
1189:
1184:
1179:
1174:
1169:
1164:
1158:
1156:
1152:
1151:
1149:
1148:
1143:
1141:Loose coupling
1138:
1133:
1128:
1123:
1118:
1113:
1108:
1103:
1098:
1092:
1090:
1083:
1079:
1078:
1071:
1070:
1063:
1056:
1048:
1040:
1039:
1024:
991:
964:
943:
922:
887:
859:
837:
796:
779:(3): 253–267.
756:
743:
715:
714:
712:
709:
708:
707:
702:
697:
690:
687:
682:
679:
673:
670:
664:
661:
660:
659:
655:
649:
648:
628:
624:
623:
619:
615:
614:
611:
606:
603:
599:error messages
593:
590:
588:
585:
532:
529:
521:integer values
515:
512:
470:
469:
467:
466:
459:
452:
444:
441:
440:
419:
414:
413:
410:
409:
396:Multistability
369:
364:
363:
360:
359:
329:Self-reference
290:
283:Systems theory
281:
280:
277:
276:
236:
231:
230:
227:
226:
182:
173:
172:
169:
168:
120:
115:
114:
111:
110:
76:Herd mentality
57:
52:
51:
48:
47:
40:
35:
34:
31:
30:
26:
25:
15:
9:
6:
4:
3:
2:
1805:
1794:
1791:
1789:
1786:
1785:
1783:
1768:
1765:
1763:
1760:
1758:
1755:
1753:
1750:
1748:
1745:
1743:
1740:
1738:
1735:
1733:
1732:Goal-oriented
1730:
1728:
1725:
1723:
1720:
1718:
1715:
1713:
1710:
1709:
1707:
1705:
1701:
1695:
1694:Geomorphology
1692:
1690:
1687:
1685:
1682:
1680:
1677:
1675:
1672:
1670:
1667:
1665:
1662:
1660:
1657:
1656:
1654:
1652:
1648:
1642:
1639:
1637:
1634:
1632:
1629:
1627:
1624:
1622:
1619:
1617:
1614:
1612:
1609:
1607:
1604:
1602:
1599:
1598:
1596:
1594:
1590:
1584:
1581:
1579:
1576:
1574:
1571:
1569:
1566:
1564:
1561:
1559:
1556:
1554:
1551:
1549:
1546:
1544:
1541:
1539:
1536:
1535:
1533:
1531:
1527:
1521:
1518:
1516:
1513:
1511:
1508:
1506:
1503:
1502:
1500:
1498:
1494:
1488:
1485:
1483:
1480:
1478:
1475:
1473:
1470:
1468:
1465:
1463:
1460:
1458:
1455:
1453:
1450:
1448:
1445:
1443:
1440:
1439:
1437:
1435:
1430:
1426:
1420:
1417:
1415:
1412:
1410:
1407:
1405:
1402:
1400:
1397:
1395:
1392:
1390:
1387:
1385:
1382:
1380:
1377:
1375:
1372:
1370:
1367:
1365:
1362:
1361:
1359:
1357:
1351:
1345:
1342:
1340:
1337:
1336:
1334:
1330:
1326:
1319:
1314:
1312:
1307:
1305:
1300:
1299:
1296:
1284:
1276:
1269:
1268:
1265:
1259:
1256:
1254:
1251:
1249:
1246:
1245:
1243:
1239:
1233:
1230:
1228:
1225:
1223:
1220:
1219:
1217:
1213:
1203:
1200:
1198:
1195:
1193:
1190:
1188:
1185:
1183:
1180:
1178:
1175:
1173:
1170:
1168:
1165:
1163:
1160:
1159:
1157:
1153:
1147:
1146:Orthogonality
1144:
1142:
1139:
1137:
1134:
1132:
1129:
1127:
1124:
1122:
1119:
1117:
1114:
1112:
1109:
1107:
1104:
1102:
1099:
1097:
1094:
1093:
1091:
1087:
1084:
1080:
1076:
1069:
1064:
1062:
1057:
1055:
1050:
1049:
1046:
1035:
1031:
1027:
1025:9781605588230
1021:
1017:
1013:
1009:
1002:
995:
981:
974:
968:
959:
954:
947:
933:
926:
912:
908:
902:
900:
898:
896:
894:
892:
877:
870:
863:
848:
841:
827:
820:
813:
811:
809:
807:
805:
803:
801:
786:
782:
778:
774:
767:
760:
750:
748:
733:
726:
720:
716:
706:
703:
701:
698:
696:
693:
692:
686:
678:
669:
656:
654:
651:
650:
645:
641:
637:
633:
629:
626:
625:
620:
617:
616:
612:
609:
608:
602:
600:
584:
582:
578:
573:
571:
567:
563:
559:
558:functionality
554:
550:
546:
542:
538:
528:
524:
522:
511:
509:
505:
501:
497:
493:
489:
485:
481:
477:
465:
460:
458:
453:
451:
446:
445:
443:
442:
436:
433:
431:
427:
422:
417:
412:
411:
406:
403:
401:
397:
393:
389:
385:
381:
377:
372:
367:
362:
361:
355:
352:
350:
346:
342:
338:
334:
330:
326:
322:
318:
314:
310:
309:Goal-oriented
306:
302:
298:
293:
288:
284:
279:
278:
273:
272:Geomorphology
270:
268:
264:
260:
256:
252:
248:
244:
239:
234:
229:
228:
223:
220:
218:
214:
210:
206:
202:
198:
194:
190:
185:
180:
176:
171:
170:
165:
162:
160:
156:
152:
148:
144:
140:
136:
132:
128:
123:
118:
113:
112:
106:
103:
101:
97:
93:
89:
85:
81:
77:
73:
69:
65:
60:
55:
50:
49:
43:
38:
33:
32:
28:
27:
24:
21:
20:
1567:
1558:Graph theory
1487:Evolvability
1222:ISO/IEC 9126
1191:
1172:Adaptability
1007:
994:
983:. Retrieved
980:Math.mit.edu
979:
967:
946:
935:. Retrieved
925:
914:. Retrieved
910:
879:. Retrieved
875:
862:
851:. Retrieved
840:
829:. Retrieved
825:
788:. Retrieved
776:
772:
759:
735:. Retrieved
731:
719:
684:
675:
666:
658:accordingly.
653:Can't happen
595:
574:
534:
525:
517:
514:Introduction
504:fuzz testing
479:
473:
424:
374:
295:
241:
222:Evolvability
187:
150:
143:Graph theory
125:
62:
1757:Autopoiesis
1752:Cybernetics
1742:Sensemaking
1712:Homeostasis
1674:Percolation
1636:Bifurcation
1611:Phase space
1497:Game theory
1354:Collective
1177:Correctness
1167:Reliability
1131:Testability
1126:Scalability
1121:Readability
1116:Reusability
1111:Portability
1106:Flexibility
754:conditions"
732:Dl.ifip.org
577:scalability
416:Game theory
400:Bifurcation
380:Phase space
345:Sensemaking
313:Homeostasis
292:Autopoiesis
287:cybernetics
255:Percolation
1782:Categories
1568:Robustness
1548:Centrality
1434:adaptation
1332:Background
1192:Robustness
1187:Efficiency
985:2016-11-13
958:2009.04131
937:2016-11-13
916:2016-11-13
881:2016-11-13
876:Cse.sc.edu
853:2016-11-13
831:2016-11-13
790:2016-11-13
737:2016-11-13
711:References
644:interfaces
605:Principles
581:efficiency
570:components
545:redundancy
531:Challenges
480:robustness
384:Attractors
179:adaptation
151:Robustness
135:Centrality
1616:Attractor
1429:Evolution
1339:Emergence
1241:Processes
1162:Usability
1082:Qualities
632:libraries
618:Stupidity
488:execution
175:Evolution
42:Emergence
1722:Feedback
1530:Networks
1356:behavior
1197:Security
1182:Accuracy
1155:External
1089:Internal
689:See also
640:pointers
610:Paranoia
537:internet
305:Feedback
238:Fractals
117:Networks
1747:Entropy
1563:Scaling
1283:Commons
1034:1147058
486:during
349:Variety
301:Entropy
147:Scaling
1553:Motifs
1202:Safety
1032:
1022:
553:Humans
549:kidney
498:, and
484:errors
139:Motifs
29:Topics
1626:Chaos
1030:S2CID
1004:(PDF)
976:(PDF)
953:arXiv
872:(PDF)
822:(PDF)
769:(PDF)
728:(PDF)
647:code.
638:, or
587:Areas
566:logic
392:Chaos
1431:and
1096:Size
1020:ISBN
579:and
285:and
177:and
1012:doi
781:doi
539:or
474:In
1784::
1028:.
1018:.
1006:.
978:.
909:.
890:^
874:.
824:.
799:^
777:30
775:.
771:.
746:^
730:.
634:,
494:,
478:,
1317:e
1310:t
1303:v
1067:e
1060:t
1053:v
1036:.
1014::
988:.
961:.
955::
940:.
919:.
884:.
856:.
834:.
793:.
783::
740:.
463:e
456:t
449:v
Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.