534:. Some selected inputs might consist of a negative number, zero, and a positive number. When using these numbers to test software in this way, the developer generalizes the set of all reals into three numbers. This is a more efficient and manageable method, but more prone to failure. Generalizing test cases is an example of just one technique to deal with failure—specifically, failure due to invalid user input. Systems generally may also fail due to other reasons as well, such as disconnecting from a network.
1283:
1290:
566:
generally only need one kidney, but having a second kidney allows room for failure. This same principle may be taken to apply to software, but there are some challenges. When applying the principle of redundancy to computer science, blindly adding code is not suggested. Blindly adding code introduces
632:
The programmer assumes users will try incorrect, bogus and malformed inputs. As a consequence, the programmer returns to the user an unambiguous, intuitive error message that does not require looking up error codes. The error message should try to be as accurate as possible without being misleading
678:
Robust machine learning typically refers to the robustness of machine learning algorithms. For a machine learning algorithm to be considered robust, either the testing error has to be consistent with the training error, or the performance is stable after adding some noise to the dataset. Recently,
529:
In general, building robust systems that encompass every point of possible failure is difficult because of the vast quantity of possible inputs and input combinations. Since all inputs and input combinations would require too much time to test, developers cannot run through all cases exhaustively.
668:
Very often, code is modified and may introduce a possibility that an "impossible" case occurs. Impossible cases are therefore assumed to be highly unlikely instead. The developer thinks about how to handle the case that is highly unlikely, and implements the handling
687:
Robust network design is the study of network design in the face of variable or uncertain demands. In a sense, robustness in network design is broad just like robustness in software design because of the vast possibilities of changes or inputs.
657:
are correctly built, users use them without finding loopholes to modify the interface. The interface should already be correctly implemented, so the user does not need to make modifications. The user therefore focuses solely on their own
764:
1990. IEEE Standard
Glossary of Software Engineering Terminology, IEEE Std 610.12-1990 defines robustness as "The degree to which a system or component can function correctly in the presence of invalid inputs or stressful environmental
537:
Regardless, complex systems should still handle any errors encountered gracefully. There are many examples of such successful systems. Some of the most robust systems are evolvable and can be easily adapted to new situations.
607:
Robust programming is a style of programming that focuses on handling unexpected termination and unexpected actions. It requires code to handle these terminations and actions gracefully by displaying accurate and unambiguous
567:
more errors, makes the system more complex, and renders it harder to understand. Code that does not provide any reinforcement to the already existing code is unwanted. The new code must instead possess equivalent
583:, and increases in size, it becomes more complex. Thus, when making a more redundant system, the system also becomes more complex and developers must consider balancing redundancy with complexity.
679:
consistently with their rise in popularity, there has been an increasing interest in the robustness of neural networks. This is particularly due their vulnerability to adverserial attacks.
624:
When building software, the programmer assumes users are out to break their code. The programmer also assumes that their own written code may fail or work incorrectly.
653:
to data structures. This information should be hidden from the user so that the user does not accidentally modify them and introduce a bug in the code. When such
546:
Programs and software are tools focused on a very specific task, and thus are not generalized and flexible. However, observations in systems such as the
1326:
776:
942:
1076:
472:
654:
554:
demonstrate adaptation to their environments. One of the ways biological systems adapt to environments is through the use of
1319:
17:
1482:
517:, are essential to showing robustness since this type of testing involves invalid or unexpected inputs. Alternatively,
219:
102:
1131:
1293:
1242:
1034:
1798:
663:
594:. One of the main reasons why there is no focus on robustness today is because it is hard to do in a general way.
1616:
1312:
571:, so that if a function is broken, another providing the same function can replace it, using manual or automated
1237:
1674:
1187:
1069:
650:
386:
829:
1207:
586:
Currently, computer science practices do not focus on building robust systems. Rather, they tend to focus on
521:
can be used to test robustness. Various commercial products perform robustness testing of software analysis.
465:
257:
253:
858:"Importance of Making Generalized Testcases - Software Testing Club - An Online Software Testing Community"
580:
1424:
1258:
1156:
857:
106:
1669:
1394:
1268:
715:
531:
82:
1111:
962:
Li, Linyi; Xie, Tao; Li, Bo (9 September 2022). "SoK: Certified
Robustness for Deep Neural Networks".
1803:
1452:
1116:
1062:
194:
1588:
1530:
1457:
1389:
1263:
555:
458:
445:
335:
199:
169:
115:
575:. To do so, the new code must know how and when to accommodate the failure point. This means more
1593:
1548:
1487:
1419:
1379:
1177:
510:
223:
174:
137:
74:
501:
and cope with erroneous input. Robustness can encompass many areas of computer science, such as
1679:
1520:
1212:
1197:
1192:
735:
591:
568:
436:
530:
Instead, the developer will try to generalize such cases. For example, imagine inputting some
1515:
1492:
1409:
710:
502:
498:
431:
364:
359:
227:
1019:
Proceedings of the 19th international symposium on
Software testing and analysis - ISSTA '10
1651:
1611:
1553:
1141:
1121:
8:
1631:
1525:
1467:
1364:
642:
440:
415:
398:
307:
207:
141:
64:
1777:
1772:
1727:
1684:
1646:
1603:
1462:
1040:
1011:
963:
917:
572:
410:
331:
327:
269:
261:
132:
879:
1689:
1661:
1384:
1354:
1146:
1030:
551:
243:
203:
94:
78:
47:
1699:
1477:
1404:
1085:
1044:
1022:
795:
791:
486:
376:
351:
319:
277:
215:
90:
1747:
1694:
1583:
1540:
1472:
1429:
1414:
1374:
1106:
705:
518:
347:
343:
273:
211:
165:
127:
110:
98:
69:
1737:
1714:
1641:
1399:
1335:
1151:
646:
406:
339:
293:
86:
33:
1792:
1742:
1704:
1563:
609:
494:
282:
149:
1026:
983:
1636:
1568:
1497:
1232:
1182:
1012:"Automatically identifying critical input regions and code in applications"
943:"What is the definition of the robustness of a machine learning algorithm?"
514:
402:
232:
153:
1767:
1762:
1752:
1722:
1621:
1573:
1507:
1136:
1126:
587:
506:
426:
390:
381:
355:
323:
302:
297:
265:
157:
612:. These error messages allow the user to more easily debug the program.
1558:
1444:
189:
145:
1626:
1439:
1349:
1172:
394:
185:
52:
1732:
1054:
968:
547:
315:
1304:
579:
needs to be added to the system. But as a system adds more logic,
1757:
311:
248:
27:
Ability of a computer system to cope with errors during execution
877:
559:
775:
Baker, Jack W.; Schubert, Matthias; Faber, Michael H. (2008).
576:
563:
696:
There exist algorithms that tolerate errors in the input.
633:
to the user, so that the problem can be fixed with ease.
912:
910:
908:
906:
904:
902:
1010:Carbin, Michael; Rinard, Martin C. (12 July 2010).
827:
774:
899:
493:is the ability of a computer system to cope with
1790:
736:"A Model-Based Approach for Robustness Testing"
1320:
1070:
466:
1009:
940:
558:. Many organs are redundant in humans. The
1327:
1313:
1077:
1063:
878:Agents on the wEb : Robust Software.
673:
473:
459:
967:
682:
961:
828:Gerald Jay Sussman (January 13, 2007).
14:
1791:
855:
760:
758:
1308:
1058:
823:
821:
819:
817:
815:
813:
811:
602:
1289:
1084:
691:
1334:
755:
24:
1483:Evolutionary developmental biology
880:"Building Robust Systems an essay"
830:"Building Robust Systems an essay"
808:
220:Evolutionary developmental biology
25:
1815:
1243:List of system quality attributes
777:"On the assessment of robustness"
1288:
1282:
1281:
641:Users should not gain access to
1617:Ordinary differential equations
1003:
976:
524:
387:Ordinary differential equations
1675:Partial differential equations
955:
934:
871:
849:
796:10.1016/j.strusafe.2006.11.004
768:
728:
258:Partial differential equations
13:
1:
721:
615:
541:
513:. Formal techniques, such as
7:
1425:Particle swarm optimization
1259:Software quality management
1238:Non-functional requirements
856:Joseph, Joby (2009-09-21).
699:
107:Particle swarm optimization
10:
1820:
1670:Reaction-diffusion systems
1395:Self-organized criticality
1269:Software quality assurance
716:Non-functional requirement
254:Reaction–diffusion systems
83:Self-organized criticality
1713:
1660:
1602:
1539:
1506:
1453:Artificial neural network
1438:
1363:
1342:
1277:
1251:
1225:
1165:
1099:
1092:
195:Artificial neural network
1531:Evolutionary game theory
1458:Evolutionary computation
1390:Collective consciousness
1264:Software quality control
597:
446:Evolutionary game theory
336:Second-order cybernetics
200:Evolutionary computation
116:Collective consciousness
1799:Reliability engineering
1549:Social network analysis
1488:Artificial intelligence
1420:Ant colony optimization
1380:Collective intelligence
1027:10.1145/1831708.1831713
1021:. ACM. pp. 37–48.
984:"Robust Network Design"
860:. Software Testing Club
674:Robust machine learning
511:Robust Security Network
507:robust machine learning
224:Artificial intelligence
138:Social network analysis
103:Ant colony optimization
75:Collective intelligence
1680:Dissipative structures
1521:Rational choice theory
437:Rational choice theory
262:Dissipative structures
1493:Evolutionary robotics
1410:Agent-based modelling
711:Defensive programming
683:Robust network design
562:is one such example.
365:Theory of computation
228:Evolutionary robotics
95:Agent-based modelling
1652:Coupled map lattices
1612:Time series analysis
1554:Small-world networks
918:"Robust Programming"
837:Groups.csail.mit.edu
638:Dangerous implements
416:Coupled map lattices
382:Time series analysis
142:Small-world networks
18:Numerical robustness
1632:Population dynamics
1526:Bounded rationality
1468:Genetic programming
1226:Standards and lists
441:Bounded rationality
399:Population dynamics
308:Conversation theory
208:Genetic programming
133:Scale-free networks
65:Collective behavior
1778:Computation theory
1773:Information theory
1728:Operationalization
1604:Nonlinear dynamics
1516:Prisoner's dilemma
1463:Genetic algorithms
941:El Sayed Mahmoud.
922:Nob.cs.ucdavis.edu
603:Robust programming
573:software diversity
552:biological systems
503:robust programming
432:Prisoner's dilemma
377:Nonlinear dynamics
332:Operationalization
328:Information theory
204:Genetic algorithms
1786:
1785:
1690:Cellular automata
1662:Pattern formation
1594:Adaptive networks
1385:Collective action
1355:Self-organization
1302:
1301:
1221:
1220:
1147:Understandability
784:Structural Safety
692:Robust algorithms
483:
482:
270:Cellular automata
244:Pattern formation
175:Adaptive networks
79:Collective action
48:Self-organization
16:(Redirected from
1811:
1804:Software quality
1700:Self-replication
1589:Dynamic networks
1478:Machine learning
1405:Phase transition
1329:
1322:
1315:
1306:
1305:
1292:
1291:
1285:
1284:
1097:
1096:
1086:Software quality
1079:
1072:
1065:
1056:
1055:
1049:
1048:
1016:
1007:
1001:
1000:
998:
997:
988:
980:
974:
973:
971:
959:
953:
952:
950:
949:
938:
932:
931:
929:
928:
914:
897:
896:
894:
893:
884:
875:
869:
868:
866:
865:
853:
847:
846:
844:
843:
834:
825:
806:
805:
803:
802:
781:
772:
766:
762:
753:
752:
750:
749:
740:
732:
487:computer science
475:
468:
461:
352:Systems thinking
278:Self-replication
216:Machine learning
170:Dynamic networks
91:Phase transition
30:
29:
21:
1819:
1818:
1814:
1813:
1812:
1810:
1809:
1808:
1789:
1788:
1787:
1782:
1748:System dynamics
1709:
1695:Spatial ecology
1656:
1598:
1584:Systems biology
1535:
1502:
1473:Artificial life
1443:
1434:
1430:Swarm behaviour
1415:Synchronization
1375:Social dynamics
1366:
1359:
1338:
1336:Complex systems
1333:
1303:
1298:
1273:
1247:
1217:
1161:
1112:Maintainability
1088:
1083:
1053:
1052:
1037:
1014:
1008:
1004:
995:
993:
986:
982:
981:
977:
960:
956:
947:
945:
939:
935:
926:
924:
916:
915:
900:
891:
889:
882:
876:
872:
863:
861:
854:
850:
841:
839:
832:
826:
809:
800:
798:
779:
773:
769:
763:
756:
747:
745:
738:
734:
733:
729:
724:
706:Fault tolerance
702:
694:
685:
676:
647:data structures
618:
605:
600:
544:
527:
519:fault injection
479:
450:
449:
448:
443:
439:
434:
429:
419:
418:
413:
409:
405:
401:
397:
393:
389:
384:
379:
369:
368:
367:
362:
358:
354:
350:
348:Systems science
346:
344:System dynamics
342:
338:
334:
330:
326:
322:
318:
314:
310:
305:
300:
286:
285:
280:
276:
274:Spatial ecology
272:
268:
264:
260:
256:
251:
246:
236:
235:
230:
226:
222:
218:
214:
212:Artificial life
210:
206:
202:
197:
192:
178:
177:
172:
168:
166:Systems biology
164:
160:
156:
152:
148:
144:
140:
135:
130:
120:
119:
118:
113:
111:Swarm behaviour
109:
105:
101:
99:Synchronization
97:
93:
89:
85:
81:
77:
72:
70:Social dynamics
67:
57:
56:
55:
50:
34:Complex systems
28:
23:
22:
15:
12:
11:
5:
1817:
1807:
1806:
1801:
1784:
1783:
1781:
1780:
1775:
1770:
1765:
1760:
1755:
1750:
1745:
1740:
1738:Self-reference
1735:
1730:
1725:
1719:
1717:
1715:Systems theory
1711:
1710:
1708:
1707:
1702:
1697:
1692:
1687:
1682:
1677:
1672:
1666:
1664:
1658:
1657:
1655:
1654:
1649:
1644:
1642:Multistability
1639:
1634:
1629:
1624:
1619:
1614:
1608:
1606:
1600:
1599:
1597:
1596:
1591:
1586:
1581:
1576:
1571:
1566:
1561:
1556:
1551:
1545:
1543:
1537:
1536:
1534:
1533:
1528:
1523:
1518:
1512:
1510:
1504:
1503:
1501:
1500:
1495:
1490:
1485:
1480:
1475:
1470:
1465:
1460:
1455:
1449:
1447:
1436:
1435:
1433:
1432:
1427:
1422:
1417:
1412:
1407:
1402:
1400:Herd mentality
1397:
1392:
1387:
1382:
1377:
1371:
1369:
1361:
1360:
1358:
1357:
1352:
1346:
1344:
1340:
1339:
1332:
1331:
1324:
1317:
1309:
1300:
1299:
1297:
1296:
1286:
1278:
1275:
1274:
1272:
1271:
1266:
1261:
1255:
1253:
1249:
1248:
1246:
1245:
1240:
1235:
1229:
1227:
1223:
1222:
1219:
1218:
1216:
1215:
1210:
1205:
1200:
1195:
1190:
1185:
1180:
1175:
1169:
1167:
1163:
1162:
1160:
1159:
1154:
1152:Loose coupling
1149:
1144:
1139:
1134:
1129:
1124:
1119:
1114:
1109:
1103:
1101:
1094:
1090:
1089:
1082:
1081:
1074:
1067:
1059:
1051:
1050:
1035:
1002:
975:
954:
933:
898:
870:
848:
807:
790:(3): 253–267.
767:
754:
726:
725:
723:
720:
719:
718:
713:
708:
701:
698:
693:
690:
684:
681:
675:
672:
671:
670:
666:
660:
659:
639:
635:
634:
630:
626:
625:
622:
617:
614:
610:error messages
604:
601:
599:
596:
543:
540:
532:integer values
526:
523:
481:
480:
478:
477:
470:
463:
455:
452:
451:
430:
425:
424:
421:
420:
407:Multistability
380:
375:
374:
371:
370:
340:Self-reference
301:
294:Systems theory
292:
291:
288:
287:
247:
242:
241:
238:
237:
193:
184:
183:
180:
179:
131:
126:
125:
122:
121:
87:Herd mentality
68:
63:
62:
59:
58:
51:
46:
45:
42:
41:
37:
36:
26:
9:
6:
4:
3:
2:
1816:
1805:
1802:
1800:
1797:
1796:
1794:
1779:
1776:
1774:
1771:
1769:
1766:
1764:
1761:
1759:
1756:
1754:
1751:
1749:
1746:
1744:
1743:Goal-oriented
1741:
1739:
1736:
1734:
1731:
1729:
1726:
1724:
1721:
1720:
1718:
1716:
1712:
1706:
1705:Geomorphology
1703:
1701:
1698:
1696:
1693:
1691:
1688:
1686:
1683:
1681:
1678:
1676:
1673:
1671:
1668:
1667:
1665:
1663:
1659:
1653:
1650:
1648:
1645:
1643:
1640:
1638:
1635:
1633:
1630:
1628:
1625:
1623:
1620:
1618:
1615:
1613:
1610:
1609:
1607:
1605:
1601:
1595:
1592:
1590:
1587:
1585:
1582:
1580:
1577:
1575:
1572:
1570:
1567:
1565:
1562:
1560:
1557:
1555:
1552:
1550:
1547:
1546:
1544:
1542:
1538:
1532:
1529:
1527:
1524:
1522:
1519:
1517:
1514:
1513:
1511:
1509:
1505:
1499:
1496:
1494:
1491:
1489:
1486:
1484:
1481:
1479:
1476:
1474:
1471:
1469:
1466:
1464:
1461:
1459:
1456:
1454:
1451:
1450:
1448:
1446:
1441:
1437:
1431:
1428:
1426:
1423:
1421:
1418:
1416:
1413:
1411:
1408:
1406:
1403:
1401:
1398:
1396:
1393:
1391:
1388:
1386:
1383:
1381:
1378:
1376:
1373:
1372:
1370:
1368:
1362:
1356:
1353:
1351:
1348:
1347:
1345:
1341:
1337:
1330:
1325:
1323:
1318:
1316:
1311:
1310:
1307:
1295:
1287:
1280:
1279:
1276:
1270:
1267:
1265:
1262:
1260:
1257:
1256:
1254:
1250:
1244:
1241:
1239:
1236:
1234:
1231:
1230:
1228:
1224:
1214:
1211:
1209:
1206:
1204:
1201:
1199:
1196:
1194:
1191:
1189:
1186:
1184:
1181:
1179:
1176:
1174:
1171:
1170:
1168:
1164:
1158:
1157:Orthogonality
1155:
1153:
1150:
1148:
1145:
1143:
1140:
1138:
1135:
1133:
1130:
1128:
1125:
1123:
1120:
1118:
1115:
1113:
1110:
1108:
1105:
1104:
1102:
1098:
1095:
1091:
1087:
1080:
1075:
1073:
1068:
1066:
1061:
1060:
1057:
1046:
1042:
1038:
1036:9781605588230
1032:
1028:
1024:
1020:
1013:
1006:
992:
985:
979:
970:
965:
958:
944:
937:
923:
919:
913:
911:
909:
907:
905:
903:
888:
881:
874:
859:
852:
838:
831:
824:
822:
820:
818:
816:
814:
812:
797:
793:
789:
785:
778:
771:
761:
759:
744:
737:
731:
727:
717:
714:
712:
709:
707:
704:
703:
697:
689:
680:
667:
665:
662:
661:
656:
652:
648:
644:
640:
637:
636:
631:
628:
627:
623:
620:
619:
613:
611:
595:
593:
589:
584:
582:
578:
574:
570:
569:functionality
565:
561:
557:
553:
549:
539:
535:
533:
522:
520:
516:
512:
508:
504:
500:
496:
492:
488:
476:
471:
469:
464:
462:
457:
456:
454:
453:
447:
444:
442:
438:
433:
428:
423:
422:
417:
414:
412:
408:
404:
400:
396:
392:
388:
383:
378:
373:
372:
366:
363:
361:
357:
353:
349:
345:
341:
337:
333:
329:
325:
321:
320:Goal-oriented
317:
313:
309:
304:
299:
295:
290:
289:
284:
283:Geomorphology
281:
279:
275:
271:
267:
263:
259:
255:
250:
245:
240:
239:
234:
231:
229:
225:
221:
217:
213:
209:
205:
201:
196:
191:
187:
182:
181:
176:
173:
171:
167:
163:
159:
155:
151:
147:
143:
139:
134:
129:
124:
123:
117:
114:
112:
108:
104:
100:
96:
92:
88:
84:
80:
76:
71:
66:
61:
60:
54:
49:
44:
43:
39:
38:
35:
32:
31:
19:
1578:
1569:Graph theory
1498:Evolvability
1233:ISO/IEC 9126
1202:
1183:Adaptability
1018:
1005:
994:. Retrieved
991:Math.mit.edu
990:
978:
957:
946:. Retrieved
936:
925:. Retrieved
921:
890:. Retrieved
886:
873:
862:. Retrieved
851:
840:. Retrieved
836:
799:. Retrieved
787:
783:
770:
746:. Retrieved
742:
730:
695:
686:
677:
669:accordingly.
664:Can't happen
606:
585:
545:
536:
528:
525:Introduction
515:fuzz testing
490:
484:
435:
385:
306:
252:
233:Evolvability
198:
161:
154:Graph theory
136:
73:
1768:Autopoiesis
1763:Cybernetics
1753:Sensemaking
1723:Homeostasis
1685:Percolation
1647:Bifurcation
1622:Phase space
1508:Game theory
1365:Collective
1188:Correctness
1178:Reliability
1142:Testability
1137:Scalability
1132:Readability
1127:Reusability
1122:Portability
1117:Flexibility
765:conditions"
743:Dl.ifip.org
588:scalability
427:Game theory
411:Bifurcation
391:Phase space
356:Sensemaking
324:Homeostasis
303:Autopoiesis
298:cybernetics
266:Percolation
1793:Categories
1579:Robustness
1559:Centrality
1445:adaptation
1343:Background
1203:Robustness
1198:Efficiency
996:2016-11-13
969:2009.04131
948:2016-11-13
927:2016-11-13
892:2016-11-13
887:Cse.sc.edu
864:2016-11-13
842:2016-11-13
801:2016-11-13
748:2016-11-13
722:References
655:interfaces
616:Principles
592:efficiency
581:components
556:redundancy
542:Challenges
491:robustness
395:Attractors
190:adaptation
162:Robustness
146:Centrality
1627:Attractor
1440:Evolution
1350:Emergence
1252:Processes
1173:Usability
1093:Qualities
643:libraries
629:Stupidity
499:execution
186:Evolution
53:Emergence
1733:Feedback
1541:Networks
1367:behavior
1208:Security
1193:Accuracy
1166:External
1100:Internal
700:See also
651:pointers
621:Paranoia
548:internet
316:Feedback
249:Fractals
128:Networks
1758:Entropy
1574:Scaling
1294:Commons
1045:1147058
497:during
360:Variety
312:Entropy
158:Scaling
1564:Motifs
1213:Safety
1043:
1033:
564:Humans
560:kidney
509:, and
495:errors
150:Motifs
40:Topics
1637:Chaos
1041:S2CID
1015:(PDF)
987:(PDF)
964:arXiv
883:(PDF)
833:(PDF)
780:(PDF)
739:(PDF)
658:code.
649:, or
598:Areas
577:logic
403:Chaos
1442:and
1107:Size
1031:ISBN
590:and
296:and
188:and
1023:doi
792:doi
550:or
485:In
1795::
1039:.
1029:.
1017:.
989:.
920:.
901:^
885:.
835:.
810:^
788:30
786:.
782:.
757:^
741:.
645:,
505:,
489:,
1328:e
1321:t
1314:v
1078:e
1071:t
1064:v
1047:.
1025::
999:.
972:.
966::
951:.
930:.
895:.
867:.
845:.
804:.
794::
751:.
474:e
467:t
460:v
20:)
Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.