157:
uncertainty about the future, and then uses this characterization to rank the desirability of alternative decision options. Importantly, this approach characterizes uncertainty without reference to the alternative options. In contrast, RDM characterizes uncertainty in the context of a particular decision. That is, the method identifies those combinations of uncertainties most important to the choice among alternative options and describes the set of beliefs about the uncertain state of the world that are consistent with choosing one option over another. This ordering provides cognitive benefits in decision support applications, allowing stakeholders to understand the key assumptions underlying alternative options before committing themselves to believing those assumptions.
226:
is judged successful from those where it is judged unsuccessful. Statistical or data-mining algorithms are applied to the database to generate simple descriptions of regions in the space of uncertain input parameters to the model that best describe the cases where the strategy is unsuccessful. That is, the algorithm for describing these cases is tuned to optimize both the predictability and interpretability by decision-makers. The resulting clusters have many characteristics of scenarios and can be used to help decision-makers understand the vulnerabilities of the proposed policies and potential response options. A review conducted by the
251:. OpenMORDM is an open source R package for RDM that includes support for defining more than one performance objective. OpenMORDM facilitates exploring the impact of different robustness criteria, including both regret-based (e.g., minimizing deviation in performance) and satisficing-based (e.g., satisfying performance constraints) criteria. Rhodium is an open source Python package that supports similar functionality to the EMA Workbench and to OpenMORDM, but also allows its application on models written in C, C++, Fortran, R and Excel, as well as the use of several multi-objective evolutionary algorithms.
196:
uncertain input parameters to the model(s), collecting the runs in a large database of cases, and analyzing this database to determine what policy-relevant statements can be supported. RDM represents a particular implementation of this concept. An RDM analysis typically creates a large database of simulation model results, and then uses this database to identify vulnerabilities of proposed strategies and the tradeoffs among potential responses. This analytic process provides several practical advantages:
109:, with a particular focus on helping decision-makers identify and design new decision options that may be more robust than those they had originally considered. Often, these more robust options represent adaptive decision strategies designed to evolve over time in response to new information. In addition, RDM can be used to facilitate group decision-making in contentious situations where parties to the decision have strong disagreements about assumptions and values.
208:
used traditional sequential decision approaches, rule-based descriptions of adaptive strategies, real options representations, complicated optimal economic growth models, spreadsheet models, agent-based models, and organization's existing suites of simulation models such as one used by the U.S. government to forecast the future state of the social security trust fund.
113:
renewable energy requirements, a comparison of long-term energy strategies for the government of Israel, an assessment of science and technology policies the government of South Korea might pursue in response to increasing economic competition from China, and an analysis of
Congress' options for reauthorizing the
144:
as a criterion to assess alternative policies. The traditional subjective utility framework ranks alternative decision options contingent on best estimated probability distributions. In general, there is a best (i.e., highest ranked) option. RDM analyses have employed several different definitions of
112:
RDM approaches have been applied to a wide range of different types of decision challenges. A study in 1996 addressed adaptive strategies for reducing greenhouse gas emissions. More recent studies include a variety of applications to water management issues, evaluation of the impacts of proposed U.S.
389:
Robust decision making describes a variety of approaches that differ from traditional optimum expected utility analysis in that they characterize uncertainty with multiple representations of the future rather than a single set of probability distributions and use robustness, rather than optimality,
225:
to facilitate the identification of vulnerabilities of proposed strategies. The process begins by specifying some performance metric, such as the total cost of a policy or its deviation from optimality (regret), which can be used to distinguish those cases in the results database where the strategy
41:
framework that aims to help identify potential robust strategies, characterize the vulnerabilities of such strategies, and evaluate the tradeoffs among them. RDM focuses on informing decisions under conditions of what is called "deep uncertainty", that is, conditions where the parties to a decision
207:
The exploratory modeling concept makes it possible to use a wide variety of decision approaches using diverse types simulation models within a common analytic framework (depending on what seems most appropriate for a particular decision application). Within this common framework RDM analyses have
165:
Robust decision methods seem most appropriate under three conditions: when the uncertainty is deep as opposed to well characterized, when there is a rich set of decision options, and the decision challenge is sufficiently complex that decision-makers need simulation models to trace the potential
195:
approach, with computer simulations used not as a device for prediction, but rather as a means for relating a set of assumptions to their implied consequences. The analyst draws useful information from such simulations by running them many times using an appropriate experimental design over the
211:
The database of cases simplifies the comparison of alternative decision frameworks because one can apply these frameworks to an identical set of model results. For instance, one can place a joint probability distribution across the cases in a database, conduct an expected utility analysis, and
230:
of the rather sparse literature evaluating how scenarios actually perform in practice when used by organizations to inform decisions identified several key weaknesses of traditional scenario approaches. Scenario-discovery methods are designed to address these weaknesses. In addition, scenario
173:
If the uncertainty is deep and a rich set of options is available, traditional qualitative scenario methods may prove most effective if the system is sufficiently simple or well understood that decision-makers can accurately connect potential actions to their consequences without the aid of
169:
When the uncertainty is well characterized, then traditional expected utility (predict-then-act) analyses are often most appropriate. In addition, if decision-makers lack a rich set of decision options they may have little opportunity to develop a robust strategy and can do no better than a
203:
Running a simulation multiple times in the forward direction can simplify the analytic challenge of representing adaptive strategies in many practical applications because it separates the running of the simulation from the analysis needed to evaluate alternative decision options using the
156:
analysis framework to characterize uncertainty and to help identify and evaluate robust strategies. This structuring of the decision problem is a key feature of RDM. The traditional decision analytic approach follows what has been called a predict-then-act approach that first characterizes
125:
RDM rests on three key concepts that differentiate it from the traditional subjective expected utility decision framework: multiple views of the future, a robustness criterion, and reversing the order of traditional decision analysis by conducting an iterative process based on a
145:
robustness. These include: trading a small amount of optimum performance for less sensitivity to broken assumptions, good performance compared to the alternatives over a wide range of plausible scenarios, and keeping options open. All incorporate some type of
182:
RDM is not a recipe of analytic steps, but rather a set of methods that can be combined in varying ways for specific decisions to implement the concept. Two key items in this toolkit are described below: exploratory modeling and scenario discovery.
231:
discovery supports analysis for multiple stressors because it characterizes vulnerabilities as combinations of very different types of uncertain parameters (e.g. climate, economic, organizational capabilities, etc.).
58:
A wide variety of concepts, methods, and tools have been developed to address decision challenges that confront a large degree of uncertainty. One source of the name "robust decision" was the field of
685:
Dessai, Suraje; Hulme, Mike (February 2007). "Assessing the robustness of adaptation decisions to climate change uncertainties: A case study on water resources management in the East of
England".
436:
Robust decision making is more analytical than intuitive. It adopts a systematic approach to remove uncertainty within the resources available to make safe and effective decisions. (1023)
133:. In some cases, these multiple views will be represented by multiple future states of the world. RDM can also incorporate probabilistic information, but rejects the view that a single
756:
Lempert, Robert J.; Nakicenovic, Nebojsa; Sarewitz, Daniel; Schlesinger, Michael (July 2004). "Characterizing climate-change uncertainties for decision-makers: an editorial essay".
137:
represents the best description of a deeply uncertain future. Rather RDM uses ranges or, more formally, sets of plausible probability distributions to describe deep uncertainty.
604:
Lempert, Robert J.; Schlesinger, Michael E.; Bankes, Steve C. (June 1996). "When we don't know the costs or the benefits: adaptive strategies for abating climate change".
66:
in the 1980s and early 1990s. Jonathan
Rosenhead and colleagues were among the first to lay out a systematic decision framework for robust decisions, in their 1989 book
353:
Lempert, Robert J.; Collins, Myles T. (August 2007). "Managing the risk of uncertain threshold responses: comparison of robust, optimum, and precautionary approaches".
149:
criteria and, in contrast to expected utility approaches, all generally describe tradeoffs rather than provide a strict ranking of alternative options.
798:
Bryant, Benjamin P.; Lempert, Robert J. (January 2010). "Thinking inside the box: a participatory, computer-assisted approach to scenario discovery".
98:
Robust decision-making (RDM) is a particular set of methods and tools developed over the last decade, primarily by researchers associated with the
907:
Kwakkel, Jan H.; Pruyt, Erik (March 2013). "Exploratory
Modeling and Analysis, an approach for model-based foresight under deep uncertainty".
239:
There is several software available to perform RDM analysis. RAND Corporation has developed CARS for exploratory modeling and the sdtoolkit
909:
800:
716:"Improving the contribution of climate model information to decision making: the value and demands of robust decision frameworks"
1040:
714:
Weaver, Christopher P.; Lempert, Robert J.; Brown, Casey; Hall, John A.; Revell, David; Sarewitz, Daniel (January 2013).
524:
Rational analysis for a problematic world revisited: problem structuring methods for complexity, uncertainty and conflict
204:
simulation. In contrast, some optimization methods make it difficult to include many types of feedbacks in a simulation.
660:
580:
535:
461:
105:
While often used by researchers to evaluate alternative options, RDM is designed and is often employed as a method for
1035:
870:
Groves, David G.; Lempert, Robert J. (February 2007). "A new analytic method for finding policy-relevant scenarios".
498:
291:"A novel project portfolio selection framework towards organizational resilience: Robust Ordinal Priority Approach"
17:
1008:
244:
200:
The database of cases provides a concrete representation of the concept of a multiplicity of plausible futures.
134:
833:
248:
114:
87:
715:
872:
687:
408:
265:
227:
758:
606:
628:
240:
83:
355:
59:
572:
560:
102:, designed to support decision-making and policy analysis under conditions of deep uncertainty.
1019:
1004:
623:
79:
971:"Rhodium: Python Library for Many-Objective Robust Decision Making and Exploratory Modeling"
615:
527:
8:
771:
490:
619:
889:
852:
775:
738:
641:
487:
Making robust decisions: decision management for technical, business, and service teams
380:
323:
290:
779:
586:
576:
541:
531:
504:
494:
467:
457:
427:
372:
368:
328:
310:
126:
vulnerability-and-response-option rather than a predict-then-act decision framework.
71:
47:
38:
885:
742:
700:
120:
982:
949:
922:
918:
881:
842:
813:
809:
767:
730:
696:
645:
633:
568:
417:
384:
364:
318:
302:
260:
106:
99:
954:
937:
893:
422:
403:
270:
247:, provides extensive exploratory modeling and scenario discovery capabilities in
63:
969:
Hadjimichael, Antonia; Gold, David; Hadka, David; Reed, Patrick (9 June 2020).
659:
Groves, David G.; Davis, Martha; Wilkinson, Robert; Lempert, Robert J. (2008).
306:
75:
936:
Hadka, David; Herman, Jonathan; Reed, Patrick; Keller, Klaus (December 2015).
1029:
453:
314:
590:
545:
508:
471:
431:
376:
332:
43:
847:
828:
1013:
146:
88:
Third
Assessment Report of the Intergovernmental Panel on Climate Change
661:"Planning for climate change in the Inland Empire, Southern California"
637:
856:
987:
970:
755:
938:"An open source framework for many-objective robust decision making"
734:
86:. An early review of many of these approaches is contained in the
121:
Differences between RDM and traditional expected-utility analysis
243:
package for scenario discovery. The EMA Workbench, developed at
212:
compare the results to an RDM analysis using the same database.
565:
High-performance government: structure, leadership, incentives
166:
consequences of their actions over many plausible scenarios.
177:
50:
distributions for the key input parameters to those models.
658:
968:
1016:, an R package for multiobjective robust decision making
603:
160:
1022:, scenario discovery toolkit for robust decision-making
713:
289:
Mahmoudi, Amin; Abbasi, Mehdi; Deng, Xiaopeng (2022).
70:. Similar themes have emerged from the literatures on
935:
793:
791:
789:
348:
346:
344:
342:
561:"High-performance government in an uncertain world"
563:. In Klitgaard, Robert E.; Light, Paul C. (eds.).
522:Mingers, John; Rosenhead, Jonathan, eds. (2001) .
288:
786:
339:
1027:
521:
723:Wiley Interdisciplinary Reviews: Climate Change
559:Lempert, Robert J.; Popper, Steven W. (2005).
352:
869:
797:
558:
906:
910:Technological Forecasting and Social Change
801:Technological Forecasting and Social Change
684:
404:"A universal model of diagnostic reasoning"
221:RDM analyses often employ a process called
829:"Exploratory modeling for policy analysis"
526:(2nd ed.). Chichester, UK; New York:
129:First, RDM characterizes uncertainty with
1007:, an "exploratory modeling workbench" by
986:
953:
846:
627:
421:
401:
322:
178:Analytic tools for robust decision-making
68:Rational Analysis for a Problematic World
46:relating actions to consequences or the
450:Quality engineering using robust design
186:
14:
1028:
942:Environmental Modelling & Software
826:
484:
447:
216:
161:Conditions for robust decision-making
27:Iterative decision analytic framework
390:as a decision criterion. (1011-1012)
84:info-gap decision theory and methods
234:
42:do not know or do not agree on the
24:
772:10.1023/B:CLIM.0000037561.75281.b3
25:
1052:
998:
975:Journal of Open Research Software
154:vulnerability-and-response-option
142:robustness rather than optimality
448:Phadke, Madhav Shridhar (1989).
369:10.1111/j.1539-6924.2007.00940.x
295:Expert Systems with Applications
962:
929:
900:
886:10.1016/j.gloenvcha.2006.11.006
863:
820:
749:
707:
701:10.1016/j.gloenvcha.2006.11.005
678:
1009:Delft University of Technology
923:10.1016/j.techfore.2012.10.005
814:10.1016/j.techfore.2009.08.002
652:
597:
552:
515:
478:
441:
402:Croskerry, Pat (August 2009).
395:
282:
245:Delft University of Technology
135:joint probability distribution
93:
13:
1:
955:10.1016/j.envsoft.2015.07.014
276:
423:10.1097/ACM.0b013e3181ace703
131:multiple views of the future
115:Terrorism Risk Insurance Act
7:
1041:Problem structuring methods
873:Global Environmental Change
827:Bankes, Steve (June 1993).
688:Global Environmental Change
266:Problem structuring methods
254:
228:European Environment Agency
170:predict-then-act analysis.
10:
1057:
307:10.1016/j.eswa.2021.116067
53:
485:Ullman, David G. (2006).
191:Many RDM analyses use an
62:popularized primarily by
1036:Decision support systems
452:. Englewood Cliffs, NJ:
668:Water Resources IMPACT
31:Robust decision-making
848:10.1287/opre.41.3.435
528:John Wiley & Sons
152:Third, RDM employs a
90:, published in 2001.
80:imprecise probability
567:. Santa Monica, CA:
193:exploratory modeling
187:Exploratory modeling
834:Operations Research
620:1996ClCh...33..235L
491:Trafford Publishing
174:simulation models.
638:10.1007/BF00140248
223:scenario discovery
217:Scenario discovery
39:decision analytics
37:) is an iterative
409:Academic Medicine
140:Second, RDM uses
72:scenario planning
48:prior probability
16:(Redirected from
1048:
993:
992:
990:
988:10.5334/jors.293
966:
960:
959:
957:
933:
927:
926:
904:
898:
897:
867:
861:
860:
850:
824:
818:
817:
795:
784:
783:
753:
747:
746:
720:
711:
705:
704:
682:
676:
675:
665:
656:
650:
649:
631:
601:
595:
594:
569:RAND Corporation
556:
550:
549:
519:
513:
512:
489:. Victoria, BC:
482:
476:
475:
445:
439:
438:
425:
416:(8): 1022–1028.
399:
393:
392:
363:(4): 1009–1026.
350:
337:
336:
326:
286:
261:Design rationale
235:Software support
107:decision support
100:RAND Corporation
21:
1056:
1055:
1051:
1050:
1049:
1047:
1046:
1045:
1026:
1025:
1001:
996:
967:
963:
934:
930:
905:
901:
868:
864:
825:
821:
796:
787:
759:Climatic Change
754:
750:
735:10.1002/wcc.202
718:
712:
708:
683:
679:
663:
657:
653:
607:Climatic Change
602:
598:
583:
557:
553:
538:
520:
516:
501:
483:
479:
464:
446:
442:
400:
396:
351:
340:
287:
283:
279:
271:System dynamics
257:
237:
219:
189:
180:
163:
123:
96:
64:Genichi Taguchi
56:
28:
23:
22:
18:Robust decision
15:
12:
11:
5:
1054:
1044:
1043:
1038:
1024:
1023:
1017:
1011:
1000:
999:External links
997:
995:
994:
961:
928:
917:(3): 419–431.
899:
862:
841:(3): 435–449.
819:
785:
748:
706:
677:
651:
629:10.1.1.20.9055
614:(2): 235–274.
596:
582:978-0833037404
581:
551:
537:978-0471495239
536:
514:
499:
493:. p. 35.
477:
463:978-0137451678
462:
440:
394:
338:
280:
278:
275:
274:
273:
268:
263:
256:
253:
236:
233:
218:
215:
214:
213:
209:
205:
201:
188:
185:
179:
176:
162:
159:
122:
119:
95:
92:
76:robust control
55:
52:
26:
9:
6:
4:
3:
2:
1053:
1042:
1039:
1037:
1034:
1033:
1031:
1021:
1018:
1015:
1012:
1010:
1006:
1005:EMA workbench
1003:
1002:
989:
984:
980:
976:
972:
965:
956:
951:
947:
943:
939:
932:
924:
920:
916:
912:
911:
903:
895:
891:
887:
883:
879:
875:
874:
866:
858:
854:
849:
844:
840:
836:
835:
830:
823:
815:
811:
807:
803:
802:
794:
792:
790:
781:
777:
773:
769:
765:
761:
760:
752:
744:
740:
736:
732:
728:
724:
717:
710:
702:
698:
694:
690:
689:
681:
673:
669:
662:
655:
647:
643:
639:
635:
630:
625:
621:
617:
613:
609:
608:
600:
592:
588:
584:
578:
574:
570:
566:
562:
555:
547:
543:
539:
533:
529:
525:
518:
510:
506:
502:
500:9781425109561
496:
492:
488:
481:
473:
469:
465:
459:
455:
454:Prentice Hall
451:
444:
437:
433:
429:
424:
419:
415:
411:
410:
405:
398:
391:
386:
382:
378:
374:
370:
366:
362:
358:
357:
356:Risk Analysis
349:
347:
345:
343:
334:
330:
325:
320:
316:
312:
308:
304:
300:
296:
292:
285:
281:
272:
269:
267:
264:
262:
259:
258:
252:
250:
246:
242:
232:
229:
224:
210:
206:
202:
199:
198:
197:
194:
184:
175:
171:
167:
158:
155:
150:
148:
143:
138:
136:
132:
127:
118:
116:
110:
108:
103:
101:
91:
89:
85:
81:
77:
73:
69:
65:
61:
60:robust design
51:
49:
45:
44:system models
40:
36:
32:
19:
978:
974:
964:
945:
941:
931:
914:
908:
902:
880:(1): 73–85.
877:
871:
865:
838:
832:
822:
808:(1): 34–49.
805:
799:
766:(1–2): 1–9.
763:
757:
751:
729:(1): 39–60.
726:
722:
709:
695:(1): 59–72.
692:
686:
680:
671:
667:
654:
611:
605:
599:
564:
554:
523:
517:
486:
480:
449:
443:
435:
413:
407:
397:
388:
360:
354:
298:
294:
284:
238:
222:
220:
192:
190:
181:
172:
168:
164:
153:
151:
141:
139:
130:
128:
124:
111:
104:
97:
67:
57:
34:
30:
29:
948:: 114–129.
674:(4): 14–17.
571:. pp.
147:satisficing
94:Application
1030:Categories
301:: 116067.
277:References
1020:sdtoolkit
1014:OpenMORDM
780:153473117
624:CiteSeerX
315:0957-4174
743:53679056
591:57344300
546:46601256
509:81600845
472:19455232
432:19638766
377:17958508
333:36818824
255:See also
117:(TRIA).
646:7945822
616:Bibcode
573:113–138
385:1722147
324:9928571
54:History
981:: 12.
894:510560
892:
857:171847
855:
778:
741:
644:
626:
589:
579:
544:
534:
507:
497:
470:
460:
430:
383:
375:
331:
321:
313:
249:Python
82:, and
890:S2CID
853:JSTOR
776:S2CID
739:S2CID
719:(PDF)
664:(PDF)
642:S2CID
381:S2CID
587:OCLC
577:ISBN
542:OCLC
532:ISBN
505:OCLC
495:ISBN
468:OCLC
458:ISBN
428:PMID
373:PMID
329:PMID
311:ISSN
983:doi
950:doi
919:doi
882:doi
843:doi
810:doi
768:doi
731:doi
697:doi
634:doi
418:doi
365:doi
319:PMC
303:doi
299:188
35:RDM
1032::
977:.
973:.
946:74
944:.
940:.
915:80
913:.
888:.
878:17
876:.
851:.
839:41
837:.
831:.
806:77
804:.
788:^
774:.
764:65
762:.
737:.
725:.
721:.
693:17
691:.
672:10
670:.
666:.
640:.
632:.
622:.
612:33
610:.
585:.
575:.
540:.
530:.
503:.
466:.
456:.
434:.
426:.
414:84
412:.
406:.
387:.
379:.
371:.
361:27
359:.
341:^
327:.
317:.
309:.
297:.
293:.
78:,
74:,
991:.
985::
979:8
958:.
952::
925:.
921::
896:.
884::
859:.
845::
816:.
812::
782:.
770::
745:.
733::
727:4
703:.
699::
648:.
636::
618::
593:.
548:.
511:.
474:.
420::
367::
335:.
305::
241:R
33:(
20:)
Text is available under the Creative Commons Attribution-ShareAlike License. Additional terms may apply.