Application of System Evaluation Theory (SET) in Evaluating Modern Day Systems


Systems thinking is an innovative evaluation approach receiving significant attention in the international evaluation community. Using interactive group discussion participants will learn the difference between using systems thinking in program evaluation versus the evaluation of modern day systems.  Participants will learn how emerging demands for considering context complexity led to the development of a cutting-edge approach for evaluating complex modern day systems, Systems Evaluation Theory (SET). SET guides evaluators in making sense of overwhelming evaluation challenges and leads to meaningful system change recommendations. Real world case evaluations will be used to teach participants how to apply SET to a) define a system and know why this is critical for evaluating system efficiency and effectiveness, b) evaluate system efficiency, including system attributes (e.g., leadership, training, information technology), feedback mechanisms, and cascading failure trigger points, and c) understand how interconnectedness and wholeness impact the evaluation of system effectiveness.

You will learn:

  • how to apply system evaluation theory (SET) to define a system and know why this is critical for evaluating system efficiency and effectiveness
  • how to use SET to evaluate a) system efficiency (e.g., leadership, training, information technology, feedback mechanisms, cascading failures), and b) system effectiveness (e.g., wholeness, connectedness)


Ralph Renger has taught graduate level courses at Canadian and American universities and currently works in the Center for Rural Health at the University of North Dakota.  He has 25 years of evaluation experience. The first 22 years were devoted to program evaluation and developing theory-driven methods such as the ATM approach to logic modeling, root cause analysis, deliberation-derived process evaluation, and constructing program theory using source documentation. Ralph has published 50 peer-reviewed evaluation articles in major international evaluation journals and recently published the first system evaluation theory (SET).

Ralph uses discussion-style facilitation in his workshops and has facilitated evaluation workshops in Europe, Australia, Canada, and the USA.

You can contact Ralph Renger via email at








Sunday, April 30 from 9:00 am to 12:00 noon

Link to CE competencies for evaluators

  • Develops evaluation designs
  • Defines evaluation methods
  • Respects the uniqueness of the site
  • Serves the information needs of intended users
  • Applies evaluation competencies to organization and program measurement challenges