Multimedia Explanation Generators for Knowledge-Based Learning Environments

James Lester

Computer Science Department
North Carolina State University

CONTACT INFORMATION

Department of Computer Science
Engineering Graduate Research Center
Box 7534
Raleigh, NC 27695-7534
Phone: (919) 515-7534
Fax : (919) 515-7925
Email: lester@csc.ncsu.edu

WWW PAGE

http://www.csc.ncsu.edu/eos/users/l/lester/www/imedia/james.html

PROGRAM AREA

Adaptive Human Interfaces.

KEYWORDS

Knowledge-based learning environments, intelligent user interfaces, multimedia, explanation generation, animated agents

PROJECT SUMMARY

Knowledge-based learning environments can provide highly customized problem-solving experiences that are tailored to the individual needs of each student. Explanation generation is the key functionality of learning environments that are designed to provide knowledgeable feedback. As a result of rapid advances in multimedia technologies, we are now presented with the opportunity to create real-time multimedia explanation generators that employ animated pedagogical agents. Prominently featured in learning environments, animated pedagogical agents could observe students' progress and provide them with visually contextualized explanations and problem-solving advice. In addition to increasing students' learning effectiveness with customized multimedia explanations, animated pedagogical agents could also play a powerful motivational role. The primary objective of the research is to create the technology for a new generation of knowledge-based learning environments. It has three major thrusts:

Developing a computational model of real-time multimedia explanation planning} that can be employed to construct pedagogically customized, multimedia explanation plans. The content, rhetorical structure, and visual arrangement of the multimedia explanations produced by the realization of these plans will be tailored to students' problem-solving actions and their knowledge of the domain.
Developing a computational model of animated pedagogical agents} that actively participate in multimedia explanations. By performing visual and verbal behaviors, animated pedagogical agents will provide students with visually compelling explanations where all actions and utterances are tailored to students' problem-solving activities.
Conducting formal empirical evaluations} of the pedagogical effectiveness of the model of explanation generation. By conducting extensive rigorous evaluations of students' interactions with an explanation generator in an implemented learning environment, we will iteratively implement and refine our model. To this end, we will employ cognitive studies of learning effectiveness and task-load studies of motivational impact.

As a CAREER project, the work also has an important educational component. Our primary objective here to create a new generation of computer scientists who can accelerate the deployment of knowledge-based learning environments into the classroom and workplace. To this end, we will be conducting the following activities: developing a teaching laboratory for multimedia technology; initiating a new multidisciplinary course on Knowledge-Based Multimedia Learning Environments; involving women, under-represented minorities, and undergraduates in research; and conducting outreach activities to local school systems and educational organizations.

PROJECT REFERENCES

William Bares and James Lester. Realtime Generation of Customized 3D Animated Explanations for Knowledge-Based Learning Environments. To appear in Proceedings of the Fourteenth National Conference on Artificial Intelligence, Providence, RI, July 1997. (AAAI-97)

William Bares and James Lester. Cinematographic User Models for Automated Realtime Camera Control in Dynamic 3D Environments. Proceedings of the Sixth International Conference on User Modeling, pp. 215-226, Sardinia, Italy, June 1997. (UM-97)

Patrick FitzGerald and James Lester. Knowledge-Based Learning Environments: A Vision for the 21st Century. In Interactive Technologies and the Social Sciences: Emerging Issues and Applications, P. Martorella (Ed.), pp. 111-127, SUNY Press, New York, 1997.

James Lester, Patrick FitzGerald, and Brian Stone. The Pedagogical Design Studio: Exploiting Artifact-Based Task Models for Constructivist Learning. Proceedings of the Third International Conference on Intelligent User Interfaces, pp. 155-162, Orlando, FL, January 1997. (IUI-97)

James Lester, Brian Stone, Michael O'Leary, and Robert Stevenson. Focusing Problem Solving in Design-Centered Learning Environments. Proceedings of the Third International Conference on Intelligent Tutoring Systems (Springer-Verlag Lecture Notes in Computer Science 1086), pp. 475-483, Montreal, June 1996. (ITS '96)

James Lester, Sharolyn Converse, Susan Kahler, Todd Barlow, Brian Stone, and Ravinder Bhogal. The Persona Effect: Affective Impact of Animated Pedagogical Agents. Proceedings of CHI '97, pp. 359-366, Atlanta, March 1997. (CHI-97)

James Lester and Brian Stone. Increasing Believability in Animated Pedagogical Agents. Proceedings of the First International Conference on Autonomous Agents, pp. 16-21, Marina del Rey, California, February 1997. (AA-97)

James Lester, Sharolyn Converse, Brian Stone, Susan Kahler, and Todd Barlow. Animated Pedagogical Agents and Problem-Solving Effectiveness: A Large-Scale Empirical Evaluation. To appear in Proceedings of the Eighth World Conference on Artificial Intelligence in Education, Kobe, Japan, August 1997. (AI&ED-97)

Brian Stone and James Lester. Dynamically Sequencing an Animated Pedagogical Agent. Proceedings of the Thirteenth National Conference on Artificial Intelligence, pp. 424-431, Portland, OR, August 1996. (AAAI-96)

AREA BACKGROUND

This project draws on three distinct research areas. (1) Knowledge-based learning environments, which grew out of work on Intelligent Tutoring Systems, seek to provide students with effective, customized learning experiences. (2) Multimedia explanation generation systems, which grew out natural language generation systems in general and explanation generation systems in particular, dynamically plan multimedia presentations that achieve given communicative goals. (3) Animated agents, which draw on agent-based methodologies and the principles of animation, orchestrate the behaviors of lifelike characters with their own goals and plans.

AREA REFERENCES

J.R. Anderson, A.T. Corbett, K.R. Koedinger and R. Pelletier. Cognitive Tutors: Lessons Learned. Journal of the Learning Sciences, 4(2), 167-207, 1995.

Elisabeth Andre and Thomas Rist. Coping with Temporal Constraints in Multimedia Presentation Planning. Proceedings of the Thirteenth National Conference on Artificial Intelligence, 142-147, 1996.

Joseph Bates. The Role of Emotion in Believable Agents. Communications of the ACM, 37(7), 122-125, 1994.

Andreas Butz and Antonio Kruger. Lean Modeling -- The Intelligent Use of Geometrical Abstraction in 3D Animations. Proceedings of the Twelfth European Conference on Artificial Intelligence, 246-250, 1996.

Alison Cawsey. Explanation and Interaction: The Computer Generation of Explanatory Dialogues. MIT Press, 1992.

Steven K. Feiner and Kathleen R. McKeown. Automating the Generation of Coordinated Multimedia Explanations. In Mark T. Maybury (Ed.), Intelligent Multimedia Interfaces, 117-138, MIT Press, 1993.

Steven K. Feiner and Doree Duncan Seligmann. Cutaways and Ghosting: Satisfying Visability Constraints in Dynamic 3D Illustrations. The Visual Computer, 8, 292-302, 1992.

Eduard H. Hovy. Automated Discourse Generation Using Discourse Structure Relations. Artificial Intelligence, 63, 341-385, 1993.

Peter Karp and Steven Feiner. Automated Presentation Planning of Animation Using Task Decomposition with Heuristic Reasoning. Proceedings of Graphics Interface '93, 118-127, 1993.

Alan Lesgold, Susanne Lajoie, Marilyn Bunzo, and Gary Eggan. SHERLOCK: A Coached Practice Environment for an Electronics Trouble-Shooting Job. In Jill H. Larkin and Ruth W. Chabay (Eds.) Computer-Assisted Instruction and Intelligent Tutoring Systems: Shared Goals and Complementary Approaches, 201-238, Lawrence Erlbaum, 1992.

James C. Lester and Bruce W. Porter. Developing and Empirically Evaluating Robust Explanation Generators: The KNIGHT Experiments. Computational Linguistics, 23(1), 65-101, 1997.

P. Maes, T. Darrell, B. Blumberg, and A. Pentland. The ALIVE System: Full-body Interaction with Autonomous Agents. Proceedings of the Computer Animation '95 Conference, 1995.

Mary A. Mark and Jim E. Greer. The VCR Tutor: Effective Instruction for Device Operation. Journal of the Learning Sciences, 4(2), 209-246, 1995.

Vibhu Mittal, Steven Roth, Johanna D. Moore, Joe Mattis, and Giuseppe Carenini. Generating Explanatory Captions for Information Graphics. Proceedings of the International Joint Conference on Artificial Intelligence, 1276-1283, 1995.

Johanna D. Moore. Participating in Explanatory Dialogues. MIT Press, 1995.

Jeff Rickel and Lewis Johnson. Intelligent Tutoring in Virtual Reality: A Preliminary Report. To appear in Proceedings of the Eighth World Conference on AI in Education, 1997.

David B. Christianson, Sean E. Anderson, Li-Wei He and David H. Salesin, Daniel S. Weld, Michael F. Cohen. Declarative Camera Control for Automatic Cinematography. Proceedings of the Thirteenth National Conference on Artificial Intelligence, 148-155, 1996.

Daniel D. Suthers. A Task-Appropriate Hybrid Architecture for Explanation. Computational Intelligence, 7(4), 315-333, 1991.

Bonnie Webber, Norman Badler, Barbara Di Eugenio, Chris Geib, Libby Levison, and Michael Moore. Instructions, Intentions and Expectations. Artificial Intelligence, 73, 253-269, 1995.

Wolfgang Wahlster, Elizabeth Andre, Wolfgang Finkler, Hans-Jurgen Profitlich, and Thomas Rist. Plan-Based Integration of Natural Language and Graphics Generation. Artificial Intelligence, 63, 387-427, 1993.

RELATED PROGRAM AREAS

Virtual Environments, Speech and Natural Understanding, Other Communication Modalities