Assessment Strategies in Engineering in the Age of Generative AI

Rethinking Evaluation for the AI Era

Andrew Katz

Learning Objectives

By the end of this lecture, you will be able to:

  1. Identify key challenges in assessing student learning when AI tools are integrated into engineering education
  2. Apply cognitive science principles to design effective AI-aware assessment strategies
  3. Implement the Assessment Alignment framework using backward design principles
  4. Evaluate student work using research-based AI literacy rubrics
  5. Design equitable assessment practices that support all learners in AI-integrated environments

Connection to Practice: These skills align with common student learning outcomes and prepare you for the evolving landscape of engineering education

Lecture Roadmap

Part I: Theoretical Foundations

  • Why theory matters for AI assessment
  • Self-regulated learning in engineering
  • Metacognitive awareness framework
  • Constructivist learning principles
  • Backward design for AI assessment

Part II: Assessment Challenges Reframed

  • Authorship ambiguity through learning theory
  • Foundational vs. enhanced skills (ZPD)
  • Process vs. product evaluation
  • Equity through inclusive pedagogy

Part III: Paradigm Shift

  • From detection to development
  • Change management theory
  • Professional identity evolution
  • Faculty concerns and solutions

Part IV: Assessment Framework

  • Progressive complexity model
  • Decision support system
  • Practical implementation models

Part V: Research-Based Grading

  • AI literacy rubrics
  • Comprehensive grading examples
  • Implementation strategies

The Engineering Assessment Challenge

A Hypothetical Scenario

“Two students submitted identical thermodynamics solutions. One documented 47 minutes of AI interaction with iterative prompt refinement. The other submitted no documentation. Both solutions were technically correct. How do you grade them?”

Is this happening in your classrooms right now?

I. Theoretical Foundations for AI-Aware Assessment

Part I Roadmap: Theoretical Foundations

Learning Objectives Focus: Apply cognitive science principles to design effective AI-aware assessment strategies

What We’ll Cover:

  • Why theory matters in practice
  • Self-regulated learning in engineering
  • Metacognitive awareness framework
  • Constructivist learning principles
  • Backward design for AI assessment
  • Theory integration summary

Key Questions We’ll Answer:

  • How do students learn with AI tools?
  • What cognitive processes should we assess?
  • How do we scaffold AI skill development?

By the End of Part I: You’ll understand the research foundation for assessment decisions we make

Why Theory Matters in Practice

Without Theoretical Grounding:

  • Ad hoc solutions that don’t scale
  • Inconsistent assessment practices
  • Student confusion about expectations
  • Difficulty defending assessment decisions
  • Missing connections to learning science

With Theoretical Foundation:

  • Evidence-based assessment design
  • Coherent progression of skills
  • Clear rationale for students and colleagues
  • Alignment with learning objectives
  • Connection to professional practice

Key Insight: Effective AI-aware assessment requires understanding how humans learn with and through technology

Self-Regulated Learning in Engineering

Self-Regulated Learning (SRL): The process by which students activate and sustain cognitions, behaviors, and affects that are systematically oriented toward attainment of learning goals (Zimmerman, 2002)

1. Forethought Phase

  • Goal setting
  • Strategic planning
  • Self-efficacy beliefs
  • Task analysis

Engineering Example: “I need to design a heat exchanger. I’ll use AI to explore initial configurations, then verify thermodynamic calculations manually.”

2. Performance Phase

  • Self-monitoring
  • Self-instruction
  • Help-seeking
  • Task strategies

Engineering Example: “The AI suggested a counterflow design. Let me check if this meets our pressure drop constraints.”

3. Self-Reflection Phase

  • Self-evaluation
  • Causal attribution
  • Self-reaction
  • Adaptivity

Engineering Example: “AI helped me consider configurations I hadn’t thought of, but I needed to verify all thermal properties independently.”

Metacognitive Awareness Framework

Metacognition: “Thinking about thinking” - awareness and understanding of one’s own thought processes (Flavell, 1979)

Metacognitive Knowledge

  • Person Knowledge: Individual learning preferences and capabilities
  • Task Knowledge: Understanding task demands and requirements
  • Strategy Knowledge: Knowing which strategies work when

Engineering Example: “I know I learn structural concepts better by working through beam calculations manually first, then using AI for complex multi-member analysis.”

Metacognitive Regulation

  • Planning: Selecting appropriate strategies and allocating resources
  • Monitoring: Tracking progress and strategy effectiveness
  • Evaluating: Assessing outcomes and strategy success

Engineering Example: “I’m monitoring whether this AI-generated circuit analysis follows Kirchhoff’s laws correctly.”

Theory in Action: Bridge Design Case

Scenario: Senior design students must design a pedestrian bridge using AI tools while demonstrating engineering competency

Self-Regulated Learning Application:

  • Forethought: “I’ll use AI to explore truss configurations, then verify calculations manually”
  • Performance: “AI suggested Warren truss - checking if member forces match my hand calculations”
  • Reflection: “AI helped explore alternatives, but I caught wind load error - need better verification”

Metacognitive Awareness Application:

  • Person Knowledge: “I struggle with dynamic analysis - extra careful with AI outputs here”
  • Task Knowledge: “Bridge needs creativity AND safety - AI helps first, I ensure second”
  • Strategy Knowledge: “AI brainstorming → manual verification → AI optimization”

Assessment Focus: Evaluate both technical solution AND quality of learning processes

Constructivist Learning Principles

Constructivism: Learning is an active process where learners construct knowledge through experience and reflection (Vygotsky, 1978)

1. Zone of Proximal Development

  • Independent vs. assisted performance
  • AI as “more knowledgeable other”
  • Scaffolding complex problems

Example: Basic fluid flow (independent) → Complex turbulence modeling (AI-assisted)

2. Social Construction

  • Learning through interaction with others and tools
  • AI as part of social learning environment
  • Peer review builds understanding

Example: Students share AI prompting strategies for circuit optimization

3. Active Construction

  • Actively engage with AI outputs
  • Critical evaluation and enhancement
  • Reflection on problem-solving changes

Example: Verify AI finite element analysis boundary conditions and mesh quality

Backward Design for AI Assessment

Backward Design: Start with desired outcomes and work backward to design instruction and assessment (Wiggins & McTighe, 2005)

Three Stages Applied to AI-Aware Assessment:

Stage 1: Desired Results

  • What should students know and be able to do with AI in professional practice?
  • How do these outcomes align with ABET criteria?
  • What are the enduring understandings about AI in engineering?

Stage 2: Assessment Evidence

  • What evidence will show that students have achieved these outcomes?
  • How can we assess both technical competence and AI literacy?
  • What authentic tasks demonstrate professional-level AI integration?

Stage 3: Learning Plan

  • What learning experiences will help students develop these capabilities?
  • How do we scaffold AI literacy development?
  • What sequence builds from novice to expert AI use?

Theoretical Integration Summary

Theoretical Framework Integration

Key Takeaway: These theories work together to inform every aspect of AI-aware assessment design, from learning objectives to grading rubrics to implementation strategies.

Think-Pair-Share: Connecting Theory to Your Practice

Activity (5 minutes)

Think (2 minutes): Which of these theoretical frameworks most resonates with your current teaching philosophy? How might it change your approach to assessment?

Pair (2 minutes): Share your thoughts with a neighbor. Discuss one specific assessment challenge you face and how these theories might help address it.

Share (1 minute): Volunteers share key insights with the group

This reflection helps you connect abstract theory to your concrete teaching context

Part I Summary: Theoretical Foundations

What We’ve Established: A research-based foundation for AI-aware assessment design

Key Theories Covered:

  • Self-Regulated Learning: Forethought → Performance → Reflection
  • Metacognitive Awareness: Knowledge + Regulation
  • Constructivist Learning: ZPD, social construction, active engagement
  • Backward Design: Outcomes → Evidence → Learning plan

Practical Implications:

  • Assessment must evaluate learning processes, not just products
  • Students need scaffolding to develop AI literacy
  • Multiple pathways support diverse learners
  • Professional outcomes drive design decisions

Connection to Learning Objectives: You can now apply cognitive science principles to design effective AI-aware assessment strategies ✓

II. The Assessment Challenges Reframed

Part II Roadmap: Assessment Challenges

Learning Objectives Focus: Identify key challenges and design equitable assessment practices

What We’ll Cover:

  • Authorship ambiguity reframed
  • Foundational vs. enhanced skills (ZPD)
  • Process vs. product evaluation
  • Accessible and inclusive pedagogy
  • Interconnected solutions framework

Key Questions We’ll Answer:

  • How do we move beyond “cheating” concerns?
  • What can students do independently vs. with AI?
  • How do we assess learning processes?
  • Who benefits from AI integration?

By the End of Part II: You’ll see assessment challenges as opportunities for better pedagogy

Challenge 1: Authorship Ambiguity Through Learning Theory Lens

Traditional View:

  • “Who did the work?”
  • Binary thinking: student or AI
  • Focus on detection and prevention
  • Adversarial relationship

Constructivist Reframe:

  • “How did learning happen?”
  • Collaborative knowledge construction
  • Focus on process and reflection
  • Partnership in learning

Self-Regulated Learning Perspective:

  • Forethought: How did student plan AI use?
  • Performance: How did student monitor AI interaction?
  • Reflection: What did student learn from the process?

Metacognitive Awareness Focus:

  • Can student articulate their thinking process?
  • Do they recognize AI’s strengths and limitations?
  • Can they transfer learning to new contexts?

Assessment Implication: Evaluate the quality of human-AI collaboration, not just the final product

Practical Application: Instead of asking “Did the student cheat?”, ask “How effectively did the student collaborate with AI to construct knowledge?”

Challenge 2: Foundational vs. Enhanced Skills Through ZPD

Independent Performance Zone:

  • Core engineering principles
  • Fundamental calculations
  • Basic problem-solving strategies
  • Professional judgment foundations

Assessment: AI-restricted components

Assisted Performance Zone:

  • Complex system analysis
  • Advanced optimization
  • Novel problem exploration
  • Creative solution generation

Assessment: AI-enhanced components with documentation

Scaffolding Progression:

  1. Novice: AI use with heavy guidance and templates
  2. Developing: Structured AI interaction with reflection
  3. Proficient: Strategic AI use with critical evaluation
  4. Expert: Innovative AI integration with professional judgment

Assessment Strategy: Design assessments that evaluate both zones while supporting progression between them

Concrete Example:

  • Independent Zone: In-class thermodynamic cycles exam (no AI)
  • Assisted Zone: Take-home power plant optimization (with AI documentation)
  • Progression Support: Scaffolded practice from simple to complex

Challenge 2: Foundational vs. Enhanced Skills Through ZPD (CONT.)

Scaffolding Progression:

  1. Novice: AI use with heavy guidance and templates
  2. Developing: Structured AI interaction with reflection
  3. Proficient: Strategic AI use with critical evaluation
  4. Expert: Innovative AI integration with professional judgment

Assessment Strategy: Design assessments that evaluate both zones while supporting progression between them

Concrete Example:

  • Independent Zone: In-class thermodynamic cycles exam (no AI)
  • Assisted Zone: Take-home power plant optimization (with AI documentation)
  • Progression Support: Scaffolded practice from simple to complex

Challenge 3: Process vs. Product Through SRL Lens

Self-Regulated Learning Process Assessment:

Forethought Assessment

  • Goal clarity and appropriateness
  • Strategic planning quality
  • Self-efficacy calibration
  • Task analysis accuracy

Example: “I will use AI to explore design alternatives, then verify the most promising option manually”

Performance Assessment

  • Self-monitoring effectiveness
  • Strategy implementation
  • Help-seeking appropriateness
  • Adaptation to feedback

Example: Documentation of prompt refinement based on AI response quality

Reflection Assessment

  • Self-evaluation accuracy
  • Causal attribution quality
  • Learning transfer evidence
  • Strategy adaptation

Example: “AI helped me consider factors I hadn’t thought of, but I needed to verify the thermal properties”

Key Insight: The learning process is as important as the final product for developing professional competence

Challenge 4: Accessible and Inclusive Pedagogy

Universal Design for Learning (UDL)

  • Multiple means of representation: Various AI tools and interfaces
  • Multiple means of engagement: Different AI interaction styles
  • Multiple means of expression: Diverse documentation formats

Cultural-Historical Activity Theory

  • Recognize cultural differences in AI interaction
  • Account for varying technological backgrounds
  • Support diverse learning communities

Self-Determination Theory

  • Autonomy: Choice in AI use strategies
  • Competence: Scaffolded skill development
  • Relatedness: Community support for learning

Growth Mindset Theory

  • AI literacy as developable skill
  • Mistakes as learning opportunities
  • Effort and strategy over innate ability

Assessment Implication: Design multiple pathways to demonstrate competence while maintaining rigorous standards

Interconnected Challenges Framework

Assessment Challenges Integration

Key Insight: These challenges are interconnected and must be addressed through a coherent theoretical framework, not piecemeal solutions

Part II Summary: Assessment Challenges

What We’ve Reframed: Traditional assessment challenges as opportunities for better learning

Challenges Addressed:

  • Authorship: From “who did it?” to “how did learning happen?”
  • Skills: Independent vs. assisted performance zones
  • Evaluation: Process AND product assessment
  • Accessibility: Multiple pathways to demonstrate competence

Key Insights:

  • Evaluate human-AI collaboration quality
  • Design assessments for both ZPD zones
  • Learning process = professional competence
  • Coherent framework beats piecemeal solutions

Connection to Learning Objectives: You can now identify key challenges and design equitable practices ✓

III. The Paradigm Shift: From Detection to Development

Part III Roadmap: Paradigm Shift

Learning Objectives Focus: Apply cognitive science principles and identify implementation challenges

What We’ll Cover:

  • Change management theory applied
  • Some faculty concerns and resistance
  • From binary to spectrum thinking

Key Questions We’ll Answer:

  • How do we manage institutional change?
  • How do we address faculty resistance?
  • What does the new paradigm look like?

By the End of Part III: You’ll understand how to lead assessment transformation in your context

Change Management Theory Applied

Kotter’s 8-Step Change Process applied to assessment transformation:

Steps 1-4: Creating Climate for Change

  1. Create Urgency: AI is disrupting traditional assessment
  2. Build Coalition: Early adopter faculty and administrators
  3. Develop Vision: AI-literate engineering graduates
  4. Communicate Vision: Benefits for students and profession

Steps 5-8: Engaging and Enabling

  1. Empower Action: Provide tools and training
  2. Generate Wins: Start with pilot successes
  3. Sustain Acceleration: Scale successful practices
  4. Institute Change: Embed in institutional culture

Faculty Development Implication: Change requires sustained support, not just one-time training

Change Management Theory Applied

Current Status Check:

  • Where is your institution in this change process?
  • Creating urgency? ✓ (AI disruption is evident)
  • Building coalition? (Need early adopters and champions)
  • Developing vision? (AI-literate engineering graduates)
  • Communicating benefits? (Professional preparation focus)
  • Empowering action? (Need training and resources)
  • Generating wins? (Pilot successes needed)
  • Sustaining acceleration? (Scaling successful practices)
  • Instituting change? (Embedding in institutional culture)

Some Faculty Concerns and Resistance

Common Concerns:

“This will lower academic standards”

  • Response: Higher-order thinking skills become more important
  • Evidence: Focus shifts to critical evaluation and professional judgment

“Students won’t learn fundamentals”

  • Response: AI-restricted components ensure foundational knowledge
  • Evidence: Layered assessment model maintains rigor

“It’s too much work to redesign everything”

  • Response: Start small with pilot approaches
  • Evidence: Templates and scaffolding reduce workload

Resistance Factors:

Cognitive: “I don’t understand AI well enough” Solution: Provide hands-on training and peer mentoring

Emotional: “This threatens my expertise” Solution: Frame as professional development opportunity

Behavioral: “I don’t have time to change” Solution: Provide ready-to-use templates and gradual implementation

Social: “My colleagues aren’t doing this” Solution: Build community of practice and early adopter network

From Binary to Spectrum Thinking

Old Paradigm: Binary Thinking

  • AI use: Yes or No
  • Assessment: Traditional or AI-proof
  • Students: Honest or Cheating
  • Faculty: Adopter or Resistor

New Paradigm: Spectrum Thinking

  • AI integration: Levels of sophistication
  • Assessment: Multiple approaches for different purposes
  • Students: Developing professional skills
  • Faculty: Learning and adapting together

Cognitive Flexibility Theory Application:

  • Multiple representations of complex concepts
  • Context-dependent knowledge application
  • Adaptive expertise development
  • Transfer across varied situations

Assessment Implication: Design flexible assessment systems that can adapt to different contexts and skill levels

Quick Poll: Where Are You Now?

Audience Response (2 minutes)

Question: Which best describes your current approach to AI in assessment?

  1. Detection Focus: Trying to prevent/detect AI use
  2. Avoidance: Ignoring the AI issue entirely
  3. Experimentation: Trying different approaches
  4. Documentation: Requiring AI use transparency
  5. Integration: Systematically incorporating AI literacy

Part III Summary: Paradigm Shift

What We’ve Established: A roadmap for transforming assessment culture

Change Management Insights:

  • Kotter’s 8-Step Process: Systematic approach to change
  • Faculty Concerns: Cognitive, emotional, behavioral, social
  • Spectrum Thinking: Beyond binary AI use decisions

Practical Strategies:

  • Address resistance with evidence and support
  • Build coalitions of early adopters
  • Frame as professional development opportunity
  • Provide ready-to-use templates and guidance

Connection to Learning Objectives: You understand the change process and can address implementation challenges ✓

IV. The Assessment Alignment Framework Redesigned

Part IV Roadmap: Assessment Framework

Learning Objectives Focus: Implement the Assessment Alignment framework using backward design principles

What We’ll Cover:

  • Progressive complexity model (4 levels)
  • Self-regulated learning integration
  • Assessment approach selection guide
  • Decision tree for implementation
  • Practical models with examples

Key Questions We’ll Answer:

  • How do students progress in AI literacy?
  • Which assessment approach fits my context?
  • How do I scaffold student development?
  • What does implementation look like?
  • How do I choose the right model?

By the End of Part IV: You’ll have concrete frameworks for designing AI-aware assessments

Progressive Complexity Model

Expertise Development Theory: Skills develop through predictable stages from novice to expert (Dreyfus & Dreyfus, 2005)

Level 1: Novice AI User

  • Follows prescribed AI interaction patterns
  • Uses basic prompts with guidance
  • Accepts AI outputs with minimal evaluation
  • Assessment Focus: Process documentation with templates

Engineering Example: Student uses provided prompt template to get AI help with basic circuit analysis, documents the interaction using a structured form.

Level 2: Developing AI User

  • Creates structured prompts for routine tasks
  • Recognizes common AI limitations
  • Applies basic verification strategies
  • Assessment Focus: Comparative analysis with scaffolding

Engineering Example: Student solves thermodynamics problem both manually and with AI, then compares approaches and identifies where AI made assumptions.

Progressive Complexity Model

Expertise Development Theory: Skills develop through predictable stages from novice to expert (Dreyfus & Dreyfus, 2005)

Level 3: Proficient AI User

  • Designs strategic AI interaction sequences
  • Systematically evaluates AI outputs
  • Enhances AI contributions with domain knowledge
  • Assessment Focus: Critical evaluation with reflection

Engineering Example: Student uses AI for initial structural design, then systematically verifies load calculations, checks code compliance, and optimizes for real-world constraints.

Level 4: Expert AI User

  • Innovates AI integration approaches
  • Anticipates and mitigates AI limitations
  • Creates novel solutions building on AI contributions
  • Assessment Focus: Meta-learning and professional application

Engineering Example: Student develops novel AI-assisted design methodology, documents limitations, and creates verification protocols for professional use.

Self-Regulated Learning Integration

Mapping Assessment Approaches to SRL Phases:

Forethought Phase

Process Documentation

  • Goal setting for AI use
  • Strategic planning documentation
  • Self-efficacy assessment

AI-Restricted Components

  • Independent goal setting
  • Strategy selection without AI
  • Self-efficacy calibration

Performance Phase

Comparative Analysis

  • Strategy implementation monitoring
  • Real-time adaptation documentation
  • Help-seeking behavior analysis

Critical Evaluation

  • Self-monitoring of AI interaction
  • Quality control processes
  • Error detection and correction

Self-Reflection Phase

Meta-Learning Assessment

  • Self-evaluation of outcomes
  • Causal attribution analysis
  • Strategy adaptation planning

All Approaches

  • Reflection on learning process
  • Transfer to new contexts
  • Professional development planning

Self-Regulated Learning Integration

Mapping Assessment Approaches to SRL Phases:

Key Insight: Each assessment approach supports different aspects of self-regulated learning development

Practical Application: Design assessment sequences that move students through SRL phases:

  1. Novice: Guided forethought with templates → Structured performance documentation → Prompted reflection
  2. Expert: Independent goal setting → Self-monitored performance → Sophisticated meta-analysis

Assessment Approach Selection Guide

Simple Decision Framework: Choose based on your primary objective and student level

If Your Primary Goal is…

Foundational Knowledge Assessment:

→ Use AI-Restricted Components

  • In-class exams, timed assessments
  • Core competency verification
  • Independent problem-solving

AI Skill Development:

→ Use Process Documentation

  • Structured AI interaction logs
  • Template-guided reflection
  • Scaffolded practice

If Your Students Are…

New to AI (Novice Level): → Process Documentation with heavy scaffolding

Some AI Experience (Developing): → Comparative Analysis with guided reflection

Comfortable with AI (Proficient): → Critical Evaluation with independent verification

Advanced AI Users (Expert): → Meta-Learning Assessment with professional application

Key Principle: Start where students are, scaffold toward expertise, maintain rigor throughout

Assessment Alignment Decision Tree

Assessment Alignment Decision Tree

Part IV Summary: Assessment Framework

What We’ve Built: A comprehensive framework for AI-aware assessment design

Framework Components:

  • Progressive Complexity: Novice → Expert development
  • SRL Integration: Forethought, performance, reflection phases
  • Decision Support: Simple selection criteria
  • Implementation Guide: Clear pathways for adoption

Key Takeaways:

  • Students develop AI literacy in predictable stages
  • Assessment approaches map to learning phases
  • Multiple models support different contexts
  • Start where students are, scaffold toward expertise

Connection to Learning Objectives: You can now implement the Assessment Alignment framework using backward design ✓

V. Practical Models Grounded in Learning Theory

Part V Roadmap: Practical Models

Learning Objectives Focus: Implement frameworks and evaluate student work effectively

What We’ll Cover:

  • Scaffolded Assessment Pyramid
  • Comparative Analysis with metacognition
  • Documentation-based reflective practice
  • Model selection guidance
  • Framework integration summary

Key Questions We’ll Answer:

  • How do I structure progressive assessment?
  • What does comparative analysis look like?
  • How do I assess reflective practice?
  • Which model fits my course?
  • How do models work together?

By the End of Part V: You’ll have specific models ready for immediate implementation

Model 1: Scaffolded Assessment Pyramid

Scaffolding Theory: Temporary support structures that help learners achieve goals they cannot reach independently (Wood et al., 1976, see Bruner, 2006)

Level 1: Core Competencies (Novice)

  • Independent demonstration of fundamental knowledge
  • Basic problem-solving without AI assistance
  • Conceptual understanding verification
  • Theory: Cognitive load management and ZPD

Level 2: Guided Practice (Developing)

  • Structured AI interaction with documentation
  • Comparative analysis of AI vs. traditional approaches
  • Reflection on learning process and outcomes
  • Theory: Self-regulated learning

Level 3: Critical Application (Proficient)

  • Systematic evaluation and enhancement of AI outputs
  • Integration of multiple AI tools and approaches
  • Domain-specific verification strategies
  • Theory: Metacognitive regulation and critical thinking

Level 4: Professional Integration (Expert)

  • Strategic AI use in complex, novel problems
  • Innovation and creative problem-solving
  • Professional judgment and ethical considerations
  • Theory: Transfer and adaptive expertise

Model 2: Comparative Analysis with Metacognitive Scaffolding

Phase 1: Metacognitive Planning

  • “What do I already know about this problem type?”
  • “What are my goals for each approach?”
  • “How will I evaluate success?”
  • “What challenges do I anticipate?”

Phase 2: Dual Problem-Solving

  • Traditional approach with self-monitoring
  • AI-assisted approach with interaction documentation
  • Real-time reflection on strategy effectiveness
  • Adaptation based on intermediate results

Phase 3: Metacognitive Evaluation

  • “Which approach was more effective and why?”
  • “What did I learn about my own thinking process?”
  • “How did AI change my problem-solving approach?”
  • “When would I use each approach in the future?”

Grading Distribution Example:

  • Traditional solution accuracy: 25%
  • AI-assisted solution quality: 25%
  • Comparative analysis depth: 30%
  • Metacognitive reflection: 20%

Model 3: Documentation-Based Reflective Practice

Reflective Practice Theory: Professional development through systematic reflection on experience (Schön, 1983)

Reflection-in-Action

  • Real-time monitoring during AI interaction
  • Immediate adjustments to prompts and strategies
  • Recognition of AI limitations as they occur

Assessment: Live documentation of thought processes

Reflection-on-Action

  • Post-task analysis of AI interaction effectiveness
  • Evaluation of outcomes and process quality
  • Identification of lessons learned

Assessment: Structured reflection essays with prompts

Reflection-for-Action

  • Forward-looking planning based on experience
  • Strategy adaptation for future tasks
  • Professional development goal setting

Assessment: Action plans and strategy refinement

Model 3: Documentation-Based Reflective Practice

Reflective Practice Theory: Professional development through systematic reflection on experience (Schön, 1983)

Professional Connection: This mirrors how practicing engineers reflect on their use of new tools and technologies

Implementation Strategy:

  • Week 1-2: Focus on reflection-in-action (real-time documentation)
  • Week 3-8: Add reflection-on-action (post-task analysis)
  • Week 9-16: Integrate reflection-for-action (forward planning)

Assessment Portfolio: Students submit documentation from all three reflection types, showing professional development progression

Model Selection: Quick Self-Assessment

Reflection Moment (1 minute)

Which of these three models best fits a specific course you teach?

  • Scaffolded Pyramid: For courses with diverse student AI experience levels
  • Comparative Analysis: For courses where traditional methods are well-established
  • Reflective Documentation: For senior design or capstone courses

Take a moment to identify one course where you could pilot one approach

Key Insight: You don’t need to transform everything at once - start with one model in one course

Framework Integration: Putting It All Together

The Complete Assessment Ecosystem:

Student Level Primary Approach Supporting Theory Assessment Focus Professional Outcome
Novice Process Documentation ZPD + Scaffolding Guided AI interaction Basic AI literacy
Developing Comparative Analysis SRL + Metacognition Critical thinking Strategic AI use
Proficient Critical Evaluation Expertise Development Independent verification Professional judgment
Expert Meta-Learning Reflective Practice Innovation & transfer AI leadership

Implementation Principle: Use multiple approaches within a single course to support different students and different learning objectives

Example Course Design: - Weeks 1-4: AI-restricted fundamentals + process documentation - Weeks 5-8: Comparative analysis assignments - Weeks 9-12: Critical evaluation projects - Weeks 13-16: Meta-learning portfolio and professional application

Part V Summary: Practical Models

What We’ve Developed: Three concrete models for immediate implementation

Models Covered:

  • Scaffolded Pyramid: Progressive complexity with clear levels
  • Comparative Analysis: Traditional vs. AI with metacognitive scaffolding
  • Reflective Documentation: Professional practice development
  • Model Integration: Using multiple approaches together

Implementation Insights:

  • Start with one model in one course
  • Build complexity over time
  • Support different learning styles
  • Connect to professional practice

Connection to Learning Objectives: You have specific frameworks ready for implementation ✓

VI. Research-Based Assessment and Grading

Part VI Roadmap: Grading and Implementation

Learning Objectives Focus: Evaluate student work using research-based rubrics and design equitable practices

What We’ll Cover:

  • AI literacy rubrics grounded in expertise research
  • Comprehensive grading examples
  • Implementation strategies and timelines
  • Equity and inclusion frameworks
  • Your action plan and next steps

Key Questions We’ll Answer:

  • How do I grade AI-enhanced work fairly?
  • What does good AI literacy look like?
  • How do I implement change systematically?
  • How do I ensure equitable outcomes?
  • What are my immediate next steps?

By the End of Part VI: You’ll have rubrics, implementation plans, and concrete next steps

AI Literacy Rubric Grounded in Expertise Research

Competency 1: AI Tool Operation

Level Cognitive Characteristics Observable Behaviors Assessment Evidence
Novice Rule-based thinking Follows prescribed prompts Uses provided templates
Developing Pattern recognition Adapts prompts to context Creates structured interactions
Proficient Strategic thinking Designs prompt sequences Demonstrates tool orchestration
Expert Intuitive expertise Innovates interaction approaches Creates novel workflows

Theoretical Grounding: This progression reflects the novice-to-expert development pattern found across professional domains

Competency Assessment with Metacognitive Focus

Competency 2: Critical Evaluation

Novice Level:

  • Metacognitive Knowledge: Limited awareness of AI limitations
  • Regulation: Basic error checking
  • Assessment: “Did you identify any errors in the AI output?”

Developing Level:

  • Metacognitive Knowledge: Recognizes common AI failure modes
  • Regulation: Systematic verification using references
  • Assessment: “What verification strategies did you use?”

Proficient Level:

  • Metacognitive Knowledge: Understands context-dependent AI reliability
  • Regulation: Multi-method verification approach
  • Assessment: “How did you adapt your verification to this specific problem?”

Expert Level:

  • Metacognitive Knowledge: Anticipates AI limitations before they occur
  • Regulation: Proactive error prevention strategies
  • Assessment: “How did you design your AI interaction to minimize errors?”

Additional AI Literacy Competencies

Competency 3: Prompt Engineering (Strategic Knowledge)

Novice Level:

  • Uses simple, direct prompts
  • Minimal context provided
  • Limited iteration on prompts
  • Assessment: “Describe your prompting approach”

Developing Level:

  • Includes relevant technical parameters
  • Provides necessary context
  • Iterates based on initial results
  • Assessment: “How did you refine your prompts?”

Proficient Level:

  • Crafts prompts with precise specifications
  • Includes constraints and boundary conditions
  • Uses systematic iteration strategies
  • Assessment: “Demonstrate your prompt development process”

Expert Level:

  • Develops multi-step prompting strategies
  • Anticipates AI limitations in prompt design
  • Uses domain-specific terminology effectively
  • Assessment: “How do you optimize prompts for engineering contexts?”

Formative vs. Summative Assessment Theory Application

Assessment for Learning vs. Assessment of Learning (Black & Wiliam, 1998)

Formative Assessment (For Learning)

AI Literacy Development:

  • Regular check-ins on AI interaction quality
  • Peer feedback on documentation approaches
  • Self-assessment using rubric criteria
  • Instructor feedback on process improvement

Benefits:

  • Supports skill development
  • Reduces anxiety about AI use
  • Builds metacognitive awareness
  • Encourages experimentation

Summative Assessment (Of Learning)

Professional Competency Demonstration:

  • Portfolio of AI-enhanced work
  • Comprehensive project with AI integration
  • Professional scenario simulation
  • Transfer to novel contexts

Benefits:

  • Validates achievement levels
  • Provides accountability
  • Demonstrates program outcomes
  • Supports credentialing decisions

Strategy: Use formative assessment to build skills, summative assessment to validate achievement

Grading Philosophy Shift

Traditional Grading Paradigm

  • Focus on final answer correctness
  • Individual achievement emphasis
  • Fixed standards and expectations
  • One-time performance evaluation
  • Deficit-based feedback

AI-Era Grading Paradigm

  • Focus on process quality and growth
  • Collaborative skill development
  • Adaptive standards for emerging skills
  • Continuous improvement tracking
  • Strength-based feedback with growth targets

Growth Mindset Integration:

  • AI literacy as developable skill set
  • Mistakes as learning opportunities
  • Effort and strategy recognition
  • Progress over perfection
  • Peer learning and support

Professional Relevance: This approach better prepares students for lifelong learning in a rapidly evolving technological landscape

Comprehensive Grading Example: Structural Design Project

Assignment: Design a pedestrian bridge using AI tools while demonstrating engineering competency (100 points total)

Traditional Grading (Old Approach)

  • Final design correctness: 40%
  • Calculations accuracy: 30%
  • Drawing quality: 20%
  • Report writing: 10%

Problems:

  • Doesn’t assess AI literacy
  • Ignores learning process
  • Binary pass/fail thinking

AI-Aware Grading (New Approach)

  • Technical accuracy: 25%
  • AI interaction quality: 20%
  • Critical evaluation: 20%
  • Enhancement & judgment: 20%
  • Process documentation: 15%

Benefits:

  • Assesses professional skills
  • Values learning process
  • Supports skill development

:::

Comprehensive Grading Example: Structural Design Project

Assignment: Design a pedestrian bridge using AI tools while demonstrating engineering competency (100 points total)

Sample Student Performance Analysis:

  • Technical Accuracy (22/25): Correct structural calculations, minor unit error caught and corrected
  • AI Interaction (16/20): Good prompts but limited iteration, could improve specificity
  • Critical Evaluation (18/20): Excellent verification methods, identified AI limitation with dynamic loads
  • Enhancement (19/20): Significant improvements beyond AI suggestions, applied professional judgment
  • Documentation (13/15): Clear process record, reflection could be deeper
  • Total: 88/100 with specific feedback for improvement

Part VI Summary: Updated Grading Approaches

What We’ve Established: Evidence-based approaches to evaluating AI-enhanced work

Grading Components:

  • AI Literacy Rubrics: 4-level competency progression
  • Comprehensive Examples: Before/after grading approaches
  • Implementation Strategies: Change management and equity
  • Practical Tools: Ready-to-use templates and checklists

Key Insights:

  • Focus on process quality and growth
  • Multiple competencies need assessment
  • Equity requires intentional design
  • Implementation needs systematic support

Connection to Learning Objectives: You can now evaluate student work using research-based rubrics and design equitable practices ✓

VII. Implementation Through Change Management Theory

Organizational Change Management

Kotter’s 8-Steps for AI Assessment Transformation:

Steps 1-2: Building Foundation

  1. Create Urgency: Share data on AI disruption of traditional assessment
  2. Build Coalition: Form AI assessment task force with diverse stakeholders

Steps 3-4: Developing Strategy

  1. Develop Vision: “Graduates prepared for AI-integrated professional practice”
  2. Communicate Vision: Regular updates, success stories, professional relevance

Steps 5-6: Implementing Change

  1. Empower Action: Provide training, resources, and institutional support
  2. Generate Wins: Celebrate pilot successes and early adopter achievements

Steps 7-8: Sustaining Change

  1. Sustain Acceleration: Scale successful practices, continuous improvement
  2. Institute Change: Embed in promotion criteria, accreditation standards

Complete Learning Objectives Review

Congratulations! You’ve achieved all five learning objectives:

✓ Learning Objective 1:

Identify key challenges - Authorship ambiguity → collaboration quality - Skills separation → ZPD zones - Process vs. product → both matter

✓ Learning Objective 2:

Apply cognitive science principles - Self-regulated learning phases - Metacognitive awareness components - Constructivist learning principles - Backward design framework

✓ Learning Objective 3:

Implement Assessment Alignment framework - Progressive complexity model - Decision support system - Practical implementation models - Framework integration approach

✓ Learning Objective 4:

Evaluate using research-based rubrics - AI literacy competency levels - Expertise development progression - Comprehensive grading examples - Multiple assessment dimensions

✓ Learning Objective 5:

Design equitable practices - Universal Design for Learning - Multiple means of demonstration

You’re now prepared to transform assessment in the age of generative AI!

VIII. Your Action Plan and Next Steps

Immediate Actions (This Week)

Self-Assessment and Planning

  1. Identify Your Starting Point: Are you an Innovator, Early Adopter, Early Majority, Late Majority, or Laggard?
  2. Audit One Course: Choose one specific course where AI has disrupted traditional assessment
  3. Map to Theory: Does SRL, metacognition, or constructivism best fit your teaching philosophy?
  4. Select Initial Approach: Choose Scaffolded Pyramid, Comparative Analysis, or Reflective Documentation

Concrete First Steps

  1. Download Templates: Get AI documentation forms from workshop materials
  2. Identify One Assignment: Select a specific assignment to redesign this semester
  3. Draft Policy Language: Write 2-3 sentences about AI use for your syllabus
  4. Test AI Tools: Spend 30 minutes using ChatGPT or similar tool for your discipline

Build Support Network

  1. Find 2-3 Colleagues: Identify potential collaborators in your department

Short-Term Implementation (Next Month)

Pilot Testing

  1. Start Small: Implement one AI-aware assessment in one course
  2. Document Everything: Keep detailed notes on what works and what doesn’t
  3. Gather Feedback: Survey students on their experience
  4. Iterate Quickly: Make adjustments based on initial results

Professional Development

  1. Join Communities: Connect with AI in education groups
  2. Share Experiences: Present at department meetings or conferences
  3. Seek Mentorship: Find experienced AI assessment practitioners
  4. Continue Learning: Stay current with research and best practices

Long-Term Vision (This Academic Year)

Systematic Integration

  1. Scale Successful Practices: Expand to multiple courses and assignments
  2. Develop Expertise: Progress from novice to proficient AI assessment user
  3. Mentor Others: Support colleagues in their adoption journey
  4. Contribute to Knowledge: Share successes and lessons learned

Professional Impact

  1. Student Outcomes: Measure impact on learning and engagement
  2. Program Alignment: Connect to ABET outcomes and program goals
  3. Institutional Change: Advocate for supportive policies and resources
  4. Professional Recognition: Seek opportunities to lead in this area

Your 90-Day Implementation Roadmap

Days 1-30: Foundation

  • Complete self-assessment
  • Choose one course to pilot
  • Select one assessment model
  • Draft syllabus language
  • Test AI tools personally
  • Identify 2-3 colleague allies
  • Customize one template
  • Plan first pilot assignment

Days 31-60: Implementation

  • Launch pilot assignment
  • Collect student feedback
  • Document what works/doesn’t
  • Refine approach based on results
  • Share experience with colleagues
  • Attend AI education webinar
  • Plan second pilot assignment
  • Begin building evidence base

Days 61-90: Expansion

  • Scale to additional assignments
  • Mentor interested colleague
  • Present at department meeting
  • Develop discipline-specific examples
  • Create student training materials
  • Plan full course integration
  • Document lessons learned
  • Advocate for institutional support

Success Metric: By day 90, you should have evidence that your AI assessment approach improves student learning outcomes

Reflection and Commitment

Personal Reflection Questions

“What is one specific change I will make to my assessment practices this semester?”

“How will this change benefit my students’ learning and professional preparation?”

“What support do I need to be successful in this change?”

Public Commitment

Share your commitment with a colleague or write it down

Tomorrow’s Workshop Preview

Building on Today’s Foundation:

Day 1: Deep Implementation

  • Hands-on template customization
  • Discipline-specific examples
  • Peer collaboration and feedback
  • Troubleshooting common challenges

Day 2: Advanced Strategies

  • Complex assessment design
  • Institutional change planning
  • Student training and support
  • Technology integration

Day 3: Professional Development

  • Leadership in AI assessment
  • Research and scholarship opportunities
  • Community building and networking
  • Long-term vision and planning

Preparation for Tomorrow:

  • Bring specific assessment challenges
  • Come ready to collaborate
  • Prepare to share and learn

Thank You and Discussion

Questions for Discussion?

Contact and Resources

Andrew Katz
Faculty Development Workshop Orchestrator
Generative AI in Engineering Education

Workshop Materials: - All slides and handouts available online

Let’s transform assessment together!

References

  • Bruner, J. S. (2006). In search of pedagogy volume I: The selected works of Jerome Bruner, 1957-1978. Routledge.
  • Dreyfus, H. L., & Dreyfus, S. E. (2005). Peripheral vision: Expertise in real world contexts. Organization studies, 26(5), 779-792.
  • Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry. American psychologist, 34(10), 906.
  • McPherson, G. E., & Zimmerman, B. J. (2002). Self-regulation of musical learning. The new handbook of research on music teaching and learning, 2, 327-347.
  • Spiro, R. J., Coulson, R. L., Feltovich, P. J., & Anderson, D. K. (2019, October). Cognitive flexibility theory: Advanced knowledge acquisition in ill-structured domains. In 10th Annual Conference Cognitive Science Society Pod (pp. 375-383). Psychology Press.
  • Wiggins, G. P., & McTighe, J. (2005). Understanding by design. Ascd.