Developing AI-Powered Methodologies for Engineering Education Research and Practice
2025-11-06
Welcome
Original Abstract
The integration of artificial intelligence and natural language processing (NLP) into engineering education research and practice represents a set of opportunities to expand how we understand student learning, assess educational outcomes, and scale qualitative research methods. This talk presents a comprehensive overview of methodological innovations developed in the IDEEAS Lab over the past five years to leverage the power of generative AI, large language models, and NLP techniques for advancing engineering education. I will present three interconnected methodological contributions: (1) AI-assisted qualitative analysis, where I demonstrate how generative text models can perform traditional thematic data analysis while maintaining rigor and validity; (2) AI-assisted assessment and feedback systems, showcasing applications of open-source language models for analyzing student responses, teamwork feedback, and ethics case studies; and (3) mental models research at scale, illustrating how NLP techniques can help uncover engineering faculty and student mental models of complex topics like sustainability, assessment, and now generative AI itself.
Original Abstract
The integration of artificial intelligence and natural language processing (NLP) into engineering education research and practice represents a set of opportunities to expand how we understand student learning, assess educational outcomes, and scale qualitative research methods. This talk presents a comprehensive overview of methodological innovations developed in the IDEEAS Lab over the past five years to leverage the power of generative AI, large language models, and NLP techniques for advancing engineering education. I will present three interconnected methodological contributions: (1) AI-assisted qualitative analysis, where I demonstrate how generative text models can perform traditional thematic data analysis while maintaining rigor and validity; (2) AI-assisted assessment and feedback systems, showcasing applications of open-source language models for analyzing student responses, teamwork feedback, and ethics case studies; and (3) mental models research at scale, illustrating how NLP techniques can help uncover engineering faculty and student mental models of complex topics like sustainability, assessment, and now generative AI itself.
Concept of a Plan
- Context: AI/NLP in engineering education research and practice
- What you’ll hopefully take away today:
- How we have applied LLMs in 3 different EngEd use cases
- Methodological lessons learned
- Opportunities and challenges ahead
Motivation: Why LLMs in EngEd?
- Opportunities:
- Expand how we understand processes
- Scale qualitative methods
- Insight into mental models
- Insight into decision-making processes
Example Use Cases
Examples Overview
- Data types
- Surveys/open-ended questions
- Interviews
- Think-aloud protocols
- Design artifacts
- Logs
- Study contexts
- Courses
- Makerspaces
- Workplace
- Online settings
- Roles for LLMs
- Data collection
- Elicitation
- Scaffolding
- Simulation
- Analysis
- Summarization
- Pattern-finding
- Coding
- Data collection
Use Case 1
“Thematic” Analysis of Large-Scale Qualitative Data
Example Study
Large-Scale Reddit Analysis of Leaving CS (JEE, 2025)
- Research question: Why do students and professionals leave CS?
- Data: Reddit posts and comments (mentioning CS and leaving)
- LLM/NLP role: Thematic coding and pattern discovery at scale
- Prompting strategy: Domain-adapted prompts; few-shot with edge cases
- Validation/benchmark: Sampled human checks; sensitivity analyses
- Main findings and caveats: Multi-factor reasons; sampling bias; ethics of social data
Workflow: GATOS pipeline
Workflow: GATOS pipeline drawing
Why leave CS? (Ross & Katz, 2025)
Use Case 2: Evaluation
Teammate Comment Quality
Use Case 2: Evaluation
Teammate Comment Quality
Example rubric (Drinkwater et al., 2025)
Use Case 2: Evaluation
Teammate Comment Quality
Results Table (Drinkwater et al., 2025)
Use Case 3:
Mental Models Research
Use Case 3
Part 1: Figurative Language Detection
Use Case 3
Part 1: Figurative Language Detection
- On learning and AI:
- Pushing through mental barriers when you want to quit and when it’s hard
- The knowledge becomes embedded in your thinking
- The illusion of learning from superficial digital content
- On AI policies:
- A legal and procedural minefield
- Dual-level approach—similar to biosafety protocols
- Competing with students is an arms race
Use Case 3
Part 1: Figurative Language Detection
- Research question: To what extent can LLMs be used to detect and classify figurative language (in educational contexts)?
- Data: Short semi-structured texts (e.g., interviews)
- LLM/NLP role: Identify candidate examples of figurative language
- Prompting strategy: persona assignment, formatting variation, output mode
Figurative Language Detection
Example language
- Life is a journey.
- Life is a journey. Some paths led to success while others ended in disappointment.
- She packed a small suitcase and left home at eighteen. Life is a journey.
| Rank | Model | Accuracy | Precision | Recall | F1 | Backend |
|---|---|---|---|---|---|---|
| 🥇 1 | Qwen3-Next-80B | 95.4% | 99.9% | 92.6% | 96.1% | MLX |
| 🥈 2 | GLM-4-9B | 93.1% | 94.9% | 94.2% | 94.5% | MLX |
| 🥉 3 | GLM-4.5-Air | 91.3% | 94.1% | 92.1% | 93.1% | MLX |
| 4 | Qwen3 30B | 91.0% | 92.7% | 92.8% | 92.8% | Ollama |
| 5 | Gemma3 12B | 89.6% | 90.0% | 93.7% | 91.8% | Ollama |
| 6 | Qwen3 4B | 88.9% | 97.1% | 84.7% | 90.5% | Ollama |
| 7 | Granite4 Small | 88.8% | 93.1% | 88.5% | 90.8% | Ollama |
| 8 | Gemma3 27B | 87.7% | 85.7% | 96.2% | 90.7% | Ollama |
| 9 | Qwen2.5 14B | 86.8% | 87.3% | 92.1% | 89.7% | Ollama |
| 10 | Mistral-Small 3.2 | 86.2% | 84.3% | 95.5% | 89.5% | Ollama |
| 11 | GPT-OSS 20B | 85.3% | 83.3% | 95.5% | 89.0% | Ollama |
| 12 | Mistral-Small 2501 | 79.9% | 79.3% | 91.4% | 84.9% | Ollama |
Figurative Language Results
Prompt format
| Format | Accuracy | Rank |
|---|---|---|
| All-Caps | 87.7% | 1 (tied) |
| Plain Text | 87.7% | 1 (tied) |
| Numbered | 87.6% | 3 |
| Emoji | 87.3% | 4 |
| Markdown | 87.1% | 5 |
| XML | 87.0% | 6 |
Figurative Language Results
Persona Impact
| Persona | Accuracy | Rank |
|---|---|---|
| Technical Writer | 88.2% | 1 |
| Computational Linguist | 87.9% | 2 |
| Neutral | 87.5% | 3 |
| Literary Scholar | 87.4% | 4 |
| Cognitive Scientist | 87.1% | 5 (tied) |
| English Teacher | 87.1% | 5 (tied) |
| Creative Writer | 86.7% | 7 |
Specific LLM Personas
Use Case 3
Part 2: Identifying Decisions and Factors
“We decided to implement carbon pricing to internalize emission externalities, even though we recognize the opposing impact on low-income households.”
Use Case 3
Part 2: Identifying Decisions and Factors
“We decided to implement carbon pricing to internalize emission externalities, even though we recognize the opposing impact on low-income households.”
Use Case 3
Part 2: Identifying Decisions and Factors
- Data
- Synthetic data
- Vary number of decisions and factors
- Vary domains (e.g., climate change, healthcare, education)
- Tests
- Models
- Personas
- Prompt formats
- Architectures
Architectures
- Single-pass
- All decisions and all factors in one shot
- Blind two-pass
- Decisions first, then factors
- Semantic two-pass
- Decisions first, then factors with relevant context
Experimental Results
Experimental Results
Use Case 3
Part 3: Causal Beliefs
Brief Detour
Directed Acyclic Graphs (DAGs)
example from Pearl (2009)
Brief Detour
Causal Influence Diagrams
Simple example of causal influence diagram (Everitt et al., 2019)
Brief Detour
Causal Influence Diagrams
Complex example causal influence diagram (Everitt et al., 2019)
Use Case 3
Part 3: Causal Beliefs
Closeup of Example Inductive Causal Belief Network
Suggestions
Prompt Engineering in Educational Contexts
- Pattern library:
- Role + constraints + examples
- Chain-of-thought alternatives (rubrics, checklists)
- Few-shot with counter-examples
- Robustness checks:
- Paraphrase prompts; seed variation; temperature sweeps
- Adversarial/edge cases from human coders
- Reproducibility:
- Prompt versioning, random seeds, model cards
prompt library
Examples of Personas in Prompts
“You are an expert in student evaluations of teaching. Your job is to generate a codebook for the following comments.”
“You are an expert qualitative researcher. Your job is to code the following comments.”
Implement as part of regular prompt or system prompt
Examples of Prompt Formatting
Examples of Output Specification
Validating AI-Generated Qualitative Codes
- Agreement metrics:
- Krippendorff’s alpha,
- Cohen’s kappa,
- exact/partial match
- Unitization:
- segments vs utterances
- window sizes
- Adjudication workflow:
- AI propose → human review → consensus rules
- Error typology:
- Overgeneralization
- Missed nuance
- Hallucination
- Deeper epistemological considerations
- Reporting standards:
- Use appendices to report prompts used
- Discuss patterns for testing/iterating on prompts
Study Design with LLMs
- When to include AI:
- Exploratory vs confirmatory phases
- Pre-registration and guardrails
- Sampling and power for qualitative+AI hybrid designs
- IRB and ethics:
- Data privacy, model terms, participant consent
- Fairness and bias audits
- Tooling:
- Open-source pipelines; notebooks; containers; reproducibility
Opportunities and Challenges Ahead
- Open-source tool development and dissemination:
- Packaging, documentation, examples, tutorials
- Community benchmarks and shared datasets
- Maintenance and reproducibility
- Implications for qualitative paradigms:
- Reliability/validity reframed; reflexivity with AI
- Division of labor: sensemaking vs sorting
- Curricula and pedagogy:
- Integrating LLMs responsibly into courses
- Assessment implications
Practical Guidance for Attendees
- Decision checklist:
- Is your task classification, extraction, generation, …something else?
- Data sensitivity; model choice; evaluation plan
- Human-in-the-loop touchpoints
- Starter templates:
- Prompt pattern for coding
- Validation plan scaffold
- Reproducibility checklist
Discussion
References
Studies + Additional Resources
Study 1a: Katz, A., Shakir, U., & Chambers, B. (2023). The utility of large language models and generative AI for education research. arXiv preprint arXiv:2305.18125.
Study 1b: Ross, A., & Katz, A. (2025). Using generative AI for large‐scale qualitative analysis of social media posts to understand why people leave computer science. Journal of Engineering Education, 114(4), e70036.
Katz, A., Gerhardt, M., & Soledad, M. (2024). Using generative text models to create qualitative codebooks for student evaluations of teaching. International Journal of Qualitative Methods, 23, 16094069241293283.
Anakok, I., Katz, A., Chew, K. J., & Matusovich, H. (2025). Leveraging Generative Text Models and Natural Language Processing to Perform Traditional Thematic Data Analysis. International Journal of Qualitative Methods, 24, 16094069251338898.
Everitt, T., Ortega, P. A., Barnes, E., & Legg, S. (2019). Understanding agent incentives using causal influence diagrams. Part I: Single action settings. arXiv preprint arXiv:1902.09980.
Pearl, J. (2009). Causality. Cambridge university press.
Extra Material
Figurative Language Experiment Results
figurative language experiment results
Major Suggestions to High School Students
Major suggestions
Surprisal-based identification
surprisal results for causal statements
Online QDA platform
QDA platform
