Welcome

• Context: AI/NLP in engineering education research and practice
• What you’ll take away today:
  • How we applied LLMs in 3 different EngEd studies
  • Methodological lessons learned
  • Opportunities and challenges ahead

Roadmap

• Motivation and framing
• Examples across study contexts and data types
• Methodological lessons:
  • Prompt engineering strategies
  • Validating AI-generated qualitative codes vs human coding
  • Study design strategies with LLMs in/out of the loop
• Emerging opportunities and challenges
• Discussion

Motivation: Why LLMs in EngEd?

• Opportunities: expand how we understand processes; scale qualitative methods
• Constraints:
  • “validity”
  • ethics
  • data access
  • reproducibility
• Positioning within EngEd research agendas

Framing and Key Terms

• LLMs vs traditional NLP (aka, what’s new and what isn’t)
• “Human-in-the-loop” for qualitative research
• What counts as “analysis” when AI is involved?

Example Studies Overview

• Data types: surveys/open-ended questions, interviews, think-aloud protocols, design artifacts, logs, forums
• Study contexts: courses, makerspaces, capstone, outreach, workplace, online settings
• Roles for LLMs:
  • During data collection (elicitation, scaffolding, simulation)
  • During analysis (coding, summarization, pattern-finding)
  • During interpretation (triangulation, explanation)…?

Study 1: Codebooks for Student Evaluations of Teaching (IJQM, 2024)

• Research question: Can LLMs help generate usable qualitative codebooks for SET?
• Data: Open-ended student comments from course evaluations
• LLM/NLP role: Codebook generation and refinement
• Prompting strategy: Role + constraints + exemplars; few-shot with counter-examples
• Validation/benchmark: Agreement vs human codebook; coder adjudication
• Main findings and caveats: Comparable coverage; nuance gaps; reproducibility practices

Workflow: SET codebook generation

Study 2: Human-in-the-Loop Analysis of Short Interviews (IJQM, 2025)

• Research question: How to combine LLMs and human coders for something similar to thematic analysis?
• Data: Short semi-structured interviews
• LLM/NLP role: Summarization, first-pass coding suggestions
• Prompting strategy: checklists; adversarial prompts
• Main findings and caveats: Efficiency gains; error modes; where humans are essential

Workflow: human-in-the-loop

Study 3: Large-Scale Reddit Analysis of Leaving CS (JEE, 2025)

• Research question: Why do students and professionals leave CS?
• Data: Reddit posts and comments (mentioning CS and leaving)
• LLM/NLP role: Thematic coding and pattern discovery at scale
• Prompting strategy: Domain-adapted prompts; few-shot with edge cases
• Validation/benchmark: Sampled human checks; sensitivity analyses
• Main findings and caveats: Multi-factor reasons; sampling bias; ethics of social data

Workflow: GATOS pipeline

Prompt Engineering in Educational Contexts

• Pattern library:
  • Role + constraints + examples
  • Chain-of-thought alternatives (rubrics, checklists)
  • Few-shot with counter-examples
• Robustness checks:
  • Paraphrase prompts; seed variation; temperature sweeps
  • Adversarial/edge cases from human coders
• Reproducibility:
  • Prompt versioning, random seeds, model cards

prompt library

Examples of Personas in Prompts

• “You are an expert in student evaluations of teaching. Your job is to generate a codebook for the following comments.”

• “You are an expert qualitative researcher. Your job is to code the following comments.”

• Implement as part of regular prompt or system prompt

Examples of Prompt Formatting

• Use markdown for readability, e.g.,

  # Heading 1
  ## Heading 2
  - Bullet 1
  - Bullet 2

• Use XML for structured output, e.g.,

  <codebook>...</codebook>

• Have sections for introduction, instructions, and output format

Examples of Output Specification

• Specify output format:

  • Text (free-form or structured)

  • JSON (for structured output), e.g.,

    {"codes": ["code1", "code2"], "themes": ["theme1", "theme2"]}`

  • XML (for complex hierarchies), e.g.,

    <codebook>...</codebook>

Validating AI-Generated Qualitative Codes

• Agreement metrics: Krippendorff’s alpha, Cohen’s kappa, exact/partial match
• Unitization: segments vs utterances; window sizes
• Adjudication workflow:
  • AI propose → human review → consensus rules
• Error typology:
  • Overgeneralization, missed nuance, hallucination
  • Deeper epistemological considerations
• Reporting standards:
  • Use appendices to report prompts used
  • Discuss patterns for testing/iterating on prompts

Study Design with LLMs

• When to include AI:
  • Exploratory vs confirmatory phases
  • Pre-registration and guardrails
• Sampling and power for qualitative+AI hybrid designs
• IRB and ethics:
  • Data privacy, model terms, participant consent
  • Fairness and bias audits
• Tooling:
  • Open-source pipelines; notebooks; containers; reproducibility

Cross-Study Synthesis: Lessons Learned

• What worked consistently
• What failed and why
• Transferability across data types/contexts
• Cost/time trade-offs and sustainability

Opportunities and Challenges Ahead

• Open-source tool development and dissemination:
  • Packaging, documentation, examples, tutorials
  • Community benchmarks and shared datasets
  • Maintenance and reproducibility
• Implications for qualitative paradigms:
  • Reliability/validity reframed; reflexivity with AI
  • Division of labor: sensemaking vs sorting
• Curricula and pedagogy:
  • Integrating LLMs responsibly into courses
  • Assessment implications

Practical Guidance for Attendees

• Decision checklist:
  • Is your task classification, extraction, generation, …something else?
  • Data sensitivity; model choice; evaluation plan
  • Human-in-the-loop touchpoints
• Starter templates:
  • Prompt pattern for coding
  • Validation plan scaffold
  • Reproducibility checklist

Discussion

References

• Study 1: Katz, A., Gerhardt, M., & Soledad, M. (2024). Using generative text models to create qualitative codebooks for student evaluations of teaching. International Journal of Qualitative Methods, 23, 16094069241293283.

• Study 2: Anakok, I., Katz, A., Chew, K. J., & Matusovich, H. (2025). Leveraging Generative Text Models and Natural Language Processing to Perform Traditional Thematic Data Analysis. International Journal of Qualitative Methods, 24, 16094069251338898.

• Study 3: Ross, A., & Katz, A. (2025). Using generative AI for large‐scale qualitative analysis of social media posts to understand why people leave computer science. Journal of Engineering Education, 114(4), e70036.