Engineering Education Research Methods Workshop

Important Concepts for Building Stronger Research

Andrew Katz, Associate Professor of Engineering Education

2025-10-24

Welcome

  • Who: UVA engineering faculty curious about education research
  • What: 90-minute speed-run through five core concepts
  • Outcome: Leave with an aligned design sketch + concrete next steps

Today’s Roadmap

  1. Alignment foundations
  2. Theory & frameworks
  3. Causality & claims
  4. Attending to uncertainty
  5. Quality criteria

Each section’s format: content → applications → guidance → mini-activity (Workbook Sections A–E).

Alignment Foundations

Alignment as Systems Engineering

  • Research design = interconnected system.
  • Misalignment at any node (question, theory, methods, analysis, claims) destabilizes the whole.
  • Think like an engineer: interfaces between components must match specifications.

The Alignment Cascade

Paradigm → Theory → Questions → Methods/Data → Analysis & Claims → Quality & Transparency

  • Start upstream (paradigm & theory) before selecting methods.
  • Downstream pieces (claims, quality language) must reflect upstream choices.

Examples of Paradigms

  • Positivist / Post-positivist
    • Tend to view reality as objective and measurable.
    • Emphasize testing relationships and establishing causation.
  • Interpretivist
    • Tend to view reality as subjective and contextual.
    • Emphasize meaning, experience, and context.
  • Critical / Transformative
    • Tend to view reality as powerful and transformative.
    • Emphasize power, equity, and action.

Paradigms Drive Alignment

  • Positivist/post-positivist
    • Causality, measurement, control.
  • Interpretivist
    • Meaning, experience, context.
  • Critical/transformative
    • Power, equity, action.
  • Paradigm choice sets expectations for data, evidence, and quality standards.

Alignment Pitfalls

  • Paradigm drift (interpretive question, survey-only methods).
  • Theory mismatch (grand theory named, never operationalized).
  • Method-claim mismatch (correlational data, causal language).
  • Quality confusion (judging qual work with quant criteria).

Application

Suboptimal Alignment (Maker Spaces)

Claim: “Maker spaces cause higher retention.”
Design executed: Compare schools with/without maker spaces.
Misalignment: No theory, no controls, causal language without causal design.
Fix: Identify paradigm (causal? interpretive?), choose theory (belonging, communities of practice), redesign methods, reframe claims.

Application

Better Alignment (Flipped Circuits Study)

  • Paradigm: Post-positivist.
  • Theory: Cognitive load theory → predicts distributed practice supports learning.
  • Design: Randomized section assignment, pre/post tests, controls for prior GPA.
  • Claims: “Flipped format produced a 0.4 SD gain under these conditions.”
  • Quality: Internal validity, transparency about context.

Guidance

Alignment Checklist

  1. Name paradigm & theory explicitly.
  2. State research question in paradigm-friendly language.
  3. Choose methods that allow theory to “do work.”
  4. Preview claims that fit design level.
  5. Match quality criteria to paradigm.

Guidance

Alignment Questions to Ask

  • “How does my theory show up in data collection or analysis?”
  • “Can my methods answer this question, in this paradigm?”
  • “What language will I use to describe findings?”
  • “Which quality criteria will reviewers expect?”

Theory & Frameworks

Theory as Design Engine

  • Blueprint (positivist)
    • Derive hypotheses, specify variables, predict relationships.
  • Lens (interpretivist)
    • Sensitize to meaning, guide coding, allow theory to emerge/refine.
  • Critique (critical)
    • Reveal power, design for action, foreground positionality.

Translating Theory into Constructs

  1. Identify constructs: What does the theory say matters?
  2. Operationalize/observe: Measures, artifacts, behaviors, narratives.
  3. Predict mechanisms: Why should change or understanding occur?
  4. Plan analysis: Statistical tests, thematic patterns, critical discourse moves.
  5. Claim alignment: Language reflects theoretical stance (growth, transfer, liberation).

Social Cognitive Career Theory (Lent, Brown, & Hackett, 2002)

Self-regulated Learning Theory (Zimmerman, 2002)

Gee’s Social Identity Theory (Gee, 2000)

Combining Frameworks Thoughtfully

  • Complementary: Pair predictive + interpretive frameworks (e.g., Expectancy-Value + Identity).
  • Sequential: Use interpretive work to surface constructs, then test via survey.
  • Overlay critical lens: Examine who benefits, whose voices are missing.
  • Guardrail: Explain why each framework is needed and how they integrate.

Theory Drift Warning Signs

  • Framework listed in intro but missing from methods/analysis.
  • Constructs don’t appear in instruments or coding scheme.
  • Claims contradict theoretical stance (“Growth mindset worked because… grit”).
  • Reviewer question: “Why did you choose this theory?” → answer is vague.

Application

Theory in Action (Makerspace Persistence)

  • Paradigm
    • Post-positivist + interpretive follow-up.
  • Frameworks
    • Self-Determination Theory + Communities of Practice.
  • Measures
    • Autonomy/competence/relatedness surveys; participation logs.
  • Analysis
    • Mixed-method integration to explain persistence.

Application

Critical Lens Example (Inclusive Design Teams)

  • Paradigm: Post-positivist / interpretive mixed approach.
  • Framework: Theory of Planned Behavior (TPB).
  • Context: Encouraging faculty to adopt inclusive project rubrics.
  • TPB Constructs: Attitudes toward inclusive practices, subjective norms, perceived behavioral control.
  • Design moves:
    • Survey faculty on TPB constructs before/after workshop.
    • Interview select faculty about implementation experiences.
    • Track actual adoption of rubric elements.
  • Claims: “Faculty with increased perceived control and supportive norms were more likely to implement inclusive rubrics; interviews revealed specific barriers.”

Guidance

Moving from Theory to Methods

  • Map each construct to at least one data source.
  • Ensure sampling captures the actors your theory highlights.
  • Use theory to craft interview protocols or observational rubrics.
  • Explicitly connect analytic codes/themes back to theoretical constructs.

Guidance

Documenting Theory Use

  • Write a paragraph in your workbook: “This theory helps me… therefore I will…”
  • Note complementary frameworks and how they integrate.
  • Prepare to justify theory choice to a skeptic (disciplinary fit, evidence base).

Activity

Alignment Check (Workbook Section A)

  1. Write your paradigm + theory on top of your puzzle.
  2. Draft a one-sentence research question.
  3. Rate alignment (1–4).
  4. Note one KEEP and one TWEAK.
  5. 2-minute partner share using Alignment Coach prompt.
⏱️ Activity 6 min

Activity

Theory Integration (Workbook Section B)

  1. List your primary theory/framework. *if you don’t have one, try looking around for one(s) online that might be relevant for your RQ
  2. Identify 2–3 constructs and how you will capture them.
  3. Note the mechanism the theory predicts.
  4. Partner prompt: “Where will your theory appear in data collection or analysis?”
  5. Capture adjustments in workbook.
⏱️ Activity 6 min

Causality & Claims

Hierarchy of Causal Evidence

  1. Randomized experiments
  2. Quasi-experiments (matching, RDD, DiD)
  3. Observational/Correlational studies
  4. Qualitative/Critical explorations (mechanism/meaning)

Remember: Claim strength should match your rung on the ladder.

Directed Acyclic Graphs (DAGs)

  • Visualize causal pathways.
  • Identify confounders (variables affecting both treatment and outcome).
  • Guide data collection and analysis.
  • Communicate assumptions and limitations.

Four DAGs (McElreath, 2018)

Confounder Playbook

  • Learner variables
    • Prior knowledge, motivation, identity.
  • Instructor/course
    • Pedagogy, class size, materials.
  • Contextual
    • Institution type, resources, culture.
  • Self-selection
    • Who opts into the ‘intervention’.

Suggestions: control, design around, or acknowledge.

Mechanisms Matter

  • Theory clarifies why a change should occur.
  • Map expected pathways (e.g., “peer mentoring → belonging → persistence”).
  • Collect data on mediators (belonging scores, narratives, observational notes).
  • Mechanisms strengthen claims even when causality is limited.

Claim Language Spectrum

Evidence level Appropriate language Avoid
Strong causal “Caused,” “produced,” “reduced” “Proved,” “guaranteed”
Suggestive “Likely contributed,” “after controlling for…” “Made,” “ensured”
Correlational “Associated with,” “students in X reported…” “Caused”
Qualitative/Critical “Participants described…,” “structures perpetuated…” “Everyone experienced,” “resulted in”

Language signals clarity about design limits.

Application 1

Office Hours Example

  • Observation: Students who attend office hours earn higher grades.
  • Confounders: Motivation, prior knowledge, scheduling flexibility.
  • Adjustments: Collect baseline data, match students, soften claims (“associated with”).
  • Mechanism: Office hours → targeted feedback → improved mastery.

Application 2

Maker Course RCT

  • Design: Random assignment to flipped vs. traditional sections.
  • Controls: Prior GPA, instructor, assessment.
  • Claims: “Flipped format caused a 0.4 SD gain under these conditions.”
  • Transparency: Limited to sophomore circuits at UVA; future replication needed.

Guidance

Aligning Claims with Design

  • Write your claim in plain language, underline verbs.
  • Ask: “Does my evidence justify that verb?”
  • Define the scope (population, context, timeframe).
  • Prepare alternative wording if reviewers push back.

Guidance

Causal Strategy Options

  • Strengthen design (randomization, counterfactuals).
  • Measure key confounders and control statistically.
  • Add qualitative data to explain mechanisms.
  • Be explicit about limitations and next steps.

Activity

Causality Check-in (Workbook Section C)

  1. Sketch a mini DAG (treatment → outcome, confounders).
  2. Identify at least two confounders + mitigation plan.
  3. Draft claim language that matches your design.
  4. Partner plays Causal Skeptic; adjust as needed.
⏱️ Activity 6 min

Attending to Uncertainty

Why Uncertainty Is Inevitable

  • Human behavior varies; contexts shift.
  • Instruments imperfectly capture constructs.
  • Researcher presence influences qualitative data.
  • Critical work deals with evolving power dynamics.

Accepting (and acknowledging) uncertainty = honest science.

Sources of Uncertainty

  • Measurement: Instrument validity, observer bias.
  • Sampling: Convenience samples, small n, context-bound settings.
  • Time: Longitudinal drift, historical events.
  • Researcher influence: Positionality, interpretation, rapport.
  • Other: Specific to your study (e.g., attrition, Hawthorne effect).

Suggestion: Catalog your top uncertainties early.

Communicating Uncertainty Well

  • Acknowledge in the limitation.
  • Explain mitigation steps (controls, member checks, triangulation).
  • Describe residual unknowns.
  • Try to avoid more definitive language.
  • Suggest future work or caution for application.
  • Be sensitive to claims about “proving” something.

Transparency increases trust with reviewers.

Uncertainty Language Examples

  • Quant: “Results may be influenced by unmeasured motivation; we controlled for prior GPA.”
  • Qual: “Findings reflect 18 students in context X; readers should judge transferability.”
  • Critical: “Insights co-created with participants; sustained change depends on institutional support.”

Consider keeping phrases ready to insert into manuscripts or talks.

Application

Retention Study Limitation

  • Study: Survey of sophomore intentions to persist.
  • Uncertainty: Self-report bias, single institution.
  • Response: Weight responses, triangulate with enrollment data, cautious claims.

Application

Qualitative Coding Example

  • Study: Interpretive interviews on failure experiences.
  • Uncertainty: Researcher positionality, transferability.
  • Response: Reflexive memoing, member checking, thick description.

Guidance

Uncertainty Planning Grid

Source Mitigation Residual risk Language
Measurement Validate instrument Limited constructs “Measures capture…”
Sampling Purposeful variation Context specific “Findings apply when…”
etc.

Try to fill this out in workbook Section D in a few minutes…

Guidance

Turning Uncertainty into Strength

  • Use uncertainty to motivate mixed-method integration.
  • Frame limitations as design choices (depth over breadth).
  • Highlight transparency as evidence of rigor.

Activity

Uncertainty Mapping (Workbook Section D)

  1. List top three uncertainties for your study.
  2. Note mitigation or transparency strategy for each.
  3. Draft one sentence per uncertainty using preferred language.
  4. Share one uncertainty statement with neighbor.
⏱️ Activity 5 min

Quality Criteria

Quality Across Paradigms

  • Positivist
    • Internal/external validity, reliability.
  • Interpretivist
    • Credibility, transferability, dependability, confirmability.
  • Critical
    • Authenticity, catalytic validity, positionality, reciprocity.

Quality = doing the right things for your paradigm, not applying one-size-fits-all rules.

Building Quality In (Lifecycle)

  • Design
    • Alignment, sampling strategy, instrument choice.
  • Data collection
    • Consistent protocols, trust building, reflexivity.
  • Analysis
    • Triangulation, negative cases, transparency.
  • Reporting
    • Thick description, limitations, positionality statements.

Key Takeaway: Plan quality moves at each stage.

Demonstrating Quality to Reviewers

  • Document procedures (audit trail, appendices).
  • Provide evidence of checks (inter-rater reliability, member feedback).
  • Show alignment with theoretical commitments.
  • Anticipate skepticism (“How do we know this is trustworthy?”).

Ethics & Positionality

  • Reflect on your relationship to participants.
  • Acknowledge potential biases and power dynamics.
  • Describe how you negotiated ethics, reciprocity, or action steps.
  • Critical work: include how findings will be used for change.

Application

Quant Quality Example

  • Study: Adaptive homework intervention.
  • Quality moves: Random assignment, power analysis, validated measures, preregistered analysis plan.
  • Transparency: Share instruments, code, acknowledge limited generalizability.

Application

Qual/Critical Quality Example

  • Study: Engineering design studio ethnography.
  • Quality moves: Prolonged engagement, multiple data sources, participant feedback on findings.
  • Positionality: Researcher as former engineer; reflexive notes included.
  • Outcomes: Credibility (design changes implemented).

Guidance

Quality Move Tracker

Phase Planned actions Evidence
Design e.g., pilot instruments Pilot summary
Collection e.g., double-coding 20% Reliability coeff
Analysis e.g., negative case search Audit memo
Reporting e.g., thick description Appendices

Fill out in workbook Section E.

Guidance

Quality Language Cheat Sheet

  • “We established credibility by…”
  • “Transferability is supported through…”
  • “Internal validity evident in…”
  • “Reliability ensured via…”

Use precise language tied to paradigm expectations.

Activity

Quality Game Plan (Workbook Section E)

  1. Identify one quality move per research phase.
  2. Note the evidence you’ll provide (appendix, audit trail, stats).
  3. Draft one sentence defending rigor to a skeptical colleague.
  4. Share with partner; capture suggestions.
⏱️ Activity 6 min

Pulling It Together

Commit & Connect

  • One alignment adjustment I’ll make before Nov. 15 is…
  • Support/resource I need is…
  • Who will I share this plan with?
⏱️ 3 min

Next Steps & Resources

  • Slides, workbook template, and resource list coming via email and on ideeaslab.com

Thank you! Reach out at [email protected].

References

Gee, J. P. (2000). Chapter 3: Identity as an analytic lens for research in education. Review of research in education, 25(1), 99-125.

Lent, R. W., Brown, S. D., & Hackett, G. (2002). Social cognitive career theory. Career choice and development, 4(1), 255-311.

McElreath, R. (2018). Statistical rethinking: A Bayesian course with examples in R and Stan. Chapman and Hall/CRC.

Zimmerman, B. J. (2002). Becoming a self-regulated learner: An overview. Theory into practice, 41(2), 64-70.

IDEEAS Lab • Engineering Education