Design Decisions for AI-Assisted Qualitative Data Analysis

A Walk-Through of Practical Choices When Building LLM-Based Workflows

Andrew Katz

Virginia Tech

2026-01-22

Quick Poll: Your Experience

Raise your hand if you have…

✋ Conducted qualitative data analysis before?
✋ Worked with datasets of 100+ observations?
✋ Used any AI tools in your research workflow?
✋ Felt overwhelmed by the volume of qualitative data?

The Challenge

Traditional Qualitative Analysis

Rich, nuanced insights
Labor-intensive
Doesn’t scale well
Consistency challenges across coders

The Scale Problem

Hundreds of survey responses
Thousands of social media posts
Years of student evaluations
Large interview corpora

The Promise of AI

Can large language models help automate parts of qualitative data analysis while maintaining rigor?

Key Tension:

Efficiency vs. Quality
Automation vs. Human Insight
Scale vs. Depth

Today’s Talk: Design Decisions

I’ll walk through the key choices we make when building AI-assisted qualitative workflows:

Workflow Structure - Mapping to established methods
Data Familiarization - Summarization approaches
Clustering & Embeddings - Grouping similar ideas
Code Generation - The heart of the process (more time here)
Theme Identification - Finding higher-level patterns
Human-in-the-Loop - Where humans remain essential (more time here)

Examples from Our Published Work

Grounding in Established Methods

We base our workflows on Braun & Clarke’s Thematic Analysis (2006, 2022):

Phase	Traditional TA	Our AI Approach
1	Familiarizing with data	LLM summarization
2	Generating initial codes	Clustering + LLM coding
3	Searching for themes	LLM theme identification
4	Reviewing themes	Cosine similarity validation
5	Defining themes	LLM + human refinement
6	Writing up	Human-led

Why This Matters

Grounding in established methodology gives credibility and structure to AI-assisted approaches.

The GATOS Workflow Overview

Design Decision #1: Data Familiarization

Traditional Approach: Read and re-read transcripts, take notes

AI Approach: Generate summary points from raw text

You are an expert text analyst reading written responses
collected in a study of faculty reactions to generative AI.

I need you to summarize the text in an enumerated list.
Each summary point should capture one distinct idea.
You MUST remove names and use gender-neutral pronouns.

If the text says "nothing", do not make up information.
There is a severe penalty for fabricating content.

Key Design Choices

Persona assignment (“expert text analyst”)
Context specification (data type, collection setting)
De-identification requirements
Guardrails against hallucination

Example: Summarization Input/Output

Input (Raw Response)

“I’m really dreading the idea of going back to the office full-time. Working from home has been a game-changer for me - I’ve been able to create a workspace that is comfortable and personalized to my needs…”

Output (Summary Points)

Dreading returning to the office full-time
Working from home has been a game-changer
Has created a comfortable workspace at home
Has created a personalized workspace at home

Tradeoff: Some nuance may be lost, but we can now process 1000+ responses systematically

Design Decision #2: Clustering Similar Ideas

Goal: Group semantically similar summary points together

Pipeline:

Clustering Pipeline

Technical Choices:

Component	Our Choice	Why
Embedding model	mxbai	Open-source, Apache 2.0 license
Dimension reduction	PCA → UMAP	Preserve structure, reduce noise
Clustering	Agglomerative	More homogeneous clusters

Try It Yourself: Mental Clustering

Quick Activity: Look at these summary points. How would you group them?

“likes flexible hours”
“concerned about commute time”
“prefers working from home”
“worried about traffic delays”
“values work-life balance”
“misses office social interactions”

Why Clustering Matters

Without Clustering

The model sees random, disconnected ideas:

“likes flexible hours”
“concerned about commute time”
“prefers working from home”
“worried about traffic delays”
“values work-life balance”
“misses office social interactions”

With Clustering

Similar ideas grouped together:

Cluster A:

“likes flexible work hours”
“prefers working from home”
“values work-life balance”

Cluster B:

“concerned about commute”
“worried about traffic delays”

Clustering enables the model to identify patterns rather than processing isolated points.

Design Decision #3: Code Generation

The Heart of the Process

Traditional approach: Read data, create code, check if code exists, repeat

Our approach: Mimic this with RAG (Retrieval-Augmented Generation)

The RAG-Based Code Generation Process

RAG Codebook Process

The Code Generation Prompt (Part 1)

Act as if you are the world's best qualitative data analyst
with expertise in generating qualitative codebooks for thematic analysis.

You specialize in creating parsimonious codebooks with
non-overlapping and non-redundant codes.

I need your help to create a qualitative codebook to analyze
written responses from a study of worker perspectives
on returning to the office after the pandemic.

Prompt Engineering Elements

Persona assignment - Establishes expertise and role
Specialization statement - Emphasizes parsimony (fewer, better codes)
Context - Data type and collection setting

The Code Generation Prompt (Part 2)

The model follows a structured reasoning process:

Step 1: Read existing codebook.
  Examine the codes in <existing_codebook> tag.

Step 2: Read the summary points.
  Identify the main theme in the current cluster.

Step 3: Try to use existing codebook.
  Can existing codes describe this theme?

Step 4: Create new code if needed.
  Only if existing codes are insufficient.

Step 5: Evaluate your suggestion.
  Check: Parsimony? Abstraction level? Non-redundancy?

Step 6: Final recommendation.
  "No new codes needed" OR provide code + definition

Evaluation Criteria in the Prompt

<evaluation_criteria>

Criterion 1. Parsimony:
  Have you made every effort to use existing codes
  before proposing a new one?

Criterion 2. Abstraction Level:
  Is any proposed new code at an appropriate level,
  consistent with existing codes?

Criterion 3. Non-Redundancy:
  Have you avoided creating codes that significantly
  overlap with existing ones?

There is a VERY significant penalty for creating
redundant or unnecessary codes.

</evaluation_criteria>

Why This Matters

These criteria force the model to think carefully before adding codes, preventing codebook bloat.

Discussion: What Would You Add?

Think-Pair-Share: What other evaluation criteria might be useful?

Take 60 seconds to think, then share in the chat.

Some ideas we’ve considered:

Theoretical alignment (does it fit the framework?)
Actionability (is this useful for the research question?)
Granularity consistency (same level of detail as others?)

Example: Model Reasoning

Cluster summary points:

requires seamless video conferencing
desires user-friendly video conferencing tools
up-to-date hardware and software required
hopes for reliable video conferencing tools

Nearest existing codes:

adaptability to change
technological challenges
flexible scheduling

Model’s reasoning:

“The existing code ‘technological challenges’ partially captures the issues but does not fully encompass the specific needs mentioned… A new code is needed to capture the specific technological requirements.”

Output: Code: reliable video conferencing tools

Evidence It Works: Code Generation Rate

Code Generation Rate

Key insight: The curve bends away from the 45° line, showing the model reuses existing codes rather than always creating new ones.

Design Decision #4: Theme Generation

After generating codes, we need higher-level themes.

Prompt structure:

You are an expert qualitative researcher specializing
in thematic analysis. Your task is to analyze a list
of codes and identify potential themes following
the guidance of Braun and Clarke.

The goal is to identify themes that help answer
the research question: "{research_question}"

Look for patterns and shared meanings among the codes.
Remember that themes don't simply "emerge" -
actively construe relationships among the codes.

Theme Generation: Structured Output

We ask for JSON output to ensure parseable results:

{
  "suggested_themes": [
    {
      "theme_name": "Flexibility in Work Arrangements",
      "concept": "Employees value control over where and when they work",
      "codes": [
        "flexible work hours",
        "hybrid work flexibility",
        "remote work preferences"
      ],
      "relationship": "These codes share concern for autonomy..."
    }
  ],
  "reflection": {
    "broad_or_narrow_themes": "...",
    "potential_subthemes": "..."
  }
}

Design Decision #5: Open Source vs. Proprietary

Why Open Source?

Data privacy (no third-party transmission)
Reproducibility (specific model versions)
Cost (no API fees at scale)
Transparency (know what you’re using)

Models We Use

Task	Model
Summarization	Llama 3.1-8b
Embedding	mxbai
Coding	Mistral-22b
Themes	Mistral-large-2
Transcription	Whisper

Reproducibility Tip

Set temperature = 0 for deterministic outputs. Document exact model versions.

Design Decision #6: Human-in-the-Loop

The AI is a tool, not a replacement for researcher judgment.

Where Humans Remain Essential

Human in the Loop

Human Validation Points

Phase	Human Role	Why It Matters
After summarization	Check for missed content	Models may oversimplify
After clustering	Verify groupings make sense	Semantic similarity ≠ conceptual similarity
After coding	Review code definitions	Ensure alignment with research questions
After themes	Refine, merge, or discard	Apply domain expertise
Final analysis	Contextualize with theory	Connect to existing literature

Reflection: Where Do You Draw the Line?

Discussion Question:

At what point does AI assistance become “too much” for qualitative research?

When AI generates initial codes?
When AI identifies themes?
When AI writes interpretations?
Never - it’s always just a tool?

There’s no universal answer - it depends on your epistemological stance and research goals.

Validation Example: Ross & Katz (2025)

Task: Classify Reddit posts about leaving computer science

Approach:

Human researcher labeled a subset manually
AI model labeled the same subset
Compared using Cohen’s κ

Result: κ = 0.72 (substantial agreement)

Validation with Synthetic Data

From the GATOS paper: Test on data with known themes

Process:

Generate synthetic data with specified themes/sub-themes
Run GATOS workflow
Compare generated codebook to original themes

Results:

Metric	Value
Sub-themes with clear match	>95%
Sub-themes with partial match	~3%
Sub-themes with no match	<2%

Example matches:

Original Sub-Theme	GATOS Output
“effective use of feedback mechanisms”	“feedback mechanism effectiveness”
“resistance to traditional office hours”	“resistance to traditional office hours”

Cosine Similarity for Theme Validation

Idea: Check if themes actually represent the original data

Cosine Similarity Validation

If the top matches make sense, the theme is well-grounded in data.

Limitations to Acknowledge

Model bias from training data may influence outputs
Replicability challenges as models evolve
Some redundancy may persist in generated codes
Computational intensity at very large scale
Nuance loss in summarization step
Cultural context may not be captured

Key Takeaways

Principles for AI-Assisted Qualitative Workflows

Ground in established methodology (e.g., Braun & Clarke)
Use open-source models for privacy and reproducibility
Invest in prompt engineering - it’s where the magic happens
Maintain human-in-the-loop at critical decision points
Validate with multiple approaches (κ, cosine similarity, synthetic data)
Document everything for reproducibility

What Will You Try?

Commit to one thing: What’s one idea from today you might explore in your own work?

Try summarizing a small dataset with an LLM?
Experiment with prompt engineering for coding?
Read one of the papers mentioned?
Discuss with your research team?

Feel free to share in the chat!

Resources

Papers:

Anakok et al. (2025) - GATA method, IJQM
Katz, Fleming & Main (2024) - GATOS workflow, arXiv
Ross & Katz (2025) - CS attrition study, JEE

Code:

GitHub: github.com/andrewskatz/ijqm-gata-2025

Contact:

Email: [email protected]
Website: ideeaslab.com

These Slides:

Available online after the session

Questions & Discussion

Let’s discuss!

What concerns do you have about these approaches?
How might this fit into your research workflow?
What barriers do you see to adoption?

I’m happy to chat more during the break or via email!

Appendix: Full Code Generation Prompt

Act as if you are the world's best qualitative data analyst with expertise
in generating qualitative codebooks for thematic analysis. You specialize
in creating parsimonious codebooks with non-overlapping and non-redundant codes.

<task_instructions>
We are trying to determine whether an existing codebook is sufficient for
analyzing one written response. Your task is to analyze summaries and determine
if the theme is already covered by existing codes or if a new code is needed.

Step 1: Read existing codebook.
  Examine the existing codebook in the <existing_codebook> tag.

Step 2: Read the summary points.
  Read the summaries in <text_to_analyze> and identify the main theme.

Step 3: Try to use existing codebook.
  Attempt to describe the main theme using existing codes.
  If no new code needed, say "No new codes needed".

Step 4: Create new code if needed.
  If existing codes are insufficient, create a new code.
  Provide: Code (short phrase) + Definition

Step 5: Evaluate your suggestion.
  Check against evaluation criteria:
  - Parsimony: Used existing codes first?
  - Abstraction Level: Consistent with existing codes?
  - Non-Redundancy: No significant overlap?

Step 6: Final recommendation.
</task_instructions>

Appendix: Theme Generation Prompt

You are an expert qualitative researcher specializing in thematic analysis.
Your task is to analyze a list of codes and identify potential themes
following the guidance of Braun and Clarke.

<instructions>
Step 1. Review the list of codes in <codes> tag.

Step 2. Look for patterns and shared meanings among the codes.

Step 3. Identify overarching narratives representing broader themes.

Step 4. Remember that themes don't simply "emerge" -
        actively construe relationships among codes.

Step 5. Consider importance and salience of potential themes.
        The number of codes is less important than meaningful patterns.

Step 6. Aim for distinctive yet coherent themes.

Step 7. Be willing to let go of codes that don't fit.

Step 8. Strive for balance - not too many, not too few themes.

Step 9. For each theme, provide: name, concept, codes, relationships.

Step 10. Reflect on your analysis.
</instructions>

Present your analysis in JSON format.