Chulalongkorn University AI Competency Assessment · Thai Hospital Personnel
Explore the Test Bank →
01 / 15

Methodology Document · Draft for project team review

AI Competency Assessment
for Thai Hospital Personnel

Framework, test design, generation pipeline, and item-level maps — built by a hybrid human–AI method grounded in real Thai hospital AI tools.

Back-Office Staff Medical Professionals Hospital Executives Chulalongkorn University
02 / 15

Overview

The shape of the assessment

5
AI competencies (C1–C5)
2
Axes — construct × depth
3
Audiences, distinct tests
6
Build phases, human + AI

One framework, three audience-specific tests, expanded by pipeline into 462 total item variants for cohort test-security.

03 / 15

Framework · Two axes

Construct × Depth

Construct — what format

Knowledge → MCQ. Recognise what is going on / what to do. ~30–90s.

Skill → open task. Produce a response, scored on a rubric. ~3–5 min.

Depth — what level of reasoning

Foundational — grasp what's going on day-to-day.

Applied — bring it to bear on one concrete case.

Advanced — set the standing rule / policy / strategy.

Depth measures reasoning, not seniority — a junior can show Applied; a senior may show only Foundational.

04 / 15

Framework · The five competencies

C1–C5

C1AI Foundations & Tool Literacy. What AI is and how it behaves — ML vs rules, LLM limits (hallucination, cutoff, bias).knowledge
C2AI-Enhanced Problem Solving. Using AI to solve problems and create value, individually and at the organisation.skill
C3Human–AI Collaboration Mindset. When human judgement leads, when to override, keeping accountability.knowledge → skill
C4Ethical, Safe & Responsible Use. Privacy (PDPA §26), bias, transparency, regulatory boundaries.knowledge → skill
C5Continuous Adaptation & Learning. Updating AI knowledge and practice as tools and rules evolve.skill
05 / 15

Framework · Construct mapping

How each competency is tested

C1
Knowledge at all depths — recognition throughout.
C2C5
Skill at all depths — performance throughout.
C3C4
Knowledge → Skill — knowledge at Foundational, skill at Applied and above.

The format always follows the construct it measures — recognition is asked as MCQ, production as an open task.

06 / 15

Framework · Operator vs governor

Two parallel tracks

Individual — the operator

Hands-on AI use by contributors. C2 / C5 progress by the complexity of one's own AI use.

  • C2 — discrete tasks → multi-step workflows → ambitious orchestration
  • C5 — seeks new tools → experiments → adapts as tech changes

Executive — the governor

Strategic AI deployment across the organisation. Never measured on operating a tool.

  • C2 — efficiency → defensible advantage → transformation & sourcing
  • C5 — keeps the team current → sets the org's staying-current strategy

Shared floor: C1 · C3 · C4 Foundational are word-for-word identical across both tracks.

07 / 15

Audiences

Three audiences, three tests

Back-office

Non-clinical staff

Foundational depth, all multiple-choice. 5 role variants from one 9-item master.

Medical

Clinical professionals

Foundational + Applied, integrated MCQs across competencies. 7 roles, 3 tracks.

Executives

AI-policy leaders

Part 1 — Applied MCQ (own AI use). Part 2 — Advanced open tasks (org rules & strategy).

08 / 15

Audiences · Medical

Three clinical tracks

Track 1

Gen-AI prompting

Conversational AI use on personal devices and scribes.

PresScribe

Track 2

Imaging AI

Interpreting an AI flag against a heatmap before reporting.

Inspectra CXR

Track 3

Real-time procedural AI

Alert-driven AI during live procedures.

gastroAI-model G

Items 1–3 differ by track; items 4–14 are shared.

09 / 15

Pipeline

Six-phase build

1Master designHuman-authored anchor test per audience
2Thai AI groundingValidated real-tool catalogue
3Role adaptationLLM — one test per role
4VariantsLLM — N parallel per item
5Thai polishLLM — natural Thai pass
6Reviewer docsJSON banks → .docx

Phases 1–2 human-led · 3–5 LLM-assisted · 6 assembly. Resumable, checkpointed per (role, item).

10 / 15

Pipeline · Grounding

Grounded in real Thai hospital AI

In use

  • Inspectra CXR
  • PresScribe (on HOSxP)
  • gastroAI-model G
  • HOSxP DDI alerts (rule-based)

In pilot

  • RAMAAI CXR
  • LiverSound
  • Inspectra MMG

Excluded

  • Autonomous diagnosis
  • US / global scribes
  • Med-recommending CDSS

Scenarios never reference tools that don't exist in Thai practice; updating the catalogue propagates on the next run.

11 / 15

Item maps

Item maps at a glance

9
Back-office · MCQ
C1 · C2 · C3 · C4, predominantly Foundational.
135 variants
14
Medical · MCQ
Foundational + Applied, often integrated. Items 1–3 by track.
294 variants
11
Executive · 5 closed + 6 open
Part 1 S1–S5 (Applied MCQ) · Part 2 O1–O5 (Advanced open).
33 variants
12 / 15

Coverage

What it measures — and doesn't

Measures

  • Recognition (knowledge) across C1–C4 in all MCQ tests
  • Executive Part 2 measures C1–C5 skill at Advanced depth

Doesn't

  • C2 / C5 skill production in MCQ — only the recognition layer
  • C5 out of scope for back-office & medical (longitudinal)

A directional signal to inform pilots and capability conversations — not a high-stakes psychometric instrument.

13 / 15

Architecture & QA

Quality assurance

Built in

  • Expert-reviewed master tests anchor each audience
  • Validated Thai AI-tools list, consulted by every prompt
  • Per-item checkpoints + tolerant JSON auto-repair

Audited automatically

  • Every item has its N variants & 4 options
  • Valid correct-answer letters; letter-distribution check
  • Competency-tag consistency across the bank

Planned: clinical reviewer sign-off per role + pilots of 10–20 test-takers per audience.

14 / 15

Limitations

Limitations & pending

Inherent
MCQs reach recognition, not production; C5 unmeasured in individual tracks.
Pending
Expert item sign-off + pilots of 10–20 per audience.
Coverage gaps
Back-office C2(b) 10th item; procurement variant awaiting input.

Extending measurement of skill production needs task-based components — flagged as future work.

15 / 15

Showcase

Explore the generated
test bank

A read-only, bilingual (Thai + English) item browser — every role, item, and variant produced by the pipeline.

AI Competency Assessment for Thai Hospital Personnel
Methodology Document · Draft for project team review · Branded with Chulalongkorn University corporate identity.