IEEE CISOSE 2026 · Fukuoka Institute of Technology · Hands-on tutorial
Building and Verifying AI Systems with Agentic AI
Build it. Trust it.
Chia-Kai Chang · National Central University, Taiwan ● presenting
Testing methods co-authored with Kuei-Hao Li · National Tsing Hua University
This slot has two independent parts, each standing alone. Visualization for AI systems — Teng-Yok Lee (Mitsubishi Electric), presented separately. Building & verifying AI systems (this deck) — Chia-Kai Chang.
How the next ~2 hours run
Agenda
1What I built — a production platform, and why you could do the same~12 min
2How I built it — one feature, start to finish live demohands-on #1~32 min
—Breakper the program
3What broke — three incidents, and a judge that was confidently wrong~15 min
4What I check now — the six questions I ended up with live demohands-on #2 #3~30 min
5How you start — the table, the artifacts, and what I would do differently~11 min
Bring a laptop. Three short hands-on blocks. The harness also runs offline, so a bad connection is fine. · Ask questions any time — in the room or in the chat. · This is an account of one system, not a course. I have paid for everything in it. · Visualization is Teng-Yok Lee's separate session, in this same slot. This session is being streamed. Everything I show is public, published, or my own system's failures. There is nothing here I would mind you sharing.
Why this talk · why now
Generative AI collapsed the cost of building real, multi-user AI systems. It did not collapse the cost of understanding — or trusting — them.
BUILD COST ↓↓↓
Days, not quarters. One engineer now ships a multi-tenant AI service with agentic development.
TRUST COST → unchanged
Is it correct? Safe? Drifting? Explainable? Compliant? None of these got cheaper.
Your colleague asks who actually read this code?
Your manager asks what will it cost next month?
Your ethics board asks who consented, and can they take it back?
Three people. Three kinds of evidence. "The tests pass" answers none of them. By the end you will have six questions that do.
Who is telling you this
I teach, I build it, and I get the bill
Chia-Kai Chang
Assistant Professor Center for General Education National Central University, Taiwan
ORCID 0000-0003-2575-2738
I TEACHGeneral education at a public university. My students are not computer-science majors, which shapes everything I build.
I BUILD & RUN ITFounder of the platform on the next slide, and the person on call when it breaks. Co-founded with Kuei-Hao Li.
I STUDY ITEducational Omics — bringing very different kinds of learner data together: what students say to a model, what they do, what their body is doing, and what the room is like.
Most people are one of these three. I ended up being all three — and that is the only reason I have the failures in this talk. You do not get an incident log by reading about systems. I chair the tutorial track at this conference. I took no part in reviewing this session.
Who is telling you this · the research
Educational Omics: six kinds of evidence about one learner
A learner leaves very different traces. Most research picks one. I try to hold them together.
COGNOMICS
What they say to a model, and at what level of thinking.
dialogue traces · running
LINGUOMICS
How they write and speak. Complexity, meaning, voice.
text & speech · running
SOCIOMICS
Who they talk to, and what they build together.
forums, group work · running
PHYSIO‑NEUROMICS
What the body is doing while they learn. Heart-rate variability, sleep, stress.
Garmin watches · running EEG & fNIRS headset · being built
ENVIRONOMICS
What the room is like. Light, heat, noise, air.
public sensor networks · running
ETHICOMICS
Consent, privacy, and what the models are biased toward.
consent records, model arena · running
Where it comes out: researchers download de-identified data through a single approved route, not by asking me for a database dump.
Why it matters here: six kinds of data, one system. A checklist that only works for web apps would be useless to me.
Who is telling you this · the instruments
What actually collects each one
Six kinds of evidence means six different things to keep working.
Dimension
Where it comes from
Who controls it
Cognomics
the chat logs of my own platform
me
Linguomics
the same text, plus speech turned into text
me
Sociomics
forum posts and group work
me
Physio-Neuromics
watches on students' wrists
a device maker
Environomics
public sensor networks around the campus
a government agency
Ethicomics
consent records, and which model students prefer
me
Three of these are mine. Two are not. A sensor can be moved, a device maker can change an interface, and nobody sends me a message when it happens. Half my data can break without anyone touching my code.
Who is telling you this · the body
A watch on the wrist, all semester
Students who opt in wear a fitness watch. Sleep, resting heart rate, heart-rate variability, activity — arriving whether or not anyone is looking.
WHY BOTHER
A learner is a body. Sleep the night before a class is not a metaphor — it is a number, and it is already being recorded by a device they chose to wear.
What I want next is several people at once, on the same clock, so a classroom can be looked at as a group rather than as a pile of individuals.
WHAT I DO NOT CLAIM
Heart-rate variability is not a stress reading, and it is not an emotion. It is a physiological measure with a long literature and a lot of noise.
In Europe, inferring emotion from this kind of signal in a school is prohibited outright. Saying "we measure stress" is not sloppy wording. It is a different legal category.
The moment a body joins the dataset, governance stops being paperwork. You cannot ask a student to re-wear last Tuesday, and you cannot un-collect a heartbeat.
Who is telling you this · hardware, in progress
And a headset we are still building
A wearable of our own design: brain activity, blood-oxygen response, and pulse, in one head-worn unit. It is in design and fabrication. It has collected nothing.
WHY BUILD IT
A watch tells you about the body. It cannot tell you much about attention. Research-grade equipment can, but it lives in a laboratory and a classroom is not a laboratory.
So the target is unglamorous: something a student can put on and forget, that still produces a signal worth analysing.
WHY IT IS ON THIS SLIDE
Because it does not exist yet, and that is exactly when the verification has to be decided.
You cannot patch hardware on a Friday afternoon. Whatever we get wrong in the design is fixed in every unit we make, and every recording it ever takes.
Software lets you postpone the question of how you will know it works. Hardware sends you the bill up front.
Who is telling you this · the room
The room is part of the data
Heat, light, noise and air quality around a campus, from public sensor networks — read continuously, and lined up against when students were actually learning.
WHAT IT IS
Weather and air-quality stations run by government agencies, sampled on a schedule and matched to the hours a course was running.
WHAT IT IS NOT
Not a sensor I installed. Not a reading from inside the classroom. The nearest station is an approximation, and it is treated as one.
THE RULE I FOLLOW
Public data is for using, not for republishing. I do not put someone else's station map back on the internet with my logo on it.
This is the only data in the system that I cannot fix. Stations are moved, recalibrated, and taken offline, and I find out from a gap in a chart weeks later. Somebody else's schedule is now part of my correctness.
Who is telling you this · measuring thinking directly
Twenty-three tests, timed to the millisecond
Classic tasks from cognitive psychology, rebuilt to run in a browser: attention, working memory, processing speed, reasoning.
WHAT IT GIVES ME
Something to compare the dialogue evidence against. If a student's conversations look sharper this month, is that the student, the course, or my measurement?
These tasks are decades old and well studied. That is the point — they are a fixed ruler in a system where everything else is moving.
ONE HARD RULE
While a test is running, the system may not call a language model. Not for a hint, not for a log, not for anything.
A model call takes a moment. The thing being measured is a moment. Answering a call would change the number I am trying to read.
Same shape as a rule you will see later: a feature is deliberately made worse so the measurement stays true. Convenience during a trial would have cost me the trial.
Who is telling you this · how anyone gets the data
One door out, and it is watched
Researchers do not email me for a database dump. They request a de-identified export through a single approved route.
ONE ROUTE
Every export goes through the same path, under the same ethics approval. There is no second way, and no private favour.
DE-IDENTIFIED
Identifiers are removed on the way out, not "handled carefully" afterwards by whoever received the file.
RECORDED
Who asked, what they got, and when. If I cannot answer those three questions, the export should not have happened.
Building an export route is also building the fastest way to lose everything at once. That is why it is the narrowest, most boring, most logged part of the whole system — and why governance here is a pipe, not a promise.
The worked example
A production, multi-tenant AI system
29
higher-education institutions
4,300+
users
420,000+
learning records
Built mostly with agentic AI — I write the spec, the agent writes the code, the tests decide.
One umbrella ethics approval — IRB reference NTU-REC 202507EM058. Built in from the start, not added later.
Very different kinds of data: chat with a model · what students click · heart rate from a watch · room sensors · markets · test scores.
These numbers do one job: they tell you how much pressure everything else in this talk was tested under. Nothing later depends on the size. It all works with one course, or with a hundred.
The philosophy · why this works at all
Programming keeps moving the source of truth up
Every step up trades control for leverage. And what you write by hand moves up too: assembly, then C, then Python — now a spec in plain English. You don't read your C compiler's assembly output. So maybe you do not need to read every line the model writes. If you can trust it.
The philosophy · the catch
From vibe coding to vibe engineering
Two named things, two years apart. This talk lives entirely on the right-hand side.
VIBE CODING · Karpathy, 2025
Accept output without understanding
Source of truth = a transient chat
Great for prototypes & throwaways
A liability in production — you gave up the understanding you needed
VIBE ENGINEERING · Willison, 2025
Keep intent-first leverage
Tie it to a spec and tests
A person checks before it ships
Verify at the new layer: behavior, not lines
You remain accountable for what ships
The LLM "compiler" is stochastic, not deterministic — so you cannot treat its output as an unread intermediate. You must verify. That's the entire second half of this talk.
This room · CISOSE 2026 federation
Wherever you sit, one layer is yours.
CISOSE federates eight conferences. Each maps to a layer of the testing checklist you leave with.
BigDataServicedata-pipeline layer · scale · redacted data template
DAPPS / JCCgovernance layer · multi-tenant service architecture
SOSEspec-driven construction & verification
IMCedge → cloud ingestion (adjacent)
Keynote / Invitedresponsible AI · IRB governance
This talk · roadmap
How this talk runs
Not a framework with examples. An account of one build, in the order it happened.
What this is not: a course on any particular agent tool, or a framework I want you to adopt. What this is: a small team built a real platform with these tools. So can you — and here is what it costs.
ACT 2 OF 5
How I built it
Spec-driven, agentic development — one feature, start to finish, including the parts that did not work.
Chia-Kai Chang · National Central University
How I built it
The spec is the source of truth
One artifact at many levels of formality. Translate the intent downward and it becomes executable. At the far end, the spec is the tests. Building and verifying share one source of truth.
How I built it
Anatomy of a good spec
Goal — one sentence of intent
Constraints — performance, security, style, what NOT to touch
Interfaces — inputs, outputs, contracts
Examples — concrete in → out pairs
Acceptance tests — how we know it's done
# spec: forgot-password rate limitgoal: cap reset requests per account
constraint: 3 / hour; do not touch login
interface: POST /reset → 429 on excess
example: 4th request in 1h → 429
accept:
- unit: counter resets after 1h
- e2e: 4th call blocked, logged
A spec an agent can implement is a spec you can test. The five parts double as your acceptance criteria.
How I built it
From prompt to spec
VAGUE PROMPT — vibe only
"add rate limiting to the
password reset so people
can't spam it"
Ambiguous limit, no interface, no test → the agent guesses, scope creeps, nothing to verify against.
SPEC — vibe, written down
goal: cap reset requests / account
limit: 3 per hour, per user_id
on_exceed: 429 + audit log
untouched: login, signup
tests: unit(counter), e2e(4th→429)
Same intent — now implementable and checkable.
Writing the spec is the thinking. The agent handles the typing; you keep the judgment.
How I built it
The agentic loop
The agent writes the code. The spec and tests keep it honest. You stay in the loop at the human gate — nothing merges unverified.
How I built it
What agentic development looks like
Spec files in the repo (Markdown) — reviewable, versioned, diffable.
Agentic coding tool drives edits across many files at once.
Version control + CI run the checks on every change.
Reviewable diffs — the human reads intent, not keystrokes.
# the input — one file, in the repo
model-arena-spec.md 12 sections# the output — next day, one pass
sql/arena.sql new
arena_elo.py new
app_arena.py new
templates/arena/index.html new
templates/arena/battle.html new
templates/arena/leaderboard.html new
templates/arena/history.html new
templates/arena/admin.html new
static/js/arena.js new
static/css/arena.css new
app.py modified11 files · one review · one change-set
How I built it
Agents work across the whole repo
SCOPE
One spec → edits in route, model, template, test, migration — coordinated in a single pass.
CONTEXT
The agent reads the codebase conventions and matches them, so the change reads like the surrounding code.
REVIEW
You review the diff as one unit: does it match the spec? You no longer read every line as it is typed.
The unit of work shifts from a line to a change-set. That's what makes the speed real — and what makes review the new bottleneck.
How I built it
● LIVE DEMO — BUILD A FEATURE
Write a 6-line spec → agent implements → tests go green → view it running
Small, visual feature. If the wifi or the model misbehaves, cut to the recorded clip — the point is the loop, not the feature.
That took four minutes. Everything else in this tutorial exists because four minutes of building creates work that takes months to find.
How I built it · worked example · 1 of 8
One spec, one day: a blind model arena
Students see two anonymous models answer the same prompt, vote, and only then learn which was which.
DAY 1
Spec written — 12 sections, including the full data model.
DAY 2
Built by an agent against that spec, end to end.
SAME DAY
Verified, gaps found, limits written down.
Why this is a fair test case, not a demo toy: two providers behind one interface · a multiplexed SSE stream feeding two panes at once · seven tables with foreign keys · Elo update math · a consent gate · an admin console · rate limits. If agentic construction only worked on CRUD, it would not be interesting.
How I built it · worked example · 2 of 8
What was actually in the spec
Twelve sections — longer to write than the feature was to build. That is where the design happened.
"Ordinary chat may fall back to another provider. The arena never does. A fallback would spoil the comparison — you would be rating a model that never answered."
§6 「兩個都不好」=不更新 Elo(僅記錄)——它不是相對排名訊號,是失敗訊號。
"'Both bad' does not update the rating, only records it — it is a failure signal, not a relative one."
A spec describes how you will know you got it. Do not specify the schema — the tables follow from the function. Write down what must never happen, and what other things bind to.
How I built it · worked example · 3 of 8
The clauses an agent cannot infer
Three lines in that spec are policy, not function. Leave them out and you get working software that measures the wrong thing.
"The rating is student preference, not capability." A rule about the words in the interface, checked by a person before release — the leaderboard may never be called a benchmark.
"Both bad" does not update the rating. It is a failure signal, not a relative one. Put it into the rating and it quietly breaks the scale.
The first clause gives up availability to keep the measurement honest. Everywhere else in the system, falling back is good engineering. Here it is a measurement error. Only knowing what the feature is for tells you which.
Every template parsed, every script syntax-checked
Note what the checks are about: not "does the page load", but the rules the spec wrote down. And note what they could not reach. "Never falls back" needs a real outage to test. "Never call it a benchmark" is enforced by a human at review. Know which of your clauses a test can hold, and which one a person has to.
How I built it · worked example · 5 of 8
Gap 1 — silent exactly where it talked most
THE SPEC SAID
An entire section on research ethics: consent on first entry, opting out without losing access, de-identified export, which approval it falls under.
THE SPEC OMITTED
Anywhere to record that consent. Six tables were specified; seven shipped — the implementer had to invent the missing one.
This one survived only because consent had a visible step in the flow — impossible to build around. A promise with no visible step just disappears, and nobody notices. For every commitment in prose, point at the thing that carries it.
How I built it · worked example · 6 of 8
Gap 2 — right sentence, untestable as written
The models introduce themselves. Blind comparison dies the moment one of them says its own name.
SPEC
"Redact model names from the stream."
REALITY
Tokens arrive in chunks. The name arrives in two of them.
chunk 1 → "...I am Taiwan"
chunk 2 → "-LLM, developed by..."
per-chunk match → nothing found, twice
holdback buffer → caught at the seam
"Redact X" only becomes testable once you say at what granularity. The sentence was right and impossible at the same time. Only a test at the boundary could show it. Your version: any rule you stated over a whole message, in a system that delivers it in pieces.
How I built it · worked example · 7 of 8
Gap 3 — the one we chose not to fix
DECISIONNot fixed. Written into the record as a known limitation, with the reason: state stays correct, only cost is wasted, and rate limits bound the abuse.
CONSEQUENCEIt became a known line item instead of an unexplained bill three months later.
A spec that records what you decided not to do is worth more than one that implies you did everything.
How I built it · worked example · 8 of 8
What "the spec is the oracle" looks like on paper
A different feature — adding multi-round tool calling to a chat engine — written to the same discipline.
THE SPEC CARRIED
Current behavior, cited to line numbers
One implementation red line: the per-round log write must stay inside the loop, or rounds 2+ vanish from the audit trail
Six numbered verification items
The expected database state, and the query to check it
AND A FAILURE TABLE
IF YOU SEE
THE CAUSE IS
one log row
write left outside the loop
rows split across parents
attached to the wrong record
only one tool fired
tools not re-offered each round
Hand this to somebody who has never opened the codebase and they can tell you whether it works. That is the whole claim.
How I built it
Code review in the agentic era
WHAT CHANGES
Volume of code per review ↑↑
Author is an agent, not a colleague
Reading line by line no longer works
WHAT TO REVIEW INSTEAD
Does the diff match the spec?
Do the tests actually cover the intent?
Any change outside the stated scope?
You are the editor, and you answer for it. "The AI wrote it" is never a defence.
How I built it · design principle
Design for verification, not only function
Bring-your-own-key, multi-provider engine
The same conversation runs on OpenAI, Claude or Grok. So the same test can run on all three — which is what the judge needs later.
Model Arena
The blind comparison from the case study — built so that real user preference keeps coming in, instead of being a one-off study nobody repeats.
Two build decisions that made the system easier to test later. Both cost more at the time. Ask it of anything you build: how will I know later that this is still true?
How I built it
Failure modes when you let agents build
Plausible-but-wrong
Compiles, reads well, does the wrong thing.
Silent scope creep
Agent "improves" beyond the spec.
Hidden coupling
A local edit breaks a distant module.
Confident hallucinated APIs
Calls that don't exist.
None of these are new. They are ordinary bugs. The speed just makes more of them, which is why the list matters.
How I built it
Every failure has a guard
Failure
Caught by
Plausible-but-wrong
unit tests + examples
Silent scope creep
spec "untouched" clause + diff review
Hidden coupling
pipeline / integration checks
Hallucinated APIs
typecheck + CI
The list is not paperwork. Each line guards one specific way an agent fails.
How I built it · honest limits
When NOT to hand it to an agent
New algorithms and research code — there is nothing for the model to copy from.
Security basics — encryption, login internals. Get it wrong once and it is very bad.
Anything you cannot write a spec for — the agent cannot read your mind.
Things you cannot undo — migrations, deleting data. Slow down and check twice.
The method works for the part you can undo and can describe. The rest still needs you.
How I built it · honest limits
The spec stops where your vocabulary stops
I can specify a rate limit exactly. I cannot specify a good-looking page. I never trained in design.
WHAT I ACTUALLY WROTE
"You are a UI/UX expert. Make this page look good."
A different result every run. Screenshots of designs I liked did not fix it either — an image shows an output, not a criterion. You cannot diff against taste.
WHAT WORKS INSTEAD
My project file now says: these five gradients are banned. Hover may change color, never position. Table-row buttons use a soft tint, never a saturated fill.
I did not learn design. I accumulated criteria — one rule per thing I rejected.
The variance was never the model being unreliable. It was an unspecified requirement. Nothing to verify against means nothing to converge to.
How I built it · the honest reality
Agentic systems have a bill
Every agent action and every judge call spends tokens. Cost is a first-class design constraint — not an afterthought.
TIER
Cheap model, cheap task
classification, extraction, background jobs on a small model; the flagship only where it earns its keep.
CACHE
Don't pay twice
reuse identical calls; cache stable prompt prefixes and retrieved context.
METER
Budget per task
cap spend per request; track token burn per feature like you track latency.
Mine: the free tier defaults to a smaller model. Grading and embeddings run on mini models. The flagship is reserved for hard reasoning. An agentic system you cannot afford to run is not shipped. Your version: the first question is which of your calls is paying flagship prices for a task a small model would pass.
How I built it · hands-on
HANDS-ON #1 · you do it
Write a 5–6 line spec for a small feature — goal · constraints · interface · example · acceptance test.
Hand it to an agent if you have one. Output: a spec you could actually ship. We'll take two or three from the room and critique against the checklist.
ACT 3 OF 5
What broke
Three incidents from production, and one judge that was confidently wrong.
Chia-Kai Chang · testing methods co-authored with Kuei-Hao Li (NTHU)
What broke · incident one
The job that kept giving itself work
The account ran dry twice in one month. The part of the system screaming loudest was not the part that was broken.
8,731
messages to analyse
195,582
times they were analysed
22×
each · about 40% of the month's tokens
A table that would have prevented this was written months earlier — and never applied to the production database. Claim the work before you do it, not after. And a migration that never reached production does not exist.
What broke · three from production
What production taught us — the hard way
ONE · THE BILLThe recompute storm.
Mechanism: the "already processed" marker was written after the work. A failed concurrent write meant the batch came back — a loop feeding itself. A table written months earlier to fix precisely this had never been applied to production. Lesson: claim work before doing it — and a migration that never reached production does not exist.
TWO · THE WRONG WORKSPACEWrong-context execution.
Mechanism: a constant rendered into the page at load time, in an interface that then switches workspace without reloading. Every later request carried the workspace the user had already left. Not a breach — both belonged to the same user — but the work, the logs and the cost went to the wrong one. Lesson: render-time constants are a time bomb in any stateful client. Side effects must sit after every gate. Otherwise a rejected request still leaves debris.
THREE · TWO MONTHS OF SILENCEThe silent downgrade.
Mechanism: a dependency vendored by its "latest" tag resolved two major versions below what the code assumed. The API had changed shape, so every call quietly did nothing — for two months. Lesson: pin versions, and treat silence as the expensive failure — a crash gets reported, nothing happening does not.
Three failures. Nothing crashed. Nobody reported anything. And no single kind of check would have caught more than one of them.
Iterate the judge prompt until agreement is acceptable.
Watch for known judge biases: length, position, self-preference, and agreeing too easily.
An uncalibrated judge is an unvalidated instrument. You wouldn't trust a scale you never checked against a known weight — same here.
What broke
See it before you test it
You cannot write a test for a failure you have never seen the shape of. Seeing comes first — here, on my own dashboards.
Read the distribution — a sudden shift in one cohort, or one week, is a signal.
I notice it by looking. Turning that into a stored case is still a manual step.
Part of the Educational Omics six-dimension view.
An honest gap. Seeing it is what tells you a test is worth writing. But nothing here converts "I noticed a shift" into a stored case — a person still has to, and a busy person will not. Visualization methods in depth → Teng-Yok Lee's separate session.
Before you go · the traps
Five anti-patterns to avoid
Blind accept — shipping agent output you never verified. (naive vibe coding)
Unversioned judge — a test prompt you can't reproduce or diff.
No drift check — "it passed once" treated as forever.
Governance on paper only — consent and redaction promised, never enforced as a check.
Happy-path only — no adversarial or red-team cases.
Each anti-pattern is just a checklist layer left empty. The scorecard is your defense against all five.
ACT 4 OF 5
What I check now
The six questions I ended up with — and the tool that answers them.
Chia-Kai Chang · National Central University
What I check now · where the list came from
Three failures, three different kinds of check
You have just heard all three. Look at what would have caught each one.
The job that kept giving itself work
WATCHING IT OVER TIME
Every functional test passed. It was only visible in the bill.
Work running in the wrong workspace
CHECKING HOW IT BEHAVES
Tests never switch workspace halfway through. A person does.
Two months of a dependency doing nothing
CHECKING THE CODE ITSELF
No crash, no error, no report. Just nothing happening.
Three failures, three kinds of check, no overlap. Pick any one kind: it catches one, and the other two go straight past you. The other three came from the same place. There was no framework here — only a bill and an incident log.
Your scorecard
A layered AI-testing checklist
Six questions, not six tests. One may take your whole test suite to answer. Another may take one person signing something off.
One word needs defining now: drift means nobody changed your code — the world changed under it.
Code
Does it run, typecheck, pass unit tests?
Pipeline
Does the data get all the way through, with nothing quietly lost?
Behavior
Right answer on normal input, and on input meant to break it?
Guardrail
Does it refuse unsafe requests, and hold up against hostile input?
Governance
Did people agree to this, can it be removed, can it be audited?
Drift
Still true next week, after a model update, at next month’s volume?
What I check now · take-home artifact
LLM-as-Judge harness
The rubric is a file — with a version and a content hash, in git.
Edit it without bumping the version and it refuses to run.
Cross-model — the same judge across providers, in one command.
Every run writes a record — rubric hash, model ids, params, raw replies.
Runs offline — no key, no network, no packages. You leave with the repo.
$ judge run --rubric dialogue_quality.v3.toml \
--models openai:...,anthropic:...REFUSING TO RUN -- rubric drift
version 'v3' is pinned to 75003a6086bd
but this file hashes to 29a083291bf3.
The prompt changed without a version bump.
An unversioned judge prompt silently invalidates every score you have ever recorded. This is the one mistake the harness will not let you make.
What I check now
Designing the judge prompt
Explicit criteria — name each dimension you score.
A fixed scale — one to five, and say what each number means.
Ask for a reason — not just a number.
Structured output — a JSON schema, so you can aggregate it.
Show an example — one for each score.
# rubric R: Socratic qualityc1 asks before answering (0-2)
c2 stays on the student goal (0-2)
c3 no direct solution leak (0-1)
# output
{"c1":2,"c2":1,"c3":1,
"reason":"gave a hint, not the answer"}
The prompt is the test spec — treat it with the same rigor as code: reviewed, versioned, and kept in a changelog.
What I check now · hands-on
Eval data is the asset
Your golden set is a small set of inputs, each with the answer you expect. It is what makes every test mean something. Models change. A good set outlives them.
Take real cases — normal ones, nasty ones, and past failures.
Label them yourself — this is what the judge is measured against.
Version it — it is a spec. Keep it in git.
Grow it — every incident becomes a case you keep forever.
HANDS-ON #2 · you do it
Build a 5-item golden set for one behavior you care about — 3 representative, 2 adversarial — with the expected verdict for each.
Your eval set is your moat — the one artifact a competitor cannot copy from your prompt.
What I check now
Live: run the judge across models
GPT
Claude
Grok
Human
item A
5
5
4
5
item B
4
4
5
4
item C
2
3
2
1
Agreement builds trust; disagreement tells you where to look. Item C splits the judges — that's your next test case. illustrative
What I check now
Cross-model reproducibility
One rubric. Several independent judges. What survives all of them is a property of the output.
With one provider you cannot tell "this output is bad" from "this vendor thinks it is bad". That is the whole reason to fan out. And it is why the multi-provider engine — a build decision — pays off here. Reproducibility was designed in.
What I check now
Pointwise vs. pairwise judging
POINTWISE — score each output
Absolute rubric score (1–5)
Good for regression thresholds & dashboards
Sensitive to scale drift between runs
PAIRWISE — compare two outputs
"A or B better?" — easier, more stable
Feeds Elo / preference ranking
Great for model selection & A/B
Rule of thumb: pointwise to gate a release against a bar; pairwise to choose between candidates. Uedu's Model Arena is pairwise at scale.
What I check now · checklist layer 3
Layer 3 — Behavior
Behavior
Right answer on normal input, and on input meant to break it?
HOW
Curate a representative + adversarial test set
Score with the calibrated judge (pointwise)
Track live with Model Arena (pairwise)
GATE
Release only above a rubric bar
Every failing case → a permanent regression test
What I check now · checklist layer 4
Layer 4 — Guardrail
Guardrails are ordinary application security on an AI surface — SE4AI, not a new discipline. Your version: whatever your system already does to untrusted user input — the model is just one more place that input arrives.
What I check now · guardrail, deeper
Red-team it like an adversary
INJECT
"Ignore previous instructions…" hidden in user content, documents, tool output.
JAILBREAK
Role-play, encoding tricks, multi-turn setups to escape scope.
EXFILTRATE
Coax out other users' data, system prompts, secrets.
Make these standing tests, not a one-off audit. The judge can grade "did it hold the line?" automatically on every release.
What I check now · checklist layer 5
Layer 5 — Governance is a test, not paperwork
Governance
Did people agree to this, can it be removed, can it be audited?
IRB-governed — single umbrella approval (NTU-REC 202507EM058); consent decoupled from grades.
Redacted, auditable data-handling template — the fourth take-home artifact.
Regulatory awareness — the EU AI Act prohibits inferring emotions from biometric data in workplaces and in education. Narrow medical and safety exceptions apply. Physiological state and inferred emotion are treated differently. That is the line our wearable work stays on.
Make it a check: is every record consented, redactable, and auditable? Then governance runs on every release, like any other test. Your version: find the strongest promise in your privacy policy. Now ask what in your test suite would fail if you broke it. Usually nothing would.
What I check now · checklist layer 6
Layer 6 — Drift, the layer everyone forgets
Drift
Still true next week, after a model update, at next month’s volume?
Weekly reruns
same suite, tracked scores
On model upgrade
re-judge before rollout
Alert on delta
score drop = investigate
Drift here means three things: the model changes under you, your data changes, and the cost per unit of work changes. Nobody edited your code in any of them. A passing test today is not a passing test forever.
What I check now · in production
Verification doesn't stop at release
LOG
Every judged interaction — prompt version, model id, score, reason — is a data point, not a black box.
MONITOR
Watch score distributions over time; a sliding drop is drift you catch before users complain.
Production is your largest, most honest test set. Instrument it so the checklist keeps getting answered after launch.
What I check now · hands-on
HANDS-ON #3 · you do it
Write a judge case: a prompt, one rubric line, and the expected verdict.
Run it in the harness across two models. Do they agree? Where they don't is where your real test lives.
Recap · the scorecard, filled in
Six layers, each with a method
Code
unit tests · typecheck · lint — earned at build
Pipeline
schema validation · end-to-end checks — earned at build
Behavior
LLM-as-Judge · Model Arena — graded, at scale
Guardrail
red-team · input sanitization — SE4AI security
Governance
IRB · redaction template — consent & audit
Drift
versioned reruns across models — catch regressions early
ACT 5 OF 5
How you start
Same tools, your domain. What I would tell myself at the beginning.
Chia-Kai Chang · National Central University
How you start · a second instance
Right now, in a completely different field
Not a platform. Not the web. Working with colleagues in cognitive neuroscience at a medical school.
THE QUESTION
While a student learns with a generative model, does where they look on the screen relate to how deeply they are thinking?
What it took to build: a normal webcam gives 15 frames a second. Far too few. So: 60 frames a second, 4K, and an infrared sensor to follow the pupil.
Days, working with an agent — not the months of reading it would have cost me.
AND THEN IT STOPPED BEING AN ENGINEERING PROBLEM
Ethics approval, before a single recording
Video of students' faces — stored, handled, never leaked
Participation that cannot cost a student anything
The equipment has been ready for a while. The equipment is not what decides when the study starts.
The same story as the opening, in a field that was new to me. AI made the building fast. Getting permission is still slow.
Take home · map YOUR system
The one table to leave with
Score your own project against each layer — mark it, and your biggest gap is where to start.
LAYER
ASK OF YOUR SYSTEM
YOUR STATUS
Code
Runs, typechecks, unit-tested?
✓ △ ✗
Pipeline
Flows end-to-end, no silent loss?
✓ △ ✗
Behavior
Right on representative + adversarial input?
✓ △ ✗
Guardrail
Refuses unsafe, resists prompt injection?
✓ △ ✗
Governance
Consented, redacted, auditable?
✓ △ ✗
Drift
Re-checked across model versions?
✓ △ ✗
Mark each row ✓ solid · △ partial · ✗ gap. Your biggest ✗ is your Monday-morning to-do. Take this table to any AI system you own.
Four things you take home
Take it home
01Runnable LLM-as-Judge harness
two real rubrics · a golden set · runs offline · clone it today
02Six-layer checklist
the self-assessment table you filled in ten minutes ago
03Spec template, with two real specs
including the one whose three gaps you just walked through
04Redacted governance template
consent and data handling — posted on the tutorial page after the session
See it at scale:uedu.tw — and the peer-reviewed work behind it (ACM L@S 2026 · Computers & Education: AI · IEEE BigDataService 2026, in this federation · ICMET 2025).
The takeaway
Build it with spec + agents. Trust it with a layered checklist.
The source of truth moved up to intent — so your discipline moves up with it. On your system, in your domain. The platform was only the proof.