From Test Automation to Agentic Test Automation for Smart Systems

Why this tutorial · Why now

AI-powered systems break the assumptions classic test automation was built on.

According to Global Market Insights, the automation-testing market is projected to grow from $15B in 2020 to more than $40B by 2027. The rapid rise of AI-powered system development and deployment is driving strong demand for AI system test automation. Since 2019, researchers and practitioners have pursued AI testing research and AI-based test automation to cope with validating AI-powered applications and systems.

Over the past six years, Dr. Gao’s group — like other research teams — has worked to address diverse AI testing problems and test-automation needs across smart chat-system validation, computer-vision test automation, and smart learning-system validation. This tutorial reviews that body of work, examines the open problems and gaps in AI test automation and intelligence-quality validation, and lays out a roadmap for transforming test automation into AI system test automation through an agentic approach that optimizes coverage and reduces cost.

The tutorial moves in three steps: first a structured review of AI system testing and automation, then an honest treatment of the open problems — including intelligence-quality validation and coverage evaluation — and finally a roadmap that transforms test automation into agentic AI test automation across AI systems, smart robots, and autonomous vehicles.

01

Review AI system test automation

A structured review of AI testing, AI-based testing, and AI test automation — covering test modeling and analysis, test generation and augmentation, smart test validation, and the use of LLMs in test automation.

02

Confront the open problems

The major problems, challenges, and needs in AI test automation — including adequate intelligence-quality validation and test-coverage evaluation for AI-powered systems.

03

Chart the agentic road

Insights on transforming test automation into AI system test automation with an agentic approach — optimizing test automation and reducing cost across AI systems, smart robots, and autonomous vehicles.

§ At a glance

From test automation to agentic test automation.

Testing evolves from human-driven checks to scripted automation, then to AI that generates and validates tests — and finally to agents that plan, optimize, and reduce the cost of testing. The same arc is applied across three smart-system domains.

The road · evolution × three domains

Each part of the tutorial follows the same shape — introduction, review, technology survey, open challenges — and ends with a roadmap toward an agentic approach for that domain.

§ Learning objectives

What you take home.

By the end of the half day you should leave with a current map of AI test automation and a concrete roadmap for where it is heading.

01 Objective

A current map of AI system testing and automation — models, methods, test generation and augmentation, automatic validation, and where LLMs fit in.

02 Objective

How to think about AI test coverage and intelligence-quality assurance for AI-powered systems.

03 Objective

The state of test automation for smart robots — platforms, validation approaches, and the challenges that remain.

04 Objective

The state of test automation for autonomous vehicles — platforms, validation approaches, and open issues.

05 Objective

A roadmap for agentic test automation across AI systems, robotics, and autonomous vehicles — and why an agentic approach lowers cost.

§ Detailed outline · three parts

Half a day, three smart-system domains.

Each part runs the same arc — introduction, statistical and literature review, technology survey, open challenges — and closes with a roadmap toward agentic test automation for that domain.

Part I

AI System Test Automation

Roadmap → agentic

Where AI system testing stands today, and the road toward an agentic approach.

An introduction to AI system test automation
Review of existing AI system testing models and methods
AI system test generation and augmentation using AI techniques
AI system automatic test validation
AI system test coverage and evaluation quality assurance

Closes with

Roadmap of agentic test automation for AI systems

Part II

Test Automation for Smart Robots and Intelligence

Roadmap → agentic

A statistical and technical review of robot test automation, and the road to agentic robotics testing.

An introduction to robot test automation
Statistical review of existing robot test automation
Literature review of robot test automation and AI validation
Technology review of smart robot test-automation platforms
Challenges, issues, and needs for smart robot testing and automation

Closes with

Roadmap of agentic test automation for smart robotics systems

Part III

Test Automation for Smart Autonomous Vehicles

Roadmap → agentic

A statistical and technical review of AV test automation, and the road to agentic AV testing.

An introduction to AV test automation
Statistical review of existing AV test automation
Literature review of AV test automation and AI validation
Technology review of AV test-automation platforms
Challenges, issues, and needs for smart AV testing and automation

Closes with

Roadmap of agentic test automation for smart AV systems

This is a half-day session; the exact time slot appears in the CISOSE 2026 program.

§ CISOSE 2026 federation

Where this tutorial sits in the congress.

CISOSE federates several constituent conferences — among them IEEE AITest, IEEE SOSE, IEEE BigDataService, IEEE Intelligent Mobile Cloud, and IEEE DAPPS. AI test automation for smart systems lands squarely on the AITest core, with substantial reach into service-oriented systems, big data, and mobile/edge intelligence.

Tutorial coverage by federated conference

●●● core · ●● substantive · ● adjacent

Federated conference	Tutorial part	Coverage
IEEE AITest — AI Testing & Quality Assurance	Parts I–III · the core subject of the tutorial
Cyber-Intelligence (overall congress)	Agentic test automation for smart systems
IEEE SOSE — Service-Oriented Systems Engineering	Part I · AI system test automation
IEEE BigDataService — Big Data & ML	Part I · ML test generation & augmentation
IEEE Intelligent Mobile Cloud	Parts II–III · edge robotics & AV systems
Smart Cities & IoT	Smart-system application context
IEEE DAPPS — Decentralized Apps / Blockchain	Out of tutorial scope	—

§ Who should attend

Targeted audience.

AI system test engineers, managers, researchers, and students, as well as industry practitioners.

Test engineers

Building or maintaining test automation for AI-powered systems.

Engineering managers

Planning AI quality assurance and test-automation strategy.

Researchers & students

Working on AI testing, robotics, or autonomous-vehicle validation.

Industry practitioners

Deploying AI systems, smart robots, or AVs at scale.

§ Speaker

Delivered by the CISOSE steering-board chair.

Jerry Zeyu Gao

Professor, Department of Computer Engineering

San Jose State University, California, USA

Steering Board Chair, IEEE CISOSE

350+

publications

10,500+

Scholar citations

388K+

ResearchGate reads

27+

years in academia

Biography

Jerry Zeyu Gao is a Professor in the Department of Computer Engineering at San Jose State University, California, USA. He is the Steering Board Chair of the IEEE International Congress on Intelligent and Service-Oriented Systems Engineering (CISOSE) since 2016, Steering Board and Organization Chair of the IEEE Future Technology Summit since 2019, and Director of the SJSU Research Center on Smart Technology, Computing, and Complex Systems. He brings over 27 years of academic research and teaching experience and more than 10 years of industry and management experience in software engineering and IT development. He has published three technical books and over 350 papers in IEEE/ACM journals, magazines, and international conferences and workshops. His current research spans machine learning, smart cities, AI test automation, AI clouds, and mobile clouds. He has served as an editorial board member and associate editor for journals including IEEE Software, the Journal of Smart Cities, Energies, and Agriculture, and has been an invited keynote speaker on AI test automation, smart cities, UAV AI clouds, green-energy AI clouds, smart agriculture, and intelligent wildfire platforms.

Leadership roles

Steering Board Chair, IEEE CISOSE (2016 – present)
Steering Board & Organization Chair, IEEE Future Technology Summit (2019 – present)
Steering Board Member, IEEE Smart World Congress
Director, SJSU Research Center on Smart Technology, Computing, and Complex Systems
General / Program Chair, IEEE CAI 2025, IEEE AITest 2025, IEEE BigDataService, IEEE MobileCloud

Selected honors

KSI Fellow (SEKE 2011)
College of Engineering Faculty Award for Excellence in Scholarship (2013)
UT Arlington Distinguished Alumna, College of Engineering 50th Anniversary (2010)
Marquis American Who’s Who (2020 – 2022)

§ Venue

Fukuoka, Japan.

IEEE International Conference on Cyber Intelligence and Software-Oriented Service Engineering (CISOSE 2026). July 27 – 30, 2026, Fukuoka, Japan.

https://cisose.fit.ac.jp/2026/

Conference info

Conference

CISOSE 2026

IEEE International Conference on Cyber Intelligence and Software-Oriented Service Engineering

Dates

July 27 – 30, 2026

Location

Fukuoka, Japan

Language

English

Registration

Registration is handled through the CISOSE 2026 main conference website. See the program for the scheduled tutorial slot.

Also at CISOSE 2026

Building and Verifying AI Systems with Agentic AI →

A companion hands-on tutorial · Chang · Li · Lee