Dr. Yafeng Wang, assistant professor in the School of Materials, Mechatronics and Manufacturing Engineering, has received a $899K award from the National Science Foundation as principal investigator for the project titled, “CPS-FR: A Cyber–Physical–LLM–Human System (CPLHS) Framework for Vision-Based Drone Applications.”

The team’s goal is to make drones more autonomous and capable in tasks that rely heavily on visual information by combining artificial intelligence, advanced control and human supervision.

“The idea is for the AI to understand the mission, help determine where and how the drone should collect data, and make adjustments when needed,” Wang said. “Instead of continuously controlling the drone, the human can provide instructions, supervise the mission and intervene when necessary.”

Wang said the idea was inspired by traditional engineering practice, where a supervising engineer provides guidance while a field engineer performs the inspection and collects data.

“We asked whether AI could take on some of these roles,” Wang said. “A large language model can provide higher-level reasoning based on engineering knowledge and mission requirements, while other AI and control systems can handle the field-level tasks needed to collect useful data.”

In the proposed system, the team refers to these two functions as Supervising AI and Field AI. The Supervising AI uses engineering knowledge, manuals and mission requirements to provide high-level guidance. The Field AI focuses on collecting useful visual data and responding to problems that arise during flight.

Human engineers will remain part of the process. Before a mission, an engineer can provide instructions and priorities. During the mission, the engineer can monitor the system and override decisions when needed. Afterward, feedback from the engineer can be used to improve future missions.

Another major part of the project focuses on image quality. Rather than simply following a predetermined flight path, the drone will be able to detect problems such as sun glare or motion blur and adjust its position, speed or camera angle to obtain a better image.

“For many inspection tasks, flying to the right location is only part of the problem,” Wang said. “The drone also needs to know whether the image it collects is actually useful.”

The team is also studying how the drone and its camera gimbal interact. Gimbal movement can affect the motion and stability of the drone, so Wang’s team is developing a Drone-Gimbal Coupling Dynamics model that treats them as an integrated system.

The project will also develop an Adaptive Priority Control system to manage competing commands during a mission. For example, a drone may be following a planned trajectory when the AI detects glare and requests a different viewpoint. The controller must accommodate that request while maintaining stable flight and continuing the overall mission.

Wang said the long-term goal is to move drones away from tools that require continuous remote control and toward intelligent robotic partners.

“We hope that through this research, an engineer can simply describe what they need, while the system handles more of the routine reasoning, data collection and control,” Wang said. “The human would remain in a supervisory role, providing expertise, judgment and intervention when needed. This could reduce workload, improve safety and make drone operations more efficient.”