Drones have come to play a major role in providing visual perspectives previously available only by flying an aircraft, offering a more cost-effective way to navigate rough terrain or monitor activity on large tracts of land.

But drones are limited by the pilot's expertise. The user decides where the drone flies, chooses camera angles and decides when images are of high enough quality.

If each new task or initiative requires lots of programming, retraining and expertise, it becomes difficult and expensive to expand drone applications. That’s why a research team in the College of Engineering, Architecture and Technology at Oklahoma State University is working to make drones more intelligent and autonomous, all without removing human supervision.

Dr. Yafeng Wang, assistant professor in the School of Materials, Mechatronics and Manufacturing Engineering, is the principal investigator on a project funded by the National Science Foundation titled “CPS-FR: A Cyber–Physical–LLM–Human System (CPLHS) Framework for Vision-based Drone Applications.”

The team’s goal is to make drones better at tasks that depend heavily on visual information. The team will combine drones, large language models, artificial intelligence, advanced control and human supervision.

“The idea is for the AI to understand the mission, help determine where and how the drone should collect data and automatically adjust the drone and camera when needed,” Wang said. “Instead of continuously controlling the drone, the human becomes more of a supervisor who can provide instructions, monitor the mission and intervene when necessary.”

Wang said he was inspired to make drones more autonomous by observing how activities such as inspecting infrastructure require an engineer to take photos, as well as a more experienced supervising engineer to interpret the images and decide where to go next.

“We asked whether a large language model could capture part of that higher-level reasoning,” Wang said. “LLMs are increasingly capable of understanding instructions, manuals, operational procedures and visual information. That creates an opportunity for them to act almost like an AI supervising engineer — understanding the mission and providing guidance to the drone — while still allowing a human expert to supervise and correct the system.”

The team is using AI to handle routine reasoning and control, while a person handles supervision. Their Hierarchical Retrieval-Augmented Generation framework involves the human operator at three stages. Before a mission, a person would provide instructions or priorities. During a mission, a person can monitor the AI’s decisions and correct/override if necessary. After a mission, the person provides feedback that becomes part of the system’s knowledge for future missions.

The team addresses several limitations by developing an integrated framework. It reduces reliance on pilots and domain experts by helping determine both how the drone should fly and what information to collect. It also enables the drone to determine which images are useful, responding to issues such as sun glare or motion blur rather than simply following a predetermined trajectory and ignoring potential photo issues.

The framework also integrates the drone and camera gimbal to account for how the gimbal movements affect the drone’s flight. It can dynamically balance commands from the mission plan, AI system and human supervisor rather than relying on predefined priorities.

The low-level control system being developed by Wang’s team includes a Drone-Gimbal Coupling Dynamics model and an Adaptive Priority Control system. The DGCD treats the drone and camera gimbal as a single system. When the gimbal moves, it generates force and torque that affect drone performance. The team will model those interactions to achieve more accurate and stable control during flight.

The APC system determines how different commands will be balanced. Wang said a drone may have a predefined flight path, but the AI may request a different viewpoint due to sun glare or other factors affecting image quality. The APC allows the system to temporarily prioritize a new requirement while remaining in stable flight and eventually returning to the mission objective.

The project will involve testing how the framework responds to battery energy storage, thermal surveillance and building roof inspection to demonstrate that it is not limited to a single type of inspection.

Drones equipped with thermal cameras will inspect a large 100MW facility and look for abnormal thermal patterns. The research will compare the autonomous system with conventional inspections in terms of detection quality, inspection time, human effort and operational effectiveness.

For roof inspections, drones with RGB cameras will inspect large buildings and identify conditions such as cracks, water pooling, membrane damage and other defects. This will allow the team to evaluate inspection coverage, accuracy, human workload, safety and cost.

“Together, these applications demonstrate whether the same underlying CPLHS framework can adapt to very different sensing technologies, environments and mission requirements,” Wang said.

The team’s effort to establish a more general framework could also support other applications in which drones collect visual information. It could aid in inspections of roads, bridges, tunnels, power lines, agriculture, wildfire response, environmental monitoring and public safety operations.

Wang said he ultimately hopes this research helps make drones more of an intelligent robotic partner rather than a remotely operated tool. He hopes an engineer can describe what needs to be accomplished rather than manually controlling every movement of the drone.

“We hope to establish principles for how large language models, physical robots, advanced control and humans can work together safely and effectively,” Wang said. “That could contribute not only to more capable autonomous drones, but also to a new generation of intelligent robotic systems in which AI provides reasoning and adaptability while humans retain meaningful supervisory control.”