Neural Networks Research In UE5

For research purposes in Unreal Engine 5, I have trained various agents using neural networks within UE. Unreal Engine 5.5 employs a new machine learning plugin known as learning agents, which is highly effective for training neural networks for specific tasks. In this post I explain 3 different use cases of neural networks in unreal engine For each use case, I describe the observation space, the action space, and the setup of the neural network.

Neural Network Car Learns to Follow The track & Avoid Obstacles

Aspect	Details
Observation Space	- distance to spline - Angle between car forward and spline tangent - Current speed - Distance to next spline point - 5 raycasts Obstacle detections (forward & side directions)
Action Space	- Throttle (0 to 1) - Steering (-1 to 1) - Brake (0 to 1)
Reward Function	bonus for staying on track and aligned bonus for avoidance of obstacles penalty for hitting obstacles   penalty for stopping End session for going off track

RL NN learns to Avoid obstacles

Neural Network Joints Learn to Crawl Like a Worm

Aspect	Details
Observation Space	- Current angle of each joint ( pitch/yaw) - Angular velocity of joints- Contact state (whether a segment is touching the ground) - velocity of body segments - Distance direction
Action Space	- motor force applied to each joint ( -1 to 1 multiple by force) - joint parameters (Smoothness, Damping, Force)
Reward Function	Bonus:distance traveled  forward bonus for maintaining ground contact bonus for moving straight or toward a goal

NN worms try to learn how to crawl

As you can see in the video, some of the worms come up with innovative ways of crawling to move forward, such as flipping, sudden flap ,etc which can be avoided by adding penalties for flipping or excessive energy use.

Neural Network Agent Learns to Navigate the Maze

This neural network consists of two hidden layers and 23 float observations. To simulate a memory (buffer)for the agent, I have stacked the observations for 10 frames into an array, (nearly 2 seconds of game time since the agent updates 5 times per second). therefore, the network has 230 inputs, 2 hidden layers, and 3 outputs.

I allowed this agent to learn for 30 minutes, and it can navigate through the maze, though it struggles with corners. This issue can be solved by adding more ray traces, more penalties for jittery movement and letting the agents learn for a longer time

Aspect	Details
Observation Space	- 15 Raycast hits in multiple directions (forward, upward, downward) , simulating 65 degree fov ( distance floats array) -15 Raycast Hit (boolean array) - Current velocity - orientation
Action Space	- Move forward (-1 to 1) -Move Right  (-1 to 1) -Turn left/right (yaw angle)
Reward Function	bonus for progressing toward penalty for time spent without progress

RL agent tries to navigate the maze

.