Visualizers

Creating custom visual debugging tools with the Apex Utility AI

Introduction

The purpose of this tutorial is to provide instructions and guidelines with regards to debugging the Apex Utility AI using real-time visualization.

Being able to visually debug AI is a powerful tool to ensure that the AI behaves as expected in in-game situations. By being able to see exactly what is going on under any circumstance, it is usually much easier to identify issues or shortcomings. The Apex Utility AI provides a few ways that real-time visualizations can be harnessed relatively easily.

Basically there are two common ways to real-time visualization for debugging with the Apex Utility AI. There are of course other approaches as well, but these two approaches can be used for most common cases. Visualizers can be made as a) context visualizers, and b) custom visualizers.

Visualizer What does it do?
Context visualizer Visualizes data in the Context and any subclasses.
Custom visualizers Visualizes specific AI entity types, e.g. qualifiers and actions, as they are executed.
Note
Custom and context visualization only works if the Utility AI Editor window is open in Unity and ‘Visual Debugging’ is enabled.

Context Visualizers

A context visualizer bases its visualization on data found in the Context. Several context visualizer base classes can be derived from, depending on what is needed. Context visualizers are global, in the sense that there should ever only be one of each type per scene, as it can handle visualization of all AIs. The first question to answer is whether Gizmos, GUI or both is needed for a particular visualization. The next question to answer is whether the base class should handle casting of the context object to the desired context type. Thus, there are a range of context visualizer base classes.

Visualizer Class Name Use Case
ContextVisualizerComponent Context visualization through a non-GUI and non-Gizmo approach, e.g. writing data to a file or using Handles to draw, rather than Gizmos
ContextGUIVisualizerComponent Context visualization through OnGUI
ContextGUIVisualizerComponent<T> Context visualization through OnGUI, with Context cast to the specified type
ContextGizmoVisualizerComponent Context visualization through Gizmos
ContextGizmoVisualizerComponent<T> Context visualization through Gizmos, with Context cast to the specified type
ContextGizmoGUIVisualizerComponent Context visualization through Gizmos and OnGUI
ContextGizmoGUIVisualizerComponent<T> Context visualization through Gizmos and OnGUI, with Context cast to the specified type

Gizmo visualization is probably the most common use case for Apex Utility AI debug visualization. Gizmos are an Editor-only feature in Unity for rendering primitive shapes such as spheres, cubes and lines inside the Unity editor. They can be immensely powerful for Utility AI visualization and cover a myriad of use cases, including showing the location of targets (move, facing, attacking, fleeing, etc.), showing the current observations held in memory, visualizing ranges (vision range, attack range, audio range, etc.) as well as using color coding to show the most desirable option, among a list of candidates.

OnGUI visualization is needed if the desire is to provide debugging through text or numbers. Typical examples include writing the score for a range of options in an ActionWithOptions<TOption>. When using OnGUI visualization, remember that most often the world coordinates (whether they are 2D or 3D) must be projected onto screen space. This can be done simply using Unity’s WorldToScreenPoint method, e.g. given a Vector3 position:

The y-coordinate needs to be reversed, because otherwise the Y-axis will appear ‘upside-down’ in-game. This is due to the Unity implementation of WorldToScreenPoint, which returns coordinates in screen-space, but if drawing with OnGUI, they need to be converted to GUI-space. There is also a Unity utility for this purpose: GUIUtility.ScreenToGUIPoint().

Tip
OnGUI calls can be expensive performance-wise, so consider removing them before performance testing or shipping your game.

If both OnGUI and Gizmos visualization is needed simultaneously, there are base classes for this purpose too. Expanding on the AttackTargetVisualizerDebugComponent introduced earlier, the following class combines GUI and Gizmo visualization:

As earlier shown, it draws a wire sphere where the attack target is currently. The addition is that it also draws the attack target’s health as a percentage of the maximum health, with color coding of the text (more red for more damaged). The example showcases the simplicity involved in writing these visualizers, through base class implementations. This visualizer component could be added to a GameObject in the scene, from which it will handle visualization of either all selected AIs in the scene, a single selected AI or some custom logic. How to choose which AIs to visualize can be selected through an Enum-based drop-down list in the Unity inspector. However, if ‘Custom’ is chosen, the protected virtual void GetContextsToVisualize(List<IAIContext> contextsBuffer, Guid relevantAIId) method must be overriden to filter the passed List so that only the desired context objects are returned for drawing.

Thus, filtering can be done based on any data found in or through the context, or through static or singleton managers, etc.

Tip
It can be helpful for debugging to expose some visualization settings in the Unity Editor inspector, e.g. through public instance variables or using the [SerializeField] attribute to expose private or protected instance variables in the Inspector. Commonly exposed settings include the color used for Gizmos drawing, sizes of spheres or cubes, and potential filters, i.e. facilitating only showing a subset of the observations, e.g. by not showing observations not matching a certain type or stat.

Contextual visualizers provide a relatively simple method for visualizing virtually any data kept in the Context. They can be immensely helpful in understanding emergent AI behaviour and in investigating why undesirable behaviour is being observed.

Custom Visualizers

Custom visualizers handle the visualization of a particular AI entity (or base class), meaning that it can visualize whenever a particular type of action or qualifier is executed by the AI. Similarly as for context visualizers, there are a few different options in regards to base class implementations of custom visualizers provided by the Apex Utility AI.

Visualizer Class Name Use Case
CustomVisualizerComponent<T, TData> The base class for all custom visualizers. T is the type of AI entity to visualize, e.g. a specific or base-class Action. TData is the type of data to visualize and can be virtually anything. Provides a single method to override, which is called every time the specified type of AI entity is executed by the AI.
CustomGizmoGUIVisualizerComponent<T, TData> The next step in custom visualization. This base class builds on the previous one, and adds abstract methods for Gizmos and OnGUI visualization, as well as options for toggling each type of visualization. Use this base class if providing visualizations for a qualifier.
ActionWithOptionsVisualizerComponent<T, TOption> The final step for custom visualization. This base class covers the expected most common use case for custom visualization: Visualizing the scores for each option used by an ActionWithOptions<TOption>. Provides an abstract method for getting the desired options through the Context object.

The perhaps most common example of such a visualization is the position score visualizer. A range of positions in a grid around the AI unit are sampled at a relatively low rate and stored in the Context, enabling other actions and scorers to access the sampled positions. However, a contextual visualizer cannot be used to visualize the scores, as the score for each position is not stored and instead is calculated on-the-fly in the AI action. The following class thus shows a full implementation of an ActionWithOptionsVisualizerComponent<T, TOption>.

Note
Performance considerations are not as prevalent for Unity Editor-only methods, since they will not have an impact on the compiled application. However, the performance should naturally not be decreased to the point where it disrupts workflow or slows the development velocity.
Tip
AI entity base classes can still be visualized, just specify the AI entity base class type as the type parameter, and not the deriving type.

This screenshot shows an example of how to visualize positions that have been given a score. Positions in a grid around the AI have all been sampled and stored in the Context, allowing for another AI entity to score each of them and select the highest scoring one as the next movement destination. See more in the “Survival Shooter” tutorial for the origin of this screen shot.

Custom visualizers provide a way to visualize data connected to a specific action. Typically, but not necessarily, this will be most relevant for ActionWithOptions, where the desire would be to visualize the score of each option, in order to understand why the AI selects a particular option.

Extensions

By combining context and custom visualizers, virtually anything that the AI does can be visualized in real-time. Usually however, a developer will want to complement visual debugging with more common debugging techniques known from programming, namely using breakpoints, profiling and writing to the console. Finally, the first test or evaluation performed on an AI should always be whether the emergent behaviour matches the designed or desired behaviour for a particular AI agent in a specific setting. If the behaviour is unexpected or undesired, only then should one employ visualization and debugging techniques, in order to understand why undesirable behaviour was observed.

Sometimes, it can also be useful to color code debugging. The actual scores can thus be supplemented with e.g. a colored gizmos (e.g. Gizmos.DrawSphere) with the lowest score being red and the highest score green. In this relation, is might be useful to color the highest scoring gizmos another color such as cyan, so it can be quickly identified for debugging whether the AI actually chooses this. See e.g. the “Survival Shooter” tutorial for an example of this.

Conclusion

Visualizations can be a powerful aid in debugging undesirable behaviour in the Apex Utility AI. There are generally two ways to visualize the Utility AI: Context visualization and custom visualization. Context visualization is based on a component found on the GameObject of an AI agent, which visualizes data stored in the Context for that agent. Custom visualization is invoked by the execution of a specific AI entity, and provides a visualization based on a real-time computation happening in the AI entity, or data that is otherwise cumbersome or impossible to reference.