Vertex normals

Small research about the differences of vertex normals between Blender and Arma 3

Introduction

Vertex normals (or smooth normals as used in other software) are an essential part of game rendering. Whithout these normals, smooth shading wouldn't be possible.

The basic idea was to introduce vertex normals instead of rendering based on face/surface normals. The normal at each vertex is a weighted average of the face normals of faces connected by the vertex. This allows us to calculate continuous shading accross interconnected meshes with finite number of faces.

For a more in-depth explanation, you can visit this page on Scratchapixel.

The idea was really revolutionary, but there is a problem. There is no universal convention or standard for the weighting of the face normals, and this creates differences between applications.

The goal of this page is to present the differences between vertex normal calculations in Blender and the Object Builder application for Arma 3.

Support in files

Mesh data exchange formats like OBJ and FBX store vertex normals as part of their standard data structure. Whether or not to actually read/use them is up to the user, and (in the case of Blender) the importer algorithm.

Same is true for the P3D file format. According to the format specifications of the MLOD P3D file format on the community wiki, every LOD of a P3D stores a list of vertex normals. These normals are then referenced by indices in other sections of the file.

Hex view of the vertex normal table of an MLOD P3D in ImHex

Blender vs Object Builder

Blender

Like other 3D modelling applications, Blender calculates vertex normals on-the-fly as the meshes are edited, and by default it uses an average where each connected face has equal weight, so the average is practically not weighted.

This can be changed with a custom split normals data layer added either manually, or with modifiers such as "Normal Edit" or "Weighted Normal"

Object Builder

The vertex normals are stored in lists in the binary P3D file. Unlike Blender, Object Builder only calculates vertex normals upon certain events like the manual command, edits made to the LOD mesh etc., and uses an average where the connected faces are weighted by their area. Vertices can be set to fixed normals in order to avoid recalculation.

The disconnect

Because of the different weights to face normals, the resulting vertex normals create a shading that is slightly different.

Blender-style smooth normals (left), and Object Builder-style smooth normals emulated in Blender (right)

Looking at the above image the difference doesn't appear to be that severe, but in practice it can cause unwanted results, especially with baked normal maps from low poly-high poly workflows.

When we export a model in FBX format (for instance) and bake a normal map from a high poly mesh in the software of our choosing, the export from Blender will have the traditional Blender-style averaged vertex normals, and the baked normal map will be generated based on this shading. Then if we export the same mesh to P3D format, and Object Builder recalculates the normals, we end up with a different base shading. This may cause smooth shaded surfaces with normal maps to appear "dented", instead of being flat as we expect.

The good news is that it's clearly the Object Builder application that causes the issue, and the vertex normals are not recalculated during the conversion to ODOL P3D format. As the following image shows, Blender-style vertex normals can indeed be forced into the game.

Blender-style smooth normals (left), and Object Builder-style smooth normals (right)

Solutions

Essentially there are two solutions to the problem:

• Make Blender use weighted normals

• Export vertex normals directly to P3D and prevent recalculation

While both solutions produce slightly different results, they both solve the disconnect between the shading in Blender and in Object Builder/Arma 3 by forcing one vertex normal calculation style over the other.

Weighted normals in Blender

The simplest solution is to use weighted normals in Blender. This doesn't require any tampering with Object Builder, or need any special add-on for Blender, since Blender comes with a built-in modifier to do the job.

Weighted Normal modifier in Blender

To get the desired results, and Weighting Mode has to be set to "Face Area", and the Weight to 50. The Keep Sharp option is recommended to be enabled too, this way the modifier respects the edges marked as sharp manually.

With this solution if we export to FBX or other exchange formats, the vertex normals will be identical to what Object Builder would calculate, therefore solving the shading issue, and providing a uniform look with baked normal maps as well.

This way is also the only way when using the ArmaToolbox Blender add-on by Alwarren to export to P3D format, since this add-on does not export vertex normals, and instead makes Object Builder recalculate them immediately after export.

Export normals to P3D

This solution is a little bit more involved than the first one, and requires the use of the Arma 3 Object Builder (A3OB) Blender add-on by me (MrClock) since this is the only add-on to export directly to MLOD P3D format from Blender, that supports proper vertex normals.

Unlike Alwarren's ArmaToolbox, the A3OB add-on properly exports the vertex normals of meshes whatever they may be. This means that a modeller is free to use any normal manipulation techniques (manual or automated), they will appear correctly in Object Builder and in Arma 3.

The only issue is that as mentioned earlier, Object Builder may automatically recalculate the vertex normals upon certain editing actions in the application. To prevent this, either the model must not be edited in Object Builder once it's exported to P3D (which is often unavoidable, since minor corrections may need to be done in Object Builder), or the vertex normals need to be fixed in place to prevent recalculation.

To fix the normals in place, the vertex flags have to be set to a specific value that instruct Object Builder to ignore the vertices during normal recalculations.

To make this easier and automated, A3OB exports all vertices with fixed normals (though this can be changed if needed). It also provides full support for all other face and vertex flag settings, should someone need them for a very specific case.

With this solution, the vertex normals in Object Builder and in Arma 3 will always match the shading in Blender (whatever it may be) which supports artistic freedom.