Mesh Deformation

Overview

An important application of 3D deep learning is the creation of 3D content purely from images. There are many different approaches to this end, but mesh deformation is a common and valuable technique that has shown up in a variety of projects, such as Mesh R-CNN and Total3DUnderstanding. We’ll introduce the technique, examine common loss functions built into PyTorch3D that we can use for these problems, and then try them out ourselves.

Why mesh deformation

Mesh deformation applies a data-centric perspective on content generation: instead of manually creating a 3D model with art tools, we can apply mesh deformation to modify an existing mesh to fit a collection of images from various perspectives. It can be a convenient and scalable alternative to manual 3D design for creating 3D models of simple objects.

When we speak of mesh deformation, we strictly mean the optimization of the vertices of a 3D model while preserving the definition of its edges and vertices. Think of it as stretching clay without breaking it. The vertices are free to change their position, but their connections to edges and faces can’t change. Mesh deformation can therefore be thought of as changes that simply shift the position of individual vertices.

In the subject of topology, deformation is limited to changes that don’t:

• Open or close holes

• Break or form connections

• Self-intersect

In other words, mesh deformation is not allowed to change the connectivity of the mesh or introduce physically implausible shapes.

How mesh deformation works

We start with a template mesh, which is a 3D mesh typically of some primitive shape like a sphere or torus, that we’ll use as our starting point. We create a copy that we call the deformable mesh, a mesh that we can freely manipulate via optimization to satisfy our objective function.

In the real world, we don’t usually have a ground truth mesh, or target mesh, for us to compare with our deformed mesh. Instead, we define proxy objectives such as an image reconstruction loss that we can optimize via our image collection. However, in testing and development cases, we should have a target mesh to validate the performance of our mesh deformation.

Drawbacks

While mesh deformation leaves us with a 3D model, it is a limited technique with a number of shortcomings that should be kept in mind.

Mesh must be homeomorphic to the target object

Homeomorphism is a property that two 3D models share if there exists a transformation that can deform one mesh (i.e., adjust the position of its vertices) to fit the shape of the other mesh. Homeomorphism is a subset of isomorphisms that applies specifically to topological spaces. More generally, it can be thought of as an invertible mapping (in other words, a bijection) between mesh A and B that preserves the constituent elements of A and B (in this case vertices) and their internal relationships (in this case their face and edge relationships). In mesh deformation, this is a crucial concept because it dictates the kinds of shapes that can be reasonably approximated with a template mesh. For instance, a sphere can be deformed to the shape of a bowl, but can never be deformed to the shape of a mug due to the hole in the handle.

Cameras must have visibility of the entire object

Mesh deformation is ultimately dependent on the images it uses as inputs. These images must have reasonable coverage of the target object in order to reconstruct its real-world dimensions in proper detail. If there are gaps in coverage, for instance, the bottom of an object resting on a platform, it will be unable to recover those details. Likewise, recovering the inside of objects from purely photographic approaches is often out of the question. While some highly concave objects such as a vase may technically be isomorphic from a sphere, the lack of internal coverage may make the interior of these objects very challenging to recover.

The optimization approach

In order to optimize the mesh, we first need to define loss functions that operate on mesh data.

Loss functions

Since we begin with a template mesh that has a predefined topology, we want changes to the locations of the vertices to respect that topology. We should expect the optimization process to respect the following reasonable properties:

• The deformed mesh should reasonably approximate the topology of the target mesh.

• Edges should be of roughly the same scale in length.

• The mesh should be reasonably smooth.

• Normals should not change drastically from face to face.

We commonly utilize any (or all) of the following loss functions to apply mesh deformations that satisfy those properties, respectively:

• Mesh edge loss

• Mesh Laplacian smoothing

• Mesh normal consistency

Mesh edge loss

When starting with a template mesh such as a sphere, it is important to consider how the source topology might change with optimization. We don’t want the edge length to distort drastically. The mesh edge loss regularizes the length of mesh edges. It is normalized by the number of edges in a mesh so that all meshes in a batch are weighted equally.

For a given edge connecting vertex $v$ to vertex $v'$, the mesh edge loss is equal to: