Shape Modelling Workflow

This tutorial will use GIAS2 tools to processes a set of meshes (segmented femurs) to generate a mean mesh and principal components of shape variation.

The workflow is composed of running three commandline applications:

  1. gias-rbfreg: Non-rigid registration of a reference mesh to all training meshes
  2. gias-rigidreg: Rigid-body registration to align all fitted meshes to the reference mesh
  3. gias-trainpcashapemodel: Perform PCA on the aligned meshes to calculate the mean mesh and principal components of shape variation.

While were are using these three GIAS2 applications for a shape modelling workflow, you can use them for other purposes, particularly gias-rbfreg and gias-rigidreg. These two applications can be generally used for non-rigid and rigid registration of surface meshes and point clouds.

Data Organisation

Each of the GIAS2 applications will read in a set of meshes and output a set of meshes. Let’s take a look at the input/output file directories prepared for this tutorial.

_images/folder_structure.png

In the data/ directory, we have

  • segmentations/: Segmented meshes for input to gias-rbfreg. In this tutorial, these are 10 femurs segemented from the Melbourne Femur Collection.
  • fitted_meshes/: Fitted meshes output by gias-rbfreg, used as input for gias-rigidreg.
  • aligned_meshes/: Aligned meshes output by gias-rigidreg, used as input for gias-trainpcashapemodel.
  • shape_model/: Shape model outputs by gias-trainpcashapemodel.

Each application takes as argument a text file containing the file paths of its input meshes. These text files are in the data/ directory:

  • rbfreg_list.txt: Segmented meshes for input into gias-rbfreg.
  • rigidreg_list.txt: Fitted meshes for input into gias-rigidreg.
  • pca_list.txt: Aligned meshes for input into gias-trainpcashapemodel.

As you run the applications in the steps below, you can view the output meshes using your preferred 3-D model viewer, e.g. MeshLab.

0. Getting Started

Open a terminal (on windows, an Anaconda prompt), and navigate to the tutorial data/ directory. This is the folder where the rbfreg_list.txt etc files are.

1. Mesh Fitting

The first step of the workflow is to non-rigidly register or fit one of our segmentations to the other segmentations, thereby representing all our segmentations by correspondent meshes. gias-rbfreg uses radial basis functions to perform registrations between clouds of non-correspondent points.

NOTE: Your segmentations MUST be in units of millimetres. The default registration parameters are tuned to work for bone-sized objects digitised in mm.

We run:

gias-rbfreg -b rbfreg_list.txt -d fitted_meshes --outext .ply

Let’s look at each argument of the command above:

  • -b rbfreg_list.txt: invokes batch mode for gias-rbfreg and gives it the list of meshes to morph. gias-rbfreg will use the FIRST mesh in the list as the reference to morph to all other meshes in the list. Note that gias-rbfreg can be used in non-batch mode to register one mesh to another.
  • -d fitted_meshes: defines the directory fitted meshes will be written into. “_rbfreg” will be appended to the end of the input file names.
  • --outext .ply: defines the output file format. In this case, we define fitted meshes to be in the PLY format.

You can add -v to the command to view each registered mesh (if Mayavi is installed). You will have to close the pop-up window for the application to continue to the next mesh. Run gias-rbfreg --help to see the full list of commandline arguments.

As the application runs, you will see output .ply mesh files appear in the data/fitted_meshes folder.

2. Rigid Alignment

The second step of the workflow is to rigidly align the fitted meshes to remove rotational and translational variations. We run:

gias-rigidreg corr_r -b rigidreg_list.txt -d aligned_meshes --outext .ply

The argument corr_r tells gias-rigidreg to use the correspondent rigid registration mode. If we want to normalise size as well, we can use corr_rs to apply scaling during alignment. As is step 1, the other arguments define the list of meshes to process, the output directory, and the output file format. “_rigidreg” will be appended to the end of input filenames.

You can add -v to the command to view each registered mesh (if Mayavi is installed). You will have to close the pop-up window for the application to continue to the next mesh. Run gias-rbfreg --help to see the full list of commandline arguments.

As the application runs, you will see output .ply mesh files appear in the data/aligned_meshes folder.

3. PCA

The final step of the workflow is to perform PCA on the aligned meshes to generate our mean mesh and principal components. In additional, we will also generate meshes along each of the first 3 principal components. We run:

gias-trainpcashapemodel pca_list.txt -n 9 -r 0 1 2 -o shape_model/femur --plot_pcs 9 -v

In the command above

  • pca_input.txt: defines the file containing the list of aligned meshes to process
  • -n 9: tells the application to calculates the first 9 principal components using an incremental PCA method. If we leave this argument out, the application will attempt to calculate all principal components, which may result in memory errors if the number of features is greater than ~10,000. In our example of 10 training meshes, there are only 9 principal components.
  • -r 0 1 2: tells the application to reconstructs meshes at +2 and -2 standard deviations along the first 3 principal components (number starts at 0).
  • -o shape_model/femur: defines the output file name. Reconstructed meshes will have this as the first part of their filenames.
  • --plot_pcs 9: tells the application to display plots of the variance explained by each of the first 9 principal components and the spread of our shapes in shape space. Make sure the number given here is equal or less than the number of principal components to calculate.
  • -v: turns on 3D visualisation of the meshes and principal components (mayavi must be working for this to work).

Run gias-trainpcashapemodel --help to see the full list of commandline arguments.

As the application runs, you will see the mean and reconstructed meshes appear in the shape_models folder. There is also the femur.pc.npz file which contains the principal component vectors and scores. This file can be loaded using numpy.load.

Two figures will also appear. The bar graph shows the proportion of variance explained by each principal component. The scatter-plot shows the spread of each meshes principal component score along the first two principal components.

Once you close these two plot windows, the 3-D model viewer will appear if Mayavi is installed and working.

_images/pca_view_1_labelled.png

To show the mean mesh, select “mean” in the “tri surfaces” (a) combobox and click “update” below the combobox (b). The mesh should appear in the interactive scene on the right of the window.

_images/pca_view_2_labelled.png

To interactively change the shape of the mesh along principal components, switch to the “statistical shape model” tab (a). Select “principal components” in the “PC Model” combobox (b), select “mean::tri” in the “PC Geometry” combobox (c). In the “Mode index” field, select the principal component (d), then drag the slider below to deform the mesh along the selected principal component (e).

4. Shape Model Registration

You will need GIAS2 0.4.15 or higher for this step.

One common use of a shape model is non-rigid registration or shape reconstruction from incomplete data. The RBF-based non-rigid registration we used in step 1 was not constrained by what the shape of an object should be, and so can result in unrealistic shapes, especially when fitting to noisy or incomplete data.

In this step we will perform registration to incomplete data by morphing the mean mesh according to the principal components we just calculated. The registration is an optimisation of the translation, rotation, and the PC scores used to deform the mean mesh so that we minimise the distance between the target surface and the deformed mesh mesh. We run:

giaspcreg shape_model/femur_mean.ply shape_model/femur.pc.npz 0 1 2 3 4 ts -a -m 0.1 -v -t segmentations_partial/2008_1526_l_partial.ply -o segmentations_partial/2008_1526_l_recon.ply

In the command above: - shape_model/femur_mean.ply: is the file path of the mean mesh. - shape_model/femur.pc.npz: is the file path of the principal components - 0 1 2 3 4: are the principal components we will use to deform the mean mesh - ts: defines the way registration error is calculated as the distance between each target point and its closest source (mean mesh) point. - -a: turns on the option to automatically perform a rigid-body registration before shape model registration. If you have already aligned your target mesh to the mean mesh, then leave this option out of the command. - -m 0.1: defines the weighting on the mahalanobis penalty term. The higher this weighting, the closer the registered mesh will be to the mean, at the cost of accuracy. - -v: turns on visualisation so we can see the registration results. - -t segmentations_partial/2008_1526_l_partial.ply: is the file path of the partial segmentation we would like to register to. - -o segmentations_partial/2008_1526_l_pcreg.ply: is the file path of the output registered mesh.

As the application runs, you will see some results printed in the command prompt. “initial rms distance” and “final rms distance” are the root-mean-squared distances between the target surface and the mean surface before and after shape model registration.

In the 3-D visualiser, selecting “target”, “mean”, and “mean morphed” and clicking update will show the target partial surface, the mean mesh, and the registered mean mesh, respectively. You can see that the target surface is missing most of the femoral shaft. The registered mean mesh is fitted to the proximal, distal, and the mid-shaft. There are some obvious errors on the femoral head and the mid-shaft, mainly due to the very small training set of the shape model. For an analysis of the accuracy of shape model reconstruction of partial data, see this paper.

5. Quantifying Shape Error [TODO]

Coming soon.