© 2026 Hanwen Ju. All rights reserved.

Back to Blogs

An Experiment with Encoder-Free Diffusion for 3D Point Clouds

This project asked a simple question: can a diffusion model generate 3D point clouds directly from raw point coordinates, without first compressing the shape into a latent code or voxelizing it into a grid?

Most diffusion-based 3D generation methods use either a latent representation or a voxel-like structure. Here, I tried a more direct setup: represent a point cloud as an unordered set \(X_0 \in \mathbb{R}^{N \times 3}\), add Gaussian noise to the xyz coordinates, and train a point-cloud network to reverse the process.

1. Starting from DDPM

I followed the DDPM formulation [1]. The forward process gradually corrupts a clean point cloud \(X_0\) into noise:

\[ q(X_{1:T} \mid X_0) = \prod_{t=1}^{T} q(X_t \mid X_{t-1}) \]
\[ q(X_t \mid X_{t-1}) = \mathcal{N}\left(X_t;\sqrt{1-\beta_t}X_{t-1}, \beta_t I\right). \]

With the standard reparameterization trick, we do not need to simulate every previous step during training:

\[ X_t = \sqrt{\bar{\alpha}_t}X_0 + \sqrt{1-\bar{\alpha}_t}\epsilon, \quad \epsilon \sim \mathcal{N}(0, I), \quad \bar{\alpha}_t = \prod_{s=1}^{t}(1-\beta_s). \]

The denoising network learns to predict the noise added to the point cloud. The simplified training objective is:

\[ \mathcal{L}_{simple}(\theta) = \mathbb{E}_{t, X_0, \epsilon} \left[ \left\| \epsilon - \epsilon_{\theta}\left( \sqrt{\bar{\alpha}_t}X_0 + \sqrt{1-\bar{\alpha}_t}\epsilon, t \right) \right\|^2 \right]. \]

During sampling, the model starts from Gaussian noise and repeatedly applies the learned reverse transition. Conceptually, the model is asked to turn a noisy unordered set of 3D points into a coherent shape.

2. Why PointNet as the Denoiser?

A point cloud is not an image: the points are unordered, and there is no fixed grid where local convolution naturally applies. This is why I used PointNet and PointNet++ ideas [2], [3]. PointNet is built around permutation-invariant set processing. A simplified form of its universal approximation statement is:

\[ f(S) \approx \gamma\left(\max_{x_i \in S} h(x_i)\right), \]

where \(S\) is a set of points, \(h\) extracts point-wise features, max pooling aggregates a global set feature, and \(\gamma\) maps that global feature to the target output. For denoising, I used a segmentation-style PointNet head so the network outputs another \(N \times 3\) tensor, matching the shape of the input point cloud.

PointNet denoising architecture with time embedding
PointNet-style denoising block with time embedding.

The timestep \(t\) is important because denoising at low noise and high noise are different tasks. I added sinusoidal time embeddings, inspired by Transformer positional embeddings [4], and injected them into the PointNet blocks. I also used residual connections so the network could preserve low-level coordinate information while learning time-dependent corrections.

3. EDM Parameterization

After building the DDPM baseline, I also tested EDM parameterization [5]. Instead of training the network to directly predict one fixed target form, EDM wraps the model output with noise-dependent scaling:

\[ D_{\theta}(x; \sigma) = c_{skip}(\sigma)x + c_{out}(\sigma) F_{\theta}\left(c_{in}(\sigma)x; c_{noise}(\sigma)\right). \]

In this form, \(F_{\theta}\) is the neural network, \(\sigma\) is the noise level, and the coefficients normalize the input and output across different noise scales. In my implementation, I used:

\[ c_{skip}(\sigma)=\frac{\sigma_{data}^2}{\sigma^2+\sigma_{data}^2}, \quad c_{out}(\sigma)= \frac{\sigma\sigma_{data}}{\sqrt{\sigma^2+\sigma_{data}^2}}, \quad c_{in}(\sigma)= \frac{1}{\sqrt{\sigma^2+\sigma_{data}^2}}. \]

Practically, this made the generation process more stable and much faster during sampling. The EDM sampler reached good metric values in far fewer steps than the DDPM sampler.

4. Generated Samples

Below are generated airplane, chair, and car point clouds from the EDM version. The results are not perfect, but they show that a direct point-coordinate diffusion process can learn recognizable 3D structures.

Generated airplane point cloud sample 1 Generated airplane point cloud sample 2 Generated airplane point cloud sample 3 Generated airplane point cloud sample 4
Generated chair point cloud sample 1 Generated chair point cloud sample 2 Generated chair point cloud sample 3 Generated chair point cloud sample 4
Generated car point cloud sample 1 Generated car point cloud sample 2 Generated car point cloud sample 3 Generated car point cloud sample 4

5. Model Comparison

I compared six variants: PointNet, PointNet++, and PointNet++ with multi-scale grouping, each under DDPM and EDM settings. Quantitatively, I used common point cloud generation metrics: MMD-CD, coverage, 1-NN accuracy, and JSD, following prior work on diffusion point cloud generation [6].

Model MMD ↓ COV ↑ 1-NNA ↓ JSD ↓
PointNet (DDPM) 0.527 0.038 0.999 0.205
PointNet++ (DDPM) 0.175 0.006 0.998 0.063
PointNet++ MSG (DDPM) 0.131 0.005 0.995 0.114
PointNet (EDM) 0.185 0.176 0.966 0.037
PointNet++ (EDM) 0.119 0.112 0.992 0.028
PointNet++ MSG (EDM) 0.126 0.152 0.991 0.027

The main pattern was clear: EDM improved generation quality and sampling efficiency. PointNet++ with EDM achieved the best MMD, while PointNet++ MSG with EDM achieved the best JSD. Interestingly, the simpler PointNet EDM model had strong coverage and 1-NNA, suggesting that even a relatively simple set network can work as a useful denoiser when the diffusion objective is well structured.

PointNet DDPM airplane generation sample PointNet++ MSG DDPM airplane generation sample PointNet EDM airplane generation sample PointNet++ MSG EDM airplane generation sample
MMD over sampling steps for diffusion point cloud models
MMD-CD across sampling steps.
JSD over sampling steps for diffusion point cloud models
JSD across sampling steps.

6. What I Learned

The main takeaway is that encoder-free point diffusion is possible, at least for recognizable shape generation. The model does not need a latent shape encoder to learn a meaningful reverse process. However, the generated point clouds can still look blurred or structurally weak compared with methods that use stronger representations, such as latent point diffusion or point-voxel diffusion [7], [8].

My next step would be to replace the denoiser with a stronger point architecture, such as a Point Transformer, or to add attention blocks into the PointNet/PointNet++ backbone. Another interesting direction is point cloud translation, where this model could potentially be combined with diffusion bridge ideas.

References

  1. Ho, J., Jain, A., & Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. NeurIPS.
  2. Qi, C. R., Su, H., Mo, K., & Guibas, L. J. (2017). PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. CVPR.
  3. Qi, C. R., Yi, L., Su, H., & Guibas, L. J. (2017). PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. NeurIPS.
  4. Vaswani, A., et al. (2017). Attention Is All You Need. NeurIPS.
  5. Karras, T., Aittala, M., Aila, T., & Laine, S. (2022). Elucidating the Design Space of Diffusion-Based Generative Models. NeurIPS.
  6. Luo, S., & Hu, W. (2021). Diffusion Probabilistic Models for 3D Point Cloud Generation. CVPR.
  7. Zhou, L., Du, Y., & Wu, J. (2021). 3D Shape Generation and Completion through Point-Voxel Diffusion. ICCV.
  8. Vahdat, A., et al. (2022). LION: Latent Point Diffusion Models for 3D Shape Generation. NeurIPS.

© 2026 Hanwen Ju. All rights reserved. The writing, original project notes, and figures from this project are copyrighted by the author unless otherwise noted. Referenced works remain the property of their respective authors and publishers.