From Pixels to Wireframes: 3D Reconstruction via CLIP-Based Sketch Abstraction

Sketch-based abstraction serves as a powerful mechanism for compactly encoding the essential structure and semantics of objects. While recent developments such as CLIPasso [1] have demonstrated the effectiveness of semantic supervision via CLIP [2] for generating 2D Bézier-based sketches from images, these methods are inherently restricted to two-dimensional representations. As such, they fail to capture the spatial consistency and geometric coherence required for applications involving 3D perception, modeling, and interaction.

This work addresses the extension of sketch abstraction to three dimensions by proposing a fully differentiable framework for 3D sketch reconstruction. Specifically, we introduce a method that optimizes a set of 3D Bézier curves, discretized via spherical Gaussian primitives, to form structured and semantically meaningful wireframe representations. Each curve is modeled as a continuous path sampled into isotropic Gaussian kernels, constrained to lie in 3D space, and rendered through a differentiable rasterization pipeline.

To supervise the optimization process, we leverage a composite CLIP-based objective comprising both global semantic alignment, captured via the final projection layer of a pretrained CLIP model, and localized geometric correspondence, derived from intermediate convolutional activations or attention maps. The optimization proceeds by iteratively projecting the 3D sketch into 2D views, computing perceptual similarity with reference RGB images, and backpropagating through the rendering pipeline to refine the control points of the Bézier curves.

By constraining the sketch abstraction problem to a subset of differentiable Gaussian splatting [3] —where the means are fixed along Bézier paths and the variances and colors remain constant, we obtain a compact and interpretable 3D representation. This formulation enables efficient optimization, multi-view supervision, and strong semantic fidelity without requiring explicit 3D ground truth. The resulting sketch abstractions demonstrate both structural coherence and semantic alignment across a variety of synthetic object datasets.

The task of generating semantically meaningful 3D sketch abstractions has gained recent attention through works such as 3Doodle [4] and Diff3DS [5], both of which propose novel pipelines for constructing view-consistent 3D stroke representations. These approaches differ from ours in their design choices: 3Doodle employs neural abstraction from mesh data via 3D stroke optimization, while Diff3DS leverages differentiable curve rendering combined with Score Distillation Sampling (SDS) method which utilizes diffusion generation for text2Image and Image2Image synthesis which is commonly used in 3D domain. These methods validate the feasibility of 3D sketch generation using neural optimization pipelines under perceptual or geometric constraints.

In contrast, our approach formulates the problem as a constrained instance of Gaussian Splatting, wherein each 3D Bézier curve is discretized into a series of spherical Gaussians with fixed appearance parameters. These primitives are optimized under CLIP-based semantic and geometric losses, enabling an interpretable and differentiable 3D sketch abstraction framework without relying on explicit depth supervision or mesh inputs. Our work builds on two key prior directions: (i) CLIP-driven 2D sketch abstraction and (ii) real-time differentiable rendering with Gaussian primitives, discussed in detail below.

Combining the semantic sketching approach of CLIPasso with the real-time rendering capabilities of Gaussian Splatting, our framework introduces a differentiable 3D sketch representation optimized using CLIP-based supervision. This positions our work uniquely at the intersection of 3D vision, neural rendering, and semantic abstraction.

In this section, we present the three key building blocks of our framework. First, we introduce a fully differentiable rasterization pipeline in which each Bézier curve is modeled as a sequence of spherical Gaussians sampled along its path and projected into 2D via camera intrinsics and extrinsics.

To steer our 3D Bézier sketch toward both semantic fidelity and geometric precision, we use a CLIP-based loss function: a high-level semantic similarity term, aligning rendered views in the semantic space, and a geometric alignment term that preserves pixel-wise detail.

Finally, we describe the model training, which alternates between projecting the current 3D curve set into 2D, computing the CLIP loss against ground-truth images for the current view, and backpropagating through the differentiable renderer to refine the Bézier control points until convergence.

To represent 3D sketches in a differentiable manner, we model 3D Bézier curves using spherical Gaussians (SGs) sampled along each curve’s path. This design allows efficient rasterization of the sketches into 2D images for training. Unlike CLIPasso’s discrete 2D rasterization, our approach supports backpropagation in 3D space and enables full differentiability with respect to the control points. Inspired by recent advances in Gaussian Splatting, we represent Bézier curves as sequences of Gaussians, each defined by a center and a fixed thickness, that can be projected onto 2D views using differentiable camera models.

3D Bézier curves are used for representing 3D sketches of objects.

Each spherical Gaussian is defined as:

$$\phi\left(\mathbf{x}; \mathbf{c}, r\right) = \exp\left(-\frac{\|\mathbf{x} - \mathbf{c}\|^2}{2r^2}\right)$$

where $\mathbf{c} \in \mathbb{R}^3$ is the center of the Gaussian and $r$ is its radius (or thickness). To sample a Bézier curve of length $L$ with a desired overlap ratio $\alpha_o$, the sampling step $d$ and the number of samples $N$ are computed as:

$$d = r \alpha_o, \quad N = \left\lceil \frac{L}{d} \right\rceil$$

For a Bézier curve $\mathbf{B}$ with control points $\mathbf{p}_0, \mathbf{p}_1, \ldots, \mathbf{p}_M \in \mathbb{R}^3$, the centers of the Gaussians sampled along the curve are given by:

$$\mathbf{c}_{n,\mathbf{B}} = \sum_{i=0}^{M} \gamma_i\,\left(1-\frac{n}{N}\right)^{M-i}\,\left(\frac{n}{N}\right)^{i}\,\mathbf{p}_i$$

where $\gamma_i$ are the Bernstein basis coefficients. This makes each Gaussian center fully differentiable with respect to the Bézier control points.

Spherical Gaussians are utilized for differentiable rasterization of the 3D sketches.

After generating the Gaussians, we render them through a differentiable rasterizer that projects the 3D Gaussians into 2D using camera intrinsics and extrinsics [6]. The resulting image $\mathbf{V} \in \mathbb{R}^{W \times H}$ is given by:

$$\mathbf{V} = T\left(\mathbf{c}, r, \mathbf{In}, \mathbf{Ex}, W, H\right)$$

where $\mathbf{In} \in \mathbb{R}^{3 \times 3}$ is the intrinsic matrix encoding camera parameters such as focal length and principal point:

$$\mathbf{In} = \begin{bmatrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix},$$

and $\mathbf{Ex} \in \mathbb{R}^{4 \times 4}$ is the extrinsic matrix representing the camera pose in the world frame, given by:

$$\mathbf{Ex} = \begin{bmatrix} \mathbf{R} & \mathbf{t} \\ \mathbf{0}^{1 \times 3} & 1 \end{bmatrix},$$

where $\mathbf{R} \in \mathbb{R}^{3 \times 3}$ is the rotation matrix and $\mathbf{t} \in \mathbb{R}^{3}$ is the translation vector. Together, these matrices map a 3D point from world coordinates to 2D pixel coordinates in the image plane.

Each pixel $\mathbf{V}_{n_1,n_2}$ corresponds to a projection and accumulation over 3D Gaussians, which are parameterized by their mean positions, covariance matrices, and opacity values:

$$\mathbf{V}_{n_1,n_2} = T\left(f(\mathbf{p}_i, \boldsymbol{\Sigma}_i, \alpha_i, \ldots), \ldots\right)$$

Notably, the strokes used in our sketch abstraction are represented as Bézier curves with 4 control points $\{\mathbf{P}_0, \mathbf{P}_1, \mathbf{P}_2, \mathbf{P}_3\}$, forming a cubic Bézier formulation:

$$\mathbf{B}(t) = (1 - t)^3 \mathbf{P}_0 + 3(1 - t)^2 t \mathbf{P}_1 + 3(1 - t) t^2 \mathbf{P}_2 + t^3 \mathbf{P}_3,\quad t \in [0, 1]$$

This parametric formulation enables fine control over stroke shape and spatial distribution in 3D. The end-to-end differentiability of the pipeline allows training the sketch representation directly using image-level loss functions such as CLIP similarity.

To guide the optimization of the 3D Bézier sketch toward both geometrically accurate and semantically meaningful representations, we leverage a CLIP-based loss function. This loss encourages the rendered sketch views to match the real images not just pixel-wise but in high-level perceptual and semantic space. Following the strategy of CLIPasso, we define a dual-component objective that combines semantic similarity and geometric alignment, each captured from different layers of a pretrained CLIP model.

The final loss is a weighted sum of the semantic and geometric components:

$$\mathcal{L} = w_s \,\mathcal{L}_{\mathrm{semantic}} + w_g \,\mathcal{L}_{\mathrm{geometric}}$$

Here, $w_s$ and $w_g$ are hyperparameters that balance the contribution of each term. The semantic loss $\mathcal{L}_{\mathrm{semantic}}$ is computed using the output of CLIP’s final fully connected layer, which captures the global alignment between rendered sketches and their corresponding RGB images in CLIP’s joint vision-language embedding space.

The geometric loss $\mathcal{L}_{\mathrm{geometric}}$ captures local spatial structure and can be extracted either from CLIP’s early convolutional layers or from token-wise attention maps, depending on the selected variant. This term ensures that the layout and fine-scale details of the projected sketches align with real-world object contours.

During training, the 3D sketch is projected into 2D using the differentiable rasterizer described in Section III-A. The resulting image is then passed through CLIP alongside its target RGB counterpart. The cosine distance between their embeddings drives the optimization, gradually refining the control points of the Bézier curves so that the rendered sketches become increasingly perceptually and semantically aligned with real views.

The training phase aims to optimize a set of 3D Bézier curves such that their 2D projections resemble real object images when viewed from different camera angles. This is achieved through an iterative loop involving projection, perceptual comparison, and gradient-based refinement. By leveraging CLIP embeddings as a supervisory signal, the model learns to align the rendered sketch views with the semantics of real RGB views. The training continues until convergence, producing a geometry-aware sketch representation that is both visually and semantically faithful.

Rasterization of images using camera matrix from the constructred 3D sketch.

In each training iteration, the current 3D Bézier sketch is projected into 2D using the known camera intrinsics and extrinsics. This rasterization step generates a view-dependent binary sketch image that mimics how the 3D curves would appear from that specific viewpoint.

Obtaining the CLIP Loss between the RGB images of the views and sampled sketch views.

The projected sketch view is then compared to the corresponding RGB image of the same view. Both are encoded using CLIP’s vision-language model, and a cosine similarity loss is computed to quantify their perceptual alignment in the embedding space.

Optimizing the control points using the backpropagated loss.

The CLIP loss is backpropagated through the differentiable projection and rendering pipeline to adjust the 3D control points of the Bézier curves. This step ensures that the curves evolve to better match the visual content of the reference images.

Training the model until loss converges.

This process is repeated iteratively for all views in the dataset. Over time, the loss steadily decreases as the curves converge to a semantically accurate and geometrically coherent representation of the object across all viewpoints.