While<\/span> long<\/span> restricted<\/span> to<\/span> an<\/span> elite<\/span> of<\/span> expert<\/span> digital<\/span> artists, 3D<\/span> content<\/span> creation<\/span> has<\/span> recently<\/span> been<\/span> greatly<\/span> simplified by<\/span> deep<\/span> learning<\/span>.<\/span> Neural<\/span> representations<\/span> of<\/span> 3D<\/span> objects<\/span> have<\/span> revolutionized<\/span> real-world<\/span> capture<\/span> from<\/span> photos, <\/span>while generative<\/span> models<\/span> are<\/span> starting<\/span> to<\/span> enable<\/span> 3D<\/span> object<\/span> synthesis<\/span> from<\/span> text<\/span> prompts. These<\/span> methods<\/span> use <\/span>differentiable <\/span>neural<\/span> rendering<\/span> that<\/span> allows<\/span> efficient<\/span> optimization<\/span> of<\/span> the<\/span> powerful<\/span> and<\/span> expressive<\/span> \u201csoft\u201d<\/span> neural <\/span>representations, but <\/span>ignores physically-based principles<\/span>, and thus has no guarantees on accuracy, severely limiting the utility of the resulting content.<\/span><\/p>\n Differentiable <\/span>physically-based rendering <\/span>on the other hand can produce 3D assets with physics-based parameters, but depends on rigid traditional \u201chard\u201d graphics representations required for light-transport <\/span>computation, that make optimization much harder and is also costly, limiting applicability. Figure 1 illustrates these concepts.<\/span><\/p>\n In<\/span> NERPHYS<\/span> we<\/span> will<\/span> combine<\/span> the<\/span> strengths<\/span> of<\/span> both<\/span> neural<\/span> and<\/span> physically-based<\/span> rendering,<\/span> lifting<\/span> their <\/span>respective limitations<\/span> by<\/span> introducing<\/span> polymorphic<\/span> 3D<\/span> representations, i.e., capable<\/span> of<\/span> morphing<\/span> between <\/span>different<\/span> states<\/span> to accommodate both efficient gradient-based optimization and physically-based light transport. <\/span>By augmenting these<\/span> representations<\/span> with<\/span> corresponding<\/span> polymorphic<\/span> differentiable<\/span> renderers<\/span>,<\/span> our <\/span>methodology<\/span> will<\/span> unleash<\/span> the potential of neural rendering to produce physically-based 3D assets with<\/span>\u00a0guarantees on accuracy.<\/span><\/p>\n NERPHYS<\/span> will<\/span> have<\/span> ground-breaking<\/span> impact<\/span> on<\/span> 3D<\/span> content<\/span> creation,<\/span> moving<\/span> beyond<\/span> today\u2019s<\/span> simplistic <\/span>plausible<\/span> imagery,<\/span> to<\/span> full<\/span> physically-based<\/span> rendering<\/span> with<\/span> guarantees<\/span> on<\/span> error,<\/span> enabling<\/span> the<\/span> use<\/span> of<\/span> powerful <\/span>neural<\/span> rendering <\/span>methods in any application requiring accuracy.<\/span> Our polymorphic approach will fundamentally change how we reason about scene representations for geometry and appearance, while our rendering algorithms will provide a new methodology for image synthesis, e.g., for training data generation or visual effects.<\/span><\/p>\n \u00a0<\/p>\n NERPHYS started Dec. 1, 2024 and runs for 5 years, the principal investigator is George Drettakis.<\/p>\n\n\n![\"\"](\"https:\/\/project.inria.fr\/nerphys\/files\/2025\/01\/teaser-300x65.jpg\")
<\/a>Figure 1: Neural<\/span> Rendering<\/span> produces<\/span> impressive<\/span> imagery<\/span> efficiently<\/span>, revolutionizing<\/span> scene<\/span> capture<\/span> (a), or<\/span> generating <\/span>images <\/span>of<\/span> stunning<\/span> realism<\/span> from<\/span> text<\/span> prompts<\/span> (b),<\/span> but<\/span> 3D<\/span> versions<\/span> (c)<\/span> are<\/span> mostly<\/span> limited<\/span> to<\/span> isolated<\/span> objects,<\/span> lacking<\/span> realism; <\/span>all<\/span> lack a<\/span> physically-based<\/span> image<\/span> formation<\/span> model.<\/span> Differentiable<\/span> Physically-Based<\/span> Rendering<\/span> (d)<\/span> has<\/span> controllable <\/span>accuracy,<\/span> but often has limited initial topology, is <\/span>costly <\/span>and tied to traditional rigid representations needed for ray intersections.<\/span><\/em><\/p>\n