Deep Shading

Convolutional Neural Networks for Screen-Space Shading

Oliver Nalbach1     Elena Arabadzhiyska1,2     Dushyant Mehta1     Hans-Peter Seidel1     Tobias Ritschel3

1 MPI Informatik    2 MMCI / Saarland University    3 University College London

An overview of Deep Shading.


In computer vision, convolutional neural networks (CNNs) have recently achieved new levels of performance for several inverse problems where RGB pixel appearance is mapped to attributes such as positions, normals or reflectance. In computer graphics, screen-space shading has recently increased the visual quality in interactive image synthesis, where per-pixel attributes such as positions, normals or reflectance of a virtual 3D scene are converted into RGB pixel appearance, enabling effects like ambient occlusion, indirect light, scattering, depth-of-field, motion blur, or anti-aliasing. In this paper we consider the diagonal problem: synthesizing appearance from given per-pixel attributes using a CNN. The resulting Deep Shading simulates various screen-space effects at competitive quality and speed while not being programmed by human experts but learned from example images.



Oliver Nalbach, Elena Arabadzhiyska, Dushyant Mehta, Hans-Peter Seidel, Tobias Ritschel
Deep Shading: Convolutional Neural Networks for Screen-Space Shading
to appear in Proc. EGSR 2017

	author		= {Oliver Nalbach and Elena Arabadzhiyska and Dushyant Mehta
					and Hans-Peter Seidel and Tobias Ritschel},
	title		= {Deep Shading: Convolutional Neural Networks for Screen-Space Shading},
	booktitle	= {Computer Graphics Forum (Proc. EGSR 2017)},
	year		= {2017},
	volume		= {36},
	number		= {4}

Interactive Comparison

This tool allows you to compare effects computed using DeepShading to ground truth images. First, select an effect from the list below. Then, move your mouse across the image to switch between different network architectures and the ground truth, respectively.



What is available?
We provide network definitions, corresponding pre-trained networks as well as our datasets for download. As we use a few custom layers and EXR images, which are not supported by a vanilla Caffe at the present, we also provide a modified version of Caffe which is hosted on GitHub.
How can I use the pre-trained networks?
Our network definitions and snapshots (pre-trained networks) are to be used with our custom version of the Caffe deep learning framework. It is available here.
How are the datasets structured?
The datasets consist of about 61,000 pairs of network inputs (split into several EXR image files) and corresponding ground truth outputs. To get an impression of what these look like consider the comparison tool above which shows exemplary pairs.
How do I need to unpack the archives you provide?
There should be a main folder containing the network definition and solver as well as the image lists. The dataset itself should be organized into different sub-folders, one per scene. The different parts of the dataset below generate the correct folder structure.
What are the "dumped blobs" below?
They contain the weights of the trained network as simple text files. The first line specifies the shape of the blob, the rest of the file lists the actual values in Caffe's canonical order.


In our datasets and paper we have used the following meshes well known in the Computer Graphics community: The conference room by Anat Grynberg and Greg Ward, Sponza by Crytek, and Sibenik Cathedral by Marko Dabrovic. Furthermore, we have used several publicly available 3D objects: Cars, robots, plants, mushrooms and a factory building by Dosch Design. Parts of the Viking Village from the Unity asset store. The Buddha and Dragon models by the Standford 3D Scanning Repository. BlendSwap and TurboSquid have been a valuable source for CC-licensed objects as well. Here, we want to thank the users Jay-Artist (for a pool ball mesh), TrueCryer (chess pieces), G. Samaniego (motorcycle), ProfiDevelopers (shed), Ringo3D (pallets), AzusaNyan (old car), YellowPanda (stones), dono (house), redwind33556 (couch), PencilBox (lamp), stendec (book), JohnsonMartin (soda can), crantisz (milk box), Jo_Anna (chair), DennisH2010 (door), Mig26 (airplane) and Geometricity (flag).