Intelligent Tools for Creative Graphics

Ariel Shamir & Niloy J. Mitra & Nobuyuki Umetani & Yuki Koyama


In recent years, much research has been dedicated to the development of “intelligent tools” that can assist both professionals as well as novices in the process of creation. Using the computational power of the machine, and involving advanced techniques, the tools handle complex and tedious tasks that were difficult or even impossible for humans, thereby freeing the human creator of many constraints and allowing her to concentrate on the creative process, while ensuring high-quality and valid design. This course is aimed at presenting some of the key technologies used to assist interactive creative processes. The course allows researchers and practitioners to understand these techniques more deeply, and possibly inspire them to research this subject and create intelligent tools themselves. More specifically, the course will concentrate on four main enabling technologies: geometric reasoning, physical constraints, data-driven techniques and machine learning, and crowdsourcing. In each of these areas the course will survey several recent papers and works and provide examples of using these in the creation of a variety of outputs: 3D models, animations, images, videos and more.


Course Notes


Course Contents

Introduction [Talk by Ariel Shamir] — 15 min

Geometric Reasoning [Talk by Niloy J. Mitra] — 45 min

An intelligent agent should be able to generate novel variations that are valid and diverse—geometrically, semantically, and topologically. Examples include generating furniture layout, floor plan generation, 3D models, and more generally virtual 3D environments. In this part of the course, we propose to focus on: i) generative 3D models ii) generative layouts incorporating both topological and geometric variations.

Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy J. Mitra, and Leonidas J. Guibas. 2019. StructureNet: hierarchical graph networks for 3D shape generation. ACM Trans. Graph. 38, 6, Article 242 (November 2019), 19 pages. DOI:

Tuanfeng Y. Wang, Duygu Ceylan, Jovan Popović, and Niloy J. Mitra. 2018. Learning a shared shape space for multimodal garment design. ACM Trans. Graph. 37, 6, Article 203 (December 2018), 13 pages. DOI:

Tao Chen, Zhe Zhu, Ariel Shamir, Shi-Min Hu, and Daniel Cohen-Or. 2013. 3-Sweep: extracting editable objects from a single photo. ACM Trans. Graph. 32, 6, Article 195 (November 2013), 10 pages. DOI:

Physical Constraints [Talk by Nobuyuki Umetani] — 45 min

Computer controlled fabrication tools such as 3D printers, laser cutters and CNC milling machines have become wildly available to the consumers. However, it is difficult to design original customized shapes while considering its physical functionality. In this talk, we present the studies to incorporate real-time physics simulation into help the design of creative functional shapes. In particular, we focus on the (i) modeling of physics (ii) data-driven approach to facilitate the interactive modeling.

Nobuyuki Umetani, Danny M. Kaufman, Takeo Igarashi, and Eitan Grinspun. 2011. Sensitive couture for interactive garment modeling and editing. ACM Trans. Graph. 30, 4, Article 90 (July 2011), 12 pages. DOI:

Nobuyuki Umetani, Takeo Igarashi, and Niloy J. Mitra. 2012. Guided exploration of physically valid shapes for furniture design. ACM Trans. Graph. 31, 4, Article 86 (July 2012), 11 pages. DOI:

Nobuyuki Umetani, Athina Panotopoulou, Ryan Schmidt, and Emily Whiting. 2016. Printone: interactive resonance simulation for free-form print-wind instrument design. ACM Trans. Graph. 35, 6, Article 184 (November 2016), 14 pages. DOI:

Nobuyuki Umetani, Yuki Koyama, Ryan Schmidt, and Takeo Igarashi. 2014. Pteromys: interactive design and optimization of free-formed free-flight model airplanes. ACM Trans. Graph. 33, 4, Article 65 (July 2014), 10 pages. DOI:

Short Break — 15 min

Data-Driven Techniques and Machine Learning [Talk by Ariel Shamir] — 45 min

Recent years have seen a revolution in the amount and ways data is used in all realms of science in general and computer science specifically. Complex problems are solved using machine learning techniques that rely on data. For example, in computer vision neural networks are now used for almost every task successfully. The question is how to make use of such techniques in the context of an intelligent tool for the creation of graphical content.

To this end, there are several machine learning techniques that enable the tool to make independent decisions without explicit input from the human user. These include:

In this part of the course we will present the basics of these technologies and show examples how they are used inside intelligent tools for the creation of various media such as images, video, and 3D objects.

Tao Chen, Ming-Ming Cheng, Ping Tan, Ariel Shamir, and Shi-Min Hu. 2009. Sketch2Photo: internet image montage. ACM Trans. Graph. 28, 5 (December 2009), 1–10. DOI:

Miao Wang, Guo-Wei Yang, Shi-Min Hu, Shing-Tung Yau, and Ariel Shamir. 2019. Write-a-video: computational video montage from themed text. ACM Trans. Graph. 38, 6, Article 177 (November 2019), 13 pages. DOI:

Adriana Schulz, Ariel Shamir, David I. W. Levin, Pitchaya Sitthi-amorn, and Wojciech Matusik. 2014. Design and fabrication by example. ACM Trans. Graph. 33, 4, Article 62 (July 2014), 11 pages. DOI:

Yuki Koyama, Shinjiro Sueda, Emma Steinhardt, Takeo Igarashi, Ariel Shamir, and Wojciech Matusik. 2015. AutoConnect: computational design of 3D-printable connectors. ACM Trans. Graph. 34, 6, Article 231 (October 2015), 11 pages. DOI:

Crowdsourcing [Talk by Yuki Koyama] — 25 min

Tools can be more intelligent if they can adequately model human perception (e.g., visual preference and perceived semantics). Crowdsourcing is an effective way to gather perceptual feedback on visual designs on demand, and researchers in computer graphics have proposed digital content creation tools with crowd intelligence in this decade. In this part of the course, especially focusing on parametric design scenarios, we will describe

For the latter, we will especially discuss two approaches: embedding crowds into a user interface to facilitate users' exploration and decision making and embedding crowds into an optimization engine to achieve automatic perception-aware parameter adjustment, with case studies on photo color enhancement and material BRDF design.

Yuki Koyama, Daisuke Sakamoto, and Takeo Igarashi. 2014. Crowd-powered parameter analysis for visual design exploration. In Proc. UIST '14. 65–74. DOI:

Yuki Koyama, Issei Sato, Daisuke Sakamoto, and Takeo Igarashi. 2017. Sequential line search for efficient visual design optimization by crowds. ACM Trans. Graph. 36, 4, Article 48 (July 2017), 11 pages. DOI:

Discussion — 15 min


Ariel Shamir

is the Dean of the Efi Arazi school of Computer Science at the Interdisciplinary Center in Israel. He received his Ph.D. from the Hebrew University in Jerusalem, and spent two years as PostDoc in the computational visualisation centre at UT Austin. Ariel has numerous publications and a number of patents. He was listed on the Thomson Reuters highly cited researchers in 2015. Ariel has a broad commercial experience consulting various companies including Disney research, Mitsubishi Electric, PrimeSense (now Apple), Verisk and more. He specializes in geometric modeling, computer graphics, image processing and machine learning.

Niloy J. Mitra

is a Professor of Geometry Processing in the Department of Computer Science, University College London (UCL). His research interests include shape analysis, data-driven geometry processing, and computational design and fabrication. Niloy received the 2013 ACM Siggraph Significant New Researcher Award, the BCS Roger Needham award in 2015, and the Eurographics Outstanding Technical Contributions Award in 2019.

Nobuyuki Umetani

is a project lecturer in the creative informatics department at the University of Tokyo, in Japan. His interests are computational fabrication, physics simulation, and interactive techniques.

Yuki Koyama

is a Researcher at National Institute of Advanced Industrial Science and Technology (AIST), Japan. He received his Ph.D. from The University of Tokyo in 2017. His research fields are mainly computer graphics and human-computer interaction. In particular, he is interested in enhancing various design activities by employing computational techniques such as mathematical optimization.