Engineering professors develop human-AI colorization tool for cartoonists

George Mason University Information and Sciences (IST) Associate Professor Sungsoo Ray Hong has always loved cartoons. Now he's helping the artists who make them. Drawing on his expertise in human-computer interaction (HCI) and artificial intelligence (AI), Hong and colleague Yotam Gingold, an associate professor in the Department of Computer Science (CS) created ShadowMagic, a new human-AI collaboration tool designed to make one of the most time-consuming parts of comic creation faster and easier.

The idea for ShadowMagic emerged from interviews with professional cartoonists. Hong, Gingold and their PhD students wanted to reduce the burden of shadowing, a task many artists find repetitive and time-consuming.

"Professional cartoonists are sometimes producing content every other day," said Hong, who is also director of George Mason's Alignment Lab. "You cannot commit to these kinds of deadlines doing everything by yourself. That's why the colorization industry is so helpful to professional cartoonists."

Colorization is a multi-step process of line drawing, flatting, base colorization, adding shadows and lighting, backgrounds, and other special effects. It is "fairly standardized, especially in South Korea and growing throughout the United States," said Hong, who is from South Korea. ShadowMagic builds on an earlier project, FlatMagic, another AI-driven colorization tool designed to support digital comic professionals.

While there are existing AI tools that help with colorization, Hong and Gingold found that these tools require more work than doing it manually. They wanted to design a system that better augments cartoonists' work and cuts down on shadowing, a labor intensive and, according to cartoonists, decidedly uncreative part of the process.

"We looked at artistic challenges, usually tedious things that aren't so fun for the artists and break down those problems, so that they can focus more on the creative parts of it," said Gingold, who directs George Mason's Computational Reality, Creativity, and Graphics Lab.

Gringold explained that there are millions of pixels in an image, and if an artist wants to move or change the color of an object, they currently have to point at one pixel among the millions.

Gingold's research applies computer science to visual, geometric, and design problems. He has "always been interested in sketching and how computers can help us express ourselves." Many of his algorithms automate filling in pixels and handling visual constraints.

"The computers will do the math where we can write formulas to update all the other millions of pixels. Another example would be keeping lines parallel and corners perpendicular, and that's where a lot of my research comes in," he said.

Chuan Yan, PhD '24, was recruited by Gingold for this project and conducted research on sketching and painting. Yan, now a postdoctoral research at Stanford University, brought expertise in computer graphics and sketch-related computer vision techniques, which he had studied extensively with Gingold.

The team built ShadowMagic as an Adobe Photoshop plug-in, and since its release, it has attracted a lot of interest from professional artists.

"Artists have expressed a strong desire for a tool that could free them from the labor-intensive flatting process," said Yan. "This response is both exciting and humbling. It confirms that our research is addressing a real need in the industry yet also highlights the remaining challenges in making [the tool] ready for widespread adoption."

Yan believes ShadowMagic is just the beginning of a broader shift in creative industries towards tools that amplify creativity rather than replace it.

"I am excited to witness and contribute to this transformation, helping to shape a future where AI acts as a powerful tool that amplifies human creativity rather than replacing it," he said.

Read the full paper: ShadowMagic: Designing Human-AI Collaborative Support for Comic Professionals' Shadowingby Amrita Ganguly, Chuan Yan, John Joon Young Chung, Tong Steven Sun, Yoon Kiheon, Yotam Gingold, and Sungsoo Ray Hong. It was presented at UIST 2024.