Awesome-Sketch-Synthesis

A collection of papers about Sketch Synthesis (Generation). Mainly focus on stroke-level vector sketch synthesis.

Feel free to create a PR or an issue.

Outlines

• 0. Survey
• 1. Datasets
• 2. Sketch-Synthesis Approaches
  • 1) Semantic Concept-to-sketch
  • 2) Photo-to-sketch
  • 3) Text/Attribute-to-sketch
  • 4) 3D shape-to-sketch
  • 5) Art-to-sketch
• 3. Vector Graphics Generation

---

0. Survey

Paper	Source	Code/Project Link
Deep Learning for Free-Hand Sketch: A Survey	TPAMI 2022	[code]

1. Datasets

Here Vector strokes means having svg data. With photos means having the photo-sketch paired data.

Level	Dataset	Source	Vector strokes	With photos	Notes
Characters	Omniglot		✔️	❌	Alphabets characters
	KanjiVG		✔️	❌	Chinese characters
	Kuzushiji		❌	❌	Japanese characters
Icon	FIGR-8-SVG		✔️	❌	Icons with text descriptions
Systematic Symbol	SketchIME	ACM MM 2023	✔️	❌	Systematic sketches with semantic annotations
Instance-level	TU-Berlin	SIGGRAPH 2012	✔️	❌	Multi-category hand sketches
	Sketchy	SIGGRAPH 2016	✔️	✔️	Multi-category photo-sketch paired
	QuickDraw	ICLR 2018	✔️	❌	Multi-category hand sketches
	QMUL-Shoe-Chair-V2	CVPR 2016	✔️	✔️	Only two categories
	Sketch Perceptual Grouping (SPG)	ECCV 2018	✔️	❌	With part-level semantic segmentation information
	FaceX	AAAI 2019	✔️	❌	Labeled facial sketches
	Creative Sketch	ICLR 2021	✔️	❌	With annotated part segmentation
	ImageNet-Sketch	NeurIPS 2019	❌	❌	50 images for each of the 1000 ImageNet classes
	SEVA	NeurIPS 2023	✔️	✔️	90K human-generated sketches that vary in detail
Scene-level	SketchyScene	ECCV 2018	❌	✔️	With semantic/instance segmentation information
	CMPlaces	TPAMI 2018	❌	✔️	Cross-modal scene dataset
	Context-Skecth	Expressive 2018	❌	✔️	Context-based scene sketches for co-classification
	SketchyCOCO	CVPR 2020	❌	✔️	Scene sketch, segmentation and normal images
	FS-COCO	ECCV 2022	✔️	✔️	Scene sketches with text description
	SFSD	VC 2022	✔️	✔️	Completely hand-drawn scene sketches with label annotation
Drawing from photos	Photo-Sketching	WACV 2019	✔️	✔️	ScenePhoto-sketch paired
	Tracing-vs-Freehand	SIGGRAPH 2021	✔️	✔️	Tracings and freehand drawings of images
Drawing from 3D models	DifferSketching	SIGGRAPH Asia 2022	✔️	❌	3D model-sketch paired, with novice and professional ones
Portrait	CUFS	TPAMI 2009	❌	✔️	Face-sketch pairs
	APDrawing	CVPR 2019	❌	✔️	Portrait-sketch paired
	SKSF-A	EG 2024	❌	✔️	Face-sketch pairs of seven styles
Children's Drawing	Amateur Drawings	TOG 2023	❌	✔️	With character bounding boxes, segmentation masks, and joint location annotations
Rough sketch	Da Vinci	CGI 2018	❌	✔️	Line drawing restoration dataset
	Rough Sketch Benchmark	SIGGRAPH Asia 2020	✔️	✔️	Rough and clean sketch pairs (only for evaluation)
CAD	ld3d	SIGGRAPH 2008	❌	❌	Line Drawings of 3D Shapes
	OpenSketch	SIGGRAPH Asia 2019	✔️	❌	Product Design Sketches
	SketchGraphs	ICML 2020 Workshop	✔️	❌	Sketches extracted from real-world CAD models
	Fusion 360 Gallery	SIGGRAPH 2021	✔️	❌	For 'sketch and extrude' designs
	FloorPlanCAD	ICCV 2021	✔️	❌	With instance and semantic annotations
Anime	Danbooru2021	/	❌	✔️	Anime images annotated with tags
	DanbooRegion	ECCV 2020	❌	❌	Anime images with region annotations
	Danbooru-Parsing	TOG 2023	❌	✔️	For anime portrait parsing and anime translation
	CreativeFlow+	CVPR 2019	❌	✔️	Large densely annotated artistic video dataset
	ATD-12K	CVPR 2021	❌	✔️	Animation frames with flow annotations
	AnimeRun	NeurIPS 2022	❌	✔️	Correspondence dataset for 2D-styled cartoons
	PaintBucket-Character	CVPR 2024	❌	✔️	Animation frames with region annotations
	Sakuga-42M	arxiv 24.05	❌	✔️	Cartoon videos with text descriptions and tags

2. Sketch-Synthesis Approaches

1) Semantic Concept-to-sketch

Level	Paper	Source	Code/Project Link
Instance-level	A Neural Representation of Sketch Drawings (sketch-rnn)	ICLR 2018	[Code] [Project] [Demo]
	Sketch-pix2seq: a Model to Generate Sketches of Multiple Categories		[Code]
	AI-Sketcher : A Deep Generative Model for Producing High-Quality Sketches	AAAI 2019	[Project]
	Stroke-based sketched symbol reconstruction and segmentation (stroke-rnn)	CGA 2019
	BézierSketch: A generative model for scalable vector sketches	ECCV 2020	[Code]
	Pixelor: A Competitive Sketching AI Agent. So you think you can beat me?	SIGGRAPH Asia 2020	[Project] [Code]
	Creative Sketch Generation	ICLR 2021	[Project] [Code]
	Computer-Aided Design as Language	arxiv 2105
	DoodleFormer: Creative Sketch Drawing with Transformers	ECCV 2022	[Project] [Code]
	SketchKnitter: Vectorized Sketch Generation with Diffusion Models	ICLR 2023	[Code]
	Self-Organizing a Latent Hierarchy of Sketch Patterns for Controllable Sketch Synthesis	TNNLS 2023	[Code]

---

2) Photo-to-sketch

• vector image generation

Data type	Paper	Source	Code/Project Link
Facial	Style and abstraction in portrait sketching	TOG 2013
Instance-level	Free-Hand Sketch Synthesis with Deformable Stroke Models	IJCV 2017	[Project] [code]
	Learning to Sketch with Shortcut Cycle Consistency	CVPR 2018	[Code1] [Code2]
	Learning Deep Sketch Abstraction	CVPR 2018
	CLIPasso: Semantically-Aware Object Sketching	SIGGRAPH 2022	[Project] [Code]
Scene-level	Sketch Generation with Drawing Process Guided by Vector Flow and Grayscale	AAAI 2021	[Code]
	CLIPascene: Scene Sketching with Different Types and Levels of Abstraction	ICCV 2023	[Project] [Code]
Technical Drawings	Deep Vectorization of Technical Drawings	ECCV 2020	[Project] [code]
3D Strokes	3Doodle: Compact Abstraction of Objects with 3D Strokes	SIGGRAPH 2024	[code]

• raster image generation

Type	Paper	Source	Code/Project Link
Facial	ArtLine	Online demo	[Code]
	APDrawingGAN: Generating Artistic Portrait Drawings from Face Photos with Hierarchical GANs	CVPR 2019	[Code] [Demo]
	Unpaired Portrait Drawing Generation via Asymmetric Cycle Mapping	CVPR 2020	[Code]
	Line Drawings for Face Portraits From Photos Using Global and Local Structure Based GANs	TPAMI 2020	[Code]
	Quality Metric Guided Portrait Line Drawing Generation from Unpaired Training Data	TPAMI 2022	[Code]
	Human-Inspired Facial Sketch Synthesis with Dynamic Adaptation	ICCV 2023	[Code]
	Stylized Face Sketch Extraction via Generative Prior with Limited Data	EG 2024	[Code] [Project]
Instance-level	Deep Factorised Inverse-Sketching	ECCV 2018
	Synthesizing human-like sketches from natural images using a conditional convolutional decoder	WACV 2020	[Code]
Anime	sketchKeras	online demo	[Code]
	LineDistiller	online demo	[Code]
	Anime2Sketch	online demo	[Code]
	Reference Based Sketch Extraction via Attention Mechanism	SIGGRAPH Asia 2022	[Code]
Scene-level	Photo-Sketching: Inferring Contour Drawings from Images	WACV 2019	[Code] [Project]
	Learning to generate line drawings that convey geometry and semantics	CVPR 2022	[Code] [Project]
Arbitrary	Semi-supervised reference-based sketch extraction using a contrastive learning	SIGGRAPH 2023	[Code] [Project]

---

3) Text/Attribute-to-sketch

• raster image generation

Type	Paper	Source	Code/Project Link
Facial	Text2Sketch: Learning Face Sketch from Facial Attribute Text	ICIP 2018
Scene-level	Sketchforme: Composing Sketched Scenes from Text Descriptions for Interactive Applications	UIST 2019
Scene-level	Scones: Towards Conversational Authoring of Sketches	IUI 2020

• vector image generation

Type	Paper	Source	Code/Project Link
Arbitrary	Modern Evolution Strategies for Creativity: Fitting Concrete Images and Abstract Concepts	arxiv 21.09	[code] [project]
Arbitrary	CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders	NeurIPS 2022	[code]
Arbitrary	StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Translation	IJCAI 2022	[code]
SVG	VectorFusion: Text-to-SVG by Abstracting Pixel-Based Diffusion Models	CVPR 2023	[project]
Arbitrary	SketchDreamer: Interactive Text-Augmented Creative Sketch Ideation	BMVC 2023	[code]
Arbitrary	DiffSketcher: Text Guided Vector Sketch Synthesis through Latent Diffusion Models	NeurIPS 2023	[project] [code]
Icon	IconShop: Text-Based Vector Icon Synthesis with Autoregressive Transformers	SIGGRAPH Asia 2023	[project]
SVG	Text-Guided Vector Graphics Customization	SIGGRAPH Asia 2023	[project]
SVG	SVGDreamer: Text Guided SVG Generation with Diffusion Model	CVPR 2024	[project] [code]
SVG	Text-to-Vector Generation with Neural Path Representation	SIGGRAPH 2024	[project]
Arbitrary	SVGCraft: Beyond Single Object Text-to-SVG Synthesis with Comprehensive Canvas Layout	arxiv 24.04	[code]
SVG	VectorPainter: A Novel Approach to Stylized Vector Graphics Synthesis with Vectorized Strokes	arxiv 24.05

---

4) 3D shape-to-sketch

Paper	Source	Code/Project Link
DeepShapeSketch : Generating hand drawing sketches from 3D objects	IJCNN 2019
Neural Contours: Learning to Draw Lines from 3D Shapes	CVPR 2020	[project] [code]
Cloud2Curve: Generation and Vectorization of Parametric Sketches	CVPR 2021	[project]
Neural Strokes: Stylized Line Drawing of 3D Shapes	ICCV 2021	[code]
Learning a Style Space for Interactive Line Drawing Synthesis from Animated 3D Models	PG 2022
CAD2Sketch: Generating Concept Sketches from CAD Sequences	SIGGRAPH Asia 2022	[project]

---

5) Art-to-sketch

Here we list sketch synthesis based on other image types, like Manga, line art, rough sketch, etc.

a) Line art

• Vector-to-Vector

Paper	Source	Code/Project Link	*-based
Closure-aware Sketch Simplification	SIGGRAPH Asia 2015	[Project]	Optimization
StrokeAggregator: Consolidating Raw Sketches into Artist-Intended Curve Drawings	SIGGRAPH 2018	[Project]	Optimization
StrokeStrip: Joint Parameterization and Fitting of Stroke Clusters	SIGGRAPH 2021	[Project] [code]	Optimization
StripMaker: Perception-driven Learned Vector Sketch Consolidation	SIGGRAPH 2023		Optimization
Region-Aware Simplification and Stylization of 3D Line Drawings	EG 2024		Optimization

• Raster-to-Vector (a.k.a. Vectorization)

Paper	Source	Code/Project Link	*-based
Topology-Driven Vectorization of Clean Line Drawings	TOG 2013		Optimization
Fidelity vs. Simplicity: a Global Approach to Line Drawing Vectorization	SIGGRAPH 2016	[Project]	Optimization
A Delaunay triangulation based approach for cleaning rough sketches	C&G 2018	[Code]	Optimization
Semantic Segmentation for Line Drawing Vectorization Using Neural Networks	EG 2018	[project] [code]	Learning
Deep Line Drawing Vectorization via Line Subdivision and Topology Reconstruction	PG 2019		Learning
Inertia-based Fast Vectorization of Line Drawings	PG 2019		Optimization
Vectorization of Line Drawings via Polyvector Fields	TOG 2019	[Code]	Optimization
Integer-Grid Sketch Simplification and Vectorization	SGP 2020	[Project] [Code]	Optimization
Deep Vectorization of Technical Drawings	ECCV 2020	[project] [code]	Learning
General Virtual Sketching Framework for Vector Line Art	SIGGRAPH 2021	[project] [code]	Learning
Keypoint-Driven Line Drawing Vectorization via PolyVector Flow	SIGGRAPH Asia 2021	[project]	Learning + Optimization
End-to-end Line Drawing Vectorization	AAAI 2022		Learning
Vectorizing Line Drawings of Arbitrary Thickness via Boundary-based Topology Reconstruction	CGF 2022		Heuristic
Singularity-Free Frame Fields for Line Drawing Vectorization	SGP 2023	[code]	Optimization
Deep Sketch Vectorization via Implicit Surface Extraction	SIGGRAPH 2024	[project] [code]	Learning

b) Rough sketch simplification / cleanup

• Datasets and benchmark

Paper	Source	Code/Project Link
A Benchmark for Rough Sketch Cleanup	SIGGRAPH Asia 2020	[Project] [Code]

• Methods

Paper	Source	Code/Project Link
Learning to Simplify: Fully Convolutional Networks for Rough Sketch Cleanup	SIGGRAPH 2016	[Code] [Project]
Mastering Sketching: Adversarial Augmentation for Structured Prediction	SIGGRAPH 2018	[Code] [Project]
Real-Time Data-Driven Interactive Rough Sketch Inking	SIGGRAPH 2018	[Code] [Project]
Perceptual-aware Sketch Simplification Based on Integrated VGG Layers	TVCG 2019

c) Manga (Comics)

Paper	Source	Code/Project Link
Deep extraction of manga structural lines	SIGGRAPH 2017	[Code]
Manga Filling Style Conversion with Screentone Variational Autoencoder	SIGGRAPH Asia 2020	[Project] [Code]
Generating Manga from Illustrations via Mimicking Manga Workflow	CVPR 2021	[Project] [Code]
MangaGAN: Unpaired Photo-to-Manga Translation Based on The Methodology of Manga Drawing	AAAI 2021
MARVEL: Raster Gray-level Manga Vectorization via Primitive-wise Deep Reinforcement Learning	TCSVT 2023

3. Vector Graphics Generation

Here we focus on learning-based vector graphics generation without depending on vector training data, and traditional vectorization algorithms.

• Learning with external black-box (non-differentiable) rendering simulator

Paper	Source	Code/Project Link
Synthesizing Programs for Images using Reinforced Adversarial Learning	ICML 2018	[Code]
Unsupervised Doodling and Painting with Improved SPIRAL	arxiv 1910	[Project]

• Learning with built-in differentiable rendering module

Paper	Source	Code/Project Link
Optimize and Reduce: A Top-Down Approach for Image Vectorization	AAAI 2024	[code]
Segmentation-Based Parametric Painting	arxiv 23.11	[code] [project]
Editable Image Geometric Abstraction via Neural Primitive Assembly	ICCV 2023
Stroke-based Neural Painting and Stylization with Dynamically Predicted Painting Region	ACM MM 2023	[code]
Intelli-Paint: Towards Developing More Human-Intelligible Painting Agents	ECCV 2022	[project]
Towards Layer-wise Image Vectorization	CVPR 2022	[code] [project]
Paint Transformer: Feed Forward Neural Painting with Stroke Prediction	ICCV 2021	[code]
Combining Semantic Guidance and Deep Reinforcement Learning For Generating Human Level Paintings	CVPR 2021	[code]
Rethinking Style Transfer: From Pixels to Parameterized Brushstrokes	CVPR 2021	[code]
Im2Vec: Synthesizing Vector Graphics without Vector Supervision	CVPR 2021	[Project] [code]
Stylized Neural Painting	CVPR 2021	[Code] [project]
Learning to Paint With Model-based Deep Reinforcement Learning	ICCV 2019	[code]
Strokenet: A neural painting environment	ICLR 2019	[Code]
Neural Painters: A learned differentiable constraint for generating brushstroke paintings	arxiv 1904	[Code]
Learning to Sketch with Deep Q Networks and Demonstrated Strokes	arxiv 1810
Unsupervised Image to Sequence Translation with Canvas-Drawer Networks	arxiv 1809

• Vectorization

Paper	Source	Code/Project Link
Depixelizing pixel art	SIGGRAPH 2011
Perception-Driven Semi-Structured Boundary Vectorization	SIGGRAPH 2018	[Webpage]
PolyFit: Perception-aligned Vectorization of Raster Clip-art via Intermediate Polygonal Fitting	SIGGRAPH 2020	[Webpage] [Code]
ClipGen: A Deep Generative Model for Clipart Vectorization and Synthesis	TVCG 2021
TCB-Spline-Based Image Vectorization	TOG 2022
Image vectorization and editing via linear gradient layer decomposition	SIGGRAPH 2023