Skip to content

Latest commit

 

History

History
668 lines (539 loc) · 61.8 KB

README.md

File metadata and controls

668 lines (539 loc) · 61.8 KB
Raw

Awesome-Sketch-Based-Applications

A collection of sketch based applications.

Feel free to create a PR or an issue.

Outline

0. Survey

Paper Source Code/Project Link
Deep Learning for Free-Hand Sketch: A Survey TPAMI 2022 [code]

1. Sketch Based Image Synthesis

1.1 Automatic Synthesis

  • Natural Image or Object
Paper Source Code/Project Link
SketchyGAN: Towards Diverse and Realistic Sketch to Image Synthesis CVPR 2018 [code]
Image Generation from Sketch Constraint Using Contextual GAN ECCV 2018 [code] [project]
Multi-Instance Sketch to Image Synthesis With Progressive Generative Adversarial Networks IEEE Access 2019
Interactive Sketch & Fill: Multiclass Sketch-to-Image Translation ICCV 2019 [code] [project]
SketchyCOCO: Image Generation from Freehand Scene Sketches CVPR 2020 [code]
Unsupervised Sketch-to-Photo Synthesis ECCV 2020 [code] [project]
Sketch Your Own GAN ICCV 2021 [code] [webpage]
Adversarial Open Domain Adaption for Sketch-to-Photo Synthesis WACV 2022 [code]
Customizing GAN Using Few-Shot Sketches ACM MM 2022
DiffSketching: Sketch Control Image Synthesis with Diffusion Models BMVC 2022 [code]
MaskSketch: Unpaired Structure-guided Masked Image Generation arxiv 23.02
Picture that Sketch: Photorealistic Image Generation from Abstract Sketches CVPR 2023 [project] [code]
  • Human Face / Portrait
Paper Source Code/Project Link
LinesToFacePhoto: Face Photo Generation from Lines with Conditional Self-Attention Generative Adversarial Network ACM MM 2019
DeepFaceDrawing: Deep Generation of Face Images from Sketches SIGGRAPH 2020 [project] [code]
DeepFacePencil: Creating Face Images from Freehand Sketches ACM MM 2020 [project] [code]
Controllable Sketch-to-Image Translation for Robust Face Synthesis TIP 2021
DrawingInStyles: Portrait Image Generation and Editing with Spatially Conditioned StyleGAN TVCG 2022
DeepPortraitDrawing: Generating Human Body Images from Freehand Sketches arxiv 22.05
Semantics-Preserving Sketch Embedding for Face Generation TMM 2023 [project] [code]
Parsing-Conditioned Anime Translation: A New Dataset and Method TOG 2023 [code]
  • 3D image
Paper Source Code/Project Link
3D-aware Conditional Image Synthesis arxiv 23.02 [project] [code]

1.2 Style-conditioned

Paper Source Code/Project Link
TextureGAN: Controlling Deep Image Synthesis with Texture Patches CVPR 2018 [code]
Multimodal Unsupervised Image-to-Image Translation ECCV 2018 [code]
SketchPatch: Sketch Stylization via Seamless Patch-level Synthesis SIGGRAPH Asia 2020
Self-Supervised Sketch-to-Image Synthesis AAAI 2021 [code]
CoGS: Controllable Generation and Search from Sketch and Style ECCV 2022
Adaptively-Realistic Image Generation from Stroke and Sketch with Diffusion Model WACV 2023 [project] [code]
Controllable Visual-Tactile Synthesis ICCV 2023 [project] [code]

1.3 Text-conditioned

Paper Source Code/Project Link
Sketch-Guided Text-to-Image Diffusion Models SIGGRAPH 2023 [project]
Adding Conditional Control to Text-to-Image Diffusion Models ICCV 2023 [code]
T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models arxiv 23.02 [code]
UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild arxiv 23.05 [code] [project]
Stable Doodle (ClipDrop) [demo]

2. Sketch Based Image Editing

  • Arbitrary Image
Paper Source Code/Project Link
Free-Form Image Inpainting with Gated Convolution (DeepFill v2) ICCV 2019 [project] [code]
Fashion Editing with Multi-scale Attention Normalization 1906.00884
DeFLOCNet: Deep Image Editing via Flexible Low-level Controls CVPR 2021 [code]
DeepSIM: Image Shape Manipulation from a Single Augmented Training Sample ICCV 2021 [code] [project]
SketchEdit: Mask-Free Local Image Manipulation with Partial Sketches CVPR 2022 [code] [project]
  • Human Face / Portrait / Hair
Paper Source Code/Project Link
FaceShop: Deep Sketch-based Face Image Editing SIGGRAPH 2018 [project]
CaricatureShop: Personalized and Photorealistic Caricature Sketching TVCG 2018
Sparse, Smart Contours to Represent and Edit Images CVPR 2018 [project]
Example-Guided Style-Consistent Image Synthesis from Semantic Labeling CVPR 2019 [code]
SC-FEGAN: Face Editing Generative Adversarial Network With User's Sketch and Color ICCV 2019 [code]
Intuitive, Interactive Beard and Hair Synthesis with Generative Models CVPR 2020
Deep Plastic Surgery: Robust and Controllable Image Editing with Human-Drawn Sketches ECCV 2020 [code] [project]
DeepFaceEditing: Deep Face Generation and Editing with Disentangled Geometry and Appearance Control SIGGRAPH 2021 [code] [project]
SketchHairSalon: Deep Sketch-based Hair Image Synthesis SIGGRAPH Asia 2021 [project]
Paint2Pix: Interactive Painting based Progressive Image Synthesis and Editing ECCV 2022 [code] [project]
NeRFFaceEditing: Disentangled Face Editing in Neural Radiance Fields SIGGRAPH Asia 2022 [project]
SketchFaceNeRF: Sketch-based Facial Generation and Editing in Neural Radiance Fields SIGGRAPH 2023
  • Anime Editing
Paper Source Code/Project Link
AniFaceDrawing: Anime Portrait Exploration during Your Sketching SIGGRAPH 2023 [project]
  • Video Editing
Paper Source Code/Project Link
DeepFaceVideoEditing: Sketch-based Deep Editing of Face Videos SIGGRAPH 2022 [project]

3. Sketch Based Image Retrieval (SBIR)

  • Object-level
Paper Source Code/Project Link
Sketch Me That Shoe CVPR 2016 [code-caffe] [code-tf] [project]
Deep Multi-task Attribute-driven Ranking for Fine-grained Sketch-based Image Retrieval BMVC 2016
Deep Sketch Hashing: Fast Free-hand Sketch-Based Image Retrieval CVPR 2017 [code]
Deep Spatial-Semantic Attention for Fine-Grained Sketch-Based Image Retrieval ICCV 2017 [project]
Zero-Shot Sketch-Image Hashing CVPR 2018
SketchMate: Deep Hashing for Million-Scale Human Sketch Retrieval CVPR 2018
Generative Domain-Migration Hashing for Sketch-to-Image Retrieval ECCV 2018 [code]
A Zero-Shot Framework for Sketch Based Image Retrieval ECCV 2018 [code]
Generalising Fine-Grained Sketch-Based Image Retrieval CVPR 2019
Doodle to Search: Practical Zero-Shot Sketch-Based Image Retrieval CVPR 2019 [project] [code]
LiveSketch: Query Perturbations for Guided Sketch-Based Visual Search CVPR 2019
Semantically Tied Paired Cycle Consistency for Zero-Shot Sketch-Based Image Retrieval CVPR 2019 [code]
Learning Structural Representations via Dynamic Object Landmarks Discovery for Sketch Recognition and Retrieval TIP 2019
Semantic-Aware Knowledge Preservation for Zero-Shot Sketch-Based Image Retrieval ICCV 2019
TC-Net for iSBIR: Triplet Classification Network for Instance-level Sketch Based Image Retrieval ACM MM 2019
Sketch-Based Image Retrieval With Multi-Clustering Re-Ranking TCSVT 2019
Semi-Heterogeneous Three-Way Joint Embedding Network for Sketch-Based Image Retrieval TCSVT 2019
Zero-Shot Sketch-Based Image Retrieval via Graph Convolution Network AAAI 2020
Sketch Less for More: On-the-Fly Fine-Grained Sketch Based Image Retrieval CVPR 2020 [code]
Solving Mixed-modal Jigsaw Puzzle for Fine-Grained Sketch-Based Image Retrieval CVPR 2020
Fine-Grained Instance-Level Sketch-Based Image Retrieval IJCV 2020
StyleMeUp: Towards Style-Agnostic Sketch-Based Image Retrieval CVPR 2021
More Photos are All You Need: Semi-Supervised Learning for Fine-Grained Sketch-Based Image Retrieval CVPR 2021 [code]
DLA-Net for FG-SBIR: Dynamic Local Aligned Network for Fine-Grained Sketch-Based Image Retrieval ACM MM 2021
Domain-Aware SE Network for Sketch-based Image Retrieval with Multiplicative Euclidean Margin Softmax ACM MM 2021 [code]
Relationship-Preserving Knowledge Distillation for Zero-Shot Sketch Based Image Retrieval ACM MM 2021
Transferable Coupled Network for Zero-Shot Sketch-Based Image Retrieval TPAMI 2021 [project]
TVT: Three-Way Vision Transformer through Multi-Modal Hypersphere Learning for Zero-Shot Sketch-Based Image Retrieval AAAI 2022
Sketching without Worrying: Noise-Tolerant Sketch-Based Image Retrieval CVPR 2022 [code]
Sketch3T: Test-time Training for Zero-Shot SBIR CVPR 2022
Augmented Multi-Modality Fusion for Generalized Zero-Shot Sketch-based Visual Retrieval TIP 2022 [code]
Adaptive Fine-Grained Sketch-Based Image Retrieval ECCV 2022 [code]
Conditional Stroke Recovery for Fine-Grained Sketch-Based Image Retrieval ECCV 2022 [code]
A Sketch Is Worth a Thousand Words: Image Retrieval with Text and Sketch ECCV 2022 [code] [project]
Multi-Level Region Matching for Fine-Grained Sketch-Based Image Retrieval ACM MM 2022 [code]
Prototype-based Selective Knowledge Distillation for Zero-Shot Sketch Based Image Retrieval ACM MM 2022
DLI-Net: Dual Local Interaction Network for Fine-Grained Sketch-Based Image Retrieval TCSVT 2022 [code]
Data-Free Sketch-Based Image Retrieval CVPR 2023 [code]
Zero-Shot Everything Sketch-Based Image Retrieval, and in Explainable Style CVPR 2023 [code]
Exploiting Unlabelled Photos for Stronger Fine-Grained SBIR CVPR 2023 [project]
CLIP for All Things Zero-Shot Sketch-Based Image Retrieval, Fine-Grained or Not CVPR 2023 [project]
Cross-Domain Alignment for Zero-Shot Sketch-Based Image Retrieval TCSVT 2023
Semi-transductive Learning for Generalized Zero-Shot Sketch-Based Image Retrieval AAAI 2023
  • Scene-level
Paper Source Code/Project Link
SceneSketcher: Fine-Grained Image Retrieval with Scene Sketches ECCV 2020
Partially Does It: Towards Scene-Level FG-SBIR with Partial Input CVPR 2022
SceneSketcher-v2: Fine-Grained Scene-Level Sketch-Based Image Retrieval using Adaptive GCNs TIP 2022
FS-COCO: Towards Understanding of Freehand Sketches of Common Objects in Context ECCV 2022 [code] [Dataset]
Scene-Level Sketch-Based Image Retrieval with Minimal Pairwise Supervision AAAI 2023
  • Video Retrieval
Paper Source Code/Project Link
Fine-Grained Instance-Level Sketch-Based Video Retrieval TCSVT 2020
Fine-Grained Video Retrieval with Scene Sketches TIP 2023 [project]

4. Sketch Based 3D Shape Retrieval

Paper Source Code/Project Link
Sketch-based 3D Shape Retrieval using Convolutional Neural Networks CVPR 2015
Learning Cross-Domain Neural Networks for Sketch-Based 3D Shape Retrieval AAAI 2016
Deep Correlated Metric Learning for Sketch-based 3D Shape Retrieval AAAI 2017
Learning Barycentric Representations of 3D Shapes for Sketch-based 3D Shape Retrieval CVPR 2017
Deep Cross-modality Adaptation via Semantics Preserving Adversarial Learning for Sketch-based 3D Shape Retrieval ECCV 2018
Unsupervised Learning of 3D Model Reconstruction from Hand-Drawn Sketches ACMMM 2018
Towards 3D VR-Sketch to 3D Shape Retrieval 3DV 2020 [code] [project]
Uncertainty Learning for Noise Resistant Sketch-Based 3D Shape Retrieval TIP 2021
Towards Fine-Grained Sketch-Based 3D Shape Retrieval TIP 2021
Domain Disentangled Generative Adversarial Network for Zero-Shot Sketch-Based 3D Shape Retrieval AAAI 2022
Retrieval-Specific View Learning for Sketch-to-Shape Retrieval TMM 2023
Doodle to Object: Practical Zero-Shot Sketch-Based 3D Shape Retrieval AAAI 2023 [code]
Democratising 2D Sketch to 3D Shape Retrieval Through Pivoting ICCV 2023

5. Sketch Based 3D Shape Modeling

  • Free-hand sketch
Paper Source Code/Project Link
3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks 3DV 2017 [code] [project]
Shape Synthesis from Sketches via Procedural Models and Convolutional Networks TVCG 2017
DeepSketch2Face: A Deep Learning Based Sketching System for 3D Face and Caricature Modeling SIGGRAPH 2017 [project] [code]
BendSketch: Modeling Freeform Surfaces Through 2D Sketching SIGGRAPH 2017 [project]
Robust Flow-Guided Neural Prediction for Sketch-Based Freeform Surface Modeling SIGGRAPH Asia 2018 [code] [project]
DeepSketchHair: Deep Sketch-based 3D Hair Modeling TVCG 2019
Interactive Liquid Splash Modeling by User Sketches SIGGRAPH Asia 2020
Monster Mash: A Single-View Approach to Casual 3D Modeling and Animation SIGGRAPH Asia 2020 [project] [code] [demo]
SAniHead: Sketching Animal-like 3D Character Heads Using a View-surface Collaborative Mesh Generative Network TVCG 2020
Towards Practical Sketch-Based 3D Shape Generation: The Role of Professional Sketches TCSVT 2020
Sketch2Model: View-Aware 3D Modeling from Single Free-Hand Sketches CVPR 2021
Sketch2Mesh: Reconstructing and Editing 3D Shapes from Sketches ICCV 2021
Real-time Skeletonization for Sketch-based Modeling SMI 2021 [code]
Sketch2Pose: Estimating a 3D Character Pose from a Bitmap Sketch SIGGRAPH 2022 [project] [code]
Sketch2PQ: Freeform Planar Quadrilateral Mesh Design via a Single Sketch TVCG 2022
Structure-aware Editable Morphable Model for 3D Facial Detail Animation and Manipulation ECCV 2022 [code]
SketchSampler: Sketch-based 3D Reconstruction via View-dependent Depth Sampling ECCV 2022 [code]
Deep Reconstruction of 3D Smoke Densities from Artist Sketches EG 2022 [project] [code]
A Diffusion-ReFinement Model for Sketch-to-Point Modeling ACCV 2022 [code]
RaBit: Parametric Modeling of 3D Biped Cartoon Characters with a Topological-consistent Dataset CVPR 2023 [project]
SketchMetaFace: A Learning-based Sketching Interface for High-fidelity 3D Character Face Modeling TVCG 2023 [project] [code]
  • NeRF
Paper Source Code/Project Link
SKED: Sketch-guided Text-based 3D Editing ICCV 2023 [project]
  • CAD sketch
Paper Source Code/Project Link
Lifting Freehand Concept Sketches into 3D SIGGRAPH Asia 2020 [project] [code]
Sketch2CAD: Sequential CAD Modeling by Sketching in Context SIGGRAPH Asia 2020 [project] [code]
Free2CAD: Parsing Freehand Drawings into CAD Commands SIGGRAPH 2022 [project] [code]
Symmetry-driven 3D Reconstruction From Concept Sketches SIGGRAPH 2022 [project]
Piecewise-smooth Surface Fitting Onto Unstructured 3D Sketches SIGGRAPH 2022 [project]
Reconstruction of Machine-Made Shapes from Bitmap Sketches SIGGRAPH Asia 2023 [project]

6. Sketch Based Garment Design

Paper Source Code/Project Link
Sketching Folds: Developable Surfaces from Non-Planar Silhouettes TOG 2015 [project]
FoldSketch: Enriching Garments with Physically Reproducible Folds SIGGRAPH 2018 [project]
Learning a Shared Shape Space for Multimodal Garment Design SIGGRAPH Asia 2018 [project]
Garment Ideation: Iterative view-aware sketch-based garment modeling 3DV 2022 [code]

7. Sketch Completion

  • Sketch completion
Paper Source Code/Project Link
SketchGAN: Joint Sketch Completion and Recognition with Generative Adversarial Network CVPR 2019
SketchHealer: A Graph-to-Sequence Network for Recreating Partial Human Sketches BMVC 2020 [code]
Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning of Sketch Gestalt CVPR 2020 [code]
Generative Sketch Healing IJCV 2022
  • Sketch gap / connectivity detection
Paper Source Code/Project Link
Joint Gap Detection and Inpainting of Line Drawings CVPR 2017 [project] [code]
Detecting Viewer-Perceived Intended Vector Sketch Connectivity SIGGRAPH 2022 [project]

8. Sketch Restoration, Retargeting and Inpainting

  • Manga
Paper Source Code/Project Link
Exploiting Aliasing for Manga Restoration CVPR 2021 [webpage] [code]
Seamless Manga Inpainting with Semantics Awareness SIGGRAPH 2021 [webpage] [code]
Screentone-Preserved Manga Retargeting arxiv 22.03
Manga Rescreening with Interpretable Screentone Representation arxiv 23.06
  • Hand-Drawn Drawings
Paper Source Code/Project Link
Restoration of Hand-Drawn Architectural Drawings using Latent Space Mapping with Degradation Generator CVPR 2023

9. Sketch and Shading

Paper Source Code/Project Link
Learning to Shadow Hand-drawn Sketches CVPR 2020 [project] [code]
SmartShadow: Artistic Shadow Drawing Tool for Line Drawings ICCV 2021 [project]

10. Sketch Enhancement

Paper Source Code/Project Link
SketchMan: Learning to Create Professional Sketches ACM MM 2020 [code]

11. Sketch-Guided Object Localization

Paper Source Code/Project Link
Sketch-Guided Object Localization in Natural Images ECCV 2020 [project] [code]
Localizing Infinity-shaped fishes: Sketch-guided object localization in the wild arxiv 21.09 [code]
Query-guided Attention in Vision Transformers for Localizing Objects Using a Single Sketch arxiv 23.03
What Can Human Sketches Do for Object Detection? CVPR 2023 [project]

12. Sketch-Guided Video Synthesis

Paper Source Code/Project Link
Deep Sketch-guided Cartoon Video Inbetweening TVCG 2021 [code]

13. Sketch Recognition

Paper Source Code/Project Link
Sketch-a-Net that Beats Humans BMVC 2015 [code]
Sketch-a-Net: A Deep Neural Network that Beats Humans IJCV 2017 [code]
Deep Self-Supervised Representation Learning for Free-Hand Sketch TCSVT 2020 [code]
S3Net:Graph Representational Network For Sketch Recognition ICME 2020 [code]
Sketchformer: Transformer-based Representation for Sketched Structure CVPR 2020 [code]
Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning of Sketch Gestalt CVPR 2020 [code]
Multi-Graph Transformer for Free-Hand Sketch Recognition TNNLS 2021 [code]
Sketch-R2CNN: An RNN-Rasterization-CNN Architecture for Vector Sketch Recognition TVCG 2021 [code]
Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting CVPR 2021 [code]
SketchAA: Abstract Representation for Abstract Sketches ICCV 2021
SketchLattice: Latticed Representation for Sketch Manipulation ICCV 2021 [code]
Multi-Stage Spatio-Temporal Networks for Robust Sketch Recognition TIP 2022

14. Sketch Segmentation and Perceptual Grouping

  • Semantic / Instance Segmentation
Type Paper Source Code/Project Link
Pixelwise SPFusionNet: Sketch Segmentation Using Multi-modal Data Fusion ICME 2019
SketchyScene: Richly-Annotated Scene Sketches (scene-level) ECCV 2018 [code]
Exploring Local Detail Perception for Scene Sketch Semantic Segmentation (scene-level) TIP 2022
Stroke-level Free Hand-Drawn Sketch Segmentation ECCV 2012
Data-driven Segmentation and Labeling of Freehand Sketches SIGGRAPH Asia 2014 [code] [project] [dataset]
Example-Based Sketch Segmentation and Labeling Using CRFs TOG 2016
SketchSegNet+: An End-to-End Learning of RNN for Multi-Class Sketch Semantic Segmentation IEEE Access 2019
Fast Sketch Segmentation and Labeling With Deep Learning CGA 2019
Stroke-based sketched symbol reconstruction and segmentation CGA 2020
SketchGNN: Semantic Sketch Segmentation with Graph Neural Networks TOG 2021 [code]
One Sketch for All: One-Shot Personalized Sketch Segmentation TIP 2022
Sketch-Segformer: Transformer-Based Segmentation for Figurative and Creative Sketches TIP 2023 [code]
  • Panoptic Segmentation
Type Paper Source Code/Project Link
Vector FloorPlanCAD: A Large-Scale CAD Drawing Dataset for Panoptic Symbol Spotting ICCV 2021 [project]
GAT-CADNet: Graph Attention Network for Panoptic Symbol Spotting in CAD Drawings CVPR 2022
CADTransformer: Panoptic Symbol Spotting Transformer for CAD Drawings CVPR 2022 [code]
VectorFloorSeg: Two-Stream Graph Attention Network for Vectorized Roughcast Floorplan Segmentation CVPR 2023 [code]
  • Perceptual Grouping
Type Paper Source Code/Project Link
Stroke-level Sketching by perceptual grouping ICIP 2013
Making Better Use of Edges via Perceptual Grouping CVPR 2015
Universal Sketch Perceptual Grouping / Toward Deep Universal Sketch Perceptual Grouper ECCV 2018 / TIP 2019 [code]

15. Sketch Representation Learning

  • Stroke order importance/saliency, sketch abstraction
Paper Source Code/Project Link Data Manner Related Applications
Learning Deep Sketch Abstraction CVPR 2018 Vector/stroke-level FG-SBIR
Goal-Driven Sequential Data Abstraction ICCV 2019 Vector/stroke-level Sketch recognition
Pixelor: a competitive sketching AI agent. So you think you can sketch? SIGGRAPH Asia 2020 [Project] [Code] Vector/stroke-level Sketch synthesis and recognition
SketchAA: Abstract Representation for Abstract Sketches ICCV 2021 Vector/stroke-level Sketch recognition, FB-SBIR, sketch healing
CLIPasso: Semantically-Aware Object Sketching SIGGRAPH 2022 [project] [code] Vector/stroke-level
Abstracting Sketches through Simple Primitives ECCV 2022 [code] Vector/stroke-level Sketch recognition, FG-SBIR
Learning Geometry-aware Representations by Sketching CVPR 2023 [code] Vector/stroke-level object attribute classification, domain transfer, stroke-based generation, FG-SBIR
SketchXAI: A First Look at Explainability for Human Sketches CVPR 2023 [project] Vector/stroke-level sketch recognition
Prediction with Visual Evidence: Sketch Classification Explanation via Stroke-Level Attributions TIP 2023 Vector/stroke-level
  • Conventional Representation Learning
Paper Source Code/Project Link Data Manner Related Applications
A Neural Representation of Sketch Drawings (Sketch-RNN) ICLR 2018 [code] Vector/stroke-level Reconstruction and interpolation
SketchHealer: A Graph-to-Sequence Network for Recreating Partial Human Sketches BMVC 2020 [code] Vector/stroke-level Sketch recognition, retrieval, completion and analogy
Sketchformer: Transformer-based Representation for Sketched Structure CVPR 2020 [code] Vector/stroke-level Sketch classification, SBIR, reconstruction and interpolation
CoSE: Compositional Stroke Embeddings NeurIPS 2020 [Code] Vector/stroke-level Auto-completing diagrams
SketchLattice: Latticed Representation for Sketch Manipulation ICCV 2021 [code] Lattice graph Sketch healing and image-to-sketch synthesis
SketchODE: Learning neural sketch representation in continuous time ICLR 2022 [Project] Vector/Stroke-level Continuous-time representation, reconstruction & interpolation
Linking Sketch Patches by Learning Synonymous Proximity for Graphic Sketch Representation AAAI 2023 [code] Vector/Stroke-level Sketch synthesis and sketch healing
SketchKnitter: Vectorized Sketch Generation with Diffusion Models ICLR 2023 [code] Vector/Stroke-level vectorized sketch generation
ChiroDiff: Modelling chirographic data with Diffusion Models ICLR 2023 [Project] Vector/Stroke-level vectorization, de-noising/healing, abstraction
Paper Source Code/Project Link Data Manner Related Applications
A Learned Representation for Scalable Vector Graphics ICCV 2019 [code] SVG Font design
DeepSVG: A Hierarchical Generative Network for Vector Graphics Animation NeurIPS 2020 [Code] [Project] SVG Vector Graphics Animation, reconstruction and interpolation
SVGformer: Representation Learning for Continuous Vector Graphics using Transformers CVPR 2023 SVG Reconstruction, classification, interpolation, retrieval
  • Self-supervised or few/zero-shot learning
Paper Source Code/Project Link Data Manner Related Applications
Sketch Fewer to Recognize More by Learning a Co-Regularized Sparse Representation TCSVT 2019 few-shot classification
On Learning Semantic Representations for Large-Scale Abstract Sketches TCSVT 2020 [Code] Bitmap and Vector Retrieval and recognition
Deep Self-Supervised Representation Learning for Free-Hand Sketch TCSVT 2020 [Code] Vector/stroke-level Retrieval and recognition
Sketch-BERT: Learning Sketch Bidirectional Encoder Representation from Transformers by Self-supervised Learning of Sketch Gestalt CVPR 2020 [Code] Vector/stroke-level Sketch recognition, retrieval, and gestalt
Vectorization and Rasterization: Self-Supervised Learning for Sketch and Handwriting CVPR 2021 [Code] Both Vector and Raster-level Recognition
  • Scene-level
Paper Source Code/Project Link
SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Text CVPR 2023

16. Sketch and Visual Correspondence

  • Datasets
Name Paper Source Code/Project Link
CreativeFlow+ Creative Flow+ Dataset CVPR 2019 [code]
ATD-12K Deep animation video interpolation in the wild CVPR 2021 [code]
AnimeRun AnimeRun: 2D Animation Correspondence from Open Source 3D Movies NeurIPS 2022 [code]
  • Methods
Paper Source Code/Project Link
Globally optimal toon tracking SIGGRAPH 2016 [project]
SketchDesc: Learning Local Sketch Descriptors for Multi-view Correspondence TCSVT 2020
SketchZooms: Deep Multi-view Descriptors for Matching Line Drawings CGF 2021 [project]
The Animation Transformer: Visual Correspondence via Segment Matching ICCV 2021
Differential Operators on Sketches via Alpha Contours SIGGRAPH 2023 [code]

17. Sketch Animation/Inbetweening

  • Inbetweening
Paper Source Representation Code/Project Link
BetweenIT: An Interactive Tool for Tight Inbetweening CGF 2010 stroke
Context-Aware Computer Aided Inbetweening TVCG 2017 stroke
FTP-SC: Fuzzy Topology Preserving Stroke Correspondence SCA 2018 stroke [webpage] [video]
Cacani: 2d animation and inbetween software / stroke [software]
Stroke-Based Drawing and Inbetweening with Boundary Strokes CGF 2021 stroke
Deep Geometrized Cartoon Line Inbetweening ICCV 2023 vertex/point [code]
Optical Flow Based Line Drawing Frame Interpolation Using Distance Transform to Support Inbetweenings ICIP 2019 raster
Bridging the Gap: Fine-to-Coarse Sketch Interpolation Network for High-Quality Animation Sketch Inbetweening arxiv 23.08 raster
  • Animation
Paper Source Representation Code/Project Link
Autocomplete Hand-drawn Animations SIGGRAPH Asia 2015 stroke [webpage] [video]
Live Sketch: Video-driven Dynamic Deformation of Static Drawings CHI 2018 vector [video]
Animating Portrait Line Drawings from a Single Face Photo and a Speech Signal SIGGRAPH 2022 image [code]
A Method for Animating Children’s Drawings of the Human Figure TOG 2023 image [code] [project] [demo]
Non-linear Rough 2D Animation using Transient Embeddings CGF 2023 stroke

18. Sketch and AR/VR

Paper Source Code/Project Link
SweepCanvas: Sketch-based 3D Prototyping on an RGB-D Image UIST 2017 [video]
Model-Guided 3D Sketching TVCG 2018 [video]
Mobi3DSketch: 3D Sketching in Mobile AR CHI 2019 [video]
Interactive Liquid Splash Modeling by User Sketches SIGGRAPH Asia 2020 [video]
Towards 3D VR-Sketch to 3D Shape Retrieval 3DV 2020 [code] [project]
3D Curve Creation on and around Physical Objects with Mobile AR TVCG 2021 [video]
HandPainter - 3D Sketching in VR with Hand-based Physical Proxy CHI 2021 [video]
Fine-Grained VR Sketching: Dataset and Insights 3DV 2021 [code]
Structure-Aware 3D VR Sketch to 3D Shape Retrieval 3DV 2022 [code]
GestureSurface: VR Sketching through Assembling Scaffold Surface with Non-Dominant Hand TVCG 2023
3D VR Sketch Guided 3D Shape Prototyping and Exploration ICCV 2023 [code]

19. Sketch Based Incremental Learning

Paper Source Code/Project Link
Doodle It Yourself: Class Incremental Learning by Drawing a Few Sketches CVPR 2022 [code]

20. Sketch Quality Measurement

Paper Source Code/Project Link
Finding Badly Drawn Bunnies CVPR 2022

21. Cloud Augmentation with Sketches

Paper Source Code/Project Link
Cloud2Sketch: Augmenting Clouds with Imaginary Sketches ACM MM 2022 [project]

22. Sketch and Re-identification

Paper Source Code/Project Link
Cross-Compatible Embedding and Semantic Consistent Feature Construction for Sketch Re-identification ACM MM 2022 [code]

23. Sketch-based Salient Object Detection

Paper Source Code/Project Link
Sketch2Saliency: Learning to Detect Salient Objects from Human Drawings CVPR 2023