Sources
Papers & sources, annotated
This is not a collection of links but a selection with reasons. Each entry says what the work contributes and where it actually takes effect in DrawLa — from the model architecture through the training details to deployment.
Sketches as sequences
-
A Neural Representation of Sketch Drawings
David Ha and Douglas Eck, 2017. Sketch-RNN models drawings as vector sequences: an LSTM encoder-decoder predicts not a single next point but a probability distribution over the next pen offset, combined with three pen states (draw, lift, end). This lets the model generate sketches, continue them and interpolate between them.
Relevance to DrawLa: the counter-position to the architecture chosen here. Why we stayed with images regardless is set out in RNN versus CNN.
The data
-
The Quick, Draw! Dataset
Google Creative Lab. Over 50 million drawings across 345 categories, published under CC BY 4.0. The documentation describes the four formats — raw with timestamps, simplified, binary and ready-made 28 × 28 bitmaps — along with the per-drawing metadata.
Relevance: DrawLa uses the simplified stroke format with 5,000 drawings per category and rasterises itself. The quirks of the set are explained in the dataset article.
-
Quick, Draw! — the original game
The context in which the data was produced. Worth knowing: a round ends as soon as Google's model guesses the word. The dataset therefore consists largely of barely recognisable drawings — a bias that every model trained on it inherits.
-
QuickDraw Component & API
An archived Google Creative Lab repository: a web component that fetches individual doodles through an API and draws or animates them on a canvas. Not training code, but a playback tool.
Relevance: discussed under Google QuickDraw on GitHub.
Architectural building blocks
-
Deep Residual Learning for Image Recognition
He, Zhang, Ren and Sun, 2015. The core idea is simple: instead of learning a mapping directly, a layer learns only the deviation from the identity, while a shortcut alongside lets the signal pass unimpeded. Only this made very deep networks trainable.
Relevance: DrawLa's model consists of four stages with two residual blocks each. Even at these eight blocks the shortcuts stabilise training noticeably.
-
Batch Normalization
Ioffe and Szegedy, 2015. Normalises each layer's inputs across the mini-batch and thereby permits considerably higher learning rates. The original explanation via "internal covariate shift" is now considered outdated — later work attributes the effect more to a smoother loss landscape. That it works is undisputed.
Relevance: behind every convolution in the model. Without it, a learning rate of 10⁻³ would not be stable at the start of training.
-
Sigmoid-Weighted Linear Units (SiLU)
Elfwing, Uchibe and Doya, 2017. The function
x · σ(x)is smooth and permits small negative values instead of clipping them hard to zero as ReLU does.Relevance: used as the activation throughout. The difference to ReLU is small at this model size, but measurable.
-
Searching for Activation Functions (Swish)
Ramachandran, Zoph and Le, 2017. Found by automated search — and with
x · σ(βx)at β = 1 identical to SiLU. A neat example of a search procedure rediscovering a solution that had already been derived elsewhere.
Training methods
-
Decoupled Weight Decay Regularization (AdamW)
Loshchilov and Hutter, 2017. Shows that Adam with an L2 term in the loss is not the same as true weight decay: the adaptive scaling weakens the regularisation exactly where it would be needed most. AdamW separates the two cleanly.
Relevance: the optimiser in use, with weight decay 10⁻⁴.
-
SGDR: Stochastic Gradient Descent with Warm Restarts
Loshchilov and Hutter, 2016. Introduced the cosine learning-rate curve that is standard today: a slow, even decay instead of stepwise drops.
Relevance: the cosine schedule lowers the learning rate from 10⁻³ to 10⁻⁵ over 40 epochs — without warm restarts.
-
Rethinking the Inception Architecture (label smoothing)
Szegedy and colleagues, 2015. Alongside the Inception-v3 architecture, this paper introduced label smoothing: instead of training towards probability 1.0 for the correct class, a little probability mass is spread across all the others.
Relevance: used with a factor of 0.1 — with a dataset containing faulty labels this is not a nicety but a necessity, so the model does not learn to be confident about wrong targets.
-
Dropout
Srivastava and colleagues, 2014. Randomly disables neurons during training and thereby prevents individual features from relying too heavily on one another.
Relevance: used with p = 0.2 before the classification layer.
Deployment and on-device inference
-
ONNX and ONNX Runtime
A vendor-neutral exchange format for neural networks and its accompanying runtime. Separates the training environment from the inference environment.
Relevance: the model is exported with opset 18; the server runs ONNX Runtime on the CPU — with no PyTorch in the container.
-
LiteRT and Google AI Edge
Google's on-device runtime succeeding TensorFlow Lite, responsible for quantised models on mobile and edge platforms.
Relevance: the route DrawLa would take if recognition moved onto the device — see Mobile & Edge AI.
-
Core ML Tools and ExecuTorch
The platform-specific counterparts: Core ML uses the Neural Engine on Apple devices, while ExecuTorch is PyTorch's own path to lean device inference.
Relevance: the export routes compared in iPhone & Android.
-
Qualcomm GenieX
A recent on-device inference runtime for Qualcomm hardware. Interesting as context: mobile networks increasingly run on the NPU rather than in the cloud.