RNN / Sketch-RNN
A recurrent network reads a sequence: point by point, stroke by stroke. Order, direction, pen lifts and timing all survive as information. Sketch-RNN uses that representation to model sketches and generate new ones.
Two model families, two points of view
An RNN on strokes, or a CNN on images?
A drawing can be understood in two ways: as movement or as a picture. Recurrent networks read the temporal sequence of pen points; CNNs look at the shape rasterised from it. Both paths work, and which one is right depends less on accuracy than on the question you are actually asking the model.
CNN / image classification
A CNN receives a rasterised image, 96 × 96 pixels in DrawLa. It learns local patterns such as edges and arcs and assembles them into shapes. Robust, fast and fitting when only one question matters: which word is most likely?
What the raw data actually contains
Quick, Draw! delivers far more than an image can show. Every drawing is a sequence of states, encoded during training preparation as a vector with four features per point:
dx, dy, pen_down, pen_up # relative movement + pen state
SEQ_LEN = 256 # padded to a fixed length
dx and dy are relative movements
rather than absolute coordinates — so the sequence is independent of
where on the canvas someone drew from the outset. From this
representation you can read the order in which somebody worked, where the
pen was lifted and how quickly the hand moved.
Rasterising into an image discards all of that. What remains is the silhouette. Whether that is a loss or a liberation is exactly the question at hand.
Head to head
| Criterion | RNN-based recognition | CNN-based recognition |
|---|---|---|
| Input format | Sequence of pen points with pen-up/pen-down states, variable length. | Rasterised greyscale image of fixed size. |
| Signal learned | Direction of movement, order, temporal build-up, stroke structure. | Shape, contour, local image patterns, spatial arrangement. |
| Typical models | LSTM, GRU, sequence encoders, transformers, Sketch-RNN. | CNNs, ResNet-style networks, lightweight real-time classifiers. |
| Drawing order | Learned — helps when it is meaningful, hurts when it is arbitrary. | Entirely irrelevant: head first or ears first gives the same image. |
| Live prediction | Can continue incrementally, processing only new points. | Recomputes the whole image each time — negligible for small models. |
| Latency | Grows with sequence length; long drawings cost more. | Constant, regardless of how many strokes there are. |
| Deployment | Recurrent state complicates export and quantisation. | Static graph, exports cleanly to ONNX, quantises well. |
| Strengths | Generative models, analysis of the drawing process, questions of style. | Fast, stable classification; simple to operate. |
| Weaknesses | Sensitive to sequence normalisation and outliers. | Loses order, direction and timing when rasterising. |
Why DrawLa takes the CNN route
Three reasons decided it, and the first is not a technical one.
Here, drawing order is noise rather than signal. In the dataset people draw the same cat in wildly different orders: some start with the outline of the head, some with the ears, some with the eyes. A sequence model has to learn to ignore that variation — capacity it no longer has for the actual task. The CNN gets that invariance for free, because rasterising simply produces it.
In a game, constant latency matters more than the last few percent. The client sends the current state eight times per second. With a recurrent model an elaborate drawing with many strokes would take longer than a sparse one — precisely when time is already running out. A fixed 96 × 96 grid makes every prediction equally expensive, whether three or thirty strokes are on the canvas.
Operating it is simply easier. A CNN with fixed input exports as a static graph to ONNX and ships without PyTorch. Recurrent networks carry state across time steps, which noticeably complicates export, quantisation and debugging.
The price is real: the model cannot know that somebody has only just started, or that a stroke was drawn hesitantly. For the question "what is this?" that does not matter. For the question "how did this come about?" it would be the decisive part.
When an RNN would be the better choice
The comparison turns out differently as soon as the drawing process itself becomes the subject. A model meant to draw has to produce strokes, not pixels — Sketch-RNN does exactly that and could not even be formulated with an image representation.
The same holds for handwriting recognition, where order and direction carry much of the information: an "O" and a circle drawn clockwise look identical as an image. Distinguishing drawing styles, or detecting hesitation and correction, likewise depends on the sequence.
A third case is early prediction: if you want to guess after the first few milliseconds, it helps that a recurrent model carries its state forward instead of starting from scratch each time. At our model's size that advantage does not matter — recomputing everything costs only a few milliseconds anyway.
The third way: both
Hybrids are common in research. One branch processes the sequence, a second the rasterised image, and their representations are merged before the classifier. That uses both sources of information and regularly leads benchmark comparisons.
For a game it is hardly worth it: you operate two models, maintain two preprocessing paths that must stay consistent, and gain a few percentage points in an application where the top-5 list decides anyway. The effort is out of proportion — for an application that analyses sketches rather than guessing them, the calculation may come out differently.
Where Sketch-RNN fits in
Sketch-RNN remains the key reference for stroke-based representations. The model learns not only which class a sketch might belong to, but what plausible stroke sequences look like in the first place. It can continue an unfinished drawing or interpolate between two sketches — abilities a classifier fundamentally lacks, because it only ever judges shapes and never creates them.
What the CNN route looks like in practice is shown in QuickDraw with PyTorch using the actual pipeline. The data behind it is explained in the Quick, Draw! dataset, and the original papers are linked under Papers & sources.