Playful Tones

Introduction

This is the second installment of my extended notes from the ADC 2025 talk in Bristol. Part 1 covered the dynamic UI loader: how we generate XML metadata from a Photoshop document, how the factory pattern builds the component tree, and how sorting layers replace JUCE's add-order Z-stack. This post picks up where that left off and covers what happens when the user resizes the window.

Three topics, in roughly the order we worked through them:

• Making components keep their relative positions and sizes at arbitrary window dimensions.
• Keeping image quality high while resampling, without burning CPU on every paint.
• Actually measuring whether either of the above is working, with an evaluation tool we call CRISP.

Same setup as before: JUCE's native renderer, no web views. Terms in bold are defined in the glossary at the end. Slides are here, public modules are here.

Table of Contents

• Introduction
  • Table of Contents
• Sorting Layers in Practice
• Smooth Resizing
  • The Naive Baseline
  • Per-Component AffineTransform
  • Container-Level Scaling
  • The Fractional-Offset Trick
• Resampling
  • Lanczos and the Alpha Edge Problem
  • Deferred Resampling
  • Skipping Resampling on Retina
• Did We Actually Get It Right?
  • Why Eyeballing Doesn't Scale
  • SSIM, GMSD, LPIPS
  • Optimizing the Hot Path
  • The Real-Time Architecture
  • What Pixel-Perfect Actually Means
• Closing
• Glossary

Sorting Layers in Practice

Part 1 named sorting layers in the glossary but left them there. This is where we actually dig into them.

Every component gets an explicit integer sort order from a SortingLayerManager. Doesn't matter when it was added or which class added it. Named constants give anchor points; the gaps are the room you actually use:

sortingLayerManager = std::make_unique<PlayfulTones::SortingLayerManager> (uiContainer);

if (auto* backgroundComp = uiLoader->getComponentByName ("Background"))
    sortingLayerManager->setSortingLayer (*backgroundComp,
        PlayfulTones::SortingLayerManager::Layer::Background);

if (auto* ledComp = uiLoader->getComponentByName ("background_LED_On"))
    sortingLayerManager->setSortingLayer (*ledComp,
        PlayfulTones::SortingLayerManager::Layer::Background + 1);

if (auto* cablesComp = uiLoader->getComponentByName ("Cables"))
    sortingLayerManager->setSortingLayer (*cablesComp,
        PlayfulTones::SortingLayerManager::Layer::Background + 1);

Background is 0. Content is 100. Overlay is 200. Tooltip is 300. Background + 1 for the LED that sits just above the wood panel but below the controls. Content + 5 for that one knob that has to render after the others because its highlight overlaps them.

If you're maintaining a derived editor class that adds plugin-specific overlays on top of a shared base layout, this pattern is the difference between "works first try" and "works until you change something."

Smooth Resizing

The Assembly PSD has a native resolution. Ours is 2430x1600 for one of the amp plugins. Every position and size in the Layout Definition is expressed in those native coordinates. The user resizes the window to 1215x800, or 1458x960, or anything else in between (aspect ratio locked to the native 243:160), and the UI has to look right at that size. No stepped increments. No perceptible quality cliff.

JUCE gives you several ways to scale a UI to an arbitrary window size, and they trade cleanliness against image quality in different places. We worked through four, from the simplest to the one we shipped. None is strictly wrong; each is the right call in some situation and the wrong one in others. Here they are, with the problem each runs into and where it actually fits.

The Naive Baseline

The obvious first version of applyLayout():

const int scaledX = (compX * targetWidth) / sourceWidth;
const int scaledY = (compY * targetHeight) / sourceHeight;
const int scaledWidth = (compWidth * targetWidth) / sourceWidth;
const int scaledHeight = (compHeight * targetHeight) / sourceHeight;
component->setBounds (scaledX, scaledY, scaledWidth, scaledHeight);

Two problems show up immediately.

Accumulated rounding error: a knob at x=1200 in a 2430-wide bitmap, rendered into a 1600-wide window, lands at floor(1200 * 1600 / 2430) = 790 instead of 790.12. Sub-pixel, sure, but the next knob over does the same in the opposite direction, and the visible spacing between them is wrong in a way you can put a ruler on.

Adjacent components no longer share an edge: a switch's right edge at native x=553 and the next switch's left edge at native x=554 both round to the same integer at some scales and to different integers at others. As you drag the resize handle, the seams open and close.

When it's fine: if the window only ever snaps to a few fixed integer scale factors (1x, 2x), every coordinate lands on a whole pixel and there's nothing to round. When it isn't: the moment the window can be any width, the integer-math approach can't be saved. We support continuous resizing, so we needed sub-pixel positioning.

Per-Component AffineTransform

Lift each component into float space and let JUCE's AffineTransform handle the math. The helper that builds the transform:

juce::AffineTransform ComponentResizer::getRectTransform (
    const juce::Rectangle<float>& source,
    const juce::Rectangle<float>& target)
{
    const float newX = target.getX();
    const float newY = target.getY();

    const float scaleX = target.getWidth()  / source.getWidth();
    const float scaleY = target.getHeight() / source.getHeight();

    return juce::AffineTransform::translation (-source.getX(), -source.getY())
        .scaled (scaleX, scaleY)
        .translated (newX, newY);
}

The order matters. A component at native (1200, 300) with size (580, 560) lives at an arbitrary offset in design space. A naive scaled(scaleX, scaleY) would scale the offset too, and the knob would end up somewhere over by the speaker grille. Translate-to-origin, then scale, then translate-to-target. Each component gets its own transform, computed once per resize.

This fixed the seams. It introduced two new symptoms.

The first is image quality. setTransform on a juce::Component that paints an image means JUCE samples the source bitmap through the transform on every paint. For photorealistic background textures the result is acceptable. For sharp edges (the LED bezel, the silver ring around a knob) it gets mushy.

The second is jitter. As the user drags the resize handle, the transform's scale factor changes continuously, and what was previously a clean pixel-aligned edge starts dancing around by fractions of a pixel. Static screenshots looked fine. Live drags didn't.

When it's fine: if your assets are all soft photographic textures and resizing is a rare, discrete action rather than a live drag, the mushiness and jitter never really surface. When it isn't: sharp edges and a continuously moving scale factor, which is exactly the skeuomorphic-UI-with-drag-resize case. So we tried the opposite extreme.

Container-Level Scaling

Lay every component out at native resolution inside the uiContainer, then scale the whole container as one transform:

void BDMEditor::resized()
{
    uiLoader->applyLayout();

    const float zoomFactor = static_cast<float> (getWidth())
                           / static_cast<float> (getRefWidth());
    uiContainer.setTransform (juce::AffineTransform().scaled (zoomFactor));
}

applyLayout() reverts to integer bounds at native coordinates; the parent setTransform does the scaling. One transform for the entire UI tree.

Cheap. Clean. Wrong.

JUCE now rasterizes the entire UI through a single affine transform on every paint pass. Every knob frame, every switch graphic, every backplate gets resampled together. That's not a CPU disaster on a modern machine, but it's worse than necessary. The real problem is image quality: when you down-resample a 2430x1600 RGBA composite to 1215x800 every frame, fine detail in any individual component is gone. And if you have SVG assets that you rasterized at native resolution, you're now down-resampling pre-rasterized SVG content, which is the worst possible input for any resampler.

The deeper reason sits in JUCE's paint model. Vector primitives (g.fillRoundedRectangle, g.drawLine) honour the current transform and rasterize once at the target resolution. They stay sharp at any scale. Images don't get that treatment: g.drawImage rasterizes the bitmap into the current coordinate system using whatever resampling quality is set. The container-level transform collapses everything into the second category.

When it's fine: prototypes, and UIs built entirely from vector primitives (g.fillRoundedRectangle, g.drawLine) that honour the transform and rasterize sharp at any scale. When it isn't: a bitmap-heavy skeuomorphic UI like ours, where the single container transform collapses every image into one blurry resample. Pixel-perfect quality trumps the "one cheap transform" appeal here.

The Fractional-Offset Trick

The insight that finally worked: we want sub-pixel positioning so the seams stay shut, but we don't want a per-component transform that re-rasterizes the image every paint. What if the component's bounds are integer, but the component knows about its own fractional offset and draws its image into a sub-pixel rect?

The layout pass computes the float-precise target rectangle, then:

auto newIntX      = juce::roundToInt (std::floor (floatX));
auto newIntY      = juce::roundToInt (std::floor (floatY));
auto newIntRight  = juce::roundToInt (std::ceil  (floatX + floatWidth));
auto newIntBottom = juce::roundToInt (std::ceil  (floatY + floatHeight));

auto fractionalX = juce::jlimit (-1.f, 1.f, floatX - static_cast<float> (newIntX));
auto fractionalY = juce::jlimit (-1.f, 1.f, floatY - static_cast<float> (newIntY));

component->setBounds (newIntX, newIntY, newIntRight - newIntX, newIntBottom - newIntY);
component->getProperties().set ("floatX", fractionalX);
component->getProperties().set ("floatY", fractionalY);
component->getProperties().set ("floatW", floatWidth);
component->getProperties().set ("floatH", floatHeight);

Two details to flag. The integer bounds always grow outward: floor for the top-left, ceil for the bottom-right. The component's surface is the smallest integer rectangle that fully contains the float-precise target. That extra fraction of a pixel of margin is what prevents the image from getting clipped at the edges when it sits at a sub-pixel offset inside the container.

The fractional offset and the float-precise dimensions get stashed on the component's properties. Components read them at paint time:

void ImageComponent::paint (juce::Graphics& g)
{
    g.setImageResamplingQuality (juce::Graphics::highResamplingQuality);
    g.drawImage (image, getFloatRect (*this), juce::RectanglePlacement::stretchToFit);
}

drawImage (image, floatRect, stretchToFit) rasterizes the bitmap straight into the float rect inside the integer-bounded surface. JUCE does the sub-pixel placement and the scaling in a single pass. There's no per-component setTransform, the container has no transform, and Component bounds are still integers so JUCE's hit testing and child layout work the way they were designed to.

The cost is a little bookkeeping: the layout pass has to stash the float rect on each component, and every image component needs a paint path that reads it back. Cheap for what it buys, and it's the one approach here with no failure mode we had to design around. This is what shipped. If you read the public bd_ui_loader source today, this is what applyLayoutToComponent does.

Resampling

Sub-pixel layout solved the placement question. Image quality at non-Retina scales was still a problem.

Lanczos and the Alpha Edge Problem

JUCE's default image resampling is fast and produces blurry results at non-trivial scale factors. We were already routing image draws through a Lanczos resampler (via the gin library's applyResize) before drawing. Lanczos is great for downscaling photorealistic content. Sharp without being noisy.

But Lanczos has a well-known property: overshoot near hard transitions. On a knob silhouette with a fully transparent background, Lanczos overshoots at the alpha edge and the result has visible halos and colour fringes around the knob.

We prototyped a fix: a two-pass composite. Lanczos the colour image at high quality, rescale a separate alpha mask at low quality (the mask only needs to define where the knob is, not what it looks like), then keep the Lanczos output only where the mask is opaque and drop everything outside it. The mask was an optional output from the Photoshop exporter, a single silhouette layer at the component's bounds.

juce::Image highQualResampledImage = gin::applyResize (img, width, height,
    gin::ResizeAlgorirm::lanczos); // the gin enum really is spelled "Algorirm"

juce::Image scaledDownMask = resamplingMask.rescaled (width, height,
    juce::Graphics::ResamplingQuality::lowResamplingQuality);

juce::Image maskedImage (juce::Image::ARGB, width, height, true);
for (int y = 0; y < highQualResampledImage.getHeight(); ++y)
    for (int x = 0; x < highQualResampledImage.getWidth(); ++x)
        if (scaledDownMask.getPixelAt (x, y).getAlpha() > 0)
            maskedImage.setPixelAt (x, y, highQualResampledImage.getPixelAt (x, y));

It didn't pan out. Clipping the smooth Lanczos output against a separately-resampled, low-quality mask just trades one edge artifact for another: the hard alpha > 0 threshold on the blocky mask gives you a stair-stepped silhouette that doesn't line up with the antialiased colour edge underneath. A different flavour of wrong, not an improvement. The mask overload still exists in the resampler, but the compositing branch is commented out and the default path is plain Lanczos.

The halo didn't go away; it just stopped being worth chasing. What defused it was deferred resampling (below): during the drag we're on the fast stretchToFit path, so there's no Lanczos and nothing shimmering along the edges while the window moves. The fringe only appears in the settled, high-quality frame after you let go, and there it's static and minor. Whatever slight offset it introduces at the edge is rarely noticeable and not jarring, which is a much lower bar than the mask composite was trying (and failing) to clear.

Deferred Resampling

Resampling isn't free. The Lanczos kernel is a 6-tap filter; running it on every paint for every animated knob during a live resize drag is not workable on modest hardware.

So we don't. Each resampled component wraps itself in a DeferredImageResampler that listens for resize events. While the window is actually moving, the component paints its source frame scaled to the target rect with stretchToFit and lets JUCE's cheap built-in scaling carry the motion:

g.drawImage (*images[imageIndex], floatRect,
    juce::RectanglePlacement::stretchToFit);

Every resize event restarts a 500ms timer. When the window finally holds still and the timer fires, a job goes onto a single dedicated background thread, Lanczos-resamples each frame to the exact target size, caches the result, and repaints. From then until the next resize the component blits the pre-resampled image at 1:1 with drawImageAt, no scaling at all. Smooth during the drag, sharp the moment you let go.

The cache is deliberately dumb: one array of resampled frames per component, discarded and rebuilt whenever the size changes. The talk demo carried a fancier version that keyed the cache on a perceptual hash of the source image, to dedupe identical frames across components without relying on the source pointer. It was a fun problem, but it never earned its complexity in a shipping plugin, so it didn't make the cut.

Skipping Resampling on Retina

The other half of the story: on a high-DPI display, the entire problem goes away.

JUCE exposes the display scale through juce::Desktop::getInstance().getDisplays(). If the component's display has scale > 1.0, the backing surface is already higher resolution than the logical pixel size, and the native compositor handles the downscale at hardware quality. Doing software resampling on top of that is pure cost, so the resampler simply doesn't: it reads the scale and only schedules a Lanczos job when the scale is 1.0. On a Retina display the component just paints its source frame scaled by the compositor, no DeferredImageResampler work at all.

Finding the right display is the fiddly part. A plugin editor lives inside whatever wrapper hierarchy the host hands it, and asking "which display is this component on?" doesn't always answer cleanly. The lookup falls back through several strategies (the component's centre point, then its full screen bounds, then the parent monitor area, then the top-level window) before giving up. Get it wrong and you either resample on a display that didn't need it or skip it on one that did.

A few months later we shipped ScaledImageSet in the public loader. Each component optionally holds a 1x set and a 2x set of frames, the Photoshop exporter emits both, and the component picks the matching set at paint time based on the current display scale. This moves the cost from runtime resampling to the asset pipeline and keeps Retina users on hardware-rate downscaling of the 2x set. Smaller binary if you ship 1x only, sharper image if you ship both.

Shipping both sets grows the asset payload, and on macOS you pay for it twice. JUCE's stock BinaryData compiles assets into the object code, so a universal (arm64 + x86_64) build embeds every byte once per architecture slice. The knob frames land in the fat binary twice over. That duplication is what pushed me to build this packed assets module: it packs the assets into a single archive that lives once in the binary and is shared across both slices, instead of being baked into each. It exposes a BinaryData::getNamedResource-compatible shim, so the BinaryAssetImageLoader from Part 1 takes the packed source as a drop-in swap rather than a rewrite. You keep the sharper 2x assets without the universal binary embedding them twice over.

Did We Actually Get It Right?

The remaining piece of the pipeline is the part that asks: is any of the above actually working?

For a long time the honest answer was "I think so?", and I never liked leaving it there with no way to actually measure it. You'd resize the window, eyeball the result against the Reference Image, decide it looked right, and move on. That holds up when the UI has six components and one designer. It stops holding up once you have a dozen plugins, multiple display scales, and three resampling modes to compare.

So we built a tool, and in the grand JUCE tradition of naming with questionable backronyms, we called it CRISP: Component Render Image Similarity Profiler. It's a JUCE module that captures the live running UI by snapshotting your juce::Component, runs full-reference image quality metrics against a designer-provided reference image, and surfaces the differences as numbers and as overlays.

Why Eyeballing Doesn't Scale

Naive pixel-by-pixel comparison gives you "100% match" or "lots of differences" with no useful structure in between. A 1px border at (10.5, 10.5) looks completely different from one at (10.0, 10.0) under pixel equality, even though to a human they're the same border. FR-IQA (Full Reference Image Quality Assessment) metrics exist exactly for this: they predict perceived quality rather than raw pixel agreement.

We surveyed the options (SSIM, MS-SSIM, FSIM, VIF, GMSD, LPIPS, DISTS), implemented several, and shipped three.

SSIM, GMSD, LPIPS

Plus a homemade Weighted Match score: 100% - (RMS pixel difference, as a percentage). Not a sophisticated metric, but readable as a single percentage you can quote at a designer without explaining what a structural similarity index is.

Optimizing the Hot Path

SSIM by the textbook is slow. A naive implementation of the 11x11 Gaussian window takes about 1.6 seconds on a 1000x1000 image. For a real-time tool you need that in milliseconds.

What got it there:

• Precompute the luminance map once and cache the Gaussian weights. (~10x.)
• Single-pass Welford-style accumulator inside the window. (Consolidated with the above.)
• Parallelize across a juce::ThreadPool by row stripes. (3-4x on big images.)
• juce::Image::BitmapData::getLinePointer() for direct pixel access instead of getPixelAt(). (Last 10%.)

End result on the same 1000x1000 image: about 51 milliseconds. The same lesson came up in the non-SSIM parts of the comparison engine: swapping setPixelColour() for direct uint32_t* writes when generating heatmaps shaved another 20% off, and std::nth_element for the median calculation (O(n)) replaced std::sort (O(n log n)).

LPIPS is a different problem. The Python POC (PyTorch + MPS on Apple Silicon) ran a 2430x1600 image through SqueezeNet in around 214ms, missing the 150ms target. The fix was a 1080p downsample before the call, which got us to about 118ms. LPIPS doesn't need full-resolution input to give a useful perceptual score. The complete SSIM + GMSD + LPIPS pipeline ended up at about 183ms total, under the 200ms target.

Then there was the model load problem. First-run LPIPS spent 3.75 seconds loading model weights. Cached runs spent 35ms. Subprocess-per-call would have paid the load cost every time. The fix was to run the Python interpreter once as a persistent subprocess with line-delimited JSON over pipes, keep the model resident, and pay the load cost once when the tool starts. Boring infrastructure work that turns "unusable" into "interactive."

The Real-Time Architecture

JUCE makes the live capture trivial. The whole snapshot primitive is this:

juce::Image MainComponent::captureComponentSnapshot (juce::Component* comp)
{
    auto bounds = comp->getLocalBounds();
    if (bounds.isEmpty())
        return {};

    juce::Image snapshot (juce::Image::ARGB,
                          bounds.getWidth(), bounds.getHeight(), true);
    juce::Graphics g (snapshot);
    comp->paintEntireComponent (g, false);
    return snapshot;
}

paintEntireComponent walks the component tree exactly as it would on the way to the screen, but into a juce::Graphics backed by an off-screen juce::Image::ARGB. No OS-level screen capture, no GPU readback, no platform-specific permissions. JUCE's own renderer is the snapshot service.

A juce::Timer fires at a configurable rate (1-60Hz, default 5) and pushes each captured frame plus the reference image into a ComparisonEngine. Each metric runs as its own juce::ThreadPoolJob, so the panel can show SSIM and GMSD numbers within a few milliseconds while LPIPS catches up. The result struct tracks each metric's state independently (NotCalculated, Calculating, Available, Failed), and the UI updates per-metric instead of waiting for the slowest one.

The default capture rate is itself a tell. SSIM + GMSD + LPIPS at full bandwidth is too much to run on every frame of an interactive resize. 5Hz gives the user roughly two updates per second of active interaction, which is enough to see whether the implementation is tracking the reference. A manual "Compare" button is the escape hatch for cases where you want a precise snapshot at a known window size.

Captured frames can also be recorded to disk as PNGs through a separate recordingThreadPool. We use this to drive the UI through animations and parameter sweeps offline and review the frames against the reference in batch. The four-way evaluation that produced our verdict on the resize approaches was a recording session like this.

What Pixel-Perfect Actually Means

Three things became clearer once we had numbers instead of intuition.

The metrics don't always agree, and that's useful. SSIM fine, GMSD bad, LPIPS fine: probably an edge-sharpness regression that the other two are too blurry to catch. SSIM bad, GMSD fine: probably a structural difference, something is in the wrong place. All three bad: actually broken, go fix.

The per-pixel maps matter more than the scalar scores. A 96.84% Weighted Match could be uniform 3% noise across the whole image, or it could be a 60% error on a 5% region (a missing LED highlight, say). The heatmap tells you which. We render the GMSD gradient map as an overlay on top of the snapshot, in the same coordinate system, so you can point at the bright region and say "that's the bit that broke."

Full pixel agreement is not the goal at all scales. At some scales an antialiased edge will pick a different alpha than the reference even with the best resampler, the metrics will hate it, and humans won't notice. "Good enough" is a judgement call informed by the metrics, not a number you lock in once and forget. CRISP is the dashboard. The decision is still yours.

Closing

If you're starting a similar project: do the sub-pixel layout work early. It's the lowest-effort, highest-payoff change in this whole post. Don't try to fix resampling quality with a container transform; fix the underlying image draw path. Build the evaluation tool sooner than you think you need it. The first time it catches a regression you didn't see is the moment it pays for itself.

The public modules and the slides are at github.com/Bogren-Digital/PSD-to-DAW. The UI loader is at bd_ui_loader. The Photoshop exporter is at skeuomorphic-ui-exporter. Issues and PRs welcome on any of them.

Glossary