Images that wouldn't load in some regions

Hello, this is the Toonkit engineering team.

We received reports like this from several users outside Korea.

"I can log in and my payment went through fine, but none of the cuts or videos I made are showing up."

We tried to check right away, but on our side the same screens loaded normally. Local, staging, and even production opened from the office all worked. With nothing to reproduce, we couldn't even see what was failing.

How token authentication works

Cuts and videos made in Toonkit are stored in object storage and delivered to the browser through a CDN. But this output should only be viewable by the person who made it — knowing the URL shouldn't be enough to download it.

So the CDN that serves user content has token authentication on it. Public assets like landing page banners and community samples are served from a separate CDN that opens without a token, while anything a user created only passes if the request carries a signed token. The token looks roughly like this.

st=startTime~exp=expiryTime~acl=allowedPath~hmac=signature

The server signs a per-user allowed path and validity window, and the frontend attaches that token as a query parameter on image and video URLs. The CDN edge verifies the signature and decides whether to allow or reject the request.

Monitoring

The first thing we did was add monitoring. We tracked the failing requests through NewRelic.

The result wasn't as detailed as we'd hoped. All we got was the fact that a request had failed — an Error — and nothing beyond that.

This wasn't really a gap in our instrumentation so much as a browser constraint. When an img or video tag fails while loading a resource, the browser fires an error event, and that event doesn't carry the HTTP status code. Whether it was a 403, a 404, or a 500, and whatever reason the response body gave, there's no way to read it from JavaScript. All that remains is the fact that a URL couldn't be fetched.

We had no choice but to look at the environment where the failure was happening.

Reproducing it from another region

So we built that environment ourselves. We spun up an EC2 instance in an overseas region and opened a SOCKS proxy tunnel over SSH.

ssh -D 1080 -N -i "pem.pem" user@IP

-D 1080 opens local port 1080 as a SOCKS proxy so all traffic exits through that server. -N means keep the tunnel alive without opening a remote shell. Anything sent to local port 1080 now leaves from that overseas region.

Then we launched a fresh Chrome through that proxy.

open -na "Google Chrome" --args --proxy-server="socks5://127.0.0.1:1080" --user-data-dir="/tmp/us-test7" --lang=en-US --accept-lang=en-US

Separating the profile with --user-data-dir mattered here. Without it, your everyday Chrome window just opens and either the proxy doesn't apply or your existing session and cache muddy the picture. A separate profile starts clean, with no cookies and no cache. --lang and --accept-lang were there to match the locale to an overseas user as well.

Opening Toonkit in that browser reproduced exactly what users had described. In the network tab, the pages and API calls were all fine — only requests to the CDN serving user content were coming back 403. This was where we first confirmed the requests were being stopped at token verification.

We suspected the token itself, but the signature was valid. The same token passed without issue from within Korea, it hadn't expired, and the allowed path was correct. Thinking it might be an origin connectivity problem, we tested the same overseas route against the public CDN that needs no token — that one was fine. It wasn't regional blocking either. The rejection response said outright that token authentication had failed.

Reaching out to the cloud provider

Once we'd narrowed it to the same token being judged differently depending on region, we contacted our cloud provider. We had checked nearly every variable under our own control. It took a while for a response to come back.

The answer was that requests work fine overseas as long as the URL isn't encoded.

Looking at our frontend code, the token was going through encodeURIComponent on its way into the URL. The = and / inside the token string were being sent as %3D and %2F.

That's where a server-side setting came into play. We sign tokens with escapeEarly turned off, which means the signature is computed over the raw string before any encoding. For the signature to match, the request has to arrive with that same raw string. If the client encodes it once more along the way, the string the CDN verifies and the string the server signed no longer match, and the signature naturally fails.

One thing was still odd. This encoding logic runs identically regardless of region, yet the problem only appeared on requests going out to certain regions. That means the same URL is handled differently from region to region, and we still don't know exactly where that difference comes from. What was clear is that removing the encoding made it work everywhere.

We changed the token attachment to send the raw string. The same function existed in two separate places, so we fixed both, and searched the whole codebase to confirm the pattern wasn't left anywhere else. After deploying we went back in through the overseas proxy and confirmed normal responses.

The single-file access token

Most screens were back to normal after that fix. One was left.

There's a screen that shows the source image passed in as a reference during image generation, and this path built its token differently. Normal screens use a token that allows access to the user's entire path, but here the file URL had to be handed directly to an external AI model. Attaching a token that opens a user's whole path to an outbound URL is risky, so we were generating a separate token allowing only the single file being handed over.

That meant the token's allowed path had no wildcard in it. It listed exactly one file path.

The overseas proxy setup we'd built earlier came in useful again. We threw requests at the same object with the same key, varying only the allowed path.

acl=/user123/animation456/cuts/image.png   → 403
acl=/user123/animation456/cuts/image.png*  → 200

Adding a single asterisk to the end of the path made it pass. Same object, same key, no encoding problem — and the result flipped on one final character. It behaved the same with a cold cache, and from within Korea both variants passed.

Since this token was built to point at a single file, we needed to confirm that appending an asterisk didn't widen its scope. In our storage layout, no other file is ever created that starts with this path. Thumbnails get a prefix ahead of the filename, and format variants only swap the extension, so neither has the original path as a prefix. In practice, still only that one file opens. We confirmed that, then shipped the asterisk.

Preventing a repeat

Regional differences can't be reproduced in test code. A test running in Korea passes in both cases, so no amount of test coverage catches this bug.

So we changed what we were verifying. Instead of testing whether the token actually passes, we added a regression test that pins the shape of the token's allowed path string. If someone later refactors this and drops the asterisk, the test breaks. We also left a code comment explaining why it has to be this shape, along with the measurements. Without that comment it reads as an unnecessary wildcard, and the odds were good that the next person would clean it up.

Looking back, most of our local development and QA happens on a domestic network, which is why we didn't catch this kind of regional difference before shipping. Now, any change touching CDN tokens gets one more check from an overseas region. It takes one EC2 instance and a single SSH command — we only put it in the process after it bit us.

With infrastructure like a CDN, where requests pass through different layers depending on region, the same request can produce different results based on where it exits from. If you hit similar symptoms, recreating that region's conditions and throwing a request at it was faster than digging further into logs.