The customs API that returned 7 different schemas (and the parser that survived all of them)

The Macedonian customs API documentation was 14 pages. Our integration broke in production 11 times in the first month. Each break came with the same 200 OK status, the same application/json header, and a completely different payload shape. The declaration_id field once arrived as a string, twice as an integer, three times nested inside a result object, and once — only once — as part of an XML envelope inside a JSON string. That last one took me two days.

This is the story of how I stopped trusting the customs API and started parsing it like a hostile witness.

What we were actually doing

Stork ships goods across the Turkey/North Macedonia border. Every shipment, no matter how small, needs a customs declaration filed with the Macedonian customs authority. There is an official API. There is one. The only alternative is driving to the customs office in Skopje with a folder of printouts.

So we used it. I wrote the first version of the integration in two weeks in February 2024. It worked beautifully in their sandbox. It worked for the first three days in production. On the fourth day, a Tuesday, at 14:22, our country manager in Skopje called me. The warehouse had 38 packages stuck. The declarations were "going through" but coming back without IDs we could store.

The API was returning 200. The body looked fine. The declaration_id field, which the docs said would be an integer at the root, was now a string. Just a string. No reason. No version header changed.

I patched it that afternoon. It broke again eleven days later, in a different way.

Eleven breaks in the first month

I kept a notebook. Here is the first month, anonymized into shape categories:

1. Flat fields, declaration_id at root, integer. The documented shape. Worked for three days.
2. Same shape, but declaration_id is now a string. No idea why. Possibly a different backend.
3. Wrapped in a result object. { "result": { "declaration_id": 12345, ... } }. Showed up only between 09:00 and 11:00 Skopje time. We still don't know why.
4. Wrapped in a data array of length one. { "data": [ { ... } ] }. Started appearing after a customs-side deploy we found out about three days later.
5. Same as (4), but the array could be empty on a successful filing. The declaration ID was elsewhere — in a separate meta object. I almost shipped a parser that read the empty array and silently filed a declaration with no ID. Caught it in code review.
6. HTML page returned with a 200. The customs gateway timing out and serving its own error page. With Content-Type: application/json. Of course.
7. JSON envelope with an XML string inside. I'll come back to this.

That XML-in-JSON one. I want to talk about it.

The two days I lost

It was a Thursday in late March. The integration started failing every few hours. The errors looked random. The body was JSON. It parsed. But the declaration_id field was a string that started with <?xml version="1.0".

I assumed it was a bug in our logging — that we were storing the wrong field somewhere upstream and the XML was leaking in. I spent a full day chasing that bug. There was no such bug.

The customs API was, in fact, returning a JSON object whose declaration_id value was a serialized XML envelope. Inside the envelope, wrapped in a <DeclarationResponse> tag with three layers of namespaces, was the actual integer ID.

I called the API team. The email signature said "Customs IT, Skopje Office, Helpdesk." A man answered. Through my colleague translating from Macedonian, he told me there had been "an upgrade", and the JSON-to-XML gateway was now serializing the inner response differently. Would it be reverted, I asked. He'd check. Two weeks later it was reverted. By then I had already written a parser branch for it.

I left the branch in. Three months later, that schema came back for one afternoon.

The defensive parser

Here's what I should have built on day one, and what I built on day twelve.

The idea is simple. Treat the response as untyped JSON first. Detect which schema you're looking at. Dispatch to a known parser. If no parser matches, alert and fall back.

// We don't trust the documented shape. We detect it.
// Each entry in the registry is a (predicate, parser, label).
type schemaEntry struct {
    label     string
    detect    func(map[string]any) bool
    parse     func(map[string]any) (Declaration, error)
    lastSeen  time.Time
}
 
var registry = []*schemaEntry{
    flatRootIntID,      // documented shape
    flatRootStringID,   // appeared 2024-02-20
    nestedInResult,     // appeared 2024-02-27
    wrappedInDataArray, // appeared 2024-03-06
    xmlInsideJSON,      // appeared 2024-03-21, 2024-06-14
}

Each schema gets its own predicate. The predicate is the cheapest thing you can write that uniquely identifies the shape — usually a single key lookup, sometimes a regex on a value.

// Detection is cheap. Don't be clever, just check the keys.
// Order matters: more specific shapes first, generic last,
// because two shapes can overlap on a happy day.
func detectSchema(body map[string]any) *schemaEntry {
    for _, entry := range registry {
        if entry.detect(body) {
            entry.lastSeen = time.Now()
            return entry
        }
    }
    return nil
}

When detectSchema returns nil, that's not an error in the API sense — the request succeeded, the body parsed as JSON, the status was 200. But it is an error in the trust sense: we've just seen a shape we've never seen before. We need to know about it before someone in the warehouse notices a package is stuck.

// Unknown shape => alert, fall back to polling.
// Polling is slow (~6 min) but it never lies to us.
func dispatchUnknown(raw []byte, shipmentID string) {
    alerts.Page("customs:unknown_schema", map[string]string{
        "shipment_id": shipmentID,
        "raw_excerpt": string(raw[:min(400, len(raw))]),
    })
    queue.Enqueue("customs:manual_review", shipmentID)
    poller.Schedule(shipmentID, 6*time.Minute)
}

The polling fallback hits a different customs endpoint — the old one that lists declarations by our internal reference. It's slow. It's old. It returns CSV. But in two years it has never returned anything we couldn't parse. When the fancy API surprises us, we fall back to the boring one and wait. Nobody's stuck in the warehouse. The page wakes me up only if the unknown schema sticks around past breakfast.

The XML-in-JSON branch, in code

Here's the parser branch that took me two days to need and twenty minutes to write once I understood the shape:

// The customs gateway sometimes serializes the inner response as XML
// and string-encodes it as a JSON field. We unwrap both layers.
// Last seen: 2024-06-14 (one afternoon, then gone again).
func parseXMLInsideJSON(body map[string]any) (Declaration, error) {
    raw, ok := body["declaration_id"].(string)
    if !ok || !strings.HasPrefix(raw, "<?xml") {
        return Declaration{}, errWrongShape
    }
    var env struct {
        XMLName xml.Name `xml:"DeclarationResponse"`
        ID      int64    `xml:"id"`
    }
    if err := xml.Unmarshal([]byte(raw), &env); err != nil {
        return Declaration{}, fmt.Errorf("xml-in-json unwrap: %w", err)
    }
    return Declaration{ID: env.ID, SchemaUsed: "xml-in-json"}, nil
}

The SchemaUsed field is written to the row alongside the rest. Every declaration in our database knows which schema variant it came from. This has been quietly useful for postmortems — when something looks wrong on a shipment three weeks later, we can group by schema and check whether a particular variant was silently degrading.

The registry, after six months

After six months in production, the registry had 14 distinct entries. Half of them we discovered through the fallback alert — the API surprised us, the polling kicked in, an engineer added the new branch within hours. Mean time from unknown-schema alert to deployed parser branch: about four hours. Total customer-visible outages caused by the customs API: zero. Total weekends I would have lost without this design: I count six. I'm being conservative.

The registry also tracks last_seen. Three of the early variants haven't appeared in eight months. We keep the branches anyway, because removing them is cheap risk and adding them back at 2 AM on a Sunday is expensive.

What I'd tell myself in February 2024

Two things.

First: don't trust Content-Type. The customs gateway will tell you the body is JSON when it's XML, HTML, or a JSON object holding an XML string. The header is a hint, not a fact. Parse the body and check the shape before you act on it.

Second: defensive parsing isn't slow. Building the registry, the dispatcher, the fallback poller, and the alert took me three days. The naive version had taken me two weeks and broken eleven times. If I had built defensively from the start, I would have shipped a week earlier and slept through six weekends I did not sleep through. The cost-benefit here is the exact opposite of what the "premature optimization" reflex would say. The expensive thing was the assumption that the documentation described reality.

The honeymoon

Last summer, for one full week — Monday to Friday, August 2024 — the customs API returned a perfect, well-typed schema. Flat fields, integer ID, no surprises. Every shape passed through the same branch. The fallback poller sat idle. The alert channel was silent.

We logged that week separately on our metrics dashboard. The chart still has a little tag on it: "the honeymoon." Our Skopje country manager sent a single Slack message on Friday evening: "Everything's smooth, what are you doing over there?" There was nothing to do. We were just watching the screen.