Which Part of Your VPN Is the Protocol?

The protocol is a narrow, specific layer: the rules by which your device and one VPN server prove who they are to each other, agree keys, and format the packets that travel between them. It is not the app, not the virtual interface, not your routing table, and not your resolver configuration — those are separate layers that sit above it on your own machine. Most symptoms people describe as protocol problems are produced by one of those other layers.

This post locates each layer so that a symptom can be traced to the right one. The two comparisons that follow it are how OpenVPN and WireGuard differ by design and WireGuard or IKEv2 when you move between networks.

The four things inside the protocol’s boundary

All of them happen between your device and one server, and none extends past the exit.

Mutual authentication. Your client establishes that the server it reached is the one it meant to reach, and the server establishes that you are entitled to a tunnel. Families do this with certificate validation, with a server public key configured in advance, or with credentials passed inside a channel that is already authenticated.

Key agreement. Both ends derive keys neither of them transmitted, and replace them periodically while the tunnel runs.

Packet format and encapsulation. Your packets are wrapped inside the protocol’s own packets, which is why a tunnel adds overhead to every packet and why the tunnel needs a second address of its own alongside the one your local network gave you.

Integrity and replay handling. Each wrapped packet carries proof it was not modified, plus a counter so a packet captured earlier cannot usefully be sent again.

That is the boundary. Notice what is absent: nothing in that list decides which of your packets are offered to the tunnel in the first place.

The layers above it, on your own machine

Four more layers, each of which can fail while the protocol works perfectly.

The virtual interface. The client creates a network adapter representing the tunnel. It exists, or it does not, independently of whether a handshake succeeded.

The routing table. Which destinations are sent to that interface, and which still go out through your local network. This is the layer that decides what the tunnel carries.

Resolver configuration. Which name servers your device asks, and whether that setting followed the tunnel or stayed as your local network left it. This is a system setting, not part of any tunnel.

The packet filter. Rules deciding whether traffic is permitted to leave by a non-tunnel path at all — the layer a fail-closed switch is built from.

The client software is what installs and removes all four, plus the account login, the server list, per-application rules, and any wrapper disguising the tunnel’s traffic pattern. Which means the client is a far larger piece of software than the protocol it speaks, and a far more common source of surprises.

Tracing a symptom to a layer

Name the layer before changing any setting, because the wrong layer will not respond.

Names resolving to your local network’s resolver while traffic is tunnelled. Resolver configuration layer. Switching protocol changes nothing; see what your VPN’s DNS setting actually changes.

Some destinations reachable without the tunnel while others use it. Routing table. Either a deliberate split-routing rule or an address family the client did not install routes for — most often the second, per why IPv6 is the leak people forget.

Traffic continuing normally after the tunnel stops. Packet filter. The routes reverted to your local default and no rule blocked what followed.

Connected, but nothing loads. Interface and routing layers, or the server. The protocol completed its handshake, which is what “connected” reports.

Handshake never completes on one particular network. Now you are in the protocol layer, and specifically its transport and port. This is the one symptom where changing the protocol setting is the correct response.

Apparent location wrong, or a service refusing you. None of these layers. That is the exit address and how the far end reads it, decided by which server you chose.

The entries name families rather than single specifications.

There is a TLS-based family, of which OpenVPN is the widely deployed example: it borrows the handshake machinery of ordinary encrypted web traffic and can run over either UDP or TCP. There is an IPsec-based family, in which IKEv2 negotiates keys and IPsec carries data, present in the networking stack of every major operating system. And there is the newer minimal design, WireGuard, which replaces most negotiation with a small fixed set of choices.

Providers also advertise proprietary protocols under brand names. Some are genuinely new work; many are WireGuard with a provider-specific layer wrapped around it for address assignment or traffic handling. When a name is unfamiliar, the question worth asking is which family it belongs to, since that determines its transport and therefore its behaviour on a filtered network.

Two entries deserve a note. “Automatic” means the client tries something and falls back, which is convenient and leaves you unable to say afterwards what ran. A TCP option is a fallback for networks that pass very little, not a stronger setting.

If you want to know what is running rather than what is selected, your own machine’s interface and route listings answer it, and they are more reliable than the app’s label.

The boundary in the other direction

Past the exit server, none of these layers exist at all.

The protocol cannot conceal from your network that you opened a persistent connection to some particular address. It can make the contents unreadable, but a connection existed and something carried it.

It cannot protect traffic after the exit, which continues to its destination under whatever protection that destination provides — so hostnames announced in ordinary encrypted connections are exposed again on the far side.

And it cannot change what the operator records. Key agreement happens between your device and their server, and their server is a party to it rather than a neutral pipe.

Using the layer model when you choose

Decide the exit and the filter rules first, because those are the layers that determine what is exposed.

Who operates the exit server decides who is in your internet provider’s old position. Whether a packet filter blocks non-tunnel paths decides what happens when the tunnel stops. Neither is a protocol question, and both are settled before the dropdown becomes interesting.

Then treat the protocol choice as a behaviour question about your particular devices and networks — the two comparisons linked at the top take that up, one on design and one on roaming. Verify the result on your own machine afterwards, because every layer above the protocol is one the label never described.