free5GC ULCL on Kubernetes

Complete implementation guide: from bare-metal kubeadm cluster to verified ULCL traffic steering between PSA-UPFs with 8 UEs.

Overview

This guide documents the full process of deploying a custom-patched free5GC v3.3.0 ULCL (Uplink Classifier) 5G core network on a 4-node kubeadm Kubernetes cluster, achieving bidirectional data-plane connectivity for 8 UEs, and implementing dynamic flow steering between two PSA-UPFs (anchor UPFs). The deployment replicates a working bare-metal testbed in a Kubernetes environment for the MARE project's autonomous self-healing research.

The work involved building 8 custom-patched NF Docker images from bare-metal binaries, debugging the full authentication chain, upgrading the gtp5g kernel module, resolving macvlan multi-route issues, and patching the SMF Go source code to enable uplink traffic steering. The result is a fully functional ULCL testbed where traffic can be dynamically steered between anchor UPFs via a REST API.

Topology & IP Addressing

Cluster Nodes

All nodes sit on a flat 10.160.101.0/24 subnet (Proxmox VMs, interface ens18). Macvlan secondary interfaces provide the 5G N3/N4/N6/N9 networks.

UPF Macvlan IP Assignments (Final)

Control plane: AMF N2 = .100, SMF N4 = .101. Slice: SST=1, SD=010203, DNN=internet.

ULCL Data Path

Reference: Bare-Metal Testbed

The K8s deployment mirrors a working bare-metal testbed on .137 (free5gc-core) with I-UPF at .143, PSA-UPF at .148, PSA-UPF2 at .198, gNB at .120, UEs at .121, and DN at .145. The bare-metal Go source tree at ~/free5gc/ on .137 contains all patched NF binaries.

Phase 1: Kubernetes Cluster Setup

kubeadm Installation

A standard 4-node kubeadm cluster (v1.36) on Ubuntu 20.04 VMs. On all 4 nodes:

# Disable swap, load modules, set sysctl
sudo swapoff -a
sudo sed -i '/swap/d' /etc/fstab
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
sudo modprobe overlay && sudo modprobe br_netfilter
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
EOF
sudo sysctl --system

# Install containerd + kubeadm/kubelet/kubectl
sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates curl containerd
sudo mkdir -p /etc/containerd
containerd config default | sudo tee /etc/containerd/config.toml
# Set SystemdCgroup = true in config.toml
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl restart containerd

# Add K8s repo and install
curl -fsSL https://pkgs.k8s.io/core:/stable:/v1.36/deb/Release.key | \
  sudo gpg --dearmor -o /etc/apt/keyrings/kubernetes-apt-keyring.gpg
echo 'deb [signed-by=/etc/apt/keyrings/kubernetes-apt-keyring.gpg] \
  https://pkgs.k8s.io/core:/stable:/v1.36/deb/ /' | \
  sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo apt-mark hold kubelet kubeadm kubectl

On the control-plane node (.91):

sudo kubeadm init --pod-network-cidr=10.244.0.0/16
mkdir -p $HOME/.kube
sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

On each worker (.92, .93, .94), join with the token from kubeadm init:

sudo kubeadm join 10.160.101.91:6443 --token <token> --discovery-token-ca-cert-hash sha256:<hash>

Flannel + Multus CNI

Flannel for pod networking, Multus for macvlan secondary interfaces (required for UPF data-plane):

# Flannel
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

# Multus
kubectl apply -f https://raw.githubusercontent.com/k8snetworkplumbingwg/multus-cni/master/deployments/multus-daemonset.yml

Network Attachment Definitions (NADs) for macvlan on ens18:

# Example NAD for N3 network
apiVersion: k8s.cni.cncf.io/v1
kind: NetworkAttachmentDefinition
metadata:
  name: n3-net
  namespace: free5gc
spec:
  config: |
    {
      "cniVersion": "0.3.1",
      "type": "macvlan",
      "master": "ens18",
      "mode": "bridge",
      "ipam": {
        "type": "static"
      }
    }

Similar NADs created for n4-net, n6-net, n9-net.

Local Container Registry

A local registry on .91 (NodePort 32000) for pushing custom images:

# Deploy registry as a K8s pod + NodePort service
kubectl apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: registry
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels: {app: registry}
  template:
    metadata:
      labels: {app: registry}
    spec:
      nodeSelector:
        kubernetes.io/hostname: free5gc-k8s
      containers:
      - name: registry
        image: registry:2
        ports:
        - containerPort: 5000
        volumeMounts:
        - {name: data, mountPath: /var/lib/registry}
      volumes:
      - name: data
        hostPath: {path: /opt/registry, type: DirectoryOrCreate}
---
apiVersion: v1
kind: Service
metadata:
  name: registry
  namespace: default
spec:
  type: NodePort
  selector: {app: registry}
  ports:
  - port: 5000
    targetPort: 5000
    nodePort: 32000
EOF

# Configure containerd on ALL nodes (.91–.94) to trust the insecure registry
for n in 91 92 93 94; do
  ssh localadmin@10.160.101.$n "sudo mkdir -p /etc/containerd/certs.d/10.160.101.91:32000 && \
    printf 'server = \"http://10.160.101.91:32000\"\n\n[host.\"http://10.160.101.91:32000\"]\n  capabilities = [\"pull\", \"resolve\"]\n  skip_verify = true\n' \
    | sudo tee /etc/containerd/certs.d/10.160.101.91:32000/hosts.toml >/dev/null && \
    sudo sed -i 's#config_path = \"\"#config_path = \"/etc/containerd/certs.d\"#g' /etc/containerd/config.toml && \
    sudo systemctl restart containerd"
done

# Verify
curl -s http://10.160.101.91:32000/v2/_catalog

Helm Chart Deployment

Deploy free5GC using the towards5gs-helm chart with ULCL topology enabled:

kubectl create ns free5gc
cd ~/towards5gs-helm/charts/free5gc
helm -n free5gc install free5gc-v1 . \
  --set global.ulcl.enabled=true

After deployment, all NF pods run but UE PDU sessions fail with stock v3.3.0 images — this is expected and resolved in Phase 2.

Phase 2: Custom Patched NF Images

Why Custom Images Are Required

The stock free5GC v3.3.0 images from the Helm chart have multiple bugs that prevent UE registration and PDU session establishment. The bare-metal testbed runs patched binaries that fix:

• AMF: NGAP decode bypass, AUSF _links format fix, NRF discovery query fixes
• SMF: BPManager FindULCL nil pointer guard, SendPfcpSessionDeletion nil check, charging URR bounds check, custom steering endpoint
• AUSF/UDM/UDR: API version v2 compatibility (nudr-dr)
• UPF: gtp5g 0.9.14 compatibility

Building Images with Buildah

Strategy: copy pre-built patched binaries from bare-metal (.137) and package them in minimal ubuntu:20.04 containers using buildah on .91.

# Install buildah on .91 (Ubuntu 20.04 — needs Kubic repo)
. /etc/os-release
echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/ /" \
  | sudo tee /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list
curl -fsSL "https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_${VERSION_ID}/Release.key" \
  | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/libcontainers.gpg
sudo apt-get update && sudo apt-get install -y buildah

# Copy binaries + certs from bare-metal to .91
# On bare-metal (.137):
cd ~/free5gc
tar czf /tmp/free5gc-patched.tgz bin/amf bin/smf bin/ausf bin/udm bin/udr bin/nssf bin/pcf bin/upf cert/
scp /tmp/free5gc-patched.tgz localadmin@10.160.101.91:/tmp/

# On .91:
mkdir -p ~/imgbuild && cd ~/imgbuild
tar xzf /tmp/free5gc-patched.tgz

Build pattern for each NF

cd ~/imgbuild
REG=10.160.101.91:32000

# Build one NF image (repeat for each)
nf=smf   # change to: amf, ausf, udm, udr, nssf, pcf, upf
c=$(sudo buildah --storage-driver vfs from ubuntu:20.04)
sudo buildah --storage-driver vfs run $c -- bash -c \
  "apt-get update && apt-get install -y libsctp1 iproute2 iptables ca-certificates && rm -rf /var/lib/apt/lists/*"
sudo buildah --storage-driver vfs run $c -- mkdir -p /free5gc/bin /free5gc/config /free5gc/cert
sudo buildah --storage-driver vfs copy $c bin/$nf /free5gc/bin/$nf
sudo buildah --storage-driver vfs copy $c cert/   /free5gc/cert/
sudo buildah --storage-driver vfs run  $c -- chmod +x /free5gc/bin/$nf
sudo buildah --storage-driver vfs config --workingdir /free5gc/bin $c
sudo buildah --storage-driver vfs commit $c $REG/free5gc-$nf:patched
sudo buildah --storage-driver vfs push --tls-verify=false $REG/free5gc-$nf:patched
sudo buildah --storage-driver vfs rm $c

All 8 NF Images

All 8 NFs were rebuilt as custom images and pushed to the local registry:

Point each deployment at the custom images:

# Example: update SMF deployment image
kubectl set image -n free5gc deploy/free5gc-v1-free5gc-smf-smf \
  smf=10.160.101.91:32000/free5gc-smf:patched

# Clear stale cached images on the node before restart
sudo ctr -n k8s.io images rm 10.160.101.91:32000/free5gc-smf:patched 2>/dev/null
kubectl rollout restart deployment/free5gc-v1-free5gc-smf-smf -n free5gc

Phase 3: Control Plane Fixes

Authentication Chain

The patched NFs use v2 of the nudr-dr API and a newer MongoDB schema. The auth chain was debugged link-by-link: AMF → AUSF → UDM → UDR → MongoDB.

Key config additions required for patched NFs:

AMF: retransmission timers

# In amfcfg.yaml — required by patched AMF
t3513:
  enable: true
  expireTime: 6s
  maxRetryTimes: 4
t3522:
  enable: true
  expireTime: 6s
  maxRetryTimes: 4
t3550:
  enable: true
  expireTime: 6s
  maxRetryTimes: 4
t3560:
  enable: true
  expireTime: 6s
  maxRetryTimes: 4
t3565:
  enable: true
  expireTime: 6s
  maxRetryTimes: 4

UDR: connector type

# In udrcfg.yaml — patched UDR requires this
info:
  version: 1.1.0
configuration:
  dbConnectorType: mongodb

Subscriber Data (Patched Schema)

Patched NFs expect a different MongoDB schema. Subscribers were inserted via mongosh into the MongoDB pod:

# Port-forward to MongoDB
kubectl port-forward -n free5gc svc/mongodb 27017:27017 &

# Insert subscriber for imsi-208930000000003 across 6 collections
mongosh mongodb://localhost:27017/free5gc <<'MONGO'
db.policyData.ues.amData.insertOne({
  "uesAmPolicyData":{"subscCats":["free5gc"]},
  "ueId":"imsi-208930000000003"
});

db.subscriptionData.authenticationData.authenticationSubscription.insertOne({
  "ueId":"imsi-208930000000003",
  "authenticationMethod":"5G_AKA",
  "encPermanentKey":"8baf473f2f8fd09487cccbd7097c6862",
  "authenticationManagementField":"8000",
  "encOpcKey":"8e27b6af0e692e750f32667a3b14605d",
  "sequenceNumber":{"sqn":"000000000000","lastIndexes":{"ausf":0}}
});

// ... plus amData, smfSelectionSubscriptionData,
// smData, provisionedData entries
MONGO

The diagnostic that cracked it: diff the working doc against the failing one

The UDR's subscription sm-data lookup returned 200 while its policy sm-data lookup returned 404 for the same UE. That proves Mongo connectivity, DB name, and the snssai key are all fine — so compare the two documents side by side:

kubectl exec -n free5gc mongodb-0 -- mongosh free5gc --quiet --eval '
print("=== subscription smData (returns 200) ===");
printjson(db.getCollection("subscriptionData.provisionedData.smData").findOne({ueId:"imsi-208930000000001"}));
print("=== policy smData (returns 404) ===");
printjson(db.getCollection("policyData.ues.smData").findOne({ueId:"imsi-208930000000001"}));
'

The working subscription doc has its fields at the top level (ueId, singleNssai, dnnConfigurations). The failing policy doc had its data buried under a smPolicyData wrapper. The UDR unmarshals the bson document straight into its SmPolicyData struct, whose field is smPolicySnssaiData at the top level — the wrapper put it one level too deep, so the UDR found nothing and returned 404.

The fix (top-level smPolicySnssaiData, no wrapper)

kubectl exec -n free5gc mongodb-0 -- mongosh free5gc --quiet --eval '
var IMSI=["imsi-208930000000001","imsi-208930000000002","imsi-208930000000003","imsi-208930000000004",
          "imsi-208930000000005","imsi-208930000000006","imsi-208930000000007","imsi-208930000000008"];
for (var i=0;i<IMSI.length;i++){
  db.getCollection("policyData.ues.smData").replaceOne(
    {ueId:IMSI[i]},
    { ueId:IMSI[i],
      smPolicySnssaiData:{ "01010203":{          // top-level, NO smPolicyData wrapper
        snssai:{sst:1,sd:"010203"},
        smPolicyDnnData:{ internet:{dnn:"internet"} } } } },
    {upsert:true});
}
var d=db.getCollection("policyData.ues.smData").findOne({ueId:IMSI[0]});
if (d.smPolicyData) print("WRONG: wrapper still present");
else print("OK: smPolicySnssaiData is top-level");
'

No NF restart is needed — the UDR reads Mongo fresh on each query. Re-attach the UEs and watch the policy lookup flip to 200:

UDR=$(kubectl get pods -n free5gc -o name | grep udr | cut -d/ -f2)
kubectl logs -n free5gc -c udr $UDR -f | grep 'policy-data.*sm-data'   # expect 200

For 8 UEs, the base subscriber (imsi-208930000000003) was cloned to 001–008:

# Clone subscriber to all 8 IMSIs
mongosh mongodb://localhost:27017/free5gc <<'MONGO'
for (let i = 1; i <= 8; i++) {
  let imsi = "imsi-20893000000000" + i;
  let src = db.subscriptionData.authenticationData.authenticationSubscription
    .findOne({"ueId":"imsi-208930000000003"});
  delete src._id;
  src.ueId = imsi;
  db.subscriptionData.authenticationData.authenticationSubscription
    .replaceOne({"ueId":imsi}, src, {upsert:true});
  // repeat for each collection...
}
MONGO

ULCL Configuration Flag

The ulcl: true flag in smfcfg.yaml is critical — without it, the ULCL path builder doesn't engage and the SMF logs "No default data path".

# In smfcfg.yaml under userplaneInformation:
userplaneInformation:
  ulcl: true
  upNodes:
    gNB1:
      type: AN
    BranchingUPF:
      type: UPF
      nodeID: 10.160.101.112
      ...

Phase 4: Data Plane

gtp5g Kernel Module Upgrade (0.8.6 → 0.9.14)

The patched UPF binary requires gtp5g 0.9.14 (stock chart installs 0.8.6). Upgrade on all 3 UPF nodes (.92, .93, .94):

# Copy gtp5g 0.9.14 source from bare-metal
# On bare-metal (.137):
cd ~/free5gc/gtp5g && tar czf /tmp/gtp5g.tgz .
scp /tmp/gtp5g.tgz localadmin@10.160.101.92:/tmp/
scp /tmp/gtp5g.tgz localadmin@10.160.101.93:/tmp/
scp /tmp/gtp5g.tgz localadmin@10.160.101.94:/tmp/

# On each UPF node (.92, .93, .94):
mkdir -p ~/gtp5g && cd ~/gtp5g && tar xzf /tmp/gtp5g.tgz

# Delete the UPF pod first (so gtp5g can be unloaded)
# On .91:
kubectl delete -n free5gc $(kubectl get pod -n free5gc -o name | grep upf-upfb)

# On the UPF node:
sudo rmmod gtp5g 2>/dev/null
cd ~/gtp5g && make clean && make && sudo make install
sudo depmod -a && sudo modprobe gtp5g
dmesg | tail -3  # should show gtp5g 0.9.14

Pitfall A — the source tree is still checked out at v0.8.6

If ~/gtp5g is at tag v0.8.6, every make rebuilds 0.8.6. Switch to a 0.9.x tag (0.9.5 ≤ v < 0.10.0; we standardised on v0.9.14 on all three nodes). The checkout aborts if the steering patch is uncommitted in src/pfcp/pdr.c — stash, switch, pop:

cd ~/gtp5g
git describe --tags                 # if v0.8.6 -> must switch
git stash                           # preserve the pdr.c steering patch
git checkout v0.9.14
git stash pop                        # may CONFLICT — see Pitfall C

Pitfall B — make install did not replace the loaded module

modprobe loads whatever .ko is installed at the modinfo path; a stale 0.8.6 there beats every rebuild. Verify with both modinfo (installed) and /sys/module (loaded), then force-overwrite the exact file:

modinfo gtp5g | grep -E 'filename|version'       # installed module + path
cat /sys/module/gtp5g/version                    # currently-loaded module

# overwrite the .ko at the path modinfo reported, then reload
sudo cp ~/gtp5g/gtp5g.ko /lib/modules/$(uname -r)/kernel/drivers/net/gtp5g.ko
sudo depmod -a
sudo rmmod gtp5g && sudo modprobe gtp5g
modinfo gtp5g | grep version    # MUST now read 0.9.14
cat /sys/module/gtp5g/version   # MUST now read 0.9.14 (not just /sys)

# persist across reboot
echo udp_tunnel | sudo tee  /etc/modules-load.d/gtp5g.conf
echo gtp5g      | sudo tee -a /etc/modules-load.d/gtp5g.conf

Pitfall C — resolving the pdr.c steering patch onto v0.9.14

v0.9.14 restructured pdr_find_by_gtp1u to use a local iph + ip_match(iph, pdr) helper, so the 0.8.6 patch (which used target_addr) will not apply cleanly. The merged result that keeps the upstream structure and the steering clause (match when the inner destination equals the UE address) is:

// src/pfcp/pdr.c, in pdr_find_by_gtp1u (~line 330)
        if (pdi->ue_addr_ipv4) {
            iph = (struct iphdr *)(skb->data + hdrlen); // inner IP header
            if ((!(pdr->af == AF_INET)) ||
                (!(ip_match(iph, pdr) || iph->daddr == pdi->ue_addr_ipv4->s_addr))) {
                continue;
            }
        }

On a node where the patch was stashed, git stash pop conflicts — resolve to the block above. On a node that is clean v0.9.14, apply the one-line change directly:

cd ~/gtp5g
cp src/pfcp/pdr.c src/pfcp/pdr.c.bak
sed -i 's/if ((!(pdr->af == AF_INET)) || (!ip_match(iph, pdr))) {/if ((!(pdr->af == AF_INET)) ||\n                (!(ip_match(iph, pdr) || iph->daddr == pdi->ue_addr_ipv4->s_addr))) {/' src/pfcp/pdr.c
grep -n 'iph->daddr == pdi->ue_addr_ipv4->s_addr' src/pfcp/pdr.c   # confirm
make clean && make
modinfo ./gtp5g.ko | grep version    # 0.9.14

Uplink: UE → DN

After gtp5g upgrade and UPF pod restart, uplink path was verified step by step:

Fix 1: AnchorUPF1 N6 egress routing

With multiple macvlan interfaces on the same /24, the default route sent N6 traffic via the wrong interface:

# Inside AnchorUPF1 pod:
ip route replace 10.160.101.145/32 dev n6
# This ensures DN-bound traffic exits via n6, not n9

Fix 2: DN server return route

The DN server (.145) is shared between bare-metal and K8s. Return routes must point at K8s anchors:

# On DN server (.145):
sudo ip route replace 10.61.0.0/16 via 10.160.101.123  # AnchorUPF1 N6
sudo ip route replace 10.62.0.0/16 via 10.160.101.133  # AnchorUPF2 N6

Fix 3: NAT removal

Bare-metal uses routed return (no MASQUERADE). K8s conntrack was 0, so NAT couldn't un-NAT replies:

# Inside each UPF pod:
iptables -t nat -F  # flush NAT rules

Downlink: DN → UE

Several issues blocked downlink GTP-U through the BranchingUPF:

Fix 4: BranchingUPF duplicate IP

N9 and N6 both had .113 — relocated N6 to .150 via deployment annotation patch.

Fix 5: BranchingUPF GTP-U binding

Set addr: 0.0.0.0 in the gtpu ifList for both N3 and N9 so the UPF listens on all macvlan interfaces for downlink GTP:

# In upfcfg.yaml for BranchingUPF:
gtpu:
  forwarder: gtp5g
  ifList:
    - addr: 0.0.0.0
      type: N3
    - addr: 0.0.0.0
      type: N9

Result: bidirectional ping

# On .95:
ping -I uesimtun0 10.160.101.145 -c 4
# 4 packets sent, 4 received, 0% loss

Multi-UE Setup (8 UEs)

# On .95 — create multi-UE config
sed "s/imsi-208930000000003/imsi-208930000000001/" \
  config/free5gc-ue.yaml > config/ue-multi.yaml

# Launch 8 UEs (keep running in foreground!)
sudo ./build/nr-ue -c config/ue-multi.yaml -n 8

# Verify all 8 tunnels
ip -br addr | grep uesimtun
# uesimtun0–7 with 10.61.0.x IPs

# All 8 ping the DN
for dev in $(ip -br addr | grep -o 'uesimtun[0-9]*' | sort -u); do
  ping -I $dev 10.160.101.145 -c 2 -W 2 &
done
wait
# All 8: 0% packet loss

Phase 5: Flow Steering

UE Routing Configuration

The uerouting.yaml configmap defines the ULCL topology and specificPaths. For the steer to work, AnchorUPF2 must have a pool, and there must be a specificPath with dest matching the traffic destination:

info:
  version: 1.0.7
  description: Routing information for UE

ueRoutingInfo:
  UE1:
    members:
    - imsi-208930000000001
    - imsi-208930000000002
    - imsi-208930000000003
    - imsi-208930000000004
    - imsi-208930000000005
    - imsi-208930000000006
    - imsi-208930000000007
    - imsi-208930000000008
    topology:
      - A: gNB1
        B: BranchingUPF
      - A: BranchingUPF
        B: AnchorUPF1
    specificPath:
      - dest: 10.160.101.145/16
        path: [BranchingUPF, AnchorUPF2]
      - dest: 10.160.101.145/16
        path: [BranchingUPF, AnchorUPF1]
      - dest: 8.8.8.8/32
        path: [BranchingUPF]

SMF Source Code Patch — Default FAR Steering

This is the critical fix that makes steering actually redirect traffic. The original HandleSteerTraffic in steering.go only modified SDF-filtered FARs on non-default paths (downlink direction). Uplink traffic (UE → DN) matched the default PDR and always went to AnchorUPF1 regardless of the steer.

Patch Add default-path uplink FAR modification to NFs/smf/internal/sbi/processor/steering.go:

// ── Insert BEFORE "if len(farList) == 0" ──

// Also redirect the DEFAULT-path uplink FAR on the I-UPF
if defaultPath.FirstDPNode != nil &&
    defaultPath.FirstDPNode.UpLinkTunnel != nil &&
    defaultPath.FirstDPNode.UpLinkTunnel.PDR != nil {
    defFAR := defaultPath.FirstDPNode.UpLinkTunnel.PDR.FAR
    if defFAR != nil && defFAR.ForwardingParameters != nil {
        existingTEID := uint32(0)
        if defFAR.ForwardingParameters.OuterHeaderCreation != nil {
            existingTEID = defFAR.ForwardingParameters.OuterHeaderCreation.Teid
        }
        defFAR.ForwardingParameters.OuterHeaderCreation = &pfcpType.OuterHeaderCreation{
            OuterHeaderCreationDescription: pfcpType.OuterHeaderCreationGtpUUdpIpv4,
            Ipv4Address:                    targetIP,
            Teid:                           existingTEID,
        }
        defFAR.State = smf_context.RULE_UPDATE
        pdrList = append(pdrList, defaultPath.FirstDPNode.UpLinkTunnel.PDR)
        farList = append(farList, defFAR)
        debugFARs += fmt.Sprintf("FARID=%d(default),UPF=%s;",
            defFAR.FARID,
            defaultPath.FirstDPNode.UPF.NodeID.ResolveNodeIdToIp().String())
        if iUPFNode == nil {
            iUPFNode = defaultPath.FirstDPNode
        }
    }
}

Also remove the early return on empty ulcfNodes (since the default-path FAR provides a valid FAR even without SDF nodes).

Build and deploy the patched SMF

# Build on .91 (Go 1.25.5 is installed)
cd ~/free5gc/NFs/smf
go build -o /tmp/smf-fixed ./cmd/
ls -la /tmp/smf-fixed  # 32442720 bytes (larger than original 32438576)

# Rebuild the SMF image
cd ~/imgbuild && cp /tmp/smf-fixed bin/smf
REG=10.160.101.91:32000
c=$(sudo buildah --storage-driver vfs from ubuntu:20.04)
sudo buildah --storage-driver vfs run $c -- bash -c \
  "apt-get update && apt-get install -y libsctp1 iproute2 iptables ca-certificates && rm -rf /var/lib/apt/lists/*"
sudo buildah --storage-driver vfs run $c -- mkdir -p /free5gc/bin /free5gc/config /free5gc/cert
sudo buildah --storage-driver vfs copy $c bin/smf /free5gc/bin/smf
sudo buildah --storage-driver vfs copy $c cert/ /free5gc/cert/
sudo buildah --storage-driver vfs run $c -- chmod +x /free5gc/bin/smf
sudo buildah --storage-driver vfs config --workingdir /free5gc/bin $c
sudo buildah --storage-driver vfs commit $c $REG/free5gc-smf:patched
sudo buildah --storage-driver vfs push --tls-verify=false $REG/free5gc-smf:patched
sudo buildah --storage-driver vfs rm $c

# Deploy
sudo ctr -n k8s.io images rm $REG/free5gc-smf:patched 2>/dev/null
kubectl rollout restart deployment/free5gc-v1-free5gc-smf-smf -n free5gc
kubectl rollout status deployment/free5gc-v1-free5gc-smf-smf -n free5gc

Steering Script

The script on .91 port-forwards to the SMF pod and calls the custom REST API for each SUPI:

# ~/steering-k8s.sh
#!/bin/bash
TARGET=$1

# Port-forward setup (run once before calling this script)
# pkill -f "port-forward.*8000"
# smf=$(kubectl get pod -n free5gc -o name | grep smf | grep -v ausf | head -1 | sed 's|pod/||')
# kubectl port-forward -n free5gc $smf 8000:80 >/tmp/pf.log 2>&1 &

SUPI_LIST=(
    "imsi-208930000000001" "imsi-208930000000002" "imsi-208930000000003"
    "imsi-208930000000004" "imsi-208930000000005" "imsi-208930000000006"
    "imsi-208930000000007" "imsi-208930000000008"
)

printf "%-25s | %-15s | %-20s\n" "SUPI" "STATUS" "FARIDs"
echo "------------------------------------------------------------"

for SUPI in "${SUPI_LIST[@]}"; do
    RESULT=$(curl -s -v -X POST http://127.0.0.1:8000/nsmf-steering/v1/steer \
        -H 'Content-Type: application/json' \
        -d "{\"supi\":\"$SUPI\",\"targetUpf\":\"$TARGET\"}" 2>&1)
    FARIDS=$(echo "$RESULT" | grep -i "x-debug-far" | grep -oP 'FARID=\K\d+' | paste -sd "," -)
    if [ -z "$FARIDS" ]; then
        printf "%-25s | %-15s | %-20s\n" "$SUPI" "FAILED" "N/A"
    else
        printf "%-25s | %-15s | %-20s\n" "$SUPI" "SUCCESS" "$FARIDS"
    fi
    sleep 0.5
done

Verification

The full end-to-end sequence — UEs must be running before steering:

# 1. Setup port-forward to SMF
pkill -f "port-forward.*8000"; sleep 1
smf=$(kubectl get pod -n free5gc -o name | grep smf | grep -v ausf | head -1 | sed 's|pod/||')
kubectl port-forward -n free5gc $smf 8000:80 >/tmp/pf.log 2>&1 &
sleep 3

# 2. Launch 8 UEs on .95 (KEEP RUNNING)
sudo killall -9 nr-ue 2>/dev/null; sleep 2
cd ~/ueransim && sudo ./build/nr-ue -c config/ue-multi.yaml -n 8
# Wait for 8x "PDU Session establishment is successful"

# 3. Start pings from .95 (second terminal)
for dev in $(ip -br addr | grep -o 'uesimtun[0-9]*' | sort -u); do
  ping -I $dev 10.160.101.145 >/dev/null 2>&1 &
done

# 4. Steer to AnchorUPF2 from .91
./steering-k8s.sh 10.160.101.131

Expected output (with the default-FAR patch)

SUPI                      | STATUS          | FARIDs
------------------------------------------------------------
imsi-208930000000001      | SUCCESS         | 27,11
imsi-208930000000002      | SUCCESS         | 31,15
imsi-208930000000003      | SUCCESS         | 19,3
...
# Two FARIDs per UE: SDF FAR + default FAR

X-Debug-FAR verification

curl -s -v -X POST http://127.0.0.1:8000/nsmf-steering/v1/steer \
  -H 'Content-Type: application/json' \
  -d '{"supi":"imsi-208930000000003","targetUpf":"10.160.101.131"}' 2>&1 | grep x-debug

# X-Debug-Far: FARID=19,UPF=10.160.101.112;FARID=3(default),UPF=10.160.101.112;
#              ^^^^^^^^ SDF FAR                ^^^^^^^^^^^^^^^ DEFAULT FAR (the fix!)

tcpdump confirms traffic moved

# On .94 (AnchorUPF2 host) — traffic appears after steering!
sudo tcpdump -ni ens18 'udp port 2152 and host 10.160.101.132' -nn

# 10.160.101.111.2152 > 10.160.101.132.2152: UDP, length 100
# 10.160.101.111.2152 > 10.160.101.132.2152: UDP, length 100
# ^^^^^^^^^^^^^ BranchingUPF         ^^^^^^^^^^^^^ AnchorUPF2 (steered!)

# Steer back to AnchorUPF1:
./steering-k8s.sh 10.160.101.144

Phase 6: Autonomous Healing Agent

The LSTM-based autonomous healing agent monitors traffic on the BranchingUPF (I-UPF), detects GTP-U flood attacks via an LSTM classifier, and steers malicious UE flows to AnchorUPF2 using the custom SMF steering API. The agent runs on .91 and uses the local port-forward to reach the SMF.

GTP-U Flood Simulator (f_k8s.py)

Sends spoofed GTP-U packets from .95 to the BranchingUPF N3 (.111), simulating a flood attack from 2 malicious UEs. The outer source IP must be the gNB IP (.95) so gtp5g accepts the packets.

Finding current TEIDs

TEIDs change on every UE relaunch. Always check before running the flood:

upfb=$(kubectl get pod -n free5gc -o name | grep upf-upfb | head -1)
kubectl exec -n free5gc $upfb -c upfb -- sh -c \
  "LD_LIBRARY_PATH=/usr/lib /tmp/gtp5g-tunnel-bin list pdr upfgtp" 2>/dev/null \
  | grep -B3 -A3 "10.61.0.7\|10.61.0.8" | grep -E "UE IPv4|In Teid"

Key configuration (top of f_k8s.py)

GNB_IP         = "10.160.101.95"         # K8s gNB (outer src)
IUPF_N3_IP     = "10.160.101.111"        # BranchingUPF N3 (outer dst)
DN_SERVER_IP   = "10.160.101.145"        # DN server (inner dst)

# ⚠ UPDATE THESE after every UE relaunch!
MALICIOUS_UES = [
    {"ip": "10.61.0.7", "teid": 0x1a},  # imsi-208930000000007
    {"ip": "10.61.0.8", "teid": 0x1e},  # imsi-208930000000008
]

Each malicious UE uses its own TEID (the bare-metal version cycled source IPs with a single TEID — incorrect for gtp5g PDR matching).

Running the flood

# On .95
sudo python3 f_k8s.py --pps 5000 --duration 60
sudo python3 f_k8s.py --pps 10000 --duration 30 --cores 2

LSTM Healing Agent (agent_kpi_k8s.py)

Adapted from the bare-metal agent_kpi.py for the K8s topology. Key differences:

• Steering is local — calls curl against localhost:8000 (port-forward), no SSH to .137
• Auto port-forward recovery — detects dead port-forwards and restarts them automatically
• I-UPF monitoring — SSH to .92 host, tcpdump on ens18 (macvlan traffic visible on host)
• Traffic confirmation — SSH to .94, tcpdump for GTP-U on .132

Key configuration (top of agent_kpi_k8s.py)

I_UPF_IP        = "10.160.101.92"         # BranchingUPF host
STEER_HOST      = "127.0.0.1"             # local port-forward
STEER_PORT      = 8000
MONITOR_UPF_N4  = "10.160.101.144"        # AnchorUPF1 N4
FAILOVER_UPF_N4 = "10.160.101.131"        # AnchorUPF2 N4
FAILOVER_HOST   = "10.160.101.94"         # AnchorUPF2 host

SUPI_NORMAL = [
    "imsi-208930000000001",
    "imsi-208930000000002",
    "imsi-208930000000003",
    "imsi-208930000000004",
    "imsi-208930000000005",
    "imsi-208930000000006",
]

SUPI_MALICIOUS = [
    "imsi-208930000000007",  # attacker UE
    "imsi-208930000000008",  # attacker UE
]

Agent loop

The agent polls every ~4 seconds, running two parallel probes:

1. Ping probe — SSH to I-UPF host, ping the AnchorUPF1 host to measure RTT and loss
2. Flood probe — SSH to I-UPF host, tcpdump on ens18 for GTP-U packets, extract 11 features

Features are scaled and fed to the LSTM classifier (5 classes, CLASS_GTP_FLOOD=4). If GTP_FLOOD is detected for CONFIRM_SECS (4s), the agent steers the malicious UE SUPIs to AnchorUPF2.

Prerequisites on .91

# Install Python dependencies
pip3 install torch joblib numpy scikit-learn

# Copy model assets to ~/agent/model/
ls ~/agent/model/
# lstm_attack_classifier.pth  scaler.pkl  label_encoder.pkl

# SSH key access (passwordless) from .91 to .92 and .94
ssh-copy-id localadmin@10.160.101.92
ssh-copy-id localadmin@10.160.101.94

SMF Rebuild Script (rebuild-smf.sh)

After any SMF source change (e.g., steering patch), this script rebuilds the image and deploys it. Critical because the K8s image cache aggressively holds stale layers — the script uses ctr images rm to force a fresh pull.

#!/bin/bash
set -e
REG=10.160.101.91:32000

echo "=== Copying patched binary ==="
cd ~/imgbuild
cp /tmp/smf-fixed bin/smf
ls -la bin/smf

echo "=== Building image ==="
c=$(sudo buildah --storage-driver vfs from ubuntu:20.04)
sudo buildah --storage-driver vfs run $c -- bash -c \
  "apt-get update && apt-get install -y libsctp1 iproute2 iptables ca-certificates && rm -rf /var/lib/apt/lists/*"
sudo buildah --storage-driver vfs run $c -- mkdir -p /free5gc/bin /free5gc/config /free5gc/cert
sudo buildah --storage-driver vfs copy $c bin/smf /free5gc/bin/smf
sudo buildah --storage-driver vfs copy $c cert/ /free5gc/cert/
sudo buildah --storage-driver vfs run $c -- chmod +x /free5gc/bin/smf
sudo buildah --storage-driver vfs config --workingdir /free5gc/bin $c
sudo buildah --storage-driver vfs commit $c $REG/free5gc-smf:patched
sudo buildah --storage-driver vfs push --tls-verify=false $REG/free5gc-smf:patched
sudo buildah --storage-driver vfs rm $c

echo "=== Clearing cache + restarting SMF ==="
sudo ctr -n k8s.io images rm $REG/free5gc-smf:patched 2>/dev/null
kubectl rollout restart deployment/free5gc-v1-free5gc-smf-smf -n free5gc
kubectl rollout status deployment/free5gc-v1-free5gc-smf-smf -n free5gc

echo "=== Restarting port-forward ==="
pkill -f "port-forward.*8000" 2>/dev/null; sleep 1
smf=$(kubectl get pod -n free5gc -o name | grep smf | grep -v ausf | head -1 | sed 's|pod/||')
kubectl port-forward -n free5gc $smf 8000:80 >/tmp/pf.log 2>&1 &
sleep 3

echo "=== Verifying binary ==="
kubectl exec -n free5gc $smf -c smf -- ls -la /free5gc/bin/smf
echo "MUST show 32442720 bytes."

Complete Demo Sequence

The full end-to-end demonstration, in order. Requires 4 terminals.

1 Ensure patched SMF is running (.91)

# Verify patched binary (32442720 bytes)
smf=$(kubectl get pod -n free5gc -o name | grep smf | grep -v ausf | head -1)
kubectl exec -n free5gc $smf -c smf -- ls -la /free5gc/bin/smf

# If wrong size, rebuild:
cd ~/free5gc/NFs/smf && go build -o /tmp/smf-fixed ./cmd/
~/rebuild-smf.sh

2 Launch 8 UEs (.95 — Terminal 1, keep running)

sudo killall -9 nr-ue 2>/dev/null; sleep 2
cd ~/ueransim && sudo ./build/nr-ue -c config/ue-multi.yaml -n 8
# Wait for 8x "PDU Session establishment is successful"

3 Start legitimate pings (.95 — Terminal 2)

# Ping from first 6 UEs (normal traffic)
for i in $(seq 0 5); do
  ping -I uesimtun$i 10.160.101.145 >/dev/null 2>&1 &
done
echo "6 pings running"

4 Update flood TEIDs and start flood (.95 — Terminal 3)

# First get current TEIDs from .91:
# upfb=$(kubectl get pod -n free5gc -o name | grep upf-upfb | head -1)
# kubectl exec -n free5gc $upfb -c upfb -- sh -c \
#   "LD_LIBRARY_PATH=/usr/lib /tmp/gtp5g-tunnel-bin list pdr upfgtp" 2>/dev/null \
#   | grep -B3 -A3 "10.61.0.7\|10.61.0.8" | grep -E "UE IPv4|In Teid"

# Update MALICIOUS_UES in f_k8s.py with new TEIDs, then:
sudo python3 f_k8s.py --pps 5000 --duration 60

5 Run the agent (.91 — Terminal 4)

cd ~/agent
python3 agent_kpi_k8s.py

The agent will:

1. Establish baseline RTT (~0.2ms)
2. Detect GTP_FLOOD when the flood starts (LSTM class=4)
3. Confirm for 4 seconds
4. Steer 2 malicious UEs to AnchorUPF2 (two FARIDs per UE: SDF + default)
5. Log KPI-5, KPI-8, KPI-10 metrics
6. Terminate

KPI Results (Verified on K8s)

Root-Cause Chain (Summary)

Every issue encountered and its resolution, in the order discovered:

Operational Notes

• kubectl only works on .91 (the control-plane node).
• Port-forward dies on every SMF restart — always restart it: pkill -f "port-forward.*8000"; smf=$(kubectl get pod -n free5gc -o name | grep smf | grep -v ausf | head -1 | sed 's|pod/||'); kubectl port-forward -n free5gc $smf 8000:80 >/tmp/pf.log 2>&1 &
• UE sessions exist only while nr-ue runs. Every SMF restart drops all sessions. Must relaunch UEs on .95 after any SMF change. Steering returns "SMContext not found" without live sessions.
• Same-tag image updates need cache clearing: sudo ctr -n k8s.io images rm <img> on the node, then delete the pod.
• Do NOT run helm upgrade — it reverts all configmaps, image overrides, and deployment annotations.
• Runtime route fixes are lost on pod restart. For persistence, add routes to the UPF wrapper.sh configmap (e.g., AnchorUPF1's /32 route via n6, DN return routes).
• gtp5g-tunnel (from libgtp5gnl) was built on .93 and copied into UPF pods for PDR/FAR inspection. Not baked into the UPF image — must re-copy after pod restarts.
• TEIDs change on every UE relaunch. Always re-check TEIDs before running the flood script. Use the PDR dump command in the f_k8s.py docstring.
• UE IPs increment if the SMF pool allocator remembers previous assignments. To reset to 10.61.0.1–8, restart the SMF first (clears the pool), then relaunch UEs.
• Agent requires SSH key access from .91 to .92 (I-UPF host for tcpdump) and .94 (AnchorUPF2 host for traffic confirmation). Set up with ssh-copy-id.
• steering.sh vs steering-k8s.sh — the bare-metal script uses 127.0.0.2:8000 (SMF runs locally on .137). The K8s script uses 127.0.0.1:8000 (port-forward on .91). Using the wrong one gives FAILED for all UEs.

Recovery Procedure

After extended operation, overnight restarts, or pod rescheduling, the testbed may need a full recovery. The symptoms and their fixes follow a predictable chain. This section provides both a quick full-restart recipe and a symptom-based troubleshooting guide.

Full Cold-Start Recovery (from scratch)

Run these steps in order on .91 unless otherwise noted. This handles every known failure mode.

Automated Recovery: cold-start.sh (updated 2026-06-15)

The cold-start.sh script on .91 performs the full bring-up in sequence. Run it as localadmin, NOT with sudo — under sudo it loses the kubeconfig and every kubectl call fails against localhost:8080.

./cold-start.sh          # NOT: sudo ./cold-start.sh

Four fixes were folded in after the 2026-06-15 session, each mapping to a failure that recurred on every cold start:

It handles: preflight → MongoDB probe fix (mongo→mongosh) → subscriber + correctly-structured policy data insert → NRF restart → ordered NF restarts → patched-SMF verification (32442720 bytes) → PFCP association check (auto-restarts SMF once if <3) → UPF NAT flush+re-add → DN return routes → port-forward on 8000:80. After it completes, launch gNB + UEs on .95.

Full cold-start.sh source

The complete script, with all four 2026-06-15 fixes baked in. Save as ~/cold-start.sh on .91, chmod +x, and run as localadmin (never sudo). The K/OPC/SQN values near the top are free5gc test defaults — replace them if your subscribers use different credentials.

#!/usr/bin/env bash
#!/bin/bash
# cold-start.sh — Full K8s ULCL testbed recovery
# Run on .91 (as localadmin, NOT sudo — sudo loses kubeconfig).
# Usage: ./cold-start.sh
#
# Auth format note: authenticationSubscription MUST use the patched-UDM format —
# encPermanentKey/encOpcKey (strings) and sequenceNumber as an OBJECT
# {sqn, lastIndexes:{ausf:0}}. The old permanentKey/opc objects with a flat-string
# sequenceNumber make the UDM throw "cannot unmarshal string into ...sequenceNumber"
# → AUSF 500 → AMF CONGESTION → all UEs rejected. Step 2 below uses the correct
# format and step 2 verifies it.
set +e

GREEN='\033[0;32m'
RED='\033[0;31m'
YELLOW='\033[1;33m'
NC='\033[0m'
step() { echo -e "\n${GREEN}=== $1 ===${NC}"; }
warn() { echo -e "${YELLOW}⚠ $1${NC}"; }

REG=10.160.101.91:32000

# ─────────────────────────────────────────────
step "1/11 — Fix MongoDB liveness probe (mongo→mongosh)"
kubectl patch statefulset mongodb -n free5gc --type='json' -p='[
  {"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/exec/command","value":["mongosh","--quiet","--disableImplicitSessions","--eval","db.adminCommand(\"ping\")"]},
  {"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/exec/command","value":["bash","-ec","mongosh --quiet --disableImplicitSessions --eval \"db.hello().isWritablePrimary || db.hello().secondary\" | grep -q true"]}
]' 2>/dev/null && echo "MongoDB probes patched (mongo→mongosh)" || echo "Probes already patched or patch failed"

echo "Waiting for MongoDB..."
for i in $(seq 1 60); do
  READY=$(kubectl get pod -n free5gc mongodb-0 -o jsonpath='{.status.containerStatuses[0].ready}' 2>/dev/null)
  if [ "$READY" = "true" ]; then
    echo "MongoDB is ready"
    break
  fi
  [ $i -eq 60 ] && warn "MongoDB not ready after 60s — continuing anyway"
  sleep 2
done

# ─────────────────────────────────────────────
step "2/11 — Insert subscriber data (patched-UDM format)"
kubectl exec -n free5gc mongodb-0 -- mongosh free5gc --eval '
var IMSI_LIST = [
  "imsi-208930000000001","imsi-208930000000002","imsi-208930000000003",
  "imsi-208930000000004","imsi-208930000000005","imsi-208930000000006",
  "imsi-208930000000007","imsi-208930000000008"
];
var KEY = "8baf473f2f8fd09487cccbd7097c6862";
var OPC = "8e27b6af0e692e750f32667a3b14605d";
for (var i = 0; i < IMSI_LIST.length; i++) {
  var ueId = IMSI_LIST[i];
  // authenticationSubscription — PATCHED-UDM format:
  //   encPermanentKey / encOpcKey are STRINGS; sequenceNumber is an OBJECT.
  //   NEVER use permanentKey/opc/milenage objects or a flat-string sequenceNumber.
  db.subscriptionData.authenticationData.authenticationSubscription.replaceOne({ueId:ueId},{ueId:ueId,authenticationMethod:"5G_AKA",encPermanentKey:KEY,authenticationManagementField:"8000",encOpcKey:OPC,sequenceNumber:{sqn:"000000000100",lastIndexes:{ausf:0}}},{upsert:true});
  db.subscriptionData.provisionedData.amData.replaceOne({ueId:ueId,servingPlmnId:"20893"},{ueId:ueId,servingPlmnId:"20893",gpsiList:["msisdn-090000000"+ueId.slice(-1)],subscribedUeAmbr:{uplink:"1 Gbps",downlink:"2 Gbps"},nssai:{defaultSingleNssais:[{sst:1,sd:"010203"}]}},{upsert:true});
  db.subscriptionData.provisionedData.smfSelectionSubscriptionData.replaceOne({ueId:ueId,servingPlmnId:"20893"},{ueId:ueId,servingPlmnId:"20893",subscribedSnssaiInfos:{"01010203":{dnnInfos:[{dnn:"internet"}]}}},{upsert:true});
  db.subscriptionData.provisionedData.smData.replaceOne({ueId:ueId,servingPlmnId:"20893"},{ueId:ueId,servingPlmnId:"20893",singleNssai:{sst:1,sd:"010203"},dnnConfigurations:{internet:{pduSessionTypes:{defaultSessionType:"IPV4",allowedSessionTypes:["IPV4"]},sscModes:{defaultSscMode:"SSC_MODE_1",allowedSscModes:["SSC_MODE_1","SSC_MODE_2","SSC_MODE_3"]},"5gQosProfile":{"5qi":9,arp:{priorityLevel:8,preemptCap:"",preemptVuln:""},priorityLevel:8},sessionAmbr:{uplink:"200 Mbps",downlink:"100 Mbps"}}}},{upsert:true});
  db.policyData.ues.amData.replaceOne({ueId:ueId},{ueId:ueId,subscCats:["free5gc"]},{upsert:true});
  db.policyData.ues.smData.replaceOne({ueId:ueId},{ueId:ueId,smPolicySnssaiData:{"01010203":{snssai:{sst:1,sd:"010203"},smPolicyDnnData:{internet:{dnn:"internet"}}}}},{upsert:true});
  db.policyData.ues.flowRule.replaceOne({ueId:ueId},{ueId:ueId,flowRules:[]},{upsert:true});
  db.policyData.ues.qosFlow.replaceOne({ueId:ueId},{ueId:ueId,qosFlows:[]},{upsert:true});
}
print("Auth: " + db.subscriptionData.authenticationData.authenticationSubscription.countDocuments());
print("SM data: " + db.subscriptionData.provisionedData.smData.countDocuments());
print("AM policy: " + db.policyData.ues.amData.countDocuments());
print("SM policy: " + db.policyData.ues.smData.countDocuments());

// ---- GUARDS: fail loud if a format regression slips in ----
var ap = db.subscriptionData.authenticationData.authenticationSubscription.findOne({ueId:IMSI_LIST[0]});
if (ap && typeof ap.sequenceNumber === "string") {
  print("WARNING: sequenceNumber is a STRING — UDM will throw unmarshal error / AUSF 500 / AMF CONGESTION!");
} else if (ap && ap.sequenceNumber && ap.sequenceNumber.sqn) {
  print("GUARD OK: sequenceNumber is object (sqn=" + ap.sequenceNumber.sqn + ")");
}
if (ap && ap.permanentKey) {
  print("WARNING: auth doc uses old permanentKey object — patched UDM expects encPermanentKey string!");
} else if (ap && ap.encPermanentKey) {
  print("GUARD OK: key format is encPermanentKey/encOpcKey (patched-UDM)");
}
var sp = db.policyData.ues.smData.findOne({ueId:IMSI_LIST[0]});
if (sp && sp.smPolicyData) {
  print("WARNING: policy smData still has smPolicyData wrapper — UDR will 404 on PDU establishment!");
} else if (sp && sp.smPolicySnssaiData) {
  print("GUARD OK: policy smData is top-level smPolicySnssaiData");
}
print("All 8 subscribers inserted (patched-UDM format, SQN=000000000100)");
'

# ─────────────────────────────────────────────
step "3/11 — Reload gtp5g on all UPF nodes"
echo "Deleting UPF pods..."
kubectl delete -n free5gc $(kubectl get pod -n free5gc -o name | grep upf) 2>/dev/null
echo "Waiting for pods to terminate..."
sleep 15

echo "Reloading gtp5g on all UPF nodes..."
for node in 92 93 94; do
  ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 localadmin@10.160.101.$node \
    "sudo rmmod gtp5g 2>/dev/null; sudo modprobe gtp5g" 2>/dev/null \
    && echo "  .$node: gtp5g reloaded ✓" || warn ".$node: gtp5g reload failed (may need password)"
done

echo "Waiting for UPF pods to restart on fresh gtp5g..."
for i in $(seq 1 60); do
  UPFB=$(kubectl get pod -n free5gc -l app=free5gc-v1-free5gc-upf-upfb -o jsonpath='{.items[0].status.containerStatuses[0].ready}' 2>/dev/null)
  UPF1=$(kubectl get pod -n free5gc -l app=free5gc-v1-free5gc-upf-upf1 -o jsonpath='{.items[0].status.containerStatuses[0].ready}' 2>/dev/null)
  UPF2=$(kubectl get pod -n free5gc -l app=free5gc-v1-free5gc-upf-upf2 -o jsonpath='{.items[0].status.containerStatuses[0].ready}' 2>/dev/null)
  if [ "$UPFB" = "true" ] && [ "$UPF1" = "true" ] && [ "$UPF2" = "true" ]; then
    echo "All 3 UPF pods running"
    break
  fi
  [ $i -eq 60 ] && warn "UPF pods not ready after 120s"
  sleep 2
done

# Verify upfgtp device exists
upf1=$(kubectl get pod -n free5gc -o name | grep upf-upf1 | head -1)
kubectl exec -n free5gc $upf1 -c upf1 -- ip link show upfgtp 2>/dev/null \
  && echo "  upfgtp device confirmed ✓" || warn "upfgtp missing — UPF may need another pod restart"

# ─────────────────────────────────────────────
step "4/11 — Ensure PCF is scaled up"
PCF_REPLICAS=$(kubectl get deploy free5gc-v1-free5gc-pcf-pcf -n free5gc -o jsonpath='{.spec.replicas}' 2>/dev/null)
if [ "$PCF_REPLICAS" = "0" ]; then
  kubectl scale deploy/free5gc-v1-free5gc-pcf-pcf -n free5gc --replicas=1
  echo "PCF scaled up from 0 → 1"
  sleep 5
else
  echo "PCF already at $PCF_REPLICAS replicas"
fi

# ─────────────────────────────────────────────
step "5/11 — Restart NRF"
kubectl rollout restart deployment/free5gc-v1-free5gc-nrf-nrf -n free5gc
kubectl rollout status deployment/free5gc-v1-free5gc-nrf-nrf -n free5gc --timeout=60s
sleep 3

# ─────────────────────────────────────────────
step "6/11 — Restart all NFs (re-register with NRF)"
for nf in ausf udm udr pcf nssf amf smf; do
  dep=$(kubectl get deploy -n free5gc -o name | grep "$nf" | head -1)
  if [ -n "$dep" ]; then
    kubectl rollout restart -n free5gc $dep
    echo "  Restarted $dep"
  fi
done
echo "Waiting for rollouts..."
for nf in ausf udm udr pcf nssf amf smf; do
  dep=$(kubectl get deploy -n free5gc -o name | grep "$nf" | head -1)
  [ -n "$dep" ] && kubectl rollout status -n free5gc $dep --timeout=60s 2>/dev/null
done
echo "All NFs restarted"

# ─────────────────────────────────────────────
step "7/11 — Verify patched SMF binary"
smf=$(kubectl get pod -n free5gc -o name | grep smf | grep -v ausf | head -1)
SMF_SIZE=$(kubectl exec -n free5gc $smf -c smf -- stat -c%s /free5gc/bin/smf 2>/dev/null)
if [ "$SMF_SIZE" = "32442720" ]; then
  echo "Patched SMF binary confirmed (32442720 bytes)"
else
  warn "SMF binary is $SMF_SIZE bytes (expected 32442720)"
  warn "Run ~/rebuild-smf.sh to fix, then re-run this script"
fi

# ─────────────────────────────────────────────
step "8/11 — Verify PFCP associations"
sleep 5
kubectl logs -n free5gc $smf -c smf | grep -i "association setup accepted" | tail -3
ASSOC_COUNT=$(kubectl logs -n free5gc $smf -c smf | grep -ci "association setup accepted")
if [ "$ASSOC_COUNT" -ge 3 ]; then
  echo "All 3 PFCP associations up"
else
  warn "Only $ASSOC_COUNT PFCP associations (expected 3)"
fi

# ─────────────────────────────────────────────
step "9/11 — Apply runtime data-plane fixes"
upf1=$(kubectl get pod -n free5gc -o name | grep upf-upf1 | head -1)
upf2=$(kubectl get pod -n free5gc -o name | grep upf-upf2 | head -1)

kubectl exec -n free5gc $upf1 -c upf1 -- ip route replace 10.160.101.145/32 dev n6 2>/dev/null && echo "  AnchorUPF1: N6 route ✓" || warn "AnchorUPF1 route failed"
kubectl exec -n free5gc $upf1 -c upf1 -- iptables -t nat -F 2>/dev/null && echo "  AnchorUPF1: NAT flushed ✓" || true
kubectl exec -n free5gc $upf2 -c upf2 -- ip route replace 10.160.101.145/32 dev n6 2>/dev/null && echo "  AnchorUPF2: N6 route ✓" || warn "AnchorUPF2 route failed"
kubectl exec -n free5gc $upf2 -c upf2 -- iptables -t nat -F 2>/dev/null && echo "  AnchorUPF2: NAT flushed ✓" || true

ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 localadmin@10.160.101.145 \
  "sudo ip route replace 10.61.0.0/16 via 10.160.101.123; sudo ip route replace 10.62.0.0/16 via 10.160.101.133" 2>/dev/null \
  && echo "  DN server: return routes ✓" || warn "DN routes failed (SSH to .145 — may need password)"

# ─────────────────────────────────────────────
step "10/11 — Restart port-forward to SMF"
pkill -f "port-forward.*8000" 2>/dev/null; sleep 1
smf_name=$(kubectl get pod -n free5gc -o name | grep smf | grep -v ausf | head -1 | sed 's|pod/||')
kubectl port-forward -n free5gc $smf_name 8000:80 >/tmp/pf.log 2>&1 &
sleep 3
PF_STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://127.0.0.1:8000/ 2>/dev/null)
if [ "$PF_STATUS" = "404" ]; then
  echo "Port-forward alive (SMF reachable)"
else
  warn "Port-forward may be dead (got $PF_STATUS)"
fi

# ─────────────────────────────────────────────
step "11/11 — Summary"
echo ""
echo "All done. Now on .95:"
echo ""
echo "  sudo killall -9 nr-gnb nr-ue 2>/dev/null; sleep 2"
echo "  cd ~/ueransim"
echo "  ./build/nr-gnb -c config/free5gc-gnb.yaml &"
echo "  sleep 3"
echo "  sudo ./build/nr-ue -c config/ue-multi.yaml -n 8"
echo ""
echo "Then test:"
echo "  ping -I uesimtun0 10.160.101.145 -c 4"
echo "  ./steering-k8s.sh 10.160.101.131"
echo ""
echo "If UEs fail with CONGESTION after an AMF SCTP-flood experiment:"
echo "  kubectl rollout restart deploy/free5gc-v1-free5gc-amf-amf -n free5gc"
echo "  then relaunch gNB+UEs. If auth still fails (AUSF 500/403), re-run this script."
echo ""

Manual Recovery Steps

If you need to run individual steps or debug a specific issue, the manual procedure follows.

1 Fix MongoDB probes + reload gtp5g on UPF nodes

MongoDB 6.0 uses mongosh, not mongo. The Helm chart installs probes with the deprecated mongo CLI, causing liveness failures, pod restarts, and data loss. Fix permanently:

# Fix MongoDB probes (one-time, persists across restarts)
kubectl patch statefulset mongodb -n free5gc --type='json' -p='[
  {"op":"replace","path":"/spec/template/spec/containers/0/livenessProbe/exec/command",
   "value":["mongosh","--quiet","--disableImplicitSessions","--eval","db.adminCommand(\"ping\")"]},
  {"op":"replace","path":"/spec/template/spec/containers/0/readinessProbe/exec/command",
   "value":["bash","-ec","mongosh --quiet --disableImplicitSessions --eval \"db.hello().isWritablePrimary || db.hello().secondary\" | grep -q true"]}
]'

# Wait for MongoDB to restart with fixed probes
kubectl get pod -n free5gc -w | grep mongo
# Wait for 1/1 Running

If gtp5g is in a bad state (upfgtp RX=0, TX errors > 0), also reload it:

# Delete UPF pods first (gtp5g can't unload while in use)
kubectl delete -n free5gc $(kubectl get pod -n free5gc -o name | grep upf-upfb)
kubectl delete -n free5gc $(kubectl get pod -n free5gc -o name | grep upf-upf1)
kubectl delete -n free5gc $(kubectl get pod -n free5gc -o name | grep upf-upf2)

# Reload gtp5g on each node
for node in 92 93 94; do
  ssh localadmin@10.160.101.$node "sudo rmmod gtp5g; sudo modprobe gtp5g"
  echo "gtp5g reloaded on .$node"
done

# Wait for K8s to recreate the UPF pods
kubectl get pod -n free5gc -w | grep upf
# Wait until all 3 show Running 1/1

2 Restart SMF (re-establish PFCP associations)

After UPF pod recreation, the SMF holds stale PFCP sessions. Restart it to re-associate with all 3 UPFs:

kubectl rollout restart deployment/free5gc-v1-free5gc-smf-smf -n free5gc
kubectl rollout status deployment/free5gc-v1-free5gc-smf-smf -n free5gc

# Verify 3 PFCP associations (.112, .144, .131)
smf=$(kubectl get pod -n free5gc -o name | grep smf | grep -v ausf | head -1)
kubectl logs -n free5gc $smf -c smf | grep -i "association setup accepted"

3 Verify patched SMF binary

smf=$(kubectl get pod -n free5gc -o name | grep smf | grep -v ausf | head -1)
kubectl exec -n free5gc $smf -c smf -- ls -la /free5gc/bin/smf
# MUST show 32442720 bytes
# If 32438576 → run ~/rebuild-smf.sh and repeat step 2

4 Reapply runtime data-plane fixes

These are lost on every pod restart:

# AnchorUPF1 — N6 route to DN + flush stale NAT
upf1=$(kubectl get pod -n free5gc -o name | grep upf-upf1 | head -1)
kubectl exec -n free5gc $upf1 -c upf1 -- ip route replace 10.160.101.145/32 dev n6
kubectl exec -n free5gc $upf1 -c upf1 -- iptables -t nat -F

# AnchorUPF2 — N6 route to DN + flush stale NAT
upf2=$(kubectl get pod -n free5gc -o name | grep upf-upf2 | head -1)
kubectl exec -n free5gc $upf2 -c upf2 -- ip route replace 10.160.101.145/32 dev n6
kubectl exec -n free5gc $upf2 -c upf2 -- iptables -t nat -F

# DN server return routes (check if already present)
ssh localadmin@10.160.101.145 "sudo ip route replace 10.61.0.0/16 via 10.160.101.123; \
  sudo ip route replace 10.62.0.0/16 via 10.160.101.133; \
  ip route | grep '10.6'"

5 Restart port-forward to SMF

pkill -f "port-forward.*8000" 2>/dev/null; sleep 1
smf=$(kubectl get pod -n free5gc -o name | grep smf | grep -v ausf | head -1 | sed 's|pod/||')
kubectl port-forward -n free5gc $smf 8000:80 >/tmp/pf.log 2>&1 &
sleep 3

# Verify
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8000/
# Should return 404 (SMF reachable)

6 Restart gNB + UEs on .95

# On .95
sudo killall -9 nr-gnb nr-ue 2>/dev/null; sleep 2
cd ~/ueransim
./build/nr-gnb -c config/free5gc-gnb.yaml &
sleep 3
# Confirm: "NG Setup procedure is successful"
sudo ./build/nr-ue -c config/ue-multi.yaml -n 8
# Wait for 8x "PDU Session establishment is successful"

7 Verify ping + steering

# On .95
ping -I uesimtun0 10.160.101.145 -c 4
# Should show 0% loss

# On .91
./steering-k8s.sh 10.160.101.131
# Should show 8x SUCCESS with two FARIDs each

Symptom-Based Troubleshooting