Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Check all deployment configurations for relevance #547

Closed
10 of 11 tasks
PapaySail opened this issue May 4, 2024 · 10 comments · Fixed by #548 or #551 · May be fixed by #552
Closed
10 of 11 tasks

Check all deployment configurations for relevance #547

PapaySail opened this issue May 4, 2024 · 10 comments · Fixed by #548 or #551 · May be fixed by #552
Assignees
@PapaySail
Labels
deployment documentation Improvements or additions to documentation

Comments

@PapaySail
Copy link
Collaborator

PapaySail commented May 4, 2024 •
edited
Loading

Need to deploy from scratch as per instructions. See if there are any bugs and what kind of bugs.
Need to strictly follow these instructions https://github.com/edgecomllc/eupf/blob/main/docs%2Fdocs-ru_ru%2Fdeployments%2FREADME.md

Docker-compose deployments:

K8s deployments
@PapaySail PapaySail self-assigned this May 4, 2024
@PapaySail
Copy link
Collaborator Author

PapaySail commented May 4, 2024 •
edited
Loading

open5gs-compose

configure firewall `bash fw.sh` flushes rules - it can be harmful
+ Hardcoded `eth0` interface
- and it's not needed for successfull iperf test! 
# flush rules
iptables -F
iptables -X
iptables -t raw -F
iptables -t raw -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X

Hardcoded eth0 interface

iptables -t nat -A POSTROUTING -s 172.20.0.0/16 -d 0.0.0.0/0 -o eth0 -j MASQUERADE
iptables -t nat -A POSTROUTING -s 10.46.0.0/16 -o eth0 -j MASQUERADE

for i in $(seq 5201 5208); do (iperf3 -s -p $i &) ; done

Iperf needed to be installed 

sergo@edgecom:~/edgecom24/eupf/docs/deployments/open5gs-compose$ for i in $(seq 5201 5208); do (iperf3 -s -p $i &) ; done

sergo@edgecom:~/edgecom24/eupf/docs/deployments/open5gs-compose$ Command 'iperf3' not found, but can be installed with:
sudo apt install iperf3
Command 'iperf3' not found, but can be installed with:
sudo apt install iperf3
Command 'iperf3' not found, but can be installed with:
sudo apt install iperf3
Command 'iperf3' not found, but can be installed with:
sudo apt install iperf3
Command 'iperf3' not found, but can be installed with:
sudo apt install iperf3
Command 'iperf3' not found, but can be installed with:
sudo apt install iperf3
Command 'iperf3' not found, but can be installed with:
sudo apt install iperf3
Command 'iperf3' not found, but can be installed with:
sudo apt install iperf3

empty env GNB_IP, UE2 can't start

Because of project name is open5gs-compose, env in docker.yaml should be
GNB_HOSTNAME: open5gs-compose-gnb-1

sergo@edgecom:~/edgecom24/eupf/docs/deployments/open5gs-compose$ sudo docker-compose -f docker-compose.ue2.yaml logs ue2-03
open5gs-compose-ue2-03-1  | GNB_IP:
open5gs-compose-ue2-03-1  | Launching ue: nr-ue -c ue.yaml
open5gs-compose-ue2-03-1  | UERANSIM v3.2.6
open5gs-compose-ue2-03-1  | terminate called after throwing an instance of 'LibError'
open5gs-compose-ue2-03-1  |   what():  Bad Inet address: null Resource temporarily unavailable
open5gs-compose-ue2-03-1  | /entrypoint.sh: line 41:    14 Aborted                 (core dumped) nr-ue -c ue.yaml $@
run iperf tests unsuccessfull

because of:

TODO to run iperf test:
Needed to set to IPERF_HOST: 172.20.0.1
sudo ethtool -K br-b8765a6637ca tx off -- for interface --gateway=172.20.0.1

When fixed, iperf tested successfully 

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-30.04  sec   240 MBytes  67.0 Mbits/sec                  receiver
[  5]  29.00-30.00  sec  11.5 MBytes  96.5 Mbits/sec
[  5]  30.00-30.04  sec   324 KBytes  68.6 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-30.04  sec   237 MBytes  66.2 Mbits/sec                  receiver
[  5]  30.00-30.04  sec   450 KBytes  84.9 Mbits/sec
[  5]  30.00-30.04  sec   288 KBytes  59.1 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-30.04  sec   246 MBytes  68.6 Mbits/sec                  receiver
[  5]   0.00-30.04  sec   252 MBytes  70.3 Mbits/sec                  receiver
[  5]  30.00-30.05  sec   462 KBytes  70.0 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-30.05  sec   242 MBytes  67.5 Mbits/sec                  receiver
[  5]  30.00-30.05  sec   710 KBytes   118 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-30.05  sec   256 MBytes  71.4 Mbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5203
-----------------------------------------------------------
-----------------------------------------------------------
Server listening on 5206
-----------------------------------------------------------
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
-----------------------------------------------------------
Server listening on 5205
-----------------------------------------------------------
-----------------------------------------------------------
Server listening on 5204
-----------------------------------------------------------
-----------------------------------------------------------
Server listening on 5202
-----------------------------------------------------------

@PapaySail PapaySail added documentation Improvements or additions to documentation deployment labels May 4, 2024
@PapaySail PapaySail linked a pull request May 4, 2024 that will close this issue
fix deploy open5gs-compose #547 #548
Merged
PapaySail added a commit that referenced this issue May 7, 2024
@PapaySail
fix deploy open5gs-compose #547 (#548)
ec283f5
@PapaySail PapaySail reopened this May 7, 2024
@PapaySail
Copy link
Collaborator Author

PapaySail commented May 8, 2024 •
edited
Loading

Free5GC & ULCL

make commands

  1. make eupf to install eUPF deploy as upfb -- unsuccessfull: 
     Events:
  Type     Reason           Age   From               Message
  ----     ------           ----  ----               -------
  Normal   Scheduled        61s   default-scheduler  Successfully assigned free5gculcl/edgecomllc-eupf-6488f49c4b-sn7p7 to edgecom-test-node01
  Warning  SysctlForbidden  61s   kubelet            forbidden sysctl: "net.ipv6.conf.all.forwarding" not allowlisted
✔ Details 
sergo@edgecom-test-node01:~$ sudo kubelet --allowed-unsafe-sysctls 'net.ipv4.ip_forward,net.ipv6.conf.all.forwarding'
Flag --allowed-unsafe-sysctls has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
I0507 20:27:57.482421   37319 server.go:399] "Kubelet version" kubeletVersion="v1.24.3"
I0507 20:27:57.482516   37319 server.go:401] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""

Solution from manual Reconfiguring the kubelet on a running node

sergo@edgecom-test-node01:~$ sudo cp /etc/kubernetes/kubelet-config.yaml new-kubelet-config.yaml
$ sudo nano new-kubelet-config.yaml
sergo@edgecom-test-node01:~$ sudo kubectl -n kube-system create configmap my-node-config --from-file=new-kubelet-config.yaml --append-hash -o yaml
apiVersion: v1
data:
  new-kubelet-config.yaml: |
    apiVersion: kubelet.config.k8s.io/v1beta1
    kind: KubeletConfiguration
    nodeStatusUpdateFrequency: "10s"
    failSwapOn: True
    authentication:
      anonymous:
        enabled: false
      webhook:
        enabled: True
      x509:
        clientCAFile: /etc/kubernetes/ssl/ca.crt
    authorization:
      mode: AlwaysAllow
    staticPodPath: /etc/kubernetes/manifests
    cgroupDriver: systemd
    containerLogMaxFiles: 5
    containerLogMaxSize: 10Mi
    maxPods: 110
    podPidsLimit: -1
    address: 10.20.253.172
    readOnlyPort: 0
    healthzPort: 10248
    healthzBindAddress: 127.0.0.1
    kubeletCgroups: /systemd/system.slice
    clusterDomain: cluster.local
    protectKernelDefaults: true
    rotateCertificates: true
    clusterDNS:
    - 169.254.25.10
    kubeReserved:
      cpu: 200m
      memory: 512Mi
    resolvConf: "/run/systemd/resolve/resolv.conf"
    allowedUnsafeSysctls:
    - net.ipv4.ip_forward
    - net.ipv6.conf.all.forwarding

    eventRecordQPS: 5
    shutdownGracePeriod: 60s
    shutdownGracePeriodCriticalPods: 20s
kind: ConfigMap
metadata:
  creationTimestamp: "2024-05-08T16:06:30Z"
  name: my-node-config-662mcffb5k
  namespace: kube-system
  resourceVersion: "72162413"
  uid: 0ace9431-2b35-494c-a7db-0139df994ef8
sergo@edgecom-test-node01:~$
sergo@edgecom-test-node01:~$ sudo kubectl edit node edgecom-test-node01
node/edgecom-test-node01 edited

add the following YAML under spec:

  configSource:
    configMap:
        name: my-node-config-662mcffb5k
        namespace: kube-system
        kubeletConfigKey: kubelet
sergo@edgecom-test-node01:~$ export NODE_NAME=edgecom-test-node01
sergo@edgecom-test-node01:~$ sudo kubectl patch node ${NODE_NAME} -p "{\"spec\":{\"configSource\":{\"configMap\":{\"name\":\"${CONFIG_MAP_NAME}\",\"namespace\":\"kube-system\",\"kubeletConfigKey\":\"kubelet\"}}}}"
node/edgecom-test-node01 patched

❌ all above does not work
✔ Ok. Working method:

sudo nano /etc/kubernetes/kubelet-config.yaml 
sudo systemctl restart kubelet

  1. make upf to install Free5gc UPFs
    CrashLoopBackOff 2024-05-08T17:00:32.904269017Z [ERRO][UPF][Main] UPF Cli Run Error: open Gtp5g: version mismatch: gtp5g version(0.7.3) should be 0.8.1 <= verion < 0.9.0 , please update it

    ✔ Updated gtp5g kernel module 

    sudo su
apt-get update; apt-get install git build-essential -y; \
cd /tmp; \
git clone --depth 1 https://github.com/free5gc/gtp5g.git; \
cd gtp5g/; \
make && make install

sergo@edgecom-test-node01:/tmp/gtp5g$ sudo modprobe -r gtp5g
sergo@edgecom-test-node01:/tmp/gtp5g$ sudo lsmod | grep ^gtp
sergo@edgecom-test-node01:/tmp/gtp5g$ sudo modprobe  gtp5g
sergo@edgecom-test-node01:/tmp/gtp5g$ sudo lsmod | grep ^gtp
gtp5g                 126976  0

  1. make ueransim to install gNodeB and UE simulators.
    UE connects and disconnects
Details 

[2024-05-08 17:11:34.717] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig]                                                                                                         [2024-05-08 17:11:36.008] [nas] [debug] PDU Session Establishment Accept received                                                                                                                               [2024-05-08 17:11:36.008] [nas] [info] PDU Session establishment is successful PSI[1]
[2024-05-08 17:11:36.041] [app] [info] Connection setup for PDU session[1] is successful, TUN interface[uesimtun0, 10.1.0.1] is up.                                                                             [2024-05-08 17:11:39.012] [rrc] [debug] Signal lost for cell[2], total [0] cells in coverage                                                                                                                    [2024-05-08 17:11:39.012] [nas] [error] Radio link failure detected
[2024-05-08 17:11:39.012] [nas] [info] UE switches to state [CM-IDLE]                                                                                                                                           [2024-05-08 17:11:39.012] [nas] [info] UE switches to state [MM-REGISTERED/PS]                                                                                                                                  [2024-05-08 17:11:39.012] [nas] [info] UE switches to state [MM-REGISTERED/PLMN-SEARCH]
[2024-05-08 17:11:39.012] [nas] [error] PLMN selection failure, no cells in coverage                                                                                                                            [2024-05-08 17:11:40.139] [rrc] [debug] New signal detected for cell[3], total [1] cells in coverage                                                                                                            [2024-05-08 17:11:41.096] [nas] [error] PLMN selection failure, no cells in coverage
[2024-05-08 17:11:42.482] [rrc] [warning] Acceptable cell selection failed in [1] cells. [1] no SI, [0] reserved, [0] barred, ftai [0]                                                                          [2024-05-08 17:11:42.482] [rrc] [error] Cell selection failure, no suitable or acceptable cell found
[2024-05-08 17:11:43.298] [nas] [error] PLMN selection failure, no cells in coverage                                                                                                                            [2024-05-08 17:11:44.399] [nas] [info] UE switches to state [MM-REGISTERED/NO-CELL-AVAILABLE]                                                                                                                   [2024-05-08 17:11:49.539] [nas] [info] Selected plmn[208/93]
[2024-05-08 17:11:49.539] [rrc] [info] Selected cell plmn[208/93] tac[1] category[SUITABLE]                                                                                                                     [2024-05-08 17:11:49.539] [nas] [info] UE switches to state [MM-REGISTERED/PS]                                                                                                                                  [2024-05-08 17:11:49.539] [nas] [info] UE switches to state [MM-REGISTERED/NORMAL-SERVICE]
[2024-05-08 17:11:54.346] [rrc] [debug] Signal lost for cell[3], total [0] cells in coverage                                                                                                                    [2024-05-08 17:11:54.346] [nas] [info] UE switches to state [MM-REGISTERED/PS]                                                                                                                                  [2024-05-08 17:11:54.346] [nas] [info] UE switches to state [MM-REGISTERED/PLMN-SEARCH]
[2024-05-08 17:11:54.346] [nas] [error] PLMN selection failure, no cells in coverage                                                                                                                            [2024-05-08 17:11:55.347] [rrc] [debug] New signal detected for cell[4], total [1] cells in coverage
[2024-05-08 17:11:56.510] [nas] [error] PLMN selection failure, no cells in coverage                                                                                                                            [2024-05-08 17:11:58.712] [nas] [error] PLMN selection failure, no cells in coverage                                                                                                                            [2024-05-08 17:11:59.540] [nas] [info] UE switches to state [MM-REGISTERED/NO-CELL-AVAILABLE]
[2024-05-08 17:11:59.541] [nas] [info] Selected plmn[208/93]                                                                                                                                                    [2024-05-08 17:11:59.541] [rrc] [info] Selected cell plmn[208/93] tac[1] category[SUITABLE]                                                                                                                     [2024-05-08 17:11:59.541] [nas] [info] UE switches to state [MM-REGISTERED/PS]
[2024-05-08 17:11:59.541] [nas] [info] UE switches to state [MM-REGISTERED/NORMAL-SERVICE]
[2024-05-08 17:12:08.216] [nas] [debug] Uplink data status changed PSI[1] pending[true]
[2024-05-08 17:12:08.216] [nas] [debug] Service request required due to [IDLE-UPLINK-DATA-PENDING]
[2024-05-08 17:12:08.216] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_data]
[2024-05-08 17:12:08.216] [nas] [debug] Sending Service Request due to [IDLE-UPLINK-DATA-PENDING]
[2024-05-08 17:12:08.216] [nas] [info] UE switches to state [MM-SERVICE-REQUEST-INITIATED]
[2024-05-08 17:12:08.216] [rrc] [debug] Sending RRC Setup Request
[2024-05-08 17:12:08.553] [rrc] [debug] Signal lost for cell[4], total [0] cells in coverage
[2024-05-08 17:12:08.553] [nas] [error] Active cell change in [CM-IDLE] state while MM specific procedure is ongoing
[2024-05-08 17:12:08.553] [nas] [info] UE switches to state [MM-DEREGISTERED/PS]
[2024-05-08 17:12:08.553] [nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH]
[2024-05-08 17:12:08.553] [nas] [error] PLMN selection failure, no cells in coverage
[2024-05-08 17:12:09.742] [rrc] [debug] New signal detected for cell[5], total [1] cells in coverage
[2024-05-08 17:12:10.823] [nas] [error] PLMN selection failure, no cells in coverage

💡 The reason of "signal lost" is a fake default gateways in GnB's deployment config:

      annotations:
        k8s.v1.cni.cncf.io/networks: >-
          [ { "name": "n2network-ueransim", "interface": "n2", "ips": [
          "10.100.50.250/29" ], "gateway": [ "10.100.50.254" ] }, { "name":
          "n3network-ueransim", "interface": "n3", "ips": [ "10.100.50.236/28"
          ], "gateway": [ "10.100.50.238" ] } ]

Solution is to remove , "gateway": [ "10.100.50.nnn" ] -- TODO: find a way how to make corrections to a helm deployment.

📝 Known bug: #465

Every peer should be configured to be monitored (see gtp_peer config option)

✔ fixed in #551

OK. See the difference in the first hop of traceroute from UE:

root@ueransim-ue-7f76db59c9-nbkq5:/ueransim/build# traceroute -i uesimtun0 www.google.com -w 1 -m3
traceroute to www.google.com (74.125.131.147), 3 hops max, 60 byte packets
 1  10.233.110.134 (10.233.110.134)  2.038 ms  1.972 ms  1.958 ms
 2  188-120-253-172.kube-prometheus-stack-kube-scheduler.kube-system.svc.cluster.local (188.120.253.172)  1.949 ms  1.941 ms  1.933 ms
 3  10.0.0.1 (10.0.0.1)  1.919 ms  1.910 ms  1.887 ms
root@ueransim-ue-7f76db59c9-nbkq5:/ueransim/build# traceroute -i uesimtun0 1.1.1.1 -w 1 -m3
traceroute to 1.1.1.1 (1.1.1.1), 3 hops max, 60 byte packets
 1  10.233.110.158 (10.233.110.158)  1.277 ms  1.263 ms  1.256 ms
 2  188-120-253-172.kubernetes.default.svc.cluster.local (188.120.253.172)  1.248 ms  1.239 ms  1.217 ms
 3  10.0.0.1 (10.0.0.1)  1.208 ms  1.197 ms  1.188 ms
root@ueransim-ue-7f76db59c9-nbkq5:/ueransim/build#

@PapaySail
Copy link
Collaborator Author

PapaySail commented May 11, 2024 •
edited
Loading

✔ Open5GS & Calico BGP - successfull

✔ Open5GS & Calico BGP with Slices - successfull

Running pods list

name namespace restarts owners node qos age status
eupf-6c4768c9b4-7k45t open5gs 0 ReplicaSet (eupf-6c4768c9b4) edgecom-test-node01 Guaranteed 1715443181000 Running
eupf2-54d595fdd4-66n4x open5gs 0 ReplicaSet (eupf2-54d595fdd4) edgecom-test-node01 Guaranteed 1715443982000 Running
gnb-ueransim-gnb-585fdf8c5b-cmg4f open5gs 0 ReplicaSet (gnb-ueransim-gnb-585fdf8c5b) edgecom-test-node01 BestEffort 1715444893000 Running
open5gs-amf-6647bbf488-276sl open5gs 0 ReplicaSet (open5gs-amf-6647bbf488) edgecom-test-node01 BestEffort 1715444455000 Running
open5gs-ausf-6b6c79777b-l94fb open5gs 0 ReplicaSet (open5gs-ausf-6b6c79777b) edgecom-test-node01 BestEffort 1715444455000 Running
open5gs-bsf-584bbc4f74-pj9dl open5gs 0 ReplicaSet (open5gs-bsf-584bbc4f74) edgecom-test-node01 BestEffort 1715444455000 Running
open5gs-mongodb-6b7cfd5c8d-sj9wg open5gs 0 ReplicaSet (open5gs-mongodb-6b7cfd5c8d) edgecom-test-node01 BestEffort 1715444455000 Running
open5gs-nrf-c7dcc4c7d-qg6ll open5gs 0 ReplicaSet (open5gs-nrf-c7dcc4c7d) edgecom-test-node01 BestEffort 1715444456000 Running
open5gs-nssf-ff6c4d95b-7wz9x open5gs 0 ReplicaSet (open5gs-nssf-ff6c4d95b) edgecom-test-node01 BestEffort 1715444455000 Running
open5gs-pcf-5c4846dd6b-47xbn open5gs 2 ReplicaSet (open5gs-pcf-5c4846dd6b) edgecom-test-node01 BestEffort 1715444455000 Running
open5gs-populate-57cbd7d569-vr2pt open5gs 0 ReplicaSet (open5gs-populate-57cbd7d569) edgecom-test-node01 BestEffort 1715444455000 Running
open5gs-smf-7df749f7-2tps5 open5gs 0 ReplicaSet (open5gs-smf-7df749f7) edgecom-test-node01 BestEffort 1715444595000 Running
open5gs-smf2-6bf4d967c-sjwqg open5gs 0 ReplicaSet (open5gs-smf2-6bf4d967c) edgecom-test-node01 BestEffort 1715444801000 Running
open5gs-udm-5f8cfcd-bdmft open5gs 0 ReplicaSet (open5gs-udm-5f8cfcd) edgecom-test-node01 BestEffort 1715444455000 Running
open5gs-udr-788f54787-xxjs9 open5gs 2 ReplicaSet (open5gs-udr-788f54787) edgecom-test-node01 BestEffort 1715444455000 Running
open5gs-webui-699c9bf9d8-lwtqw open5gs 0 ReplicaSet (open5gs-webui-699c9bf9d8) edgecom-test-node01 BestEffort 1715444455000 Running
ueransim1-ueransim-ues-567d89c5dd-hc8p2 open5gs 0 ReplicaSet (ueransim1-ueransim-ues-567d89c5dd) edgecom-test-node01 BestEffort 1715444957000 Running
ueransim2-ueransim-ues-79669647f6-wzc4f open5gs 0 ReplicaSet (ueransim2-ueransim-ues-79669647f6) edgecom-test-node01 BestEffort 1715445106000 Running

@PapaySail
Copy link
Collaborator Author

✔ Open5GS & Load Balanced eUPF - successfull

Details 

PS C:\Users\SergO\Documents\GitHub\eupf\docs\deployments\open5gs-with-scaling-eupf> kubectl get po -n open5gs -l "app.kubernetes.io/name=eupf" -o wide
NAME     READY   STATUS    RESTARTS   AGE     IP               NODE                  NOMINATED NODE   READINESS GATES
eupf-0   1/1     Running   0          3m40s   10.233.110.136   edgecom-test-node01   <none>           <none>
eupf-1   1/1     Running   0          3m20s   10.233.110.132   edgecom-test-node01   <none>           <none>

SMF log:

05/13 16:35:22.646: [smf] INFO: [Added] Number of SMF-UEs is now 1 (../src/smf/context.c:1010)                                       05/13 16:35:22.646: [smf] INFO: [Added] Number of SMF-Sessions is now 1 (../src/smf/context.c:3051)
05/13 16:35:22.647: [sbi] WARNING: Try to discover [nudm-sdm] (../lib/sbi/path.c:348)
05/13 16:35:22.648: [sbi] INFO: [1bafd4d6-1146-41ef-bd74-d3c6b28ef87a] (NF-discover) NF registered (../lib/sbi/nnrf-handler.c:829)
05/13 16:35:22.648: [sbi] INFO: [1bafd4d6-1146-41ef-bd74-d3c6b28ef87a] (NF-discover) NF Profile updated (../lib/sbi/nnrf-handler.c:8505/13 16:35:22.653: [sbi] WARNING: Try to discover [npcf-smpolicycontrol] (../lib/sbi/path.c:348)                                    05/13 16:35:22.654: [sbi] INFO: [2b90c5d6-1146-41ef-9d4f-e5658e726d87] (NF-discover) NF registered (../lib/sbi/nnrf-handler.c:829)
05/13 16:35:22.654: [sbi] INFO: [2b90c5d6-1146-41ef-9d4f-e5658e726d87] (NF-discover) NF Profile updated (../lib/sbi/nnrf-handler.c:85
05/13 16:35:22.661: [smf] INFO: UE SUPI[imsi-999700000000001] DNN[internet] IPv4[10.11.0.2] IPv6[] (../src/smf/npcf-handler.c:528)   
05/13 16:35:22.663: [gtp] INFO: gtp_connect() [10.233.110.136]:2152 (../lib/gtp/path.c:60)

05/13 16:38:04.317: [smf] INFO: [Added] Number of SMF-UEs is now 2 (../src/smf/context.c:1010)
05/13 16:38:04.317: [smf] INFO: [Added] Number of SMF-Sessions is now 2 (../src/smf/context.c:3051)
05/13 16:38:04.325: [smf] INFO: UE SUPI[imsi-999700000000002] DNN[internet] IPv4[10.11.0.3] IPv6[] (../src/smf/npcf-handler.c:528)
05/13 16:38:04.327: [gtp] INFO: gtp_connect() [10.233.110.132]:2152 (../lib/gtp/path.c:60)

UE1 console:

bash-5.1# ping -I uesimtun0 10.233.110.136
PING 10.233.110.136 (10.233.110.136): 56 data bytes
64 bytes from 10.233.110.136: seq=0 ttl=64 time=0.815 ms
64 bytes from 10.233.110.136: seq=1 ttl=64 time=0.858 ms
64 bytes from 10.233.110.136: seq=2 ttl=64 time=1.915 ms
64 bytes from 10.233.110.136: seq=3 ttl=64 time=1.381 ms
64 bytes from 10.233.110.136: seq=4 ttl=64 time=1.500 ms
^C
--- 10.233.110.136 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.815/1.293/1.915 ms
bash-5.1# ping -I uesimtun0 10.233.110.132
PING 10.233.110.132 (10.233.110.132): 56 data bytes
^C
--- 10.233.110.132 ping statistics ---
15 packets transmitted, 0 packets received, 100% packet loss

bash-5.1# ip a s dev uesimtun0
4: uesimtun0: <POINTOPOINT,PROMISC,NOTRAILERS,UP,LOWER_UP> mtu 1400 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none
    inet 10.11.0.2/32 scope global uesimtun0

UE2 console:

bash-5.1# ping -I uesimtun0 10.233.110.136 -c 5
PING 10.233.110.136 (10.233.110.136): 56 data bytes

--- 10.233.110.136 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss
bash-5.1# ping -I uesimtun0 10.233.110.132 -c 5
PING 10.233.110.132 (10.233.110.132): 56 data bytes
64 bytes from 10.233.110.132: seq=0 ttl=64 time=0.939 ms
64 bytes from 10.233.110.132: seq=1 ttl=64 time=1.139 ms
64 bytes from 10.233.110.132: seq=2 ttl=64 time=1.100 ms
64 bytes from 10.233.110.132: seq=3 ttl=64 time=6.400 ms
64 bytes from 10.233.110.132: seq=4 ttl=64 time=0.976 ms

--- 10.233.110.132 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.939/2.110/6.400 ms
bash-5.1# ip a s dev uesimtun0
4: uesimtun0: <POINTOPOINT,PROMISC,NOTRAILERS,UP,LOWER_UP> mtu 1400 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none
    inet 10.11.0.3/32 scope global uesimtun0

@PapaySail
Copy link
Collaborator Author

PapaySail commented May 14, 2024 •
edited
Loading

✔ Open5GS & srsRAN & Calico BGP
Check steps

  1. exec shell in UE pod -- wrong command. Correct will be: kubectl -n srs-open5gs exec -ti statefulset/srsran-srsran5gue-zmq-gnb -- /bin/bash
  2. run ICMP test -- unsuccessfull
Details 

PS C:\Users\SergO\Documents\GitHub\eupf\docs\deployments\srsran-gnb> kubectl -n srs-open5gs exec -ti statefulset/srsran-srsran5gue-zmq-gnb -- /bin/bash
Defaulted container "gnb" out of: gnb, ue
root@srsran-srsran5gue-zmq-gnb-0:/# ping -I tun_srsue 1.1.1.1
PING 1.1.1.1 (1.1.1.1) from 10.11.0.2 tun_srsue: 56(84) bytes of data.
^C
--- 1.1.1.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2049ms

root@srsran-srsran5gue-zmq-gnb-0:/# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
3: eth0@if409: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1430 qdisc noqueue state UP group default
    link/ether 66:fc:26:53:2b:14 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.233.110.137/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::64fc:26ff:fe53:2b14/64 scope link
       valid_lft forever preferred_lft forever
4: tun_srsue: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none
    inet 10.11.0.2/24 scope global tun_srsue
       valid_lft forever preferred_lft forever
root@srsran-srsran5gue-zmq-gnb-0:/#
root@srsran-srsran5gue-zmq-gnb-0:/# ping -I tun_srsue 10.233.64.11
PING 10.233.64.11 (10.233.64.11) from 10.11.0.2 tun_srsue: 56(84) bytes of data.
^C
--- 10.233.64.11 ping statistics ---
10 packets transmitted, 0 received, 100% packet loss, time 9224ms

root@srsran-srsran5gue-zmq-gnb-0:/#

SRSran GnB log:

+ /opt/srsRAN_Project/target/bin/gnb -c gnb.yml
Warning: Scheduling priority of thread "gnb_ctrl" not changed. Cause: Not enough privileges.
Warning: Scheduling priority of thread "du_cell#0" not changed. Cause: Not enough privileges.
Warning: Scheduling priority of thread "phy_worker" not changed. Cause: Not enough privileges.
2024-05-14T18:20:30.022340 [GNB     ] [W] Could not check scaling governor. filename=/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor error="No such file or directory"
2024-05-14T18:20:30.022387 [GNB     ] [W] DRM KMS polling is enabled, which may hinder performance. You can disable it using the "srsran_performance" script
2024-05-14T18:20:30.025034 [IO-EPOLL] [E] epoll_ctl failed for fd=0
2024-05-14T18:20:30.025035 [GNB     ] [E] Couldn't register stdin handler
2024-05-14T18:20:30.213949 [zmq:tx:0:0] [I] Binding to address tcp://127.0.0.1:2000.
2024-05-14T18:20:30.289730 [zmq:rx:0:0] [I] Connecting to address tcp://127.0.0.1:2001.
2024-05-14T18:20:35.823437 [MAC     ] [W] [    66.8] UL rnti=0x4601 lcid=61 CE: Discarding MAC CE. Cause: C-RNTI is not associated with any existing UE
2024-05-14T18:22:13.317015 [GTPU    ] [W] ue=0 DL teid=0x000001: Incomplete PDU at NG-U interface: missing or invalid PDU session container. pdu_len=92 teid=0x000001
2024-05-14T18:22:14.341467 [GTPU    ] [W] ue=0 DL teid=0x000001: Incomplete PDU at NG-U interface: missing or invalid PDU session container. pdu_len=92 teid=0x000001
2024-05-14T18:22:15.351123 [GTPU    ] [W] ue=0 DL teid=0x000001: Incomplete PDU at NG-U interface: missing or invalid PDU session container. pdu_len=92 teid=0x000001
2024-05-14T18:22:18.415462 [GTPU    ] [W] ue=0 DL teid=0x000001: Incomplete PDU at NG-U interface: missing or invalid PDU session container. pdu_len=92 teid=0x000001

OK. Solution is to change image to eupf:main
🆒 fact: after restart of upf, smf processes this events and sends command to create session as it was. So, UE gets connectivity immediately.

Details: PFCP restoration

eUPF log after restart:

[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:    export GIN_MODE=release
 - using code:    gin.SetMode(gin.ReleaseMode)

[GIN-debug] GET    /metrics                  --> github.com/edgecomllc/eupf/cmd/api/rest.(*ApiHandler).InitMetricsRoute.func1.1 (4 handlers)
2024/05/31 17:41:21 INF running on :8080
2024/05/31 17:41:21 INF running on :9090
2024/05/31 17:41:22 INF Got Association Setup Request from: 10.233.110.143.

2024/05/31 17:41:22 INF
Association Setup Request:
  Node ID: 10.233.110.143
  Recovery Time: 2024-05-30 08:13:36 +0000 UTC

2024/05/31 17:41:22 INF Saving new association: &{ID:10.233.110.143 Addr:10.233.110.143 NextSessionID:1 NextSequenceID:1 Sessions:map[] HeartbeatChannel:0xc0000ae4e0 FailedHeartbeats:0 HeartbeatsActive:false Mutex:{state:0 sema:0}}
2024/05/31 17:41:22 INF Got Session Establishment Request from: 10.233.110.143.
2024/05/31 17:41:22 INF
Session Establishment Request:
  CreatePDR ID: 1
    FAR ID: 1
    QER ID: 1
    URR ID: 1
    Source Interface: 1
    UE IPv4 Address: 10.11.0.3
  CreatePDR ID: 2
    Outer Header Removal: 0
    FAR ID: 2
    QER ID: 1
    Source Interface: 0
    TEID: 5034
    Ipv4: 10.233.64.11
    Ipv6: <nil>
    UE IPv4 Address: 10.11.0.3
  CreatePDR ID: 3
    Outer Header Removal: 0
    FAR ID: 1
    QER ID: 1
    Source Interface: 3
    TEID: 8912
    Ipv4: 10.233.64.11
    Ipv6: <nil>
  CreatePDR ID: 4
    Outer Header Removal: 0
    FAR ID: 3
    Source Interface: 0
    TEID: 5034
    Ipv4: 10.233.64.11
    Ipv6: <nil>
    SDF Filter: permit out 58 from ff02::2/128 to assigned
  CreateFAR ID: 1
    Apply Action: [2 0]
    Forwarding Parameters:
      Network Instance: internet
      Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:1 IPv4Address:10.233.110.176 IPv6Address:<nil> PortNumber:0 CTag:0 STag:0}
  CreateFAR ID: 2
    Apply Action: [2 0]
    Forwarding Parameters:
      Network Instance: internet
  CreateFAR ID: 3
    Apply Action: [2 0]
    Forwarding Parameters:
      Network Instance: internet
      Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:2 IPv4Address:10.233.110.143 IPv6Address:<nil> PortNumber:0 CTag:0 STag:0}
  CreateQER ID: 1
    Gate Status DL: 0
    Gate Status UL: 0
    Max Bitrate DL: 999999
    Max Bitrate UL: 999999
    QFI: 1
  CreateURR ID: 1
    Measurement Method: 2
    Volume Threshold: &{Flags:1 TotalVolume:104857600 UplinkVolume:0 DownlinkVolume:0}
  CreateBAR ID: 1

2024/05/31 17:41:22 INF Saving FAR info to session: 1, {Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:2960058634 LocalIP:188803338 TransportLevelMarking:0}
2024/05/31 17:41:22 INF WARN: No OuterHeaderCreation
2024/05/31 17:41:22 INF Saving FAR info to session: 2, {Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:188803338 TransportLevelMarking:0}
2024/05/31 17:41:22 INF Saving FAR info to session: 3, {Action:2 OuterHeaderCreation:1 Teid:2 RemoteIP:2406410506 LocalIP:188803338 TransportLevelMarking:0}
2024/05/31 17:41:22 INF Saving QER info to session: 1, {GateStatusUL:0 GateStatusDL:0 Qfi:1 MaxBitrateUL:999999000 MaxBitrateDL:999999000 StartUL:0 StartDL:0}
2024/05/31 17:41:22 Matched groups: ["permit out 58 from ff02::2/128 to assigned" "58" "ff02::2" "128" "" "assigned" "" ""]
2024/05/31 17:41:22 INF Session Establishment Request from 10.233.110.143 accepted.

SMF log:

05/31 16:28:02.911: [smf] INFO: [Added] Number of SMF-UEs is now 1 (../src/smf/context.c:1010)
05/31 16:28:02.911: [smf] INFO: [Added] Number of SMF-Sessions is now 1 (../src/smf/context.c:3051)
05/31 16:28:02.919: [smf] INFO: UE SUPI[imsi-999700000000001] DNN[internet] IPv4[10.11.0.3] IPv6[] (../src/smf/npcf-handler.c:528)
05/31 17:17:05.321: [sbi] INFO: [da4a3af0-15b8-41ef-8e58-3d6de1144d1a] NF expired (../lib/sbi/nf-sm.c:318)
05/31 17:17:05.326: [sbi] INFO: [e9c93eb8-15b8-41ef-aacb-15e80733941f] NF expired (../lib/sbi/nf-sm.c:318)
05/31 17:17:05.374: [sbi] INFO: [34d464ce-1e5a-41ef-9bcf-bd90bf5b1752] NF expired (../lib/sbi/nf-sm.c:318)
05/31 17:41:22.721: [pfcp] ERROR: Remote PFCP restarted [3926161136<3926166081] in Heartbeat RSP (../lib/pfcp/handler.c:76)
05/31 17:41:22.721: [smf] INFO: PFCP de-associated [10.233.64.11]:8805 (../src/smf/pfcp-sm.c:201)
05/31 17:41:22.722: [pfcp] WARNING: F-TEID allocation/release not supported with peer [10.233.64.11]:8805 (../lib/pfcp/handler.c:182)
05/31 17:41:22.722: [smf] INFO: PFCP associated [10.233.64.11]:8805 (../src/smf/pfcp-sm.c:186)
05/31 17:41:22.722: [smf] INFO: UE SUPI[imsi-999700000000001] DNN[internet] IPv4[10.11.0.3] IPv6[] (../src/smf/pfcp-sm.c:442)
05/31 17:41:22.722: [smf] ERROR: PFCP restoration (../src/smf/pfcp-sm.c:197)

Fixed in #552

@PapaySail
Copy link
Collaborator Author

PapaySail commented May 16, 2024 •
edited
Loading

Free5GC & Calico BGP

  1. install eupf make upf :: problem: hardcoded "master": "eth0" in values/eupf.yaml
Pod can't create. Status: ContainerCreating

Pod description:

  Warning  FailedCreatePodSandBox  74s (x16 over 103s)  kubelet            (combined from similar events): Failed to create pod sandbox: rpc err
or: code = Unknown desc = failed to setup network for sandbox "7b793c609a914881a32db2f7e3350992cfd1162b17790a985e4e62b6f841f864": plugin type="m
ultus" name="multus-cni-network" failed (add): [free5gc24/eupf-57ccfb74bd-m4js5:n3network-free5gc-free5gc-upf]: error adding container to networ
k "n3network-free5gc-free5gc-upf": failed to lookup master "eth0": Link not found

  1. exec shell in UE pod :: error in doc : correct command is:
    kubectl -n free5gc exec -ti deployment/ueransim1-ue -- /bin/bash

✔ fixed in #551

@PapaySail PapaySail linked a pull request May 16, 2024 that will close this issue
deploy: remove fake routes from GnB's NetworkAttachmentDefinition in free5gc deployments #551
Merged
@PapaySail
Copy link
Collaborator Author

✔ Docker-compose Free5GC

docker-compose up -d 

sergo@edgecom:~/edgecom24/free5gc-compose$ sudo docker-compose up -d
[+] Running 21/21
 _ Network free5gc-compose_privnet_n6  Created                                                                                                      0.1s
 _ Network free5gc-compose_default     Created                                                                                                      0.1s
 _ Network free5gc-compose_privnet     Created                                                                                                      0.1s
 _ Container prometheus                Started                                                                                                      1.3s
 _ Container upf                       Started                                                                                                      1.6s
 _ Container grafana                   Started                                                                                                      1.7s
 _ Container eupf                      Started                                                                                                      2.6s
 _ Container nat                       Started                                                                                                      2.4s
 _ Container mongodb                   Started                                                                                                      0.9s
 _ Container ui                        Started                                                                                                      1.3s
 _ Container webui                     Started                                                                                                      2.1s
 _ Container nrf                       Started                                                                                                      2.4s
 _ Container nssf                      Started                                                                                                      3.9s
 _ Container udm                       Started                                                                                                      3.9s
 _ Container ausf                      Started                                                                                                      3.9s
 _ Container amf                       Started                                                                                                      3.4s
 _ Container pcf                       Started                                                                                                      4.1s
 _ Container smf                       Started                                                                                                      4.0s
 _ Container udr                       Started                                                                                                      3.9s
 _ Container ueransim                  Started                                                                                                      4.2s
 _ Container n3iwf                     Started                                                                                                      4.5s
sergo@edgecom:~/edgecom24/free5gc-compose$

sergo@edgecom:~/edgecom24/free5gc-compose$ sudo docker-compose exec  ueransim bash
root@1d4b478b1c91:/ueransim# ./nr-ue -c config/uecfg.yaml
UERANSIM v3.2.6
[2024-05-16 17:37:49.842] [nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH]
[2024-05-16 17:37:49.843] [rrc] [debug] New signal detected for cell[1], total [1] cells in coverage
[2024-05-16 17:37:49.846] [nas] [info] Selected plmn[208/93]
[2024-05-16 17:37:52.343] [rrc] [info] Selected cell plmn[208/93] tac[1] category[SUITABLE]
[2024-05-16 17:37:52.343] [nas] [info] UE switches to state [MM-DEREGISTERED/PS]
[2024-05-16 17:37:52.343] [nas] [info] UE switches to state [MM-DEREGISTERED/NORMAL-SERVICE]
[2024-05-16 17:37:52.343] [nas] [debug] Initial registration required due to [MM-DEREG-NORMAL-SERVICE]
[2024-05-16 17:37:52.343] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig]
[2024-05-16 17:37:52.343] [nas] [debug] Sending Initial Registration
[2024-05-16 17:37:52.343] [nas] [info] UE switches to state [MM-REGISTER-INITIATED]
[2024-05-16 17:37:52.343] [rrc] [debug] Sending RRC Setup Request
[2024-05-16 17:37:52.346] [rrc] [info] RRC connection established
[2024-05-16 17:37:52.346] [rrc] [info] UE switches to state [RRC-CONNECTED]
[2024-05-16 17:37:52.346] [nas] [info] UE switches to state [CM-CONNECTED]
[2024-05-16 17:37:52.378] [nas] [debug] Authentication Request received
[2024-05-16 17:37:52.389] [nas] [debug] Security Mode Command received
[2024-05-16 17:37:52.389] [nas] [debug] Selected integrity[2] ciphering[0]
[2024-05-16 17:37:52.440] [nas] [debug] Registration accept received
[2024-05-16 17:37:52.440] [nas] [info] UE switches to state [MM-REGISTERED/NORMAL-SERVICE]
[2024-05-16 17:37:52.440] [nas] [debug] Sending Registration Complete
[2024-05-16 17:37:52.440] [nas] [info] Initial Registration is successful
[2024-05-16 17:37:52.440] [nas] [debug] Sending PDU Session Establishment Request
[2024-05-16 17:37:52.463] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig]
[2024-05-16 17:37:52.709] [nas] [debug] PDU Session Establishment Accept received
[2024-05-16 17:37:52.709] [nas] [info] PDU Session establishment is successful PSI[1]
[2024-05-16 17:37:52.725] [app] [info] Connection setup for PDU session[1] is successful, TUN interface[uesimtun0, 10.60.0.1] is up.

### at another console
sergo@edgecom:~/edgecom24/free5gc-compose$ sudo docker-compose exec  ueransim bash
root@1d4b478b1c91:/ueransim# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: ip_vti0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
3: uesimtun0: <POINTOPOINT,PROMISC,NOTRAILERS,UP,LOWER_UP> mtu 1400 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none
    inet 10.60.0.1/32 scope global uesimtun0
       valid_lft forever preferred_lft forever
689: eth0@if690: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:0a:64:c8:10 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.100.200.16/24 brd 10.100.200.255 scope global eth0
       valid_lft forever preferred_lft forever
root@1d4b478b1c91:/ueransim#
root@1d4b478b1c91:/ueransim# ping -I uesimtun0 1.1.1.1

PING 1.1.1.1 (1.1.1.1) from 10.60.0.1 uesimtun0: 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=55 time=2.19 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=55 time=1.58 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=55 time=1.68 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=55 time=1.66 ms
64 bytes from 1.1.1.1: icmp_seq=5 ttl=55 time=1.59 ms
^C
--- 1.1.1.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4004ms
rtt min/avg/max/mdev = 1.584/1.738/2.190/0.228 ms
root@1d4b478b1c91:/ueransim#

eupf logs:
sudo docker-compose logs edgecom-upf |less

eupf  | 2024/05/16 17:37:52 Handling PFCP message from 10.100.200.15:8805
eupf  | 2024/05/16 17:37:52 Got Session Modification Request from: 10.100.200.15.
eupf  | 2024/05/16 17:37:52 Finding association for 10.100.200.15
eupf  | 2024/05/16 17:37:52 Finding session 2
eupf  | 2024/05/16 17:37:52
eupf  | Session Modification Request:
eupf  |   UpdatePDR ID: 2
eupf  |     FAR ID: 2
eupf  |     Source Interface: 1
eupf  |     UE IPv4 Address: 10.60.0.1
eupf  |   UpdateFAR ID: 2
eupf  |     Apply Action: [2]
eupf  |     Update forwarding Parameters:
eupf  |       Network Instance: internet
eupf  |       Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:1 IPv4Address:10.100.200.16 IPv6Address:<nil> PortNumber:0 CTag:0 STag:0}
eupf  | 2024/05/16 17:37:52 Updating FAR info: 2, {FarInfo:{Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:281568266 LocalIP:248013834 TransportLevelMark
ing:0} GlobalId:1}
eupf  | 2024/05/16 17:37:52 EBPF: Update FAR: internalId=1, farInfo={Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:281568266 LocalIP:248013834 Transport
LevelMarking:0}
eupf  | 2024/05/16 17:37:52 Both F-TEID IE and UE IP Address IE are missing
eupf  | 2024/05/16 17:37:57 Sent Heartbeat Request to: 10.100.200.15
eupf  | 2024/05/16 17:37:57 Received 16 bytes from 10.100.200.15:8805

✔ Free5GC with UpLink Classifier config throught three eUPFs

docker-compose -f docker-compose.yaml -f docker-compose.ulcl.yml up -d 

sergo@edgecom:~/edgecom24/free5gc-compose$ git checkout ulcl-n9upf-experimetns
Branch 'ulcl-n9upf-experimetns' set up to track remote branch 'ulcl-n9upf-experimetns' from 'origin'.
Switched to a new branch 'ulcl-n9upf-experimetns'

docker-compose -f docker-compose.yaml -f docker-compose.ulcl.yml up -d

sergo@edgecom:~/edgecom24/free5gc-compose$ sudo docker-compose ps
NAME                IMAGE                           COMMAND                  SERVICE             CREATED             STATUS              PORTS
amf                 free5gc/amf:v3.3.0              "./amf -c ./config/a_"   free5gc-amf         2 minutes ago       Up 2 minutes        8000/tcp
ausf                free5gc/ausf:v3.3.0             "./ausf -c ./config/_"   free5gc-ausf        2 minutes ago       Up 2 minutes        8000/tcp
grafana             grafana/grafana                 "/run.sh"                grafana             2 minutes ago       Up 2 minutes        0.0.0.0:3000->3000/tcp, :::3000->3000/tcp
i-eupf              ghcr.io/edgecomllc/eupf:0.5.0   "sh /app/bin/entrypo_"   edgecom-i-upf       2 minutes ago       Up 2 minutes        0.0.0.0:8082->8080/tcp, :::8082->8080/tcp, 0.0.0.0:9092->9090/tcp, :::9092->9090/tcp
mongodb             mongo                           "docker-entrypoint.s_"   db                  2 minutes ago       Up 2 minutes        27017/tcp
n3iwf               free5gc/n3iwf:v3.3.0            "sh -c './n3iwf-ipse_"   free5gc-n3iwf       2 minutes ago       Up 2 minutes
nat                 free5gc-compose-edgecom-nat     "sh /app/bin/busy-po_"   edgecom-nat         2 minutes ago       Up 2 minutes
nrf                 free5gc/nrf:v3.3.0              "./nrf -c ./config/n_"   free5gc-nrf         2 minutes ago       Up 2 minutes        8000/tcp
nssf                free5gc/nssf:v3.3.0             "./nssf -c ./config/_"   free5gc-nssf        2 minutes ago       Up 2 minutes        8000/tcp
pcf                 free5gc/pcf:v3.3.0              "./pcf -c ./config/p_"   free5gc-pcf         2 minutes ago       Up 2 minutes        8000/tcp
prometheus          prom/prometheus                 "/bin/prometheus --c_"   prometheus          2 minutes ago       Up 2 minutes        0.0.0.0:9091->9090/tcp, :::9091->9090/tcp
psa-eupf            ghcr.io/edgecomllc/eupf:0.5.0   "sh /app/bin/entrypo_"   edgecom-psa-upf     2 minutes ago       Up 2 minutes        0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp
psa-eupf-2          ghcr.io/edgecomllc/eupf:0.5.0   "sh /app/bin/entrypo_"   edgecom-psa-upf-2   2 minutes ago       Up 2 minutes
smf                 free5gc/smf:v3.3.0              "./smf -c ./config/s_"   free5gc-smf         2 minutes ago       Up 2 minutes        8000/tcp
udm                 free5gc/udm:v3.3.0              "./udm -c ./config/u_"   free5gc-udm         2 minutes ago       Up 2 minutes        8000/tcp
udr                 free5gc/udr:v3.3.0              "./udr -c ./config/u_"   free5gc-udr         2 minutes ago       Up 2 minutes        8000/tcp
ueransim            free5gc/ueransim:v3.3.0         "./nr-gnb -c ./confi_"   ueransim            2 minutes ago       Up 2 minutes
webui               free5gc/webui:v3.3.0            "./webui -c ./config_"   free5gc-webui       2 minutes ago       Up 2 minutes        0.0.0.0:5000->5000/tcp, :::5000->5000/tcp
sergo@edgecom:~/edgecom24/free5gc-compose$
sergo@edgecom:~/edgecom24/free5gc-compose$ sudo docker-compose exec ueransim bash
root@aaad991d992b:/ueransim# ./nr-ue -c config/uecfg.yaml &
[1] 24
root@aaad991d992b:/ueransim# UERANSIM v3.2.6
[2024-05-16 18:01:03.356] [nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH]
[2024-05-16 18:01:03.356] [rrc] [debug] New signal detected for cell[1], total [1] cells in coverage
[2024-05-16 18:01:03.356] [nas] [info] Selected plmn[208/93]
[2024-05-16 18:01:03.356] [rrc] [info] Selected cell plmn[208/93] tac[1] category[SUITABLE]
[2024-05-16 18:01:03.356] [nas] [info] UE switches to state [MM-DEREGISTERED/PS]
[2024-05-16 18:01:03.356] [nas] [info] UE switches to state [MM-DEREGISTERED/NORMAL-SERVICE]
[2024-05-16 18:01:03.356] [nas] [debug] Initial registration required due to [MM-DEREG-NORMAL-SERVICE]
[2024-05-16 18:01:03.357] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig]
[2024-05-16 18:01:03.357] [nas] [debug] Sending Initial Registration
[2024-05-16 18:01:03.357] [nas] [info] UE switches to state [MM-REGISTER-INITIATED]
[2024-05-16 18:01:03.357] [rrc] [debug] Sending RRC Setup Request
[2024-05-16 18:01:03.357] [rrc] [info] RRC connection established
[2024-05-16 18:01:03.357] [rrc] [info] UE switches to state [RRC-CONNECTED]
[2024-05-16 18:01:03.357] [nas] [info] UE switches to state [CM-CONNECTED]
[2024-05-16 18:01:03.387] [nas] [debug] Authentication Request received
[2024-05-16 18:01:03.398] [nas] [debug] Security Mode Command received
[2024-05-16 18:01:03.398] [nas] [debug] Selected integrity[2] ciphering[0]
[2024-05-16 18:01:03.451] [nas] [debug] Registration accept received
[2024-05-16 18:01:03.451] [nas] [info] UE switches to state [MM-REGISTERED/NORMAL-SERVICE]
[2024-05-16 18:01:03.451] [nas] [debug] Sending Registration Complete
[2024-05-16 18:01:03.451] [nas] [info] Initial Registration is successful
[2024-05-16 18:01:03.451] [nas] [debug] Sending PDU Session Establishment Request
[2024-05-16 18:01:03.451] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig]
[2024-05-16 18:01:03.720] [nas] [debug] PDU Session Establishment Accept received
[2024-05-16 18:01:03.720] [nas] [info] PDU Session establishment is successful PSI[1]
[2024-05-16 18:01:03.735] [app] [info] Connection setup for PDU session[1] is successful, TUN interface[uesimtun0, 10.62.0.1] is up.

root@aaad991d992b:/ueransim# ping -I uesimtun0 1.1.1.1
PING 1.1.1.1 (1.1.1.1) from 10.62.0.1 uesimtun0: 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=55 time=2.01 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=55 time=1.61 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=55 time=1.66 ms
^C
--- 1.1.1.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 1.608/1.759/2.007/0.176 ms
root@aaad991d992b:/ueransim# ping -I uesimtun0 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 10.62.0.1 uesimtun0: 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=58 time=16.9 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=58 time=17.0 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=58 time=17.0 ms
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.873/16.951/17.002/0.056 ms

Logs

sudo docker-compose -f docker-compose.yaml -f docker-compose.ulcl.yml logs edgecom-i-upf |less -R
look for 8.8.8.8

i-eupf  | 2024/05/16 18:01:03 INF Got Session Modification Request from: 10.100.200.17.
i-eupf  |
i-eupf  |
i-eupf  | 2024/05/16 18:01:03 INF Finding association for 10.100.200.17
i-eupf  | 2024/05/16 18:01:03 INF Finding session 2
i-eupf  | 2024/05/16 18:01:03 INF
i-eupf  | Session Modification Request:
i-eupf  |   CreatePDR ID: 3
i-eupf  |     Outer Header Removal: 0
i-eupf  |     FAR ID: 3
i-eupf  |     QER ID: 1
i-eupf  |     Source Interface: 0
i-eupf  |     TEID: 1
i-eupf  |     Ipv4: 10.100.200.14
i-eupf  |     Ipv6: <nil>
i-eupf  |     UE IPv4 Address: 10.62.0.1
i-eupf  |     SDF Filter: permit out ip from 10.62.0.1 to 8.8.8.8/32
i-eupf  |   CreatePDR ID: 4
i-eupf  |     Outer Header Removal: 0
i-eupf  |     FAR ID: 4
i-eupf  |     QER ID: 1
i-eupf  |     Source Interface: 1
i-eupf  |     TEID: 4
i-eupf  |     Ipv4: 10.100.200.14
i-eupf  |     Ipv6: <nil>
i-eupf  |   CreateFAR ID: 3
i-eupf  |     Apply Action: [2]
i-eupf  |     Forwarding Parameters:
i-eupf  |       Network Instance: internet
i-eupf  |       Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:1 IPv4Address:10.100.200.16 IPv6Address:<nil> PortNumber:0 CTag:0 STag:0}
i-eupf  |   CreateFAR ID: 4
i-eupf  |     Apply Action: [2]
i-eupf  |     Forwarding Parameters:
i-eupf  |       Network Instance: internet
i-eupf  |       Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:1 IPv4Address:10.100.200.19 IPv6Address:<nil> PortNumber:0 CTag:0 STag:0}
i-eupf  |
i-eupf  |
i-eupf  | 2024/05/16 18:01:03 INF Saving FAR info to session: 3, {Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:281568266 LocalIP:248013834 TransportLevelMarking:0}
i-eupf  | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=2, qerInfo={Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:281568266 LocalIP:248013834 TransportLevelMarking:0}
i-eupf  | 2024/05/16 18:01:03 INF Saving FAR info to session: 4, {Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:331899914 LocalIP:248013834 TransportLevelMarking:0}
i-eupf  | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=3, qerInfo={Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:331899914 LocalIP:248013834 TransportLevelMarking:0}
i-eupf  | 2024/05/16 18:01:03 INF EBPF: Put PDR Uplink: teid=1, pdrInfo={OuterHeaderRemoval:0 FarId:2 QerId:1 SdfFilter:{Protocol:1 SrcAddress:{Type:1 Ip:10.62.0.1 Mask:ffffffff} SrcPortRange:{LowerBound:0 UpperBound:65535} DstAddress:{Type:1 Ip:8.8.8.8 Mask:ffffffff} DstPortRange:{LowerBound:0 UpperBound:65535}}}
i-eupf  | 2024/05/16 18:01:03 INF EBPF: Put PDR Uplink: teid=4, pdrInfo={OuterHeaderRemoval:0 FarId:3 QerId:1 SdfFilter:<nil>}
i-eupf  | 2024/05/16 18:01:04 INF Received 16 bytes from 10.100.200.17:8805

sudo docker-compose -f docker-compose.yaml -f docker-compose.ulcl.yml logs edgecom-psa-upf |less -R

psa-eupf  | 2024/05/16 18:01:03 INF Got Session Establishment Request from: 10.100.200.17.
psa-eupf  | 2024/05/16 18:01:03 INF
psa-eupf  | Session Establishment Request:
psa-eupf  |   CreatePDR ID: 1
psa-eupf  |     Outer Header Removal: 0
psa-eupf  |     FAR ID: 1
psa-eupf  |     QER ID: 2
psa-eupf  |     QER ID: 1
psa-eupf  |     Source Interface: 0
psa-eupf  |     TEID: 1
psa-eupf  |     Ipv4: 10.100.200.15
psa-eupf  |     Ipv6: <nil>
psa-eupf  |     UE IPv4 Address: 10.62.0.1
psa-eupf  |   CreatePDR ID: 2
psa-eupf  |     FAR ID: 2
psa-eupf  |     QER ID: 2
psa-eupf  |     QER ID: 1
psa-eupf  |     Source Interface: 1
psa-eupf  |     UE IPv4 Address: 10.62.0.1
psa-eupf  |   CreateFAR ID: 1
psa-eupf  |     Apply Action: [2]
psa-eupf  |     Forwarding Parameters:
psa-eupf  |       Network Instance: internet
psa-eupf  |   CreateFAR ID: 2
psa-eupf  |     Apply Action: [2]
psa-eupf  |     Forwarding Parameters:
psa-eupf  |       Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:2 IPv4Address:10.100.200.14 IPv6Address:<nil> PortNumber:0 CTag:0 STag:0}
psa-eupf  |   CreateQER ID: 2
psa-eupf  |     Gate Status DL: 0
psa-eupf  |     Gate Status UL: 0
psa-eupf  |     QFI: 1
psa-eupf  |   CreateQER ID: 1
psa-eupf  |     Gate Status DL: 0
psa-eupf  |     Gate Status UL: 0
psa-eupf  |     Max Bitrate DL: 1000000
psa-eupf  |     Max Bitrate UL: 2000000
psa-eupf  |     QFI: 0
psa-eupf  |
psa-eupf  |
psa-eupf  | 2024/05/16 18:01:03 INF WARN: No OuterHeaderCreation
psa-eupf  | 2024/05/16 18:01:03 INF Saving FAR info to session: 1, {Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:264791050 TransportLevelMarking:0}
psa-eupf  | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=0, qerInfo={Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:264791050 TransportLevelMarking:0}
psa-eupf  | 2024/05/16 18:01:03 INF Saving FAR info to session: 2, {Action:2 OuterHeaderCreation:1 Teid:2 RemoteIP:248013834 LocalIP:264791050 TransportLevelMarking:0}
psa-eupf  | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=1, qerInfo={Action:2 OuterHeaderCreation:1 Teid:2 RemoteIP:248013834 LocalIP:264791050 TransportLevelMarking:0}
psa-eupf  | 2024/05/16 18:01:03 INF Saving QER info to session: 2, {GateStatusUL:0 GateStatusDL:0 Qfi:1 MaxBitrateUL:0 MaxBitrateDL:0 StartUL:0 StartDL:0}
psa-eupf  | 2024/05/16 18:01:03 INF EBPF: Put QER: internalId=0, qerInfo={GateStatusUL:0 GateStatusDL:0 Qfi:1 MaxBitrateUL:0 MaxBitrateDL:0 StartUL:0 StartDL:0}
psa-eupf  | 2024/05/16 18:01:03 INF Saving QER info to session: 1, {GateStatusUL:0 GateStatusDL:0 Qfi:0 MaxBitrateUL:2000000000 MaxBitrateDL:1000000000 StartUL:0 StartDL:0}
psa-eupf  | 2024/05/16 18:01:03 INF EBPF: Put QER: internalId=1, qerInfo={GateStatusUL:0 GateStatusDL:0 Qfi:0 MaxBitrateUL:2000000000 MaxBitrateDL:1000000000 StartUL:0 StartDL:0}
psa-eupf  | 2024/05/16 18:01:03 INF EBPF: Put PDR Uplink: teid=1, pdrInfo={OuterHeaderRemoval:0 FarId:0 QerId:0 SdfFilter:<nil>}
psa-eupf  | 2024/05/16 18:01:03 INF EBPF: Put PDR Downlink: ipv4=10.62.0.1, pdrInfo={OuterHeaderRemoval:0 FarId:1 QerId:0 SdfFilter:<nil>}
psa-eupf  | 2024/05/16 18:01:03 INF Session Establishment Request from 10.100.200.17 accepted.
psa-eupf  | 2024/05/16 18:01:04 INF Received 16 bytes from 10.100.200.17:8805

sudo docker-compose -f docker-compose.yaml -f docker-compose.ulcl.yml logs edgecom-psa-upf-2 |less -R

psa-eupf-2  | 2024/05/16 18:01:03 INF Handling PFCP message from 10.100.200.17:8805
psa-eupf-2  | 2024/05/16 18:01:03 INF Got Session Establishment Request from: 10.100.200.17.
psa-eupf-2  | 2024/05/16 18:01:03 INF
psa-eupf-2  | Session Establishment Request:
psa-eupf-2  |   CreatePDR ID: 1
psa-eupf-2  |     Outer Header Removal: 0
psa-eupf-2  |     FAR ID: 1
psa-eupf-2  |     QER ID: 1
psa-eupf-2  |     Source Interface: 0
psa-eupf-2  |     TEID: 1
psa-eupf-2  |     Ipv4: 10.100.200.16
psa-eupf-2  |     Ipv6: <nil>
psa-eupf-2  |     UE IPv4 Address: 10.62.0.1
psa-eupf-2  |   CreateFAR ID: 1
psa-eupf-2  |     Apply Action: [2]
psa-eupf-2  |     Forwarding Parameters:
psa-eupf-2  |       Network Instance: internet
psa-eupf-2  |   CreateQER ID: 1
psa-eupf-2  |     Gate Status DL: 0
psa-eupf-2  |     Gate Status UL: 0
psa-eupf-2  |     Max Bitrate DL: 1000000
psa-eupf-2  |     Max Bitrate UL: 2000000
psa-eupf-2  |     QFI: 0
psa-eupf-2  |
psa-eupf-2  |
psa-eupf-2  | 2024/05/16 18:01:03 INF WARN: No OuterHeaderCreation
psa-eupf-2  | 2024/05/16 18:01:03 INF Saving FAR info to session: 1, {Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:281568266 TransportLevelMarking:0}
psa-eupf-2  | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=0, qerInfo={Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:281568266 TransportLevelMarking:0}
psa-eupf-2  | 2024/05/16 18:01:03 INF Saving QER info to session: 1, {GateStatusUL:0 GateStatusDL:0 Qfi:0 MaxBitrateUL:2000000000 MaxBitrateDL:1000000000 StartUL:0 StartDL:0}
psa-eupf-2  | 2024/05/16 18:01:03 INF EBPF: Put QER: internalId=0, qerInfo={GateStatusUL:0 GateStatusDL:0 Qfi:0 MaxBitrateUL:2000000000 MaxBitrateDL:1000000000 StartUL:0 StartDL:0}
psa-eupf-2  | 2024/05/16 18:01:03 INF EBPF: Put PDR Uplink: teid=1, pdrInfo={OuterHeaderRemoval:0 FarId:0 QerId:0 SdfFilter:<nil>}
psa-eupf-2  | 2024/05/16 18:01:03 INF Session Establishment Request from 10.100.200.17 accepted.
psa-eupf-2  | 2024/05/16 18:01:03 INF Received 137 bytes from 10.100.200.17:8805
psa-eupf-2  | 2024/05/16 18:01:03 INF Handling PFCP message from 10.100.200.17:8805
psa-eupf-2  | 2024/05/16 18:01:03 INF Got Session Modification Request from: 10.100.200.17.
psa-eupf-2  |
psa-eupf-2  |
psa-eupf-2  | 2024/05/16 18:01:03 INF Finding association for 10.100.200.17
psa-eupf-2  | 2024/05/16 18:01:03 INF Finding session 2
psa-eupf-2  | 2024/05/16 18:01:03 INF
psa-eupf-2  | Session Modification Request:
psa-eupf-2  |   CreatePDR ID: 2
psa-eupf-2  |     FAR ID: 2
psa-eupf-2  |     QER ID: 1
psa-eupf-2  |     Source Interface: 1
psa-eupf-2  |     UE IPv4 Address: 10.62.0.1
psa-eupf-2  |   CreateFAR ID: 2
psa-eupf-2  |     Apply Action: [2]
psa-eupf-2  |     Forwarding Parameters:
psa-eupf-2  |       Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:4 IPv4Address:10.100.200.14 IPv6Address:<nil> PortNumber:0 CTag:0 STag:0}
psa-eupf-2  |
psa-eupf-2  |
psa-eupf-2  | 2024/05/16 18:01:03 INF Saving FAR info to session: 2, {Action:2 OuterHeaderCreation:1 Teid:4 RemoteIP:248013834 LocalIP:281568266 TransportLevelMarking:0}
psa-eupf-2  | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=1, qerInfo={Action:2 OuterHeaderCreation:1 Teid:4 RemoteIP:248013834 LocalIP:281568266 TransportLevelMarking:0}
psa-eupf-2  | 2024/05/16 18:01:03 INF EBPF: Put PDR Downlink: ipv4=10.62.0.1, pdrInfo={OuterHeaderRemoval:0 FarId:1 QerId:0 SdfFilter:<nil>}
psa-eupf-2  | 2024/05/16 18:01:04 INF Received 16 bytes from 10.100.200.17:8805
psa-eupf-2  | 2024/05/16 18:01:04 INF Handling PFCP message from 10.100.200.17:8805

1 similar comment
@PapaySail
Copy link
Collaborator Author

✔ Docker-compose Free5GC

docker-compose up -d 

sergo@edgecom:~/edgecom24/free5gc-compose$ sudo docker-compose up -d
[+] Running 21/21
 _ Network free5gc-compose_privnet_n6  Created                                                                                                      0.1s
 _ Network free5gc-compose_default     Created                                                                                                      0.1s
 _ Network free5gc-compose_privnet     Created                                                                                                      0.1s
 _ Container prometheus                Started                                                                                                      1.3s
 _ Container upf                       Started                                                                                                      1.6s
 _ Container grafana                   Started                                                                                                      1.7s
 _ Container eupf                      Started                                                                                                      2.6s
 _ Container nat                       Started                                                                                                      2.4s
 _ Container mongodb                   Started                                                                                                      0.9s
 _ Container ui                        Started                                                                                                      1.3s
 _ Container webui                     Started                                                                                                      2.1s
 _ Container nrf                       Started                                                                                                      2.4s
 _ Container nssf                      Started                                                                                                      3.9s
 _ Container udm                       Started                                                                                                      3.9s
 _ Container ausf                      Started                                                                                                      3.9s
 _ Container amf                       Started                                                                                                      3.4s
 _ Container pcf                       Started                                                                                                      4.1s
 _ Container smf                       Started                                                                                                      4.0s
 _ Container udr                       Started                                                                                                      3.9s
 _ Container ueransim                  Started                                                                                                      4.2s
 _ Container n3iwf                     Started                                                                                                      4.5s
sergo@edgecom:~/edgecom24/free5gc-compose$

sergo@edgecom:~/edgecom24/free5gc-compose$ sudo docker-compose exec  ueransim bash
root@1d4b478b1c91:/ueransim# ./nr-ue -c config/uecfg.yaml
UERANSIM v3.2.6
[2024-05-16 17:37:49.842] [nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH]
[2024-05-16 17:37:49.843] [rrc] [debug] New signal detected for cell[1], total [1] cells in coverage
[2024-05-16 17:37:49.846] [nas] [info] Selected plmn[208/93]
[2024-05-16 17:37:52.343] [rrc] [info] Selected cell plmn[208/93] tac[1] category[SUITABLE]
[2024-05-16 17:37:52.343] [nas] [info] UE switches to state [MM-DEREGISTERED/PS]
[2024-05-16 17:37:52.343] [nas] [info] UE switches to state [MM-DEREGISTERED/NORMAL-SERVICE]
[2024-05-16 17:37:52.343] [nas] [debug] Initial registration required due to [MM-DEREG-NORMAL-SERVICE]
[2024-05-16 17:37:52.343] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig]
[2024-05-16 17:37:52.343] [nas] [debug] Sending Initial Registration
[2024-05-16 17:37:52.343] [nas] [info] UE switches to state [MM-REGISTER-INITIATED]
[2024-05-16 17:37:52.343] [rrc] [debug] Sending RRC Setup Request
[2024-05-16 17:37:52.346] [rrc] [info] RRC connection established
[2024-05-16 17:37:52.346] [rrc] [info] UE switches to state [RRC-CONNECTED]
[2024-05-16 17:37:52.346] [nas] [info] UE switches to state [CM-CONNECTED]
[2024-05-16 17:37:52.378] [nas] [debug] Authentication Request received
[2024-05-16 17:37:52.389] [nas] [debug] Security Mode Command received
[2024-05-16 17:37:52.389] [nas] [debug] Selected integrity[2] ciphering[0]
[2024-05-16 17:37:52.440] [nas] [debug] Registration accept received
[2024-05-16 17:37:52.440] [nas] [info] UE switches to state [MM-REGISTERED/NORMAL-SERVICE]
[2024-05-16 17:37:52.440] [nas] [debug] Sending Registration Complete
[2024-05-16 17:37:52.440] [nas] [info] Initial Registration is successful
[2024-05-16 17:37:52.440] [nas] [debug] Sending PDU Session Establishment Request
[2024-05-16 17:37:52.463] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig]
[2024-05-16 17:37:52.709] [nas] [debug] PDU Session Establishment Accept received
[2024-05-16 17:37:52.709] [nas] [info] PDU Session establishment is successful PSI[1]
[2024-05-16 17:37:52.725] [app] [info] Connection setup for PDU session[1] is successful, TUN interface[uesimtun0, 10.60.0.1] is up.

### at another console
sergo@edgecom:~/edgecom24/free5gc-compose$ sudo docker-compose exec  ueransim bash
root@1d4b478b1c91:/ueransim# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: ip_vti0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
3: uesimtun0: <POINTOPOINT,PROMISC,NOTRAILERS,UP,LOWER_UP> mtu 1400 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none
    inet 10.60.0.1/32 scope global uesimtun0
       valid_lft forever preferred_lft forever
689: eth0@if690: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:0a:64:c8:10 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.100.200.16/24 brd 10.100.200.255 scope global eth0
       valid_lft forever preferred_lft forever
root@1d4b478b1c91:/ueransim#
root@1d4b478b1c91:/ueransim# ping -I uesimtun0 1.1.1.1

PING 1.1.1.1 (1.1.1.1) from 10.60.0.1 uesimtun0: 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=55 time=2.19 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=55 time=1.58 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=55 time=1.68 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=55 time=1.66 ms
64 bytes from 1.1.1.1: icmp_seq=5 ttl=55 time=1.59 ms
^C
--- 1.1.1.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4004ms
rtt min/avg/max/mdev = 1.584/1.738/2.190/0.228 ms
root@1d4b478b1c91:/ueransim#

eupf logs:
sudo docker-compose logs edgecom-upf |less

eupf  | 2024/05/16 17:37:52 Handling PFCP message from 10.100.200.15:8805
eupf  | 2024/05/16 17:37:52 Got Session Modification Request from: 10.100.200.15.
eupf  | 2024/05/16 17:37:52 Finding association for 10.100.200.15
eupf  | 2024/05/16 17:37:52 Finding session 2
eupf  | 2024/05/16 17:37:52
eupf  | Session Modification Request:
eupf  |   UpdatePDR ID: 2
eupf  |     FAR ID: 2
eupf  |     Source Interface: 1
eupf  |     UE IPv4 Address: 10.60.0.1
eupf  |   UpdateFAR ID: 2
eupf  |     Apply Action: [2]
eupf  |     Update forwarding Parameters:
eupf  |       Network Instance: internet
eupf  |       Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:1 IPv4Address:10.100.200.16 IPv6Address:<nil> PortNumber:0 CTag:0 STag:0}
eupf  | 2024/05/16 17:37:52 Updating FAR info: 2, {FarInfo:{Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:281568266 LocalIP:248013834 TransportLevelMark
ing:0} GlobalId:1}
eupf  | 2024/05/16 17:37:52 EBPF: Update FAR: internalId=1, farInfo={Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:281568266 LocalIP:248013834 Transport
LevelMarking:0}
eupf  | 2024/05/16 17:37:52 Both F-TEID IE and UE IP Address IE are missing
eupf  | 2024/05/16 17:37:57 Sent Heartbeat Request to: 10.100.200.15
eupf  | 2024/05/16 17:37:57 Received 16 bytes from 10.100.200.15:8805

✔ Free5GC with UpLink Classifier config throught three eUPFs

docker-compose -f docker-compose.yaml -f docker-compose.ulcl.yml up -d 

sergo@edgecom:~/edgecom24/free5gc-compose$ git checkout ulcl-n9upf-experimetns
Branch 'ulcl-n9upf-experimetns' set up to track remote branch 'ulcl-n9upf-experimetns' from 'origin'.
Switched to a new branch 'ulcl-n9upf-experimetns'

docker-compose -f docker-compose.yaml -f docker-compose.ulcl.yml up -d

sergo@edgecom:~/edgecom24/free5gc-compose$ sudo docker-compose ps
NAME                IMAGE                           COMMAND                  SERVICE             CREATED             STATUS              PORTS
amf                 free5gc/amf:v3.3.0              "./amf -c ./config/a_"   free5gc-amf         2 minutes ago       Up 2 minutes        8000/tcp
ausf                free5gc/ausf:v3.3.0             "./ausf -c ./config/_"   free5gc-ausf        2 minutes ago       Up 2 minutes        8000/tcp
grafana             grafana/grafana                 "/run.sh"                grafana             2 minutes ago       Up 2 minutes        0.0.0.0:3000->3000/tcp, :::3000->3000/tcp
i-eupf              ghcr.io/edgecomllc/eupf:0.5.0   "sh /app/bin/entrypo_"   edgecom-i-upf       2 minutes ago       Up 2 minutes        0.0.0.0:8082->8080/tcp, :::8082->8080/tcp, 0.0.0.0:9092->9090/tcp, :::9092->9090/tcp
mongodb             mongo                           "docker-entrypoint.s_"   db                  2 minutes ago       Up 2 minutes        27017/tcp
n3iwf               free5gc/n3iwf:v3.3.0            "sh -c './n3iwf-ipse_"   free5gc-n3iwf       2 minutes ago       Up 2 minutes
nat                 free5gc-compose-edgecom-nat     "sh /app/bin/busy-po_"   edgecom-nat         2 minutes ago       Up 2 minutes
nrf                 free5gc/nrf:v3.3.0              "./nrf -c ./config/n_"   free5gc-nrf         2 minutes ago       Up 2 minutes        8000/tcp
nssf                free5gc/nssf:v3.3.0             "./nssf -c ./config/_"   free5gc-nssf        2 minutes ago       Up 2 minutes        8000/tcp
pcf                 free5gc/pcf:v3.3.0              "./pcf -c ./config/p_"   free5gc-pcf         2 minutes ago       Up 2 minutes        8000/tcp
prometheus          prom/prometheus                 "/bin/prometheus --c_"   prometheus          2 minutes ago       Up 2 minutes        0.0.0.0:9091->9090/tcp, :::9091->9090/tcp
psa-eupf            ghcr.io/edgecomllc/eupf:0.5.0   "sh /app/bin/entrypo_"   edgecom-psa-upf     2 minutes ago       Up 2 minutes        0.0.0.0:8080->8080/tcp, :::8080->8080/tcp, 0.0.0.0:9090->9090/tcp, :::9090->9090/tcp
psa-eupf-2          ghcr.io/edgecomllc/eupf:0.5.0   "sh /app/bin/entrypo_"   edgecom-psa-upf-2   2 minutes ago       Up 2 minutes
smf                 free5gc/smf:v3.3.0              "./smf -c ./config/s_"   free5gc-smf         2 minutes ago       Up 2 minutes        8000/tcp
udm                 free5gc/udm:v3.3.0              "./udm -c ./config/u_"   free5gc-udm         2 minutes ago       Up 2 minutes        8000/tcp
udr                 free5gc/udr:v3.3.0              "./udr -c ./config/u_"   free5gc-udr         2 minutes ago       Up 2 minutes        8000/tcp
ueransim            free5gc/ueransim:v3.3.0         "./nr-gnb -c ./confi_"   ueransim            2 minutes ago       Up 2 minutes
webui               free5gc/webui:v3.3.0            "./webui -c ./config_"   free5gc-webui       2 minutes ago       Up 2 minutes        0.0.0.0:5000->5000/tcp, :::5000->5000/tcp
sergo@edgecom:~/edgecom24/free5gc-compose$
sergo@edgecom:~/edgecom24/free5gc-compose$ sudo docker-compose exec ueransim bash
root@aaad991d992b:/ueransim# ./nr-ue -c config/uecfg.yaml &
[1] 24
root@aaad991d992b:/ueransim# UERANSIM v3.2.6
[2024-05-16 18:01:03.356] [nas] [info] UE switches to state [MM-DEREGISTERED/PLMN-SEARCH]
[2024-05-16 18:01:03.356] [rrc] [debug] New signal detected for cell[1], total [1] cells in coverage
[2024-05-16 18:01:03.356] [nas] [info] Selected plmn[208/93]
[2024-05-16 18:01:03.356] [rrc] [info] Selected cell plmn[208/93] tac[1] category[SUITABLE]
[2024-05-16 18:01:03.356] [nas] [info] UE switches to state [MM-DEREGISTERED/PS]
[2024-05-16 18:01:03.356] [nas] [info] UE switches to state [MM-DEREGISTERED/NORMAL-SERVICE]
[2024-05-16 18:01:03.356] [nas] [debug] Initial registration required due to [MM-DEREG-NORMAL-SERVICE]
[2024-05-16 18:01:03.357] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig]
[2024-05-16 18:01:03.357] [nas] [debug] Sending Initial Registration
[2024-05-16 18:01:03.357] [nas] [info] UE switches to state [MM-REGISTER-INITIATED]
[2024-05-16 18:01:03.357] [rrc] [debug] Sending RRC Setup Request
[2024-05-16 18:01:03.357] [rrc] [info] RRC connection established
[2024-05-16 18:01:03.357] [rrc] [info] UE switches to state [RRC-CONNECTED]
[2024-05-16 18:01:03.357] [nas] [info] UE switches to state [CM-CONNECTED]
[2024-05-16 18:01:03.387] [nas] [debug] Authentication Request received
[2024-05-16 18:01:03.398] [nas] [debug] Security Mode Command received
[2024-05-16 18:01:03.398] [nas] [debug] Selected integrity[2] ciphering[0]
[2024-05-16 18:01:03.451] [nas] [debug] Registration accept received
[2024-05-16 18:01:03.451] [nas] [info] UE switches to state [MM-REGISTERED/NORMAL-SERVICE]
[2024-05-16 18:01:03.451] [nas] [debug] Sending Registration Complete
[2024-05-16 18:01:03.451] [nas] [info] Initial Registration is successful
[2024-05-16 18:01:03.451] [nas] [debug] Sending PDU Session Establishment Request
[2024-05-16 18:01:03.451] [nas] [debug] UAC access attempt is allowed for identity[0], category[MO_sig]
[2024-05-16 18:01:03.720] [nas] [debug] PDU Session Establishment Accept received
[2024-05-16 18:01:03.720] [nas] [info] PDU Session establishment is successful PSI[1]
[2024-05-16 18:01:03.735] [app] [info] Connection setup for PDU session[1] is successful, TUN interface[uesimtun0, 10.62.0.1] is up.

root@aaad991d992b:/ueransim# ping -I uesimtun0 1.1.1.1
PING 1.1.1.1 (1.1.1.1) from 10.62.0.1 uesimtun0: 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=55 time=2.01 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=55 time=1.61 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=55 time=1.66 ms
^C
--- 1.1.1.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 1.608/1.759/2.007/0.176 ms
root@aaad991d992b:/ueransim# ping -I uesimtun0 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 10.62.0.1 uesimtun0: 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=58 time=16.9 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=58 time=17.0 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=58 time=17.0 ms
^C
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 16.873/16.951/17.002/0.056 ms

Logs

sudo docker-compose -f docker-compose.yaml -f docker-compose.ulcl.yml logs edgecom-i-upf |less -R
look for 8.8.8.8

i-eupf  | 2024/05/16 18:01:03 INF Got Session Modification Request from: 10.100.200.17.
i-eupf  |
i-eupf  |
i-eupf  | 2024/05/16 18:01:03 INF Finding association for 10.100.200.17
i-eupf  | 2024/05/16 18:01:03 INF Finding session 2
i-eupf  | 2024/05/16 18:01:03 INF
i-eupf  | Session Modification Request:
i-eupf  |   CreatePDR ID: 3
i-eupf  |     Outer Header Removal: 0
i-eupf  |     FAR ID: 3
i-eupf  |     QER ID: 1
i-eupf  |     Source Interface: 0
i-eupf  |     TEID: 1
i-eupf  |     Ipv4: 10.100.200.14
i-eupf  |     Ipv6: <nil>
i-eupf  |     UE IPv4 Address: 10.62.0.1
i-eupf  |     SDF Filter: permit out ip from 10.62.0.1 to 8.8.8.8/32
i-eupf  |   CreatePDR ID: 4
i-eupf  |     Outer Header Removal: 0
i-eupf  |     FAR ID: 4
i-eupf  |     QER ID: 1
i-eupf  |     Source Interface: 1
i-eupf  |     TEID: 4
i-eupf  |     Ipv4: 10.100.200.14
i-eupf  |     Ipv6: <nil>
i-eupf  |   CreateFAR ID: 3
i-eupf  |     Apply Action: [2]
i-eupf  |     Forwarding Parameters:
i-eupf  |       Network Instance: internet
i-eupf  |       Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:1 IPv4Address:10.100.200.16 IPv6Address:<nil> PortNumber:0 CTag:0 STag:0}
i-eupf  |   CreateFAR ID: 4
i-eupf  |     Apply Action: [2]
i-eupf  |     Forwarding Parameters:
i-eupf  |       Network Instance: internet
i-eupf  |       Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:1 IPv4Address:10.100.200.19 IPv6Address:<nil> PortNumber:0 CTag:0 STag:0}
i-eupf  |
i-eupf  |
i-eupf  | 2024/05/16 18:01:03 INF Saving FAR info to session: 3, {Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:281568266 LocalIP:248013834 TransportLevelMarking:0}
i-eupf  | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=2, qerInfo={Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:281568266 LocalIP:248013834 TransportLevelMarking:0}
i-eupf  | 2024/05/16 18:01:03 INF Saving FAR info to session: 4, {Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:331899914 LocalIP:248013834 TransportLevelMarking:0}
i-eupf  | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=3, qerInfo={Action:2 OuterHeaderCreation:1 Teid:1 RemoteIP:331899914 LocalIP:248013834 TransportLevelMarking:0}
i-eupf  | 2024/05/16 18:01:03 INF EBPF: Put PDR Uplink: teid=1, pdrInfo={OuterHeaderRemoval:0 FarId:2 QerId:1 SdfFilter:{Protocol:1 SrcAddress:{Type:1 Ip:10.62.0.1 Mask:ffffffff} SrcPortRange:{LowerBound:0 UpperBound:65535} DstAddress:{Type:1 Ip:8.8.8.8 Mask:ffffffff} DstPortRange:{LowerBound:0 UpperBound:65535}}}
i-eupf  | 2024/05/16 18:01:03 INF EBPF: Put PDR Uplink: teid=4, pdrInfo={OuterHeaderRemoval:0 FarId:3 QerId:1 SdfFilter:<nil>}
i-eupf  | 2024/05/16 18:01:04 INF Received 16 bytes from 10.100.200.17:8805

sudo docker-compose -f docker-compose.yaml -f docker-compose.ulcl.yml logs edgecom-psa-upf |less -R

psa-eupf  | 2024/05/16 18:01:03 INF Got Session Establishment Request from: 10.100.200.17.
psa-eupf  | 2024/05/16 18:01:03 INF
psa-eupf  | Session Establishment Request:
psa-eupf  |   CreatePDR ID: 1
psa-eupf  |     Outer Header Removal: 0
psa-eupf  |     FAR ID: 1
psa-eupf  |     QER ID: 2
psa-eupf  |     QER ID: 1
psa-eupf  |     Source Interface: 0
psa-eupf  |     TEID: 1
psa-eupf  |     Ipv4: 10.100.200.15
psa-eupf  |     Ipv6: <nil>
psa-eupf  |     UE IPv4 Address: 10.62.0.1
psa-eupf  |   CreatePDR ID: 2
psa-eupf  |     FAR ID: 2
psa-eupf  |     QER ID: 2
psa-eupf  |     QER ID: 1
psa-eupf  |     Source Interface: 1
psa-eupf  |     UE IPv4 Address: 10.62.0.1
psa-eupf  |   CreateFAR ID: 1
psa-eupf  |     Apply Action: [2]
psa-eupf  |     Forwarding Parameters:
psa-eupf  |       Network Instance: internet
psa-eupf  |   CreateFAR ID: 2
psa-eupf  |     Apply Action: [2]
psa-eupf  |     Forwarding Parameters:
psa-eupf  |       Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:2 IPv4Address:10.100.200.14 IPv6Address:<nil> PortNumber:0 CTag:0 STag:0}
psa-eupf  |   CreateQER ID: 2
psa-eupf  |     Gate Status DL: 0
psa-eupf  |     Gate Status UL: 0
psa-eupf  |     QFI: 1
psa-eupf  |   CreateQER ID: 1
psa-eupf  |     Gate Status DL: 0
psa-eupf  |     Gate Status UL: 0
psa-eupf  |     Max Bitrate DL: 1000000
psa-eupf  |     Max Bitrate UL: 2000000
psa-eupf  |     QFI: 0
psa-eupf  |
psa-eupf  |
psa-eupf  | 2024/05/16 18:01:03 INF WARN: No OuterHeaderCreation
psa-eupf  | 2024/05/16 18:01:03 INF Saving FAR info to session: 1, {Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:264791050 TransportLevelMarking:0}
psa-eupf  | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=0, qerInfo={Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:264791050 TransportLevelMarking:0}
psa-eupf  | 2024/05/16 18:01:03 INF Saving FAR info to session: 2, {Action:2 OuterHeaderCreation:1 Teid:2 RemoteIP:248013834 LocalIP:264791050 TransportLevelMarking:0}
psa-eupf  | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=1, qerInfo={Action:2 OuterHeaderCreation:1 Teid:2 RemoteIP:248013834 LocalIP:264791050 TransportLevelMarking:0}
psa-eupf  | 2024/05/16 18:01:03 INF Saving QER info to session: 2, {GateStatusUL:0 GateStatusDL:0 Qfi:1 MaxBitrateUL:0 MaxBitrateDL:0 StartUL:0 StartDL:0}
psa-eupf  | 2024/05/16 18:01:03 INF EBPF: Put QER: internalId=0, qerInfo={GateStatusUL:0 GateStatusDL:0 Qfi:1 MaxBitrateUL:0 MaxBitrateDL:0 StartUL:0 StartDL:0}
psa-eupf  | 2024/05/16 18:01:03 INF Saving QER info to session: 1, {GateStatusUL:0 GateStatusDL:0 Qfi:0 MaxBitrateUL:2000000000 MaxBitrateDL:1000000000 StartUL:0 StartDL:0}
psa-eupf  | 2024/05/16 18:01:03 INF EBPF: Put QER: internalId=1, qerInfo={GateStatusUL:0 GateStatusDL:0 Qfi:0 MaxBitrateUL:2000000000 MaxBitrateDL:1000000000 StartUL:0 StartDL:0}
psa-eupf  | 2024/05/16 18:01:03 INF EBPF: Put PDR Uplink: teid=1, pdrInfo={OuterHeaderRemoval:0 FarId:0 QerId:0 SdfFilter:<nil>}
psa-eupf  | 2024/05/16 18:01:03 INF EBPF: Put PDR Downlink: ipv4=10.62.0.1, pdrInfo={OuterHeaderRemoval:0 FarId:1 QerId:0 SdfFilter:<nil>}
psa-eupf  | 2024/05/16 18:01:03 INF Session Establishment Request from 10.100.200.17 accepted.
psa-eupf  | 2024/05/16 18:01:04 INF Received 16 bytes from 10.100.200.17:8805

sudo docker-compose -f docker-compose.yaml -f docker-compose.ulcl.yml logs edgecom-psa-upf-2 |less -R

psa-eupf-2  | 2024/05/16 18:01:03 INF Handling PFCP message from 10.100.200.17:8805
psa-eupf-2  | 2024/05/16 18:01:03 INF Got Session Establishment Request from: 10.100.200.17.
psa-eupf-2  | 2024/05/16 18:01:03 INF
psa-eupf-2  | Session Establishment Request:
psa-eupf-2  |   CreatePDR ID: 1
psa-eupf-2  |     Outer Header Removal: 0
psa-eupf-2  |     FAR ID: 1
psa-eupf-2  |     QER ID: 1
psa-eupf-2  |     Source Interface: 0
psa-eupf-2  |     TEID: 1
psa-eupf-2  |     Ipv4: 10.100.200.16
psa-eupf-2  |     Ipv6: <nil>
psa-eupf-2  |     UE IPv4 Address: 10.62.0.1
psa-eupf-2  |   CreateFAR ID: 1
psa-eupf-2  |     Apply Action: [2]
psa-eupf-2  |     Forwarding Parameters:
psa-eupf-2  |       Network Instance: internet
psa-eupf-2  |   CreateQER ID: 1
psa-eupf-2  |     Gate Status DL: 0
psa-eupf-2  |     Gate Status UL: 0
psa-eupf-2  |     Max Bitrate DL: 1000000
psa-eupf-2  |     Max Bitrate UL: 2000000
psa-eupf-2  |     QFI: 0
psa-eupf-2  |
psa-eupf-2  |
psa-eupf-2  | 2024/05/16 18:01:03 INF WARN: No OuterHeaderCreation
psa-eupf-2  | 2024/05/16 18:01:03 INF Saving FAR info to session: 1, {Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:281568266 TransportLevelMarking:0}
psa-eupf-2  | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=0, qerInfo={Action:2 OuterHeaderCreation:0 Teid:0 RemoteIP:0 LocalIP:281568266 TransportLevelMarking:0}
psa-eupf-2  | 2024/05/16 18:01:03 INF Saving QER info to session: 1, {GateStatusUL:0 GateStatusDL:0 Qfi:0 MaxBitrateUL:2000000000 MaxBitrateDL:1000000000 StartUL:0 StartDL:0}
psa-eupf-2  | 2024/05/16 18:01:03 INF EBPF: Put QER: internalId=0, qerInfo={GateStatusUL:0 GateStatusDL:0 Qfi:0 MaxBitrateUL:2000000000 MaxBitrateDL:1000000000 StartUL:0 StartDL:0}
psa-eupf-2  | 2024/05/16 18:01:03 INF EBPF: Put PDR Uplink: teid=1, pdrInfo={OuterHeaderRemoval:0 FarId:0 QerId:0 SdfFilter:<nil>}
psa-eupf-2  | 2024/05/16 18:01:03 INF Session Establishment Request from 10.100.200.17 accepted.
psa-eupf-2  | 2024/05/16 18:01:03 INF Received 137 bytes from 10.100.200.17:8805
psa-eupf-2  | 2024/05/16 18:01:03 INF Handling PFCP message from 10.100.200.17:8805
psa-eupf-2  | 2024/05/16 18:01:03 INF Got Session Modification Request from: 10.100.200.17.
psa-eupf-2  |
psa-eupf-2  |
psa-eupf-2  | 2024/05/16 18:01:03 INF Finding association for 10.100.200.17
psa-eupf-2  | 2024/05/16 18:01:03 INF Finding session 2
psa-eupf-2  | 2024/05/16 18:01:03 INF
psa-eupf-2  | Session Modification Request:
psa-eupf-2  |   CreatePDR ID: 2
psa-eupf-2  |     FAR ID: 2
psa-eupf-2  |     QER ID: 1
psa-eupf-2  |     Source Interface: 1
psa-eupf-2  |     UE IPv4 Address: 10.62.0.1
psa-eupf-2  |   CreateFAR ID: 2
psa-eupf-2  |     Apply Action: [2]
psa-eupf-2  |     Forwarding Parameters:
psa-eupf-2  |       Outer Header Creation: &{OuterHeaderCreationDescription:256 TEID:4 IPv4Address:10.100.200.14 IPv6Address:<nil> PortNumber:0 CTag:0 STag:0}
psa-eupf-2  |
psa-eupf-2  |
psa-eupf-2  | 2024/05/16 18:01:03 INF Saving FAR info to session: 2, {Action:2 OuterHeaderCreation:1 Teid:4 RemoteIP:248013834 LocalIP:281568266 TransportLevelMarking:0}
psa-eupf-2  | 2024/05/16 18:01:03 INF EBPF: Put FAR: internalId=1, qerInfo={Action:2 OuterHeaderCreation:1 Teid:4 RemoteIP:248013834 LocalIP:281568266 TransportLevelMarking:0}
psa-eupf-2  | 2024/05/16 18:01:03 INF EBPF: Put PDR Downlink: ipv4=10.62.0.1, pdrInfo={OuterHeaderRemoval:0 FarId:1 QerId:0 SdfFilter:<nil>}
psa-eupf-2  | 2024/05/16 18:01:04 INF Received 16 bytes from 10.100.200.17:8805
psa-eupf-2  | 2024/05/16 18:01:04 INF Handling PFCP message from 10.100.200.17:8805

@PapaySail
Copy link
Collaborator Author

OAI Kubernetes

The deployment still not ready. 🤷‍♂️ Nothing to check.

  1. install eupf
    ⚠️ not finished, work in progress
    make upf

  2. configure calico BGP settings. Here, we configure Calico BGP peer, create Calico IP Pool (for NAT) and configure Felix for save external routes (recevied by BGP from eUPF BIRD)
    ⚠️ not finished, work in progress
    make calico

@PapaySail
Copy link
Collaborator Author

PapaySail commented May 21, 2024 •
edited
Loading

✔ Docker-compose OAI 5G SA mode with L2 nFAPI simulator

  1. Start containers in the following order:

    service "oai-spgwu" has neither an image nor a build context specified: invalid compose project 

    sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ cp -ar ~/edgecom24/eupf/docs/deployments/oai-nfapi-sim-compose/* ./
sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ ls
docker-compose.override.yml  docker-compose.yaml  eupf  nat  README.md
sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$
sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ sudo docker-compose up -d mysql oai-nrf oai-amf oai-smf edgecom-nat edgecom-upf
service "oai-spgwu" has neither an image nor a build context specified: invalid compose project

    ✔ service renamed to [oai-upf](https://gitlab.eurecom.fr/oai/openairinterface5g/-/blob/develop/ci-scripts/yaml_files/5g_l2sim_tdd/docker-compose.yaml#L46) :: update to docker-compose.override.yaml needed 
    sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ sudo docker-compose up -d mysql oai-nrf oai-amf oai-smf edgecom-nat edgecom-upf
service "prometheus" refers to undefined volume prom_data: invalid compose project

    ✔ update to docker-compose.override.yaml needed:

    volumes:
  prom_data:

    ✔ no such service: oai-nrf

    SMF exited: [smf_app] [error] Could not read network interface configuration

    • due to new config file
    • removed image: oaisoftwarealliance/oai-smf:v1.5.1
    • added
        networks:
      public_net:
        aliases:
          - oai-upf
docker pull oaisoftwarealliance/oai-gnb:develop 

sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ sudo docker pull oaisoftwarealliance/oai-gnb:develop
develop: Pulling from oaisoftwarealliance/oai-gnb
edaedc954fb5: Already exists
90e8c9a558e0: Already exists
544dfcb95356: Pull complete
5aaeb55fa076: Pull complete
943517f19fa8: Pull complete
c5ce95fed1cd: Pull complete
650a9bbfdc1e: Pull complete
6c40c55a2a3e: Pull complete
b116ff4d25cb: Pull complete
e46081b32958: Pull complete
955f4243c199: Pull complete
4f4fb700ef54: Pull complete
a07e06ffa612: Pull complete
Digest: sha256:58dac7ea7322641c664ae58f1d6e4bbb05a5843d12c257913038fd085338840c
Status: Downloaded newer image for oaisoftwarealliance/oai-gnb:develop
docker.io/oaisoftwarealliance/oai-gnb:develop
sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ sudo docker-compose up -d  oai-gnb

❌ UE not connected

oaitun_ue1 interface without ip address 

sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ ip a |grep oaitun
769: oaitun_ue1: <POINTOPOINT,MULTICAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 500

Troubleshooting

SMF logs regarding UPF: Did not find a UPF config for UPF-associated association 192.168.71.134. Use empty UPF profile

l2sim-oai-smf  | [2024-05-22 21:09:36.870] [smf_app] [debug] Reverse Resolving Try #0
l2sim-oai-smf  | [2024-05-22 21:09:36.871] [smf_app] [debug] Resolve IP Addr 192.168.71.134, FQDN eupf.l2sim-oai-public-net
l2sim-oai-smf  | [2024-05-22 21:09:36.871] [smf_app] [debug] Did not find a UPF config for UPF-associated association 192.168.71.134. Use empty UPF profile with one assumed N3 and N6 interface.
l2sim-oai-smf  | [2024-05-22 21:09:36.871] [smf_app] [debug] Could not find other edges for UPF, just add UPF as a node
l2sim-oai-smf  | [2024-05-22 21:09:36.871] [smf_app] [debug] Successfully added UPF node: 192.168.71.134, (2252843200)
l2sim-oai-smf  | [2024-05-22 21:09:36.871] [smf_app] [debug] Successfully added UPF graph edge for 192.168.71.134: N3()
l2sim-oai-smf  | [2024-05-22 21:09:36.871] [smf_app] [debug] Successfully added UPF graph edge for 192.168.71.134: N6()
l2sim-oai-smf  | [2024-05-22 21:09:36.871] [smf_app] [debug] UPF graph
l2sim-oai-smf  | [2024-05-22 21:09:36.871] [smf_app] [debug] * 192.168.71.134 --> N3(), N6(),
l2sim-oai-smf  |
l2sim-oai-smf  |
l2sim-oai-smf  | [2024-05-22 21:09:36.871] [smf_app] [debug] Added PFCP assocation with UPF config:
l2sim-oai-smf  |  + 192.168.71.134
l2sim-oai-smf  |       + host...................................: 192.168.71.134
l2sim-oai-smf  |       + port...................................: 8805

⚠ The original deployment itself is faulty:

sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ sudo docker-compose ps -a
NAME                IMAGE                                    COMMAND                  SERVICE             CREATED              STATUS                        PORTS
l2sim-mysql         mysql:8.0                                "docker-entrypoint.s_"   mysql               40 minutes ago       Up 40 minutes (healthy)       3306/tcp, 33060/tcp
l2sim-oai-amf       oaisoftwarealliance/oai-amf:v2.0.0       "/openair-amf/bin/oa_"   oai-amf             40 minutes ago       Up 40 minutes (healthy)       80/tcp, 9090/tcp, 38412/sctp
l2sim-oai-ext-dn    oaisoftwarealliance/trf-gen-cn5g:focal   "/bin/bash -c ' ipta_"   oai-ext-dn          40 minutes ago       Up 40 minutes (healthy)
l2sim-oai-gnb       oaisoftwarealliance/oai-gnb:develop      "/tini -v -- /opt/oa_"   oai-gnb             39 minutes ago       Up 39 minutes (healthy)
l2sim-oai-nr-ue0    oaisoftwarealliance/oai-nr-ue:develop    "/tini -v -- /opt/oa_"   oai-nr-ue0          About a minute ago   Up About a minute (healthy)
l2sim-oai-smf       oaisoftwarealliance/oai-smf:v2.0.0       "/openair-smf/bin/oa_"   oai-smf             40 minutes ago       Up 40 minutes (healthy)       80/tcp, 8080/tcp, 8805/udp
l2sim-oai-upf       oaisoftwarealliance/oai-upf:v2.0.0       "/openair-upf/bin/oa_"   oai-upf             40 minutes ago       Up 40 minutes (healthy)       2152/udp, 8805/udp
l2sim-proxy         oai-lte-multi-ue-proxy:latest            "/oai-lte-multi-ue-p_"   proxy               About a minute ago   Up About a minute (healthy)
sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ ip a |grep oaitun
825: oaitun_ue1: <POINTOPOINT,MULTICAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 500
sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ sudo docker-compose logs  oai-amf |less
sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ sudo docker-compose logs -t oai-nr-ue0 |less
sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ ip a |grep oaitun
825: oaitun_ue1: <POINTOPOINT,MULTICAST,NOARP> mtu 1500 qdisc noop state DOWN group default qlen 500

Let's get back in history an start from commit chore(ci): updating 5G rfsimulator scenarios to release v2.0
It is parent to commit with changes to docker-compose and config files.

sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ git checkout f1d080d31515d3cc9a31a41fd548753ccba4f09f
Note: switching to 'f1d080d31515d3cc9a31a41fd548753ccba4f09f'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at f1d080d315 chore(ci): updating 5G rfsimulator scenarios to release v2.0
sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$

Trying to start it as is = unsuccessful, no IP addres assigned to oaitun_ue1

Next, will try to use image tag 2023.w43 for oai-nr-ue and oai-gnb.
OK! Yes, this is working images.

sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ ip a |grep oaitun
846: oaitun_ue1: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 500
    inet 12.1.1.2/24 brd 12.1.1.255 scope global oaitun_ue1
sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ sudo docker-compose ps -a
NAME                IMAGE                                       COMMAND                  SERVICE             CREATED              STATUS                        PORTS
l2sim-mysql         mysql:8.0                                   "docker-entrypoint.s_"   mysql               54 minutes ago       Up 54 minutes (healthy)       3306/tcp, 33060/tcp
l2sim-oai-amf       oaisoftwarealliance/oai-amf:v1.5.0          "python3 /openair-am_"   oai-amf             54 minutes ago       Up 54 minutes (healthy)       80/tcp, 9090/tcp, 38412/sctp
l2sim-oai-ext-dn    oaisoftwarealliance/trf-gen-cn5g:focal      "/bin/bash -c ' ipta_"   oai-ext-dn          54 minutes ago       Up 54 minutes (healthy)
l2sim-oai-gnb       oaisoftwarealliance/oai-gnb:2023.w43        "/opt/oai-gnb/bin/en_"   oai-gnb             2 minutes ago        Up 2 minutes (healthy)
l2sim-oai-nr-ue0    oaisoftwarealliance/oai-nr-ue:2023.w43      "/opt/oai-nr-ue/bin/_"   oai-nr-ue0          About a minute ago   Up About a minute (healthy)
l2sim-oai-nrf       oaisoftwarealliance/oai-nrf:v1.5.0          "python3 /openair-nr_"   oai-nrf             54 minutes ago       Up 54 minutes (healthy)       80/tcp, 9090/tcp
l2sim-oai-smf       oaisoftwarealliance/oai-smf:v1.5.0          "python3 /openair-sm_"   oai-smf             54 minutes ago       Up 54 minutes (healthy)       80/tcp, 8080/tcp, 8805/udp
l2sim-oai-spgwu     oaisoftwarealliance/oai-spgwu-tiny:v1.5.0   "python3 /openair-sp_"   oai-spgwu           54 minutes ago       Up 54 minutes (healthy)       2152/udp, 8805/udp
l2sim-proxy         oai-lte-multi-ue-proxy:latest               "/oai-lte-multi-ue-p_"   proxy               About a minute ago   Up About a minute (healthy)
sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ ping -I oaitun_ue1 -c 3 1.1.1.1
PING 1.1.1.1 (1.1.1.1) from 12.1.1.2 oaitun_ue1: 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=55 time=79.7 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=55 time=112 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=55 time=151 ms

--- 1.1.1.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 79.722/114.461/151.339/29.276 ms

OK, with eUPF and oaisoftwarealliance/oai-gnb:2023.w43, oaisoftwarealliance/oai-nr-ue:2023.w43. Just no icmp replies.

Details & troubleshooting 

sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ ip a |grep tun
865: oaitun_ue1: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 500
    inet 12.1.1.2/24 brd 12.1.1.255 scope global oaitun_ue1
sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ ping -I oaitun_ue1 -c 3 1.1.1.1
PING 1.1.1.1 (1.1.1.1) from 12.1.1.2 oaitun_ue1: 56(84) bytes of data.

--- 1.1.1.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2040ms

sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ ping -I oaitun_ue1 -c 3 172.17.0.1
PING 172.17.0.1 (172.17.0.1) from 12.1.1.2 oaitun_ue1: 56(84) bytes of data.
^C
--- 172.17.0.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2054ms

sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ ping -I oaitun_ue1 -c 3 10.100.250.1
PING 10.100.250.1 (10.100.250.1) from 12.1.1.2 oaitun_ue1: 56(84) bytes of data.

--- 10.100.250.1 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2037ms

sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ sudo docker-compose ps -a
NAME                IMAGE                                                              COMMAND                  SERVICE             CREATED             STATUS                   PORTS
eupf                ghcr.io/edgecomllc/eupf:7086b71cb1dbc254db84c30e14fa4a68d3beb132   "sh /app/bin/entrypo_"   edgecom-upf         5 minutes ago       Up 5 minutes             0.0.0.0:9090->9090/tcp, :::9090->9090/tcp, 0.0.0.0:8880->8080/tcp, :::8880->8080/tcp
l2sim-mysql         mysql:8.0                                                          "docker-entrypoint.s_"   mysql               5 minutes ago       Up 5 minutes (healthy)   3306/tcp, 33060/tcp
l2sim-oai-amf       oaisoftwarealliance/oai-amf:v1.5.0                                 "python3 /openair-am_"   oai-amf             5 minutes ago       Up 5 minutes (healthy)   80/tcp, 9090/tcp, 38412/sctp
l2sim-oai-gnb       oaisoftwarealliance/oai-gnb:2023.w43                               "/opt/oai-gnb/bin/en_"   oai-gnb             5 minutes ago       Up 5 minutes (healthy)
l2sim-oai-nr-ue0    oaisoftwarealliance/oai-nr-ue:2023.w43                             "/opt/oai-nr-ue/bin/_"   oai-nr-ue0          4 minutes ago       Up 4 minutes (healthy)
l2sim-oai-nrf       oaisoftwarealliance/oai-nrf:v1.5.0                                 "python3 /openair-nr_"   oai-nrf             5 minutes ago       Up 5 minutes (healthy)   80/tcp, 9090/tcp
l2sim-oai-smf       oaisoftwarealliance/oai-smf:v1.5.1                                 "python3 /openair-sm_"   oai-smf             5 minutes ago       Up 5 minutes (healthy)   80/tcp, 8080/tcp, 8805/udp
l2sim-proxy         oai-lte-multi-ue-proxy:latest                                      "/oai-lte-multi-ue-p_"   proxy               4 minutes ago       Up 4 minutes (healthy)
nat                 5g_l2sim_tdd-edgecom-nat                                           "sh /app/bin/busy-po_"   edgecom-nat         5 minutes ago       Up 5 minutes
sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$

Troubleshooting:

sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ ping -I oaitun_ue1 -c 1 172.17.0.1
PING 172.17.0.1 (172.17.0.1) from 12.1.1.2 oaitun_ue1: 56(84) bytes of data.

--- 172.17.0.1 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ ping -I oaitun_ue1 -c 1 1.1.1.1
PING 1.1.1.1 (1.1.1.1) from 12.1.1.2 oaitun_ue1: 56(84) bytes of data.

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms


sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ sudo tcpdump -i br-oainfapi-n6
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on br-oainfapi-n6, link-type EN10MB (Ethernet), snapshot length 262144 bytes
14:30:04.784634 IP 12.1.1.2 > 172.17.0.1: ICMP echo request, id 16, seq 1, length 64
14:30:04.784716 IP 10.100.250.2 > 172.17.0.1: ICMP echo request, id 16, seq 1, length 64
14:30:04.784745 IP 172.17.0.1 > 10.100.250.2: ICMP echo reply, id 16, seq 1, length 64
14:30:04.784754 IP 172.17.0.1 > 12.1.1.2: ICMP echo reply, id 16, seq 1, length 64
14:30:09.931117 ARP, Request who-has 10.100.250.2 tell 10.100.250.1, length 28
14:30:09.931144 ARP, Request who-has 10.100.250.1 tell 10.100.250.2, length 28
14:30:09.931182 ARP, Reply 10.100.250.1 is-at 02:42:10:2b:4a:d7 (oui Unknown), length 28
14:30:09.931168 ARP, Reply 10.100.250.2 is-at 02:42:0a:64:fa:02 (oui Unknown), length 28
14:30:30.008260 IP 12.1.1.2 > one.one.one.one: ICMP echo request, id 17, seq 1, length 64
14:30:30.008351 IP 10.100.250.2 > one.one.one.one: ICMP echo request, id 17, seq 1, length 64
14:30:30.009622 IP one.one.one.one > 10.100.250.2: ICMP echo reply, id 17, seq 1, length 64
14:30:30.009640 IP one.one.one.one > 12.1.1.2: ICMP echo reply, id 17, seq 1, length 64
sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ sudo docker-compose exec edgecom-nat sh
/app # ip r
default via 10.100.250.1 dev eth0
10.100.250.0/24 dev eth0 scope link  src 10.100.250.2
/app # ip route add 12.1.1.0/24 via `nslookup upfn6.free5gc.org | awk '/^Address: / { print $2 }'`
/app # ip r
default via 10.100.250.1 dev eth0
10.100.250.0/24 dev eth0 scope link  src 10.100.250.2
12.1.1.0/24 via 10.100.250.3 dev eth0
/app # exit

sergo@edgecom:~/edgecom24/openairinterface5g/ci-scripts/yaml_files/5g_l2sim_tdd$ ping -I oaitun_ue1 -c 1 1.1.1.1
PING 1.1.1.1 (1.1.1.1) from 12.1.1.4 oaitun_ue1: 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=55 time=82.7 ms

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 82.736/82.736/82.736/0.000 ms

Solution will be to remove dev eth1 from nat's script command ip route add.
✔ fixed in #552

@PapaySail PapaySail linked a pull request May 23, 2024 that will close this issue
fix: deploy commit & image tags for oai-nfapi-sim-compose #552
Open
@PapaySail PapaySail mentioned this issue Jun 11, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@PapaySail PapaySail

Labels
deployment documentation Improvements or additions to documentation
Projects
None yet
Milestone
No milestone
1 participant
@PapaySail