Upgrade to Pro — share decks privately, control downloads, hide ads and more …

LINE's next-generation SDN architecture

LINE's next-generation SDN architecture

Toshiki Tsuchiya
LINE Service Network Team Infra Engineer
https://linedevday.linecorp.com/jp/2019/sessions/E1-2

LINE DevDay 2019

November 20, 2019
Tweet

More Decks by LINE DevDay 2019

Other Decks in Technology

Transcript

  1. LINE Services and Infrastructure Production Infrastructure Development Infrastructure Exclusive Infrastructure

    Common Services 
 (Messenger, Family Service, …) Fintech Services Exclusive Infrastructure
  2. LINE’s Infrastructure >Many works to design and build infrastructure >Lack

    of infrastructure flexibility >Many fragmented infrastructure Challenges
  3. Solution: Multi-Tenant Network Underlay Network Overlay Network for Service C

    Overlay Network for Service B Overlay Network for Service A Overlay Network > Service Specific Virtual Network > Flexible Network Policy > Simple & Scalable Physical Network Underlay Network
  4. Underlay Network CLOS Network > High-Capacity Network > Horizontally Scalable

    Architecture Full L3 Network > All Nodes Are Connected With BGP • Simple, Stateless Network • Reduce Operation Cost … Spine Leaf ToR Server BGP
  5. > Less users, informations > Less device support > IP

    CLOS awareness > Flexible instruction Pros Cons > Lose advance of Full-L3 underlay > Additional protocols for
 more flexible networking > More users, informations > Wider devices support Pros Cons How To Build Overlay Network? L2 Base Technology: VXLAN L3 Base Technology: SRv6 Adopted SRv6
  6. Segment Routing > Source Routing Technology > Segment: Network Device,

    Interface … • Segment ID (SID): IPv6 Address = 128bit SRv6 (IPv6 Segment Routing) A B D I/F I/F I/F I/F Segment
  7. SRv6 (IPv6 Segment Routing) A B C D X Y

    SID: A::1 SID: B::1 SID: C::1 SID: D::1 Packet
  8. SRv6 (IPv6 Segment Routing) A B C D Insert Segment

    List (Encap) X Y Packet SRH B::1,C::1,D::1 IPv6 H DA=B::1 SID: A::1 SID: B::1 SID: C::1 SID: D::1
  9. SRv6 (IPv6 Segment Routing) A B C D X Y

    SID: A::1 SID: B::1 SID: C::1 SID: D::1 Packet SRH B::1,C::1,D::1 IPv6 H DA=C::1
  10. SRv6 (IPv6 Segment Routing) A B C D X Y

    SID: A::1 SID: B::1 SID: C::1 SID: D::1 Packet SRH B::1,C::1,D::1 IPv6 H DA=D::1
  11. SRv6 (IPv6 Segment Routing) A B C D X Y

    SID: A::1 SID: B::1 SID: C::1 SID: D::1 Remove Segment List (Decap) Packet SRH B::1,C::1,D::1 IPv6 H DA=D::1
  12. Multi-Tenant Network Underlay Network Tenant B Tenant A VM VM

    VM VM Control-Plane > Data-Plane: Packet Processing > Control-Plane: Manage Tenants & Configure SRv6 Rules
  13. Network Architecture SRv6 Node Tenant IPv4 IPv6 IPv4 NFV (Firewall,

    IDS, …) CLOS Network Network Node (NN) A B Network Node (NN) A B Hypervisor1 (HV) A B VM VM Hypervisor2 (HV) A B VM VM Internet, …
  14. Network Architecture SRv6 Node Tenant IPv4 IPv6 IPv4 NFV (Firewall,

    IDS, …) CLOS Network Network Node (NN) A B Network Node (NN) A B Hypervisor1 (HV) A B VM VM Hypervisor2 (HV) A B VM VM Internet, … Linux Server Linux Server Linux Server Linux Server
  15. Network Architecture SRv6 Node Tenant IPv4 IPv6 IPv4 NFV (Firewall,

    IDS, …) Network Node (NN) A B Network Node (NN) A B Hypervisor1 (HV) A B VM VM Hypervisor2 (HV) A B VM VM Internet, … SID = NN::A SID = HV1::A SID = NN::B SID = HV2::B NN::/96 HV1::/96 HV2::/96 NN::/96
  16. Packet Flow in a Tenant NFV (Firewall, IDS, …) SRv6

    Node Tenant Network Node A B Network Node A B Hypervisor1 (HV) A B VM1 VM2 Hypervisor2 (HV) A B VM3 VM4 IPv4 IPv6 IPv4 To VM3 HV2::A Encap Decap To VM3 To VM4
  17. Packet Flow Between Tenants NFV (Firewall, IDS, …) SRv6 Node

    Tenant Network Node A B Network Node A B Hypervisor1 (HV) A B VM1 VM2 Hypervisor2 (HV) A B VM3 VM4 IPv4 IPv6 IPv4 To VM4 NN::A Encap To VM4 HV2::B To VM3 To VM3 To VM4 Decap Decap Encap To VM4
  18. > Manage Tenants on Network Node & Hypervisor > Configure

    Encap/Decap Rule to Network Node & Hypervisor Control-Plane Control-Plane NFV (Firewall, IDS, …) CLOS Network Network Node A B Network Node A B Hypervisor A B VM VM Hypervisor A B VM VM
  19. SRv6 Control-Plane Choices >ISIS >OSPF >BGP >SDN Controller LINE uses

    OpenStack as Private Cloud Controller so adopted SDN Controller
  20. Neutron SRv6 Plugin - networking-sr > ML2 Mechanism/Type Driver and

    Agent > Gateway Agent on Network Nodes > Service Plugin for New API To Add SRv6 Encap Rule Controller (Neutron) Service Plugin srv6_encap_network Type Driver srv6 Mechanism Driver mech_sr Network Node Hypervisor srgw_agent ml2 agent sr-agent
  21. Nova, Neutron Behavior - VM Create Neutron Controller Hypervisor Nova

    nova-compute neutron-agent VM TAP 3. VM Info 4. Run VM 5. Create TAP 1. Create Network 1. Create Network 2. Create VM 2. Create VM 3. VM Info 4. Run VM 5. Create TAP
  22. Nova, Neutron Behavior - VM Create Neutron Controller Hypervisor Nova

    nova-compute neutron-agent VM TAP 7. Get/Update Port Info VRF 6. Detect Tap 6. Detect Tap 7. Get/Update Port Info 8. Config Tap 9. Create VRF 10. Set SRv6 Encap/Decap Rules 8. Config Tap 9. Create VRF 10. Set SRv6 Encap/Decap Rules
  23. Packets for VM Encap/Decap on VRF Neutron Controller Hypervisor Nova

    nova-compute neutron-agent VM TAP VRF IPv4 SRv6 IPv4
  24. Set Encap Rule of Each VM Hypervisor 3 nova-compute neutron-agent

    Hypervisor 1 neutron-agent VRF 1 VM1 VM2 VRF 1 VM5 TAP Set SRv6 Encap/Decap Rule Encap: VM1, VM2 → VRF1 of Hypervisor1 Encap: VM3, VM4 → VRF1 of Hypervisor2 Encap: VM5 → VRF1 of Hypervisor3 Hypervisor 2 neutron-agent VRF 1 VM3 VM4 Encap: VM5 → VRF1 of Hypervisor3 VM1 VM2 VM3 VM4
  25. VM Configuration Summary > Communication Between VMs in the Same

    Tenant Is Possible > Next: Communication Between VM and Other Networks Hypervisor VRF 1 VM VM Hypervisor VRF 1 VM VM VRF 2 Hypervisor VRF 2 VM VM Network Node VRF 1 VRF 2 Network Node VRF 1 VRF 2
  26. Network Node Requirements: Multi Clusters Network Node VRF 1 VRF

    2 VRF 3 Network Node VRF 1 VRF 2 VRF 3 OpenStack Cluster 1 OpenStack Cluster 2 OpenStack Cluster N ɾɾɾ
  27. Network Node Requirements: Scale Hypervisor VRF 1 VM VM Network

    Node VRF 1 VRF 2 VRF 3 Network Node VRF 1 VRF 2 VRF 3 Hypervisor VRF 2 VM VM Hypervisor VRF 3 VM VM ɾɾɾ Network Node VRF 1 VRF 2 VRF 3
  28. Etcd + Agent Model Network Node VRF 1 VRF 2

    VRF 3 OpenStack Cluster 1 OpenStack Cluster 2 OpenStack Cluster N ɾɾɾ etcd Agent Network Node VRF 1 VRF 2 VRF 3 Agent
  29. Notify New Encap/Decap Rule via Etcd Network Node VRF Agent

    etcd Neutron Controller Hypervisor Nova nova-compute neutron-agent VM TAP VRF 3. Put Port Info 4. Create VRF and Set SRv6 Encap/Decap Rules 2. Get/Update Port Info 1. Detect Tap 1. Detect Tap 2. Get/Update Port Info 3. Put Port Info 4. Create VRF and Set SRv6 Encap/Decap Rules
  30. Configuration Summary Hypervisor VRF 1 VM VM Hypervisor VRF 1

    VM VM VRF 2 Hypervisor VRF 2 VM VM Network Node VRF 1 VRF 2 Network Node VRF 1 VRF 2 > Communication Between VMs in the Same Tenant Is Possible > Communication Between VM and Other Networks Is Possible
  31. > Follow the Design and Philosophy of OpenStack > Keep

    It Simple Without Complicated Logic > Loose Coupling of Data-Plane and Control-Plane Control-Plane Design Policy
  32. > Architecture Improvement • Service Chaining Future Plan NFV (Firewall,

    IDS, …) Network Node (NN) A B Hypervisor2 (HV) A B VM3 VM4 VM Hypervisor1 (HV) A B VM2 VM1 NFV (Function Pool) Firewall (VM) IDS (VM) … IPv4 IPv6 IPv4
  33. > Architecture Improvement • Service Chaining Future Plan Network Node

    (NN) A B Hypervisor2 (HV) A B VM3 VM4 VM Hypervisor1 (HV) A B VM2 VM1 NFV (Function Pool) IPv4 IPv6 IPv4 Packet [to VM4] SRH NFV::FW, HV4::B Firewall (VM) IDS (VM) …