Table of Content

Introduction

The Internet Protocol provides a best effortd datagram service between two end systems on the Internet. In between the two end systems, there are forwarding nodes that we often call routers. The end systems and these forwarding nodes forwards IP packets from one linke (the incoming link) to another link (the outgoing link) independently from each other based on the information in the IP packet header, specifically, the source and the destination IP addresses and a forwarding table on the forwarding node (including the end systems) that we often call a routing table depending on how we present it.

Internetworks for Experiments

To understand this concept, we design internetworks with four Linux virtual machines with two different topologies, and observe packet forwarding on the internetworks. For ease of discussion, we name the four Linux virtual machines, as bushwick, eastny flatbush, and midwood, and we call the two topologies the diamond topology and the line topology.

Diamond topology

Line (or linear) topology

Setting up Diamond and Line Ethernets

  1. We add two Ethernet NICs to each of the four Linux virtual machines and connect one of the two NICs to the same Ethernet that we call the “DiamondEthernet” VirtualBox Internal Networks, and the other “LineEthernet”. We can complete this via the VirtualBox GUI interface or via the following commands on the command line. However, since you can only access 4 of the 8 NICs a VM can have via VirtualBox’s GUI, you cannot have both the diamond Ethernets and the line Ethernets simultaneously.

      #!/bin/bash -x

  vms=(Bushwick EastNewYork Flatbush Midwood)
  for vm in "${vms[@]}"; do
          vboxmanage controlvm DebianLnx${vm} poweroff
  done

  # Diamond Ethernets
  # DiamondMidwoodFlatbush (3 - 3)
  vboxmanage modifyvm DebianLnxMidwood --nic3 intnet 
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxMidwood --intnet3 "DiamondMidwoodFlatbush"
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxFlatbush --nic3 intnet
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxFlatbush --intnet3 "DiamondMidwoodFlatbush"
  [ $? -eq 0 ] || exit $?

  # DiamondMidwoodBushwick (4 - 3)
  vboxmanage modifyvm DebianLnxMidwood --nic4 intnet 
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxMidwood --intnet4 "DiamondMidwoodBushwick"
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxBushwick --nic3 intnet
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxBushwick --intnet3 "DiamondMidwoodBushwick"
  [ $? -eq 0 ] || exit $?
  # DiamondFlatbushEastNY (4 - 3)
  vboxmanage modifyvm DebianLnxFlatbush --nic4 intnet
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxFlatbush --intnet4 "DiamondFlatbushEastNY"
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxEastNewYork --nic3 intnet
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxEastNewYork --intnet3 "DiamondFlatbushEastNY"
  [ $? -eq 0 ] || exit $?
  # DiamondBushwickEastNY (4 - 4)
  vboxmanage modifyvm DebianLnxBushwick --nic4 intnet
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxBushwick --intnet4 "DiamondBushwickEastNY"
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxEastNewYork --nic4 intnet
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxEastNewYork --intnet4 "DiamondBushwickEastNY"
  [ $? -eq 0 ] || exit $?


  # Line Ethernets
  # LineMidwoodFlatbush (5 - 5)
  vboxmanage modifyvm DebianLnxMidwood --nic5 intnet 
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxMidwood --intnet5 "LineMidwoodFlatbush"
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxFlatbush --nic5 intnet
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxFlatbush --intnet5 "LineMidwoodFlatbush"
  [ $? -eq 0 ] || exit $?
  # LineFlatbushBushwick (6 - 5)
  vboxmanage modifyvm DebianLnxFlatbush --nic6 intnet
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxFlatbush --intnet6 "LineFlatbushBushwick"
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxBushwick --nic5 intnet
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxBushwick --intnet5 "LineFlatbushBushwick"
  [ $? -eq 0 ] || exit $?
  # LineBushwickEastNY (6 - 5)
  vboxmanage modifyvm DebianLnxBushwick --nic6 intnet
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxBushwick --intnet6 "LineBushwickEastNY"
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxEastNewYork --nic5 intnet
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxEastNewYork --intnet5 "LineBushwickEastNY"
  [ $? -eq 0 ] || exit $?


  for vm in "${vms[@]}"; do
          vboxmanage startvm DebianLnx${vm} --type headless
  done

    If you insist on using GUI to set up this, you can only set up one internetwork at a time. In the GUI, make sure you choose right adapter, select “internal network” as “Attach to”, select or enter the correct Ethernet name to attach to.

Example of Virtual Box Networking Setup for a VM via GUI

Verifying Diamond and Line Ethernets Setup

We need to configure 8 NICs on 4 VM s to set up 4 Ethernets for the Diamond ethernets, and 6 NICs on 4 VMs for the Line ethernets. It is easy to make a mistake during the process. It is important to verify the Ethernet setup. This takes a few steps.

  1. Query Ethernet NIC information. For instance, on VM Flatbush, we run the following,

      brooklyn@flatbush:~$ ip link show
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
          link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:45:bf:a2 brd ff:ff:ff:ff:ff:ff
  3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:59:ef:a7 brd ff:ff:ff:ff:ff:ff
  4: enp0s9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
          link/ether 08:00:27:cb:67:1d brd ff:ff:ff:ff:ff:ff
  5: enp0s10: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
          link/ether 08:00:27:29:17:39 brd ff:ff:ff:ff:ff:ff
  6: enp0s16: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
          link/ether 08:00:27:e9:34:5f brd ff:ff:ff:ff:ff:ff
  7: enp0s17: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
          link/ether 08:00:27:d5:de:96 brd ff:ff:ff:ff:ff:ff
  brooklyn@flatbush:~$

    Similarly, on VM Midwood, we do

      brooklyn@midwood:~$ ip link show
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
          link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:5a:ff:ec brd ff:ff:ff:ff:ff:ff
  3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:a1:33:b4 brd ff:ff:ff:ff:ff:ff
  4: enp0s9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
          link/ether 08:00:27:08:0d:a1 brd ff:ff:ff:ff:ff:ff
  5: enp0s10: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
          link/ether 08:00:27:a2:b4:f0 brd ff:ff:ff:ff:ff:ff
  6: enp0s16: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
          link/ether 08:00:27:a3:d1:e4 brd ff:ff:ff:ff:ff:ff
  brooklyn@midwood:~$

  2. We now need to verify that one NIC on VM midwood and one on Flatbush on the DiamondMidwoodFlatbush Ethernet as designed.

    In VirtualBox we can identify a NIC by a number or a name like “nic1” or “adapter1”, but a VM names each NIC indepently. Because of this, we use a NIC’s Ethernet address to identify it.

    For instance, from Virtubox’s UI or from the command line on the host, we can retrieve the Ethernet address of the NIC tapped on the DiamondMidwoodFlatbush Ethernet on VM midwood,

      $ vboxmanage showvminfo DebianLnxMidwood | grep DiamondMidwoodFlatbush
  NIC 3:                       MAC: 080027080DA1, Attachment: Internal Network 'Di
  amondMidwoodFlatbush', Cable connected: on, Trace: off (file: none), Type: 82540
  EM, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: deny, Bandwidth gr
  oup: none

    where it shows that the Ethernet’s address of NIC 3 is 080027080DA1. Comparing with the information of the NICs on VM midwood queried in Step 1, we see that NIC enp0s9 has the same address (shown as 08:00:27:08:0d:a1).

    Similarly, from Virtubox’s UI or from the command line on the host, we can the Ethernet address of the NIC tapped on the DiamondMidwoodFlatbush Ethernet on VM Flatbush,

      $ vboxmanage showvminfo DebianLnxFlatbush | grep DiamondMidwoodFlatbush
  NIC 3:                       MAC: 080027CB671D, Attachment: Internal Network 'Di
  amondMidwoodFlatbush', Cable connected: on, Trace: off (file: none), Type: 82540
  EM, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: deny, Bandwidth gr
  oup: none

    Likewise, we observe that the Ethernet address of VM Flatbush’s NIC enp0s9 matches NIC 3’s address that we retrieved on the host from VirtualBox in Step 1.

    We should examine every Ethernet (DiamondMidwoodFlatbush, DiamondMidwoodBushwick, and …, i.e., 4 Diamond Ethernets and 3 Line Ethernets) to make sure we have set them up correctly.

  3. To be sure, we now transmit Ethernet frames from a NIC on an Ethernet we set up. Conceptually, we can view an Ethernet as a physical link that NICs can tap on it. When we transmit an Ethernet frame via one NIC on an Ethernet, any other NICs tapped on the same Ethernet should “see” the frame. However, any other NICs that aren’t on the Ethernet won’t “see” it.

    For instance, in the following, we transmit an Ethernet frame via NIC enp0s9 on midwood. Since the only other NIC among all the NICs of the 4 VMs is also on the same Etherneet on which midwood’s enp0s9 is tapped is enp0s9 on Flatbush. We can only capture this frame on Flatbush from the very NIC (enp0s9), not from any other NIC on Flatbush, and not from any NICs (other than enp0s9 on Flatbush and enp0s9 on Midwood) on other VMs.

    The following steps are to do and to verify just that. Notice that we do this step on 3 VMs, midwood, flatbush, and bushwick.

    1. Prepare flatbush for sniffing packets.

        brooklyn@flatbush:~$ hostname
  flatbush
  brooklyn@flatbush:~$ ip link show
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
          link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:45:bf:a2 brd ff:ff:ff:ff:ff:ff
  3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:59:ef:a7 brd ff:ff:ff:ff:ff:ff
  4: enp0s9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:cb:67:1d brd ff:ff:ff:ff:ff:ff
  5: enp0s10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:29:17:39 brd ff:ff:ff:ff:ff:ff
  6: enp0s16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:e9:34:5f brd ff:ff:ff:ff:ff:ff
  7: enp0s17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:d5:de:96 brd ff:ff:ff:ff:ff:ff
  brooklyn@flatbush:~$ sudo ip link set enp0s9 up
  brooklyn@flatbush:~$ sudo ip link set enp0s10 up
  brooklyn@flatbush:~$ sudo ip link set enp0s16 up
  brooklyn@flatbush:~$ sudo ip link set enp0s17 up
  brooklyn@flatbush:~$ sudo scapy3
  >>> get_if_list()
  ['enp0s17', 'enp0s10', 'enp0s9', 'enp0s3', 'enp0s8', 'lo', 'enp0s16']
  >>> packets = sniff(prn=lambda p:p.summary(), filter='ether host 08:00:27:cb:67:
  ...:1d', iface='enp0s9')

    2. Prepare bushwick for sniffing packets.

        brooklyn@bushwick:~$ hostname
  bushwick
  brooklyn@bushwick:~$ ip link show
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
          link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:47:cb:70 brd ff:ff:ff:ff:ff:ff
  3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:4c:b6:a8 brd ff:ff:ff:ff:ff:ff
  4: enp0s9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:3d:c4:e4 brd ff:ff:ff:ff:ff:ff
  5: enp0s10: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
          link/ether 08:00:27:6f:89:e6 brd ff:ff:ff:ff:ff:ff
  6: enp0s16: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
          link/ether 08:00:27:70:e7:16 brd ff:ff:ff:ff:ff:ff
  7: enp0s17: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
          link/ether 08:00:27:53:90:ff brd ff:ff:ff:ff:ff:ff
  brooklyn@bushwick:~$ sudo ip link set enp0s9 up
  brooklyn@bushwick:~$ sudo ip link set enp0s10 up
  brooklyn@bushwick:~$ sudo ip link set enp0s16 up
  brooklyn@bushwick:~$ sudo ip link set enp0s17 up
  brooklyn@bushwick:~$ sudo scapy3
  >>> packets = sniff(prn=lambda p:p.summary(), filter='ether host 08:00:27:cb:67: ...:1d')

    3. Prepare eastny for sniffing packets.

        brooklyn@eastny:~$ hostname
  eastny
  brooklyn@eastny:~$ sudo ip link show
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
          link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:17:a1:40 brd ff:ff:ff:ff:ff:ff
  3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:ee:90:20 brd ff:ff:ff:ff:ff:ff
  4: enp0s9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
          link/ether 08:00:27:1e:20:a8 brd ff:ff:ff:ff:ff:ff
  5: enp0s10: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
          link/ether 08:00:27:34:ea:d3 brd ff:ff:ff:ff:ff:ff
  6: enp0s16: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
          link/ether 08:00:27:d4:f5:25 brd ff:ff:ff:ff:ff:ff
  brooklyn@eastny:~$ sudo ip link set enp0s9 up
  brooklyn@eastny:~$ sudo ip link set enp0s10 up
  brooklyn@eastny:~$ sudo ip link set enp0s16 up
  brooklyn@eastny:~$ sudo scapy3
  >>> packets = sniff(prn=lambda p:p.summary(), filter='ether host 08:00:27:cb:67:
  ...:1d')

    4. Prepare midwood for sending packets.

      brooklyn@midwood:~$ hostname
midwood
brooklyn@midwood:~$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
        link/ether 08:00:27:5a:ff:ec brd ff:ff:ff:ff:ff:ff
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
        link/ether 08:00:27:a1:33:b4 brd ff:ff:ff:ff:ff:ff
4: enp0s9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
        link/ether 08:00:27:08:0d:a1 brd ff:ff:ff:ff:ff:ff
5: enp0s10: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
        link/ether 08:00:27:a2:b4:f0 brd ff:ff:ff:ff:ff:ff
6: enp0s16: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
        link/ether 08:00:27:a3:d1:e4 brd ff:ff:ff:ff:ff:ff
brooklyn@midwood:~$ sudo ip link set enp0s9 up
brooklyn@midwood:~$ sudo ip link set enp0s10 up
brooklyn@midwood:~$ sudo ip link set enp0s16 up
brooklyn@midwood:~$ sudo scapy3
>>> get_if_list()
['enp0s8', 'enp0s16', 'enp0s10', 'enp0s9', 'lo', 'enp0s3']
>>> get_if_hwaddr(enp0s9)
'08:00:27:08:0d:a1'
>>> packet = Ether()/'Hello, Flatbush. I am tapping on DiamondMidwoodFlatbush'
>>> packet.dst = '08:00:27:cb:67:1d'
>>> packet.src = '08:00:27:08:0d:a1'
>>> hexdump(packet)
0000  080027CB671D080027080DA190004865 ..'.g...'.....He
0010  6C6C6F2C20466C6174627573682E2049 llo, Flatbush. I
0020  20616D2074617070696E67206F6E2044  am tapping on D
0030  69616D6F6E644D6964776F6F64466C61 iamondMidwoodFla
0040  7462757368                       tbush
>>> sendp(packet, iface='enp0s9')
.
Sent 1 packets.
>>>

    5. Return to flatbush, we should observe,

      >>> packets = sniff(prn=lambda p:p.summary(), filter='ether host 08:00:27:cb:67:
...:1d', iface='enp0s9')
08:00:27:08:0d:a1 > 08:00:27:cb:67:1d (0x9000) / Raw
^C>>> hexdump(packets[0])
0000  080027CB671D080027080DA190004865 ..'.g...'.....He
0010  6C6C6F2C20466C6174627573682E2049 llo, Flatbush. I
0020  20616D2074617070696E67206F6E2044  am tapping on D
0030  69616D6F6E644D6964776F6F64466C61 iamondMidwoodFla
0040  7462757368                       tbush
>>>
      1. On other two VMs, bushwick and eastny, we should observe that we don’t see those packets.

    6. A few variants of packet sniffing scenario provide how these physical links work including capturing packets from “all” interface and a designated interface other than enp0s9, e.g.,

          >>> packets = sniff(prn=lambda p:p.summary(), filter='ether host 08:00:27:cb:67:
    ...:1d')
    08:00:27:08:0d:a1 > 08:00:27:cb:67:1d (0x9000) / Raw
    >>> packets = sniff(prn=lambda p:p.summary(), filter='ether host 08:00:27:cb:67:
    ...:1d', iface='enp0s10')

    which shows the packet indeed arrives at the NIC tapped on the
    ethernet because we would not see such a packet if we only
    capture traffic comes to a NIC tapped on a different Ethernet.

Setting up Diamond IP Network

It is important to remember all of the setup below wouldn’t survive a reboot. A prudent approach would be to save each command in a shell script, and you can easily repeat without retyping them.

Perhaps, a related question is how we configure the host so that the setup via the length commands would survive the reboot. Appendix I provides a gist of of the answer to the question.

It is also important to note that this section does not contain all the commands to set up the Diamond IP network as we envisioned completely. But it gives you all the ingredients you need to finish “cooking” by yourself.

  1. We plan to deploy the following IP networks to the Diamond Ethernets.

    Ethernet IPv4 Network IPv4 Broadcast Address
    DiamondMidwoodFlatbush 10.1.1.0/28 10.1.1.15
    DiamondMidwoodbushwick 10.1.1.16/28 10.1.1.31
    DiamondFlatbushEastNY 10.1.1.32/28 10.1.1.47
    DiamondBushwickEastNY 10.1.1.48/28 10.1.1.63

    Following the above plan, we have the following IPv4 address assignments to the NICs on the four VMs.

    Host name NIC:IP Address Linux Command
    midwood enp0s9: 10.1.1.1 ip address add 10.1.1.1/28 broadcast 10.1.1.15 dev enp0s9
    midwood enp0s10: 10.1.1.17 ip address add 10.1.1.17/28 broadcast 10.1.1.31 dev enp0s10
    flatbush enp0s9: 10.1.1.2 ip address add 10.1.1.2/28 broadcast 10.1.1.15 dev enp0s9
    flatbush enp0s10: 10.1.1.33 ip address add 10.1.1.33/28 broadcast 10.1.1.47 dev enp0s10
    bushwick enp0s9: 10.1.1.18  
    bushwick enp0s10: 10.1.1.49  
    eastny enp0s9: 10.1.1.34 ip address add 10.1.1.34/28 broadcast 10.1.1.47 dev enp0s9
    eastny enp0s10: 10.1.1.50 ip address add 10.1.1.50/28 broadcast 10.1.1.63 dev enp0s10

  2. We realize these using the commands in the able on the VMs, for instance, on midwood, as follows,

      brooklyn@midwood:~$ sudo ip address add 10.1.1.1/28 broadcast 10.1.1.15 dev enp0s9
  brooklyn@midwood:~$ sudo ip address add 10.1.1.17/28 broadcast 10.1.1.31 dev enp0s10
  brooklyn@midwood:~$ ip address show
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
          link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
          inet 127.0.0.1/8 scope host lo
                 valid_lft forever preferred_lft forever
          inet6 ::1/128 scope host
                 valid_lft forever preferred_lft forever
  2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
          link/ether 08:00:27:5a:ff:ec brd ff:ff:ff:ff:ff:ff
          inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
                 valid_lft 83644sec preferred_lft 83644sec
          inet6 fe80::a00:27ff:fe5a:ffec/64 scope link
                 valid_lft forever preferred_lft forever
  3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
          link/ether 08:00:27:a1:33:b4 brd ff:ff:ff:ff:ff:ff
          inet 192.168.56.103/24 brd 192.168.56.255 scope global dynamic enp0s8
                 valid_lft 591sec preferred_lft 591sec
          inet6 fe80::a00:27ff:fea1:33b4/64 scope link
                 valid_lft forever preferred_lft forever
  4: enp0s9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
          link/ether 08:00:27:08:0d:a1 brd ff:ff:ff:ff:ff:ff
          inet 10.1.1.1/28 brd 10.1.1.15 scope global enp0s9
                 valid_lft forever preferred_lft forever
          inet6 fe80::a00:27ff:fe08:da1/64 scope link
                 valid_lft forever preferred_lft forever
  5: enp0s10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
          link/ether 08:00:27:a2:b4:f0 brd ff:ff:ff:ff:ff:ff
          inet 10.1.1.17/28 brd 10.1.1.31 scope global enp0s10
                 valid_lft forever preferred_lft forever
          inet6 fe80::a00:27ff:fea2:b4f0/64 scope link
                 valid_lft forever preferred_lft forever
  6: enp0s16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
          link/ether 08:00:27:a3:d1:e4 brd ff:ff:ff:ff:ff:ff
          inet6 fe80::a00:27ff:fea3:d1e4/64 scope link
                 valid_lft forever preferred_lft forever
  brooklyn@midwood:~$

  3. For a packet originated at midwood to reach eastny (2 hops away), we need to configure flatbush to act as a forwarding node.

      brooklyn@flatbush:~$ sudo sysctl -w net.ipv4.ip_forward=1
  net.ipv4.ip_forward = 1
  brooklyn@flatbush:~$

  4. We also need to update the forwarding table at midwood, and the forwarding table at eastny. Without the former, midwood does not know where it should transmit a packet whose destination is on eastny. Without the later, easny does not know where to transmit a reply.

    1. At midwood, we do

        sudo ip route add 10.1.1.32/28 via 10.1.1.2

    2. At eastny, we do

        sudo ip route add 10.1.1.0/28 via 10.1.1.33

  5. It’s the time to see we can send an IP packet from midwood to eastny and get a reply from it. So at midwood, we do

      brooklyn@midwood:~$ sudo scapy3
  >>> reply = sr1(IP(dst='10.1.1.34')/ICMP())
  Begin emission:
  ..Finished sending 1 packets.
  .*
  Received 4 packets, got 1 answers, remaining 0 packets
  >>> reply
  <IP  version=4 ihl=5 tos=0x0 len=28 id=36717 flags= frag=0 ttl=63 proto=icmp chksum=0xd64f src=10.1.1.34 dst=10.1.1.1 options=[] |<ICMP  type=echo-reply code=0 chksum=0xffff id=0x0 seq=0x0 |<Padding  load='\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' |>>>
  >>> reply.summary()
  'IP / ICMP 10.1.1.34 > 10.1.1.1 echo-reply 0 / Padding'
  >>>

Experiments and Demonstration

Setting up Diamond or Line IP Network

This is your exercise and exploration. Review the steps we carry out for the Diamond IP network, and repeat them for the Line IP network.

Observing IP Forwarding

With these two networks, we can observe how IP forwarding works. To view the forwarding table, for instance, on host midwood, we do,

ip route show

Appendix I. Network Configuration Files

On Debian Linux, we can configure host with the steps outlined in the above in a configuration file, and the configuration file would come into effect when the system boots.

Enabling or Disabling IPv4 Forwarding

In /etc/sysct.conf, uncomment to enable or comment the following line to disable IPv4 forwarding,

#net.ipv4.ip_forward=1

Setting up IP Address and Routes

In /etc/network/interfaces file, add a section of configuration code for each network interface you wish to set up, e.g., for enps09 on midwood, we may append the following lines to it,

auto enps09
iface enps09 inet static
  address 10.1.1.1/28
  post−up ip route add 10.1.1.32/28 via 10.1.1.2 
  pre−down ip route del 10.1.1.32/28 via 10.1.1.2