Table of Content

Introduction

The Internet Protocol provides a best effortd datagram service between two end systems on the Internet. In between the two end systems, there are forwarding nodes that we often call routers. The end systems and these forwarding nodes forwards IP packets from one linke (the incoming link) to another link (the outgoing link) independently from each other based on the information in the IP packet header, specifically, the source and the destination IP addresses and a forwarding table on the forwarding node (including the end systems) that we often call a routing table depending on how we present it.

Internetworks for Experiments

To understand this concept, we design internetworks with four Linux virtual machines with two different topologies, and observe packet forwarding on the internetworks. For ease of discussion, we name the four Linux virtual machines, as bushwick, eastny flatbush, and midwood, and we call the two topologies the diamond topology and the line topology.

Diamond topology

Line (or linear) topology

Setting up Diamond and Line Ethernets

  1. We add two Ethernet NICs to each of the four Linux virtual machines and connect one of the two NICs to the same Ethernet that we call the “DiamondEthernet” VirtualBox Internal Networks, and the other “LineEthernet”. We can complete this via the VirtualBox GUI interface or via the following commands on the command line. However, since you can only access 4 of the 8 NICs a VM can have via VirtualBox’s GUI, you cannot have both the diamond Ethernets and the line Ethernets simultaneously.

      #!/bin/bash -x

  vms=(Bushwick EastNewYork Flatbush Midwood)
  for vm in "${vms[@]}"; do
          vboxmanage controlvm DebianLnx${vm} poweroff
  done

  # Diamond Ethernets
  # DiamondMidwoodFlatbush (3 - 3)
  vboxmanage modifyvm DebianLnxMidwood --nic3 intnet 
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxMidwood --intnet3 "DiamondMidwoodFlatbush"
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxFlatbush --nic3 intnet
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxFlatbush --intnet3 "DiamondMidwoodFlatbush"
  [ $? -eq 0 ] || exit $?

  # DiamondMidwoodBushwick (4 - 3)
  vboxmanage modifyvm DebianLnxMidwood --nic4 intnet 
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxMidwood --intnet4 "DiamondMidwoodBushwick"
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxBushwick --nic3 intnet
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxBushwick --intnet3 "DiamondMidwoodBushwick"
  [ $? -eq 0 ] || exit $?
  # DiamondFlatbushEastNY (4 - 3)
  vboxmanage modifyvm DebianLnxFlatbush --nic4 intnet
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxFlatbush --intnet4 "DiamondFlatbushEastNY"
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxEastNewYork --nic3 intnet
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxEastNewYork --intnet3 "DiamondFlatbushEastNY"
  [ $? -eq 0 ] || exit $?
  # DiamondBushwickEastNY (4 - 4)
  vboxmanage modifyvm DebianLnxBushwick --nic4 intnet
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxBushwick --intnet4 "DiamondBushwickEastNY"
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxEastNewYork --nic4 intnet
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxEastNewYork --intnet4 "DiamondBushwickEastNY"
  [ $? -eq 0 ] || exit $?


  # Line Ethernets
  # LineMidwoodFlatbush (5 - 5)
  vboxmanage modifyvm DebianLnxMidwood --nic5 intnet 
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxMidwood --intnet5 "LineMidwoodFlatbush"
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxFlatbush --nic5 intnet
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxFlatbush --intnet5 "LineMidwoodFlatbush"
  [ $? -eq 0 ] || exit $?
  # LineFlatbushBushwick (6 - 5)
  vboxmanage modifyvm DebianLnxFlatbush --nic6 intnet
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxFlatbush --intnet6 "LineFlatbushBushwick"
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxBushwick --nic5 intnet
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxBushwick --intnet5 "LineFlatbushBushwick"
  [ $? -eq 0 ] || exit $?
  # LineBushwickEastNY (6 - 5)
  vboxmanage modifyvm DebianLnxBushwick --nic6 intnet
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxBushwick --intnet6 "LineBushwickEastNY"
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxEastNewYork --nic5 intnet
  [ $? -eq 0 ] || exit $?
  vboxmanage modifyvm DebianLnxEastNewYork --intnet5 "LineBushwickEastNY"
  [ $? -eq 0 ] || exit $?


  for vm in "${vms[@]}"; do
          vboxmanage startvm DebianLnx${vm} --type headless
  done

    If you insist on using GUI to set up this, you can only set up one internetwork at a time. In the GUI, make sure you choose right adapter, select “internal network” as “Attach to”, select or enter the correct Ethernet name to attach to.

Example of Virtual Box Networking Setup for a VM via GUI

Verifying Diamond and Line Ethernets Setup

We need to configure 8 NICs on 4 VM s to set up 4 Ethernets for the Diamond ethernets, and 6 NICs on 4 VMs for the Line ethernets. It is easy to make a mistake during the process. It is important to verify the Ethernet setup. This takes a few steps.

  1. Query Ethernet NIC information. For instance, on VM Flatbush, we run the following,

      brooklyn@flatbush:~$ ip link show
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
          link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:45:bf:a2 brd ff:ff:ff:ff:ff:ff
  3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:59:ef:a7 brd ff:ff:ff:ff:ff:ff
  4: enp0s9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
          link/ether 08:00:27:cb:67:1d brd ff:ff:ff:ff:ff:ff
  5: enp0s10: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
          link/ether 08:00:27:29:17:39 brd ff:ff:ff:ff:ff:ff
  6: enp0s16: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
          link/ether 08:00:27:e9:34:5f brd ff:ff:ff:ff:ff:ff
  7: enp0s17: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
          link/ether 08:00:27:d5:de:96 brd ff:ff:ff:ff:ff:ff
  brooklyn@flatbush:~$

    Similarly, on VM Midwood, we do

      brooklyn@midwood:~$ ip link show
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
          link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:5a:ff:ec brd ff:ff:ff:ff:ff:ff
  3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:a1:33:b4 brd ff:ff:ff:ff:ff:ff
  4: enp0s9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
          link/ether 08:00:27:08:0d:a1 brd ff:ff:ff:ff:ff:ff
  5: enp0s10: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
          link/ether 08:00:27:a2:b4:f0 brd ff:ff:ff:ff:ff:ff
  6: enp0s16: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
          link/ether 08:00:27:a3:d1:e4 brd ff:ff:ff:ff:ff:ff
  brooklyn@midwood:~$

  2. We now need to verify that one NIC on VM midwood and one on Flatbush on the DiamondMidwoodFlatbush Ethernet as designed.

    In VirtualBox we can identify a NIC by a number or a name like “nic1” or “adapter1”, but a VM names each NIC indepently. Because of this, we use a NIC’s Ethernet address to identify it.

    For instance, from Virtubox’s UI or from the command line on the host, we can retrieve the Ethernet address of the NIC tapped on the DiamondMidwoodFlatbush Ethernet on VM midwood,

      $ vboxmanage showvminfo DebianLnxMidwood | grep DiamondMidwoodFlatbush
  NIC 3:                       MAC: 080027080DA1, Attachment: Internal Network 'Di
  amondMidwoodFlatbush', Cable connected: on, Trace: off (file: none), Type: 82540
  EM, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: deny, Bandwidth gr
  oup: none

    where it shows that the Ethernet’s address of NIC 3 is 080027080DA1. Comparing with the information of the NICs on VM midwood queried in Step 1, we see that NIC enp0s9 has the same address (shown as 08:00:27:08:0d:a1).

    Similarly, from Virtubox’s UI or from the command line on the host, we can the Ethernet address of the NIC tapped on the DiamondMidwoodFlatbush Ethernet on VM Flatbush,

      $ vboxmanage showvminfo DebianLnxFlatbush | grep DiamondMidwoodFlatbush
  NIC 3:                       MAC: 080027CB671D, Attachment: Internal Network 'Di
  amondMidwoodFlatbush', Cable connected: on, Trace: off (file: none), Type: 82540
  EM, Reported speed: 0 Mbps, Boot priority: 0, Promisc Policy: deny, Bandwidth gr
  oup: none

    Likewise, we observe that the Ethernet address of VM Flatbush’s NIC enp0s9 matches NIC 3’s address that we retrieved on the host from VirtualBox in Step 1.

    We should examine every Ethernet (DiamondMidwoodFlatbush, DiamondMidwoodBushwick, and …, i.e., 4 Diamond Ethernets and 3 Line Ethernets) to make sure we have set them up correctly.

  3. To be sure, we now transmit Ethernet frames from a NIC on an Ethernet we set up. Conceptually, we can view an Ethernet as a physical link that NICs can tap on it. When we transmit an Ethernet frame via one NIC on an Ethernet, any other NICs tapped on the same Ethernet should “see” the frame. However, any other NICs that aren’t on the Ethernet won’t “see” it.

    For instance, in the following, we transmit an Ethernet frame via NIC enp0s9 on midwood. Since the only other NIC among all the NICs of the 4 VMs is also on the same Etherneet on which midwood’s enp0s9 is tapped is enp0s9 on Flatbush. We can only capture this frame on Flatbush from the very NIC (enp0s9), not from any other NIC on Flatbush, and not from any NICs (other than enp0s9 on Flatbush and enp0s9 on Midwood) on other VMs.

    The following steps are to do and to verify just that. Notice that we do this step on 3 VMs, midwood, flatbush, and bushwick.

    1. Prepare flatbush for sniffing packets.

        brooklyn@flatbush:~$ hostname
  flatbush
  brooklyn@flatbush:~$ ip link show
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
          link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:45:bf:a2 brd ff:ff:ff:ff:ff:ff
  3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:59:ef:a7 brd ff:ff:ff:ff:ff:ff
  4: enp0s9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:cb:67:1d brd ff:ff:ff:ff:ff:ff
  5: enp0s10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:29:17:39 brd ff:ff:ff:ff:ff:ff
  6: enp0s16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:e9:34:5f brd ff:ff:ff:ff:ff:ff
  7: enp0s17: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:d5:de:96 brd ff:ff:ff:ff:ff:ff
  brooklyn@flatbush:~$ sudo ip link set enp0s9 up
  brooklyn@flatbush:~$ sudo ip link set enp0s10 up
  brooklyn@flatbush:~$ sudo ip link set enp0s16 up
  brooklyn@flatbush:~$ sudo ip link set enp0s17 up
  brooklyn@flatbush:~$ sudo scapy3
  >>> get_if_list()
  ['enp0s17', 'enp0s10', 'enp0s9', 'enp0s3', 'enp0s8', 'lo', 'enp0s16']
  >>> packets = sniff(prn=lambda p:p.summary(), filter='ether host 08:00:27:cb:67:
  ...:1d', iface='enp0s9')

    2. Prepare bushwick for sniffing packets.

        brooklyn@bushwick:~$ hostname
  bushwick
  brooklyn@bushwick:~$ ip link show
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
          link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:47:cb:70 brd ff:ff:ff:ff:ff:ff
  3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:4c:b6:a8 brd ff:ff:ff:ff:ff:ff
  4: enp0s9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:3d:c4:e4 brd ff:ff:ff:ff:ff:ff
  5: enp0s10: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
          link/ether 08:00:27:6f:89:e6 brd ff:ff:ff:ff:ff:ff
  6: enp0s16: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
          link/ether 08:00:27:70:e7:16 brd ff:ff:ff:ff:ff:ff
  7: enp0s17: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
          link/ether 08:00:27:53:90:ff brd ff:ff:ff:ff:ff:ff
  brooklyn@bushwick:~$ sudo ip link set enp0s9 up
  brooklyn@bushwick:~$ sudo ip link set enp0s10 up
  brooklyn@bushwick:~$ sudo ip link set enp0s16 up
  brooklyn@bushwick:~$ sudo ip link set enp0s17 up
  brooklyn@bushwick:~$ sudo scapy3
  >>> packets = sniff(prn=lambda p:p.summary(), filter='ether host 08:00:27:cb:67: ...:1d')

    3. Prepare eastny for sniffing packets.

        brooklyn@eastny:~$ hostname
  eastny
  brooklyn@eastny:~$ sudo ip link show
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
          link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
  2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:17:a1:40 brd ff:ff:ff:ff:ff:ff
  3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
          link/ether 08:00:27:ee:90:20 brd ff:ff:ff:ff:ff:ff
  4: enp0s9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
          link/ether 08:00:27:1e:20:a8 brd ff:ff:ff:ff:ff:ff
  5: enp0s10: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
          link/ether 08:00:27:34:ea:d3 brd ff:ff:ff:ff:ff:ff
  6: enp0s16: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
          link/ether 08:00:27:d4:f5:25 brd ff:ff:ff:ff:ff:ff
  brooklyn@eastny:~$ sudo ip link set enp0s9 up
  brooklyn@eastny:~$ sudo ip link set enp0s10 up
  brooklyn@eastny:~$ sudo ip link set enp0s16 up
  brooklyn@eastny:~$ sudo scapy3
  >>> packets = sniff(prn=lambda p:p.summary(), filter='ether host 08:00:27:cb:67:
  ...:1d')

    4. Prepare midwood for sending packets.

      brooklyn@midwood:~$ hostname
midwood
brooklyn@midwood:~$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
        link/ether 08:00:27:5a:ff:ec brd ff:ff:ff:ff:ff:ff
3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
        link/ether 08:00:27:a1:33:b4 brd ff:ff:ff:ff:ff:ff
4: enp0s9: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
        link/ether 08:00:27:08:0d:a1 brd ff:ff:ff:ff:ff:ff
5: enp0s10: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
        link/ether 08:00:27:a2:b4:f0 brd ff:ff:ff:ff:ff:ff
6: enp0s16: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
        link/ether 08:00:27:a3:d1:e4 brd ff:ff:ff:ff:ff:ff
brooklyn@midwood:~$ sudo ip link set enp0s9 up
brooklyn@midwood:~$ sudo ip link set enp0s10 up
brooklyn@midwood:~$ sudo ip link set enp0s16 up
brooklyn@midwood:~$ sudo scapy3
>>> get_if_list()
['enp0s8', 'enp0s16', 'enp0s10', 'enp0s9', 'lo', 'enp0s3']
>>> get_if_hwaddr(enp0s9)
'08:00:27:08:0d:a1'
>>> packet = Ether()/'Hello, Flatbush. I am tapping on DiamondMidwoodFlatbush'
>>> packet.dst = '08:00:27:cb:67:1d'
>>> packet.src = '08:00:27:08:0d:a1'
>>> hexdump(packet)
0000  080027CB671D080027080DA190004865 ..'.g...'.....He
0010  6C6C6F2C20466C6174627573682E2049 llo, Flatbush. I
0020  20616D2074617070696E67206F6E2044  am tapping on D
0030  69616D6F6E644D6964776F6F64466C61 iamondMidwoodFla
0040  7462757368                       tbush
>>> sendp(packet, iface='enp0s9')
.
Sent 1 packets.
>>>

    5. Return to flatbush, we should observe,

      >>> packets = sniff(prn=lambda p:p.summary(), filter='ether host 08:00:27:cb:67:
...:1d', iface='enp0s9')
08:00:27:08:0d:a1 > 08:00:27:cb:67:1d (0x9000) / Raw
^C>>> hexdump(packets[0])
0000  080027CB671D080027080DA190004865 ..'.g...'.....He
0010  6C6C6F2C20466C6174627573682E2049 llo, Flatbush. I
0020  20616D2074617070696E67206F6E2044  am tapping on D
0030  69616D6F6E644D6964776F6F64466C61 iamondMidwoodFla
0040  7462757368                       tbush
>>>
      1. On other two VMs, bushwick and eastny, we should observe that we don’t see those packets.

    6. A few variants of packet sniffing scenario provide how these physical links work including capturing packets from “all” interface and a designated interface other than enp0s9, e.g.,

          >>> packets = sniff(prn=lambda p:p.summary(), filter='ether host 08:00:27:cb:67:
    ...:1d')
    08:00:27:08:0d:a1 > 08:00:27:cb:67:1d (0x9000) / Raw
    >>> packets = sniff(prn=lambda p:p.summary(), filter='ether host 08:00:27:cb:67:
    ...:1d', iface='enp0s10')

    which shows the packet indeed arrives at the NIC tapped on the
    ethernet because we would not see such a packet if we only
    capture traffic comes to a NIC tapped on a different Ethernet.

Setting up Diamond IP Network

It is important to remember all of the setup below wouldn’t survive a reboot. A prudent approach would be to save each command in a shell script, and you can easily repeat without retyping them.

Perhaps, a related question is how we configure the host so that the setup via the length commands would survive the reboot. Appendix I provides a gist of of the answer to the question.

It is also important to note that this section does not contain all the commands to set up the Diamond IP network as we envisioned completely. But it gives you all the ingredients you need to finish “cooking” by yourself.

  1. We plan to deploy the following IP networks to the Diamond Ethernets.

    Ethernet IPv4 Network IPv4 Broadcast Address
    DiamondMidwoodFlatbush 10.1.1.0/28 10.1.1.15
    DiamondMidwoodbushwick 10.1.1.16/28 10.1.1.31
    DiamondFlatbushEastNY 10.1.1.32/28 10.1.1.47
    DiamondBushwickEastNY 10.1.1.48/28 10.1.1.63

    Following the above plan, we have the following IPv4 address assignments to the NICs on the four VMs.

    Host name NIC:IP Address Linux Command
    midwood enp0s9: 10.1.1.1 ip address add 10.1.1.1/28 broadcast 10.1.1.15 dev enp0s9
    midwood enp0s10: 10.1.1.17 ip address add 10.1.1.17/28 broadcast 10.1.1.31 dev enp0s10
    flatbush enp0s9: 10.1.1.2 ip address add 10.1.1.2/28 broadcast 10.1.1.15 dev enp0s9
    flatbush enp0s10: 10.1.1.33 ip address add 10.1.1.33/28 broadcast 10.1.1.47 dev enp0s10
    bushwick enp0s9: 10.1.1.18  
    bushwick enp0s10: 10.1.1.49  
    eastny enp0s9: 10.1.1.34 ip address add 10.1.1.34/28 broadcast 10.1.1.47 dev enp0s9
    eastny enp0s10: 10.1.1.50 ip address add 10.1.1.50/28 broadcast 10.1.1.63 dev enp0s10

  2. We realize these using the commands in the able on the VMs, for instance, on midwood, as follows,

      brooklyn@midwood:~$ sudo ip address add 10.1.1.1/28 broadcast 10.1.1.15 dev enp0s9
  brooklyn@midwood:~$ sudo ip address add 10.1.1.17/28 broadcast 10.1.1.31 dev enp0s10
  brooklyn@midwood:~$ ip address show
  1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
          link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
          inet 127.0.0.1/8 scope host lo
                 valid_lft forever preferred_lft forever
          inet6 ::1/128 scope host
                 valid_lft forever preferred_lft forever
  2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
          link/ether 08:00:27:5a:ff:ec brd ff:ff:ff:ff:ff:ff
          inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
                 valid_lft 83644sec preferred_lft 83644sec
          inet6 fe80::a00:27ff:fe5a:ffec/64 scope link
                 valid_lft forever preferred_lft forever
  3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
          link/ether 08:00:27:a1:33:b4 brd ff:ff:ff:ff:ff:ff
          inet 192.168.56.103/24 brd 192.168.56.255 scope global dynamic enp0s8
                 valid_lft 591sec preferred_lft 591sec
          inet6 fe80::a00:27ff:fea1:33b4/64 scope link
                 valid_lft forever preferred_lft forever
  4: enp0s9: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
          link/ether 08:00:27:08:0d:a1 brd ff:ff:ff:ff:ff:ff
          inet 10.1.1.1/28 brd 10.1.1.15 scope global enp0s9
                 valid_lft forever preferred_lft forever
          inet6 fe80::a00:27ff:fe08:da1/64 scope link
                 valid_lft forever preferred_lft forever
  5: enp0s10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
          link/ether 08:00:27:a2:b4:f0 brd ff:ff:ff:ff:ff:ff
          inet 10.1.1.17/28 brd 10.1.1.31 scope global enp0s10
                 valid_lft forever preferred_lft forever
          inet6 fe80::a00:27ff:fea2:b4f0/64 scope link
                 valid_lft forever preferred_lft forever
  6: enp0s16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
          link/ether 08:00:27:a3:d1:e4 brd ff:ff:ff:ff:ff:ff
          inet6 fe80::a00:27ff:fea3:d1e4/64 scope link
                 valid_lft forever preferred_lft forever
  brooklyn@midwood:~$

  3. For a packet originated at midwood to reach eastny (2 hops away), we need to configure flatbush to act as a forwarding node.

      brooklyn@flatbush:~$ sudo sysctl -w net.ipv4.ip_forward=1
  net.ipv4.ip_forward = 1
  brooklyn@flatbush:~$

  4. We also need to update the forwarding table at midwood, and the forwarding table at eastny. Without the former, midwood does not know where it should transmit a packet whose destination is on eastny. Without the later, easny does not know where to transmit a reply.

    1. At midwood, we do

        sudo ip route add 10.1.1.32/28 via 10.1.1.2

    2. At eastny, we do

        sudo ip route add 10.1.1.0/28 via 10.1.1.33

  5. It’s the time to see we can send an IP packet from midwood to eastny and get a reply from it. So at midwood, we do

      brooklyn@midwood:~$ sudo scapy3
  >>> reply = sr1(IP(dst='10.1.1.34')/ICMP())
  Begin emission:
  ..Finished sending 1 packets.
  .*
  Received 4 packets, got 1 answers, remaining 0 packets
  >>> reply
  <IP  version=4 ihl=5 tos=0x0 len=28 id=36717 flags= frag=0 ttl=63 proto=icmp chksum=0xd64f src=10.1.1.34 dst=10.1.1.1 options=[] |<ICMP  type=echo-reply code=0 chksum=0xffff id=0x0 seq=0x0 |<Padding  load='\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00' |>>>
  >>> reply.summary()
  'IP / ICMP 10.1.1.34 > 10.1.1.1 echo-reply 0 / Padding'
  >>>

Experiments and Demonstration

Setting up Diamond or Line IP Network

This is your exercise and exploration. Review the steps we carry out for the Diamond IP network, and repeat them for the Line IP network.

Observing IP Forwarding

With these two networks, we can observe how IP forwarding works. To view the forwarding table, for instance, on host midwood, we do,

ip route show

Observing Packet Fragmentation and Reassembly

Before attempting this exercise, please review Appendix II. Below we consider two scenraios.

  1. Sending a large packet from the source, and the packet will be fragmented at the source. We do this in ScaPy,

      >>> packet = IP(dst='10.1.1.34')/Raw("A"*5000)
  >>> send(packet)
  .
  Sent 1 packets.
  >>>

    Before you send the packet, you should prepare packet capturing; otherwise, you will miss the chance of observing IP packet fragmentation.

  2. Sending a packet that will be fragmented at a forwarding node along the path.

    For this, we need to set the MTU of the forwarding node to a smaller value with a commmand like,

      ip link set dev YOUR_NIC mtu 1400

    Using this command set a smaller MTU at a forwarding node, and observe how an IPv4 packet is fragmented and forwarded, e.g.,

      brooklyn@flatbush:~$ sudo ip link set dev enp0s10 mtu 1280
  brooklyn@flatbush:~$

    After you complete this, like before, you will set up packet capturing at along the path, craft an IP packet larger than the MTU (like larger than 1280 bytes), and send the packet.

Appendix I. Network Configuration Files

On Debian Linux, we can configure host with the steps outlined in the above in a configuration file, and the configuration file would come into effect when the system boots.

Enabling or Disabling IPv4 Forwarding

In /etc/sysct.conf, uncomment to enable or comment the following line to disable IPv4 forwarding,

#net.ipv4.ip_forward=1

Setting up IP Address and Routes

In /etc/network/interfaces file, add a section of configuration code for each network interface you wish to set up, e.g., for enps09 on midwood, we may append the following lines to it,

auto enps09
iface enps09 inet static
  address 10.1.1.1/28
  post−up ip route add 10.1.1.32/28 via 10.1.1.2 
  pre−down ip route del 10.1.1.32/28 via 10.1.1.2

Appendix II. (Possible) ScaPy Bug in Version 2.4.0

It appears that is a bug in ScaPy verison 2.4.0 that prevents any IP packet larger than MTU to be sent successfully. The error is like the following,

brooklyn@flatbush:~$ sudo scapy3

                     aSPY//YASa
             apyyyyCY//////////YCa       |
            sY//////YSpcs  scpCY//Pp     | Welcome to Scapy
 ayp ayyyyyyySCP//Pp           syY//C    | Version 2.4.0
 AYAsAYYYYYYYY///Ps              cY//S   |
         pCCCCY//p          cSSps y//Y   | https://github.com/secdev/scapy
         SPPPP///a          pP///AC//Y   |
              A//A            cyP////C   | Have fun!
              p///Ac            sC///a   |
              P////YCpc           A//A   | Craft packets like I craft my beer.
       scccccp///pSP///p          p//Y   |               -- Jean De Clerck
      sY/////////y  caa           S//P   |
       cayCyayP//Ya              pY/Ya
        sY/PsY////YCc          aC//Yp
         sc  sccaCY//PCypaapyCP//YSs
                  spCPY//////YPSps
                       ccaacs
                                       using IPython 5.8.0
>>> packet = IP(dst='10.1.1.1')/Raw("A"*5000)
>>> send(packet)
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
/usr/lib/python3/dist-packages/scapy/arch/linux.py in send(self, x)
    485         try:
--> 486             self.outs.sendto(sx, sdto)
    487         except socket.error as msg:

OSError: [Errno 90] Message too long

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
<ipython-input-2-b8a536fe3f5f> in <module>()
----> 1 send(packet)

/usr/lib/python3/dist-packages/scapy/sendrecv.py in send(x, inter, loop, count, verbose, realtime, return_packets, socket, *args, **kargs)
    300         socket = conf.L3socket(*args, **kargs)
    301     return __gen_send(socket, x, inter=inter, loop=loop, count=count,verbose=verbose,
--> 302                       realtime=realtime, return_packets=return_packets)
    303
    304 @conf.commands.register

/usr/lib/python3/dist-packages/scapy/sendrecv.py in __gen_send(s, x, inter, loop, count, verbose, realtime, return_packets, *args, **kargs)
    274                     else:
    275                         dt0 = ct-p.time
--> 276                 s.send(p)
    277                 if return_packets:
    278                     sent_packets.append(p)

/usr/lib/python3/dist-packages/scapy/arch/linux.py in send(self, x)
    486             self.outs.sendto(sx, sdto)
    487         except socket.error as msg:
--> 488             if msg[0] == 22 and len(sx) < conf.min_pkt_size:
    489                 self.outs.send(sx + b"\x00" * (conf.min_pkt_size - len(sx)))
    490             elif conf.auto_fragment and msg[0] == 90:

TypeError: 'OSError' object is not subscriptable
>>>

The bug does not appear in versions 2.4.3 and 2.4.4. To resolve this issue, on the Virtual Machine you intend to use it as the sending node, do the following to upgrade ScaPy to version 2.4.4,

sudo apt-get install --no-install-recommends python3-pip
sudo python -m pip install scapy --upgrade
sudo python -m pip install pyparsing --upgrade
sudo python -m pip install prompt_toolkit --upgrade
sudo python -m pip install IPython --upgrade

This isn’t the recommended method to upgrade a Python package since we installed Python and Python packages via the Operating System’s package manager. It is preferable that you do these on a clone, other than the original. In this case, if you discover something isn’t working due to this upgrade, you can always create a new clone.