In this section, we provide details on how the experimental environment is set up and how the measurements are conducted. The testing and measurements are carried out on both virtual and physical equipment. The virtual equipment is constructed with the network simulation software GNS3 at Atea. Experimenting on physical equipment is conducted at Mälardalens University.
8.1 Virtual Network Setup
The network topology is constructed in collaboration with Atea using their equipment. The net-work architecture consists of a firewall, one router, two switches, one VMware ESXi server, one Access Point, and one Wireless Controller. See Figure 2 for a visual representation of how the network is designed.
VMware Esxi
Router Firewall
Switch 1
Internet
Access Point Switch 2
Wireless Controller
Figure 2: Network Topology built at Atea
The network is secluded from the production network used at Atea which means no external traffic hampers the result. The router is configured as a "router on a stick" where several Virtual Local Area Network (VLAN) are configured. Three VLANs are configured (10, 15, and 20). VLAN 10 acts as a server network where all the physical equipment is allocated with an IP address from this VLAN, VLAN 15 acts as a management VLAN to enable management traffic on the network, and VLAN 20 is created as a client network so different clients (such as computers and virtual machines) can connect to the network. The Access Point and Wireless Controller are part of the network topology to enable a wireless connection to the network.
8.1.1 VMware ESXi
The VMware ESXi server is configured with five different virtual machines. One GNS3 server, an Ansible machine, a Puppet machine, a SaltStack machine, and a Windows Server 2019 machine.
The Windows Server 2019 acts as a Dynamic Host Configuration Protocol (DHCP) server. With a DHCP server on the network, there is no need for us to manually set static IP addresses on every single device. The DHCP server allows us to easily join new devices to the network. To enable the virtual machines to connect to the rest of the network all of the virtual machines are configured within VMware ESXi with a bridged network adapter. Enabling this setting will connect the vir-tual machines’ network card to the physical machine’s network card.
The GNS3 server is installed according to Appendix A as a virtual machine on the VMware ESXi server. When the GNS3 server is installed the remaining configuration is done by remote connection to the GNS3 server, either via the GNS3 desktop application or by connecting through
a web browser. After establishing a connection to the GNS3 server via the IP address and port number specified by the GNS3 server’s interface on the virtual machine a testing setup is con-structed. The test setup consists of virtual routers that run a Cisco IOS and more specifically the c7200-advipservicesk9-mz.152-4.S5 OS.
The Ansible, Puppet, and SaltStack virtual machines are all installed with Ubuntu version 20.04.
For the automation tools to work on equal footing with each other, it is decided they should operate with the same OS. As seen in Table 1 an OS based on Linux would be able to run all automation tools. Ubuntu is a popular Linux distribution with a package management system based on Debian and it is chosen for its user-friendly User Interface [24]. Communication between the virtual machines hosting the automation tools to the virtual network topology hosted by GNS3 is internal since all the virtual machines are directly connected through the VMware ESXi server.
8.1.2 Virtual Network Topology
The virtual network topology is built within GNS3 for testing the automation tools. The topology is built on the VMware ESXi server with a Cisco image12for a Cisco 7200 router. Due to resource limitations of the server, the virtual environment is limited to twenty network devices. Figure3 shows the topology of the virtual network.
Bridge to Physical Network
Virtual environment
Figure 3: Virtual Network Topology made in GNS3 hosted on the VMware ESXI server The switches seen in Figure3 are integrated GNS3 Ethernet switches. These switches are used to connect the virtual routers and to connect the virtual routers to the bridged network adapter which enables them to communicate with the network outside of the GNS3 server. The reason for using several switches is that the GNS3 standard switches have a limited amount of interfaces.
This means that several switches are needed to connect all the virtual routers. All routers are preconfigured with a default configuration to enable an SSH connection. The default configuration consisted of: assigning an IP address to an interface, creating a user to enable login, and the activation of SSH.
8.2 Physical Network Setup
The measurements performed on the physical equipment are identical with those we performed on the virtual equipment. The automation tools are configured identically and tasked to configure the same configuration changes done on the virtual equipment. The automation tools are installed on virtual machines hosted on VirtualBox.
The experiments with physical equipment, use a total sixteen Cisco 2811 routers with Cisco IOS 12.4 are used. Sixteen routers are used because of a limited number of routers available. The physical routers are preconfigured with an identical default configuration as the virtual routers to
12The Cisco IOS is built using a binary image which is uploaded to the network device [25].
enable an SSH connection. The physical routers are all connected to a single Cisco 2960 switch.
The switch connects all physical routers. The virtual machine connects to the switch which enables the automation tools to configure the physical routers. Configuring the virtual machines within VirtualBox to use a bridged network adapter, enables the virtual machine to communicate with the physical network. See Figure4 for a visual representation of the physical network topology.
Figure 4: Physical Network Topology
8.3 Execution of Measurements
To gather enough data to analyze, a script13is written to automate our automation configuration.
The script is written in the command language Bash, which specializes in Linux shell scripting.
The script allows us to execute a task several times after each other without the need for manual intervention. This provides us the ability to gather more data to analyze which in turn provides us a more precise result. The script consists of three different tasks. The first task is to initiate the automation tool and measure the time to complete the configuration changes. The execution time is then directed to a text file where it is stored for us to collect once the script is finished. The second task of the script is to measure the amount of data that is sent during the first task. The calculated amount of data is redirected to a separate file for us to collect once the script is finished.
After the second task is done the automation tool has configured all routers that are specified to be configured. The last task of the script is to clear and revert all the network devices to a state they held before the script is initiated. After the third task is completed the script restarts and continues the process once again. Each of the experiments are repeated 50 times. See AppendixE for the script used during the measurements. The automation tools perform the following changes:
Set login banner:
- This sets the specified text as a banner when someone logs in to the device.
Apply IP address to two interfaces:
- This applies two different IP addresses to two different interfaces and activates them.
13Scripting is used to automate certain tasks with programming.
Set a static IP route:
- This sets a static IP route and applies it to the router’s routing table.
Configure a management Virtual Routing and Forwarding (VRF):
- This sets and configures a VRF named management.
Set a Network Time Protocol (NTP) server:
- This configures a NTP server on the router.
Set a Access Control List (ACL):
- This will configure an ACL to limit entry to the network.
Set a new hostname:
- Configure a new hostname on the router.
Add a new user:
- This will configure a new user on the router with a username and a password.
Add a description to an interface:
- This will add a description to an interface to document its properties.
8.4 Ansible
The installation of Ansible is done on a virtual Ubuntu machine through the Command Line Interface (CLI), see AppendixBfor installation and deployment. Once the installation is completed an Ansible configuration file is created and by default found at /etc/ansible/ansible.cfg. This file needs some tweaking to allow us to configure network devices via SSH. We also change a setting that will allow Ansible to configure several devices at once. Per default, Ansible will only configure five devices simultaneously. This is done to put Ansible on equal footing with the other automation tools. The last edit we make to the configuration file is to make Ansible display the execution time after it is finished. The changes we make in the configuration file are:
fork = 20
host_key_checking = False callback_whitelist = timer
After these changes are made to the configuration file a hosts file is created. The hosts file contains the information needed for Ansible to establish a connection. Within the hosts file groups can be constructed to simplify management. If the same configuration is to be established on several devices, a group can be created and called upon instead of individually calling several network devices. The next step is to create a playbook file. A playbook file contains the network device’s configuration changes. The configuration changes we write in the playbook file use YAML syntax.
For our measurements, two different playbook files are created. One contains three configuration changes and one contains ten configuration changes. Since we make a change to Ansible’s config-uration file it displays the time to execute the playbook file. This time is captured by our script and stored in a text file for us to collect once the measurements are completed. For instruction on how to construct a hosts file and a playbook file in detail, see AppendixB.
8.5 Puppet
The installation of Puppet Bolt is done on a virtual Ubuntu machine through the CLI, see Ap-pendixC for installation and deployment. Unlike Ansible we do not change the configuration of Puppet Bolt since it operates the way we intend with a fresh installation. After the installation is complete we create an inventory file that uses YAML syntax. Similar to Ansible, groups can be constructed within the inventory file to simplify management. The inventory file contains the necessary information for Puppet Bolt to establish an SSH connection to the network devices. The next step is to create a manifest file. A manifest file contains all the configuration changes which
are to be carried out by Puppet Bolt. Two manifest files are created, one containing three con-figuration changes and one containing ten concon-figuration changes. Puppet Bolt displays by default the run time after each execution. This run time is captured by our script and stored in a text file for us to collect once the measurements are completed. For instructions on how to construct an inventory file and a manifest file in detail, see AppendixC.
8.6 SaltStack
The installation of SaltStack is done on a virtual Ubuntu machine through the CLI, see AppendixD for installation and deployment. After the installation is completed we do not change the configur-ation of SaltStack, because like Puppet Bolt it operates the way we intend with a fresh installconfigur-ation.
Once the installation is complete the next step is to create pillar files. The pillar files contain all the necessary information needed for SaltStack to establish an SSH connection. One pillar file is created for every network device we wish to configure. The pillar files are then summarized in a top file. The top file contains a destination to every pillar file and associates the pillar files with an alias. Both the pillar file and the top file use YAML syntax. After constructing the top file, the Salt Master and the Salt proxy agents can be initiated. One Salt proxy agent is initialized for every network device we wish to configure. Once the Salt proxy agents establish a connection to their target device they will wait for instructions from the Salt Master and keep the connection established to the target device. The configuration changes are written with Cisco IOS commands in an ordinary text file. When the Salt Master initiates the configuration with proxy agents, it will send the configuration file which contains the commands. Then the Salt proxy agents distribute the file to their target devices via the data communication protocol Secure Copy (SCP). Two of these configuration files are created which contain three and ten configuration changes. To capture the time it takes to configure all the changes we use the Bash command time. The time command displays the time it takes to complete a command. This time is then captured by our script and stored in a text file for us to collect once the measurements are completed. For instruction on how to construct a pillar file and a top file in detail, see AppendixD.