OpenStack Interface Hot Plugging

Sometimes it’s useful to dynamically add or remove interfaces to already running instances without having to recreate the instance. For example, if you want to reorganize which networks you originally spawned your instances on or perhaps if creating a service VM that acts as a router between different subnets.

In Grizzly, this was made possible. In order demonstrate this I’ve created two networks net1 and net2 as shown:

$ quantum net-list
+--------------------------------------+---------+----------------------------------------------------+
| id                                   | name    | subnets                                            |
+--------------------------------------+---------+----------------------------------------------------+
| d58e9f6b-d9af-468e-a0cb-f4eea18b6065 | net1    | 4671d053-ac63-4b0f-afe8-18d9444ad8c0 10.2.0.0/24   |
| e79f5d1a-289e-44b6-8070-943535cbbeae | net2    | e73a25ca-d211-44b4-a376-64791ea1dabe 10.3.0.0/24   |
+--------------------------------------+---------+----------------------------------------------------+

Create an instance on net1 via:

$ nova boot --image cirros-0.3.1-x86_64-uec --flavor 1 --nic net-id=d58e9f6b-d9af-468e-a0cb-f4eea18b6065 vm1

Nova list shows the running instance and it’s id:

$ nova list
+--------------------------------------+------+--------+------------+-------------+---------------+
| ID                                   | Name | Status | Task State | Power State | Networks      |
+--------------------------------------+------+--------+------------+-------------+---------------+
| 54ca2943-46e7-4e2e-b470-a015f23797c0 | vm1  | ACTIVE | None       | Running     | net1=10.2.0.3 |
+--------------------------------------+------+--------+------------+-------------+---------------+

Running ifconfig -a in the instance returns the following:

$ ifconfig -a
eth0      Link encap:Ethernet  HWaddr FA:16:3E:59:08:AC  
          inet addr:10.2.0.3  Bcast:10.2.0.255  Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe59:8ac/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:89 errors:0 dropped:0 overruns:0 frame:0
          TX packets:140 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:14652 (14.3 KiB)  TX bytes:14688 (14.3 KiB)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:46 errors:0 dropped:0 overruns:0 frame:0
          TX packets:46 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:4192 (4.0 KiB)  TX bytes:4192 (4.0 KiB)

Now if we want to attach an additional interface on net2 running the following command achieves this:

$ nova interface-attach --net-id e79f5d1a-289e-44b6-8070-943535cbbeae 54ca2943-46e7-4e2e-b470-a015f23797c0 # <-- instance id
$ ifconfig -a
eth0      Link encap:Ethernet  HWaddr FA:16:3E:59:08:AC  
          inet addr:10.2.0.3  Bcast:10.2.0.255  Mask:255.255.255.0
          inet6 addr: fe80::f816:3eff:fe59:8ac/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:144 errors:0 dropped:0 overruns:0 frame:0
          TX packets:178 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:21919 (21.4 KiB)  TX bytes:21250 (20.7 KiB)

eth1      Link encap:Ethernet  HWaddr FA:16:3E:EA:83:F5  
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:46 errors:0 dropped:0 overruns:0 frame:0
          TX packets:46 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:4192 (4.0 KiB)  TX bytes:4192 (4.0 KiB)

As you can see eth1 was added. We can also remove the interface attached to net1 (eth0) via:

$ nova interface-detach  54ca2943-46e7-4e2e-b470-a015f23797c0 d4911c36-2c8d-4dd3-a128-2d7e411ce877 # <--port-uuid

and ifconfig -a shows that it was removed:

$ ifconfig -a
eth1      Link encap:Ethernet  HWaddr FA:16:3E:EA:83:F5  
          BROADCAST MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

lo        Link encap:Local Loopback  
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:46 errors:0 dropped:0 overruns:0 frame:0
          TX packets:46 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:4192 (4.0 KiB)  TX bytes:4192 (4.0 KiB)

Heads up if you are running the first RC release of Grizzly there was a bug that caused this not to work when using quantum but this was shortly by https://github.com/openstack/nova/commit/bba57e9fa63b7f55d403d9f6950c4cde425d83b0.

Posted in openstack | 1 Comment

Building a Multi-Tier Application With OpenStack

In this blog post I’m going to give you a walk through on how to build a simple multi-tier application on OpenStack. Before I do this, I want to give a little background about the features that went into OpenStack during the Grizzly release cycle that enabled this. First I’ll talk about security groups and then about Load-balancer-as-a-Service.

During the Grizzly cycle of OpenStack there was a lot of work that went into security groups, in particular security groups involving Quantum. If you aren’t familiar with Quantum, it is a relatively new OpenStack project that became core in Folsom and its goal is to handle the networking part of OpenStack, which makes sense that security groups should be implemented there. In Folsom, if one wanted to use security groups, one would need to use Nova’s security group implementation, which had a few limitations that we wanted to fix. The first limitation was that Nova security groups implementation did not work if one wanted to use overlapping ip addresses. In addition, Nova’s security groups did not support egress filtering unless a tenant enforced this themselves within the instance. Egress filtering allows tenants to enforce which end hosts and protocols their instances are able to initiate communication with, which is useful if one wants to lock down who their instances can communicate with.

The last part of the security group work was to implement the ability for Nova to proxy its security group calls to Quantum. This is important because it allows one to create an instance via Nova and specific security groups in Quantum, which helps reduce orchestration requirements (i.e: first creating a port in quantum with specific security groups and then tell nova to use that port). In addition, this proxy layer allows existing scripts and tools to continue to work even if using quantum security groups, allows Nova’s EC2 security group implementation to work with quantum, and lastly allows the Nova security group api to work in conjunction with overlapping ips as the calls are proxied to Quantum, which handles this.

In addition, to security groups, Load-balancer-as-a-Service (LBaaS) was another feature added to Quantum during grizzly. LBaaS allows the ability to provision on demand loadbalancers pragmatically, which in my opinion is pretty freaking cool! This allows one to create several instances all running the same application and then distribute the load across them in order to scale out an application and provide high availability.

Multi-Tier Application Walk Through

Consider the scenario where a tenant wants a multi-tier application that consists of web servers and database servers. The tenant only wants to allow HTTP port 80 to be accessible to the internet from the web servers and only allows the web servers to be able to communicate with the database servers over port 3306 to access the database. The tenant also wants to have a jump box accessibly from the internet to ssh to and then ssh from that to any of the web servers or database servers if needed. This increases the security by not having your web servers and database severs directly ssh-able from the internet. Lastly, the tenant wants the requests to be loadbalanced between the two web servers. To demonstrate this using OpenStack, we’ll use a project called devstack to quickly setup OpenStack.

Get the devstack code:

$ git clone https://github.com/openstack-dev/devstack
$ cd devstack

Next, create a file called localrc which is used to tell devstack which components we want it to setup and install and put the following content below in it. For this demo we’ll be using the Open vSwitch Quantum plugin.

ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-obj,n-cpu,n-sch,n-cauth,horizon,mysql,rabbit,sysstat,cinder,c-api,c-vol,c-sch,n-cond,quantum,q-svc,q-agt,q-dhcp,q-l3,q-meta,q-lbaas,n-novnc,n-xvnc,q-lbaas
DATABASE_PASSWORD=password
RABBIT_PASSWORD=password
SERVICE_TOKEN=password
SERVICE_PASSWORD=password
ADMIN_PASSWORD=password

Next run ./stack.sh and grab a cup of coffee as this will take a few minutes to complete as it downloads all the required packages and code.

In order to start using OpenStack, you’ll need to authenticate as a tenant. To do this, run the following command in order to put the demo user’s credentials in your environment.

$ source openrc demo demo

The devstack script automatically creates two networks for you ‘private’ and ‘public’. The ‘public’ network we’ll use to allocate floating ips out of later. Running quantum net-list will show this:

$ quantum net-list
+--------------------------------------+---------+--------------------------------------------------+
| id                                   | name    | subnets                                          |
+--------------------------------------+---------+--------------------------------------------------+
| 02e0a203-8349-47fc-8d61-8987b4197f1c | public  | c1787d38-fa26-48c9-9f36-d2f563b38c70             |
| 812fb7bd-3ad9-4583-b9f0-55c9ab5a7d55 | private | a00e7146-d77c-4835-93d0-5ab743f3aee6 10.0.0.0/24 |
+--------------------------------------+---------+--------------------------------------------------+

First we’ll create the three security groups we’ll need to contain the members: web, database and ssh.

$ quantum security-group-create web
$ quantum security-group-create database
$ quantum security-group-create ssh

Now we’ll add rules into these security groups for their desired functionality.

Allow all HTTP Port 80 traffic to web security group:

$ quantum security-group-rule-create --direction ingress --protocol TCP --port-range-min 80 --port-range-max 80 web

Allow database severs to be accessed from web servers:

$ quantum security-group-rule-create --direction ingress --protocol TCP --port-range-min 3306 --port-range-max 3306 --remote-group-id web database

Allow Jump host to ssh into database servers and webservers

$ quantum security-group-rule-create --direction ingress --protocol TCP --port-range-min 22 --port-range-max 22 --remote-group-id ssh database
$ quantum security-group-rule-create --direction ingress --protocol TCP --port-range-min 22 --port-range-max 22 --remote-group-id ssh web

Allow outside world to be able to access the jumpbox over port 22 for ssh:

$ quantum security-group-rule-create --direction ingress --protocol tcp  --port-range-min 22 --port-range-max 22 ssh

Let’s now boot some vms using these security groups. First, run quantum net-list to obtain the private network uuid that we are going to be using:

$ quantum net-list
+--------------------------------------+---------+--------------------------------------------------+
| id                                   | name    | subnets                                          |
+--------------------------------------+---------+--------------------------------------------------+
| 02e0a203-8349-47fc-8d61-8987b4197f1c | public  | c1787d38-fa26-48c9-9f36-d2f563b38c70             |
| 812fb7bd-3ad9-4583-b9f0-55c9ab5a7d55 | private | a00e7146-d77c-4835-93d0-5ab743f3aee6 10.0.0.0/24 |
+--------------------------------------+---------+--------------------------------------------------+

Then, we’ll run nova image-list to determine the images available to boot our instances with. Since we’re using devstack, the script automatically uploaded an image to glance for us to use.

$ nova image-list
+--------------------------------------+---------------------------------+--------+--------+
| ID                                   | Name                            | Status | Server |
+--------------------------------------+---------------------------------+--------+--------+
| fe2a01fe-202a-4b4d-b98c-2cbcc5c72d18 | cirros-0.3.1-x86_64-uec         | ACTIVE |        |
| 79b967f1-4f4e-4876-a2c0-edf647337e38 | cirros-0.3.1-x86_64-uec-kernel  | ACTIVE |        |
| 4d14dc5d-ef4a-451e-9ef8-bba3f8aaf66d | cirros-0.3.1-x86_64-uec-ramdisk | ACTIVE |        |
+--------------------------------------+---------------------------------+--------+--------+

Boot four instances: two web severs, one database server, and our ssh jump box.

Note: When running nova boot without –nic net-id= an instance will be allocated a port on every network the tenant owns. In this case the tenant only owns one network so we are leaving off the –nic net-id=812fb7bd-3ad9-4583-b9f0-55c9ab5a7d55 for simplicity so the commands can be copied and pasted.

Boots two instances named web_server1 and web_server2 on the private network using the cirros image and part of the web security group:

$ nova boot --image cirros-0.3.1-x86_64-uec --security_groups web --flavor 1 web_server1
$ nova boot --image cirros-0.3.1-x86_64-uec --security_groups web --flavor 1 web_server2

Boot database server:

$ nova boot --image cirros-0.3.1-x86_64-uec --security_groups database --flavor 1 database_server1

Boot ssh jump host:

$ nova boot --image cirros-0.3.1-x86_64-uec --security_groups ssh --flavor 1 jumpbox

Boot client instance that we’ll use to access the web servers from (Since we did not specify a security group this instance will be part of a ‘default’ security group which allows the instance to make outgoing connections to anyone but only accept incoming connections from members of this same security group):

$ nova boot --image cirros-0.3.1-x86_64-uec --flavor 1 client

Running nova list will display the status of the instances. After a few seconds all of the instances should go to an ACTIVE status.

$ nova list
+--------------------------------------+------------------+--------+------------------+
| ID                                   | Name             | Status | Networks         |
+--------------------------------------+------------------+--------+------------------+
| d4305afe-4985-45c1-9337-f6fd4f1c83f1 | client           | ACTIVE | private=10.0.0.7 |
| 5c4128a7-3133-43ee-8a98-ba8ff8fda65f | database_server1 | ACTIVE | private=10.0.0.5 |
| 50cbfad8-004a-4385-aa94-23d0a335fffa | jumpbox          | ACTIVE | private=10.0.0.6 |
| ec0ada74-c978-468e-9f5f-251c5dc0c710 | webserver1       | ACTIVE | private=10.0.0.3 |
| c66d0545-44af-476c-8d27-6d45c91e7e71 | webserver2       | ACTIVE | private=10.0.0.4 |
+--------------------------------------+------------------+--------+------------------+

To make the jumpbox publicly accessible on the internet we’ll need to assign a floating IP to it. To do this first create a floating IP via:

$ quantum floatingip-create public
Created a new floatingip:
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| fixed_ip_address    |                                      |
| floating_ip_address | 172.24.4.227                         |
| floating_network_id | 02e0a203-8349-47fc-8d61-8987b4197f1c |
| id                  | d67ec5ae-04e3-4e87-a6a5-7cc04a3d5608 |
| port_id             |                                      |
| router_id           |                                      |
| tenant_id           | d8293c068214472d8008000a79d369da     |
+---------------------+--------------------------------------+

Next, we need to determine the port id of the jumpbox:

$ quantum port-list
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
| id                                   | name | mac_address       | fixed_ips                                                                       |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+
| 4cfc707f-f78a-4bd9-9c27-bb6f8b01dfcc |      | fa:16:3e:03:f7:c4 | {"subnet_id": "82294d13-5e55-4ae2-bc95-9d4e09ab4ebf", "ip_address": "10.0.0.2"} |
| 5620c0d6-046e-49de-b4ab-5d8948a78cbf |      | fa:16:3e:12:3b:75 | {"subnet_id": "82294d13-5e55-4ae2-bc95-9d4e09ab4ebf", "ip_address": "10.0.0.5"} |
| 5912c994-be8e-4b9e-8fe9-6bcb1f1c89dd |      | fa:16:3e:ba:04:b7 | {"subnet_id": "82294d13-5e55-4ae2-bc95-9d4e09ab4ebf", "ip_address": "10.0.0.1"} |
| 5cfcf996-4a3a-4a68-84e5-5c68d06832b4 |      | fa:16:3e:47:2e:72 | {"subnet_id": "82294d13-5e55-4ae2-bc95-9d4e09ab4ebf", "ip_address": "10.0.0.6"} |
| a219f389-c792-4087-a3a6-d8b9c681c170 |      | fa:16:3e:71:ce:0a | {"subnet_id": "82294d13-5e55-4ae2-bc95-9d4e09ab4ebf", "ip_address": "10.0.0.4"} |
| edeb0b7b-89f1-4018-9eae-fbf70bcee7ac |      | fa:16:3e:59:c8:ea | {"subnet_id": "82294d13-5e55-4ae2-bc95-9d4e09ab4ebf", "ip_address": "10.0.0.3"} |
| faa8718a-2efd-443f-a1b0-29c06c281e72 |      | fa:16:3e:5a:9d:a9 | {"subnet_id": "82294d13-5e55-4ae2-bc95-9d4e09ab4ebf", "ip_address": "10.0.0.7"} |
+--------------------------------------+------+-------------------+---------------------------------------------------------------------------------+

and find the id that matches the IP address of the jumphost (10.0.0.6) and associate it via:

$ quantum floatingip-associate d67ec5ae-04e3-4e87-a6a5-7cc04a3d5608 5cfcf996-4a3a-4a68-84e5-5c68d06832b4
Associated floatingip d67ec5ae-04e3-4e87-a6a5-7cc04a3d5608

Now you should be able to ssh to the jumbox via with password cubswin:) :

$ ssh cirros@172.24.4.227

(Optional) To access your instances from horizon, point your web browser at the IP address of this box and it should take you to the horizon landing page. Log in via demo/password and then set the current project to ‘demo’ (if not already set) and click on the instances tab. This should display all of your running instances and should allow you to access them via VNC.

After logging into the jumpbox you’ll be able to ssh into your webserver1, webserver2, and database server via:

$ ssh 10.0.0.3
$ ssh 10.0.0.4
$ ssh 10.0.0.5

but none of those instances will be able to ssh to each other. The point of this instance is so that you do not need to have all of your other instances publicly addressable and directly accessible via the internet.

Now let’s log in to web_server1 and web_server2 (via ssh or via horizon) and setup a simple web server to handle requests and reply with who they are:

# On web_server 1 
$ while true; do echo -e 'HTTP/1.0 200 OK\r\n\r\nweb_server1' | sudo nc -l -p 80 ; done 

# on web_server 2
$ while true; do echo -e 'HTTP/1.0 200 OK\r\n\r\nweb_server2' | sudo nc -l -p 80 ; done

Now, log in to your client vm. From there if you run:

$ wget -O - http://10.0.0.3/
Connecting to 10.0.0.3 (10.0.0.3:80)
web_server1
               100% |************************************| 12 0:00:00 ETA

$ wget -O - http://10.0.0.4/
Connecting to 10.0.0.4 (10.0.0.4:80)
web_server2
               100% |************************************| 12 0:00:00 ETA

This demonstrates that our simple web server is working on our two web server instances.

We can demonstrate that the web security group is working correctly by killing our simple web server and changing the port number. (Note: to kill the web server you may need to hold control + c for a second in order for it to break out of the while loop before another instance of nc is created.)

# On web_server 1 
$ while true; do echo -e 'HTTP/1.0 200 OK\r\n\r\nweb_server1' | sudo nc -l -p 81 ; done

Now on the client run:

$ wget -O - http://10.0.0.3:81
Connecting to 10.0.0.3 (10.0.0.3:81)
wget: can't connect to remote host (10.0.0.3): Connection timed out

As you can see the request is never answered as expected because our web security group does not allow port 81 ingress. Now let’s set the web server to run on port 80 again.

At this point were going to provision loadbalancer via quantum order to load balance requests between our two web server instances.

First we determine the subnet uuid of the private network:

$ quantum subnet-list
+--------------------------------------+------+-------------+--------------------------------------------+
| id                                   | name | cidr        | allocation_pools                           |
+--------------------------------------+------+-------------+--------------------------------------------+
| a00e7146-d77c-4835-93d0-5ab743f3aee6 |      | 10.0.0.0/24 | {"start": "10.0.0.2", "end": "10.0.0.254"} |
+--------------------------------------+------+-------------+--------------------------------------------+

Create a loadbalancer pool:

$ quantum lb-pool-create --lb-method ROUND_ROBIN --name mypool --protocol HTTP --subnet-id a00e7146-d77c-4835-93d0-5ab743f3aee6

and then associate our two web servers with this pool.

$ quantum lb-member-create --address 10.0.0.3 --protocol-port 80 mypool
$ quantum lb-member-create --address 10.0.0.4 --protocol-port 80 mypool

Now, let’s create a health monitor, which checks to make sure our instances are still running and associate that with the pool:

$ quantum lb-healthmonitor-create --delay 3 --type HTTP --max-retries 3 --timeout 3
Created a new health_monitor:
+----------------+--------------------------------------+
| Field          | Value                                |
+----------------+--------------------------------------+
| admin_state_up | True                                 |
| delay          | 3                                    |
| expected_codes | 200                                  |
| http_method    | GET                                  |
| id             | ba9ed221-848a-4001-be0e-3a0314981396 |
| max_retries    | 3                                    |
| status         | PENDING_CREATE                       |
| tenant_id      | 53b0a7fe38b747b78e84207a34e46857     |
| timeout        | 3                                    |
| type           | HTTP                                 |
| url_path       | /                                    |
+----------------+--------------------------------------+

$ quantum lb-healthmonitor-associate ba9ed221-848a-4001-be0e-3a0314981396 mypool
Associated health monitor ba9ed221-848a-4001-be0e-3a0314981396

Let’s now create a VIP (Virtual IP Address) that when accessed the loadblancer will direct the request to either instance to loadbalance the request.

$ quantum lb-vip-create --name myvip --protocol-port 80 --protocol HTTP --subnet-id  a00e7146-d77c-4835-93d0-5ab743f3aee6 mypool
Created a new vip:
+------------------+--------------------------------------+
| Field            | Value                                |
+------------------+--------------------------------------+
| address          | 10.0.0.8                             |
| admin_state_up   | True                                 |
| connection_limit | -1                                   |
| description      |                                      |
| id               | e91236dc-b920-4302-bdfa-a6ec029ea128 |
| name             | myvip                                |
| pool_id          | 6aea6769-2cdf-4258-bbe7-60fed0965be9 |
| port_id          | bd9609f3-4c26-4b85-90f0-3e3139fc5c0e |
| protocol         | HTTP                                 |
| protocol_port    | 80                                   |
| status           | PENDING_CREATE                       |
| subnet_id        | a00e7146-d77c-4835-93d0-5ab743f3aee6 |
| tenant_id        | 53b0a7fe38b747b78e84207a34e46857     |
+------------------+--------------------------------------+

Finally, let’s test out the loadbalancer. From the client instance we should be able to run wget at 10.0.0.8 and see that it loadbalancers our requests.

$ wget -O - http://10.0.0.8/
Connecting to 10.0.0.8 (10.0.0.8:80)
web_server1
               100% |************************************| 12 0:00:00 ETA

$ wget -O - http://10.0.0.8/
Connecting to 10.0.0.8 (10.0.0.8:80)
web_server2
               100% |************************************| 12 0:00:00 ETA

$ wget -O - http://10.0.0.8/
Connecting to 10.0.0.8 (10.0.0.8:80)
web_server1
               100% |************************************| 12 0:00:00 ETA

$ wget -O - http://10.0.0.8/
Connecting to 10.0.0.8 (10.0.0.8:80)
web_server2
               100% |************************************| 12 0:00:00 ETA

From the output above you can see that the the requests are being handled by web_server1 then web_server2 in a round robin fashion.

Now to make our VIP publicly accessible via the internet we need to create another floating IP:

$ quantum floatingip-create public
Created a new floatingip:
+---------------------+--------------------------------------+
| Field               | Value                                |
+---------------------+--------------------------------------+
| fixed_ip_address    |                                      |
| floating_ip_address | 172.24.4.228                         |
| floating_network_id | 02e0a203-8349-47fc-8d61-8987b4197f1c |
| id                  | c1dad50e-fd98-4863-9d42-8d383ad441e4 |
| port_id             |                                      |
| router_id           |                                      |
| tenant_id           | d8293c068214472d8008000a79d369da     |
+---------------------+--------------------------------------+

Determine the port_id for the VIP:

$ quantum port-list
+--------------------------------------+------------------------------------------+-------------------+---------------------------------------------------------------------------------+
| id                                   | name                                     | mac_address       | fixed_ips                                                                       |
+--------------------------------------+------------------------------------------+-------------------+---------------------------------------------------------------------------------+
| 1334c522-14ad-4420-bc1a-53031ed68df9 | vip-37a3eeff-c24b-425f-8f29-c8288e1081c6 | fa:16:3e:e6:87:de | {"subnet_id": "82294d13-5e55-4ae2-bc95-9d4e09ab4ebf", "ip_address": "10.0.0.8"} |
| 4cfc707f-f78a-4bd9-9c27-bb6f8b01dfcc |                                          | fa:16:3e:03:f7:c4 | {"subnet_id": "82294d13-5e55-4ae2-bc95-9d4e09ab4ebf", "ip_address": "10.0.0.2"} |
| 5620c0d6-046e-49de-b4ab-5d8948a78cbf |                                          | fa:16:3e:12:3b:75 | {"subnet_id": "82294d13-5e55-4ae2-bc95-9d4e09ab4ebf", "ip_address": "10.0.0.5"} |
| 5912c994-be8e-4b9e-8fe9-6bcb1f1c89dd |                                          | fa:16:3e:ba:04:b7 | {"subnet_id": "82294d13-5e55-4ae2-bc95-9d4e09ab4ebf", "ip_address": "10.0.0.1"} |
| 5cfcf996-4a3a-4a68-84e5-5c68d06832b4 |                                          | fa:16:3e:47:2e:72 | {"subnet_id": "82294d13-5e55-4ae2-bc95-9d4e09ab4ebf", "ip_address": "10.0.0.6"} |
| a219f389-c792-4087-a3a6-d8b9c681c170 |                                          | fa:16:3e:71:ce:0a | {"subnet_id": "82294d13-5e55-4ae2-bc95-9d4e09ab4ebf", "ip_address": "10.0.0.4"} |
| edeb0b7b-89f1-4018-9eae-fbf70bcee7ac |                                          | fa:16:3e:59:c8:ea | {"subnet_id": "82294d13-5e55-4ae2-bc95-9d4e09ab4ebf", "ip_address": "10.0.0.3"} |
| faa8718a-2efd-443f-a1b0-29c06c281e72 |                                          | fa:16:3e:5a:9d:a9 | {"subnet_id": "82294d13-5e55-4ae2-bc95-9d4e09ab4ebf", "ip_address": "10.0.0.7"} |
+--------------------------------------+------------------------------------------+-------------------+---------------------------------------------------------------------------------+

Associate VIP port with floating IP:

$ quantum floatingip-associate c1dad50e-fd98-4863-9d42-8d383ad441e4  1334c522-14ad-4420-bc1a-53031ed68df9
Associated floatingip c1dad50e-fd98-4863-9d42-8d383ad441e4

At this point the VIP port is a member of the ‘default’ security group which does not allow ingress traffic unless you are also part of this security group so we need to update the VIP port to be a member of the web security group so that requests from the internet are allowed to pass (not just from our client instance).
Get the web security group uuid:

$ quantum security-group-list
+--------------------------------------+----------+-------------+
| id                                   | name     | description |
+--------------------------------------+----------+-------------+
| 4a65c786-b7a2-4158-a8fd-8ceff9d40337 | ssh      |             |
| 51eaf14f-d873-4e6b-ac12-648ba0bf4ce0 | web      |             |
| e11e7617-2683-48f2-a474-8634721248d2 | default  | default     |
| e637afb3-831e-4298-8e54-ceddedcd33ee | database |             |
+--------------------------------------+----------+-------------+

Update VIP port to be a member of the web security group:

$ quantum port-update 1334c522-14ad-4420-bc1a-53031ed68df9 --security_groups list=true 51eaf14f-d873-4e6b-ac12-648ba0bf4ce0
Updated port: 1334c522-14ad-4420-bc1a-53031ed68df9

At this point your VIP is publicly addressable:

$ wget -O - http://172.24.4.228
Connecting to 172.24.4.228 (172.24.4.228:80)
web_server1
               100% |************************************| 12 0:00:00 ETA

$ wget -O - http://172.24.4.228
Connecting to 172.24.4.228 (172.24.4.228:80)
web_server2
               100% |************************************| 12 0:00:00 ETA

To demonstration high availability, we’ll go and delete our web_server1 instance to simulate a failure.

$ nova delete web_server1

After our health monitor detects this. it will stop sending request to web_server1 and we can see this happening here as web_server2 handles all the requests:

$ wget -O - http://172.24.4.228
Connecting to 172.24.4.228 (172.24.4.228:80)
web_server2
               100% |************************************| 12 0:00:00 ETA

$ wget -O - http://172.24.4.228
Connecting to 172.24.4.228 (172.24.4.228:80)
web_server2
               100% |************************************| 12 0:00:00 ETA

I’d like to give a shout out to the whole OpenStack community on the amazing amount of work that was done in Grizzly and I’m excited for what’s in store for Havana!

Posted in openstack, Uncategorized | 9 Comments

Open vSwitch and Libvirt

Recently I started playing with libvirt in order to manage all of my virtual machines since it’s able to handle most hypervisors through one API. One issue I ran into though is that I use Open vSwitch as my bridging mechanism. For those of you who do not know what Open Vswitch is, it’s basically a virtual switch that has a ton of cool features like OpenFlow, 802.1Q , sFlow, and many other things. I use Open vSwitch mostly  as a software OpenFlow switch for my thesis work. Open vSwitch has something called brcompat which makes brctl compatibly with OVS bridges.  This said it seems like libvirt is not compatible with it. For example when I used virsh in order to start a vm and attach it to an OVS bridge it said, “error: Failed to add tap interface to bridge ‘br0′: Invalid argument”. Though, when I ran brctl by hand I could attach the tap interface to br0 just fine. I’m not sure if this is a bug or I was missing something in my setup.

Anyways.. Libvirt now has support for interfacing with OVS bridges, so I was interested in trying that out. The first step to doing this was to get the latest libvirt source code with these new changes. Since the underlying host runs gentoo linux and provides a live-ebuild for libvirt I started using that. Unfortunately the ebuild for that is not up to date with the current source in libvirt repo. In order to fix this I needed to add an overlay which required the following steps:

mkdir -p /usr/local/portage/profiles/
echo "my_local_overlay" > /usr/local/portage/profiles/repo_name
Added PORTDIR_OVERLAY="/usr/local/portage/" to  /etc/make.conf
mkdir -p /usr/local/portage/app-emulation/libvirt
cp /usr/portage/app-emulation/libvirt/libvirt-9999.ebuild \
    /usr/local/portage/app-emulation/libvirt/
cp -r /usr/portage/app-emulation/libvirt/files \
   /usr/local/portage/app-emulation/libvirt/

Now libvirt-9999.ebuild needed to be corrected. Thankfully someone already figured out what needed to be changed and filed a bug report here (also thanks to hasufell from #gentoo to  pointing me to this!)  so I was able to easily make those changes and finally compile libvirt successfully.

ebuild libvirt-9999.ebuild manifest
emerge app-emulation/libvirt

(I also emerged the virt-manager from ebuild live though I don’t think the newest version is required. )

of course I could have just cloned libvirt directly from source:

git://libvirt.org/libvirt.git

though I wanted portage to do all of the heavy lifting for me such as handling dependencies, etc. This also lets me use smart-live-rebuild in order to pull the latest source code for all of my live-ebuilds so I don’t need to manage them all individuality.

Now to actually using Open vSwitch and libvirt. Open vSwitch can be downloaded from:

git clone git://openvswitch.org/openvswitch

after getting the source  follow INSTALL.Linux for instruction on how to build and configure it. First, I created a br0 and attach eth0 to it.

ovs-vsctl add-br br0
ovs-vsctl add-port br0 eth0

Next,  I changed the interface section of the configuration file for the vm telling it to use my  OVS bridge interface.

virsh edit <domain>

<interface type='bridge'>
    <mac address='52:54:00:43:1f:f4'/>
    <source bridge='br0'/>
    <virtualport type='openvswitch'>
    </virtualport>
</interface>

Now, when the vm is started it will be added to the OVS bridge. As I was saying before I use Open vSwitch as an OpenFlow switch to experiment with, though I also run several other virtual machines for different tasks on the network (Cacti/Nagios VM, etc). In order to experiment with things on the network and keep other VMs unaffected I create a special ovsbr datapath and attach a patch port between ovsbr and br0.

ovs-vsctl add-br ovsbr
ovs-vsctl add-port br0 patch-to-ovsbr
ovs-vsctl set Interface patch-to-ovsbr type=patch
ovs-vsctl set Interface patch-to-ovsbr options:peer=patch-to-br0
ovs-vsctl add-port ovsbr patch-to-br0
ovs-vsctl set Interface patch-to-br0 type=patch
ovs-vsctl set Interface patch-to-br0 options:peer=patch-to-ovsbr

After doing that I also enable OpenFlow on the ovsbr datapath and point that at my OpenFlow controller:

ovs-vsctl set-controller dp0 tcp:130.127.39.177:6633

This allows me to try out crazy things on the ovsbr and isolate this from the rest of the VMs attached to br0.

Posted in Uncategorized | Leave a comment

Blog Configuration

I figured my first post on here would be about how I setup this blog. So here it goes… blog.aaronorosen.com runs as a virtual machine using kvm. Instead of setting up and installing the vm directly using the kvm commands, I decided to try out libvirt since I had not played with it before. Libvirt is a virtualization API that supports KVM, XEN, LXC, OpenVZ, and several other hypervisors.

Since I only have one public IP address outside of the firewall and didn’t want to run this website on that, a reverse proxy was needed. For this I chose Nginx and was surprised how fully featured it was and how easy it was to use. Below shows what I needed to add to the configuration in order to make it work.

server {
    listen 80;
    server_name blog.aaronorosen.com;
    access_log /var/log/nginx/blog.aaronorosen.access.log;
    error_log /var/log/nginx/blog.aaronorosen.error.log;

    location / {
        proxy_set_header Host $host;
        proxy_pass http://130.127.39.238;
    }
}

What this does is when request comes to blog.aaronorosen.com, nginx looks at the HTTP header and forwards it to the proxy_pass location. The only gotcha I ran into is that I used a vhost entry for the domain on the vm and unless proxy_set_header is specified the host headers from the request are not forward.  This cause vhost entry to obviously not work.

As far as the VM goes it runs gentoo linux and installing wordpress on it was very easy following this guide.  The only issue I ran into with wordpress is that wordpress uses the URL that you use to configure it as it’s / location. I initially configured it using http://130.127.39.238/wordpress so all of the linkage on the site was set to that address which was definitely not what I wanted. To correct this the following commands were needed to be run against the database.

UPDATE wp_options set `option_value` = 'http://blog.aaronorosen.com' where `option_name` = 'siteurl';
UPDATE wp_options set `option_value` = 'http://blog.aaronorosen.coml' where `option_name` = 'home';

After that that everything was up and running!

Posted in Uncategorized | 1 Comment

Hello World!

So I figured I’d setup a blog where I could share random tidbits of things that I found interesting. More to come later. Stay tuned!

Posted in Uncategorized | 2 Comments