Deploying Openstack Icehouse with Vagrant and Puppet Enterprise

2014-08-29 - Reading time: 13 minutes #openstack #vagrant #puppet

If you want to learn about OpenStack and want a more serious environment than what DevStack is designed to provide for you, then this might be a good place for you to start.

This blog serves to document the process I went through to bring up a multi-node OpenStack Icehouse system using Puppet Enterprise. Little to no knowledge of Puppet is required, as most of the setup is automated, but of course if you do know some Puppet, all the better!

The repo is available to clone here: vagrant-puppet-openstack

I’m a big fan of Vagrant so my target was to bring up a working OpenStack environment from the basic “vagrant up” command (which ended up being overly ambitious using this method). I’ve tested all this with VirtualBox on OSX Mavericks. The base image we’re going to use is a PuppetLabs Centos 6.5 box, and we’re going to end up with the following nodes:

  • Puppet Master
  • Controller Node
  • Network Node
  • Compute Node
  • Storage Node

Needless to say all these VMs are going to take up a bit of memory on your host. I wouldn’t recommend this for anyone with less than 8GB RAM.

This is what the system we are building will look like: (Credits to Hodgepodge from Puppet Labs for this information, as we’ll more or less be following his procedure described in this video)

If you checked out the video, you’ll notice that the “vagrant up” part of the process has been skipped. Since that was my target to build the entire system here, then it’s kind of important that we understand these steps!

So lets go through the Vagrantfile piece by piece. It’s important to note that I am using a Vagrant plugin called oscar to provision the puppet master and nodes and also provision the hosts files. You can install this plugin by vagrant plugin install oscar. If you don’t install it your build will break when you run vagrant up.

The Vagrantfile

Below we have some general system setup such as the base box and the version of Puppet Enterprise to use. Next is some provisioning that happens on all the nodes in the environment. The lines to disable the iptables is really just there to help the initial provisioning of the system. When the puppet agent runs it will enable the firewall again, as it’s critical to OpenStack functionality. We also need to fix the hosts file to make sure that the FQDN added by the :hosts provisioner points to the loopback address rather than 127.0.1.1 otherwise we’ll have problems with RabbitMQ.

Vagrant.configure('2') do |config|
  config.vm.box                      = 'centos-64-x64-vbox4210-nocm'
  config.vm.box_url                  = 'http://puppet-vagrant-boxes.puppetlabs.com/centos-64-x64-vbox4210-nocm.box'
  config.pe_build.download_root      = 'https://s3.amazonaws.com/pe-builds/released'
  config.pe_build.version            = '3.2.3'
  config.pe_build.filename           = 'puppet-enterprise-3.2.3-el-6-x86_64.tar.gz'
  
  config.vm.provision 'shell', inline: 'yum update -y'
  # Only here to assist with initial provisioning of machines. After puppet agent run iptables is reenabled and started 
  config.vm.provision 'shell', inline: 'chkconfig iptables off'
  config.vm.provision 'shell', inline: 'service iptables stop'
  config.vm.provision :hosts
  # Fix this for RabbitMQ
  config.vm.provision 'shell', inline: 'sed -i "s/^127.0.1.1/127.0.0.1/g" /etc/hosts'

Now we take a look at the Puppet Master config. As already mentioned, Puppet Enterprise is actually installed by the pe_build plugin (part of osar) using the :pe_bootstrap provisioner. We make sure to specify the FQDN for the puppet.conf, and then we run a bunch of scripts to install the puppet modules, configure hiera and added the nodes and classes.

  # Puppet Master config
  config.vm.define 'master' do |master|
    master.vm.hostname = 'master.localnet'
    master.vm.network :private_network, ip: "192.168.11.3"
    master.vm.network :private_network, ip: "192.168.22.3"
    master.vm.network :private_network, ip: "172.16.33.3"
    master.vm.network :private_network, ip: "172.16.44.3"
    master.vm.provider :virtualbox do |vbox|
      vbox.customize ["modifyvm", :id, "--memory", "1024"]
    end

    master.vm.provision :pe_bootstrap do |provisioner|
      provisioner.role = :master
      provisioner.master = "master.localnet"
    end

    master.vm.provision "shell", inline: $install_puppet_modules
    master.vm.provision "shell", inline: $hiera_setup
    master.vm.provision "shell", inline: $add_node_classes
    master.vm.provision "shell", inline: $add_nodes

  end # master

The install_puppet_modules script explicitly declares the desired versions of the puppet modules that we use to build the environment. This is necessary because the version pinning of the module dependencies in the puppetlabs-openstack module is loose, which means you will get different versions over time if you rebuild the environment. I had a multitude of issues with this, from not finding a compatible sets of modules and the install failing, to having sets of modules that produced 500 internal server errors when the environment finished provisioning. This set of modules I have at least verified to produce a basic working environment.

$install_puppet_modules = <<SCRIPT
/usr/local/bin/puppet module install puppetlabs-openstack  --version 4.1.0 --ignore-dependencies --modulepath '/etc/puppetlabs/puppet/modules'
/usr/local/bin/puppet module install dprince-qpid          --version 1.0.2 --ignore-dependencies --modulepath '/etc/puppetlabs/puppet/modules'
/usr/local/bin/puppet module install duritong-sysctl       --version 0.0.4 --force --modulepath '/etc/puppetlabs/puppet/modules'
/usr/local/bin/puppet module install garethr-erlang        --version 0.3.0 --ignore-dependencies --modulepath '/etc/puppetlabs/puppet/modules'
/usr/local/bin/puppet module install puppetlabs-apache     --version 1.1.0 --ignore-dependencies --modulepath '/etc/puppetlabs/puppet/modules'
/usr/local/bin/puppet module install puppetlabs-ceilometer --version 4.1.0 --ignore-dependencies --modulepath '/etc/puppetlabs/puppet/modules'
/usr/local/bin/puppet module install puppetlabs-cinder     --version 4.1.0 --ignore-dependencies --modulepath '/etc/puppetlabs/puppet/modules'
/usr/local/bin/puppet module install puppetlabs-glance     --version 4.1.0 --ignore-dependencies --modulepath '/etc/puppetlabs/puppet/modules'
/usr/local/bin/puppet module install puppetlabs-heat       --version 4.1.0 --ignore-dependencies --modulepath '/etc/puppetlabs/puppet/modules'
/usr/local/bin/puppet module install puppetlabs-horizon    --version 4.1.0 --ignore-dependencies --modulepath '/etc/puppetlabs/puppet/modules'
/usr/local/bin/puppet module install puppetlabs-keystone   --version 4.1.0 --ignore-dependencies --modulepath '/etc/puppetlabs/puppet/modules'
/usr/local/bin/puppet module install puppetlabs-rabbitmq   --version 3.1.0 --ignore-dependencies --modulepath '/etc/puppetlabs/puppet/modules'
/usr/local/bin/puppet module install puppetlabs-mongodb    --version 0.8.0 --ignore-dependencies --modulepath '/etc/puppetlabs/puppet/modules'
/usr/local/bin/puppet module install puppetlabs-mysql      --version 2.3.0 --ignore-dependencies --modulepath '/etc/puppetlabs/puppet/modules'
/usr/local/bin/puppet module install puppetlabs-neutron    --version 4.2.0 --ignore-dependencies --modulepath '/etc/puppetlabs/puppet/modules'
/usr/local/bin/puppet module install puppetlabs-nova       --version 4.1.0 --ignore-dependencies --modulepath '/etc/puppetlabs/puppet/modules'
/usr/local/bin/puppet module install puppetlabs-rsync      --version 0.3.0 --ignore-dependencies --modulepath '/etc/puppetlabs/puppet/modules'
/usr/local/bin/puppet module install puppetlabs-swift      --version 4.1.0 --ignore-dependencies --modulepath '/etc/puppetlabs/puppet/modules'
/usr/local/bin/puppet module install puppetlabs-tempest    --version 3.0.0 --ignore-dependencies --modulepath '/etc/puppetlabs/puppet/modules'
/usr/local/bin/puppet module install puppetlabs-vcsrepo    --version 0.2.0 --force --modulepath '/etc/puppetlabs/puppet/modules'
/usr/local/bin/puppet module install puppetlabs-vswitch    --version 0.3.0 --ignore-dependencies --modulepath '/etc/puppetlabs/puppet/modules'
/usr/local/bin/puppet module install puppetlabs-xinetd     --version 1.3.0 --ignore-dependencies --modulepath '/etc/puppetlabs/puppet/modules'
/usr/local/bin/puppet module install saz-memcached         --version 2.5.0 --ignore-dependencies --modulepath '/etc/puppetlabs/puppet/modules'
/usr/local/bin/puppet module install saz-ssh               --version 1.4.0 --ignore-dependencies --modulepath '/etc/puppetlabs/puppet/modules'
/usr/local/bin/puppet module install stahnma-epel          --version 0.1.0 --ignore-dependencies --modulepath '/etc/puppetlabs/puppet/modules'
/usr/local/bin/puppet module install puppetlabs-ntp        --version 3.0.3 --ignore-dependencies --modulepath '/etc/puppetlabs/puppet/modules'
# hackery needed to get the module version above to play nice
chmod u+w /etc/puppetlabs/puppet/modules/openstack/metadata.json
sed -i -e 's@duritong/sysctl\",\"version_requirement\":\"@duritong/sysctl\",\"version_requirement\":\"\>=@g' /etc/puppetlabs/puppet/modules/openstack/metadata.json
SCRIPT

The hiera_setup is more or less following the instructions from Hodepodge. We need to switch the libvirt type from KVM to qemu otherwise your instances won’t start because hvm is unavailable in the VMs in virtualbox. Youmay need to change the network configuration here depending on the local network you host is connected too. I’ll go into a bit more detail about this toward the end of the post where we need to set up NAT and IP forwarding so that your instances get internet access.

$hiera_setup = <<SCRIPT
mkdir /etc/puppetlabs/puppet/hieradata
cp /etc/puppetlabs/puppet/modules/openstack/examples/common.yaml /etc/puppetlabs/puppet/hieradata/openstack.yaml
chgrp -R pe-puppet /etc/puppetlabs/puppet/hieradata
sed -i 's/^openstack::network::external::gateway:.*/openstack::network::external::gateway: 192.168.22.1/g' /etc/puppetlabs/puppet/hieradata/openstack.yaml
sed -i 's/^openstack::network::external::dns:.*/openstack::network::external::dns: 8.8.8.8/g' /etc/puppetlabs/puppet/hieradata/openstack.yaml
sed -i "s@^openstack::network::neutron::private:.*@openstack::network::neutron::private: \'10.10.10.0/24\'@g" /etc/puppetlabs/puppet/hieradata/openstack.yaml
sed -i "s@^openstack::nova::libvirt_type:.*@openstack::nova::libvirt_type: \'qemu\'@g" /etc/puppetlabs/puppet/hieradata/openstack.yaml
sed -i "s/^openstack::keystone::admin_email:.*/openstack::keystone::admin_email: \'philip.cheong@elastx.se\'/g" /etc/puppetlabs/puppet/hieradata/openstack.yaml
awk '{print} sub(/- global/,"- openstack")' /etc/puppetlabs/puppet/hiera.yaml > hiera.new && mv -f hiera.new /etc/puppetlabs/puppet/hiera.yaml
sed -i 's@:datadir:@:datadir: /etc/puppetlabs/puppet/hieradata@g' /etc/puppetlabs/puppet/hiera.yaml
service pe-puppet restart
SCRIPT

Finally we add classes, create nodes and allocate roles to them so that when the agents connect they will have the correct config.

$add_node_classes = <<SCRIPT
/opt/puppet/bin/rake -f /opt/puppet/share/puppet-dashboard/Rakefile RAILS_ENV=production nodeclass:add[openstack::role::compute,skip]
/opt/puppet/bin/rake -f /opt/puppet/share/puppet-dashboard/Rakefile RAILS_ENV=production nodeclass:add[openstack::role::controller,skip]
/opt/puppet/bin/rake -f /opt/puppet/share/puppet-dashboard/Rakefile RAILS_ENV=production nodeclass:add[openstack::role::network,skip]
/opt/puppet/bin/rake -f /opt/puppet/share/puppet-dashboard/Rakefile RAILS_ENV=production nodeclass:add[openstack::role::storage,skip]
SCRIPT

$add_nodes = <<SCRIPT
/opt/puppet/bin/rake -f /opt/puppet/share/puppet-dashboard/Rakefile RAILS_ENV=production node:add[control.localnet,,openstack::role::controller,skip]
/opt/puppet/bin/rake -f /opt/puppet/share/puppet-dashboard/Rakefile RAILS_ENV=production node:add[compute.localnet,,openstack::role::compute,skip]
/opt/puppet/bin/rake -f /opt/puppet/share/puppet-dashboard/Rakefile RAILS_ENV=production node:add[network.localnet,,openstack::role::network,skip]
/opt/puppet/bin/rake -f /opt/puppet/share/puppet-dashboard/Rakefile RAILS_ENV=production node:add[storage.localnet,,openstack::role::storage,skip]
SCRIPT

All of the nodes look similar in their setup, so there is not much more to say about them except for noting that the network and compute nodes have an important configuration on eth2 to set them to promiscuous mode with the --nicpromisc3 flag (the 3 is the third interface, eth2). This needs to be done in order to make sure that the instances see the traffic that occurs on these network interfaces, which act as the external network.

  # Network node config
  config.vm.define 'network' do |network|
    network.vm.hostname = 'network.localnet'
    network.vm.network :private_network, ip: "192.168.11.6"
    network.vm.network :private_network, ip: "192.168.22.6"
    network.vm.network :private_network, ip: "172.16.33.6"
    network.vm.network :private_network, ip: "172.16.44.6"
    network.vm.provider :virtualbox do |vbox|
      vbox.customize ["modifyvm", :id, "--memory", "640"]
      vbox.customize ["modifyvm", :id, "--nicpromisc3", "allow-all"]
    end

    network.vm.provision :pe_bootstrap do |provisioner|
      provisioner.role = :agent
      provisioner.master = "master.localnet"
    end
    
    network.vm.provision "shell", inline: "/usr/local/bin/puppet agent -tvd &"
  end # network

When you execute vagrant up, it will take a bit over an hour for the whole environment to be built. This time could be sped up quite a bit by repackaging the base image after a yum update had been performed.

Puppet Master login information The puppet master is available on https://192.168.11.3. These login details are created by vagrant-pe_build.

user: admin@puppetlabs.com

pass: puppetlabs

The Puppet Master should report that the nodes converged successfully. If it reports failures, it may not necessarily be a problem, so long as the total number of resources converged on each node are the same as above. After initial successful convergence, it is not uncommon to have non-critical puppet run failures afterwards. So long as you get an complete convergence at least once then any subsequent failures (often due to firewalls) should not affect the functioning of the system.

Horizon login information

Openstack horizon dashboard is available on http://192.168.11.4. These login details are configured in the openstack.yaml hiera file.

user: test

pass: abc123

It’s always important to check the state of all the OpenStack services in the System Info. After the initial convergence, the State of everything should be “Up”, but if you restart the environment some will be “Down” as I show here above (this is after a reboot).

Make sure you restart each service that is indicated as Down.

After the initial convergence all the services should appear as “Up”, but from the experience I’ve had working with this system you will hit a number of common issues that you’ll need to learn how to identify and fix.

Debugging

Either due to the behaviour of Openstack or how the puppet modules configure the Openstack services, there are a bunch of issues that you will run into with this system that you need to know how to detect and fix to make it work properly. I will try to document here all the main things that I’ve found (at least with this version and method of building the system).

  1. Check that your network namespaces have been created successfully. If these aren’t here then you need to reboot the network node.

    [root@network ~]# ip netns
    qdhcp-9fb0a8c7-3568-4aa0-a836-0472e9b7c56c
    qrouter-5a2d57bc-96d7-4855-9dc5-2271cce5b602
  2. Check that iptables on the network and compute node show a bunch of rules beginning with “neutron-openvswi-”. If not, you need to restart the neutron-openvswitch-agent on the node missing the rules. Check the iptables again and the rules should appear. Here is a snippet of some of the rules you should see on the compute node.

    [root@compute ~]# iptables -S
    -P INPUT ACCEPT
    -P FORWARD ACCEPT
    -P OUTPUT ACCEPT
    -N neutron-filter-top
    -N neutron-openvswi-FORWARD
    -N neutron-openvswi-INPUT
    -N neutron-openvswi-OUTPUT
    -N neutron-openvswi-i45ba26ee-6
    -N neutron-openvswi-local
  3. Check that the router has the correct iptables rules. You should see rules beginning with “neutron-l3-agent-” and “neutron-vpn-agen-”. If you don’t, then you need to restart the neutron-l3-agent or the neutron-vpn-agent on the network node.

    [root@network ~]# ip netns exec qrouter-5a2d57bc-96d7-4855-9dc5-2271cce5b602 iptables -S
    -P INPUT ACCEPT
    -P FORWARD ACCEPT
    -P OUTPUT ACCEPT
    -N neutron-filter-top
    -N neutron-l3-agent-FORWARD
    -N neutron-l3-agent-INPUT
    -N neutron-l3-agent-OUTPUT
    -N neutron-l3-agent-local
    -N neutron-vpn-agen-FORWARD
    -N neutron-vpn-agen-INPUT
    -N neutron-vpn-agen-OUTPUT
    -N neutron-vpn-agen-local
    -A INPUT -j neutron-vpn-agen-INPUT
    -A INPUT -j neutron-l3-agent-INPUT
  4. A common problem is DHCP discovery not working correctly, meaning that the created instance never get’s an IP address from the DHCP server. You can easily tell this in the console log by these messages:

    Starting network...
    udhcpc (v1.20.1) started
    Sending discover...
    Sending discover...
    Sending discover...
    No lease, failing
    WARN: /etc/rc3.d/S40-network failed

There are multiple reason why this can happen. DHCP discovery messages are sent from the instance, down the network stack of the compute node, up the network stack of the network node and to the DHCP server. Discovery Offers from the server have to traverse a similar path back. There are multiple services that set up all this networking and each can have different problems making the traffic disappear anywhere along this route in either direction of packet flow. Here is a good link to explain more about Neutron networking in OpenStack and also how to troubleshoot networking issues.

This is what a DHCP discovery packet should look like when it is seen correctly on the integration bridge interface (br-int) of the compute node.

[root@compute ~]# tcpdump -i br-int -vvv -s 1500 '((port 67 or port 68) and (udp\[8:1] = 0x1))'
tcpdump: WARNING: br-int: no IPv4 address assigned
tcpdump: listening on br-int, link-type EN10MB (Ethernet), capture size 1500 bytes
11:18:56.194909 IP (tos 0x0, ttl 64, id 0, offset 0, flags \[none], proto UDP (17), length 308)
0.0.0.0.bootpc > 255.255.255.255.bootps: [udp sum ok] BOOTP/DHCP, Request from

fa:16:3e:21:be:cf (oui Unknown), length 280, xid 0x4d42147f, Flags \[none] (0x0000)
  Client-Ethernet-Address fa:16:3e:21:be:cf (oui Unknown)
  Vendor-rfc1048 Extension
  Magic Cookie 0x63825363
  DHCP-Message Option 53, length 1: Discover
  Client-ID Option 61, length 7: ether fa:16:3e:21:be:cf
  MSZ Option 57, length 2: 576
  Parameter-Request Option 55, length 7:
    Subnet-Mask, Default-Gateway, Domain-Name-Server, Hostname
    Domain-Name, BR, NTP
  Vendor-Class Option 60, length 12: "udhcp 1.20.1"
   END Option 255, length 0 
  • If you are not seeing DHCP discovery messages on the br-int interface of the compute node then you should restart the openstack-nova-compute and neutron-openvswitch-agent.
  • If you are not seeing DHCP discovery messages on the br-int interface of the network node then you should restart the neutron-openvswitch-agent and neutron-l3-agent.
  • If you are not seeing DHCP offer responses on the br-int interface of the network node then you should restart the neutron-dhcp-agent.
  1. If you have problems with the Horizon user interface with an error page (might be a 50x or 40x) saying “Oops something went wrong” then check to see if the rabbitmq-server is running on the control node.

  2. If you can ssh to the created instance by don’t have internet access you probably haven’t got internet access you go back through the directions to set up packet forwarding and NAT to the instance.

Additional host setup for NAT and IP forwarding

Because the OpenStack external network is actually setup as a host-only network in virtualbox, the only way to give created instances internet access is to setup NAT for them. At the moment, I don’t have a better way to do this than to manually set this up on the host machine. I think it might be possible to specify the external network to use a bridged network, but then you’re probably going to run into issues with static ips on a network with dhcp enabled (unless you’re on a network that doesn’t have dhcp, but when is that ever the case?).

So here is how to setup ip forwarding and NAT on your (OSX) host: Enable ip forwarding:

sudo sysctl -w net.inet.ip.forwarding=1

Add this line to the file /etc/pf.conf on your host after the line: ‘nat-anchor “com.apple/*“’

nat on { en0 } from 192.168.22.101/32 to any -> { (en0) }

Flush and reload the new rules:

sudo pfctl -F all -f /etc/pf.conf

Enable the packet filter:

sudo pfctl -e

There are a couple of caveats with this method:

  • If you are connecting to the net on an interface other than en0, then make sure you change the nat rule to the correct interface
  • If you have more than once instance you will need to add another rule for that too, or fiddle around to get a correct CIDR that only your instances are on. You can’t just forward everything on 192.168.22.0/24 because some traffic on that network is destined for node-node communication. I never bothered to do this because I recognise there are limits to how much I can torture my poor little laptop
  • If you reboot your host, then you’ll need to run the commands above again
  • If you’re on linux, you can do the same with iptables. If you’re on Windows… I’m sorry

I hope that you found this post interesting and useful. If you have any questions or comments please email me at philip.cheong@elastx.se

Philip Cheong

#openstack #vagrant #puppet