Work done by China Mobile in the Big Cloud Elastic Computing System

The aim of this post is to describe a new OpenNebula LXC driver (OneLXC) developed by China Mobile to allow the management of hosts and the deployment of lxc domains in OpenNebula using the LXC hypervisor.

Features

OneLXC mainly consists of two components:

• IM_MAD: a series of remote scripts that are able to monitor the remote hosts
• VMM_MAD: a series of remote scripts to manage lxc domains.

OneLXC is very similar with kvm driver because libvirt can support kvm and lxc. They both use virsh command to monitoring hosts and operating virtual machines.

Currently OneLXC supports some simple functions as follows:

• monitoring host information, for example, cpu and memory
• deploy/delete/monitoring(poll) lxc domains and their info

Developing Enviroment

• three host machines with ubuntu 12.04 operating system（amd64
• opennebula-3.2.1
• libvirt-0.9.8
• lxc-0.7.5

How to install and use OneLXC Driver?

To install the OneLXC driver run the “install.sh” script provided. This script will copy the necessary files into the OpenNebula installation tree alongside OpenNebula itself.

Driver Configuration

In order to enable the OneLXC driver, it is necessary to modify “oned.conf” accordingly. This is achieved by setting the IM_MAD and VM_MAD options as follows:

#-------------------------------------------------------------------------------
# LXC Information Driver Manager Configuration
# -r number of retries when monitoring a host
# -t number of threads, i.e. number of hosts monitored at the same time
#-------------------------------------------------------------------------------
IM_MAD = [
name = "im_lxc",
executable = "one_im_ssh",
arguments = "-r 0 -t 15 lxc" ]

#-------------------------------------------------------------------------------
VM_MAD = [
name = "vmm_lxc",
executable = "one_vmm_exec",
arguments = "-t 15 -r 0 lxc",
default = "vmm_exec/vmm_exec_lxc.conf",
type = "lxc" ]
#-------------------------------------------------------------------------------

The name of the driver needs to be provided at the time of adding a new host to OpenNebula. For example, we can use command “onehost create” and “onehost list” to create and show the hosts.

After adding hosts, we can use onevm create and onevm show <vm_id> to deploy a lxc domain and show its informations, for  example as follows:

Driver files

The OneLXC driver package contains the following files. Note that they are referenced using $ONE_LOCATION as the base directory, therefore meaning a self-contained installation of OpenNebula.

• $ONE_LOCATION/etc/vmm_exec/vmm_exec_lxc.conf: Configuration file to define the default values for the LXC domain definitions.
• $ONE_LOCATION/var/remotes/vmm/lxc/: Scripts used to perform the operations on the lxc domains. These files are called “remotes”, meaning they are copied to the remote hosts and executed there.
• $ONE_LOCATION/var/remotes/im/lxc.d/: Scripts used to fetch information from the remote hosts (memory, cpu use…). These scripts are copied to the remote hosts and executed there.
• oned.conf: Example OpenNebula configuration file with the LXC drivers enabled.

Source files

• src/vmm/LibVirtDriverLXC.cc: The libvirt driver to generate the lxc domain’s deployment configuration file.

Image files

Different from kvm and xen image file, the lxc domain’s image actually is a directory called “rootfs”. For the sake of transfering easily, we compress it, copy it to the target host and decompress it again in the destination. Note： the file permission of  /usr/bin/sudo of lxc domain must be 4755.Otherwise, it can not use root privilege to execute command in lxc domain.

Virtual Machine’s Configuration file

NAME = lxc_2
CPU = 1
MEMORY = 1024
VCPU = 2

DISK = [
source = "/opt/nebula/images/lxc.tar.gz"
]

Bugs and problems

• There is a synchronous problem between the “cancel” operation and “tm_delete.sh” for OneLXC driver.
• OneLXC driver cann’t implement “reboot”, “shutdown” and “restart” operations because maybe libvirt or LXC doesn’t support.
• How to generate the lxc “config” file in “rootfs” directory dynamically? Because each lxc domain has different image file path.
• perhaps there are a lot of bugs but could not find them

Source Code Download

Because lxc driver function is not perfect，I will submit the code later. If someone want it now, you can download the draft version from here:

https://github.com/cmri/opennebula-3.2.1-lxc.git

Work done by China Mobile in the Big Cloud Elastic Computing System 

The answer is less than 5 minutes!

We recently made an interesting test to see how fast can OpenNebula deploy a huge amount of VMs in short time. The testbed includes 200 hosts and runs VMs in local disk. For some reasons the scheduler policy must be setup as Packing Policy. To accelerate the deployment speed we use the cache patch (available at dev.opennebula.org). In this test we call “onevm create” command 100 times in a script to create 100 VMs as fast as possible.

In the first test we used OpenNebula mm_sched scheduler with the following configuration:

SCHED_INTERVAL = 60
MAX_VM = 300
MAX_DISPATCH = 30
MAX_HOST = 1

The result is that about 30VMs booted in 1 minute, the next 30 VMs  in 20 minutes, and the last 40 VMs booted finally in 60 minutes.  The next figure shows the overall evolution of the VM set (X-axis is the number of VMs, and Y-axis is the deployment time in seconds).

We thought that image caching can help in this scenario, but also a couple of optimizations where needed. So we made the second test with the following configurations:

SCHED_INTERVAL = 180
MAX_VM = 300
MAX_DISPATCH = 200
MAX_HOST = 1

Additionally, we increased the concurrency for the TM ssh driver (up to 100 threads) to not to limit the number of concurrent image transfers. Thanks Ruben (OpenNebula Chief Architect) for this hint.

As the figure shows, in this round, all VMs can be finished in one sched interval, being roughly 5 minutes the slowest boot time.

The conclusion is :

1. Image Caching can greatly accelerate the deployment speed when OpenNebula uses local storage.
2. To make image cache efficient some additional optimizations to the scheduler configuration may be needed.
3. OpenNebula is good at scheduling a big amount of VM requests in short time.

Big Cloud is the cloud computing software stack developed by China Mobile Research Institute to support China Mobile‘s operation platform and provide services to its more than 600 million customers.

BC-EC, Big Cloud Elastic Computing, chose OpenNebula as its core component to manage and schedule the virtualization infrastructure in 2008. Since then, we are glad to see OpenNebula to be more full-fledged each day. And BC-EC, matured with OpenNebula, was used both in China Mobile’s internal business and ready to provide public service.

BC-EC includes 3 parts: web portal providing self-service entry, front-end management service providing service and operation management, user management and billing etc, and service database.  Front-end can provide similar functions as OZone to manage several OpenNebula back-ends.

Recently China Mobile prepared to publish a public cloud service based on BC-EC solution. This cloud includes 1000 servers, 700 servers of those will provide virtual machine computing service based on BC-EC and another 300 servers will provide cloud storage based on Big Cloud’s object store system, named Onest.

We hope to make deep cooperation with OpenNebula community to improve it with our requirements and experiences, contributing bug fixes and developing new features.