I've been experimenting with the OpenStack placement service in a container recently. This is a part of exploratory research to see what will be involved in doing one or both of the following:
Making a lightweight, but working, install of the placement service that includes as few imports and as small a configuration as possible with minimal knobs.
Eventually extracting placement from the large and convoluted nova repo into its own thing.
What follows are some notes, mostly to myself but published here in case others find them useful, about playing with putting placement in a docker-managed container and peeling off some of the useless bits.
Note: The reason for doing this is not to create a working container for placement. If that were the case I'd be collaborating with Kolla and OpenStack Ansible people. No, the goal here is to do things the manual, crufty, hard way to expose issues and learn some stuff.
I tried to keep track of myself in a git repo, called
placedock, where, when I
remembered, I would commit some state. I tested myself as I went by
having a running devstack where the placement service from devstack
was replaced by the one managed by the container. In both cases the
requests to placement are proxied through apache2 using
mod_proxy_uwsgi so all I had to was change the target of the proxy.
Testing was driven by the gabbi-tempest
plugin. This worked very well.
provides just a Dockerfile that uses an Alpine base, installs some
binary requirements that allow a
pip install -e of nova to succeed.
The nova used there is master plus a long running
change that allows
the database connection for placement to be explicitly for placement
[placement_database]/connection in the configuration file). Note
that just because there's a different connection configuration setting
doesn't mean it has to be a different database. In my tests placement
data continues to be stored in the
results in a running placement service but it uses a very large
nova.conf with plenty of stuff that is not needed. Because of a
placement bug (now
fixed), extra steps of linking the conf file within the container are
required. That was cleared in a later
The commit that trims the nova.conf is nice because it removes a lot of meaningless cruft and results in a reasonable conf file that can be the same across several instancesj of the placement service:
[DEFAULT] debug = True [placement_database] connection = mysql+pymysql://root:email@example.com/nova_api?charset=utf8 [keystone_authtoken] memcached_servers = 192.168.1.76:11211 project_domain_name = Default project_name = service user_domain_name = Default password = secret username = nova auth_type = password service_token_roles_required = True auth_url = http://192.168.1.76/identity www_authenticate_uri = http://192.168.1.76/identity
Even this is longer than it really should be. Ideally all of this could be communicated via environment variables passed to the container and no file would be needed at all. There are other options too.
The most fun commit is the one that trims the python
requirements.txt is 65 lines long and the cascade of
transitive dependencies is huge. This change gets the explicit
requirements down to less than 30, but the transitive dependencies remain huge
because of the package structure in
Nova. It's fairly common in
__init__.py to contain imports, which means any packages
within suffer those imports too. This makes fine sense if there is
only one monolithic entry-point, but in nova there are several
different api services and command line tools, running in different
processes, not all of which need all that stuff.
There are some short term fixes linked from that bug, but the long
term fix to get a lightweight placement is to either change the
__init__.py files that impact it, or (much better) extract to its
own repo. Extraction comes with its own challenges but there may be
some tricks to make it less painful (more to be written about that
The final commit in the repo (thus far) switches the uwsgi configuration to use a TCP socket instead of a unix socket. This makes it possible to distribute multiple placement containers across multiple hosts and address then from a reverse proxy. The commit message describes how things are hooked up to apache2 (the http server usually used with devstack).
Things to do Next
- Make it smaller. It ought to be possible to incrementally chip away at the imports, but it is quite a tangled mess.
- Copy the config files into the containers rather than sharing them over a volume.
- Go a step further and explore the work on etcd backends for oslo.config. This wont help the uwsgi config, but that ought to be fairly static. The dynamic configuration values are database connection strings, and authtoken settings.
- Try it with more than two placement services (works fine with two).
- Try it with multiple database servers (galera?).