One of the nice features that's been built into gabbi from near the start is a command line runner that allows gabbi tests to be run against a running web service. Given args.yaml:

tests:
- name: check args
  GET: /get?foo=bar
  response_json_paths:
      $.args.foo: bar

You can do:

gabbi-run http://httpbin.org < args.yaml

and get:

... ✓ gabbi-runner.input_check_args

----------------------------------------------------------------------
Ran 1 test in 0.379s

In OpenStack development, gabbi is frequently used in functional test for the HTTP services, sometimes paired with wsgi-intercept. In this context it is relatively easy to set up. Using it for integration tests, however, gets a bit more complicated because there's a lot to set up so that gabbi is properly aware of the services in the cloud that's being tested. The gnocchi and heat projects have done a fair bit of work to make tools that work for them.

gabbi-tempest

Based off that work, I've created a plugin for Tempest (an integration testing framework for OpenStack) called gabbi-tempest that tries to take care of some of the necesary set up while preserving the ease of use you get with gabbi-run. The result allows commands like:

GABBI_TEMPEST_PATH=/home/cdent/ds1/gabbits tempest run --regex gabbi

gabbi-tempest itself is responsible for establishing simple credentials and providing gabbi with enough environment variables to be able to find the endpoints for all the services in the service catalog. From there it is possible to write gabbi test files that uses those services.

Trying It Out

I've made a little git repo, gabbi-tempest-sample that shows the results of experimenting with gabbi-tempest to talk to a cloud (created by devstack in this case, but that's not too relevant). You can repeat the experiment yourself with the following instructions.

Install tempest and gabbi-tempest somewhere (if you prefer to work in a virtualenv, please do):

git clone https://git.openstack.org/openstack/tempest
pip install tempest/
pip install gabbi-tempest

Tempest plugins work using entry points, so if tempest and gabbi-tempest are in the same Python environment, tempest will be aware of the plugin without further effort.

Create a workspace from which testing will happen. This will create a basic tempest.conf that will be used to find the target cloud. I've named my workspace ds1 because that's the host where my devstack-based cloud lives.

tempest init ds1

Edit tempest.conf to add information for accessing the Keystone service and identifying an existing user and password in your cloud. In my tests I'm using Keystone v3. If you are not you will need to make somewhat different changes to your configuration. At the moment, the gabbi-tempest plugin requires a user named 'admin' but that will change.

The intent here is have a minimal configuration. Here's what my file looks like:

[DEFAULT]
log_dir = /home/cdent/ds1/logs
log_file = tempest.log

[oslo_concurrency]
lock_path = /home/cdent/ds1/tempest_lock

[identity]
auth_version = v3
admin_domain_scope = True
uri_v3 = http://192.168.1.76/identity/v3

[auth]
admin_domain_name = Default
admin_project_name = admin
admin_password = replace_me
admin_username = admin

uri_v3 is the URI for accessing the keystone service.

Create some YAML to confirm things are working. It's common for gabbi YAML files to be called gabbits. Create a directory with that name and in there create a new file (replace vim with your preferred editor):

mkdir gabbits
vim gabbits/sample.yaml

Gabbi YAML files are a sequence of HTTP requests in a tests attribute, each of which uses any defaults defined in a defaults attribute. We will set some defaults that will be used in every request. The primary purpose here is to make sure that authentication is handled and that the proper content types and microversions are used. For the sake of this experiment we will use the latest microversions, but usually that's not what you want. Add a default section:

defaults:
    request_headers:
        x-auth-token: $ENVIRON['SERVICE_TOKEN']
        content-type: application/json
        accept: application/json
        openstack-api-version: 'compute latest, placement latest'

The SERVICE_TOKEN has been established by the plugin. Any test which does not override these headers will use them.

Now let's create one simple test that lists the servers on the cloud and tests that there are 0 of them:

tests:
    - name: list servers
      GET: $ENVIRON['COMPUTE_SERVICE']/servers
      response_json_paths:
          $.servers.`len`: 0

It also tests that the response status code is 200. If you want to test for a different status each test can take a status field. See gabbi test format docs for more information.

Run that test. To do so we need to tell tempest to only run gabbi tests. We do that with the --regex parameter (but see the caveats, below) By default the plugin will look in the current working directory for a directory named gabbits to find one or more YAML files:

tempest run --regex gabbi

If there are in fact zero servers, you'll see test output that looks something like this:

{0} tempest.scenario.home.cdent.ds1.gabbits.sample_list_servers.test_request [0.260659s] ... ok

Make the test verbose to see what's really happening. By changing the test to add verbose: True we can see the request and response. Make the list servers test look like this:

    - name: list servers
      verbose: True
      GET: $ENVIRON['COMPUTE_SERVICE']/servers
      response_json_paths:
          $.servers.`len`: 0

Now when you run the tests (as above), you'll also see verbose representations of the request and response bodies and headers. You can make this the default for the file by putting verbose: True in the defaults section instead.

Getting More Complicated

The repo includes a considerably more complex example. This one talks to three services, accepts information from environment variables passed on the command line, and uses advance features of both YAML and gabbi to make reference to data in other responses in the sequence. What it tests is the behavior of the placement service when booting a single server and resizing it.

To try it out, the user in your tempest.conf must have the admin role, as the placement service requires that. Move away your existing sample.yaml and replace it with the complex one. When running it we'll demonstrate some additional features:

GABBI_TEMPEST_PATH=`pwd`/gabbits:/some/other/gabbits FLAVOR_NAME=m1.nano \
   FLAVOR_NAME_ALT=m1.micro IMAGE_NAME=cirros-0.3.5-x86_64-disk \
   tempest run --regex gabbi

GABBI_TEMPEST_PATH contains one or more directory paths in which the gabbi-tempest plugin should look for gabbi YAML files. It is formatted in the same fashion as the standard Unix PATH (colon separated directories). This allows a caller to keep their gabbits in lots of different places, if that's useful.

The NAME variables shows ways in which the caller can pass information into the tests. In this case we are telling the tests which flavors and images to work with, without needing to know their ids (the ids are discovered by tests and referred to in subsequent tests).

Hopefully you can see that this has some potential. I would love some feedback and of course any offers of help are very welcome.

Also, if you've been paying attention, you've probably noticed that tempest isn't really needed here. It's a convenience that allows you to be integrated with the way that tempest accounts for tests and gathers information about the environment. If you don't need that, you could set all the environment variables that are used for substituting service location and auth yourself and use gabbi-run directly.

Caveats

While the above experiments have worked for me there are some areas of concern that are going to need some additional learning or work:

• tempest run does test discovery in a way that means the --regex handling does not match on the test names that gabbi generates in its dynamic processing. It is matching on the plugin's code. It may be the case that this situation will change when tests are run in parallel (where tests are listed, grouped, and fed back to the runner) but that introduces other issues.
• If the --regex parameter is not used, then all tests that tempest can find are run, including the tests provided by the gabbi-tempest plugin and any other plugin active in the environment.
• In each individual YAML file the test are expected to be, and enforced as, a sequence. In a concurrent environment where care is not taken, this can mean that tests could be run multiple times in separate processes. This can be controlled by manipulating the group_regex (see how this done for nova functional tests) but it is not yet clear how to integrate this cleanly.
• User management is a bit limited, requiring a user named 'admin' at the moment, but that should be relatively easy to fix.
• Plenty of other things I don't now but you may. Tell me.