There's a thread on the openstack-discuss mailing list, started in September and then continuing in October, about limiting planned scope for Nova in the Ussuri cycle so that stakeholders' expectations are properly managed. Although Nova gets a vast amount done per cycle there is always some stuff left undone and some people surprised by that. In the midst of the thread, Kashyap points out:

I welcome scope reduction, focusing on fewer features, stability, and bug fixes than "more gadgetries and gongs". Which also means: less frenzy, less split attention, fewer mistakes, more retained concentration, and more serenity. [...] If we end up with bags of "spare time", there's loads of tech-debt items, performance (it's a feature, let's recall) issues, and meaningful clean-ups waiting to be tackled.

Yes, there are.

When Placement was extracted from Nova, one of the agreements the new project team made was to pay greater attention to tech-debt items, performance, and meaningful clean-ups. One of the reasons this was possible was that by being extracted, Placement vastly limited its scope and feature drive. Focused attention is easier and the system is contained enough that unintended consequences from changes are less frequent.

Another reason was that for several months my employer allowed me to devote effectively 100% of my time to upstream work. That meant that there was long term continuity of attention in my work. Minimal feature work combined with maximal attention leads to some good results.

In August I wrote up an analysis of some of that work in Placement Performance Analysis, explaining some of the things that were learned and changed. However that analysis was comparing Placement code from the start of Train to Train in August. I've since repeated some of the measurement, comparing:

1. Running Placement from the Nova codebase, using the stable/stein branch.
2. Running Placement from the Placement codebase, using the stable/stein/ branch.
3. Running Placement from the Placement codebase, using master, which at the moment is the same as what will become stable/train and be released as 2.0.0.

The same database (PostgreSQL) and web server (uwsgi using four processes of ten threads each) is used with each version of the code. The database is pre-populated with 7000 resource providers representing a suite of 1000 compute hosts with a moderately complex nested provider topology that is similar to what might be used for a virtualized network function.

The same query is used, whatever the latest microversion is for that version:

http://ds1:8000/allocation_candidates? \
                resources=DISK_GB:10& \
                required=COMPUTE_VOLUME_MULTI_ATTACH& \
                resources1=VCPU:1,MEMORY_MB:256& \
                required1=CUSTOM_FOO& \
                resources2=FPGA:1& \
                group_policy=none

(This is similar to what is used in the nested-perfload peformance job in the testing gate, modified to work with all available microversions.)

Here are some results, with some discussion after.

10 Serial Requests

Placement in Nova (stein)

Requests per second:    0.06 [#/sec] (mean)
Time per request:       16918.522 [ms] (mean)

Extracted Placement (stein)

Requests per second:    0.34 [#/sec] (mean)
Time per request:       2956.959 [ms] (mean)

Extracted Placement (train)

Requests per second:    1.37 [#/sec] (mean)
Time per request:       730.566 [ms] (mean)

100 Requests, 10 at a time

Placement in Nova (stein)

This one failed. The numbers say:

Requests per second:    0.18 [#/sec] (mean)
Time per request:       56567.575 [ms] (mean)

But of the 100 requests, 76 failed.

Extracted Placement (stein)

Requests per second:    0.41 [#/sec] (mean)
Time per request:       24620.759 [ms] (mean)

Extracted Placement (train)

Requests per second:    2.65 [#/sec] (mean)
Time per request:       3774.854 [ms] (mean)

The improvement between the versions in Stein (16.9s to 2.9s per request) were mostly made through fairly obvious architecture and code improvments found by inspection (or simply knowing it was not ideal when first made, and finally getting around to fixing it). Things like removing the use of oslo versioned objects and changes to cache management to avoid redundant locks.

From Stein to Train (2.9s to .7s per request) the improvements were made by doing detailed profiling and benchmarking and pursuing a very active process of iteration (some of which is described by Placement Performance Analysis).

In both cases this was possible because people (especially me) had the "retained concentration" desired above by Kashyap. As a community OpenStack needs to figure out how it can enshrine and protect that attention and the associated experimentation and consideration for long term health. I was able to do it in part because I was able to get my employer to let me and in part because I overcommitted myself.

Neither of these things are true any more. My employer has called me inside, my upstream time will henceforth drop to "not much". I'm optimistic that we've established a precedent and culture for doing the right things in Placement, but it will be a challenge and I don't think it is there in general for the whole community.

I've written about some of these things before. If the companies making money off OpenStack are primarily focused on features (and being disappointed when they can't get those features into Nova) who will be focused on tech-debt, performance, and meaningful clean-ups? Who will be aware of the systems well enough to effectively and efficiently review all these proposed features? Who will clear up tech-debt enough that the systems are easier to extend without unintended consequences or risks?

Let's hit that Placement performance improvement some more, just to make it clear:

In the tests above, "Placement in Nova (stein)" failed with a concurrency of 10. I wanted to see at what concurrency "Extracted Placement (train)" would fail: At 150 concurrency of 1000 requests some requests fail. At 140 all requests work, albeit slow per request (33s). Based on the error messages seen, the failing at 150 is tied to the sizing and configuration of the web server and nothing to do with the placement code itself. The way to have higher concurrency is to have more or larger web servers.

Remember that the nova version fails at concurrency of 10 with the exact same web server setup. Find the time to fix your debt. It will be worth it.