Open source C++ projects

1

In our discussions about what C++ style to adopt for our codebase, and which aspects of the language and standard library to adopt and which to stay away from, Richard made the excellent point that it would be worth looking at some other open source C++ projects. So I’m making this thread to collect examples of C++ projects that appear to be worth examining, and as a place for any discussion. When this came up, we were discussing use of std::unique_ptr and std::move, so I will call those out in particular, but I don’t want to restrict this thread to those specifically.

Here’s a few to start:

  • http://seastar.io/ — “Seastar is an advanced, open-source C++ framework for high-performance server applications on modern hardware. Seastar is used in Scylla, a high-performance NoSQL database compatible with Apache Cassandra. Applications using Seastar can run on Linux or OSv.”

    I’m a little disappointed I hadn’t looked at this before. From a first cursory glance, they appear to be in a very similar problem space as we are.

    In their docs here, under the section “Lifetime Management”, they discuss making use of std::move, and both std::unique_ptr and their own single-threaded implementation of std::shared_ptr called (natually) seastar::shared_ptr. A search in github shows significant use of unique_ptr. I will definitely need to look through their codebase more to see if there is anything worth discussing with the team.

  • http://www.includeos.org/ — “IncludeOS allows you to run your application in the cloud without an operating system. IncludeOS adds operating system functionality to your application allowing you to create performant, secure and resource efficient virtual machines. IncludeOS applications boot in tens of milliseconds and require only a few megabytes of disk and memory.”

    In terms of C++ style, they say they follow the C++ Core Guidelines pretty strictly, which are very clear about their suggested usage of smart pointers for resource management and ownership control. A search in github shows heavy usage of unique_ptr, where many of the search results are type aliases like

    class Request;
using Request_ptr = std::unique_ptr<Request>;

class Response;
using Response_ptr = std::unique_ptr<Response>;

class Server;
using Server_ptr = std::unique_ptr<Server>;

  • Cap’n Proto — They use their own unique pointer implementation, kj::Own. Their docs explain how they manage ownership of resources:

    Every object has an “owner”. The owner may be another object, or it may be a stack frame (which is in turn owned by its parent stack frame, and so on up to the top frame, which is owned by the thread, which itself is an object which is owned by something).

    The owner decides when to destroy an object. If the owner itself is destroyed, everything it owns must be transitively destroyed. This should be accomplished through RAII style.

    The owner specifies the lifetime of the object and how the object may be accessed. This specification may be through documented convention or actually enforced through the type system; the latter is preferred when possible.

    An object can never own itself, including transitively.

    When declaring a pointer to an object which is owned by the scope, always use kj::Own. Regular C++ pointers and references always point to objects that are not owned.

  • Abseil — Google’s open source C++ library. The lead of Google’s C++ libraries group, Titus Winters, is also the chair of the Library Evolutions group for the C++ standards committee, so it’s not surprising that they are strong advocates of the “modern” resource management style. Titus is also the lead of C++ education in Google, and with the open source release of Abseil he has started publicly publishing some of the “Tip of the Week” C++ articles they have written. The article TotW55 is a nice explanation of usage of unique_ptr different from others I’ve seen. Unique_ptr is the subject of several articles: TotW123, TotW126, TotW134. TotW77 is a nice way of thinking about move semantics continuing from TotW55.

  • Folly — Facebook’s open source C++ library. A search shows extensive use of unique_ptr.

2

I gathered some statistics from the 2.0M lines of
llvm source ( including tests). On average there’s
one unique_ptr per 444 lines, and one std::move
per 562 lines. I need to look close to figure out
the distribution.

I’m trying to remember the specific problems
caused by unique_ptr at Oracle. IIRC it was
mostly

a) Using naked refs or pointers to objects owned
by a unique_ptr, but then it turns out the object
gets deleted leaving dangling refs. Sutter et al
talk about passing naked refs when you’re not
“concerned with ownership” - but that turns out to
be a slippery slope to normal C-style shoot-your-
foot-off chaos, since who is and isn’t “concerned
with ownership” is in general difficult to
design, review, and test.

b) Using a unique_ptr which has been move’d
and set to nullptr. The rules say don’t do that - but there’s nothing in the compiler or analysis
tools or unique_ptr which will warn you about it before you dereference the nullptr, at which point you have to work backwards to figure out when and why the move happened.

3

I have a little script to find occurrences of “unique_ptr” and “std::move”, trying it on these projects I get:

seastar:

Number of files = 128
Lines of code = 37640 total
Occurrences of unique_ptr = 160
Occurrences of std::move = 646
Num files with either = 84

IncludeOS:

Number of files = 479
Lines of code = 72441 total
Occurrences of unique_ptr = 61
Occurrences of std::move = 315
Num files with either = 95

abseil-cpp:

Number of files = 267
Lines of code = 85477 total
Occurrences of unique_ptr = 158
Occurrences of std::move = 150
Num files with either = 42

folly:

Lines of code = 330039 total
Occurrences of unique_ptr = 842
Occurrences of std::move = 2308
Num files with either = 434

llvm:

Number of files = 4391
Lines of code = 2010549 total
Occurrences of unique_ptr = 4502
Occurrences of std::move = 3559
Num files with either = 1261

I also read through the seastar discussion of lifetime management - the main takeaway
is that it encourages a lot of stuff to be written as lambdas with captured arguments, and
the lambda function being run at some unknown time in the future. In that style, you absolutely need to get the lambda arguments to be heap-allocated - they give several
examples, and the first is using a std::move, but the most general (and to be honest, in
a multi-thread server running event-driven code I think it’s by far the safest choice) is to
use shared_ptr.

As the flow of control becomes more complicated and event-driven, it becomes harder
and harder to get a correct design with unique_ptr.

Looking a bit closer at abseil-cpp, it seems that almost all the occurrences of “unique_ptr”
are in test and benchmark code, with only 21 occurrences in 3 files of the actual library code.

4

I don’t see this as a unique_ptr issue, this is one of the fundamental problems of resource management. I think you’re arguing “don’t pass naked refs or pointers, only shared_ptr”.

The goal of unique_ptr is to aid in proving the correctness of code which makes any use of non-reference counted pointers and references. But the basic pattern of analyzing such code is the same whether you use smart pointers or not:

  1. Every heap object has an owner (at least conceptually), which may change but is well defined at any point in the code. Any non-owning reference to the object may be assumed to stay valid for as long as the owner is not modified.

  2. If you pass a raw reference to a function foo(Widget*), then it is the caller’s responsibility to ensure that the reference will remain valid for the duration of the call. The nested scopes of function calls help but don’t guarantee this. To avoid subtle errors you have to be able to ensure at the call site foo(w) that foo is not able to modify the owner of w. Then foo’s pointer to w cannot be invalidated by foo or any functions called by foo.

    I agree this is not an easy problem in general (though I think there are plenty of cases where this is easy to prove). I’ve seen this called the #1 correctness issue with smart pointers (as opposed to the #1 performance problem, which is unnecessarily passing smart pointer arguments), and it sounds like this is the issue you ran in to. But as I said, I don’t think this is a problem with smart pointers, it’s a fundamental problem of resource management. I think the danger with smart pointers is thinking they free you from this kind of problem. Their primary job is to aid in the reasoning about ownership, and to automate destruction using RAII.

    Note that reference counting doesn’t avoid this problem either, unless you make a rule to exclusively use reference-counted pointers (even in library code). Consider

    std::shared_ptr<int> gsp = std::make_shared<int>(); // global shared pointer
void foo(int*);
void bar(std::shared_ptr<int>&, int*);
int main() {
  foo(gsp.get()); // bad
  auto sp = gsp; // make local copy
  foo(sp.get()); // safe
  bar(sp, sp.get()); // bad
  bar(gsp, sp.get()); // safe
}

    In the first call, foo has access to the global shared pointer gsp, so it (or a descendant) could potentially modify gsp, which would invalidate foo’s argument, but in the second foo cannot access sp, so the ref count can’t drop below 1. Similarly, in the first bar call, bar is given access to sp, so bar is able to modify sp (and gsp) and could invalidate it’s raw pointer argument, but the second call is safe.

  3. Given 2., any function taking a raw pointer or reference is free to assume that pointer will remain valid for the duration of the function.

One of the goals of the core guidelines is to develop a set of statically checkable rules which provide warnings when, for example, calling a function which can modify the owner of one of its pointer arguments. I think VisualStudio has the most complete implementation of the core guidelines. Clang-tidy has support for some of them, but I don’t think it can do lifetime analysis yet.

I want to resolve this and get on with working on Hail as much as you do, so I want to be sure we’re getting to the core of where we disagree. It looks to me like the problems you ran in to using unique_ptr are common problems people face when learning how to use them effectively. I think the options to avoid those problems are

  • Try to learn effective usage patterns, idioms, and guidelines, communicate them with the team, and enforce them in code review. Pros: better performance, we learn together how to use parts of modern C++ more effectively, can always add reference counting later. Cons: We will have subtle bugs while we learn how to avoid them, higher learning curve for team members.
  • Use reference counting everywhere (or at least only allow non-reference counted pointers in small self-contained places). Pros: Less opportunity for subtle bugs, reduced learning curve. Cons: higher overhead on parameter passing, less opportunity to get familiar with aspects of the direction in which C++ is moving/has moved, like move semantics, harder to remove reference counting if performance cost becomes unacceptable.

Does this seem like a fair summary?

This is much easier. Unless you go out of your way, the only way for a variable foo to get moved from is to write std::move(foo). If you see that, then foo may only be assigned to or deleted. Clang-tidy has what appears to be a fairly thorough enforcement of this, see here.

5

Yes. I am arguing that you should use shared_ptr, and pass shared_ptr, everywhere except where
it is obvious, or non-obvious but has been measured, that the runtime memory or time overheads
of shared_ptr are a significant problem.

In practice that tends to mean using a lot of shared_ptr’s being passed down, but at the very lowest
levels of the most-frequently called functions you do whatever the heck you want to get good performance, of which using naked pointers is one trick, but vectorization and assembler code
are also in the bag of tricks.

Typically the parts of the code which really matter for performance are less than 5% of the total
lines of code.

The global optimization here is to not spend time on design and debugging and review of
ownership-based memory allocation in the bulk of the code - just use shared_ptr and forget about
it - in order to have more developer-hours to spend on activites which give a higher ROI - viz
getting the high-level code design right so that you fully exploit whatever IO/network/compute resources you might have; and doing extreme (and possible dangerous) optimization of the
few parts of the code which are really performance-critical.

IMO any strategy which distinguishes owning-references from non-owning references (and doesn’t have the kind of language and analysis support that Rust guarantees, or the non-dangling guarantee of weak_ptr) is only epsilon above using C and naked pointers. Murphy’s
Law (and my experience at Oracle) says that as soon as you start passing around
naked refs and pointers, you’ll find yourself debugging dangling refs and mistakenly-
invalidated ownership pointers. And that’s time better spent on other activities with vastly
higher ROI.

And in my view of the wider world of software engineering, most of the world has already
voted with their feet by adopting managed-memory languages such as Java, Scala, Python,
even Javascript in order to get higher productivity in spite of enormous performance and memory
overheads. What’s great about std::shared_ptr is that it gives you most of those productivity
benefits, in most kinds of code, with only a small fraction of the performance overhead (and
opportunities to make that overhead insignificant with optimization of a small fraction of the
code). And selection bias also helps to explain why “the C++ community” still seems to put
effort into trying to explain how to use unique_ptr and move semantics - a lot of the pragmatists
who got sick and tired of complicated lifetime-management rules have left C++ behind to go to
Java or Python, or to try to “do it right” with Rust. Anyone who is currently in “the C++ community” has been through the pain of doing years of explicit memory management, and has
at least tolerated it.

And what’s bad about unique_ptr is that it’s a stalking horse that looks like something new
and modern and better, but it actually dumps you right back into the same old problem of having
to spend effort on matching your lifetime-management design to your control flow and passing
around totally-unsafe naked refs and pointers (or only-safe-because-of-uncheckable-global-
properties-of-the-program).

We’re all smart enough to do it, I have no doubt. I just think the better development strategy is
to get away from hard-to-design hard-to-debug ownership-based lifetime management (except
in the few parts of the code where it’s a clear performance win to use something other than shared_ptr).

6

You make some good points, and I’m looking forward to talking this afternoon to settle on a plan. Until then, I’ll just respond to one point:

passing around totally-unsafe naked refs and pointers (or only-safe-because-of-uncheckable-global-
properties-of-the-program)

It is explicitly the goal of the guidelines around pointer passing and ownership to ensure safety using only local properties of the code. Otherwise you’re right they wouldn’t be worth much in practice.

7

Also your example isn’t really a problem with shared_ptr, it’s the wider issue of value vs reference
parameters. If you want to be really sure that something can’t be modified, then you take your
own copy that no-one else can see - with a value parameter being one way to do that.

If you take a reference to something that’s visible to other code, then it’s a higher-level design issue
to make sure that it will be bound to the right value for as long as you need it.

The use of lambdas adds the related problem of having the correct values in/through any
captured variables.

I’m definitely not arguing that lifetime-management is the only way to shoot your foot off in
C++. Just a very common one, and one that tends to be particularly time-consuming to debug
since it usually involves backtracking from the first symptom to find the point at which references
were screwed up/invalidated.

8

unique_ptr misses that goal. You end up passing naked refs and pointers down the call stack,
and then the bugs arise because those naked ref’s or pointers get stored in something which
outlives the duration of the call. That can happen deep down the call stack (which may also involve
callbacks to higher-level functions that weren’t even written at the time you were designing the
lifetime-management). “Does this pointer get stored somewhere off the stack before this call returns
?” is a non-local property of the code which - in the presence of callbacks and virtual functions -
might not be answerable by any kind of static analysis, let alone a local analysis.

Though of course there are simple cases where it’s provable. It just tends to be the complex
cases where you get trouble - and the trends towards multithreading and functional code
and callback/observer pattern lead towards that kind of complexity.

9

I adjusted my script to treat capnproto’s “kj::Own” as being equivalent to unique_ptr,
and to exclude files with names containing “test” or “bench”.

abseil-cpp
Number of files = 165
Lines of code = 45678 total
Occurrences of unique_ptr = 61
Occurrences of std::move = 47
Num files with either = 21

capnproto
Number of files = 161
Lines of code = 121036 total
Occurrences of unique_ptr = 660
Occurrences of std::move = 1
Num files with either = 48

folly
Number of files = 757
Lines of code = 191257 total
Occurrences of unique_ptr = 420
Occurrences of std::move = 1363
Num files with either = 279

IncludeOS
Number of files = 323
Lines of code = 51791 total
Occurrences of unique_ptr = 39
Occurrences of std::move = 259
Num files with either = 74

seastar
Number of files = 61
Lines of code = 26156 total
Occurrences of unique_ptr = 122
Occurrences of std::move = 476
Num files with either = 44

llvm
Number of files = 3942
Lines of code = 1860692 total
Occurrences of unique_ptr = 3962
Occurrences of std::move = 3113
Num files with either = 1130

On inspection, CapnProto’s kj::Own can be used to manage refcount’ed objects
(see c++/src/kj/refcount.h), so it’s hard to figure out what mixture of single-owner
vs refcounted objects are being used in that project.