5

Typical implementations of malloc use brk/sbrk as the primary means of claiming memory from the OS. However, they also use mmap to get chunks for large allocations. Is there a real benefit to using brk instead of mmap, or is it just tradition? Wouldn't it work just as well to do it all with mmap?

(Note: I use sbrk and brk interchangeably here because they are interfaces to the same Linux system call, brk.)


For reference, here are a couple of documents describing the glibc malloc:

GNU C Library Reference Manual: The GNU Allocator
https://www.gnu.org/software/libc/manual/html_node/The-GNU-Allocator.html

glibc wiki: Overview of Malloc
https://sourceware.org/glibc/wiki/MallocInternals

What these documents describe is that sbrk is used to claim a primary arena for small allocations, mmap is used to claim secondary arenas, and mmap is also used to claim space for large objects ("much larger than a page").

The use of both the application heap (claimed with sbrk) and mmap introduces some additional complexity that might be unnecessary:

Allocated Arena - the main arena uses the application's heap. Other arenas use mmap'd heaps. To map a chunk to a heap, you need to know which case applies. If this bit is 0, the chunk comes from the main arena and the main heap. If this bit is 1, the chunk comes from mmap'd memory and the location of the heap can be computed from the chunk's address.

[Glibc malloc is derived from ptmalloc, which was derived from dlmalloc, which was started in 1987.]


The jemalloc manpage (http://jemalloc.net/jemalloc.3.html) has this to say:

Traditionally, allocators have used sbrk(2) to obtain memory, which is suboptimal for several reasons, including race conditions, increased fragmentation, and artificial limitations on maximum usable memory. If sbrk(2) is supported by the operating system, this allocator uses both mmap(2) and sbrk(2), in that order of preference; otherwise only mmap(2) is used.

So, they even say here that sbrk is suboptimal but they use it anyway, even though they've already gone to the trouble of writing their code so that it works without it.

[Writing of jemalloc started in 2005.]

UPDATE: Thinking about this more, that bit about "in order of preference" gives me a line on inquiry. Why the order of preference? Are they just using sbrk as a fallback in case mmap is not supported (or lacks necessary features), or is it possible for the process to get into some state where it can use sbrk but not mmap? I'll look at their code and see if I can figure out what it's doing.


I'm asking because I'm implementing a garbage collection system in C, and so far I see no reason to use anything besides mmap. I'm wondering if there's something I'm missing, though.

(In my case I have an additional reason to avoid brk, which is that I might need to use malloc at some point.)

5

The system call brk() has the advantage of having only a single data item to track memory use, which happily is also directly related to the total size of the heap.

This has been in the exact same form since 1975's Unix V6. Mind you, V6 supported a user address space of 65,535 bytes. So there wasn't a lot of thought given for managing much more than 64K, certainly not terabytes.

Using mmap seems reasonable until I start wondering how altered or added-on garbage collection could use mmap but without rewriting the allocation algorithm too.

Will that work nicely with realloc(), fork(), etc.?

  • The thing is, modern allocators have rewritten their allocation algorithms extensively since then. One, jemalloc, wasn't even written until 2005. And modern allocators do use mmap extensively, so it seems they've figured out how to make it work. However, the ones I've been looking at mix it with calls to sbrk, as I've now described in some updates to the question. – Nate C-K Apr 20 at 12:50
4

The obvious advantage is that you can grow the last allocation in place, which is something you can't do with mmap(2) (mremap(2) is a Linux extension, not portable).

For naive (and not-so-naive) programs which are using realloc(3) eg. to append to a string, this translates in a 1 or 2 orders of magnitude speed boost ;-)

  • Couldn't I just claim a large amount of RAM with mmap, say 1 GB, and then start filling it in from the bottom up? I don't think that would actually consume physical memory until the pages are touched, right? I could also mark the unused pages with PROT_NONE to make sure they aren't accessed accident before I mean to use them. – Nate C-K Apr 19 at 23:35
  • 2
    without thinking too deep about it: claiming a bing chunk of contiguous virtual memory (eg. 1G) will create big problems on 32bit machines. This is usually done only with lisp vms, emulators and such; it's not acceptable that the implementation of a library function like malloc() hog most of the available address space. – mosvy Apr 19 at 23:47
  • Yes, that makes sense. Maybe that's part I was missing: malloc has to stay out of the way of other programs, and growing the heap from the bottom means that other programs know how to stay out of its way. But you could grow the heap from the bottom using mmap without bothering to call brk, couldn't you? Maybe if you're going to do that then you might as well do brk, though. – Nate C-K Apr 20 at 0:09
  • 1
    Google suggests that Go's memory allocator is entirely mmap based – that other guy Apr 20 at 1:44
  • 1
    @thatotherguy and so was FreeBSD's malloc iirc. – mosvy Apr 20 at 1:45
2

mmap() didn't exist in the early versions of Unix. brk() was the only way to increase the size of the data segment of the process at that time. The first version of Unix with mmap() was SunOS in the mid 80's, the first open-source version was BSD-Reno in 1990.

And to be usable for malloc() you don't want to require a real file to back up the memory. In 1988 SunOS implemented /dev/zero for this purpose, and in the 1990's HP-UX implemented the MAP_ANONYMOUS flag.

There are now versions of mmap() that offer a variety of methods to allocate the heap.

  • That explains why mmap wasn't used in the past, but modern versions do use it, so I'm not sure if the history explains why they don't use it exclusively. Maybe they were originally written to use brk only and then added mmap calls later as an improvement? But jemalloc only dates back to 2005 and it uses both sbrk and mmap. – Nate C-K Apr 19 at 23:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.