Tramasys - linux

mirror, mirror on the wall, which cpu am I running on?

2023-10-29T00:00:00+00:00

experimentation in python

Starting with the obvious version:

#!/usr/bin/python3
import psutil

nr = psutil.Process().cpu_num()
print(nr)

How does psutil find out which CPU we're running on?

I first stepped through it with pdb:

python3 -m pdb ./allocs/getcpu.py

n goes to the next line, s steps into a call, and ll lists the current function. With psutil 7.2.2 the relevant path is:

psutil.Process.__init__
  -> psutil.Process.cpu_num
  -> self._proc.cpu_num()
  -> psutil._pslinux.Process.cpu_num
  -> self._parse_stat_file()['cpu_num']

The public Process.cpu_num method is only the platform-independent wrapper. On Linux it lands in psutil/_pslinux.py, where cpu_num still calls _parse_stat_file and converts the result to an integer.

The interesting part of _parse_stat_file is small:

data = bcat(f"{self._procfs_path}/{self.pid}/stat")
rpar = data.rfind(b')')
fields = data[rpar + 2:].split()

ret['cpu_num'] = fields[36]

The file is /proc/<pid>/stat. The process name inside it is surrounded by parentheses and may itself contain spaces or parentheses, so splitting the whole line on spaces would be wrong. psutil finds the last ), removes the pid and (comm) prefix, and only then splits the remaining fields.

The procfs interface calls this value field 39, processor: the CPU number the task last executed on. After removing the first two fields, procfs field 39 becomes Python index 36. So that slightly magic-looking number is at least accounted for.

Tracing the script makes the route visible:

$ strace -e openat,read,getcpu,rseq python3 getcpu.py
rseq(..., 33, 0, 0x53053053) = 0
openat(AT_FDCWD, "/proc/208862/stat", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/proc/208862/stat", O_RDONLY|O_CLOEXEC) = 3
13

There are two reads because constructing Process() also records the process start time to protect against PID reuse. cpu_num() then reads the same stat file for the CPU field. The rseq call belongs to glibc/Python startup. psutil does not use it for this answer.

There is a semantic wrinkle here. /proc/<pid>/stat describes the task with that PID. For a multithreaded process, /proc/<pid>/stat is the thread-group leader's stat file. It does not tell us which CPU every thread in the process is using. Individual threads have their own /proc/<pid>/task/<tid>/stat files.

It also says "last executed on", which is useful when inspecting another process or a sleeping task, but is not quite the same question as "which CPU is this calling thread executing on at this instruction?"

experimentation in rust

For the calling thread, Linux exposes sched_getcpu(). The Rust libc crate contains the FFI binding:

fn main() {
    let cpu = unsafe { libc::sched_getcpu() };
    if cpu < 0 {
        panic!("sched_getcpu failed: {}", std::io::Error::last_os_error());
    }
    println!("{cpu}");
}

On a GNU/Linux target this is a call to glibc, not an implementation inside the Rust crate. The resulting binary has a call through the dynamic symbol:

call sched_getcpu@PLT

Setting a breakpoint at sched_getcpu with GDB lands in sysdeps/unix/sysv/linux/sched_getcpu.c. My old notes had this as THREAD_GETMEM_VOLATILE(THREAD_SELF, rseq_area.cpu_id). Current glibc spells the same operation with RSEQ_GETMEM_ONCE:

int
sched_getcpu (void)
{
  int cpu_id = RSEQ_GETMEM_ONCE (cpu_id);
  return __glibc_likely (cpu_id >= 0) ? cpu_id : vsyscall_sched_getcpu ();
}

So the normal path is now almost disappointingly short: read one integer from the current thread's rseq area and return it.

new world: rseq

rseq means restartable sequences. Its main purpose is to let userspace do small per-CPU operations that the kernel can abort and restart if a thread is preempted, migrated, or interrupted by a signal. A useful side effect is that the registered userspace area contains cpu_id.

glibc allocates that area in thread-local storage. On x86-64, RSEQ_GETMEM_ONCE is a volatile single-copy read relative to the fs thread pointer and the exported __rseq_offset. In the installed glibc 2.44 binary the important part looks like this:

mov  rax, QWORD PTR [rip + __rseq_offset]
mov  rax, QWORD PTR [rax]
mov  eax, DWORD PTR fs:[rax + 0x4]
test eax, eax
js   fallback

Nonnegative values are CPU IDs. A negative value means rseq registration was disabled, failed, or has not completed, so glibc uses the fallback.

Who registered this memory with the kernel? glibc did, before main.

For the initial thread of a statically linked program the current chain is:

_start
  -> __libc_start_main_impl
  -> ARCH_SETUP_TLS
  -> __libc_setup_tls
  -> call_tls_init_tp
  -> __tls_init_tp
  -> rseq_register_current_thread
  -> rseq syscall

For a dynamically linked program, ld-linux performs the corresponding TLS setup and calls __tls_init_tp before transferring control to the program. New pthreads register their own area from start_thread in nptl/pthread_create.c.

The allocation is not something to hard-code. The kernel advertises the feature size and alignment through the ELF auxiliary vector. On this system:

$ LD_SHOW_AUXV=1 /bin/true | grep RSEQ
AT_RSEQ_FEATURE_SIZE: 33
AT_RSEQ_ALIGN:        64

That explains the rseq(..., 33, ...) seen by strace. glibc obtains these values, allocates suitably aligned extra TLS, initializes cpu_id, and calls rseq_register_current_thread with the kernel's rseq syscall.

On the kernel side, migration marks the registered IDs as changed. In the current source this runs through rseq_sched_set_ids_changed and rseq_sched_switch_event. Before returning to userspace, rseq_set_ids_get_csaddr writes the new cpu_id_start, cpu_id, NUMA node, and memory-map concurrency ID into the registered area. This is slightly more precise than saying "the scheduler writes the field during migration": the update is arranged so it is correct when the task resumes in userspace.

The field is extremely cheap for glibc to read because the kernel already did the work during scheduling. No syscall is required for each sched_getcpu().

It is possible to turn the glibc registration off:

$ strace -e rseq,getcpu ./getcpu
rseq(...) = 0
18

$ GLIBC_TUNABLES=glibc.pthread.rseq=0 \
    strace -e rseq,getcpu ./getcpu
13

In the second run neither syscall appears. rseq is disabled, yet there is still no getcpu syscall. That drops us into the next layer: the vDSO.

fallback one: the vDSO

vsyscall_sched_getcpu is still the current glibc helper name, although this is the vDSO path rather than the old fixed-address vsyscall page. It uses INLINE_VSYSCALL(getcpu, ...). During startup, setup_vdso_pointers looks up __vdso_getcpu in the ELF image supplied by the kernel and stores its address in glibc's read-only loader state.

The vDSO is kernel code mapped into every process, but it executes in user mode. That is why strace sees no getcpu call above.

Hard-coding a mapping such as 0x00007ffff7fc6000 works once and then ASLR moves it. A less fragile GDB route is:

(gdb) set disassembly-flavor intel
(gdb) info auxv
(gdb) info proc mappings
(gdb) dump binary memory /tmp/vdso.bin START END

AT_SYSINFO_EHDR in info auxv is the vDSO ELF header. readelf -Ws and objdump -d -M intel can then find __vdso_getcpu in the dumped image.

The current x86 __vdso_getcpu calls vdso_read_cpunode. That helper has two implementations packed into one instruction-alternative site:

baseline:  LSL with the __CPUNODE_SEG selector
patched:   RDPID when X86_FEATURE_RDPID is present

The baseline starts by loading selector 0x7b. LSL returns the limit from a per-CPU GDT descriptor whose bits contain the CPU and NUMA node. Linux uses the low 12 bits for the CPU and the bits above them for the node.

On the machine I am checking now, CPUID leaf 7, subleaf 0, ECX bit 22 is set, and /proc/cpuinfo contains rdpid. The kernel therefore patched the vDSO to the newer instruction. With rseq disabled, the live mapping disassembles as:

mov   eax, 0x7b
rdpid rax
test  rdi, rdi
je    no_cpu_pointer
mov   edx, eax
and   edx, 0xfff
mov   DWORD PTR [rdi], edx

The apparently unused mov eax, 0x7b remains because the alternatives mechanism patches the LSL instruction in place. On hardware without RDPID, that selector is consumed by LSL instead.

RDTSCP can also expose IA32_TSC_AUX in ecx, and older notes often list it as another assembly-level way to obtain per-CPU data. It also reads the timestamp and has stronger ordering properties than needed here. Current x86 vDSO source specifically chooses between RDPID and LSL. It does not use RDTSCP for this function.

fallback two: the actual syscall

If the rseq value is negative and the vDSO symbol is unavailable, or the vDSO returns ENOSYS, glibc finally executes the real getcpu syscall. On x86-64 its number is 309, or 0x135, which is visible in the last branch of the glibc disassembly.

The kernel implementation is almost the answer in plain C: SYSCALL_DEFINE3(getcpu) takes raw_smp_processor_id(), optionally converts it to a NUMA node, and copies the values to userspace.

So the Rust path is:

libc::sched_getcpu()
  -> glibc sched_getcpu()
  -> rseq cpu_id TLS read
  -> __vdso_getcpu() if rseq is unavailable
  -> getcpu syscall if the vDSO is unavailable

and the Python path was separate all along:

psutil.Process().cpu_num()
  -> /proc/<pid>/stat
  -> field 39: processor

in the wild: MongoDB and TCMalloc

A place where the answer matters on a very hot path is Google's TCMalloc. MongoDB 8.0 and newer ship a version with per-CPU caches, so an ordinary small allocation may end up taking an object from a cache belonging to the current logical CPU. That avoids putting every allocating thread through the same lock.

At upstream revision 622a1d1, the most direct helper in percpu.h is almost identical to the glibc fast path:

inline int GetRealCpuUnsafe() { return __rseq_abi.cpu_id; }

__rseq_abi is thread-local storage registered with the kernel. The nonnegative values are real CPU IDs. -1 means uninitialized and -2 means unsupported. GetRealCpu() first tries this load and falls back to sched_getcpu() if no rseq value is available. TCMalloc uses that general helper in places such as its deallocation profiler, where it records the CPU, L3 cache, and NUMA node associated with a sample.

The Unsafe suffix is the important part. The load itself is fine, but its answer can become stale as soon as the thread migrates. This would not be enough for the allocator fast path:

cpu = current_cpu()
object = slabs[cpu].pop()

A migration between those two lines could make the thread modify the old CPU's cache while running on the new CPU. TCMalloc therefore does more than ask for a number. InitThreadPerCpu registers its own TLS area with the raw rseq syscall, and the x86-64 push and pop paths run as actual restartable sequences. The last store updates the slab header and commits the operation. If the kernel preempts or migrates the thread before that store, execution goes through the abort trampoline and starts the operation again.

The current implementation also avoids calculating slabs[cpu] on every allocation. It caches the address of the current CPU's slab in the thread-local tcmalloc_slabs. The layout is slightly evil in a nice way:

TLS address:  __rseq_abi - 4       __rseq_abi
              +--------------------+--------------------+
8-byte load:  | pointer bits 0..31 | pointer bits 32..63|
              +--------------------+--------------------+
rseq field:                        | cpu_id_start       |

The two TLS objects overlap by four bytes on little-endian systems. Bit 63 marks the cached pointer as valid. When the scheduler moves the thread, the kernel writes the new, small CPU number into cpu_id_start, overwriting that part of the pointer and clearing the validity bit as a side effect.

The x86 fast path can therefore load the cached slab pointer, test and clear bit 63 with BTR, and continue if the bit was set. If it was clear, it takes CacheCpuSlabSlow, reads the current rseq CPU ID, computes the new slab address, and caches it again with another small rseq operation. A scheduler-maintained CPU field has become an automatic cache invalidation mechanism.

There is one current integration wrinkle. TCMalloc registers its own rseq area, while modern glibc registers a different one before main, and Linux only permits one registration per thread. MongoDB's deployment documentation consequently tells operators to start mongod with GLIBC_TUNABLES=glibc.pthread.rseq=0. That leaves the rseq slot free for TCMalloc. Otherwise the allocator falls back to its legacy per-thread caches.

So in this real path, finding the CPU is not an informational query. It is part of deciding which memory may be touched, and rseq supplies the missing guarantee that a plain CPU-number snapshot cannot provide.

what does "currently" mean?

None of these interfaces freezes the scheduler. A thread may migrate after the CPU number is observed and before the caller uses it. Even the rseq load can be followed by preemption before sched_getcpu() returns. The API only gives a snapshot.

If the CPU must remain stable across some work, reading its number is not enough. Pinning the thread to one logical CPU with sched_setaffinity makes that property explicit. TCMalloc's push and pop paths show the other option: an rseq critical section lets a short per-CPU operation detect migration and restart instead of trying to prevent it.

For observing another task, psutil's procfs answer is convenient. For cheaply asking about the calling thread, sched_getcpu() is the direct interface.

source trail

vDSO - why tho?

2023-06-04T00:00:00+00:00

what is the vDSO?

The vDSO is the virtual dynamic shared object. Linux maps this small shared library into every new userspace process, and the C library uses functions from it when that is faster than entering the kernel with a normal system call.

The name is used for two closely related things:

an ELF shared object built as part of the kernel
the [vdso] memory mapping containing that object in a process.

The code in that mapping still executes in userspace, at ring 3. It does not gain kernel privileges just because the kernel supplied it. Its advantage is that it can combine ordinary userspace instructions with data the kernel has mapped read-only next to it.

That second mapping is normally called [vvar]:

7ffff7fc4000-7ffff7fc6000 r--p 00000000 00:00 0  [vvar]
7ffff7fc6000-7ffff7fc8000 r-xp 00000000 00:00 0  [vdso]

[vdso] contains executable code. [vvar] contains kernel-maintained data used by that code. Recent x86 kernels may also map a [vvar_vclock] region for paravirtual clock pages. The exact addresses, sizes, and available symbols depend on the architecture, kernel version, and configuration.

Both mappings are normally randomized on each exec. A hard-coded vDSO address would therefore be useless. The kernel instead gives the new process the address of the ELF header in its auxiliary vector as AT_SYSINFO_EHDR.

The older x86 [vsyscall] page is a different mechanism. It lived at a fixed address and had a much more rigid ABI. Some current glibc identifiers still contain VSYSCALL, but in the paths below they mean a call into the vDSO, not the old fixed page.

finding the image

I started by checking the process mappings:

$ rg '\[(vdso|vvar|vvar_vclock)\]' /proc/self/maps
7ffff7fc4000-7ffff7fc6000 r--p 00000000 00:00 0  [vvar]
7ffff7fc6000-7ffff7fc8000 r-xp 00000000 00:00 0  [vdso]

There is a small trap in that command: /proc/self refers to the process which opens it, so a short-lived tool may actually show its own mapping. For examining a particular program, GDB is less ambiguous:

(gdb) set disassembly-flavor intel
(gdb) start
(gdb) info proc mappings
(gdb) info auxv

info auxv includes the entry used to find the image:

AT_SYSINFO_EHDR      System-supplied DSO's ELF header  0x7ffff7fc6000

It is also visible before main through the dynamic loader:

$ LD_SHOW_AUXV=1 /bin/true | rg SYSINFO_EHDR
AT_SYSINFO_EHDR:      0x7ffce917e000

The address can be obtained programmatically with getauxval(AT_SYSINFO_EHDR). This gives the ELF base, not the address of a particular function.

Since the mapping is a real ELF image, it can be dumped and inspected like one. After taking the start and end addresses from info proc mappings:

(gdb) dump binary memory /tmp/vdso.bin 0x7ffff7fc6000 0x7ffff7fc8000
(gdb) shell file /tmp/vdso.bin
(gdb) shell readelf -Ws /tmp/vdso.bin
(gdb) shell objdump -d -M intel /tmp/vdso.bin

The first command produces something along these lines:

/tmp/vdso.bin: ELF 64-bit LSB shared object, x86-64, dynamically linked, stripped

My original dump, from Linux 6.4.4, contained the following interesting dynamic symbols:

clock_gettime@@LINUX_2.6
__vdso_clock_gettime@@LINUX_2.6
gettimeofday@@LINUX_2.6
__vdso_gettimeofday@@LINUX_2.6
time@@LINUX_2.6
__vdso_time@@LINUX_2.6
clock_getres@@LINUX_2.6
__vdso_clock_getres@@LINUX_2.6
getcpu@@LINUX_2.6
__vdso_getcpu@@LINUX_2.6
__vdso_sgx_enter_enclave@@LINUX_2.6

The exact table is not permanent. The current x86-64 linker script also exports getrandom and __vdso_getrandom, for example. New functions can be added because userspace performs an ordinary versioned ELF symbol lookup at runtime. Programs must not assume that every kernel or every architecture exports the same names.

who put it there?

The mappings arrive during exec, before the program gets control.

For an x86-64 ELF executable the current route through the kernel is roughly:

load_elf_binary
  -> ARCH_SETUP_ADDITIONAL_PAGES
  -> arch_setup_additional_pages
  -> map_vdso(&vdso_image_64, 0)
  -> vdso_install_vvar_mapping
  -> create_elf_tables
  -> ARCH_DLINFO
  -> AT_SYSINFO_EHDR = current->mm->context.vdso

map_vdso first finds an unused randomized area. It installs the image as a special readable and executable mapping, then installs the vvar pages next to it. The resulting vDSO base is saved in the process's mm context.

With the error handling removed, the relevant part of the current x86 kernel code is small:

text_start = addr - image->sym_vvar_start;

_install_special_mapping(mm, text_start, image->size,
    VM_READ | VM_EXEC | VM_MAYREAD | VM_MAYWRITE | VM_MAYEXEC,
    &vdso_mapping);

vdso_install_vvar_mapping(mm, addr);
mm->context.vdso = (void *)text_start;

text_start is the beginning of the ELF image. addr is positioned so the vvar pages described by the image land at their expected relative offset. The last assignment is the value which will later become AT_SYSINFO_EHDR.

The mapping has VM_MAYWRITE so a debugger can create private copy-on-write pages when it inserts a breakpoint. That does not make the normal mapping writable. Its effective permissions are still read and execute.

Later, create_elf_tables constructs the initial stack, including the auxiliary vector. On x86, ARCH_DLINFO adds AT_SYSINFO_EHDR with the base saved by map_vdso.

The dynamic loader then takes over. glibc's _dl_parse_auxv stores AT_SYSINFO_EHDR in dl_sysinfo_dso. setup-vdso.h treats the already mapped image as an abridged link_map, reads its dynamic section and hash tables, and adds it to the loader namespace. This is why tools such as ldd can display it as if it were another shared library:

linux-vdso.so.1 (0x00007ffd5c3f8000)

Finally, dl-vdso.h performs versioned weak symbol lookups. On x86 the expected version is LINUX_2.6. The resolved addresses are stored in glibc's read-only loader state as pointers such as dl_vdso_clock_gettime.

No vDSO file was opened from the filesystem in this process. The kernel had already mapped the object and told the loader where it was.

where did the ELF come from?

The x86 vDSO is built with the kernel, but it is not linked into the ordinary kernel text and executed there.

The vDSO sources are compiled as position-independent userspace code and linked with their own linker script. The exported symbol set and symbol version are defined in vdso.lds.S. The result is stripped and checked because the runtime image cannot contain relocations that would need a normal dynamic linker.

The x86 build then runs vdso2c. That tool reads the finished ELF image and emits the data and metadata used to construct a vdso_image in the kernel. The 64-bit image eventually becomes vdso_image_64, which is the object passed to map_vdso above.

So the lifecycle looks like this:

kernel source tree
  -> compile and link a small position-independent ELF
  -> turn that ELF into an image embedded in the kernel
  -> exec maps the image into a process
  -> AT_SYSINFO_EHDR publishes its address
  -> libc resolves versioned functions from it

This also explains why the vDSO matches the running kernel rather than the distribution's libc package.

code in vdso, data in vvar

clock_gettime needs data that changes as the kernel maintains time, but letting userspace write that data would obviously be a problem. Linux splits the mechanism in two:

[vdso] is executable code supplied by the kernel.
[vvar] is kernel-owned data mapped read-only to userspace.

The kernel writes the data through its own mapping. The process can read it but cannot change the shared kernel state.

My old notes described this as one struct vdso_data. That is no longer the current layout. In Linux 7.1, include/vdso/datapage.h defines separate pieces including:

struct vdso_clock, with the sequence counter, clock mode, cycle base, mask, multiplier, shift, and base times.
struct vdso_time_data, containing the clock data, auxiliary clocks, timezone values, and timer resolution.
struct vdso_rng_data, used by the newer vDSO getrandom path.
pages for time namespaces and architecture-specific data.

lib/vdso/datastore.c maps these pages and services their faults. The generic order currently includes time data, time-namespace data, and RNG data, followed by architecture-specific pages where required.

That matters if new vDSO data is added. DECLARE_VVAR and DEFINE_VVAR from older architecture code are not a generic extension interface. Shared time data belongs in the generic data structures. Architecture-specific data belongs in the corresponding asm/vdso structures and pages. Every vDSO variant which consumes the layout has to stay in agreement, and the exposed layout has to remain compatible with older userspace code.

following clock_gettime from c

Starting with a normal C program:

#include <stdio.h>
#include <time.h>

int main(void)
{
    struct timespec ts;

    if (clock_gettime(CLOCK_REALTIME, &ts) == -1) {
        perror("clock_gettime");
        return 1;
    }

    printf("time: %lld.%09ld\n", (long long) ts.tv_sec, ts.tv_nsec);
    return 0;
}

On x86-64, clock_gettime is also system call 228. However, tracing this program usually does not show that system call:

$ cc -O2 clock.c -o clock
$ strace -e clock_gettime ./clock
time: 1786430552.887265796
+++ exited with 0 +++

The public function came from glibc. In glibc 2.44 its Linux implementation is __clock_gettime64. Depending on the architecture and its time_t size, the compiled path uses either the time64 or native-time vDSO pointer. Reduced to the relevant decisions, it does this:

if (vdso_clock_gettime != NULL) {
    r = INTERNAL_VSYSCALL_CALL(vdso_clock_gettime, 2, clock_id, tp);
    if (r == 0)
        return 0;
    return an_error;
}

return INTERNAL_SYSCALL_CALL(clock_gettime, clock_id, tp);

INTERNAL_VSYSCALL_CALL is an indirect userspace function call through the address resolved by the dynamic loader. The historic macro name does not mean it performs a system call.

With an optimized build, GDB therefore reaches an instruction resembling:

call rax

The value in rax is inside [vdso] and resolves to __vdso_clock_gettime@@LINUX_2.6. Since execution never crossed into the kernel, strace had nothing to report.

resolving it directly

It is possible to reproduce part of glibc's lookup by hand. This is useful for investigating the object, although ordinary programs should continue to call clock_gettime:

#define _GNU_SOURCE
#include <dlfcn.h>
#include <elf.h>
#include <stdio.h>
#include <string.h>
#include <sys/auxv.h>
#include <time.h>

typedef int (*vdso_clock_gettime_fn)(clockid_t, struct timespec *);

int main(void)
{
    void *handle = dlopen("linux-vdso.so.1", RTLD_LAZY | RTLD_LOCAL);
    if (handle == NULL) {
        fprintf(stderr, "dlopen: %s\n", dlerror());
        return 1;
    }

    dlerror();
    void *symbol = dlvsym(
        handle, "__vdso_clock_gettime", "LINUX_2.6");
    const char *error = dlerror();
    if (error != NULL) {
        fprintf(stderr, "dlvsym: %s\n", error);
        return 1;
    }

    vdso_clock_gettime_fn fn;
    _Static_assert(sizeof(fn) == sizeof(symbol), "incompatible pointer sizes");
    memcpy(&fn, &symbol, sizeof(fn));

    struct timespec ts;
    if (fn(CLOCK_REALTIME, &ts) != 0)
        return 1;

    printf("AT_SYSINFO_EHDR:     %#lx\n", getauxval(AT_SYSINFO_EHDR));
    printf("__vdso_clock_gettime: %p\n", symbol);
    printf("time:                 %lld.%09ld\n",
           (long long) ts.tv_sec, ts.tv_nsec);
}

Compiling and running it shows that the symbol address is a small offset from the auxiliary-vector address:

$ cc vdso-direct.c -ldl -o vdso-direct
$ ./vdso-direct
AT_SYSINFO_EHDR:      0x7ffcdecd6000
__vdso_clock_gettime: 0x7ffcdecd6ff0
time:                 1786430552.887265796

Linux's own vDSO correctness selftest uses the same general dlopen and versioned lookup technique.

inside __vdso_clock_gettime

On x86, __vdso_clock_gettime is a small wrapper around the generic vDSO time implementation. For CLOCK_REALTIME, the current call path is approximately:

__vdso_clock_gettime
  -> __cvdso_clock_gettime
  -> __cvdso_clock_gettime_data
  -> __cvdso_clock_gettime_common
  -> do_hres
  -> vdso_get_timestamp
  -> __arch_get_hw_counter

The important part is that [vvar] does not contain a timestamp which the kernel updates on every nanosecond. It contains a recent base time and the values needed to convert a hardware counter into elapsed nanoseconds.

The kernel source compiled into the vDSO makes that read loop visible. With the capability check and declarations left out, current do_hres contains:

do {
    if (vdso_read_begin_timens(vc, &seq))
        return do_hres_timens(vd, vc, clk, ts);

    if (!vdso_get_timestamp(vd, vc, clk, &sec, &ns))
        return false;
} while (vdso_read_retry(vc, seq));

vdso_set_timespec(ts, sec, ns);
return true;

This is an unusual-looking piece of kernel source: after being built into the vDSO image, these instructions execute in the calling process. A false result tells the outer clock path that this case cannot be answered safely here, so it should use the syscall fallback. A successful read produces sec and ns without leaving userspace.

For a TSC clocksource, the simplified calculation is:

$$ \text{now} = \text{base} + \frac{(\text{cycles} - \text{cycle_last}) \times \text{mult}}{2^{\text{shift}}} $$

The actual vdso_calc_ns code also deals with masking, overflow, and the possibility of a slightly backward TSC observation. The base seconds and nanoseconds, cycle_last, mult, and shift come from [vvar]. The current cycle count comes directly from an x86 instruction.

__arch_get_hw_counter selects that instruction from the clock mode. For VDSO_CLOCKMODE_TSC it calls rdtsc_ordered(). The kernel's alternatives mechanism patches this site for the CPU's available ordering instruction. In Intel syntax the possible sequences are equivalent to:

rdtsc

lfence
rdtsc

or:

rdtscp

The two halves of the TSC are then combined into one 64-bit cycle value. A paravirtual guest may instead read a pvclock or Hyper-V clock page from the additional vvar mapping.

reading while the kernel writes

There is still a race to handle. The kernel could update the multiplier or base time while the process is halfway through reading them.

The vDSO data therefore includes a sequence counter. The high-resolution path does roughly this:

read sequence
if sequence is odd, wait and retry
read base and conversion values
read hardware cycles
read sequence again
if it changed, retry everything
calculate the timestamp

An odd sequence value means a writer is in progress. Equal even values before and after the read mean the snapshot was consistent. The current helpers live in include/vdso/helpers.h.

The writer is on the kernel side. update_vsyscall opens the sequence, updates the clock mode, cycle base, multiplier, shift, base times, and related values, then closes the sequence. Userspace gets a coherent snapshot without a lock and without entering the kernel.

Time namespaces add one more step. A process in a time namespace sees namespace offsets in its vvar page while the underlying host clock data comes from the host page. The helper combines the two before returning the result.

Coarse clocks take a shorter route: they return the kernel's cached base time without reading a hardware cycle counter. They are cheaper but intentionally have lower resolution.

when it still becomes a syscall

The vDSO is a fast path, not a promise that every clock_gettime call avoids the kernel. It falls back when, for example:

the requested clock ID has no vDSO implementation.
the active clocksource cannot be read safely from userspace.
the kernel does not export the expected symbol.
an architecture-specific time-width path is unavailable.

The generic vDSO code uses its inline syscall fallback for unsupported cases. glibc also has a syscall path when symbol resolution failed.

To compare the two paths deliberately, the libc wrapper can be bypassed:

#include <sys/syscall.h>
#include <time.h>
#include <unistd.h>

struct timespec ts;
syscall(SYS_clock_gettime, CLOCK_REALTIME, &ts);

Now strace -e clock_gettime sees the call. On current Linux the real kernel handler is still defined in kernel/time/posix-timers.c with SYSCALL_DEFINE2(clock_gettime, ...). That path selects the clock's kernel implementation and copies the resulting timespec back to userspace.

rust reaches the same code

The Rust standard library example is just as small as before:

use std::time::SystemTime;

fn main() {
    println!("{:?}", SystemTime::now());
}

On a normal GNU/Linux target, Rust does not parse the vDSO itself. Its current Unix time backend eventually asks for CLOCK_REALTIME, and Timespec::now calls libc::clock_gettime.

The path is therefore:

SystemTime::now
  -> Rust's Unix time backend
  -> Timespec::now(CLOCK_REALTIME)
  -> libc::clock_gettime
  -> glibc's resolved vDSO function
  -> __vdso_clock_gettime

There is extra handling on 32-bit GNU/Linux for the time64 transition, but the decision still ends in a libc clock function. Other targets, especially non-glibc or statically linked ones, can use a different libc implementation while presenting the same Rust API.

This explains why strace normally shows no clock_gettime system call for the Rust program either.

python reaches the same code

Python adds more layers but ends up at the same interface:

import time

print(time.time_ns())

The names in my original notes had become stale. In current CPython the route is:

time.time / time.time_ns
  -> time_time / time_time_ns
  -> PyTime_Time
  -> py_get_system_clock
  -> clock_gettime(CLOCK_REALTIME)
  -> glibc's vDSO path

The module entry points are in Modules/timemodule.c. The platform work happens in Python/pytime.c, where py_get_system_clock uses clock_gettime(CLOCK_REALTIME) on a Unix system which provides it, and keeps gettimeofday as a fallback.

So CPython does not contain a separate vDSO implementation. It calls the normal C library API and inherits glibc's resolved fast path. time.time() converts the result to a Python float, while time.time_ns() keeps integer nanoseconds. The operating-system clock lookup below them is shared.

putting the pieces together

For the common CLOCK_REALTIME case, the complete route is:

C clock_gettime / Rust SystemTime::now / Python time.time
  -> libc clock_gettime
  -> function pointer resolved from the vDSO ELF
  -> __vdso_clock_gettime executes in userspace
  -> read a consistent base and conversion data from [vvar]
  -> read the hardware or paravirtual cycle counter
  -> calculate the current timestamp
  -> return without a system call

The vDSO works because the kernel split the operation into two parts. The kernel maintains the privileged and slowly changing clock state, while a small kernel-supplied userspace function combines that state with a counter the process is allowed to read. The sequence counter makes concurrent updates detectable, and the syscall path remains available whenever the shortcut cannot answer safely.

That is more machinery than the original clock_gettime call suggests, but it moves one of the most frequently requested pieces of kernel information out of the syscall path without giving the process control over it.

Tramasys - linux

mirror, mirror on the wall, which cpu am I running on?

experimentation in python#

experimentation in rust#

new world: rseq#

fallback one: the vDSO#

fallback two: the actual syscall#

in the wild: MongoDB and TCMalloc#

what does "currently" mean?#

source trail#

vDSO - why tho?

what is the vDSO?#

finding the image#

who put it there?#

where did the ELF come from?#

code in vdso, data in vvar#

following clock_gettime from c#

resolving it directly#

inside __vdso_clock_gettime#

reading while the kernel writes#

when it still becomes a syscall#

rust reaches the same code#

python reaches the same code#

putting the pieces together#

source trail#

experimentation in python

experimentation in rust

new world: rseq

fallback one: the vDSO

fallback two: the actual syscall

in the wild: MongoDB and TCMalloc

what does "currently" mean?

source trail

what is the vDSO?

finding the image

who put it there?

where did the ELF come from?

code in vdso, data in vvar

following clock_gettime from c

resolving it directly

inside __vdso_clock_gettime

reading while the kernel writes

when it still becomes a syscall

rust reaches the same code

python reaches the same code

putting the pieces together

source trail