User Space: execve(path, argv, envp)
    ↓
Syscall Entry: SYSCALL_DEFINE3(execve, ...)
    ↓
do_execve() → do_execveat_common(AT_FDCWD, ...)
    ↓
1. Validation & Setup
   - Check RLIMIT_NPROC
   - Allocate linux_binprm structure
   - Count and validate argc/envc
    ↓
2. Argument/Environment Copying
   - copy_string_kernel(filename)
   - copy_strings(envc, envp)
   - copy_strings(argc, argv)
   - Handle empty argv edge case
    ↓
3. bprm_execve(bprm)
   ├─ prepare_bprm_creds() - Setup credentials
   ├─ check_unsafe_exec() - Security validation
   ├─ current->in_execve = 1
   ├─ sched_exec() - CPU migration optimization
   ├─ security_bprm_creds_for_exec() - LSM hook
   └─ exec_binprm(bprm)
       └─ search_binary_handler(bprm)
           └─ Iterate registered binfmt handlers
               └─ fmt->load_binary(bprm) [e.g., load_elf_binary]
                   └─ begin_new_exec(bprm) [Point of no return]
    ↓
4. Post-Execution Cleanup
   - sched_mm_cid_after_execve()
   - rseq_execve()
   - current->in_execve = 0
   - user_events_execve()
   - acct_update_integrals()
   - task_numa_free()

search_binary_handler()
    ↓
Iterate through registered formats:
    1. binfmt_script (#! interpreter)
    2. binfmt_elf (ELF binaries)
    3. binfmt_misc (Custom handlers)
    4. binfmt_flat (Flat binaries, embedded)
    ↓
First successful load_binary() wins
    ↓
If interpreter needed (scripts):
    Recursively call exec_binprm() with interpreter
    Max depth: 5 levels (prevents infinite loops)

struct linux_binprm {
    // Memory management
    struct vm_area_struct *vma;          // VMA for argument stack
    unsigned long vma_pages;
    unsigned long argmin;                // Stack limit marker
    struct mm_struct *mm;
    unsigned long p;                     // Current top of memory

    // Flags
    unsigned int
        have_execfd:1,                   // Exec fd passed to userspace
        execfd_creds:1,                  // Use script creds (binfmt_misc)
        secureexec:1,                    // Privilege-gaining exec occurred
        point_of_no_return:1,            // Cannot return errors to userspace
        comm_from_dentry:1,              // Comm from dentry
        is_check:1;                      // Check executability only

    // File references
    struct file *executable;             // Executable file
    struct file *interpreter;            // Interpreter (for scripts)
    struct file *file;                   // Current binary file

    // Credentials
    struct cred *cred;                   // New credentials
    int unsafe;                          // LSM_UNSAFE_* mask
    unsigned int per_clear;              // Personality bits to clear

    // Arguments
    int argc, envc;                      // Argument/environment counts
    const char *filename;                // Binary name (for procps)
    const char *interp;                  // Actual executed binary
    const char *fdpath;                  // Generated path for execveat

    // Execution state
    unsigned interp_flags;
    int execfd;                          // Executable file descriptor
    unsigned long exec;                  // Entry point address

    // Resource limits
    struct rlimit rlim_stack;            // Saved RLIMIT_STACK

    // Magic number buffer (first 128 bytes of file)
    char buf[BINPRM_BUF_SIZE];
};

struct linux_binfmt {
    struct list_head lh;                 // Linked list node
    struct module *module;               // Owning module
    int (*load_binary)(struct linux_binprm *);  // Load handler
#ifdef CONFIG_COREDUMP
    int (*core_dump)(struct coredump_params *); // Core dump handler
    unsigned long min_coredump;          // Minimum core dump size
#endif
};

System Call Interface

Primary Syscalls

execve(const char *filename, char *const argv[], char *const envp[])

Standard POSIX execve implementation.

Parameters:

• filename: Path to executable
• argv: NULL-terminated argument array
• envp: NULL-terminated environment array

Implementation:

SYSCALL_DEFINE3(execve,
    const char __user *, filename,
    const char __user *const __user *, argv,
    const char __user *const __user *, envp)
{
    return do_execve(getname(filename), argv, envp);
}

execveat(int fd, const char *filename, char *const argv[], char *const envp[], int flags)

Extended version supporting relative paths and additional flags.

Flags:

• AT_EMPTY_PATH: Allow empty filename (operate on fd directly)
• AT_SYMLINK_NOFOLLOW: Don't follow symlinks

Implementation:

SYSCALL_DEFINE5(execveat,
    int, fd, const char __user *, filename,
    const char __user *const __user *, argv,
    const char __user *const __user *, envp,
    int, flags)
{
    return do_execveat(fd, getname_uflags(filename, flags), argv, envp, flags);
}

Compatibility Syscalls

For 32-bit compatibility on 64-bit kernels:

• COMPAT_SYSCALL_DEFINE3(execve, ...)
• COMPAT_SYSCALL_DEFINE5(execveat, ...)

Kernel Internal API

kernel_execve(const char *filename, const char *const *argv, const char *const *envp)

Execute a program from kernel space (e.g., init process, hotplug helpers).

Restrictions:

• Cannot be called from kernel threads (PF_KTHREAD)
• Uses kernel pointers instead of user pointers

Binary Format Handlers

Registered Formats

Registration API

// Register at end of list (lower priority)
void register_binfmt(struct linux_binfmt *fmt);

// Insert at beginning (higher priority)
void insert_binfmt(struct linux_binfmt *fmt);

// Unregister format
void unregister_binfmt(struct linux_binfmt *fmt);

Example (from binfmt_elf.c):

static struct linux_binfmt elf_format = {
    .module     = THIS_MODULE,
    .load_binary = load_elf_binary,
#ifdef CONFIG_COREDUMP
    .core_dump  = elf_core_dump,
    .min_coredump = ELF_EXEC_PAGESIZE,
#endif
};

static int __init init_elf_binfmt(void)
{
    register_binfmt(&elf_format);
    return 0;
}

Security Mechanisms

Credential Transitions

Execve handles privilege changes based on file permissions:

1. SUID/SGID Detection:

• Check executable's setuid/setgid bits
• Validate filesystem supports SUID (nosuid mount check)
• Apply new UID/GID from file ownership

1. Capability Handling:

• File capabilities (if supported by filesystem)
• Ambient capability inheritance
• Securebits preservation

1. LSM Hooks:

security_bprm_check(bprm);              // Pre-execution check
security_bprm_creds_for_exec(bprm);     // Credential setup
security_bprm_committing_creds(bprm);   // Before committing
security_bprm_committed_creds(bprm);    // After committing

Unsafe Execution States

Tracked via bprm->unsafe flags:

When unsafe states detected:

• SUID/SGID bits ignored
• File capabilities dropped
• AT_SECURE auxv flag set for glibc

Point of No Return

After begin_new_exec() succeeds:

• Old process image destroyed
• Cannot return errors to userspace
• Fatal signal (SIGSEGV) if subsequent failure occurs
• Traced via bprm->point_of_no_return

Memory Management

Argument Stack Layout

High Addresses
┌─────────────────────┐
│  Environment Strings │ ← envp[0], envp[1], ...
├─────────────────────┤
│  Argument Strings    │ ← argv[0], argv[1], ...
├─────────────────────┤
│  Filename String     │ ← Program path
├─────────────────────┤
│  Auxiliary Vector    │ ← AT_ENTRY, AT_PHDR, etc.
├─────────────────────┤
│  Environment Pointers│ ← envp[] array + NULL
├─────────────────────┤
│  Argument Pointers   │ ← argv[] array + NULL
├─────────────────────┤
│  Argument Count      │ ← argc
├─────────────────────┤
Low Addresses

Stack Limits

• Maximum argument strings: MAX_ARG_STRINGS (typically 2^31)
• Maximum string length: MAX_ARG_STRLEN (PAGE_SIZE * 32)
• Stack expansion: Limited by RLIMIT_STACK
• Guard page: Prevents stack overflow into other regions

ASLR (Address Space Layout Randomization)

Randomization applied to:

• Stack base address
• Mmap base address
• VDSO location
• Executable base (PIE binaries)

Controlled via:

• /proc/sys/kernel/randomize_va_space
• personality(ADDR_NO_RANDOMIZE) flag

Key Functions

Core Execution Functions

do_execveat_common()

Main entry point for all exec variants.

Responsibilities:

1. Validate filename and resource limits
2. Allocate and initialize linux_binprm
3. Count and copy arguments/environment
4. Handle empty argv edge case
5. Call bprm_execve()

bprm_execve()

Orchestrate the execution process.

Flow:

prepare_bprm_creds(bprm);
check_unsafe_exec(bprm);
current->in_execve = 1;
sched_exec();
security_bprm_creds_for_exec(bprm);
exec_binprm(bprm);  // May not return on success
// Post-exec cleanup on failure

exec_binprm()

Search for and invoke appropriate binary format handler.

Features:

• Iterates through registered formats
• Handles interpreter chains (scripts)
• Maximum recursion depth: 5 levels
• Audit and tracing integration

search_binary_handler()

Try each registered binary format until one succeeds.

Algorithm:

read_lock(&binfmt_lock);
list_for_each_entry(fmt, &formats, lh) {
    if (!try_module_get(fmt->module))
        continue;
    read_unlock(&binfmt_lock);

    retval = fmt->load_binary(bprm);

    read_lock(&binfmt_lock);
    put_binfmt(fmt);
    if (retval != -ENOEXEC)
        return retval;
}
return -ENOEXEC;

Helper Functions

Lifecycle Phases

Phase 1: Preparation

1. Validation:

• Check filename validity
• Verify RLIMIT_NPROC not exceeded
• Clear PF_NPROC_EXCEEDED flag

1. Allocation:

• Allocate linux_binprm structure
• Read first 128 bytes of executable (magic number detection)
• Open executable file

1. Counting:

• Count argc and envc
• Validate against MAX_ARG_STRINGS
• Calculate stack requirements

Phase 2: Copying

1. String Copying:

• Copy filename to bprm stack
• Copy environment strings
• Copy argument strings
• Handle empty argv (add "" as argv[0])

1. Stack Setup:

• Create temporary VMA for argument stack
• Track current stack pointer (bprm->p)
• Enforce stack limits

Phase 3: Execution

1. Credential Preparation:

• Prepare new credentials
• Check for unsafe execution states
• LSM security checks

1. Binary Loading:

• Search for matching binary format
• Invoke load_binary() handler
• Handle interpreter recursion

1. Commit (Point of No Return):

• begin_new_exec() called by loader
• Destroy old memory mappings
• Install new credentials
• Setup new stack frame

Phase 4: Completion

1. Post-Exec Actions:

• Update scheduling statistics
• RSEQ (Restartable Sequences) notification
• User events cleanup
• Accounting updates
• NUMA balancing reset

1. Cleanup:

• Free bprm structure
• Release filename reference
• Clear current->in_execve

Error Handling

Pre Point-of-No-Return

Errors returned to userspace normally:

• -ENOENT: File not found
• -EACCES: Permission denied
• -ENOMEM: Out of memory
• -E2BIG: Argument list too long
• -ENOEXEC: Invalid executable format
• -ELOOP: Too many levels of symbolic links or interpreters

Post Point-of-No-Return

Cannot return to userspace safely:

• Send fatal signal (SIGSEGV) if error occurs
• Process termination guaranteed
• No userspace recovery possible

Integration Points

Tracing & Auditing

trace_sched_process_exec(current, old_pid, bprm);  // Scheduler trace
audit_bprm(bprm);                                   // Audit subsystem
ptrace_event(PTRACE_EVENT_EXEC, old_vpid);         // Ptrace notification
proc_exec_connector(current);                       // Proc connector

Scheduler Integration

• sched_exec(): Optimize CPU selection after exec
• sched_mm_cid_before_execve(): Save concurrency ID
• sched_mm_cid_after_execve(): Restore concurrency ID

Security Subsystem

• LSM hooks at multiple stages
• AppArmor/SELinux policy enforcement
• Capability bounding set updates
• Secure computing mode (seccomp) preservation

Resource Accounting

• Update integral accounting (CPU time)
• Reset NUMA balancing state
• Clear io_uring context
• Update task statistics

Configuration Options

Common Issues

Issue 1: "Argument List Too Long" (E2BIG)

Cause: Total size of arguments + environment exceeds stack limit

Solution:

• Reduce argument count or length
• Increase RLIMIT_STACK
• Use response files (@file) pattern

Issue 2: "Permission Denied" on SUID Binary

Cause: 

• Filesystem mounted with nosuid
• Process has no_new_privs bit set
• SELinux/AppArmor policy denial

Debugging:

mount | grep nosuid
cat /proc/self/status | grep NoNewPrivs
dmesg | grep AVC  # SELinux denials

Issue 3: Interpreter Loop Detection

Cause: Script interpreter chain exceeds 5 levels

Example:

#!/usr/bin/python3  # Level 1
#!/usr/bin/env python3  # Level 2 (via env)
...

Solution: Reduce interpreter nesting depth

Issue 4: Memory Corruption During Exec

Cause: 

• Buggy binary format handler
• Race condition in credential transition
• Stack overflow during argument copying

Debugging:

• Enable KASAN (Kernel Address Sanitizer)
• Check dmesg for OOPS messages
• Review custom binfmt handlers

Best Practices

For Kernel Developers

1. Use EXPORT_SYMBOL_GPL for exec-related exports
2. Validate all user pointers before dereferencing
3. Check return values from all allocation functions
4. Handle partial failures gracefully before point of no return
5. Document binary format handlers with kerneldoc
6. Test with various configurations (SUID, namespaces, LSM enabled)

For Userspace Applications

1. Check execve return value - never assume success
2. Limit argument size to avoid E2BIG
3. Use execveat for better security (relative paths, O_CLOEXEC)
4. Clear sensitive data from environment before exec
5. Handle SIGCHLD properly in parent processes

Related Files

Testing

KUnit Tests

Enable with CONFIG_EXEC_KUNIT_TEST=y:

#ifdef CONFIG_EXEC_KUNIT_TEST
#include "tests/exec_kunit.c"
#endif

Manual Testing

# Test basic execve
strace -e execve ./test_program

# Test with different formats
./elf_binary
./script.sh
echo "test" | binfmt_misc_handler

# Test SUID behavior
chmod u+s suid_program
ls -l suid_program
./suid_program

# Test resource limits
ulimit -s 8192  # 8MB stack
./large_argv_program

Performance Considerations

Optimization Opportunities

1. sched_exec(): Migrates process to optimal CPU
2. Demand loading: Page faults load executable on-demand
3. Shared libraries: mmap() with MAP_PRIVATE for COW
4. VDSO: Virtual dynamic shared object avoids syscalls

Bottlenecks

1. Argument copying: O(n) where n = total arg+env size
2. Binary parsing: ELF header/program header processing
3. Library resolution: Dynamic linker overhead
4. Page faults: Initial code/data access patterns

Profiling

# Profile execve latency
perf record -e syscalls:sys_enter_execve,syscalls:sys_exit_execve ./program
perf report

# Trace binary format selection
ftrace -p function_graph -g search_binary_handler

Historical Notes

• 1991: Original exec implementation by Linus Torvalds
• 1993: Demand-loading added by Eric Youngdale (mmap-based)
• 1990s: Binary format dispatch table introduced
• 2000s: Security enhancements (LSM, capabilities)
• 2010s: execveat syscall added (Linux 3.17)
• 2020s: Enhanced auditing and tracing support

References

• POSIX execve Specification
• Linux man page: execve(2)
• ELF Specification
• Linux Security Modules
• Binary Formats

Last Updated: 2026-04-09
Maintainer: Linux Kernel Process Management Maintainers
Status: Stable - Core System Call