Abstract
A System Call is the official application programming interface (API) between a user application and the operating system. Because user applications run in unprivileged User Mode, they invoke system calls using software trap instructions to safely request privileged operations from the kernel.
- Category: OS Interface & API Architecture
- Core Categories: Process, Memory, File, Device, and Communication Management.
- Key Mechanism: Software Trap Instruction (
syscall,sysenter, orint 0x80).
System Call Categories
The operating system exports a defined set of system calls, accessible via standard C libraries (e.g., POSIX, Win32):
- Process Management:
fork(),exec(),exit(),wait() - Memory Management:
mmap(),brk(),sbrk() - File Management:
open(),read(),write(),close() - Device Management:
ioctl(),read(),write() - Communication & Inter-Process Communication (IPC):
socket(),bind(),connect(),pipe()
The System Call Trap Mechanism
User programs cannot call kernel C function pointers directly. Instead, they trigger a deliberate software exception (a trap) that switches execution to Kernel Mode:
Execution Sequence (read() Example)
- API Invocation: The user application calls
read(fd, buffer, n). - Trap Setup: The standard C library puts the system call identifier for
read()into a specific CPU register (e.g.,%eaxor%rax). - Software Trap: The library executes a hardware trap instruction (e.g.,
syscallorsysenter). - Mode Switch & Dispatch: The CPU switches to Kernel Mode, saves user register states, and jumps to the kernel’s central
SyscallHandlervia the interrupt vector table. - Service Execution: The kernel reads the system call ID from the register, validates arguments, and executes
sys_read(). - Return To User Space: Upon completion, the kernel writes the return value to a register and executes a return-from-trap instruction (
sysret), which restores user registers, sets the mode bit back to User Mode (), and resumes execution.
Referencing Kernel Objects: Handles vs. Pointers
Because user processes and the OS kernel operate in separate memory address spaces, applications cannot pass raw kernel memory pointers to system calls.
Safe Object Descriptors
The kernel uses integer handles or descriptors (e.g., Unix File Descriptors like
fd = 3) instead of direct pointers. The kernel indexes these integers into private process lookup tables, ensuring applications cannot forge pointers to corrupt kernel memory structures.