101
0/240
Loading content...
The history of real-time Linux is a story of competing philosophies, technical innovation, legal battles, and ultimately convergence. From the earliest attempts to add real-time capabilities to Linux in the mid-1990s to today's mainlined PREEMPT_RT, the community has explored fundamentally different architectural approaches.
Understanding this history—and the distinctions between approaches—is essential for any real-time Linux practitioner. Legacy systems using older RT-Linux variants still exist in production worldwide, and the design decisions of these projects continue to influence modern real-time computing.
By the end of this page, you will understand: (1) The dual-kernel vs single-kernel architectural debate; (2) The history and design of RTLinux, RTAI, and Xenomai; (3) How these approaches compare to PREEMPT_RT; (4) When each approach is appropriate; (5) The path toward a unified real-time Linux; and (6) The current state of the real-time Linux ecosystem.
The challenge of making Linux real-time has been tackled via two fundamentally different architectural approaches, each with distinct trade-offs:
12345678910111213141516171819202122232425262728
Dual-Kernel Architecture (RTLinux, RTAI, Xenomai Cobalt):┌───────────────────────────────────────────────────────────────┐│ User Space Applications (Normal Linux Apps) │├───────────────────────────────────────────────────────────────┤│ Linux Kernel ││ (Runs as a task under the RT kernel - can be preempted) │├───────────────────────────────────────────────────────────────┤│ Real-Time Microkernel / Interrupt Virtualization Layer ││ - Handles all hardware interrupts ││ - Real-time tasks run here with hard guarantees ││ - Linux interrupts are virtualized (can be delayed) │├───────────────────────────────────────────────────────────────┤│ Hardware │└───────────────────────────────────────────────────────────────┘ Single-Kernel Architecture (PREEMPT_RT):┌───────────────────────────────────────────────────────────────┐│ User Space Applications ││ (Both RT and normal apps as regular processes) │├───────────────────────────────────────────────────────────────┤│ Linux Kernel with PREEMPT_RT ││ - Fully preemptible (nearly everywhere) ││ - RT tasks are normal processes with RT scheduling ││ - All Linux APIs available ││ - Threaded interrupts compete via scheduler │├───────────────────────────────────────────────────────────────┤│ Hardware │└───────────────────────────────────────────────────────────────┘| Aspect | Dual-Kernel | Single-Kernel (PREEMPT_RT) |
|---|---|---|
| Latency Guarantee | Harder guarantees (microseconds) | Soft guarantees (tens of microseconds) |
| Linux API in RT | Limited (special APIs required) | Full Linux API available |
| Driver Reuse | Must port or write RT drivers | Uses standard Linux drivers |
| Maintenance | Separate codebase to maintain | Part of mainline Linux |
| Debugging | Separate RT debugging tools | Standard Linux tools (gdb, etc.) |
| Complexity | Higher (two execution domains) | Lower (unified environment) |
| Worst-Case Analysis | Easier (isolated RT domain) | Harder (more paths through kernel) |
Early in RT-Linux history, dual-kernel approaches offered clearly superior worst-case latency. As PREEMPT_RT matured—and hardware became faster—the gap has narrowed significantly. For many applications, PREEMPT_RT now provides sufficient guarantees with much simpler development.
RTLinux was the first significant attempt to add hard real-time capabilities to Linux, developed by Victor Yodaiken and Michael Barabanov at New Mexico Institute of Mining and Technology (NMT) beginning in 1995.
The Key Insight:
RTLinux was based on a deceptively simple observation: making the entire Linux kernel real-time is extremely difficult due to its size and complexity. But what if you ran Linux as a task under a tiny, simple real-time kernel that you could guarantee?
123456789101112131415161718192021222324252627282930
RTLinux Interrupt Virtualization: Hardware Interrupt Occurs │ ▼┌─────────────────────────────────────────────────────────────────┐│ RTLinux Microkernel Interrupt Handler ││ ││ 1. Is this for an RT task? ──────YES──► Run RT handler ││ │ │ ││ NO Done ││ │ ││ ▼ ││ 2. Queue interrupt for Linux (virtualized) ││ Linux will see it when scheduled ││ │└─────────────────────────────────────────────────────────────────┘ Linux's View:┌─────────────────────────────────────────────────────────────────┐│ Linux thinks it controls hardware interrupts with cli/sti ││ But these are EMULATED by RTLinux! ││ ││ cli (disable interrupts): ││ Actually: sets flag "Linux doesn't want interrupts" ││ Linux still runs; RT tasks still preempt ││ ││ sti (enable interrupts): ││ Actually: clears flag, delivers any pending virtual IRQs │└─────────────────────────────────────────────────────────────────┘Writing RTLinux Real-Time Tasks:
RT tasks in RTLinux were written as kernel modules—not normal user-space processes. They executed in kernel space with direct hardware access but without memory protection.
1234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465
/* * Example RTLinux Real-Time Task (Historical) * * Note: This is kernel module code running in the RT domain, * NOT a normal user-space application! */ #include <rtl.h>#include <time.h>#include <pthread.h> pthread_t rt_thread;int keep_running = 1; /* Real-time task - runs in RT kernel domain */void* rt_task_function(void* arg) { struct timespec next_period; const struct timespec period = { 0, 1000000 }; /* 1ms */ /* Get current time */ clock_gettime(CLOCK_REALTIME, &next_period); while (keep_running) { /* * DO RT WORK HERE * - Direct hardware access OK * - No memory protection! * - Bug = kernel crash */ do_control_loop(); /* Calculate next period */ timespec_add(&next_period, &period); /* Sleep until next period */ clock_nanosleep(CLOCK_REALTIME, TIMER_ABSTIME, &next_period, NULL); } return NULL;} /* Module initialization - called when module loads */int init_module(void) { pthread_attr_t attr; struct sched_param param; pthread_attr_init(&attr); /* Set SCHED_FIFO (in RT domain, not Linux!) */ param.sched_priority = 1; pthread_attr_setschedparam(&attr, ¶m); pthread_attr_setschedpolicy(&attr, SCHED_FIFO); /* Create RT thread */ pthread_create(&rt_thread, &attr, rt_task_function, NULL); return 0;} /* Module cleanup */void cleanup_module(void) { keep_running = 0; pthread_join(rt_thread, NULL);}RTLinux History and Fate:
The RTLinux patent—covering running a general-purpose OS as a task under an RT kernel—created significant controversy. While FSMLabs offered free licensing for GPL projects, the patent's existence and enforcement attempts damaged community trust and motivated development of alternatives.
RTAI (Real-Time Application Interface) emerged from the Politecnico di Milano in Italy, initially as an alternative implementation of the RTLinux concept that sought to avoid patent issues and provide a more community-friendly project.
Key RTAI Innovations:
While architecturally similar to RTLinux, RTAI introduced several significant improvements:
12345678910111213141516171819202122
RTAI LXRT: User-Space Hard Real-Time┌─────────────────────────────────────────────────────────────────┐│ ││ User Space RT Task (via LXRT) ││ ┌─────────────────────────────────────────────────────────────┐││ │ Regular user-space process │││ │ - Memory protection! │││ │ - Debuggable with standard tools │││ │ - BUT: runs under RTAI scheduler when doing RT work │││ └─────────────────────────────────────────────────────────────┘││ │ ││ │ rt_task_make_periodic() ││ ▼ (switch to RT scheduling) ││ ┌─────────────────────────────────────────────────────────────┐││ │ RTAI RT Domain │││ │ - Task now scheduled by RTAI, not Linux │││ │ - Hard real-time guarantees │││ │ - Linux blocked while RT task runs │││ └─────────────────────────────────────────────────────────────┘││ ││ Innovation: Get memory protection AND hard RT! │└─────────────────────────────────────────────────────────────────┘123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566
/* * Example RTAI LXRT User-Space Real-Time Task (Historical) * * This runs as a normal user-space process but gets * hard real-time scheduling when in RT mode. */ #include <rtai_lxrt.h>#include <rtai_sem.h> #define PERIOD_NS 1000000 /* 1ms */ int main(void) { RT_TASK *task; RTIME period; /* Lock memory (prevent page faults) */ mlockall(MCL_CURRENT | MCL_FUTURE); /* * Initialize LXRT - tells RTAI about this process * nam2num() converts string to unique ID */ task = rt_task_init(nam2num("MYTASK"), 0, /* priority (0 = highest) */ 0, /* stack size (0 = default) */ 0); /* message size */ if (!task) { fprintf(stderr, "Failed to initialize RT task"); return 1; } /* * Switch to hard real-time mode * From this point, Linux cannot preempt us! */ rt_make_hard_real_time(); /* Calculate period in RTAI time units */ period = nano2count(PERIOD_NS); /* Make task periodic starting now */ rt_task_make_periodic(task, rt_get_time(), period); /* Real-time loop */ while (!should_stop()) { /* === RT WORK HAPPENS HERE === */ /* Hard real-time guaranteed! */ do_control_computation(); /* === END RT WORK === */ /* Wait for next period */ rt_task_wait_period(); } /* Switch back to soft real-time (Linux scheduling) */ rt_make_soft_real_time(); /* Cleanup */ rt_task_delete(task); return 0;}RTAI Status:
RTAI remains available and maintained, though development has slowed significantly. It's still used in:
RTAI is appropriate when you need proven hard real-time guarantees, are willing to use its specific APIs, and can accept the maintenance burden of an out-of-tree kernel patch. For new projects, Xenomai or PREEMPT_RT are generally better choices.
Xenomai is the most sophisticated and actively maintained dual-kernel real-time Linux solution. Begun in 2001, it introduced crucial innovations that address many limitations of earlier approaches.
Xenomai Architecture (Cobalt Core):
Xenomai 3.x features a dual architecture: Cobalt (dual-kernel) for hard real-time and Mercury (single-kernel on PREEMPT_RT) for soft real-time.
1234567891011121314151617181920212223242526272829
Xenomai Cobalt (Dual-Kernel Mode):┌───────────────────────────────────────────────────────────────┐│ User Space ││ ┌─────────────────────────────────────────────────────────┐ ││ │ Xenomai Application (uses POSIX or RTDM APIs) │ ││ │ │ ││ │ • When calling RT functions: handled by Cobalt │ ││ │ • When calling Linux functions: handled by Linux │ ││ │ • Seamless switching between domains │ ││ └─────────────────────────────────────────────────────────┘ │├───────────────────────────────────────────────────────────────┤│ Xenomai Libraries (libcobalt) ││ • POSIX skin (pthreads, mutexes, etc. - RT implementations) ││ • VxWorks skin (emulation for porting) ││ • RTDM (Real-Time Driver Model) │├───────────────────────────────────────────────────────────────┤│ Linux Kernel │ Cobalt Core (RT Co-Kernel) ││ ┌──────────────────┐ │ ┌───────────────────────────────────┐ ││ │ Standard Linux │ │ │ • Hard RT scheduler │ ││ │ kernel with │ │ │ • RT resource management │ ││ │ I-pipe patch │◄┼►│ • Interrupt pipeline (I-pipe) │ ││ │ │ │ │ • Microsecond-level guarantees │ ││ └──────────────────┘ │ └───────────────────────────────────┘ │├───────────────────────────────────────────────────────────────┤│ I-pipe (Interrupt Pipeline) ││ • All interrupts flow through I-pipe first ││ • Cobalt gets first chance at every interrupt ││ • Linux sees virtualized interrupts │└───────────────────────────────────────────────────────────────┘123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081
/* * Xenomai 3.x Real-Time Application (Cobalt) * * Uses standard POSIX APIs - but they're the RT versions! * Compile with: xeno-config --posix --ldflags --cflags */ #include <pthread.h>#include <time.h>#include <stdio.h>#include <sys/mman.h>#include <signal.h> #define PERIOD_NS 1000000 /* 1ms period */ static int running = 1; void* rt_thread(void* arg) { struct timespec next_wake; printf("RT thread started on Xenomai/Cobalt"); clock_gettime(CLOCK_MONOTONIC, &next_wake); while (running) { /* Calculate next wake time */ next_wake.tv_nsec += PERIOD_NS; if (next_wake.tv_nsec >= 1000000000) { next_wake.tv_nsec -= 1000000000; next_wake.tv_sec++; } /* * Do RT work here * * IMPORTANT: Calling pure POSIX functions stays in RT domain * Calling Linux-specific functions (e.g., printf) causes * automatic switch to Linux domain (mode switch) */ do_control_loop(); /* Sleep until next period - RT precise! */ clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &next_wake, NULL); } return NULL;} int main(void) { pthread_t thread; pthread_attr_t attr; struct sched_param param; /* Lock memory */ mlockall(MCL_CURRENT | MCL_FUTURE); /* Create RT thread with SCHED_FIFO */ pthread_attr_init(&attr); pthread_attr_setinheritsched(&attr, PTHREAD_EXPLICIT_SCHED); pthread_attr_setschedpolicy(&attr, SCHED_FIFO); param.sched_priority = 80; pthread_attr_setschedparam(&attr, ¶m); /* * This pthread_create is intercepted by libcobalt * Thread will be scheduled by Cobalt (hard RT), not Linux! */ if (pthread_create(&thread, &attr, rt_thread, NULL) != 0) { perror("pthread_create"); return 1; } pthread_attr_destroy(&attr); /* Wait for thread (in Linux domain - that's fine) */ pthread_join(thread, NULL); return 0;}Mode Switching in Xenomai:
Xenomai applications can switch between RT (Cobalt) and non-RT (Linux) domains. This flexibility is powerful but requires understanding:
123456789101112131415161718192021222324252627282930
Mode Switching Behavior: RT Domain (Cobalt) Linux Domain┌──────────────────┐ ┌──────────────────┐│ • Hard RT timing │ │ • Normal Linux ││ • POSIX RT APIs │ Mode │ • All syscalls ││ • RTDM drivers │◄─Switch──────►│ • File I/O ││ • Low jitter │ │ • Networking │└──────────────────┘ └──────────────────┘ Automatic Mode Switch Triggers (RT → Linux):• Calling Linux system calls (open, read, write, etc.)• Page faults• Signals from Linux• Non-RT-safe library calls Returning to RT Domain:• Calling Xenomai POSIX function• Waiting on RT synchronization object WARNING: Mode switches add latency!• Overhead: ~1-5 microseconds per switch• During switch: briefly non-deterministic• Avoid mode switches in timing-critical code paths Best Practice:• Do all Linux I/O before entering RT loop• Use RTDM drivers for RT-safe I/O• Keep RT work purely computational• Use shared memory for non-RT communicationXenomai/Cobalt remains the best choice when you need microsecond-level hard real-time guarantees on Linux and can work within its constraints. Its POSIX compatibility makes porting easier than RTAI, and active development ensures modern kernel support.
With multiple RT Linux solutions available, choosing the right one requires understanding their characteristics in context of your requirements:
| Aspect | PREEMPT_RT | Xenomai (Cobalt) | RTAI |
|---|---|---|---|
| Worst-Case Latency | 20-100 μs (typical hardware) | 1-10 μs | 1-10 μs |
| Architecture | Single kernel | Dual kernel (I-pipe) | Dual kernel |
| Mainline Status | Merged / merging into mainline | Out-of-tree patch (I-pipe) | Out-of-tree patch |
| Linux Kernel Support | Current mainline | Selective kernels (lag) | Selective kernels (larger lag) |
| API Model | Standard Linux POSIX | POSIX (libcobalt) + RTDM | Custom + LXRT |
| Driver Reuse | All Linux drivers work | Some need RTDM port | May need porting |
| Development Effort | Lowest (standard Linux dev) | Medium (POSIX familiar) | Higher (specific APIs) |
| Debugging | Standard Linux tools | Specialized tools needed | Specialized tools needed |
| Active Development | Very active (Linux community) | Active (smaller community) | Limited maintenance |
| Commercial Support | Multiple vendors | Limited | Very limited |
Decision Framework:
123456789101112131415161718192021222324252627282930313233343536
Decision Tree: Which RT Linux? 1. What are your latency requirements? └── Need < 10μs worst-case guaranteed? └── YES → Xenomai Cobalt (or possibly RTAI) └── NO → Continue to question 2 2. What's your tolerance for kernel patches? └── Prefer mainline kernel with minimal patches? └── YES → PREEMPT_RT (largely mainlined) └── NO → Consider Xenomai for tighter guarantees 3. What's your development team's expertise? └── Standard Linux development skills? └── YES → PREEMPT_RT (familiar environment) └── NO (have RTOS experience) → Xenomai skins help 4. Do you need to port from VxWorks/pSOS/etc.? └── YES → Xenomai (has emulation skins) └── NO → Continue 5. Is long-term maintenance critical? └── YES → PREEMPT_RT (mainline = maintained forever) └── NO → Any solution works if requirements met RECOMMENDED PATH (2024+):┌─────────────────────────────────────────────────────────────────┐│ For most new projects: Start with PREEMPT_RT ││ ││ • Measure actual latency requirements ││ • If PREEMPT_RT doesn't meet them (< 10μs needed) ││ → Then invest in Xenomai/Cobalt ││ ││ Don't assume you need microsecond guarantees! ││ Many "real-time" requirements are actually 100μs-1ms range. │└─────────────────────────────────────────────────────────────────┘The Linux community is converging on PREEMPT_RT as the primary real-time solution. Xenomai's Mercury mode (using PREEMPT_RT) shows even dual-kernel proponents recognize the value of single-kernel simplicity for many applications. For new projects, start with PREEMPT_RT unless you have proven sub-10μs requirements.
The real-time Linux landscape continues to evolve, with clear trends pointing toward a more unified future:
The Mainlining Journey:
1234567891011121314151617181920212223242526272829303132
PREEMPT_RT Mainline Integration Progress: [===================================================] ~95% Complete Already Mainlined:✓ Threaded interrupt handlers (IRQF_THREAD)✓ Priority inheritance futexes ✓ High-resolution timers✓ Preemptible RCU✓ Raw spinlock infrastructure✓ Generic mutex implementation✓ Sleeping spinlocks core✓ PREEMPT_RT config option In Progress / Remaining (~2024+):○ Printk subsystem rework (nearly complete)○ Remaining locking edge cases○ Some architecture-specific bits Goal State:┌─────────────────────────────────────────────────────────────────┐│ $ make menuconfig ││ ││ General setup ---> ││ Preemption Model (Fully Preemptible Kernel (Real-Time)) ← ││ ( ) No Forced Preemption (Server) ││ ( ) Voluntary Kernel Preemption (Desktop) ││ ( ) Preemptible Kernel (Low-Latency Desktop) ││ (X) Fully Preemptible Kernel (Real-Time) ← JUST CONFIG! ││ ││ No patches, no out-of-tree code, just a standard kernel option │└─────────────────────────────────────────────────────────────────┘The mainlining of PREEMPT_RT represents two decades of patient engineering work. It proves that a community-driven effort can transform a general-purpose OS into a real-time platform while maintaining compatibility and gaining acceptance from thousands of kernel developers.
If you're working with legacy RT Linux systems or considering migration between approaches, here are key considerations:
The most successful migrations start by clearly documenting actual timing requirements (not just 'real-time'), then systematically verifying the new solution meets them. Many projects discover their requirements were over-specified, simplifying the migration.
The quest for real-time Linux has produced multiple sophisticated solutions, each with its place in the ecosystem. Let's consolidate the key concepts:
What's Next:
With a complete understanding of RT Linux variants, we'll conclude with application guidelines—practical advice for developing real-time applications on Linux, including design patterns, common pitfalls, and best practices for achieving deterministic behavior.
You now understand the history, architecture, and trade-offs of different RT Linux approaches. This context enables informed decisions about which solution fits your specific requirements and constraints.