IPC Overview

Message Passing vs. Shared Memory Conceptual Model
═══════════════════════════════════════════════════
 
SHARED MEMORY MODEL:
┌─────────────────┐                   ┌─────────────────┐
│   Process A     │                   │   Process B     │
│                 │                   │                 │
│  Write to addr  │──────┬──────────►│  Read from addr │
│                 │      │            │                 │
└─────────────────┘      │            └─────────────────┘
                         │
                    ┌────▼────┐
                    │ Shared  │
                    │ Memory  │   ← Both directly access
                    │ Region  │      the same memory
                    └─────────┘
 
 
MESSAGE PASSING MODEL:
┌─────────────────┐                   ┌─────────────────┐
│   Process A     │                   │   Process B     │
│                 │                   │                 │
│  send(B, msg)   │                   │  recv(A, &msg)  │
│                 │                   │                 │
└────────┬────────┘                   └────────▲────────┘
         │                                     │
         │  ┌─────────────────────────────┐   │
         └─►│       Message Channel       │───┘
            │  (kernel-managed buffer)    │
            │                             │
            │  [msg1] [msg2] [msg3] ...   │  ← Kernel owns
            └─────────────────────────────┘     the channel

1
2
3
4
5
6
7
8
9
10
11
12
13
14

// Direct Naming - Symmetric (both sides name each other)
// Process A:
send(process_B, message);
 
// Process B:
receive(process_A, &message);
 
// Direct Naming - Asymmetric (sender names, receiver accepts any)
// Process A:
send(process_B, message);  // Sender specifies recipient
 
// Process B (server pattern):
receive(&sender_id, &message);  // Accept from any sender
reply(sender_id, response);     // Reply to whoever sent

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

// Indirect Naming - Mailbox Model
// Both processes send to/receive from a mailbox, not each other
 
// Create a mailbox
mailbox_t mailbox = create_mailbox("/my_mailbox");
 
// Process A (sender):
send(mailbox, message);  // Send to mailbox, not to a process
 
// Process B (receiver):
receive(mailbox, &message);  // Receive from mailbox
 
// Multiple senders, multiple receivers possible!
// Process C (another sender):
send(mailbox, another_message);
 
// Process D (another receiver):
receive(mailbox, &yet_another_message);
 
// Who gets which message? Depends on the mailbox semantics:
// - One-to-one: Exactly one receiver gets each message
// - Broadcasting: All receivers get all messages (rare)

Blocking Communication Timeline
════════════════════════════════
 
BLOCKING SEND (waits for receiver):
─────────────────────────────────────────────────────────────────────►
Sender:    |--send()--| (blocked) |------- continues -------
           │                      ▲
           │    message in        │ receiver accepted
           ▼    kernel buffer     │ the message
─────────────────────────────────────────────────────────────────────►
Receiver:                    |--receive()--|
 
 
BLOCKING RECEIVE (waits for sender):
─────────────────────────────────────────────────────────────────────►
Receiver:  |--receive()--| (blocked) |---- continues (with message)
           │                         ▲
           │    waiting for          │ sender provided
           ▼    a message            │ a message
─────────────────────────────────────────────────────────────────────►
Sender:                         |--send()--|
 
 
With blocking semantics:
• Sender cannot proceed until receiver acknowledges
• Receiver cannot proceed until a message arrives
• Provides natural flow control and synchronization
• Simple to reason about - no race conditions

Non-Blocking Communication Timeline
═════════════════════════════════════
 
NON-BLOCKING SEND:
─────────────────────────────────────────────────────────────────────►
Sender:    |--send()--| message queued, returns immediately
                      | continues without waiting for receiver
                      ▼
           ┌─────────────────────┐
           │   Message Buffer   │ (kernel holds message)
           │   [msg in queue]   │
           └─────────────────────┘
                                ▲
Receiver:                       └── receive() later gets the message
 
 
NON-BLOCKING RECEIVE:
─────────────────────────────────────────────────────────────────────►
Receiver:  |--receive()--| returns immediately (may be empty!)
           │ if message available: returns message
           │ if no message: returns error/indication
           ▼
           continue doing other work...
           │
           └─► poll again later with receive()
 
 
Non-blocking semantics:
• Sender doesn't wait - messages buffered
• Receiver doesn't wait - checks and moves on
• Requires explicit polling or event notification
• Can lead to buffer overflow if sender is faster
• Enables parallelism and responsiveness

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

// Go's unbuffered channels provide rendezvous semantics
 
ch := make(chan int)  // Unbuffered channel
 
// Goroutine 1 (sender)
go func() {
    fmt.Println("Sender: about to send")
    ch <- 42  // Blocks until receiver is ready
    fmt.Println("Sender: sent!")
}()
 
// Goroutine 2 (receiver)
go func() {
    time.Sleep(time.Second)  // Delay receiver
    fmt.Println("Receiver: about to receive")
    value := <-ch  // Blocks until sender is ready
    fmt.Println("Receiver: got", value)
}()
 
// Output:
// Sender: about to send
// (1 second delay)
// Receiver: about to receive
// Sender: sent!
// Receiver: got 42
 
// Note: Sender was blocked for ~1 second waiting for receiver!

Effect of Buffer Size on Producer-Consumer Communication
═══════════════════════════════════════════════════════════
 
Scenario: Producer generates messages at 100/sec, Consumer processes at 80/sec
 
Buffer Size = 0 (Rendezvous):
────────────────────────────────────────────────────────────────────────
Producer blocks on EVERY send waiting for consumer.
Effective rate: 80 messages/sec (limited by consumer)
Producer is idle 20% of the time.
 
Buffer Size = 10:
────────────────────────────────────────────────────────────────────────
Initially: Buffer fills at 20 messages/sec (100 produced - 80 consumed)
After 0.5 seconds: Buffer full (10 messages)
Then: Producer blocks until consumer catches up.
Effective rate: Still 80 messages/sec average
But: Produces in bursts, handles temporary producer speed variations better.
 
Buffer Size = 1000:
────────────────────────────────────────────────────────────────────────  
Takes 50 seconds to fill the buffer
Allows producer to work ahead significantly
But: 50 seconds of work lost if consumer crashes before draining buffer
Memory usage: 1000 messages worth of RAM
 
Buffer Size = Unbounded (conceptual):
────────────────────────────────────────────────────────────────────────
Buffer grows indefinitely at 20 messages/sec
After 1 hour: 72,000 messages buffered (potentially gigabytes!)
Eventually: System runs out of memory
This is why "unbounded" is dangerous.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

#include <mqueue.h>
#include <fcntl.h>
#include <stdio.h>
 
int main() {
    // POSIX message queues have configurable max messages
    struct mq_attr attr;
    attr.mq_flags = 0;
    attr.mq_maxmsg = 10;      // Maximum 10 messages in queue
    attr.mq_msgsize = 1024;   // Maximum 1024 bytes per message
    attr.mq_curmsgs = 0;      // Current messages (read-only after creation)
    
    mqd_t mq = mq_open("/flow_controlled_queue",
                       O_CREAT | O_RDWR,
                       0666,
                       &attr);
    
    if (mq == (mqd_t)-1) {
        perror("mq_open");
        return 1;
    }
    
    // Now sending behavior:
    // - First 10 sends succeed immediately (buffer has space)
    // - 11th send BLOCKS until a message is received
    // - This provides automatic flow control!
    
    char msg[] = "test message";
    for (int i = 0; i < 20; i++) {
        printf("Sending message %d... ", i);
        // mq_send blocks when queue is full (unless O_NONBLOCK)
        if (mq_send(mq, msg, sizeof(msg), 0) == 0) {
            printf("sent!
");
        }
    }
    
    // With O_NONBLOCK flag:
    // - mq_send returns -1 with errno=EAGAIN when full
    // - Caller decides whether to retry, drop, or wait
    
    mq_close(mq);
    mq_unlink("/flow_controlled_queue");
    return 0;
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

// Problem: Byte streams don't preserve message boundaries
 
// Sender sends two messages:
write(pipe_fd, "Hello", 5);   // First message
write(pipe_fd, "World", 5);   // Second message
 
// Receiver might see:
// - "HelloWorld" in one read (messages merged)
// - "He" then "lloWor" then "ld" (split across reads)
// - "Hello" then "World" (lucky alignment, not guaranteed!)
 
// Solution: Message framing protocol
 
// Length-prefixed framing:
void send_message(int fd, const char *msg, size_t len) {
    uint32_t netLen = htonl(len);  // Network byte order
    write(fd, &netLen, sizeof(netLen));   // Send length first
    write(fd, msg, len);                   // Then message body
}
 
int recv_message(int fd, char *buf, size_t bufSize) {
    uint32_t netLen;
    if (read(fd, &netLen, sizeof(netLen)) != sizeof(netLen)) {
        return -1;  // Connection closed or error
    }
    size_t len = ntohl(netLen);
    if (len > bufSize) {
        return -1;  // Message too large for buffer
    }
    // Read exactly 'len' bytes (may need loop for partial reads)
    size_t total = 0;
    while (total < len) {
        ssize_t n = read(fd, buf + total, len - total);
        if (n <= 0) return -1;
        total += n;
    }
    return len;
}
 
// Alternative: Delimiter-based framing (like HTTP headers)
// Each message ends with "
" or "\r
"
// Simpler but fails if message contains the delimiter

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

#include <mqueue.h>
#include <fcntl.h>
#include <string.h>
#include <stdio.h>
 
int main() {
    struct mq_attr attr = {
        .mq_maxmsg = 10,      // Max 10 messages
        .mq_msgsize = 256     // Max 256 bytes per message
    };
    
    // Server: Create and receive from queue
    mqd_t mq = mq_open("/example_queue", O_CREAT | O_RDONLY, 0666, &attr);
    
    char buffer[256];
    unsigned int priority;
    
    // Receive a complete message (boundaries preserved automatically!)
    ssize_t bytes = mq_receive(mq, buffer, sizeof(buffer), &priority);
    
    if (bytes > 0) {
        printf("Received message: '%.*s' (priority: %u)
",
               (int)bytes, buffer, priority);
    }
    
    //Client: Open and send to queue
    mqd_t client_mq = mq_open("/example_queue", O_WRONLY);
    
    const char *msg1 = "First message";
    const char *msg2 = "Second message";
    
    // These are sent as SEPARATE messages
    mq_send(client_mq, msg1, strlen(msg1), 1);  // Priority 1
    mq_send(client_mq, msg2, strlen(msg2), 2);  // Priority 2
    
    // Receiver gets them as separate messages, in priority order!
    // Higher priority (2) comes first, then (1)
    
    mq_close(mq);
    mq_close(client_mq);
    mq_unlink("/example_queue");  // Cleanup
    
    return 0;
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69

// Unix Domain Socket Server (SOCK_STREAM example)
 
#include <sys/socket.h>
#include <sys/un.h>
#include <unistd.h>
#include <stdio.h>
#include <string.h>
 
#define SOCKET_PATH "/tmp/my_socket"
 
int main() {
    // Create socket
    int server_fd = socket(AF_UNIX, SOCK_STREAM, 0);
    if (server_fd == -1) {
        perror("socket");
        return 1;
    }
    
    // Bind to a filesystem path
    struct sockaddr_un addr;
    memset(&addr, 0, sizeof(addr));
    addr.sun_family = AF_UNIX;
    strncpy(addr.sun_path, SOCKET_PATH, sizeof(addr.sun_path) - 1);
    
    // Remove existing socket file if present
    unlink(SOCKET_PATH);
    
    if (bind(server_fd, (struct sockaddr*)&addr, sizeof(addr)) == -1) {
        perror("bind");
        return 1;
    }
    
    // Listen for connections
    if (listen(server_fd, 5) == -1) {
        perror("listen");
        return 1;
    }
    
    printf("Server listening on %s
", SOCKET_PATH);
    
    // Accept a connection
    int client_fd = accept(server_fd, NULL, NULL);
    if (client_fd == -1) {
        perror("accept");
        return 1;
    }
    
    printf("Client connected!
");
    
    // Exchange data
    char buffer[256];
    ssize_t n = read(client_fd, buffer, sizeof(buffer) - 1);
    if (n > 0) {
        buffer[n] = '\0';
        printf("Received: %s
", buffer);
        
        const char *response = "Hello from server!";
        write(client_fd, response, strlen(response));
    }
    
    close(client_fd);
    close(server_fd);
    unlink(SOCKET_PATH);
    
    return 0;
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

// Sending a file descriptor over Unix domain socket
 
#include <sys/socket.h>
#include <sys/un.h>
#include <fcntl.h>
 
void send_fd(int socket, int fd_to_send) {
    struct msghdr msg = {0};
    struct iovec iov[1];
    char buf[1] = {0};  // Must send at least 1 byte of data
    
    iov[0].iov_base = buf;
    iov[0].iov_len = 1;
    msg.msg_iov = iov;
    msg.msg_iovlen = 1;
    
    // Control message carries the file descriptor
    char cmsgbuf[CMSG_SPACE(sizeof(int))];
    msg.msg_control = cmsgbuf;
    msg.msg_controllen = sizeof(cmsgbuf);
    
    struct cmsghdr *cmsg = CMSG_FIRSTHDR(&msg);
    cmsg->cmsg_level = SOL_SOCKET;
    cmsg->cmsg_type = SCM_RIGHTS;      // "Sending file descriptor rights"
    cmsg->cmsg_len = CMSG_LEN(sizeof(int));
    *((int*)CMSG_DATA(cmsg)) = fd_to_send;
    
    sendmsg(socket, &msg, 0);
}
 
// Use case: A privileged daemon opens /etc/shadow (which only root can read)
// and passes the open fd to an unprivileged process for processing.
// The unprivileged process can read the file through the fd even though
// it couldn't have opened it itself!

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

// Getting peer credentials (SO_PEERCRED)
 
#include <sys/socket.h>
#include <stdio.h>
 
void check_peer(int socket) {
    struct ucred cred;
    socklen_t len = sizeof(cred);
    
    if (getsockopt(socket, SOL_SOCKET, SO_PEERCRED, &cred, &len) == 0) {
        printf("Peer process ID: %d
", cred.pid);
        printf("Peer user ID: %d
", cred.uid);
        printf("Peer group ID: %d
", cred.gid);
        
        // Security decision based on peer identity
        if (cred.uid != 0 && cred.uid != getuid()) {
            printf("Rejecting connection from unprivileged user
");
            close(socket);
            return;
        }
    }
}
 
// This is how D-Bus, systemd, and many security-sensitive daemons
// authenticate clients without passwords!