Writing Inter-Process Communication (IPC) Library in C (Part 1)

Writing Inter-Process Communication (IPC) Library in C

Welcome to a project-based tutorial on how to write an inter-process communication library in C. In this tutorial, I will start by giving an overview of how you can work with processes in C and some basic strategies to make multiple processes communicate with each other, then I will define the goal for the project and divide the whole process of implementing a custom IPC library into multiple parts. The reason why I’ll not cover everything in a single post is that I don’t want the readers (who might be relatively new to this stuff) to feel frustrated or overwhelmed by the length of the post. Because sometimes, people may easily get demotivated just by looking at how long a marathon is instead of looking at what it entails. That being said, let’s start looking at the main disadvantage of using a single process when you can use multiple.

Is using a single process slow? f**k it!

I didn’t actually mean what you might have thought. What I meant was to just use the fork function to create an extra (so-called child) process along with the process (so-called parent) that called it. These processes have their own separate stack, copies of the variables, virtual memory, and so on. This essentially means that what one process does with its memory blocks is unknown to the other process. Can you believe this? While you might think that such a parent-child relationship shouldn’t exist, there are good reasons for it to exist in computers, one of them being the freedom given to processes that perform independent and separate tasks. But why? Because those independent processes don’t have to share the same memory. What would happen if they do, though? Nothing extreme would happen… except more frustrated programmers cracking their heads at carefully managing memory between the processes to prevent one process from intervening in the other by erasing or overwriting the memory sections (supposedly by accident) that don’t belong to it. And maybe also security risks — when one process purposefully accesses a memory section used by another process to manipulate it. But maybe there is even a simpler answer, like, would you feel comfortable sharing your house with someone else who has nothing to do with you (even though you may call him your child for the sake of this tutorial)? Enough talking; let’s see how you can fork stuff up.

#include <stdio.h>
#include <unistd.h> // to use fork()
#include <sys/wait.h> // to use waitpid()

int main(){
    pid_t pid = fork();
    if(pid == -1){
        fprintf(stderr, "fork() failed\n");
        exit(1);
    }
    else if(!pid){
        printf("Hello from child process!\n");
    }
    else{
        printf("Hello from parent process!\n");
        waitpid(pid, 0, 0);
        printf("Child process exited\n");
    }
    return 0;
}

I would still recommend you take a look at the fork man page for more details. One of the bullet points mentioned on that page is the following:

The child process is created with a single thread-the one that called fork(). The entire virtual address space of the parent is replicated in the child, including the states of mutexes, condition variables, and other pthreads objects; the use of pthread_atfork() may be helpful for dealing with problems that this can cause.

Replication of the entire virtual address space of the parent in the child process essentially means that the pointers before the fork() would still seemingly point to the same address virtually after the fork() while being mapped to different physical memory addresses. We can observe this by executing a simple program:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/wait.h>

int main(){
    int *val = malloc(sizeof(int));
    *val = 6;

    pid_t pid = fork();
    if(pid == -1) exit(1);
    else if(!pid){ // child proc
        ++*val;
        printf("child incremented val by one stored at %p: %d\n", val, *val);
    }
    else{ // parent proc
        waitpid(pid, 0, 0);
        printf("parent says the val stored at %p is %d\n", val, *val);
    }

    free(val); // both processes will call free(), but no SIGSEGV will occur
    return 0;
}

What this output tells us is that programmers work with virtual addresses more often than not. You can suppose that these virtual addresses always begin from 0 and end at some big value LAST_ADDR. Each time a new process is run, a new virtual address space is given to the process to be used. However, to avoid mixing the physical addresses that different processes write to or read from, those virtual addresses are mapped to different physical regions of the memory by the OS. So, even though two processes may have a pointer to a (virtual) address 0xdeadbeef, this virtual address is mapped to real addresses 0x0000beef and 0x1111beef on the physical memory. That’s why calling free() twice doesn’t cause any issue; each process frees a different physical memory block that it has access to by using the same virtual address. (OS knows that for process A, free(0xdeadbeef) means deallocation of 0x0000beef, and for process B, free(0xdeadbeef) means deallocation of 0x1111beef on the physical memory.)

Another complicated father-son relationship…

The parent process doesn’t want to have anything to do with the child process, and the child process thinks that the feelings are reciprocal. Now what? The reason these processes don’t get along well is not because they inherently hate each other but because using fork separates their physical memory addresses so that one process doesn’t unintentionally or intentionally fork other’s stuff up. However, there is a way to overcome this. But wait, didn’t we want to have separate memory addresses in the first place? Why are we trying to ruin what fork has already given us? Well, yes, we wanted to have separate (private) memory addresses for security reasons and have less frustrated programmers. But I never said we didn’t want a shared (public) memory address along with the private one. I hope this answer didn’t make you feel stupid or anything like that. If it did, it’s also fine, though; you can overcome anything in life (maybe not really). Anyways, having public memory blocks between multiple processes is nice and very useful when those processes may want to tackle a hard problem altogether by communicating with each other. They may use their private memory for the stuff that only concerns each of them separately and then share their final results with the others or let them know what to do next. You can imagine all sorts of messages that can be passed between the processes if they just had a shared public memory section. Just to give you a very simple and high-level example, you can build a chatting application in C that uses a single process, which results in each person in the group chat waiting for the person who actively uses the process to read previous messages or write a new message, to finish and free the process so that it can be used by someone else in the same manner, or you can have multiple processes for each person to let them do their own thing separately on their own and make these processes to communicate once in a while to know the temporal ordering of the messages sent in the chat. What a nice feature to have for users, huh? Now, let’s see how you can start doing something like this in C by using a message queue.

#include <stdio.h>
#include <unistd.h>
#include <sys/wait.h>

typedef struct {
    long int type;
    char data[1024];
} msg_t;

void *proc1(int msg_id){
    msg_t msg = {.type = 1};
    strncpy(msg.data, "proc1 says hello\n", 1024);
    printf("sending: (%zu, %s)\n", msg.type, msg.data);
    msgsend(msg_id, &msg, sizeof(msg_t), 0);
    return 0;
}

void *proc2(int msg_id){
    msg_t msg = {0};
    msgrcv(msg_id, &msg, sizeof(msg_t), 1, 0);
    printf("received: (%zu, %s)\n", msg.type, msg.data);
    return 0;
}

int main(){
    key_t key = ftok("/etc/passwd/", 'A');
    if(key == -1) exit(1); // ftok failed

    int msg_id = msgget(key, 0666 | IPC_CREAT);
    if(msg_id == -1) exit(2); // msgget failed

    pid_t pid = fork();
    if(pid == -1) exit(3); // fork failed
    else if(!pid) proc1(msg_id); // child executes proc1 function
    else{ proc2(msg_id); waitpid(pid, 0, 0); } // parent executes proc2 function
    return 0;
}

One disadvantage of using a message queue is that it forces processes to use a fixed data structure required by the functions on the message queue to function properly. This data structure must have the following fields: long int to hold the message type and char[] to hold the message data. Yeah, it is true that C wizards can cast that char* to any godly customized struct as long as the allocated size is fixed, but wouldn’t it be nice for C peasants to pass their own custom structs between processes as normal? Also, some C folks may prefer to have built-in support to work with sender and receiver IDs when sending and receiving messages. In addition to that, a message queue is a one-way pipe, meaning that either process A sends data and process B reads it, or process B sends data and process A reads it. Both processes cannot read and write data simultaneously. To overcome this limitation, you might want to use a bidirectional pipe (as opposed to a unidirectional pipe), but this won’t make all the C peasants’ wishes come true.

The project

The main objective of this project is to build a C library that provides processes with a set of simple functionalities to send and receive messages. Another way of thinking about it is that the project is simply about creating a chatting app for processes on your computer so that they can socialize occasionally when you load them up with heavy tasks. Yes, it’s true — our poor processes are mostly introverted, just like we programmers are. That’s why we want to make them become what we have failed to become. Such a prideful moment for us! Let’s shed some tears on a prototype program that is also an example of how I would like other programmers to use this library.

#include "fastcpee.h" // [fast] [c]pu [p]arallel [e]xecution [e]ngine
#include <unistd.h> // to call fork()
#include <sys/wait.h> // to call waitpid()
#include <stdio.h>

#define FMB_ID 'A'          // FMB IDentifier character
#define SLOT_COUNT 3        // 3 mail slots per channel
#define CHANNEL_COUNT 1     // 1 (concurrent) channel

typedef struct {
    int i;
    char c;
} T;

void *proc1(void *arg){
    fmb_t fmb = (fmb_t)arg;
    T data = {.i=1, .c='A'};
    printf("sending: (%d, %c)\n", data.i, data.c);
    fmb_send(fmb, 0, 1, &data); // to be implemented
    return 0;
}

void *proc2(void *arg){
    fmb_t fmb = (fmb_t)arg;
    T data;
    fmb_recv(fmb, 1, 0, &data); // to be implemented
    printf("received: (%d, %c)\n", data.i, data.c);
    return 0;
}

int main(){
    fmb_t fmb; // [f]ast [m]ail[b]ox [_t]ype
    fmb = fmb_new(FMB_ID, sizeof(T), SLOT_COUNT, CHANNEL_COUNT); // to be implemented
    pid_t pid = fork();

    if(pid == -1) exit(1);
    else if(!pid) proc1(fmb); // this is what the child process will execute
    else{                     // this is what the parent will execute
        proc2(fmb);
        waitpid(pid, 0, 0);
        fmb_free(fmb);        // to be implemented
    }
    return 0;
}

Now, we kind of have an idea of what this library should be capable of doing. The code above was just an example to show you the kind of functionality I would like this library to provide to other programmers. It is not the complete set of features, though. That list is yet to come in the next post. Then we will have much clear idea of how we should start thinking about this project, what it does, and how to actually implement it in C.

A brief introduction to IPC $\leftarrow$ you are here
High-level specification of the project
Functional specification and design choices
What data structure(s) should be used?
Implementing 2-way asynchronous functions
Implementing (1-way) asynchronous functions
Implementing synchronous functions
Extra cherry on top
Testing the library
Conclusion

Writing Inter-Process Communication (IPC) Library in C (Part 1)