{"id":414110,"date":"2024-06-29T23:33:32","date_gmt":"2024-06-29T23:33:32","guid":{"rendered":"http:\/\/savepearlharbor.com\/?p=414110"},"modified":"-0001-11-30T00:00:00","modified_gmt":"-0001-11-29T21:00:00","slug":"","status":"publish","type":"post","link":"https:\/\/savepearlharbor.com\/?p=414110","title":{"rendered":"<span>Kernel Queue: The Complete Guide On The Most Essential Technology For High-Performance I\/O<\/span>"},"content":{"rendered":"<div><!--[--><!--]--><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-2\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<p>When talking about high-performance software we probably think of server software (such as nginx) which processes millions requests from thousands clients in parallel.  Surely, what makes server software work so fast is high-end CPU running with huge amount of memory and a very fast network link.  But even then, the software must utilize these hardware resources at maximum efficiency level, otherwise it will end up wasting the most of the valuable CPU power for unnecessary kernel-user context switching or while waiting for slow I\/O operations to complete.<\/p>\n<p>Thankfully, the Operating Systems have a solution to this problem, and it&#8217;s called <em>kernel event queue<\/em>.  Server software and OS kernel use this mechanism together to achieve minimum latency and maximum scalability (when serving a very large number of clients in parallel).  In this article we are going to talk about <strong>FreeBSD, macOS and kqueue<\/strong>, <strong>Linux and epoll<\/strong>, <strong>Windows and I\/O Completion Ports<\/strong>.  They all have their similarities and differences which we&#8217;re going to discuss here.  The goal of this article is for you to understand the whole mechanism behind kernel queues and to understand how to work with each API.<\/p>\n<p><em>I assume you are already familiar with socket programming and with asynchronous operations, but anyway, in case you think there&#8217;s something I should define or explain in more detail &#8212; send me a message, I&#8217;ll try to update the article.<\/em><\/p>\n<p><em>Although I tried to keep this article clean of any unnecessary sentences (it&#8217;s not a novel, after all), I sometimes can&#8217;t stop myself from expressing my thoughts about something I like or dislike.<\/em><\/p>\n<p>Contents:<\/p>\n<ul>\n<li>\n<p><a href=\"#what-is-kernel-queue\" rel=\"noopener noreferrer nofollow\">What is kernel queue?<\/a> <\/p>\n<ul>\n<li>\n<p><a href=\"#api-principles\" rel=\"noopener noreferrer nofollow\">API Principles<\/a><\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><a href=\"#freebsdmacos-and-kqueue\" rel=\"noopener noreferrer nofollow\">FreeBSD\/macOS and kqueue<\/a> <\/p>\n<ul>\n<li>\n<p><a href=\"#accepting-socket-connections-with-kqueue\" rel=\"noopener noreferrer nofollow\">Accepting socket connections with kqueue<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#creating-and-closing-kqueue-object\" rel=\"noopener noreferrer nofollow\">Creating and closing kqueue object<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#attaching-socket-descriptor-to-kqueue\" rel=\"noopener noreferrer nofollow\">Attaching socket descriptor to kqueue<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#receiving-events-from-kqueue\" rel=\"noopener noreferrer nofollow\">Receiving events from kqueue<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#processing-received-events-from-kqueue\" rel=\"noopener noreferrer nofollow\">Processing received events from kqueue<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#establishing-tcp-connection-with-kqueue\" rel=\"noopener noreferrer nofollow\">Establishing TCP connection with kqueue<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#processing-stale-cached-events\" rel=\"noopener noreferrer nofollow\">Processing stale cached events<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#user-triggered-events-with-kqueue\" rel=\"noopener noreferrer nofollow\">User-triggered events with kqueue<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#system-timer-events-with-kqueue\" rel=\"noopener noreferrer nofollow\">System timer events with kqueue<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#unix-signals-from-kqueue\" rel=\"noopener noreferrer nofollow\">UNIX signals from kqueue<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#asynchronous-file-io-with-kqueue\" rel=\"noopener noreferrer nofollow\">Asynchronous file I\/O with kqueue<\/a><\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><a href=\"#linux-and-epoll\" rel=\"noopener noreferrer nofollow\">Linux and epoll<\/a> <\/p>\n<ul>\n<li>\n<p><a href=\"#accepting-socket-connections-with-epoll\" rel=\"noopener noreferrer nofollow\">Accepting socket connections with epoll<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#creating-and-closing-epoll-object\" rel=\"noopener noreferrer nofollow\">Creating and closing epoll object<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#attaching-socket-descriptor-to-epoll\" rel=\"noopener noreferrer nofollow\">Attaching socket descriptor to epoll<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#receiving-events-from-epoll\" rel=\"noopener noreferrer nofollow\">Receiving events from epoll<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#processing-received-events-from-epoll\" rel=\"noopener noreferrer nofollow\">Processing received events from epoll<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#establishing-tcp-connection-with-epoll\" rel=\"noopener noreferrer nofollow\">Establishing TCP connection with epoll<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#user-triggered-events-with-epoll\" rel=\"noopener noreferrer nofollow\">User-triggered events with epoll<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#system-timer-events-with-epoll\" rel=\"noopener noreferrer nofollow\">System timer events with epoll<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#unix-signals-from-epoll\" rel=\"noopener noreferrer nofollow\">UNIX signals from epoll<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#asynchronous-file-io-with-epoll\" rel=\"noopener noreferrer nofollow\">Asynchronous file I\/O with epoll<\/a><\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><a href=\"#windows-and-i-o-completion-ports\" rel=\"noopener noreferrer nofollow\">Windows and I\/O Completion Ports<\/a> <\/p>\n<ul>\n<li>\n<p><a href=\"#accepting-connections-to-a-named-pipe-with-iocp\" rel=\"noopener noreferrer nofollow\">Accepting connections to a named pipe with IOCP<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#creating-and-closing-iocp-object\" rel=\"noopener noreferrer nofollow\">Creating and closing IOCP object<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#attaching-socket-descriptor-to-iocp\" rel=\"noopener noreferrer nofollow\">Attaching file descriptor to IOCP<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#receiving-events-from-iocp\" rel=\"noopener noreferrer nofollow\">Receiving events from IOCP<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#processing-received-events-from-iocp\" rel=\"noopener noreferrer nofollow\">Processing received events from IOCP<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#user-triggered-events-with-iocp\" rel=\"noopener noreferrer nofollow\">User-triggered events with IOCP<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#establishing-tcp-connection-with-iocp\" rel=\"noopener noreferrer nofollow\">Establishing TCP connection with IOCP<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#writing-data-to-a-tcp-socket-with-iocp\" rel=\"noopener noreferrer nofollow\">Writing data to a TCP socket with IOCP<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#reading-data-from-a-tcp-socket-with-iocp\" rel=\"noopener noreferrer nofollow\">Reading data from a TCP socket with IOCP<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#reading-and-writing-data-fromto-a-udp-socket-with-iocp\" rel=\"noopener noreferrer nofollow\">Reading and writing data from\/to a UDP socket with IOCP<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#io-cancellation-with-iocp\" rel=\"noopener noreferrer nofollow\">I\/O Cancellation with IOCP<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#accepting-socket-connections-with-iocp\" rel=\"noopener noreferrer nofollow\">Accepting socket connections with IOCP<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#system-timer-events-with-iocp\" rel=\"noopener noreferrer nofollow\">System timer events with IOCP<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#asynchronous-file-io-with-iocp\" rel=\"noopener noreferrer nofollow\">Asynchronous file I\/O with IOCP<\/a><\/p>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><a class=\"anchor\" name=\"what-is-kernel-queue\" id=\"what-is-kernel-queue\"><\/a><\/p>\n<h3>What is kernel queue?<\/h3>\n<p>Kernel event queue (which I&#8217;m gonna call KQ from now on) is a <strong>fast signal-delivery mechanism<\/strong> which allows server software to process events from OS in a very effective way.  KQ is a bunch of data living in kernel memory and a bunch of kernel code that operates with this data to notify a user-level application about various system events.  A user app can&#8217;t access KQ data directly (it&#8217;s managed by kernel) and so it operates with KQ via the API that OS provides.  There are 3 different API we&#8217;re going to use here: kqueue, epoll, IOCP.  However, this section describes kernel queues in general so the API doesn&#8217;t matter for now.<\/p>\n<p>Because the main purpose of KQ is to deliver notifications from network sockets, let me formulate the key idea in a different way:<\/p>\n<blockquote>\n<p>A user application wants to be notified when any of its sockets is ready to read or write some data, and the OS kernel serves this purpose by maintaining the list of all registered and signalled events.<\/p>\n<\/blockquote>\n<h4>Use-case N1<\/h4>\n<p>What an application achieves through KQ technology is that the app is notified about an I\/O signal  such as when a <strong>network packet is received<\/strong>.  For example:<\/p>\n<ol start=\"0\">\n<li>\n<p>Suppose some user app created a UDP socket and registered it with a KQ along with some app-defined data (i.e. cookie).<\/p>\n<\/li>\n<li>\n<p>At some point the last chunk of a UDP packet is received by network device.<\/p>\n<\/li>\n<li>\n<p>OS now has a complete UDP packet and is ready to notify the user process as soon as it calls the KQ waiting function.<\/p>\n<\/li>\n<li>\n<p>At some time the user app calls KQ waiting function which tells the kernel: <code>Give me something new<\/code>.<\/p>\n<\/li>\n<li>\n<p>OS responds with <code>Got a READ event from the socket associated with your cookie<\/code>.<\/p>\n<\/li>\n<li>\n<p>This cookie is the object pointer which the app then uses to handle the signal &#8212; <em>read a message from UDP socket<\/em>, in our case.<\/p>\n<\/li>\n<\/ol>\n<blockquote>\n<p>Note that neither the opening of a socket, neither reading from a socket after the signal is received isn&#8217;t normally the part of KQ mechanism.  On UNIX we always use conventional socket functions and we use KQ functions to receive events associated with sockets.  However, IOCP on Windows is different.  There, I\/O functions and their associated events are a part of a single mechanism.  Anyway, we&#8217;ll deal with IOCP later, so for now just don&#8217;t bother with it &#8212; let us always think by default that KQ just delivers signals.<\/p>\n<\/blockquote>\n<h4>Use-case N2<\/h4>\n<p>Consider the next example where the user app receives a <strong>notification after a TCP socket connects to its peer<\/strong>:<\/p>\n<ol start=\"0\">\n<li>\n<p>User app creates a TCP socket and registers it with a KQ along with some app-defined data (i.e. cookie).<\/p>\n<\/li>\n<li>\n<p>Now the app begins the procedure to connect to a remote host.  Obviously, this operation can&#8217;t finish immediately most of the time, because it takes some time to transmit 2 TCP packets needed for TCP connection.  Moreover, what if the network link is very busy and the packets get dropped?  Needless to say that TCP connection may take a long time to finish.  Because of that, OS returns the control back to the app with the result <code>Can't finish the operation immediately<\/code>.  While packets are being sent and received, the app keeps doing some other stuff, relying on OS to do its best to complete the connection procedure.<\/p>\n<\/li>\n<li>\n<p>Finally, a <code>SYN+ACK<\/code> TCP packet is received from the remote host, which means it&#8217;s willing to establish a TCP connection with our app.  Now OS is ready to signal the app as soon as the latter becomes ready.<\/p>\n<\/li>\n<li>\n<p>At some point the user app calls the KQ waiting function which tells the kernel: <code>Give me something new<\/code>.<\/p>\n<\/li>\n<li>\n<p>OS responds with <code>Got a WRITE event from the socket associated with your cookie<\/code>.<\/p>\n<\/li>\n<li>\n<p>This cookie is the object pointer which the app then uses to handle the signal &#8212; <em>write some data to the TCP socket<\/em>, in our case.<\/p>\n<\/li>\n<\/ol>\n<p>Although the primary use of KQ is I\/O event notifications, it also can be used for other purposes, for example KQ can notify when a child process signals its parent (i.e. <strong>UNIX signals delivery<\/strong>), or KQ can be used to receive <strong>notifications from a system timer<\/strong>.  I also explain these use-cases and show the example code in this article.<\/p>\n<h4>Internal representation example<\/h4>\n<p>Let&#8217;s see a diagram with an example of how KQ may look like internally after a user app has registered 6 different events there (user-triggered event, I\/O events, system timer), 3 of which have signalled already.<\/p>\n<pre><code>         KQ table example ================================= Event    | Descriptor | Signalled? ---------+------------+----------- USER     | #789       | READ     | #1         | READ     | #2         | yes WRITE    | #2         | yes WRITE    | #3         | TIMER    | #456       | yes <\/code><\/pre>\n<p>In this example, both READ and WRITE events for socket #2 are in signalled state which means we can read and write data from\/to this socket.  And the timer event is in signalled state too which means the system timer interval has expired.  The signalled flag also means that after a user app calls the function to receive events from KQ, it will receive an array of these 3 signalled events so it can process them.  The kernel then may clear the signalled flag so that it won&#8217;t deliver the same signals over and over again unless necessary.<\/p>\n<p>Of course in reality KQ is much more complex but we don&#8217;t need to know exactly how the KQ is implemented internally &#8212; we need just to understand what and when it delivers to us and how me may use it effectively.  I&#8217;m not a kernel developer so I don&#8217;t know much about how it&#8217;s implemented inside &#8212; you have to read some Linux\/FreeBSD kernel manuals and epoll\/kqueue code if you are interested in this subject.<\/p>\n<p><a class=\"anchor\" name=\"api-principles\" id=\"api-principles\"><\/a><\/p>\n<h4>API Principles<\/h4>\n<p>Now let&#8217;s talk about what features all those API provide us with.  In general, working with a KQ API consists of 4 steps:<\/p>\n<ol>\n<li>\n<p><strong>Create KQ object<\/strong>.  It&#8217;s the easiest part, where we just call a function which returns the descriptor for our new KQ.  We may create KQ objects as many as we want, but I don&#8217;t see the point of creating more than 1 per process or thread.<\/p>\n<\/li>\n<li>\n<p><strong>Attach file\/socket descriptor<\/strong> along with opaque user data to KQ.  We have to tell the OS that we want it to notify us about any particular descriptor through a particular KQ object.  How else the kernel should know what to notify us about?  Here we also associate some data with the descriptor, which is usually a pointer to some kind of a structure object.  How else are we going to handle the received signal?  The attachment is needed only once for each descriptor, usually it&#8217;s done right after the descriptor is configured and ready for I\/O operations (though we can delay that until absolutely necessary to probably save a context switch).  The detachment procedure usually is not needed (with the right design), so we won&#8217;t even talk about it here.<\/p>\n<\/li>\n<li>\n<p><strong>Wait for incoming events from KQ<\/strong>.  When the user app has nothing more important to do, it calls a KQ waiting function.  We specify the output array of events and timeout value as parameters when calling this function.  It fills our array with the information about which events signalled and how they signalled.  By using an array of events rather than a single event we save CPU time on somewhat costly kernel-userspace context switches.  By using timeout value we control how much time this KQ function can block internally.  If we specify a positive value, then the function will block for this amount of time in case it has no events to give us.  If we specify 0, it won&#8217;t block at all and return immediately.<\/p>\n<blockquote>\n<p>Some people use a small timeout value for KQ waiting functions so that they can check for some flags and variables and probably exit the waiting loop if some condition is met.  But when using a small timeout value, like 50ms, they waste a lot of context switches unnecessarily.  In this case OS periodically wakes up their process, even if it has nothing to do except calling the same KQ waiting function again in the next loop iteration.  If you use this technique, it&#8217;s most likely that there&#8217;s something you do wrong.  All normal software should use inifinite timeout, so the process wakes only when it is necessary.<\/p>\n<\/blockquote>\n<\/li>\n<li>\n<p><strong>Destroy KQ object<\/strong>.  When we don&#8217;t need a KQ object anymore, we close it so the OS can free all associated memory.  Obviously, after KQ object is closed, you won&#8217;t be able to receive any notifications for the file descriptors attached to it.<\/p>\n<\/li>\n<\/ol>\n<p>What I like the most about this whole KQ idea is that user code is very clear and straightforward.  I think the OS must deliver a nice, clear and convenient API for their users &#8212; the API which everybody understands how it works.  And the way I understand it, a canonical KQ mechanism shouldn&#8217;t do or require users to do anything else except registering an event inside KQ and delivering this event from KQ to the user once it signals.  There is one single promise to the user: <code>When an event you care about signals, I will notify you about it<\/code>.  It allows the user code to be very flexible and free to do whatever it wants.  Let&#8217;s see an example with pseudo code which proves my point.<\/p>\n<h4>Pseudo code example<\/h4>\n<pre><code class=\"cpp\">\/\/ Pseudo code for an asynchronous HTTP\/1 client func do_logic(kq) { conn := new conn.socket = socket(TCP, NONBLOCK) kq.attach(conn.socket, conn) \/\/ attach our socket along with the object pointer to KQ conn.connect_to_peer() }  func connect_to_peer(conn) { addr := \"1.2.3.4:80\" result := conn.socket.connect_async(addr) \/\/ initiate connection or get the result of the previously initiated connection procedure if result == EINPROGRESS { conn.write_handler = connect_to_peer return }  print(\"connected to %1\", addr) conn.write_data() }  func write_data(conn) { data[] := \"GET \/ HTTP\/1.1\\r\\nHost: hostname\\r\\n\\r\\n\" result := conn.socket.send(data) if result == EAGAIN { conn.write_handler = write_data return }  print(\"written %1 bytes to socket\", result) conn.read_data() }  func read_data(conn) { data[], result := conn.socket.receive() if result == EAGAIN { conn.read_handler = read_data return }  print(\"received %1 bytes from socket: %2\", result, data) }  func worker(kq) { for { events[] := kq_wait(kq)  for ev := events {  conn := ev.user_data if ev.event == READ { conn.read_handler() } if ev.event == WRITE { conn.write_handler() } } } } <\/code><\/pre>\n<p>Here we have 3 operations: connect, socket write, socket read.  All 3 may block with a normal socket descriptor, so we set a non-blocking flag when creating the socket.  Then we attach our socket to KQ along with the pointer to our object <code>conn<\/code>.  Now we are ready to use the socket as we want, in our example we have a client socket which needs to be connected to a server.<\/p>\n<p>We begin a socket connection procedure which may or may not complete immediately.  In case it completes immediately &#8212; we continue with our program logic as usual, but in case it can&#8217;t complete immediately it just returns <code>EINPROGRESS<\/code> error code.  If it does so, we set <code>write_handler<\/code> function pointer to the name of our function we want to be called when connection is established.  And then we just return from our function, because there&#8217;s nothing else for us to do &#8212; we must wait.  At this point our application is free to do whatever it wants &#8212; process something else or just wait until some events are received from the kernel.  Which is why we have a <code>worker()<\/code> function.  It receives events from KQ and processes them one by one, calling the appropriate handler function.  In our case, <code>connect_to_peer()<\/code> function will be called after the TCP socket connection is established (or failed).  Now we&#8217;re inside this function the second time and we now get the result of our previous connect request.  It may be a failure, but I don&#8217;t check it here for the simplicity of our example.<\/p>\n<p>In case the connection was successul, we continue by calling <code>write_data<\/code> function which sends an HTTP request to a server.  Again, it may or may not complete immediately, so in case it returns with <code>EAGAIN<\/code> error, we set <code>write_handler<\/code> and return.  As simple as that.  After some time we are back inside our function again and we try to send the data once more.  We may go back and forth with this logic until we have sent the complete data for our HTTP request.  Once it&#8217;s sent we start reading the HTTP response from server.<\/p>\n<p>We are inside the <code>read_data()<\/code> function which starts reading data from socket.  It may or may not complete immediately.  If it returns with <code>EAGAIN<\/code>, we set <code>read_handler<\/code> and return.  Why do we use a different name for function pointer depending on whether it&#8217;s READ or WRITE event?  For our example it doesn&#8217;t matter, but in real life when we use full-duplex sockets, <em>both<\/em> events may fire at once, therefore we must be prepared to handle both READ and WRITE events in parallel.<\/p>\n<p>Isn&#8217;t it simple?  The only thing we need inside our program logic is to check for return values and error codes and set handling function pointers, then after some time we&#8217;re back <strong>in the same function<\/strong> to try once more <strong>with the same code<\/strong>.  I love this approach &#8212; it makes everything seem very clear, even though we&#8217;ve just written the program logic that can easily handle thousands of connections in parallel.<\/p>\n<p><a class=\"anchor\" name=\"freebsdmacos-and-kqueue\" id=\"freebsdmacos-and-kqueue\"><\/a><\/p>\n<h3>FreeBSD\/macOS and kqueue<\/h3>\n<p>Now I think we&#8217;re ready for some real code with a real API.  kqueue API is the one I like the most for some reason, so let&#8217;s start with it.  <strong>I strongly advise you to read this section and try to completely understand it, even if you won&#8217;t use FreeBSD in your work.<\/strong>  There&#8217;s just one syscall <code>kevent()<\/code> which we use with several different flags to control its behaviour.<\/p>\n<p><a class=\"anchor\" name=\"accepting-socket-connections-with-kqueue\" id=\"accepting-socket-connections-with-kqueue\"><\/a><\/p>\n<h4>Accepting socket connections with kqueue<\/h4>\n<p>Let&#8217;s see an easy example of a server which accepts a new connection by an event from kqueue.<\/p>\n<pre><code class=\"cpp\">\/* Kernel Queue The Complete Guide: kqueue-accept.c: Accept socket connection Usage: $ .\/kqueue-accept $ curl 127.0.0.1:64000\/ *\/ #include &lt;assert.h> #include &lt;stdio.h> #include &lt;unistd.h> #include &lt;netinet\/in.h> #include &lt;sys\/types.h> #include &lt;sys\/socket.h> #include &lt;sys\/event.h> #include &lt;sys\/ioctl.h>  int kq;  \/\/ the structure associated with a socket descriptor struct context { int sk; void (*rhandler)(struct context *obj); };  void accept_handler(struct context *obj) { printf(\"Received socket READ event via kqueue\\n\");  int csock = accept(obj->sk, NULL, 0); assert(csock != -1); close(csock); }  void main() { \/\/ create kqueue object kq = kqueue(); assert(kq != -1);  struct context obj = {}; obj.rhandler = accept_handler;  \/\/ create and prepare a socket obj.sk = socket(AF_INET, SOCK_STREAM | SOCK_NONBLOCK, 0); assert(obj.sk != -1); int val = 1; setsockopt(obj.sk, SOL_SOCKET, SO_REUSEADDR, &amp;val, 4);  struct sockaddr_in addr = {}; addr.sin_family = AF_INET; addr.sin_port = ntohs(64000); assert(0 == bind(obj.sk, (struct sockaddr*)&amp;addr, sizeof(addr))); assert(0 == listen(obj.sk, 0));  \/\/ attach socket to kqueue struct kevent events[2]; EV_SET(&amp;events[0], obj.sk, EVFILT_READ, EV_ADD | EV_CLEAR, 0, 0, &amp;obj); EV_SET(&amp;events[1], obj.sk, EVFILT_WRITE, EV_ADD | EV_CLEAR, 0, 0, &amp;obj); assert(0 == kevent(kq, events, 2, NULL, 0, NULL));  \/\/ wait for incoming events from kqueue struct timespec *timeout = NULL; \/\/ wait indefinitely int n = kevent(kq, NULL, 0, events, 1, timeout); assert(n > 0);  \/\/ process the received event struct context *o = events[0].udata; if (events[0].filter == EVFILT_READ) o->rhandler(o); \/\/ handle read event  close(obj.sk); close(kq); } <\/code><\/pre>\n<p>This code creates a TCP socket, attaches it to the newly created KQ, and waits for incoming connections.  After a client is connected, it prints a message to stdout and sends an HTTP response.  Inline comments explain in short form what each block is for.  Now we&#8217;re going to describe all this in detail.<\/p>\n<p><a class=\"anchor\" name=\"creating-and-closing-kqueue-object\" id=\"creating-and-closing-kqueue-object\"><\/a><\/p>\n<h4>Creating and closing kqueue object<\/h4>\n<p>To create a new KQ object, we call <code>kqueue()<\/code> function which returns the descriptor or <code>-1<\/code> on error.  KQ object is usually stored in the global context (in our case &#8212; it&#8217;s just a global variable) because we need it all the time while our app is running.  We close KQ object with <code>close()<\/code>.<\/p>\n<pre><code class=\"cpp\">kq = kqueue(); ... close(kq); <\/code><\/pre>\n<p><a class=\"anchor\" name=\"attaching-socket-descriptor-to-kqueue\" id=\"attaching-socket-descriptor-to-kqueue\"><\/a><\/p>\n<h4>Attaching socket descriptor to kqueue<\/h4>\n<p>So how do we attach a file descriptor to our kqueue object?  First, we prepare an object where we define:<\/p>\n<ul>\n<li>\n<p>Which file descriptor we want to associate with KQ.  This value can be a socket, UNIX signal, or an arbitrary user-defined ID, depending on the type of event we want to register.<\/p>\n<\/li>\n<li>\n<p>Which event we are interested in.  For I\/O events this value must be either <code>EVFILT_READ<\/code> or <code>EVFILT_WRITE<\/code>.  For UNIX signals it&#8217;s <code>EVFILT_SIGNAL<\/code>, for timers it&#8217;s <code>EVFILT_TIMER<\/code>, for user events it&#8217;s <code>EVFILT_USER<\/code>, but they will be explained later in separate sections.<\/p>\n<\/li>\n<li>\n<p>What we want <code>kevent()<\/code> to do: <code>EV_ADD<\/code> attaches descriptor to KQ.  <code>EV_CLEAR<\/code> flag prevents the event from unnecessary signalling (it&#8217;s explained later).<\/p>\n<\/li>\n<li>\n<p>What object pointer we associate with our file descriptor.  Normally, this object contains at least 2 fields: file descriptor itself and the function pointer which handles the event.<\/p>\n<\/li>\n<\/ul>\n<p>To set the above parameters we use <code>EV_SET()<\/code> macro for convenience but you may also use the <code>struct kevent<\/code> fields directly.<\/p>\n<p>Then, we call <code>kevent()<\/code> function which processes all our events we supplied to it and returns <code>0<\/code> on success.<\/p>\n<pre><code class=\"cpp\">struct kevent events[2]; EV_SET(&amp;events[0], sk, EVFILT_READ, EV_ADD | EV_CLEAR, 0, 0, obj); EV_SET(&amp;events[1], sk, EVFILT_WRITE, EV_ADD | EV_CLEAR, 0, 0, obj); kevent(kq, events, 2, NULL, 0, NULL); <\/code><\/pre>\n<p><a class=\"anchor\" name=\"receiving-events-from-kqueue\" id=\"receiving-events-from-kqueue\"><\/a><\/p>\n<h4>Receiving events from kqueue<\/h4>\n<p>To receive events from KQ we must have a place to store them so the kernel can prepare the data for us &#8212; we need an array for signalled events, array of <code>struct kevent<\/code> objects.  Note that it doesn&#8217;t necessarily mean that events are stored in kernel memory in the same form &#8212; it&#8217;s just what the kernel prepares for us.  Normally, the events are stored in the chronological order, meaning that the event with index 0 has signalled before the event with a larger index, but this isn&#8217;t important for us, because user software is prepared to handle the events in any order anyway.  In our example we use an array for the maximum of 1 events.  Only very busy software can benefit from using a large array here to minimize the number of context switching.<\/p>\n<p>We call <code>kevent()<\/code> function and pass the array to it along with a timeout value defining how long it can block in case there are no signalled events.  The function returns the number of signalled events, or <code>0<\/code> if timeout value has passed before anything signalled, or <code>-1<\/code> on error.  Normally, we call KQ waiting functions in a loop like so:<\/p>\n<pre><code class=\"cpp\">while (!quit) { struct timespec *timeout = NULL; \/\/ wait indefinitely struct kevent events[1]; int n = kevent(kq, NULL, 0, events, 1, &amp;timeout); if (n &lt; 0 &amp;&amp; errno == EINTR) continue; \/\/ kevent() interrupts when UNIX signal is received } <\/code><\/pre>\n<p>As you can see we also check the return value for error <code>EINTR<\/code> which means that while <code>kevent()<\/code> was waiting for events a UNIX signal has been received and processed by a UNIX signal handling function.  This behaviour allows us to easily handle some important global flags that signal handlers may set.  For example we may handle <code>SIGINT<\/code> signal which is sent after the user presses <code>Ctrl+C<\/code> within the terminal window.  Then <code>SIGINT<\/code> signal handling function may set some kind of <code>quit<\/code> flag to indicate we should exit our app.  <code>kevent()<\/code> then returns with <code>EINTR<\/code>, and we check for <code>quit<\/code> value and exit the loop in this case.<\/p>\n<p><a class=\"anchor\" name=\"processing-received-events-from-kqueue\" id=\"processing-received-events-from-kqueue\"><\/a><\/p>\n<h4>Processing received events from kqueue<\/h4>\n<p>To process an event which we have received from KQ previously we have to know what to do with it.  But all events look pretty much the same to us at this point.  That&#8217;s why we used an object pointer with <code>EV_SET()<\/code> to associate it with each event.  Now we can simply call an event handling function.  We get this pointer by accessing <code>struct kevent.udata<\/code> field.  For full-duplex I\/O we need either 2 different handling functions or a single handler which will check itself which filter has signalled.  Since all KQ mechanisms have their different ways, I recommend you to go with 2-handlers approach and choose which handler to execute here, at the lowest level, to simplify the higher level code.<\/p>\n<pre><code class=\"cpp\">struct context *o = events[i].udata; if (events[i].filter == EVFILT_READ) o->rhandler(o); \/\/ handle read event else if (events[i].filter == EVFILT_WRITE) o->whandler(o); \/\/ handle write event <\/code><\/pre>\n<p>Do you remember the <code>EV_CLEAR<\/code> flag we supplied to KQ when we attached socket to it?  Here&#8217;s why we need to use it.  For example, after KQ returns a READ event to us, it won&#8217;t signal again until we drain all the data from this socket, i.e. until <code>recv()<\/code> returns with <code>EAGAIN<\/code> error.  This mechanism prevents from signalling the same event over and over again each time we call KQ waiting function, thus improving overall performance.  The software that can&#8217;t deal with <code>EV_CLEAR<\/code> behaviour most probably has a design flaw.<\/p>\n<p><a class=\"anchor\" name=\"establishing-tcp-connection-with-kqueue\" id=\"establishing-tcp-connection-with-kqueue\"><\/a><\/p>\n<h4>Establishing TCP connection with kqueue<\/h4>\n<p>Now let&#8217;s see how to correctly use <code>connect()<\/code> on a TCP socket with kqueue.  What makes this use-case special is that there is no function that could return the result of previous <code>connect()<\/code> operation.  Instead, we must use <code>struct kevent<\/code> to get the error code.  Here&#8217;s an example.<\/p>\n<pre><code class=\"cpp\">int r = connect(sk, ...); if (r == 0) { ... \/\/ connection completed successfully  } else if (errno == EINPROGRESS) { \/\/ connection is in progress struct kevent events[1]; int n = kevent(kq, NULL, 0, events, 1, &amp;timeout);  if (events[0].filter == EVFILT_WRITE) { errno = 0; if (events[i].flags &amp; EV_EOF) errno = events[0].fflags; ... \/\/ handle TCP connection result depending on `errno` value }  } else { ... \/\/ fatal error } <\/code><\/pre>\n<p>Suppose that we created a non-blocking TCP socket, attached it to KQ, and now we begin the connection procedure.  If it completes successfully right away, then we can read or write data to it immediately, and it isn&#8217;t what we are talking about here.  But if it returns <code>-1<\/code> with <code>EINPROGRESS<\/code> error, we should wait until OS notifies about with the result of the procedure.  And here&#8217;s the main thing: when <code>EVFILT_WRITE<\/code> event is received we test <code>struct kevent.flags<\/code> field for <code>EV_EOF<\/code> and if it&#8217;s set, then it means that <code>connect()<\/code> has failed.  In this case <code>struct kevent.fflags<\/code> field contains the error number &#8212; the same error that a blocking <code>connect()<\/code> call would set.<\/p>\n<blockquote>\n<p>I have to say that I don&#8217;t like this whole logic with getting an error code from <code>struct kevent<\/code> because it forces me to multiply branches in my code.  Another reason behind that is because it&#8217;s kqueue-specific stuff &#8212; for example on Linux we have to call <code>getsockopt(..., SOL_SOCKET, SO_ERROR, ...)<\/code> to get error code.  But on the other hand, on FreeBSD we don&#8217;t need to perform another syscall which probably outweighs both of my points above, so in the end I think it&#8217;s alright.<\/p>\n<\/blockquote>\n<p>Let&#8217;s see the complete example of a simple HTTP\/1 client that connects, sends request and receives response &#8212; all via KQ.  But of course you may notice that our code for handling WRITE event from KQ is useless here, because our request is very small and should always fit into the empty socket buffer, i.e. <code>send()<\/code> will always complete immediately.  But I think it&#8217;s OK for a sample program &#8212; the goal is to show you the general principle.<\/p>\n<pre><code class=\"cpp\">\/* Kernel Queue The Complete Guide: kqueue-connect.c: HTTP\/1 client Usage: $ nc -l 127.0.0.1 64000 $ .\/kqueue-connect *\/ #include &lt;assert.h> #include &lt;errno.h> #include &lt;stdio.h> #include &lt;unistd.h> #include &lt;netinet\/in.h> #include &lt;netinet\/tcp.h> #include &lt;sys\/types.h> #include &lt;sys\/socket.h> #include &lt;sys\/event.h>  int kq; int quit;  \/\/ the structure associated with a socket descriptor struct context { int sk; void (*rhandler)(struct context *obj); void (*whandler)(struct context *obj); int data_offset; };  void obj_write(struct context *obj); void obj_read(struct context *obj);  void obj_prepare(struct context *obj) { \/\/ create and prepare socket obj->sk = socket(AF_INET, SOCK_STREAM | SOCK_NONBLOCK, 0); assert(obj->sk != -1);  int val = 1; assert(0 == setsockopt(obj->sk, 0, TCP_NODELAY, (char*)&amp;val, sizeof(int)));  \/\/ attach socket to KQ struct kevent events[2]; EV_SET(&amp;events[0], obj->sk, EVFILT_READ, EV_ADD | EV_CLEAR, 0, 0, obj); EV_SET(&amp;events[1], obj->sk, EVFILT_WRITE, EV_ADD | EV_CLEAR, 0, 0, obj); assert(0 == kevent(kq, events, 2, NULL, 0, NULL)); }  void obj_connect(struct context *obj) { if (obj->whandler == NULL) { \/\/ begin asynchronous connection struct sockaddr_in addr = {}; addr.sin_family = AF_INET; addr.sin_port = ntohs(64000); char ip4[] = {127,0,0,1}; *(int*)&amp;addr.sin_addr = *(int*)ip4;  int r = connect(obj->sk, (struct sockaddr*)&amp;addr, sizeof(struct sockaddr_in)); if (r == 0) { \/\/ connection completed successfully } else if (errno == EINPROGRESS) { \/\/ connection is in progress obj->whandler = obj_connect; return; } else { assert(0); \/\/ fatal error }  } else { assert(errno == 0); \/\/ connection is successful obj->whandler = NULL; \/\/ we don't want any more signals from KQ }  printf(\"Connected\\n\"); obj_write(obj); }  void obj_write(struct context *obj) { const char data[] = \"GET \/ HTTP\/1.1\\r\\nHost: hostname\\r\\nConnection: close\\r\\n\\r\\n\"; int r = send(obj->sk, data + obj->data_offset, sizeof(data)-1 - obj->data_offset, 0); if (r > 0) { \/\/ sent some data obj->data_offset += r; if (obj->data_offset != sizeof(data)-1) { \/\/ we need to send the complete request obj_write(obj); return; } obj->whandler = NULL;  } else if (r &lt; 0 &amp;&amp; errno == EAGAIN) { \/\/ the socket's write buffer is full obj->whandler = obj_write; return; } else { assert(0); \/\/ fatal error }  printf(\"Sent HTTP request.  Receiving HTTP response...\\n\"); obj_read(obj); }  void obj_read(struct context *obj) { char data[64*1024]; int r = recv(obj->sk, data, sizeof(data), 0); if (r > 0) { \/\/ received some data printf(\"%.*s\", r, data); obj_read(obj); return;  } else if (r == 0) { \/\/ server has finished sending data  } else if (r &lt; 0 &amp;&amp; errno == EAGAIN) { \/\/ the socket's read buffer is empty obj->rhandler = obj_read; return; } else { assert(0); \/\/ fatal error }  quit = 1; }  void main() { \/\/ create KQ object kq = kqueue(); assert(kq != -1);  struct context obj = {}; obj_prepare(&amp;obj); obj_connect(&amp;obj);  \/\/ wait for incoming events from KQ and process them while (!quit) { struct kevent events[1]; struct timespec *timeout = NULL; \/\/ wait indefinitely int n = kevent(kq, NULL, 0, events, 1, timeout); if (n &lt; 0 &amp;&amp; errno == EINTR) continue; \/\/ kevent() interrupts when UNIX signal is received assert(n > 0);  \/\/ now process each signalled event for (int i = 0;  i != n;  i++) { struct context *o = events[i].udata;  errno = 0; if (events[i].flags &amp; EV_EOF) errno = events[i].fflags;  if (events[i].filter == EVFILT_READ &amp;&amp; o->rhandler != NULL) o->rhandler(o); \/\/ handle read event  if (events[i].filter == EVFILT_WRITE &amp;&amp; o->whandler != NULL) o->whandler(o); \/\/ handle write event } }  close(obj.sk); close(kq); } <\/code><\/pre>\n<p><em>Note that macOS doesn&#8217;t support <\/em><code>SOCK_NONBLOCK<\/code> flag in <code>socket()<\/code> &#8212; you should set the socket as nonblocking manually.<\/p>\n<p><a class=\"anchor\" name=\"processing-stale-cached-events\" id=\"processing-stale-cached-events\"><\/a><\/p>\n<h4>Processing stale cached events<\/h4>\n<p>One of the most interesting aspects of programming with KQ is handling the events in which we in fact are not interested anymore.  Here&#8217;s what may happen when we use KQ <em>carelessly<\/em>:<\/p>\n<ul>\n<li>\n<p>We attach a socket to KQ for both READ and WRITE events.<\/p>\n<\/li>\n<li>\n<p>We keep performing normal operations on a socket, reading and writing to it occasionally.<\/p>\n<\/li>\n<li>\n<p>At some point both READ and WRITE events get signalled.<\/p>\n<\/li>\n<li>\n<p>We use an array of events for KQ waiting function and it <strong>returns 2 events to us for the same socket<\/strong>.<\/p>\n<\/li>\n<li>\n<p>We start handling the first event which happens to be a READ event.<\/p>\n<\/li>\n<li>\n<p>We call READ event handling function, it processes this event and comes to a conclusion that the client object should be closed, because it has sent some invalid data.  We close the socket and destroy the object.  And everything seems to be correct, because after the socket is closed KQ won&#8217;t signal us with it anymore.  But remember that we have called a KQ waiting function some time ago and it has already returned 2 events to us.  We&#8217;ve handled the first event just now, but the second event is still in our array of event objects and is yet to be processed in the next iteration of the loop.<\/p>\n<\/li>\n<li>\n<p>We start handling the second event which is WRITE event in our case.  We take our object data associated with the event and we try to call the handling function.  BAM!  We hit the memory region we have just destroyed while handling the READ event.<\/p>\n<\/li>\n<\/ul>\n<p>This can happen at any time while our app is running and we don&#8217;t know anything in advance &#8212; we need to correctly determine such cases and handle them as they occur.  Note that this situation isn&#8217;t just limited to full-duplex sockets, but to any KQ event in general.  Suppose a timer signal has fired and we decided to close a client connection, but its socket has signalled already, and there&#8217;s an associated event already in our cache &#8212; we just don&#8217;t know it yet, because the timer signal has occurred just before it.  So unless we always limit the number of received events from KQ to 1, we must always be ready to handle this situation &#8212; we can&#8217;t prevent it from happening.  And of course, there&#8217;s the same problem on Linux with epoll, so it&#8217;s not just kqueue-only stuff.<\/p>\n<p>So how are we going to solve this problem?  Thankfully, it&#8217;s already solved by Igor Sysoev (the great man who initially wrote nginx) long time ago.  Here&#8217;s the trick.<\/p>\n<pre><code class=\"cpp\">struct context { int sk; void (*handler)(struct context *obj); int flag; };  void handler_func(struct context *obj) { if (...) { goto finish; \/\/ an error occurred } ...  finish: close(obj->sk); obj->flag = !obj->flag; \/\/ turn over the safety flag }  void main() { ... struct context *obj = ...; obj->flag = 0; struct kevent events[2]; void *ptr = (void*)((size_t)obj | obj->flag); \/\/ pass additional flag along with the object pointer to KQ EV_SET(&amp;events[0], obj->sk, EVFILT_READ, EV_ADD | EV_CLEAR, 0, 0, ptr); EV_SET(&amp;events[1], obj->sk, EVFILT_WRITE, EV_ADD | EV_CLEAR, 0, 0, ptr); kevent(kq, events, 2, NULL, 0, NULL);  ...  int n = kevent(kq, NULL, 0, events, 8, &amp;timeout); for (...) { void *ptr = events[i].udata; struct context *obj = (void*)((size_t)ptr &amp; ~1); \/\/ clear the lowest bit  int flag = (size_t)ptr &amp; 1; \/\/ check the lowest bit if (flag != obj->flag) continue; \/\/ don't process this event  obj->handler(obj); } } <\/code><\/pre>\n<p>Explanation:<\/p>\n<ul>\n<li>\n<p>When we attach a socket to KQ, we also associate our object pointer with it.  But we can actually <strong>store some more information<\/strong> there &#8212; not just the pointer.  In our example here, we set the value of <code>struct context.flag<\/code> field (which is <code>0<\/code> at first) as the lowest bit value along with our object pointer.  It works because all structure objects that contain a pointer are aligned to at least 4 or 8 bytes by default.  In other words, the lowest bit for any object pointer is always 0, and we can use it for our own purposes.<\/p>\n<\/li>\n<li>\n<p>After we have received an event from KQ, we <strong>clear the lowest bit<\/strong> when converting the user data pointer to our object pointer.  All event handlers are called as usual without any problem.<\/p>\n<\/li>\n<\/ul>\n<pre><code class=\"cpp\">kevent(EV_ADD, 0x???????0)  events[] = kevent() object = events[0].udata.bits[1..31]  \/\/ object   | events[0] \/\/ ---------+------------------- \/\/ {flag=0} | {udata=0x???????0}  if 0 == 0 \/\/ TRUE   object.handler() <\/code><\/pre>\n<ul>\n<li>\n<p>But if inside the event handling function we decided to close the socket, we <strong>mark our object as unused<\/strong> &#8212; in our case we set an internally stored safety flag to <code>1<\/code>.  We don&#8217;t free the memory associated with our object so that we can access this value later.<\/p>\n<\/li>\n<\/ul>\n<pre><code class=\"cpp\">read_handler() { close(object.socket) object.flag = 1 } <\/code><\/pre>\n<ul>\n<li>\n<p>When we start the processing of the next (cached) event for the same socket, we <strong>compare the lowest bit<\/strong> from the associated data pointer with the flag stored within our object.  In our case they don&#8217;t match, which means that we don&#8217;t want this event to be processed.  Note that C doesn&#8217;t allow logical bit operations on pointers, hence the somewhat ugly cast to integer type and back.<\/p>\n<\/li>\n<\/ul>\n<pre><code class=\"cpp\">object = events[1].udata.bits[1..31]  \/\/ object   | events[1] \/\/ ---------+------------------- \/\/ {flag=1} | {udata=0x???????0}  if 1 == 0 \/\/ FALSE <\/code><\/pre>\n<ul>\n<li>\n<p>After we have processed all cached events we may free the memory allocated for our objects marked as unused.  Or we may decide not to free the memory for our objects at all &#8212; because the next iteration may need to create a new object and we would need to allocate memory again.  Instead, we may <strong>store all unused objects in a list<\/strong> (or array) and free them only when our app is closing the whole KQ subsystem.<\/p>\n<\/li>\n<li>\n<p>Next time we use the same object pointer (if we didn&#8217;t free its memory), the flag is still set to <code>1<\/code> and so it is passed to KQ as the lowest bit of the user data pointer when we attach a new socket descriptor to KQ.<\/p>\n<\/li>\n<\/ul>\n<pre><code class=\"cpp\">kevent(EV_ADD, 0x???????1)  events[] = kevent() object = events[0].udata.bits[1..31]  \/\/ object   | events[0] \/\/ ---------+------------------- \/\/ {flag=1} | {udata=0x???????1}  if 1 == 1 \/\/ TRUE   object.handler() <\/code><\/pre>\n<ul>\n<li>\n<p>And again, when we have finished working with this object, we turn it over and set the flag to <code>0<\/code>.  After that, the values from KQ (the old bit value <code>1<\/code>) and the flag value inside the object (now <code>0<\/code>) don&#8217;t match, therefore the handling function won&#8217;t be called which is exactly what we want.<\/p>\n<\/li>\n<\/ul>\n<pre><code class=\"cpp\">read_handler() { close(object.socket) object.flag = 0 }  object = events[1].udata.bits[1..31]  \/\/ object   | events[1] \/\/ ---------+------------------- \/\/ {flag=0} | {udata=0x???????1}  if 0 == 1 \/\/ FALSE <\/code><\/pre>\n<p>Some people don&#8217;t use the lowest bit approach to handle the problem with stale events.  They just use a list where they put the unused objects until they can free them (after each iteration or by a timer signal).  Imagine how this can slow down the processing when <strong>every cached event starting at index 1 should be checked against some data in a container<\/strong> &#8212; a search must be performed which wastes CPU cycles.  I don&#8217;t know what&#8217;s the reasoning for doing so, but I really don&#8217;t see how it&#8217;s too hard to use the lowest bit trick and turn it over once in a while.  So I advise using a list of unused objects only to save on countless memory allocations and deallocations and not for deciding whether an event is stale or not.<\/p>\n<p><a class=\"anchor\" name=\"user-triggered-events-with-kqueue\" id=\"user-triggered-events-with-kqueue\"><\/a><\/p>\n<h4>User-triggered events with kqueue<\/h4>\n<p>When we use an infinite timeout in KQ waiting function, it blocks forever until it can return an event.  Similar to how it returns with <code>EINTR<\/code> after UNIX signal is processed, we can force it to return at any time by sending a user event to KQ.  To do it we first register a <code>EVFILT_USER<\/code> event in KQ, then we can trigger this event via <code>NOTE_TRIGGER<\/code>.  Here&#8217;s an example.<\/p>\n<pre><code class=\"cpp\">\/* Kernel Queue The Complete Guide: kqueue-user.c: User-triggered events *\/ #include &lt;assert.h> #include &lt;stdio.h> #include &lt;unistd.h> #include &lt;sys\/types.h> #include &lt;sys\/event.h>  int kq;  struct context { void (*handler)(struct context *obj); };  struct context user_event_obj;  void user_event_obj_handler(struct context *obj) { printf(\"Received user event via kqueue\\n\"); }  \/\/ application calls this function whenever it wants to add a new event to KQ \/\/ which will execute user_event_obj_handler() void trigger_user_event() { user_event_obj.handler = user_event_obj_handler;  struct kevent events[1]; EV_SET(&amp;events[0], 1234, EVFILT_USER, 0, NOTE_TRIGGER, 0, &amp;user_event_obj); assert(0 == kevent(kq, events, 1, NULL, 0, NULL)); }  void main() { \/\/ create kqueue object kq = kqueue(); assert(kq != -1);  \/\/ register user event with any random ID \/\/ note that user data is NULL here struct kevent events[1]; EV_SET(&amp;events[0], 1234, EVFILT_USER, EV_ADD | EV_ENABLE | EV_CLEAR, 0, 0, NULL); assert(0 == kevent(kq, events, 1, NULL, 0, NULL));  trigger_user_event();  struct timespec *timeout = NULL; \/\/ wait indefinitely int n = kevent(kq, NULL, 0, events, 1, timeout); assert(n > 0);  struct context *o = events[0].udata; if (events[0].filter == EVFILT_USER) o->handler(o); \/\/ handle user event  close(kq); } <\/code><\/pre>\n<p>To register a new user event in KQ we have to supply an arbitrary ID which we later will use to trigger it, I use <code>1234<\/code> just for example.  Note also that contrary to when attaching socket descriptor, we don&#8217;t set user data pointer at this point.<\/p>\n<pre><code class=\"cpp\">EV_SET(&amp;events[0], 1234, EVFILT_USER, EV_ADD | EV_ENABLE | EV_CLEAR, 0, 0, NULL); <\/code><\/pre>\n<p>Then at some point we decide to trigger the event with <code>NOTE_TRIGGER<\/code>.  And now we can pass an object pointer which the waiting function will return back to us after the event signals.<\/p>\n<pre><code class=\"cpp\">EV_SET(&amp;events[0], 1234, EVFILT_USER, 0, NOTE_TRIGGER, 0, obj); <\/code><\/pre>\n<p>After this event is returned from KQ and gets processed, it won&#8217;t signal again until we trigger it next time &#8212; that&#8217;s because we set <code>EV_CLEAR<\/code> flag when registering the event.<\/p>\n<p><a class=\"anchor\" name=\"system-timer-events-with-kqueue\" id=\"system-timer-events-with-kqueue\"><\/a><\/p>\n<h4>System timer events with kqueue<\/h4>\n<p>Another facility that KQ offers us is system timers &#8212; we can order KQ to periodically send us a timer event.  A timer is necessary when we want to close connections for the clients that are silent for too long, for example.  In kqueue we register a timer with <code>EVFILT_TIMER<\/code> and process its events as usual.  For example:<\/p>\n<pre><code class=\"cpp\">\/* Kernel Queue The Complete Guide: kqueue-timer.c: System timer events *\/ #include &lt;assert.h> #include &lt;stdio.h> #include &lt;unistd.h> #include &lt;sys\/types.h> #include &lt;sys\/event.h>  int kq;  struct context { void (*handler)(struct context *obj); };  void timer_handler(struct context *obj) { static int n; printf(\"Received timer event via kqueue: %d\\n\", n++); }  void main() { kq = kqueue(); assert(kq != -1);  struct context obj = {}; obj.handler = timer_handler;  \/\/ start system timer int period_ms = 1000; struct kevent events[1]; EV_SET(&amp;events[0], 1234, EVFILT_TIMER, EV_ADD | EV_ENABLE, 0, period_ms, &amp;obj); assert(0 == kevent(kq, events, 1, NULL, 0, NULL));  for (;;) { struct timespec *timeout = NULL; \/\/ wait indefinitely int n = kevent(kq, NULL, 0, events, 1, timeout); assert(n > 0);  struct context *o = events[0].udata; if (events[0].filter == EVFILT_TIMER) o->handler(o); \/\/ handle timer event }  close(kq); } <\/code><\/pre>\n<p>Sometimes we don&#8217;t need a periodic timer, but the timer which will signal us only once, i.e. a <strong>one-shot timer<\/strong>.  It&#8217;s simple with kqueue &#8212; we just use <code>EV_ONESHOT<\/code> flag:<\/p>\n<pre><code class=\"cpp\">EV_SET(&amp;events[0], 1234, EVFILT_TIMER, EV_ADD | EV_ENABLE | EV_ONESHOT, 0, period_ms, obj); <\/code><\/pre>\n<p>Keep in mind that KQ timers are not designed so that you can use a million of them &#8212; you just need 1.  Even if our software handles a million clients, we still need 1 system timer, because all we need is to just periodically wake up and process the oldest entries in our <em>timer queue<\/em> which we handle ourselves with our own code.  Timer queue mechanism isn&#8217;t a part of KQ, it isn&#8217;t in scope of this article, but it&#8217;s just a linked-list or rbtree container where the first item is the oldest.<\/p>\n<p><a class=\"anchor\" name=\"unix-signals-from-kqueue\" id=\"unix-signals-from-kqueue\"><\/a><\/p>\n<h4>UNIX signals from kqueue<\/h4>\n<p>Another convenient feature of KQ is handling UNIX signals.  We register a UNIX signal handler with <code>EVFILT_SIGNAL<\/code> and pass the signal number we want to attach to.  When processing an event, we get the signal number from <code>struct kevent.ident<\/code> field.  Example:<\/p>\n<pre><code class=\"cpp\">\/* Kernel Queue The Complete Guide: kqueue-signal.c: UNIX signal handler Usage: $ .\/kqueue-signal $ killall -SIGUSR1 kqueue-signal *\/ #include &lt;assert.h> #include &lt;signal.h> #include &lt;stdio.h> #include &lt;unistd.h> #include &lt;sys\/types.h> #include &lt;sys\/event.h>  int kq;  struct context { void (*handler)(int sig); };  void unix_signal_handler(int sig) { printf(\"Received UNIX signal via kqueue: %d\\n\", sig); }  void main() { kq = kqueue(); assert(kq != -1);  struct context obj = {}; obj.handler = unix_signal_handler;  \/\/ block default signal handler int sig = SIGUSR1; sigset_t mask; sigemptyset(&amp;mask); sigaddset(&amp;mask, sig); sigprocmask(SIG_BLOCK, &amp;mask, NULL);  \/\/ register UNIX signal handler struct kevent events[1]; EV_SET(&amp;events[0], sig, EVFILT_SIGNAL, EV_ADD | EV_ENABLE, 0, 0, &amp;obj); assert(0 == kevent(kq, events, 1, NULL, 0, NULL));  struct timespec *timeout = NULL; \/\/ wait indefinitely int n = kevent(kq, NULL, 0, events, 1, timeout); assert(n > 0);  struct context *o = events[0].udata; if (events[0].filter == EVFILT_SIGNAL) { int sig = events[0].ident; obj.handler(sig); \/\/ handle UNIX signal }  close(kq); } <\/code><\/pre>\n<p>Note that KQ will return the event only after the signal has been processed with its normal mechanism, i.e. the handlers registered with <code>sigaction()<\/code> &#8212; KQ can&#8217;t completely replace this mechanism.  But KQ makes it easier to handle signals such as <code>SIGCHLD<\/code>, when a child process signals its parent about its closure.<\/p>\n<p><a class=\"anchor\" name=\"asynchronous-file-io-with-kqueue\" id=\"asynchronous-file-io-with-kqueue\"><\/a><\/p>\n<h4>Asynchronous file I\/O with kqueue<\/h4>\n<p>Trying to read or write data from\/to files on disk is a little bit harder than performing I\/O on sockets.  One of the reasons behind this is because sockets don&#8217;t have an offset to read at &#8212; we always read from one end, while with files we may issue several parallel operations at different offsets.  And how can OS notify us about which particular operation has completed?  That&#8217;s why the kernel has to provide us with a new API for dealing with file AIO.  And FreeBSD is the only OS that has a complete implementation of asynchronous file read\/write operations.  Sadly, it doesn&#8217;t look anything like the rest of what we&#8217;ve talked about here so far.  What we would have expected from kqueue is the mechanism which just signals us when some data is available to read or write from\/to a file &#8212; exactly the same way we work with sockets.  But no, what we have here instead is the mechanism of asynchronous file operations (only read\/write operations are supported) which hold (or &#171;lock&#187;) the user data buffer internally and don&#8217;t signal at all unless the whole data chunk is transferred, depriving us from the flexibility we already got used to when working with sockets.  <em>Now all this looks more like IOCP, which can&#8217;t be a good sign.<\/em>  But anyway, since I promised you to show everything I know about KQ, let&#8217;s see how this mechanism works with files.<\/p>\n<ul>\n<li>\n<p>First, we enable the AIO subsystem by loading the appropriate kernel module:<\/p>\n<pre><code>  % kldload aio <\/code><\/pre>\n<\/li>\n<li>\n<p>The next step is to prepare an AIO object of type <code>struct aiocb<\/code> and call the appropriate function, that is <code>aio_read()<\/code> or <code>aio_write()<\/code>.<\/p>\n<\/li>\n<li>\n<p>Then we immediately check for operation status with <code>aio_error()<\/code>, because it may have finished already before we even called KQ waiting function.<\/p>\n<\/li>\n<li>\n<p>When we know that everything is fine and the operation is in progress, we wait for a signal from KQ as usual.<\/p>\n<\/li>\n<li>\n<p>When we receive an event of type <code>EVFILT_AIO<\/code> from KQ we may read <code>struct kevent.ident<\/code> field to get <code>struct aiocb*<\/code> object pointer associated with the operation.  This is how we can distinguish several parallel operations on the same file descriptor from each other.<\/p>\n<\/li>\n<\/ul>\n<p>Here&#8217;s the minimal example of how to read from a file asynchronously:<\/p>\n<pre><code class=\"cpp\">\/* Kernel Queue The Complete Guide: kqueue-file.c: Asynchronous file reading Usage: $ echo 'Hello file AIO' >.\/kqueue-file.txt $ .\/kqueue-file *\/ #include &lt;aio.h> #include &lt;assert.h> #include &lt;errno.h> #include &lt;fcntl.h> #include &lt;stdio.h> #include &lt;stdlib.h> #include &lt;string.h> #include &lt;unistd.h> #include &lt;sys\/types.h> #include &lt;sys\/event.h>  int kq;  struct context { int fd; struct aiocb acb; int (*handler)(struct context *obj, struct aiocb *acb); };  void file_io_result(const char *via, int res) { printf(\"Read from file via %s: %d\\n\", via, res); }  int file_aio_handler(struct context *obj, struct aiocb *acb) { int r = aio_error(acb); if (r == EINPROGRESS) { return 0; \/\/ AIO in progress } else if (r == -1) { file_io_result(\"kqueue\", -1); \/\/ AIO completed with error return -1; }  r = aio_return(acb); file_io_result(\"kqueue\", r); \/\/ AIO completed successfully return 1; }  void main() { \/\/ create KQ object kq = kqueue(); assert(kq != -1);  \/\/ open file descriptor and prepare the associated object int fd = open(\".\/kqueue-file.txt\", O_RDONLY, 0); assert(fd != -1); struct context obj = {}; obj.handler = file_aio_handler;  \/\/ associate the AIO operation with KQ and user object pointer memset(&amp;obj.acb, 0, sizeof(obj.acb)); obj.acb.aio_sigevent.sigev_notify_kqueue = kq; obj.acb.aio_sigevent.sigev_notify = SIGEV_KEVENT; obj.acb.aio_sigevent.sigev_notify_kevent_flags = EV_CLEAR; obj.acb.aio_sigevent.sigev_value.sigval_ptr = &amp;obj;  void *buf = malloc(4*1024);  \/\/ specify operation parameters obj.acb.aio_fildes = fd; obj.acb.aio_buf = buf; \/\/ destination buffer obj.acb.aio_nbytes = 4*1024; \/\/ max number of bytes to read obj.acb.aio_offset = 0; \/\/ offset to begin reading at  \/\/ begin file AIO operation obj.acb.aio_lio_opcode = LIO_READ; if (0 != aio_read(&amp;obj.acb)) { if (errno == EAGAIN || errno == ENOSYS || errno == EOPNOTSUPP) { \/\/ no resources to complete this I\/O operation \/\/ or AIO module isn't loaded \/\/ or the system can't perform AIO on this file } else { file_io_result(\"aio_read\", -1); return; \/\/ fatal error }  \/\/ AIO doesn't work - perform synchronous reading at the specified offset int r = pread(fd, buf, obj.acb.aio_nbytes, obj.acb.aio_offset); file_io_result(\"pread\", r); return; }  \/\/ asynchronous file reading has started, but might be finished already if (0 != file_aio_handler(&amp;obj, &amp;obj.acb)) return;  \/\/ asynchronous file reading is in progress, now wait for the signal from KQ struct kevent events[1]; struct timespec *timeout = NULL; \/\/ wait indefinitely int n = kevent(kq, NULL, 0, events, 1, timeout);  struct context *o = events[0].udata; if (events[0].filter == EVFILT_AIO) { struct aiocb *acb = (void*)events[0].ident; o->handler(o, acb); \/\/ handle file AIO event }  free(buf); close(fd); close(kq); } <\/code><\/pre>\n<p>The main function here is <code>aio_read()<\/code>, which processes our request and starts the asynchronous operation.  It returns <code>0<\/code> if the operation has started successfully.<\/p>\n<pre><code class=\"cpp\">struct aiocb acb = ...; \/\/ fill in `acb` object acb.aio_lio_opcode = LIO_READ; aio_read(&amp;acb); \/\/ begin file AIO operation <\/code><\/pre>\n<p>However, if something is wrong, it returns with an error, and we must handle several cases here:<\/p>\n<pre><code class=\"cpp\">if (errno == EAGAIN || errno == ENOSYS || errno == EOPNOTSUPP) { \/\/ no resources to complete this I\/O operation \/\/ or AIO module isn't loaded \/\/ or the system can't perform AIO on this file } else { \/\/ fatal error } <\/code><\/pre>\n<p>We can handle some types of errors by issuing a <strong>synchronous file reading<\/strong>:<\/p>\n<pre><code class=\"cpp\">\/\/ AIO doesn't work - perform synchronous reading at the specified offset int r = pread(fd, buf, size, off); <\/code><\/pre>\n<p>For <strong>writing data to a file asynchronously<\/strong> we use the same template, except the opcode and the function are different, while everything else is the same:<\/p>\n<pre><code class=\"cpp\">struct aiocb acb = ...; \/\/ fill in `acb` object acb.aio_lio_opcode = LIO_WRITE; aio_write(&amp;acb); \/\/ begin file AIO operation <\/code><\/pre>\n<p>And of course we use a different function for the <strong>synchronous file writing<\/strong> in case file AIO doesn&#8217;t work:<\/p>\n<pre><code class=\"cpp\">int r = pwrite(fd, buf, size, off); <\/code><\/pre>\n<blockquote>\n<p>Though file AIO is good to have, it&#8217;s still not enough for high-performance file servers, because such software needs not just file AIO in terms of reading\/writing file data, but disk AIO in general.  What for?  In our example above we use asynchronous file reading only, but we know that the very first step in working with files in UNIX is opening a file descriptor.  However, we just can&#8217;t perform an asynchronous file open &#8212; it&#8217;s not supported.  And in the real life an <code>open()<\/code> syscall may take a whole second to complete on a busy machine &#8212; it may block our worker thread for a long time.  In real life we also want to call <code>stat()<\/code> or <code>fstat()<\/code> on a file path or a file descriptor.  And OS doesn&#8217;t provide a way to call them asynchronously either, except calling them inside another thread.  So even if file AIO can help sometimes, it&#8217;s still somewhat lame and incomplete.  And considering the fact that other OS don&#8217;t have an appropriate file AIO implementation at all, it may be a better choice not to use any of those APIs at all.  It may be better to use a thread pool with a file operations queue and dispatch operations to another thread.  Inside a new thread the operations will be performed <em>synchronously<\/em>.  And then it will signal the main thread when the operation is complete.<\/p>\n<\/blockquote>\n<p><a class=\"anchor\" name=\"linux-and-epoll\" id=\"linux-and-epoll\"><\/a><\/p>\n<h3>Linux and epoll<\/h3>\n<p>epoll API is very similar to kqueue for socket I\/O notifications, though it&#8217;s quite different for other purposes.  Here we use <code>epoll_ctl()<\/code> to attach file descriptors to KQ and we use <code>epoll_wait()<\/code> to receive events from KQ.  There are several more syscalls we&#8217;re going to use for user events, timers and UNIX signals: <code>eventfd()<\/code>, <code>timerfd_create()<\/code>, <code>timerfd_settime()<\/code>, <code>signalfd()<\/code>.  Overall, the functionality of <code>epoll<\/code> is the same as <code>kqueue<\/code> but sometimes with a slightly different approach.<\/p>\n<p>The key <strong>differences of epoll and kqueue<\/strong> are:<\/p>\n<ul>\n<li>\n<p><code>kevent()<\/code> supports attaching many file descriptors in a single syscall, <code>epoll_ctl()<\/code> does not &#8212; we must call it once for each fd.  <code>kevent()<\/code> even allows us to attach fd&#8217;s AND wait for new events in a single syscall, <code>epoll_wait()<\/code> can&#8217;t do that.<\/p>\n<\/li>\n<li>\n<p>epoll may join 2 events (<code>EPOLLIN<\/code> and <code>EPOLLOUT<\/code>) into 1 event object in case both READ and WRITE events signal.  This is contrary to kqueue which always returns 1 event object per 1 event (<code>EVFILT_READ<\/code> or <code>EVFILT_WRITE<\/code>).  Be careful with epoll here, always check if it&#8217;s alright to execute event handling functions, because otherwise you risk calling WRITE event handler after you have finalized the object in READ event handler.<\/p>\n<\/li>\n<li>\n<p>epoll makes it somewhat harder to use additional KQ functionality such as UNIX signals, system timers or user events &#8212; see below.  In kqueue, however, it looks all the same.<\/p>\n<\/li>\n<\/ul>\n<p>What is different but also similar between epoll and kqueue:<\/p>\n<ul>\n<li>\n<p>We attach a socket to epoll using <code>EPOLLIN | EPOLLOUT<\/code> flags which is the same as registering 2 separate events <code>EVFILT_READ<\/code> and <code>EVFILT_WRITE<\/code> with kqueue.<\/p>\n<\/li>\n<li>\n<p><code>EPOLLET<\/code> flag in epoll is the same thing as <code>EV_CLEAR<\/code> flag in kqueue &#8212; it prevents epoll from signalling us about the same event more than once until we drain all data from the socket.  And it&#8217;s the same situation when writing to a streaming (e.g. TCP) socket &#8212; we must keep calling <code>write()<\/code> until it returns with <code>EAGAIN<\/code> error &#8212; only then we may expect epoll to signal us.<\/p>\n<\/li>\n<\/ul>\n<p><a class=\"anchor\" name=\"accepting-socket-connections-with-epoll\" id=\"accepting-socket-connections-with-epoll\"><\/a><\/p>\n<h4>Accepting socket connections with epoll<\/h4>\n<p>Here&#8217;s a minimal example for accepting a socket connection.<\/p>\n<pre><code class=\"cpp\">\/* Kernel Queue The Complete Guide: epoll-accept.c: Accept socket connection Usage: $ .\/epoll-accept $ curl 127.0.0.1:64000\/ *\/ #include &lt;assert.h> #include &lt;fcntl.h> #include &lt;netinet\/in.h> #include &lt;signal.h> #include &lt;stdio.h> #include &lt;unistd.h> #include &lt;sys\/ioctl.h> #include &lt;sys\/epoll.h> #include &lt;sys\/socket.h>  int kq;  \/\/ the structure associated with a socket descriptor struct context { int sk; void (*rhandler)(struct context *object); };  void accept_handler(struct context *obj) { printf(\"Received socket READ event via epoll\\n\");  int csock = accept(obj->sk, NULL, 0); assert(csock != -1); close(csock); }  void main() { \/\/ create KQ object kq = epoll_create(1); assert(kq != -1);  struct context obj = {}; obj.rhandler = accept_handler;  \/\/ create and prepare a socket obj.sk = socket(AF_INET, SOCK_STREAM | SOCK_NONBLOCK, 0); assert(obj.sk != -1); int val = 1; setsockopt(obj.sk, SOL_SOCKET, SO_REUSEADDR, &amp;val, 4);  struct sockaddr_in addr = {}; addr.sin_family = AF_INET; addr.sin_port = ntohs(64000); assert(0 == bind(obj.sk, (struct sockaddr*)&amp;addr, sizeof(addr))); assert(0 == listen(obj.sk, 0));  \/\/ attach socket to KQ struct epoll_event event; event.events = EPOLLIN | EPOLLOUT | EPOLLET; event.data.ptr = &amp;obj; assert(0 == epoll_ctl(kq, EPOLL_CTL_ADD, obj.sk, &amp;event));  \/\/ wait for incoming events from KQ struct epoll_event events[1]; int timeout_ms = -1; \/\/ wait indefinitely int n = epoll_wait(kq, events, 1, timeout_ms); assert(n > 0);  \/\/ process the received event struct context *o = events[0].data.ptr; if (events[0].events &amp; (EPOLLIN | EPOLLERR)) o->rhandler(o); \/\/ handle read event  close(obj.sk); close(kq); } <\/code><\/pre>\n<p><a class=\"anchor\" name=\"creating-and-closing-epoll-object\" id=\"creating-and-closing-epoll-object\"><\/a><\/p>\n<h4>Creating and closing epoll object<\/h4>\n<p><code>epoll_create()<\/code> function returns new KQ object descriptor which we close as usual with <code>close()<\/code>.<\/p>\n<pre><code class=\"cpp\">kq = epoll_create(1); ... close(kq); <\/code><\/pre>\n<p><a class=\"anchor\" name=\"attaching-socket-descriptor-to-epoll\" id=\"attaching-socket-descriptor-to-epoll\"><\/a><\/p>\n<h4>Attaching socket descriptor to epoll<\/h4>\n<p><code>EPOLLIN<\/code> flag means that we want the kernel to notify us when a READ event signals, and <code>EPOLLOUT<\/code> flag is the same for a WRITE event.  <code>EPOLLET<\/code> prevents epoll from returning to us the same signal unnecessarily.  We set our object pointer with <code>struct epoll_event.data.ptr<\/code> field.<\/p>\n<pre><code class=\"cpp\">struct epoll_event event; event.events = EPOLLIN | EPOLLOUT | EPOLLET; event.data.ptr = obj; epoll_ctl(kq, EPOLL_CTL_ADD, sk, &amp;event); <\/code><\/pre>\n<p><a class=\"anchor\" name=\"receiving-events-from-epoll\" id=\"receiving-events-from-epoll\"><\/a><\/p>\n<h4>Receiving events from epoll<\/h4>\n<p><code>epoll_wait()<\/code> function blocks until it has something to return to us, or until the timeout value expires.  The function returns the number of signalled events which may be <code>0<\/code> only in case of timeout.  It also returns with <code>EINTR<\/code> exactly like <code>kevent()<\/code> after a UNIX signal has been received, so we must always handle this case.<\/p>\n<pre><code class=\"cpp\">while (!quit) { struct epoll_event events[1]; int timeout_ms = -1; \/\/ wait indefinitely int n = epoll_wait(kq, events, 1, timeout_ms); if (n &lt; 0 &amp;&amp; errno == EINTR) continue; \/\/ epoll_wait() interrupts when UNIX signal is received } <\/code><\/pre>\n<p><a class=\"anchor\" name=\"processing-received-events-from-epoll\" id=\"processing-received-events-from-epoll\"><\/a><\/p>\n<h4>Processing received events from epoll<\/h4>\n<p>We get the events flags by reading <code>struct epoll_event.events<\/code> which we should always test for <code>EPOLLERR<\/code> too, because otherwise we can miss an event we&#8217;re waiting for.  To get the associated user data pointer, we read <code>struct epoll_event.data.ptr<\/code> value.<\/p>\n<p>I emphasize once again that you should be careful not to invalidate user object memory region inside READ event handler, or the program may crash inside the WRITE event handler which is executed next.  To handle this situation you may just always set event handler function pointers to <code>NULL<\/code> when you don&#8217;t expect them to signal.  An alternative solution may be to clear <code>EPOLLOUT | EPOLLERR<\/code> flags from <code>struct epoll_event.events<\/code> field from inside READ event handler.<\/p>\n<pre><code class=\"cpp\">struct context *o = events[i].data.ptr;  if ((events[i].events &amp; (EPOLLIN | EPOLLERR)) &amp;&amp; o->rhandler != NULL) o->rhandler(o); \/\/ handle read event  if ((events[i].events &amp; (EPOLLOUT | EPOLLERR)) &amp;&amp; o->whandler != NULL) o->whandler(o); \/\/ handle write event <\/code><\/pre>\n<p><a class=\"anchor\" name=\"establishing-tcp-connection-with-epoll\" id=\"establishing-tcp-connection-with-epoll\"><\/a><\/p>\n<h4>Establishing TCP connection with epoll<\/h4>\n<p>As with kqueue, epoll also has a way to notify us about the status of TCP connection.  But unlike kqueue which sets the error number for us inside <code>struct kevent<\/code> object, epoll doesn&#8217;t do that (it can&#8217;t do that).  Instead, we get the error number associated with our socket via <code>getsockopt(..., SOL_SOCKET, SO_ERROR, ...)<\/code>.<\/p>\n<pre><code class=\"cpp\">int err; socklen_t len = 4; getsockopt(obj->sk, SOL_SOCKET, SO_ERROR, &amp;err, &amp;len); errno = err; ... \/\/ handle TCP connection result depending on `errno` value <\/code><\/pre>\n<p><a class=\"anchor\" name=\"user-triggered-events-with-epoll\" id=\"user-triggered-events-with-epoll\"><\/a><\/p>\n<h4>User-triggered events with epoll<\/h4>\n<p>There are slightly more things to do with epoll rather than with kqueue to handle user-triggered events:<\/p>\n<ul>\n<li>\n<p>First, we create a new file descriptor with <code>eventfd()<\/code> which we then attach to KQ.<\/p>\n<\/li>\n<li>\n<p>We trigger a user event at any time by writing an 8 byte value to our eventfd descriptor.  In our case this value is an object pointer.<\/p>\n<\/li>\n<li>\n<p>After we receive an event from KQ, we read an 8 byte value from eventfd descriptor.  We can convert this value to an object pointer.  Remember that we need to keep reading data from eventfd descriptor until it returns with <code>EAGAIN<\/code> error, because we use <code>EPOLLET<\/code>.<\/p>\n<\/li>\n<\/ul>\n<pre><code class=\"cpp\">\/* Kernel Queue The Complete Guide: epoll-user.c: User-triggered events *\/ #include &lt;assert.h> #include &lt;errno.h> #include &lt;signal.h> #include &lt;stdio.h> #include &lt;unistd.h> #include &lt;sys\/epoll.h> #include &lt;sys\/eventfd.h>  int kq; int efd;  struct context { void (*handler)(struct context *obj); };  struct context eventfd_obj; struct context user_event_obj;  void user_event_obj_handler(struct context *obj) { printf(\"Received user event via epoll\\n\"); }  \/\/ application calls this function whenever it wants to add a new event to KQ \/\/ which will execute user_event_obj_handler() void trigger_user_event() { struct context *obj = &amp;user_event_obj; obj->handler = user_event_obj_handler;  unsigned long long val = (size_t)obj; int r = write(efd, &amp;val, 8); assert(r == 8); }  \/\/ handle event from eventfd-descriptor void handle_eventfd(struct context *obj) { unsigned long long val; for (;;) { int r = read(efd, &amp;val, 8); if (r &lt; 0 &amp;&amp; errno == EAGAIN) break; assert(r == 8);  struct context *o = (void*)(size_t)val; o->handler(o); } }  void main() { \/\/ create kqueue object kq = epoll_create(1); assert(kq != -1);  struct context obj = {}; obj.handler = handle_eventfd;  \/\/ prepare eventfd-descriptor for user events efd = eventfd(0, EFD_NONBLOCK); assert(efd != -1);  \/\/ register eventfd in KQ struct epoll_event event; event.events = EPOLLIN | EPOLLET; event.data.ptr = &amp;obj; assert(0 == epoll_ctl(kq, EPOLL_CTL_ADD, efd, &amp;event));  trigger_user_event();  struct epoll_event events[1]; int timeout_ms = -1; int n = epoll_wait(kq, events, 1, timeout_ms); assert(n > 0);  struct context *o = events[0].data.ptr; if (events[0].events &amp; (EPOLLIN | EPOLLERR)) o->handler(o); \/\/ handle eventfd event  close(efd); \/\/ close eventfd descriptor close(kq); } <\/code><\/pre>\n<p><a class=\"anchor\" name=\"system-timer-events-with-epoll\" id=\"system-timer-events-with-epoll\"><\/a><\/p>\n<h4>System timer events with epoll<\/h4>\n<p>To receive the notifications from system timer with epoll we must use a timerfd object &#8212; a special file descriptor we can attach to epoll.<\/p>\n<pre><code class=\"cpp\">\/* Kernel Queue The Complete Guide: epoll-timer.c: System timer events *\/ #include &lt;assert.h> #include &lt;signal.h> #include &lt;stdio.h> #include &lt;unistd.h> #include &lt;sys\/epoll.h> #include &lt;sys\/timerfd.h>  int kq; int tfd;  struct context { void (*handler)(struct context *obj); };  void timer_handler(struct context *obj) { static int n; printf(\"Received timerfd event via epoll: %d\\n\", n++);  unsigned long long val; read(tfd, &amp;val, 8); }  void main() { \/\/ create kqueue object kq = epoll_create(1); assert(kq != -1);  struct context obj = {}; obj.handler = timer_handler;  \/\/ prepare timerfd-descriptor tfd = timerfd_create(CLOCK_MONOTONIC, 0); assert(tfd != -1);  \/\/ register timerfd in KQ struct epoll_event event; event.events = EPOLLIN | EPOLLET; event.data.ptr = &amp;obj; assert(0 == epoll_ctl(kq, EPOLL_CTL_ADD, tfd, &amp;event));  \/\/ start periodic timer struct itimerspec its; its.it_value.tv_sec = 1; its.it_value.tv_nsec = 0; its.it_interval = its.it_value; assert(0 == timerfd_settime(tfd, 0, &amp;its, NULL));  for (;;) { struct epoll_event events[1]; int timeout_ms = -1; int n = epoll_wait(kq, events, 1, timeout_ms); assert(n > 0);  struct context *o = events[0].data.ptr; if (events[0].events &amp; (EPOLLIN | EPOLLERR)) o->handler(o); \/\/ handle timerfd event }  close(tfd); \/\/ close timerfd descriptor close(kq); } <\/code><\/pre>\n<p>To create a timerfd object we call <code>timerfd_create()<\/code> with <code>CLOCK_MONOTONIC<\/code> parameter which means the timer isn&#8217;t affected by system time\/date changes.<\/p>\n<pre><code class=\"cpp\">tfd = timerfd_create(CLOCK_MONOTONIC, 0); <\/code><\/pre>\n<p>To start a periodic timer, we set <code>struct itimerspec.it_interval<\/code> field which defines the timer interval.<\/p>\n<pre><code class=\"cpp\">\/\/ start periodict timer struct itimerspec its; its.it_value.tv_sec = 1; its.it_value.tv_nsec = 0; its.it_interval = its.it_value; timerfd_settime(tfd, 0, &amp;its, NULL); <\/code><\/pre>\n<p>After we have received the timer event we always need to read the data from timerfd descriptor, otherwise we won&#8217;t get any more events from KQ because we use <code>EPOLLET<\/code> flag.<\/p>\n<pre><code class=\"cpp\">unsigned long long val; read(tfd, &amp;val, 8); <\/code><\/pre>\n<p>To create a <strong>one-shot timer<\/strong>, we set <code>struct itimerspec.it_interval<\/code> fields to <code>0<\/code>:<\/p>\n<pre><code class=\"cpp\">\/\/ start one-shot timer struct itimerspec its; its.it_value.tv_sec = 1; its.it_value.tv_nsec = 0; its.it_interval.tv_sec = its.it_interval.tv_nsec = 0; timerfd_settime(tfd, 0, &amp;its, NULL); <\/code><\/pre>\n<p><a class=\"anchor\" name=\"unix-signals-from-epoll\" id=\"unix-signals-from-epoll\"><\/a><\/p>\n<h4>UNIX signals from epoll<\/h4>\n<p>We can also receive UNIX signals with epoll.  We must use a signalfd descriptor to be able to do that.  To get the information about the UNIX signal we receive, we read some data from this signalfd object.<\/p>\n<pre><code class=\"cpp\">\/* Kernel Queue The Complete Guide: epoll-signal.c: UNIX signal handler Usage: $ .\/epoll-signal $ killall -SIGUSR1 epoll-signal *\/ #include &lt;assert.h> #include &lt;signal.h> #include &lt;stdio.h> #include &lt;unistd.h> #include &lt;sys\/epoll.h> #include &lt;sys\/signalfd.h>  int kq; int sfd;  struct context { void (*handler)(struct context *obj); };  void unix_signal_handler(struct context *obj) { struct signalfd_siginfo si; int r = read(sfd, &amp;si, sizeof(si)); assert(r == sizeof(si));  int sig = si.ssi_signo; printf(\"Received UNIX signal via epoll: %d\\n\", sig); }  void main() { \/\/ create kqueue object kq = epoll_create(1); assert(kq != -1);  struct context obj = {}; obj.handler = unix_signal_handler;  \/\/ block default signal handler int sig = SIGUSR1; sigset_t mask; sigemptyset(&amp;mask); sigaddset(&amp;mask, sig); sigprocmask(SIG_BLOCK, &amp;mask, NULL);  \/\/ prepare signalfd-descriptor sfd = signalfd(-1, &amp;mask, SFD_NONBLOCK); assert(sfd != -1);  \/\/ register signalfd in KQ struct epoll_event event; event.events = EPOLLIN | EPOLLET; event.data.ptr = &amp;obj; assert(0 == epoll_ctl(kq, EPOLL_CTL_ADD, sfd, &amp;event));  struct epoll_event events[1]; int timeout_ms = -1; int n = epoll_wait(kq, events, 1, timeout_ms); assert(n > 0);  struct context *o = events[0].data.ptr; if (events[0].events &amp; (EPOLLIN | EPOLLERR)) o->handler(o); \/\/ handle signalfd event  close(sfd); \/\/ close signalfd descriptor close(kq); } <\/code><\/pre>\n<p>We create a signalfd object by calling <code>signalfd()<\/code>.<\/p>\n<pre><code class=\"cpp\">sfd = signalfd(-1, &amp;mask, SFD_NONBLOCK); <\/code><\/pre>\n<p>As with timerfd object, we also need to read from signalfd descriptor otherwise we won&#8217;t get more events from KQ.  But another reason why we need this is we want to know which signal has fired.  We do it by reading <code>struct signalfd_siginfo<\/code> data from signalfd object and then reading <code>struct signalfd_siginfo.ssi_signo<\/code> field that contains UNIX signal number.<\/p>\n<pre><code class=\"cpp\">struct signalfd_siginfo si; read(sfd, &amp;si, sizeof(si)); int sig = si.ssi_signo; <\/code><\/pre>\n<p><a class=\"anchor\" name=\"asynchronous-file-io-with-epoll\" id=\"asynchronous-file-io-with-epoll\"><\/a><\/p>\n<h4>Asynchronous file I\/O with epoll<\/h4>\n<p>It&#8217;s even harder to work with files asynchronously when using epoll compared to the file AIO API we used earlier with kqueue.  First of all, there&#8217;s the requirement to use <code>O_DIRECT<\/code> flag when opening a file descriptor if we want it to support asynchronous operations.  This flag alone creates 2 new restrictions for us: we must implement our own data caching mechanism and we must use buffers aligned to disk block size.  We solve the first problem by not solving it &#8212; in our example we don&#8217;t implement any data caching here.  However, in the real life our performance may be very poor without it.  Linux kernel has a very good disk data caching mechanism and it&#8217;s a shame when we can&#8217;t use it.  In general, this means that there are not so many use-cases for using file AIO on Linux.  The second restriction which is the requirement of using aligned buffers is just inconvenient, but at least we can handle it.  We can&#8217;t use a normal buffer pointer allocated by <code>malloc()<\/code> &#8212; we must use a pointer aligned to disk block size (in our example we use <code>posix_memalign()<\/code> function which returns an aligned pointer).  Furthermore, we can&#8217;t read or write an arbitrary amount of data, say, 1000 bytes, neither we can use an arbitrary file offset &#8212; we must also use an aligned number.<\/p>\n<p>Another inconvenient thing is that GLIBC still doesn&#8217;t have the wrappers for the syscalls we need to use here.  We solve this problem by writing our own wrappers.  It isn&#8217;t hard, but just inconvenient.  <em>Does this mean that only some crazy people use this functionality on Linux?<\/em><\/p>\n<p>The file AIO API on Linux sometimes looks similar to what we were using on FreeBSD (at least, submitting a new operation looks similar).  However, the signal delivery mechanism is completely different.  It uses eventfd descriptor we are already familiar with to notify us about that AIO subsystem has signalled.  We then receive events from AIO and process the results of our operations.  Though there&#8217;s nothing here too hard to implement, the whole code just looks a little bit overcomplex and ugly.  There are too many things we must control here, so the use of convenient wrappers is absolutely required in real life code.<\/p>\n<p>OK, let&#8217;s see the example already, or else I won&#8217;t stop complaining how bad all this is.<\/p>\n<pre><code class=\"cpp\">\/* Kernel Queue The Complete Guide: epoll-file.c: Asynchronous file reading Usage: $ echo 'Hello file AIO' >.\/epoll-file.txt $ .\/epoll-file *\/ #define _GNU_SOURCE #include &lt;assert.h> #include &lt;errno.h> #include &lt;fcntl.h> #include &lt;stdio.h> #include &lt;stdlib.h> #include &lt;string.h> #include &lt;unistd.h> #include &lt;sys\/epoll.h> #include &lt;sys\/syscall.h> #include &lt;sys\/eventfd.h> #include &lt;linux\/aio_abi.h>  int kq; int efd; aio_context_t aioctx;  struct context { struct iocb acb; int (*handler)(struct context *obj); };  void file_io_result(const char *via, int res) { printf(\"Read from file via %s: %d\\n\", via, res); }  \/\/ GLIBC doesn't have wrappers for these syscalls, so we make our own wrappers static inline int io_setup(unsigned nr_events, aio_context_t *ctx_idp) { return syscall(SYS_io_setup, nr_events, ctx_idp); } static inline int io_destroy(aio_context_t ctx_id) { return syscall(SYS_io_destroy, ctx_id); } static inline int io_submit(aio_context_t ctx_id, long nr, struct iocb **iocbpp) { return syscall(SYS_io_submit, ctx_id, nr, iocbpp); } static inline int io_getevents(aio_context_t ctx_id, long min_nr, long nr, struct io_event *events, struct timespec *timeout) { return syscall(SYS_io_getevents, ctx_id, min_nr, nr, events, timeout); }  int file_aio_handler(struct context *obj) { unsigned long long n; for (;;) { int r = read(efd, &amp;n, 8); if (r &lt; 0 &amp;&amp; errno == EAGAIN) break; assert(r == 8); \/\/ we've got `n` unprocessed events from file AIO  for (;;) {  struct io_event events[64]; struct timespec timeout = {}; r = io_getevents(aioctx, 1, 64, events, &amp;timeout); if (r &lt; 0 &amp;&amp; errno == EINTR) { continue; \/\/ interrupted due to UNIX signal } else if (r == 0) { break; \/\/ no more events } assert(r > 0);  \/\/ process result value for each event for (int i = 0;  i != r;  i++) { struct context *obj = (void*)(size_t)events[i].data; int result = events[i].res; if (result &lt; 0) { errno = -result; result = -1; } file_io_result(\"epoll\", result); } } } return 1; }  void main() { \/\/ create KQ object kq = epoll_create(1); assert(kq != -1);  \/\/ prepare the associated object struct context obj = {}; obj.handler = file_aio_handler;  \/\/ open file descriptor, O_DIRECT is mandatory int fd = open(\".\/epoll-file.txt\", O_DIRECT | O_RDONLY, 0); assert(fd != -1);  \/\/ initialize file AIO subsystem int aio_workers = 64; assert(0 == io_setup(aio_workers, &amp;aioctx));  \/\/ open eventfd descriptor which will pass signals from file AIO efd = eventfd(0, EFD_NONBLOCK); assert(efd != -1);  \/\/ attach eventfd to KQ struct epoll_event event; event.events = EPOLLIN | EPOLLET; event.data.ptr = &amp;obj; assert(0 == epoll_ctl(kq, EPOLL_CTL_ADD, efd, &amp;event));  \/\/ associate the AIO operation with KQ and user object pointer memset(&amp;obj.acb, 0, sizeof(obj.acb)); obj.acb.aio_data = (size_t)&amp;obj; obj.acb.aio_flags = IOCB_FLAG_RESFD; obj.acb.aio_resfd = efd;  void *buf; assert(0 == posix_memalign(&amp;buf, 512, 4*1024)); \/\/ allocate 4k buffer aligned by 512  \/\/ specify operation parameters obj.acb.aio_fildes = fd; obj.acb.aio_buf = (size_t)buf; \/\/ destination buffer obj.acb.aio_nbytes = 4*1024; \/\/ max number of bytes to read obj.acb.aio_offset = 0; \/\/ offset to begin reading at  \/\/ begin file AIO operation obj.acb.aio_lio_opcode = IOCB_CMD_PREAD; struct iocb *cb = &amp;obj.acb; if (1 != io_submit(aioctx, 1, &amp;cb)) { if (errno == EAGAIN || errno == ENOSYS) { \/\/ no resources to complete this I\/O operation \/\/ or the system can't perform AIO on this file } else { file_io_result(\"io_submit\", -1); return; \/\/ fatal error }  \/\/ AIO doesn't work - perform synchronous reading at the specified offset int r = pread(fd, buf, obj.acb.aio_nbytes, obj.acb.aio_offset); file_io_result(\"pread\", r); return; }  \/\/ asynchronous file reading is in progress, now wait for the signal from KQ struct epoll_event events[1]; int timeout_ms = -1; \/\/ wait indefinitely int n = epoll_wait(kq, events, 1, timeout_ms);  struct context *o = events[0].data.ptr; if (events[0].events &amp; (EPOLLIN | EPOLLERR)) { o->handler(o); \/\/ handle file AIO event via eventfd }  free(buf); close(fd); io_destroy(aioctx); close(kq); } <\/code><\/pre>\n<p>First, we initialize AIO subsystem by calling <code>io_setup()<\/code> which returns a pointer to AIO context.  We specify how many I\/O workers the system should allocate for us.  I don&#8217;t have any good advice for you on how many workers you should allocate &#8212; the official documentation is very short &#8212; so I just use <code>64<\/code> for this example.<\/p>\n<pre><code class=\"cpp\">aio_context_t aioctx; int aio_workers = 64; io_setup(aio_workers, &amp;aioctx); <\/code><\/pre>\n<p>We need an object of type <code>struct iocb<\/code> which will define the operation we want to perform.  We start filling it by specifiyng how it should signal us about its completion.  Here we order it to notify us via eventfd descriptor.<\/p>\n<pre><code class=\"cpp\">struct iocb acb = {}; acb.aio_data = (size_t)obj; \/\/ user object pointer acb.aio_flags = IOCB_FLAG_RESFD; acb.aio_resfd = efd; \/\/ eventfd descriptor <\/code><\/pre>\n<p>Then we set the details on our operation: which file descriptor to use, where to put the data and so on.<\/p>\n<pre><code class=\"cpp\">acb.aio_fildes = fd; acb.aio_buf = (size_t)buf; \/\/ destination buffer acb.aio_nbytes = 4*1024; \/\/ max number of bytes to read acb.aio_offset = 0; \/\/ offset to begin reading at <\/code><\/pre>\n<p>Finally, we call the <code>io_submit()<\/code> function which will begin reading from a file and signal us via eventfd on completion.<\/p>\n<pre><code class=\"cpp\">acb.aio_lio_opcode = IOCB_CMD_PREAD; struct iocb *cb = &amp;acb; io_submit(aioctx, 1, &amp;cb); <\/code><\/pre>\n<p>Just like on FreeBSD, we also must check for error codes here on Linux and be ready to perform our I\/O operation synchronously if we need to.<\/p>\n<pre><code class=\"cpp\">if (1 != io_submit(aioctx, 1, &amp;cb)) { if (errno == EAGAIN || errno == ENOSYS) { \/\/ no resources to complete this I\/O operation \/\/ or the system can't perform AIO on this file } else { file_io_result(-1); return; \/\/ fatal error }  \/\/ AIO doesn't work - perform synchronous reading at the specified offset int r = pread(fd, buf, size, offset); file_io_result(r); return; } <\/code><\/pre>\n<p>When we receive a signal from epoll for our eventfd we&#8217;ve associated the file AIO operation with, we ask AIO about which operations are complete using <code>io_getevents()<\/code> function.  It returns the number of completed operations and fills our array of <code>struct io_event<\/code> from where we can determine the results of each operation.  Timeout value defines how long the function can block, but we don&#8217;t want it to block at all, obviously, so we use a zero timeout value.<\/p>\n<pre><code class=\"cpp\">struct io_event events[64]; struct timespec timeout = {}; n = io_getevents(aioctx, 1, 64, events, &amp;timeout); <\/code><\/pre>\n<p>And lastly, we process each event, get the user object associated with it by reading <code>struct io_event.data<\/code> field and the result of operation from <code>struct io_event.res<\/code> field.  The result is the number of bytes read or written, but in case an error has occurred, this value is negative and contains the error code.<\/p>\n<pre><code class=\"cpp\">struct context *obj = (void*)(size_t)events[i].data; int result = events[i].res; if (result &lt; 0) { errno = -result; result = -1; } <\/code><\/pre>\n<blockquote>\n<p>Once again I&#8217;m gonna say that this whole system has a poor design and it may be better to use a thread-pool mechanism which performs file operations synchronously.  A good file AIO system must have more configuration options such as how many workers should be used for each physical disk.  And it shouldn&#8217;t have restrictions such as buffer alignment.  And it shouldn&#8217;t force the user to disable in-kernel caching for a file descriptor.  And it should support more file functions including <code>open()<\/code> and <code>fstat()<\/code>.  But I think the main point is that the code for using file AIO should look similar to what we have with KQ and network sockets.  That is, we want an API which allows us to perform a simple operation such as reading a 64k byte chunk from a file, and <em>if it can&#8217;t complete immediately<\/em>, it should just return with <code>EINPROGRESS<\/code>.  This function should also start a new background operation internally which will read at least 1 block of data from our file (GLIBC could handle that).  Then we should receive a signal from KQ which tells us that now we may try again.  And again we call <em>the same code<\/em> but now it returns, say, 4k bytes to us &#8212; and even so it isn&#8217;t 64k we asked for, it is still fine, because we can start processing this data immediately.  This is a very straightforward approach that works very well with sockets (except the poorly designed <code>connect()+EINPROGRESS+getsockopt()<\/code> logic).  I just wish the file functions could work this way too.  But I guess that since it isn&#8217;t still implemented, no one needs this functionality, that&#8217;s all.<\/p>\n<\/blockquote>\n<p><a class=\"anchor\" name=\"windows-and-io-completion-ports\" id=\"windows-and-io-completion-ports\"><\/a><\/p>\n<h3>Windows and I\/O Completion Ports<\/h3>\n<p>So are you ready for some more programming?  I have to warn you that IOCP may be a little bit harder to use and understand.  But even if you don&#8217;t plan to use IOCP on Windows in the near future, I recommend you to at least walk through all these code samples just to learn and remember <strong>how not to design a KQ API<\/strong>.  A bad example is still an example.  I&#8217;m also very surprised to see that some developers even try to mimic the behaviour of IOCP in their own KQ\/AIO libraries.  They think that the API design of pending asynchronous operations that forces the user to have long living locked context data and buffer memory regions is better than the mechanism of simple and plain event signalling?  Maybe for someone, but not for me.  To prove my point, just see how the code for kqueue looks comparing to the same code for IOCP and judge for yourselves.  But first, let&#8217;s start with a minimal example which surprisingly looks quite clear, because we use pipes and not network sockets.<\/p>\n<p><a class=\"anchor\" name=\"accepting-connections-to-a-named-pipe-with-iocp\" id=\"accepting-connections-to-a-named-pipe-with-iocp\"><\/a><\/p>\n<h4>Accepting connections to a named pipe with IOCP<\/h4>\n<pre><code class=\"cpp\">\/* Kernel Queue The Complete Guide: iocp-pipe.c: Accept connections to a named pipe *\/ #include &lt;windows.h> #include &lt;assert.h> #include &lt;stdio.h>  HANDLE kq;  struct context { HANDLE p; void (*handler)(struct context *obj); OVERLAPPED accept_ctx; };  void pipe_handler(struct context *obj) { printf(\"Accepted pipe connection via IOCP\\n\"); }  void main() { \/\/ create KQ object kq = CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, 0, 0); assert(kq != NULL);  struct context obj = {}; obj.handler = pipe_handler;  \/\/ create a named pipe obj.p = CreateNamedPipeW(L\"\\\\\\\\.\\\\pipe\\\\iocp-pipe\" , PIPE_ACCESS_DUPLEX | FILE_FLAG_FIRST_PIPE_INSTANCE | FILE_FLAG_OVERLAPPED , PIPE_TYPE_BYTE | PIPE_READMODE_BYTE | PIPE_WAIT , PIPE_UNLIMITED_INSTANCES, 512, 512, 0, NULL); assert(obj.p != INVALID_HANDLE_VALUE);  \/\/ attach pipe to KQ assert(NULL != CreateIoCompletionPort(obj.p, kq, (ULONG_PTR)&amp;obj, 0));  memset(&amp;obj.accept_ctx, 0, sizeof(obj.accept_ctx)); BOOL ok = ConnectNamedPipe(obj.p, &amp;obj.accept_ctx); assert(ok || GetLastError() == ERROR_IO_PENDING);  \/\/ wait for incoming events from KQ and process them for (;;) { OVERLAPPED_ENTRY events[1]; ULONG n = 0; int timeout_ms = -1; \/\/ wait indefinitely BOOL ok = GetQueuedCompletionStatusEx(kq, events, 1, &amp;n, timeout_ms, 0); assert(ok);  \/\/ now process each signalled event for (int i = 0;  i != (int)n;  i++) { struct context *o = (void*)events[i].lpCompletionKey; o->handler(o); \/\/ handle event } }  DisconnectNamedPipe(obj.p); \/\/ close accepted pipe CloseHandle(obj.p); \/\/ close listening pipe CloseHandle(kq); } <\/code><\/pre>\n<p>Note that some functions that return <code>HANDLE<\/code> type value on Windows return <code>NULL<\/code> on error while others return <code>INVALID_HANDLE_VALUE<\/code>.  It&#8217;s a typical design flaw from Microsoft.<\/p>\n<p><a class=\"anchor\" name=\"creating-and-closing-iocp-object\" id=\"creating-and-closing-iocp-object\"><\/a><\/p>\n<h4>Creating and closing IOCP object<\/h4>\n<p>To create a new KQ object we call <code>CreateIoCompletionPort()<\/code> function and we pass <code>INVALID_HANDLE_VALUE<\/code> as the first parameter and <code>NULL<\/code> as the second, which means that we want to create a new KQ object and not use the already existing object.  Remember that it returns <code>NULL<\/code> on error and not <code>INVALID_HANDLE_VALUE<\/code>.  We close this object with <code>CloseHandle()<\/code> as usual.<\/p>\n<pre><code class=\"cpp\">HANDLE kq = CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, 0, 0); ... CloseHandle(kq); <\/code><\/pre>\n<p><a class=\"anchor\" name=\"attaching-socket-descriptor-to-iocp\" id=\"attaching-socket-descriptor-to-iocp\"><\/a><\/p>\n<h4>Attaching file descriptor to IOCP<\/h4>\n<p>To attach a file or socket descriptor to KQ we call the same function <code>CreateIoCompletionPort()<\/code> and pass file descriptor, KQ object and a user object pointer.  Remember that the function name is deceiving in this case: it doesn&#8217;t create a new &#171;completion port&#187; but only registers the file descriptor with it.  And we must not close the file descriptor it returns &#8212; we only check if it&#8217;s <code>NULL<\/code>, which means error.  When registering a socket descriptor rather than a file, an explicit cast to <code>HANDLE<\/code> is required, because <code>SOCKET != HANDLE<\/code>, even though they are essentially of the same pointer type and can be safely cast to <code>void*<\/code>.  Also, a cast to <code>ULONG_PTR<\/code> is required when passing a user object pointer, because <code>ULONG_PTR != void*<\/code>.<\/p>\n<pre><code class=\"cpp\">CreateIoCompletionPort(fd, kq, (ULONG_PTR)obj, 0); \/\/ attach file to KQ CreateIoCompletionPort((HANDLE)sk, kq, (ULONG_PTR)obj, 0); \/\/ attach socket to KQ <\/code><\/pre>\n<p><a class=\"anchor\" name=\"receiving-events-from-iocp\" id=\"receiving-events-from-iocp\"><\/a><\/p>\n<h4>Receiving events from IOCP<\/h4>\n<p>The new function for waiting and receiving events from IOCP is <code>GetQueuedCompletionStatusEx()<\/code> which is available since Windows 7.  Older Windows versions don&#8217;t have this function, so we must use <code>GetQueuedCompletionStatus()<\/code> there.  I don&#8217;t think it will benefit this article to resurrect old Windows, so we don&#8217;t talk about old functions here.  Thankfully, <code>GetQueuedCompletionStatusEx()<\/code> works exactly the same as in kqueue or epoll &#8212; we pass KQ object, array of events and timeout value for how long the function may block.  It returns 1 on success and sets the number of signalled events.<\/p>\n<pre><code class=\"cpp\">OVERLAPPED_ENTRY events[1]; ULONG n = 0; int timeout_ms = -1; \/\/ wait indefinitely BOOL ok = GetQueuedCompletionStatusEx(kq, events, 1, &amp;n, timeout_ms, 0); <\/code><\/pre>\n<p><a class=\"anchor\" name=\"processing-received-events-from-iocp\" id=\"processing-received-events-from-iocp\"><\/a><\/p>\n<h4>Processing received events from IOCP<\/h4>\n<p>Now for each received event we read our object pointer from <code>OVERLAPPED_ENTRY.lpCompletionKey<\/code> field.  Then we determine which event has signalled by reading <code>OVERLAPPED_ENTRY.lpOverlapped<\/code> field and comparing it with the appropriate <code>OVERLAPPED<\/code> object we use for our read and write operations.<\/p>\n<pre><code class=\"cpp\">struct context *o = events[i].lpCompletionKey; if (events[i].lpOverlapped == &amp;o->read_context) \/\/ handle read event else if (events[i].lpOverlapped == &amp;o->write_context) \/\/ handle write event <\/code><\/pre>\n<blockquote>\n<p>I could never understand why the context objects for active I\/O operations are called overlapped objects\/events in Windows.  And I still can&#8217;t comprehend this idea &#8212; what overlaps with what here exactly?<\/p>\n<\/blockquote>\n<p><a class=\"anchor\" name=\"user-triggered-events-with-iocp\" id=\"user-triggered-events-with-iocp\"><\/a><\/p>\n<h4>User-triggered events with IOCP<\/h4>\n<p>Let&#8217;s see how we can post a user event to KQ.<\/p>\n<pre><code class=\"cpp\">\/* Kernel Queue The Complete Guide: iocp-user.c: User-triggered events *\/ #include &lt;windows.h> #include &lt;assert.h> #include &lt;stdio.h>  HANDLE kq;  struct context { void (*handler)(struct context *obj); };  void user_event_handler(struct context *obj) { printf(\"Received user event via IOCP\\n\"); }  void main() { \/\/ create KQ object kq = CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, 0, 0); assert(kq != NULL);  struct context obj = {}; obj.handler = user_event_handler;  assert(0 != PostQueuedCompletionStatus(kq, 0, (ULONG_PTR)&amp;obj, NULL));  \/\/ wait for incoming events from KQ and process them OVERLAPPED_ENTRY events[1]; ULONG n = 0; int timeout_ms = -1; \/\/ wait indefinitely BOOL ok = GetQueuedCompletionStatusEx(kq, events, 1, &amp;n, timeout_ms, 0); assert(ok); assert(n == 1);  struct context *o = (void*)events[0].lpCompletionKey; o->handler(o); \/\/ handle the event  CloseHandle(kq); } <\/code><\/pre>\n<p>There&#8217;s no need to register or prepare a user event as in kqueue or epoll.  We just trigger the event at any time by calling <code>PostQueuedCompletionStatus()<\/code> and we pass any user object pointer to it.<\/p>\n<pre><code class=\"cpp\">PostQueuedCompletionStatus(kq, 0, (ULONG_PTR)obj, NULL); <\/code><\/pre>\n<p><a class=\"anchor\" name=\"establishing-tcp-connection-with-iocp\" id=\"establishing-tcp-connection-with-iocp\"><\/a><\/p>\n<h4>Establishing TCP connection with IOCP<\/h4>\n<p>Now that we start talking about sockets in Windows, you can finally see the whole picture of how difficult it is.  There are many things that seem unnecessary and overcomplicated, and the code looks ugly and bloated.  Here&#8217;s the code for our HTTP\/1 client.<\/p>\n<pre><code class=\"cpp\">\/* Kernel Queue The Complete Guide: iocp-connect.c: HTTP\/1 client Link with -lws2_32 *\/ #include &lt;ws2tcpip.h> #include &lt;mswsock.h> #include &lt;windows.h> #include &lt;assert.h> #include &lt;stdio.h>  HANDLE kq; int quit; LPFN_CONNECTEX KQConnectEx;  \/\/ the structure associated with a socket descriptor struct context { SOCKET sk; void (*rhandler)(struct context *obj); void (*whandler)(struct context *obj); OVERLAPPED read_ctx; OVERLAPPED write_ctx; char sendbuf[1]; int data_offset; };  \/\/ some forward declarations void obj_write(struct context *obj); void obj_read(struct context *obj);  void obj_prepare(struct context *obj) { \/\/ create and prepare socket obj->sk = socket(AF_INET, SOCK_STREAM, 0); assert(obj->sk != INVALID_SOCKET);  \/\/ make socket as non-blocking int nonblock = 1; ioctlsocket(obj->sk, FIONBIO, (unsigned long*)&amp;nonblock);  int val = 1; assert(0 == setsockopt(obj->sk, 0, TCP_NODELAY, (char*)&amp;val, sizeof(int)));  \/\/ attach socket to KQ assert(NULL != CreateIoCompletionPort((HANDLE)obj->sk, kq, (ULONG_PTR)obj, 0));  \/\/ get extended socket function pointers void *func = NULL; DWORD res; GUID guid = WSAID_CONNECTEX; WSAIoctl(obj->sk, SIO_GET_EXTENSION_FUNCTION_POINTER, (void*)&amp;guid, sizeof(GUID), &amp;func, sizeof(void*), &amp;res, 0, 0); KQConnectEx = func; assert(KQConnectEx != NULL); }  void obj_connect(struct context *obj) { if (obj->whandler == NULL) { struct sockaddr_in baddr = {}; baddr.sin_family = AF_INET; char ip4[] = {127,0,0,1}; *(int*)&amp;baddr.sin_addr = *(int*)ip4; assert(0 == bind(obj->sk, (struct sockaddr*)&amp;baddr, sizeof(struct sockaddr_in)));  \/\/ begin asynchronous connection struct sockaddr_in addr = {}; addr.sin_family = AF_INET; addr.sin_port = ntohs(64000); *(int*)&amp;addr.sin_addr = *(int*)ip4; BOOL ok = KQConnectEx(obj->sk, (struct sockaddr*)&amp;addr, sizeof(struct sockaddr_in), NULL, 0, NULL, &amp;obj->write_ctx); assert(ok || GetLastError() == ERROR_IO_PENDING); obj->whandler = obj_connect; return;  } else { DWORD res; BOOL ok = GetOverlappedResult(NULL, &amp;obj->write_ctx, &amp;res, 0); assert(ok); }  printf(\"Connected\\n\"); obj_write(obj); }  void obj_write(struct context *obj) { const char data[] = \"GET \/ HTTP\/1.1\\r\\nHost: hostname\\r\\nConnection: close\\r\\n\\r\\n\"; int r; if (obj->whandler == NULL) { r = send(obj->sk, data + obj->data_offset, sizeof(data)-1 - obj->data_offset, 0); if (r > 0) { \/\/ sent some data  } else if (r &lt; 0 &amp;&amp; GetLastError() == WSAEWOULDBLOCK) { \/\/ the socket's write buffer is full memset(&amp;obj->write_ctx, 0, sizeof(obj->write_ctx)); obj->sendbuf[0] = data[obj->data_offset]; DWORD wr; BOOL ok = WriteFile((HANDLE)obj->sk, obj->sendbuf, 1, &amp;wr, &amp;obj->write_ctx); assert(ok || GetLastError() == ERROR_IO_PENDING); obj->whandler = obj_write; return;  } else { assert(0); \/\/ fatal error }  } else { DWORD res; BOOL ok = GetOverlappedResult(NULL, &amp;obj->write_ctx, &amp;res, 0); assert(ok); r = res; obj->whandler = NULL; }  \/\/ sent some data obj->data_offset += r; if (obj->data_offset != sizeof(data)-1) { \/\/ we need to send the complete request obj_write(obj); return; }  printf(\"Sent HTTP request.  Receiving HTTP response...\\n\"); obj_read(obj); }  void obj_read(struct context *obj) { char data[64*1024]; if (obj->rhandler == NULL) { int r = recv(obj->sk, data, sizeof(data), 0); if (r > 0) { \/\/ received some data printf(\"%.*s\", r, data); obj_read(obj); return;  } else if (r == 0) { \/\/ server has finished sending data  } else if (r &lt; 0 &amp;&amp; GetLastError() == WSAEWOULDBLOCK) { \/\/ the socket's read buffer is empty memset(&amp;obj->read_ctx, 0, sizeof(obj->read_ctx)); DWORD rd; BOOL ok = ReadFile((HANDLE)obj->sk, NULL, 0, &amp;rd, &amp;obj->read_ctx); assert(ok || GetLastError() == ERROR_IO_PENDING); obj->rhandler = obj_read; return; } else { assert(0); \/\/ fatal error }  } else { DWORD res; BOOL ok = GetOverlappedResult(NULL, &amp;obj->read_ctx, &amp;res, 0); assert(ok); obj->rhandler = NULL; obj_read(obj); return; }  quit = 1; }  void main() { \/\/ initialize sockets WSADATA wsa; WSAStartup(MAKEWORD(2, 2), &amp;wsa);  \/\/ create KQ object kq = CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, 0, 0); assert(kq != NULL);  struct context obj = {}; obj_prepare(&amp;obj); obj_connect(&amp;obj);  \/\/ wait for incoming events from KQ and process them while (!quit) { OVERLAPPED_ENTRY events[1]; ULONG n = 0; int timeout_ms = -1; \/\/ wait indefinitely BOOL ok = GetQueuedCompletionStatusEx(kq, events, 1, &amp;n, timeout_ms, 0); assert(ok); assert(n == 1);  \/\/ now process each signalled event for (int i = 0;  i != n;  i++) { struct context *o = (void*)events[i].lpCompletionKey; if (events[i].lpOverlapped == &amp;o->read_ctx) o->rhandler(o); \/\/ handle read event else if (events[i].lpOverlapped == &amp;o->write_ctx) o->whandler(o); \/\/ handle connect\/write event } }  closesocket(obj.sk); CloseHandle(kq); } <\/code><\/pre>\n<p>When we&#8217;re dealing with sockets on Windows, the first thing we need to do is initialize socket subsystem.  We do it by calling <code>WSAStartup()<\/code> function as following:<\/p>\n<pre><code class=\"cpp\">WSADATA wsa; WSAStartup(MAKEWORD(2, 2), &amp;wsa); <\/code><\/pre>\n<p>There are several Windows-specific socket functions that can be used for KQ, but they are not exported by any <code>.dll<\/code> library, so we must get their pointers at runtime using this method:<\/p>\n<pre><code class=\"cpp\">void *func = NULL; DWORD b; GUID guid = WSAID_CONNECTEX; WSAIoctl(sk, SIO_GET_EXTENSION_FUNCTION_POINTER, (void*)&amp;guid, sizeof(GUID), &amp;func, sizeof(void*), &amp;b, 0, 0); LPFN_CONNECTEX KQConnectEx = func; <\/code><\/pre>\n<p>Here, <code>WSAID_CONNECTEX<\/code> is a predefined GUID value (array of bytes, to be specific).  We pass it to <code>WSAIoctl()<\/code> function along with the valid socket descriptor to get the real function pointer, in our case <code>ConnectEx()<\/code> &#8212; the function that asynchronously initiates a new TCP connection.<\/p>\n<p>In order for <code>ConnectEx()<\/code> to work, we must first bind our socket to a local address:<\/p>\n<pre><code class=\"cpp\">struct sockaddr_in addr = {}; addr.sin_family = AF_INET; char ip4[] = {127,0,0,1}; *(int*)&amp;addr.sin_addr = *(int*)ip4; bind(obj->sk, (struct sockaddr*)&amp;addr, sizeof(struct sockaddr_in)); <\/code><\/pre>\n<p>In this example we bind to 127.0.0.1 because we initiate connection to the loopback interface.<\/p>\n<p>And then we can call the function and pass to it remote address and the <code>OVERLAPPED<\/code> object which of course must stay valid until the operation completes.<\/p>\n<pre><code class=\"cpp\">addr = ...; KQConnectEx(sk, (struct sockaddr*)&amp;addr, sizeof(struct sockaddr_in), NULL, 0, NULL, &amp;obj->write_ctx); <\/code><\/pre>\n<p>When IOCP signals about the completion of operation, we get the result with <code>GetOverlappedResult()<\/code>.  Normally it sets the number of transferred bytes upon return, but in our case the number will be 0.<\/p>\n<pre><code class=\"cpp\">DWORD res; BOOL ok = GetOverlappedResult(NULL, &amp;obj->write_ctx, &amp;res, 0); <\/code><\/pre>\n<p><a class=\"anchor\" name=\"writing-data-to-a-tcp-socket-with-iocp\" id=\"writing-data-to-a-tcp-socket-with-iocp\"><\/a><\/p>\n<h4>Writing data to a TCP socket with IOCP<\/h4>\n<p>After the connection is established, we send the request data to server.  And this block of code looks worse than with kqueue or epoll where we just try to read or write some data and return when the functions can&#8217;t fulfil our request.  It&#8217;s not that easy with IOCP, because here we must initiate an ongoing I\/O operation to make IOCP signal us when the state of our socket changes.  These are the steps of the write-data algorithm:<\/p>\n<ol>\n<li>\n<p>The first thing to do is to check whether we already have a result from the previously initiated WRITE operation &#8212; we do it simply by checking the WRITE event handling function <code>whandler<\/code>.<\/p>\n<\/li>\n<li>\n<p>On the first entry the function pointer is unset, so we enter the first branch and try to send some data using common <code>send()<\/code> function.  Unlike on UNIX it doesn&#8217;t return with <code>EAGAIN<\/code> error code if it can&#8217;t complete the request immediately, but on Windows it returns with <code>WSAEWOULDBLOCK<\/code>.<\/p>\n<\/li>\n<li>\n<p>Here&#8217;s where the things get more interesting.  Now we need to emulate the algorithm we have on UNIX to make IOCP signal us when the kernel is ready to accept some more data from us for this socket, but we can&#8217;t do it exactly the UNIX-way.  The solution is that we start an asynchronous WRITE operation which will send just 1 byte for us.  It also means that the next time we are back in our function, <code>send()<\/code> will be able to copy some more data from us.<\/p>\n<\/li>\n<\/ol>\n<p>Why do we send just 1 byte via asynchronous operation and not the whole data we have for sending?  Because the IOCP won&#8217;t signal us at all until the whole data is sent.  If the request is quite large, it&#8217;s more likely that we&#8217;ll hit our own timer signal which will kill the connection (we would think that it&#8217;s stalled).  That&#8217;s why we send only 1 byte this way &#8212; it&#8217;s much more flexible and it effectively emulates UNIX behaviour (which is canonical asynchronous I\/O via KQ in my opinion).  However, if you think you can benefit from sending larger chunk &#8212; go ahead and do it.  Anyway, to send some data asynchronously we call <code>WriteFile()<\/code> and pass to it the data to send along with the <code>OVERLAPPED<\/code> object.  Needless to say that the data buffer associated with the operation must stay valid until its completion.<\/p>\n<ol start=\"4\">\n<li>\n<p>Another interesting moment about asynchronous I\/O with IOCP is that it seems like it always tries to deceive me.  In our case, <code>WriteFile()<\/code> function may return <code>1<\/code> or it may return <code>0<\/code> with <code>ERROR_IO_PENDING<\/code> error, but in fact it&#8217;s all the same to me.  In the first case it says it has completed immediately, while in the second case it says it can&#8217;t complete immediately.  But in both cases IOCP will fire a signal anyway.  I don&#8217;t like the idea of increasing the complexity of my code while trying to support both cases correctly, therefore I just support 1 case &#8212; start an asynchronous operation then wait for its completion.  Even if it&#8217;s completed already &#8212; I still wait for the explicit signal.  It&#8217;s likely that you can adapt your own code to support both cases and save some context-switches and latency &#8212; I encourage you to try.<\/p>\n<\/li>\n<li>\n<p>Now, the final step is handling the result from the previously initiated operation.  We enter the <code>else<\/code> branch in this case and call <code>GetOverlappedResult()<\/code> with the same <code>OVERLAPPED<\/code> object.  It returns the result of the operation and sets the number of bytes sent which is always 1 in our case.<\/p>\n<\/li>\n<\/ol>\n<pre><code class=\"cpp\">if (obj->whandler == NULL) { r = send(obj->sk, data + obj->data_offset, sizeof(data)-1 - obj->data_offset, 0); if (r > 0) { \/\/ sent some data  } else if (r &lt; 0 &amp;&amp; GetLastError() == WSAEWOULDBLOCK) { \/\/ the socket's write buffer is full memset(&amp;obj->write_ctx, 0, sizeof(obj->write_ctx)); obj->sendbuf[0] = data[obj->data_offset]; DWORD wr; BOOL ok = WriteFile((HANDLE)obj->sk, obj->sendbuf, 1, &amp;wr, &amp;obj->write_ctx); assert(ok || GetLastError() == ERROR_IO_PENDING); obj->whandler = obj_write; return;  } else { assert(0); \/\/ fatal error }  } else { DWORD res; BOOL ok = GetOverlappedResult(NULL, &amp;obj->write_ctx, &amp;res, 0); assert(ok); r = res; obj->whandler = NULL; } <\/code><\/pre>\n<p><a class=\"anchor\" name=\"reading-data-from-a-tcp-socket-with-iocp\" id=\"reading-data-from-a-tcp-socket-with-iocp\"><\/a><\/p>\n<h4>Reading data from a TCP socket with IOCP<\/h4>\n<p>Reading data is very similar to writing the data &#8212; the process I described above.  However, unlike what we did earlier with sending 1 byte of data to emulate UNIX behaviour, here we may achieve the exact UNIX behaviour by initiating an operation which reads 0 bytes.  When we do so, IOCP will signal us at the moment the socket buffer becomes non-empty, and we can then read some data from it with <code>recv()<\/code>.  I like this behaviour from IOCP very much &#8212; I wish everything in IOCP would work this way.<\/p>\n<pre><code class=\"cpp\">if (obj->rhandler == NULL) { int r = recv(obj->sk, data, sizeof(data), 0); if (r > 0) { \/\/ received some data printf(\"%.*s\", r, data); obj_read(obj); return;  } else if (r == 0) { \/\/ server has finished sending data  } else if (r &lt; 0 &amp;&amp; GetLastError() == WSAEWOULDBLOCK) { \/\/ the socket's read buffer is empty memset(&amp;obj->read_ctx, 0, sizeof(obj->read_ctx)); DWORD rd; BOOL ok = ReadFile((HANDLE)obj->sk, NULL, 0, &amp;rd, &amp;obj->read_ctx); assert(ok || GetLastError() == ERROR_IO_PENDING); obj->rhandler = obj_read; return; } else { assert(0); \/\/ fatal error }  } else { DWORD res; BOOL ok = GetOverlappedResult(NULL, &amp;obj->read_ctx, &amp;res, 0); assert(ok); obj->rhandler = NULL; obj_read(obj); return; } <\/code><\/pre>\n<p>Note that I call our <code>obj_read()<\/code> function again after receiving some data or after processing the event &#8212; I just think it looks clearer this way than a loop, although I don&#8217;t recommend doing so in real life code, because there&#8217;s a small possibility to crash the app with stack memory overuse.<\/p>\n<p><a class=\"anchor\" name=\"reading-and-writing-data-fromto-a-udp-socket-with-iocp\" id=\"reading-and-writing-data-fromto-a-udp-socket-with-iocp\"><\/a><\/p>\n<h4>Reading and writing data from\/to a UDP socket with IOCP<\/h4>\n<p>I have to explain the difference between reading data from a TCP and UDP sockets with IOCP.  Although we used a beautiful way to get the signal from IOCP while using empty read buffer with <code>ReadFile()<\/code>, unfortunately it works only for TCP sockets.  For UDP sockets we must provide the correct buffer with the maximum packet size, otherwise the message data will be cut (same as in UNIX, by the way).  This buffer must stay valid until the operation completes.  Then, after each asynchronous <code>ReadFile()<\/code> operation completes, I&#8217;m sure it&#8217;s worth calling <code>recv()<\/code> in succession until it fails with <code>WSAEWOULDBLOCK<\/code>.<\/p>\n<pre><code class=\"cpp\">char buffer[64*1024]; ReadFile((HANDLE)udp_socket, buffer, sizeof(buffer), &amp;rd, &amp;obj->read_ctx); <\/code><\/pre>\n<p>It&#8217;s slightly different situation with writing data to UDP sockets.  We must provide the whole buffer with UDP message to <code>send()<\/code> and we never need to issue asynchronous operations with <code>WriteFile()<\/code>.<\/p>\n<pre><code class=\"cpp\">char data[] = ...; send(udp_socket, data, size, 0); <\/code><\/pre>\n<p>Someone once told me that he tried to force <code>WriteFile()<\/code> to return with <code>ERROR_IO_PENDING<\/code> with calling it relentlessly in a tight loop but he couldn&#8217;t do it &#8212; it always succeeded.  That&#8217;s because sending data to a UDP socket just works this way &#8212; the kernel either sends the whole packet or the packet just gets dropped.  After all, it&#8217;s the user code that is responsible for UDP packet delivery &#8212; not the system.<\/p>\n<p><a class=\"anchor\" name=\"io-cancellation-with-iocp\" id=\"io-cancellation-with-iocp\"><\/a><\/p>\n<h4>I\/O Cancellation with IOCP<\/h4>\n<p>Now let me explain why this approach of IOCP with its <code>ReadFile\/WriteFile<\/code> for socket I\/O is worse than the UNIX approach with <code>recv\/send<\/code> and KQ events.  On UNIX, when the kernel can&#8217;t complete my I\/O request immediately, the functions just return with <code>EAGAIN<\/code> error and do nothing else.  And then I must wait for a signal from KQ which means that now the same functions will give me some result.  During the time I wait for a signal all kinds of stuff is happening in parallel, such as timer events, user interaction, signals from other processes, hardware errors, input data corruptions &#8212; all this stuff can easily change the whole picture for my app.  And I want my code to be flexible in order to easily adapt to any changes.<\/p>\n<p>The most basic example is the timer event.  Imagine that while we&#8217;re waiting for a specific socket event from KQ on UNIX, the timer has expired for this connection and I must now close the socket and destroy all data associated with this connection (i.e. destroy its context).  It&#8217;s very easy to do so with epoll or kqueue because I just go ahead and destroy everything immediately except the small portion which prevents from the stale cached events being handled by mistake.  With IOCP, however, the <code>OVERLAPPED<\/code> object and the data associated with the ongoing asynchronous operation must both stay valid until I receive the notifications.  And it&#8217;s just very annoying because why would I want to receive unnecessary notifications from a thousand contexts when I have just initiated the closure of their associated sockets by myself?  How does the notion of <code>receiving an event from a closed socket<\/code> sound in general?  It sounds like nonsense to me.  Maybe a schematic look can make my point clearer:<\/p>\n<pre><code>IOCP I\/O Cancellation ==================== ReadFile(sk, rctx) WriteFile(sk, wctx) ... CloseHandle(sk) ... GetQueuedCompletionStatusEx() handle\/skip wctx event ... GetQueuedCompletionStatusEx() handle\/skip rctx event and destroy object data <\/code><\/pre>\n<p>while the same on UNIX is:<\/p>\n<pre><code>kqueue\/epoll I\/O Cancellation ============================= recv(sk) write(sk) ... close(sk) destroy object data and turn over the safety bit <\/code><\/pre>\n<p>In the end, IOCP&#8217;s design is just different from that of kqueue and epoll.  Maybe it&#8217;s just me, but I have a feeling that there&#8217;s not enough flexibility when I work with IOCP.  It seems harder, it requires more code, more logical branches to support.  But it is what it is.<\/p>\n<p><a class=\"anchor\" name=\"accepting-socket-connections-with-iocp\" id=\"accepting-socket-connections-with-iocp\"><\/a><\/p>\n<h4>Accepting socket connections with IOCP<\/h4>\n<p>Now let&#8217;s see how we can use IOCP to accept incoming socket connections.<\/p>\n<pre><code class=\"cpp\">\/* Kernel Queue The Complete Guide: iocp-accept.c: Accept socket connection Link with -lws2_32 *\/ #include &lt;ws2tcpip.h> #include &lt;mswsock.h> #include &lt;windows.h> #include &lt;assert.h> #include &lt;stdio.h>  HANDLE kq; LPFN_ACCEPTEX KQAcceptEx; LPFN_GETACCEPTEXSOCKADDRS KQGetAcceptExSockaddrs;  struct context { void (*handler)(struct context *obj); OVERLAPPED accept_ctx; unsigned char local_peer_addrs[(sizeof(struct sockaddr_in6) + 16) * 2]; SOCKET client_sock; };  void accept_handler(struct context *obj) { DWORD res; BOOL ok = GetOverlappedResult(NULL, &amp;obj->accept_ctx, &amp;res, 0); assert(ok);  printf(\"Accepted socket connection via IOCP\\n\");  \/\/ get local and peer network address of the accepted socket int len_local = 0, len_peer = 0; struct sockaddr *addr_local, *addr_peer; KQGetAcceptExSockaddrs(obj->local_peer_addrs, 0, sizeof(struct sockaddr_in6) + 16, sizeof(struct sockaddr_in6) + 16, &amp;addr_local, &amp;len_local, &amp;addr_peer, &amp;len_peer);  char buf[1000]; int r = recv(obj->client_sock, buf, 1000, 0); assert(r >= 0);  char data[] = \"HTTP\/1.1 200 OK\\r\\n\\r\\nHello\"; assert(sizeof(data)-1 == send(obj->client_sock, data, sizeof(data)-1, 0)); }  void main() { \/\/ initialize sockets WSADATA wsa; WSAStartup(MAKEWORD(2, 2), &amp;wsa);  \/\/ create KQ object kq = CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, 0, 0); assert(kq != NULL);  struct context obj = {}; obj.handler = accept_handler;  \/\/ create the listening socket SOCKET lsock = socket(AF_INET, SOCK_STREAM, 0); assert(lsock != INVALID_SOCKET);  \/\/ make socket as non-blocking int nonblock = 1; ioctlsocket(lsock, FIONBIO, (unsigned long*)&amp;nonblock);  struct sockaddr_in addr = {}; addr.sin_family = AF_INET; addr.sin_port = ntohs(64000); assert(0 == bind(lsock, (struct sockaddr*)&amp;addr, sizeof(struct sockaddr_in))); assert(0 == listen(lsock, SOMAXCONN));  assert(NULL != CreateIoCompletionPort((HANDLE)lsock, kq, (ULONG_PTR)&amp;obj, 0));  \/\/ get extended socket function pointers void *func = NULL; DWORD res; GUID guid = WSAID_ACCEPTEX; WSAIoctl(lsock, SIO_GET_EXTENSION_FUNCTION_POINTER, (void*)&amp;guid, sizeof(GUID), &amp;func, sizeof(void*), &amp;res, 0, 0); KQAcceptEx = func; func = NULL; GUID guid2 = WSAID_GETACCEPTEXSOCKADDRS; WSAIoctl(lsock, SIO_GET_EXTENSION_FUNCTION_POINTER, (void*)&amp;guid2, sizeof(GUID), &amp;func, sizeof(void*), &amp;res, 0, 0); KQGetAcceptExSockaddrs = func; assert(KQAcceptEx != NULL &amp;&amp; KQGetAcceptExSockaddrs != NULL);  \/\/ try to accept a connection synchronously obj.client_sock = accept(lsock, NULL, 0); assert(obj.client_sock == INVALID_SOCKET &amp;&amp; GetLastError() == WSAEWOULDBLOCK); \/\/ we require this for our example  \/\/ begin asynchronous operation obj.client_sock = socket(AF_INET, SOCK_STREAM, 0); assert(obj.client_sock != INVALID_SOCKET); memset(&amp;obj.accept_ctx, 0, sizeof(obj.accept_ctx)); BOOL ok = KQAcceptEx(lsock, obj.client_sock, obj.local_peer_addrs, 0, sizeof(struct sockaddr_in6) + 16, sizeof(struct sockaddr_in6) + 16, &amp;res, &amp;obj.accept_ctx); assert(ok || GetLastError() == ERROR_IO_PENDING);  \/\/ wait for incoming events from KQ OVERLAPPED_ENTRY events[1]; ULONG n = 0; int timeout_ms = -1; \/\/ wait indefinitely ok = GetQueuedCompletionStatusEx(kq, events, 1, &amp;n, timeout_ms, 0); assert(ok); assert(n == 1);  struct context *o = (void*)events[0].lpCompletionKey; o->handler(o); \/\/ handle socket accept event  closesocket(obj.client_sock); closesocket(lsock); CloseHandle(kq); } <\/code><\/pre>\n<p>First, we call <code>accept()<\/code> as usual &#8212; we can always hope it will return immediately with a new client connetion and we won&#8217;t need to go the long way with IOCP events.<\/p>\n<pre><code class=\"cpp\">csock = accept(lsock, NULL, 0); <\/code><\/pre>\n<p>But if it returns <code>INVALID_SOCKET<\/code> with <code>WSAEWOULDBLOCK<\/code>, it means that there are no clients at the moment and we must wait.  Now we must use IOCP and a couple of new Windows-specific functions.  <code>AcceptEx()<\/code> function begins an asynchronous operation which will signal us via IOCP when a new client connects to our listening socket.  But since it&#8217;s a real asynchronous operation and not just a signal like in kqueue or epoll, we must also provide it with a client socket descriptor created beforehand.  Of course it must stay valid until the operation completes.<\/p>\n<p>Another inconvenient thing is that we need to also supply it with the buffer where local and peer address will be stored.  For some reason we must make 16 bytes more room in our buffer for each address.  Please don&#8217;t mind me allocating the space for IPv6 while using an IPv4 socket.<\/p>\n<pre><code class=\"cpp\">csock = socket(AF_INET, SOCK_STREAM, 0);  unsigned char local_peer_addrs[(sizeof(struct sockaddr_in6) + 16) * 2]; DWORD res; OVERLAPPED ctx = {}; KQAcceptEx(lsock, csock, local_peer_addrs, 0, sizeof(struct sockaddr_in6) + 16, sizeof(struct sockaddr_in6) + 16, &amp;res, &amp;ctx); <\/code><\/pre>\n<p>After we receive a signal from IOCP, our client socket descriptor is a valid connected socket which we would get by calling <code>accept()<\/code>.  The only difference now is that we must get the local and peer address by calling a new special function <code>GetAcceptExSockaddrs()<\/code> which finally returns us <code>struct sockaddr*<\/code> pointers.<\/p>\n<pre><code class=\"cpp\">int len_local = 0, len_peer = 0; struct sockaddr *addr_local, *addr_peer; KQGetAcceptExSockaddrs(local_peer_addrs, 0, sizeof(struct sockaddr_in6) + 16, sizeof(struct sockaddr_in6) + 16, &amp;addr_local, &amp;len_local, &amp;addr_peer, &amp;len_peer); <\/code><\/pre>\n<p>Don&#8217;t forget that after processing each <code>AcceptEx()<\/code> signal we must proceed by calling <code>accept()<\/code> again, and if it fails &#8212; begin a new operation with <code>AcceptEx()<\/code>.  We need so much more stuff to handle here in IOCP rather than very simple behaviour of the asynchronous <code>accept()<\/code> we have with kqueue and epoll.  But this is the reality.<\/p>\n<p><a class=\"anchor\" name=\"system-timer-events-with-iocp\" id=\"system-timer-events-with-iocp\"><\/a><\/p>\n<h4>System timer events with IOCP<\/h4>\n<p>Now let&#8217;s see how we can work with timers and receive timer signals with IOCP.<\/p>\n<pre><code class=\"cpp\">\/* Kernel Queue The Complete Guide: iocp-timer.c: System timer events *\/ #include &lt;windows.h> #include &lt;assert.h> #include &lt;stdio.h>  HANDLE kq; HANDLE tmr; HANDLE evt;  struct context { void (*handler)(struct context *obj); };  void timer_handler(struct context *obj) { static int n; printf(\"Received timer event via IOCP: %d\\n\", n++); }  void __stdcall timer_func(void *arg, DWORD dwTimerLowValue, DWORD dwTimerHighValue) { assert(0 != PostQueuedCompletionStatus(kq, 0, (ULONG_PTR)arg, NULL)); }  int __stdcall timer_thread(void *param) { int period_ms = 1000; long long due_ns100 = (long long)period_ms * 1000 * -10; assert(SetWaitableTimer(tmr, (LARGE_INTEGER*)&amp;due_ns100, period_ms, timer_func, param, 1));  for (;;) { int r = WaitForSingleObjectEx(evt, INFINITE, \/*alertable*\/ 1);  if (r == WAIT_IO_COMPLETION) {  } else if (r == WAIT_OBJECT_0) {  } else { assert(0); break; } } return 0; }  void main() { \/\/ create KQ object kq = CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, 0, 0); assert(kq != NULL);  struct context obj = {}; obj.handler = timer_handler;  \/\/ create timer object tmr = CreateWaitableTimer(NULL, 0, NULL); assert(tmr != NULL);  \/\/ create event object to control the timer thread evt = CreateEvent(NULL, 0, 0, NULL); assert(evt != NULL);  \/\/ start a new thread which will receive timer notifications HANDLE thd = CreateThread(NULL, 0, (PTHREAD_START_ROUTINE)timer_thread, &amp;obj, 0, NULL); assert(thd != NULL);  \/\/ wait for incoming events from KQ and process them for (;;) { OVERLAPPED_ENTRY events[1]; ULONG n = 0; int timeout_ms = -1; \/\/ wait indefinitely BOOL ok = GetQueuedCompletionStatusEx(kq, events, 1, &amp;n, timeout_ms, 0); assert(ok);  struct context *o = (void*)events[0].lpCompletionKey; o->handler(o); \/\/ handle event }  CloseHandle(tmr); \/\/ Note that we should correctly exit the thread with WaitForSingleObject(), but it's OK for our example CloseHandle(thd); CloseHandle(evt); CloseHandle(kq); } <\/code><\/pre>\n<p>There are 3 things we need to do first &#8212; create timer object, create a thread that will process timer events and create an event object for controlling this timer thread.  All 3 functions return <code>NULL<\/code> on error.<\/p>\n<pre><code class=\"cpp\">HANDLE tmr = CreateWaitableTimer(NULL, 0, NULL); HANDLE evt = CreateEvent(NULL, 0, 0, NULL); HANDLE thd = CreateThread(NULL, 0, (PTHREAD_START_ROUTINE)timer_thread, NULL, 0, NULL); <\/code><\/pre>\n<p>Inside the thread function we start the timer by calling <code>SetWaitableTimer()<\/code> and we pass to it the timer interval along with the timer function.<\/p>\n<pre><code class=\"cpp\">int period_ms = 1000; long long due_ns100 = (long long)period_ms * 1000 * -10; assert(SetWaitableTimer(htmr, (LARGE_INTEGER*)&amp;due_ns100, period_ms, timer_func, param, 1)); <\/code><\/pre>\n<p>The timer function looks this way:<\/p>\n<pre><code class=\"cpp\">void __stdcall timer_func(void *arg, DWORD dwTimerLowValue, DWORD dwTimerHighValue) { } <\/code><\/pre>\n<p>The function is called each time the timer interval expires.  But it won&#8217;t be called unless we wait for incoming timer events.  We can&#8217;t do it via IOCP directly &#8212; we need to use another waiting function, for example <code>WaitForSingleObjectEx()<\/code> with <em>alertable state<\/em>.  We also wait for events on our event descriptor we created earlier, but in this example I don&#8217;t have a code which processes it.<\/p>\n<pre><code class=\"cpp\">for (;;) { WaitForSingleObjectEx(evt, INFINITE, \/*alertable*\/ 1); } <\/code><\/pre>\n<p>But the call to our timer function alone won&#8217;t wake up our main thread, so we have to post a user event to it.  And only then we have a working system timer via KQ.  The signal delivery mechanism for a timer can be best explained by this diagram:<\/p>\n<pre><code>[Thread 2]                               [Thread 1]                                          main()                                          |- CreateThread(timer_thread) timer_thread()                           | |- SetWaitableTimer(timer_func)          | |- WaitForSingleObjectEx()               |    |- timer_func()                       |       |- PostQueuedCompletionStatus(obj) |          with obj.handler=timer_handler  |                                          |- GetQueuedCompletionStatusEx()                                             |- timer_handler() <\/code><\/pre>\n<p>But why do we need to post a user event into KQ while we can just process timer events in another thread?  The reason for this is because in real world the timers are often used to close stalled connections.  If we would try to do that directly from the timer thread, we will absolutely require a locking mechanism, because the KQ thread may operate with the same object at this time.  So to simplify our code it&#8217;s much better to do all the work from the same thread, thus eliminating the requirement to use per-object locks.  The second reason for this is to emulate the behaviour of epoll and kqueue, thus making our code look similar everywhere.  The third reason is just pure human logic which tells us that all we need is a single kernel queue mechanism for everything &#8212; we need a way to receive <em>any<\/em> event from the kernel and process it with the same code.  We wouldn&#8217;t need another thread for timer at all if Windows didn&#8217;t force us.<\/p>\n<p><a class=\"anchor\" name=\"asynchronous-file-io-with-iocp\" id=\"asynchronous-file-io-with-iocp\"><\/a><\/p>\n<h4>Asynchronous file I\/O with IOCP<\/h4>\n<p>File I\/O with IOCP looks similar to file AIO API we used with kqueue.  But file AIO on FreeBSD is still better than anything else because it supports unaligned and bufferred I\/O, while file AIO on Linux and Windows require us to switch off kernel disk caching and require us using aligned buffers and offsets.  It&#8217;s interesting that due to the nature of IOCP the code doesn&#8217;t look &#171;off-topic&#187; as with kqueue or epoll (there, file AIO tries to glue asynchronous operations with a pure event delivery mechanism).  So in this situation I can&#8217;t tell anything bad about IOCP because it delivers what it&#8217;s designed for &#8212; asynchronous operations.  Here&#8217;s the code that reads from a file asynchronously.<\/p>\n<pre><code class=\"cpp\">\/* Kernel Queue The Complete Guide: iocp-file.c: Asynchronous file reading Usage: echo Hello file AIO >iocp-file.txt iocp-file *\/ #include &lt;windows.h> #include &lt;assert.h> #include &lt;stdio.h>  HANDLE kq;  struct context { void (*handler)(struct context *obj); HANDLE fd; OVERLAPPED rctx; };  void file_io_result(struct context *obj) { DWORD res; BOOL ok = GetOverlappedResult(NULL, &amp;obj->rctx, &amp;res, 0); if (ok) ; else if (GetLastError() == ERROR_HANDLE_EOF) res = 0; else assert(0);  printf(\"Read from file with IOCP: %d\\n\", res); }  void main() { \/\/ create KQ object kq = CreateIoCompletionPort(INVALID_HANDLE_VALUE, NULL, 0, 0); assert(kq != NULL);  struct context obj = {}; obj.handler = file_io_result;  \/\/ create a named pipe, FILE_FLAG_NO_BUFFERING | FILE_FLAG_OVERLAPPED are mandatory obj.fd = CreateFileW(L\"iocp-file.txt\" , GENERIC_READ, FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE, NULL, OPEN_EXISTING , FILE_ATTRIBUTE_NORMAL | FILE_FLAG_NO_BUFFERING | FILE_FLAG_OVERLAPPED, NULL); assert(obj.fd != INVALID_HANDLE_VALUE);  \/\/ attach file to KQ assert(NULL != CreateIoCompletionPort(obj.fd, kq, (ULONG_PTR)&amp;obj, 0));  void *buf = HeapAlloc(GetProcessHeap(), 0, 4*1024); assert(buf != NULL);  \/\/ begin asynchronous file read operation memset(&amp;obj.rctx, 0, sizeof(obj.rctx)); unsigned int size = 4*1024; unsigned long long off = 0; obj.rctx.Offset = (unsigned int)off; obj.rctx.OffsetHigh = (unsigned int)(off >> 32); BOOL ok = ReadFile(obj.fd, buf, size, NULL, &amp;obj.rctx); assert(GetLastError() != ERROR_HANDLE_EOF); assert(ok || GetLastError() == ERROR_IO_PENDING);  \/\/ asynchronous file reading is in progress, now wait for the signal from KQ OVERLAPPED_ENTRY events[1]; ULONG n = 0; int timeout_ms = -1; \/\/ wait indefinitely ok = GetQueuedCompletionStatusEx(kq, events, 1, &amp;n, timeout_ms, 0); assert(ok);  struct context *o = (void*)events[0].lpCompletionKey; o->handler(o); \/\/ handle read event  HeapFree(GetProcessHeap(), 0, buf); CloseHandle(obj.fd); CloseHandle(kq); } <\/code><\/pre>\n<p>File descriptor must be opened with <code>FILE_FLAG_NO_BUFFERING | FILE_FLAG_OVERLAPPED<\/code> flags, otherwise asynchronous operations won&#8217;t work.<\/p>\n<p>To initiate an asynchronous file reading operation we call <code>ReadFile()<\/code> on a file descriptor and supply the parameters: aligned buffer pointer, aligned maximum number of bytes to read, aligned file offset.  Remember that if the function fails immediately or asynchronously with <code>ERROR_HANDLE_EOF<\/code> error it simply means that we&#8217;ve reached file end.  It&#8217;s always interesting to see how Windows functions are designed: here we pass buffer and size as function parameters, but we pass file offset via <code>OVERLAPPED<\/code> object even though all 3 parameters are bound together for a single operation.<\/p>\n<pre><code class=\"cpp\">void *buf = ...; unsigned int size = ...; unsigned long long off = ...; OVERLAPPED ctx = {}; ctx.Offset = (unsigned int)off; ctx.OffsetHigh = (unsigned int)(off >> 32); ReadFile(fd, buf, size, NULL, &amp;ctx); <\/code><\/pre>\n<p>Writing asynchronously to a file is the same as reading, except we would call <code>WriteFile()<\/code>.  However, writing into a new file (into an unallocated disk space) on NTFS is <strong>always blocking<\/strong> and it isn&#8217;t in fact asynchronous at all.  So you should first allocate enough disk space before writing new data to a file using this method to achieve truely asynchronous behaviour.<\/p>\n<h4>Conclusion<\/h4>\n<p>So I&#8217;ve written here nearly everything that I know about kernel queues and I hope that this guide helped you at least with something.  I&#8217;ve made a new repository on GitHub so that you can easily download all examples and so that we all can improve the code together: <a href=\"https:\/\/github.com\/stsaz\/kernel-queue-the-complete-guide\" rel=\"noopener noreferrer nofollow\">https:\/\/github.com\/stsaz\/kernel-queue-the-complete-guide<\/a>.<\/p>\n<p>I understand that the low-level KQ APIs aren&#8217;t very popular because the developers would rather use some library which handles all these complexities inside than waste their time trying to understand how it all works&#8230;  But come on, since when did we start to ignore the true nature of the things?  If you would sit down and write your own function wrappers for all KQs you will have a small library where you will understand every single line.  You will have a complete experience which will absolutely help you with your future work.  It will make you smarter, it will make you better.  After all, isn&#8217;t it what we are here in this world for?  To make ourselves better with each passing day, to grow in confidence, to grow in experience, in knowledge?<\/p>\n<\/div>\n<\/div>\n<\/div>\n<p><!----><!----><\/div>\n<p><!----><!----><br \/> \u0441\u0441\u044b\u043b\u043a\u0430 \u043d\u0430 \u043e\u0440\u0438\u0433\u0438\u043d\u0430\u043b \u0441\u0442\u0430\u0442\u044c\u0438 <a href=\"https:\/\/habr.com\/ru\/articles\/600123\/\"> https:\/\/habr.com\/ru\/articles\/600123\/<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<div><!--[--><!--]--><\/div>\n<div id=\"post-content-body\">\n<div>\n<div class=\"article-formatted-body article-formatted-body article-formatted-body_version-2\">\n<div xmlns=\"http:\/\/www.w3.org\/1999\/xhtml\">\n<p>When talking about high-performance software we probably think of server software (such as nginx) which processes millions requests from thousands clients in parallel.  Surely, what makes server software work so fast is high-end CPU running with huge amount of memory and a very fast network link.  But even then, the software must utilize these hardware resources at maximum efficiency level, otherwise it will end up wasting the most of the valuable CPU power for unnecessary kernel-user context switching or while waiting for slow I\/O operations to complete.<\/p>\n<p>Thankfully, the Operating Systems have a solution to this problem, and it&#8217;s called <em>kernel event queue<\/em>.  Server software and OS kernel use this mechanism together to achieve minimum latency and maximum scalability (when serving a very large number of clients in parallel).  In this article we are going to talk about <strong>FreeBSD, macOS and kqueue<\/strong>, <strong>Linux and epoll<\/strong>, <strong>Windows and I\/O Completion Ports<\/strong>.  They all have their similarities and differences which we&#8217;re going to discuss here.  The goal of this article is for you to understand the whole mechanism behind kernel queues and to understand how to work with each API.<\/p>\n<p><em>I assume you are already familiar with socket programming and with asynchronous operations, but anyway, in case you think there&#8217;s something I should define or explain in more detail &#8212; send me a message, I&#8217;ll try to update the article.<\/em><\/p>\n<p><em>Although I tried to keep this article clean of any unnecessary sentences (it&#8217;s not a novel, after all), I sometimes can&#8217;t stop myself from expressing my thoughts about something I like or dislike.<\/em><\/p>\n<p>Contents:<\/p>\n<ul>\n<li>\n<p><a href=\"#what-is-kernel-queue\" rel=\"noopener noreferrer nofollow\">What is kernel queue?<\/a> <\/p>\n<ul>\n<li>\n<p><a href=\"#api-principles\" rel=\"noopener noreferrer nofollow\">API Principles<\/a><\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><a href=\"#freebsdmacos-and-kqueue\" rel=\"noopener noreferrer nofollow\">FreeBSD\/macOS and kqueue<\/a> <\/p>\n<ul>\n<li>\n<p><a href=\"#accepting-socket-connections-with-kqueue\" rel=\"noopener noreferrer nofollow\">Accepting socket connections with kqueue<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#creating-and-closing-kqueue-object\" rel=\"noopener noreferrer nofollow\">Creating and closing kqueue object<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#attaching-socket-descriptor-to-kqueue\" rel=\"noopener noreferrer nofollow\">Attaching socket descriptor to kqueue<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#receiving-events-from-kqueue\" rel=\"noopener noreferrer nofollow\">Receiving events from kqueue<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#processing-received-events-from-kqueue\" rel=\"noopener noreferrer nofollow\">Processing received events from kqueue<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#establishing-tcp-connection-with-kqueue\" rel=\"noopener noreferrer nofollow\">Establishing TCP connection with kqueue<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#processing-stale-cached-events\" rel=\"noopener noreferrer nofollow\">Processing stale cached events<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#user-triggered-events-with-kqueue\" rel=\"noopener noreferrer nofollow\">User-triggered events with kqueue<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#system-timer-events-with-kqueue\" rel=\"noopener noreferrer nofollow\">System timer events with kqueue<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#unix-signals-from-kqueue\" rel=\"noopener noreferrer nofollow\">UNIX signals from kqueue<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#asynchronous-file-io-with-kqueue\" rel=\"noopener noreferrer nofollow\">Asynchronous file I\/O with kqueue<\/a><\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><a href=\"#linux-and-epoll\" rel=\"noopener noreferrer nofollow\">Linux and epoll<\/a> <\/p>\n<ul>\n<li>\n<p><a href=\"#accepting-socket-connections-with-epoll\" rel=\"noopener noreferrer nofollow\">Accepting socket connections with epoll<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#creating-and-closing-epoll-object\" rel=\"noopener noreferrer nofollow\">Creating and closing epoll object<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#attaching-socket-descriptor-to-epoll\" rel=\"noopener noreferrer nofollow\">Attaching socket descriptor to epoll<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#receiving-events-from-epoll\" rel=\"noopener noreferrer nofollow\">Receiving events from epoll<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#processing-received-events-from-epoll\" rel=\"noopener noreferrer nofollow\">Processing received events from epoll<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#establishing-tcp-connection-with-epoll\" rel=\"noopener noreferrer nofollow\">Establishing TCP connection with epoll<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#user-triggered-events-with-epoll\" rel=\"noopener noreferrer nofollow\">User-triggered events with epoll<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#system-timer-events-with-epoll\" rel=\"noopener noreferrer nofollow\">System timer events with epoll<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#unix-signals-from-epoll\" rel=\"noopener noreferrer nofollow\">UNIX signals from epoll<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#asynchronous-file-io-with-epoll\" rel=\"noopener noreferrer nofollow\">Asynchronous file I\/O with epoll<\/a><\/p>\n<\/li>\n<\/ul>\n<\/li>\n<li>\n<p><a href=\"#windows-and-i-o-completion-ports\" rel=\"noopener noreferrer nofollow\">Windows and I\/O Completion Ports<\/a> <\/p>\n<ul>\n<li>\n<p><a href=\"#accepting-connections-to-a-named-pipe-with-iocp\" rel=\"noopener noreferrer nofollow\">Accepting connections to a named pipe with IOCP<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#creating-and-closing-iocp-object\" rel=\"noopener noreferrer nofollow\">Creating and closing IOCP object<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#attaching-socket-descriptor-to-iocp\" rel=\"noopener noreferrer nofollow\">Attaching file descriptor to IOCP<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#receiving-events-from-iocp\" rel=\"noopener noreferrer nofollow\">Receiving events from IOCP<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#processing-received-events-from-iocp\" rel=\"noopener noreferrer nofollow\">Processing received events from IOCP<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#user-triggered-events-with-iocp\" rel=\"noopener noreferrer nofollow\">User-triggered events with IOCP<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#establishing-tcp-connection-with-iocp\" rel=\"noopener noreferrer nofollow\">Establishing TCP connection with IOCP<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#writing-data-to-a-tcp-socket-with-iocp\" rel=\"noopener noreferrer nofollow\">Writing data to a TCP socket with IOCP<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#reading-data-from-a-tcp-socket-with-iocp\" rel=\"noopener noreferrer nofollow\">Reading data from a TCP socket with IOCP<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#reading-and-writing-data-fromto-a-udp-socket-with-iocp\" rel=\"noopener noreferrer nofollow\">Reading and writing data from\/to a UDP socket with IOCP<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#io-cancellation-with-iocp\" rel=\"noopener noreferrer nofollow\">I\/O Cancellation with IOCP<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#accepting-socket-connections-with-iocp\" rel=\"noopener noreferrer nofollow\">Accepting socket connections with IOCP<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#system-timer-events-with-iocp\" rel=\"noopener noreferrer nofollow\">System timer events with IOCP<\/a><\/p>\n<\/li>\n<li>\n<p><a href=\"#asynchronous-file-io-with-iocp\" rel=\"noopener noreferrer nofollow\">Asynchronous file I\/O with IOCP<\/a><\/p>\n<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><a class=\"anchor\" name=\"what-is-kernel-queue\" id=\"what-is-kernel-queue\"><\/a><\/p>\n<h3>What is kernel queue?<\/h3>\n<p>Kernel event queue (which I&#8217;m gonna call KQ from now on) is a <strong>fast signal-delivery mechanism<\/strong> which allows server software to process events from OS in a very effective way.  KQ is a bunch of data living in kernel memory and a bunch of kernel code that operates with this data to notify a user-level application about various system events.  A user app can&#8217;t access KQ data directly (it&#8217;s managed by kernel) and so it operates with KQ via the API that OS provides.  There are 3 different API we&#8217;re going to use here: kqueue, epoll, IOCP.  However, this section describes kernel queues in general so the API doesn&#8217;t matter for now.<\/p>\n<p>Because the main purpose of KQ is to deliver notifications from network sockets, let me formulate the key idea in a different way:<\/p>\n<blockquote>\n<p>A user application wants to be notified when any of its sockets is ready to read or write some data, and the OS kernel serves this purpose by maintaining the list of all registered and signalled events.<\/p>\n<\/blockquote>\n<h4>Use-case N1<\/h4>\n<p>What an application achieves through KQ technology is that the app is notified about an I\/O signal  such as when a <strong>network packet is received<\/strong>.  For example:<\/p>\n<ol start=\"0\">\n<li>\n<p>Suppose some user app created a UDP socket and registered it with a KQ along with some app-defined data (i.e. cookie).<\/p>\n<\/li>\n<li>\n<p>At some point the last chunk of a UDP packet is received by network device.<\/p>\n<\/li>\n<li>\n<p>OS now has a complete UDP packet and is ready to notify the user process as soon as it calls the KQ waiting function.<\/p>\n<\/li>\n<li>\n<p>At some time the user app calls KQ waiting function which tells the kernel: <code>Give me something new<\/code>.<\/p>\n<\/li>\n<li>\n<p>OS responds with <code>Got a READ event from the socket associated with your cookie<\/code>.<\/p>\n<\/li>\n<li>\n<p>This cookie is the object pointer which the app then uses to handle the signal &#8212; <em>read a message from UDP socket<\/em>, in our case.<\/p>\n<\/li>\n<\/ol>\n<blockquote>\n<p>Note that neither the opening of a socket, neither reading from a socket after the signal is received isn&#8217;t normally the part of KQ mechanism.  On UNIX we always use conventional socket functions and we use KQ functions to receive events associated with sockets.  However, IOCP on Windows is different.  There, I\/O functions and their associated events are a part of a single mechanism.  Anyway, we&#8217;ll deal with IOCP later, so for now just don&#8217;t bother with it &#8212; let us always think by default that KQ just delivers signals.<\/p>\n<\/blockquote>\n<h4>Use-case N2<\/h4>\n<p>Consider the next example where the user app receives a <strong>notification after a TCP socket connects to its peer<\/strong>:<\/p>\n<ol start=\"0\">\n<li>\n<p>User app creates a TCP socket and registers it with a KQ along with some app-defined data (i.e. cookie).<\/p>\n<\/li>\n<li>\n<p>Now the app begins the procedure to connect to a remote host.  Obviously, this operation can&#8217;t finish immediately most of the time, because it takes some time to transmit 2 TCP packets needed for TCP connection.  Moreover, what if the network link is very busy and the packets get dropped?  Needless to say that TCP connection may take a long time to finish.  Because of that, OS returns the control back to the app with the result <code>Can't finish the operation immediately<\/code>.  While packets are being sent and received, the app keeps doing some other stuff, relying on OS to do its best to complete the connection procedure.<\/p>\n<\/li>\n<li>\n<p>Finally, a <code>SYN+ACK<\/code> TCP packet is received from the remote host, which means it&#8217;s willing to establish a TCP connection with our app.  Now OS is ready to signal the app as soon as the latter becomes ready.<\/p>\n<\/li>\n<li>\n<p>At some point the user app calls the KQ waiting function which tells the kernel: <code>Give me something new<\/code>.<\/p>\n<\/li>\n<li>\n<p>OS responds with <code>Got a WRITE event from the socket associated with your cookie<\/code>.<\/p>\n<\/li>\n<li>\n<p>This cookie is the object pointer which the app then uses to handle the signal &#8212; <em>write some data to the TCP socket<\/em>, in our case.<\/p>\n<\/li>\n<\/ol>\n<p>Although the primary use of KQ is I\/O event notifications, it also can be used for other purposes, for example KQ can notify when a child process signals its parent (i.e. <strong>UNIX signals delivery<\/strong>), or KQ can be used to receive <strong>notifications from a system timer<\/strong>.  I also explain these use-cases and show the example code in this article.<\/p>\n<h4>Internal representation example<\/h4>\n<p>Let&#8217;s see a diagram with an example of how KQ may look like internally after a user app has registered 6 different events there (user-triggered event, I\/O events, system timer), 3 of which have signalled already.<\/p>\n<pre><code>         KQ table example ================================= Event    | Descriptor | Signalled? ---------+------------+----------- USER     | #789       | READ     | #1         | READ     | #2         | yes WRITE    | #2         | yes WRITE    | #3         | TIMER    | #456       | yes <\/code><\/pre>\n<p>In this example, both READ and WRITE events for socket #2 are in signalled state which means we can read and write data from\/to this socket.  And the timer event is in signalled state too which means the system timer interval has expired.  The signalled flag also means that after a user app calls the function to receive events from KQ, it will receive an array of these 3 signalled events so it can process them.  The kernel then may clear the signalled flag so that it won&#8217;t deliver the same signals over and over again unless necessary.<\/p>\n<p>Of course in reality KQ is much more complex but we don&#8217;t need to know exactly how the KQ is implemented internally &#8212; we need just to understand what and when it delivers to us and how me may use it effectively.  I&#8217;m not a kernel developer so I don&#8217;t know much about how it&#8217;s implemented inside &#8212; you have to read some Linux\/FreeBSD kernel manuals and epoll\/kqueue code if you are interested in this subject.<\/p>\n<p><a class=\"anchor\" name=\"api-principles\" id=\"api-principles\"><\/a><\/p>\n<h4>API Principles<\/h4>\n<p>Now let&#8217;s talk about what features all those API provide us with.  In general, working with a KQ API consists of 4 steps:<\/p>\n<ol>\n<li>\n<p><strong>Create KQ object<\/strong>.  It&#8217;s the easiest part, where we just call a function which returns the descriptor for our new KQ.  We may create KQ objects as many as we want, but I don&#8217;t see the point of creating more than 1 per process or thread.<\/p>\n<\/li>\n<li>\n<p><strong>Attach file\/socket descriptor<\/strong> along with opaque user data to KQ.  We have to tell the OS that we want it to notify us about any particular descriptor through a particular KQ object.  How else the kernel should know what to notify us about?  Here we also associate some data with the descriptor, which is usually a pointer to some kind of a structure object.  How else are we going to handle the received signal?  The attachment is needed only once for each descriptor, usually it&#8217;s done right after the descriptor is configured and ready for I\/O operations (though we can delay that until absolutely necessary to probably save a context switch).  The detachment procedure usually is not needed (with the right design), so we won&#8217;t even talk about it here.<\/p>\n<\/li>\n<li>\n<p><strong>Wait for incoming events from KQ<\/strong>.  When the user app has nothing more important to do, it calls a KQ waiting function.  We specify the output array of events and timeout value as parameters when calling this function.  It fills our array with the information about which events signalled and how they signalled.  By using an array of events rather than a single event we save CPU time on somewhat costly kernel-userspace context switches.  By using timeout value we control how much time<\/p>\n<\/li>\n<\/ol>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-414110","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/414110","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=414110"}],"version-history":[{"count":0,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=\/wp\/v2\/posts\/414110\/revisions"}],"wp:attachment":[{"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=414110"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=414110"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/savepearlharbor.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=414110"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}