Library

Networking — sockets and HTTP

TCP streams, listeners and the in-language HTTP request and response model, all driven by the event loop.

since 0.1.0-alpha.1linux

Description

Networking is built from the stream model (see lang.streams) and the event loop (see lang.event-loop). Nothing blocks: a socket is read through callbacks, and the loop calls them when data arrives.

Sockets. A TcpStream is a connected socket. You send with send(text) or <<, receive with onData(callback), learn that the peer went away with onClose(callback), and release it with close(). A TcpListener is a listening socket seen as a stream of connections: connections(callback) hands a new TcpStream to the callback for every client. Both are described in their own entries.

Connecting. std::connectTimeout(host, port, ms, callback) opens a connection without blocking and calls callback with the connected descriptor, ready to wrap in a TcpStream, or with -1 if the connection was refused, the host was unreachable or malformed, or ms milliseconds passed first. A host written as an IPv6 literal (anything containing :) connects over IPv6.

Sending never silently loses data. The socket is non-blocking, so a large payload may not fit in one write. send queues the part the kernel did not take and drains it as room appears. flush() is for the rare caller that wants to know: it suspends until everything queued has been handed to the transport, and returns true once it has, or false if the connection closed or failed first. It is a barrier on the local queue, not a confirmation from the peer.

HTTP is written in the language on top of those streams, so its building blocks are plain objects you can create and inspect without any network:

HeaderMap is an ordered, case-insensitive multimap of headers. add appends, set replaces every entry of that name and appends once, first(name) returns string?, all(name) returns every value, has, remove, entries(), and render() produce the wire text. Because order and duplicates are kept, several Set-Cookie headers survive.
HttpRequest has method, path, version, body and headers, and header(name) (an empty string when absent). parse(raw) reads a request that is already buffered, and feed(chunk) reads one incrementally: it returns true once the head and the whole body (as declared by Content-Length) have arrived.
HttpResponse(status, body) has headers, withHeader(name, value), reason(), render() and parse(raw). render() computes Content-Length and Connection, and drops any Content-Length, Transfer-Encoding or Connection header you set, so the values on the wire are always consistent. parse decodes a Transfer-Encoding: chunked body for you.
HttpResponse::ofStream(status, headers, writer) is the streaming body mode. The server sends a Transfer-Encoding: chunked head and then calls writer with a live ChunkedSink. sink.write(data) sends one chunk and suspends until the transport has taken it (so a slow client slows the writer down), sink.end() finishes the body, sink.onClose(callback) runs if the connection ends prematurely, and sink.isClosed() says whether the sink is finished either way. write and end after the sink is finished do nothing, and calls that overlap are kept in order so chunks never interleave. The writer may return and keep the sink open for later writes from a timer or subscription, which is the shape of server-sent events. isStreaming() tells the two modes apart. render() throws on a streaming response, because only HttpServer may drive it.
ChunkedDecoder decodes chunked transfer encoding from fragments of any size (feed(chunk), isDone). std::chunkEncode(data) and std::chunkEnd() are the encoder side.

Headers: ordered, case-insensitive, duplicates kept

HeaderMap h = HeaderMap();
h.add("Set-Cookie", "a=1").add("Set-Cookie", "b=2");
h.set("Content-Type", "text/plain");
console.writeln(h.has("content-type"));
string? ct = h.first("CONTENT-TYPE");
console.writeln(ct ?? "none");
console.writeln(h.all("set-cookie").length());
h.set("Content-Type", "text/html");
console.writeln(h.length());
h.remove("Set-Cookie");
console.writeln(h.render().replace("\r\n", "|"));

Run it ▶

true
text/plain
2
3
Content-Type: text/html|

Rules

Text only. HTTP bodies are strings; binary bodies are not supported yet.
One connection per client request. HttpClient opens a connection per request and sends Connection: close. There is no connection pooling.
Keep-alive on the server. HttpServer keeps a connection open for further requests unless the request sent Connection: close or the handler failed, up to 100 requests per connection. A finite streamed response keeps the connection only after its last chunk has drained.
No pipelining. If bytes arrive while a response is still being sent, the server closes the connection.
Handler errors become responses. An uncaught exception in a handler becomes a 500 response with Connection: close, and the server keeps running.
Bodiless responses. For a HEAD request, or a 1xx, 204 or 304 status, a streaming response sends only its head and never calls the writer. Chunked streaming needs HTTP/1.1: a streaming response to a request of any other version is replaced by 505 and the connection is closed.
Secure connections use the same objects; see lang.tls.

Examples

A request can be parsed from text, whether it arrives all at once or in fragments:

Parsing HTTP requests

HttpRequest req = HttpRequest();
req.parse("POST /submit?x=1 HTTP/1.1\r\nHost: example.com\r\nContent-Length: 5\r\n\r\nhello");
console.writeln(req.method);
console.writeln(req.path);
console.writeln(req.version);
console.writeln(req.header("host"));
console.writeln(req.body);
string absent = req.header("missing");
console.writeln("missing header length: ${absent.length()}");

HttpRequest inc = HttpRequest();
console.writeln(inc.feed("GET /a HTTP/1.1\r\nHo"));
console.writeln(inc.feed("st: x\r\n\r\n"));
console.writeln(inc.path);

Run it ▶

POST
/submit?x=1
HTTP/1.1
example.com
hello
missing header length: 0
false
true
/a

A response renders to the exact text sent on the wire. Note that the caller's own Content-Length: 999 was dropped and the correct 5 computed. Parsing decodes chunked bodies:

Rendering, parsing and chunked bodies

HttpResponse ok = HttpResponse(200, "hello");
ok.withHeader("Content-Type", "text/plain").withHeader("Content-Length", "999");
string wire = ok.render();
console.writeln(wire.replace("\r\n", "<CRLF>\n"));
console.writeln(HttpResponse(404, "").reason());

HttpResponse parsed = HttpResponse(0, "");
parsed.parse("HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n5\r\nhello\r\n6\r\n world\r\n0\r\n\r\n");
console.writeln(parsed.status);
console.writeln(parsed.body);
console.writeln(std::chunkEncode("abc").replace("\r\n", "<CRLF>"));
console.writeln(std::chunkEnd().replace("\r\n", "<CRLF>"));

Run it ▶

HTTP/1.1 200 OK<CRLF>
Content-Length: 5<CRLF>
Connection: close<CRLF>
Content-Type: text/plain<CRLF>
<CRLF>
hello
Not Found
200
hello world
3<CRLF>abc<CRLF>
0<CRLF><CRLF>

A streaming response is a different mode, and only a server may render it:

Streaming responses

HeaderMap h = HeaderMap();
h.set("Content-Type", "text/event-stream");
HttpResponse live = HttpResponse::ofStream(200, h, (sink) => {
    sink.write("data: hi\n\n");
    sink.end();
});
console.writeln(live.isStreaming());
console.writeln(HttpResponse(200, "x").isStreaming());
try {
    live.render();
} catch (RuntimeException e) {
    console.writeln(e.message);
}

Run it ▶

true
false
render() called on a streaming HttpResponse

Opening a connection with a deadline:

std::connectTimeout("example.com", 80, 2000, (fd) => {
    if (fd < 0) {
        console.writeln("could not connect");
        return;
    }
    TcpStream conn = TcpStream(fd);
    conn.onData((chunk) => console.writeln("received ${chunk.length()} bytes"));
    conn.onClose(() => console.writeln("peer closed"));
    conn << "GET / HTTP/1.0\r\n\r\n";
});

not run — needs the network

Notes

Not implemented: client-side redirects, parsing a URL string into host, port and path (the client takes them separately), a request timeout, request pipelining, chunked uploads from the client, and client-side connection pooling.

Networking — sockets and HTTP

Description

Rules

Examples

Notes

See also