mirror of
https://github.com/yhirose/cpp-httplib.git
synced 2026-10-13 06:43:17 +00:00
Fix docs pages that no longer match the code
Samples that did not compile or run as shown: - res.user_data.get<T>() inside a generic lambda needs the `template` keyword; use explicit parameter types (tour 09, cookbook s15). - listen() on a Unix domain socket fails with port 0 (tour 09, s22). - "*.dev.local" is not a NO_PROXY pattern (c16). - ssl_backend_error() holds a verify result, not an ERR_get_error() value, after a verification failure; decode each with the matching OpenSSL function (c18). - The content provider's `length` is everything that remains, so the sample read the whole file in one call (s05). Statements corrected: - Client keep-alive is off by default; c14 is rewritten around set_keep_alive(true). - Mounted files are looked up before GET handlers (tour 04, s04). - Params keep insertion order, and to_string(Error::Connection) reads "Could not establish connection" (tour 02). - A chunked provider ends with sink.done(), and post_routing_handler runs before the response is sent (tour 09). - Timeouts surface as Error::Read; Error::Timeout comes from the stream API (c17). The max timeout cuts off the wait for the response only (c13). The progress callback needs Content-Length (c11). - Encoding selection follows q-values, then Brotli, gzip, Zstd (s08), and the client compresses with the first of those it was built with (c15). - stop() cuts a provider-driven response short (s19); a rejected content_reader already gets 400 or 413 (s07); user_data values must be copyable (s12); Client accepts a client certificate too (t04); on_message() is the fallback for every unhandled event and 204/403/404 end reconnection (e04); the pong timeout takes two to three intervals and ends a waiting read() (w02). In the LLM app tutorial, an uncaught exception does not crash the server, so say what it does instead. Drop the server and client timeout settings whose stated purpose, covering inference and download time, they do not serve: those timeouts bound a single socket wait. Update the llama.cpp server layout in chapter 7.
This commit is contained in:
yhirose
@@ -17,7 +17,7 @@ auto res = cli.Post("/api/users", j.dump(), "application/json");
|
||||
|
||||
Pass the JSON string as the second argument to `Post()` and the Content-Type as the third. The same pattern works with `Put()` and `Patch()`.
|
||||
|
||||
> **Warning:** If you omit the Content-Type (the third argument), the server may not recognize the body as JSON. Always specify `"application/json"`.
|
||||
> **Warning:** If the Content-Type (the third argument) is anything other than `"application/json"`, the server may not recognize the body as JSON. Always specify `"application/json"`.
|
||||
|
||||
## Receive a JSON response
|
||||
|
||||
|
||||
@@ -21,7 +21,7 @@ auto res = cli.Get("/large-file",
|
||||
std::cout << std::endl;
|
||||
```
|
||||
|
||||
The callback fires each time data arrives. `total` comes from the Content-Length header — if the server doesn't send one, it may be `0`. In that case, you can't compute a percentage, so just display bytes received.
|
||||
The callback fires each time data arrives. `total` comes from the Content-Length header. For a response without one (chunked transfer, for example), the progress callback is not called at all.
|
||||
|
||||
## Upload progress
|
||||
|
||||
@@ -54,6 +54,8 @@ auto res = cli.Get("/large-file",
|
||||
});
|
||||
```
|
||||
|
||||
> **Note:** A response without Content-Length never calls the progress callback, so it cannot be cancelled this way. Return `false` from a `ContentReceiver` instead.
|
||||
|
||||
> **Note:** `ContentReceiver` and the progress callback can be used together. When you want to stream to a file and show progress at the same time, pass both.
|
||||
|
||||
> For a concrete example of saving to a file, see [C01. Get the response body / save to a file](../c01-get-response-body).
|
||||
|
||||
@@ -16,7 +16,7 @@ cli.set_max_timeout(5000); // 5 seconds (in milliseconds)
|
||||
auto res = cli.Get("/slow-endpoint");
|
||||
```
|
||||
|
||||
The value is in milliseconds. Connection, send, and receive together — the whole request is aborted if it exceeds the limit.
|
||||
The value is in milliseconds. Once that much time has passed since the request started, waiting for the response is cut off. The limit does not shorten the connection and write waits themselves, so bound those with `set_connection_timeout` and `set_write_timeout`.
|
||||
|
||||
## Use `std::chrono`
|
||||
|
||||
|
||||
@@ -4,30 +4,29 @@ order: 14
|
||||
status: "draft"
|
||||
---
|
||||
|
||||
When you send multiple requests through the same `httplib::Client` instance, the TCP connection is reused automatically. HTTP/1.1 Keep-Alive does the work for you — you don't pay the TCP and TLS handshake cost on every call.
|
||||
By default, `httplib::Client` closes the connection after every request (it sends `Connection: close`). Call `set_keep_alive(true)` and the requests you send through the same instance share one TCP connection, so you don't pay the TCP and TLS handshake cost on every call.
|
||||
|
||||
## Connections are reused automatically
|
||||
## Enable Keep-Alive
|
||||
|
||||
```cpp
|
||||
httplib::Client cli("https://api.example.com");
|
||||
cli.set_keep_alive(true);
|
||||
|
||||
auto res1 = cli.Get("/users/1");
|
||||
auto res2 = cli.Get("/users/2"); // reuses the same connection
|
||||
auto res3 = cli.Get("/users/3"); // reuses the same connection
|
||||
```
|
||||
|
||||
No special config required. Just hold on to `cli` — internally, the socket stays open across calls. The effect is especially noticeable over HTTPS, where the TLS handshake is expensive.
|
||||
After that, just hold on to `cli`. Internally, the socket stays open across calls. The effect is especially noticeable over HTTPS, where the TLS handshake is expensive.
|
||||
|
||||
## Disable Keep-Alive explicitly
|
||||
## Turn Keep-Alive back off
|
||||
|
||||
To force a fresh connection every time, call `set_keep_alive(false)`. Mostly useful for testing.
|
||||
To go back to a fresh connection every time, call `set_keep_alive(false)`. This is the default behavior.
|
||||
|
||||
```cpp
|
||||
cli.set_keep_alive(false);
|
||||
```
|
||||
|
||||
For normal use, leave it on (the default).
|
||||
|
||||
## Don't create a `Client` per request
|
||||
|
||||
If you create a `Client` inside a loop and let it fall out of scope each iteration, you lose the reuse benefit. Create the instance outside the loop.
|
||||
@@ -36,11 +35,13 @@ If you create a `Client` inside a loop and let it fall out of scope each iterati
|
||||
// Bad: a new connection every iteration
|
||||
for (auto id : ids) {
|
||||
httplib::Client cli("https://api.example.com");
|
||||
cli.set_keep_alive(true);
|
||||
cli.Get("/users/" + id);
|
||||
}
|
||||
|
||||
// Good: the connection is reused
|
||||
httplib::Client cli("https://api.example.com");
|
||||
cli.set_keep_alive(true);
|
||||
for (auto id : ids) {
|
||||
cli.Get("/users/" + id);
|
||||
}
|
||||
@@ -50,4 +51,4 @@ for (auto id : ids) {
|
||||
|
||||
If you want to send requests in parallel from multiple threads, give each thread its own `Client` instance. A single `Client` uses a single TCP connection, so firing concurrent requests at the same instance ends up serializing them anyway.
|
||||
|
||||
> **Note:** If the server closes the connection after its Keep-Alive timeout, cpp-httplib reconnects and retries transparently. You don't need to handle this in application code.
|
||||
> **Note:** If the server closes the connection after its Keep-Alive timeout, cpp-httplib notices before it sends the next request and reconnects. You don't need to handle this in application code.
|
||||
|
||||
@@ -28,7 +28,7 @@ std::string big_payload = build_payload();
|
||||
auto res = cli.Post("/api/data", big_payload, "application/json");
|
||||
```
|
||||
|
||||
With `set_compress(true)`, the body of POST or PUT requests gets gzipped before sending. The server needs to handle compressed bodies too.
|
||||
With `set_compress(true)`, the body of POST or PUT requests is compressed before sending. The encoding is the first one enabled in your build, in the order Brotli, gzip, Zstd. The server needs to handle compressed bodies too.
|
||||
|
||||
## Decompress the response
|
||||
|
||||
@@ -44,4 +44,4 @@ With `set_decompress(true)`, the client automatically decompresses responses tha
|
||||
|
||||
It's on by default, so normally you don't need to do anything. Set it to `false` only if you want the raw compressed bytes.
|
||||
|
||||
> **Warning:** If you build without `CPPHTTPLIB_ZLIB_SUPPORT`, calling `set_compress()` or `set_decompress()` does nothing. If compression isn't working, check the macro definition first.
|
||||
> **Warning:** If you build without any compression library, `set_compress(true)` leaves the request uncompressed. And a response compressed with an encoding your build lacks makes the request fail with `Error::UnsupportedContentEncoding`. If compression isn't working, check the macro definitions first.
|
||||
|
||||
@@ -55,7 +55,7 @@ You often want internal endpoints to skip the proxy. Configure a bypass list wit
|
||||
|
||||
```cpp
|
||||
cli.set_proxy("proxy.internal", 8080);
|
||||
cli.set_no_proxy({"internal.corp", "10.0.0.0/8", "*.dev.local"});
|
||||
cli.set_no_proxy({"internal.corp", "10.0.0.0/8", ".dev.local"});
|
||||
```
|
||||
|
||||
Each entry is one of:
|
||||
|
||||
@@ -29,8 +29,8 @@ Use `if (res)` to check success. On failure, `res.error()` returns a `httplib::E
|
||||
| --- | --- |
|
||||
| `Error::Connection` | Couldn't connect to the server |
|
||||
| `Error::ConnectionTimeout` | Connection timeout (`set_connection_timeout`) |
|
||||
| `Error::Read` / `Error::Write` | Error during send or receive |
|
||||
| `Error::Timeout` | Overall timeout set via `set_max_timeout` |
|
||||
| `Error::Read` / `Error::Write` | Error during send or receive. A timeout from `set_read_timeout` or `set_max_timeout` is also reported as `Error::Read` |
|
||||
| `Error::Timeout` | A body read timed out in `stream::Get()` or `SSEClient` |
|
||||
| `Error::ExceedRedirectCount` | Too many redirects |
|
||||
| `Error::SSLConnection` | TLS handshake failed |
|
||||
| `Error::SSLServerVerification` | Server certificate verification failed |
|
||||
|
||||
@@ -24,19 +24,24 @@ if (!res) {
|
||||
}
|
||||
```
|
||||
|
||||
`ssl_error()` returns the error code from the SSL library (e.g., OpenSSL's `SSL_get_error()`). `ssl_backend_error()` gives you the backend's more detailed error value — for OpenSSL, that's `ERR_get_error()`.
|
||||
`ssl_error()` is a backend-independent TLS error category: an `httplib::tls::ErrorCode` cast to `int`. `ssl_backend_error()` gives you the backend's own error value. With OpenSSL that is `ERR_get_error()` when the handshake failed, and the verify result (`X509_V_ERR_*`) when certificate verification failed.
|
||||
|
||||
## Format OpenSSL errors as strings
|
||||
|
||||
When you have a value from `ssl_backend_error()`, pass it to OpenSSL's `ERR_error_string()` to get a readable message.
|
||||
When you have a value from `ssl_backend_error()`, pass it to the OpenSSL function that matches the kind of failure to get a readable message.
|
||||
|
||||
```cpp
|
||||
#include <openssl/err.h>
|
||||
#include <openssl/x509.h>
|
||||
|
||||
if (res.ssl_backend_error() != 0) {
|
||||
if (res.error() == httplib::Error::SSLConnection) {
|
||||
char buf[256];
|
||||
ERR_error_string_n(res.ssl_backend_error(), buf, sizeof(buf));
|
||||
std::cerr << "openssl: " << buf << std::endl;
|
||||
} else if (res.error() == httplib::Error::SSLServerVerification ||
|
||||
res.error() == httplib::Error::SSLServerHostnameVerification) {
|
||||
auto code = static_cast<long>(res.ssl_backend_error());
|
||||
std::cerr << "openssl: " << X509_verify_cert_error_string(code) << std::endl;
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
@@ -41,7 +41,7 @@ sse.on_event("leave", [](const auto &msg) {
|
||||
});
|
||||
```
|
||||
|
||||
`on_message()` serves as a generic fallback for unnamed events (the default `message` type).
|
||||
`on_message()` is the generic fallback: it receives every event that has no handler registered through `on_event()`. With `on_event("message", ...)` registered as above, `message` events go there instead.
|
||||
|
||||
## Connection lifecycle and errors
|
||||
|
||||
@@ -55,7 +55,7 @@ sse.on_error([](httplib::Error err) {
|
||||
});
|
||||
```
|
||||
|
||||
Hook into connection open and error events. Even when the error handler fires, `SSEClient` keeps trying to reconnect in the background.
|
||||
Hook into connection open and error events. Even when the error handler fires, `SSEClient` keeps trying to reconnect in the background. The exceptions are a 204, 403, or 404 response, after which it stops.
|
||||
|
||||
## Run asynchronously
|
||||
|
||||
|
||||
@@ -30,7 +30,7 @@ You can even mount multiple directories at the same path — they're searched in
|
||||
|
||||
## Combine with API handlers
|
||||
|
||||
Static files and API handlers coexist happily. Handlers registered with `Get()` and friends take priority; the mount points are searched only when nothing matches.
|
||||
Static files and API handlers coexist happily. For GET and HEAD, the mount points are searched first; handlers registered with `Get()` and friends run only when no file is found.
|
||||
|
||||
```cpp
|
||||
svr.Get("/api/users", [](const auto &req, auto &res) {
|
||||
|
||||
@@ -15,14 +15,15 @@ svr.Get("/download", [](const httplib::Request &req, httplib::Response &res) {
|
||||
res.set_content_provider(
|
||||
total_size, "application/octet-stream",
|
||||
[](size_t offset, size_t length, httplib::DataSink &sink) {
|
||||
auto data = read_range_from_file("large.bin", offset, length);
|
||||
auto n = std::min(length, size_t(64 * 1024));
|
||||
auto data = read_range_from_file("large.bin", offset, n);
|
||||
sink.write(data.data(), data.size());
|
||||
return true;
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
The lambda is called repeatedly with `offset` and `length`. Read just that range and write it to `sink`. Only a small chunk sits in memory at any given time.
|
||||
The lambda is called repeatedly with `offset`, the position sent so far, and `length`, the number of bytes still to send. Cap how much you read per call and write it to `sink`, and only a small chunk sits in memory at any given time.
|
||||
|
||||
## Just send a file
|
||||
|
||||
|
||||
@@ -96,7 +96,7 @@ svr.Post("/upload",
|
||||
});
|
||||
```
|
||||
|
||||
When `content_reader` returns `false`, set the response status yourself. The rest of the body is left unread and the connection is closed, so a client that is still sending sees the connection drop.
|
||||
When `content_reader` returns `false`, the response status becomes 400 (413 if the body exceeded the size limit). Set it yourself if you want a different one. The rest of the body is left unread and the connection is closed, so a client that is still sending sees the connection drop.
|
||||
|
||||
> **Warning:** When you use `HandlerWithContentReader`, `req.body` stays **empty**. Handle the body yourself inside the callbacks.
|
||||
|
||||
|
||||
@@ -32,7 +32,7 @@ That's it. If the client sent `Accept-Encoding: gzip`, cpp-httplib compresses th
|
||||
|
||||
## Encoding priority
|
||||
|
||||
When the client accepts multiple encodings, cpp-httplib picks in this order (among those enabled at build time): Brotli → Zstd → gzip. Your code doesn't need to care — you always get the most efficient option available.
|
||||
When the client accepts multiple encodings, cpp-httplib picks the one with the highest q-value in `Accept-Encoding`. On a tie the order is Brotli → gzip → Zstd (among those enabled at build time).
|
||||
|
||||
## Streaming responses are compressed too
|
||||
|
||||
|
||||
@@ -37,7 +37,7 @@ svr.Get("/me", [](const httplib::Request &req, httplib::Response &res) {
|
||||
|
||||
## Typical value types
|
||||
|
||||
Strings, numbers, structs, `std::shared_ptr` — anything copyable or movable works.
|
||||
Strings, numbers, structs, `std::shared_ptr`: anything copyable works.
|
||||
|
||||
```cpp
|
||||
res.user_data.set("user_id", std::string{"42"});
|
||||
|
||||
@@ -48,7 +48,7 @@ svr.set_pre_routing_handler([](const auto &req, auto &res) {
|
||||
return httplib::Server::HandlerResponse::Unhandled;
|
||||
});
|
||||
|
||||
svr.set_logger([](const auto &req, const auto &res) {
|
||||
svr.set_logger([](const httplib::Request &req, const httplib::Response &res) {
|
||||
auto *start = res.user_data.get<std::chrono::steady_clock::time_point>("start");
|
||||
auto elapsed = start
|
||||
? std::chrono::duration_cast<std::chrono::milliseconds>(
|
||||
|
||||
@@ -52,6 +52,6 @@ int main() {
|
||||
|
||||
## What happens to in-flight requests
|
||||
|
||||
When you call `stop()`, new connections are refused, but requests already being processed are **allowed to finish**. Once all workers drain, `listen()` returns. That's what makes it graceful.
|
||||
When you call `stop()`, new connections are refused, but handlers that are already running are **allowed to finish**. A response still being sent by a content provider (a streaming response, for example) is cut short, though. Once all workers drain, `listen()` returns. That's what makes it graceful.
|
||||
|
||||
> **Warning:** There's a wait between calling `stop()` and `listen()` returning — it's the time in-flight requests take to finish. To enforce a timeout, you'll need to add your own shutdown timer in application code.
|
||||
|
||||
@@ -19,7 +19,7 @@ svr.Get("/", [](const auto &, auto &res) {
|
||||
svr.listen("/tmp/httplib.sock", 80);
|
||||
```
|
||||
|
||||
Call `set_address_family(AF_UNIX)` first, then pass the socket file path as the first argument to `listen()`. The port number is unused but required by the signature — pass any value.
|
||||
Call `set_address_family(AF_UNIX)` first, then pass the socket file path as the first argument to `listen()`. The port number is unused but required by the signature. Pass any value other than `0`, which makes `listen()` fail.
|
||||
|
||||
## Client side
|
||||
|
||||
|
||||
@@ -55,7 +55,7 @@ httplib::SSLClient cli("api.example.com", 443,
|
||||
auto res = cli.Get("/");
|
||||
```
|
||||
|
||||
Note you're using `SSLClient` directly, not `Client`. If the private key has a password, pass it as the fifth argument.
|
||||
If the certificate and key paths are all you need, `Client` takes them too: `httplib::Client cli("https://api.example.com", "client-cert.pem", "client-key.pem")`. When the private key has a password, use `SSLClient` and pass it as the fifth argument.
|
||||
|
||||
The client side has the same `PemMemory` struct too, letting you set the client certificate from PEM in memory.
|
||||
|
||||
|
||||
@@ -67,7 +67,7 @@ cli.set_websocket_max_missed_pongs(2); // close after 2 consecutive unacked ping
|
||||
|
||||
The server side has the same `set_websocket_max_missed_pongs()`.
|
||||
|
||||
With a 30-second ping interval and `max_missed_pongs = 2`, a dead peer is detected within roughly 60 seconds and the connection is closed with `CloseStatus::GoingAway` and the reason `"pong timeout"`.
|
||||
With a 30-second ping interval and `max_missed_pongs = 2`, a dead peer is detected 60 to 90 seconds after it stops answering, and the connection is closed with `CloseStatus::GoingAway` and the reason `"pong timeout"`. A `read()` waiting on the peer at that moment returns `Fail`.
|
||||
|
||||
The counter is reset whenever `read()` consumes an incoming Pong frame, so this only works if your code is actively calling `read()` in a loop — which is what a normal WebSocket client does anyway.
|
||||
|
||||
|
||||
@@ -18,10 +18,6 @@ Simply pass the path to a model file to `llamalib::Llama`, and model loading, co
|
||||
int main() {
|
||||
auto llm = llamalib::Llama{"models/gemma-2-2b-it-Q4_K_M.gguf"};
|
||||
|
||||
// LLM inference takes time, so set a longer timeout (default is 5 seconds)
|
||||
svr.set_read_timeout(300);
|
||||
svr.set_write_timeout(300);
|
||||
|
||||
// ... Build and start the HTTP server ...
|
||||
}
|
||||
```
|
||||
@@ -78,7 +74,7 @@ svr.Post("/translate",
|
||||
});
|
||||
```
|
||||
|
||||
`llm.chat()` can throw exceptions during inference (for example, when the context length is exceeded). By catching them with `try/catch` and returning the error as JSON, we prevent the server from crashing.
|
||||
`llm.chat()` can throw exceptions during inference (for example, when the context length is exceeded). We catch them with `try/catch` and return the error as JSON. Left uncaught, cpp-httplib would still answer with a 500, but the client would not learn the cause.
|
||||
|
||||
## 2.3 Complete Code
|
||||
|
||||
@@ -111,10 +107,6 @@ int main() {
|
||||
// Load the model downloaded in Chapter 1
|
||||
auto llm = llamalib::Llama{"models/gemma-2-2b-it-Q4_K_M.gguf"};
|
||||
|
||||
// LLM inference takes time, so set a longer timeout (default is 5 seconds)
|
||||
svr.set_read_timeout(300);
|
||||
svr.set_write_timeout(300);
|
||||
|
||||
// Log requests and responses
|
||||
svr.set_logger([](const auto &req, const auto &res) {
|
||||
std::cout << req.method << " " << req.path << " -> " << res.status
|
||||
|
||||
@@ -73,7 +73,7 @@ A few key points:
|
||||
- After writing to `sink.os`, you can check whether the client is still connected with `sink.os.good()`. If the client has disconnected, it returns `false` to stop inference
|
||||
- Each token is escaped as a JSON string using `json(token).dump()` before sending. This is safe even for tokens containing newlines or quotes
|
||||
- The first three arguments of `dump(-1, ' ', false, ...)` are the defaults. What matters is the fourth argument, `json::error_handler_t::replace`. Since the LLM returns tokens at the subword level, multi-byte characters (such as Japanese) can be split mid-character across tokens. Passing an incomplete UTF-8 byte sequence directly to `dump()` would throw an exception, so `replace` safely substitutes them. The browser reassembles the bytes on its end, so everything displays correctly
|
||||
- The entire lambda is wrapped in `try/catch`. `llm.chat()` can throw exceptions for reasons such as exceeding the context window. If an exception goes uncaught inside the lambda, the server will crash, so we return the error as an SSE event instead
|
||||
- The entire lambda is wrapped in `try/catch`. `llm.chat()` can throw exceptions for reasons such as exceeding the context window. If an exception goes uncaught inside the lambda, cpp-httplib just drops the connection and the client never learns the cause, so we return the error as an SSE event instead
|
||||
- `data: [DONE]` follows the OpenAI API convention to signal the end of the stream to the client
|
||||
|
||||
## 3.4 Complete Code
|
||||
@@ -107,10 +107,6 @@ int main() {
|
||||
// Load the GGUF model
|
||||
auto llm = llamalib::Llama{"models/gemma-2-2b-it-Q4_K_M.gguf"};
|
||||
|
||||
// LLM inference takes time, so set a longer timeout (default is 5 seconds)
|
||||
svr.set_read_timeout(300);
|
||||
svr.set_write_timeout(300);
|
||||
|
||||
// Log requests and responses
|
||||
svr.set_logger([](const auto &req, const auto &res) {
|
||||
std::cout << req.method << " " << req.path << " -> " << res.status
|
||||
|
||||
@@ -219,7 +219,6 @@ bool download_model(const ModelInfo &model,
|
||||
std::function<bool(int)> progress_cb) {
|
||||
httplib::Client cli("https://huggingface.co");
|
||||
cli.set_follow_location(true);
|
||||
cli.set_read_timeout(std::chrono::hours(1));
|
||||
|
||||
auto url = "/" + model.repo + "/resolve/main/" + model.filename;
|
||||
auto path = get_models_dir() / model.filename;
|
||||
@@ -469,7 +468,6 @@ bool download_model(const ModelInfo &model,
|
||||
std::function<bool(int)> progress_cb) {
|
||||
httplib::Client cli("https://huggingface.co");
|
||||
cli.set_follow_location(true); // Hugging Face redirects to a CDN
|
||||
cli.set_read_timeout(std::chrono::hours(1)); // Set a long timeout for large models
|
||||
|
||||
auto url = "/" + model.repo + "/resolve/main/" + model.filename;
|
||||
auto path = get_models_dir() / model.filename;
|
||||
@@ -539,10 +537,6 @@ int main() {
|
||||
auto llm = llamalib::Llama{path};
|
||||
std::mutex llm_mutex; // Protect access during model switching
|
||||
|
||||
// Set a long timeout since LLM inference takes time (default is 5 seconds)
|
||||
svr.set_read_timeout(300);
|
||||
svr.set_write_timeout(300);
|
||||
|
||||
svr.set_logger([](const auto &req, const auto &res) {
|
||||
std::cout << req.method << " " << req.path << " -> " << res.status
|
||||
<< std::endl;
|
||||
|
||||
@@ -939,7 +939,6 @@ bool download_model(const ModelInfo &model,
|
||||
std::function<bool(int)> progress_cb) {
|
||||
httplib::Client cli("https://huggingface.co");
|
||||
cli.set_follow_location(true); // Hugging Face redirects to CDN
|
||||
cli.set_read_timeout(std::chrono::hours(1)); // Long timeout for large models
|
||||
|
||||
auto url = "/" + model.repo + "/resolve/main/" + model.filename;
|
||||
auto path = get_models_dir() / model.filename;
|
||||
@@ -1008,10 +1007,6 @@ int main() {
|
||||
}
|
||||
auto llm = llamalib::Llama{path};
|
||||
|
||||
// LLM inference takes time, so set a longer timeout (default is 5 seconds)
|
||||
svr.set_read_timeout(300);
|
||||
svr.set_write_timeout(300);
|
||||
|
||||
svr.set_logger([](const auto &req, const auto &res) {
|
||||
std::cout << req.method << " " << req.path << " -> " << res.status
|
||||
<< std::endl;
|
||||
|
||||
@@ -407,7 +407,6 @@ bool download_model(const ModelInfo &model,
|
||||
std::function<bool(int)> progress_cb) {
|
||||
httplib::Client cli("https://huggingface.co");
|
||||
cli.set_follow_location(true); // Hugging Face redirects to a CDN
|
||||
cli.set_read_timeout(std::chrono::hours(1)); // Long timeout for large models
|
||||
|
||||
auto url = "/" + model.repo + "/resolve/main/" + model.filename;
|
||||
auto path = get_models_dir() / model.filename;
|
||||
@@ -469,10 +468,6 @@ int main() {
|
||||
auto llm = llamalib::Llama{path};
|
||||
std::mutex llm_mutex; // Protect access during model switching
|
||||
|
||||
// Set a long timeout since LLM inference takes time (default is 5 seconds)
|
||||
svr.set_read_timeout(300);
|
||||
svr.set_write_timeout(300);
|
||||
|
||||
svr.set_logger([](const auto &req, const auto &res) {
|
||||
std::cout << req.method << " " << req.path << " -> " << res.status
|
||||
<< std::endl;
|
||||
|
||||
@@ -11,13 +11,17 @@ Over the course of six chapters, we built a translation desktop app from scratch
|
||||
## 7.1 Source Code Location
|
||||
|
||||
```ascii
|
||||
llama.cpp/tools/server/
|
||||
├── server.cpp # Main server implementation
|
||||
├── httplib.h # cpp-httplib (bundled version)
|
||||
└── ...
|
||||
llama.cpp/
|
||||
├── tools/server/
|
||||
│ ├── server.cpp # Entry point
|
||||
│ ├── server-http.cpp # HTTP server (the layer that uses cpp-httplib)
|
||||
│ ├── server-context.cpp # Inference and slot management
|
||||
│ └── ...
|
||||
└── vendor/cpp-httplib/
|
||||
└── httplib.h # cpp-httplib (bundled version)
|
||||
```
|
||||
|
||||
The code is contained in a single `server.cpp`. It runs to several thousand lines, but once you understand the structure, you can narrow down the parts worth reading.
|
||||
The implementation is split across several files by role. It is a large code base, but once you understand the structure, you can narrow down the parts worth reading.
|
||||
|
||||
## 7.2 OpenAI-Compatible API
|
||||
|
||||
|
||||
@@ -200,8 +200,8 @@ auto res = cli.Post("/submit", httplib::Params{
|
||||
});
|
||||
if (res) {
|
||||
std::cout << res->body << std::endl;
|
||||
// age = 30
|
||||
// name = Alice
|
||||
// age = 30
|
||||
}
|
||||
```
|
||||
|
||||
@@ -241,7 +241,7 @@ auto res = cli.Get("/hi");
|
||||
if (!res) {
|
||||
// Connection error
|
||||
std::cout << "Error: " << httplib::to_string(res.error()) << std::endl;
|
||||
// Error: Connection
|
||||
// Error: Could not establish connection
|
||||
return 1;
|
||||
}
|
||||
|
||||
|
||||
@@ -95,7 +95,7 @@ svr.set_mount_point("/", "./public");
|
||||
svr.listen("0.0.0.0", 8080);
|
||||
```
|
||||
|
||||
Handlers take priority. The handler responds to `/api/hello`. For every other path, the server looks for a file in `./public`.
|
||||
The server looks for a file in `./public` first and calls the handler when there is none. So the handler responds to `/api/hello` unless you put a file at `./public/api/hello`.
|
||||
|
||||
## Adding response headers
|
||||
|
||||
|
||||
@@ -50,7 +50,7 @@ svr.Get("/stream", [](const auto &, auto &res) {
|
||||
res.set_chunked_content_provider("text/plain",
|
||||
[](size_t offset, httplib::DataSink &sink) {
|
||||
sink.write("chunk\n", 6);
|
||||
return true; // Return false to finish
|
||||
return true; // Call sink.done() to finish
|
||||
});
|
||||
});
|
||||
```
|
||||
@@ -155,7 +155,7 @@ svr.set_pre_routing_handler([](const auto &req, auto &res) {
|
||||
});
|
||||
|
||||
svr.set_post_routing_handler([](const auto &req, auto &res) {
|
||||
// Runs after the response is sent
|
||||
// Runs just before the response is sent
|
||||
res.set_header("X-Server", "cpp-httplib");
|
||||
});
|
||||
```
|
||||
@@ -168,7 +168,7 @@ svr.set_pre_routing_handler([](const auto &req, auto &res) {
|
||||
return httplib::Server::HandlerResponse::Unhandled;
|
||||
});
|
||||
|
||||
svr.Get("/me", [](const auto &req, auto &res) {
|
||||
svr.Get("/me", [](const httplib::Request &req, httplib::Response &res) {
|
||||
auto *user = res.user_data.get<std::string>("auth_user");
|
||||
res.set_content("Hello, " + *user, "text/plain");
|
||||
});
|
||||
@@ -205,7 +205,7 @@ In addition to TCP, we support Unix Domain Sockets. You can use them for inter-p
|
||||
// Server
|
||||
httplib::Server svr;
|
||||
svr.set_address_family(AF_UNIX);
|
||||
svr.listen("/tmp/httplib.sock", 0);
|
||||
svr.listen("/tmp/httplib.sock", 80); // The port is unused, but must not be 0
|
||||
```
|
||||
|
||||
```cpp
|
||||
|
||||
Reference in New Issue
Block a user