mirror of
https://github.com/yhirose/cpp-httplib.git
synced 2026-10-23 12:10:57 +00:00
Fix docs pages that no longer match the code
Samples that did not compile or run as shown: - res.user_data.get<T>() inside a generic lambda needs the `template` keyword; use explicit parameter types (tour 09, cookbook s15). - listen() on a Unix domain socket fails with port 0 (tour 09, s22). - "*.dev.local" is not a NO_PROXY pattern (c16). - ssl_backend_error() holds a verify result, not an ERR_get_error() value, after a verification failure; decode each with the matching OpenSSL function (c18). - The content provider's `length` is everything that remains, so the sample read the whole file in one call (s05). Statements corrected: - Client keep-alive is off by default; c14 is rewritten around set_keep_alive(true). - Mounted files are looked up before GET handlers (tour 04, s04). - Params keep insertion order, and to_string(Error::Connection) reads "Could not establish connection" (tour 02). - A chunked provider ends with sink.done(), and post_routing_handler runs before the response is sent (tour 09). - Timeouts surface as Error::Read; Error::Timeout comes from the stream API (c17). The max timeout cuts off the wait for the response only (c13). The progress callback needs Content-Length (c11). - Encoding selection follows q-values, then Brotli, gzip, Zstd (s08), and the client compresses with the first of those it was built with (c15). - stop() cuts a provider-driven response short (s19); a rejected content_reader already gets 400 or 413 (s07); user_data values must be copyable (s12); Client accepts a client certificate too (t04); on_message() is the fallback for every unhandled event and 204/403/404 end reconnection (e04); the pong timeout takes two to three intervals and ends a waiting read() (w02). In the LLM app tutorial, an uncaught exception does not crash the server, so say what it does instead. Drop the server and client timeout settings whose stated purpose, covering inference and download time, they do not serve: those timeouts bound a single socket wait. Update the llama.cpp server layout in chapter 7.
This commit is contained in:
@@ -18,10 +18,6 @@ Simply pass the path to a model file to `llamalib::Llama`, and model loading, co
|
||||
int main() {
|
||||
auto llm = llamalib::Llama{"models/gemma-2-2b-it-Q4_K_M.gguf"};
|
||||
|
||||
// LLM inference takes time, so set a longer timeout (default is 5 seconds)
|
||||
svr.set_read_timeout(300);
|
||||
svr.set_write_timeout(300);
|
||||
|
||||
// ... Build and start the HTTP server ...
|
||||
}
|
||||
```
|
||||
@@ -78,7 +74,7 @@ svr.Post("/translate",
|
||||
});
|
||||
```
|
||||
|
||||
`llm.chat()` can throw exceptions during inference (for example, when the context length is exceeded). By catching them with `try/catch` and returning the error as JSON, we prevent the server from crashing.
|
||||
`llm.chat()` can throw exceptions during inference (for example, when the context length is exceeded). We catch them with `try/catch` and return the error as JSON. Left uncaught, cpp-httplib would still answer with a 500, but the client would not learn the cause.
|
||||
|
||||
## 2.3 Complete Code
|
||||
|
||||
@@ -111,10 +107,6 @@ int main() {
|
||||
// Load the model downloaded in Chapter 1
|
||||
auto llm = llamalib::Llama{"models/gemma-2-2b-it-Q4_K_M.gguf"};
|
||||
|
||||
// LLM inference takes time, so set a longer timeout (default is 5 seconds)
|
||||
svr.set_read_timeout(300);
|
||||
svr.set_write_timeout(300);
|
||||
|
||||
// Log requests and responses
|
||||
svr.set_logger([](const auto &req, const auto &res) {
|
||||
std::cout << req.method << " " << req.path << " -> " << res.status
|
||||
|
||||
@@ -73,7 +73,7 @@ A few key points:
|
||||
- After writing to `sink.os`, you can check whether the client is still connected with `sink.os.good()`. If the client has disconnected, it returns `false` to stop inference
|
||||
- Each token is escaped as a JSON string using `json(token).dump()` before sending. This is safe even for tokens containing newlines or quotes
|
||||
- The first three arguments of `dump(-1, ' ', false, ...)` are the defaults. What matters is the fourth argument, `json::error_handler_t::replace`. Since the LLM returns tokens at the subword level, multi-byte characters (such as Japanese) can be split mid-character across tokens. Passing an incomplete UTF-8 byte sequence directly to `dump()` would throw an exception, so `replace` safely substitutes them. The browser reassembles the bytes on its end, so everything displays correctly
|
||||
- The entire lambda is wrapped in `try/catch`. `llm.chat()` can throw exceptions for reasons such as exceeding the context window. If an exception goes uncaught inside the lambda, the server will crash, so we return the error as an SSE event instead
|
||||
- The entire lambda is wrapped in `try/catch`. `llm.chat()` can throw exceptions for reasons such as exceeding the context window. If an exception goes uncaught inside the lambda, cpp-httplib just drops the connection and the client never learns the cause, so we return the error as an SSE event instead
|
||||
- `data: [DONE]` follows the OpenAI API convention to signal the end of the stream to the client
|
||||
|
||||
## 3.4 Complete Code
|
||||
@@ -107,10 +107,6 @@ int main() {
|
||||
// Load the GGUF model
|
||||
auto llm = llamalib::Llama{"models/gemma-2-2b-it-Q4_K_M.gguf"};
|
||||
|
||||
// LLM inference takes time, so set a longer timeout (default is 5 seconds)
|
||||
svr.set_read_timeout(300);
|
||||
svr.set_write_timeout(300);
|
||||
|
||||
// Log requests and responses
|
||||
svr.set_logger([](const auto &req, const auto &res) {
|
||||
std::cout << req.method << " " << req.path << " -> " << res.status
|
||||
|
||||
@@ -219,7 +219,6 @@ bool download_model(const ModelInfo &model,
|
||||
std::function<bool(int)> progress_cb) {
|
||||
httplib::Client cli("https://huggingface.co");
|
||||
cli.set_follow_location(true);
|
||||
cli.set_read_timeout(std::chrono::hours(1));
|
||||
|
||||
auto url = "/" + model.repo + "/resolve/main/" + model.filename;
|
||||
auto path = get_models_dir() / model.filename;
|
||||
@@ -469,7 +468,6 @@ bool download_model(const ModelInfo &model,
|
||||
std::function<bool(int)> progress_cb) {
|
||||
httplib::Client cli("https://huggingface.co");
|
||||
cli.set_follow_location(true); // Hugging Face redirects to a CDN
|
||||
cli.set_read_timeout(std::chrono::hours(1)); // Set a long timeout for large models
|
||||
|
||||
auto url = "/" + model.repo + "/resolve/main/" + model.filename;
|
||||
auto path = get_models_dir() / model.filename;
|
||||
@@ -539,10 +537,6 @@ int main() {
|
||||
auto llm = llamalib::Llama{path};
|
||||
std::mutex llm_mutex; // Protect access during model switching
|
||||
|
||||
// Set a long timeout since LLM inference takes time (default is 5 seconds)
|
||||
svr.set_read_timeout(300);
|
||||
svr.set_write_timeout(300);
|
||||
|
||||
svr.set_logger([](const auto &req, const auto &res) {
|
||||
std::cout << req.method << " " << req.path << " -> " << res.status
|
||||
<< std::endl;
|
||||
|
||||
@@ -939,7 +939,6 @@ bool download_model(const ModelInfo &model,
|
||||
std::function<bool(int)> progress_cb) {
|
||||
httplib::Client cli("https://huggingface.co");
|
||||
cli.set_follow_location(true); // Hugging Face redirects to CDN
|
||||
cli.set_read_timeout(std::chrono::hours(1)); // Long timeout for large models
|
||||
|
||||
auto url = "/" + model.repo + "/resolve/main/" + model.filename;
|
||||
auto path = get_models_dir() / model.filename;
|
||||
@@ -1008,10 +1007,6 @@ int main() {
|
||||
}
|
||||
auto llm = llamalib::Llama{path};
|
||||
|
||||
// LLM inference takes time, so set a longer timeout (default is 5 seconds)
|
||||
svr.set_read_timeout(300);
|
||||
svr.set_write_timeout(300);
|
||||
|
||||
svr.set_logger([](const auto &req, const auto &res) {
|
||||
std::cout << req.method << " " << req.path << " -> " << res.status
|
||||
<< std::endl;
|
||||
|
||||
@@ -407,7 +407,6 @@ bool download_model(const ModelInfo &model,
|
||||
std::function<bool(int)> progress_cb) {
|
||||
httplib::Client cli("https://huggingface.co");
|
||||
cli.set_follow_location(true); // Hugging Face redirects to a CDN
|
||||
cli.set_read_timeout(std::chrono::hours(1)); // Long timeout for large models
|
||||
|
||||
auto url = "/" + model.repo + "/resolve/main/" + model.filename;
|
||||
auto path = get_models_dir() / model.filename;
|
||||
@@ -469,10 +468,6 @@ int main() {
|
||||
auto llm = llamalib::Llama{path};
|
||||
std::mutex llm_mutex; // Protect access during model switching
|
||||
|
||||
// Set a long timeout since LLM inference takes time (default is 5 seconds)
|
||||
svr.set_read_timeout(300);
|
||||
svr.set_write_timeout(300);
|
||||
|
||||
svr.set_logger([](const auto &req, const auto &res) {
|
||||
std::cout << req.method << " " << req.path << " -> " << res.status
|
||||
<< std::endl;
|
||||
|
||||
@@ -11,13 +11,17 @@ Over the course of six chapters, we built a translation desktop app from scratch
|
||||
## 7.1 Source Code Location
|
||||
|
||||
```ascii
|
||||
llama.cpp/tools/server/
|
||||
├── server.cpp # Main server implementation
|
||||
├── httplib.h # cpp-httplib (bundled version)
|
||||
└── ...
|
||||
llama.cpp/
|
||||
├── tools/server/
|
||||
│ ├── server.cpp # Entry point
|
||||
│ ├── server-http.cpp # HTTP server (the layer that uses cpp-httplib)
|
||||
│ ├── server-context.cpp # Inference and slot management
|
||||
│ └── ...
|
||||
└── vendor/cpp-httplib/
|
||||
└── httplib.h # cpp-httplib (bundled version)
|
||||
```
|
||||
|
||||
The code is contained in a single `server.cpp`. It runs to several thousand lines, but once you understand the structure, you can narrow down the parts worth reading.
|
||||
The implementation is split across several files by role. It is a large code base, but once you understand the structure, you can narrow down the parts worth reading.
|
||||
|
||||
## 7.2 OpenAI-Compatible API
|
||||
|
||||
|
||||
Reference in New Issue
Block a user