1. Prompt Chaining Prompt chaining breaks a task down into a series of steps, where each LLM call processes the output of the previous step. 2. Routing Routing classifies the input and directs it to a specialized follow-up task. 3. Parallelization An LLM can sometimes work on a task concurrently and aggregate its outputs programmatically.

1. Handling Deterministic Faults For faults that have a specific handling procedure, just let the Agent handle them and send a notification to spread the word. For similar automated handling, we have exception load deletion at the application layer, disk cleanup at the node layer, masking an offline GPU card, masking a node, and so on.

1. Testing the Original Model Start the environment 1 2 3 4 5 6 7 8 9 10 11 12 nerdctl run -it \ --security-opt apparmor=unconfined \ --security-opt seccomp=unconfined \ --gpus all \ --ipc=host \ -p 8000:8000 \ --ulimit memlock=-1 \ --ulimit stack=67108864 \ --name vllm \ --volume /data/models:/data/models \

1. Introduction to LMCache TTFT is the time from when a request is issued until the model generates its first token. Because the Prefill phase has to encode the input context into a KV Cache before generation can begin, producing that first token requires a large amount of computation, which makes TTFT high.

1. What Is Prefix Cache In model inference scenarios, caching mechanisms are often used to improve throughput and performance. There are two common caching mechanisms: Key-Value Cache (KV Cache), which is oriented toward the internals of a single request, caching the intermediate computation results (Key and Value) of the Transformer model to avoid redundant computation.