https://www.chenshaowen.com/en/tags/llm/Recent content in LLM on Shaowen Chen's WebsiteHugo -- gohugo.ioen©2016 - {year}, All Rights Reserved.Wed, 24 Jun 2026 00:00:00 +0000weeklyhttps://www.chenshaowen.com/en/blog/litellm-unified-model-api-format.htmlWed, 24 Jun 2026 00:00:00 +0000Wed, 24 Jun 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/litellm-unified-model-api-format.html1. What LiteLLM Is LiteLLM is an open-source LLM adapter: call 100+ APIs (OpenAI, Anthropic, Gemini, Bedrock, Azure, Ollama, vLLM, Qwen, and more) using the standard OpenAI format. Two ways to use it: Python SDK — called directly from your code AI Gateway — a team-level proxy with authentication, billing,微信公众号AILiteLLMLLMGatewayR&Dhttps://www.chenshaowen.com/en/blog/installation-and-configuration-of-openclaw.htmlThu, 26 Feb 2026 00:00:00 +0000Thu, 26 Feb 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/installation-and-configuration-of-openclaw.html1. Preparing the Node Environment Install nvm 1 curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.4/install.sh | bash Set the nvm environment variables You need to edit Bash’s config file and add the following: 1 2 3 export NVM_DIR="$HOME/.nvm" [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" [ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion" Install Node 1 nvm install 22 default 2.微信公众号OpenClawAIAgentLLMToolsR&DLearninghttps://www.chenshaowen.com/en/blog/using-mcp-to-power-ai-agent.htmlThu, 29 Jan 2026 00:00:00 +0000Thu, 29 Jan 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-mcp-to-power-ai-agent.html1. What Is MCP MCP is short for Model Context Protocol, a protocol for passing context between models and applications. The greatest value of MCP is that, as the model wars are winding down and AI applications are about to explode, it gives developers a unified standard for invoking capabilities beyond the model itself.微信公众号AIAI AgentMCPLLMR&Dhttps://www.chenshaowen.com/en/blog/using-ops-event-to-power-ai-agent.htmlWed, 14 Jan 2026 10:00:00 +0000Wed, 14 Jan 2026 10:00:00 +0000https://www.chenshaowen.com/en/blog/using-ops-event-to-power-ai-agent.html1. Why Operations Events Less invasive Operations teams usually already have observations accumulated in the Metrics and Log dimensions. Operations events can be a new entry point: introduce new operational capabilities without affecting the stability of existing systems. Better suited to automation Metrics represent system state, Log represents specific code behavior, while Event represents changes in components.微信公众号AI AgentAIOperationsLLMhttps://www.chenshaowen.com/en/blog/using-llama-factory-to-fine-tune-qwen3-model.htmlSun, 21 Sep 2025 00:00:00 +0000Sun, 21 Sep 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-llama-factory-to-fine-tune-qwen3-model.html1. Testing the Original Model Start the environment 1 2 3 4 5 6 7 8 9 10 11 12 nerdctl run -it \ --security-opt apparmor=unconfined \ --security-opt seccomp=unconfined \ --gpus all \ --ipc=host \ -p 8000:8000 \ --ulimit memlock=-1 \ --ulimit stack=67108864 \ --name vllm \ --volume /data/models:/data/models \微信公众号LLaMA-FactoryQwen3Fine-TuningAILLMhttps://www.chenshaowen.com/en/blog/use-lmcache-to-improve-ttft.htmlWed, 17 Sep 2025 00:00:00 +0000Wed, 17 Sep 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/use-lmcache-to-improve-ttft.html1. Introduction to LMCache TTFT is the time from when a request is issued until the model generates its first token. Because the Prefill phase has to encode the input context into a KV Cache before generation can begin, producing that first token requires a large amount of computation, which makes TTFT high.微信公众号AILLMvLLMInferencelmcacheOperationshttps://www.chenshaowen.com/en/blog/nvidia-rtx-5090-inference-test.htmlTue, 09 Sep 2025 00:00:00 +0000Tue, 09 Sep 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/nvidia-rtx-5090-inference-test.html1. Installing the Driver Download the driver Go to https://www.nvidia.com/en-us/drivers/ and select the corresponding driver version to download. 1 wget https://us.download.nvidia.com/XFree86/Linux-x86_64/580.76.05/NVIDIA-Linux-x86_64-580.76.05.run Install the driver 1 bash NVIDIA-Linux-x86_64-580.76.05.run Check the GPUs 1 nvidia-smi 1 2 3 GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-92fcdc58-4754-73c7-af6c-56740936817d) GPU 1: NVIDIA GeForce RTX 5090 (UUID:微信公众号AIGPUNVIDIALLMInferencePerformance TestingOperationshttps://www.chenshaowen.com/en/blog/what-is-model-quantization.htmlSat, 06 Sep 2025 00:00:00 +0000Sat, 06 Sep 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/what-is-model-quantization.html1. What Is Model Quantization Model quantization is the process of converting the weights and activations of a high-precision model (usually 32-bit floating point FP32 or 16-bit floating point FP16) into a low-precision model (such as 8-bit integer INT8). The value range of FP32 is -3.4*10^38 to 3.4*10^38, with 4 billion values.微信公众号AILLMModel QuantizationOptimizationOperationshttps://www.chenshaowen.com/en/blog/using-claude-code-on-server.htmlFri, 29 Aug 2025 00:00:00 +0000Fri, 29 Aug 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-claude-code-on-server.htmlThe same applies to a desktop environment. 1. Prepare the node.js environment Install nvm 1 curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.3/install.sh | bash 1 2 echo 'export NVM_DIR="$([ -z "${XDG_CONFIG_HOME-}" ] && printf %s "${HOME}/.nvm" || printf %s "${XDG_CONFIG_HOME}/nvm")"' >> ~/.bashrc echo '[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"' >> ~/.bashrc 1 source微信公众号ClaudeCLIAILLMToolsR&DOperationshttps://www.chenshaowen.com/en/blog/ai-related-papers.htmlSat, 05 Jul 2025 00:00:00 +0000Sat, 05 Jul 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/ai-related-papers.html2025 Intrinsic Fingerprint of LLMs [Published: 07-04] View The paper proposes a robust fingerprinting method based on the inter-layer standard-deviation distribution pattern of attention parameter matrices (Q/K/V/O), used to detect lineage relationships between large language models (LLMs) — for example, whether one model was derived from another through continued training, fine-tuning, or an upgrade.微信公众号AIPapersStorageLLMhttps://www.chenshaowen.com/en/blog/using-gemini-cli-on-server.htmlFri, 04 Jul 2025 00:00:00 +0000Fri, 04 Jul 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-gemini-cli-on-server.html1. Get the Project ID and API Key Visit https://aistudio.google.com/apikey Get the project ID and API key, and set them as environment variables 1 2 echo "export GOOGLE_CLOUD_PROJECT=" >> ~/.bashrc echo "export GEMINI_API_KEY=" >> ~/.bashrc 1 source ~/.bashrc 2. Prepare the Node.js Environment Install nvm 1 curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.3/install.sh |微信公众号GeminiCLIAILLMLearningToolsOperationsProgramming Assistanthttps://www.chenshaowen.com/en/blog/deploy-deepseek-3fs-storage-system-under-containers.htmlSat, 15 Mar 2025 00:00:00 +0000Sat, 15 Mar 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/deploy-deepseek-3fs-storage-system-under-containers.html1. Deployment Plan Before starting the containerized deployment, a few requirements: To simplify delivery, only one image is needed For reliability, deploy as many replicas as possible Start different services through different parameters Inject configuration through environment variables and render it into configuration files Below is the deployment plan for DeepSeek 3FS:微信公众号StorageAIContainersKubernetesDockerOperationsLLM3FShttps://www.chenshaowen.com/en/blog/structure-of-large-models-with-qwen.htmlSun, 09 Feb 2025 00:00:00 +0000Sun, 09 Feb 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/structure-of-large-models-with-qwen.html1. Introduction to the Qwen Model In April 2023, Alibaba released the beta version of Qwen. In December 2023, Alibaba open-sourced the first version of Qwen. In September 2024, Alibaba released Qwen2.5. In January 2025, Alibaba released Qwen 2.5-Max. Qwen 2.5 is the latest series of the Qwen large language微信公众号QwenLLMAILearningArchitectureTransformerOpen SourceR&Dhttps://www.chenshaowen.com/en/blog/use-vllm-for-inference.htmlSat, 18 Jan 2025 00:00:00 +0000Sat, 18 Jan 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/use-vllm-for-inference.html1. Environment Preparation Download Miniforge 1 wget "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh" Install Miniforge 1 bash Miniforge3-$(uname)-$(uname -m).sh 1 2 echo "export PATH=$HOME/miniforge3/bin:$PATH" >> ~/.bashrc source ~/.bashrc Create the environment 1 conda create -n vllm python=3.12 vLLM currently requires Python 3.9+. Activate the environment 1 conda activate vllm Install dependencies 1 conda install微信公众号vLLMAILLMInferenceOperationshttps://www.chenshaowen.com/en/blog/use-vllm-verify-inference-node.htmlThu, 16 Jan 2025 00:00:00 +0000Thu, 16 Jan 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/use-vllm-verify-inference-node.html1. Building the Image To make testing convenient, the model files are packaged into the image here. Download the model 1 2 3 4 git clone https://huggingface.co/Qwen/Qwen1.5-1.8B-Chat cd Qwen1.5-1.8B-Chat && git lfs pull rm -rf .git cd .. Write the Dockerfile 1 2 3 4 5 cat <<EOF > Dockerfile微信公众号vLLMInferenceAILLMOperationshttps://www.chenshaowen.com/en/blog/ai-application-development-tech-stack.htmlSun, 12 Jan 2025 00:00:00 +0000Sun, 12 Jan 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/ai-application-development-tech-stack.htmlEmbedding Models Embedding models map high-dimensional data into a lower-dimensional space, which makes the data easier to process and analyze. Text Models Here is a leaderboard, https://huggingface.co/spaces/mteb/leaderboard The leaderboard above gives each model’s score, parameter count, memory usage, vector dimension, maximum tokens, and other information. Below are some commonly used open-source models:微信公众号AIApplication DevelopmentTech StackLLMR&DLearningRAGEmbeddingBest Practiceshttps://www.chenshaowen.com/en/blog/develop-a-copilot-to-handle-exceptions.htmlWed, 14 Aug 2024 00:00:00 +0000Wed, 14 Aug 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/develop-a-copilot-to-handle-exceptions.htmlThis post comes mainly from an internal sharing session, and is also a summary of some recent work. 1. The Common Failure Handling Process The figure above shows a typical operations exception handling process. Along the timeline, there are the following key points in time: Failure occurs Failure detected Failure微信公众号LLMOperationsExceptionsFailuresSharinghttps://www.chenshaowen.com/en/blog/ops-copilot-has-been-updated-using-pipeline-and-llm.htmlWed, 01 May 2024 00:00:00 +0000Wed, 01 May 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/ops-copilot-has-been-updated-using-pipeline-and-llm.html1. Making Ops Copilot an Ops Coilot In September 2023, I wrote a version of Ops Copilot, and published an article about it, I’m Writing a Copilot for My Ops Tool. The effect it achieved was like this: 1 2 3 4 5 6 7 8 9 10 Opscli> 打微信公众号AICopilotAgentLLMOpsOperationshttps://www.chenshaowen.com/en/blog/what-is-mlops.htmlSat, 27 Apr 2024 10:00:00 +0000Sat, 27 Apr 2024 10:00:00 +0000https://www.chenshaowen.com/en/blog/what-is-mlops.html1. What Is MLOps MLOps is short for Machine Learning Operations, and it describes the standardization and engineering of the entire lifecycle around model development. MLOps includes the following key steps: Data management: storing, accessing, cleaning, and transforming data Model development: algorithm development and model construction Model training and tuning: training models on data, adjusting hyperparameters to optimize the model, and fine-tuning models Model evaluation: testing a model’s accuracy, generalization ability, and performance metrics Model deployment: deploying the model to the target environment and converting the model Model monitoring: monitoring the model’s behavior, performance degradation, data drift, and robustness Another similar concept is ModelOps.微信公众号Machine LearningMLOpsLLMR&DWhat Ishttps://www.chenshaowen.com/en/blog/common-ai-base-images-and-run-command.htmlThu, 28 Mar 2024 00:01:00 +0000Thu, 28 Mar 2024 00:01:00 +0000https://www.chenshaowen.com/en/blog/common-ai-base-images-and-run-command.html1. What the Image Tag Suffixes Mean base/cuda: includes the CUDA runtime runtime: builds on base, adding the CUDA math libraries and the NCCL and cuDNN runtimes devel: builds on runtime, adding headers and the development tools used to build CUDA images; particularly useful for multi-stage builds cuddn: builds on the above, adding the cuDNN neural network acceleration library微信公众号AIImagesContainersDockerGPULLMOperationsR&Dhttps://www.chenshaowen.com/en/blog/provide-a-way-to-develop-ai-agent.htmlSat, 16 Mar 2024 08:01:28 +0000Sat, 16 Mar 2024 08:01:28 +0000https://www.chenshaowen.com/en/blog/provide-a-way-to-develop-ai-agent.html1. Keep Trying to Ship the AI Application Side Based on my understanding of operations, I developed an open-source operations tool, https://github.com/shaowenchen/ops . The Ops tool divides operations work into two categories — script execution and file distribution — and both operations targets, hosts and Kubernetes clusters, implement both of微信公众号ThoughtsAIAgentLLMOpsOperationshttps://www.chenshaowen.com/en/blog/speeding-up-model-inference-with-tensorrt.htmlTue, 06 Feb 2024 00:00:00 +0000Tue, 06 Feb 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/speeding-up-model-inference-with-tensorrt.html1. What Is TensorRT TensorRT is a C++ library mainly used for high-performance inference acceleration on NVIDIA GPUs. It provides both a C++ API and a Python API for integration. The mainstream deep learning frameworks TensorRT supports are: Caffe, where TensorRT can read the prototxt format directly TensorFlow, where the微信公众号AITensorRTNVIDIAGPUInferenceLLMhttps://www.chenshaowen.com/en/blog/using-triton-server-and-tensorrt-llm-under-container.htmlSat, 03 Feb 2024 08:05:48 +0000Sat, 03 Feb 2024 08:05:48 +0000https://www.chenshaowen.com/en/blog/using-triton-server-and-tensorrt-llm-under-container.html1. Compiling Models with TensorRT-LLM 1.1 Introduction to TensorRT-LLM When using TensorRT, you usually need to convert the model to ONNX format, then convert the ONNX to TensorRT format, and finally run inference in TensorRT or Triton Server. But this conversion process is not simple, and you often run into微信公众号AITritonTensorRTLLMInferenceGPUOperationsContainersLearninghttps://www.chenshaowen.com/en/blog/how-to-access-gpt-service-on-wechat.htmlTue, 26 Dec 2023 00:00:00 +0000Tue, 26 Dec 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-access-gpt-service-on-wechat.htmlPaid integration service available at 200 RMB/year; technical support is also available at 200 RMB per session, capped at 1 hour. Follow the official account to get contact details. 1. Information You Need to Provide Go to https://mp.weixin.qq.com/ and open 【Settings and微信公众号LLMGPTOfficial AccountWeChatAIR&DLearningIntegrationOpenAIPracticehttps://www.chenshaowen.com/en/blog/large-model-application-design-and-implementation-guide.htmlSat, 23 Dec 2023 11:22:55 +0000Sat, 23 Dec 2023 11:22:55 +0000https://www.chenshaowen.com/en/blog/large-model-application-design-and-implementation-guide.html1. Problems with Using LLMs Directly Unstable output One characteristic of generative AI is the diversity of its output. Ask an LLM the same question several times and you may get different answers. This uncertainty in output is a pleasant surprise for users in conversation and creative scenarios. But in微信公众号LLMAIBest PracticesR&DArchitectureLearningRAGAgenthttps://www.chenshaowen.com/en/blog/how-to-run-llama-on-cpu.htmlSat, 16 Sep 2023 00:00:00 +0000Sat, 16 Sep 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-run-llama-on-cpu.html1. Running Locally in Containers Start the LLM 1 docker run --security-opt apparmor=unconfined --security-opt seccomp=unconfined --rm -p 8000:8000 shaowenchen/chinese-alpaca-2-7b-gguf:Q2_K The API documentation is available at http://localhost:8000/docs, as shown below: Deploy a simple Chat UI Note that the OPENAI_API_HOST parameter must be set to your host machine’s IP address rather than localhost or 127.微信公众号CPULLMAIInferenceOperationshttps://www.chenshaowen.com/en/blog/llama-cpp-that-is-a-llm-deployment-tool.htmlTue, 05 Sep 2023 00:00:00 +0000Tue, 05 Sep 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/llama-cpp-that-is-a-llm-deployment-tool.html1. LLM Deployment Tool llama.cpp Research on large models is split into two parts: training and inference. The training process is essentially the process of finding model parameters that minimize the model’s loss function and optimize the inference results. Once training is complete, the model’s parameters are fixed, and at微信公众号AILLMToolsllama.cppInferenceDeploymenthttps://www.chenshaowen.com/en/blog/usage-of-transformers-lib.htmlTue, 22 Aug 2023 00:00:00 +0000Tue, 22 Aug 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/usage-of-transformers-lib.htmltransformers is a Python library developed by Hugging Face for using and training pretrained Transformer models in natural language processing (NLP) tasks. It provides many powerful tools and features that make working with text data and building NLP models much easier. The library is widely used across a variety of微信公众号TransformerAILLMNLPPythonhttps://www.chenshaowen.com/en/blog/models-and-datasets-on-huggingface.htmlMon, 21 Aug 2023 00:00:00 +0000Mon, 21 Aug 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/models-and-datasets-on-huggingface.htmlHuggingFace offers shared models, datasets, and hosted spaces, giving AI researchers and developers a complete ecosystem. This article explains how to work with HuggingFace models and datasets. 1. Model Operations and Usage 1.1 Custom Storage Directory 1 export HF_HOME=/Volumes/Data/HuggingFace Otherwise the default is the ~/.cache/huggingface directory. 1.2 Downloading Models The微信公众号TransformerAILLMHuggingFaceDatasetR&Dhttps://www.chenshaowen.com/en/blog/ai-basic-knowledge.htmlFri, 18 Aug 2023 11:22:55 +0000Fri, 18 Aug 2023 11:22:55 +0000https://www.chenshaowen.com/en/blog/ai-basic-knowledge.html1. Keywords Machine Learning (ML) The technology of automatically acquiring knowledge from data. Neural Network (NN) A model that imitates the structure and learning mechanism of biological neural networks; one of the branches of machine learning. The structure of a neural network consists of an input layer, hidden layers, and an output layer.微信公众号AILLMMachine LearningKnowledge PointsLearninghttps://www.chenshaowen.com/en/blog/call-functions-through-dialogue-using-openai-and-langchain.htmlWed, 16 Aug 2023 00:00:00 +0000Wed, 16 Aug 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/call-functions-through-dialogue-using-openai-and-langchain.html1. LLMs and Langchain Many people may never get the chance to train, or even fine-tune, a large model, but using large models is the wave of the future. So how should we embrace this change? The answer is Langchain. A large model provides a broad, general-purpose foundation. So far微信公众号OpenAILangchainAILLMR&DLearningFunction CallingApplication DevelopmentBest Practices