https://www.chenshaowen.com/en/Recent content on Shaowen Chen's WebsiteHugo -- gohugo.ioen©2016 - {year}, All Rights Reserved.Sun, 27 Sep 2026 00:00:00 +0000weeklyhttps://www.chenshaowen.com/en/blog/applab.htmlSun, 27 Sep 2026 00:00:00 +0000Sun, 27 Sep 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/applab.htmlThis is an open-source project: https://github.com/shaowenchen/applab 1. Building an Application with AppLab Clone the repository 1 git clone https://x:xxx@applab.chenshaowen.com/applab/git/app.git Let the Agent build it 1 claude Once you describe what you need, just sit back and wait — you will see output like this: 1 2 3 4 5 6WeChat Official AccountAppLabVibeCodingAI AgentPlatformDeploymentR&Dhttps://www.chenshaowen.com/en/blog/llms-txt.htmlSun, 20 Sep 2026 00:00:00 +0000Sun, 20 Sep 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/llms-txt.html1. What llms.txt Is llms.txt is a convention: put a Markdown file at the root of your site, and use plain language to tell large language models and AI agents what the site is, which content is worth reading, and what each piece covers. It was proposed by Jeremy Howard (Answer.WeChat Official AccountAIllms.txtGEOR&Dhttps://www.chenshaowen.com/en/blog/deepseek-harness-remote-node.htmlSat, 12 Sep 2026 00:00:00 +0000Sat, 12 Sep 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/deepseek-harness-remote-node.html1. Insufficient Context Is the Key Constraint on AI Agent Adoption The quality of an Agent’s work depends both on model capability and on the environmental context it can reach. For development and operations, the harder part is that the environment is often fragmented: logs live on production, docs in a knowledge base, code in a Git repository.WeChat Official AccountAIDeepSeek HarnessAgentRemote ExecutionR&Dhttps://www.chenshaowen.com/en/blog/rdma-ops.htmlSat, 22 Aug 2026 00:00:00 +0000Sat, 22 Aug 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/rdma-ops.html1. RoCE 1.1 Connectivity Requirements RDMA requires the same class of network end to end, for example the same /16 subnet. Single switch. RoCE: host NIC ── Ethernet switch (PFC) ── peer NIC Multiple switches. RoCE: host NIC ── Leaf ── Spine ── Leaf ── peer NIC When there are multiple hops, note:WeChat Official AccountRDMARoCEInfiniBandNetworkOperationsHigh Performance Computinghttps://www.chenshaowen.com/en/blog/will-middle-platform-disappear-under-ai-agent.htmlThu, 20 Aug 2026 00:00:00 +0000Thu, 20 Aug 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/will-middle-platform-disappear-under-ai-agent.html1. Business Bypasses the Middle Platform At the moment I work on Infra at a fairly large AI middle-platform team. After a department meeting early this year, I started wondering whether the middle platform would disappear. OpenClaw was very popular at the time. Business colleagues using OpenClaw with a large model could already get a great many things done without cooperating with other teams — the division of labor disappeared.WeChat Official AccountAIAgentMiddle PlatformPlatformThoughtsR&Dhttps://www.chenshaowen.com/en/blog/ops-storage-knowledge.htmlSat, 15 Aug 2026 00:00:00 +0000Sat, 15 Aug 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/ops-storage-knowledge.html1 Volatile Memory 1.1 REG REG (Register) is a storage unit inside the CPU, sitting right next to the arithmetic unit — the fastest and smallest level of the storage hierarchy. General-purpose registers (RAX/RBX/RDI on x86, and so on) take part directly in instruction execution, with access latency around 0.2–0.5 ns (roughly one clock cycle), lost on power loss, and typically only tens to hundreds of bytes.WeChat Official AccountOperationsStorageSystemsArchitecturehttps://www.chenshaowen.com/en/blog/openspec-spec-driven-development-for-ai.htmlFri, 31 Jul 2026 00:00:00 +0000Fri, 31 Jul 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/openspec-spec-driven-development-for-ai.html1. What OpenSpec Is OpenSpec is an open-source Spec-Driven Development framework maintained by Fission-AI. It does not try to teach AI new coding tricks. Instead it adds a lightweight “agreement layer” between you and the AI — before any code is written, both sides settle on what to do, in writing.WeChat Official AccountAIOpenSpecClaude CodeCursorR&Dhttps://www.chenshaowen.com/en/blog/add-huawei-npu-node-to-kubernetes-cluster.htmlWed, 22 Jul 2026 00:00:00 +0000Wed, 22 Jul 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/add-huawei-npu-node-to-kubernetes-cluster.html1. Disk Handling 1.1 Inspect Disks Inspect the new disk 1 fdisk -l Disk /dev/nvme1n1: 3.91 TiB, 4294967296000 bytes, 8388608000 sectors 1.2 Build a RAID0 If you have several small disks, a better approach is to build a RAID0 — this gives you not only a larger storage directory but also higher speed.WeChat Official AccountAINPUAscendHuaweiHardwareDriversKuberneteshttps://www.chenshaowen.com/en/blog/loop-device-lifecycle-management.htmlMon, 13 Jul 2026 00:00:00 +0000Mon, 13 Jul 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/loop-device-lifecycle-management.html1. Creating the Image File 1.1 When Loop Storage Fits Treat an ordinary file as a /dev/loopN block device: reads and writes go through the block device interface, while the backing file is still just a file on the host. Mounting ISO and disk images Local practice and stress testingWeChat Official AccountLinuxStorageOperationsloophttps://www.chenshaowen.com/en/blog/ubuntu-package-dependency-and-change-management.htmlMon, 13 Jul 2026 00:00:00 +0000Mon, 13 Jul 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/ubuntu-package-dependency-and-change-management.html1. Viewing Package Dependencies 1.1 Forward Dependencies Which packages it depends on List dependencies 1 apt-cache depends apparmor 1 2 3 4 5 6 7 8 9 10 11 12 13 14 apparmor Depends: debconf # 必须 Depends: lsb-base # 必须 |Depends: debconf # 必须 Depends: <debconf-2.0>WeChat Official AccountUbuntuLinuxaptOperationsPackage Managementhttps://www.chenshaowen.com/en/blog/container-deploy-ceph.htmlSun, 12 Jul 2026 00:00:00 +0000Sun, 12 Jul 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/container-deploy-ceph.htmlThis article covers single-node validation and three-node production deployment with cephadm. For architecture, operations, and functional testing, see Ceph Architecture and Operations. 1. Disk Preparation An OSD can only use a block device (/dev/sdX, a partition, /dev/loopN). 1.1 Inspecting Available Devices 1 2 3 4 5 6 7 lsblk -oWeChat Official AccountMiddlewareContainersCephStorageOperationshttps://www.chenshaowen.com/en/blog/container-deploy-ceph-architecture-operations-and-testing.htmlSat, 11 Jul 2026 00:00:00 +0000Sat, 11 Jul 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/container-deploy-ceph-architecture-operations-and-testing.htmlCeph is a unified distributed storage system that supports block storage (RBD), object storage (RGW), and file storage (CephFS). The official cephadm can orchestrate each daemon as a container (Docker, or Podman + Containerd); see Deploying Ceph in Containers for details. 1. Architecture 1.1 Topology and Data Paths The diagram below shows a common five-node topology (3 MON + 2 MGR + 5 OSD): MONs only need to come in odd numbers, and in a five-node deployment 3 is enough (tolerating one MON failure); 5 MONs tolerate 2 failures, but the metadata overhead is larger, so there is generally no need to run a MON on every machine.WeChat Official AccountMiddlewareContainersCephStorageOperationsArchitecturehttps://www.chenshaowen.com/en/blog/container-deploy-victoriametrics.htmlSat, 11 Jul 2026 00:00:00 +0000Sat, 11 Jul 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/container-deploy-victoriametrics.htmlVictoriaMetrics (VM for short) is a time-series database compatible with the Prometheus ecosystem. It uses fewer resources than Prometheus and comes in both single-node and cluster editions. The single-node edition can directly replace Prometheus storage; the cluster edition splits reads, writes, and storage across vmstorage / vminsert / vmselect, giving it stronger horizontal scaling.WeChat Official AccountMiddlewareContainersVictoriaMetricsPrometheusMonitoringOperationshttps://www.chenshaowen.com/en/blog/agent-skills-engineering-workflows-for-ai-agents.htmlSun, 05 Jul 2026 00:00:00 +0000Sun, 05 Jul 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/agent-skills-engineering-workflows-for-ai-agents.html1. What Agent Skills Is Agent Skills is a production-grade engineering skill pack for AI coding assistants. It encodes the way senior engineers work — their working methods, quality gates, and best practices across the full Define → Plan → Build → Verify → Review → Ship lifecycle — into Markdown workflows (Skills) that an agent can execute.WeChat Official AccountAICursorClaudeAgent SkillsR&Dhttps://www.chenshaowen.com/en/blog/headroom-reduce-ai-agent-tokens.htmlFri, 03 Jul 2026 00:00:00 +0000Fri, 03 Jul 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/headroom-reduce-ai-agent-tokens.html1. What Headroom Is Headroom is a context compression layer for AI agents. Before the LLM receives a request, it performs content-aware compression on tool output, logs, RAG chunks, file contents, and conversation history, cutting token usage to 5%–40% of the original while keeping answer quality as unchanged as possible.WeChat Official AccountAIMCPCursorHeadroomR&Dhttps://www.chenshaowen.com/en/blog/claude-code-custom-model-config-and-tips.htmlWed, 01 Jul 2026 00:00:00 +0000Wed, 01 Jul 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/claude-code-custom-model-config-and-tips.htmlReferences: Claude Code Configuration, Environment Variables, Connecting to an LLM Gateway Claude Code talks to the Anthropic Messages API. When connecting to a custom gateway, use the env block of settings.json to override the endpoint and the model. 1. Where the Configuration File Lives The user configuration is at ~/.WeChat Official AccountAIClaude CodeProgramming AssistantR&Dhttps://www.chenshaowen.com/en/blog/codex-custom-model-config-and-tips.htmlWed, 01 Jul 2026 00:00:00 +0000Wed, 01 Jul 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/codex-custom-model-config-and-tips.htmlReferences: Codex Configuration Basics, Codex Advanced Configuration Codex uses the OpenAI Responses API. When connecting to a custom gateway, define the endpoint with [model_providers.<id>], and the top-level model and model_provider determine the actual call. 1. Where the Configuration File Lives The user configuration is at ~/.codex/config.toml by default (on Windows,WeChat Official AccountAICodexProgramming AssistantR&Dhttps://www.chenshaowen.com/en/blog/opencode-custom-model-config-and-tips.htmlWed, 01 Jul 2026 00:00:00 +0000Wed, 01 Jul 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/opencode-custom-model-config-and-tips.htmlReferences: OpenCode Config, Providers OpenCode uses the OpenAI Chat Completions API. When connecting to a custom gateway, declare the endpoint and the model list in the provider block of opencode.jsonc. 1. Installation and Configuration File 1 npm install -g opencode-ai The user configuration is at ~/.config/opencode/opencode.jsonc by default (opencode.json alsoWeChat Official AccountAIOpenCodeProgramming AssistantR&Dhttps://www.chenshaowen.com/en/blog/separate-kubernetes-events-to-dedicated-etcd.htmlMon, 29 Jun 2026 00:00:00 +0000Mon, 29 Jun 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/separate-kubernetes-events-to-dedicated-etcd.htmlThe disk pressure on the original etcd is reduced to below 5%, the risk factor is very low, and the cluster shows no anomalies. 1. Deploying the Events Etcd Cluster Events have lower consistency requirements than the main Etcd. Deploying a 3-node cluster is recommended. SSD is recommended for the disk, and the data directory should be kept separate from the main Etcd.WeChat Official AccountKubernetesEtcdEventsOperationsPerformance Optimizationbloghttps://www.chenshaowen.com/en/blog/litellm-unified-model-api-format.htmlWed, 24 Jun 2026 00:00:00 +0000Wed, 24 Jun 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/litellm-unified-model-api-format.html1. What LiteLLM Is LiteLLM is an open-source LLM adapter: call 100+ APIs (OpenAI, Anthropic, Gemini, Bedrock, Azure, Ollama, vLLM, Qwen, and more) using the standard OpenAI format. Two ways to use it: Python SDK — called directly from your code AI Gateway — a team-level proxy with authentication, billing,WeChat Official AccountAILiteLLMLLMGatewayR&Dhttps://www.chenshaowen.com/en/blog/codegraph-help-ai-understand-codebase.htmlThu, 18 Jun 2026 00:00:00 +0000Thu, 18 Jun 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/codegraph-help-ai-understand-codebase.html1. What CodeGraph Is CodeGraph is a local-first code intelligence tool. It parses a codebase with tree-sitter, stores symbols, relationships, and files in a local SQLite database, and exposes them as a queryable knowledge graph through MCP, a CLI, and a TypeScript API. Put simply, it turns “grep, glob, and Read all over the code” into an index built ahead of time, so an AI assistant can answer structural questions with just a few queries.WeChat Official AccountAIMCPCursorCodeGraphR&Dhttps://www.chenshaowen.com/en/blog/superpowers-software-engineering-methodology-for-ai.htmlMon, 01 Jun 2026 00:00:00 +0000Mon, 01 Jun 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/superpowers-software-engineering-methodology-for-ai.html1. What Superpowers Is Superpowers is a plugin that bolts “engineering discipline” onto a coding agent, maintained by Prime Radiant. It is not a library of prompt templates but a complete software engineering methodology built on top of Skills — at session startup a bootstrap mechanism makes the AI automaticallyWeChat Official AccountAISuperpowersCursorClaude CodeR&Dhttps://www.chenshaowen.com/en/blog/openspec-align-requirements-before-coding.htmlTue, 19 May 2026 00:00:00 +0000Tue, 19 May 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/openspec-align-requirements-before-coding.html1. What OpenSpec Is OpenSpec is an open-source, lightweight AI-assisted spec-driven development framework. It stores specs and changes in Markdown files, drives the AI assistant with slash commands (such as /opsx:propose and /opsx:apply), and uses a CLI to manage state, validate format, and archive changes. In short, it turns “requirements talked out in a conversation” into a “traceable, iterable, archivable contract” — align before writing code, correct at any point during implementation, and when you are done, let the change settle into the source of truth for system behavior.WeChat Official AccountAIOpenSpecCursorR&Dhttps://www.chenshaowen.com/en/blog/comfyui-wan2-2-video-generation-examples.htmlTue, 05 May 2026 00:00:00 +0000Tue, 05 May 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/comfyui-wan2-2-video-generation-examples.htmlReference: ComfyUI official Wan2.2 tutorial 1. Download the Models Enter the ComfyUI models directory 1 cd models Download the repository 1 GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/Comfy-Org/Wan_2.2_ComfyUI_Repackaged.git Download the models 1 2 cd Wan_2.2_ComfyUI_Repackaged git lfs pull& 2. 5B Hybrid Edition (Text-to-Video + Image-to-Video) 2.1 Download the Workflow 1 wget https://raw.githubusercontent.com/Comfy-Org/workflow_templates/refs/heads/main/templates/video_wan2_2_5B_ti2v.json Or click Workflow → Browse Templates → Video → Wan2.WeChat Official AccountAIComfyUIWan2.2Video Generationhttps://www.chenshaowen.com/en/blog/inference-app-troubleshooting-checklist.htmlFri, 01 May 2026 00:00:00 +0000Fri, 01 May 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/inference-app-troubleshooting-checklist.htmlPod View the child processes of PID 1 1 ps --ppid 1 -f View TIME_WAIT connections 1 netstat -anp | grep TIME_WAIT Istio Ingress 1 kubectl -n istio-ingress logs -l app=istio-ingress |grep vlm-tag-v1 EastWestGateway 1 kubectl -n istio-ingress logs -l app=istio-eastwestgateway |grep vlm-tag-v1 Slow XDS Push 1 2 3 4 5 histogram_quantile(0.WeChat Official AccountAIInferenceTroubleshootingKubernetesIstioOperationshttps://www.chenshaowen.com/en/blog/elasticsearch-troubleshooting.htmlThu, 30 Apr 2026 00:00:00 +0000Thu, 30 Apr 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/elasticsearch-troubleshooting.htmlHandling an Elasticsearch Index in the Red State GET _cat/shards?v=true&h=index,shard,prirep,state,node,unassigned.reason&s=state 1 2 ops-pod-loggie-2026.06.11 0 p UNASSIGNED NODE_LEFT ops-pod-loggie-2026.06.11 0 r UNASSIGNED ALLOCATION_FAILED Try to reallocate. 1 POST _cluster/reroute?retry_failed=true Or delete it outright. 1 DELETE ops-pod-loggie-2026.06.11 Scheduled Index Cleanup Keep all indexes for only 7 days. Create an ILM policy 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 PUT _ilm/policy/ops-delete-after-7d { "policy": { "phases": { "hot": { "actions": {} }, "delete": { "min_age": "7d", "actions": { "delete": {} } } } } } Create an index template 1 2 3 4 5 6 7 8 9 10 PUT _index_template/ops-delete-after-7d { "index_patterns": ["*", "-.WeChat Official AccountElasticsearchOperationsTroubleshootinghttps://www.chenshaowen.com/en/blog/deploy-hermes-agent-under-containers.htmlSun, 19 Apr 2026 00:00:00 +0000Sun, 19 Apr 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/deploy-hermes-agent-under-containers.html1. Starting the Container Set the image 1 export IMAGE=nousresearch/hermes-agent:v2026.6.5 Set directory permissions 1 2 mkdir -p hermes-agent-home chmod -R 777 hermes-agent-home Set environment variables 1 2 3 cat <<EOF > hermes-agent-home/.env GATEWAY_ALLOW_ALL_USERS=true EOF Start hermes-agent 1 2 3 4 5 6 7 8 9 10 nerdctl run -d \WeChat Official AccountHermesAIContainersDockerOperationsAgenthttps://www.chenshaowen.com/en/blog/deploy-juicefs-csi-driver.htmlSat, 18 Apr 2026 00:00:00 +0000Sat, 18 Apr 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/deploy-juicefs-csi-driver.html1. Installation and Deployment Download the deployment file 1 wget https://raw.githubusercontent.com/juicedata/juicefs-csi-driver/master/deploy/k8s.yaml Check kubelet’s install directory on any non-master node 1 ps -ef|grep kubelet |grep root-dir If there is a value, you need to perform the replacement; otherwise install directly. 1 sed 's@/var/lib/kubelet@{{KUBELET_DIR}}@g' k8s.yaml 1 kubectl apply -f k8s.yaml Check theWeChat Official AccountJuiceFSStoragehttps://www.chenshaowen.com/en/blog/use-a2a-gateway-to-connect-multiple-openclaw.htmlWed, 15 Apr 2026 00:00:00 +0000Wed, 15 Apr 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/use-a2a-gateway-to-connect-multiple-openclaw.html1. Architecture 1 2 3 4 5 6 7 ┌──────────────────────┐ A2A/JSON-RPC ┌───────────────WeChat Official AccountOpenClawA2AAIAgenthttps://www.chenshaowen.com/en/blog/loggie-deployment-and-configuration.htmlTue, 14 Apr 2026 00:00:00 +0000Tue, 14 Apr 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/loggie-deployment-and-configuration.html1. Download the Chart Package 1 git clone https://github.com/loggie-io/installation 2. Install loggie 1 cd installation/helm-chart 1 helm install loggie ./ -nloggie --create-namespace 3. Global Configuration Changes Remove the timestamp in front of stdout 1 kubectl -n loggie edit cm loggie-config-loggie Set parseStdout to true to remove the timestamp in front of stdout.WeChat Official AccountLoggieLoggingStorageDeploymentKubernetesOperationshttps://www.chenshaowen.com/en/blog/building-the-flywheel-for-openclaw-operation.htmlSat, 11 Apr 2026 00:00:00 +0000Sat, 11 Apr 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/building-the-flywheel-for-openclaw-operation.html1. Production Cases Cleaning up a node Restarting a node Deleting an abnormal Pod Inspecting the cluster 2. Two Steps to Get Configured Step one: configure MCP. Hand this configuration to OpenClaw and it will set it up for you. 1 2 3 4 5 6 7 8 { "mcpServers": { "ops-mcp-server": { "baseUrl": "http://ops-mcp-server/mcp", "headers": { "Authorization": "Bearer xxx" } } } } Step two: install skills.WeChat Official AccountOpenClawTroubleshootingOperationsAgenthttps://www.chenshaowen.com/en/blog/how-to-land-ai-agent-sops.htmlSat, 28 Mar 2026 00:00:00 +0000Sat, 28 Mar 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-land-ai-agent-sops.html1. Establish a SOPS Baseline SOPS stands for Standard Operating Procedures. In the production activities we carry out, there are often some basic, repetitive operational flows, and these operations are the SOPS. You need to build up a set of SOPS; by combining them you can finish tasks faster. In the past, establishing standard procedures required a senior engineer, but now with AI, producing SOPS is very easy.WeChat Official AccountAIAI AgentMCPLLMR&Dhttps://www.chenshaowen.com/en/blog/agfs-file-system-performance-test.htmlThu, 26 Mar 2026 00:00:00 +0000Thu, 26 Mar 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/agfs-file-system-performance-test.html1. Installing wrk 1 apt-get install -y wrk 2. localfs Performance Test 2.1 1KB Small File Test Write file 1 2 3 4 5 6 7 8 9 10 11 12 13 14 cat > put-1k.lua <<"EOF" wrk.method = "PUT" wrk.body = string.rep("a", 1024) wrk.headers["Content-Type"] = "text/plain" counter = 0 local WRAP = 500000 request = function() counter = counter + 1 local n = ((counter - 1) % WRAP) + 1 local filename = "/local/perf-1k/test_" .WeChat Official AccountAIAGFSStoragePerformance Testinghttps://www.chenshaowen.com/en/blog/agfs-file-system-for-ai-agent.htmlWed, 25 Mar 2026 11:22:55 +0000Wed, 25 Mar 2026 11:22:55 +0000https://www.chenshaowen.com/en/blog/agfs-file-system-for-ai-agent.html1. Introduction to AGFS Overview AGFS is a project that provides storage services through a RESTful API. It supports a variety of storage backends, including memory, files, databases, message queues, and more. Because it does not need POSIX interface support, data can be stored through remote calls, which is very useful in AI Agents.WeChat Official AccountAIAGFSStorageValidationDeploymenthttps://www.chenshaowen.com/en/blog/openviking-usage-and-openclaw-integration.htmlTue, 24 Mar 2026 00:00:00 +0000Tue, 24 Mar 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/openviking-usage-and-openclaw-integration.html1. Introduction to OpenViking 1.1 Use Cases Problem How OpenViking Handles It Fragmented context Unified management with a filesystem paradigm: memory, resources, and skills all map to a viking:// virtual directory, so context is managed the way files are High token consumption L0/L1/L2 tiering: the L0 summary (~100 tokens) isWeChat Official AccountOpenVikingOpenClawAI Agenthttps://www.chenshaowen.com/en/blog/how-to-send-messages-to-openclaw-remotely.htmlThu, 05 Mar 2026 00:00:00 +0000Thu, 05 Mar 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-send-messages-to-openclaw-remotely.html1. Configure hooks Configure hooks in openclaw.json 1 2 3 4 5 6 7 "hooks": { "enabled": true, "token": "xxx", "path": "/openclaw/hooks", "allowRequestSessionKey": true, "allowedAgentIds": ["*"] }, See https://docs.openclaw.ai/automation/webhook The token here must not be the same as the gateway’s token. 2. Sending a message 1 2 3 4 5 6 7 8 9 10 11 12 curl --location 'https://xxx.WeChat Official AccountOpenClawMessagingAutomationOperationsAI Agenthttps://www.chenshaowen.com/en/blog/deploy-openclaw-under-containers.htmlWed, 04 Mar 2026 00:00:00 +0000Wed, 04 Mar 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/deploy-openclaw-under-containers.html1. Start the Container Set the image 1 export IMAGE=ghcr.io/openclaw/openclaw:2026.3.2 Generate a random token 1 openssl rand -hex 32 Set the gateway token 1 export OPENCLAW_GATEWAY_TOKEN=xxx This token is used for authentication on both the remote web UI and the local TUI. Set directory permissions 1 2 mkdir -p openclaw-homeWeChat Official AccountOpenClawAIContainersDockerOperationsAgenthttps://www.chenshaowen.com/en/blog/using-skills-and-mcp-to-extend-openclaw.htmlSun, 01 Mar 2026 00:00:00 +0000Sun, 01 Mar 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-skills-and-mcp-to-extend-openclaw.html1. What Skills Are 1 2 3 openclaw skills list Skills (7/51 ready) OpenClaw ships with a number of built-in skills, which you can view with openclaw skills list. The concrete implementations of these Skills can be found at https://github.com/openclaw/openclaw/tree/main/skills. A Skills file mainly describes the Skill’s metadata, when itWeChat Official AccountAIAI AgentSkillsMCPOpenClawOperationsLearninghttps://www.chenshaowen.com/en/blog/installation-and-configuration-of-openclaw.htmlThu, 26 Feb 2026 00:00:00 +0000Thu, 26 Feb 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/installation-and-configuration-of-openclaw.html1. Preparing the Node Environment Install nvm 1 curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.4/install.sh | bash Set the nvm environment variables You need to edit Bash’s config file and add the following: 1 2 3 export NVM_DIR="$HOME/.nvm" [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" [ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion" Install Node 1 nvm install 22 default 2.WeChat Official AccountOpenClawAIAgentLLMToolsR&DLearninghttps://www.chenshaowen.com/en/blog/lvm-daily-maintenance.htmlFri, 20 Feb 2026 00:00:00 +0000Fri, 20 Feb 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/lvm-daily-maintenance.html1. Basic Concepts Physical Volume A PV (Physical Volume) is made up of one or more physical hard disks, and can also be a RAID device. Volume Group A VG (Volume Group) is made up of one or more PVs. Logical Volume Space carved out of a VG as an LV (Logical Volume); after formatting, it is mounted to a directory for use.WeChat Official AccountLVMStorageOperationshttps://www.chenshaowen.com/en/blog/minio-infrastructure-selection-and-optimization.htmlThu, 19 Feb 2026 00:00:00 +0000Thu, 19 Feb 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/minio-infrastructure-selection-and-optimization.html1. Hardware 1.1 CPU Support for AVX/AVX-512, used for erasure coding computation and SIMD operations Support for AES-NI, used for object encryption Total storage per node Recommended vCPU count Up to 1 TB 8 Up to 10 TB 16 Up to 100 TB 32 Up to 1 PB 64 MoreWeChat Official AccountMinIOStorageDiskCPUMemoryNetworkTuninghttps://www.chenshaowen.com/en/blog/container-deploy-clickhouse.htmlWed, 11 Feb 2026 00:00:00 +0000Wed, 11 Feb 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/container-deploy-clickhouse.html1. ClickHouse Single Node 1.1 Configure Environment Variables 1 2 3 4 5 6 7 8 export CONTAINER_CLI=nerdctl export IMAGE=clickhouse/clickhouse-server:24 export CLICKHOUSE_INSTANCE_NAME=clickhouse export CH_DATA=/data/ops/clickhouse/$CLICKHOUSE_INSTANCE_NAME export CLICKHOUSE_PORT=9000 export CLICKHOUSE_PROMETHEUS_PORT=9363 export CLICKHOUSE_USER=default export CLICKHOUSE_PASSWORD=xxxxxx 1.2 Generate the Configuration Files 1 2 3 4 5 6 7 8 9 10 11 12 13WeChat Official AccountMiddlewareContainersClickHousePrometheusMonitoringOperationshttps://www.chenshaowen.com/en/blog/container-deploy-redis.htmlWed, 11 Feb 2026 00:00:00 +0000Wed, 11 Feb 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/container-deploy-redis.html1. Redis Single Instance Configure environment variables 1 2 3 4 5 6 7 8 9 export CONTAINER_CLI=nerdctl export IMAGE=redis:7 export REDIS_INSTANCE_NAME=redis-instance export REDIS_PASSWORD=xxxxxx export REDIS_PORT=6379 export REDIS_DATA=/data/ops/redis/$REDIS_INSTANCE_NAME mkdir -p $REDIS_DATA/data $REDIS_DATA/log Start the service 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 $CONTAINER_CLI run -d \ --name $REDIS_INSTANCE_NAME \ --restart always \ --security-opt apparmor=unconfined \ --security-opt seccomp=unconfined \ --network host \ --ulimit memlock=-1 \ --ulimit stack=67108864 \ --ulimit nofile=1048576:1048576 \ --memory-swappiness=0 \ -v $REDIS_DATA/data:/data \ -v $REDIS_DATA/log:/var/log/redis \ $IMAGE redis-server --port $REDIS_PORT --requirepass $REDIS_PASSWORD \ --maxmemory 64gb \ --maxmemory-policy noeviction \ --appendonly yes \ --appendfsync everysec Test the connection A PONG response means the connection succeededWeChat Official AccountMiddlewareContainersRedisOperationshttps://www.chenshaowen.com/en/blog/container-deploy-multi-node-foundationdb-and-maintenance.htmlTue, 10 Feb 2026 00:00:00 +0000Tue, 10 Feb 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/container-deploy-multi-node-foundationdb-and-maintenance.html1. Generate a Cluster ID 1 cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 16 | head -n 1 The examples below use CLUSTER_ID=fKbIga9RHP79OIx1. 2. On the First Node Configure environment variables 1 2 3 4 5 6 7 8 9 10 export CONTAINER_CLI=nerdctl export IMAGE=foundationdb/foundationdb:7.1.26 export CLUSTER_ID=fKbIga9RHP79OIx1 export FDB_INSTANCE_NAME=fdb_server export FDB_CLUSTER_FIRST_IP=$(hostname -I | awk '{print $1}') export FDB_PUBLIC_IP=$(hostname -I | awk '{print $1}') export FDB_PORT=4500 export FDB_DIR=/data/ops/fdb/$FDB_INSTANCE_NAME Clean up old data 1 2 3 $CONTAINER_CLI rm -f $FDB_INSTANCE_NAME mv $FDB_DIR $FDB_DIR.WeChat Official AccountFoundationDBDatabaseMiddlewareStorageContainers3FSOperationshttps://www.chenshaowen.com/en/blog/multi-node-multi-disk-minio-cluster-performance-test-in-different-erasure-code-configurations.htmlSun, 08 Feb 2026 00:00:00 +0000Sun, 08 Feb 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/multi-node-multi-disk-minio-cluster-performance-test-in-different-erasure-code-configurations.html1. Erasure Code Configuration The default erasure parity configuration is as follows: Erasure Set Size Default Parity (EC:M) 1 EC:0 2-3 EC:1 4-5 EC:2 6 - 7 EC:3 8 - 16 EC:4 The maximum parity is ERASURE_SET_SIZE/2, meaning the number of parity blocks cannot exceed the number of data blocks.WeChat Official AccountMinIOStorageOperationsPerformanceTestingBenchmarkingPerformance Testinghttps://www.chenshaowen.com/en/blog/multi-node-multi-disk-minio-cluster-performance-test-in-different-erasure-code-configurations-on-ipoib.htmlSun, 08 Feb 2026 00:00:00 +0000Sun, 08 Feb 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/multi-node-multi-disk-minio-cluster-performance-test-in-different-erasure-code-configurations-on-ipoib.htmlThe earlier tests found the bottleneck in the NIC; this post adds supplementary testing under IPoIB. 1. MinIO Cluster Environment 1.1 Creating the MinIO Cluster 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 export CONTAINER_CLI=nerdctl export IMAGE=minio/minio:RELEASE.WeChat Official AccountMinIOStorageDistributed SystemsErasure CodingIPoIBPerformance TestingBenchmarkinghttps://www.chenshaowen.com/en/blog/using-haproxy-to-proxy-minio-and-performance-testing.htmlSun, 08 Feb 2026 00:00:00 +0000Sun, 08 Feb 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-haproxy-to-proxy-minio-and-performance-testing.html1. Preparing the Configuration Files MinIO cluster 1 mc admin info local 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 ● minio1:9000 Uptime: 55 minutes Version: 2025-04-22T22:12:26Z Network: 2/2 OK Drives: 4/4 OK Pool: 1 ● minio2:9000 Uptime: 55 minutes Version: 2025-04-22T22:12:26Z Network: 2/2 OK Drives: 4/4 OK Pool: 1 ┌──────┬───────────────────────┬─────────────────────┬──────────────┐ │ Pool │ Drives Usage │ Erasure stripe size │ Erasure sets │ │ 1st │ 0.WeChat Official AccountMinIOHAProxyStorageProxyPerformance TestingBenchmarkinghttps://www.chenshaowen.com/en/blog/minio-multi-node-multi-disk-deployment-and-maintenance.htmlSat, 07 Feb 2026 00:00:00 +0000Sat, 07 Feb 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/minio-multi-node-multi-disk-deployment-and-maintenance.html1. Environment Preparation 1.1 Preparing Data Disks Inspect the data disks 1 lsblk -d -o NAME,SIZE,TYPE | grep nvme 1 2 3 4 nvme0n1 745.2G disk nvme1n1 745.2G disk nvme2n1 745.2G disk nvme3n1 745.2G disk Prepare the storage directories 1 2 3 for i in {0..3}; do mkdir -p /mnt/data${i} done Format the data disks 1 2 3 for i in {0.WeChat Official AccountMinIOStorageDistributed SystemsOperationsLearningObject StorageS3DeploymentHigh Availabilityhttps://www.chenshaowen.com/en/blog/how-to-survive-in-the-cursor-world.htmlSun, 01 Feb 2026 10:00:00 +0000Sun, 01 Feb 2026 10:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-survive-in-the-cursor-world.html1. The Quantitative Change AI IDEs Have Brought AI generates a large amount of code that enters engineering work. AI IDEs are simply too good to use: from naming completion in the past, to generating function implementations, and now to delivering the code for an entire feature. You do not even need to know how to write code — through chatting you can complete many software iteration tasks.WeChat Official AccountThoughtsAIAI AgentAgentCursorhttps://www.chenshaowen.com/en/blog/using-mcp-to-power-ai-agent.htmlThu, 29 Jan 2026 00:00:00 +0000Thu, 29 Jan 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-mcp-to-power-ai-agent.html1. What Is MCP MCP is short for Model Context Protocol, a protocol for passing context between models and applications. The greatest value of MCP is that, as the model wars are winding down and AI applications are about to explode, it gives developers a unified standard for invoking capabilities beyond the model itself.WeChat Official AccountAIAI AgentMCPLLMR&Dhttps://www.chenshaowen.com/en/blog/test-and-deploy-pd-disagg-app-with-erdma-on-aliyun.htmlSat, 17 Jan 2026 00:00:00 +0000Sat, 17 Jan 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/test-and-deploy-pd-disagg-app-with-erdma-on-aliyun.htmlIn a PD disaggregated deployment, heterogeneous GPU models are often used to deploy the model across machines, which multiplies the cross-machine communication pressure. RDMA devices are usually brought in to accelerate kvcache transfer between nodes so as to achieve a lower FTTL. This post describes how to test eRDMA devicesWeChat Official AccountAlibaba CloudeRDMARDMAPDDisaggregationAIApplicationOperationshttps://www.chenshaowen.com/en/blog/using-ops-event-to-power-ai-agent.htmlWed, 14 Jan 2026 10:00:00 +0000Wed, 14 Jan 2026 10:00:00 +0000https://www.chenshaowen.com/en/blog/using-ops-event-to-power-ai-agent.html1. Why Operations Events Less invasive Operations teams usually already have observations accumulated in the Metrics and Log dimensions. Operations events can be a new entry point: introduce new operational capabilities without affecting the stability of existing systems. Better suited to automation Metrics represent system state, Log represents specific code behavior, while Event represents changes in components.WeChat Official AccountAI AgentAIOperationsLLMhttps://www.chenshaowen.com/en/blog/some-common-llm-agent-infrastructure.htmlSun, 21 Dec 2025 00:00:00 +0000Sun, 21 Dec 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/some-common-llm-agent-infrastructure.html1. Prompt Chaining Prompt chaining breaks a task down into a series of steps, where each LLM call processes the output of the previous step. 2. Routing Routing classifies the input and directs it to a specialized follow-up task. 3. Parallelization An LLM can sometimes work on a task concurrently and aggregate its outputs programmatically.WeChat Official AccountAILLMAgentR&DArchitectureLearningBest Practiceshttps://www.chenshaowen.com/en/blog/some-case-of-ai-agent.htmlSat, 29 Nov 2025 00:00:00 +0000Sat, 29 Nov 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/some-case-of-ai-agent.html1. Handling Deterministic Faults For faults that have a specific handling procedure, just let the Agent handle them and send a notification to spread the word. For similar automated handling, we have exception load deletion at the application layer, disk cleanup at the node layer, masking an offline GPU card, masking a node, and so on.WeChat Official AccountAIAI AgentLLMOperationsAutomationhttps://www.chenshaowen.com/en/blog/ext4-xfs-zfs-btrfs-performance-test-in-raid6.htmlMon, 06 Oct 2025 00:00:00 +0000Mon, 06 Oct 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/ext4-xfs-zfs-btrfs-performance-test-in-raid6.html1. Single Disk 1.1 Creating the File System Wipe the disk 1 wipefs -a /dev/nvme0n1 Format the disk 1 mkfs.xfs /dev/nvme0n1 Mount the disk 1 2 mkdir -p /mnt/nvme0n1-test mount /dev/nvme0n1 /mnt/nvme0n1-test 1.2 Performance Test 4k random write 1 2 3 4 echo 3 > /proc/sys/vm/drop_caches fio --name=4k_randwrite --rw=randwrite --bs=4k --ioengine=libaio --direct=1 \ --numjobs=8 --iodepth=32 --size=20G --time_based --runtime=120 --group_reporting \ --allow_mounted_write=1 --filename=/mnt/nvme0n1-test/fio-4k-randwrite.WeChat Official AccountFilesystemext4xfszfsbtrfsRAIDStoragePerformance TestingOperationshttps://www.chenshaowen.com/en/blog/raid-zfs-operations.htmlSun, 05 Oct 2025 00:00:00 +0000Sun, 05 Oct 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/raid-zfs-operations.htmlZFS does not support online shrinking; this post focuses on operational tasks such as creating pools, replacing disks, and expanding capacity. 1. ZFS vdev Types mirror: mirroring, multiple disks hold copies of each other, equivalent to RAID 1 raidz3: triple parity, tolerates 3 disk failures raidz2: double parity, tolerates 2 disk failures, equivalent to RAID 6 raidz1: single parity, tolerates 1 disk failure, equivalent to RAID 5 striped: striping, no redundancy, equivalent to RAID 0 They can also be combined:WeChat Official AccountFilesystemzfsRAIDStorageOperationshttps://www.chenshaowen.com/en/blog/raid-xfs-operations.htmlSat, 04 Oct 2025 00:00:00 +0000Sat, 04 Oct 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/raid-xfs-operations.htmlThe xfs filesystem does not support shrinking. This post is mainly about disk replacement and expansion under RAID 10. 1. RAID States and Disk States In mdadm, the RAID states are: clean, normal degraded, still readable and writable, but with reduced fault tolerance reshaping, reorganization in progress, appears during expansion and shrinking resyncing, data synchronization in progress recovering, data recovery in progress, accompanied by degradation as a disk is rebuilt failed In mdadm, the disk states are:WeChat Official AccountFile SystemxfsRAIDmdadmStorageOperationshttps://www.chenshaowen.com/en/blog/raid-btrfs-operations.htmlFri, 03 Oct 2025 00:00:00 +0000Fri, 03 Oct 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/raid-btrfs-operations.htmlBtrfs supports online expansion, shrinking, disk replacement, and RAID mode conversion; this post covers creating a filesystem under RAID 0/RAID 1/RAID 10/RAID 6 topologies, adding disks, replacing disks, shrinking, and common operational tasks. 1. Btrfs Profiles single: single copy, no redundancy, works with one disk or can aggregate multiple disks.WeChat Official AccountFilesystembtrfsRAIDStorageOperationshttps://www.chenshaowen.com/en/blog/xfs-performance-test-in-various-raid.htmlThu, 02 Oct 2025 00:00:00 +0000Thu, 02 Oct 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/xfs-performance-test-in-various-raid.html1. Disk Baseline Testing 1.1 4k Random Write 1 2 3 4 5 6 7 8 9 10 11 12 13 14 fio \ --name=4k_randwrite \ --rw=randwrite \ --bs=4k \ --ioengine=libaio \ --direct=1 \ --numjobs=8 \ --iodepth=32 \ --size=20G \ --time_based \ --runtime=120 \ --group_reporting \ --allow_mounted_write=1 \ --filename=/data1/fio-4k-randwrite.dat 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 Starting 8 processes 4k_randwrite: Laying out IO file (1 file / 20480MiB) Jobs: 8 (f=8): [w(8)][100.WeChat Official AccountFilesystemxfsRAIDmdadmStoragePerformance Testinghttps://www.chenshaowen.com/en/blog/raid-storage-technology.htmlWed, 01 Oct 2025 00:00:00 +0000Wed, 01 Oct 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/raid-storage-technology.html1. What Is RAID RAID technology combines multiple physical hard drives into a single logical hard drive, so the operating system sees it as one storage device. There are two ways to implement RAID: hardware and software. A hardware implementation is a RAID controller, a standalone hardware device responsible for RAID control and management.WeChat Official AccountRAIDStorageHardwareOperationsLearninghttps://www.chenshaowen.com/en/blog/using-llama-factory-to-fine-tune-qwen3-model.htmlSun, 21 Sep 2025 00:00:00 +0000Sun, 21 Sep 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-llama-factory-to-fine-tune-qwen3-model.html1. Testing the Original Model Start the environment 1 2 3 4 5 6 7 8 9 10 11 12 nerdctl run -it \ --security-opt apparmor=unconfined \ --security-opt seccomp=unconfined \ --gpus all \ --ipc=host \ -p 8000:8000 \ --ulimit memlock=-1 \ --ulimit stack=67108864 \ --name vllm \ --volume /data/models:/data/models \WeChat Official AccountLLaMA-FactoryQwen3Fine-TuningAILLMhttps://www.chenshaowen.com/en/blog/using-vllm-to-deploy-pd-disagg-app.htmlSat, 20 Sep 2025 00:00:00 +0000Sat, 20 Sep 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-vllm-to-deploy-pd-disagg-app.html1. Why Deploy LLM Applications with PD Disaggregation In the process of LLM inference, there are two serial stages: Process the entire input context and generate the KV Cache (Prefill stage) Incrementally generate new tokens (Decode stage) These two stages have different resource requirements. The Prefill stage has to computeWeChat Official AccountvLLMDeploymentPDDisaggregationApplicationhttps://www.chenshaowen.com/en/blog/use-lmcache-to-improve-ttft.htmlWed, 17 Sep 2025 00:00:00 +0000Wed, 17 Sep 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/use-lmcache-to-improve-ttft.html1. Introduction to LMCache TTFT is the time from when a request is issued until the model generates its first token. Because the Prefill phase has to encode the input context into a KV Cache before generation can begin, producing that first token requires a large amount of computation, which makes TTFT high.WeChat Official AccountAILLMvLLMInferencelmcacheOperationshttps://www.chenshaowen.com/en/blog/what-is-prefix-cache.htmlThu, 11 Sep 2025 00:00:00 +0000Thu, 11 Sep 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/what-is-prefix-cache.html1. What Is Prefix Cache In model inference scenarios, caching mechanisms are often used to improve throughput and performance. There are two common caching mechanisms: Key-Value Cache (KV Cache), which is oriented toward the internals of a single request, caching the intermediate computation results (Key and Value) of the Transformer model to avoid redundant computation.WeChat Official AccountAIvLLMLLMCacheInferenceOperationshttps://www.chenshaowen.com/en/blog/nvidia-rtx-5090-inference-test.htmlTue, 09 Sep 2025 00:00:00 +0000Tue, 09 Sep 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/nvidia-rtx-5090-inference-test.html1. Installing the Driver Download the driver Go to https://www.nvidia.com/en-us/drivers/ and select the corresponding driver version to download. 1 wget https://us.download.nvidia.com/XFree86/Linux-x86_64/580.76.05/NVIDIA-Linux-x86_64-580.76.05.run Install the driver 1 bash NVIDIA-Linux-x86_64-580.76.05.run Check the GPUs 1 nvidia-smi 1 2 3 GPU 0: NVIDIA GeForce RTX 5090 (UUID: GPU-92fcdc58-4754-73c7-af6c-56740936817d) GPU 1: NVIDIA GeForce RTX 5090 (UUID:WeChat Official AccountAIGPUNVIDIALLMInferencePerformance TestingOperationshttps://www.chenshaowen.com/en/blog/what-is-model-quantization.htmlSat, 06 Sep 2025 00:00:00 +0000Sat, 06 Sep 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/what-is-model-quantization.html1. What Is Model Quantization Model quantization is the process of converting the weights and activations of a high-precision model (usually 32-bit floating point FP32 or 16-bit floating point FP16) into a low-precision model (such as 8-bit integer INT8). The value range of FP32 is -3.4*10^38 to 3.4*10^38, with 4 billion values.WeChat Official AccountAILLMModel QuantizationOptimizationOperationshttps://www.chenshaowen.com/en/blog/kubernetes-base-environment-requirements.htmlThu, 04 Sep 2025 00:00:00 +0000Thu, 04 Sep 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/kubernetes-base-environment-requirements.html1. Base Packages Kubernetes version ≥ 1.18 socat Required conntrack Required ebtables Optional, but recommended ipset Optional, but recommended ipvsadm Optional, but recommended 1 apt-get -y install socat conntrack ebtables ipset ipvsadm 2. Port Requirements 2.1 Base Node Services Service Protocol Action Start Port End Port Comment ssh TCP allow 22 22 Node remote management docker TCP allow 2375 2376 Docker remote communication etcd TCP allow 2379 2380 etcd cluster communication 2.WeChat Official AccountKubernetesClusterConfigurationSecurityOperationsLearningInstallationDependenciesBest Practiceshttps://www.chenshaowen.com/en/blog/how-upgrade-kubernetes-node-cgroup-version.htmlTue, 02 Sep 2025 00:00:00 +0000Tue, 02 Sep 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-upgrade-kubernetes-node-cgroup-version.html1. Interface Path Differences Between cgroup v1 and v2 v1 1 2 3 4 /sys/fs/cgroup/cpu/cpu.cfs_quota_us /sys/fs/cgroup/cpu/cpuacct.usage /sys/fs/cgroup/memory/memory.limit_in_bytes /sys/fs/cgroup/memory/memory.usage_in_bytes v2 1 2 3 4 /sys/fs/cgroup/cpu.max /sys/fs/cgroup/cpu.stat /sys/fs/cgroup/memory.max /sys/fs/cgroup/memory.current cgroup v2 is the upgraded version of v1, with advantages such as more unified resource hierarchy management and precise resource isolation. But it also means that when writing code, the paths for reading the relevant interface files are different, so compatibility handling is required.WeChat Official AccountKubernetesUpgradeOperationscgroupLearningNodesResource IsolationTroubleshootingKubelethttps://www.chenshaowen.com/en/blog/using-claude-code-on-server.htmlFri, 29 Aug 2025 00:00:00 +0000Fri, 29 Aug 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-claude-code-on-server.htmlThe same applies to a desktop environment. 1. Prepare the node.js environment Install nvm 1 curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.3/install.sh | bash 1 2 echo 'export NVM_DIR="$([ -z "${XDG_CONFIG_HOME-}" ] && printf %s "${HOME}/.nvm" || printf %s "${XDG_CONFIG_HOME}/nvm")"' >> ~/.bashrc echo '[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"' >> ~/.bashrc 1 sourceWeChat Official AccountClaudeCLIAILLMToolsR&DOperationshttps://www.chenshaowen.com/en/blog/common-npu-operation-and-fault-handling.htmlWed, 20 Aug 2025 00:00:00 +0000Wed, 20 Aug 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/common-npu-operation-and-fault-handling.htmlContent referenced or recorded while troubleshooting, continuously updated 1. Mounting Devices into a Container 1 export IMAGE=ascendai/pytorch:2.1.0 1 2 3 4 5 6 7 8 9 10 nerdctl run --security-opt apparmor=unconfined --security-opt seccomp=unconfined --rm -it --ipc=host \ --device=/dev/davinci7 \ --device=/dev/davinci_manager \ --device=/dev/devmm_svm \ --device=/dev/hisi_hdc \ -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \ -v /usr/local/Ascend/add-ons/:/usr/local/Ascend/add-ons/ \ -v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \ ${IMAGE} \ /bin/bash 2.WeChat Official AccountNPUAIHardwareOperationsFault Handlinghttps://www.chenshaowen.com/en/blog/the-use-guide-of-nvidia-dcgm.htmlWed, 13 Aug 2025 00:00:00 +0000Wed, 13 Aug 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-use-guide-of-nvidia-dcgm.html1. What DCGM Is DCGM (Data Center GPU Manager) is a toolset NVIDIA provides for managing and monitoring GPUs in the data center. It offers the following capabilities: GPU behavior monitoring GPU configuration management GPU policy supervision GPU health and diagnostics GPU accounting and process statistics NVSwitch configuration and monitoringWeChat Official AccountAINVIDIADCGMGPUMonitoringOperationsTroubleshootinghttps://www.chenshaowen.com/en/blog/how-to-clean-zombie-processes.htmlThu, 07 Aug 2025 00:00:00 +0000Thu, 07 Aug 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-clean-zombie-processes.html1. What a Zombie Process Is How a process is created: The parent process calls fork() to create a child process The child process calls exec() to load a new program The child process finishes executing and calls exit() or returns The parent process calls wait() or waitpid() If the parent process never calls wait() or waitpid(), the child stays in the system after it finishes, becoming a zombie process.WeChat Official AccountOperationsCleanupGPUNVIDIALinuxhttps://www.chenshaowen.com/en/blog/using-vllm-benchmark-to-test-model-performance.htmlTue, 05 Aug 2025 00:00:00 +0000Tue, 05 Aug 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-vllm-benchmark-to-test-model-performance.htmlVLLM Benchmark is a tool provided by VLLM for testing model performance, supporting a variety of inference backends. This article mainly records some of the process of using VLLM Benchmark to test model performance. 1. Starting the Model Service 1 2 3 4 5 6 7 8 9 10 11 12 13 14 python -m vllm.WeChat Official AccountAIVLLMBenchmarkingModel PerformanceTestinghttps://www.chenshaowen.com/en/blog/a-journey-with-kubesphere.htmlFri, 01 Aug 2025 00:00:00 +0000Fri, 01 Aug 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/a-journey-with-kubesphere.html1. A Slap in the Face: Deleting the Repo, the Images, the Docs On Ray’s last day at QingCloud as the project’s initiator, the frontend, docs, and images of the open source project KubeSphere were deleted, and users received an [Announcement on the Adjustment of the KubeSphere Open Source Project] through various channels.WeChat Official AccountKubeSphereKubernetesOpen SourceThoughtshttps://www.chenshaowen.com/en/blog/gb-standards.htmlWed, 30 Jul 2025 00:00:00 +0000Wed, 30 Jul 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/gb-standards.htmlGB/T 22239-2019 Information Security Technology — Baseline for Classified Protection of Cybersecurity - PDF Download GB/T 43697-2024 Data Security Technology — Rules for Data Classification and Grading - PDF DownloadWeChat Official AccountSecurityStandardshttps://www.chenshaowen.com/en/blog/deploy-dragonfly-on-kubernetes.htmlMon, 28 Jul 2025 00:00:00 +0000Mon, 28 Jul 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/deploy-dragonfly-on-kubernetes.html1. Deploy NFS Storage 1.1 Configure the NFS Service on the Storage Node Install the NFS service on the storage node 1 apt-get install -y nfs-kernel-server Start the NFS service and enable it at boot 1 systemctl start nfs-server.service & systemctl enable nfs-server.service Configure the NFS shared directory 1 mkdir -p /data/nfs 1 vim /etc/exports /data/nfs *(rw,sync,no_root_squash,no_all_squash) Restart the NFS service 1 systemctl restart nfs-server.WeChat Official AccountKubernetesDragonflyImagesImage AccelerationAIOperationshttps://www.chenshaowen.com/en/blog/the-use-of-history.htmlSun, 13 Jul 2025 00:00:00 +0000Sun, 13 Jul 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-use-of-history.html1. The history Command The history command lets you view the command history of the current session. When you exit the shell, the command history is saved to the corresponding history file. For bash, the command history is saved in the ~/.bash_history file; for zsh, it is saved in ~/.zsh_history. You can also use the HISTFILE environment variable to specify the location of the history file: export HISTFILE=/data/ops/bash_history.WeChat Official AccounthistoryCommand LineToolsLinuxOperationshttps://www.chenshaowen.com/en/blog/ai-related-papers.htmlSat, 05 Jul 2025 00:00:00 +0000Sat, 05 Jul 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/ai-related-papers.html2025 Intrinsic Fingerprint of LLMs [Published: 07-04] View The paper proposes a robust fingerprinting method based on the inter-layer standard-deviation distribution pattern of attention parameter matrices (Q/K/V/O), used to detect lineage relationships between large language models (LLMs) — for example, whether one model was derived from another through continued training, fine-tuning, or an upgrade.WeChat Official AccountAIPapersStorageLLMhttps://www.chenshaowen.com/en/blog/using-gemini-cli-on-server.htmlFri, 04 Jul 2025 00:00:00 +0000Fri, 04 Jul 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-gemini-cli-on-server.html1. Get the Project ID and API Key Visit https://aistudio.google.com/apikey Get the project ID and API key, and set them as environment variables 1 2 echo "export GOOGLE_CLOUD_PROJECT=" >> ~/.bashrc echo "export GEMINI_API_KEY=" >> ~/.bashrc 1 source ~/.bashrc 2. Prepare the Node.js Environment Install nvm 1 curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.3/install.sh |WeChat Official AccountGeminiCLIAILLMLearningToolsOperationsProgramming Assistanthttps://www.chenshaowen.com/en/blog/ascend-dmi-tool-usage-guide.htmlTue, 17 Jun 2025 00:00:00 +0000Tue, 17 Jun 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/ascend-dmi-tool-usage-guide.html1. Installing Dependencies MindCluster ToolBox is a system-level toolset for cluster operations and hardware management, mainly used for operations such as device monitoring, performance testing, and log collection. MindCluster ToolBox includes the Ascend DMI tool, the log collection tool, and the Ascend Cert tool. MindCluster ToolBox works by calling low-level DCMI (Device Control Management Interface) and AscendCL (Ascend Computing Language) interfaces to perform the relevant detection functions; system-level information queries are implemented by calling the general-purpose libraries provided by the system.WeChat Official AccountAIDMIAscendHuaweiNPUTroubleshootingPerformance Testinghttps://www.chenshaowen.com/en/blog/kubernetes-gpu-virtualization.htmlSat, 14 Jun 2025 00:00:00 +0000Sat, 14 Jun 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/kubernetes-gpu-virtualization.html1. k8s-device-plugin https://github.com/NVIDIA/k8s-device-plugin is the official Kubernetes device plugin from NVIDIA, used to manage and allocate NVIDIA GPU resources in a Kubernetes cluster. By interacting with kubelet, k8s-device-plugin automatically discovers and registers GPU devices and exposes them to the Kubernetes scheduler as resources. It supports many GPU models and is able to handle GPU partitioning and sharing.WeChat Official AccountKubernetesGPUAIVirtualizationhttps://www.chenshaowen.com/en/blog/foundationdb-data-backup-and-restore.htmlTue, 10 Jun 2025 00:00:00 +0000Tue, 10 Jun 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/foundationdb-data-backup-and-restore.html1. Set Environment Variables Set the foundationdb cluster name and version 1 2 3 4 5 export CLUSTER_NAME= export NAMESPACE= export DEPLOY_NAME=${CLUSTER_NAME}-deployment export SECRET_NAME=fdb-backup-secret export VERSION=7.1.26 Set the S3 backup bucket and credentials 1 2 3 4 5 export BUCKET= export HOST=s3.us-west-2.amazonaws.com export AK=x export SK=x export S3_BACKUP_URI="blobstore://${AK}:${SK}@${HOST}/fdb-backup?bucket=${BUCKET}&sc=0" 2. Create the Credentials 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 cat <<EOF | kubectl apply -f - apiVersion: v1 kind: Secret metadata: name: ${SECRET_NAME} namespace: ${NAMESPACE} type: Opaque stringData: credentials: | { "accounts": { "${BUCKET}@${HOST}": { "access_key": "${AK}", "secret": "${SK}", "endpoint": "${HOST}" } } } EOF 3.WeChat Official AccountDatabaseFoundationDBKubernetesBackupOperationshttps://www.chenshaowen.com/en/blog/book/the-cognitive-awakening.htmlSun, 08 Jun 2025 00:00:00 +0000Sun, 08 Jun 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/book/the-cognitive-awakening.htmlSubtitle: The Driving Force That Initiates Self-Change Author: Zhou Ling Publisher: Posts & Telecom Press Publication Year: 2020-10 ISBN: 9787115543424 Notes: The book is divided into two parts. The first part is inward-looking: understanding yourself. Starting from the triune brain theory, it explains how the brain works. The instinctive brain maintains basic physiological needs, the emotional brain handles emotion and social interaction, and the rational brain processes logic and decision-making.WeChat Official AccountBooksCognitionPsychologyGrowthhttps://www.chenshaowen.com/en/blog/how-to-add-basic-auth-for-node-exporter.htmlSat, 24 May 2025 00:00:00 +0000Sat, 24 May 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-add-basic-auth-for-node-exporter.html1. Background Node Exporter is a common component in the Prometheus ecosystem for collecting host metrics, but by default it does not provide access authentication. This article describes how to add basic authentication to Node Exporter in a Kubernetes environment to improve security. 2. Configuring Credentials for Node Exporter 2.1 Generating an Encrypted Password Use the htpasswd tool to generate an encrypted password:WeChat Official AccountexporterCollectorSecurityKubernetesPrometheusMonitoringOperationsLearningNode ExporterBasic AuthAuthenticationhttps://www.chenshaowen.com/en/blog/deploy-smokeping-prober.htmlSun, 11 May 2025 00:00:00 +0000Sun, 11 May 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/deploy-smokeping-prober.html1. What smokeping-prober Is smokeping-prober is a tool for probing network quality. It probes network quality by sending ICMP requests to target nodes. 2. Deploying smokeping-prober 2.1 Generating the smokeping-prober.yaml Configuration File 1 2 3 4 5 6 7 8 9 10 11 cat > smokeping_prober.yaml <<EOF --- targets: - hosts: - 1.WeChat Official AccountsmokepingexporterNetworkMonitoringOperationsPrometheusProbingLearninghttps://www.chenshaowen.com/en/blog/opscli-support-mcp-provide-cline-cursor-integration.htmlFri, 04 Apr 2025 00:00:00 +0000Fri, 04 Apr 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/opscli-support-mcp-provide-cline-cursor-integration.html1. Using Ops from Cline Query which clusters exist Get the details of a node The case is very simple, but the data is real, which shows that Cline is already able to connect to Ops for operations tasks. More complex cases still need more testing. The specific approach might lie in: providing certain Workspaces as context information, setting custom instructions to reinforce certain characteristics, and so on.WeChat Official AccountOpsClineCursorAI AgentMCPOperationshttps://www.chenshaowen.com/en/blog/kube-proxy-abnormal-causes-pod-on-node-unable-to-access-service.htmlMon, 31 Mar 2025 00:00:00 +0000Mon, 31 Mar 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/kube-proxy-abnormal-causes-pod-on-node-unable-to-access-service.html1. Problem Description Related Pods 1 2 3 4 5 6 kubectl -n istio-system get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES istiod-647c7c9d95-7n7n6 1/1 Running 0 77m 10.244.173.51 docs-ai-a800-4 <none> <none> istiod-647c7c9d95-k6l88 1/1 Running 0 30m 10.244.210.160 ai-a40-2 <none> <none> istiod-647c7c9d95-pj82r 1/1 Running 0 51m 10.WeChat Official AccountKuberneteskube-proxyPodServiceOperationshttps://www.chenshaowen.com/en/blog/nfs-over-rdma-vs-nfs-fio-large-block-read-write-performance.htmlSun, 30 Mar 2025 00:00:00 +0000Sun, 30 Mar 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/nfs-over-rdma-vs-nfs-fio-large-block-read-write-performance.html1. Background While testing the performance of NFS Over RDMA, I found that reading a 4M file could actually reach 45GB/s. 1 2 3 fio -numjobs=128 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=read -bs=4M --group_reporting -size=100m -time_based -runtime=30 -name=fio-test -directory=/data1/nfs READ: bw=42.3GiB/s (45.4GB/s), 42.3GiB/s-42.3GiB/s (45.4GB/s-45.4GB/s), io=1269GiB (1363GB), run=30019-30019msec Yet the disk’s 4M multi-threaded read performance is only 6 GB/s.WeChat Official AccountNFSRDMAFIOPerformancehttps://www.chenshaowen.com/en/blog/3fs-key-technologies-and-design.htmlMon, 24 Mar 2025 00:00:00 +0000Mon, 24 Mar 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/3fs-key-technologies-and-design.html1. Direct IO Direct IO bypasses the operating system’s page cache and interacts with hardware devices directly. Characteristics of Direct IO: Lots of new data, so no caching is needed Low memory footprint Sequential reads and writes of large files For synchronous read operations above a threshold (1MB by default), the 3FS client converts them into AIO operations (opening the file in Direct IO mode) to improve read performance.WeChat Official Account3FSStorageAIOperationshttps://www.chenshaowen.com/en/blog/deepseek-3fs-operations-guide.htmlThu, 20 Mar 2025 00:00:00 +0000Thu, 20 Mar 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/deepseek-3fs-operations-guide.htmlNotes on some DeepSeek 3FS operations, continually updated. 1. Basic Concepts and Cautions Chain A Chain is made up of several Targets, and each Target is a replica of the storage. When all commits are ready, all Targets in a Chain are consistent. Targets on the same Chain cannot be on the same node.WeChat Official AccountDeepSeek3FSAIStorageOperationshttps://www.chenshaowen.com/en/blog/deploy-deepseek-3fs-storage-system-under-containers.htmlSat, 15 Mar 2025 00:00:00 +0000Sat, 15 Mar 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/deploy-deepseek-3fs-storage-system-under-containers.html1. Deployment Plan Before starting the containerized deployment, a few requirements: To simplify delivery, only one image is needed For reliability, deploy as many replicas as possible Start different services through different parameters Inject configuration through environment variables and render it into configuration files Below is the deployment plan for DeepSeek 3FS:WeChat Official AccountStorageAIContainersKubernetesDockerOperationsLLM3FShttps://www.chenshaowen.com/en/blog/3fs-performance-test.htmlTue, 11 Mar 2025 00:00:00 +0000Tue, 11 Mar 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/3fs-performance-test.html1. Hardware Tests There are two NVMe SSDs (SAMSUNG MZQL27T6HBLA-00A07) and a 3FS cluster made up of two nodes, each with 4 IB ports. 1.1 Single Thread Disk FIO 128 K, read 1 fio -numjobs=1 -fallocate=none -iodepth=2 -ioengine=libaio -direct=1 -rw=read -bs=128K --group_reporting -size=100M -time_based -runtime=30 -name=fio-test -directory=/data/fio 1 Jobs: 1WeChat Official Account3FSStorageAIPerformance TestingOperationshttps://www.chenshaowen.com/en/blog/fluid-using-3fs-as-pvc.htmlSun, 09 Mar 2025 00:00:00 +0000Sun, 09 Mar 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/fluid-using-3fs-as-pvc.html1. Why Connect 3FS to Fluid 3FS is a distributed storage system open-sourced by DeepSeek. Its exceptionally impressive performance test results have made it a hot topic, and its star count has risen rapidly. The team I work on has also been tracking 3FS technically, looking for suitable application scenarios to get the most value out of our AI hardware infrastructure.WeChat Official AccountFluidJuiceFSAIDatahttps://www.chenshaowen.com/en/blog/nfs-over-rdma.htmlThu, 06 Mar 2025 00:00:00 +0000Thu, 06 Mar 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/nfs-over-rdma.html1. Prerequisites The storage and the nodes that use it form an RDMA network The RDMA devices are configured with IP addresses If you are using Mellanox NICs, add the -with-nfsrdma parameter when installing the driver 1 ./mlnxofedinstall --with-nfsrdma 2. Starting the Server Install dependencies 1 apt install nfs-kernel-server rdma-core -y Load the kernel module 1 modprobe svcrdma Add the port to portlist 1 echo 'rdma 20049' | tee /proc/fs/nfsd/portlist Export directory 1 mkdir /data1/nfs 1 2 3 vim /etc/exports /data1/nfs *(rw,sync,no_root_squash,no_all_squash) Start the NFS service 1 systemctl start nfs-server.WeChat Official AccountNFSRDMAHigh Performance StorageStorageNetworkOperationsLearninghttps://www.chenshaowen.com/en/blog/kubectl-logs-not-found-error.htmlSat, 22 Feb 2025 00:00:00 +0000Sat, 22 Feb 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/kubectl-logs-not-found-error.html1. Symptoms Pod information can be viewed 1 2 3 4 kubectl -n my-testns get pod my-testpod NAME READY STATUS RESTARTS AGE my-testpod 1/1 Running 0 2d13h Pod logs cannot be viewed 1 2 3 kubectl -n my-testns logs my-testpod -f Error from server (NotFound): the server could not find the requested resource ( pods/log my-testpod) On the host where the Pod runs, the container logs can be viewed with docker logs.WeChat Official AccountKuberneteskubectlPodTroubleshootingOperationsLoggingLearningcontainerdhttps://www.chenshaowen.com/en/blog/dante-socks5-proxy.htmlThu, 20 Feb 2025 00:00:00 +0000Thu, 20 Feb 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/dante-socks5-proxy.htmlBecause the servers in the data center are bound to a small bandwidth, this post is mainly about using Dante to provide SOCKS5 forwarding, using some high-bandwidth servers for traffic forwarding to speed up dependency package downloads. 1. Find a High-Bandwidth Server Install speedtest-cli 1 pip3 install speedtest-cli List peer test servers 1 2 3 4 5 6 7 8 9 10 11 12 13 speedtest-cli --secure --list Retrieving speedtest.WeChat Official AccountDanteOperationsBandwidthProxyNetworkhttps://www.chenshaowen.com/en/blog/using-calico-to-limit-pod-bandwidth.htmlWed, 12 Feb 2025 00:00:00 +0000Wed, 12 Feb 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-calico-to-limit-pod-bandwidth.html1. Testing the Full Bandwidth Start the iperf3 server on the target host 1 2 3 4 5 iperf3 -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Test from the client host 1 iperf3 -c x.x.x.x -p 5201 -t 10 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Connecting to host x.WeChat Official AccountCalicoPodBandwidthLimitingKubernetesNetworkOperationsLearningCNITraffic Controlhttps://www.chenshaowen.com/en/blog/what-is-ray.htmlSun, 09 Feb 2025 10:00:00 +0000Sun, 09 Feb 2025 10:00:00 +0000https://www.chenshaowen.com/en/blog/what-is-ray.html1. What Ray Is In 2016, UC Berkeley’s RISELab released a new distributed computing framework called Ray. In 2017, after the Ray paper was published, it drew broad attention across the industry; in China it was mainly Ant Group that adopted and contributed to Ray. In 2020, Ray released versionWeChat Official AccountDistributed SystemsComputing FrameworkRayTrainingInferencehttps://www.chenshaowen.com/en/blog/structure-of-large-models-with-qwen.htmlSun, 09 Feb 2025 00:00:00 +0000Sun, 09 Feb 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/structure-of-large-models-with-qwen.html1. Introduction to the Qwen Model In April 2023, Alibaba released the beta version of Qwen. In December 2023, Alibaba open-sourced the first version of Qwen. In September 2024, Alibaba released Qwen2.5. In January 2025, Alibaba released Qwen 2.5-Max. Qwen 2.5 is the latest series of the Qwen large languageWeChat Official AccountQwenLLMAILearningArchitectureTransformerOpen SourceR&Dhttps://www.chenshaowen.com/en/blog/nvidia-environment-variable-configuration.htmlMon, 20 Jan 2025 01:00:00 +0000Mon, 20 Jan 2025 01:00:00 +0000https://www.chenshaowen.com/en/blog/nvidia-environment-variable-configuration.htmlNVIDIA_VISIBLE_DEVICES Specifies the GPU devices visible to the program 1 CUDA_VISIBLE_DEVICES=0,1 Available values: 1,2, a comma-separated list of GPU UUIDs or indices all, all GPUs none, load the driver but with no GPU access void, do not load the driver NVIDIA_DRIVER_CAPABILITIES Controls which driver libraries/binaries will be installed inside the containerWeChat Official AccountAINVIDIAGPUEnvironment VariablesConfigurationhttps://www.chenshaowen.com/en/blog/ascend-environment-variable-configuration.htmlMon, 20 Jan 2025 00:00:00 +0000Mon, 20 Jan 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/ascend-environment-variable-configuration.htmlHCCL_IF_IP Configures the IP of the root communication NIC for HCCL initialization. Priority: environment variable HCCL_IF_IP > environment variable HCCL_SOCKET_IFNAME > NICs other than docker/lo (in ascending lexicographic order of NIC name) > docker NIC > lo NIC. 1 export HCCL_IF_IP=10.0.0.1 HCCL_IF_BASE_PORT Specifies the starting port number of the Host NIC.WeChat Official AccountAIAscendHuaweiNPUEnvironment VariablesConfigurationhttps://www.chenshaowen.com/en/blog/fluid-juicefs-enterprise-maintenance.htmlSat, 18 Jan 2025 01:00:00 +0000Sat, 18 Jan 2025 01:00:00 +0000https://www.chenshaowen.com/en/blog/fluid-juicefs-enterprise-maintenance.html1. Setting Environment Variables 1 2 export NAMESPACE=xxx export PVC=xxx 2. The Dataset Will Not Become Ready 2.1 Fluid Component Problems 1 kubectl -n fluid-system get pod -o wide | grep -v "Running" There may be cases where something did not start up properly. 2.2 Abnormally Terminated Datasets Abnormal resources can cause Fluid resources to restart repeatedly, and manual intervention is required to delete them.WeChat Official AccountFluidJuiceFSKubernetesStorageAIDataOperationshttps://www.chenshaowen.com/en/blog/use-vllm-for-inference.htmlSat, 18 Jan 2025 00:00:00 +0000Sat, 18 Jan 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/use-vllm-for-inference.html1. Environment Preparation Download Miniforge 1 wget "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh" Install Miniforge 1 bash Miniforge3-$(uname)-$(uname -m).sh 1 2 echo "export PATH=$HOME/miniforge3/bin:$PATH" >> ~/.bashrc source ~/.bashrc Create the environment 1 conda create -n vllm python=3.12 vLLM currently requires Python 3.9+. Activate the environment 1 conda activate vllm Install dependencies 1 conda installWeChat Official AccountvLLMAILLMInferenceOperationshttps://www.chenshaowen.com/en/blog/use-nerdctl-to-build-multi-architecture-images.htmlFri, 17 Jan 2025 00:00:00 +0000Fri, 17 Jan 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/use-nerdctl-to-build-multi-architecture-images.html1. Installing Nerdctl Install Opscli 1 curl -sfL https://raw.githubusercontent.com/shaowenchen/ops/main/getcli.sh |VERSION=latest sh - Install Nerdctl 1 opscli task -f install-nerdctl --arch amd64 2. BuildKit Download Buildkit 1 wget https://github.com/moby/buildkit/releases/download/v0.19.0-rc2/buildkit-v0.19.0-rc2.linux-amd64.tar.gz Install Buildkit 1 2 tar xvf buildkit-*.tar.gz mv bin/* /usr/local/bin/ Configure Buildkitd 1 mkdir -p /etc/buildkit /data/buildkit 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 cat > /etc/buildkit/buildkitd.WeChat Official AccountMulti-ArchImagesCINerdctlKubernetesOperationsLearningCICDcontainerdBuildhttps://www.chenshaowen.com/en/blog/use-vllm-verify-inference-node.htmlThu, 16 Jan 2025 00:00:00 +0000Thu, 16 Jan 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/use-vllm-verify-inference-node.html1. Building the Image To make testing convenient, the model files are packaged into the image here. Download the model 1 2 3 4 git clone https://huggingface.co/Qwen/Qwen1.5-1.8B-Chat cd Qwen1.5-1.8B-Chat && git lfs pull rm -rf .git cd .. Write the Dockerfile 1 2 3 4 5 cat <<EOF > DockerfileWeChat Official AccountvLLMInferenceAILLMOperationshttps://www.chenshaowen.com/en/blog/ai-application-development-tech-stack.htmlSun, 12 Jan 2025 00:00:00 +0000Sun, 12 Jan 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/ai-application-development-tech-stack.htmlEmbedding Models Embedding models map high-dimensional data into a lower-dimensional space, which makes the data easier to process and analyze. Text Models Here is a leaderboard, https://huggingface.co/spaces/mteb/leaderboard The leaderboard above gives each model’s score, parameter count, memory usage, vector dimension, maximum tokens, and other information. Below are some commonly used open-source models:WeChat Official AccountAIApplication DevelopmentTech StackLLMR&DLearningRAGEmbeddingBest Practiceshttps://www.chenshaowen.com/en/blog/how-to-deploy-jumpserver-on-kubernetes.htmlThu, 09 Jan 2025 00:00:00 +0000Thu, 09 Jan 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-deploy-jumpserver-on-kubernetes.html1. Deploying Jumpserver You need a StorageClass ready in advance to store Jumpserver’s data. Besides the database mentioned below, each component — jms-core, jms-web, jms-koko, jms-lion, jms-chen — also needs a PV for storage. 1.1 Deploying MySQL Refer to https://github.com/shaowenchen/ops-hub/blob/main/database/mysql8.yaml to deploy MySQL. You need to adjust the StorageClass fieldWeChat Official AccountKubernetesJumpserverSecurityOperationsBastion HostDeploymentLearninghttps://www.chenshaowen.com/en/blog/use-ops-to-view-and-monitor-cluster-events.htmlWed, 11 Dec 2024 00:00:00 +0000Wed, 11 Dec 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/use-ops-to-view-and-monitor-cluster-events.htmlhttps://github.com/shaowenchen/ops 1. Alerting on Kubernetes Cluster Events Monitor specified keywords 1 2 3 4 5 6 7 8 9 10 11 apiVersion: crd.chenshaowen.com/v1 kind: EventHooks metadata: name: kube-pod-falid namespace: ops-system spec: type: xiezuo url: https://xxx.com/api/v1/webhook/send?key=xxx subject: "ops.clusters.*.namespaces.*.pods.*.event" keywords: - failed A simple configuration like this yields a large number of related alerts.WeChat Official AccountOpsKubernetesOperationsEventsMonitoringAutomationhttps://www.chenshaowen.com/en/blog/using-fluid-and-s3fs-to-access-s3-storage-and-performance-testing.htmlThu, 05 Dec 2024 00:00:00 +0000Thu, 05 Dec 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-fluid-and-s3fs-to-access-s3-storage-and-performance-testing.htmlThis post uses Fluid 1.0. In later versions the configuration file paths changed, so adjust according to your actual situation. 1. Building the Image 1.1 fluid_config_init.py 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 #!WeChat Official Accounts3FluidJuiceFSAIDatahttps://www.chenshaowen.com/en/blog/using-fluid-to-access-s3-storage-and-performance-testing.htmlWed, 04 Dec 2024 00:00:00 +0000Wed, 04 Dec 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-fluid-to-access-s3-storage-and-performance-testing.html1. Jindo Mounting S3 Set the environment variables 1 2 3 4 export ENDPOINT=obs.ap-southeast-3.myhuaweicloud.com export BUCKET= export AK= export SK= Create the credentials 1 2 3 4 5 6 7 8 9 10 kubectl apply -f - <<EOF apiVersion: v1 kind: Secret metadata: name: mys3secret type: Opaque stringData: fs.s3.accessKeyId: ${AK} fs.WeChat Official AccountKubernetesFluidJuiceFSS3StorageData ManagementPerformance TestingAIOperationshttps://www.chenshaowen.com/en/blog/using-tensorboard-to-visualize-pytorch-training-process.htmlSun, 17 Nov 2024 00:00:00 +0000Sun, 17 Nov 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-tensorboard-to-visualize-pytorch-training-process.html1. What TensorBoard Is TensorBoard is mainly used to monitor how a model’s various metrics change — accuracy, loss, the weight distributions of each layer, and so on. TensorBoard is a visualization tool from TensorFlow that supports visualizing scalar, text, image, audio, video, and embedding data, among other types. But PyTorch can use TensorBoard too.WeChat Official AccountAIPyTorchTensorBoardTrainingR&Dhttps://www.chenshaowen.com/en/blog/using-pytorch-to-train-model-on-mnist-dataset.htmlSat, 16 Nov 2024 00:00:00 +0000Sat, 16 Nov 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-pytorch-to-train-model-on-mnist-dataset.html1. Create the Training Script Create a training script mnist.py with the following contents: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36WeChat Official AccountAIPyTorchTrainingContainershttps://www.chenshaowen.com/en/blog/set-specific-kernel-version-in-ubuntu/Thu, 14 Nov 2024 00:00:00 +0000Thu, 14 Nov 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/set-specific-kernel-version-in-ubuntu/1. Installing a New Kernel Version Installing from the apt repositories is recommended, so you avoid installing an incompatible kernel version. 1.1 Installing from the Repository List the available versions 1 2 3 4 apt list linux-headers-5.15.*-*-generic linux-image-5.15.*-*-generic linux-image-5.15.0-94-generic/focal-updates,focal-security 5.15.0-94.104~20.04.1 amd64 linux-image-5.15.0-97-generic/focal-updates,focal-security 5.15.0-97.107~20.04.1 amd64 Install the kernel 1 apt install linux-image-5.WeChat Official AccountUbuntuLinuxKernelOperationshttps://www.chenshaowen.com/en/blog/mpi-communication-primitives-and-python-programming.htmlSun, 10 Nov 2024 00:00:00 +0000Sun, 10 Nov 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/mpi-communication-primitives-and-python-programming.html1. What MPI Is MPI, the Message Passing Interface, is a communication protocol used for parallel computing. MPI provides a set of standardized interfaces for transferring data between different compute nodes, and is widely used in scientific computing, machine learning, deep learning, and other fields. MPI has multiple implementations; theWeChat Official AccountMPIAIDistributed SystemsCommunicationhttps://www.chenshaowen.com/en/blog/ops-v1.0.0.htmlThu, 07 Nov 2024 11:22:55 +0000Thu, 07 Nov 2024 11:22:55 +0000https://www.chenshaowen.com/en/blog/ops-v1.0.0.htmlBased on real production needs, I recently made several important updates to https://github.com/shaowenchen/ops and released version v1.0.0. Here I mainly introduce the key features of this release. 1. Support for Multi-Cluster Task Execution In practice, the recommendation is: Create the hosts of the current cluster as Hosts You can create multiple Clusters, and the Cluster objects you own are the clusters under management Task and Pipeline objects are automatically synchronized to all Clusters under the cluster, with no manual trigger required.WeChat Official AccountOpsOperationsKubernetesToolshttps://www.chenshaowen.com/en/blog/common-network-topology.htmlWed, 06 Nov 2024 00:00:00 +0000Wed, 06 Nov 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/common-network-topology.html1. Fat-Tree In 1985, Charles E. Leiserson of MIT invented the Fat-Tree network. As shown in the figure below, a fat-tree network is a binary tree in which bandwidth gradually increases from the root nodes down to the leaf nodes. In August 2008, a group of computer scientists at UC San Diego published a scalable network architecture design that adopted a topology inspired by the fat-tree topology, achieving better scalability than previous hierarchical networks.WeChat Official AccountAINetworkTopologyOperationsLearningData CenterFat-Treehttps://www.chenshaowen.com/en/blog/rdma-technique.htmlTue, 05 Nov 2024 00:00:00 +0000Tue, 05 Nov 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/rdma-technique.html1. What RDMA Is RDMA (Remote Direct Memory Access) is a technology created to solve the delay of server-side data processing in network transmission. With TCP/IP transmission, data passes through the network stack, then through the NIC to be sent; after the receiver receives it, the data is reassembled according to sequence numbers.WeChat Official AccountRDMANetworkHigh Performance ComputingHardwareAIOperationshttps://www.chenshaowen.com/en/blog/sr-iov-technique.htmlSun, 03 Nov 2024 00:00:00 +0000Sun, 03 Nov 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/sr-iov-technique.html1. What SR-IOV Technology Is SR-IOV (Single Root I/O Virtualization) is a virtualization technology that lets virtual machines and containers access physical hardware resources directly, improving I/O performance and reducing host CPU consumption. As shown in the figure above, SR-IOV partitions a single physical device (for example a network interfaceWeChat Official AccountVirtualizationSR-IOVNetworkOperationsGPUhttps://www.chenshaowen.com/en/blog/infiniband-network-and-useful-commands.htmlSat, 02 Nov 2024 00:00:00 +0000Sat, 02 Nov 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/infiniband-network-and-useful-commands.html1. InfiniBand Network InfiniBand (abbreviated IB) is a computer network communication standard for high-performance computing. It offers extremely high throughput and very low latency, and is used for data interconnection between computers. InfiniBand is also used as a direct or switched interconnect between servers and storage systems, as well as between storage systems themselves.WeChat Official AccountInfiniBandNetworkHigh Performance ComputingHardwareOperationshttps://www.chenshaowen.com/en/blog/using-fluid-to-access-obs-storage-and-performance-testing.htmlTue, 22 Oct 2024 00:00:00 +0000Tue, 22 Oct 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-fluid-to-access-obs-storage-and-performance-testing.html1. Jindo Mounting OBS Set the environment variables 1 2 3 4 export ENDPOINT=obs.cn-north-4.myhuaweicloud.com export BUCKET= export AK= export SK= Create the credentials 1 2 3 4 5 6 7 8 9 10 kubectl apply -f - <<EOF apiVersion: v1 kind: Secret metadata: name: myobssecret type: Opaque stringData: fs.obs.accessKeyId: ${AK} fs.WeChat Official AccountOBSFluidJuiceFSKubernetesAIDataOperationshttps://www.chenshaowen.com/en/blog/use-iceberg-and-spark-on-kubernetes.htmlThu, 12 Sep 2024 00:00:00 +0000Thu, 12 Sep 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/use-iceberg-and-spark-on-kubernetes.html1. Data Processing Architecture It is mainly divided into four layers: Processing capability layer: Spark on Kubernetes provides streaming data processing capability Data management layer: Iceberg provides dataset access operations such as ACID and tables Storage layer: Hive MetaStore manages Iceberg table metadata, PostgreSQL serves as the storage backend forWeChat Official AccountSparkIcebergKubernetesBig DataData LakeOperationsData ProcessingLearninghttps://www.chenshaowen.com/en/blog/how-to-enable-gds-on-gpu-host.htmlWed, 11 Sep 2024 00:00:00 +0000Wed, 11 Sep 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-enable-gds-on-gpu-host.html1. What GDS (GPUDirectStorage) Is GDS lets an RDMA NIC access GPU memory directly, which helps increase the IO bandwidth for GPU applications reading and writing files, reduces IO latency, and lowers their CPU load. Once the client has GDS enabled, files are opened in O_DIRECT mode and the client no longer caches file data.WeChat Official AccountGDSGPUAIOperationsStoragehttps://www.chenshaowen.com/en/blog/what-is-token.htmlTue, 10 Sep 2024 10:00:00 +0000Tue, 10 Sep 2024 10:00:00 +0000https://www.chenshaowen.com/en/blog/what-is-token.htmlA token is a unit tightly bound to data. It can be used to measure how much corpus is needed to train a model, and also to measure the input and output length at inference time. 1. What a token is A token can be a whole word, a subword,WeChat Official AccountMachine LearningLLMAITokenLearningNLPInferencehttps://www.chenshaowen.com/en/blog/what-is-flops.htmlMon, 09 Sep 2024 10:00:00 +0000Mon, 09 Sep 2024 10:00:00 +0000https://www.chenshaowen.com/en/blog/what-is-flops.html1. About FLOPs FLOPs (Floating Point Operations Per Second) refers to the number of floating-point operations performed per second. Specifically: A single floating-point addition, such as a + b, counts as one floating-point operation. A single floating-point multiplication, such as a * b, also counts as one floating-point operation. Other basic floating-point operations, such as division and square root, can also be counted as one floating-point operation.WeChat Official AccountAIMachine LearningLLMFLOPsLearningComputePerformanceMetricshttps://www.chenshaowen.com/en/blog/what-is-pd-separation.htmlSun, 08 Sep 2024 10:00:00 +0000Sun, 08 Sep 2024 10:00:00 +0000https://www.chenshaowen.com/en/blog/what-is-pd-separation.html1. Definition There are two completely different phases in LLM inference, and PD separation is The compute-intensive Prefill phase, LLM processes all users’ input and computes the corresponding KV Cache The memory-intensive Decode phase, sequentially producing tokens one by one, computing only a single token per memory access 2. Metrics 2.WeChat Official AccountMachine LearningLLMPDWhat Ishttps://www.chenshaowen.com/en/blog/kubernetes-dlrover-workflow-analysis.htmlTue, 27 Aug 2024 00:00:00 +0000Tue, 27 Aug 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/kubernetes-dlrover-workflow-analysis.htmlThe DLRover version used in this article is 0.3.7 1. DLRover Operator 1.1 Starting the ElasticJob and ScalePlan Controllers Implementation code: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 // 创建 ElasticJob 的控制器 if err = controllers.NewElasticJobReconciler(mgr, masterImage).SetupWithManager(mgr); errWeChat Official AccountDLRoverKubernetesAITrainingFault Self-HealingOperationshttps://www.chenshaowen.com/en/blog/nvidia-gpu-cores-and-architecture-evolution-history.htmlSun, 25 Aug 2024 00:00:00 +0000Sun, 25 Aug 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/nvidia-gpu-cores-and-architecture-evolution-history.html1. Product Lines GeForce Aimed at gamers, offering powerful graphics processing capabilities and advanced gaming technologies. Common ones include the NVIDIA GTX series, the high-end RTX series, and the Titan series. Quadro Aimed at the professional market, such as designers, engineers, scientists, and content creators. Common ones include the Quadro P series and the high-end Quadro RTX series.WeChat Official AccountAIGPUNVIDIAHardwareArchitectureHistoryhttps://www.chenshaowen.com/en/blog/data-parallel-architecture-in-distributed-train.htmlWed, 21 Aug 2024 00:00:00 +0000Wed, 21 Aug 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/data-parallel-architecture-in-distributed-train.html1. Parameter Server Architecture In the Parameter Server architecture, the nodes in the cluster are divided into two categories: parameter server nodes (Parameter Server) and worker server nodes (Worker). 1.1 Parameter Server The Parameter Server is used to store the model’s parameters. Each parameter server node is responsible for managing and updating one part of the model’s parameters, while each worker node only processes the subset of data corresponding to it.WeChat Official AccountDistributed SystemsTrainingData ParallelismAIOperationshttps://www.chenshaowen.com/en/blog/use-dlrover-to-manage-training-job.htmlSat, 17 Aug 2024 00:00:00 +0000Sat, 17 Aug 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/use-dlrover-to-manage-training-job.html1. Problems Facing Distributed Training Estimating training resources is difficult and cannot be automated How much compute, how much time, how much bandwidth, how many CPUs, how much memory — without enough accumulated experience it is hard to estimate accurately. The result is over-requesting and over-allocation, causing enormous resource waste.WeChat Official AccountDLRoverAITrainingKubernetesElastic TrainingFault-Tolerant TrainingOperationshttps://www.chenshaowen.com/en/blog/develop-a-copilot-to-handle-exceptions.htmlWed, 14 Aug 2024 00:00:00 +0000Wed, 14 Aug 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/develop-a-copilot-to-handle-exceptions.htmlThis post comes mainly from an internal sharing session, and is also a summary of some recent work. 1. The Common Failure Handling Process The figure above shows a typical operations exception handling process. Along the timeline, there are the following key points in time: Failure occurs Failure detected FailureWeChat Official AccountLLMOperationsExceptionsFailuresSharinghttps://www.chenshaowen.com/en/blog/migrate-juicefs-metadata-from-redis-to-pgsql.htmlMon, 12 Aug 2024 00:00:00 +0000Mon, 12 Aug 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/migrate-juicefs-metadata-from-redis-to-pgsql.html1. Why Migrate Metadata Storage from Redis to PGSQL PGSQL is cheaper Redis stores metadata in memory, while PGSQL stores metadata on disk, so the cost difference is obvious. PGSQL performance is tunable Different PGSQL services offer different performance IOPS. If you do not have a continuously high performance requirement, PGSQL is a good choice.WeChat Official AccountJuiceFSRedisPGSQLOperationsStorageLearningMigrationMetadataTroubleshootinghttps://www.chenshaowen.com/en/blog/use-volcano-to-run-hccl-test.htmlSun, 11 Aug 2024 00:00:00 +0000Sun, 11 Aug 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/use-volcano-to-run-hccl-test.html1. Building the hccl-test Image Download the dependency packages Python-3.8.18.tgz Ascend-cann-toolkit_8.0.RC2_linux-x86_64.run Ascend-cann-kernels-910b_8.0.RC2_linux.run mpich-3.2.1.tar.gz If downloading is inconvenient, you can also copy them straight out of the image I built. Write the Dockerfile 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 FROM ubuntu:22.WeChat Official AccountVolcanoHCCLAscendAITestinghttps://www.chenshaowen.com/en/blog/use-volcano-to-run-nccl-test.htmlSun, 11 Aug 2024 00:00:00 +0000Sun, 11 Aug 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/use-volcano-to-run-nccl-test.html1. Building the nccl-test Image Check the CUDA version 1 2 3 nvidia-smi | grep "CUDA Version" | awk '{print $9}' 12.2 Write the Dockerfile 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 cat > Dockerfile << EOF FROM nvidia/cuda:12.1.0-cudnn8-devel-ubuntu22.04 ENV DEBIAN_FRONTEND=noninteractive ARG CONDA_VERSION WORKDIR /workspace ENV DEBIAN_FRONTEND=noninteractive RUN apt-get update && apt install -y openmpi-bin libopenmpi-dev ssh openssh-server net-tools vim git iputils-ping nfs-common RUN git clone https://github.WeChat Official AccountVolcanoNCCLNvidiaAITestinghttps://www.chenshaowen.com/en/blog/why-top-node-free-grafana-data-not-match.htmlFri, 26 Jul 2024 01:00:00 +0000Fri, 26 Jul 2024 01:00:00 +0000https://www.chenshaowen.com/en/blog/why-top-node-free-grafana-data-not-match.html1. top Shows Node Resource Usage Above 100% 1 2 3 4 5 6 kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% master-1 995m 16% 13760Mi 118% master-2 827m 13% 10672Mi 92% master-3 889m 14% 10244Mi 88% This is because usage is computed against allocatable resources by default, which excludes the portion reserved by Kubelet.WeChat Official AccountMonitoringMemoryKubernetesGrafanaOperationsTroubleshootinghttps://www.chenshaowen.com/en/blog/how-to-get-device-information-on-server.htmlThu, 25 Jul 2024 00:08:00 +0000Thu, 25 Jul 2024 00:08:00 +0000https://www.chenshaowen.com/en/blog/how-to-get-device-information-on-server.html1. Checking the CPU Check the CPU model 1 lscpu Check the instruction set 1 cat /proc/cpuinfo | grep -iE "flags|instruction set" 2. Checking Memory Memory usage and size 1 free -h Memory module model 1 dmidecode -t memory 3. Checking Disks Check disk mounts 1 lsblk -o NAME,TYPE,SIZE,MODEL,UUID,MOUNTPOINT Check disk usage 1 df -H | grep -vE '^Filesystem|tmpfs|cdrom|loop|udev' | awk '{ print $5 "/" $2 " " $1 }' |grep " "/ Check disk speed 1 hdparm -t /dev/sda3 Check disk temperature and more 1 apt install smartmontools 1 (for disk in /dev/sd?WeChat Official AccountServerDevicesOperationsHardwareTroubleshootingLearningLinuxhttps://www.chenshaowen.com/en/blog/some-things-about-storage-system-in-model-development-process.htmlTue, 23 Jul 2024 00:00:00 +0000Tue, 23 Jul 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/some-things-about-storage-system-in-model-development-process.htmlThis article is compiled from part of an internal sharing session I gave. 1. Core Elements of a Storage System 1.1 Security Credentials for object storage buckets, authorization when using storage PVCs, and control over access sources — these are all issues security needs to address. But they are also very easy to overlook, and when something goes wrong it is a big problem.WeChat Official AccountAIModel DataR&D ProcessStorageLLMOperationsArchitectureThoughtshttps://www.chenshaowen.com/en/blog/something-about-memoryfs-storage-system.htmlWed, 12 Jun 2024 00:00:00 +0000Wed, 12 Jun 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/something-about-memoryfs-storage-system.html1. Using Memory as a Storage Medium The image above is the storage pyramid, showing the relationship between the price of a storage medium and its speed. Today, enterprises widely adopt tape and disks as storage media mainly because of their price advantage. Market prices are driven by supply and demand: even for an expensive storage medium, if the production cost is low, the market is large enough, and the yield rate in the production process is high, its market price will come down.WeChat Official AccountStorageMemoryMemoryFSAIThoughtsArchitectureHigh PerformanceLearninghttps://www.chenshaowen.com/en/blog/using-fluid-to-access-oss-storage-and-performance-testing.htmlTue, 11 Jun 2024 00:00:00 +0000Tue, 11 Jun 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-fluid-to-access-oss-storage-and-performance-testing.html1. Jindo Directly Accelerating OSS Set the environment variables 1 2 3 4 5 6 7 export ENDPOINT=oss-cn-beijing-internal.aliyuncs.com export BUCKET= export AK= export SK= export NAMESPACE=default export PVC=myoss-jindo export URI=datacenter Create the credentials 1 2 3 4 5 6 7 8 9 10 11 kubectl apply -f - <<EOF apiVersion:WeChat Official AccountOSSFluidJuiceFSAIDatahttps://www.chenshaowen.com/en/blog/how-to-warmup-juicefs-data.htmlSun, 09 Jun 2024 00:00:00 +0000Sun, 09 Jun 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-warmup-juicefs-data.html1. About JuiceFS Caching On a host, warmed cache is placed directly on the host. In a cluster, there are two levels of cache: Worker, which provides cache shared at the cluster level Fuse, which provides cache at the current node level only 2. Warming Data with the JuiceFS Client Note that warming at the Fuse layer is only effective for the current node; if you need to warm the whole cluster, you must warm at the Worker layer.WeChat Official AccountFluidJuiceFSAIDataKubernetesStorageOperationsLearningCacheLLMhttps://www.chenshaowen.com/en/blog/high-frequency-io-pod-not-suitable-for-setting-limit.htmlMon, 03 Jun 2024 21:00:16 +0000Mon, 03 Jun 2024 21:00:16 +0000https://www.chenshaowen.com/en/blog/high-frequency-io-pod-not-suitable-for-setting-limit.html1. Symptoms Elasticsearch on Kubernetes was restarting frequently, leaving the service almost unusable. During data import, the Pod’s memory usage kept growing Once the Pod’s memory usage approached the Limit, continued importing triggered an abnormal Pod exit, with the error log ERROR: Elasticsearch exited unexpectedly The Pod’s memory usage did not drop; it stayed near the Limit, and before long the Pod exited abnormally again The Elasticsearch Pod’s memory limit was 64GB, while the JVM memory limit was 32GB.WeChat Official AccountIOKubernetesElasticsearchOperationshttps://www.chenshaowen.com/en/blog/deploy-elasticsearch-using-memory-storage.htmlWed, 29 May 2024 00:00:00 +0000Wed, 29 May 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/deploy-elasticsearch-using-memory-storage.html1. Mount a Memory Storage Directory on the Host Create a directory to mount 1 mkdir /mnt/memory_storage Mount the tmpfs filesystem 1 mount -t tmpfs -o size=800G tmpfs /mnt/memory_storage Storage is allocated on demand, so using 100G of storage only consumes 100G of memory. The host node has 2T of memory, and here 800G is allocated to store Elasticsearch data.WeChat Official AccountElasticsearchHigh PerformanceFull-Text SearchOperationsStorageSearchhttps://www.chenshaowen.com/en/blog/ascend-npu-driver-installation.htmlTue, 28 May 2024 00:00:00 +0000Tue, 28 May 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/ascend-npu-driver-installation.html1. Install the Driver Create the HwHiAiUser user 1 2 groupadd -g 1000 HwHiAiUser useradd -g HwHiAiUser -u 1000 -d /home/HwHiAiUser -m HwHiAiUser -s /bin/bash Add directory permissions 1 2 chown -R HwHiAiUser /usr/local/Ascend chmod -R 755 /usr/local/Ascend Download the driver and firmware Go to https://www.hiascend.ru/hardware/firmware-drivers/community?product=1&model=30&cann=All&driver=1.0.26.alpha to find the matching driver and firmware.WeChat Official AccountAINPUAscendHuaweiHardwareDriversOperationshttps://www.chenshaowen.com/en/blog/data-storage-in-model-development-cycle.htmlSun, 26 May 2024 00:00:00 +0000Sun, 26 May 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/data-storage-in-model-development-cycle.html1. Data Delivery Based on Object Storage As shown above, model R&D mainly involves three sub-platforms: Data platform The data platform is mainly responsible for data-related management, such as data ingestion and data processing, ultimately producing the data needed for training. The data platform stores raw data in object storage. During processing, it fetches data from object storage, processes it, and stores the result back in object storage.WeChat Official AccountLLMModelJuiceFSData Storagehttps://www.chenshaowen.com/en/blog/comparison-of-storage.htmlSat, 25 May 2024 00:00:00 +0000Sat, 25 May 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/comparison-of-storage.html1. Storage Tiering and Interconnect 2. Various Storage Types Storage Type Bandwidth Capacity Response Latency Cost per Unit of Storage Storage Principle Use Case CPU L1 Cache 256-512GB/s 32-64KB/core <1ns ~$2000/MB SRAM Closest cache layer to CPU core CPU L2 Cache 64-256GB/s 256KB-2MB/core group 2-5ns ~$1000/MB Partly SRAM, partly embedded DRAM Next-level cache for the CPU CPU L3 Cache 32-128GB/s 4-16MB/CPU 5-15ns ~$100-200/MB Embedded DRAM Last-level cache for the CPU GPU L1 Cache 8-32TB/s 16-96KB/SM <1ns ~$2000/MB SRAM Closest cache layer to GPU core GPU L2 Cache 1-4TB/s 512KB-4MB/SMC 5-15ns ~$500-1000/MB Partly SRAM, partly embedded DRAM Next-level cache for the GPU Main memory (RAM) 10-80GB/s 4-128GB 30-60ns ~$2-7/GB DDR4/5 DRAM System main storage space GPU VRAM 768GB/s-4TB/s 4-24GB 25-75ns ~$7-20/GB GDDR6/6X SDRAM Dedicated high-speed buffer storage for the GPU SSD 500MB/s-7000MB/s 128GB-8TB 0.WeChat Official AccountStorageStorage TieringAIOperationshttps://www.chenshaowen.com/en/blog/store-elasticsearch-data-in-juicefs.htmlWed, 22 May 2024 00:00:00 +0000Wed, 22 May 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/store-elasticsearch-data-in-juicefs.html1. Storage Options There are three storage approaches: Isolate by directory and share a single JuiceFS Elasticsearch nodes share one JuiceFS instance, mounting different Elasticsearch nodes through subdirectories. /0/ corresponds to node Node-0 /1/ corresponds to node Node-1 /2/ corresponds to node Node-2 The main advantage of this approach is that it is easy to scale and simple to configure.WeChat Official AccountKubernetesJuiceFSElasticsearchStorageOperationshttps://www.chenshaowen.com/en/blog/fluid-using-s3-as-pvc.htmlSun, 19 May 2024 00:00:00 +0000Sun, 19 May 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/fluid-using-s3-as-pvc.html1. Create the Dataset 1 2 3 4 5 6 7 8 9 10 kubectl apply -f - <<EOF apiVersion: v1 kind: Secret metadata: name: my-s3 type: Opaque stringData: aws.accessKeyId: xxx aws.secretKey: xxx EOF 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 kubectl apply -f - <<EOF apiVersion: data.WeChat Official AccountFluidJuiceFSKubernetesStorageAIDatahttps://www.chenshaowen.com/en/blog/fluid-mount-s3-and-performance-testing.htmlThu, 16 May 2024 00:00:00 +0000Thu, 16 May 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/fluid-mount-s3-and-performance-testing.html1. Packaging the Fluid Runtime Image Create the fluid_config_init.py script 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 #!/usr/bin/env python import json rawStr = "" with open("/etc/fluid/config.json", "r") as f: rawStr = f.WeChat Official AccountFluidJuiceFSKubernetesStorageAIDatahttps://www.chenshaowen.com/en/blog/fluid-using-lustre-runtime-and-performance-testing.htmlThu, 16 May 2024 00:00:00 +0000Thu, 16 May 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/fluid-using-lustre-runtime-and-performance-testing.html1. Analyzing How Fluid Mounts NFS Storage Inspect the Fuse Pod 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 kubectl get pod nfs-demo-fuse-f9wg8 -oyaml apiVersion: v1 kind: Pod metadata: generateName: nfs-demo-fuse- spec: containers: - command: - /usr/local/bin/entrypoint.WeChat Official AccountFluidJuiceFSAIDataLustrehttps://www.chenshaowen.com/en/blog/fluid-using-nfs-runtime-and-performance-testing.htmlTue, 14 May 2024 00:00:00 +0000Tue, 14 May 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/fluid-using-nfs-runtime-and-performance-testing.html1. Creating a Dataset 1 2 3 4 5 6 7 8 9 10 kubectl apply -f - <<EOF apiVersion: data.fluid.io/v1alpha1 kind: Dataset metadata: name: nfs-demo spec: mounts: - mountPoint: x.x.x.x:/x-x/ name: nfs-demo EOF 2. Creating a Runtime 1 2 3 4 5 6 7 8 9 10 11 12 13 14 kubectl apply -f - <<EOF apiVersion: data.WeChat Official AccountFluidJuiceFSKubernetesStorageAIDatahttps://www.chenshaowen.com/en/blog/ops-copilot-has-been-updated-using-pipeline-and-llm.htmlWed, 01 May 2024 00:00:00 +0000Wed, 01 May 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/ops-copilot-has-been-updated-using-pipeline-and-llm.html1. Making Ops Copilot an Ops Coilot In September 2023, I wrote a version of Ops Copilot, and published an article about it, I’m Writing a Copilot for My Ops Tool. The effect it achieved was like this: 1 2 3 4 5 6 7 8 9 10 Opscli> 打WeChat Official AccountAICopilotAgentLLMOpsOperationshttps://www.chenshaowen.com/en/blog/what-is-mlops.htmlSat, 27 Apr 2024 10:00:00 +0000Sat, 27 Apr 2024 10:00:00 +0000https://www.chenshaowen.com/en/blog/what-is-mlops.html1. What Is MLOps MLOps is short for Machine Learning Operations, and it describes the standardization and engineering of the entire lifecycle around model development. MLOps includes the following key steps: Data management: storing, accessing, cleaning, and transforming data Model development: algorithm development and model construction Model training and tuning: training models on data, adjusting hyperparameters to optimize the model, and fine-tuning models Model evaluation: testing a model’s accuracy, generalization ability, and performance metrics Model deployment: deploying the model to the target environment and converting the model Model monitoring: monitoring the model’s behavior, performance degradation, data drift, and robustness Another similar concept is ModelOps.WeChat Official AccountMachine LearningMLOpsLLMR&DWhat Ishttps://www.chenshaowen.com/en/blog/model-parallel-training-techniques.htmlThu, 04 Apr 2024 00:00:00 +0000Thu, 04 Apr 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/model-parallel-training-techniques.html1. Data Parallelism Training steps: The master device loads the model and copies the model parameters to every worker device The master device splits the training data along the batch dimension and passes each batch to every worker device Each worker device performs training The master device aggregates the gradients from each worker device and updates the model parameters The master device broadcasts the model parameters to every worker device, preparing for the next batch of training Core idea:WeChat Official AccountAITrainingModelLLMDistributed SystemsLearningGPUhttps://www.chenshaowen.com/en/blog/common-ai-base-images-and-run-command.htmlThu, 28 Mar 2024 00:01:00 +0000Thu, 28 Mar 2024 00:01:00 +0000https://www.chenshaowen.com/en/blog/common-ai-base-images-and-run-command.html1. What the Image Tag Suffixes Mean base/cuda: includes the CUDA runtime runtime: builds on base, adding the CUDA math libraries and the NCCL and cuDNN runtimes devel: builds on runtime, adding headers and the development tools used to build CUDA images; particularly useful for multi-stage builds cuddn: builds on the above, adding the cuDNN neural network acceleration libraryWeChat Official AccountAIImagesContainersDockerGPULLMOperationsR&Dhttps://www.chenshaowen.com/en/blog/conda-install-and-use.htmlThu, 28 Mar 2024 00:00:00 +0000Thu, 28 Mar 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/conda-install-and-use.html1. Installing conda 1 2 3 wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh bash Miniconda3-latest-Linux-x86_64.sh rm -rf Miniconda3-latest-Linux-x86_64.sh However, Miniconda cannot be used for free at large-scale commercial deployments, so you can use Miniforge as a drop-in alternative. 1 2 wget "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh" bash Miniforge3-$(uname)-$(uname -m).sh 2. Changing the default configuration 2.1 Initializing the Shell Without initialization, activating an environment reports CondaError: Run 'conda init' before 'conda activate'.WeChat Official AccountCondaPythonAIEnvironmentOperationshttps://www.chenshaowen.com/en/blog/event-driven-workflow-using-argo-events.htmlWed, 27 Mar 2024 00:00:00 +0000Wed, 27 Mar 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/event-driven-workflow-using-argo-events.html1. How Argo Events Works The diagram above is from the official Argo Events website. An event processing system has three important parts: Ingesting event sources, which corresponds to the Event Source Distributing events, which corresponds to the Event Sensor Consuming events, which corresponds to the Event Trigger The event messages are stored in the EventBus, which uses NATS by default.WeChat Official AccountArgoArgo EventsKubernetesCICDEvent-DrivenOperationshttps://www.chenshaowen.com/en/blog/the-basic-of-volcano.htmlSun, 24 Mar 2024 00:00:00 +0000Sun, 24 Mar 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-basic-of-volcano.html1. Introduction to Volcano Volcano is a Kubernetes-based resource scheduling system open-sourced by Huawei. Compared with the native scheduler, its notable features are: Supports gang scheduling Scheduling batch jobs easily runs into deadlock: for example, two jobs each need 10 Pods running simultaneously to start. When both jobs are submitted at the same time, it is possible that only part of the Pods of each are scheduled, so neither job can run properly and they wait on each other.WeChat Official AccountVolcanoAIKubernetesSchedulingOperationshttps://www.chenshaowen.com/en/blog/basic-usage-of-npu-smi.htmlWed, 20 Mar 2024 00:00:00 +0000Wed, 20 Mar 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/basic-usage-of-npu-smi.html1. What Is npu-smi npu-smi is a command-line tool provided by Huawei, purpose-built for managing and monitoring the status and performance of Huawei Ascend series Neural Processing Units (NPUs), similar to NVIDIA’s nvidia-smi. 2. Meaning of npu-smi Fields 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 npu-smi info +------------------------------------------------------------------------------------------------+ | npu-smi 23.WeChat Official AccountAINPUHuaweiAscendHardwareMonitoring ToolsOperationshttps://www.chenshaowen.com/en/blog/ai-chip-high-speed-connection-solution.htmlTue, 19 Mar 2024 00:00:00 +0000Tue, 19 Mar 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/ai-chip-high-speed-connection-solution.htmlI have been studying the infrastructure around model training lately, and I found that AI chip interconnect topology determines the scheduling and resource allocation of training cluster jobs. So I spent a little time putting together the common AI chip interconnect solutions. 1. Point-to-Point Interconnect In a traditional PCIe system, data transfer between AI chips goes over PCIe, which cannot meet the demands of large-scale data transfer.WeChat Official AccountAIChipInterconnectHardwareOperationshttps://www.chenshaowen.com/en/blog/common-gpu-operation-and-fault-handling.htmlMon, 18 Mar 2024 00:00:00 +0000Mon, 18 Mar 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/common-gpu-operation-and-fault-handling.htmlContent referenced or recorded while handling faults, continuously updated. 1. XID Error Events XID is NVIDIA’s error code. You can retrieve it with the command: 1 dmesg -T | grep -i "NVRM: Xid" or 1 journalctl --since `date -d "10 days ago" "+%Y-%m-%d"`|grep Xid The XID lets you locate the fault.WeChat Official AccountGPUNVIDIAAIHardwareOperationsTroubleshootinghttps://www.chenshaowen.com/en/blog/provide-a-way-to-develop-ai-agent.htmlSat, 16 Mar 2024 08:01:28 +0000Sat, 16 Mar 2024 08:01:28 +0000https://www.chenshaowen.com/en/blog/provide-a-way-to-develop-ai-agent.html1. Keep Trying to Ship the AI Application Side Based on my understanding of operations, I developed an open-source operations tool, https://github.com/shaowenchen/ops . The Ops tool divides operations work into two categories — script execution and file distribution — and both operations targets, hosts and Kubernetes clusters, implement both ofWeChat Official AccountThoughtsAIAgentLLMOpsOperationshttps://www.chenshaowen.com/en/blog/how-to-quickly-create-juicefs-enterprise-pvc-in-kubernetes.htmlFri, 08 Mar 2024 00:00:00 +0000Fri, 08 Mar 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-quickly-create-juicefs-enterprise-pvc-in-kubernetes.htmlThis post mainly records the script for creating an enterprise-edition JuiceFS PVC, to make configuration quick. 1. Set Environment Variables 1.1 Basic Configuration 1 2 3 export NAMESPACE= export PVC_NAME= export JUICEFS_VOLUME= 1.2 JuiceFS Credentials 1 2 3 export TOKEN= export ACCESS_KEY= export SECRET_KEY= 1.3 JuiceFS Service Configuration 1 2 export BASE_URL=http://x.WeChat Official AccountAIStorageJuiceFSKubernetesFluidOperationsPVCEnterprise Editionhttps://www.chenshaowen.com/en/blog/how-to-quickly-create-juicefs-pvc-in-kubernetes.htmlThu, 07 Mar 2024 00:00:00 +0000Thu, 07 Mar 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-quickly-create-juicefs-pvc-in-kubernetes.htmlThis post mainly records the script for creating a community-edition JuiceFS PVC, to make configuration quick. For component deployment, see Managing Data in Kubernetes with Fluid and JuiceFS. 1. Set Environment Variables Bucket configuration 1 2 3 4 5 6 export ACCESS_KEY= export SECRET_KEY= export BUCKET= export ENDPOINT=ks3-cn-beijing-internal.ksyun.com export BUCKET_ENPOINT=$BUCKET.WeChat Official AccountAIStorageJuiceFSKubernetesFluidOperationsPVChttps://www.chenshaowen.com/en/blog/introduction-to-argo-core-components.htmlFri, 23 Feb 2024 00:00:00 +0000Fri, 23 Feb 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/introduction-to-argo-core-components.html1. Introduction to Argo Argo is an open source project that provides a set of tools on Kubernetes for building and running applications. The core components of Argo mainly include the following: Argo CD Argo CD is a GitOps-based continuous delivery tool that allows users to declaratively manage the deployment of applications in a Kubernetes cluster.WeChat Official AccountArgoArgo CDKubernetesCICDDevOpsOperationshttps://www.chenshaowen.com/en/blog/ops-added-server-and-ui-services.htmlWed, 14 Feb 2024 00:00:00 +0000Wed, 14 Feb 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/ops-added-server-and-ui-services.html1. What the Ops Project Is I introduced a commonly used Ops tool in an earlier article. The design philosophy behind Ops is that the core of an operations tool lies in text distribution and script execution; once these two capabilities are in place, they can satisfy operational requirements. My main operational targets at the moment are Host machines and Kubernetes clusters, so at the OpsObject layer I implemented the Host and Cluster objects, corresponding to hosts and Kubernetes clusters respectively.WeChat Official AccountOpsCopilotKubernetesOperationshttps://www.chenshaowen.com/en/blog/speeding-up-model-inference-with-tensorrt.htmlTue, 06 Feb 2024 00:00:00 +0000Tue, 06 Feb 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/speeding-up-model-inference-with-tensorrt.html1. What Is TensorRT TensorRT is a C++ library mainly used for high-performance inference acceleration on NVIDIA GPUs. It provides both a C++ API and a Python API for integration. The mainstream deep learning frameworks TensorRT supports are: Caffe, where TensorRT can read the prototxt format directly TensorFlow, where theWeChat Official AccountAITensorRTNVIDIAGPUInferenceLLMhttps://www.chenshaowen.com/en/blog/practice-guide-to-kind.htmlMon, 05 Feb 2024 00:00:00 +0000Mon, 05 Feb 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/practice-guide-to-kind.html1. Project Overview kind is a tool for managing Kubernetes clusters using containers. Project address https://github.com/kubernetes-sigs/kind . It is mainly used for: Local development environments Temporary environments for learning Automated testing 2. Installing kind macOS 1 brew install kind Linux 1 2 curl -Lo /usr/local/bin/kind https://kind.sigs.k8s.io/dl/v0.21.0/kind-linux-amd64 chmod +x /usr/local/bin/kind 3. Creating a kind Cluster If you have a PROXY configured locally, it is recommended to reset the environment variables before creating:WeChat Official AccountKuberneteskindPracticeOperationshttps://www.chenshaowen.com/en/blog/ai-related-exporters-in-kubernetes.htmlSun, 04 Feb 2024 00:00:00 +0000Sun, 04 Feb 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/ai-related-exporters-in-kubernetes.html1. dcgm-exporter dcgm-exporter is the GPU monitoring tool provided by the official NVIDIA community. Project address https://github.com/NVIDIA/dcgm-exporter 1.1 Installation Add the Helm chart repository 1 helm repo add gpu-helm-charts https://nvidia.github.io/dcgm-exporter/helm-charts 1 helm repo update Install 1 2 3 4 5 helm install dcgm-exporter gpu-helm-charts/dcgm-exporter --namespace monitor --create-namespace \ --set serviceMonitor.enabled=false \ --set image.WeChat Official AccountKubernetesAIMetricsCollectionhttps://www.chenshaowen.com/en/blog/using-triton-server-and-tensorrt-llm-under-container.htmlSat, 03 Feb 2024 08:05:48 +0000Sat, 03 Feb 2024 08:05:48 +0000https://www.chenshaowen.com/en/blog/using-triton-server-and-tensorrt-llm-under-container.html1. Compiling Models with TensorRT-LLM 1.1 Introduction to TensorRT-LLM When using TensorRT, you usually need to convert the model to ONNX format, then convert the ONNX to TensorRT format, and finally run inference in TensorRT or Triton Server. But this conversion process is not simple, and you often run intoWeChat Official AccountAITritonTensorRTLLMInferenceGPUOperationsContainersLearninghttps://www.chenshaowen.com/en/blog/basic-usage-of-nvidia-smi.htmlThu, 01 Feb 2024 00:00:00 +0000Thu, 01 Feb 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/basic-usage-of-nvidia-smi.html1. What Is nvidia-smi nvidia-smi stands for NVIDIA System Management Interface, the interface NVIDIA provides for managing and monitoring GPUs. nvidia-smi calls NVML. NVML stands for NVIDIA Management Library, a library that provides a set of C APIs for NVIDIA GPU monitoring and management. 1.1 Queryable State ECC error counts GPU utilization Active compute processes Clocks and PState Temperature and fan speed Power management Hardware identification 1.WeChat Official AccountAIGPUNVIDIAHardwareMonitoring ToolsOperationshttps://www.chenshaowen.com/en/blog/managing-data-in-kubernetes-using-fluid-and-juicefs.htmlSat, 27 Jan 2024 00:00:00 +0000Sat, 27 Jan 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/managing-data-in-kubernetes-using-fluid-and-juicefs.html1. Introduction to Fluid Below is the Fluid architecture diagram from https://github.com/fluid-cloudnative/fluid: Fluid abstracts two concepts: Dataset, a collection of data, the abstraction from the user’s point of view Runtime, the abstraction of the real services such as data storage and acceleration Fluid mainly solves the problems of using traditional caching systems on Kubernetes:WeChat Official AccountKubernetesFluidJuiceFSStorageData ManagementAIOperationshttps://www.chenshaowen.com/en/blog/performance-testing-and-comparison-of-juicefs-ce-ee-and-dragonfly.htmlWed, 24 Jan 2024 00:00:00 +0000Wed, 24 Jan 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/performance-testing-and-comparison-of-juicefs-ce-ee-and-dragonfly.htmlNote that --block-size 4096 in this post means 4GB; using --block-size 4 would be more reasonable and would give better write performance. 1. Environment Preparation Enter a safe directory 1 mkdir -p /data/test && cd /data/test All the test tasks are carried out in this directory. Create a separate directory for Redis 1 mkdir redis-data It is recommended to create a new directory, because Redis will change the Owner of the files in the current directory to systemd-coredump.WeChat Official AccountJuiceFSDragonflyStoragePerformance TestingKubernetesOperationshttps://www.chenshaowen.com/en/blog/health-check-runs-out-of-root-run-storage.htmlTue, 23 Jan 2024 00:00:00 +0000Tue, 23 Jan 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/health-check-runs-out-of-root-run-storage.htmlIf you use nvidia-container-runtime, this is worth your attention — especially if JuiceFS is also in the picture. 1. An Alert Out of Nowhere, and I Panicked Over the weekend I was studying TensorRT LLM, and while I was at it I installed Dragonfly on the largest production cluster. ThenWeChat Official AccountKubernetesNVIDIAJuiceFSExperienceTroubleshootingOperationshttps://www.chenshaowen.com/en/blog/distributing-image-with-dragonfly-v2.htmlSat, 13 Jan 2024 11:22:55 +0000Sat, 13 Jan 2024 11:22:55 +0000https://www.chenshaowen.com/en/blog/distributing-image-with-dragonfly-v2.html1. Introduction to Dragonfly The Dragonfly documentation at https://d7y.io/zh/docs/ already covers things in detail. Here is just a brief introduction to the main components of V2: Manager, which provides the UI, user management, cluster monitoring, task management, and other features Scheduler, which schedules traffic between Peers and provides preheating and other features Seed Peer, the back-to-source node used to download data from origin sites (Harbor, Docker.WeChat Official AccountAIDragonflyKubernetesImagesImageNydusConfigurationPracticeOperationshttps://www.chenshaowen.com/en/blog/nydus-configuration-and-practice.htmlSat, 06 Jan 2024 11:22:55 +0000Sat, 06 Jan 2024 11:22:55 +0000https://www.chenshaowen.com/en/blog/nydus-configuration-and-practice.htmlStatistics show that most files inside a container are never used. Based on this characteristic, Nydus defines a custom filesystem in the Rafs format, enabling on-demand loading of image files to solve the slow startup and storage consumption caused by large images. In AI scenarios, whether for inference or training, images often start at several GB, or even tens of GB, so Nydus is very well suited.WeChat Official AccountAINydusDockerImagesConfigurationPracticeOperationshttps://www.chenshaowen.com/en/blog/the-increment-is-no-longer-chaos-is-standing.htmlMon, 01 Jan 2024 07:21:38 +0000Mon, 01 Jan 2024 07:21:38 +0000https://www.chenshaowen.com/en/blog/the-increment-is-no-longer-chaos-is-standing.html1. Only a Wheel Turning at High Speed Stays Balanced 1.1 The Consumer-Side Boom Is Over As of June 2023, China had 1.079 billion internet users, up 11.09 million from December 2022, with an internet penetration rate of 76.4%. The consumer-side demographic dividend has passed, time spent online is also growing slowly, and there is little room left for growth in applications built on existing scenarios.WeChat Official AccountChaosThoughtsIndustryOperationshttps://www.chenshaowen.com/en/blog/nvidia-gpu-driver-installation.htmlThu, 28 Dec 2023 00:00:00 +0000Thu, 28 Dec 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/nvidia-gpu-driver-installation.html1. Disk Handling 1.1 Inspect Disks Inspect the new disk 1 fdisk -l Disk /dev/nvme1n1: 3.91 TiB, 4294967296000 bytes, 8388608000 sectors 1.2 Build a RAID0 If you have several small disks, a better approach is to build a RAID0 — this gives you not only a larger storage directory but also higher speed.WeChat Official AccountAIGPUNVIDIAHardwareDrivershttps://www.chenshaowen.com/en/blog/how-to-access-gpt-service-on-wechat.htmlTue, 26 Dec 2023 00:00:00 +0000Tue, 26 Dec 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-access-gpt-service-on-wechat.htmlPaid integration service available at 200 RMB/year; technical support is also available at 200 RMB per session, capped at 1 hour. Follow the official account to get contact details. 1. Information You Need to Provide Go to https://mp.weixin.qq.com/ and open 【Settings andWeChat Official AccountLLMGPTOfficial AccountWeChatAIR&DLearningIntegrationOpenAIPracticehttps://www.chenshaowen.com/en/blog/large-model-application-design-and-implementation-guide.htmlSat, 23 Dec 2023 11:22:55 +0000Sat, 23 Dec 2023 11:22:55 +0000https://www.chenshaowen.com/en/blog/large-model-application-design-and-implementation-guide.html1. Problems with Using LLMs Directly Unstable output One characteristic of generative AI is the diversity of its output. Ask an LLM the same question several times and you may get different answers. This uncertainty in output is a pleasant surprise for users in conversation and creative scenarios. But inWeChat Official AccountLLMAIBest PracticesR&DArchitectureLearningRAGAgenthttps://www.chenshaowen.com/en/blog/chaos-engineering-and-practice.htmlSat, 16 Dec 2023 11:15:11 +0000Sat, 16 Dec 2023 11:15:11 +0000https://www.chenshaowen.com/en/blog/chaos-engineering-and-practice.htmlTwo months ago, I gave a talk to a business team about chaos engineering in practice. This post mainly organizes the content of that talk. Click to view the presentation 1. The Origin of Chaos 1.1 The Birth of the Discipline of Chaos Before discussing chaos, let us first think about the relationship between chaos, chaos engineering, and our production services.WeChat Official AccountChaosPracticeProcessConventionsChaos EngineeringOperationsReliabilityLearningFault Drillshttps://www.chenshaowen.com/en/blog/how-experience-sre-in-small-and-medium-sized-companies.htmlFri, 08 Dec 2023 11:33:41 +0000Fri, 08 Dec 2023 11:33:41 +0000https://www.chenshaowen.com/en/blog/how-experience-sre-in-small-and-medium-sized-companies.html1. Two Years Ago I Took a Different Path Looking back now, 2021 should have been a golden year for internet workers in Wuhan to change jobs. The pandemic had passed, and we were full of hope for the future; monetary policy was loose, and companies were confident about market prospects.WeChat Official AccountSREThoughtsValueOperationsTeamsGrowthCareerhttps://www.chenshaowen.com/en/blog/my-restful-api-specification.htmlThu, 07 Dec 2023 00:00:00 +0000Thu, 07 Dec 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/my-restful-api-specification.htmlMainly for standardizing the APIs I write myself, and also to help LLMs understand my APIs better. 1. Domain Use a dedicated domain whenever possible, for example api.example.com. 2. Path Always use lowercase letters. Do not include an /api prefix. Do not include file extensions. Do not put / at the end.WeChat Official AccountInterfaceAPIR&DSpecificationLLMhttps://www.chenshaowen.com/en/blog/openai-vs-azure-openai-api.htmlSun, 03 Dec 2023 11:22:55 +0000Sun, 03 Dec 2023 11:22:55 +0000https://www.chenshaowen.com/en/blog/openai-vs-azure-openai-api.htmlBecause pricing and limits change over time, this article is only a reference for the time it was written. 1. Terminology RPM (requests per minute) Requests per minute RPD (requests per day) Requests per day TPM (tokens per minute) Tokens per minute TPD (tokens per day), Tokens per dayWeChat Official AccountOpenAIAzureAPIAILLMR&Dhttps://www.chenshaowen.com/en/blog/configurate-envoyfilter-in-istio.htmlSun, 12 Nov 2023 00:00:00 +0000Sun, 12 Nov 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/configurate-envoyfilter-in-istio.html1. What Is EnvoyFilter EnvoyFilter is a CRD resource in Istio that allows users to modify Envoy’s configuration to satisfy their customization needs for different scenarios. 1 2 3 4 5 6 7 8 9 kubectl get envoyfilter -A NAMESPACE NAME AGE istio-system add-request-id-into-ingressgateway 54d istio-system compression-gzip 18d istio-system custom-access-log 3d istio-system ingressgateway-settings 52d istio-system preserve-request-header-us-test-ingress-gateway 95d istio-system preserve-x-request-id 54d When using Istio, you will usually end up using at least a few EnvoyFilters.WeChat Official AccountIstioEnvoyFilterKubernetesService MeshConfigurationOperationshttps://www.chenshaowen.com/en/blog/optimization-go-applications-in-containers.htmlSat, 11 Nov 2023 08:05:48 +0000Sat, 11 Nov 2023 08:05:48 +0000https://www.chenshaowen.com/en/blog/optimization-go-applications-in-containers.html1. Memory Alignment Arrange the fields inside a struct from largest to smallest. This reduces memory footprint. Install the fieldalignment tool 1 go install golang.org/x/tools/go/analysis/passes/fieldalignment/cmd/fieldalignment@latest Analyze and fix memory alignment 1 2 3 4 5 6 7 8 9 fieldalignment -fix ./... /Users/shaowenchen/Code/app/config/config.go:136:14: struct with 32 pointer bytes could be 24 /Users/shaowenchen/Code/app/config/config.WeChat Official AccountGoPerformance OptimizationContainersBest PracticesR&DLearningDockerKuberneteshttps://www.chenshaowen.com/en/blog/how-to-intialize-pci-ssd.htmlThu, 09 Nov 2023 00:00:00 +0000Thu, 09 Nov 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-intialize-pci-ssd.html1. Check the PCI Interface Device Use lspci to check whether the device is already installed. 1 2 3 lspci |grep storage d8:00.0 Mass storage controller: Shannon Systems Device 2275 Here a Shannon Systems device shows up, which means the SSD is already installed. 1 2 3 4 5 6WeChat Official AccountDSMDiskStorageOperationshttps://www.chenshaowen.com/en/blog/record-a-troubleshooting-process-for-application-slowness.htmlWed, 08 Nov 2023 00:00:00 +0000Wed, 08 Nov 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/record-a-troubleshooting-process-for-application-slowness.html1. Symptoms The business side reported that the API of application app-a was slow. Looking at the logs, one particular Pod was slow, and deleting that Pod so it moved to another node fixed it. From the monitoring metrics you can see that the Pod’s CPU usage did indeed spike sharply.WeChat Official AccountCPUNetworkApplicationTroubleshootingKubernetesOperationshttps://www.chenshaowen.com/en/blog/source-analysis-kubernetes-management-of-pod-ip.htmlThu, 02 Nov 2023 19:45:31 +0000Thu, 02 Nov 2023 19:45:31 +0000https://www.chenshaowen.com/en/blog/source-analysis-kubernetes-management-of-pod-ip.html1. Subnet Management by kube-controller-manager kube-controller-manager has many controllers; the one related to Pod IP is NodeIpamController. The NodeIpamController mainly manages the podcidr of nodes: when a new node joins the cluster, it allocates a subnet to that node; when a node is deleted, it reclaims the subnet. The subnetsWeChat Official AccountSource Code AnalysisKubernetesPodIPNetworkLearningOperationshttps://www.chenshaowen.com/en/blog/simple-tutorial-on-using-ffmpeg.htmlWed, 11 Oct 2023 00:00:00 +0000Wed, 11 Oct 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/simple-tutorial-on-using-ffmpeg.html1. Installing FFmpeg Run this command on macOS: 1 brew install ffmpeg 2. Using FFmpeg 1 ffmpeg {1} {2} -i {3} {4} {5} The five parts of the arguments are, in order: Global arguments, such as -y, -loglevel, -preset, which control the overall behavior Input file arguments, such as -i, -ss, -t, -stream_loop, which control how the input file is read Input file, the path to the audio/video input file to process Output file arguments, such as -c:v, -c:a, -vf, -af, which control the encoding and filtering of the output Output file, the path to the processed output audio/video file Common output arguments:WeChat Official AccountFFmpegVideoTutorialToolsLearningMultimediahttps://www.chenshaowen.com/en/blog/some-tips-for-optimizing-program-building.htmlThu, 28 Sep 2023 00:00:00 +0000Thu, 28 Sep 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/some-tips-for-optimizing-program-building.htmlContinuously updated… 1. Golang Use trimpath to remove the build path, avoiding exposure of build path information 1 go build -gcflags="all=-trimpath=${PWD}" -asmflags="all=-trimpath=${PWD}" -o ./bin/opscli ./cmd/cli/main.go Before adding it, a panic prints the path /Users/shaowenchen/Code/Github/ops/main.go; after adding it, a panic prints main.go. -ldflags “-w -s” removes debug information and reduces the binary size 1 go build -ldflags "-w -s" -o .WeChat Official AccountCICDDevOpsBest PracticesR&DBuildOptimizationGoLearninghttps://www.chenshaowen.com/en/blog/writing-copilot-for-my-ops-tool.htmlSat, 23 Sep 2023 00:00:00 +0000Sat, 23 Sep 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/writing-copilot-for-my-ops-tool.html1. What Is an Ops Tool https://www.chenshaowen.com/ops/ is one of the tools I use most often in day-to-day operations. To operate on machines, I can reuse my previous scripts and perform operations in bulk. To operate on clusters, I can reuse my previous scripts and operate machines without logging into the nodes.WeChat Official AccountLLMAIThoughtsOpsCopilotOperationshttps://www.chenshaowen.com/en/blog/how-to-run-llama-on-cpu.htmlSat, 16 Sep 2023 00:00:00 +0000Sat, 16 Sep 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-run-llama-on-cpu.html1. Running Locally in Containers Start the LLM 1 docker run --security-opt apparmor=unconfined --security-opt seccomp=unconfined --rm -p 8000:8000 shaowenchen/chinese-alpaca-2-7b-gguf:Q2_K The API documentation is available at http://localhost:8000/docs, as shown below: Deploy a simple Chat UI Note that the OPENAI_API_HOST parameter must be set to your host machine’s IP address rather than localhost or 127.WeChat Official AccountCPULLMAIInferenceOperationshttps://www.chenshaowen.com/en/blog/llama-cpp-that-is-a-llm-deployment-tool.htmlTue, 05 Sep 2023 00:00:00 +0000Tue, 05 Sep 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/llama-cpp-that-is-a-llm-deployment-tool.html1. LLM Deployment Tool llama.cpp Research on large models is split into two parts: training and inference. The training process is essentially the process of finding model parameters that minimize the model’s loss function and optimize the inference results. Once training is complete, the model’s parameters are fixed, and atWeChat Official AccountAILLMToolsllama.cppInferenceDeploymenthttps://www.chenshaowen.com/en/blog/free-quota-gpt-4-32k-for-fans.htmlMon, 04 Sep 2023 00:00:00 +0000Mon, 04 Sep 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/free-quota-gpt-4-32k-for-fans.htmlTaken offline on 2024.06.26 1. My GPT-4-32K API Application Was Approved I had previously submitted an application on Azure to use GPT-4, and a few days ago I received the email saying the review had passed. The pricing is as follows: Context Length Prompt per 1,000 tokens (8k) Completion per 1,000 tokens (32k) 8k $0.WeChat Official AccountGPT-4OpenAIAILLMhttps://www.chenshaowen.com/en/blog/some-potential-pitfalls-when-using-cloud-infrastructure.htmlWed, 23 Aug 2023 00:00:00 +0000Wed, 23 Aug 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/some-potential-pitfalls-when-using-cloud-infrastructure.html1. Quota Limits Every resource in the cloud has a cap on how much of it you can use. This is how cloud vendors prevent resource abuse and reduce the mutual impact between tenants. For example: the number of enterprise projects under one account, the number of elastic compute instances, the number of elastic public IPs, the bandwidth of an elastic public IP, and so on.WeChat Official AccountCloud VendorInfrastructureSREOperationshttps://www.chenshaowen.com/en/blog/usage-of-transformers-lib.htmlTue, 22 Aug 2023 00:00:00 +0000Tue, 22 Aug 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/usage-of-transformers-lib.htmltransformers is a Python library developed by Hugging Face for using and training pretrained Transformer models in natural language processing (NLP) tasks. It provides many powerful tools and features that make working with text data and building NLP models much easier. The library is widely used across a variety ofWeChat Official AccountTransformerAILLMNLPPythonhttps://www.chenshaowen.com/en/blog/models-and-datasets-on-huggingface.htmlMon, 21 Aug 2023 00:00:00 +0000Mon, 21 Aug 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/models-and-datasets-on-huggingface.htmlHuggingFace offers shared models, datasets, and hosted spaces, giving AI researchers and developers a complete ecosystem. This article explains how to work with HuggingFace models and datasets. 1. Model Operations and Usage 1.1 Custom Storage Directory 1 export HF_HOME=/Volumes/Data/HuggingFace Otherwise the default is the ~/.cache/huggingface directory. 1.2 Downloading Models TheWeChat Official AccountTransformerAILLMHuggingFaceDatasetR&Dhttps://www.chenshaowen.com/en/blog/learning-notes-of-transformer.htmlSun, 20 Aug 2023 00:00:00 +0000Sun, 20 Aug 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/learning-notes-of-transformer.html1. Why Transformer Fully connected self-attention In earlier RNN models, each word could only relate to its neighboring words, whereas with the Attention mechanism in the Transformer model, a word can relate to a word at any position, so it can capture global context information. No vanishing gradient problem An RNN applies repeatedly to the same weight matrix, so when its largest eigenvalue is less than 1 the vanishing gradient problem appears.WeChat Official AccountTransformerAILLMLearningDeep LearningNLPhttps://www.chenshaowen.com/en/blog/the-key-factors-while-using-large-models.htmlSat, 19 Aug 2023 00:00:00 +0000Sat, 19 Aug 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-key-factors-while-using-large-models.html1. What a Large Model Actually Is First, let us ask two large models this question and see what they answer. Claude says that a large model is essentially a probabilistic expression of linguistic knowledge: through statistical learning it models the regularities at every level of language and represents the prior distribution of language generation, thereby gaining the ability to predict and generate language.WeChat Official AccountAILLMThoughtsLearningTechnology Selectionhttps://www.chenshaowen.com/en/blog/ai-basic-knowledge.htmlFri, 18 Aug 2023 11:22:55 +0000Fri, 18 Aug 2023 11:22:55 +0000https://www.chenshaowen.com/en/blog/ai-basic-knowledge.html1. Keywords Machine Learning (ML) The technology of automatically acquiring knowledge from data. Neural Network (NN) A model that imitates the structure and learning mechanism of biological neural networks; one of the branches of machine learning. The structure of a neural network consists of an input layer, hidden layers, and an output layer.WeChat Official AccountAILLMMachine LearningKnowledge PointsLearninghttps://www.chenshaowen.com/en/blog/call-functions-through-dialogue-using-openai-and-langchain.htmlWed, 16 Aug 2023 00:00:00 +0000Wed, 16 Aug 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/call-functions-through-dialogue-using-openai-and-langchain.html1. LLMs and Langchain Many people may never get the chance to train, or even fine-tune, a large model, but using large models is the wave of the future. So how should we embrace this change? The answer is Langchain. A large model provides a broad, general-purpose foundation. So farWeChat Official AccountOpenAILangchainAILLMR&DLearningFunction CallingApplication DevelopmentBest Practiceshttps://www.chenshaowen.com/en/blog/monitor-service-accessibility-with-upptime-for-free.htmlSat, 12 Aug 2023 00:00:00 +0000Sat, 12 Aug 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/monitor-service-accessibility-with-upptime-for-free.html1. What Needs Dial Testing GPT was all the rage this year, and I deployed and developed several applications and mini programs to learn from. Of course, following the principle of helping vendors test their features, right now only the GPT 3.5 API costs a little every day — theWeChat Official AccountGitHubActionsMonitoringDial TestingOperationsLearningAvailabilityFreeCICDBest Practiceshttps://www.chenshaowen.com/en/blog/common-problems-using-apline.htmlFri, 04 Aug 2023 00:00:00 +0000Fri, 04 Aug 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/common-problems-using-apline.html1. DNS Request Timeout Cause: alpine uses the musl library, which has some limitations in DNS resolution [1] Solution: Do not use an apline image, and add the options single-request-reopen setting to the container’s resolv.conf file. This is because the single-request-reopen option only takes effect for the glibc library, while apline images use the musl library.WeChat Official AccountDockerAlpineImagesTroubleshootingOperationshttps://www.chenshaowen.com/en/blog/the-scope-of-credential-in-building.htmlThu, 06 Jul 2023 00:00:00 +0000Thu, 06 Jul 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-scope-of-credential-in-building.htmlIn the client, we have already seen the Docker CLI passing credentials to the Docker Daemon via X-Registry-Config when sending the build context. But recent build feedback still shows some phenomena that cannot be explained. This post mainly runs some basic tests to make troubleshooting easier. 1. Building under host Docker The Docker Daemon is started with root privileges.WeChat Official AccountCICDDevOpsDockerCredentialsKubernetesOperationsTroubleshootinghttps://www.chenshaowen.com/en/blog/the-practice-of-github-copilot-chat.htmlTue, 04 Jul 2023 00:00:00 +0000Tue, 04 Jul 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-practice-of-github-copilot-chat.html1. Applying to Use GitHub Copilot Chat Application link https://github.com/github-copilot/chat_waitlist_signup/join Once the application is approved, you will receive an email: 2. What Is VS Code Insiders What is VS Code insiders VS Code insiders is the preview version of VS Code, offering some of the latest features and improvements, withWeChat Official AccountCopilotGitHubAIR&DProgramming AssistanceLearningEfficiencyLLMhttps://www.chenshaowen.com/en/blog/how-to-set-proxy-to-server-using-nodejs.htmlSat, 24 Jun 2023 00:00:00 +0000Sat, 24 Jun 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-set-proxy-to-server-using-nodejs.html1. Background Under a microservices architecture, the dependency relationships between services are complex. During development, multiple services often need joint debugging. There are two ways to do this: Deploy services to the online development environment, the Kubernetes cluster Use telepresence to connect local and online cluster communication, so you get a relatively stable joint debugging environment.WeChat Official AccountNodejsProxyProxyMicroservicesR&DToolshttps://www.chenshaowen.com/en/blog/how-to-set-hpa-for-kubernetes-app.htmlThu, 08 Jun 2023 00:00:00 +0000Thu, 08 Jun 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-set-hpa-for-kubernetes-app.html1. Business Background Once an enterprise reaches a certain scale, relying entirely on public cloud infrastructure makes IT costs very high. The cost of purchasing physical machines can be amortized over the next 3 to 5 years, and afterwards the machines are not scrapped; instead they continue to serve beyond their planned lifespan.WeChat Official AccountKubernetesHPAPodElastic ScalingOperationsHybrid CloudLearninghttps://www.chenshaowen.com/en/blog/autoscale-kubernetes-applications-with-keda.htmlThu, 18 May 2023 00:00:00 +0000Thu, 18 May 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/autoscale-kubernetes-applications-with-keda.html1. HPA VS KEDA HPA also provides: Elasticity based on custom metrics Scale to Zero Compared with KEDA, these are no longer disadvantages. The real difference is that HPA can only scale using monitoring data, whereas KEDA can scale using many more data sources — queue messages, databases, Redis, and so on, including monitoring data as well.WeChat Official AccountKubernetesKEDAAutoscalingElastic ScalingOperationsLearningHPAEvent-Drivenhttps://www.chenshaowen.com/en/blog/get-webassembly-programs-using-go.htmlSat, 13 May 2023 00:00:00 +0000Sat, 13 May 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/get-webassembly-programs-using-go.html1. Introduction to WebAssembly Cross-platform: it can run on any platform that supports WebAssembly, including web browsers, servers, and mobile devices High performance: it uses a compact binary format that can be loaded and parsed quickly in the browser, improving application performance Security: it uses a sandbox model that isolates the code running inside it, protecting the system from malicious codeWeChat Official AccountGoWebAssemblyFrontendR&Dhttps://www.chenshaowen.com/en/blog/webassembly-serverless-will-be-a-common-arch-in-the-future.htmlTue, 09 May 2023 00:00:00 +0000Tue, 09 May 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/webassembly-serverless-will-be-a-common-arch-in-the-future.html1. Container-Based Serverless Cannot Support the Next Generation of Applications As the figure above shows, we are going through a change in runtime form. From bare metal to virtual machines, applications are no longer limited by the number of local servers or the stability of the data center, and gain better elasticity and availability.WeChat Official AccountWebAssemblyServerlessKubernetesApplicationshttps://www.chenshaowen.com/en/blog/custom-cluster-scheduler-to-optimize-tekton.htmlWed, 26 Apr 2023 10:00:00 +0000Wed, 26 Apr 2023 10:00:00 +0000https://www.chenshaowen.com/en/blog/custom-cluster-scheduler-to-optimize-tekton.html1. A Constrained Build Environment Cannot Meet Build Requirements Tekton is a CICD engine based on a Kubernetes cluster, and it is more cloud-native than Jenkins. In plain terms, that means it is easier to develop plugins for, easier to scale, easier to observe, and more fun. Because code can only be stored on the company intranet, the build cluster can only be deployed on the office intranet.WeChat Official AccountOptimizationTektonCICDDevOpsKubernetesOperationsLearningSchedulingBuildMulti-Clusterhttps://www.chenshaowen.com/en/blog/infinite-io-capability-for-tekton.htmlWed, 26 Apr 2023 00:00:00 +0000Wed, 26 Apr 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/infinite-io-capability-for-tekton.html1. After Turning Off affinity-assistant In an earlier post, by turning off affinity-assistant and using NFS storage, I saved nearly 30 seconds on average per pipeline execution.[1] The impact of affinity-assistant Before turning it off, the sequence diagram for creating a Pod looked like this: Because affinity-assistant was on, every pipeline was bound to a single node for execution.WeChat Official AccountTektonKubernetesCloud NativeCICDOperationshttps://www.chenshaowen.com/en/blog/integrate-chatgpt-into-mp-by-cloudflare-workers.html.htmlFri, 14 Apr 2023 00:00:00 +0000Fri, 14 Apr 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/integrate-chatgpt-into-mp-by-cloudflare-workers.html.html1. Create a ChatGPT Account Visit https://chat.openai.com/ to test your network If you see the following page, your network does not support ChatGPT Normally you should see this page The solution is to switch networks. Fortunately, the company network supports it, so I registered my account on the company network.WeChat Official AccountCloudflareWorkersOfficial AccountWeChatOpenAIChatGPThttps://www.chenshaowen.com/en/blog/a-open-source-ops-tool-project.htmlTue, 11 Apr 2023 00:00:00 +0000Tue, 11 Apr 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/a-open-source-ops-tool-project.html1. What Documentation as Tooling Means “Documentation as tooling, tooling as a product” is a slogan I have repeated many times in my previous blog posts. A good document is no match for a tool that is pleasant to use. A script or a single command is more direct than reading documentation and can solve the problem faster.WeChat Official AccountOpsToolsProjectOpen SourceOperationshttps://www.chenshaowen.com/en/blog/a-dynamic-proxy-envoy.htmlThu, 23 Feb 2023 00:00:00 +0000Thu, 23 Feb 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/a-dynamic-proxy-envoy.htmlEnvoy is the third project to graduate from the CNCF, and thanks to features like dynamic configuration and high performance it has become the de facto data plane standard in cloud native. Many projects rely on Envoy to handle data plane traffic while focusing their own efforts on adapting the control plane to their application scenarios, writing user input into Envoy over the xDS protocol.WeChat Official AccountEnvoyData PlaneCloud NativeProxyNetworkOperationsService Meshhttps://www.chenshaowen.com/en/blog/developing-istio-wasm-plugin-by-tinygo.htmlWed, 22 Feb 2023 01:00:00 +0000Wed, 22 Feb 2023 01:00:00 +0000https://www.chenshaowen.com/en/blog/developing-istio-wasm-plugin-by-tinygo.htmlwasme only supports up to Istio 1.9, while I am using Istio 1.14, so this post goes straight to tinygo for verification and learning. 1. Installing tinygo Requirements Go v1.18+ Install tinygo 1 2 brew tap tinygo-org/tools brew install tinygo Check the version 1 2 3 tinygo version tinygo version 0.WeChat Official AccounttinygoIstioWasmCloud NativeService MeshR&DGoLearningExtensionhttps://www.chenshaowen.com/en/blog/network-performance-testing-tool-iperf.htmlWed, 22 Feb 2023 00:00:00 +0000Wed, 22 Feb 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/network-performance-testing-tool-iperf.html1. Installation Ubuntu 1 apt-get install -y iperf3 CentOS 1 yum install -y iperf3 2. Parameters The principle behind iperf3 is that the client sends packets to the server so the network can be analyzed. It has two running modes: client and server. The iperf3 parameters fall into three groups: common parameters, client parameters, and server parameters.WeChat Official AccountNetworkPerformanceTestingiperfOperationsToolshttps://www.chenshaowen.com/en/blog/several-ways-to%20inject%20Sidecar.htmlTue, 21 Feb 2023 00:01:00 +0000Tue, 21 Feb 2023 00:01:00 +0000https://www.chenshaowen.com/en/blog/several-ways-to%20inject%20Sidecar.htmlThe template Istio uses to inject a Sidecar lives in the istio-sidecar-injector ConfigMap. Annotations let you customize the Sidecar’s various parameters, such as CPU usage, proxyImage, and so on. What follows is mainly a summary of the Sidecar injection methods. 1. Add a label to a namespace -> takes effect for the whole namespace Inject the label 1 kubectl label namespace default istio-injection=enabled --overwrite After the application restarts, the Sidecar container will be injected automatically.WeChat Official AccountIstioSidecarKubernetesService MeshOperationshttps://www.chenshaowen.com/en/blog/several-traffic-paths-under-istio-gateway.htmlTue, 21 Feb 2023 00:00:00 +0000Tue, 21 Feb 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/several-traffic-paths-under-istio-gateway.html1. Common Object Configuration 1.1 Gateway selector Selects the Envoy on which the rules take effect servers The domains to match Port Protocol TLS certificate 1.2 VirtualService gateways Specifies the gateway on which this takes effect; the default value mesh is for east-west traffic, while specifying a Gateway object makes it north-south traffic http Layer 7 routing Redirect Rewrite Retry Condition rules Timeout Fault injection CORS policy tcp Layer 7 routing tls Routing with certificates TLS certificate 1.WeChat Official AccountIstioGatewayKubernetesService MeshTrafficOperationshttps://www.chenshaowen.com/en/blog/transform-ops-system-using-event-bus.htmlFri, 17 Feb 2023 00:00:00 +0000Fri, 17 Feb 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/transform-ops-system-using-event-bus.html1. An Operations Service System That Has Piled Up Beyond Repair Developing a corresponding operations service for a clear operations need, so that operations and business users can use it, is entirely reasonable. But if you are satisfied with only that, the following situation easily appears: Users constantly look for the entry point of each system, jumping back and forth between systems, busy hunting for buttons and copying parameters.WeChat Official AccountOperations SystemPaaSPlatformEventArchitectureThoughtsEvent-DrivenLearningAutomationhttps://www.chenshaowen.com/en/blog/optimizing-the-slow-of-tekton-clone-task.htmlThu, 16 Feb 2023 00:00:00 +0000Thu, 16 Feb 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/optimizing-the-slow-of-tekton-clone-task.html1. Symptom - Tekton’s Clone Task Is Slow When running the clone task, Tekton is very time-consuming; with multiple repositories it generally takes around 2 minutes 30 seconds. As shown below: A pipeline that only clones takes 2 minutes 16 seconds to run, while the clone script itself actually runs for only 1-3 seconds.WeChat Official AccountTektonKubernetesCICDDevOpsOperationsPerformance Optimizationhttps://www.chenshaowen.com/en/blog/listen-runtime-security-using-falco.htmlSat, 11 Feb 2023 00:00:00 +0000Sat, 11 Feb 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/listen-runtime-security-using-falco.html1. What Falco Is Falco is a cloud-native runtime security project contributed to the CNCF by Sysdig. Falco implements an extensible event rule filtering engine. Through a series of operations — capturing events, matching security rules, and producing alert notifications — it can surface security problems in a system. The events come from system calls, and it also supports an ebpf probe; the rules are open source [1] and you can define your own extensions [2].WeChat Official AccountFalcoKubernetesSecurityToolsOperationshttps://www.chenshaowen.com/en/blog/how-to-run-stable-diffusion-in-macos-without-gpu.htmlFri, 10 Feb 2023 00:00:00 +0000Fri, 10 Feb 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-run-stable-diffusion-in-macos-without-gpu.html1. Recommended Configuration for Running Stable Diffusion Memory: at least 16 GB DDR4 or DDR5 Storage: at least 10 GB of free space GPU: an NVIDIA card with at least 6 GB of VRAM If your hardware falls short, you can also use one of the various optimized forks that support lower-spec hardware, but generation time will increase.WeChat Official AccountAImacOSGPUStable DiffusionImage GenerationLearninghttps://www.chenshaowen.com/en/blog/using-prometheus-in-the-cluster-to-collect-etcd-metrics.htmlThu, 09 Feb 2023 00:00:00 +0000Thu, 09 Feb 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-prometheus-in-the-cluster-to-collect-etcd-metrics.html1. Skipping Certificate Verification Prevents Scraping If you can skip TLS authentication when scraping metrics, that is the easiest path. The Prometheus ConfigMap configuration looks like this: 1 2 3 4 5 6 7 8 9 - job_name: etcd metrics_path: /metrics scheme: https tls_config: insecure_skip_verify: true static_configs: - targets: ["1.1.1.1:2379"] - targets: ["2.WeChat Official AccountEtcdKubernetesPrometheusMonitoringOperationshttps://www.chenshaowen.com/en/blog/how-to-add-basic-auth-to-kubernetes-service.htmlSun, 05 Feb 2023 00:00:00 +0000Sun, 05 Feb 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-add-basic-auth-to-kubernetes-service.html1. Deploy the Ingress Controller Check the Kubernetes version 1 2 3 4 kubectl version --short Client Version: v1.21.4 Server Version: v1.21.4 Find a compatible Nginx Ingress version Helm Chart version Highest available Helm Chart version Compatible K8s version 3.x.x 3.36.0 1.16+ 4.x.x 4.4.2 1.19+ Reference: https://github.com/kubernetes/ingress-nginx Install the Nginx Ingress Controller 1 2 3 helm upgrade --install ingress-nginx ingress-nginx \ --repo https://kubernetes.WeChat Official AccountKubernetesAuthenticationIngressSecurityOperationshttps://www.chenshaowen.com/en/blog/troubleshoot-slow-io-when-building-dockerfile.htmlThu, 02 Feb 2023 00:00:00 +0000Thu, 02 Feb 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/troubleshoot-slow-io-when-building-dockerfile.html1. The Problem Project details: File size 5.6 GB File count 529352 Dockerfile 1 2 3 FROM golang:1.13 COPY ./ /go/src/code The build command and its output are as follows: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 time DOCKER_BUILDKIT=1 docker build --no-cache -t test:v3 -f Dockerfile .WeChat Official AccountDockerImagesCICDBuildTroubleshootingOperationshttps://www.chenshaowen.com/en/blog/run-mv-is-slower-than-cp-in-dockerfile.htmlMon, 30 Jan 2023 00:00:00 +0000Mon, 30 Jan 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/run-mv-is-slower-than-cp-in-dockerfile.htmlUnlike on CentOS or Ubuntu, where we feel that mv is faster than cp, when you build an image with a Dockerfile, using Run cp is faster than Run mv. This post presents the test and verification data behind that. 1. Test Setup Machine environment Ubuntu 20.04.1 LTS 32CWeChat Official AccountDockerContainersCICDImagesOperationshttps://www.chenshaowen.com/en/blog/how-to-repair-k8s-master-after-reinstalling-os.htmlTue, 17 Jan 2023 00:00:00 +0000Tue, 17 Jan 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-repair-k8s-master-after-reinstalling-os.htmlRecently I ran into this twice: a host had to be reinstalled because of a failure. In one case only a single Etcd node was left, and the whole cluster stayed down for half an hour before it recovered. This post mainly records the process of initializing a fresh UbuntuWeChat Official AccountKubernetesRepairOperationsEtcdTroubleshootingClusterKubekeyLearninghttps://www.chenshaowen.com/en/blog/auto-add-or-remove-nodes-incluster.htmlMon, 16 Jan 2023 00:00:00 +0000Mon, 16 Jan 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/auto-add-or-remove-nodes-incluster.htmlElastic scaling mainly has three dimensions: HPA, which automatically scales the number of Pods based on utilization VPA, which automatically sets a Pod’s Request and Limit based on historical data CA, which automatically scales the number of Nodes based on usage This post focuses on the node scaling part. 1. The autoscaler Autoscaling Component autoscaler is a project maintained by the Kubernetes community.WeChat Official AccountClusterElastic ScalingNodesKubernetesOperationshttps://www.chenshaowen.com/en/blog/tekton-stress-test-and-optimize-k8s-cluster.htmlThu, 10 Nov 2022 00:00:00 +0000Thu, 10 Nov 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/tekton-stress-test-and-optimize-k8s-cluster.html1. Purpose of the Test Tune the parameters of the build cluster Probe the upper limit on the number of concurrent Tekton pipelines Give the optimal concurrency limit for a single cluster 2. Related Components and Machine Configuration Kubernetes version v1.21.4 Tekton version v0.24.1, kept consistent with production OpenEBS version localpv version 3.WeChat Official AccountTektonKubernetesStress TestingLarge ClustersOptimizationCICDOperationshttps://www.chenshaowen.com/en/blog/how-to-estimate-the-local-and-memory-storage-consumption-of-prometheus.htmlWed, 02 Nov 2022 00:00:00 +0000Wed, 02 Nov 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-estimate-the-local-and-memory-storage-consumption-of-prometheus.html1. Local Storage Capacity Required disk size (GB) = data retention period _ samples fetched per second _ metric data size / 1024 / 1024 / 1024 Where Samples fetched per second: rate(prometheus_tsdb_head_samples_appended_total[1d]) Average size of a sample within one hour: rate(prometheus_tsdb_compaction_chunk_size_bytes_sum[1d])/rate(prometheus_tsdb_compaction_chunk_samples_sum[1d]) The disk consumption for one day (86400 seconds) can be queried directly in Prometheus:WeChat Official AccountPrometheusMonitoringKubernetesOperationsLearningCapacity PlanningTroubleshootinghttps://www.chenshaowen.com/en/blog/kubernetes-app-troubleshooting.htmlTue, 01 Nov 2022 03:00:00 +0000Tue, 01 Nov 2022 03:00:00 +0000https://www.chenshaowen.com/en/blog/kubernetes-app-troubleshooting.htmlSetting Reasonable Req and Limit Without Req and Limit set, when an application’s CPU or MEM spikes, it harms other Pods on the same node, and can even cause cluster nodes to be crushed one after another. Req and Limit have four values in total. If only some of them are set, then when node resource usage reaches the threshold preset by Kubelet, Kubelet evicts Pods in the order Guaranteed > Burstable > Best-EffortWeChat Official AccountKubernetesTroubleshootingOperationsPodResourcesLearningBest Practiceshttps://www.chenshaowen.com/en/blog/kubernetes-cluster-troubleshooting.htmlTue, 01 Nov 2022 01:00:00 +0000Tue, 01 Nov 2022 01:00:00 +0000https://www.chenshaowen.com/en/blog/kubernetes-cluster-troubleshooting.htmlFailedCreatePodSandBox Error Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:402: getting the final child's pid from pipe caused: EOF: unknown Resolution Clear the cache 1 echo 3 > /proc/sys/vm/drop_caches Cause Too much memory fragmentation calico-node Keeps Restarting increase max user Error runtime: failed toWeChat Official AccountKubernetesClusterTroubleshootingOperationsLearningFault DiagnosisPodBest PracticesProblem Localizationhttps://www.chenshaowen.com/en/blog/how-to-repair-the-kubernetes-cluster-after-changing-ip.htmlTue, 25 Oct 2022 00:00:00 +0000Tue, 25 Oct 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-repair-the-kubernetes-cluster-after-changing-ip.htmlA record of repairing a cluster failure caused by an IP change. There are two clusters: one single-node (allinone) cluster, and one four-node cluster (3 masters, 1 node). 1. Update the Etcd Certificates [On every Etcd node] Back up the Etcd certificates 1 cp -R /etc/ssl/etcd/ssl /etc/ssl/etcd/ssl-bak View the domainsWeChat Official AccountKubernetesOperationsRepairTroubleshootingLearningEtcdCertificatesClusterNetworkhttps://www.chenshaowen.com/en/blog/kubernetes-network-packets.htmlSat, 17 Sep 2022 00:00:00 +0000Sat, 17 Sep 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/kubernetes-network-packets.htmlThis article is translated from https://learnk8s.io/kubernetes-network-packets, not word for word, with some of my own understanding worked in. By reading this article, you can learn how packets are forwarded inside and outside Kubernetes, starting from the original web request all the way to the container hosting the application. Kubernetes Network Requirements Before diving into the details of how packets flow through a Kubernetes cluster, let’s first clarify Kubernetes’ requirements for networking.WeChat Official AccountTranslationKubernetesNetworkOperationsLearningiptablesCNIhttps://www.chenshaowen.com/en/blog/sre-to-left-devops-to-right.htmlSat, 10 Sep 2022 08:00:00 +0000Sat, 10 Sep 2022 08:00:00 +0000https://www.chenshaowen.com/en/blog/sre-to-left-devops-to-right.html1. A Process Divorced from Responsibility Is Meaningless Software architecture matches organizational architecture, and this shows up not only in functional boundaries but even more in the division of responsibilities. Only clear responsibility boundaries can build good team collaboration and growth. Every team and every person should understand their own goals — what they should take on and what they should stay out of — and invest their time and energy in things that benefit the primary goal, rather than sinking into trivia.WeChat Official AccountSREDevOpsThoughtsOperationsTeamsManagementhttps://www.chenshaowen.com/en/blog/pull-a-large-image-and-get-an-error.htmlSat, 10 Sep 2022 02:00:00 +0000Sat, 10 Sep 2022 02:00:00 +0000https://www.chenshaowen.com/en/blog/pull-a-large-image-and-get-an-error.html1. Picking Up from Last Time: Shared Storage Optimizes Pulling Images Overseas Building on An Image Management and Distribution Case Based on Harbor and Registry, I recently made another optimization. The previous approach was to deploy a Mirror Cache in each region on a machine with low specs and a large disk to cache images.WeChat Official AccountKubernetesImage RegistryImagesOperationsTroubleshootingHarborhttps://www.chenshaowen.com/en/blog/insight-kubernetes-network-by-kindling.htmlSat, 10 Sep 2022 00:00:00 +0000Sat, 10 Sep 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/insight-kubernetes-network-by-kindling.htmlI recently had a requirement to collect Kubernetes’ external network access. So I set up and tried out some related projects. This post is mainly about how to install Kindling and configure Grafana to view Kubernetes network connection data. 1. What Is Kindling Kindling solves the problem of how to observe the network without intruding on the application; its functionality is mainly achieved by exposing kernel events for observation.WeChat Official AccountKubernetesKindlingNetworkObservabilityOperationseBPFMonitoringhttps://www.chenshaowen.com/en/blog/expiration-of-the-openebs-cert.htmlFri, 09 Sep 2022 00:00:00 +0000Fri, 09 Sep 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/expiration-of-the-openebs-cert.html1. Installation Method 1 2 kubectl apply -f https://openebs.github.io/charts/openebs-operator.yaml kubectl patch storageclass openebs-hostpath -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}' OpenEBS is mainly used as the default storage for Tekton pipelines. I had also tried Longhorn before, but it could not hold up during peak hours and the pipelines went Pending. And after uninstalling Longhorn there were leftovers, which caused kube-apiserver to keep reporting errors, and in the end it took a great deal of effort to remove.WeChat Official AccountOpenEBSStorageKubernetesCertificatesTroubleshootingOperationsLearningTLSStorageClasshttps://www.chenshaowen.com/en/blog/some-thoughts-on-the-construction-of-platform.htmlWed, 31 Aug 2022 00:00:00 +0000Wed, 31 Aug 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/some-thoughts-on-the-construction-of-platform.html1. Only Platformization Lets You Go Further As long as you respond to the market a little faster than your competitors and live a little longer, you can win more room to survive. A brilliant business model or market opportunity is more like magic: it can grow a company fast, but it is not something human effort can control.WeChat Official AccountPlatformThoughtsConstructionPaaSR&DOperationshttps://www.chenshaowen.com/en/blog/how-to-estimate-the-resource-request-at-monitoring-kubernetes.htmlTue, 23 Aug 2022 00:00:00 +0000Tue, 23 Aug 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-estimate-the-resource-request-at-monitoring-kubernetes.htmlThe monitoring metrics described in this article cover only the basic Kubernetes metrics, not business-related metrics. The components involved are prometheus-server, kube-state-metrics, and node-exporter, and the data retention period is 3 days. 1. Monitoring-related components in the cluster 1 2 3 4 helm -n monitor list NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION prom-k8s monitor 1 2022-05-12 16:47:53.WeChat Official AccountKubernetesMonitoringCalculationEstimationOperationshttps://www.chenshaowen.com/en/blog/some-common-promql-for-alarming.htmlSun, 21 Aug 2022 11:22:55 +0000Sun, 21 Aug 2022 11:22:55 +0000https://www.chenshaowen.com/en/blog/some-common-promql-for-alarming.htmlHosts Host memory usage exceeds threshold 1 - node_memory_MemAvailable_bytes{mode!="idle"} / node_memory_MemTotal_bytes Threshold: 0.9 Host CPU usage exceeds threshold 1 - avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) by (host_name) Threshold: 0.85 Host disk usage exceeds threshold 1 - avg without (fstype)(node_filesystem_free_bytes{fstype!='rootfs',mountpoint!~'/(run|var|snap).*'} / node_filesystem_size_bytes{fstype!='rootfs',mountpoint!~'/(run|var|snap).*'}) Threshold: 0.8 Windows Windows host memory usage exceeds threshold 1 - 1 * windows_os_physical_memory_free_bytes{job="windows_exporter",mode!WeChat Official AccountPrometheusAlertingMonitoringOperationsPromQLhttps://www.chenshaowen.com/en/blog/set-port-to-be-accessible-only-to-a-specified-ip.htmlSat, 20 Aug 2022 08:00:00 +0000Sat, 20 Aug 2022 08:00:00 +0000https://www.chenshaowen.com/en/blog/set-port-to-be-accessible-only-to-a-specified-ip.html1. Host Service Ports 1 2 iptables -I INPUT -p tcp --dport 80 -j DROP iptables -I INPUT -p tcp -s 1.2.3.4 --dport 80 -j ACCEPT This allows only 1.2.3.4 to access port 80 on the local host. 2. Docker Service Ports For services started like docker run -d -p 80:80 shaowenchen/demo:whoami, the method above does not work; the rule has to be added to the DOCKER-USER chain.WeChat Official AccountDockerLinuxSecurityRestrictionOperationsLearningiptablesFirewallNetworkhttps://www.chenshaowen.com/en/blog/traffic-restriction-using-linux-tc.htmlSat, 20 Aug 2022 00:00:00 +0000Sat, 20 Aug 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/traffic-restriction-using-linux-tc.html1. Traffic Control Principles Under Linux By queuing packets, we can control how data packets are sent. This kind of control is called data shaping (shape the data), and it includes the following operations on data: Adding latency Dropping packets Reordering Duplicating, corrupting Rate control Under the qdisc-class-filter structure, controlling traffic requires three steps:WeChat Official AccountLinuxTCTraffic ControlOperationsNetworkLearningRate LimitingTroubleshootinghttps://www.chenshaowen.com/en/blog/kubernetes-job-create-3w-pod-attack-cluster.htmlWed, 17 Aug 2022 00:00:00 +0000Wed, 17 Aug 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/kubernetes-job-create-3w-pod-attack-cluster.html1. Background At 10:00 in the morning, a colleague asked me to run a task on the cluster, so I created a Kubernetes Job through a tool. When the tool creates a Job, it first gets all the nodes on the cluster, then binds to each node one by one to create Jobs.WeChat Official AccountKubernetesJobPodIncidentTroubleshootingOperationshttps://www.chenshaowen.com/en/blog/how-to-access-a-unified-auth-system-quickly.htmlTue, 26 Jul 2022 00:00:00 +0000Tue, 26 Jul 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-access-a-unified-auth-system-quickly.html1. Authentication and Authorization Problems Brought by Heterogeneous Systems Enterprise systems can be divided into the following types: Commercial software you purchase, such as JumpServer Open-source software, such as Kibana and Grafana Software developed in-house, such as an application management platform One thing worth clarifying here: authentication and authorization are two different functions:WeChat Official AccountAuthenticationAuthorizationSecurityOperationsArchitectureSSOhttps://www.chenshaowen.com/en/blog/hijack-docker-io-req-to-private-repository.htmlMon, 18 Jul 2022 00:00:00 +0000Mon, 18 Jul 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/hijack-docker-io-req-to-private-repository.html1. Self-signing a *.docker.io Domain Certificate 1.1 Creating a CA Certificate Generate the CA certificate private key 1 openssl genrsa -out ca.key 4096 Generate the CA certificate 1 2 3 4 openssl req -x509 -new -nodes -sha512 -days 3650 \ -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=chenshaowen.com" \ -key ca.key \ -out ca.crt 1.2 Creating a *.WeChat Official AccountJenkinsKubernetesDockerImagesOperationsSecurityhttps://www.chenshaowen.com/en/blog/a-image-management-and-distribution-case-based-on-harbor-and-registry.htmlSat, 16 Jul 2022 00:00:00 +0000Sat, 16 Jul 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/a-image-management-and-distribution-case-based-on-harbor-and-registry.html1. The Challenge of a Cross-Region Harbor If all you need is to store image data in a simple way, Registry is an excellent choice as an image repository. Registry not only supports multiple storage backends, but can also be configured with HTTPS certificates and access credentials. Worth noting: Harbor also uses Registry to store image data.WeChat Official AccountHarborRegistryImagesDistributionKubernetesOperationsLearningMulti-RegionAccelerationCICDhttps://www.chenshaowen.com/en/blog/how-to-deploy-the-elasticsearch-stack-on-kubernetes.htmlWed, 06 Jul 2022 00:00:00 +0000Wed, 06 Jul 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-deploy-the-elasticsearch-stack-on-kubernetes.htmlIf you use Logstash to centrally receive the log input from Filebeat, it easily creates a single point of bottleneck; if you use Kafka to receive the log input from Filebeat, the timeliness of the logs is not guaranteed. Here we output the logs collected by Filebeat directly to Elasticsearch.WeChat Official AccountKubernetesElasticsearchLoggingOperationsELKMonitoringhttps://www.chenshaowen.com/en/blog/how-to-set-up-pod-to-run-to-a-specified-node.htmlMon, 27 Jun 2022 00:00:00 +0000Mon, 27 Jun 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-set-up-pod-to-run-to-a-specified-node.html1. Specify a Node via nodeSelector When Creating a Workload Add a label to the node 1 kubectl label node node2 project=A Create the workload with the specified nodeSelector 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 cat <<EOF | kubectl apply -f - apiVersion: apps/v1 kind: Deployment metadata: name: nginx-nodeselector spec: replicas: 1 selector: matchLabels: app: nginx-nodeselector template: metadata: labels: app: nginx-nodeselector spec: nodeSelector: project: A containers: - name: nginx image: nginx EOF Check the workload 1 2 3 4 kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-nodeselector-7bb75b7687-7r5xk 1/1 Running 0 19s 10.WeChat Official AccountPodKubernetesSchedulingOperationsnodeSelectorLearninghttps://www.chenshaowen.com/en/blog/using-blackbox-exporter-to-test-network.htmlSat, 25 Jun 2022 00:00:00 +0000Sat, 25 Jun 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-blackbox-exporter-to-test-network.htmlIf you need to monitor the network status between two hosts, or between a host and an external service, then the Blackbox Exporter mentioned in this article is worth a try. 1. Installing Blackbox 1.1 Deploying on a Host Download the binary package 1 2 3 4 5 wget https://github.WeChat Official AccountPrometheusMonitoringProbingOperationsLearningNetworkBlackbox ExporterAvailabilityhttps://www.chenshaowen.com/en/blog/evolution-of-kubernetes-under-multipl-location.htmlSat, 18 Jun 2022 12:09:53 +0000Sat, 18 Jun 2022 12:09:53 +0000https://www.chenshaowen.com/en/blog/evolution-of-kubernetes-under-multipl-location.html1. Application Architecture Matches Business Growth and Operations Capability At industry conferences, in documents and blogs, we constantly see all kinds of excellent solutions — but if you copy them straight into your own business, you keep hitting walls. Because these technical solutions were incubated in specific business scenarios; different business forms, different business scales, and different stages of business development all affect how technology lands.WeChat Official AccountMulti-LocationKubernetesArchitectureTopologyOperationshttps://www.chenshaowen.com/en/blog/descheduler-makes-kubernetes-load-more-balanced.htmlSat, 11 Jun 2022 00:00:00 +0000Sat, 11 Jun 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/descheduler-makes-kubernetes-load-more-balanced.html1. Why Secondary Scheduling Is Needed The job of the Kubernetes scheduler is to bind a Pod to a single best node. To do this, the scheduler runs through a series of filtering and scoring steps. Kubernetes scheduling is based on Requests, but the actual usage of each Pod changes dynamically.WeChat Official AccountKubernetesdeschedulerSchedulingOperationshttps://www.chenshaowen.com/en/blog/how-to-renew-kubernetes-certs-manually.htmlFri, 10 Jun 2022 00:00:00 +0000Fri, 10 Jun 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-renew-kubernetes-certs-manually.htmlBy default, Kubernetes certificates need to be renewed once a year. Below is a record of one certificate renewal process. 1. Check the Certificates Check the certificate expiration time on a master node: 1 kubeadm certs check-expiration On older cluster versions this command errors out; use: kubeadm alpha certs check-expirationWeChat Official AccountKubernetesCertificatesOperationsSecuritykubeadmTroubleshootingLearninghttps://www.chenshaowen.com/en/blog/how-to-insight-the-pipeline-of-tekton.htmlTue, 07 Jun 2022 00:00:00 +0000Tue, 07 Jun 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-insight-the-pipeline-of-tekton.html1. Scraping Tekton Metrics Add a ConfigMap configuration file 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 cat <<EOF | kubectl apply -f - apiVersion: v1 kind: ConfigMap metadata: name: config-observability namespace: tekton-pipelines labels: app.kubernetes.io/instance: default app.kubernetes.io/part-of: tekton-pipelines data: metrics.backend-destination: prometheus metrics.taskrun.level: "task" metrics.WeChat Official AccountKubernetesTektonMonitoringMetricsCICDOperationshttps://www.chenshaowen.com/en/blog/how-to-collect-labels-and-annotations-of-kubernetes-objects.htmlThu, 02 Jun 2022 01:00:00 +0000Thu, 02 Jun 2022 01:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-collect-labels-and-annotations-of-kubernetes-objects.html1. Why kube-status-metrics is needed Kubernetes monitoring mainly focuses on two kinds of metrics: Basic performance metrics CPU, memory, disk, network, and so on. These can be collected by deploying node-exporter as a DaemonSet and having Prometheus scrape the relevant metrics. Resource object metrics The replica count of a Deployment, the running state of a Pod, and so on.WeChat Official AccountKubernetesMonitoringMetricsCollectionPrometheusOperationsLearningObservabilitykube-status-metricshttps://www.chenshaowen.com/en/blog/peer-domain.htmlThu, 02 Jun 2022 00:00:00 +0000Thu, 02 Jun 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/peer-domain.html1. Meaning If a service has multiple domain entry points and accessing it through any of them yields the same content, those domains are called peer domains. For example, through peer domains you can provide three identical file or API services. https://server.chenshaowen.com/static/index.html https://server-peer-a.chenshaowen.com/static/index.html https://server-peer-b.chenshaowen.com/static/index.html 2. Uses 2.1 Raising the Browser Concurrency Limit Browsers usually cap the maximum number of concurrent requests to a single domain at no more than 6 per second.WeChat Official AccountDomain NamesArchitectureOverseasRoutingOperationsNetworkhttps://www.chenshaowen.com/en/blog/traffic-distribution-cross-multi-region.htmlTue, 31 May 2022 00:00:00 +0000Tue, 31 May 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/traffic-distribution-cross-multi-region.html1. About Traffic Distribution Traffic governance is divided into north-south and east-west. In the typical drawing convention for Network Diagrams, core network components are drawn at the top and clients at the bottom, with different service levels drawn in between — hence the names north-south and east-west traffic. North-south traffic refers to traffic initiated by clients, while east-west traffic refers to traffic between services.WeChat Official AccountOverseasTrafficGatewayOperationsNetworkhttps://www.chenshaowen.com/en/blog/how-to-configure-scheduled-task-using-systemd-timer.htmlThu, 26 May 2022 00:00:00 +0000Thu, 26 May 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-configure-scheduled-task-using-systemd-timer.html1. Preparing the Scheduled Script If it is a Bash script, the first line needs to specify the interpreter. 1 2 mkdir -p /root/scripts vim /root/scripts/quick-clear.sh 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 #!/bin/sh /usr/local/bin/kubectl --kubeconfig /root/.kube/config get pods --all-namespaces -o wide | grep Evicted | awk '{print $1,$2}' | xargs --no-run-if-empty -L1 /usr/local/bin/kubectl --kubeconfig /root/.WeChat Official AccountLinuxsystemdScheduled TasksOperationsLearningcronAutomationBest Practiceshttps://www.chenshaowen.com/en/blog/the-monitor-system-cross-multi-region.htmlWed, 25 May 2022 00:00:00 +0000Wed, 25 May 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-monitor-system-cross-multi-region.html1. Background When you are at a job, you always have to do something, and you want to do something new. But it is not the case that if you want to do it you get the chance, and can do it well. Whether you do it alone or can do it together with everyone else, the final result is different.WeChat Official AccountMonitoringOverseasMulti-RegionOperationsMulti-DatacenterObservabilityhttps://www.chenshaowen.com/en/blog/install-thanos-using-binary.htmlWed, 11 May 2022 19:51:48 +0000Wed, 11 May 2022 19:51:48 +0000https://www.chenshaowen.com/en/blog/install-thanos-using-binary.html1. Download the Binaries 1 2 3 4 wget https://github.com/thanos-io/thanos/releases/download/v0.26.0/thanos-0.26.0.linux-amd64.tar.gz tar xvf thanos-0.26.0.linux-amd64.tar.gz mv thanos-0.26.0.linux-amd64/thanos /usr/bin/ rm -rf thanos-0.26.0.linux-amd64* 2. Install Thanos Query 1 vim /etc/systemd/system/thanos-query.service 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 [Unit] Description=Thanos Query After=network-online.target [Service] Restart=on-failure ExecStart=/usr/bin/thanos query \ --log.WeChat Official AccountMonitoringThanosProtocolPrometheusOperationsObservabilityLearninghttps://www.chenshaowen.com/en/blog/book/a-book-to-understand-the-geoworld.htmlWed, 11 May 2022 00:00:00 +0000Wed, 11 May 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/book/a-book-to-understand-the-geoworld.htmlAuthor: Wang Wei Publisher: CITIC Press Publication Year: 2017-10-15 ISBN: 9787508667751 Notes: Within ten thousand years, geography is an unchangeable fact and a field that strategists continue to study in depth. To China’s east, past Japan and across the North Pacific, lies the United States; to China’s southwest, through the Indian Ocean and the Atlantic, China is also connected to the United States.WeChat Official AccountBooksGeopoliticsPoliticsInternationalhttps://www.chenshaowen.com/en/blog/the-differences-between-ops-and-business-while-designing-the-system.htmlMon, 25 Apr 2022 00:00:00 +0000Mon, 25 Apr 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-differences-between-ops-and-business-while-designing-the-system.html1. Choosing a Communication Protocol Ops systems are better suited to HTTP than to gRPC. More ops and R&D engineers are familiar with HTTP than with other protocols. Once you know the HTTP protocol, learning a RESTful-style HTTP API is quick. Being familiar to more people and easier to learn means better communication and lower handover cost, because they share more common domain background.WeChat Official AccountDesignThoughtsOperationsSystemsArchitectureAPIR&DBest Practiceshttps://www.chenshaowen.com/en/blog/how-to-add-imagepullsecrets-for-kubernetes.htmlFri, 15 Apr 2022 00:00:00 +0000Fri, 15 Apr 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-add-imagepullsecrets-for-kubernetes.html1. Adding via kubectl create 1 kubectl create secret docker-registry mypullsecret --docker-server=harbor.chenshaowen.com --docker-username=robot-test --docker-password=xxxxxx With kubectl create you can add credentials for pulling images directly. 2. Adding via ~/.docker/config.json Log in to the image registry with an account and password 1 docker login harbor.chenshaowen.com:5000 1 docker login harbor.chenshaowen.com You can add more than one.WeChat Official AccountKubernetesImagesOperationsConfigurationhttps://www.chenshaowen.com/en/blog/some-problems-using-obsutil-commands-in-containers.htmlFri, 15 Apr 2022 00:00:00 +0000Fri, 15 Apr 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/some-problems-using-obsutil-commands-in-containers.html1. The container’s ulimit is set too low Error message: 1 2 3 4 5 6 7 CheckpointDir: /root/.obsutil_checkpoint OutputDir: /root/.obsutil_output runtime: mlock of signal stack failed: 12 runtime: increase the mlock limit (ulimit -l) or runtime: update your kernel to 5.3.15+, 5.4.2+, or 5.5+ fatal error: mlock failed Under Docker, you can add the ulimit parameter at startup 1 docker run -it --ulimit memlock=-1 .WeChat Official AccountObject StorageContainersDockerOperationsTroubleshootinghttps://www.chenshaowen.com/en/blog/book/symmetry-and-asymmetry.htmlSun, 03 Apr 2022 00:00:00 +0000Sun, 03 Apr 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/book/symmetry-and-asymmetry.htmlAuthor: [US] T.D. Lee Publisher: Tsinghua University Press Published: 2000-05 ISBN: 9787302038665 Notes: I used to enjoy reading about mathematics and physics. After starting work, I was consumed by solving all kinds of project and technical problems, and gradually lost my curiosity about what the world really is. Thanks to the author’s deep mastery, this book presents symmetry and asymmetry across many fields and scenarios with ease, giving the reader a fairly direct feel for the consistency that lies within this contradiction.WeChat Official AccountBooksPhysicsScienceReadingGeneral Knowledgehttps://www.chenshaowen.com/en/blog/an-advanced-user-guide-about-thanos.htmlMon, 28 Mar 2022 19:51:48 +0000Mon, 28 Mar 2022 19:51:48 +0000https://www.chenshaowen.com/en/blog/an-advanced-user-guide-about-thanos.html1. Aggregating Data with Query As shown above, the components that Thanos Query can connect to are: Thanos Store Gateway Thanos Query Thanos Receive Prometheus, by way of Sidecar By cascading Thanos Queries, we can perform correlated queries across components and build an extremely large monitoring system. This also means that every connected component should provide a sufficiently fast Prometheus API.WeChat Official AccountThanosMonitoringStorageMetricsPrometheusOperationsLearningObservabilityLong-term Storagehttps://www.chenshaowen.com/en/blog/a-global-images-distribution-network.htmlSun, 27 Mar 2022 14:00:00 +0000Sun, 27 Mar 2022 14:00:00 +0000https://www.chenshaowen.com/en/blog/a-global-images-distribution-network.html1. Global Network Planning Many global, multi-region infrastructures do not put much thought into network planning at the outset. Once the business grows complex enough, they are forced into network adjustments and optimization. Yet any major network change has a huge impact on the business. In the end you are stuck in a dilemma, able to do nothing but invest more people, carry the baggage of history, and walk along the edge of a cliff again and again.WeChat Official AccountImagesHarborNetworkKubernetesOperationsDistributionMulti-RegionLearninghttps://www.chenshaowen.com/en/blog/manage-multiple-prometheus-using-thanos.htmlTue, 22 Mar 2022 00:00:00 +0000Tue, 22 Mar 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/manage-multiple-prometheus-using-thanos.html1. Layering of Monitoring As shown above, when building a monitoring system, two strategies are used: Layered monitoring. The benefit of separating IaaS, MySQL middleware, and App layer monitoring is that the systems have high availability and fault tolerance between them. When App layer monitoring stops working, IaaS layer monitoring will immediately make that visible.WeChat Official AccountThanosPrometheusMonitoringStorageKubernetesOperationshttps://www.chenshaowen.com/en/blog/add-ssh-key-login-os-and-troubleshooting.htmlThu, 10 Mar 2022 00:00:00 +0000Thu, 10 Mar 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/add-ssh-key-login-os-and-troubleshooting.html1. Steps to Add a Key 1.1 Generate an SSH Key on the Client There are two key formats: Old format, the private key begins with -----BEGIN RSA PRIVATE KEY----- 1 ssh-keygen -m PEM -t rsa -b 4096 -C "mail@chenshaowen.com" New format, the private key begins with -----BEGIN OPENSSH PRIVATE KEY----- 1 ssh-keygen -t rsa -b 4096 -C "mail@chenshaowen.WeChat Official AccountSSHOperationsConfigurationSecurityTroubleshootinghttps://www.chenshaowen.com/en/blog/how-upgrade-kubernetes-cluster.htmlMon, 07 Mar 2022 00:00:00 +0000Mon, 07 Mar 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-upgrade-kubernetes-cluster.htmlThe idea of the upgrade is: drain the workloads and take the traffic off, then upgrade the control plane nodes first and the worker nodes afterwards. 1. Check the cluster version 1 2 3 4 kubectl version Client Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.4", GitCommit:"e87da0bd6e03ec3fea7933c4b5263d151aafd07c", GitTreeState:"clean", BuildDate:"2021-02-18T16:12:00Z", GoVersion:"go1.15.8", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.WeChat Official AccountKubernetesUpgradeOperationsClusterkubeadmTroubleshootingLearninghttps://www.chenshaowen.com/en/blog/injecting-env-vars-to-kubernetes-pod-and-priority.htmlFri, 04 Mar 2022 00:00:00 +0000Fri, 04 Mar 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/injecting-env-vars-to-kubernetes-pod-and-priority.html1. Several Ways for a Kubernetes Pod to Reference Environment Variables 1.1 Directly Key/Value You can set the Value directly, and you can also use the current Pod’s information as the Value. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 apiVersion: v1 kind: Pod metadata: name: envar-demo labels: purpose: demonstrate-envars spec: containers: - name: envar-demo-container image: gcr.WeChat Official AccountKubernetesEnvironment VariablesConfigurationOperationshttps://www.chenshaowen.com/en/blog/add-outer-dns-server-to-kubernetes-cluster.htmlThu, 24 Feb 2022 01:00:00 +0000Thu, 24 Feb 2022 01:00:00 +0000https://www.chenshaowen.com/en/blog/add-outer-dns-server-to-kubernetes-cluster.html1. Add DNS to the Host 1.1 CentOS The first method: manage DNS with /etc/resolv.conf Disable NetworkManager If you do not disable NetworkManager, the DNS records you add directly to /etc/resolv.conf are lost after you restart the NetworkManager component. 1 2 3 4 vim /etc/NetworkManager/NetworkManager.conf [main] dns=none Add dns=none to the [main] section.WeChat Official AccountKubernetesDNSNetworkOperationshttps://www.chenshaowen.com/en/blog/how-to-migrate-docker-storage-to-new-disk.htmlThu, 24 Feb 2022 00:00:00 +0000Thu, 24 Feb 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-migrate-docker-storage-to-new-disk.html1. Formatting the Disk Check the new disk 1 fdisk -l Usually, the second disk is named /dev/sdb. Partition the disk 1 fdisk /dev/sdb You will be prompted to enter parameters: command (m for help):n Partition number(1-4):1 First cylinder (1-22800,default 1):Enter command (m for help):w Format the disk as ext4 1 mkfs.WeChat Official AccountDockerStorageOperationsMigrationTroubleshootinghttps://www.chenshaowen.com/en/blog/how-to-cache-third-party-package-whilie-building-in-stage.htmlTue, 22 Feb 2022 00:00:00 +0000Tue, 22 Feb 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-cache-third-party-package-whilie-building-in-stage.htmlIn a non-staged build scenario, when we use a container to build, we can mount the cache directory inside the container onto the build host, run the build task, and then copy the artifacts into the runtime image to produce the application image. But with a staged build, the build image and the runtime image live in the same Dockerfile, which makes it harder to optimize the caching of third-party dependencies.WeChat Official AccountCICDCIDevOpsCacheOptimizationhttps://www.chenshaowen.com/en/blog/some-common-scripts-while-using-mysql.htmlFri, 18 Feb 2022 00:00:00 +0000Fri, 18 Feb 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/some-common-scripts-while-using-mysql.html1. Passwordless Login Edit /etc/my.cnf and add a line under [mysqld]: skip-grant-tables=1 Restart the mysqld service 1 systemctl restart mysqld Log in to MySQL as the root user 1 mysql -u root 2. Allow All Access Sources Log in to MySQL 1 mysql -u root -p At the MySQL interactive command line, enter:WeChat Official AccountMySQLScriptingDatabaseOperationsR&DLearninghttps://www.chenshaowen.com/en/blog/monitor-and-alarm-system-101.htmlTue, 15 Feb 2022 00:00:00 +0000Tue, 15 Feb 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/monitor-and-alarm-system-101.htmlThe hard part of a monitoring system is storing large volumes of time-series data and providing high-performance query capability; the hard part of an alerting system is designing an efficient alert engine and implementing a flexible alert escalation mechanism. I have been tracking monitoring and alerting systems for a while now, and this post mainly organizes some of the concepts, components, and research options related to monitoring and alerting.WeChat Official AccountMonitoringAlertingOperationsPrometheusArchitecturehttps://www.chenshaowen.com/en/blog/taxonomy-of-educational-objectives.htmlTue, 08 Feb 2022 00:00:00 +0000Tue, 08 Feb 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/taxonomy-of-educational-objectives.html1. Bloom’s Taxonomy Bloom’s taxonomy is a classification scheme proposed by the American educational psychologist Benjamin Bloom in 1956 at the University of Chicago. It classifies educators’ teaching objectives so that each objective can be achieved more effectively. According to Bloom’s theory, knowledge can be divided into the following threeWeChat Official AccountGeneral KnowledgeEducationLearningCognitionMethodologyThoughtshttps://www.chenshaowen.com/en/blog/interviewer-training-about-how-to-choose-right-people.htmlTue, 18 Jan 2022 00:00:00 +0000Tue, 18 Jan 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/interviewer-training-about-how-to-choose-right-people.html1. Interview Process Establish hiring criteria Align on hiring criteria Resume analysis Opening small talk Competency assessment Values assessment Verifying authenticity Motivation assessment Attracting strong candidates Closing the interview Filling out the interview evaluation form Assisting with onboarding follow-up 2. Before the Interview 2.1 Establish Hiring Criteria why Evaluate candidates with a unified assessment standardWeChat Official AccountInterviewTrainingManagementTeamsRecruitingTalentLearningHuman Resourceshttps://www.chenshaowen.com/en/blog/several-open-source-kubernetes-web-management-tools.htmlMon, 27 Dec 2021 01:00:00 +0000Mon, 27 Dec 2021 01:00:00 +0000https://www.chenshaowen.com/en/blog/several-open-source-kubernetes-web-management-tools.htmlI have recently been researching open-source Kubernetes management platforms. The requirement is to manage hundreds of clusters on an intranet. The functional positioning is to assist operations and provide capabilities to the application layer, rather than serving end users directly. 1. Kubernetes Dashboard Project address: https://github.com/kubernetes/dashboard Tech stack: Angular + GoWeChat Official AccountKubernetesComparisonManagement ToolsOperationshttps://www.chenshaowen.com/en/blog/a-easy-tutorial-of-installation-dsm-system.htmlTue, 21 Dec 2021 00:00:00 +0000Tue, 21 Dec 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/a-easy-tutorial-of-installation-dsm-system.html1. Installation Preparation A USB drive with PE installed On MacOS, you can use balenaEtcher to burn the wepe for mac image. Download the synoboot file http://down.nas2x.com/synology/dsm/6.2/synoboot/ Download the DSM.pat file https://archive.synology.com/download/Os/DSM 2. Getting Started Boot through PE and enter the system Write synoboot to the built-in SSD storage If you boot from a USB drive, you may also need to modify the PID and VID values in grup.WeChat Official AccountDSMSynologyDiskOperationshttps://www.chenshaowen.com/en/blog/how-to-config-host-monitor-on-grafana.htmlMon, 20 Dec 2021 00:00:00 +0000Mon, 20 Dec 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-config-host-monitor-on-grafana.html1. Install node exporter 1 2 3 wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz tar xvfz node_exporter-1.3.1.linux-amd64.tar.gz mv node_exporter-1.3.1.linux-amd64/node_exporter /usr/local/bin/ If you cannot reach GitHub, refer to: Several ways to access GitHub from mainland China . 2. Configure and start the exporter Add a Systemd configuration 1 vim /etc/systemd/system/node_exporter.service Add the following content: 1 2 3 4 5 6 7 8 9 10 11 12 [Unit] Description=Node Exporter After=network.WeChat Official AccountGrafanaPrometheusMonitoringHostOperationshttps://www.chenshaowen.com/en/blog/how-to-reading-btrfs-partition-files-under-macos.htmlSun, 19 Dec 2021 00:00:00 +0000Sun, 19 Dec 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-reading-btrfs-partition-files-under-macos.html1. Background and Prerequisites Compared with Ext4, Btrfs is a younger file system with more features to play with — snapshots, subvolumes, checksum self-verification, software RAID, and even transparent compression. That said, if you do not have the operations capability behind you, it is best not to use Btrfs. This article records the process of pulling a Btrfs disk out of a Synology DSM system and reading its data.WeChat Official AccountMacOSBtrfsFilesFile SystemOperationshttps://www.chenshaowen.com/en/blog/some-questions-of-using-go-mod.htmlFri, 10 Dec 2021 00:00:00 +0000Fri, 10 Dec 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/some-questions-of-using-go-mod.html1. go.sum Version Mismatch After Upgrading Go Running the command go build reports an error. Error message: missing go.sum entry for module providing package golang.org/x/time/rate; to add Solution: Update go.sum before go build by running the command go mod tidy. 2. TLS Error Running the command go mod download reports an error.WeChat Official AccountGoR&DDependency ManagementTroubleshootinghttps://www.chenshaowen.com/en/blog/common-commands-for-cleaning-up-kubernetes-cluster-resources.htmlWed, 08 Dec 2021 00:00:00 +0000Wed, 08 Dec 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/common-commands-for-cleaning-up-kubernetes-cluster-resources.htmlClusters that run for a long time often face various kinds of resource exhaustion. On top of that, when disk space runs low, Kubelet will proactively clean up images on its own, adding another source of uncertainty. This article provides some command snippets for cleanup work. 1. Cleaning Up Basic Kubernetes Objects Clean up Pods in Evicted state 1 sudo kubectl get pods --all-namespaces -o wide | grep Evicted | awk '{print $1,$2}' | sudo xargs -L1 kubectl delete pod -n Clean up Pods in Error state 1 sudo kubectl get pods --all-namespaces -o wide | grep Error | awk '{print $1,$2}' | sudo xargs -L1 kubectl delete pod -n Clean up Pods in Completed state 1 sudo kubectl get pods --all-namespaces -o wide | grep Completed | awk '{print $1,$2}' | sudo xargs -L1 kubectl delete pod -n Clean up unused PVs 1 sudo kubectl describe -A pvc | grep -E "^Name:.WeChat Official AccountKubernetesPracticeCleanupOperationsLearningkubectlTroubleshootingResource Managementhttps://www.chenshaowen.com/en/blog/the-basic-configuration-of-logstash.htmlTue, 07 Dec 2021 00:00:00 +0000Tue, 07 Dec 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-basic-configuration-of-logstash.html1. How Logstash Works Logstash is a component for data transport and processing. Through combinations of plugins, Logstash can handle a wide variety of log collection scenarios: logstash->elasticsearch filebeat->logstash->elasticsearch filebeat->kafka->logstash->elasticsearch filebeat->redis->logstash->elasticsearch 2. Basic Logstash Configuration Here is the format of a Logstash configuration: 1 2 3 4 5 6 7WeChat Official AccountLoggingConfigurationPracticeELKOperationsLearninghttps://www.chenshaowen.com/en/blog/the-tips-of-time-and-tz-in-go.htmlTue, 07 Dec 2021 00:00:00 +0000Tue, 07 Dec 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-tips-of-time-and-tz-in-go.html1. Time and Time Zones 1.1 Time Standards UTC, Coordinated Universal Time, is the current time standard, counted by atomic time. GMT, Greenwich Mean Time, is the older time standard, which defines the moment the sun passes over the Royal Observatory in Greenwich, on the outskirts of London, as 12 noon.WeChat Official AccountGoR&DTimeTime Zonehttps://www.chenshaowen.com/en/blog/cant-connect-cloudflare-while-pulling-docker-image.htmlFri, 12 Nov 2021 00:00:00 +0000Fri, 12 Nov 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/cant-connect-cloudflare-while-pulling-docker-image.html1. Error message When pulling an image, you will occasionally run into the following error: 1 2 3 4 5 6 7 8 docker pull node:10.16-alpine 10.16-alpine: Pulling from library/node e7c96db7181b: Already exists 50958466d97a: Pulling fs layer 56174ae7ed1d: Pulling fs layer 284842a36c0d: Pulling fs layer error pulling image configuration: Get https://production.WeChat Official AccountDockerImagesTroubleshootingNetworkOperationshttps://www.chenshaowen.com/en/blog/using-assembly-and-template-model-to-manage-app-config.htmlWed, 10 Nov 2021 00:00:00 +0000Wed, 10 Nov 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-assembly-and-template-model-to-manage-app-config.htmlApplication configuration management is about the configuration that an application depends on at runtime, which is different from the static configuration of a project. This article is a summary drawn from real development work, offered for your reference; discussion is welcome too. 1. About Configuration Management 1.1 Terminology Configuration item A single key=value pairWeChat Official AccountConfiguration ManagementPaaSPlatformConfigurationR&Dhttps://www.chenshaowen.com/en/blog/the-scale-and-node-size-of-kubernetes-cluster.htmlMon, 01 Nov 2021 10:00:00 +0000Mon, 01 Nov 2021 10:00:00 +0000https://www.chenshaowen.com/en/blog/the-scale-and-node-size-of-kubernetes-cluster.htmlJudging from the survey data, more people lean toward choosing small clusters. Below is a comparison list from the perspective of node count and node configuration. Below is a comparison of choosing a few large clusters versus many small clusters: Feature A few large clusters Many small clusters Resource utilization High Low Node management overhead Low High Resource scaling range Large Small Resource scheduling range Large Small Efficiency of inter-application communication High Low Cluster operations difficulty High Low Cluster version diversity Low High Blast radius Large Small Below is a comparison of choosing a few high-spec nodes versus many low-spec nodes:WeChat Official AccountKubernetesClusterPlanningOperationsNodehttps://www.chenshaowen.com/en/blog/a-few-communication-methods-of-jenkins-agent.htmlMon, 01 Nov 2021 00:00:00 +0000Mon, 01 Nov 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/a-few-communication-methods-of-jenkins-agent.html1. SSH Connection direction: Master actively connects to Agent Requirements: Master can connect directly to Agent over SSH The SSHD service is running on Agent Jenkins has the SSH Credentials Plugin and the SSH Build Agents plugin installed 2. JNLP Connection direction: Agent actively connects to Master Requirements: Agent can reach Master Agent needs a JVM environment to run Master needs to expose port 50000 (the default) for Agent communication Port 50000 is the service port of the Jenkins Master; no other service program needs to be run for it.WeChat Official AccountJenkinsCICDDevOpsOperationshttps://www.chenshaowen.com/en/blog/book/measure-what-matters.htmlSat, 23 Oct 2021 00:00:00 +0000Sat, 23 Oct 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/book/measure-what-matters.htmlSubtitle: The Working Method Behind the Explosive Growth of Google and Amazon Author: John Doerr (US) Publisher: CITIC Press Publication Year: 2018-12 ISBN: 9787508696881 Notes: OKR stands for Objectives and Key Results, a working method for managing goals and their completion invented by Intel founder Andy Grove. The O stands for Objective, which should be important and specific, action-oriented, and inspiring — for example, improving the stability of a system.WeChat Official AccountBooksWorkMethodsManagementOKRGoalshttps://www.chenshaowen.com/en/blog/what-makes-you-common.htmlThu, 14 Oct 2021 00:00:00 +0000Thu, 14 Oct 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/what-makes-you-common.html1. Lacking a Desire That Comes from Within Desire is what creates motivation. The desire to survive and the desire for something better are the wellspring that drives human progress. The stronger the desire, the stronger the motivation. An intense desire makes a person charge at the goal regardless of everything.WeChat Official AccountThoughtsGrowthLifePersonal GrowthMethodologyhttps://www.chenshaowen.com/en/blog/the-construction-of-two-places-and-three-centers-under-the-container.htmlSun, 10 Oct 2021 00:00:00 +0000Sun, 10 Oct 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-construction-of-two-places-and-three-centers-under-the-container.html1. About Two Places and Three Centers As shown above, the two-places-three-centers architecture exists to improve the fault tolerance and disaster recovery capability of a system. When one data center becomes unavailable, traffic for critical business can be switched to another data center, which makes the system resilient to city-level natural disasters.WeChat Official AccountTwo Places Three CentersContainersKubernetesOperationsDisaster Recoveryhttps://www.chenshaowen.com/en/blog/book/low-iq-society.htmlTue, 05 Oct 2021 12:09:51 +0000Tue, 05 Oct 2021 12:09:51 +0000https://www.chenshaowen.com/en/blog/book/low-iq-society.htmlSubtitle: How to Break Out of the Decline of IQ Author: (Japan) Kenichi Ohmae Publisher: CITIC Press Publication Year: 2010-4 ISBN: 9787508619262 Notes: What is worth noting is when this book was written. More than a decade later, reading it still conveys its weight. The more developed the network, the lazier the brain.WeChat Official AccountBooksThoughtsSocietyDevelopmenthttps://www.chenshaowen.com/en/blog/the-definition-of-application-runtime-under-multiple-env.htmlSat, 18 Sep 2021 00:00:00 +0000Sat, 18 Sep 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-definition-of-application-runtime-under-multiple-env.html1. Why We Need to Define the Application Runtime There are more runtime choices. Traditional application runtimes include physical machines, virtual machines, and cloud hosts. In the container era, the common runtimes are Docker and Kubernetes. These runtimes no longer present us with a single runtime choice. Application topologies are more complex.WeChat Official AccountLabelsDevelopmentEnvironmentOperationsR&Dhttps://www.chenshaowen.com/en/blog/multi-cluster-applications-under-kubevela.htmlFri, 17 Sep 2021 00:00:00 +0000Fri, 17 Sep 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/multi-cluster-applications-under-kubevela.htmlKubevela is currently at version 1.1. We usually consider 1.x releases to be relatively stable and safe to try in production. Through continuous tracking and learning, I have also come to appreciate some of the things Kubevela does well. This is a summary document. 1. What Problem Kubevela Solves Aimed at platform developers Several roles need to be distinguished: development, operations, and operations development.WeChat Official AccountKubevelaKubernetesApplicationOperationsMulti-ClusterOAMLearningCloud Nativehttps://www.chenshaowen.com/en/blog/how-to-use-cilium-to-replace-calico.htmlThu, 16 Sep 2021 08:01:28 +0000Thu, 16 Sep 2021 08:01:28 +0000https://www.chenshaowen.com/en/blog/how-to-use-cilium-to-replace-calico.html1. System Requirements Kubernetes, >= 1.16.0-0 Helm, >= 3.0 Linux kernel >= 4.9.17 For the process of upgrading the kernel on CentOS 7, see How to Switch the Data Plane to eBPF Under Calico. 2. Uninstalling Calico Delete the cluster resources 1 2 3 4 5 6 kubectl -n kube-system delete ds calico-node kubectl -n kube-system delete deploy calico-kube-controllers kubectl -n kube-system delete sa calico-node kubectl -n kube-system delete sa calico-kube-controllers kubectl -n kube-system delete cm calico-config kubectl -n kube-system delete secret calico-config 1 kubectl get crd | grep calico | awk '{print $1}' | xargs kubectl delete crd Bring down the Tunl0 interface 1 ifconfig tunl0 down Remove the Calico configuration files 1 rm -rf /etc/cni/net.WeChat Official AccountCiliumCalicoKubernetesContainersNetworkhttps://www.chenshaowen.com/en/blog/how-to-switch-data-plane-to-ebpf.htmlWed, 15 Sep 2021 07:33:55 +0000Wed, 15 Sep 2021 07:33:55 +0000https://www.chenshaowen.com/en/blog/how-to-switch-data-plane-to-ebpf.html1. Environment Preparation 1.1 Calico eBPF Requirements System requirements Ubuntu 18.04.4+ Red Hat v8.2 Linux kernel v5.3+ If Calico does not detect a compatible kernel, it falls back to standard mode. Every node needs a BPF filesystem mounted at /sys/fs/bpf Calico version no lower than 3.13 1.2 Upgrading the Kernel This uses CentOS 7:WeChat Official AccountNetworkCalicoeBPFKubernetesOperationshttps://www.chenshaowen.com/en/blog/the-question-and-answer-of-kubernetes-installation.htmlFri, 10 Sep 2021 00:00:00 +0000Fri, 10 Sep 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-question-and-answer-of-kubernetes-installation.html1. Calico 1.1 BIRD is not ready 1 kubectl -n kube-system get pod calico-node-xxx stays at 0/1 and never comes up, reporting calico/node is not ready: BIRD is not ready: BGP not established with Solution: Calico uses first-found by default, meaning it takes the NodeIP from the first NIC it finds.WeChat Official AccountKubernetesInstallationTroubleshootingCalicoOperationshttps://www.chenshaowen.com/en/blog/the-nload-is-a-the-terminal-tool-of-linux-network-traffic-monitor.htmlFri, 10 Sep 2021 00:00:00 +0000Fri, 10 Sep 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-nload-is-a-the-terminal-tool-of-linux-network-traffic-monitor.htmlnload is a terminal tool for traffic monitoring. It reads the /proc/net/dev file and draws traffic graphs. Below is a brief introduction to using it. 1. Installation CentOS 1 yum install -y nload If your system’s repositories cannot find the nload package, run the following command to add a newWeChat Official AccountLinuxToolsnloadMonitoringOperationshttps://www.chenshaowen.com/en/blog/how-to-add-entrance-to-kubernetes-apiserver.htmlWed, 08 Sep 2021 00:00:00 +0000Wed, 08 Sep 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-add-entrance-to-kubernetes-apiserver.html1. How to Access a Remote Cluster Locally During development, you often need to connect directly to a remote Kubernetes cluster. The usual approach is to copy /etc/kubernetes/admin.conf to your local ~/.kube/kubeconfig. But the kubeconfig’s server address is kubernetes.default.svc. So we need to configure a hosts entry: 1 1.1.1.1 kubernetes.default.svc If you need to switch between different clusters, you not only have to swap the kubeconfig but also modify hosts.WeChat Official AccountKubernetesR&DCertificatesOperationsSecurityConfigurationhttps://www.chenshaowen.com/en/blog/the-monitor-of-kubernetes-using-prometheus-grafana.htmlFri, 03 Sep 2021 00:00:00 +0000Fri, 03 Sep 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-monitor-of-kubernetes-using-prometheus-grafana.htmlThe Prometheus community moves so fast that some of the documents I wrote earlier have become outdated. I’ve recently started paying attention to observability again, filling in some of the gaps in my operations knowledge. 1. Terminology Grafana A visualization tool that provides a variety of visualization panels and supports many data sources, including Prometheus, OpenTSDB, MySQL, and others.WeChat Official AccountPrometheusGrafanaKubernetesMonitoringOperationsObservabilityhttps://www.chenshaowen.com/en/blog/why-does-jenkins-agent-like-going-to-the-same-node.htmlThu, 02 Sep 2021 00:00:00 +0000Thu, 02 Sep 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/why-does-jenkins-agent-like-going-to-the-same-node.html1. Background Many build nodes were added to Jenkins using the same Label for pipelines to use, but Jenkins tends to run builds on the same node every time. This leads to concurrency problems: a single node is overloaded while other nodes sit idle, so the load is extremely unbalanced.WeChat Official AccountJenkinsCICDDevOpsKubernetesSchedulingOperationsTroubleshootinghttps://www.chenshaowen.com/en/blog/a-gateway-under-kubernetes-named-apisix.htmlWed, 01 Sep 2021 00:00:00 +0000Wed, 01 Sep 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/a-gateway-under-kubernetes-named-apisix.html1. Comparing a Few Common Gateways Nginx, a reverse proxy with a modular design, written in C OpenResty, a web development platform built around Nginx that can parse and execute Lua scripts Kong, an application on top of OpenResty, an API gateway with API management and request proxying, using PostgreSQL for storage APISIX, which replaces Kong’s PostgreSQL with Etcd and is built on Nginx’s core libraries APISIX’s advantage lies in the API management and extensibility it provides, so the gateway no longer merely forwards traffic to services but can be configured and customized.WeChat Official AccountKubernetesAPISIXMicroservicesGatewayOperationsProxyLearningCloud Nativehttps://www.chenshaowen.com/en/blog/develop-a-wechat-backend-using-tencent-serverless.htmlSat, 28 Aug 2021 00:00:00 +0000Sat, 28 Aug 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/develop-a-wechat-backend-using-tencent-serverless.html1. Where Serverless Fits If cloud computing is meant to make resources flow like tap water — turn it on and use it, turn it off and it stops — then Serverless is one of the future directions of cloud computing. Compared with runtimes such as IaaS and Kubernetes, ServerlessWeChat Official AccountGitHubCICDServerlessWeChatR&DOperationshttps://www.chenshaowen.com/en/blog/a-cicd-system-based-on-tekton.htmlWed, 25 Aug 2021 00:00:00 +0000Wed, 25 Aug 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/a-cicd-system-based-on-tekton.html1. Background 1.1 Problems with the Current Use of Jenkins The orchestration engine is unstable Jenkins is an orchestration engine written in Java, and it stops the world (STW) during a full GC. Under large-scale builds, STW can prevent Jenkins from handling new requests. Heavy builds stall Jenkins stores data in disk files; every pipeline and every build occupies a file directory, producing a huge number of files.WeChat Official AccountCICDDevOpsTektonKubernetesOperationsPipelineLearninghttps://www.chenshaowen.com/en/blog/how-to-trigger-github-action-remotely.htmlWed, 25 Aug 2021 00:00:00 +0000Wed, 25 Aug 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-trigger-github-action-remotely.htmlUsually, we need to perform some operation on GitHub in order to trigger a GitHub Action. This post introduces a way to trigger a GitHub Action remotely through an API call. 1. Common Ways to Trigger a GitHub Action Here is an example GitHub Action: 1 2 3 4 5 6 7 name: GitHub Actions Demo on: [push, pull_request] jobs: Explore-GitHub-Actions: runs-on: ubuntu-latest steps: - run: echo "Hello World!WeChat Official AccountGitHubGitHub ActionsCICDDevOpsToolsR&Dhttps://www.chenshaowen.com/en/blog/using-kubefed-to-distribute-tekton-resource-cross-cluster.htmlSat, 26 Jun 2021 00:00:00 +0000Sat, 26 Jun 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-kubefed-to-distribute-tekton-resource-cross-cluster.html1. The Advantages of Multi-Cluster Tekton Builds Thanks to Kubernetes, Tekton already has good elasticity and can support large-scale builds. At the same time, developing Tasks mainly uses Yaml and Shell, which widens the range of scenarios Tekton can adapt to. The figure above is a sketch of Tekton across multiple clusters.WeChat Official AccountTektonKubeFedKubernetesMulti-ClusterCICDOperationsLearningPipelineBuildDistributed Systemshttps://www.chenshaowen.com/en/blog/building-and-using-go-private-packages.htmlThu, 24 Jun 2021 00:00:00 +0000Thu, 24 Jun 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/building-and-using-go-private-packages.html1. Create a Go Modules Project Create the directory 1 2 mkdir go-test cd go-test Initialize the package 1 2 3 4 5 go mod init gitlab.private.com/shaowenchen/go-test go: creating new go.mod: module gitlab.private.com/shaowenchen/go-test go: to add module requirements and sums: go mod tidy Add the business code main.go 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 package main import ( "github.WeChat Official AccountGoGetting StartedBuildR&Dhttps://www.chenshaowen.com/en/blog/how-to-implement-approval-function-in-tekton.htmlThu, 24 Jun 2021 00:00:00 +0000Thu, 24 Jun 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-implement-approval-function-in-tekton.html1. Basic Features of a CICD Platform A common CICD engine is not suitable to hand directly to the business side. The main reasons are the high learning cost for users, the lack of necessary authentication, and the difficulty of maintenance and upgrades. We usually build on top of a process engine, adapting it to the business to improve usability and encapsulating it for specific scenarios to reduce complexity.WeChat Official AccountTektonKubernetesCICDDevOpsOperationshttps://www.chenshaowen.com/en/blog/how-to-change-kubelet-config.htmlWed, 23 Jun 2021 00:00:00 +0000Wed, 23 Jun 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-change-kubelet-config.html1. Edit the Kubelet Configuration File 1 vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf Modify the relevant Kubelet parameters ExecStart=/usr/local/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS You can list the available parameters with the following command: 1 kubelet --help For example, if you need to change --sync-frequency in KUBELET_KUBECONFIG_ARGS, the period for syncing container configuration, the default value is 1 min.WeChat Official AccountKubernetesKubeletConfigurationOperationsLearningNodesystemdTroubleshootinghttps://www.chenshaowen.com/en/blog/how-to-configure-networkpolicy-for-nodeport.htmlWed, 23 Jun 2021 00:00:00 +0000Wed, 23 Jun 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-configure-networkpolicy-for-nodeport.html1. Background As shown above, the business team needs to isolate the services in a namespace, forbidding access from workloads in the bar namespace while allowing users to reach the service through a Load Balancer (LB) via NodePort. It is easy to write a network policy: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 apiVersion: networking.WeChat Official AccountNetworkPolicyNetworkKubernetesOperationsSecurityLearningNodePortNetwork Isolationhttps://www.chenshaowen.com/en/blog/can-we-mount-var-lib-docker-to-remote-storage.htmlTue, 22 Jun 2021 00:00:00 +0000Tue, 22 Jun 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/can-we-mount-var-lib-docker-to-remote-storage.htmlNo. 1. Background When building a reliable, stable operations system on top of Kubernetes, destroying and creating virtual machines (VMs) is the norm. VMs provide compute and memory resources, while external storage is exposed to PVCs in the cluster through a StorageClass. Against this backdrop, how to initialize a VM quickly becomes a new challenge.WeChat Official AccountDockerContainersStorageKubernetesCan Wehttps://www.chenshaowen.com/en/blog/how-to-customize-your-jenkins.htmlSat, 05 Jun 2021 00:00:00 +0000Sat, 05 Jun 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-customize-your-jenkins.html1. How to Deploy Jenkins To make the tests below easier, let me first introduce two ways to deploy Jenkins. Here I use the shaowenchen/jenkins:2.277.4 image. In a production environment, you should replace it with the official jenkins/jenkins image or your own custom image. 1.1 Running with docker-compose The docker-compose.yaml fileWeChat Official AccountDevOpsCICDJenkinsDockerOperationshttps://www.chenshaowen.com/en/blog/best-practices-for-jenkins-on-kubernetes.htmlFri, 04 Jun 2021 00:00:00 +0000Fri, 04 Jun 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/best-practices-for-jenkins-on-kubernetes.html1. Configure Larger -Xms -Xmx Parameters Jenkins is an orchestration engine written in Java, and it stops the world (STW) during a Full GC. At large build scale, STW can prevent Jenkins from handling new requests. To avoid frequent STW while also increasing concurrency, it is recommended to set a larger heap, -Xms3g -Xmx6g -XX:MaxRAM=6g.WeChat Official AccountDevOpsCICDJenkinsKubernetesOperationshttps://www.chenshaowen.com/en/blog/a-universal-design-for-pipeline.htmlWed, 19 May 2021 00:00:00 +0000Wed, 19 May 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/a-universal-design-for-pipeline.html1. Decoupling the Engine to Unlock Pipeline Capabilities When designing a system, we often face a dilemma. Do we contain the complexity and offer a single, easy-to-use capability to the outside; or do we release the complexity and return flexibility to the user? This is a real test of productWeChat Official AccountCloud NativeDevOpsCICDThoughtsR&Dhttps://www.chenshaowen.com/en/blog/network-policy-of-kubernetes.htmlTue, 18 May 2021 00:00:00 +0000Tue, 18 May 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/network-policy-of-kubernetes.html1. Network Isolation in Kubernetes Kubernetes introduced Network Policy in 1.3. It defines entities through ipBlock, podSelector, and namespaceSelector, and controls their From (Ingress) and To (Egress) traffic behavior. But Kubernetes only defines the network policy; the actual implementation depends on the network plugin. Today, network plugins such as Calico, Cilium, and Weave Net all support network isolation.WeChat Official AccountKubernetesNetwork IsolationNetworkPolicyOperationshttps://www.chenshaowen.com/en/blog/blueking-is-a-platform-worth-having.htmlSat, 15 May 2021 00:00:00 +0000Sat, 15 May 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/blueking-is-a-platform-worth-having.html1. My Experience Building Platforms After graduation I joined Tencent BlueKing, mainly working on SaaS development. After nearly three years there, I went back to my hometown of Wuhan and joined QingCloud, where I was responsible for DevOps R&D. After nearly two years, I joined a new company, took part in building a business support platform, and thought about how the business side puts Kubernetes into practice.WeChat Official AccountPaaSBlueKingThoughtsPlatformDevOpsOperationshttps://www.chenshaowen.com/en/blog/some-tips-to-access-github.htmlFri, 14 May 2021 00:00:00 +0000Fri, 14 May 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/some-tips-to-access-github.htmlFor well-known reasons, accessing Github from a network environment in China is either blocked or very slow. This article provides several ways to get around it. 1. Download Using a Github Mirror Just prepend a Proxy address to the GitHub repository; the way different Mirrors are concatenated may vary. TheWeChat Official AccountGitHubR&DOperationsNetworkProxyhttps://www.chenshaowen.com/en/blog/jenkins-vs-tekton.htmlSat, 01 May 2021 00:00:00 +0000Sat, 01 May 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/jenkins-vs-tekton.htmlBelow is a list comparing Jenkins and Tekton: Feature Jenkins Tekton Programming language Java Golang Language for developing plugins Java Shell, Yaml Pipeline description language Groovy, Shell Yaml, Shell Plugin ecosystem Many plugins, LDAP, GitLab Lacking Number of plugins 1500+ 100+ Compatibility between plugins May conflict, cannot be upgraded freely Fully compatible Secondary development Wrapping the API Composing Tasks High availability Integrates Gearman, master-slave mode Relies on Kubernetes’ high availability Concurrent build scale per instance A few hundred concurrent builds Relies on Kubernetes’ Pod management capability, so it can be very large Data storage Local disk Etcd Supports automatic triggering Yes Yes Has commercial support No NoWeChat Official AccountCICDJenkinsTektonKubernetesDevOpsComparisonhttps://www.chenshaowen.com/en/blog/pull-images-unlinkat.htmlThu, 29 Apr 2021 00:00:00 +0000Thu, 29 Apr 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/pull-images-unlinkat.htmlThis is on CentOS 7, kernel version 3.10.0-327. 1. Check the current Docker version 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 docker version Client: Docker Engine - Community Version: 20.WeChat Official AccountDockerImagesTroubleshootingOperationsLearningStorage Driveroverlay2https://www.chenshaowen.com/en/blog/how-to-run-a-private-registry-mirror.htmlThu, 29 Apr 2021 00:00:00 +0000Thu, 29 Apr 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-run-a-private-registry-mirror.htmlA Docker mirror can only accelerate images from docker.io, not images from private registries. 1. Why You Need a Private Registry Mirror Rate limiting on the public network Docker Hub pull rate limits Reduce image pull time 2. Create a Registry Image Acceleration Service Generate a configuration file version: 0.WeChat Official AccountDockerContainersImagesOperationsRegistryAccelerationhttps://www.chenshaowen.com/en/blog/how-to-add-physical-machines-to-tekton.htmlWed, 28 Apr 2021 00:00:00 +0000Wed, 28 Apr 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-add-physical-machines-to-tekton.html1. Why Physical Build Machines Are Needed In the article How to Add a Remote macOS Physical Machine for Jenkins Pipeline Builds, I described how to add a physical build machine in Jenkins. This was not a requirement I made up on a whim — at the time, real ToB commercial customers were actually asking for a solution.WeChat Official AccountTektonCloud NativeKubernetesCICDDevOpsOperationshttps://www.chenshaowen.com/en/blog/how-to-install-macos-on-dell-optiplex-7080.htmlMon, 26 Apr 2021 00:00:00 +0000Mon, 26 Apr 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-install-macos-on-dell-optiplex-7080.htmlAbout ten years ago, I installed macOS on a laptop. The biggest headache back then was that only specific hardware could succeed, and drivers for all sorts of things were missing. Later I bought my own Mac laptop and stopped paying attention to how to install macOS on generic machines for a long time.WeChat Official AccountHackintoshmacOSHardwareOperationshttps://www.chenshaowen.com/en/blog/some-common-computer-interview-questions.htmlMon, 26 Apr 2021 00:00:00 +0000Mon, 26 Apr 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/some-common-computer-interview-questions.htmlComputer Fundamentals Heap and Stack When do you use the heap? When do you use the stack? What is the stack for? Are Go variables on the stack or the heap? Do the heap and stack have limits? If so, what are they related to? Data Structures How is space allocated for a slice?WeChat Official AccountR&DInterviewLearningComputer Fundamentalshttps://www.chenshaowen.com/en/blog/how-to-distribute-files-and-scripts-to-vm-under-kubernetes.htmlSat, 24 Apr 2021 00:00:00 +0000Sat, 24 Apr 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-distribute-files-and-scripts-to-vm-under-kubernetes.html1. What This Article Mainly Discusses Never build a high tower on shifting sand. Business growth and the evolution of business forms both need solid, powerful IT systems to support them. Business content is transparent to the market, but IT systems cannot be built to perfection overnight. In the future, competition between companies will mainly come from competition between their IT systems, and the ability to respond quickly to business requirements will be the key to winning.WeChat Official AccountKubernetesDockerPaaSOperationsConfiguration ManagementVirtual MachinesLearningCloud Nativehttps://www.chenshaowen.com/en/blog/how-to-build-istio-under-centos.htmlThu, 08 Apr 2021 00:00:00 +0000Thu, 08 Apr 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-build-istio-under-centos.htmlThis article mainly describes how to build the https://github.com/istio/istio project on CentOS 8.2 using local tools. Another approach is to use the BUILD_WITH_CONTAINER parameter to build inside a container, which depends only on make and docker. 1. Install and Upgrade Ruby >= 2.6 Install Ruby 1 yum install -y ruby Check the Ruby version 1 2 3 ruby -v ruby 2.WeChat Official AccountCentOSIstioKubernetesOperationsLearningService MeshCompilationOpen Sourcehttps://www.chenshaowen.com/en/blog/the-road-to-open-source-for-tob.htmlTue, 30 Mar 2021 00:00:00 +0000Tue, 30 Mar 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-road-to-open-source-for-tob.html1. An Open-Source-Centric Business Model One of the charms of open source is its inclusiveness. It accepts people with all kinds of intentions — the engineer obsessed with technology, the merchant with ulterior motives, the volunteer passionate about public welfare, even the casual bystander — all of them can collide and mingle here, forming a force.WeChat Official AccountOpen SourceToBThoughtsKubeSphereKuberneteshttps://www.chenshaowen.com/en/blog/install-zsh-in-centos-7.htmlMon, 29 Mar 2021 00:00:00 +0000Mon, 29 Mar 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/install-zsh-in-centos-7.html1. Install zsh 1 sudo yum install -y zsh 2. Install oh-my-zsh 1 sh -c "$(curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh)" It is recommended to set Zsh as the default shell. 3. Install Plugins 3.1 Install autojump 1 sudo yum install -y epel-release 1 sudo yum install -y autojump-zsh 3.2 Install zsh-autosuggestions 1 git clone git://github.WeChat Official AccountCentOSLinuxEnvironmentZshOperationshttps://www.chenshaowen.com/en/blog/install-homebrew-in-centos-7.htmlSun, 28 Mar 2021 00:00:00 +0000Sun, 28 Mar 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/install-homebrew-in-centos-7.htmlWhen installing packages on CentOS you usually have to hunt down all sorts of repositories; with Homebrew that hassle goes away. This article uses CentOS 7.6 — other system versions may differ. 1. Create a Non-root User Homebrew refuses to run as root, so a new user has to be created.WeChat Official AccountCentOSEnvironmentHomebrewOperationsPackage ManagementLearninghttps://www.chenshaowen.com/en/blog/devops-platform-under-cloud-native.html.htmlWed, 24 Mar 2021 00:00:00 +0000Wed, 24 Mar 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/devops-platform-under-cloud-native.html.html1. What Is Cloud Native Cloud native is a rapidly evolving field. In 2013, Pivotal proposed the concept of cloud native and has continued to interpret it. Below is Pivotal’s description of the characteristics of cloud-native architecture: 2015: 12-Factor, microservices-oriented, anti-fragile 2017: Observability, modularity, replaceability, disposability 2019: DevOps, continuous delivery, microservices, containers Many people’s first encounter with cloud native may have been through the CNCF.WeChat Official AccountDevOpsThoughtsCloud NativeOperationsPlatformCICDKuberneteshttps://www.chenshaowen.com/en/blog/book/musings-on-linux-and-open-source.htmlTue, 16 Mar 2021 10:00:00 +0000Tue, 16 Mar 2021 10:00:00 +0000https://www.chenshaowen.com/en/blog/book/musings-on-linux-and-open-source.htmlAuthor: Eric S. Raymond (US) Publisher: China Machine Press Original title: The Cathedral & the Bazaar: Musings on Linux and Open Source by an Accidental Revolutionary Translator: Wei Jianfan Publication Year: 2014-5 ISBN: 9787111452478 PDF: View Notes: The cathedral metaphor stands for a closed development model in which a handful of experts create the work and do not release it until it is finished; the bazaar metaphor stands for an agile development model of collective effort and small, rapid steps.WeChat Official AccountBooksOpen SourceCommunityThoughtshttps://www.chenshaowen.com/en/blog/top-to-bottom-vs-bottom-to-top.htmlTue, 16 Mar 2021 00:00:00 +0000Tue, 16 Mar 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/top-to-bottom-vs-bottom-to-top.htmlIn ToB Thinking at Large and Small Companies, I analyzed the ToB strategy of large companies in some detail; this post supplements the thinking on small companies’ ToB strategy from another angle. 1. What Is a ToB Service A commercial company usually maintains at least one value chain and makes its profit from it.WeChat Official AccountToBThoughtsBusiness ModelManagementTeamshttps://www.chenshaowen.com/en/blog/lighthouse-of-devops-tool-chain.htmlFri, 05 Mar 2021 00:00:00 +0000Fri, 05 Mar 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/lighthouse-of-devops-tool-chain.htmlThis post introduces Lighthouse, a ChatOps tool; the content comes mainly from the official documentation. The Kubernetes community uses Prow to drive its collaboration on GitHub, but Prow does not work for other repositories. Lighthouse is broadly applicable to more types of Git repositories. 1. What Is Lighthouse Lighthouse is a lightweight ChatOps tool built on webhooks.WeChat Official AccountTranslationDevOpsProwKubernetesCICDLighthousehttps://www.chenshaowen.com/en/blog/using-podman-to-build-images-under-kubernetes-and-jenkins.htmlThu, 25 Feb 2021 00:00:00 +0000Thu, 25 Feb 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-podman-to-build-images-under-kubernetes-and-jenkins.htmlIn principle, Jenkins can use Podman to build images in any Kubernetes cluster; this article uses Containerd as the example. 1. Dropping Docker Brings New Challenges to CICD In CICD scenarios, we often need to build and push images within a pipeline. In an earlier document, Creating Jenkins Slaves Dynamically on Kubernetes, I described how mounting the /var/run/docker.WeChat Official AccountJenkinsDevOpsCICDDockerKubernetesPodmanhttps://www.chenshaowen.com/en/blog/tekton-hub-is-live.htmlWed, 24 Feb 2021 00:00:00 +0000Wed, 24 Feb 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/tekton-hub-is-live.htmlThe original was written by Puneet Punamiya from Red Hat (a CDF member). Tekton is an open source project under the Continuous Delivery Foundation (CDF). It provides a framework for building CI/CD systems in a cloud native way. Put simply, people can define an entire CI/CD pipeline as Kubernetes resources.WeChat Official AccountTranslationTektonKubernetesCICDDevOpshttps://www.chenshaowen.com/en/blog/a-sample-intro-to-cloud-native-buildpacks.htmlFri, 05 Feb 2021 00:00:00 +0000Fri, 05 Feb 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/a-sample-intro-to-cloud-native-buildpacks.html1. Buildpacks, an Old Tree Blooms Anew The Buildpacks project was first started by Heroku in 2011 and was widely adopted by PaaS platforms such as Cloud Foundry. In an earlier document, Buildpack for PaaS Deployment, I demonstrated how to deploy a Django application to Heroku. The shortcoming of Buildpacks was that the artifact it produced was in Droplet format, which could not be adapted directly to container platforms.WeChat Official AccountDevOpsBuildpackContainersImageshttps://www.chenshaowen.com/en/blog/argocd-of-devops-tool-chain.htmlThu, 04 Feb 2021 00:00:00 +0000Thu, 04 Feb 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/argocd-of-devops-tool-chain.html1. What Problem Argo CD Solves 1.1 Starting from GitOps GitOps originated from a blog post published by Weaveworks in 2017: GitOps - Operations by Pull Request. In it, Alexis introduced a deployment approach that treats Git as the single source of truth. In GitOps practice, we need to define software infrastructure in a Git repository for management.WeChat Official AccountDevOpsArgoCDKubernetesCICDGitOpsOperationsLearningDeploymentBest Practiceshttps://www.chenshaowen.com/en/blog/cli-banner-tool-pyfiglet.htmlWed, 03 Feb 2021 22:00:00 +0000Wed, 03 Feb 2021 22:00:00 +0000https://www.chenshaowen.com/en/blog/cli-banner-tool-pyfiglet.html1. What Is pyfiglet pyfiglet is an ASCII art text generation tool implemented in Python. It can generate graphics like the following from a string: 1 2 3 4 5 6 _ _ _ _ _ _ | |__ ___| | | ___ __ _____ _ __| | __| | | | '_ \ / _ \ | |/ _ \ \ \ /\ / / _ \| '__| |/ _` | | | | | | __/ | | (_) | \ V V / (_) | | | | (_| |_| |_| |_|\___|_|_|\___( ) \_/\_/ \___/|_| |_|\__,_(_) |/ When writing CLI tools, you can use pyfiglet to generate a Banner to display on the command line.WeChat Official AccountPythonCommandToolsR&Dhttps://www.chenshaowen.com/en/blog/local-is-fine-but-error-in-github-actions.htmlTue, 02 Feb 2021 00:00:00 +0000Tue, 02 Feb 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/local-is-fine-but-error-in-github-actions.html1. A Confusing Problem 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 name: Go on: [push, pull_request] jobs: build: name: CI runs-on: ubuntu-latest steps: - name: Set up Go 1.13 uses: actions/setup-go@v1 with: go-version: 1.13 - name: Check out code into the Go module directory uses: actions/checkout@v2 - name: Check pr is properly formatted run: diff -u <(echo -n) <(gofmt -d .WeChat Official AccountGitHubActionsCICDTroubleshootingOperationsGoLearninghttps://www.chenshaowen.com/en/blog/10-tips-of-github-action.htmlSun, 31 Jan 2021 00:00:00 +0000Sun, 31 Jan 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/10-tips-of-github-action.htmlIf you already use GitHub Actions, this article will show you more interesting and useful ways to use it. By the time I finished writing, I had added several more workflows to my repositories. 1. Passing Parameters When a Workflow Runs When a workflow runs, you can allow parameters to be entered on the GitHub Actions page to control the execution logic.WeChat Official AccountGitHubGitHub ActionsCICDDevOpsR&Dhttps://www.chenshaowen.com/en/blog/attack-vectors-under-kubernetes.htmlFri, 22 Jan 2021 00:00:00 +0000Fri, 22 Jan 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/attack-vectors-under-kubernetes.htmlThis document is mainly meant to demonstrate the dangers of Docker privileged mode, so please proceed with caution. Users without CLI access can copy the example YAML and directly create cluster workloads such as Pod, Job, and DaemonSet to carry out the operations. 1. Directly Deleting All Resources If you can log in to the machine, pack your things, and run the command:WeChat Official AccountKubernetesSecurityAttack VectorsOperationsDockerTroubleshootinghttps://www.chenshaowen.com/en/blog/free-kubernetes-env-using-github-actions.htmlWed, 20 Jan 2021 01:00:00 +0000Wed, 20 Jan 2021 01:00:00 +0000https://www.chenshaowen.com/en/blog/free-kubernetes-env-using-github-actions.html1. Log in to Ngrok and get an Authtoken 1.1 Log in to the Ngrok website and get an Authtoken Visit the Ngrok website, https://dashboard.ngrok.com/ , and log in with a GitHub or Google account. Go to the Authentication page and find your Authtoken, as shown below: 1.2 Configure Secrets under the GitHub project On the project’s Settings page, add a new Secret, as shown below:WeChat Official AccountGitHubActionsDevOpsCICDdemohttps://www.chenshaowen.com/en/blog/a-debugger-for-actions-v2.htmlWed, 20 Jan 2021 00:00:00 +0000Wed, 20 Jan 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/a-debugger-for-actions-v2.html1. What Is Ngrok Ngrok is an intranet penetration tool that can publish services on the internal network to the public internet. The figure below illustrates its function well: Compared with a similar tool like Frp (which requires running both a Server and a Client), Ngrok turns intranet penetration intoWeChat Official AccountGitHubGitHub ActionsDevOpsCICDR&Dhttps://www.chenshaowen.com/en/blog/a-debugger-for-actions.htmlMon, 18 Jan 2021 00:00:00 +0000Mon, 18 Jan 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/a-debugger-for-actions.html1. Debugging Until You Want to Quit GitHub Actions is the continuous integration service GitHub launched in October 2018. For open-source projects it offers unlimited free build time, and it supports Linux, MacOs, and Windows systems, which makes it very appealing. But a recent experience changed my view. I submitted a commit, improvement: build and ci, to a colleague’s repository, to improve its continuous build setup.WeChat Official AccountGitHubActionsDevOpsCICDdemohttps://www.chenshaowen.com/en/blog/operate-host-in-container-and-debug-container-on-host.htmlMon, 11 Jan 2021 00:00:00 +0000Mon, 11 Jan 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/operate-host-in-container-and-debug-container-on-host.html1. A Strange Requirement My boss had a strange requirement: use a single kubeconfig file to fetch all kinds of host status information, such as the process list and process states. My first reaction was that he must not understand containers — how could anyone use them that way, and if you were going to use them that way, why use containers at all?WeChat Official AccountContainersSecurityKubernetesDockerOperationshttps://www.chenshaowen.com/en/blog/writing-and-coding-follow-the-same-pattern.htmlSat, 09 Jan 2021 00:00:00 +0000Sat, 09 Jan 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/writing-and-coding-follow-the-same-pattern.html1. Writing Matters as Much as Writing Code From a behavioral standpoint, writing and coding are both typing; the only difference is that writing addresses people while coding addresses machines. Coding exists to control the state of a machine and make it transition according to preset instructions; writing exists toWeChat Official AccountThoughtsTeamsWritingR&DGrowthSharinghttps://www.chenshaowen.com/en/blog/using-github-actions-to-build-a-chatops-in-three-steps.htmlWed, 06 Jan 2021 00:00:00 +0000Wed, 06 Jan 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-github-actions-to-build-a-chatops-in-three-steps.htmlIn the previous post, Using ChatOps to Improve the R&D Process, providing preview links for Pull Requests through ChatOps improved our agility. This post describes how to implement this feature quickly. 1. Step One: Configure a Trigger 1.1 Choose a Trigger GitHub has three kinds of Workflow triggers: scheduled, manual, and automatic.WeChat Official AccountGitHubGitHub ActionsR&DDevOpsChatOpshttps://www.chenshaowen.com/en/blog/using-chatops-to-improve-the-r-and-d-process.htmlWed, 06 Jan 2021 00:00:00 +0000Wed, 06 Jan 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-chatops-to-improve-the-r-and-d-process.html1. What Is ChatOps GitOps, ChatOps, AIOps, and the like (hereafter NewOps) are emerging operations philosophies that have appeared in recent years. NewOps separates Ops out of its chaotic state into two parts: one facing users, trending toward being more human, auditable, and traceable; the other facing infrastructure, trending toward being more programmatic, automated, and intelligent.WeChat Official AccountR&DDevOpsChatOpsOperationsAutomationLearningCollaborationProcesshttps://www.chenshaowen.com/en/blog/all-src-list.htmlFri, 01 Jan 2021 00:00:00 +0000Fri, 01 Jan 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/all-src-list.html1. Industry Standards [TPSA19-22] SRC Industry Security Testing Standard: https://security.tencent.com/announcement/msg/180 2. SRC List No. Launch Time SRC Name 01 2012 TSRC (Tencent) 02 2013 ZSRC (Zhubajie) 03 2013 NSRC (NetEase) 04 2013 KSRC (Kingsoft) 05 2013 JSRC (JD) 06 2013 BSRC (Baidu) 07 2013 ASRC (Alibaba) 08 2013 360SRC (360) 09 2014 YSCR (EZVIZ) 10 2014 CSRC (Ctrip) 11 2014 MISRC (Xiaomi) 12 2014 SSRC (Sina) 13 2014 YSRC (JOYY) 14 2014 SGSRC (Sogou) 15 2014 QSRC (Qunar) 16 2015 VSRC (Vipshop) 17 2015 SNSRC (Suning) 18 2015 PSRC (Ping An) 19 2015 LYSRC (Tongcheng-Elong) 20 2016 SNSRC (Tuniu) 21 2016 JJSRC (Jingji World) 22 2016 DSRC (DiDi) 23 2016 100TALSRC (TAL Education) 24 2016 ESRC (Ele.WeChat Official AccountSecuritySRCIncident ResponseVulnerabilityhttps://www.chenshaowen.com/en/blog/using-s3-to-store-terraform-state.htmlSun, 27 Dec 2020 00:00:00 +0000Sun, 27 Dec 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-s3-to-store-terraform-state.html1. How Terraform Manages Resource State After running terraform init, Terraform downloads the dependent plugins into a local plugins directory. 1 2 3 4 5 6 7 8 9 10 11 12 13 tree -aL 5 . |-- myresource.tf |-- .terraform | `-- plugins | |-- registry.terraform.io | | |-- hashicorp | | | `-- null | | `-- shaowenchen | | `-- qingcloud | `-- selections.WeChat Official AccountS3TerraformOperationsIaCStoragehttps://www.chenshaowen.com/en/blog/the-summary-of-devops-workshop.htmlThu, 24 Dec 2020 00:00:00 +0000Thu, 24 Dec 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-summary-of-devops-workshop.html1. What Is a Workshop A Workshop is a learning format centered on an experienced speaker in the field, guiding members to discuss a topic together through activities, discussion, talks, and other means. Compared with traditional formats, a Workshop has the following characteristics: More targeted A bit like a seminar, it can concentrate and fully discuss one class of problem.WeChat Official AccountSummaryThoughtsDevOpsWorkshopOperationshttps://www.chenshaowen.com/en/blog/make-decisions-through-data.htmlFri, 18 Dec 2020 00:00:00 +0000Fri, 18 Dec 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/make-decisions-through-data.htmlFor more techniques, keep an eye on the updates to this post. 1. Operations Is No Longer Guesswork Everything is interconnected, and the internet’s model of the physical world is becoming more and more accurate. Our geographic location, our movement trajectories, and our operations on electronic devices are all being recorded.WeChat Official AccountDataDataficationThoughtsStrategyBusinessOperationsDecision MakingData Analysishttps://www.chenshaowen.com/en/blog/using-terraform-and-github-actions-to-test-iac.htmlSun, 13 Dec 2020 00:00:00 +0000Sun, 13 Dec 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-terraform-and-github-actions-to-test-iac.html1. Testing Is a Beacon at Sea The more complex and larger a project is, the more it demonstrates the value and importance of testing. Testing guarantees that the direction is right. Like a beacon appearing at sea while sailing, it can be used to check and correct the course, so the helmsman can stay aware of the situation at any moment and make adjustments.WeChat Official AccountTerraformGitHubGitHub ActionsCICDKubernetesOperationshttps://www.chenshaowen.com/en/blog/the-concept-of-tekton-a-big-game.htmlSat, 12 Dec 2020 00:00:00 +0000Sat, 12 Dec 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-concept-of-tekton-a-big-game.html1. On the Aesthetics of CICD Tools In the document A Software Product Is the Output of a Team’s Capability, I mentioned that a software product is the delivery vehicle for a solution, and that its quality depends on the team’s understanding of the core problem. Only with a deep understanding of the domain can the delivered product have a chance of being good.WeChat Official AccountDevOpsTektonKubernetesCICDThoughtshttps://www.chenshaowen.com/en/blog/the-daemon-less-tools-of-kaniko.htmlFri, 11 Dec 2020 08:00:00 +0000Fri, 11 Dec 2020 08:00:00 +0000https://www.chenshaowen.com/en/blog/the-daemon-less-tools-of-kaniko.html1. daemon-less Image Build Tools 1.1 What Is a daemon-less Image Build Tool In a CICD pipeline, image builds come up often, and the conventional approach is to build with Docker in Docker or Docker out of Docker. For details, see: How to Use Docker in Docker In fact, to avoid monopolies and advance the industry, a unified OCI image format specification based on the Docker image format was defined long ago.WeChat Official AccountKanikoDevOpsImagesCICDKubernetesOperationshttps://www.chenshaowen.com/en/blog/how-to-cross-the-limit-of-dockerhub.htmlWed, 09 Dec 2020 00:00:00 +0000Wed, 09 Dec 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-cross-the-limit-of-dockerhub.html1. DockerHub Limits In the end, there is no getting around this error: 1 Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit Starting November 2, 2020, DockerHub officially began limiting the pull rate for non-paying users:WeChat Official AccountDockerImagesDockerHubOperationsTroubleshootingRate LimitingProxyhttps://www.chenshaowen.com/en/blog/a-canvas-cover-devops-servicemesh-serverless.htmlSat, 05 Dec 2020 00:01:00 +0000Sat, 05 Dec 2020 00:01:00 +0000https://www.chenshaowen.com/en/blog/a-canvas-cover-devops-servicemesh-serverless.htmlMany cloud native open source components start with the service and add visualization later. Today I had a brainstorm and went the opposite way: first the visual interaction, then the underlying design. 1. One Canvas Covers Half of Cloud Native Users only need one canvas. Define the Workflow, and they can roam freely across all kinds of fancy Cloud Native applications.WeChat Official AccountCloud NativeDevOpsServiceMeshServerlessThoughtshttps://www.chenshaowen.com/en/blog/build-a-scalable-team.htmlSat, 05 Dec 2020 00:00:00 +0000Sat, 05 Dec 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/build-a-scalable-team.htmlFor engineers in the internet industry, what comes to mind most often is the Scalability of systems — for example, how to improve a system as traffic, compute, and storage grow, with all kinds of horizontal and vertical scaling options. Besides services, the Scalability of a team is also critical.WeChat Official AccountThoughtsR&DTeamsDevOpsManagementOrganizationGrowthScalingBest Practiceshttps://www.chenshaowen.com/en/blog/how-to-add-self-hosted-runners-to-github-action.htmlFri, 04 Dec 2020 00:00:00 +0000Fri, 04 Dec 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-add-self-hosted-runners-to-github-action.html1. What GitHub Actions Is In an earlier post I put the CI capabilities GitLab provides into practice — click here to see. The advantages of using GitLab are that it can be deployed privately, it allows an unlimited number of private repositories, its CI configuration is simple, and it can connect to self-hosted Runners.WeChat Official AccountGitHubGitHub ActionsCICDDevOpsToolsR&DOperationshttps://www.chenshaowen.com/en/blog/practical-algorithm-of-bloom-filter.htmlTue, 01 Dec 2020 00:00:00 +0000Tue, 01 Dec 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/practical-algorithm-of-bloom-filter.html1. What Is Bloom Filter A Bloom Filter is a data structure proposed by Bloom in 1970. It maps elements (x, y, z) through a series of functions into a binary vector (a 0101 sequence), and is used to quickly determine whether an element w is in a set. As shown below (from Wikipedia):WeChat Official AccountAlgorithmsR&DGoOptimizationBloom Filterhttps://www.chenshaowen.com/en/blog/capability-maturity-model-of-devops.htmlFri, 27 Nov 2020 00:00:00 +0000Fri, 27 Nov 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/capability-maturity-model-of-devops.htmlUnveiling the Capability Maturity Model of DevOps R&D Operations (DevOps) Solution Capability Rating Requirements - PDF Download Part 1: Overall Architecture - PDF Download Part 2: Agile Development Management Process - PDF Download Part 3: Continuous Delivery Process - PDF Download Part 4: Technical Operations - PDF Download Part 5: Application Design - PDF DownloadWeChat Official AccountDevOpsCICDStandardsModelshttps://www.chenshaowen.com/en/blog/how-to-run-the-pipeline-on-remote-osx.htmlSun, 22 Nov 2020 00:00:00 +0000Sun, 22 Nov 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-run-the-pipeline-on-remote-osx.htmlThis article also applies to connecting physical machines with ARM or MIPS architectures and FreeBSD or Windows systems. If Jenkins can reach the build machine, you can skip the Frp part. 1. The Problem In scenarios where Kubernetes is the infrastructure, when Jenkins runs a build pipeline it creates a separate Pod for each pipeline for building.WeChat Official AccountJenkinsmacOSCICDCIDevOpshttps://www.chenshaowen.com/en/blog/how-to-use-docker-in-docker.htmlSat, 21 Nov 2020 00:00:00 +0000Sat, 21 Nov 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-use-docker-in-docker.html1. Typical Use Cases In CI, there is usually a CI Engine responsible for parsing the pipeline and controlling the whole build process, while the actual build is delegated to an Agent. Jenkins and GitLab both work this way. As shown below, there are many kinds of Agents that connect to the CI Engine.WeChat Official AccountDockerKubernetesContainersCICICDOperationsLearningDinDBuildJenkinshttps://www.chenshaowen.com/en/blog/how-to-get-the-real-ip-of-client-in-kubernetes.htmlFri, 20 Nov 2020 17:22:45 +0000Fri, 20 Nov 2020 17:22:45 +0000https://www.chenshaowen.com/en/blog/how-to-get-the-real-ip-of-client-in-kubernetes.htmlKubernetes relies on the kube-proxy component to implement Service communication and load balancing. In this process, because SNAT is used to translate the source address, a service in a Pod cannot obtain the real client IP address. This article mainly answers the question of how a workload in a Kubernetes cluster can obtain the real IP address of the client.WeChat Official AccountKubernetesIPNetworkOperationshttps://www.chenshaowen.com/en/blog/the-core-of-open-source-is-community.htmlMon, 09 Nov 2020 00:00:00 +0000Mon, 09 Nov 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-core-of-open-source-is-community.html1. Open Source Is Growing More and More Popular The 2019 IDC North America Open Source Software Usage Survey showed that 71% of enterprises are using open source software, and 54% plan to expand their usage. The 2020 Red Hat Enterprise Open Source Survey showed that 95% of IT leaders consider enterprise open source critical to their enterprise infrastructure software strategy.WeChat Official AccountThoughtsOpen SourceCommunityOperationsCloud Nativehttps://www.chenshaowen.com/en/blog/whiteboard-jenkins-on-kubernetes.htmlMon, 19 Oct 2020 00:08:00 +0000Mon, 19 Oct 2020 00:08:00 +0000https://www.chenshaowen.com/en/blog/whiteboard-jenkins-on-kubernetes.htmlWeChat Official AccountJenkinsKubernetesWhiteboard SharingDevOpsOperationshttps://www.chenshaowen.com/en/blog/book/the-38-letters-of-rockefeller-to-children.htmlMon, 19 Oct 2020 00:00:00 +0000Mon, 19 Oct 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/book/the-38-letters-of-rockefeller-to-children.htmlAuthor: John D. Rockefeller (USA) Publisher: Jilin Publishing Group Co., Ltd. Published: 2012-1-1 ISBN: 9787546369785 Notes: The road to success is direct and bright, while the roads to failure are each different.WeChat Official AccountBooksLifeWorkChoiceAttitudehttps://www.chenshaowen.com/en/blog/what-is-whiteboard-sharing.htmlSat, 17 Oct 2020 00:00:00 +0000Sat, 17 Oct 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/what-is-whiteboard-sharing.html1. Past Attempts In the post Without Tags, No One Can Recall You, I argued that in a homogeneous group the individual becomes background noise, while tags become effective information. In the image below, it is hard for us to remember any particular person, yet very easy to find someone through the A, B, C, D, E tags.WeChat Official AccountThoughtsTeamsR&DWhiteboard SharingWhat Ishttps://www.chenshaowen.com/en/blog/book/the-guide-to-kubernetes-network.htmlFri, 16 Oct 2020 00:00:00 +0000Fri, 16 Oct 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/book/the-guide-to-kubernetes-network.htmlAuthor: Du Jun Publisher: Publishing House of Electronics Industry Published: 2019-10 ISBN: 9787121373398 Notes: Networking is one of the harder parts of Kubernetes to master. Network failures directly affect the running workloads, and they are usually very urgent problems. Networking knowledge is also lower-level than application development, with many details that require long-term accumulation.WeChat Official AccountBooksKubernetesNetworkLearningOperationshttps://www.chenshaowen.com/en/blog/the-operation-guide-of-kubesphere-devops-3-0-pipeline.htmlWed, 14 Oct 2020 00:00:00 +0000Wed, 14 Oct 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-operation-guide-of-kubesphere-devops-3-0-pipeline.htmlThis post mainly covers how to operate and maintain a DevOps pipeline, and how to solve some common problems. The problems fall mainly into two categories: one is Kubernetes-related and fairly general in nature; the other is business-related and requires some understanding of the domain before you can solve problemsWeChat Official AccountKubeSphereKubernetesDevOpsJenkinsOperationsCICDhttps://www.chenshaowen.com/en/blog/the-development-guide-of-kubesphere-devops-3-0-pipeline.htmlTue, 13 Oct 2020 00:00:00 +0000Tue, 13 Oct 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-development-guide-of-kubesphere-devops-3-0-pipeline.htmlKubeSphere DevOps consists of two parts: S2I and Pipeline. In the community, OpenShift provides an application packaging tool called S2I; for details, see Building Cloud-Native Applications with S2I . KubeSphere turned it into a service, using a CRD managed by a separate Operator, and its functionality is fairly self-contained. In 3.WeChat Official AccountKubeSphereKubernetesDevOpsJenkinsDevelopment EnvironmentR&DOperationshttps://www.chenshaowen.com/en/blog/great-changes-and-opportunities-in-the-next-decade.htmlFri, 18 Sep 2020 00:00:00 +0000Fri, 18 Sep 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/great-changes-and-opportunities-in-the-next-decade.html1. Drawing the Heavy Sword, Challenging the World Since 1840, China endured more than a century of humiliation; yet starting in 1978, it took only a little over forty years to turn from a poor and weak great power into a wealthy and strong one. This is a manifestation of a nation’s resilience and wisdom.WeChat Official AccountChangeEconomyThoughtsIndustryGrowthMacrohttps://www.chenshaowen.com/en/blog/stress-testing-plan-of-kubernetes-platform-management-software.htmlThu, 17 Sep 2020 00:00:00 +0000Thu, 17 Sep 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/stress-testing-plan-of-kubernetes-platform-management-software.htmlKubernetes platform management software runs on Kubernetes and is used to manage the resource objects running on Kubernetes. 1. Testing Approach Test the management capability of the platform software under a given load and a given cluster scale, not the management capability of Kubernetes. The platform software’s management capability is mainly reflected in the ability to create, read, update, and delete workloads and PVs through the UI, and to view the monitoring and logs of workloads directly in the UI.WeChat Official AccountKubernetesStress TestingPlanEtcdLoadhttps://www.chenshaowen.com/en/about.htmlWed, 16 Sep 2020 00:00:00 +0000Wed, 16 Sep 2020 00:00:00 +0000https://www.chenshaowen.com/en/about.htmlContact Email: mail(at)chenshaowen.com WeChat Follow my WeChat Official Account to get my WeChat ID. Free Consultation Please send an email, or leave a message after adding me on WeChat. Skip the small talk and describe your question directly. I generally reply to everything, but not always right away. Paid Consultation 200 RMB / 0.WeChat Official Accounthttps://www.chenshaowen.com/en/blog/deploy-static-server-by-docker-image.htmlWed, 16 Sep 2020 00:00:00 +0000Wed, 16 Sep 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/deploy-static-server-by-docker-image.html1. Why Deploy Independently with an Image Better SEO With GitHub Pages, deploying a static site is very convenient. Configure it once, and every commit deploys and updates automatically. Add jsDelivr and Cloudflare acceleration on top, and among free options it is already production-ready. Unfortunately, Baidu’s search engine indexes GitHub Pages sites very slowly, or not at all.WeChat Official AccountHexoDockerImagesBlogOperationshttps://www.chenshaowen.com/en/blog/github-container-registry.htmlSat, 05 Sep 2020 00:00:00 +0000Sat, 05 Sep 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/github-container-registry.html1. Github Container Registry On September 1, GitHub announced that Github Container Registry had entered public beta, offering a free, unlimited-capacity Docker image registry service during the testing period. No more worrying about docker.io wiping images out of the blue. How nice, how sweet! GitHub is using hosted code repositories as its entry point and gradually covering the entire R&D toolchain to build a one-stop DevOps platform.WeChat Official AccountGitHubDockerImagesServicesContainersOperationshttps://www.chenshaowen.com/en/blog/let-us-start-learning-go-6.htmlFri, 04 Sep 2020 00:00:00 +0000Fri, 04 Sep 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/let-us-start-learning-go-6.html1. Interface-Oriented Programming 1.1 Characteristics Interface-oriented programming emphasizes interaction between modules through interfaces. First, the caller specifies a set of method signatures; then, the callee implements that set of methods. What sets interface programming apart from other programming styles is that it focuses on methods, not attributes. In many programming scenarios, methods are defined around attributes.WeChat Official AccountGoLet's Learn GoInterfaceLearninghttps://www.chenshaowen.com/en/blog/let-us-start-learning-go-5.htmlThu, 03 Sep 2020 00:00:00 +0000Thu, 03 Sep 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/let-us-start-learning-go-5.html1. The Concurrency Model in Go 1.1 The Communication Model: CSP CSP stands for Communicating Sequential Process, a model for concurrent communication. A Process can use many Channels, and a Channel does not care who is using it — it is only responsible for sending and receiving data. In the Go community there is a very famous maxim: do not communicate by sharing memory; instead, share memory by communicating.WeChat Official AccountGoConcurrencyParallelismGoroutineChannelLearningLet's Learn Gohttps://www.chenshaowen.com/en/blog/what-is-functional-programming.htmlWed, 02 Sep 2020 00:00:00 +0000Wed, 02 Sep 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/what-is-functional-programming.html1. What a Programming Paradigm Is A programming paradigm is a class of typical programming conventions. On one hand, it provides engineers with a way to model entities, linking the physical world to code; on the other hand, it provides engineers with a way of thinking about code and programs.WeChat Official AccountFunctional ProgrammingGoProgramming ParadigmPythonWhat Ishttps://www.chenshaowen.com/en/blog/how-to-integrate-kata-in-kubernetes-cluster.htmlSun, 30 Aug 2020 00:00:00 +0000Sun, 30 Aug 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-integrate-kata-in-kubernetes-cluster.html1. What Problem Kata Solves Security and isolation are what make Kata Container stand out from Docker Container. Kata Container comes from the merger of the Intel Clear Containers and Hyper runV projects. Intel Clear Containers used Intel VT-x technology to provide containers through lightweight virtual machines, solving the security problem while delivering excellent performance.WeChat Official AccountKubernetesKataContainersSecurityOperationsVirtualizationhttps://www.chenshaowen.com/en/blog/curry-and-partial-function.htmlSat, 29 Aug 2020 00:00:00 +0000Sat, 29 Aug 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/curry-and-partial-function.html1. What Currying Is According to the Wikipedia entry, in computer science currying is the technique of converting a function that takes multiple arguments into a function that takes a single argument (the first argument of the original function), and returns a new function that accepts the remaining arguments and returns a result.WeChat Official AccountGoPythonCurryingFunctional ProgrammingPartial FunctionsR&Dhttps://www.chenshaowen.com/en/blog/how-to-use-terraform-to-provide-iac-platform.htmlSat, 22 Aug 2020 00:00:00 +0000Sat, 22 Aug 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-use-terraform-to-provide-iac-platform.html1. Terraform Vs Kubernetes Infrastructure as Code (Iac) is built on immutable infrastructure: it uses orchestration tools to turn infrastructure into text, so you can manage infrastructure the way you manage code. In 2018 I was working on SaaS development, deploying on the Kubernetes platform, and that was the year Terraform was very hot.WeChat Official AccountTerraformIacKubernetesDevOpsCICDR&Dhttps://www.chenshaowen.com/en/blog/the-best-era-of-cloud-native.htmlWed, 19 Aug 2020 00:00:00 +0000Wed, 19 Aug 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-best-era-of-cloud-native.html1. Little Money, Lots of Work, and Everyone Pushes You Around The operations department is a cost center. There is a phrase for it: cost optimization. When a CXO sees how low machine utilization is, they think about cutting machines — spend as little as possible — and that means more work for operations.WeChat Official AccountThoughtsCloud NativeOperationsKuberneteshttps://www.chenshaowen.com/en/blog/jenkins-x-is-not-jenkins-but-stack.htmlSun, 16 Aug 2020 00:00:00 +0000Sun, 16 Aug 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/jenkins-x-is-not-jenkins-but-stack.html1. Introduction to Jenkins X Relying on its massive plugin ecosystem, Jenkins currently holds most of the enterprise CICD engine market share. But in the cloud native era, Jenkins has also exposed many problems: a single point of service, disk storage, memory usage, and so on. Around Kubernetes, Jenkins X offers a DevOps approach better suited to the cloud native era.WeChat Official AccountTech StackJenkinsJenkins-XKubernetesDevOpsCICDOperationshttps://www.chenshaowen.com/en/blog/the-use-of-etcd-and-etcdctl.htmlTue, 11 Aug 2020 00:00:00 +0000Tue, 11 Aug 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-use-of-etcd-and-etcdctl.html1. Basic Introduction to Etcd Etcd is a distributed Key/Value storage system. Through distributed locks, leader election, and write barriers it achieves distributed coordination, offering highly available, persistent data storage and retrieval services. How it works Every Etcd node stores a complete copy of the data, and at any momentWeChat Official AccountEtcdEtcdctlKubernetesOperationsLearningDistributed StorageConfigurationClusterhttps://www.chenshaowen.com/en/blog/the-stress-test-about-kubernetes-dynamically-creates-jenkins-agent.htmlSun, 02 Aug 2020 00:00:00 +0000Sun, 02 Aug 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-stress-test-about-kubernetes-dynamically-creates-jenkins-agent.htmlIn the previous document, we took advantage of the elasticity Kubernetes provides to dynamically create Jenkins Slaves on Kubernetes. This document is mainly a stress test of Jenkins under large-scale builds. 1. Cluster Configuration 1.1 Kubernetes Version The version used here is v1.16.7 1 2 3 4 kubectl version Client Version: version.WeChat Official AccountJenkinsKubernetesDevOpsCICDTestingOperationshttps://www.chenshaowen.com/en/blog/some-summaries-of-the-interviewer.htmlSun, 26 Jul 2020 00:00:00 +0000Sun, 26 Jul 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/some-summaries-of-the-interviewer.htmlRecently quite a few teammates have left, and we are fairly short-staffed. I have interviewed nearly ten candidates one after another; the first-round pass rate is around 50%, and there is still no confirmed HC. 1. Preparation Collecting resumes. Posts, email announcements, WeChat Moments, and job boards are all sourcesWeChat Official AccountR&DInterviewTeamsManagementThoughtshttps://www.chenshaowen.com/en/blog/docker-image-under-multi-arch.htmlSun, 05 Jul 2020 00:00:00 +0000Sun, 05 Jul 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/docker-image-under-multi-arch.html1. Enabling Docker’s experimental Feature First, enable Docker’s experimental feature, so the commands below are available. Edit vim ~/.docker/config.json and add the following: 1 2 3 4 { "experimental": "enabled", "debug": true } Note that this is not the /etc/docker/daemon.json file, and Docker does not need to be restarted. 2. Docker Images Starting with Docker 1.WeChat Official AccountDockerImagesMulti-ArchmanifestOperationshttps://www.chenshaowen.com/en/blog/install-jdk-in-centos-7.htmlWed, 01 Jul 2020 00:00:00 +0000Wed, 01 Jul 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/install-jdk-in-centos-7.html1. Compilation Error: java.nio.file.NoSuchFileException When compiling a Jenkins plugin, the following error is reported: 1 2 3 4 mvn package Compilation failure [ERROR] java.nio.file.NoSuchFileException: /root/java/target/classes/META-INF/annotations/hudson.Extension It turns out that Maven uses JAVA_HOME rather than PATH, so it cannot find a Java runtime. Java 11 does not work either — only Java 1.WeChat Official AccountJavaEnvironmentCentOSR&Dhttps://www.chenshaowen.com/en/blog/book/the-essentials-of-devops.htmlTue, 23 Jun 2020 00:00:00 +0000Tue, 23 Jun 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/book/the-essentials-of-devops.htmlAuthor: (Russia) Oleg Skrynnik Publisher: Tsinghua University Press Publication year: 2020-05-01 ISBN: 9787302547143 Notes: As the title suggests, the book focuses on DevOps from a business perspective. From its origins to its foundations, from principles to practice, you can take something away at every level. DevOps rests on lean production (the Toyota production line) and agile development, with the value chain at its core.WeChat Official AccountBooksDevOpsCICDR&Dhttps://www.chenshaowen.com/en/blog/the-question-of-lightweight-checkout-in-jenkins.htmlMon, 22 Jun 2020 00:00:00 +0000Mon, 22 Jun 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-question-of-lightweight-checkout-in-jenkins.html1. What Problem We Hit A Jenkins build log threw an error: 1 2 3 4 5 6 Started by user admin Lightweight checkout support not available, falling back to full checkout. Checking out git https://github.com/shaowenchen/pipeline-test.git into /var/jenkins_home/workspace/abc@script to read Jenkinsfile ... ... Unable to access '.git/index.lock': File exists. Cause analysis:WeChat Official AccountJenkinsGitDevOpsTroubleshootinghttps://www.chenshaowen.com/en/blog/artifacts-and-cache-in-jenkins.htmlSun, 21 Jun 2020 00:00:00 +0000Sun, 21 Jun 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/artifacts-and-cache-in-jenkins.htmlIn a CICD flow, there are two main kinds of things that need to be saved: build artifacts and caches. Build artifacts are the final execution result, and caches exist to speed up the next build. This post mainly describes how to archive build artifacts and caches in Jenkins, along with a hands-on example backed by object storage.WeChat Official AccountJenkinsS3DevOpsCachehttps://www.chenshaowen.com/en/blog/how-to-develop-the-plugin-of-jenkins.htmlSat, 20 Jun 2020 00:00:00 +0000Sat, 20 Jun 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-develop-the-plugin-of-jenkins.htmlUsing Jenkins always means dealing with all kinds of plugins. To practice DevOps better, we should also be capable of developing plugins ourselves, so the whole process can converge inside Jenkins. 1. Jenkins Plugins 1.1 The Plugin Ecosystem Jenkins’ predecessor Hudson started in 2004, and 16 years on it is still a mainstream CI/CD engine.WeChat Official AccountJenkinsDevOpsCICDR&DPluginJavaLearninghttps://www.chenshaowen.com/en/blog/windows-node-to-dynamicly-provide-jenkins-agent-on-k8s.htmlSun, 14 Jun 2020 00:00:00 +0000Sun, 14 Jun 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/windows-node-to-dynamicly-provide-jenkins-agent-on-k8s.htmlBuilding on the previous two documents, Creating Jenkins Slaves Dynamically on Kubernetes and Adding Windows Nodes to Kubernetes to Provide Dynamic Jenkins Build Agents, this document mainly attempts to dynamically provide Windows build agents on Kubernetes. 1. Adding Pipelines For the Kubernetes and Jenkins integration part, refer to the two documents above; here we simply create two pipelines to test with.WeChat Official AccountJenkinsKubernetesWindowsDevOpshttps://www.chenshaowen.com/en/blog/the-strategy-for-rd-and-ops.htmlSun, 14 Jun 2020 00:00:00 +0000Sun, 14 Jun 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-strategy-for-rd-and-ops.html1. Thought Drives Behavior Thought drives behavior, and behavior produces results. How we treat people, handle things, and approach affairs all come from culture, and they do not leap forward just because technology advances rapidly. The Chinese nation, with its long history, has accumulated remarkably resilient intellectual wisdom. Dynasties rose and fell, regimes changed, and none of it could bring her down.WeChat Official AccountR&DThoughtsTeamsManagementOperationsCulturehttps://www.chenshaowen.com/en/blog/add-windows-node-for-k8s.htmlSat, 13 Jun 2020 00:00:00 +0000Sat, 13 Jun 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/add-windows-node-for-k8s.htmlHere a Windows node is used mainly as a Worker, while the Master control plane stays on Linux. 1. System Configuration 1.1 Kubernetes Control Plane Since version 1.14, Kubernetes has added production-grade support for Windows nodes. Because Microsoft’s official documentation mainly covers installing the flannel network plugin, it is recommended here that Kubernetes also use the flannel plugin.WeChat Official AccountKubernetesWindowsNodesInstallationOperationshttps://www.chenshaowen.com/en/blog/incomplete-data-using-base64-decoding-jwt-playload.htmlMon, 08 Jun 2020 00:00:00 +0000Mon, 08 Jun 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/incomplete-data-using-base64-decoding-jwt-playload.htmlWhen Base64-decoding a JWT, I found that the JSON data was incomplete. This article mainly introduces the relevant knowledge and solves this problem. 1. Introduction to JWT JWT passes authentication by setting Authorization: Bearer <token> in the Header. A JWT Token is a Base64-encoded string joined by dots, something likeWeChat Official AccountGoJWTEncodingBase64SecurityR&Dhttps://www.chenshaowen.com/en/blog/batch-command-of-kubernetes.htmlSun, 07 Jun 2020 00:00:00 +0000Sun, 07 Jun 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/batch-command-of-kubernetes.html1. Delete Pods in the Evicted State in Bulk 1 kubectl get pods --all-namespaces -o wide | grep Evicted | awk '{print $1,$2}' | xargs -L1 kubectl delete pod -n 2. Delete Pods in a Given State in a Given Namespace in Bulk Using field-selectors, you can delete Pods in a given state in a given namespace.WeChat Official AccountKuberneteskubectlOperationsTipshttps://www.chenshaowen.com/en/blog/why-should-we-use-remote-build.htmlThu, 04 Jun 2020 00:00:00 +0000Thu, 04 Jun 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/why-should-we-use-remote-build.html1. Raise the Level of Automation Remote builds usually need the help of some CI tool, such as GitHub Actions, Travis CI, and so on. In an intranet environment, you can use GitLab CI and add a CI node for automated builds — see my GitLab documentation. The benefits of automation go without saying: it standardizes processes, saves a great deal of time, and delivers clear returns.WeChat Official AccountThoughtsR&DBuildCICDDevOpsGitHub ActionsBest Practiceshttps://www.chenshaowen.com/en/blog/affinity-of-kubernetes-scheduler.htmlSat, 30 May 2020 00:00:00 +0000Sat, 30 May 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/affinity-of-kubernetes-scheduler.html1. The Scheduler in Kubernetes kube-scheduler is the component in Kubernetes that decides which Node a Pending Pod runs on; it is called the scheduler. Kubernetes ships with a large number of built-in scheduling policies and also provides some advanced scheduling policies (nodeAffinity, podAffinity, and so on) for users, whichWeChat Official AccountKubernetesSchedulingAffinityLearningOperationshttps://www.chenshaowen.com/en/blog/a-complete-git-submission-process.htmlTue, 19 May 2020 00:00:00 +0000Tue, 19 May 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/a-complete-git-submission-process.htmlThis is also a complete Git process for submitting a PR to an open source project. 1. Local Configuration Set the commit user information 1 2 git config --global user.name "username" git config --global user.email "user@email.com" GPG configuration Reference: GPG Verified Commits 2. Cloning the Code First fork the original repositoryWeChat Official AccountGitR&DOpen SourceLearninghttps://www.chenshaowen.com/en/blog/the-difference-of-tree-ways-of-merging-code-in-github.htmlMon, 18 May 2020 00:00:00 +0000Mon, 18 May 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-difference-of-tree-ways-of-merging-code-in-github.html1. Create a merge commit PR commit log: 1 2 commit c1 commit c2 Integration branch commit log: 1 2 3 commit PR #NUM commit c2 commit c1 Create a merge commit carries every commit in the PR over to the integration branch intact, and adds one PR commit entry on top.WeChat Official AccountR&DGitHubGitDevOpsCICDhttps://www.chenshaowen.com/en/blog/how-to-process-work-order.htmlMon, 18 May 2020 00:00:00 +0000Mon, 18 May 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-process-work-order.html1. When We’re on Ticket Duty, What Are We On Duty For? Most people have probably only ever filed a ticket, never worked one. And people who have worked tickets generally don’t file them lightly. A ticket is a service bundled free with the product a customer paid for, dedicated to solving every kind of usage problem.WeChat Official AccountTicketsR&DThoughtsTeamsOperationsCustomer SupportProductGrowthhttps://www.chenshaowen.com/en/blog/git-reference-not-a-tree-in-jenkins.htmlSun, 17 May 2020 00:00:00 +0000Sun, 17 May 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/git-reference-not-a-tree-in-jenkins.html1. Problem Description When a Webhook was configured to automatically trigger a Jenkins pipeline, the following error occurred: 1 2 3 hudson.plugins.git.GitException: Command "git checkout -f 23b446ea" returned status code 128: stdout: stderr: fatal: reference is not a tree: 23b446ea 2. How Git Manages Versions Git is a content-addressable filesystem. Git maintains a sha tree, and any historical node can be traced back to via its sha value.WeChat Official AccountTroubleshootingJenkinsDevOpsGithttps://www.chenshaowen.com/en/blog/how-to-configure-efficient-k8s-terminal.htmlSat, 16 May 2020 00:00:00 +0000Sat, 16 May 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-configure-efficient-k8s-terminal.htmlSharpening the axe never delays the woodcutting. No matter when, the time you spend on your toolchain is always worth it. 1. Autocompletion - kubectl OS X install command: 1 brew install bash-complete@2 It is not just for kubectl — it provides completion prompts for other command lines too.WeChat Official AccountKubernetesCommand LineToolskubectlOperationsEfficiencyLearninghttps://www.chenshaowen.com/en/blog/the-use-of-homebrew.htmlFri, 15 May 2020 00:00:00 +0000Fri, 15 May 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-use-of-homebrew.htmlHomebrew is the package manager for OS X, similar to yum on CentOS or apt-get on Ubuntu. From the OS X command line you can install applications with the brew and brew cask commands. 1. Install brew cask 1 brew install brew-cask-completion 2. The Difference Between brew and brew cask The difference between brew and brew cask lies in the type of application they favor.WeChat Official AccountHomebrewMacOSCommand LineToolsOperationshttps://www.chenshaowen.com/en/blog/the-advanced-usage-of-curl.htmlThu, 14 May 2020 00:00:00 +0000Thu, 14 May 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-advanced-usage-of-curl.html1. RESTful Requests 1 curl -X POST --data 'keyword=value' http://domain.com/mypath/ After -X you can also use DELETE, PUT, and so on. 2. Adding Headers 1 curl -H 'Content-Type:application/json' -H 'Authorization: bearer MyToken' http://domain.com/mypath/ 3. Basic Authentication 1 curl -u username:password http://domain.com/mypath/ 4. Download and Execute 1 curl -sSL http://domain.com/my.sh | bash 5.WeChat Official AccountcurlCommand LineTipsOperationsLearningHTTPAPIDebuggingNetworkhttps://www.chenshaowen.com/en/blog/surface-pen-lose-feeling.htmlMon, 11 May 2020 00:00:00 +0000Mon, 11 May 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/surface-pen-lose-feeling.htmlRecently, while using my Surface, I dug out the stylus and found it was out of battery. I swapped in a new AAAA battery, and then discovered the pen tip had stopped responding to touch. Many documents online recommend uninstalling Marvell AVASTAR Bluetooth and rebooting for this to take effect. But that method did not work for me; uninstalling Precise Touch Device is what actually worked.WeChat Official AccountTipsSurfaceHardwareTroubleshootingWindowsOperationsLearninghttps://www.chenshaowen.com/en/blog/10-common-software-architectural-patterns-in-a-nutshell.htmlMon, 04 May 2020 00:00:00 +0000Mon, 04 May 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/10-common-software-architectural-patterns-in-a-nutshell.htmlHave you ever thought about how to design a large enterprise system? Before starting any major software development, we first have to pick a suitable architecture. That architecture has to give us the functionality and the quality guarantees we need. So before applying these architectures to our design, we should understand the different architectural systems.WeChat Official AccountTranslationDesign PatternsArchitectureSoftwareR&Dhttps://www.chenshaowen.com/en/blog/how-to-set-proxy-for-kubernetes.htmlSun, 03 May 2020 00:00:00 +0000Sun, 03 May 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-set-proxy-for-kubernetes.htmlUsually, running a command of the form export http_proxy/https_proxy on the host is enough to set a Proxy. However, settings on the host do not take effect inside containers. Below are several configuration approaches. 1. Configure Docker’s proxy - Node level Configure this on the node that needs the Proxy.WeChat Official AccountDockerKubernetesProxyOperationsConfigurationNetworkLearninghttps://www.chenshaowen.com/en/blog/how-to-create-labels-by-pygithub.htmlFri, 01 May 2020 00:00:00 +0000Fri, 01 May 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-create-labels-by-pygithub.html1. About Labels In an earlier document, How to Create GitLab Labels Automatically with python-gitlab, I already explained that Labels can be used for simple project management. A team usually has more than one code repository. To reduce the cost of communication and learning, besides using a consistent toolchain, it should also reach a certain basic consensus.WeChat Official AccountGitHubProcessR&DPythonAutomationProject ManagementLearninghttps://www.chenshaowen.com/en/blog/the-product-is-the-output-of-teams-ability.htmlSun, 26 Apr 2020 00:00:00 +0000Sun, 26 Apr 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-product-is-the-output-of-teams-ability.html1. People Matter More and More Why is the valuation of the internet industry significantly higher than that of manufacturing? Among equally excellent companies in their respective industries, Tencent’s PE is as high as 30+, while Gree’s PE is only 10+. The internet industry is characterized by being asset-light: no expensive complex equipment, no need for large production plants.WeChat Official AccountThoughtsSoftwareProductTeamsManagementR&DOrganizationCapabilityhttps://www.chenshaowen.com/en/blog/helm-2-vs-helm-3.htmlSat, 25 Apr 2020 00:00:00 +0000Sat, 25 Apr 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/helm-2-vs-helm-3.htmlHelm 3 is finally released. We can say goodbye to Tiller, but Helm 3’s changes do not stop there. Let’s continue exploring the other changes. 1. Saying Goodbye to Tiller Removing Tiller in Helm 3 was a good decision. But to understand why it is good, we first need to understand the background that produced Tiller.WeChat Official AccountTranslationHelmKubernetesComparisonOperationshttps://www.chenshaowen.com/en/blog/book/the-iot-land.htmlSat, 25 Apr 2020 00:00:00 +0000Sat, 25 Apr 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/book/the-iot-land.htmlSubtitle: IoT Profit-Making Models and All-New Practices Author: (Japan) Kenichi Ohmae Translator: Zhu Yuewei Publisher: Beijing Times Chinese Press Co., Ltd. Publication Year: 2019-2-1 ISBN: 9787569932171 Notes: The author is a very famous management scholar; I have previously read another book of his, The Art of Thinking. This book mainly explains the importance of IoT and how to put IoT into practice to generate benefits.WeChat Official AccountBooksIoTTechnologyBusinesshttps://www.chenshaowen.com/en/blog/dns-server-in-kubernetes.htmlFri, 24 Apr 2020 00:00:00 +0000Fri, 24 Apr 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/dns-server-in-kubernetes.html1. About DNS 1.1 What the DNS Service Is For DNS provides a mapping service from domain names to IPs. For example, you type https://www.chenshaowen.com into a browser to open a page, but the data link communicates over IP and cannot recognize www.chenshaowen.com. This is where a DNS lookup comes in: the input is www.WeChat Official AccountKubernetesDNSDomain NamesNetworkService Discoveryhttps://www.chenshaowen.com/en/blog/configure-helm-mirror-in-china.htmlTue, 21 Apr 2020 00:00:00 +0000Tue, 21 Apr 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/configure-helm-mirror-in-china.htmlThe official Helm repo https://charts.helm.sh/stable is unreachable from some machines in China, so you need to configure a mirror. 1. Official Repo 1 helm repo add stable https://charts.helm.sh/stable 2. Git Pages Mirror 1 helm repo add stable https://burdenbear.github.io/kube-charts-mirror/ See kube-charts-mirror for a reference on building a self-hosted, self-controlled mirror.WeChat Official AccountHelmKubernetesChinaOperationsMirrorsNetworkConfigurationhttps://www.chenshaowen.com/en/blog/support-https-access-harbor-using-self-signed-cert.htmlSat, 18 Apr 2020 00:00:00 +0000Sat, 18 Apr 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/support-https-access-harbor-using-self-signed-cert.htmlIn an earlier post, Installing Harbor with Helm, I described in detail how to install Ingress and Harbor and finally push an image successfully. There the domain was publicly reachable and the certificate was issued by a certificate authority. In an intranet environment, however, we need to use an internal domain name for access.WeChat Official AccountHarborKubernetesHttpsImagesCertificatesOperationshttps://www.chenshaowen.com/en/blog/prow-of-devops-tool-chain.htmlFri, 17 Apr 2020 13:17:44 +0000Fri, 17 Apr 2020 13:17:44 +0000https://www.chenshaowen.com/en/blog/prow-of-devops-tool-chain.html1. About Prow In the GitHub repositories of well-known projects such as Kubernetes and Istio, we often see an xxx-bot user adding labels to issues and merging PRs. That bot account is driven by Prow. Prow is a project of the Kubernetes Testing Special Interest Group and is currently part of kubernetes/test-infra.WeChat Official AccountDevOpsProwKubernetesCICDhttps://www.chenshaowen.com/en/blog/how-to-install-kubeflow-using-kfctl.htmlSun, 12 Apr 2020 00:00:00 +0000Sun, 12 Apr 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-install-kubeflow-using-kfctl.html1. Installing the Base Environment Install Kubernetes Reference link: Install a Kubernetes Cluster Using Kubeadm . It is worth noting that Kubeflow is not compatible with every version of Kubernetes, see system-requirements. 1 2 3 4 kubectl version Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.12", GitCommit:"e2a822d9f3c2fdb5c9bfbe64313cf9f657f0a725", GitTreeState:"clean", BuildDate:"2020-05-06T05:17:59Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.WeChat Official AccountKubeflowKubernetesMachine LearningAIOperationsDeploymentLearninghttps://www.chenshaowen.com/en/blog/can-not-download-packages-in-vs-code.htmlSat, 11 Apr 2020 00:00:00 +0000Sat, 11 Apr 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/can-not-download-packages-in-vs-code.htmlAfter installing the Go plugin in VS Code, when you open a Go project the editor prompts you to install the toolchain. But golang.org is not directly accessible, so we need to download the packages manually into the GOPATH directory and install them there. Create the package directory 1 2 cd $GOPATH mkdir -p src/golang.WeChat Official AccountGoVSCodePluginR&DEnvironmenthttps://www.chenshaowen.com/en/blog/engineers-need-more-sense-of-service.htmlMon, 06 Apr 2020 00:00:00 +0000Mon, 06 Apr 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/engineers-need-more-sense-of-service.html1. About the Service Industry The primary sector, represented by agriculture, solves the problem of survival; the secondary sector, represented by industry, solves the problem of efficiency; the tertiary sector, represented by services, solves other people’s problems. Unless some factor puts people’s physical existence under threat, the service industry has extremely broad room to grow.WeChat Official AccountThoughtsCultureTeamsGrowthhttps://www.chenshaowen.com/en/blog/docker-skills-you-do-not-notice.htmlWed, 18 Mar 2020 00:00:00 +0000Wed, 18 Mar 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/docker-skills-you-do-not-notice.html1. Multi-Stage Builds Compiling a project requires a set of specific tools, but those tools are not needed at runtime. To reduce the image size, you can build in stages. Build in the first stage, then carry the compiled output into the next stage to produce a smaller image. 1 2 3 4 5 6 7 8 9 10 11 FROM golang:1.WeChat Official AccountDockerTipsOperationsR&DImagesLearningBest Practiceshttps://www.chenshaowen.com/en/blog/run-tensorflow-on-docker.htmlThu, 05 Mar 2020 00:00:00 +0000Thu, 05 Mar 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/run-tensorflow-on-docker.htmlI wrote about this in an earlier post, How to Install a GPU Driver on CentOS. This one looks at how to run TensorFlow with Docker. 1. Check Which TensorFlow Version the Current CPU Supports On a CPU that does not support AVX instructions, running TensorFlow > 1.15 fails with Illegal instruction (core dumped).WeChat Official AccountGPUTensorflowDockerAIMachine LearningOperationsLearninghttps://www.chenshaowen.com/en/blog/book/on-china.htmlTue, 25 Feb 2020 00:00:00 +0000Tue, 25 Feb 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/book/on-china.htmlOriginal title: On China Author: Henry Kissinger (USA) Translators: Hu Liping / Lin Hua / Yang Yunqin / Zhu Jingwen Publisher: CITIC Press Published: 2012-10 ISBN: 9787508635583 Notes: The author is a renowned American scholar and statesman, and he offers a different perspective on the book, documenting China’s modern, contemporary, and present-day foreign policy and decision-making mechanisms.WeChat Official AccountBooksHistoryPolicyhttps://www.chenshaowen.com/en/blog/what-is-left-shift.htmlSun, 23 Feb 2020 00:00:00 +0000Sun, 23 Feb 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/what-is-left-shift.html1. What Left Shift Is It should be noted that “left” here is not the political left, standing for conservatism or radicalism, but the left that describes a process — moving things earlier in time. I first came across “left shift” while researching automated testing. The idea is appealing, and putting it into practice is simple.WeChat Official AccountThoughtsTestingAgileR&DDevOpsLearningProcessQualityhttps://www.chenshaowen.com/en/blog/book/a-little-monk-in-one-zen.htmlThu, 20 Feb 2020 00:00:00 +0000Thu, 20 Feb 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/book/a-little-monk-in-one-zen.htmlAuthor: Yi Chan Xiao He Shang Publisher: Jiangsu Phoenix Literature and Art Publishing House Publication Year: 2017-09 ISBN: 9787559411013 Notes: With its clean, simple art style, A Little Monk in One Zen leads readers into a world full of goodwill and wisdom. If you feel trapped or troubled by life’s trivialities, this book is sure to give you inspiration.WeChat Official AccountBooksComicsLifeReadingHealinghttps://www.chenshaowen.com/en/blog/how-to-clear-commit-history-of-git.htmlThu, 20 Feb 2020 00:00:00 +0000Thu, 20 Feb 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-clear-commit-history-of-git.html Here we take clearing the entire commit history of the main branch as the example. Switch to the main branch 1 git checkout main Create a clean branch 1 git checkout --orphan new_main Commit all files 1 2 git add -A git commit -m "msg" Delete the main branch 1 git branch -D main Rename the new branch to main 1 git branch -m main Force push to the remote repository 1 git push -f origin mainWeChat Official AccountGitTipsOperationsR&Dhttps://www.chenshaowen.com/en/blog/book/the-wind-through-kitchen.htmlWed, 19 Feb 2020 00:00:00 +0000Wed, 19 Feb 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/book/the-wind-through-kitchen.htmlAuthor: Shi Jia Fan Publisher: Beijing United Publishing Company Publication Year: 2017-3-2 ISBN: 9787550294509 Notes: In the kitchen where the wind blows through, there is a breath of the human world. Dish after delicious dish, enough to make your mouth water as you read. The author must surely be full of love for life.WeChat Official AccountBooksKitchenLifeFoodhttps://www.chenshaowen.com/en/blog/book/roses-of-time.htmlTue, 18 Feb 2020 00:00:00 +0000Tue, 18 Feb 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/book/roses-of-time.htmlAuthor: Dan Bin Subtitle: Dan Bin’s Investment Notes Publisher: CITIC Press Publication Year: 2018-06 ISBN: 9787508688640 Notes: This book mainly collects the author’s thoughts and analyses on investment from the 200X years. Bei Dao’s The Rose of Time accompanied the author through his hardest times, so he borrowed the same name.WeChat Official AccountBooksFinanceInvestmentReadingValue Investinghttps://www.chenshaowen.com/en/blog/how-to-install-gpu-driver-in-centos.htmlSat, 15 Feb 2020 00:00:00 +0000Sat, 15 Feb 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-install-gpu-driver-in-centos.htmlUsing CentOS 7.7 and a Tesla P100 GPU as an example. 1. Preparing the Base Environment Install the lspci command 1 yum install -y pciutils Check whether the GPU supports CUDA 1 2 3 lspci | grep -i nvidia 00:09.0 3D controller: NVIDIA Corporation GP100GL [Tesla P100 PCIe 12GB] (rev a1) List of CUDA-capable GPUs: https://developer.WeChat Official AccountDockerGPUCentOSOperationsDrivershttps://www.chenshaowen.com/en/blog/backup-kubernetes-cluster-using-velero.htmlWed, 25 Dec 2019 00:00:00 +0000Wed, 25 Dec 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/backup-kubernetes-cluster-using-velero.html1. Introduction to Velero Velero is an open-source Kubernetes cluster backup and migration tool from the heptio team (acquired by VMWare). Velero stores cluster resources in object storage. It supports AWS, Azure, and GCP object storage by default, is compatible with the S3 protocol, and can be extended to other platforms such as Aliyun OSS through plugins.WeChat Official AccountKubernetesVeleroBackupOperationsLearningDisaster RecoveryMigrationClusterhttps://www.chenshaowen.com/en/blog/kubernetes-cheat-sheet.htmlTue, 24 Dec 2019 01:00:00 +0000Tue, 24 Dec 2019 01:00:00 +0000https://www.chenshaowen.com/en/blog/kubernetes-cheat-sheet.htmlWeChat Official AccountKuberneteskubectlCheat SheetOperationshttps://www.chenshaowen.com/en/blog/how-to-compile-a-java-project.htmlFri, 20 Dec 2019 00:00:00 +0000Fri, 20 Dec 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-compile-a-java-project.htmlFirst, the compiler needs to compile .java text files into .class bytecode, and then the JVM executes the .class bytecode files. The process is not complicated; this article mainly records some of the related steps during compilation and runtime. 1. A Single Source File Create a text file Hello.java 1WeChat Official AccountJavaMavenBuildTipsR&Dhttps://www.chenshaowen.com/en/blog/ip-not-in-ansible-all-ipv4-addresses-while-using-kubespray.htmlWed, 18 Dec 2019 00:00:00 +0000Wed, 18 Dec 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/ip-not-in-ansible-all-ipv4-addresses-while-using-kubespray.htmlWhen installing Kubernetes with KubeSpray, the following error occurs 1 2 3 4 5 6 fatal: [node0]: FAILED! => { "assertion": "ip in ansible_all_ipv4_addresses", "changed": false, "evaluated_to": false, "failed": true } Check the inventory.ini configuration 1 2 3 4 5 6 7 cat inventory.ini # ## Configure 'ip' variable to bind kubernetes services on a # ## different ip than the default iface # ## We should set etcd_member_name for etcd cluster.WeChat Official AccountKubernetesKubeSprayErrorInstallationTroubleshootingOperationshttps://www.chenshaowen.com/en/blog/build-an-automated-test-system-using-kubernetes-and-jenkins.htmlThu, 12 Dec 2019 00:00:00 +0000Thu, 12 Dec 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/build-an-automated-test-system-using-kubernetes-and-jenkins.html1. Test Layering The purpose of testing is to verify expected behavior and uncover latent defects. Testing strengthens confidence that a qualified product can be delivered, and it is what makes agile iteration possible. You could say that testing determines the development pace of a product. The network model has the seven-layer OSI and the four-layer TCP, while development patterns include MTV, MVC, MVP, MVVM, and so on.WeChat Official AccountTestingAutomationKubernetesJenkinsDevOpsCICDOperationshttps://www.chenshaowen.com/en/blog/creating-jenkins-slave-dynamically-on-kubernetes.htmlFri, 06 Dec 2019 00:00:00 +0000Fri, 06 Dec 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/creating-jenkins-slave-dynamically-on-kubernetes.html1. Jenkins Working Modes Jenkins has a single-Master, multi-Slave architecture. The Master assigns tasks and manages services. Slaves execute the actual tasks. Even when multiple Masters are deployed, they remain independent of one another and cannot coordinate scheduling. A high-availability Jenkins solution requires an external task distribution framework, such asWeChat Official AccountJenkinsKubernetesCICDDevOpsOperationshttps://www.chenshaowen.com/en/blog/how-to-dial-up-testing-using-jenkins.htmlWed, 04 Dec 2019 00:00:00 +0000Wed, 04 Dec 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-dial-up-testing-using-jenkins.html1. Why Dial Testing For some of the critical services in a system, we usually set up monitoring. When a failure occurs, it can be detected as quickly as possible and a notification sent to the people who care. Once a failure has occurred, the failure process can be tracedWeChat Official AccountJenkinsDial TestingMonitoringDevOpsOperationshttps://www.chenshaowen.com/en/blog/kubernetes-ingress-certificates-and-ops.htmlThu, 10 Oct 2019 00:00:00 +0000Thu, 10 Oct 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/kubernetes-ingress-certificates-and-ops.html1. Automatically Issuing Ingress Certificates Install cert-manager 1 2 3 4 5 6 7 8 9 10 kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.10/deploy/manifests/00-crds.yaml kubectl create namespace cert-manager kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true helm repo add jetstack https://charts.jetstack.io helm repo update helm install \ --name cert-manager \ --namespace cert-manager \ --version v0.10.0 \ jetstack/cert-manager Create a global issuer Create the file issuer.WeChat Official AccountKubernetesCertificatesFailuresIngressTroubleshootingOperationshttps://www.chenshaowen.com/en/blog/basic-operation-of-deployment-in-kubernetes.htmlFri, 27 Sep 2019 00:00:00 +0000Fri, 27 Sep 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/basic-operation-of-deployment-in-kubernetes.htmlA Deployment controls the number and state of Pods by creating ReplicaSets. This post mainly introduces some common Deployment operations. 1. Deployment yaml Format Passing the --dry-run flag means the command is not actually executed, only YAML output is generated: 1 kubectl create deployment nginx --image=nginx --dry-run -o yaml apiVersion: apps/v1 kind: Deployment metadata: creationTimestamp: null labels: app: nginx name: nginx spec: replicas: 1 selector: matchLabels: app: nginx strategy: {} template: metadata: creationTimestamp: null labels: app: nginx spec: containers: - image: nginx name: nginx resources: {} status: {} 2.WeChat Official AccountKubernetesDeploymentDeploymentLearningOperationshttps://www.chenshaowen.com/en/blog/certificates-in-kubernetes.htmlThu, 26 Sep 2019 00:00:00 +0000Thu, 26 Sep 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/certificates-in-kubernetes.htmlI am using Kubernetes 1.15.3; the way different versions handle this may vary. 1. About Certificates A root certificate is self-signed A root certificate is issued by itself. Browsers have the CA certificates of common certificate vendors built in. That is why browsers trust the next-level certificates issued by those vendors.WeChat Official AccountKubernetesCertificatesSecurityOperationsTLSConfigurationhttps://www.chenshaowen.com/en/blog/how-to-develop-a-operator-using-kubebuilder.htmlWed, 25 Sep 2019 00:00:00 +0000Wed, 25 Sep 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-develop-a-operator-using-kubebuilder.htmlWith the Operator approach, Kubernetes functionality can be extended in a friendly way. Operator = CRD + Controller. First you generate the CRD from a yaml definition, then the Controller continuously watches the data in etcd and performs the corresponding actions. There is a lot of tedious and repetitive work involved in developing an Operator.WeChat Official AccountKubernetesGoOperatorR&DOperationsCRDLearninghttps://www.chenshaowen.com/en/blog/summary-of-installation-problems-for-kubernetes-1.6.0.htmlFri, 20 Sep 2019 00:00:00 +0000Fri, 20 Sep 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/summary-of-installation-problems-for-kubernetes-1.6.0.html1. CNI Problems Error log 1 2 journalctl -u kubelet ...Unable to update cni config: No networks found in /etc/cni/net.d Because no CNI is installed, you need to remove --network-plugin=cni from the parameters in /var/lib/kubelet/kubeadm-flags.env. 1 2 cat /var/lib/kubelet/kubeadm-flags.env KUBELET_KUBEADM_ARGS="--cgroup-driver=systemd --pod-infra-container-image=k8s.gcr.io/pause:3.1" 2. Node NotReady There can be many reasons for a node being NotReady.WeChat Official AccountKubernetesInstallationTroubleshootingOperationsCNILearningkubeadmhttps://www.chenshaowen.com/en/blog/an-enterprise-class-registry-of-harbor.htmlThu, 12 Sep 2019 00:00:00 +0000Thu, 12 Sep 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/an-enterprise-class-registry-of-harbor.html1. Requirements for Building Harbor Harbor hardware requirements: CPU, at least 2 cores, 4 cores is better Mem, at least 4 GB, 8 GB is better Disk, at least 40 GB, 160 GB is better Docker version requirements: 17.06.0 or later To build Harbor on Kubernetes, you can refer to the documentation, Building Harbor with Helm.WeChat Official AccountHarborImagesContainersKubernetesOperationshttps://www.chenshaowen.com/en/blog/how-docker-pull-images.htmlFri, 06 Sep 2019 08:00:00 +0000Fri, 06 Sep 2019 08:00:00 +0000https://www.chenshaowen.com/en/blog/how-docker-pull-images.html1. Pulling images with docker pull When you pull an image with docker pull {IMAGE_NAME}, there are two cases: The IMAGE_NAME prefix points to a registry Docker recognizes IMAGE_NAME as an image provided by the specified repository. For example, for myregistry.io/space1/image1:latest, Docker requests the image data from the server that myregistry.WeChat Official AccountDockerImagesLearningOperationshttps://www.chenshaowen.com/en/blog/install-harbor-using-helm.htmlFri, 30 Aug 2019 00:00:00 +0000Fri, 30 Aug 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/install-harbor-using-helm.htmlPrerequisites: (1) Helm is already installed, see Installing Helm; (2) the cluster has default dynamic storage available, see Providing PV Dynamic Storage with StorageClass 1. Installing Ingress with Helm Ingress consists of two parts: Ingress and the Ingress Controller. In Kubernetes, the Ingress object describes routing rules; the Ingress Controller interacts with the Apiserver and writes the Ingress rules into the Nginx Pod.WeChat Official AccountKubernetesHarborIngressHelmImage RegistryImagesOperationshttps://www.chenshaowen.com/en/blog/using-s2i-to-build-cloud-native-applications.htmlMon, 26 Aug 2019 00:00:00 +0000Mon, 26 Aug 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-s2i-to-build-cloud-native-applications.html1. What Problem S2I Solves It is foreseeable that a large number of applications will be deployed as containers in the future. Container platforms care about images and containers, while application development cares about business code, and ultimately the code needs to be deployed as an image. From code to image — that is Source To Image, or S2I.WeChat Official AccountS2IDockerCloud NativeImagesOperationsR&Dhttps://www.chenshaowen.com/en/blog/using-emptydir-hostpath-localvolume-in-kubernetes.htmlSat, 24 Aug 2019 00:00:00 +0000Sat, 24 Aug 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-emptydir-hostpath-localvolume-in-kubernetes.htmlEarlier, in Kubernetes Volumes, we got some understanding of Volumes. This post focuses on practice, learning how to use the three local storage options: emptydir, hostpath, and localvolume. 1. Basic Properties of a PV 1.1 PV Lifecycle PV states: Available: available, not yet bound by any PVC Bound: already bound to a PVC Released: the PVC was deleted, but the resource has not yet been reclaimed Failed: automatic reclamation failed 1.WeChat Official AccountKubernetesVolumeStorageOperationsPVCLearninghttps://www.chenshaowen.com/en/blog/kubernetes-cluster-operations-practices.htmlThu, 22 Aug 2019 00:00:00 +0000Thu, 22 Aug 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/kubernetes-cluster-operations-practices.htmlCompiled from the “Development Tips” series, gathering the common problems and solutions from day-to-day Kubernetes cluster operations. 1. Configuring Multiple Clusters with Kubectl When doing Kubernetes-related development you usually end up managing several clusters. Kubectl provides multi-cluster context management. Kubectl’s configuration usually lives in $HOME/.kube/config or /etc/kubernetes/admin.conf. Log into the machine, look at the cluster configuration, and edit it using the format below.WeChat Official AccountKubernetesOperationsR&DkubectlHelmhttps://www.chenshaowen.com/en/blog/sonarqube-of-code-quality-analysis-tool.htmlWed, 21 Aug 2019 00:00:00 +0000Wed, 21 Aug 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/sonarqube-of-code-quality-analysis-tool.html1. Who Needs SonarQube Code review is an indispensable part of high-quality software development. It helps developers spot bugs in the code in time and improves code quality and maintainability. The dimensions of code review include language conventions, code style, and soundness of design. Checking these things by hand consumesWeChat Official AccountSonarQubeDevOpsCICDR&DCode Qualityhttps://www.chenshaowen.com/en/blog/using-kubeadm-to-install-the-kubernetes-cluster.htmlThu, 15 Aug 2019 00:00:00 +0000Thu, 15 Aug 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-kubeadm-to-install-the-kubernetes-cluster.html1. Cluster Planning Prepare three hosts: one Master and two Nodes. Operating system: CentOS 7 Specs: 2 Core 4 GB Docker version: 18.06.3 Kubernetes version: 1.15.3 If you are using a purchased cloud host, open the following ports: 1 2 3 4 5 6 7 8 9 10 11 12WeChat Official AccountKubernetesKubeadmInstallationOperationsClusterLearninghttps://www.chenshaowen.com/en/blog/let-us-start-learning-go-4.htmlSat, 10 Aug 2019 00:00:00 +0000Sat, 10 Aug 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/let-us-start-learning-go-4.html1. The main and init Functions A package can contain multiple init functions, but it must contain exactly one main function. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 package main import ( "fmt" ) func init() { fmt.WeChat Official AccountGoLet's Learn GoFunctionsLearninghttps://www.chenshaowen.com/en/blog/ceph-in-kubernetes.htmlWed, 07 Aug 2019 00:00:00 +0000Wed, 07 Aug 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/ceph-in-kubernetes.html1. Different Kinds of Storage 1.1 File Storage File storage is storage based on files. When accessing data, you need to provide the corresponding lookup path. It is suitable for services such as FTP and NFS. 1.2 Block Storage Block storage splits data into blocks and stores each part separately. When accessing data, the underlying storage software reassembles these scattered pieces.WeChat Official AccountKubernetesCephStorageOperationsRBDLearningDistributed Storagehttps://www.chenshaowen.com/en/blog/volumes-of-kubernetes.htmlMon, 05 Aug 2019 00:00:00 +0000Mon, 05 Aug 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/volumes-of-kubernetes.html1. Docker Storage Volumes 1.2 Volumes in Docker A Docker volume mounts a host directory into a container. Files modified inside the container are persisted to the host. Even if the container is deleted, the files on the host are retained. Docker uses /var/lib/docker/volumes/ to store container volumes. List local volumes:WeChat Official AccountKubernetesStorageLearningOperationsPVCStorage VolumeArchitectureDockerhttps://www.chenshaowen.com/en/blog/labels-and-selectors-of-kubernetes.htmlSun, 04 Aug 2019 00:00:00 +0000Sun, 04 Aug 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/labels-and-selectors-of-kubernetes.html1. Labels 1.1 What Labels Are Labels are key-value pairs attached to an object. You can add labels directly when the object is created, or modify them dynamically afterwards. Label format: 1 2 3 4 "labels": { "key1" : "value1", "key2" : "value2" } Format requirements: Key: must be unique Value: must start with a letter or a digit, may contain letters, digits, hyphens, dots, and underscores, and is at most 63 characters long 1.WeChat Official AccountKubernetesLearningOperationsLabelsSelectorshttps://www.chenshaowen.com/en/blog/open-source-is-refactoring-the-business-model.htmlThu, 01 Aug 2019 00:00:00 +0000Thu, 01 Aug 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/open-source-is-refactoring-the-business-model.htmlAdobe acquired Magento for $1.67 billion, Microsoft acquired GitHub for $7.5 billion, IBM acquired Red Hat for $34 billion — open source turns out to be a good business too. This article is mainly a record of, and some thoughts about, open source. 1. What Open Source Is 1.1 History In 1969, Bell Labs shared the Unix source code with the community, laying an important foundation for open source.WeChat Official AccountOpen SourceThoughtsBusiness ModelIndustryBusinessToBhttps://www.chenshaowen.com/en/blog/networks-of-kubernetes.htmlSun, 28 Jul 2019 00:00:00 +0000Sun, 28 Jul 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/networks-of-kubernetes.html1. Docker’s Network Model 1.1 bridge Mode bridge mode is used by default, and you can also specify it with --net=bridge. In bridge mode, containers connect to the same virtual bridge, docker0. docker0 usually occupies the 172.17.0.1/16 subnet. Containers on the same bridge can communicate directly over IP. 1.2 host Mode Use --net=host to specify host mode.WeChat Official AccountKubernetesNetworkLearningOperationsDockerCNIArchitecturehttps://www.chenshaowen.com/en/blog/quickly-deploy-prometheus-using-helm-and-operator.htmlFri, 26 Jul 2019 00:00:00 +0000Fri, 26 Jul 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/quickly-deploy-prometheus-using-helm-and-operator.htmlHelm is the package manager for Kubernetes; Operator is used to manage stateful distributed applications on Kubernetes. This article mainly describes how to use Helm and Operator to quickly deploy Prometheus on a Minikube cluster and use Grafana to view monitoring data. For Minikube installation, see Building a Remote Kubernetes Development Environment; for Helm configuration, see The Package Manager for Kubernetes – Helm; for Operator, see Kubernetes Complex Stateful Application Management Framework – Operator.WeChat Official AccountKubernetesHelmOperatorPrometheusMonitoringOperationsLearninghttps://www.chenshaowen.com/en/blog/api-of-kubernetes.htmlFri, 26 Jul 2019 00:00:00 +0000Fri, 26 Jul 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/api-of-kubernetes.html1. Objects in Kubernetes Kubernetes objects are persistent entities in the system, used to represent the state of the cluster. Users interact with Kubernetes by operating on objects, telling the system the workload they expect. Objects are manipulated through the Kubernetes API. Every Kubernetes object contains two nested object fields, Spec and Status.WeChat Official AccountKubernetesAPIObjectsOperationsLearningArchitectureApiserverhttps://www.chenshaowen.com/en/blog/complex-application-management-framework-operator-for-kubernetes.htmlThu, 25 Jul 2019 00:00:00 +0000Thu, 25 Jul 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/complex-application-management-framework-operator-for-kubernetes.html1. Origins Originally, two CoreOS employees, in order to make deploying etcd clusters easier, bound corresponding logical operations to the create, delete, and update events of the etcdCluster object, using Kubernetes to automate etcd cluster management. A few months later, at a KubeCon conference, they shared this approach, which they called Operator, and it received a strong response from the community.WeChat Official AccountKubernetesHelmPracticeOperatorOperationsStateful ApplicationsLearninghttps://www.chenshaowen.com/en/blog/package-manager-helm-of-kubernetes.htmlWed, 24 Jul 2019 00:00:00 +0000Wed, 24 Jul 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/package-manager-helm-of-kubernetes.html1. Why Helm Is Needed One important design philosophy in Kubernetes is declarative operation. Users change the system by setting the system’s expected state. For example, the current replica count is 2 and it needs to be adjusted to 3. The declarative way is to modify the replica count inWeChat Official AccountKubernetesHelmPracticeOperationsPackage ManagementLearningChartDeploymenthttps://www.chenshaowen.com/en/blog/let-us-start-learning-go-3.htmlTue, 23 Jul 2019 00:00:00 +0000Tue, 23 Jul 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/let-us-start-learning-go-3.html1. Go’s Package Management Mechanisms 1.1 GOPATH GOPATH pulls code with the go get command and places it in the GOPATH directory. The problems with GOPATH are: It cannot manage package versions It uses a global repository, so it cannot isolate projects effectively 1.2 Vendor Starting with version 1.5, GoWeChat Official AccountGoGoModulesLet's Learn GoLearningR&DPackage Managementhttps://www.chenshaowen.com/en/blog/how-to-optimize-the-blog-solution-step-by-step.htmlSun, 21 Jul 2019 00:00:00 +0000Sun, 21 Jul 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-optimize-the-blog-solution-step-by-step.htmlFour or five years have passed since I first started blogging. My original intent was to record, think, and organize — a little more for my own benefit, and if it could help others too, so much the better. I went from csdn at the very beginning, to deploying Ghost on Linode, to Pages hosting, and finally to settling on a CDN.WeChat Official AccountBlogPlanOperationsCDNLearninghttps://www.chenshaowen.com/en/blog/practice-guide-to-kubectl.htmlFri, 19 Jul 2019 01:00:00 +0000Fri, 19 Jul 2019 01:00:00 +0000https://www.chenshaowen.com/en/blog/practice-guide-to-kubectl.html1. What Is kubectl kubectl is the command-line tool for Kubernetes. It interacts with the cluster through the API server. 2. Configuring kubectl kubectl can be configured through ~/.kube/config to connect to one or more clusters. For details on how to configure it, see: Configure Access to Multiple Clusters. If you need to configure a remote cluster, see: Building a Remote Kubernetes Development Environment.WeChat Official AccountKuberneteskubectlPracticeLearningOperationshttps://www.chenshaowen.com/en/blog/basic-concepts-in-kubernetes.htmlFri, 19 Jul 2019 00:00:00 +0000Fri, 19 Jul 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/basic-concepts-in-kubernetes.html1. Cluster 1.1 Master The Master is responsible for managing and maintaining Kubernetes cluster information, dispatching tasks to Nodes, and receiving feedback. The components running on the Master include kube-apiserver, kube-scheduler, kube-controller-manager, cloud-controller-manager, and so on. 1.2 Node A Node carries the cluster workload; it can be a physical machine or a virtual machine.WeChat Official AccountKubernetesConceptsLearningOperationshttps://www.chenshaowen.com/en/blog/building-a-remote-kubernetes-development-environment.htmlThu, 18 Jul 2019 00:00:00 +0000Thu, 18 Jul 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/building-a-remote-kubernetes-development-environment.htmlMinikube is a single-node distribution of Kubernetes, suited to product evaluation and day-to-day development. Here we use Minikube to build a development environment: Kubernetes runs on a CentOS cloud server, and development happens remotely from a local OS X machine. 1. Install Minikube on the Cloud Server On the MinikubeWeChat Official AccountKubernetesR&DEnvironmentPracticeMiniKubeLearningOperationsDevelopment Environmenthttps://www.chenshaowen.com/en/blog/tob-thinking-of-large-and-small-companies.htmlTue, 16 Jul 2019 00:00:00 +0000Tue, 16 Jul 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/tob-thinking-of-large-and-small-companies.html1. Understanding ToB It is only because money has become harder and harder to make from ToC that ToB has flourished. The era in which a single idea could be turned into a good product and ultimately accepted by everyone has been crushed beneath the wheels. In the early days of the internet, a single portal directory made Sohu, a single chat tool made Tencent, and even hao123 — nothing more than a handful of jump pages — sold for tens of millions.WeChat Official AccountThoughtsToBBusiness ModelProductManagementTeamshttps://www.chenshaowen.com/en/blog/nfs-deployment-and-mount.htmlFri, 12 Jul 2019 00:00:00 +0000Fri, 12 Jul 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/nfs-deployment-and-mount.htmlNFS (Network File System) is used to share directories over the network; once the client mounts them, they can be read and written like a local disk. This post covers deploying an NFS service on CentOS and Ubuntu, and how to mount a remote NFS directory on each system. 1.WeChat Official AccountNFSCentOSUbuntumacOSLinuxOperationsStoragehttps://www.chenshaowen.com/en/blog/book/the-way-to-go.htmlThu, 06 Jun 2019 07:16:21 +0000Thu, 06 Jun 2019 07:16:21 +0000https://www.chenshaowen.com/en/blog/book/the-way-to-go.htmlEnglish title: The Way To Go Subtitle: A Thorough Introduction To The Go Programming Language Author: Ivo Balbaert Publisher: iUniverse Publication year: 2012-3-8 ISBN: 9781469769165 Notes: This is a Go primer I would personally recommend. Because Go has evolved rapidly in recent years, parts of the original text are already out of date.WeChat Official AccountBooksGoGetting StartedLearninghttps://www.chenshaowen.com/en/blog/let-us-start-learning-go-2.htmlWed, 05 Jun 2019 00:00:00 +0000Wed, 05 Jun 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/let-us-start-learning-go-2.html1. Go’s Data Structures Basic types Boolean: bool Integer: byte, int, int8, int16, uint, uintptr Floating point: float32, float64 Complex: complex64, complex128 String: string Character: rune Error: error Composite types Pointer: pointer Array: array Slice: slice Dictionary: map Channel: chan Struct: struct Interface: interface A Go variable identifier is made up of letters, digits, and underscores, and its first character cannot be a digit.WeChat Official AccountGoLet's Learn GoTypesData Structureshttps://www.chenshaowen.com/en/blog/how-to-prevent-xss-in-web-rich-text.htmlTue, 04 Jun 2019 00:00:00 +0000Tue, 04 Jun 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-prevent-xss-in-web-rich-text.html1. How XSS Works and Several Common Attack Types An XSS attack is an attack that executes malicious scripts to steal a user’s login state, hijack a session, and so on. The input sources for malicious scripts include Cookies, Post forms, Get requests, HTTP header content, and more. We usually call a code fragment used for an XSS attack an XSS vector.WeChat Official AccountXSSRich TextSecurityWebR&Dhttps://www.chenshaowen.com/en/blog/django-xss-cleaner.htmlMon, 03 Jun 2019 00:00:00 +0000Mon, 03 Jun 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/django-xss-cleaner.htmldjango-xss-cleaner is a bleach-based Django XSSFilter toolkit that implements whitelist XSS filtering for GET and POST request parameters. The package ships with a set of built-in whitelisted HTML tags and attribute settings, and also supports custom extensions. Project address, https://github.com/shaowenchen/django-xss-cleaner 1. Installing and Configuring settings.py Add the middleware xss_cleaner.middlewares.CleanXssMiddleware toWeChat Official AccountXSSDjangoPythonSecurityR&Dhttps://www.chenshaowen.com/en/blog/book/cloud-work-era.htmlSat, 01 Jun 2019 00:00:00 +0000Sat, 01 Jun 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/book/cloud-work-era.htmlAuthor: Wang Jianguo / Wu Bin Publication year: 2016-11-1 ISBN: 9787508668024 Notes: The book’s main point is that in the internet era, working from home and collaborating remotely is a better fit. From there it goes on to discuss how business owners and workers can harness this way of working, and the author offers some of his own thoughts.WeChat Official AccountBooksFormCloudWorkhttps://www.chenshaowen.com/en/blog/let-us-start-learning-go-1.htmlTue, 28 May 2019 00:00:00 +0000Tue, 28 May 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/let-us-start-learning-go-1.html1. The Background of the Go Language The development of C/C++ did not keep pace with the advance of computing, and for over a decade no mainstream systems programming language suited to the era appeared. To solve this problem, Google developed a new programming language – Go. Google’s situation at the time:WeChat Official AccountGoLet's Learn GoBackgroundLearninghttps://www.chenshaowen.com/en/blog/different-stages-of-the-saas-development-team.htmlFri, 24 May 2019 00:00:00 +0000Fri, 24 May 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/different-stages-of-the-saas-development-team.htmlThe team I am on is responsible for SaaS development. We have a few flagship SaaS products and also some supporting ones. The flagship SaaS products have complex architectures, large technical challenges, and heavy staffing, and they are the focus of our KPIs. But a flagship project cannot stay the flagship forever — projects have life cycles and phases.WeChat Official AccountThoughtsTeamsSaaSManagementR&Dhttps://www.chenshaowen.com/en/blog/git-skills-you-do-not-notice.htmlFri, 17 May 2019 00:00:00 +0000Fri, 17 May 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/git-skills-you-do-not-notice.html1. Pages GitHub, GitLab, Bitbucket and others all offer a free static page hosting service called Pages. With the Pages service you can publish documentation, blogs, and more. Taking GitHub as an example, using Pages usually takes just a few simple steps: Create a new project: [username].github.io Commit static htmlWeChat Official AccountTipsGitProcessesDevelopmentR&Dhttps://www.chenshaowen.com/en/blog/output-as-a-service.htmlFri, 03 May 2019 00:00:00 +0000Fri, 03 May 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/output-as-a-service.html1. IT Infrastructure Is Undergoing Drastic Change Traditional enterprises are building more efficient business models on top of internet technology to strengthen their competitiveness in their industries. Lower R&D costs, faster product iteration, closer proximity to customers, better service quality… This whole set of changes will push the productivity and living standards of society as a whole onto a new level.WeChat Official AccountThoughtsCloud ServicesOperationsIndustryToBhttps://www.chenshaowen.com/en/blog/book/microservice-design.htmlTue, 16 Apr 2019 08:47:38 +0000Tue, 16 Apr 2019 08:47:38 +0000https://www.chenshaowen.com/en/blog/book/microservice-design.htmlAuthor: [UK] Sam Newman Publication year: 2016-5 ISBN: 9787115420268 Notes: It mainly answers a series of questions about microservices. What microservices are, what characteristics they have, how systems evolve, how to divide services, and integration, deployment, and testing strategies. It gives a fairly comprehensive introduction to microservices, and many of its views resonate with me.WeChat Official AccountBooksServicesArchitecturehttps://www.chenshaowen.com/en/blog/continue-to-run.htmlFri, 05 Apr 2019 00:00:00 +0000Fri, 05 Apr 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/continue-to-run.html1. On Running When you talk about running, it is easy to think of Haruki Murakami’s What I Talk About When I Talk About Running. The sentence pattern “What I talk about when I talk about XX” has been widely quoted, which says something about the influence of the book.WeChat Official AccountThoughtsRunningLifeGrowthhttps://www.chenshaowen.com/en/blog/from-account-security-to-secondary-verification.htmlThu, 21 Mar 2019 00:00:00 +0000Thu, 21 Mar 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/from-account-security-to-secondary-verification.html1. Security Problems Are Getting Worse As the internet permeates every layer of life, we depend more and more on various online IDs. These IDs make up our digital selves and are extremely important account assets. On the other hand, account information leaks and security incidents happen from time to time: CSDN, 12306, Huazhu Hotels, Facebook, Twitter, Uber… the list goes on.WeChat Official AccountSecurityVerificationTwo-Factor AuthenticationAccount SecurityLearningBest Practiceshttps://www.chenshaowen.com/en/blog/database-query-optimization-of-django-performance.htmlWed, 20 Mar 2019 00:00:00 +0000Wed, 20 Mar 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/database-query-optimization-of-django-performance.htmlThis article mainly offers optimization advice on Django fields and queries, and also introduces a performance analysis tool called Django-silk. I hope it helps you develop high-performance Django projects. 1. DBA’s Advice 1.1 Table Field Design Avoid null values; null values are hard to optimize queries around and take up extra index space Prefer INT over BIGINT, and describe fields as accurately as possible Use enums or integers instead of string types Use TIMESTAMP instead of DATETIME Do not put more than 20 fields in a single table Store IPs as integers 1.WeChat Official AccountDatabaseOptimizationPerformanceDjangoPythonR&Dhttps://www.chenshaowen.com/en/blog/sub-db-and-table-of-django-performance.htmlTue, 19 Mar 2019 00:00:00 +0000Tue, 19 Mar 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/sub-db-and-table-of-django-performance.html1. The Problem We Hit The frontend requests are heavy, concurrency is high, and access is slow. The bottlenecks show up mainly as: Large single tables Large single databases Slow network IO Slow disk IO Optimizing network and disk IO mainly relies on hardware upgrades. In theory, a database imposes no limit on the size of a single database or a single table, but an oversized single database or table means more requests land on a single machine, putting pressure on IO.WeChat Official AccountDjangoDatabaseShardingPerformanceDatabase ShardingTable Shardinghttps://www.chenshaowen.com/en/blog/r-and-d-process-of-agile-development.htmlThu, 14 Mar 2019 00:00:00 +0000Thu, 14 Mar 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/r-and-d-process-of-agile-development.html1. What Agile Development Is In the traditional software development model, the cycle from raising a requirement to final delivery is long. The waterfall model proceeds through six steps: requirements analysis, design, coding, integration, testing, and maintenance. Once a requirement changes, the work invested earlier is not only wasted but also hard to adjust.WeChat Official AccountBranchingAgileProcessTestingR&DLearningManagementTeamsIterationBest Practiceshttps://www.chenshaowen.com/en/blog/build-an-overall-team.htmlSat, 02 Mar 2019 00:00:00 +0000Sat, 02 Mar 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/build-an-overall-team.html1. Consistent Understanding In a large organization, we can think of a small team as a microservice. As early as 1967, Conway proposed the concept of the microservice. Conway argued that any organization designing a system will deliver a design whose structure is consistent with that organization’s communication structure. When developing complex systems, we usually split the system into modules.WeChat Official AccountThoughtsTeamsR&DManagementCulturehttps://www.chenshaowen.com/en/blog/mock-of-pytest.htmlSun, 27 Jan 2019 00:00:00 +0000Sun, 27 Jan 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/mock-of-pytest.html1. Why We Need Mock When writing unit tests, the function under test is sometimes not a self-contained executable unit. It depends on external resources, such as the return value of another function or a value in a row of a database. To shield the test from interference from externalWeChat Official AccountTestingPythonPytestLearninghttps://www.chenshaowen.com/en/blog/buildpack-for-paas-deployment.htmlThu, 24 Jan 2019 00:00:00 +0000Thu, 24 Jan 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/buildpack-for-paas-deployment.htmlI have been learning Go recently, and the internal PaaS platform we commonly use happens to support Go as well as related Web frameworks. A PaaS system that supports multiple languages cannot do without the buildpack mechanism. Although PaaS platforms keep being upgraded, the buildpack mechanism has always been retained.WeChat Official AccountPaaSBuildpackOperationsR&DDeploymentLearningCloud NativeMulti-languagehttps://www.chenshaowen.com/en/blog/configuration-of-developing-go.htmlWed, 23 Jan 2019 00:00:00 +0000Wed, 23 Jan 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/configuration-of-developing-go.html1. Common Environment Variables 1.1 GOROOT $GOROOT is the local directory where the Go package is installed. 1 2 3 4 cd /c/Go ls AUTHORS CONTRIBUTORS PATENTS VERSION bin favicon.ico misc robots.txt test CONTRIBUTING.md LICENSE README.md api doc lib pkg src 1.2 GOPATH $GOPATH is the Go workspace directory, used to hold code, third-party libraries, compiled intermediate files, and so on.WeChat Official AccountGoGetting StartedEnvironment VariablesDevelopment EnvironmentR&Dhttps://www.chenshaowen.com/en/blog/domain-knowledge-is-the-key-of-paas.htmlSat, 12 Jan 2019 00:00:00 +0000Sat, 12 Jan 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/domain-knowledge-is-the-key-of-paas.html1. What I Am Thinking About At a big company, you get more opportunities to learn about industry trends and take part in industry change. A large platform does not run on the strength of one person or a few people. If it really could, it would not deserve to be called a large platform.WeChat Official AccountPaaSThoughtsOperationsPlatformManagementhttps://www.chenshaowen.com/en/blog/101-of-pytest.htmlSat, 29 Dec 2018 00:00:00 +0000Sat, 29 Dec 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/101-of-pytest.htmlThe Pytest framework is simple to use, rich in plugins, and powerful, and it is widely used for Python automated testing. This article introduces some basic concepts and usage of Pytest. 1. How It Runs Step one, Pytest reads its configuration from the command line or a file. Step two,WeChat Official AccountTestingPythonPytestR&DLearningAutomated TestingUnit TestingBest Practiceshttps://www.chenshaowen.com/en/blog/how-to-use-jenkins-docker-gitlab-to-build-django-automated-deployment-process.htmlSun, 25 Nov 2018 00:00:00 +0000Sun, 25 Nov 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-use-jenkins-docker-gitlab-to-build-django-automated-deployment-process.htmlOne of the illusions that programmers at big companies easily fall into is mistaking platform capabilities for their own abilities. In a large team, we should not focus only on our own little patch of ground; we need to understand every part of the platform. On one hand, this helps us make better use of the platform’s features; on the other, it helps our own technical growth.WeChat Official AccountDjangoGitLabJenkinsDockerDevOpsCICDR&DOperationshttps://www.chenshaowen.com/en/blog/docker-operations-practices.htmlSat, 24 Nov 2018 00:00:00 +0000Sat, 24 Nov 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/docker-operations-practices.htmlCompiled from the “Development Tips” series, a roundup of common problems and solutions in Docker usage and operations. 1. Basic Docker Machine Commands Create the my-vm-name virtual machine 1 docker-machine create --driver virtualbox my-vm-name List all docker-machine instances 1 2 3 4 docker-machine ls NAME ACTIVE DRIVER STATE URL SWARMWeChat Official AccountDockerOperationsR&DContainershttps://www.chenshaowen.com/en/blog/api-of-viewset-and-serializer-in-rest-framework.htmlTue, 20 Nov 2018 00:00:00 +0000Tue, 20 Nov 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/api-of-viewset-and-serializer-in-rest-framework.html1. Serializer 1.1 Data Validation When deserializing data, the validity of the data needs to be checked. At this point you can call is_valid() for validation, and if a validation error occurs, you can get the error message from the .errors attribute. For example: 1 2 3 4 serializer.is_valid() # False serializer.WeChat Official AccountDjangoPythonAPIInterfaceR&Dhttps://www.chenshaowen.com/en/blog/viewset-and-serializer-in-rest-framework.htmlSun, 18 Nov 2018 00:00:00 +0000Sun, 18 Nov 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/viewset-and-serializer-in-rest-framework.html1. View Class in Django First, recall how Django handles a request. After receiving a request, Django creates a handler of type WSGIHandler, and the handler controls the entire processing flow. So how are the request URL and the View associated with each other? Django first loads the URLconf according to the ROOT_URLCONF setting, then matches the URLpatterns in the URLconf one by one in order, stopping as soon as a match is found.WeChat Official AccountDjangoPythonAPIFrameworkInterfaceR&Dhttps://www.chenshaowen.com/en/blog/django-full-stack-optimization-guide.htmlThu, 15 Nov 2018 00:00:00 +0000Thu, 15 Nov 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/django-full-stack-optimization-guide.htmlAs the data volume exploded, the system responded very slowly. We carried out a series of optimizations on the application, and the system’s response time improved by an order of magnitude. Overall, optimizations in file compression and faster network access gave a noticeable boost to frontend performance, while optimizations in stored procedures, caching, and logic code gave a noticeable boost to backend performance.WeChat Official AccountDjangoPythonOptimizationFrontendBackendGuideR&Dhttps://www.chenshaowen.com/en/blog/how-to-build-tob-products.htmlWed, 14 Nov 2018 00:00:00 +0000Wed, 14 Nov 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-build-tob-products.htmlThe team I work with is building a ToB product that has considerable influence in its vertical. This article is a review of key events for the team, and also contains some personal reflections. 1. Be the Best in a Familiar Setting On the internet there is far too much shoddy content, and products that win broad attention and influence are few and far between.WeChat Official AccountThoughtsCommunityToBProductTeamsManagementhttps://www.chenshaowen.com/en/blog/centos-server-operations-practices.htmlSun, 11 Nov 2018 00:00:00 +0000Sun, 11 Nov 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/centos-server-operations-practices.htmlCompiled from the “Development Tips” series, collecting common problems and solutions in CentOS server operations. 1. Upgrading the Docker Version Uninstall the old version of Docker 1 2 3 4 5 6 $yum remove docker \ docker-common \ container-selinux \ docker-selinux \ docker-engine \ docker-engine-selinux Images, containers, data, and networks stored in /var/lib/docker/ are all preserved.WeChat Official AccountCentOSLinuxOperationsR&DDockerhttps://www.chenshaowen.com/en/blog/using-frp-to-publish-local-service-to-public-network.htmlWed, 07 Nov 2018 00:00:00 +0000Wed, 07 Nov 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-frp-to-publish-local-service-to-public-network.html1. What frp Is frp is a high-performance reverse proxy application. It provides: Using a machine behind a private network or firewall to expose http or https services to the external network For http and https services, it supports domain-based virtual hosts and custom domain binding, so multiple domains canWeChat Official AccountfrpFRPNAT TraversalToolsOperationshttps://www.chenshaowen.com/en/blog/development-tools-and-productivity-tips.htmlFri, 02 Nov 2018 00:00:00 +0000Fri, 02 Nov 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/development-tools-and-productivity-tips.htmlCompiled from the “Development Tips” series, collecting common development tools and productivity-related techniques. 1. The Babel Transpiler ES6 provides many new features, but not every browser supports them perfectly, while ES5 support is far better. Babel is an ES6 transpiler that converts ES6 code into ES5 code. This means youWeChat Official AccountToolsR&DProductivityVSCodeLearninghttps://www.chenshaowen.com/en/blog/solving-the-problem-of-celery-processe-large-file-failure.htmlWed, 24 Oct 2018 00:00:00 +0000Wed, 24 Oct 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/solving-the-problem-of-celery-processe-large-file-failure.html1. A Small Requirement We run into small requirements all the time, but they are not always simple to implement. Here is a simple file upload requirement, broken into the following steps: The user uploads a large file on the page The large file is temporarily stored in the internalWeChat Official AccountCeleryPythonDjangoFailureTroubleshootingLarge FilesR&Dhttps://www.chenshaowen.com/en/blog/after-install-centos.htmlSun, 21 Oct 2018 00:00:00 +0000Sun, 21 Oct 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/after-install-centos.html1. Replacing the Repository Mirror First, back up the official repository. 1 mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.backup Download the new repository to /etc/yum.repos.d/ CentOS 5 1 wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-5.repo CentOS 6 1 wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-6.repo CentOS 7 1 wget -O /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo Generate the metadata cache. 1 yum makecache 2. Accessing NTFS File Systems The Linux kernel currently only supports reading Microsoft’s NTFS file system.WeChat Official AccountTipsCentOSLinuxOperationsLearningInitializationConfigurationhttps://www.chenshaowen.com/en/blog/writing-python2-python3-compatible-code.htmlSat, 20 Oct 2018 14:51:47 +0000Sat, 20 Oct 2018 14:51:47 +0000https://www.chenshaowen.com/en/blog/writing-python2-python3-compatible-code.htmlPython2 to Python3 is a major version update. Today, a large number of projects in production still run on Python2. But that does not mean those projects will stay on Python2 forever — developers also need to consider the project’s compatibility with Python3, both to make migration easier and asWeChat Official AccountPythonCompatibilityR&DLearningMigrationhttps://www.chenshaowen.com/en/blog/git-and-development-environment-setup.htmlFri, 19 Oct 2018 00:00:00 +0000Fri, 19 Oct 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/git-and-development-environment-setup.htmlCompiled from the “Development Tips” series, gathering practices around Git collaboration and development environment setup. 1. warning: LF will be replaced by CRLF Windows, Linux, and Mac are inconsistent in how they mark line endings in files. Windows uses CRLF as the terminator, while Linux and Mac use LF. GitWeChat Official AccountGitPythonWindowsR&DEnvironmenthttps://www.chenshaowen.com/en/blog/python-and-django-development-practices.htmlFri, 19 Oct 2018 00:00:00 +0000Fri, 19 Oct 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/python-and-django-development-practices.htmlCompiled from the “Development Tips” series, collecting common problems and solutions in Python and Django development. 1. Sorting Python Iterables Iterable objects fall into 3 main categories: All sequence types, such as list, str, and tuple Some non-sequence types, such as dict and file Objects that contain an __iter__() orWeChat Official AccountPythonDjangoR&DLearningBackendhttps://www.chenshaowen.com/en/blog/how-to-use-django-signal-decoupling-module.htmlFri, 12 Oct 2018 00:00:00 +0000Fri, 12 Oct 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-use-django-signal-decoupling-module.htmlRecently I was responsible for developing a backend-heavy application. The data flow in this application is complex, and the processing logic has piled up in a redundant mess. The project’s tech stack is Django + Vuejs. The frontend is bundled with Webpack, managed in modules, and mainly displays data. The backend involves many modules, many processing rules, and many data tables, so every time I had to modify an earlier feature I spent a lot of time reviewing the code.WeChat Official AccountDjangoPythonDesignDecouplingR&Dhttps://www.chenshaowen.com/en/blog/open-k8s-monitoring-and-stress-testing..htmlMon, 24 Sep 2018 00:00:00 +0000Mon, 24 Sep 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/open-k8s-monitoring-and-stress-testing..html1. Kubectl Basic Commands 1.1 Creating Objects 1 2 3 4 5 6 7 8 # Create resources; a remote URL also works kubectl create -f ./my.yaml # Create resources from multiple files kubectl create -f ./my1.yaml -f ./my2.yaml # Create resources using all manifest files in a directory kubectl create -f .WeChat Official AccountKubernetesTestingMonitoringOperationsLearningkubectlStress TestingObservabilityhttps://www.chenshaowen.com/en/blog/how-to-use-minikube-under-windows7.htmlSun, 23 Sep 2018 00:00:00 +0000Sun, 23 Sep 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-use-minikube-under-windows7.html1. Basic Concepts 1.1 Kubernetes Kubernetes (K8s for short), the successor to Google’s Borg, is an open-source system for automatically deploying, scaling, and managing containerized applications. The features it provides: Automated deployment of containers Automated scaling up and down Automated application/service upgrades Grouping containers to serve traffic externally, with load balancing support Health checks for services, with automatic restarts 1.WeChat Official AccountDockerKubernetesWindowsMiniKubeLearningOperationshttps://www.chenshaowen.com/en/blog/some-tips-under-windows7-using-docker.htmlFri, 21 Sep 2018 00:00:00 +0000Fri, 21 Sep 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/some-tips-under-windows7-using-docker.html1. Customizing the docker-machine Location When using Docker on Windows 7, docker-machine is stored by default in the C:\users\your name\.docker\machine\machines directory. To avoid taking up storage space on the system drive, you can change this as follows. If you have not created a virtual machine yet, you can specify the location by setting the MACHINE_STORAGE_PATH environment variable.WeChat Official AccountDockerWindowsTroubleshootingOperationshttps://www.chenshaowen.com/en/blog/selinux-and-iptables-on-linux.htmlFri, 21 Sep 2018 00:00:00 +0000Fri, 21 Sep 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/selinux-and-iptables-on-linux.htmlWhile working on servers recently, I ran into a few concepts I wasn’t clear about, so I looked up some references and organized them as follows. 1. The BLP Security Model The model divides the entities in an information system into two parts: Subject: the entity performing the operations, suchWeChat Official AccountSecurityLinuxiptablesSELinuxOperationshttps://www.chenshaowen.com/en/blog/learning-flask-1.htmlThu, 20 Sep 2018 00:00:00 +0000Thu, 20 Sep 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/learning-flask-1.html1. Introduction Flask was born in 2010 and is a lightweight Web application framework written in Python. Flask depends on two libraries. Jinja2, a template engine similar to mako Werkzeug, a Python library that follows the WSGI protocol and implements much of what sits underneath a Web framework, such as the request and response objects.WeChat Official AccountFlaskWebPythonBackendLearninghttps://www.chenshaowen.com/en/blog/how-to-package-a-docker-image-of-robot-framework.htmlTue, 18 Sep 2018 00:00:00 +0000Tue, 18 Sep 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-package-a-docker-image-of-robot-framework.htmlTo make it easier for CI to integrate UI automation tests, the Robot Framework runtime needs to be packaged as a Docker image. This post covers the configuration and scripts involved in the packaging process. 1. Packaging Directory Structure 1 2 3 4 5 6 tree . ├──WeChat Official AccountDockerRobotFrameworkPythonTestingCICDOperationshttps://www.chenshaowen.com/en/blog/ci-script-of-how-to-use-cdn-to-separate-frontend-and-backend-more.htmlSat, 15 Sep 2018 00:00:00 +0000Sat, 15 Sep 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/ci-script-of-how-to-use-cdn-to-separate-frontend-and-backend-more.htmlIn a team, any change to the development workflow must be paired with the corresponding automation tooling. Without a low enough cost of use, the change is meaningless, because nobody will actually use it. In the previous post we covered how to use a CDN for further frontend/backend separation; thisWeChat Official AccountCICDNOptimizationDeploymentPaaSScriptingDevOpsR&Dhttps://www.chenshaowen.com/en/blog/how-to-use-cdn-to-separate-frontend-and-backend-more.htmlThu, 13 Sep 2018 00:00:00 +0000Thu, 13 Sep 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-use-cdn-to-separate-frontend-and-backend-more.htmlI have been optimizing a project recently. After mapping out the whole chain, I started improving it step by step and found many points that could be improved. What follows are some thoughts on the development model and the deployment approach. I hope they give you some inspiration. 1. Development Background 1.WeChat Official AccountCICDNOptimizationDeploymentPaaSFrontendBackendR&Dhttps://www.chenshaowen.com/en/blog/some-installation-problems-about-nodejs.htmlMon, 10 Sep 2018 00:00:00 +0000Mon, 10 Sep 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/some-installation-problems-about-nodejs.html1. Installing node-sass When installing node-sass, the node scripts/install stage downloads a .node file from github.com, and network problems often cause this to fail. You can add an .npmrc file inside the project to configure a proxy for the installation: 1 2 sass_binary_site=https://npm.taobao.org/mirrors/node-sass/ registry=https://registry.npm.taobao.org Or install with cnpm: 1 2 npm install -g cnpm --registry=https://registry.WeChat Official AccountFrontendWebpackNodejsR&DEnvironmenthttps://www.chenshaowen.com/en/blog/install-python3-and-go-kernel-in-jupyter.htmlSun, 09 Sep 2018 00:00:00 +0000Sun, 09 Sep 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/install-python3-and-go-kernel-in-jupyter.htmlEarlier we mentioned that Jupyter Notebook is an interactive notebook that supports running more than 40 programming languages and is very well suited to teaching. Recently, while learning the Go language, I thought of Jupyter. This article mainly describes how to install the Python3 and Go kernels inside Jupyter. 1.WeChat Official AccountJupyterGoPython3LearningR&Dhttps://www.chenshaowen.com/en/blog/interesting-games.htmlTue, 04 Sep 2018 00:00:00 +0000Tue, 04 Sep 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/interesting-games.htmlFruit Ninja Newton’s Cradle Talking TomWeChat Official AccountDemoGamesFrontendJavaScriptEntertainmenthttps://www.chenshaowen.com/en/blog/month-share-1.htmlSat, 01 Sep 2018 00:00:00 +0000Sat, 01 Sep 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/month-share-1.htmlSharing with you some of the tech news I have been following over the past month. There is also some interesting content here too: tools, learning resources, tips, and more. Frontend Webpack Back in February this year, Webpack had already released version 4.0.0. Webpack 4 no longer supports Node.js 4,WeChat Official AccountMonthly ShareTech NewsSharingR&Dhttps://www.chenshaowen.com/en/blog/book/growth-hacker.htmlTue, 21 Aug 2018 08:47:38 +0000Tue, 21 Aug 2018 08:47:38 +0000https://www.chenshaowen.com/en/blog/book/growth-hacker.htmlAuthor: Fan Bing Subtitle: A Secret Playbook for User and Revenue Growth in Startups Publication year: 2015-7 ISBN: 9787121263606 Notes: This is a fairly marketing-oriented book aimed at ToC internet products. Through all kinds of real cases, it shows you the playbook of internet products: how to create a product, acquire users, spark activity, improve retention, increase revenue, and spread virally.WeChat Official AccountBooksUsersGrowthhttps://www.chenshaowen.com/en/blog/how-to-use-cloudready-make-chrome-os-boot-disk.htmlSat, 18 Aug 2018 00:00:00 +0000Sat, 18 Aug 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-use-cloudready-make-chrome-os-boot-disk.html1. Download the Image File Click here, and from the CloudReady website download the DOWNLOAD 64-BIT or DOWNLOAD 32-BIT image. After extracting it, you get a cloudready-free-xxx-bit.bin file. 2. Install the Chromebook Recovery Utility First install the Chrome browser, then open this page and install the Chromebook Recovery Utility. 3. Create the Image Step one, open the Chromebook Recovery Utility.WeChat Official AccountChromeOSCloudReadySystemsOperationsLearningHardwareBoot Diskhttps://www.chenshaowen.com/en/blog/senior-of-robot-framework-2.htmlThu, 16 Aug 2018 00:00:00 +0000Thu, 16 Aug 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/senior-of-robot-framework-2.htmlSettings Table The settings table in Robot Framework mainly serves two purposes. For a detailed explanation, see here. Import test, resource, and variable files. Define metadata. In the Setting table: 1 2 3 4 5 Library 引入库 Resource 引入资源文件 Variables 引入变WeChat Official AccountRobotFrameworkPythonTestingAutomationDevOpshttps://www.chenshaowen.com/en/blog/robot-framework-common-library.htmlThu, 16 Aug 2018 00:00:00 +0000Thu, 16 Aug 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/robot-framework-common-library.html1. Standard Libraries The libraries that Robot Framework can import and use directly, found in the directory python\Lib\site-packages\robot\libraries. For details, see the documentation. Builtin, contains the keywords that are needed most often. It is imported automatically without an import, so it is always available. Dialogs, provides ways to pause test execution and to take input from the user.WeChat Official AccountTestingAutomationDevOpsRobotFrameworkR&DLearninghttps://www.chenshaowen.com/en/blog/senior-of-robot-framework-1.htmlWed, 15 Aug 2018 00:00:00 +0000Wed, 15 Aug 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/senior-of-robot-framework-1.htmlpybot Command Run all test cases 1 pybot . Run a certain test suite 1 pybot testsuite.txt Run a test case in a certain test suite 1 pybot --test case_name testsuit.txt Output the test results to a fixed path 1 pybot --ouputdir your_ouput_dir testsuit.txt Run test cases that contain aWeChat Official AccountRobotFrameworkPythonTestingAutomationDevOpshttps://www.chenshaowen.com/en/blog/correct-way-to-use-vim-on-windows.htmlSun, 12 Aug 2018 00:00:00 +0000Sun, 12 Aug 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/correct-way-to-use-vim-on-windows.htmlI had used Cygwin before, and Git Bash too. But neither of them works well with Vim, so I finally settled on MSYS2 as my console. 1. Comparing Cygwin, MinGW, and MSYS2 On Windows, the key to using Vim is finding a suitable console. When installing Vim plugins, a suitableWeChat Official AccountVimToolsWindowsDevelopmenthttps://www.chenshaowen.com/en/blog/vim-necessary-plugin.htmlTue, 07 Aug 2018 00:00:00 +0000Tue, 07 Aug 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/vim-necessary-plugin.html1. nerdtree — Generate a File Tree 1 2 " 文件目录树 Plugin 'scrooloose/nerdtree' 1 2 3 4 5 6 7 8 9 10 11 12 13 "===================================================== "" NERDTree 配置 "===================================================== let NERDTreeChDirMode=1 "显示书签&#3WeChat Official AccountVimPluginToolsDevelopmentR&DLearningEditorEfficiencyhttps://www.chenshaowen.com/en/blog/vim-installation-and-configuration-on-windows-os.htmlMon, 06 Aug 2018 00:00:00 +0000Mon, 06 Aug 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/vim-installation-and-configuration-on-windows-os.html1. Introduction to Vim Vim is an editor that evolved from vi; its first version was released by Bram Moolenaar in 1991. The original name was Vi IMitation, and as features accumulated it was officially renamed Vi IMproved. The image above shows the learning curves of common editors. For mostWeChat Official AccountVimToolsWindowsR&Dhttps://www.chenshaowen.com/en/blog/how-to-create-gitlab-label-using-python-gitlab.htmlSun, 05 Aug 2018 10:00:00 +0000Sun, 05 Aug 2018 10:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-create-gitlab-label-using-python-gitlab.htmlUsing GitLab issues for project management is a very Dev-friendly thing. Entering issues, opening Merge Requests, creating milestones — these all advance dynamically during development. But creating a pile of Labels for every new project is a headache. This article is mainly about solving that problem. 1. GitLab Label BeforeWeChat Official AccountGitLabPythonProject ManagementDevOpsR&Dhttps://www.chenshaowen.com/en/blog/jenkins-integrated-robot-framework-testing.htmlSun, 05 Aug 2018 00:00:00 +0000Sun, 05 Aug 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/jenkins-integrated-robot-framework-testing.html1. Installing a Headless Browser 1.1 Installing PhantomJS on CentOS Download and extract Visit Phantomjs, find the download link for Download phantomjs-2.1.1-linux-x86_64.tar.bz2, and copy it. Run the following commands on CentOS: 1 2 3 4 5 wget https://bitbucket.org/ariya/phantomjs/downloads/phantomjs-2.1.1-linux-x86_64.tar.bz2 # 如果没有安装 bzip2 可能会报WeChat Official AccountRobotFrameworkAutomationContinuous IntegrationTestingDevOpsJenkinsR&Dhttps://www.chenshaowen.com/en/blog/how-to-configure-the-hhkb-keyboard.htmlSat, 28 Jul 2018 00:00:00 +0000Sat, 28 Jul 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-configure-the-hhkb-keyboard.html1. About Keyboard Types There are mainly three types of keyboards: Mechanical keyboard Every key on a mechanical keyboard has its own separate switch, also called a “switch” (axis). Classified by the micro switch type, mechanical keyboards are divided into brown, blue, white, black, and red switches. Mechanical keyboards feel comfortable to type on, but they are on the expensive side.WeChat Official AccountPeripheralsKeyboardToolsLifehttps://www.chenshaowen.com/en/blog/no-tag-no-chance.htmlThu, 26 Jul 2018 00:00:00 +0000Thu, 26 Jul 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/no-tag-no-chance.html1. Everyone Is Equal Before the Machine Every day we take in a huge amount of digital information, but very little of it actually helps us. Most of it is like junk food: it provides no nourishment, yet it takes up capacity, leaving you unable to dig into the truth and form your own thinking and judgment.WeChat Official AccountThoughtsInfluenceGrowthPersonal Brandhttps://www.chenshaowen.com/en/blog/basic-of-robot-framework.htmlTue, 24 Jul 2018 00:00:00 +0000Tue, 24 Jul 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/basic-of-robot-framework.htmlThis post mainly introduces some basic concepts of Robot Framework and how to write a test case. 1. Basic Elements 1.1 Keywords Robot Framework keywords are similar to functions. They are divided into system keywords and user-defined keywords. System keywords, imported by loading a Library User keywords, imported by loadingWeChat Official AccountPythonTestingRobotFrameworkDevOpsLearningAutomated TestingR&DCICDhttps://www.chenshaowen.com/en/blog/how-to-install-robot-framework-ide-on-windows.htmlSat, 21 Jul 2018 00:00:00 +0000Sat, 21 Jul 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-install-robot-framework-ide-on-windows.html1. Introduction 1.1 Robot Framework Robot Framework is an automated functional testing framework written in Python. It is highly extensible and supports keyword-driven testing It can test several types of clients or interfaces at the same time It can run distributed tests It is mainly used for acceptance testing withWeChat Official AccountRobotFrameworkTestingPythonRIDEAutomationR&Dhttps://www.chenshaowen.com/en/blog/some-common-scripts-in-ci.htmlTue, 17 Jul 2018 00:00:00 +0000Tue, 17 Jul 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/some-common-scripts-in-ci.htmlBecause I am responsible for the team’s shared CI matters, I often need to configure CI pipelines or help others solve problems, so I put together the CI scripts I use most often for easy reference. 1. Structure of .gitlab-ci.yml Below is the structure of a GitLab CI configuration file.WeChat Official AccountGitLabGitLab-CICIDevOpsContinuous IntegrationR&Dhttps://www.chenshaowen.com/en/blog/book/love-and-math.htmlSun, 15 Jul 2018 08:47:38 +0000Sun, 15 Jul 2018 08:47:38 +0000https://www.chenshaowen.com/en/blog/book/love-and-math.htmlAuthor: Edward Frenkel (USA) Original title: Love and Math: The Heart of Hidden Reality Publication year: 2016-3 ISBN: 9787508658070 Notes: I finished this book on the high-speed train during a round trip. It is a fairly light, autobiographical kind of mathematics book. It mainly follows the timeline of the author’s schooling, and tells the story of how a boy interested in physics, step by step and through unremitting effort, became a world-class mathematician.WeChat Official AccountBooksMathhttps://www.chenshaowen.com/en/blog/h-function-in-vue.htmlSun, 08 Jul 2018 00:00:00 +0000Sun, 08 Jul 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/h-function-in-vue.html1. The createElement Method The common DOM operations in HTML are modifying DOM nodes and accessing DOM nodes. Beyond those, the W3C has also published technical standards for creating and deleting DOM nodes. The createElement method is used to create a DOM node. createElement() usually needs to be used togetherWeChat Official AccountFrontendVueFunctionsR&Dhttps://www.chenshaowen.com/en/blog/how-to-raise-a-good-question.htmlSat, 07 Jul 2018 00:00:00 +0000Sat, 07 Jul 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-raise-a-good-question.html1. Motivation for Asking Questions There are two kinds of motivation for asking questions: passive and active. 2 Passive Passive questioning is the forced action you take when you run into a problem, search for a solution, and come up empty. 2.1 Try to Solve It Yourself When you hit a problem, do not rush to ask others.WeChat Official AccountThoughtsQuestionsCommunicationLearningGrowthhttps://www.chenshaowen.com/en/blog/how-to-secure-get-request-in-django.htmlTue, 26 Jun 2018 00:00:00 +0000Tue, 26 Jun 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-secure-get-request-in-django.htmlIn Django, request holds all the information for a single request. Backend processing logic frequently needs information from request. For example, the DRF framework may want to be able to obtain request at any time, or pass some parameters globally. There are third-party Django apps with tools that can satisfyWeChat Official AccountDjangoPythonR&DTipsLearningDRFThread Safetyhttps://www.chenshaowen.com/en/blog/using-axios-in-vue.htmlMon, 25 Jun 2018 15:00:00 +0000Mon, 25 Jun 2018 15:00:00 +0000https://www.chenshaowen.com/en/blog/using-axios-in-vue.html1. Installing axios Install with npm 1 npm install axios --save There are two ways to register it globally: Bind it to the prototype 1 2 import axios from "axios"; Vue.prototype.axios = axios; With this approach, every Vue object gets a new axios object. 1 2 3 this.axios.post(apiUrl).then((res) => {WeChat Official AccountFrontendInterfaceAPIVuejsAxiosR&Dhttps://www.chenshaowen.com/en/blog/how-to-ensure-concurrence-of-data-in-django.htmlMon, 25 Jun 2018 12:00:00 +0000Mon, 25 Jun 2018 12:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-ensure-concurrence-of-data-in-django.htmlThe first part mainly covers optimistic and pessimistic locking. Locking guarantees data consistency under concurrency at the database level. Understanding locks helps you understand concurrency solutions. The second part mainly covers how to guarantee data consistency in concurrent scenarios within Django. 1. About Locks 1.1 Optimistic Locking The starting pointWeChat Official AccountDjangoPythonConcurrencyDatabaseR&Dhttps://www.chenshaowen.com/en/blog/using-utf8mb4-in-django-to-support-emoji-expression.htmlMon, 25 Jun 2018 00:00:00 +0000Mon, 25 Jun 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-utf8mb4-in-django-to-support-emoji-expression.html1. What Is utf8 In theory, utf8 uses 1-6 characters, In practice, the latest utf8 specification uses only one to four bytes, and can encode at most 21 bits, which is exactly enough to represent all 17 Unicode planes. 2. What Is utf8mb4 utf8mb4 is a superset of utf8. In theory, if you were using utf8 before and then change the charset to utf8mb4, it will not cause any problem when reading the existing utf8-encoded data.WeChat Official AccountDjangoPythonDatabaseCharsetEncodingR&Dhttps://www.chenshaowen.com/en/blog/common-commands-list-of-npm.htmlSun, 24 Jun 2018 22:00:00 +0000Sun, 24 Jun 2018 22:00:00 +0000https://www.chenshaowen.com/en/blog/common-commands-list-of-npm.html** By default, a command operates on the current directory environment. Adding the -g flag makes it apply globally instead.** Upgrade npm 1 npm install npm -g Show the current version 1 npm -v Show the npm registry configuration 1 npm config get registry Change the npm registry 1 2 npm config set registry http://registry.WeChat Official AccountCommandsFrontendNPMNode.jsR&DLearninghttps://www.chenshaowen.com/en/blog/note-of-strategic-thinking-and-expression.htmlSun, 24 Jun 2018 00:00:00 +0000Sun, 24 Jun 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/note-of-strategic-thinking-and-expression.htmlThis mainly records analysis methods. 1. Strategic Analysis McKinsey’s Seven-Step Problem-Solving Method State the problem Analyze the problem Prioritize Lay out a detailed work plan Perform key analyses Synthesize results and build a structured conclusion Tell the story 2. Recognizing the Problem Most problems are a failure to recognize the problem — unknown unknowns.WeChat Official AccountNotesExpressionThoughtsLearningManagementMethodshttps://www.chenshaowen.com/en/blog/gitlab-ci-configuring-runner.htmlTue, 08 May 2018 00:00:00 +0000Tue, 08 May 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/gitlab-ci-configuring-runner.html1. Downloading and Configuring the Runner 1.1 Downloading the Runner The GitLab CI Runner is a program package written in Go; you can download it from the official site to your local machine. 1 2 yum install -y wget wget -O /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64 Add execute permission 1 chmod +x /usr/local/bin/gitlab-runner 1.WeChat Official AccountGitLabGitLab-CICIRunnerDevOpsOperationshttps://www.chenshaowen.com/en/blog/how-to-synchronize-the-backup-of-git-repository.htmlThu, 19 Apr 2018 00:00:00 +0000Thu, 19 Apr 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-synchronize-the-backup-of-git-repository.htmlBecause the Git repository currently in use is about to become unavailable, the repository code needs to be migrated to a new Git repository. The specific procedure is as follows: Log in to the new repository and create a repository. After creation, the new repository address is: http://gitlab.your-new-domain.com/project.git Clone the old repository as a mirror locally: 1 2 3 git clone --mirror http://gitlab.WeChat Official AccountGitBackupMigrationR&DOperationshttps://www.chenshaowen.com/en/blog/nginx-configuration-problem-record.htmlWed, 18 Apr 2018 00:00:00 +0000Wed, 18 Apr 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/nginx-configuration-problem-record.html1. server_name Has No Effect Symptom: Nginx reverse-proxies two applications, and the configuration details are as follows. It turned out that requests to b.chenshaowen.com and a.chenshaowen.com both returned responses from service A. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 serverWeChat Official AccountNginxConfigurationTroubleshootingOperationsLearningReverse ProxyBest PracticesWebhttps://www.chenshaowen.com/en/blog/introduction-of-various-environmental-isolating-tools-under-python.htmlTue, 17 Apr 2018 00:00:00 +0000Tue, 17 Apr 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/introduction-of-various-environmental-isolating-tools-under-python.html1. Non-standard Library 1.1 virtualenv virtualenv is a very popular tool for creating isolated python libraries environments. I strongly recommend you learn about it, because it is very practical and widely used; many people use it to set up python development environments. The other tools below are mainly compared against virtualenv to illustrate the differences.WeChat Official AccountPythonToolsEnvironmentvirtualenvR&DLearningpyenvCondahttps://www.chenshaowen.com/en/blog/how-to-use-docker-to-help-individual-projects.htmlMon, 16 Apr 2018 00:00:00 +0000Mon, 16 Apr 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-use-docker-to-help-individual-projects.html1. About Personal Projects Why emphasize that they are personal projects? Commercial projects have lengthy processes around code hosting, development, operations, and deployment. For an individual developer, such processes are too expensive and work against rapid project iteration. A personal project might be a Demo for learning. Building a DemoWeChat Official AccountDockerGitR&DOperationsLearningBest PracticesPersonal Projectshttps://www.chenshaowen.com/en/blog/based-on-git-development-workflow-of-front-and-back.htmlSat, 31 Mar 2018 00:00:00 +0000Sat, 31 Mar 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/based-on-git-development-workflow-of-front-and-back.htmlRecently I took part in a project developed collaboratively by several people, and ran into quite a few problems along the way. For example: a frontend commit overwrote backend code, conflicts were left unresolved and turned into rework, someone committed to branch B, or committed straight to the Master branch.WeChat Official AccountGitEfficiencyCollaborationCIDevOpsR&Dhttps://www.chenshaowen.com/en/blog/common-commands-in-django-development.htmlThu, 22 Mar 2018 00:00:00 +0000Thu, 22 Mar 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/common-commands-in-django-development.htmlCreating a Django Project 1 django-admin startproject your_project_name Creating an App 1 2 3 django-admin.py startapp your_app_name # 或者 python manage.py startapp your_app_name python manage.py and django-admin do essentially the same thing. The difference is that python manage.py also sets the DJANGO_SETTINGS_MODULE environment variable and adds the project’s pathWeChat Official AccountPythonDjangoCommandsR&Dhttps://www.chenshaowen.com/en/blog/dictionary-and-list-structure-in-python2.htmlMon, 12 Mar 2018 11:22:55 +0000Mon, 12 Mar 2018 11:22:55 +0000https://www.chenshaowen.com/en/blog/dictionary-and-list-structure-in-python2.html1. Why Dictionary Lookups Are Faster Than List Lookups First, take a look at the following piece of code. 1 2 3 4 5 6 7 8 9 10 11 12 13 from time import time t = time() data = [chr(i) for i in range(97, 123)] # data =WeChat Official AccountPythonSource CodeLearningR&Dhttps://www.chenshaowen.com/en/blog/python2-vs-python3.htmlFri, 09 Mar 2018 17:22:45 +0000Fri, 09 Mar 2018 17:22:45 +0000https://www.chenshaowen.com/en/blog/python2-vs-python3.html1. Differences Between Python 2 and 3 Feature \ Version Python 2 Python 3 print as a function print"abc" print(“abc”) Unified class Old-style and new-style classes New-style classes only Floating-point division 1/2=0 1/2=0.5 String formatting %, Format Format,% xrange replaces range xrange range long renamed to int Long,int Int Package imports Relative imports Absolute imports Source file encoding Ascii utf8 The official Python recommendation is strongly in favor of learning Python 3 directly, since Python 2 is only maintained through 2020.WeChat Official AccountPythonComparisonR&DLearningMigrationhttps://www.chenshaowen.com/en/blog/compiling-python2-source-code-under-windows.htmlFri, 09 Mar 2018 11:22:55 +0000Fri, 09 Mar 2018 11:22:55 +0000https://www.chenshaowen.com/en/blog/compiling-python2-source-code-under-windows.html1. Compiler Preparation Prepare the Python source code From the official Python website, download the source code for the Python version you need to compile. The one chosen here is the latest Python 2 release, Python-2.7.14.tar.xz, click here to go there. Prepare a compiler On Windows, the project files builtWeChat Official AccountPythonSource CodeWindowsLearningCompilationR&Dhttps://www.chenshaowen.com/en/blog/python2-source-of-pyc.htmlFri, 09 Mar 2018 00:00:00 +0000Fri, 09 Mar 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/python2-source-of-pyc.htmlThis article uses the Python 2.7.8 source code as an example. 1. Common File Formats in Python py file A Python source code file, which can be modified with a text editor. pyc file The bytecode file generated after Python source code is compiled. pyw file When a pyc file is executed, a console window appears; when a pyw file is executed, it does not.WeChat Official AccountPythonSource CodeCompilationLearningR&Dhttps://www.chenshaowen.com/en/blog/notes-on-the-art-of-writing-readable-code.htmlThu, 08 Mar 2018 17:22:45 +0000Thu, 08 Mar 2018 17:22:45 +0000https://www.chenshaowen.com/en/blog/notes-on-the-art-of-writing-readable-code.html1. About Names Choose professional words Avoid overly generic names Use concrete names instead of abstract ones Use prefixes or suffixes to attach more information to a name Decide how long a name should be Use the formatting of a name to express meaning 2. Packing Information into Names Generally speaking, adding words like is, has, can, or should makes a boolean value more explicit.WeChat Official AccountBooksConventionsR&DCodeLearninghttps://www.chenshaowen.com/en/blog/notes-on-making-strategy-of-the-presentation.htmlTue, 06 Mar 2018 20:55:29 +0000Tue, 06 Mar 2018 20:55:29 +0000https://www.chenshaowen.com/en/blog/notes-on-making-strategy-of-the-presentation.html1. Overview Course prerequisite: the logical structure and content of the PowerPoint have already been worked out and written. The course enhances the expressiveness of PowerPoint mainly through the following six parts: Color Text Shapes Charts Slide Master Animation 2. Color 2.1 Understanding Color Hue Chromatic colors, such as purple,WeChat Official AccountPPTToolsLearningSharinghttps://www.chenshaowen.com/en/blog/how-to-build-a-pypi-server-using-devpi.htmlSun, 04 Mar 2018 00:00:00 +0000Sun, 04 Mar 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-build-a-pypi-server-using-devpi.htmlWhen you install Python packages with pip, it looks up the corresponding package on the https://pypi.python.org/simple/ index by default, then downloads and installs it. But in an intranet environment, or when you need to publish some private packages to designated users, you have to build your own PyPI Server. This post mainly records the process of building a PyPI Server index with the devpi tool, along with some commonly used commands.WeChat Official AccountPythondevpiToolsOperationsLearningPyPIPrivate IndexPackage ManagementIntranethttps://www.chenshaowen.com/en/blog/django-model-and-er-diagram.htmlFri, 02 Mar 2018 00:00:00 +0000Fri, 02 Mar 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/django-model-and-er-diagram.htmlAs requirements are gradually confirmed, the system’s DB model is gradually confirmed as well. Sometimes you use a tool like MySQL Workbench to draw an E-R diagram and then generate MySQL database tables, and at that point you need to reverse-generate Django Models; other times you write the Django Models directly, but then you need to look at the E-R diagram.WeChat Official AccountDjangoPythonDatabaseDesignModelR&Dhttps://www.chenshaowen.com/en/blog/service-discovery-etcd-confd-nginx.htmlThu, 08 Feb 2018 14:51:16 +0000Thu, 08 Feb 2018 14:51:16 +0000https://www.chenshaowen.com/en/blog/service-discovery-etcd-confd-nginx.htmlThis post mainly explains why the service discovery feature is needed and compares several service discovery tools. It also puts service discovery with Etcd, Confd, and Nginx into practice on CentOS. 1. Service Registration and Discovery 1.1 Why Service Registration and Discovery Are Needed With the rise of microservices, aWeChat Official AccountToolsMicroservicesEtcdConfdNginxService Discoveryhttps://www.chenshaowen.com/en/blog/monitor-about-prometheus.htmlWed, 07 Feb 2018 00:00:00 +0000Wed, 07 Feb 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/monitor-about-prometheus.htmlInternet services often depend on tens of thousands of hosts, and those hosts are deployed all over the world, on top of a tangled mess of user environments. To keep a service stable and reliable, you need to monitor it. The goals of monitoring can be considered from several angles: from the system dimension, understand CPU, memory, and disk usage; from the network monitoring dimension, understand upstream and downstream rates; from the service monitoring dimension, understand the health and availability of the application.WeChat Official AccountPrometheusGrafanaDockerMonitoringOperationshttps://www.chenshaowen.com/en/blog/elk-log-search-practice.htmlSat, 03 Feb 2018 00:00:00 +0000Sat, 03 Feb 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/elk-log-search-practice.htmlThis article gives a brief introduction to the ELK stack and provides a Docker compose orchestration configuration. After reading it, you can get ELK running locally with Docker. A later article will cover deploying ELK on servers and adding the related configuration. 1. Introduction to the ELK Stack ELK is not actually a single piece of software but a whole solution — an acronym formed from the initials of three software products: Elasticsearch, Logstash, and Kibana.WeChat Official AccountELKToolsLoggingSearchElasticsearchOperationshttps://www.chenshaowen.com/en/blog/how-to-do-codereview-better.htmlTue, 30 Jan 2018 00:00:00 +0000Tue, 30 Jan 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-do-codereview-better.html1. How to Do Code Review Better Requirement Items Requirements for the committer Small granularity Atomicity With comments Tool checks Requirements for the reviewer First conventions Second logic & performance Third design Requirements for the team Reviewing the old to learn the new 2. Define a Code Review Checklist Requirement Items Naming Variable namesWeChat Official AccountQualityNotesR&DTeamsCodeReviewLearningBest Practiceshttps://www.chenshaowen.com/en/blog/development-specification-1-of-django.htmlSat, 27 Jan 2018 00:00:00 +0000Sat, 27 Jan 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/development-specification-1-of-django.htmlThis article mainly lays out a few things to watch out for during Django development. A consistent coding style and sound design principles help a project’s development and maintenance, and are worth developers studying and discussing continuously. 1. Encoding Declaration When the Python interpreter executes code, it needs to beWeChat Official AccountDjangoPythonConventionsR&DLearningBest PracticesCode StyleArchitecturehttps://www.chenshaowen.com/en/blog/component-list-of-front-library.htmlFri, 19 Jan 2018 00:00:00 +0000Fri, 19 Jan 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/component-list-of-front-library.htmltoastr Toast notification popups artDialog Dialog component DataTables A jQuery-based table plugin KendoUI A very complete UI framework, with many components wangEditor Web rich text editor jstree Web tree structure component highcharts echarts D3js Data charts select2 Dropdown select validate jQuery-based form validation Vuejs Can be used for two-way data binding, and also for building single-page applicationsWeChat Official AccountFrontendJavaScriptComponentsR&DLearningLibrarieshttps://www.chenshaowen.com/en/blog/toolbox.htmlTue, 16 Jan 2018 00:00:00 +0000Tue, 16 Jan 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/toolbox.html1. Documentation Pandoc Pandoc is a text conversion tool written in Haskell. It is fast and supports a wide range of formats, converting Markdown, LaTeX, and others into HTML, Docs, PDF, and more. All it takes is a simple command: 1 pandoc mypaper.md -o mypaper.docx Graphviz An open-source toolkit developedWeChat Official AccountToolsEfficiencyR&DLearningSharinghttps://www.chenshaowen.com/en/blog/rate-limiting-of-api.htmlThu, 11 Jan 2018 00:00:00 +0000Thu, 11 Jan 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/rate-limiting-of-api.html1. Flow Control Caching, degradation, and rate limiting are common methods for protecting high-concurrency systems. Caching trades space for time and cuts the time spent on CPU and network calls; degradation protects the high availability of core services by delaying or refusing to handle non-core requests during peak periods; rateWeChat Official AccountAPIMicroservicesR&DArchitectureRate Limitinghttps://www.chenshaowen.com/en/blog/101-of-go-language.htmlSun, 07 Jan 2018 00:00:00 +0000Sun, 07 Jan 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/101-of-go-language.html1. Background For the past decade or so, C/C++ has not developed well in the computing field and has not kept pace with the advance of computers. With existing programming languages, development speed and system efficiency often cannot be had at the same time. Either execution is fast but development and compilation are inefficient, or execution is slow but compilation is efficient.WeChat Official AccountGoProgrammingGetting StartedLearninghttps://www.chenshaowen.com/en/blog/process-and-thread-in-python.htmlSat, 06 Jan 2018 11:22:55 +0000Sat, 06 Jan 2018 11:22:55 +0000https://www.chenshaowen.com/en/blog/process-and-thread-in-python.html1. Processes A process is an instance of a running program, and it is the most basic unit by which the kernel allocates resources. A process has its own independent heap and stack, its own address space, and its own resource handles. Processes are scheduled by the OS, the schedulingWeChat Official AccountPythonConcurrencyProcessesThreadsCoroutinesR&DLearninghttps://www.chenshaowen.com/en/blog/stateless-service.htmlWed, 03 Jan 2018 17:22:45 +0000Wed, 03 Jan 2018 17:22:45 +0000https://www.chenshaowen.com/en/blog/stateless-service.html1. Problem Description Background: a SaaS application developed with Django, offering document services externally. Its search feature is implemented through Django Haystack. Problem: the search feature is sometimes available and sometimes not. After repeated testing, the available and unavailable states were found to alternate, each occurring with a probability ofWeChat Official AccountDjangoServicesPaaSSaaShttps://www.chenshaowen.com/en/blog/content-type-http-header.htmlSat, 30 Dec 2017 00:00:00 +0000Sat, 30 Dec 2017 00:00:00 +0000https://www.chenshaowen.com/en/blog/content-type-http-header.html1. HTTP Header The HTTP protocol is an application-layer specification built on top of TCP/IP, transmitted as ASCII text. The HTTP specification divides an HTTP request into three parts: the status line, the request headers, and the message body. Something like this: 1 2 3 4 5 <method><request-URL><version> <headers> <entity-body> An HTTP header falls into four categories: general headers, request headers, response headers, and entity headers.WeChat Official AccountHTTPInterfaceBackendProtocolR&Dhttps://www.chenshaowen.com/en/blog/class-function-method-of-python.htmlFri, 29 Dec 2017 00:00:00 +0000Fri, 29 Dec 2017 00:00:00 +0000https://www.chenshaowen.com/en/blog/class-function-method-of-python.html1. Function Methods A function is an organized, reusable snippet of code used to implement a single or related piece of functionality. Functions improve an application’s modularity and the reuse rate of code. Python provides many built-in functions, such as print() and str(). At the same time, Python also allowsWeChat Official AccountPythonConventionsFunctionsR&Dhttps://www.chenshaowen.com/en/blog/interactive-notebook-jupyter.htmlMon, 25 Dec 2017 17:22:45 +0000Mon, 25 Dec 2017 17:22:45 +0000https://www.chenshaowen.com/en/blog/interactive-notebook-jupyter.html1. Introduction Jupyter Notebook (formerly IPython notebook) is an interactive notebook that supports running more than 40 programming languages. Jupyter Notebook is in fact a web application that lets you create and share program documents, with support for live code, mathematical equations, visualizations, and Markdown. Its uses include data cleaningWeChat Official AccountToolsPythonNotesJupyterLearningR&DData AnalysisMachine LearningVisualizationhttps://www.chenshaowen.com/en/blog/message-queue-about-rabbitmq.htmlSat, 23 Dec 2017 20:51:22 +0000Sat, 23 Dec 2017 20:51:22 +0000https://www.chenshaowen.com/en/blog/message-queue-about-rabbitmq.html1. Where a Message Queue Fits 1.1 Asynchronous Processing Use case: after a user registers, a registration email and a registration SMS must be sent. With the synchronous approach, system performance (concurrency, throughput, response time) hits a bottleneck. 1.2 Application Decoupling Use case: after a user places an order, theWeChat Official AccountRabbitMQMiddlewareCeleryPythonMessageQueuehttps://www.chenshaowen.com/en/blog/common-git-commands.htmlMon, 18 Dec 2017 11:22:55 +0000Mon, 18 Dec 2017 11:22:55 +0000https://www.chenshaowen.com/en/blog/common-git-commands.html1. Basic Concepts Workspace: the working area, the project files Index: the staging area, also called the pending-commit area; before a commit enters the repo, all updates are placed in the staging area Local Repository: the local repo, the version repository kept locally; HEAD points at the current development branchWeChat Official AccountGitToolsVersion ControlCommand LineR&Dhttps://www.chenshaowen.com/en/presentation/Sat, 16 Dec 2017 17:42:04 +0000Sat, 16 Dec 2017 17:42:04 +0000https://www.chenshaowen.com/en/presentation/Chaos Engineering and Practice University Training Week 2 | DevOps University Training Week 1 | Containers and Cloud Native China DevOps Community | A CICD Platform Based on Tekton KubeSphere Open Source: Thoughts on R&D Process DevOps Platform Under Cloud Native 2020 Cloud Native Industry Conference | Building an Enterprise DevOps Platform Across ClustersWeChat Official Accounthttps://www.chenshaowen.com/en/blog/how-to-migrating-blogs-to-hexo.htmlSun, 10 Dec 2017 11:22:55 +0000Sun, 10 Dec 2017 11:22:55 +0000https://www.chenshaowen.com/en/blog/how-to-migrating-blogs-to-hexo.html1. Introduction to Hexo Hexo is a concise, efficient, open-source static blog generator built with Node.js. Hexo can render Markdown into static HTML files extremely quickly. Like Jekyll, Ghost, and Octopress, it can be used to create a blog. Hexo has the following advantages: It uses Node.js, so setting up the environment is simple.WeChat Official AccountToolsHexoBlogGhostOperationsMigrationhttps://www.chenshaowen.com/en/blog/api-interface-specification.htmlFri, 01 Dec 2017 00:00:00 +0000Fri, 01 Dec 2017 00:00:00 +0000https://www.chenshaowen.com/en/blog/api-interface-specification.htmlIn the process of Web application development, backend developers need to deliver API interfaces frequently, and frontend developers need to call API interfaces frequently. To lower communication costs and prevent possible security risks, and following the principle of convention over configuration, it is necessary to standardize the API specification. RestfulWeChat Official AccountDjangoInterfaceAPIR&DSpecificationhttps://www.chenshaowen.com/en/blog/front-webpack-practice-of-gitlab-ci.htmlWed, 08 Nov 2017 09:15:56 +0000Wed, 08 Nov 2017 09:15:56 +0000https://www.chenshaowen.com/en/blog/front-webpack-practice-of-gitlab-ci.htmlStarting with GitLab 8.0, GitLab began integrating CI (Continuous Integration) functionality. You only need to add a Runner on the server and a .gitlab-ci.yml file in the project, and you can run CI. In GitLab Setup and Configuration I recorded the whole process of setting up a GitLab service fromWeChat Official AccountFrontendContinuous IntegrationToolsDevOpsGitLabCICDhttps://www.chenshaowen.com/en/blog/error-code-design-and-unified-processing-in-django.htmlMon, 30 Oct 2017 21:50:33 +0000Mon, 30 Oct 2017 21:50:33 +0000https://www.chenshaowen.com/en/blog/error-code-design-and-unified-processing-in-django.htmlI currently use Django for SaaS development, and I develop and maintain several SaaS applications at the same time. Many SaaS applications have agreed-upon error codes, some used to handle login state, others to mark business logic status. For a feature that is so strongly shared across projects, it isWeChat Official AccountDjangoPythonError CodesMiddlewareR&DLearningAPIBest Practiceshttps://www.chenshaowen.com/en/blog/how-to-learn-a-new-programming-language-taking-python-as-an-example.htmlFri, 13 Oct 2017 23:00:06 +0000Fri, 13 Oct 2017 23:00:06 +0000https://www.chenshaowen.com/en/blog/how-to-learn-a-new-programming-language-taking-python-as-an-example.html1. Understand the Background of the New Language ABC was a teaching language designed specifically for non-professional programmers, but because it was closed, it did not succeed. During Christmas 1989, Guido van Rossum developed a new script interpreter and named it Python, as a kind of successor to the ABCWeChat Official AccountPythonLearningProgrammingR&Dhttps://www.chenshaowen.com/en/blog/how-to-pack-a-python-package-and-upload-it-to-pypi.htmlFri, 06 Oct 2017 20:50:41 +0000Fri, 06 Oct 2017 20:50:41 +0000https://www.chenshaowen.com/en/blog/how-to-pack-a-python-package-and-upload-it-to-pypi.htmlRecently I have been thinking about how to improve development efficiency. From web crawlers to data processing, and then to artificial intelligence, Python’s strength lies in having a large set of out-of-the-box toolkits, so you do not have to reinvent the wheel, which greatly improves development efficiency. So why not modularize the functional pieces of a project and package them as reusable Python packages?WeChat Official AccountPythonDjangoDeploymentPyPIR&DOpen SourceLearninghttps://www.chenshaowen.com/en/blog/django-debug-toolbar.htmlFri, 06 Oct 2017 10:45:58 +0000Fri, 06 Oct 2017 10:45:58 +0000https://www.chenshaowen.com/en/blog/django-debug-toolbar.html1. About Django’s Performance Django is a web application framework written in Python. With Django you can develop complex, data-driven websites very simply and efficiently. At the same time, Django pays close attention to component reusability, pluggability, agile development, and DRY (Don’t Repeat Yourself). To a certain extent, Django is an application framework very well suited to implementing business logic.WeChat Official AccountDjangoPythonToolsTestingPerformanceDebuggingR&Dhttps://www.chenshaowen.com/en/blog/control-of-static-file-version-in-django.htmlFri, 06 Oct 2017 09:58:43 +0000Fri, 06 Oct 2017 09:58:43 +0000https://www.chenshaowen.com/en/blog/control-of-static-file-version-in-django.htmlTo respond quickly to user needs and meet the requirements of marketing campaigns, internet products usually have a very high release frequency. Adopting agile development shortens the delivery cycle and speeds up product iteration, but it also brings challenges to a project’s file management. Frontend engineering faces users directly, bearsWeChat Official AccountDjangoFrontendJavaScriptFilesR&DVersionhttps://www.chenshaowen.com/en/blog/django-static-file-processing.htmlThu, 05 Oct 2017 15:42:27 +0000Thu, 05 Oct 2017 15:42:27 +0000https://www.chenshaowen.com/en/blog/django-static-file-processing.html1. Categories of Django Static Files Django static files fall into two categories: static and media. static: files referenced by pages, such as JS, CSS, and images media: files uploaded by users 2. Production Environment Configuration In production, you usually configure Nginx to forward static file requests while Django handles dynamic requests.WeChat Official AccountDjangoFrontendFilesPythonR&Dhttps://www.chenshaowen.com/en/blog/django-big-file-transfer.htmlTue, 26 Sep 2017 18:28:56 +0000Tue, 26 Sep 2017 18:28:56 +0000https://www.chenshaowen.com/en/blog/django-big-file-transfer.html1. Returning the File Directly If the static file is at media/test.zip in the project root, you first have to read the file into memory and then transfer it. The code looks like this: settings.py configuration 1 2 PROJECT_ROOT = os.path.dirname(os.path.abspath(__file__)) MEDIA_ROOT = os.path.join(PROJECT_ROOT, 'media/') yourapp/views.py 1 2 3 4 5 6 7 8 9 10 11 12 from django.WeChat Official AccountPythonDjangoLarge FilesNetworkR&Dhttps://www.chenshaowen.com/en/blog/build-django-development-environment-using-vagrant.htmlMon, 25 Sep 2017 21:56:05 +0000Mon, 25 Sep 2017 21:56:05 +0000https://www.chenshaowen.com/en/blog/build-django-development-environment-using-vagrant.htmlDocker solves the problem of environment isolation on the same machine and improves the efficiency of operations and deployment. Vagrant gives developers a unified, near-fully-isolated environment for development and testing. This article mainly discusses how to build a Django development environment with Vagrant. Versions: VirtualBox 5.0, Vagrant 1.8. 1. BasicWeChat Official AccountDjangoPythonVagrantToolsDevelopmentR&DOperationshttps://www.chenshaowen.com/en/blog/how-to-package-django-development-environments-using-docker-6.htmlSun, 24 Sep 2017 18:03:22 +0000Sun, 24 Sep 2017 18:03:22 +0000https://www.chenshaowen.com/en/blog/how-to-package-django-development-environments-using-docker-6.html1. Directory Structure The django-devops-uwsgi directory holds the image build files that package uWSGI and Django. It installs the necessary packages such as uWSGI, pip, and virtualenv into the image. When a container is created from the image, the start.sh script runs: it creates the Python virtual runtime environment, installs the dependency packages from Django’s requirements.WeChat Official AccountDockerDjangoPythonServicesToolsR&DOperationshttps://www.chenshaowen.com/en/blog/how-to-package-django-development-environments-using-docker-5.htmlSun, 24 Sep 2017 14:34:35 +0000Sun, 24 Sep 2017 14:34:35 +0000https://www.chenshaowen.com/en/blog/how-to-package-django-development-environments-using-docker-5.html1. Directory Structure 1 2 3 4 5 6 7 8 9 10 11 12 ├── django-devops-nginx │ ├── Dockerfile │ └── nginx.repo ├── django-devops-compose │ ├── docker-compose.yml │ ├── www │ ├── conf │ │ ├── nginx.conf │ ├── log │ │ ├── nginx │ │ │ ├── access.log │ │ │ ├── error.WeChat Official AccountDockerServicesToolsDjangoNginxPythonR&DOperationshttps://www.chenshaowen.com/en/blog/how-to-package-django-development-environments-using-docker-4.htmlSun, 24 Sep 2017 13:32:40 +0000Sun, 24 Sep 2017 13:32:40 +0000https://www.chenshaowen.com/en/blog/how-to-package-django-development-environments-using-docker-4.html1. Deployment Architecture The program processing flow when a browser visits a page: (1) The browser sends a request to Nginx. If it matches a static URL on the Nginx side, for example files such as js, css, or 404.html under the /static directory, Nginx returns the file directly. Other request URLs are forwarded to uWSGI for processing via the uwsgi_pass configuration.WeChat Official AccountDockerServicesToolsDjangoPythonR&DOperationshttps://www.chenshaowen.com/en/blog/how-to-package-django-development-environments-using-docker-3.htmlSat, 23 Sep 2017 19:56:03 +0000Sat, 23 Sep 2017 19:56:03 +0000https://www.chenshaowen.com/en/blog/how-to-package-django-development-environments-using-docker-3.html1. Basic Concepts Docker Compose is a Docker tool for defining and running complex applications. With Docker Compose, you can define a multi-container application in a single file, then use a single command to start your application and finish all the preparation work. Docker Compose positions itself as ‘defining and running complex applications with Docker’.WeChat Official AccountDockerServicesToolsDjangoPythonR&DOperationshttps://www.chenshaowen.com/en/blog/how-to-package-django-development-environments-using-docker-2.htmlFri, 22 Sep 2017 19:52:27 +0000Fri, 22 Sep 2017 19:52:27 +0000https://www.chenshaowen.com/en/blog/how-to-package-django-development-environments-using-docker-2.html1. Basic Concepts A Dockerfile is a collection of instructions for building a Docker image. Docker reads Dockerfile instructions to build an image automatically. A Dockerfile is similar to a Makefile: both are text files that organize all the instructions in the order the image is built. The command to build a Docker image:WeChat Official AccountDockerDjangoPythonServicesToolsR&DOperationshttps://www.chenshaowen.com/en/blog/how-to-package-django-development-environments-using-docker-1.htmlThu, 21 Sep 2017 09:38:47 +0000Thu, 21 Sep 2017 09:38:47 +0000https://www.chenshaowen.com/en/blog/how-to-package-django-development-environments-using-docker-1.htmlVagrant is suited to managing virtual machines, while Docker is suited to managing application environments. To better simulate a real runtime environment, this series of articles uses Docker and Docker Compose to build a development environment of Nginx + uWSGI + Django + MySQL + Redis + Rabbit. 1. BasicWeChat Official AccountDockerServicesToolsDjangoPythonR&DOperationshttps://www.chenshaowen.com/en/blog/django-application-performance-test-based-on-docker.htmlThu, 07 Sep 2017 19:08:09 +0000Thu, 07 Sep 2017 19:08:09 +0000https://www.chenshaowen.com/en/blog/django-application-performance-test-based-on-docker.html1. Basic Website Performance Metrics Concurrency: the number of sessions the server handles per unit of time Throughput: the number of requests the server handles per unit of time Response time: the total elapsed time from when the user sends a request to when the response is fully received, made up of network transmission time, service processing time, and other parts Average response time: the average of the response times of all requests 2.WeChat Official AccountTestingDemoDockerDjangohttps://www.chenshaowen.com/en/blog/free-app-domain-name-service-of-paas-platform.htmlTue, 05 Sep 2017 19:00:58 +0000Tue, 05 Sep 2017 19:00:58 +0000https://www.chenshaowen.com/en/blog/free-app-domain-name-service-of-paas-platform.html Provider Domain Google App Engine appspot.com second-level domain Sina App Engine sinaapp.com second-level domain Baidu App Engine duapp.com second-level domain Heroku herokuapp.com second-level domain AWS Elastic Beanstalk elasticbeanstalk.com second-level domain OpenShift rhcloud.com second-level domain DaoCloud daocloud.io second-level domain TenxCloud tenxapp.com second-level domain Alauda myalauda.cn second-level domainWeChat Official AccountDomain NamesServicesPaaSPlatformOperationshttps://www.chenshaowen.com/en/blog/container-and-ai-are-the-future-directions-in-the-development-of-paas.htmlSun, 03 Sep 2017 15:57:50 +0000Sun, 03 Sep 2017 15:57:50 +0000https://www.chenshaowen.com/en/blog/container-and-ai-are-the-future-directions-in-the-development-of-paas.html1. About PaaS 1.1 What Is PaaS PaaS is short for Platform as a Service, a type of cloud computing service that provides a computing platform along with solution services. The emergence of PaaS accelerated the growth of SaaS, and in particular sped up the development of SaaS applications. For example, when developing SaaS, using the login, database, cache, and other services uniformly provided by a PaaS platform greatly reduces the manpower and cost of building those services.WeChat Official AccountThoughtsCloud ServicesPaaSContainersAIhttps://www.chenshaowen.com/en/blog/hybrid-cloud-of-cloud-deployment.htmlSun, 03 Sep 2017 13:18:17 +0000Sun, 03 Sep 2017 13:18:17 +0000https://www.chenshaowen.com/en/blog/hybrid-cloud-of-cloud-deployment.html1. User Scenarios for Hybrid Cloud 1.1 Handling Explosive Business Growth Internet businesses grow exponentially, and it is hard to predict how much infrastructure capacity to reserve. Usually, for a long stretch in the early days of a business, a small colocation footprint of physical machines is enough to meet demand.WeChat Official AccountCloudPaaSCloud ServicesHybrid Cloudhttps://www.chenshaowen.com/en/blog/jenkins-pipeline-usging-and-debug.htmlMon, 14 Aug 2017 13:13:55 +0000Mon, 14 Aug 2017 13:13:55 +0000https://www.chenshaowen.com/en/blog/jenkins-pipeline-usging-and-debug.html1. Basic Concepts master master is where Jenkins is installed and runs; it is responsible for parsing job scripts, handling tasks, and scheduling compute resources. agent agent is responsible for handling tasks dispatched from master; the actual operations are executed through an executor. executor An executor is the compute resourceWeChat Official AccountJenkinsPipelineContinuous IntegrationDevOpsCICDDebuggingR&Dhttps://www.chenshaowen.com/en/blog/ci-practice-of-using-gitlab.htmlSun, 13 Aug 2017 08:03:17 +0000Sun, 13 Aug 2017 08:03:17 +0000https://www.chenshaowen.com/en/blog/ci-practice-of-using-gitlab.html1. Basic Concepts GitLab-CI: the continuous integration system provided by GitLab. It manages the build status of a project and runs build tasks through GitLab Runner. GitLab-Runner: used to execute build tasks. The script portion of .gitlab-ci.yml is run by GitLab-Runner. .gitlab-ci.yml: a file in the root directory of a git project that records a series of stages and execution rules.WeChat Official AccountDevOpsGitLabGitLab-CICIToolsContinuous IntegrationR&Dhttps://www.chenshaowen.com/en/blog/build-and-configure-of-gitlab.htmlSat, 12 Aug 2017 00:51:07 +0000Sat, 12 Aug 2017 00:51:07 +0000https://www.chenshaowen.com/en/blog/build-and-configure-of-gitlab.html1. Introduction GitLab is an open-source application developed with Ruby on Rails. It implements self-hosted Git project repositories, and public or private projects can be accessed through a web interface. It has features similar to GitHub, and can browse source code, manage defects and comments. It can manage team accessWeChat Official AccountToolsManagementContinuous IntegrationGitLabDevOpsOperationshttps://www.chenshaowen.com/en/blog/growingio-collect-js-field-analysis.htmlTue, 08 Aug 2017 22:01:35 +0000Tue, 08 Aug 2017 22:01:35 +0000https://www.chenshaowen.com/en/blog/growingio-collect-js-field-analysis.html1. Field Meanings 1.1 u, User ID 1 2 3 4 5 6 if cookie.has(gr_user_id){ //使用 cookie 中的 gr_user_id 有效期十年 } else { //按照 "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" 格式，随机生成一个 gr_user_id } 1.2 s, SessionWeChat Official AccountDataFrontendAnalysisTrackingOperationsR&DLearningData Collectionhttps://www.chenshaowen.com/en/blog/django-initializes-fixture-of-data.htmlMon, 07 Aug 2017 21:57:43 +0000Mon, 07 Aug 2017 21:57:43 +0000https://www.chenshaowen.com/en/blog/django-initializes-fixture-of-data.html1. Characteristics of Fixtures Fixtures are a new way to provide initial data, and they are used by Django’s test framework to handle test data for unit tests. Unlike SQL files, a fixture can provide a serialized file that Django’s serialization system understands; it is read, automatically converted into the corresponding model, and then saved into the database.WeChat Official AccountDjangoPythonDataInitializationR&Dhttps://www.chenshaowen.com/en/blog/atom-create-lightweight-python-full-stack-ide.htmlSun, 06 Aug 2017 14:51:16 +0000Sun, 06 Aug 2017 14:51:16 +0000https://www.chenshaowen.com/en/blog/atom-create-lightweight-python-full-stack-ide.htmlAt the moment I use PyCharm as my integrated development tool for both front end and back end. My main stack is JavaScript, CSS, ECMAScript 6, Vue, Python, Django, Mako, and Markdown. Front-end work has been growing steadily lately, and PyCharm seems to be struggling. Atom, Sublime Text, and Visual Studio Code became the candidates; Atom is maintained by GitHub, has better community support, and is more configurable and more fun to play with.WeChat Official AccountToolsPythonR&DIDEAtomFull-Stackhttps://www.chenshaowen.com/en/blog/programming-editor-atom.htmlFri, 04 Aug 2017 23:46:56 +0000Fri, 04 Aug 2017 23:46:56 +0000https://www.chenshaowen.com/en/blog/programming-editor-atom.html1. Introduction to Atom Atom is a code editor that supports the three major desktop platforms Windows, Mac, and Linux, is completely free, and is open source on GitHub. Go now 1.1 Pros and Cons Main advantages A strong development and maintenance team, an open source project, fast Bug fixes,WeChat Official AccountToolsDevelopmentR&DEditorLearninghttps://www.chenshaowen.com/en/blog/axios-ajax.htmlFri, 04 Aug 2017 07:07:11 +0000Fri, 04 Aug 2017 07:07:11 +0000https://www.chenshaowen.com/en/blog/axios-ajax.html1. Basic Concepts 1.1 Ajax Ajax stands for “Asynchronous JavaScript and XML”. At its core it is made up of the JavaScript, XmlHttpRequest, and DOM objects: through the XmlHttpRequest object it sends an asynchronous request to the server, gets data back from the server, and then uses JavaScript to manipulateWeChat Official AccountFrontendToolsAxiosAjaxAPIhttps://www.chenshaowen.com/en/blog/vuejs-component-library.htmlThu, 03 Aug 2017 21:53:02 +0000Thu, 03 Aug 2017 21:53:02 +0000https://www.chenshaowen.com/en/blog/vuejs-component-library.html1. UI Components element - A Vue 2 web UI toolkit from Ele.me Vux - A component library based on Vue and WeUI iview - An open-source UI component library based on Vuejs mint-ui - Mobile UI elements for Vue 2 muse-ui - A responsive UI library with consistent styling across all three platforms vue-material - Build beautiful apps with Vue Material and Vue 2 vuetify - A Vue JS 2 component framework built for mobile Keen-UI - A lightweight collection of basic UI components vonic - Quickly build mobile single-page applications vue-multiselect - A Vue.WeChat Official AccountFrontendToolsComponentsVuejsRepostedhttps://www.chenshaowen.com/en/blog/base64-code.htmlWed, 02 Aug 2017 17:00:31 +0000Wed, 02 Aug 2017 17:00:31 +0000https://www.chenshaowen.com/en/blog/base64-code.html1. Application Scenarios 1.1 Mail RFC821 requires that email content be ASCII. When a message contains other non-ASCII characters or binary data, Content-Transfer-Encoding is needed, and Base64 is one such method. 1.2 URL Some applications need to put binary data into a URL, but URLs only allow a specific setWeChat Official AccountFrontendEncodingDataBase64R&Dhttps://www.chenshaowen.com/en/blog/page-uses-a-custom-font.htmlTue, 01 Aug 2017 19:30:47 +0000Tue, 01 Aug 2017 19:30:47 +0000https://www.chenshaowen.com/en/blog/page-uses-a-custom-font.html1. Font-family The fonts you can use on a web page are limited to the handful of fonts already installed on the PC the browser is running on. Windows operating system, Chinese fonts: SimHei (Hei) SimSun (Song) NSimSun (New Song) FangSong (FangSong) KaiTi (Kai) FangSongGB2312 (FangSong GB2312) KaiTiGB2312 (Kai GB2312)WeChat Official AccountFrontendCSSR&DFontsLearninghttps://www.chenshaowen.com/en/blog/basis-of-docker.htmlFri, 28 Jul 2017 15:30:00 +0000Fri, 28 Jul 2017 15:30:00 +0000https://www.chenshaowen.com/en/blog/basis-of-docker.html1. What Docker Is Docker was born in early 2013 as a side project inside dotCloud (the former name of Docker Inc.). Its source code is hosted on GitHub, it is written in Go, and it follows the Apache 2.0 open-source license. The goal of the Docker project is to provide a lightweight operating system virtualization solution.WeChat Official AccountPaaSDevOpsDockerContainersLearninghttps://www.chenshaowen.com/en/blog/symmetric-encrypted-transmission-of-aes.htmlWed, 26 Jul 2017 18:52:18 +0000Wed, 26 Jul 2017 18:52:18 +0000https://www.chenshaowen.com/en/blog/symmetric-encrypted-transmission-of-aes.html1. Basic Concepts Symmetric encryption: Symmetric encryption is an encryption method that uses a single-key cryptosystem, in which the same key is used both to encrypt and to decrypt the information. Because it is fast, it is often used to encrypt the transmission of large amounts of data. DES (DataWeChat Official AccountSecurityFrontendBackendEncryptionR&Dhttps://www.chenshaowen.com/en/blog/one-day-make-report-great.htmlSat, 22 Jul 2017 20:27:55 +0000Sat, 22 Jul 2017 20:27:55 +0000https://www.chenshaowen.com/en/blog/one-day-make-report-great.htmlWeChat Official AccountRepostResearch ReportMethodshttps://www.chenshaowen.com/en/blog/how-to-quickly-understand-an-industry.htmlSat, 22 Jul 2017 20:18:50 +0000Sat, 22 Jul 2017 20:18:50 +0000https://www.chenshaowen.com/en/blog/how-to-quickly-understand-an-industry.html1. Methods 2. In PracticeWeChat Official AccountRepostResearch ReportMethodsLearningThoughtshttps://www.chenshaowen.com/en/blog/automated-testing-tool-about-selenium.htmlSat, 22 Jul 2017 15:19:42 +0000Sat, 22 Jul 2017 15:19:42 +0000https://www.chenshaowen.com/en/blog/automated-testing-tool-about-selenium.html1. Introduction to Selenium Selenium is an integration testing tool from Thoughtworks. At its core, Selenium Core is based on JSUnit, written entirely in JavaScript, and can run in any browser that supports JavaScript. 1.1 Main Features and Characteristics Open source and free. Multi-browser support: Firefox, Chrome, IE, Opera — usable for compatibility testing.WeChat Official AccountToolsTestingAutomationR&DSeleniumLearninghttps://www.chenshaowen.com/en/blog/cloud-industry-research-report-of-2017.htmlSat, 22 Jul 2017 15:00:49 +0000Sat, 22 Jul 2017 15:00:49 +0000https://www.chenshaowen.com/en/blog/cloud-industry-research-report-of-2017.html1. Classification of Cloud Services IaaS, which provides computing and storage services. PaaS, which provides the platform environment in which software runs. SaaS, which delivers software services directly. 2. Application Scenarios for Cloud Services Adopting cloud services can significantly reduce IT spending. Among IaaS services, cloud hosts and storage are the products with the highest adoption rates, followed by databases and network acceleration.WeChat Official AccountResearch ReportPaaSOpinionsCloud Serviceshttps://www.chenshaowen.com/en/blog/django-forms-function.htmlFri, 21 Jul 2017 21:29:40 +0000Fri, 21 Jul 2017 21:29:40 +0000https://www.chenshaowen.com/en/blog/django-forms-function.html1. Automatically Generating HTML Form Elements A Widget is the tool used to render an HTML element. Specifying a widget 1 2 3 4 5 6 from django import forms class CommentForm(forms.Form): name = forms.CharField() url = forms.URLField() comment = forms.CharField(widget=forms.Textarea) Output of CommentForm().as_table() 1 2 3 4 5 6WeChat Official AccountDjangoPythonSecurityR&Dhttps://www.chenshaowen.com/en/blog/operation-buried-point-and-design.htmlFri, 21 Jul 2017 13:34:15 +0000Fri, 21 Jul 2017 13:34:15 +0000https://www.chenshaowen.com/en/blog/operation-buried-point-and-design.html1. Terminology xpath: There are two approaches, the exact path and the approximate path. Exact path: body>div[0]>div[3]>ul>li[5]>a[0], walking up from the clicked element all the way to the root node and recording every node along the way. Approximate path: body>div.header>div.nav>a[23], which on top of the former omits from the upward path any node that is not on the whitelist.WeChat Official AccountDataDesignPlanR&DTrackingOperationsLearningData Analysishttps://www.chenshaowen.com/en/blog/sql-code-about-django-orm.htmlFri, 21 Jul 2017 13:21:08 +0000Fri, 21 Jul 2017 13:21:08 +0000https://www.chenshaowen.com/en/blog/sql-code-about-django-orm.html1. Basic Concepts ORM: Object Relational Mapping. Its job is to map between a relational database and objects. No complex SQL statements are needed — operating on data is as simple as operating on objects. QuerySet: the list of objects for a given model. A QuerySet lets you read dataWeChat Official AccountDjangoPythonDataSQLR&Dhttps://www.chenshaowen.com/en/blog/haystack-full-text-search.htmlFri, 21 Jul 2017 13:00:03 +0000Fri, 21 Jul 2017 13:00:03 +0000https://www.chenshaowen.com/en/blog/haystack-full-text-search.htmlA quick word on the project requirements: the team needed to publish documentation externally. The documents are written in Markdown and need to be published as HTML. At first we used an Nginx + Jekyll solution. As the documentation grew, the document system developed a strong need for search. IWeChat Official AccountDjangoPythonDataDesignSearchR&Dhttps://www.chenshaowen.com/en/blog/jenkins-integrated-github-development.htmlTue, 18 Jul 2017 21:29:41 +0000Tue, 18 Jul 2017 21:29:41 +0000https://www.chenshaowen.com/en/blog/jenkins-integrated-github-development.html1. Basic Concepts Continuous Integration Continuous integration emphasizes that once developers commit new code, a build and (unit) test run immediately. From the test results we can determine whether the new code and the existing code integrate correctly. Continuous Delivery Building on continuous integration, continuous delivery deploys the integrated codeWeChat Official AccountToolsDevelopmentJenkinsGitHubCIDevOpsR&Dhttps://www.chenshaowen.com/en/blog/cross-departmental-collaborative-curriculum-notes.htmlFri, 14 Jul 2017 20:55:29 +0000Fri, 14 Jul 2017 20:55:29 +0000https://www.chenshaowen.com/en/blog/cross-departmental-collaborative-curriculum-notes.htmlCollaboration is a journey. 1. Obstacles to Cross-Departmental Collaboration Resistance to collaborating on new business Gray areas in responsibilities Employees’ ability to collaborate Information asymmetry The management mechanisms for cross-departmental collaboration Misaligned goals and interests Limited resources 2. The Core Philosophy of Collaboration Win-win thinking An open mindset The three aspects at the heart of collaborationWeChat Official AccountManagementCollaborationTeamsR&Dhttps://www.chenshaowen.com/en/blog/drawing-toolkit-of-dot-using-graphviz.htmlFri, 14 Jul 2017 12:32:44 +0000Fri, 14 Jul 2017 12:32:44 +0000https://www.chenshaowen.com/en/blog/drawing-toolkit-of-dot-using-graphviz.html1. Introduction Graphviz is an open-source drawing toolkit developed at Bell Labs. It uses a specific DSL (domain-specific language), DOT, as its scripting language. It uses layout engines to parse DOT scripts and complete automatic layout. It supports a rich set of export formats, such as PNG, JPG, PostScript, SVG,WeChat Official AccountDiagramToolsGraphvizDrawingR&Dhttps://www.chenshaowen.com/en/blog/jekyll-search-option.htmlThu, 13 Jul 2017 19:24:52 +0000Thu, 13 Jul 2017 19:24:52 +0000https://www.chenshaowen.com/en/blog/jekyll-search-option.htmlIntroduction: Jekyll is a static HTML generation tool. Our documentation system is deployed with Jekyll, which ultimately converts Markdown into HTML files for output. The search requirement is: given a keyword, match it against a document’s title or content and list the matching documents. 1. Option One: Simple-Jekyll-Search Reference: https://github.com/christian-fei/Simple-Jekyll-SearchWeChat Official AccountJekyllBlogSolutionsDataSearchFrontendhttps://www.chenshaowen.com/en/blog/signal-of-django.htmlThu, 13 Jul 2017 13:06:10 +0000Thu, 13 Jul 2017 13:06:10 +0000https://www.chenshaowen.com/en/blog/signal-of-django.html1. Basic Concepts Django ships with a signal dispatcher built in. Signals help decouple program modules. When an event occurs elsewhere in the application, a designated function is notified. Signals let certain senders notify a group of receivers that some action has taken place. 2. Using Signals 2.1 Declaring aWeChat Official AccountDjangoBackendPythonR&DLearningDecouplingEventshttps://www.chenshaowen.com/en/blog/101-of-vue-and-webpack-project.htmlWed, 12 Jul 2017 17:26:29 +0000Wed, 12 Jul 2017 17:26:29 +0000https://www.chenshaowen.com/en/blog/101-of-vue-and-webpack-project.html1, Initializing the Project vue-cli is the official project scaffold released by Vue; using vue-cli, you can quickly create a Vue + Webpack project. 1 2 3 4 5 6 7 8 9 10 npm install -g vue-cli # Install vue-cli globally vue init webpack myproject # Create a vue project; when it runs, it prompts for information about the project cd myproject # Enter the project npm install # Install the dependency packages; a new node_modules folder is created and the packages are installed there npm run dev # Run locally, by default at http://localhost:8080, starting the service 2, Directory Structure 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |--build //folder for webpack-related code | |--build.WeChat Official AccountFrontendToolsVueWebpackR&Dhttps://www.chenshaowen.com/en/blog/class-based-view-of-django.htmlTue, 11 Jul 2017 16:36:56 +0000Tue, 11 Jul 2017 16:36:56 +0000https://www.chenshaowen.com/en/blog/class-based-view-of-django.htmlDjango has two kinds of views: function-based views and class-based views. The role of a view is mainly to fill in logic and return a response body. Function-based views are hard to extend and have a low rate of code reuse. Class-based views, on the other hand, can use inheritanceWeChat Official AccountDjangoBackendPythonClass-Based ViewsR&Dhttps://www.chenshaowen.com/en/blog/auto-testing-of-django.htmlMon, 10 Jul 2017 21:39:27 +0000Mon, 10 Jul 2017 21:39:27 +0000https://www.chenshaowen.com/en/blog/auto-testing-of-django.htmlAutomated testing in Django can be done with doctests or unit tests. The logic of automated testing is to pass test data into the function under test, then use whether the output matches the expected result as the criterion for whether the test passes. There are a few key pointsWeChat Official AccountDjangoTestingDemoDevOpsCICDhttps://www.chenshaowen.com/en/blog/permissions-of-django-rest-framework.htmlFri, 07 Jul 2017 06:47:19 +0000Fri, 07 Jul 2017 06:47:19 +0000https://www.chenshaowen.com/en/blog/permissions-of-django-rest-framework.html1. Permission Management in DRF Permission management in Django REST Framework consists of two parts. One is Authentication. It specifies how the user is authenticated, obtaining request.user. The other is Permissions. It performs permission control over Django resources and user categories. 1.1 Authentication Methods The relevant source is in the rest_framework/authentication.WeChat Official AccountDjangoPythonBackendSecurityAPIPermissionsR&Dhttps://www.chenshaowen.com/en/blog/third-party-component-list-of-django.htmlThu, 06 Jul 2017 09:27:50 +0000Thu, 06 Jul 2017 09:27:50 +0000https://www.chenshaowen.com/en/blog/third-party-component-list-of-django.htmlAwesome-Django A collection of Django apps, projects, and resources. Django-Packages A large number of Django third-party components. Django-Mptt Store hierarchical data structures in the database. Django-Rest-Framework Automatically generate RESTful APIs. Django-Rest-Framework-Jwt Provides token authentication for Django-Rest-Framework. Django-Tastypie Automatically generate RESTful APIs. Django-Cors-Headers Solves the problem of cross-origin access. Django-Ckeditor Rich text support for Django.WeChat Official AccountDjangoPythonBackendComponentsR&Dhttps://www.chenshaowen.com/en/blog/django-template-inherit.htmlMon, 03 Jul 2017 19:17:11 +0000Mon, 03 Jul 2017 19:17:11 +0000https://www.chenshaowen.com/en/blog/django-template-inherit.html1. The Scenario In a project, elements such as the header and footer are often reused. To avoid rewriting these elements on every page, and so that a change does not require editing each page individually, the common parts need to be extracted — that is Django template inheritance. 2.WeChat Official AccountDjangoPythonTemplatesR&Dhttps://www.chenshaowen.com/en/blog/book/the-101-of-ecma-script.htmlTue, 27 Jun 2017 20:05:01 +0000Tue, 27 Jun 2017 20:05:01 +0000https://www.chenshaowen.com/en/blog/book/the-101-of-ecma-script.htmlAuthor: Ruan Yifeng Publisher: Publishing House of Electronics Industry Publication year: 2014-8-1 ISBN: 9787121238369 E-bookWeChat Official AccountBooksJavaScriptFrontendLearninghttps://www.chenshaowen.com/en/blog/django-decorator.htmlThu, 22 Jun 2017 23:55:59 +0000Thu, 22 Jun 2017 23:55:59 +0000https://www.chenshaowen.com/en/blog/django-decorator.htmlWhen developing with a separated front end and back end, some of the API endpoints provided to the front end use GET requests and some use POST requests. To keep the back end from throwing an error while reading values from request in views.py, you had to check the requestWeChat Official AccountDjangoDemoDecoratorsPythonhttps://www.chenshaowen.com/en/blog/snippets-of-django.htmlThu, 22 Jun 2017 23:52:06 +0000Thu, 22 Jun 2017 23:52:06 +0000https://www.chenshaowen.com/en/blog/snippets-of-django.html1. Auto-Register All Models in Admin admin.py 1 2 3 4 5 6 7 8 9 10 # -*- coding: utf-8 -*- import inspect from django.contrib import admin from . import models for name, obj in inspect.getmembers(models): try: if inspect.isclass(obj): admin.site.register(getattr(models, name)) except Exception as e: pass 2. Get All View Names Get all view names of the projectWeChat Official AccountDjangoPythonR&DLearningTipshttps://www.chenshaowen.com/en/blog/101-of-django-rest-framework.htmlThu, 22 Jun 2017 23:50:15 +0000Thu, 22 Jun 2017 23:50:15 +0000https://www.chenshaowen.com/en/blog/101-of-django-rest-framework.htmlIn the SaaS development I work on, there is a fairly high demand for development efficiency. From project initiation, prototype design and evaluation, requirements confirmation, frontend design, and backend development through to final acceptance, we complete one iteration in just a few weeks. Guided by agile development, we began rollingWeChat Official AccountDataBackendInterfaceDjangoPythonAPIR&Dhttps://www.chenshaowen.com/en/blog/how-to-reflect-backend-value-in-front-and-backend-separation-mode.htmlThu, 22 Jun 2017 11:14:21 +0000Thu, 22 Jun 2017 11:14:21 +0000https://www.chenshaowen.com/en/blog/how-to-reflect-backend-value-in-front-and-backend-separation-mode.htmlIn recent years, with the rise of single-page web applications, all kinds of frameworks have kept emerging. The front end has begun to enter a stage of engineering and componentization. Single-page web technology has expanded the boundaries of the front end, which can now also route and render pages without depending on the back end.WeChat Official AccountManagementBackendValueR&DArchitectureThoughtsTeamshttps://www.chenshaowen.com/en/blog/how-to-develop-document-system-using-jekyll.htmlSat, 17 Jun 2017 00:02:01 +0000Sat, 17 Jun 2017 00:02:01 +0000https://www.chenshaowen.com/en/blog/how-to-develop-document-system-using-jekyll.htmlFor a company that produces technical output, an excellent documentation management and publishing system is indispensable. A documentation system is a kind of content management system; it has high accessibility requirements, a low update frequency per document, and a high publishing frequency. In my opinion, publishing documentation as pure staticWeChat Official AccountJekyllDocumentationDesignOperationsR&Dhttps://www.chenshaowen.com/en/blog/mock-server.htmlFri, 16 Jun 2017 21:10:54 +0000Fri, 16 Jun 2017 21:10:54 +0000https://www.chenshaowen.com/en/blog/mock-server.htmlIn frontend-backend separated development, when the backend API is not finished, the frontend cannot continue debugging. So that frontend and backend can develop in parallel, the frontend needs a set of API interface environments — this is the Mock API. The diagram below lays out the development flow nicely. Without a Mock data step, frontend-backend integration takes up a great deal of time.WeChat Official AccountFrontendBackendMockR&Dhttps://www.chenshaowen.com/en/blog/source-code-analysis-of-arachni-1.htmlFri, 16 Jun 2017 21:02:22 +0000Fri, 16 Jun 2017 21:02:22 +0000https://www.chenshaowen.com/en/blog/source-code-analysis-of-arachni-1.htmlArachni is a web security vulnerability scanner built on the Ruby on Rails framework. 1. Ruby on Rails Ruby on Rails, abbreviated ROR, is a web framework that consists of two parts: the Ruby language and the Rails framework. Ruby had always been popular in Japan, until 2004, when a 26-year-old Dane, David Heinemeier Hansson, proposed the web framework Rails.WeChat Official AccountToolsSecuritySource CodeR&DRubyhttps://www.chenshaowen.com/en/blog/practice-and-automation-generation-of-apidoc.htmlWed, 14 Jun 2017 17:22:45 +0000Wed, 14 Jun 2017 17:22:45 +0000https://www.chenshaowen.com/en/blog/practice-and-automation-generation-of-apidoc.htmlIn a frontend/backend separated architecture, API documentation changes hands frequently. When third-party interfaces are involved, API and documentation changes can be even faster in multi-party collaboration scenarios. To make maintaining the API and handing over documentation easier, here is a documentation generation tool worth recommending — apidoc. 1. Introduction toWeChat Official AccountDjangoInterfaceAPIDocumentationR&Dhttps://www.chenshaowen.com/en/blog/restful-api-of-using-django-tastypie.htmlTue, 13 Jun 2017 21:31:13 +0000Tue, 13 Jun 2017 21:31:13 +0000https://www.chenshaowen.com/en/blog/restful-api-of-using-django-tastypie.html1. Restful REST is short for Representational State Transfer, meaning “representation state transfer”. Restful is a development philosophy and a software architecture style for the World Wide Web. 1.1 Features of Restful Abstract resources Images, text, songs, and videos are all resource entities; on the network they are abstracted asWeChat Official AccountDjangoPythonInterfaceAPITastypieR&Dhttps://www.chenshaowen.com/en/blog/same-origin-policy.htmlTue, 13 Jun 2017 17:19:03 +0000Tue, 13 Jun 2017 17:19:03 +0000https://www.chenshaowen.com/en/blog/same-origin-policy.html1. Same-Origin Policy The same-origin policy is the cornerstone of browser security. The definition of same-origin covers three aspects: Same protocol Same domain Same port Scope of the restrictions: Cookie, LocalStorage, and IndexDB cannot be read The DOM cannot be obtained AJAX requests cannot be sent Simply put, two URLs whose protocol, domain, and port differ in any way are not allowed to communicate with each other, and the scope covers reading each other’s cookies and DOM and sending AJAX requests.WeChat Official AccountFrontendNetworkSecurityR&DCORSLearningBrowserhttps://www.chenshaowen.com/en/blog/linux-performance-command-tools.htmlSun, 28 May 2017 15:47:41 +0000Sun, 28 May 2017 15:47:41 +0000https://www.chenshaowen.com/en/blog/linux-performance-command-tools.htmlThere are many performance tools on the Linux platform, enough to make your head spin, but long-term exploration and experience have shown that the most useful ones are still those time-tested, simple little utilities. Below are the command-line tools collected from system performance expert Brendan D. Gregg’s talk on Linux performance (Linux Performance Tools).WeChat Official AccountLinuxPerformanceOperationsMonitoringLearningTroubleshootingTuningCommandsObservabilityhttps://www.chenshaowen.com/en/blog/book/interactive-data-visualization-for-the-web.htmlFri, 26 May 2017 19:06:09 +0000Fri, 26 May 2017 19:06:09 +0000https://www.chenshaowen.com/en/blog/book/interactive-data-visualization-for-the-web.htmlSubtitle: Using D3 to Build Interactive Charts Original title: Interactive Data Visualization for the Web Author: [US] Scott Murray Translator: Li Songfeng Publisher: Posts & Telecom Press Publication year: 2013-6 ISBN: 9787115320117WeChat Official AccountBooksDataFrontendVisualizationD3R&Dhttps://www.chenshaowen.com/en/blog/qiniu-storage-batch-operation-using-qshell.htmlSun, 21 May 2017 21:47:49 +0000Sun, 21 May 2017 21:47:49 +0000https://www.chenshaowen.com/en/blog/qiniu-storage-batch-operation-using-qshell.html1. Download the Tool - qshell qshell is a command-line tool built on the public APIs documented by Qiniu to make it easy for developers to test and use Qiniu API services, written in the Go language. The tool currently bundles convenient commands for Qiniu Storage, CDN, and other methods commonly used in Qiniu services.WeChat Official AccountDataDemoCDNQiniuBatchhttps://www.chenshaowen.com/en/blog/xss-cheat-sheet.htmlSat, 20 May 2017 09:12:31 +0000Sat, 20 May 2017 09:12:31 +0000https://www.chenshaowen.com/en/blog/xss-cheat-sheet.htmlIf you use the Chrome browser to test XSS vectors, disable the browser’s XSS interception first. First, close all Chrome browser processes, then run: 1 chrome.exe -args --disable-xss-auditor --args --disable-web-security Enter non-web-safe mode. 1. Injecting HTML/JavaScript with the <> tags 1 <script>alert('XSS')</script> 1 '';!--"<XSS>=&{()} 1 2 //The quotes around SRC can be omitted <SCRIPT SRC=http://ha.WeChat Official AccountSecurityXSSR&DVulnerabilityFrontendhttps://www.chenshaowen.com/en/blog/how-to-export-excel-quickly-in-web-development.htmlThu, 18 May 2017 17:51:05 +0000Thu, 18 May 2017 17:51:05 +0000https://www.chenshaowen.com/en/blog/how-to-export-excel-quickly-in-web-development.htmlIn web development you often run into the need to export data. This post mainly covers how to quickly export data and save it as an Excel file. 1. Frontend In web development, formatted data is usually displayed as a table. Below is an employee salary table; we will useWeChat Official AccountDjangoFrontendDataDemohttps://www.chenshaowen.com/en/blog/principle-and-construction-of-xss.htmlTue, 16 May 2017 20:04:53 +0000Tue, 16 May 2017 20:04:53 +0000https://www.chenshaowen.com/en/blog/principle-and-construction-of-xss.htmlXSS is an attack technique that executes JavaScript in the frontend. With the popularity of UGC sites, the data generated by users has surged, and the ever-denser network of data blocks makes it easier to carry out and spread XSS. The damage XSS causes includes: stealing user cookies and personal information; hijacking sessions and manipulating users’ network data; launching DDoS attacks; tampering with pages, popping up ads, and so on.WeChat Official AccountSecurityJavaScriptXSSR&Dhttps://www.chenshaowen.com/en/blog/manage-command-in-django.htmlMon, 15 May 2017 21:14:48 +0000Mon, 15 May 2017 21:14:48 +0000https://www.chenshaowen.com/en/blog/manage-command-in-django.htmlEvery Django project automatically generates a manage.py file. manage.py is a simple wrapper around django-admin; its job is to put the Django Project into the sys.path directory and set the DJANGO_SETTINGS_MODULE environment variable to the current Project’s settings.py file. In other words, the difference between django-admin and manage is thatWeChat Official AccountDjangoPythonToolsCommandR&Dhttps://www.chenshaowen.com/en/blog/django-model-meta.htmlThu, 11 May 2017 15:48:41 +0000Thu, 11 May 2017 15:48:41 +0000https://www.chenshaowen.com/en/blog/django-model-meta.htmlDefine metadata for a model through an inner class “class Meta”, like this: 1 2 3 4 class Foo(models.Model): bar = models.CharField(max_length=30) class Meta: #...... Model metadata means: any data that is not a field such as ordering options, admin options, and so on. Below are all the Meta options you might use.WeChat Official AccountDjangoPythonDatabaseModelR&Dhttps://www.chenshaowen.com/en/blog/django-middleware.htmlWed, 10 May 2017 19:39:12 +0000Wed, 10 May 2017 19:39:12 +0000https://www.chenshaowen.com/en/blog/django-middleware.html1. __init__(self) In a middleware class, the __init__() method is used to perform system-wide initialization settings. For performance reasons, each enabled middleware is initialized only once per server process. That is, __init__() is called only when the server process starts, and is not executed while handling an individual request. For a middleware, the usual reason to define an __init__() method is to check whether it is needed at all.WeChat Official AccountDjangoPythonMiddlewareR&DLearningRequest HandlingArchitecturehttps://www.chenshaowen.com/en/blog/introduction-of-django-standard-library.htmlWed, 10 May 2017 19:22:23 +0000Wed, 10 May 2017 19:22:23 +0000https://www.chenshaowen.com/en/blog/introduction-of-django-standard-library.htmlDjango’s standard library lives in the django.contrib package. Each subpackage is a standalone add-on feature package. These subpackages are generally independent of one another, though some django.contrib subpackages depend on other subpackages. There is no hard requirement on the kinds of functions in django.contrib. Some of these packages ship with models (and therefore require you to install the corresponding database tables), while others consist of standalone middleware and template tags.WeChat Official AccountDjangoPythonCompilationLearningR&Dhttps://www.chenshaowen.com/en/blog/how-to-build-reusable-django-app.htmlFri, 05 May 2017 21:43:47 +0000Fri, 05 May 2017 21:43:47 +0000https://www.chenshaowen.com/en/blog/how-to-build-reusable-django-app.htmlThe team I work on is responsible for SaaS development and carries almost all of the center’s SaaS needs. Among these there are long-term maintenance projects and short-term crunch demo projects, and everyone wears several hats. Of course, the development platform is also open to other people to use, and the whole platform has hundreds if not thousands of SaaS applications.WeChat Official AccountDjangoPythonManagementR&DArchitecturehttps://www.chenshaowen.com/en/blog/book/large-web-technology-architecture.htmlThu, 04 May 2017 21:39:11 +0000Thu, 04 May 2017 21:39:11 +0000https://www.chenshaowen.com/en/blog/book/large-web-technology-architecture.htmlSubtitle: Core Principles and Case Studies Author: Li Zhihui Publisher: Publishing House of Electronics Industry Year: 2013-9-1 ISBN: 9787121212000 Notes: Without business demand for technology, you cannot cultivate an outstanding architect. The author grew up alongside Taobao’s business growth. The book explains the key technical points of website architecture mainly from several aspects: system performance, availability, scalability, extensibility, and security.WeChat Official AccountBooksArchitectureDesignhttps://www.chenshaowen.com/en/blog/third-party-login-and-single-sign-on.htmlThu, 04 May 2017 20:41:35 +0000Thu, 04 May 2017 20:41:35 +0000https://www.chenshaowen.com/en/blog/third-party-login-and-single-sign-on.htmlThe session mechanism for login: HTTP is a stateless protocol, and every request a browser makes is independent of the others. But not every HTTP request is unrelated to state, so the browser and the server have to jointly maintain a state — this is the session mechanism. One approach: the first time the browser sends a request to the server, the server returns a sessionID to the browser, and every subsequent request from the browser carries that sessionID to verify the same user.WeChat Official AccountSecurityDevOpsAuthenticationSSOR&DArchitecturehttps://www.chenshaowen.com/en/blog/book/the-big-data.htmlSun, 23 Apr 2017 21:10:24 +0000Sun, 23 Apr 2017 21:10:24 +0000https://www.chenshaowen.com/en/blog/book/the-big-data.htmlSubtitle: A Revolution That Will Transform How We Live, Work, and Think Author: [UK] Viktor Mayer-Schönberger Publisher: Zhejiang People’s Publishing House Publication Year: 2012-12 ISBN: 9787213052545WeChat Official AccountBooksDataBig DataThinkinghttps://www.chenshaowen.com/en/blog/using-python-read-and-write-excel-data.htmlSun, 16 Apr 2017 01:35:32 +0000Sun, 16 Apr 2017 01:35:32 +0000https://www.chenshaowen.com/en/blog/using-python-read-and-write-excel-data.htmlPython and Excel are both commonly used for data processing, so passing data back and forth and handling computations between them is unavoidable. This article mainly introduces the Python-Excel family of libraries, and how to use the xlrd and xlwt libraries. 1. Common Libraries xlwings, openpyxl, pandas, win32com, xlsxwriter, DataNitro,WeChat Official AccountPythonDataR&DExcelLearningData Processingpandashttps://www.chenshaowen.com/en/blog/collect-free-open-data-source.htmlSat, 15 Apr 2017 11:06:28 +0000Sat, 15 Apr 2017 11:06:28 +0000https://www.chenshaowen.com/en/blog/collect-free-open-data-source.html1. Economy World Bank Gapminder Economic, health, and social statistics covered by the World Health Organization and the World Bank CIA, The World Factbook Contains information on history, population, economy, government, infrastructure, and military affairs for 267 countries National Bureau of Statistics of China latest macroeconomic data Ministry of Industry and Information Technology A good deal of data is published here, especially data related to industrial operations and informatizationWeChat Official AccountDataData SourceCompilationR&Dhttps://www.chenshaowen.com/en/blog/book/the-moon-and-sixpence.htmlSun, 09 Apr 2017 12:57:38 +0000Sun, 09 Apr 2017 12:57:38 +0000https://www.chenshaowen.com/en/blog/book/the-moon-and-sixpence.htmlAuthor: W. Somerset Maugham Publisher: Tianjin People’s Publishing House Publication year: 2016-1-20 ISBN: 9787201100159WeChat Official AccountBooksLifeLiteratureReadingMaughamhttps://www.chenshaowen.com/en/blog/book/a-maverick-pig.htmlSat, 08 Apr 2017 00:03:08 +0000Sat, 08 Apr 2017 00:03:08 +0000https://www.chenshaowen.com/en/blog/book/a-maverick-pig.htmlAuthor: Wang Xiaobo Publisher: Northern Literature and Art Publishing House Publication Year: 2006-4 ISBN: 9787531719199WeChat Official AccountBooksLifeLiteratureThoughtshttps://www.chenshaowen.com/en/blog/book/you-just-look-very-hard.htmlTue, 04 Apr 2017 20:26:10 +0000Tue, 04 Apr 2017 20:26:10 +0000https://www.chenshaowen.com/en/blog/book/you-just-look-very-hard.htmlAuthor: Li Shanglong Publisher: China Friendship Publishing Company Publication Year: 2015-7 ISBN: 9787505735361 Notes: Better suited to college students who feel lost. The author dropped out of a military academy, became an English teacher at New Oriental, and eventually wrote screenplays and made films, living the life he chose through his own effort.WeChat Official AccountBooksLifeGrowthThoughtshttps://www.chenshaowen.com/en/blog/app-engine-2-0.htmlTue, 04 Apr 2017 15:14:40 +0000Tue, 04 Apr 2017 15:14:40 +0000https://www.chenshaowen.com/en/blog/app-engine-2-0.htmlNote: A few brief words on a sharing session by a teammate. App Engine frees development from the tedious deployment and configuration process and provides full-fledged automated support services. Its Controller is the brain of App Engine, coordinating and controlling the operation of the whole engine. The slides mainly cover App Engine’s design approach and evolution, and also give thought to resource scheduling and failure disaster recovery.WeChat Official AccountPaaSArchitecturePlatformIntroductionhttps://www.chenshaowen.com/en/blog/unified-login-service.htmlTue, 04 Apr 2017 10:20:45 +0000Tue, 04 Apr 2017 10:20:45 +0000https://www.chenshaowen.com/en/blog/unified-login-service.htmlThe platform currently includes multiple subsystems and multiple versions, and all versions use the same operations system. To make it easy for users to use multiple versions, we need to bind their QQ and WeChat accounts. In addition, to minimize the chance that a failure in some login system leavesWeChat Official AccountRepostServicesAuthenticationSecurityArchitectureR&Dhttps://www.chenshaowen.com/en/blog/book/marketing-research.htmlMon, 03 Apr 2017 21:15:45 +0000Mon, 03 Apr 2017 21:15:45 +0000https://www.chenshaowen.com/en/blog/book/marketing-research.htmlAuthor: Ronald F. Bush / Alvin C. Burns Publisher: Renmin University of China Press Publication year: 2011-3-1 ISBN: 9787300133362WeChat Official AccountBooksMarketingManagementReadingResearchhttps://www.chenshaowen.com/en/blog/book/the-economic-trend-reality.htmlSat, 25 Mar 2017 12:15:01 +0000Sat, 25 Mar 2017 12:15:01 +0000https://www.chenshaowen.com/en/blog/book/the-economic-trend-reality.htmlAuthor: Shi Hanbing Publisher: Shanghai University of Finance and Economics Press Publication Year: 2014-7-1 ISBN: 9787564219239WeChat Official AccountBooksFinanceEconomicsThoughtshttps://www.chenshaowen.com/en/blog/book/the-economic-trend-future.htmlSat, 25 Mar 2017 12:09:51 +0000Sat, 25 Mar 2017 12:09:51 +0000https://www.chenshaowen.com/en/blog/book/the-economic-trend-future.htmlAuthor: Shi Hanbing Publisher: Shanghai University of Finance and Economics Press Publication Year: 2014-7-1 ISBN: 9787564219352 Notes: The author applies a fractal analysis to China’s debt, real estate, demographics, and food supply, and is deeply anxious about the current situation. He applies the same fractal analysis to the economic and political circumstances of geopolitical nations, explaining the reach of the US dollar, the unrivaled supremacy of financial warfare, and the contests between China and its neighbors such as Russia, India, Vietnam, and Japan.WeChat Official AccountBooksFinanceEconomicsThoughtshttps://www.chenshaowen.com/en/blog/plugin-development-of-jquery.htmlSun, 05 Mar 2017 17:16:43 +0000Sun, 05 Mar 2017 17:16:43 +0000https://www.chenshaowen.com/en/blog/plugin-development-of-jquery.html1. Features of jQuery jQuery eliminates browser compatibility issues Traversing the DOM tree and selecting elements A large plugin library A utility function library, such as $.type(), $.extent(), and so on The core function of jQuery usually refers to a factory object, that is, the jQuery() function, or its more commonly used alias, the $() function.WeChat Official AccountFrontendJavaScriptjQueryR&DPluginLearninghttps://www.chenshaowen.com/en/blog/book/professional-jquery.htmlSun, 05 Mar 2017 15:39:16 +0000Sun, 05 Mar 2017 15:39:16 +0000https://www.chenshaowen.com/en/blog/book/professional-jquery.htmlEnglish title: Professional jQuery Author: [US] Otero, [US] Lawrence Publisher: Tsinghua University Press Translator: Shi Hongbin Publication year: 2013-04 ISBN: 9787302317845WeChat Official AccountBooksJavaScriptFrontendLearninghttps://www.chenshaowen.com/en/blog/book/data-decision-making.htmlTue, 28 Feb 2017 21:07:03 +0000Tue, 28 Feb 2017 21:07:03 +0000https://www.chenshaowen.com/en/blog/book/data-decision-making.htmlAuthor: Douglas W. Hubbard Subtitle: The Big Data Era — the quantitative decision-making method used by Fortune 500 companies Publisher: World Publishing Corporation Translator: Deng Hongtao Publication Year: 2013-09-01 ISBN: 9787510067327 Notes: Measurement is used to reduce uncertainty, not to eliminate it. The key to measurement is figuring out what to measure, and why the thing being measured matters.WeChat Official AccountBooksDataDecision MakingManagementBig Datahttps://www.chenshaowen.com/en/blog/book/antifragile-things-that-gain-from-disorder.htmlSat, 25 Feb 2017 12:53:20 +0000Sat, 25 Feb 2017 12:53:20 +0000https://www.chenshaowen.com/en/blog/book/antifragile-things-that-gain-from-disorder.htmlAuthor: Nassim Nicholas Taleb (US) Subtitle: Gaining from Uncertainty Original title: Antifragile: Things That Gain from Disorder Publisher: CITIC Press Publication Year: 2014-01-01 ISBN: 9787508643335WeChat Official AccountBooksFinanceGame TheoryThoughtsRiskhttps://www.chenshaowen.com/en/blog/book/dive-into-python.htmlFri, 24 Feb 2017 13:11:11 +0000Fri, 24 Feb 2017 13:11:11 +0000https://www.chenshaowen.com/en/blog/book/dive-into-python.htmlAuthor: Mark Pilgrim Publisher: Apress Published: 2004-07-19 ISBN: 9781590593561WeChat Official AccountBooksPythonLearningProgrammingReadinghttps://www.chenshaowen.com/en/blog/ddos-101.htmlThu, 23 Feb 2017 20:17:24 +0000Thu, 23 Feb 2017 20:17:24 +0000https://www.chenshaowen.com/en/blog/ddos-101.html1. What Is DDoS Take the Spring Festival ticket rush: the moment high-speed rail tickets go on sale, people all over the country send connection requests to the 12306 servers. The 12306 servers then either stop responding or respond slowly to web requests. That constitutes a DDoS attack — through a large number of concurrent requests, the target service is forced to become degraded, or even to shut down.WeChat Official AccountNetworkSecurityDDoSGetting Startedhttps://www.chenshaowen.com/en/blog/reflection-of-python.htmlFri, 17 Feb 2017 20:46:34 +0000Fri, 17 Feb 2017 20:46:34 +0000https://www.chenshaowen.com/en/blog/reflection-of-python.html1. Application Scenarios Control the program’s runtime flow through a configuration file. What a configuration file usually holds is a string, not an object. When debugging a program, inspect all the attribute values of an object. Dynamic module import. For the first scenario, reflection is the widely adopted approach. ManyWeChat Official AccountPythonReflectionR&DLearninghttps://www.chenshaowen.com/en/blog/book/the-interactive-training.htmlFri, 17 Feb 2017 09:29:44 +0000Fri, 17 Feb 2017 09:29:44 +0000https://www.chenshaowen.com/en/blog/book/the-interactive-training.htmlSubtitle: New adult-training methods that make the learning process active and enjoyable Author: Harold D. Stolovitch, Erica J. Keeps (US) Publisher: Enterprise Management Publishing House Publication year: 2012-1 ISBN: 9787802558861 Note: It mainly discusses how interactive training can achieve very good training results, analyzes the training and learning process and psychology, and proposes four training methods along with some suggestions.WeChat Official AccountBooksTrainingThoughtshttps://www.chenshaowen.com/en/blog/block-chain-an-innovation-triggered-by-credit-cost-reduction.htmlTue, 14 Feb 2017 09:19:48 +0000Tue, 14 Feb 2017 09:19:48 +0000https://www.chenshaowen.com/en/blog/block-chain-an-innovation-triggered-by-credit-cost-reduction.html1. How Society Runs on Credit First, let’s do a Q&A: @ask:Why can you ride the bus with just a card? @answer:Because I topped it up. @ask:Money? Then where did your money come from? @answer:My monthly salary. @ask:Why does the company pay you a salary every month? @answer:We signed a contract.WeChat Official AccountThoughtsDataBlockchainFinancehttps://www.chenshaowen.com/en/blog/black-scholes-option-pricing-model.htmlSun, 12 Feb 2017 10:23:51 +0000Sun, 12 Feb 2017 10:23:51 +0000https://www.chenshaowen.com/en/blog/black-scholes-option-pricing-model.html1. The Significance of the Black-Scholes Option Pricing Model The Black-Scholes model and some of its variants have been widely used by option traders, investment banks, financial managers, insurers, and others. The expansion of derivative instruments has made international financial markets more efficient, but it has also made global markets more volatile.WeChat Official AccountFinanceTradingOptionsPricinghttps://www.chenshaowen.com/en/blog/book/the-black-swan.htmlSat, 11 Feb 2017 17:09:35 +0000Sat, 11 Feb 2017 17:09:35 +0000https://www.chenshaowen.com/en/blog/book/the-black-swan.htmlEnglish title: The Black Swan Subtitle: How to Deal with an Unpredictable Future Author: Nassim Nicholas Taleb Translator: Wan Dan Publisher: CITIC Press Publication Year: 2009-8 ISBN: 9787508616018 Notes: The author uses his own experiences — the Lebanese war, a journalist’s war diary, a stock market crash, Yevgenia’s book — as his entry points, describing from personal experience this world that lives in uncertainty.WeChat Official AccountBooksFinanceLifeThoughtsRiskhttps://www.chenshaowen.com/en/blog/built-in-functions-in-python.htmlSat, 11 Feb 2017 13:48:34 +0000Sat, 11 Feb 2017 13:48:34 +0000https://www.chenshaowen.com/en/blog/built-in-functions-in-python.htmlFunction Description abs(number) Returns the absolute value of a number apply(function[, args[, kwds]]) Calls the given function, optionally providing arguments all(iterable) Returns True if all elements of the iterable are true, otherwise False any(iterable) Returns True if any element of the iterable is true, otherwise False basestring() Abstract superclass of str and unicode, used for type checking bool(object) Returns True or False depending on the boolean value of object callable(object) Checks whether an object is callable chr(number) Returns the character whose ASCII code is the given number classmethod(func) Creates a class method from an instance method cmp(x, y) Compares x and y — returns a positive number if x>y; returns 0 if x==y complex(real[, imag]) Returns the complex number with the given real part (and optional imaginary part) delattr(object, name) Deletes the given attribute from the given object dict([mapping-or-sequence]) Constructs a dictionary, optionally from a mapping or a list of (key, value) pairs.WeChat Official AccountPythonAPIFunctionsLearningR&Dhttps://www.chenshaowen.com/en/blog/what-is-the-quant.htmlFri, 10 Feb 2017 19:23:05 +0000Fri, 10 Feb 2017 19:23:05 +0000https://www.chenshaowen.com/en/blog/what-is-the-quant.html1. What a Quant Does A quant’s job is to design and implement mathematical models for finance, including derivative pricing, risk valuation, and predicting market behavior. 2. Types of Quants Desk quant, who develops pricing models used directly by traders. The advantage is being close to the money and the opportunities encountered in trading.WeChat Official AccountFinanceQuantQuantitativeLearninghttps://www.chenshaowen.com/en/blog/drawing-2d-fractal-graph-using-python.htmlWed, 25 Jan 2017 23:28:00 +0000Wed, 25 Jan 2017 23:28:00 +0000https://www.chenshaowen.com/en/blog/drawing-2d-fractal-graph-using-python.html1. Koch Curve The Swedish mathematician Helge von Koch proposed the Koch curve in his 1904 paper “On a continuous curve without tangents, constructible from elementary geometry.” It is described as follows: Specify the length of a line segment \(l\) (this can be thought of as iteration 0) Divide thisWeChat Official AccountPythonMathR&DFractalLearningVisualizationDrawinghttps://www.chenshaowen.com/en/blog/fractal-theory.htmlMon, 23 Jan 2017 17:53:31 +0000Mon, 23 Jan 2017 17:53:31 +0000https://www.chenshaowen.com/en/blog/fractal-theory.html1. Fractal Theory 1.1 The Definition of a Fractal A fractal usually refers to a geometric shape that can be divided into several parts, each of which is approximately a reduced-scale copy of the whole — that is, it has the property of self-similarity. 1.2 The History of Fractal Research In the 17th century, Leibniz already pondered the problem of self-similarity.WeChat Official AccountDiagramMathLearningFractalGeneral Knowledgehttps://www.chenshaowen.com/en/blog/django-permissions-control.htmlSun, 22 Jan 2017 09:23:45 +0000Sun, 22 Jan 2017 09:23:45 +0000https://www.chenshaowen.com/en/blog/django-permissions-control.html1. Django Built-in Permission Management 1.1 Permission Categories Permission Used to define User A’s permission on Task. User If User A has permission on Model B, then User A has the corresponding permission on all instances in Model B. The user_permission field of the User object is used to manageWeChat Official AccountDjangoDevelopmentPermissionsPythonR&DLearningSecurityRBACAuthenticationhttps://www.chenshaowen.com/en/blog/several-methods-of-django-data-migration.htmlTue, 17 Jan 2017 23:29:15 +0000Tue, 17 Jan 2017 23:29:15 +0000https://www.chenshaowen.com/en/blog/several-methods-of-django-data-migration.html1. Background In the course of Web development, migrating data between multiple environments (local, testing, and production) comes up often. This article mainly discusses the possible data migration paths involved in Django development, and seeks feasible methods for them. 2. Scenarios There are four data migration targets in total: theWeChat Official AccountDjangoPythonDataMigrationR&Dhttps://www.chenshaowen.com/en/blog/thinking-on-the-cooperative-way-between-paas-and-saas-team.htmlTue, 10 Jan 2017 22:13:42 +0000Tue, 10 Jan 2017 22:13:42 +0000https://www.chenshaowen.com/en/blog/thinking-on-the-cooperative-way-between-paas-and-saas-team.html1. Work Scenario The team I am on has 20+ people and carries the responsibilities of both PaaS and SaaS development. The people who develop PaaS, the PaaSer, are mainly responsible for developing and maintaining the PaaS platform, with the goal of providing the SaaS developers with the APIs and documentation they need.WeChat Official AccountCloud ServicesThoughtsPaaSSaaSR&Dhttps://www.chenshaowen.com/en/blog/101-about-vuejs.htmlMon, 02 Jan 2017 19:55:16 +0000Mon, 02 Jan 2017 19:55:16 +0000https://www.chenshaowen.com/en/blog/101-about-vuejs.html1. What Problem We Ran Into If the features you implement are simple and both development and maintenance are easy, there would be no research into design patterns, frameworks, and related topics. It is precisely because Web systems are complex, requirements change quickly, there is a lot of reuse, there are many developers, and personnel handovers are frequent.WeChat Official AccountFrontendVueVuejsLearningR&Dhttps://www.chenshaowen.com/en/blog/automation-operations-with-python-technique-and-best-practices.htmlMon, 02 Jan 2017 18:12:29 +0000Mon, 02 Jan 2017 18:12:29 +0000https://www.chenshaowen.com/en/blog/automation-operations-with-python-technique-and-best-practices.htmlSubtitle: Technique and Best Practices English title: Automation Operations With Python Technique And Best Practices Author: Liu Tiansi Publisher: China Machine Press Published: 2014-11-1 ISBN: 9787111483069 Notes: Drawing on his own operations experience, the author describes in detail the use and configuration of a wide range of Python-based tools. It is a reference book well suited to Python operations personnel.WeChat Official AccountBooksPythonOperationsAutomationReadingLearningBest Practiceshttps://www.chenshaowen.com/en/blog/best-file-structure-of-django.htmlSun, 01 Jan 2017 15:07:01 +0000Sun, 01 Jan 2017 15:07:01 +0000https://www.chenshaowen.com/en/blog/best-file-structure-of-django.html1. What Django Is Django is a web development framework written in Python. It was initially used by its developers to manage news-content-focused websites, was open-sourced under the BSD license in July 2005, and formally established its foundation on June 17, 2008. Other Python web frameworks alongside Django include TornadoWeChat Official AccountDjangoPythonArchitectureR&Dhttps://www.chenshaowen.com/en/blog/what-is-devops.htmlSat, 31 Dec 2016 13:01:05 +0000Sat, 31 Dec 2016 13:01:05 +0000https://www.chenshaowen.com/en/blog/what-is-devops.html1. Overview DevOps (a portmanteau of Development and Operations) is a culture and a movement that values communication and collaboration between software developers (Dev) and IT operations staff (Ops). By automating the processes of “software delivery” and “architecture change,” it makes building, testing, and releasing software faster, more frequent, and more reliable.WeChat Official AccountDevelopmentThoughtsDevOpsLearningCultureAutomationOperationsR&Dhttps://www.chenshaowen.com/en/blog/how-to-create-excellent-product-interaction.htmlWed, 28 Dec 2016 20:05:13 +0000Wed, 28 Dec 2016 20:05:13 +0000https://www.chenshaowen.com/en/blog/how-to-create-excellent-product-interaction.htmlMy role covers full-stack development: I not only write backend logic in Python, but also build the frontend pages and interactions myself. The programs we develop are deployed on a PaaS platform, and since the developers are not professional frontend engineers, a Dreamweaver-like tool is provided to lower the barrier to entry.WeChat Official AccountSharingFrontendProductR&DInteractionUXhttps://www.chenshaowen.com/en/blog/study-of-bwapp.htmlSun, 25 Dec 2016 15:38:42 +0000Sun, 25 Dec 2016 15:38:42 +0000https://www.chenshaowen.com/en/blog/study-of-bwapp.html1. Introduction to BWAPP buggy web Application, abbreviated BWAPP, is an open-source web application that packs in all kinds of common and recent vulnerabilities, with the goal of helping security enthusiasts, developers, and students find and prevent web vulnerabilities. It contains more than 100 vulnerabilities, covering every major known web vulnerability, including the OWASP Top 10 security risks, and most importantly it already includes the OpenSSL and ShellShock vulnerabilities.WeChat Official AccountSecurityNetworkLearningBWAPPhttps://www.chenshaowen.com/en/blog/micro-service.htmlSun, 25 Dec 2016 00:19:01 +0000Sun, 25 Dec 2016 00:19:01 +0000https://www.chenshaowen.com/en/blog/micro-service.html1. SOA Service-Oriented Architecture (SOA) holds that a complex enterprise IT system should be divided along different, reusable granularities, organizing a group of functionally related service providers together to serve consumers. Its purpose is to solve the information silo problem caused by different IT resources within an enterprise being unable to interconnect.WeChat Official AccountMicroservicesServicesArchitectureSOAR&Dhttps://www.chenshaowen.com/en/blog/strategy-of-company-domains.htmlSat, 24 Dec 2016 21:14:05 +0000Sat, 24 Dec 2016 21:14:05 +0000https://www.chenshaowen.com/en/blog/strategy-of-company-domains.htmlToday’s enterprises run many lines of business and often need a large number of domains to support their growth. This article looks at four aspects — choosing enterprise domains, domains and brand building, site SEO, and site security — and asks how an enterprise should plan and build its own domain system.WeChat Official AccountDomain NamesOperationsSecuritySEOManagementhttps://www.chenshaowen.com/en/blog/book/convict-conditioning.htmlWed, 21 Dec 2016 19:38:05 +0000Wed, 21 Dec 2016 19:38:05 +0000https://www.chenshaowen.com/en/blog/book/convict-conditioning.htmlSubtitle: How to Build Superhuman Strength and Survival Power Using Lost Techniques Original title: Convict Conditioning Author: Paul Wade (USA) Publisher: Beijing Science and Technology Press Published: 2013-10-1 ISBN: 9787530467558WeChat Official AccountBooksFitnessExerciseReadingHealthhttps://www.chenshaowen.com/en/blog/burpsuite-web-attack-detection.htmlFri, 16 Dec 2016 16:15:54 +0000Fri, 16 Dec 2016 16:15:54 +0000https://www.chenshaowen.com/en/blog/burpsuite-web-attack-detection.html1. Introduction to Burpsuite Burp Suite is an integrated platform for attacking web applications. It contains many tools and designs many interfaces for those tools, in order to speed up the process of attacking an application. All the tools share one powerful, extensible framework that can handle and display HTTPWeChat Official AccountNetworkSecurityAttackWebBurpsuitehttps://www.chenshaowen.com/en/blog/a-successful-git-branching-model.htmlSun, 04 Dec 2016 20:34:03 +0000Sun, 04 Dec 2016 20:34:03 +0000https://www.chenshaowen.com/en/blog/a-successful-git-branching-model.htmlThis article is built around Git, the tool that controls all of our source code. 1. Why Git? For an in-depth discussion of the pros and cons of git versus centralized source control systems, see here. There has been far too much heated argument on this subject. As a developer, I currently prefer Git over other tools.WeChat Official AccountTranslationToolsGitR&DBranching Modelhttps://www.chenshaowen.com/en/blog/how-to-develop-chrome-plugin.htmlSun, 04 Dec 2016 15:19:06 +0000Sun, 04 Dec 2016 15:19:06 +0000https://www.chenshaowen.com/en/blog/how-to-develop-chrome-plugin.html1. Ways to Extend Chrome: Extensions, Plugins, Apps Google Chrome has come a long way. It is no longer just a browser but more like a platform, where you can install the services you need in a personalized way, and through which you can also publish services of your own. Today there are three main ways to extend the browser’s functionality:WeChat Official AccountDevelopmentFrontendChromeExtensionhttps://www.chenshaowen.com/en/blog/book/professional-javascript-for-web.htmlSat, 03 Dec 2016 11:44:19 +0000Sat, 03 Dec 2016 11:44:19 +0000https://www.chenshaowen.com/en/blog/book/professional-javascript-for-web.htmlEnglish title: Professional JavaScript for Web Author: [US] Nicholas C. Zakas Publisher: Turing Programming Series Publication year: 2012-3-29 ISBN: 9787115275790WeChat Official AccountBooksFrontendJavaScriptLearninghttps://www.chenshaowen.com/en/blog/js-event-handle-mechanism.htmlTue, 29 Nov 2016 21:45:06 +0000Tue, 29 Nov 2016 21:45:06 +0000https://www.chenshaowen.com/en/blog/js-event-handle-mechanism.html1. Event Handling Models: Bubbling and Capturing 1 2 3 <div id="outer"> <p id="inner">Click!</p> </div> In the code above, a div element contains a p child element. If both elements have a click handler, how do we know which one will be triggered first? To solve this problem, Microsoft and Netscape proposed two almost completely opposite concepts.WeChat Official AccountFrontendJavaScriptEventsR&Dhttps://www.chenshaowen.com/en/blog/python-coding.htmlTue, 08 Nov 2016 20:21:42 +0000Tue, 08 Nov 2016 20:21:42 +0000https://www.chenshaowen.com/en/blog/python-coding.htmlBackground: I had just finished a Django data-query web project. The data came from internal API queries, and every query had to call several APIs to fetch the data and render it on the front-end page. Since the relevant data does not change often, I designed a cache to improveWeChat Official AccountPythonEncodingR&DLearninghttps://www.chenshaowen.com/en/blog/using-celery-in-django.htmlTue, 25 Oct 2016 07:57:59 +0000Tue, 25 Oct 2016 07:57:59 +0000https://www.chenshaowen.com/en/blog/using-celery-in-django.htmlScenario one: the company is preparing to push an email announcement to all employees. Using a wrapped mail library API and looping over every employee to push the emails will leave the main program stuck in the loop and unable to respond, which is not what we want — we want to be able to carry on with other operations normally after clicking execute.WeChat Official AccountCeleryDjangoPythonAsynchronousR&Dhttps://www.chenshaowen.com/en/blog/bond-term.htmlFri, 26 Aug 2016 16:23:14 +0000Fri, 26 Aug 2016 16:23:14 +0000https://www.chenshaowen.com/en/blog/bond-term.html1. Class C Accounts Settlement members of the interbank bond market are divided into Class A members, Class B members, and Class C members. Class A accounts are mainly commercial banks; Class B accounts are rural credit institutions, securities firms, funds, and insurers; Class C accounts are mainly non-financial institutional legal entities, whose trading settlement must be entrusted to a Class A member to handle on their behalf.WeChat Official AccountBondsFinanceLearningTerminologyPersonal Financehttps://www.chenshaowen.com/en/blog/book/liars-poker.htmlWed, 24 Aug 2016 17:17:23 +0000Wed, 24 Aug 2016 17:17:23 +0000https://www.chenshaowen.com/en/blog/book/liars-poker.htmlSubtitle: Wall Street’s Investment Game English title: Liar’s Poker Author: [US] Michael Lewis Publisher: CITIC Press Publication year: 2007-1 ISBN: 9787508607559WeChat Official AccountBooksGame TheoryFinanceWall Streethttps://www.chenshaowen.com/en/blog/investment-clock.htmlWed, 24 Aug 2016 11:06:59 +0000Wed, 24 Aug 2016 11:06:59 +0000https://www.chenshaowen.com/en/blog/investment-clock.htmlThe Merrill Lynch Investment Clock theory is a method that links assets, industry rotation, the bond yield curve, and the four stages of the economic cycle. It is a very practical tool for guiding investment cycles. The analytical framework of the Merrill Lynch Investment Clock can help investors identify the important turning points of the economic cycle.WeChat Official AccountBooksInvestingFinanceEconomicshttps://www.chenshaowen.com/en/blog/book/core-python-programming-2nd.htmlTue, 23 Aug 2016 09:55:49 +0000Tue, 23 Aug 2016 09:55:49 +0000https://www.chenshaowen.com/en/blog/book/core-python-programming-2nd.htmlChinese Title: Python核心编程（第二版 English Title: Core Python Programming, 2nd Edition Author: [US] Wesley J. Chun Publication Year: 2008-06 ISBN: 9787115178503WeChat Official AccountBooksPythonLearningProgrammingReadinghttps://www.chenshaowen.com/en/blog/influence-of-new-gtld-on-the-internet.htmlThu, 18 Aug 2016 09:00:48 +0000Thu, 18 Aug 2016 09:00:48 +0000https://www.chenshaowen.com/en/blog/influence-of-new-gtld-on-the-internet.html1. The Past and Present of ICANN ICANN, The International Corporation for Assigned Names and Numbers, was founded in 1998 and is responsible for allocating Internet resources such as domain names and IP addresses. ICANN’s predecessor was IANA (Internet Assigned number Authority), which for a long time was funded directly by the U.WeChat Official AccountDomain NamesInternetOperationsThoughtshttps://www.chenshaowen.com/en/blog/book/the-troubled-wall-street.htmlThu, 18 Aug 2016 08:47:38 +0000Thu, 18 Aug 2016 08:47:38 +0000https://www.chenshaowen.com/en/blog/book/the-troubled-wall-street.htmlAuthor: Yu Yang Subtitle: The Experience of a Chinese Trader Publication Year: 2011-3-2 ISBN: 9787300132440 Notes: ROE = 净利润／股东权益(Net Profit / Equity) =（净利润/销售收入）*（销售收入WeChat Official AccountBooksFinanceTradinghttps://www.chenshaowen.com/en/blog/value-of-domain.htmlThu, 11 Aug 2016 09:05:06 +0000Thu, 11 Aug 2016 09:05:06 +0000https://www.chenshaowen.com/en/blog/value-of-domain.html1. What Is a Domain Name 1.1 The Composition of a Domain Name The complete domain name of this site should look like this: www.chenshaowen.com. Note the dot at the very end, which can usually be omitted. Reading a domain name from back to front, .com is the top-level domain, chenshaowen is the second-level domain, and www is the communication protocol.WeChat Official AccountDomain NamesValueOperationsLearninghttps://www.chenshaowen.com/en/blog/story-unicorn.htmlWed, 10 Aug 2016 14:07:30 +0000Wed, 10 Aug 2016 14:07:30 +0000https://www.chenshaowen.com/en/blog/story-unicorn.htmlLong, long ago, there was a unicorn forest with many, many great trees. In the forest lived a mother unicorn and a little unicorn, along with lots and lots of small animals. The mother unicorn had a very beautiful horn growing on her head, and it looked lovely there. Her horn was magical: when a little rabbit got hurt, the mother unicorn touched it with her horn, and the sparkles that fell from the horn landed on the rabbit’s wound, and the rabbit’s injury healed right away.WeChat Official AccountUnicornStoryLifeLiteratureParentinghttps://www.chenshaowen.com/en/blog/book/the-ascent-of-money.htmlTue, 26 Jul 2016 10:43:50 +0000Tue, 26 Jul 2016 10:43:50 +0000https://www.chenshaowen.com/en/blog/book/the-ascent-of-money.htmlChinese title: 货币崛起 - How Finance Affects World History English title: The Ascent of Money: A Financial History of the World Author: Niall Ferguson (UK) Publication Year: 2009-6 ISBN: 9787508615073 Notes: Money is in fact a matter of trust, even a kind of faith: trust is embodiedWeChat Official AccountBooksCurrencyFinanceHistoryhttps://www.chenshaowen.com/en/blog/ctp-101.htmlWed, 20 Jul 2016 08:55:13 +0000Wed, 20 Jul 2016 08:55:13 +0000https://www.chenshaowen.com/en/blog/ctp-101.html1. Introduction to the Comprehensive Trading Platform (CTP) The Comprehensive Trading Platform is a futures brokerage business management system developed by Shanghai Futures Information Technology Co., Ltd. for futures brokerage firms, based on the trading system of the Shanghai Futures Exchange. The whole system consists of three major subsystems: trading, risk control (abbreviated as risk control) and settlement.WeChat Official AccountProgram TradingTrading PlatformCompilationFinanceFuturesLearninghttps://www.chenshaowen.com/en/blog/book/reserve-crisis.htmlTue, 19 Jul 2016 08:34:34 +0000Tue, 19 Jul 2016 08:34:34 +0000https://www.chenshaowen.com/en/blog/book/reserve-crisis.htmlChinese title: 金融的本质 - Bernanke’s Four Lectures on the Federal Reserve English title: The Federal Reserve and the Financial Crisis Author: Ben S. Bernanke (US) Publication Year: 2014-4 ISBN: 9787508644097 Notes: The main cause of the 2008 financial crisis was this: financial institutions were overconfident inWeChat Official AccountBooksFinanceEconomicsFederal Reservehttps://www.chenshaowen.com/en/blog/book/positioning.htmlThu, 07 Jul 2016 08:21:53 +0000Thu, 07 Jul 2016 08:21:53 +0000https://www.chenshaowen.com/en/blog/book/positioning.htmlChinese title: 定位 - The Most Influential Marketing Concept in American History English title: Positioning - The Battle for Your Mind Author: Jack Trout / Al Ries Notes: For a new industry or product, the first company finds it easy to make a deep impression in the customer’s mind;WeChat Official AccountBooksMarketMarketingReadingBrandinghttps://www.chenshaowen.com/en/blog/source-analysis-of-ghost-1.htmlThu, 30 Jun 2016 08:53:53 +0000Thu, 30 Jun 2016 08:53:53 +0000https://www.chenshaowen.com/en/blog/source-analysis-of-ghost-1.htmlGhost version:“version”: “0.9.0-beta.2” |— content Content directory | |— apps directory, currently empty; apps may be deployed directly on Ghost in the future | |— data Database folder | |— images Image folder | |— themes Ghost theme folder |— core Core module directory | |— client Client-side code folderWeChat Official AccountGhostSource CodeNode.jsBlogLearningR&Dhttps://www.chenshaowen.com/en/blog/front-auto-build-tools-gulp.htmlTue, 28 Jun 2016 21:59:04 +0000Tue, 28 Jun 2016 21:59:04 +0000https://www.chenshaowen.com/en/blog/front-auto-build-tools-gulp.html1. Frontend Automated Build Tools When dealing with frontend scenarios, you write CSS with Less, HTML with Jade, modularize with Browserify, and add MD5 hashes to resources for non-overwriting deployments. If all this work were done purely by hand, the efficiency would be extremely low. A frontend automated build toolWeChat Official AccountFrontendgulpToolsR&DBuildLearningAutomationhttps://www.chenshaowen.com/en/blog/generate-letsencrypt-ssl.htmlTue, 28 Jun 2016 10:28:53 +0000Tue, 28 Jun 2016 10:28:53 +0000https://www.chenshaowen.com/en/blog/generate-letsencrypt-ssl.html1. SSL Certificates Let’s start with a few concepts. 1.1 Digital Certificates A digital certificate contains the public or private key used for encryption, along with some additional identity information. Anyone can use the relevant tools to generate their own digital certificate, for encrypting files and email, or for securing communications.WeChat Official AccountHTTPSSecuritySecurity CertificatesCertificatesOperationshttps://www.chenshaowen.com/en/blog/telnet-ssh-key.htmlMon, 27 Jun 2016 17:58:19 +0000Mon, 27 Jun 2016 17:58:19 +0000https://www.chenshaowen.com/en/blog/telnet-ssh-key.htmlThis article mainly introduces the communication principles of Telnet and SSH, and analyzes their working flow during communication. 1. Telnet No matter what type of terminal a Telnet protocol connection uses, communication is converted to NVT (Net Virtual Terminal) format. The Network Virtual Terminal, NVT, is the foundation of Telnet’s heterogeneous, cross-platform nature.WeChat Official AccountSSHTelnetProtocolsNetworkOperationshttps://www.chenshaowen.com/en/blog/anonymous-network.htmlMon, 27 Jun 2016 11:24:44 +0000Mon, 27 Jun 2016 11:24:44 +0000https://www.chenshaowen.com/en/blog/anonymous-network.htmlAnonymous communication was proposed by Chaum, who put forward an anonymous communication algorithm based on Mix nodes. A Mix node receives messages from multiple senders, mixes and processes these messages, and then transmits them to the receivers, thereby concealing the identity information of the senders and receivers and achieving anonymity.WeChat Official AccountAnonymous NetworkTorSecurityNetworkLearninghttps://www.chenshaowen.com/en/blog/book/peersinc.htmlMon, 27 Jun 2016 08:24:42 +0000Mon, 27 Jun 2016 08:24:42 +0000https://www.chenshaowen.com/en/blog/book/peersinc.htmlChinese title: 共享经济 - Rebuilding the Future Business Model English title: Peers Inc: How People and Platforms Are Inventing the Collaborative Economy and Reinventing Capitalism Author: Robin Chase (US) Publication Year: 2015-9-25 ISBN: 9787213067853 Notes: Taking Zipcar, the company she founded, as her entry point, and byWeChat Official AccountBooksSharingMarkethttps://www.chenshaowen.com/en/blog/nginx-open-http2.htmlThu, 23 Jun 2016 21:20:17 +0000Thu, 23 Jun 2016 21:20:17 +0000https://www.chenshaowen.com/en/blog/nginx-open-http2.htmlThis article mainly covers how to install Nginx from source on Linux and enable HTTP/2, and it also compares the speed of accessing this site over HTTP/1.1 versus HTTP/2.0. 1. Building and Installing Nginx First you need to download Nginx and a few of its dependencies. Zlib compresses data when a web request is transferred, saving bandwidth; PCRE matches URLs with regular expressions for routing and dispatch; OpenSSL or LibreSSL, the secure sockets algorithm library, provides HTTPS support.WeChat Official AccountNginxHTTP/2OperationsNetworkPerformance