https://www.chenshaowen.com/en/tags/learning/Recent content in Learning on Shaowen Chen's WebsiteHugo -- gohugo.ioen©2016 - {year}, All Rights Reserved.Sun, 01 Mar 2026 00:00:00 +0000weeklyhttps://www.chenshaowen.com/en/blog/using-skills-and-mcp-to-extend-openclaw.htmlSun, 01 Mar 2026 00:00:00 +0000Sun, 01 Mar 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-skills-and-mcp-to-extend-openclaw.html1. What Skills Are 1 2 3 openclaw skills list Skills (7/51 ready) OpenClaw ships with a number of built-in skills, which you can view with openclaw skills list. The concrete implementations of these Skills can be found at https://github.com/openclaw/openclaw/tree/main/skills. A Skills file mainly describes the Skill’s metadata, when it微信公众号AIAI AgentSkillsMCPOpenClawOperationsLearninghttps://www.chenshaowen.com/en/blog/installation-and-configuration-of-openclaw.htmlThu, 26 Feb 2026 00:00:00 +0000Thu, 26 Feb 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/installation-and-configuration-of-openclaw.html1. Preparing the Node Environment Install nvm 1 curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.4/install.sh | bash Set the nvm environment variables You need to edit Bash’s config file and add the following: 1 2 3 export NVM_DIR="$HOME/.nvm" [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" [ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion" Install Node 1 nvm install 22 default 2.微信公众号OpenClawAIAgentLLMToolsR&DLearninghttps://www.chenshaowen.com/en/blog/minio-multi-node-multi-disk-deployment-and-maintenance.htmlSat, 07 Feb 2026 00:00:00 +0000Sat, 07 Feb 2026 00:00:00 +0000https://www.chenshaowen.com/en/blog/minio-multi-node-multi-disk-deployment-and-maintenance.html1. Environment Preparation 1.1 Preparing Data Disks Inspect the data disks 1 lsblk -d -o NAME,SIZE,TYPE | grep nvme 1 2 3 4 nvme0n1 745.2G disk nvme1n1 745.2G disk nvme2n1 745.2G disk nvme3n1 745.2G disk Prepare the storage directories 1 2 3 for i in {0..3}; do mkdir -p /mnt/data${i} done Format the data disks 1 2 3 for i in {0.微信公众号MinIOStorageDistributed SystemsOperationsLearningObject StorageS3DeploymentHigh Availabilityhttps://www.chenshaowen.com/en/blog/kubernetes-base-environment-requirements.htmlThu, 04 Sep 2025 00:00:00 +0000Thu, 04 Sep 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/kubernetes-base-environment-requirements.html1. Base Packages Kubernetes version ≥ 1.18 socat Required conntrack Required ebtables Optional, but recommended ipset Optional, but recommended ipvsadm Optional, but recommended 1 apt-get -y install socat conntrack ebtables ipset ipvsadm 2. Port Requirements 2.1 Base Node Services Service Protocol Action Start Port End Port Comment ssh TCP allow 22 22 Node remote management docker TCP allow 2375 2376 Docker remote communication etcd TCP allow 2379 2380 etcd cluster communication 2.微信公众号KubernetesClusterConfigurationSecurityOperationsLearningInstallationDependenciesBest Practiceshttps://www.chenshaowen.com/en/blog/how-upgrade-kubernetes-node-cgroup-version.htmlTue, 02 Sep 2025 00:00:00 +0000Tue, 02 Sep 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-upgrade-kubernetes-node-cgroup-version.html1. Interface Path Differences Between cgroup v1 and v2 v1 1 2 3 4 /sys/fs/cgroup/cpu/cpu.cfs_quota_us /sys/fs/cgroup/cpu/cpuacct.usage /sys/fs/cgroup/memory/memory.limit_in_bytes /sys/fs/cgroup/memory/memory.usage_in_bytes v2 1 2 3 4 /sys/fs/cgroup/cpu.max /sys/fs/cgroup/cpu.stat /sys/fs/cgroup/memory.max /sys/fs/cgroup/memory.current cgroup v2 is the upgraded version of v1, with advantages such as more unified resource hierarchy management and precise resource isolation. But it also means that when writing code, the paths for reading the relevant interface files are different, so compatibility handling is required.微信公众号KubernetesUpgradeOperationscgroupLearningNodesResource IsolationTroubleshootingKubelethttps://www.chenshaowen.com/en/blog/using-gemini-cli-on-server.htmlFri, 04 Jul 2025 00:00:00 +0000Fri, 04 Jul 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-gemini-cli-on-server.html1. Get the Project ID and API Key Visit https://aistudio.google.com/apikey Get the project ID and API key, and set them as environment variables 1 2 echo "export GOOGLE_CLOUD_PROJECT=" >> ~/.bashrc echo "export GEMINI_API_KEY=" >> ~/.bashrc 1 source ~/.bashrc 2. Prepare the Node.js Environment Install nvm 1 curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.40.3/install.sh |微信公众号GeminiCLIAILLMLearningToolsOperationsProgramming Assistanthttps://www.chenshaowen.com/en/blog/nfs-over-rdma.htmlThu, 06 Mar 2025 00:00:00 +0000Thu, 06 Mar 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/nfs-over-rdma.html1. Prerequisites The storage and the nodes that use it form an RDMA network The RDMA devices are configured with IP addresses If you are using Mellanox NICs, add the -with-nfsrdma parameter when installing the driver 1 ./mlnxofedinstall --with-nfsrdma 2. Starting the Server Install dependencies 1 apt install nfs-kernel-server rdma-core -y Load the kernel module 1 modprobe svcrdma Add the port to portlist 1 echo 'rdma 20049' | tee /proc/fs/nfsd/portlist Export directory 1 mkdir /data1/nfs 1 2 3 vim /etc/exports /data1/nfs *(rw,sync,no_root_squash,no_all_squash) Start the NFS service 1 systemctl start nfs-server.微信公众号NFSRDMAHigh Performance StorageStorageNetworkOperationsLearninghttps://www.chenshaowen.com/en/blog/using-calico-to-limit-pod-bandwidth.htmlWed, 12 Feb 2025 00:00:00 +0000Wed, 12 Feb 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-calico-to-limit-pod-bandwidth.html1. Testing the Full Bandwidth Start the iperf3 server on the target host 1 2 3 4 5 iperf3 -s ----------------------------------------------------------- Server listening on 5201 ----------------------------------------------------------- Test from the client host 1 iperf3 -c x.x.x.x -p 5201 -t 10 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Connecting to host x.微信公众号CalicoPodBandwidthLimitingKubernetesNetworkOperationsLearningCNITraffic Controlhttps://www.chenshaowen.com/en/blog/structure-of-large-models-with-qwen.htmlSun, 09 Feb 2025 00:00:00 +0000Sun, 09 Feb 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/structure-of-large-models-with-qwen.html1. Introduction to the Qwen Model In April 2023, Alibaba released the beta version of Qwen. In December 2023, Alibaba open-sourced the first version of Qwen. In September 2024, Alibaba released Qwen2.5. In January 2025, Alibaba released Qwen 2.5-Max. Qwen 2.5 is the latest series of the Qwen large language微信公众号QwenLLMAILearningArchitectureTransformerOpen SourceR&Dhttps://www.chenshaowen.com/en/blog/use-nerdctl-to-build-multi-architecture-images.htmlFri, 17 Jan 2025 00:00:00 +0000Fri, 17 Jan 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/use-nerdctl-to-build-multi-architecture-images.html1. Installing Nerdctl Install Opscli 1 curl -sfL https://raw.githubusercontent.com/shaowenchen/ops/main/getcli.sh |VERSION=latest sh - Install Nerdctl 1 opscli task -f install-nerdctl --arch amd64 2. BuildKit Download Buildkit 1 wget https://github.com/moby/buildkit/releases/download/v0.19.0-rc2/buildkit-v0.19.0-rc2.linux-amd64.tar.gz Install Buildkit 1 2 tar xvf buildkit-*.tar.gz mv bin/* /usr/local/bin/ Configure Buildkitd 1 mkdir -p /etc/buildkit /data/buildkit 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 cat > /etc/buildkit/buildkitd.微信公众号Multi-ArchImagesCINerdctlKubernetesOperationsLearningCICDcontainerdBuildhttps://www.chenshaowen.com/en/blog/ai-application-development-tech-stack.htmlSun, 12 Jan 2025 00:00:00 +0000Sun, 12 Jan 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/ai-application-development-tech-stack.htmlEmbedding Models Embedding models map high-dimensional data into a lower-dimensional space, which makes the data easier to process and analyze. Text Models Here is a leaderboard, https://huggingface.co/spaces/mteb/leaderboard The leaderboard above gives each model’s score, parameter count, memory usage, vector dimension, maximum tokens, and other information. Below are some commonly used open-source models:微信公众号AIApplication DevelopmentTech StackLLMR&DLearningRAGEmbeddingBest Practiceshttps://www.chenshaowen.com/en/blog/how-to-deploy-jumpserver-on-kubernetes.htmlThu, 09 Jan 2025 00:00:00 +0000Thu, 09 Jan 2025 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-deploy-jumpserver-on-kubernetes.html1. Deploying Jumpserver You need a StorageClass ready in advance to store Jumpserver’s data. Besides the database mentioned below, each component — jms-core, jms-web, jms-koko, jms-lion, jms-chen — also needs a PV for storage. 1.1 Deploying MySQL Refer to https://github.com/shaowenchen/ops-hub/blob/main/database/mysql8.yaml to deploy MySQL. You need to adjust the StorageClass field微信公众号KubernetesJumpserverSecurityOperationsBastion HostDeploymentLearninghttps://www.chenshaowen.com/en/blog/use-iceberg-and-spark-on-kubernetes.htmlThu, 12 Sep 2024 00:00:00 +0000Thu, 12 Sep 2024 00:00:00 +0000https://www.chenshaowen.com/en/blog/use-iceberg-and-spark-on-kubernetes.html1. Data Processing Architecture It is mainly divided into four layers: Processing capability layer: Spark on Kubernetes provides streaming data processing capability Data management layer: Iceberg provides dataset access operations such as ACID and tables Storage layer: Hive MetaStore manages Iceberg table metadata, PostgreSQL serves as the storage backend for微信公众号SparkIcebergKubernetesBig DataData LakeOperationsData ProcessingLearninghttps://www.chenshaowen.com/en/blog/using-triton-server-and-tensorrt-llm-under-container.htmlSat, 03 Feb 2024 08:05:48 +0000Sat, 03 Feb 2024 08:05:48 +0000https://www.chenshaowen.com/en/blog/using-triton-server-and-tensorrt-llm-under-container.html1. Compiling Models with TensorRT-LLM 1.1 Introduction to TensorRT-LLM When using TensorRT, you usually need to convert the model to ONNX format, then convert the ONNX to TensorRT format, and finally run inference in TensorRT or Triton Server. But this conversion process is not simple, and you often run into微信公众号AITritonTensorRTLLMInferenceGPUOperationsContainersLearninghttps://www.chenshaowen.com/en/blog/how-to-access-gpt-service-on-wechat.htmlTue, 26 Dec 2023 00:00:00 +0000Tue, 26 Dec 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-access-gpt-service-on-wechat.htmlPaid integration service available at 200 RMB/year; technical support is also available at 200 RMB per session, capped at 1 hour. Follow the official account to get contact details. 1. Information You Need to Provide Go to https://mp.weixin.qq.com/ and open 【Settings and微信公众号LLMGPTOfficial AccountWeChatAIR&DLearningIntegrationOpenAIPracticehttps://www.chenshaowen.com/en/blog/large-model-application-design-and-implementation-guide.htmlSat, 23 Dec 2023 11:22:55 +0000Sat, 23 Dec 2023 11:22:55 +0000https://www.chenshaowen.com/en/blog/large-model-application-design-and-implementation-guide.html1. Problems with Using LLMs Directly Unstable output One characteristic of generative AI is the diversity of its output. Ask an LLM the same question several times and you may get different answers. This uncertainty in output is a pleasant surprise for users in conversation and creative scenarios. But in微信公众号LLMAIBest PracticesR&DArchitectureLearningRAGAgenthttps://www.chenshaowen.com/en/blog/chaos-engineering-and-practice.htmlSat, 16 Dec 2023 11:15:11 +0000Sat, 16 Dec 2023 11:15:11 +0000https://www.chenshaowen.com/en/blog/chaos-engineering-and-practice.htmlTwo months ago, I gave a talk to a business team about chaos engineering in practice. This post mainly organizes the content of that talk. Click to view the presentation 1. The Origin of Chaos 1.1 The Birth of the Discipline of Chaos Before discussing chaos, let us first think about the relationship between chaos, chaos engineering, and our production services.微信公众号ChaosPracticeProcessConventionsChaos EngineeringOperationsReliabilityLearningFault Drillshttps://www.chenshaowen.com/en/blog/source-analysis-kubernetes-management-of-pod-ip.htmlThu, 02 Nov 2023 19:45:31 +0000Thu, 02 Nov 2023 19:45:31 +0000https://www.chenshaowen.com/en/blog/source-analysis-kubernetes-management-of-pod-ip.html1. Subnet Management by kube-controller-manager kube-controller-manager has many controllers; the one related to Pod IP is NodeIpamController. The NodeIpamController mainly manages the podcidr of nodes: when a new node joins the cluster, it allocates a subnet to that node; when a node is deleted, it reclaims the subnet. The subnets微信公众号Source Code AnalysisKubernetesPodIPNetworkLearningOperationshttps://www.chenshaowen.com/en/blog/ai-basic-knowledge.htmlFri, 18 Aug 2023 11:22:55 +0000Fri, 18 Aug 2023 11:22:55 +0000https://www.chenshaowen.com/en/blog/ai-basic-knowledge.html1. Keywords Machine Learning (ML) The technology of automatically acquiring knowledge from data. Neural Network (NN) A model that imitates the structure and learning mechanism of biological neural networks; one of the branches of machine learning. The structure of a neural network consists of an input layer, hidden layers, and an output layer.微信公众号AILLMMachine LearningKnowledge PointsLearninghttps://www.chenshaowen.com/en/blog/call-functions-through-dialogue-using-openai-and-langchain.htmlWed, 16 Aug 2023 00:00:00 +0000Wed, 16 Aug 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/call-functions-through-dialogue-using-openai-and-langchain.html1. LLMs and Langchain Many people may never get the chance to train, or even fine-tune, a large model, but using large models is the wave of the future. So how should we embrace this change? The answer is Langchain. A large model provides a broad, general-purpose foundation. So far微信公众号OpenAILangchainAILLMR&DLearningFunction CallingApplication DevelopmentBest Practiceshttps://www.chenshaowen.com/en/blog/monitor-service-accessibility-with-upptime-for-free.htmlSat, 12 Aug 2023 00:00:00 +0000Sat, 12 Aug 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/monitor-service-accessibility-with-upptime-for-free.html1. What Needs Dial Testing GPT was all the rage this year, and I deployed and developed several applications and mini programs to learn from. Of course, following the principle of helping vendors test their features, right now only the GPT 3.5 API costs a little every day — the微信公众号GitHubActionsMonitoringDial TestingOperationsLearningAvailabilityFreeCICDBest Practiceshttps://www.chenshaowen.com/en/blog/how-to-set-hpa-for-kubernetes-app.htmlThu, 08 Jun 2023 00:00:00 +0000Thu, 08 Jun 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-set-hpa-for-kubernetes-app.html1. Business Background Once an enterprise reaches a certain scale, relying entirely on public cloud infrastructure makes IT costs very high. The cost of purchasing physical machines can be amortized over the next 3 to 5 years, and afterwards the machines are not scrapped; instead they continue to serve beyond their planned lifespan.微信公众号KubernetesHPAPodElastic ScalingOperationsHybrid CloudLearninghttps://www.chenshaowen.com/en/blog/autoscale-kubernetes-applications-with-keda.htmlThu, 18 May 2023 00:00:00 +0000Thu, 18 May 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/autoscale-kubernetes-applications-with-keda.html1. HPA VS KEDA HPA also provides: Elasticity based on custom metrics Scale to Zero Compared with KEDA, these are no longer disadvantages. The real difference is that HPA can only scale using monitoring data, whereas KEDA can scale using many more data sources — queue messages, databases, Redis, and so on, including monitoring data as well.微信公众号KubernetesKEDAAutoscalingElastic ScalingOperationsLearningHPAEvent-Drivenhttps://www.chenshaowen.com/en/blog/custom-cluster-scheduler-to-optimize-tekton.htmlWed, 26 Apr 2023 10:00:00 +0000Wed, 26 Apr 2023 10:00:00 +0000https://www.chenshaowen.com/en/blog/custom-cluster-scheduler-to-optimize-tekton.html1. A Constrained Build Environment Cannot Meet Build Requirements Tekton is a CICD engine based on a Kubernetes cluster, and it is more cloud-native than Jenkins. In plain terms, that means it is easier to develop plugins for, easier to scale, easier to observe, and more fun. Because code can only be stored on the company intranet, the build cluster can only be deployed on the office intranet.微信公众号OptimizationTektonCICDDevOpsKubernetesOperationsLearningSchedulingBuildMulti-Clusterhttps://www.chenshaowen.com/en/blog/developing-istio-wasm-plugin-by-tinygo.htmlWed, 22 Feb 2023 01:00:00 +0000Wed, 22 Feb 2023 01:00:00 +0000https://www.chenshaowen.com/en/blog/developing-istio-wasm-plugin-by-tinygo.htmlwasme only supports up to Istio 1.9, while I am using Istio 1.14, so this post goes straight to tinygo for verification and learning. 1. Installing tinygo Requirements Go v1.18+ Install tinygo 1 2 brew tap tinygo-org/tools brew install tinygo Check the version 1 2 3 tinygo version tinygo version 0.微信公众号tinygoIstioWasmCloud NativeService MeshR&DGoLearningExtensionhttps://www.chenshaowen.com/en/blog/how-to-repair-k8s-master-after-reinstalling-os.htmlTue, 17 Jan 2023 00:00:00 +0000Tue, 17 Jan 2023 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-repair-k8s-master-after-reinstalling-os.htmlRecently I ran into this twice: a host had to be reinstalled because of a failure. In one case only a single Etcd node was left, and the whole cluster stayed down for half an hour before it recovered. This post mainly records the process of initializing a fresh Ubuntu微信公众号KubernetesRepairOperationsEtcdTroubleshootingClusterKubekeyLearninghttps://www.chenshaowen.com/en/blog/kubernetes-cluster-troubleshooting.htmlTue, 01 Nov 2022 01:00:00 +0000Tue, 01 Nov 2022 01:00:00 +0000https://www.chenshaowen.com/en/blog/kubernetes-cluster-troubleshooting.htmlFailedCreatePodSandBox Error Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:402: getting the final child's pid from pipe caused: EOF: unknown Resolution Clear the cache 1 echo 3 > /proc/sys/vm/drop_caches Cause Too much memory fragmentation calico-node Keeps Restarting increase max user Error runtime: failed to微信公众号KubernetesClusterTroubleshootingOperationsLearningFault DiagnosisPodBest PracticesProblem Localizationhttps://www.chenshaowen.com/en/blog/how-to-repair-the-kubernetes-cluster-after-changing-ip.htmlTue, 25 Oct 2022 00:00:00 +0000Tue, 25 Oct 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-repair-the-kubernetes-cluster-after-changing-ip.htmlA record of repairing a cluster failure caused by an IP change. There are two clusters: one single-node (allinone) cluster, and one four-node cluster (3 masters, 1 node). 1. Update the Etcd Certificates [On every Etcd node] Back up the Etcd certificates 1 cp -R /etc/ssl/etcd/ssl /etc/ssl/etcd/ssl-bak View the domains微信公众号KubernetesOperationsRepairTroubleshootingLearningEtcdCertificatesClusterNetworkhttps://www.chenshaowen.com/en/blog/kubernetes-network-packets.htmlSat, 17 Sep 2022 00:00:00 +0000Sat, 17 Sep 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/kubernetes-network-packets.htmlThis article is translated from https://learnk8s.io/kubernetes-network-packets, not word for word, with some of my own understanding worked in. By reading this article, you can learn how packets are forwarded inside and outside Kubernetes, starting from the original web request all the way to the container hosting the application. Kubernetes Network Requirements Before diving into the details of how packets flow through a Kubernetes cluster, let’s first clarify Kubernetes’ requirements for networking.微信公众号TranslationKubernetesNetworkOperationsLearningiptablesCNIhttps://www.chenshaowen.com/en/blog/traffic-restriction-using-linux-tc.htmlSat, 20 Aug 2022 00:00:00 +0000Sat, 20 Aug 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/traffic-restriction-using-linux-tc.html1. Traffic Control Principles Under Linux By queuing packets, we can control how data packets are sent. This kind of control is called data shaping (shape the data), and it includes the following operations on data: Adding latency Dropping packets Reordering Duplicating, corrupting Rate control Under the qdisc-class-filter structure, controlling traffic requires three steps:微信公众号LinuxTCTraffic ControlOperationsNetworkLearningRate LimitingTroubleshootinghttps://www.chenshaowen.com/en/blog/a-image-management-and-distribution-case-based-on-harbor-and-registry.htmlSat, 16 Jul 2022 00:00:00 +0000Sat, 16 Jul 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/a-image-management-and-distribution-case-based-on-harbor-and-registry.html1. The Challenge of a Cross-Region Harbor If all you need is to store image data in a simple way, Registry is an excellent choice as an image repository. Registry not only supports multiple storage backends, but can also be configured with HTTPS certificates and access credentials. Worth noting: Harbor微信公众号HarborRegistryImagesDistributionKubernetesOperationsLearningMulti-RegionAccelerationCICDhttps://www.chenshaowen.com/en/blog/how-to-set-up-pod-to-run-to-a-specified-node.htmlMon, 27 Jun 2022 00:00:00 +0000Mon, 27 Jun 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-set-up-pod-to-run-to-a-specified-node.html1. Specify a Node via nodeSelector When Creating a Workload Add a label to the node 1 kubectl label node node2 project=A Create the workload with the specified nodeSelector 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 cat <<EOF | kubectl apply -f - apiVersion: apps/v1 kind: Deployment metadata: name: nginx-nodeselector spec: replicas: 1 selector: matchLabels: app: nginx-nodeselector template: metadata: labels: app: nginx-nodeselector spec: nodeSelector: project: A containers: - name: nginx image: nginx EOF Check the workload 1 2 3 4 kubectl get pod -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-nodeselector-7bb75b7687-7r5xk 1/1 Running 0 19s 10.微信公众号PodKubernetesSchedulingOperationsnodeSelectorLearninghttps://www.chenshaowen.com/en/blog/using-blackbox-exporter-to-test-network.htmlSat, 25 Jun 2022 00:00:00 +0000Sat, 25 Jun 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-blackbox-exporter-to-test-network.htmlIf you need to monitor the network status between two hosts, or between a host and an external service, then the Blackbox Exporter mentioned in this article is worth a try. 1. Installing Blackbox 1.1 Deploying on a Host Download the binary package 1 2 3 4 5 wget https://github.微信公众号PrometheusMonitoringProbingOperationsLearningNetworkBlackbox ExporterAvailabilityhttps://www.chenshaowen.com/en/blog/how-to-collect-labels-and-annotations-of-kubernetes-objects.htmlThu, 02 Jun 2022 01:00:00 +0000Thu, 02 Jun 2022 01:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-collect-labels-and-annotations-of-kubernetes-objects.html1. Why kube-status-metrics is needed Kubernetes monitoring mainly focuses on two kinds of metrics: Basic performance metrics CPU, memory, disk, network, and so on. These can be collected by deploying node-exporter as a DaemonSet and having Prometheus scrape the relevant metrics. Resource object metrics The replica count of a Deployment, the running state of a Pod, and so on.微信公众号KubernetesMonitoringMetricsCollectionPrometheusOperationsLearningObservabilitykube-status-metricshttps://www.chenshaowen.com/en/blog/a-global-images-distribution-network.htmlSun, 27 Mar 2022 14:00:00 +0000Sun, 27 Mar 2022 14:00:00 +0000https://www.chenshaowen.com/en/blog/a-global-images-distribution-network.html1. Global Network Planning Many global, multi-region infrastructures do not put much thought into network planning at the outset. Once the business grows complex enough, they are forced into network adjustments and optimization. Yet any major network change has a huge impact on the business. In the end you are stuck in a dilemma, able to do nothing but invest more people, carry the baggage of history, and walk along the edge of a cliff again and again.微信公众号ImagesHarborNetworkKubernetesOperationsDistributionMulti-RegionLearninghttps://www.chenshaowen.com/en/blog/interviewer-training-about-how-to-choose-right-people.htmlTue, 18 Jan 2022 00:00:00 +0000Tue, 18 Jan 2022 00:00:00 +0000https://www.chenshaowen.com/en/blog/interviewer-training-about-how-to-choose-right-people.html1. Interview Process Establish hiring criteria Align on hiring criteria Resume analysis Opening small talk Competency assessment Values assessment Verifying authenticity Motivation assessment Attracting strong candidates Closing the interview Filling out the interview evaluation form Assisting with onboarding follow-up 2. Before the Interview 2.1 Establish Hiring Criteria why Evaluate candidates with a unified assessment standard微信公众号InterviewTrainingManagementTeamsRecruitingTalentLearningHuman Resourceshttps://www.chenshaowen.com/en/blog/common-commands-for-cleaning-up-kubernetes-cluster-resources.htmlWed, 08 Dec 2021 00:00:00 +0000Wed, 08 Dec 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/common-commands-for-cleaning-up-kubernetes-cluster-resources.htmlClusters that run for a long time often face various kinds of resource exhaustion. On top of that, when disk space runs low, Kubelet will proactively clean up images on its own, adding another source of uncertainty. This article provides some command snippets for cleanup work. 1. Cleaning Up Basic Kubernetes Objects Clean up Pods in Evicted state 1 sudo kubectl get pods --all-namespaces -o wide | grep Evicted | awk '{print $1,$2}' | sudo xargs -L1 kubectl delete pod -n Clean up Pods in Error state 1 sudo kubectl get pods --all-namespaces -o wide | grep Error | awk '{print $1,$2}' | sudo xargs -L1 kubectl delete pod -n Clean up Pods in Completed state 1 sudo kubectl get pods --all-namespaces -o wide | grep Completed | awk '{print $1,$2}' | sudo xargs -L1 kubectl delete pod -n Clean up unused PVs 1 sudo kubectl describe -A pvc | grep -E "^Name:.微信公众号KubernetesPracticeCleanupOperationsLearningkubectlTroubleshootingResource Managementhttps://www.chenshaowen.com/en/blog/multi-cluster-applications-under-kubevela.htmlFri, 17 Sep 2021 00:00:00 +0000Fri, 17 Sep 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/multi-cluster-applications-under-kubevela.htmlKubevela is currently at version 1.1. We usually consider 1.x releases to be relatively stable and safe to try in production. Through continuous tracking and learning, I have also come to appreciate some of the things Kubevela does well. This is a summary document. 1. What Problem Kubevela Solves Aimed at platform developers Several roles need to be distinguished: development, operations, and operations development.微信公众号KubevelaKubernetesApplicationOperationsMulti-ClusterOAMLearningCloud Nativehttps://www.chenshaowen.com/en/blog/a-gateway-under-kubernetes-named-apisix.htmlWed, 01 Sep 2021 00:00:00 +0000Wed, 01 Sep 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/a-gateway-under-kubernetes-named-apisix.html1. Comparing a Few Common Gateways Nginx, a reverse proxy with a modular design, written in C OpenResty, a web development platform built around Nginx that can parse and execute Lua scripts Kong, an application on top of OpenResty, an API gateway with API management and request proxying, using PostgreSQL for storage APISIX, which replaces Kong’s PostgreSQL with Etcd and is built on Nginx’s core libraries APISIX’s advantage lies in the API management and extensibility it provides, so the gateway no longer merely forwards traffic to services but can be configured and customized.微信公众号KubernetesAPISIXMicroservicesGatewayOperationsProxyLearningCloud Nativehttps://www.chenshaowen.com/en/blog/a-cicd-system-based-on-tekton.htmlWed, 25 Aug 2021 00:00:00 +0000Wed, 25 Aug 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/a-cicd-system-based-on-tekton.html1. Background 1.1 Problems with the Current Use of Jenkins The orchestration engine is unstable Jenkins is an orchestration engine written in Java, and it stops the world (STW) during a full GC. Under large-scale builds, STW can prevent Jenkins from handling new requests. Heavy builds stall Jenkins stores data in disk files; every pipeline and every build occupies a file directory, producing a huge number of files.微信公众号CICDDevOpsTektonKubernetesOperationsPipelineLearninghttps://www.chenshaowen.com/en/blog/using-kubefed-to-distribute-tekton-resource-cross-cluster.htmlSat, 26 Jun 2021 00:00:00 +0000Sat, 26 Jun 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-kubefed-to-distribute-tekton-resource-cross-cluster.html1. The Advantages of Multi-Cluster Tekton Builds Thanks to Kubernetes, Tekton already has good elasticity and can support large-scale builds. At the same time, developing Tasks mainly uses Yaml and Shell, which widens the range of scenarios Tekton can adapt to. The figure above is a sketch of Tekton across multiple clusters.微信公众号TektonKubeFedKubernetesMulti-ClusterCICDOperationsLearningPipelineBuildDistributed Systemshttps://www.chenshaowen.com/en/blog/how-to-configure-networkpolicy-for-nodeport.htmlWed, 23 Jun 2021 00:00:00 +0000Wed, 23 Jun 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-configure-networkpolicy-for-nodeport.html1. Background As shown above, the business team needs to isolate the services in a namespace, forbidding access from workloads in the bar namespace while allowing users to reach the service through a Load Balancer (LB) via NodePort. It is easy to write a network policy: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 apiVersion: networking.微信公众号NetworkPolicyNetworkKubernetesOperationsSecurityLearningNodePortNetwork Isolationhttps://www.chenshaowen.com/en/blog/how-to-distribute-files-and-scripts-to-vm-under-kubernetes.htmlSat, 24 Apr 2021 00:00:00 +0000Sat, 24 Apr 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-distribute-files-and-scripts-to-vm-under-kubernetes.html1. What This Article Mainly Discusses Never build a high tower on shifting sand. Business growth and the evolution of business forms both need solid, powerful IT systems to support them. Business content is transparent to the market, but IT systems cannot be built to perfection overnight. In the future, competition between companies will mainly come from competition between their IT systems, and the ability to respond quickly to business requirements will be the key to winning.微信公众号KubernetesDockerPaaSOperationsConfiguration ManagementVirtual MachinesLearningCloud Nativehttps://www.chenshaowen.com/en/blog/how-to-build-istio-under-centos.htmlThu, 08 Apr 2021 00:00:00 +0000Thu, 08 Apr 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-build-istio-under-centos.htmlThis article mainly describes how to build the https://github.com/istio/istio project on CentOS 8.2 using local tools. Another approach is to use the BUILD_WITH_CONTAINER parameter to build inside a container, which depends only on make and docker. 1. Install and Upgrade Ruby >= 2.6 Install Ruby 1 yum install -y ruby Check the Ruby version 1 2 3 ruby -v ruby 2.微信公众号CentOSIstioKubernetesOperationsLearningService MeshCompilationOpen Sourcehttps://www.chenshaowen.com/en/blog/install-homebrew-in-centos-7.htmlSun, 28 Mar 2021 00:00:00 +0000Sun, 28 Mar 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/install-homebrew-in-centos-7.htmlWhen installing packages on CentOS you usually have to hunt down all sorts of repositories; with Homebrew that hassle goes away. This article uses CentOS 7.6 — other system versions may differ. 1. Create a Non-root User Homebrew refuses to run as root, so a new user has to be created.微信公众号CentOSEnvironmentHomebrewOperationsPackage ManagementLearninghttps://www.chenshaowen.com/en/blog/argocd-of-devops-tool-chain.htmlThu, 04 Feb 2021 00:00:00 +0000Thu, 04 Feb 2021 00:00:00 +0000https://www.chenshaowen.com/en/blog/argocd-of-devops-tool-chain.html1. What Problem Argo CD Solves 1.1 Starting from GitOps GitOps originated from a blog post published by Weaveworks in 2017: GitOps - Operations by Pull Request. In it, Alexis introduced a deployment approach that treats Git as the single source of truth. In GitOps practice, we need to define software infrastructure in a Git repository for management.微信公众号DevOpsArgoCDKubernetesCICDGitOpsOperationsLearningDeploymentBest Practiceshttps://www.chenshaowen.com/en/blog/how-to-use-docker-in-docker.htmlSat, 21 Nov 2020 00:00:00 +0000Sat, 21 Nov 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-use-docker-in-docker.html1. Typical Use Cases In CI, there is usually a CI Engine responsible for parsing the pipeline and controlling the whole build process, while the actual build is delegated to an Agent. Jenkins and GitLab both work this way. As shown below, there are many kinds of Agents that connect to the CI Engine.微信公众号DockerKubernetesContainersCICICDOperationsLearningDinDBuildJenkinshttps://www.chenshaowen.com/en/blog/let-us-start-learning-go-6.htmlFri, 04 Sep 2020 00:00:00 +0000Fri, 04 Sep 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/let-us-start-learning-go-6.html1. Interface-Oriented Programming 1.1 Characteristics Interface-oriented programming emphasizes interaction between modules through interfaces. First, the caller specifies a set of method signatures; then, the callee implements that set of methods. What sets interface programming apart from other programming styles is that it focuses on methods, not attributes. In many programming scenarios, methods are defined around attributes.微信公众号GoLet's Learn GoInterfaceLearninghttps://www.chenshaowen.com/en/blog/let-us-start-learning-go-5.htmlThu, 03 Sep 2020 00:00:00 +0000Thu, 03 Sep 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/let-us-start-learning-go-5.html1. The Concurrency Model in Go 1.1 The Communication Model: CSP CSP stands for Communicating Sequential Process, a model for concurrent communication. A Process can use many Channels, and a Channel does not care who is using it — it is only responsible for sending and receiving data. In the Go community there is a very famous maxim: do not communicate by sharing memory; instead, share memory by communicating.微信公众号GoConcurrencyParallelismGoroutineChannelLearningLet's Learn Gohttps://www.chenshaowen.com/en/blog/the-use-of-etcd-and-etcdctl.htmlTue, 11 Aug 2020 00:00:00 +0000Tue, 11 Aug 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/the-use-of-etcd-and-etcdctl.html1. Basic Introduction to Etcd Etcd is a distributed Key/Value storage system. Through distributed locks, leader election, and write barriers it achieves distributed coordination, offering highly available, persistent data storage and retrieval services. How it works Every Etcd node stores a complete copy of the data, and at any moment微信公众号EtcdEtcdctlKubernetesOperationsLearningDistributed StorageConfigurationClusterhttps://www.chenshaowen.com/en/blog/how-to-develop-the-plugin-of-jenkins.htmlSat, 20 Jun 2020 00:00:00 +0000Sat, 20 Jun 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-develop-the-plugin-of-jenkins.htmlUsing Jenkins always means dealing with all kinds of plugins. To practice DevOps better, we should also be capable of developing plugins ourselves, so the whole process can converge inside Jenkins. 1. Jenkins Plugins 1.1 The Plugin Ecosystem Jenkins’ predecessor Hudson started in 2004, and 16 years on it is still a mainstream CI/CD engine.微信公众号JenkinsDevOpsCICDR&DPluginJavaLearninghttps://www.chenshaowen.com/en/blog/affinity-of-kubernetes-scheduler.htmlSat, 30 May 2020 00:00:00 +0000Sat, 30 May 2020 00:00:00 +0000https://www.chenshaowen.com/en/blog/affinity-of-kubernetes-scheduler.html1. The Scheduler in Kubernetes kube-scheduler is the component in Kubernetes that decides which Node a Pending Pod runs on; it is called the scheduler. Kubernetes ships with a large number of built-in scheduling policies and also provides some advanced scheduling policies (nodeAffinity, podAffinity, and so on) for users, which微信公众号KubernetesSchedulingAffinityLearningOperationshttps://www.chenshaowen.com/en/blog/backup-kubernetes-cluster-using-velero.htmlWed, 25 Dec 2019 00:00:00 +0000Wed, 25 Dec 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/backup-kubernetes-cluster-using-velero.html1. Introduction to Velero Velero is an open-source Kubernetes cluster backup and migration tool from the heptio team (acquired by VMWare). Velero stores cluster resources in object storage. It supports AWS, Azure, and GCP object storage by default, is compatible with the S3 protocol, and can be extended to other platforms such as Aliyun OSS through plugins.微信公众号KubernetesVeleroBackupOperationsLearningDisaster RecoveryMigrationClusterhttps://www.chenshaowen.com/en/blog/basic-operation-of-deployment-in-kubernetes.htmlFri, 27 Sep 2019 00:00:00 +0000Fri, 27 Sep 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/basic-operation-of-deployment-in-kubernetes.htmlA Deployment controls the number and state of Pods by creating ReplicaSets. This post mainly introduces some common Deployment operations. 1. Deployment yaml Format Passing the --dry-run flag means the command is not actually executed, only YAML output is generated: 1 kubectl create deployment nginx --image=nginx --dry-run -o yaml apiVersion: apps/v1 kind: Deployment metadata: creationTimestamp: null labels: app: nginx name: nginx spec: replicas: 1 selector: matchLabels: app: nginx strategy: {} template: metadata: creationTimestamp: null labels: app: nginx spec: containers: - image: nginx name: nginx resources: {} status: {} 2.微信公众号KubernetesDeploymentDeploymentLearningOperationshttps://www.chenshaowen.com/en/blog/how-to-develop-a-operator-using-kubebuilder.htmlWed, 25 Sep 2019 00:00:00 +0000Wed, 25 Sep 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-develop-a-operator-using-kubebuilder.htmlWith the Operator approach, Kubernetes functionality can be extended in a friendly way. Operator = CRD + Controller. First you generate the CRD from a yaml definition, then the Controller continuously watches the data in etcd and performs the corresponding actions. There is a lot of tedious and repetitive work involved in developing an Operator.微信公众号KubernetesGoOperatorR&DOperationsCRDLearninghttps://www.chenshaowen.com/en/blog/using-emptydir-hostpath-localvolume-in-kubernetes.htmlSat, 24 Aug 2019 00:00:00 +0000Sat, 24 Aug 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-emptydir-hostpath-localvolume-in-kubernetes.htmlEarlier, in Kubernetes Volumes, we got some understanding of Volumes. This post focuses on practice, learning how to use the three local storage options: emptydir, hostpath, and localvolume. 1. Basic Properties of a PV 1.1 PV Lifecycle PV states: Available: available, not yet bound by any PVC Bound: already bound to a PVC Released: the PVC was deleted, but the resource has not yet been reclaimed Failed: automatic reclamation failed 1.微信公众号KubernetesVolumeStorageOperationsPVCLearninghttps://www.chenshaowen.com/en/blog/using-kubeadm-to-install-the-kubernetes-cluster.htmlThu, 15 Aug 2019 00:00:00 +0000Thu, 15 Aug 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/using-kubeadm-to-install-the-kubernetes-cluster.html1. Cluster Planning Prepare three hosts: one Master and two Nodes. Operating system: CentOS 7 Specs: 2 Core 4 GB Docker version: 18.06.3 Kubernetes version: 1.15.3 If you are using a purchased cloud host, open the following ports: 1 2 3 4 5 6 7 8 9 10 11 12微信公众号KubernetesKubeadmInstallationOperationsClusterLearninghttps://www.chenshaowen.com/en/blog/complex-application-management-framework-operator-for-kubernetes.htmlThu, 25 Jul 2019 00:00:00 +0000Thu, 25 Jul 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/complex-application-management-framework-operator-for-kubernetes.html1. Origins Originally, two CoreOS employees, in order to make deploying etcd clusters easier, bound corresponding logical operations to the create, delete, and update events of the etcdCluster object, using Kubernetes to automate etcd cluster management. A few months later, at a KubeCon conference, they shared this approach, which they called Operator, and it received a strong response from the community.微信公众号KubernetesHelmPracticeOperatorOperationsStateful ApplicationsLearninghttps://www.chenshaowen.com/en/blog/package-manager-helm-of-kubernetes.htmlWed, 24 Jul 2019 00:00:00 +0000Wed, 24 Jul 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/package-manager-helm-of-kubernetes.html1. Why Helm Is Needed One important design philosophy in Kubernetes is declarative operation. Users change the system by setting the system’s expected state. For example, the current replica count is 2 and it needs to be adjusted to 3. The declarative way is to modify the replica count in微信公众号KubernetesHelmPracticeOperationsPackage ManagementLearningChartDeploymenthttps://www.chenshaowen.com/en/blog/let-us-start-learning-go-3.htmlTue, 23 Jul 2019 00:00:00 +0000Tue, 23 Jul 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/let-us-start-learning-go-3.html1. Go’s Package Management Mechanisms 1.1 GOPATH GOPATH pulls code with the go get command and places it in the GOPATH directory. The problems with GOPATH are: It cannot manage package versions It uses a global repository, so it cannot isolate projects effectively 1.2 Vendor Starting with version 1.5, Go微信公众号GoGoModulesLet's Learn GoLearningR&DPackage Managementhttps://www.chenshaowen.com/en/blog/practice-guide-to-kubectl.htmlFri, 19 Jul 2019 01:00:00 +0000Fri, 19 Jul 2019 01:00:00 +0000https://www.chenshaowen.com/en/blog/practice-guide-to-kubectl.html1. What Is kubectl kubectl is the command-line tool for Kubernetes. It interacts with the cluster through the API server. 2. Configuring kubectl kubectl can be configured through ~/.kube/config to connect to one or more clusters. For details on how to configure it, see: Configure Access to Multiple Clusters. If you need to configure a remote cluster, see: Building a Remote Kubernetes Development Environment.微信公众号KuberneteskubectlPracticeLearningOperationshttps://www.chenshaowen.com/en/blog/mock-of-pytest.htmlSun, 27 Jan 2019 00:00:00 +0000Sun, 27 Jan 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/mock-of-pytest.html1. Why We Need Mock When writing unit tests, the function under test is sometimes not a self-contained executable unit. It depends on external resources, such as the return value of another function or a value in a row of a database. To shield the test from interference from external微信公众号TestingPythonPytestLearninghttps://www.chenshaowen.com/en/blog/buildpack-for-paas-deployment.htmlThu, 24 Jan 2019 00:00:00 +0000Thu, 24 Jan 2019 00:00:00 +0000https://www.chenshaowen.com/en/blog/buildpack-for-paas-deployment.htmlI have been learning Go recently, and the internal PaaS platform we commonly use happens to support Go as well as related Web frameworks. A PaaS system that supports multiple languages cannot do without the buildpack mechanism. Although PaaS platforms keep being upgraded, the buildpack mechanism has always been retained.微信公众号PaaSBuildpackOperationsR&DDeploymentLearningCloud NativeMulti-languagehttps://www.chenshaowen.com/en/blog/101-of-pytest.htmlSat, 29 Dec 2018 00:00:00 +0000Sat, 29 Dec 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/101-of-pytest.htmlThe Pytest framework is simple to use, rich in plugins, and powerful, and it is widely used for Python automated testing. This article introduces some basic concepts and usage of Pytest. 1. How It Runs Step one, Pytest reads its configuration from the command line or a file. Step two,微信公众号TestingPythonPytestR&DLearningAutomated TestingUnit TestingBest Practiceshttps://www.chenshaowen.com/en/blog/development-tools-and-productivity-tips.htmlFri, 02 Nov 2018 00:00:00 +0000Fri, 02 Nov 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/development-tools-and-productivity-tips.htmlCompiled from the “Development Tips” series, collecting common development tools and productivity-related techniques. 1. The Babel Transpiler ES6 provides many new features, but not every browser supports them perfectly, while ES5 support is far better. Babel is an ES6 transpiler that converts ES6 code into ES5 code. This means you微信公众号ToolsR&DProductivityVSCodeLearninghttps://www.chenshaowen.com/en/blog/python-and-django-development-practices.htmlFri, 19 Oct 2018 00:00:00 +0000Fri, 19 Oct 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/python-and-django-development-practices.htmlCompiled from the “Development Tips” series, collecting common problems and solutions in Python and Django development. 1. Sorting Python Iterables Iterable objects fall into 3 main categories: All sequence types, such as list, str, and tuple Some non-sequence types, such as dict and file Objects that contain an __iter__() or微信公众号PythonDjangoR&DLearningBackendhttps://www.chenshaowen.com/en/blog/open-k8s-monitoring-and-stress-testing..htmlMon, 24 Sep 2018 00:00:00 +0000Mon, 24 Sep 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/open-k8s-monitoring-and-stress-testing..html1. Kubectl Basic Commands 1.1 Creating Objects 1 2 3 4 5 6 7 8 # Create resources; a remote URL also works kubectl create -f ./my.yaml # Create resources from multiple files kubectl create -f ./my1.yaml -f ./my2.yaml # Create resources using all manifest files in a directory kubectl create -f .微信公众号KubernetesTestingMonitoringOperationsLearningkubectlStress TestingObservabilityhttps://www.chenshaowen.com/en/blog/how-to-use-minikube-under-windows7.htmlSun, 23 Sep 2018 00:00:00 +0000Sun, 23 Sep 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-use-minikube-under-windows7.html1. Basic Concepts 1.1 Kubernetes Kubernetes (K8s for short), the successor to Google’s Borg, is an open-source system for automatically deploying, scaling, and managing containerized applications. The features it provides: Automated deployment of containers Automated scaling up and down Automated application/service upgrades Grouping containers to serve traffic externally, with load balancing support Health checks for services, with automatic restarts 1.微信公众号DockerKubernetesWindowsMiniKubeLearningOperationshttps://www.chenshaowen.com/en/blog/install-python3-and-go-kernel-in-jupyter.htmlSun, 09 Sep 2018 00:00:00 +0000Sun, 09 Sep 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/install-python3-and-go-kernel-in-jupyter.htmlEarlier we mentioned that Jupyter Notebook is an interactive notebook that supports running more than 40 programming languages and is very well suited to teaching. Recently, while learning the Go language, I thought of Jupyter. This article mainly describes how to install the Python3 and Go kernels inside Jupyter. 1.微信公众号JupyterGoPython3LearningR&Dhttps://www.chenshaowen.com/en/blog/basic-of-robot-framework.htmlTue, 24 Jul 2018 00:00:00 +0000Tue, 24 Jul 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/basic-of-robot-framework.htmlThis post mainly introduces some basic concepts of Robot Framework and how to write a test case. 1. Basic Elements 1.1 Keywords Robot Framework keywords are similar to functions. They are divided into system keywords and user-defined keywords. System keywords, imported by loading a Library User keywords, imported by loading微信公众号PythonTestingRobotFrameworkDevOpsLearningAutomated TestingR&DCICDhttps://www.chenshaowen.com/en/blog/how-to-secure-get-request-in-django.htmlTue, 26 Jun 2018 00:00:00 +0000Tue, 26 Jun 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-secure-get-request-in-django.htmlIn Django, request holds all the information for a single request. Backend processing logic frequently needs information from request. For example, the DRF framework may want to be able to obtain request at any time, or pass some parameters globally. There are third-party Django apps with tools that can satisfy微信公众号DjangoPythonR&DTipsLearningDRFThread Safetyhttps://www.chenshaowen.com/en/blog/dictionary-and-list-structure-in-python2.htmlMon, 12 Mar 2018 11:22:55 +0000Mon, 12 Mar 2018 11:22:55 +0000https://www.chenshaowen.com/en/blog/dictionary-and-list-structure-in-python2.html1. Why Dictionary Lookups Are Faster Than List Lookups First, take a look at the following piece of code. 1 2 3 4 5 6 7 8 9 10 11 12 13 from time import time t = time() data = [chr(i) for i in range(97, 123)] # data =微信公众号PythonSource CodeLearningR&Dhttps://www.chenshaowen.com/en/blog/python2-source-of-pyc.htmlFri, 09 Mar 2018 00:00:00 +0000Fri, 09 Mar 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/python2-source-of-pyc.htmlThis article uses the Python 2.7.8 source code as an example. 1. Common File Formats in Python py file A Python source code file, which can be modified with a text editor. pyc file The bytecode file generated after Python source code is compiled. pyw file When a pyc file is executed, a console window appears; when a pyw file is executed, it does not.微信公众号PythonSource CodeCompilationLearningR&Dhttps://www.chenshaowen.com/en/blog/notes-on-making-strategy-of-the-presentation.htmlTue, 06 Mar 2018 20:55:29 +0000Tue, 06 Mar 2018 20:55:29 +0000https://www.chenshaowen.com/en/blog/notes-on-making-strategy-of-the-presentation.html1. Overview Course prerequisite: the logical structure and content of the PowerPoint have already been worked out and written. The course enhances the expressiveness of PowerPoint mainly through the following six parts: Color Text Shapes Charts Slide Master Animation 2. Color 2.1 Understanding Color Hue Chromatic colors, such as purple,微信公众号PPTToolsLearningSharinghttps://www.chenshaowen.com/en/blog/how-to-build-a-pypi-server-using-devpi.htmlSun, 04 Mar 2018 00:00:00 +0000Sun, 04 Mar 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/how-to-build-a-pypi-server-using-devpi.htmlWhen you install Python packages with pip, it looks up the corresponding package on the https://pypi.python.org/simple/ index by default, then downloads and installs it. But in an intranet environment, or when you need to publish some private packages to designated users, you have to build your own PyPI Server. This post mainly records the process of building a PyPI Server index with the devpi tool, along with some commonly used commands.微信公众号PythondevpiToolsOperationsLearningPyPIPrivate IndexPackage ManagementIntranethttps://www.chenshaowen.com/en/blog/development-specification-1-of-django.htmlSat, 27 Jan 2018 00:00:00 +0000Sat, 27 Jan 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/development-specification-1-of-django.htmlThis article mainly lays out a few things to watch out for during Django development. A consistent coding style and sound design principles help a project’s development and maintenance, and are worth developers studying and discussing continuously. 1. Encoding Declaration When the Python interpreter executes code, it needs to be微信公众号DjangoPythonConventionsR&DLearningBest PracticesCode StyleArchitecturehttps://www.chenshaowen.com/en/blog/toolbox.htmlTue, 16 Jan 2018 00:00:00 +0000Tue, 16 Jan 2018 00:00:00 +0000https://www.chenshaowen.com/en/blog/toolbox.html1. Documentation Pandoc Pandoc is a text conversion tool written in Haskell. It is fast and supports a wide range of formats, converting Markdown, LaTeX, and others into HTML, Docs, PDF, and more. All it takes is a simple command: 1 pandoc mypaper.md -o mypaper.docx Graphviz An open-source toolkit developed微信公众号ToolsEfficiencyR&DLearningSharinghttps://www.chenshaowen.com/en/blog/process-and-thread-in-python.htmlSat, 06 Jan 2018 11:22:55 +0000Sat, 06 Jan 2018 11:22:55 +0000https://www.chenshaowen.com/en/blog/process-and-thread-in-python.html1. Processes A process is an instance of a running program, and it is the most basic unit by which the kernel allocates resources. A process has its own independent heap and stack, its own address space, and its own resource handles. Processes are scheduled by the OS, the scheduling微信公众号PythonConcurrencyProcessesThreadsCoroutinesR&DLearninghttps://www.chenshaowen.com/en/blog/interactive-notebook-jupyter.htmlMon, 25 Dec 2017 17:22:45 +0000Mon, 25 Dec 2017 17:22:45 +0000https://www.chenshaowen.com/en/blog/interactive-notebook-jupyter.html1. Introduction Jupyter Notebook (formerly IPython notebook) is an interactive notebook that supports running more than 40 programming languages. Jupyter Notebook is in fact a web application that lets you create and share program documents, with support for live code, mathematical equations, visualizations, and Markdown. Its uses include data cleaning微信公众号ToolsPythonNotesJupyterLearningR&DData AnalysisMachine LearningVisualizationhttps://www.chenshaowen.com/en/blog/error-code-design-and-unified-processing-in-django.htmlMon, 30 Oct 2017 21:50:33 +0000Mon, 30 Oct 2017 21:50:33 +0000https://www.chenshaowen.com/en/blog/error-code-design-and-unified-processing-in-django.htmlI currently use Django for SaaS development, and I develop and maintain several SaaS applications at the same time. Many SaaS applications have agreed-upon error codes, some used to handle login state, others to mark business logic status. For a feature that is so strongly shared across projects, it is微信公众号DjangoPythonError CodesMiddlewareR&DLearningAPIBest Practiceshttps://www.chenshaowen.com/en/blog/how-to-learn-a-new-programming-language-taking-python-as-an-example.htmlFri, 13 Oct 2017 23:00:06 +0000Fri, 13 Oct 2017 23:00:06 +0000https://www.chenshaowen.com/en/blog/how-to-learn-a-new-programming-language-taking-python-as-an-example.html1. Understand the Background of the New Language ABC was a teaching language designed specifically for non-professional programmers, but because it was closed, it did not succeed. During Christmas 1989, Guido van Rossum developed a new script interpreter and named it Python, as a kind of successor to the ABC微信公众号PythonLearningProgrammingR&Dhttps://www.chenshaowen.com/en/blog/programming-editor-atom.htmlFri, 04 Aug 2017 23:46:56 +0000Fri, 04 Aug 2017 23:46:56 +0000https://www.chenshaowen.com/en/blog/programming-editor-atom.html1. Introduction to Atom Atom is a code editor that supports the three major desktop platforms Windows, Mac, and Linux, is completely free, and is open source on GitHub. Go now 1.1 Pros and Cons Main advantages A strong development and maintenance team, an open source project, fast Bug fixes,微信公众号ToolsDevelopmentR&DEditorLearninghttps://www.chenshaowen.com/en/blog/page-uses-a-custom-font.htmlTue, 01 Aug 2017 19:30:47 +0000Tue, 01 Aug 2017 19:30:47 +0000https://www.chenshaowen.com/en/blog/page-uses-a-custom-font.html1. Font-family The fonts you can use on a web page are limited to the handful of fonts already installed on the PC the browser is running on. Windows operating system, Chinese fonts: SimHei (Hei) SimSun (Song) NSimSun (New Song) FangSong (FangSong) KaiTi (Kai) FangSongGB2312 (FangSong GB2312) KaiTiGB2312 (Kai GB2312)微信公众号FrontendCSSR&DFontsLearninghttps://www.chenshaowen.com/en/blog/basis-of-docker.htmlFri, 28 Jul 2017 15:30:00 +0000Fri, 28 Jul 2017 15:30:00 +0000https://www.chenshaowen.com/en/blog/basis-of-docker.html1. What Docker Is Docker was born in early 2013 as a side project inside dotCloud (the former name of Docker Inc.). Its source code is hosted on GitHub, it is written in Go, and it follows the Apache 2.0 open-source license. The goal of the Docker project is to provide a lightweight operating system virtualization solution.微信公众号PaaSDevOpsDockerContainersLearninghttps://www.chenshaowen.com/en/blog/automated-testing-tool-about-selenium.htmlSat, 22 Jul 2017 15:19:42 +0000Sat, 22 Jul 2017 15:19:42 +0000https://www.chenshaowen.com/en/blog/automated-testing-tool-about-selenium.html1. Introduction to Selenium Selenium is an integration testing tool from Thoughtworks. At its core, Selenium Core is based on JSUnit, written entirely in JavaScript, and can run in any browser that supports JavaScript. 1.1 Main Features and Characteristics Open source and free. Multi-browser support: Firefox, Chrome, IE, Opera — usable for compatibility testing.微信公众号ToolsTestingAutomationR&DSeleniumLearninghttps://www.chenshaowen.com/en/blog/signal-of-django.htmlThu, 13 Jul 2017 13:06:10 +0000Thu, 13 Jul 2017 13:06:10 +0000https://www.chenshaowen.com/en/blog/signal-of-django.html1. Basic Concepts Django ships with a signal dispatcher built in. Signals help decouple program modules. When an event occurs elsewhere in the application, a designated function is notified. Signals let certain senders notify a group of receivers that some action has taken place. 2. Using Signals 2.1 Declaring a微信公众号DjangoBackendPythonR&DLearningDecouplingEventshttps://www.chenshaowen.com/en/blog/book/the-101-of-ecma-script.htmlTue, 27 Jun 2017 20:05:01 +0000Tue, 27 Jun 2017 20:05:01 +0000https://www.chenshaowen.com/en/blog/book/the-101-of-ecma-script.htmlAuthor: Ruan Yifeng Publisher: Publishing House of Electronics Industry Publication year: 2014-8-1 ISBN: 9787121238369 E-book微信公众号BooksJavaScriptFrontendLearninghttps://www.chenshaowen.com/en/blog/snippets-of-django.htmlThu, 22 Jun 2017 23:52:06 +0000Thu, 22 Jun 2017 23:52:06 +0000https://www.chenshaowen.com/en/blog/snippets-of-django.html1. Auto-Register All Models in Admin admin.py 1 2 3 4 5 6 7 8 9 10 # -*- coding: utf-8 -*- import inspect from django.contrib import admin from . import models for name, obj in inspect.getmembers(models): try: if inspect.isclass(obj): admin.site.register(getattr(models, name)) except Exception as e: pass 2. Get All View Names Get all view names of the project微信公众号DjangoPythonR&DLearningTipshttps://www.chenshaowen.com/en/blog/book/dive-into-python.htmlFri, 24 Feb 2017 13:11:11 +0000Fri, 24 Feb 2017 13:11:11 +0000https://www.chenshaowen.com/en/blog/book/dive-into-python.htmlAuthor: Mark Pilgrim Publisher: Apress Published: 2004-07-19 ISBN: 9781590593561微信公众号BooksPythonLearningProgrammingReadinghttps://www.chenshaowen.com/en/blog/drawing-2d-fractal-graph-using-python.htmlWed, 25 Jan 2017 23:28:00 +0000Wed, 25 Jan 2017 23:28:00 +0000https://www.chenshaowen.com/en/blog/drawing-2d-fractal-graph-using-python.html1. Koch Curve The Swedish mathematician Helge von Koch proposed the Koch curve in his 1904 paper “On a continuous curve without tangents, constructible from elementary geometry.” It is described as follows: Specify the length of a line segment \(l\) (this can be thought of as iteration 0) Divide this微信公众号PythonMathR&DFractalLearningVisualizationDrawinghttps://www.chenshaowen.com/en/blog/django-permissions-control.htmlSun, 22 Jan 2017 09:23:45 +0000Sun, 22 Jan 2017 09:23:45 +0000https://www.chenshaowen.com/en/blog/django-permissions-control.html1. Django Built-in Permission Management 1.1 Permission Categories Permission Used to define User A’s permission on Task. User If User A has permission on Model B, then User A has the corresponding permission on all instances in Model B. The user_permission field of the User object is used to manage微信公众号DjangoDevelopmentPermissionsPythonR&DLearningSecurityRBACAuthenticationhttps://www.chenshaowen.com/en/blog/study-of-bwapp.htmlSun, 25 Dec 2016 15:38:42 +0000Sun, 25 Dec 2016 15:38:42 +0000https://www.chenshaowen.com/en/blog/study-of-bwapp.html1. Introduction to BWAPP buggy web Application, abbreviated BWAPP, is an open-source web application that packs in all kinds of common and recent vulnerabilities, with the goal of helping security enthusiasts, developers, and students find and prevent web vulnerabilities. It contains more than 100 vulnerabilities, covering every major known web vulnerability, including the OWASP Top 10 security risks, and most importantly it already includes the OpenSSL and ShellShock vulnerabilities.微信公众号SecurityNetworkLearningBWAPPhttps://www.chenshaowen.com/en/blog/python-coding.htmlTue, 08 Nov 2016 20:21:42 +0000Tue, 08 Nov 2016 20:21:42 +0000https://www.chenshaowen.com/en/blog/python-coding.htmlBackground: I had just finished a Django data-query web project. The data came from internal API queries, and every query had to call several APIs to fetch the data and render it on the front-end page. Since the relevant data does not change often, I designed a cache to improve微信公众号PythonEncodingR&DLearninghttps://www.chenshaowen.com/en/blog/front-auto-build-tools-gulp.htmlTue, 28 Jun 2016 21:59:04 +0000Tue, 28 Jun 2016 21:59:04 +0000https://www.chenshaowen.com/en/blog/front-auto-build-tools-gulp.html1. Frontend Automated Build Tools When dealing with frontend scenarios, you write CSS with Less, HTML with Jade, modularize with Browserify, and add MD5 hashes to resources for non-overwriting deployments. If all this work were done purely by hand, the efficiency would be extremely low. A frontend automated build tool微信公众号FrontendgulpToolsR&DBuildLearningAutomation