Embedding Models
Embedding models map high-dimensional data into a lower-dimensional space, which makes the data easier to process and analyze.
Text Models
Here is a leaderboard, https://huggingface.co/spaces/mteb/leaderboard
The leaderboard above gives each model’s score, parameter count, memory usage, vector dimension, maximum tokens, and other information. Below are some commonly used open-source models:
- gte-Qwen2-7B-instruct
https://huggingface.co/Alibaba-NLP/gte-Qwen2-7B-instruct
Dimension 3584, max tokens 131072, memory 28 GB
- gte-Qwen2-1.5B-instruct
https://huggingface.co/Alibaba-NLP/gte-Qwen2-1.5B-instruct
Dimension 1536, max tokens 131072, memory 6 GB
Image Models
Look for ViT models; Vision Transformer (ViT) is a new architecture based on the Transformer.
- clip-vit-base-patch32
https://huggingface.co/openai/clip-vit-base-patch32
Dimension 512
- clip-vit-large-patch14
https://huggingface.co/openai/clip-vit-large-patch14
Dimension 768
Vector Databases
Vector databases are purpose-built for the storage and retrieval of vectors, and when handling vector data they are faster and more accurate than traditional databases.
postgresql
Since version 13, postgresql has been able to perform vector search through the open-source PGVector extension.
Suited to scenarios with relaxed latency requirements and cost sensitivity.
redis
Since version 2.4, redis has officially supported vector search, and version 7.2 brought a major update and optimization of vector search, including support for HNSW search.
Suited to scenarios with strict latency requirements, frequent queries, and a need for simple, fast deployment.
elasticsearch
elasticsearch added support for vector fields starting with version 7.0, and version 8.0 introduced native ANN and HNSW search.
Suited to large-scale data and to scenarios that mix text, structured data, and vector data.
Chroma
https://github.com/chroma-core/chroma
Chroma is a vector database built with Rust and Python, known for its simplicity and ease of use. It has good support for handling multimedia content and is particularly suited to rapid prototyping and experimentation.
Milvus
https://github.com/milvus-io/milvus
Milvus is a distributed vector database developed in Go and C++, offering powerful general-purpose data processing capabilities and performing well in scenarios such as recommendation systems and language and vision analysis.
Faiss
https://github.com/facebookresearch/faiss
Faiss is a C++ vector retrieval library open-sourced by Facebook; its standout feature is GPU acceleration, which lets it handle large-scale vector search tasks efficiently.
Weaviate
https://github.com/weaviate/weaviate
Weaviate is a vector search engine implemented in Go, providing a GraphQL interface and being particularly good at knowledge graph integration and complex query handling.
Qdrant
https://github.com/qdrant/qdrant
Qdrant is a vector similarity search engine developed in Rust, supporting complex metadata-filtered queries and performing excellently in recommendation systems, semantic search, and multimodal search.
Agent Frameworks
Agent frameworks enable the ability to plan, analyze, and automatically execute tasks.
AutoGPT
https://github.com/Significant-Gravitas/AutoGPT
AutoGPT is a powerful autonomous AI agent system that can independently accomplish complex goals, supporting features such as internet search, code generation, and task execution. It uses GPT-4 as its decision engine and comes with a rich set of built-in tool integrations.
AutoGen
https://github.com/microsoft/autogen
AutoGen is a multi-agent conversation framework developed by Microsoft, supporting collaborative dialogue among multiple AI agents. It provides a rich set of preset agent templates and can accomplish complex tasks such as code generation and mathematical reasoning.
Langfuse
https://github.com/langfuse/langfuse
Langfuse is a platform focused on LLM application development and monitoring, offering complete performance analysis, logging, and version management to help developers build and optimize AI applications more effectively.
ChatDev
https://github.com/OpenBMB/ChatDev
ChatDev is an agent framework for software development that automates the path from requirements to code by simulating a complete development team. It supports multiple programming languages and provides full-lifecycle project management.
BabyAGI
https://github.com/yoheinakajima/babyagi
BabyAGI is a lightweight task planning and execution framework that uses a simple task decomposition model and is suited to learning and experimentation. It has basic memory storage and a clear, easy-to-understand code structure.
SuperAGI
https://github.com/TransformerOptimus/SuperAGI
SuperAGI is an enterprise-grade AI agent development framework that provides a graphical interface and complete agent management. It supports many tool integrations and offers detailed execution logs and analysis.
MetaGPT
https://github.com/geekan/MetaGPT
MetaGPT is an agent framework focused on software engineering, supporting the full lifecycle from requirements analysis to system design. It provides a variety of predefined role templates and supports automated code generation and documentation writing.
Workflow Orchestration
LangChain
https://github.com/langchain-ai/langchain
LangChain links together the steps that may be involved in the application development process through a series of Chains, and it also provides features such as prompts and memory, letting developers build AI applications faster.
FastGPT
https://github.com/labring/FastGPT
FastGPT is a knowledge base question-answering system based on the LLM large language model, offering out-of-the-box capabilities such as data processing and model invocation. It can also orchestrate workflows visually through Flow, making complex question-answering scenarios possible!
Dify
https://github.com/langgenius/dify
Dify provides a Workflow feature that lets users create and manage task flows in a visual way.
Model Deployment
Ollama
https://github.com/ollama/ollama
Ollama provides a simple, easy-to-use interface that lets users run and manage large models on local devices with ease.
vLLM
https://github.com/vllm-project/vllm
vLLM is a project for deploying large-model inference services, using the PageddAttention mechanism to improve memory utilization. It is an excellent tool for deploying models in production. vLLM can adapt to various kinds of hardware, including CPU, GPU, and TPU.
Triton Server
https://github.com/triton-inference-server/server
Triton Server is NVIDIA’s open-source inference serving framework, supporting multiple model formats including ONNX, TensorRT, PyTorch, and TensorFlow. If you are targeting NVIDIA GPU hardware only, Triton Server is a very good choice.
Model Fine-Tuning
Llama-factory
https://github.com/hiyouga/LLaMA-Factory
Llama-factory is a unified LLM fine-tuning framework that supports training a variety of mainstream models (such as Llama, Mistral, and Qwen). It provides a rich set of training methods (including LoRA, QLoRA, and full-parameter fine-tuning) and a wide range of training task types, and it is simple to use while being comprehensive.
unsloth
https://github.com/unslothai/unsloth
unsloth is an efficient LLM fine-tuning tool focused on delivering faster training speed and lower VRAM usage. By optimizing the underlying implementation and using innovative training methods, it can significantly improve the efficiency of fine-tuning techniques such as LoRA, making it especially suitable for model training under limited compute.
Model Training
Pytorch
https://github.com/pytorch/pytorch
Pytorch is a deep learning framework open-sourced by Facebook, very well suited to training large models. It provides users with a variety of easy-to-use machine learning libraries and has good support for various kinds of hardware and topologies.
Megatron-LM
https://github.com/NVIDIA/Megatron-LM
Megatron-LM is built on PyTorch but optimized for large-model training, supporting multi-GPU and multi-node training and suited to large-scale model training.
DeepSpeed
https://github.com/microsoft/DeepSpeed
DeepSpeed is a deep learning optimization library developed by Microsoft; its ZeRO optimization technique can partition model parameters, gradients, and optimizer states across multiple GPUs or nodes, thereby reducing memory usage and improving training efficiency.
