1. Problems with Using LLMs Directly
- Unstable output
One characteristic of generative AI is the diversity of its output. Ask an LLM the same question several times and you may get different answers.
This uncertainty in output is a pleasant surprise for users in conversation and creative scenarios. But in scenarios that demand a high degree of determinism, it keeps LLMs from reaching the adoption stage.
- Insufficient data freshness
Training an LLM is an expensive and time-consuming process. The training dataset cannot be updated in time, and it is very common to train an LLM on data that is a year or two out of date.
This lack of data timeliness greatly diminishes the value of an LLM’s output and limits the scope in which it can be applied.
- Aimed only at humans, detached from the physical world
If we compare an LLM to a human brain, then the way Chat opens it up is by fitting the human with ears and a mouth. The LLM gives a corresponding answer based on what it hears. Whether single-modal or multi-modal, an LLM only responds with output to input; it takes no other form of action.
This usage pattern, custom-built for humans, leaves an LLM unable to perceive the physical world a human inhabits, and unable to truly help a human complete all those repetitive, dull, tedious, and dangerous tasks in real life.
2. Agent = Sensor + Action
As shown above, let us analyze the characteristics an LLM application has from the perspective of an interaction flow. Below, we will describe the design and implementation of an LLM application around this diagram.
When Agent comes up, we easily think of the LLM Agent, but the LLM Agent overemphasizes the Auto capability, that is, running continuously and handling things automatically. That form is excellent. Take open-interpreter: once running locally, it can automatically make a plan from the input, then execute it, automatically handle exceptions, and keep going until the task is done and it waits for the next interaction. open-interpreter is more like an application, with a complete design and implementation, not merely an Agent.
Analyzing it functionally, I believe the Agent an LLM needs consists of two parts: Sensor and Action.
Sensor is responsible for perceiving the environment, such as the current device, the current date, the current interaction context, the latest news, and so on — information about the current application state and environment state.
Action is responsible for executing actions, such as opening an app on a phone, shutting down a device, sending an email, or moving a robot’s arm. These are the response actions the application takes based on the LLM’s output.
A well-designed Agent can solve the problems of an LLM application perceiving the world and connecting to surrounding systems.
3. Memory
The core of multi-turn conversation is Memory. Memory is a storage space that simulates the human memory system, used to store the context information of a conversation.
Memory is divided into long-term memory and short-term memory.
- Long-term memory
Long-term memory is similar to a knowledge base, providing a storage space for long-term knowledge fragments. For example, some factual knowledge, some commonly used answers, and so on.
- Short-term memory
Short-term memory is used to store the context information of the current conversation, such as the history of the conversation, the conversational context, and so on.
In my practice, short-term memory is generally just kept in memory, but if the application restarts, the short-term memory disappears; moreover, if the volume of conversation is large, the memory footprint of short-term memory is also large. Using an external database for persistent storage is a better choice.
The implementation details of Memory go far beyond this. How Memory is managed based on the conversation, how it is retrieved, how it is cleaned up, how it is updated, how it is stored, how long it is stored — all of these are issues to consider. Memory can be designed and implemented as a separate module.
4. RAG Applications
I believe many people have heard of, or even used, Chat applications based on a knowledge base. RAG applications are really an umbrella term for this kind of application. RAG stands for Retrieval Augmented Generation.
The core idea of RAG is to find one or more relevant knowledge fragments through retrieval, then feed these knowledge fragments as input to the LLM to generate the final output.
There are several key points here; let us think them through together:
- Why RAG works
RAG can solve the problems of LLM output stability and data freshness. The input of a RAG application is the relevant content retrieved from the knowledge base, which prevents the LLM from free-associating without a goal; the knowledge base of a RAG application is updated dynamically, which guarantees data freshness, and by configuring a threshold, only the most relevant knowledge fragments can be selected, ensuring the accuracy of the output.
- The retrieval method
Usually vector retrieval is used: during preprocessing, each knowledge fragment in the knowledge base is converted into a vector and stored; during retrieval, the input is converted into a vector, and the similarity (that is, the cosine value) between the input vector and each knowledge fragment vector in the knowledge base is computed, and the several knowledge fragments with the highest similarity are selected.
Milvus, PG Vector, Redis, or even llama_index with local files can all implement vector storage and retrieval. But how to choose an Embedding model, and whether multiple Embedding models are needed, may be a question worth thinking about. The role of an Embedding model is to complete the conversion of text to vectors; the commonly used one is OpenAI’s text-embedding-ada-002.
- The size of knowledge fragments
Microsoft has a research paper saying that the best knowledge fragment size is 512 tokens, followed by 256 tokens. Too small a knowledge fragment leads to incomplete knowledge fragments being retrieved; too large a knowledge fragment leads to irrelevant knowledge fragments being retrieved. The exact number of characters that is best can be adjusted according to your actual situation, combined with the conclusion above.
- How to split knowledge fragments
In the actual chunking process, the best approach is to split semantically, but this is difficult to implement in code and requires an algorithmic model. The usual approach is to split by delimiters, such as periods, newlines, and so on. An article or a book is split by paragraph into 512-token knowledge fragments, which are vectorized and stored in the database.
But this approach has flaws too. For example, the pronouns he, she, it in the first sentence of a chunk will cause the split knowledge fragment to be incomplete. In this case, it can be solved by redundancy: concatenate the last sentence of the previous knowledge fragment onto the first sentence of the next knowledge fragment to ensure the completeness of the knowledge fragment. Another idea is to make use of a knowledge graph.
- What flaws do RAG applications have
Maintaining the validity of the knowledge base has a certain cost, retrieving from the knowledge base takes time, and feeding the knowledge fragments into the LLM increases the LLM’s computation and lengthens the response time.
5. Prompt
In many Web GPT Chat applications, a large number of conversational roles are built in, such as a Xiaohongshu copywriter, a psychologist, and so on; the playing of these roles is done by setting a Prompt.
The importance of the Prompt to using an LLM is self-evident. Without proper guidance, the LLM’s output easily deviates from our true intent.
A Prompt framework is a method of writing prompts. Through the definition of some key elements, it not only helps the LLM better understand the input, but also lets users write high-quality prompts faster.
There are many kinds of Prompt frameworks, such as ICIO, CRISPE, BROKE, CREATE, TAG, RTF, ROSES, APE, RACE, TRACE, LangGPT, and so on. These frameworks emphasize different elements. Take LangGPT as an example: it emphasizes elements such as Role, Skill, Rules, Workflow, and Initialization, and is written in Markdown format. For example:
| |
Being proficient in one or two Prompt frameworks is very helpful for developing LLM applications. Nowadays, when I write a Prompt, I basically always use the LangGPT framework; the structure is clear and easy to understand and maintain.
6. Model Fine-Tuning
- lora
lora freezes the weights of the pretrained model and adds an extra network layer on top of the original model. Training only this lora layer is enough to achieve generalization of the model’s capabilities.
In the Stable Diffusion ecosystem, a complete toolchain for lora model fine-tuning has already formed. By loading a lora plugin shared by others, you can directly generate images in a specified style.
If this kind of ecosystem could also form in the NLP field, the efficiency of sharing domain knowledge would be greatly improved. When developing LLM applications, we would no longer need to clean data and enter data, but could instead directly use LLM + lora plugins to quickly connect to a specified application scenario.
- fine-tuning
fine-tuning means directly fine-tuning the model parameters on top of a pretrained model. This approach requires a large amount of data and computing resources to accomplish, and the result may not necessarily be good.
Some examples I have seen online are that more than 1k high-quality data entries are needed to complete a good fine-tuning task, and training is prone to overfitting, damaging the original model’s generalization ability. I think this flaw is not unavoidable, but rather requires a rich reserve of relevant knowledge and accumulated experience to achieve good results.
Model fine-tuning will be a very challenging thing for application developers. If it can be replaced by other means, it is advisable not to spend too much time on it.
7. Thoughts on the Development of Short-Term Model Applications
The LLMs I use are OpenAI’s GPT-3.5, Anthropic’s claude, and GitHub Copilot’s LLM (recently when I asked Chat it said it uses GPT-4), and they can basically meet daily office needs. In addition, in order to use LLMs for several projects, I need to buy 20-30 OpenAI accounts every month.
There are many cloud LLM API services in China, and they are all comparing themselves with GPT-3.5. So I came to know that using GPT-3.5 directly is the right move, and if you are going to production you can buy Azure’s OpenAI GPT service. For LLM application developers, there is no need to switch frequently among various LLMs, or even to pay attention to the release of new LLMs; you should focus on how application scenarios integrate with LLMs.
From selling OpenAI accounts at the very beginning to selling LLM APIs now, we can clearly feel the rapid development of the LLM ecosystem. But there is often a sense of powerlessness too: we cannot grasp the opportunities in it. There are too many people willing to learn new knowledge, but wanting to monetize it and generate actual returns is not an easy thing. This requires breaking through the original way of thinking. Wherever there is a cognition gap or an information gap, there is an opportunity; do not think something is worthless just because it seems low-end or simple. Simple means the market will be larger and the audience broader.
Thinking from the perspective of functionality, which scenarios are suited to early LLM applications?
- New flowers on an old tree. Existing applications that add an LLM bypass without affecting the original functionality. For example, intelligent customer service, new process creation, new product interactions, and so on.
- Niche scenarios. Mass-demand scenarios are too competitive, and there are already many classic similar products in existence. The real breakthrough point of LLMs is their generality; these niche needs are not fundamentally different for an LLM, and we only need to make some small adjustments to quickly satisfy these long-tail needs. For example, product quotation customer service, scenic-area guides, museum narration, and so on.
- Small tools. B2B efficiency tools that improve work efficiency and streamline work processes. For example, RPA, automated product listing, automated shipping, and so on.
The whole economic environment is very bad at present, and the first thing an LLM application must consider is still the profit problem. If it cannot generate revenue from the start, then it will be hard to have any later on. Whether it is new flowers on an old tree, or niche products and small tools, the profit model should be thought through clearly before developing an LLM application.
