1. What a Large Model Actually Is

First, let us ask two large models this question and see what they answer.

Claude says that a large model is essentially a probabilistic expression of linguistic knowledge: through statistical learning it models the regularities at every level of language and represents the prior distribution of language generation, thereby gaining the ability to predict and generate language.

ChatGPT says that a large model is essentially a deep neural network that, through a huge number of parameters and data, learns abstract representations to solve complex tasks — but it demands heavy computation and resource investment.

Judging by their answers, a large model is essentially a statistically derived knowledge distribution. So why did earlier statistical approaches not achieve the effect of GPT-3.5? There are many main reasons, most of which emphasize training data volume, compute power, model parameters, Transformer, and so on.

But as a user, I care more about how to optimize my use of large models from the input side, in order to improve the results.

2. Technical Factors That Affect How You Use Large Models

2.1 Prompt learning - provide some examples and expectations

Zero-shot uses no training examples at all, relying entirely on the model’s own knowledge to generate a prompt and then complete the downstream task.

One-shot uses just a very small example prompt, letting the model learn the format of the downstream task and then generate a new prompt to finish it.

Few-shot uses roughly 2 to 10 example prompts, letting the model learn the characteristics of the downstream task and then use them to generate more prompts.

The more prompts you provide, the better the result. The closer the format of your examples is to the downstream task, the better the result.

Input

1
2
3
4
5
6

Generate a set of random floats Let's think step by step

Here are some examples

Generate a set of random numbers -> [1, 5, 6]
Generate a set of random variables -> [a, b, c]

It will generate

1

Generate a set of random floats -> [2.3, 4.8, 1.5]

Whereas the input

1

Generate a set of random floats

will generate Python and JavaScript scripts for generating random numbers.

To get better results, we need to provide some prompt that explains what role the large model should play, a description of the problem, the context in which it arises, and the expected result, so the model knows what we actually want.

2.2 Chain-of-Thought - let the large model think step by step

A traditional large model is like a black box: it just outputs a result and cannot make full use of its knowledge. The chain-of-thought technique makes the large model decompose the problem, solve it piece by piece, and then arrive at the final answer.

To keep the large model from entering a wrong knowledge space, we can even provide some solution ideas and frameworks.

Input

1

A juggler has 16 balls. Half of the balls are golf balls, and half of the golf balls are blue. How many blue golf balls are there?

It will generate

1

According to the description in the problem, there are 16 balls, half of which are golf balls, that is, 8. And half of the golf balls are blue, so the number of blue golf balls is half the number of golf balls, that is, 8.

This answer is wrong; the correct answer is 4.

But if, after the input, you append a hint to each sentence — [please think step by step] — you get the correct answer.

1

A juggler has 16 balls. Half of the balls are golf balls, and half of the golf balls are blue. How many blue golf balls are there? Please think step by step.

It will generate

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

When we think about this problem step by step:

A juggler has 16 balls.
Half of the balls are golf balls, so 8 balls are golf balls.
Half of the golf balls are blue.
Now let us calculate how many blue golf balls there are:

Since half of the golf balls are blue, we can divide the number of golf balls by 2 to get the number of blue golf balls.

Number of blue golf balls = half the number of golf balls = half of 8 balls = 4 blue golf balls.

So the juggler has 4 blue golf balls in hand.

You can see that the accuracy of the result improved markedly. This is because, under its own guidance, the large model entered the correct knowledge space; of course, you can also directly give the large model a thinking framework and have it think along the lines you intend.

2.3 The longer the input text, the worse the result - control input length

If the input text exceeds the large model’s maximum token length, the model will automatically truncate the input and lose contextual information, leading to inaccurate results.

Even without exceeding the model’s maximum token length, an increase in input text length will still degrade the model’s performance.

Research shows that when processing long text, a large model tends to ignore the information in the middle and focus only on the beginning and the end. https://arxiv.org/pdf/2307.03172.pdf

Therefore, do not actually feed in 100K tokens just because the model supports a 100K token input; instead, split it into several smaller chunks, feed them in separately, and then aggregate the results.

3. References

• https://github.com/openai/openai-cookbook/blob/main/techniques_to_improve_reliability.md