Because pricing and limits change over time, this article is only a reference for the time it was written.
1. Terminology
- RPM (requests per minute)
Requests per minute
- RPD (requests per day)
Requests per day
- TPM (tokens per minute)
Tokens per minute
- TPD (tokens per day),
Tokens per day
At https://platform.openai.com/tokenizer you can look up the token count for a given text. At https://github.com/openai/tiktoken/blob/main/tiktoken/model.py you can see that text-embedding-ada-002, gpt-3.5, and gpt-4 all use the cl100k_base vocabulary. If you are only doing vectorization, the cheaper text-embedding-ada-002 is a good choice.
2. OpenAI API
OpenAI adjusts quotas based on usage; after more than $5 paid, the quota increases noticeably:
< $5 paid
| Models | Input | Output | RPM | RPD | TPM |
|---|---|---|---|---|---|
| gpt-3.5-turbo-1106 | $0.0010 | $0.0020 | 3 | 200 | 40K |
| gpt-3.5-turbo-instruct | $0.0015 | $0.0020 | 3 | 200 | 40K |
> $5 paid, < $50 paid
| Models | Input | Output | RPM | RPD | TPM |
|---|---|---|---|---|---|
| gpt-3.5-turbo-1106 | $0.0010 | $0.0020 | 3500 | 10000 | 60K |
| gpt-3.5-turbo-instruct | $0.0015 | $0.0020 | 3500 | 10000 | 60K |
| gpt-4 | $0.03 | $0.06 | 500 | 10000 | 10K |
| gpt-4-32k | $0.06 | $0.12 | 500 | 10000 | 10K |
Model pricing https://openai.com/pricing
Model rate limits https://platform.openai.com/docs/guides/rate-limits/usage-tiers
3. Azure OpenAI API
Prices and limits differ by region; East US 2 is used as the example here:
| Models | Context | Prompt (Per 1k tokens) | Completion (Per 1k tokens) | TPM |
|---|---|---|---|---|
| GPT-3.5-Turbo | 4K | $0.0015 | $0.002 | 300K |
| GPT-3.5-Turbo | 16K | $0.003 | $0.004 | 300K |
| GPT-4 | 8K | $0.03 | $0.06 | 40K |
| GPT-4 | 32K | $0.06 | $0.12 | 80K |
Model pricing https://azure.microsoft.com/en-us/pricing/details/cognitive-services/openai-service/
Model rate limits https://learn.microsoft.com/en-us/azure/ai-services/openai/quotas-limits
4. Problems You May Hit When Calling the Azure OpenAI API
4.1 404 Resource Not Found
If the Api Version set in the request is wrong, you get a 404 Resource Not Found. Reference:
https://learn.microsoft.com/en-us/azure/ai-services/openai/reference
Only the following versions are currently available:
- 2022-12-01
- 2023-03-15-preview
- 2023-05-15
- 2023-06-01-preview
- 2023-07-01-preview
- 2023-08-01-preview
- 2023-09-01-preview
4.2 Unsupported data type
The Azure OpenAI Api endpoint should have this format:
https://{your-resource-name}.openai.azure.com/openai/deployments/{deployment-id}/completions?api-version={api-version}
If you are using:
https://{your-resource-name}.openai.azure.com/openai/deployments/{deployment-id}/chat/completions?api-version={api-version}
Note that the second format has an extra chat in the URL; that format is not supported and returns Unsupported data type.
5. How to Optimize When You Are Being Rate-Limited
5.1 Application Side
- Add retry logic to the application
- Avoid overly concentrated requests; spread them across multiple time windows
5.2 Deployment Side
- Deploy several more models in Azure
- Register several more OpenAI accounts
- Use a proxy pool to spread requests across a set of API Keys
6. Summary
On account acquisition: in China there are plenty of people selling OpenAI accounts and offering registration on your behalf; there is a free quota, and anything beyond it requires spending with a foreign credit card. Azure OpenAI API, by contrast, is post-paid based on consumption with a foreign credit card, and enabling GPT-4 requires a separate application.
On price: OpenAI API is a little cheaper than Azure OpenAI API. Bulk-purchased OpenAI accounts cost under 2 yuan and come with a $5 usage quota valid within three months, whereas Azure bills by usage. Counted that way, the gap is dozens of times over.
On request limits: the free OpenAI API is far more restricted than Azure OpenAI API, so you must build an API proxy pool to take advantage of OpenAI API. The hard part is building a stable proxy pool, and OpenAI also bans a single IP from using too many SKs. Azure OpenAI API’s quota limits are looser, and you can raise the quota further by deploying multiple instances of the same model.
On networking: OpenAI requires an international network egress, which means relying on an overseas proxy and carries a certain legal risk. Azure OpenAI, by contrast, is directly reachable in China and needs no proxy.
Overall, for production it is advisable to use Azure OpenAI API, while for development and testing you can use OpenAI API. If a scenario has low real-time requirements, allows repeated attempts, and is extremely cost-sensitive, you can consider using OpenAI API with a proxy pool.
