Google’s Longer Context Window Ushers in New Era of Chatbot Capabilities
Longer context windows will increase AI's capacity to handle task like coding. Photo by Jaap Arriens/NurPhoto via Getty Images
Key Takeaways
- Google’s latest Gemini 1.5 foundational model can handle up to a million tokens.
- The longer context window marks a significant leap forward compared to previous solutions.
- With a higher token capacity, chatbots can handle much longer prompts.
When Google announced the next generation of its Gemini foundational model last week, it boasted that Gemini 1.5 represented “a breakthrough in long-context understanding.”
Able to process inputs of up to a million tokens, the new model, which has been released for early testing, is miles ahead of its predecessor, which has a 32,000-token capacity. Gemini 1.5 even beats OpenAI’s most advanced GPT model, which can run inputs of up to 128,000 tokens. But what do longer context windows for chatbot users?
Next-Gen AI Models Can Handle Longer, More Detailed Prompts
In the field of Natural Language Programming, tokenization breaks down phrases and sentences into smaller fragments.
As a general rule of thumb, 750 English words equates to around 1000 tokens. Complex words are typically divided into 2 or more smaller chunks and punctuation marks are each represented by individual tokens.
With longer context windows running to tens or hundreds of thousands of tokens, chatbots can handle much more detailed prompts that include references to lengthy texts.
For example, with a 128,000 token limit, ChatGPT could digest and summarize a short- to mid-length novel. With a million tokens, Gemini 1.5 could do the same for a whole series.
Internally, Google even said it has tested Gemini with up to 10 million tokens – more than enough for it to handle the complete works of Shakespeare.
“The raw data that 1.5 Pro can handle opens up whole new ways to interact with the model,” the firm observed.
“Instead of summarizing a document dozens of pages long, for example, it can summarize documents thousands of pages long. Where the old model could help analyze thousands of lines of code, thanks to its breakthrough long context window, 1.5 Pro can analyze tens of thousands of lines of code at once.”
Longer Context Windows Promise to Boost Chatbot Functionality
Google’s emphasis on coding is telling. The market for generative AI programming tools is fast emerging as one of the sector’s most lucrative subfields and is expected to grow at an annual rate of over 25% from now until 2030.
Discussing the latest version of Gemini, Google Researcher Machel Reid said they tested the longer context window by feeding the model an entire codebase, which it was then able to write documentation for.
“There was another test where it was able to accurately answer questions about the 1924 film Sherlock Jr. after we gave the model the entire 45-minute movie to ‘watch,’” he added.
Longer context windows are especially impactful when it comes to languages other than English, which tend to tokenize less efficiently.
A Boost for Non-English Users
Among Latin-script languages, communicating information can require up to 100% more tokens than the equivalent English translation. Languages like German are especially inefficient to tokenize as they often combine multi-word phrases into a single complex word.
The problem is even more pronounced when prompting AI with languages that aren’t scripted with a Latin alphabet. For example, an Amharic prompt might be broken down into 10x more tokens than its English translation.
Pointing to the consequences of longer context windows for non-English languages, Reid said Gemini 1.5 was able to learn Kalamang, a rare language spoken by fewer than 200 people around the world using the one grammar manual available in English:
“The model can’t speak it on its own if you just ask it to translate into this language, but with the expanded long context window, you can put the entire grammar manual and some examples of sentences into context, and the model was able to learn to translate from English to Kalamang at a similar level to a person learning from the same content.”
Sora Release Date: OpenAI’s Video Generator Public Timeline, Details
