This is an automated archive made by the Lemmit Bot.

The original was posted on /r/singularity by /u/sdmat on 2024-09-25 01:10:09+00:00.

I see a lot of people fuming at the lack of a major version bump but from my testing the new 1.5 Pro is real progress. More than just the speed improvements and cost reduction.

Long context has always been the strength of Gemini 1.5, but it was frustratingly flaky once you got past the first 100-200K tokens.

My main use case for this is analyzing documents and log files. Testing the new model shows a big improvement. Specifically it seems to be able to pull in information for “wide” prompts at 2-3x the context depth that -0827 managed.

E.g. if you give the model a half million token stack of documents and asked for summaries, previously the model tended to forget about most of the documents in the deeper half of the context. Now it’s much more consistent. I A/B tested the new model against -0827 with a few runs to be sure.

To be clear it’s not like long context was useless previously - a specific prompt would pull out information. But it was more like long term memory than the kind of deeply associative and enumerable / recitable top-of-mind functionality we expect from SOTA models in short context.

TL;DR: the new model makes long context much more useful. Still not for the full 2 million tokens, but it’s getting there.