Are we operating on a similar question with two different approaches?

https://github.com/space-bacon/Semiotic-Analysis-Tool

I’m going to put some time into your question today and hopefully return with a more useful response.

I wanted to shamelessly get my early stage work on semiotic analysis in front of your eyeballs in the meantime as I see this as one of the more valuable pieces of content I’ve consumed today that could help improve my scripts direction as well.

When I see people evaluating LLMs in this way I can't help but think they are letting emotion get the better of them:

> I am using the incorrect phrasing, but Ive been heavily using claude's "Project Folders" (paid account) - and when I put "context" files into the project folders, it will "forget" that the files are there - and Ill call it out when it switches back to boilerplate response language -- and it apologizes and says "You are correct I SHOULD HAVE BEEN using the project files for context.

You should probably try to implement your own RAG system with free models locally (ollama, langchain, chromadb can do it, it's very straightforward) so you can understand the process a bit more under the hood.

> How/why is it occurring mid conversation?

I dunno, usually when something is retrieved it is added to context. But a key part of RAG is determining how to chunk up your content so that the prompt embedding matches up to information that is actually targeted and concise.

So if RAG is behaving suboptimally the first thing to check is, are the input documents targeted and concise? If too much context is being stuffed into the prompt context, then the results will be poor.

You can see this with even just very large prompts, the larger the context the worse the quality, despite model developers claiming ever larger context windows.

I don't think inventing your own terms (Which therefore have extremely weak embeddings to match with the embeddings of the models training content) is the right way to go.

If I chuck in what is a discernment lattice to gpt4o I get:

> A "discernment lattice" isn't a widely recognized term in most common fields of study, but it can be interpreted in a few ways depending on context. Here's a breakdown of potential meanings and applications:....

So it's not really giving the model valuable tokens to work with.

> context windows in all the GPTs are a lie

The above is a very very bold claim, to say the least.-

What is a discernment lattice and what are you smoking?