Most of the other tools like RAGAS and DeepEval are:
- Written in Python (vs. JavaScript like promptfoo), whereas our codebase is in Rails
- Get too into the weeds of AI, whereas I just want to write simple tests for my webapp not stare at a bunch of math equations
- Are hosted SaaS solutions (vs. open source like promptfoo) and I don't want to deal with the 6 month enterprise software sales cycle just to add a couple tests
So we basically ended up with promptfoo. That said, it felt really awkward to use promptfoo and I don't really love it. I'd love a super simple way to add some integration tests for an LLM so I can just keep iterating
Yet this is used in Shopify, Microsoft, etc. supposedly. Are there people here who have chosen promptfoo over other alternatives? Why?