A public rubric compares agentic AI vendor docs on cost and safety
A Reddit user says they built a tool that scores agentic AI vendor documentation with a 15-question rubric. The tool takes a folder of public vendor docs and returns a structured report with quoted evidence for each score. The author tested Anthropic Claude Agent SDK, OpenAI Agents SDK, LangGraph, Sierra, and Salesforce Agentforce using only public material. The rubric covers tool-call correctness, loop termination, multi-step state coherence, and visible step or cost ceilings.
Key points
- The author made a 15-question rubric for checking agentic AI vendor documentation.
- The scoring used only public vendor material.
- Anthropic Claude Agent SDK scored 3.27 out of 5, and OpenAI Agents SDK scored 3.20 out of 5.
- Salesforce Agentforce scored lower even though it lists many enterprise compliance credentials.
- The author asks whether the method unfairly rewards vendors with deeper public documentation.
Quick term guide
- agentic AI
- AI that tries to complete a goal by taking several steps, not just answering one question.
- documentation
- Written notes that explain how a task or process is done.
- Claude Agent SDK
- A developer tool for building AI agents that use Claude to work through tasks.
- LangGraph
- An open-source framework that lets you design an AI agent's decision steps as a visual graph, making complex multi-step logic easier to manage.
- enterprise
- A large business or company, which usually buys special software plans for better security and privacy guarantees.
- compliance
- Following required rules, laws, or policies for a specific field.
- credentials
- Secret keys or tokens used to access an account or service.
- credential
- Information that proves permission to use an account, card, or service.