Show HN: Pi Labs – AI scoring and optimization tools for software engineers
9 by achintms | 0 comments on Hacker News.
Hey HN, after years building some of the core AI and NLU systems in Google Search, we decided to leave and build outside. Our goal was to put the advanced ML and DS techniques we’ve been using in the hands of all software engineers, so that everyone can build AI and Search apps at the same level of performance and sophistication as the big labs. This was a hard technical challenge but we were very inspired by the MVC architecture for Web development. The intuition there was that when a data model changes, its view would get auto-updated. We built a similar architecture for AI. On one side is a scoring system, which encapsulates in a set of metrics what’s good about the AI application. On the other side is a set of optimizers that “compile” against this scorer - prompt optimization, data filtering, synthetic data generation, supervised learning, RL, etc. The scoring system can be calibrated using developer, user or rater feedback, and once it’s updated, all the optimizers get recompiled against it. The result is a setup that makes it easy to incrementally improve the quality of your AI in a tight feedback loop: You update your scorers, they auto-update your optimizers, your app gets better, you see that improvement in interpretable scores, and then you repeat, progressing from simpler to more advanced optimizers and from off-the-shelf to calibrated scorers. We would love your feedback on this approach. https://build.withpi.ai has a set of playgrounds to help you quickly build a scorer and multiple optimizers. No sign in required. https://code.withpi.ai has the API reference and Notebook links. Finally, we have a Loom demo [1]. More technical details Scorers: Our scoring system has three key differences from the common LLM-as-a-judge pattern. First, rather than a single label or metric from an LLM judge, our scoring system is represented as a tunable tree of metrics, with 20+ dimensions which get combined into a final (non-linear) weighted score. The tree structure makes scores easily interpretable (just look at the breakdown by dimension), extensible (just add/remove a dimension), and adjustable (just re-tune the weights). Training the scoring system with labeled/preference data adjusts the weights. You can automate this process with user feedback signals, resulting in a tight feedback loop. Second, our scoring system handles natural language dimensions (great for free-form, qualitative questions requiring NLU) alongside quantitative dimensions (like computations over dates or doc length, which can be provided in Python) in the same tree. When calibrating with your labeled or preference data, the scorer learns how to balance these. Third, for natural language scoring, we use specialized smaller encoder models rather than autoregressive models. Encoders are a natural fit for scoring as they are faster and cheaper to run, easier to fine-tune, and more suitable architecturally (bi-directional attention with regression or classification head) than similar sized decoder models. For example, we can score 20+ dimensions in sub-100ms, making it possible to use scoring everywhere from evaluation to agent orchestration to reward modeling. Optimizers: We took the most salient ML techniques and reformulated them as optimizers against our scoring system e.g. for DSPy, the scoring system acts as its validator. For GRPO, the scoring system acts as its reward model. We’re keen to hear the community’s feedback on which techniques to add next. Overall stack: Playgrounds next.js and Vercel. AI: Runpod and GCP for training GPUs, TRL for training algos, ModernBert & Llama as base models. GCP and Azure for 4o and Anthropic calls. We’d love your feedback and perspectives: Our team will be around to answer questions and discuss. If there’s a lot of interest, happy to host a live session! - Achint, co-founder of Pi Labs [1] https://ift.tt/dVe1Avc
Friday, March 14, 2025
Thursday, March 13, 2025
Wednesday, March 12, 2025
New top story on Hacker News: Show HN: Nuanced – Help AI understand code structure, not just text
Show HN: Nuanced – Help AI understand code structure, not just text
12 by aymandfire | 3 comments on Hacker News.
Hi HN! We built Nuanced ( https://ift.tt/j1J5e47 ), an open-source Python library that makes AI coding tools smarter about code structure. The problem: current AI coding assistants see code as just text. They don't understand which functions call which, or how code components depend on each other. This is why they often suggest changes that break dependencies they should have known about. Nuanced solves this by generating call graphs that map these relationships. When you ask "What would break if I change this function?", instead of guessing, the AI can see the actual dependencies. How it works: 1. Run `nuanced init .` to analyze a Python module in your codebase 2. Use `nuanced enrich app/file.py function_name` to get relationship data 3. Include this data in your AI prompts or integrate it into tools We're already working with teams building AI coding assistants and security review tools to integrate this capability. Our initial release supports Python with plans for JavaScript/TypeScript next. I'd love your feedback, especially if you're building dev tools that could benefit from better code structure understanding!
12 by aymandfire | 3 comments on Hacker News.
Hi HN! We built Nuanced ( https://ift.tt/j1J5e47 ), an open-source Python library that makes AI coding tools smarter about code structure. The problem: current AI coding assistants see code as just text. They don't understand which functions call which, or how code components depend on each other. This is why they often suggest changes that break dependencies they should have known about. Nuanced solves this by generating call graphs that map these relationships. When you ask "What would break if I change this function?", instead of guessing, the AI can see the actual dependencies. How it works: 1. Run `nuanced init .` to analyze a Python module in your codebase 2. Use `nuanced enrich app/file.py function_name` to get relationship data 3. Include this data in your AI prompts or integrate it into tools We're already working with teams building AI coding assistants and security review tools to integrate this capability. Our initial release supports Python with plans for JavaScript/TypeScript next. I'd love your feedback, especially if you're building dev tools that could benefit from better code structure understanding!
Tuesday, March 11, 2025
Monday, March 10, 2025
Sunday, March 9, 2025
New top story on Hacker News: Show HN: Evolving Agents Framework
Show HN: Evolving Agents Framework
27 by matiasmolinas | 3 comments on Hacker News.
Hey HN, I've been working on an open-source framework for creating AI agents that evolve, communicate, and collaborate to solve complex tasks. The Evolving Agents Framework allows agents to: Reuse, evolve, or create new agents dynamically based on semantic similarity Communicate and delegate tasks to other specialized agents Continuously improve by learning from past executions Define workflows in YAML, making it easy to orchestrate agent interactions Search for relevant tools and agents using OpenAI embeddings Support multiple AI frameworks (BeeAI, etc.) Current Status & Roadmap This is still a draft and a proof of concept (POC). Right now, I’m focused on validating it in real-world scenarios to refine and improve it. Next week, I'm adding a new feature to make it useful for distributed multi-agent systems. This will allow agents to work across different environments, improving scalability and coordination. Why? Most agent-based AI frameworks today require manual orchestration. This project takes a different approach by allowing agents to decide and adapt based on the task at hand. Instead of always creating new agents, it determines if existing ones can be reused or evolved. Example Use Case: Let’s say you need an invoice analysis agent. Instead of manually configuring one, our framework: Checks if a similar agent exists (e.g., a document analyzer) Decides whether to reuse, evolve, or create a new agent Runs the best agent and returns the extracted information Here's a simple example in Python: import asyncio from evolving_agents.smart_library.smart_library import SmartLibrary from evolving_agents.core.llm_service import LLMService from evolving_agents.core.system_agent import SystemAgent async def main(): library = SmartLibrary("agent_library.json") llm = LLMService(provider="openai", model="gpt-4o") system = SystemAgent(library, llm) result = await system.decide_and_act( request="I need an agent that can analyze invoices and extract the total amount", domain="document_processing", record_type="AGENT" ) print(f"Decision: {result['action']}") # 'reuse', 'evolve', or 'create' print(f"Agent: {result['record']['name']}") if __name__ == "__main__": asyncio.run(main()) Next Steps Validating in real-world use cases and improving agent evolution strategies Adding distributed multi-agent support for better scalability Full integration with BeeAI Agent Communication Protocol (ACP) Better visualization tools for debugging Would love feedback from the HN community! What features would you like to see? Repo: https://ift.tt/WDfXTZw
27 by matiasmolinas | 3 comments on Hacker News.
Hey HN, I've been working on an open-source framework for creating AI agents that evolve, communicate, and collaborate to solve complex tasks. The Evolving Agents Framework allows agents to: Reuse, evolve, or create new agents dynamically based on semantic similarity Communicate and delegate tasks to other specialized agents Continuously improve by learning from past executions Define workflows in YAML, making it easy to orchestrate agent interactions Search for relevant tools and agents using OpenAI embeddings Support multiple AI frameworks (BeeAI, etc.) Current Status & Roadmap This is still a draft and a proof of concept (POC). Right now, I’m focused on validating it in real-world scenarios to refine and improve it. Next week, I'm adding a new feature to make it useful for distributed multi-agent systems. This will allow agents to work across different environments, improving scalability and coordination. Why? Most agent-based AI frameworks today require manual orchestration. This project takes a different approach by allowing agents to decide and adapt based on the task at hand. Instead of always creating new agents, it determines if existing ones can be reused or evolved. Example Use Case: Let’s say you need an invoice analysis agent. Instead of manually configuring one, our framework: Checks if a similar agent exists (e.g., a document analyzer) Decides whether to reuse, evolve, or create a new agent Runs the best agent and returns the extracted information Here's a simple example in Python: import asyncio from evolving_agents.smart_library.smart_library import SmartLibrary from evolving_agents.core.llm_service import LLMService from evolving_agents.core.system_agent import SystemAgent async def main(): library = SmartLibrary("agent_library.json") llm = LLMService(provider="openai", model="gpt-4o") system = SystemAgent(library, llm) result = await system.decide_and_act( request="I need an agent that can analyze invoices and extract the total amount", domain="document_processing", record_type="AGENT" ) print(f"Decision: {result['action']}") # 'reuse', 'evolve', or 'create' print(f"Agent: {result['record']['name']}") if __name__ == "__main__": asyncio.run(main()) Next Steps Validating in real-world use cases and improving agent evolution strategies Adding distributed multi-agent support for better scalability Full integration with BeeAI Agent Communication Protocol (ACP) Better visualization tools for debugging Would love feedback from the HN community! What features would you like to see? Repo: https://ift.tt/WDfXTZw