G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment · HackerPapers discussions

G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment — mentioned by nimitkalra

Fri, 23 May 2025 18:04:48 GMT

One method to get a better estimate is to extract the token log-probabilities of "YES" and "NO" from the final logits of the LLM and take a weighted sum [1] [2]. If the LLM is calibrated for your task, there should be roughly a ~50% chance of sampling YES (1) and ~50% chance of NO (0) — yielding 0.5. But generally you wouldn't use a binary outcome when you can have samples that are 50/50 pass/fail. Better to use a discrete scale of 1..3 or 1..5 and specify exactly what makes a sample a 2/5 vs a 4/5, for example You are correct to question the weaknesses of a panel. This class of methods…

Paper and discussion history. A mention is not an endorsement.

G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment — HN submission

Mon, 02 Dec 2024 07:36:10 GMT

Garvata: Observability and Debugging for AI Agent Stack

We are stepping into the era of Multi-Agentic Applications, where these apps rely on the interaction between agents, LLMs, tools, vector databases, and other agents. Developers are spending extensive time debugging inaccurate agent outputs and refining their prompts. In the next five years, SMBs might have hundreds of agents managing workflows, while enterprises could be handling thousands. Our mission is to exponentially enhance developer capabilities and improve agent accuracy. Debugging Multi-Agentic Apps is Challenging: 1. Primary issue: Inaccurate outputs — irrelevant, incorrect, or…

3 points; 1 discussion reply on HN. Paper and source history. A submission is not an endorsement.

G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment — mentioned by visarga

Fri, 30 Jun 2023 19:44:17 GMT

> A lot of the AI doomerism comes from folks who do not understand the real complexities in making systems which really can function in a fully autonomous way in an environment which is hostile and dynamic. I agree. Autonomy is hard. But a weaker version of it is possible - self replication in software. An AI model can generate text that can train new models from scratch[1]. They can also generate the model code, explain it, make meaningful changes[2], monitor the training run and evaluate the "child" models[3]. So AI can "pull everything from inside" to make a child, no external materials…

Paper and discussion history. A mention is not an endorsement.