Expert Talks Chennai

AI Evaluation Strategy & Rethinking Enterprise Authorisation Beyond JWTs

Sat, Aug 29 · 10:30 AM IST

IndiQube Alpine, IndiQube Alpine, next to Ekkatutthangal metro station Labour Colony, SIDCO Industrial Estate, Ekkatuthangal, Guindy, Chennai, Ta, Chennai, IN

Hi Everyone,
We’re back with the fifth edition of Expert Talks for the year and this one's extra special as it's the first from our new office space!

Please join us on 29th Aug 2026, from 10:30 AM to 1:00 PM @ Equal Experts, IndiQube Alpine, Ekkatuthangal, Chennai.

Agenda :
1. Welcome & Intros (10 mins)
2. Talk 1: Does Your AI Actually Work? Making Sense of LLM Quality
3. Tea Break (15 mins)
4. Talk 2: Rethinking Enterprise Authorisation Beyond JWTs: A Live Demo

Talk 1: Does Your AI Actually Work? Making Sense of LLM Quality
by Divya Murali & Bala Murugan

AI is moving fast, from experiments to systems that make real decisions, every single day, at scale. With that speed comes a simple question we can't skip: how do we actually know what we've built works?

Traditional software gave us a clear promise: the same input, the same output, every time. AI doesn't work that way. It's probabilistic by design, and "correct" isn't always a single answer. A good summary, a good tag, or a good response can look slightly different each time and still be right. That means the old way of testing, checking output against one expected answer, doesn't hold up anymore. We need real metrics: is it accurate, is it consistent, does it stay true to its source.

In this talk, we'll take a step back and ask a simple but important question: how do we actually know an AI system is doing its job well? We'll look at what quality means for an LLM, the metrics that help us measure it, and the tools that make this practical. We will see, step by step, how to move from "this looks good to me" to something you can genuinely measure. Because the goal isn't just to ship AI. It's to ship AI that works, reliably, every time.

Outline
1. Problem
Why testing AI breaks the old rules: there's no single "right answer" to check against. What quietly goes wrong when AI isn't evaluated properly, and what "good" actually means for a system like this.

2. AI Evaluation Metrics, Made Simple
The handful of things that really matter: is it accurate, is it consistent, does it stay true to its source, is it complete. Why simple text-matching falls short, and why we don't need every metric, just the right ones for the task at hand.

3. Pre-requisites
Choosing the Right Metrics for the Use Case
A simple way to decide what to measure, instead of guessing — because the right metric depends entirely on the task. A summary needs to be checked for faithfulness and completeness; a classification needs to be checked for accuracy; a tagging system needs precision and recall. In plain terms: know what "good" looks like for your specific use case before you try to measure it.

Choosing the Right Tool
A simple way to decide what kind of help you actually need, instead of guessing. What different tools are generally good at, in plain terms — no single tool does everything, and that's fine.

The Ground Truth
Creating a small set of "known good" examples to test against. How to check quality without manually reviewing everything by hand, and how to handle the tricky, borderline cases.

4. Live Demo
A small, realistic change is made to an AI feature that quietly breaks something. Watch it get caught automatically, with a clear read on what went wrong — pass/fail results in real time, the same way you'd see a failed build check.

5. Making This Part of the Workflow
Running these checks automatically every time something changes, and tracking quality over time — so slow decline gets noticed, not just sudden breaks.

6. Q&A & Next Steps
A simple starting point for teams beginning from scratch, and an open floor for questions.

Talk 2: Rethinking Enterprise Authorisation Beyond JWTs: A Live Demo
by Mowrish Dev

Somewhere in your system right now, there is probably a JSON Web Token (JWT) granting access its owner should no longer have. We sign permissions into JWTs and trust them everywhere, but the moment access changes in the database, the token and the truth disagree.

Revoke a user's access to a document and their token keeps working until it expires or grant them access and it never arrives until they log in again. Google's Zanzibar paper named this the "new enemy problem", and OWASP ranks broken access control the #1 web application risk. It is why Google, Airbnb, and Carta all built centralized authorization systems instead of trusting claims frozen inside tokens.

In this talk, we will see why JWTs answer “who are you?” well but cannot answer “what may you do right now?”, then answer one question live: can this user access this document? Two ways: with JWT scopes, where revokes and grants silently fail, and with OpenFGA, where they take effect on the very next request. You will leave knowing where JWTs belong, where they don't, and what to reach for instead.

Outline
One Token to Rule Them All

• What is JWT
• How the industry drifted into using JWTs for authorization

Three Ways Your JWT Lies to You

• Revocation is impossible: access stays live until expiry
• Live permission changes never propagate: the "new enemy problem" from Google's Zanzibar paper.
• Scopes are too coarse: Token bloating with overpermissioning

Is This a Real Enterprise Problem?

• Why Google (Zanzibar), Airbnb (Himeji) and Carta all built centralized authorization services.

Live Demo: Can This User Access This Document?

• A real running system answering one authorization question two ways and showcase the limitations of JWT and how FGA solves it for authorisation
• a document API protected by JWT scopes
• same API backed by OpenFGA
• The Modern Toolbox, and the Honest Trade-offs (7 mins)
• Relationship-based access control (SpiceDB, OpenFGA, Ory Keto, Permify) vs policy engines (OPA, Cedar, Cerbos): how to choose.
• The costs nobody should hide: a network hop on every check, a new critical dependency, and migration effort.
• Where JWTs still shine: identity transport, SSO, short-lived single-purpose tokens.

Q&A and Next Steps

• A starting point for teams: how to pilot centralized authorization on one service without a big-bang migration, and an open floor for questions

About the Speakers

Divya Murali is a Software Engineer with 13+ years of experience in event-driven systems, microservices, and cloud-native applications. She has led large-scale platform migrations, delivered performance optimizations, and contributed to open source. Passionate about AI-driven automation, She focuses on building resilient, scalable systems that solve real-world business challenges.

Bala Murugan is a Software Engineer with 9+ years of experience having built scalable and resilient applications across mobile, web, and backend platforms. She has worked across diverse domains and tech stacks, delivering solutions across various scales and complexities. Driven by a deep curiosity for how things work, She enjoys diving beneath the surface to understand systems at a fundamental level. Passionate about AI, She is now expanding her horizons into LLMs, retrieval systems, and the science behind how models learn and evolve.

Mowrish Dev is a software engineer with 7+ years of experience designing, developing and optimising scalable enterprise applications. He has proven expertise in building cloud native solutions, event driven systems and system modernisation. He is passionate about taking up new engineering challenges and solving them and excited to share the insights along the way.