[Emerging Tech] Cutting AI Costs Without Sacrificing Quality With Max Schuman
Max Schuman is the President and Co-Founder of Trimio, an AI FinOps platform that helps businesses control and optimize their AI spending. Trimio provides cost visibility, financial guardrails, and intelligent model routing to reduce AI expenses while maintaining quality. Max leads the company’s revenue and go-to-market strategy, drawing on extensive experience in AI sales, strategic partnerships, and technology channel leadership. He brings a strong track record of helping businesses accelerate AI adoption, develop infrastructure strategies, and navigate complex technology decisions.
Here’s a glimpse of what you’ll learn:
[2:35] Max Schuman discusses how Trimio helps businesses gain financial control over AI usage and spending
[6:32] AI governance through observability, financial guardrails, budgets, and usage tracking
[11:15] Choosing different AI models for different business workflows
[14:07] Open-weight AI models and their potential cost advantages
[17:34] Max talks about evaluating AI models based on quality, workload, and associated costs
[21:35] Why AI models may be more equivalent than businesses realize
[25:34] How to build evaluation frameworks to measure AI model efficiency
[32:03] AI FinOps, granular usage reporting, and controlling expensive model usage
In this episode…
As businesses rapidly adopt AI, many struggle to understand what they are spending, whether their tools are delivering sufficient value, and how to prevent runaway costs. With models constantly changing in price, capabilities, and performance, choosing the most expensive or powerful option does not necessarily deliver the best results. How can businesses control AI spending while maintaining quality and maximizing ROI?
Max Schuman, an expert in AI FinOps, governance, and model optimization, recommends giving businesses greater visibility into AI usage and establishing financial guardrails such as budgets and off switches. He explains that companies should evaluate models according to specific workflows, comparing cost, speed, quality, consistency, and token usage rather than automatically choosing the most powerful model. Max also discusses testing alternative models on a portion of workloads and using intelligent routing to direct tasks toward the most economical model capable of delivering the required quality.
In this episode of The Customer Wins, Richard Walker interviews Max Schuman, President and Co-Founder of Trimio, about controlling AI costs while improving performance and ROI. Max discusses AI governance, model evaluation, and intelligent model routing.
Resources Mentioned in this episode
Quotable Moments:
“It’s kind of an interesting way to think about governance if it's really financial governance.”
“How do I know that we’re getting ROI out of this project?”
“You have to have that data, though, or you don’t know what’s happening.”
“Sometimes the cheapest model is by far and above the answer because it’s just giving you what you need.”
“The hardest part about evaluating new models is, “How’s it gonna affect my business, and how do I do it with scale?”’
Action Steps:
Establish financial guardrails for AI usage: Set budgets, spending limits, and off switches to prevent unexpected costs while making it easier to measure whether AI investments are producing meaningful ROI.
Evaluate AI models for specific workflows: Compare models based on cost, speed, quality, consistency, and task requirements because the most powerful or expensive model may not always provide the best business outcome.
Track AI usage across the organization: Create visibility into which employees, projects, and business units are using specific AI models so leaders can identify inefficient usage and make informed decisions.
Test alternative AI models before scaling them: Route a small percentage of workloads to alternative models and compare their performance to identify lower-cost options that maintain the required quality.
Measure AI by value rather than token consumption: Focus on how quickly and economically a model produces the desired output so your organization can maximize the ROI of its AI investments.
Sponsor for this episode...
This is brought to you by Quik!
At Quik!, we provide forms automation and management solutions for companies seeking to maximize their potential productivity.
Using our FormXtract API, you can submit your completed forms and get clean, context-rich data that is 99.9% accurate.
Our vision is to become the leading forms automation company by making paperwork the easiest part of every transaction.
Meanwhile, our mission is to help the top firms in the financial industry raise their bottom line by streamlining the customer experience with automated, convenient solutions.
Go to www.quickforms.com to learn more, or contact us with questions at support@quikforms.com.
Powered by Rise25 Podcast Production Company
Episode Transcript:
Intro: 00:02
Welcome to The Customer Wins podcast, where business leaders discuss their secrets and techniques for helping their customers succeed and in turn grow their business.
Richard Walker: 00:16
Hi, I'm Rich Walker, the host of The Customer Wins, where I talk to business leaders about how they help their customers win and how their focus on customer experience leads to growth. Some of my past guests have included Brian Herr of StartingBlocks, Trunal Bhanse of Clazar, and Jagdeesh Prakasam of Qdeck. Today is a special episode of my series on new and emerging solutions, and today's guest is Max Schuman of Trimio. And today's episode is brought to you by Quik!, the leader in enterprise forms processing. When your business relies upon processing forms, don't waste your team's valuable time manually reviewing the forms.
Instead, get Quik!. Using Quik!, you'll be able to generate completed forms and get back clean, context rich data that reduces manual reviews to only one out of 1000 submissions. Visit quickforms.com to get started. All right. I've been getting to know this guest for a couple of weeks now.
A couple of months later I even really, really like him. So I'm excited to bring him on the show. Max Schuman is the CEO and co-founder of Trimio, the economic engine for enterprise AI. Trimio sits between companies and their AI providers, cutting spend and adding control without slowing anything down. The platform is built on four patent pending inventions.
He runs the company from Phoenix with his co-founder Brad Reynolds, and a small team that punches well above its size. I love that, Max. Max, welcome to The Customer Wins.
Max Schuman: 01:43
Thank you. Yeah. Very excited to be here. What started off as me making a comment on your LinkedIn post has led to many, many conversations. So thrilled to be here.
We love talking about AI. We love learning about AI. We love just seeing where things are going. So very, very excited to talk.
Richard Walker: 02:05
Yeah, man. It's so great how LinkedIn works and makes new friends, and creates opportunities. And so I'll encourage others to reach out to me for that purpose too. Yeah. If anybody, if anybody's hearing my podcast for the first time, I love to talk to business leaders about what they're doing to help their customers win, how they build and deliver a great customer experience and the challenges to growing their own company. So Max, let's understand your business a lot better.
I mean, this is a new concept for most people. So how does your company help people?
Max Schuman: 02:35
Yeah. So the last company I was at was my business partner. So I'm actually restructuring the business. I'm president now. I essentially run all go to market motion sales, anything like that.
Brad is our CEO. So the cool part about it is the last place that we were actually at. Brad was the founder of the company, sold it back in 2005, came back three years ago and created the AI platform. So I really like learning AI. Call it within like really the last two years, you know, I'd always heard of companies like ChatGPT didn't fully understand the concept of it. Now in a very, very different position, but as part of leading the sales team at the last place I was at and the AI stack, we noticed that there was or I noticed there was this, there was this huge problem about AI.
And there's a lot of different levers into, you know, how you get involved with AI. But the biggest thing is businesses are tasked with identifying their business. It can be automating a task. It can be trying to automate an entire workflow and something that several people do to help those people go and scale. You know, as much as you go and read the news that robots are taking over everybody's lives, it's really not what we're seeing.
And we don't think we're as close to that it is really like, you know, the, the market tells us. But anyways, we notice there's this, this polar universe happening right now where people are looking at identifying those processes, users, you know, like, like you rich have become very sophisticated. You're seeing a ton of automation and ROI behind using it all the while. It's very, very, very expensive. And I believe we're in a very experimental phase within AI.
And as part of that, there's a big price tag with it. There's no off switches. Agents are very good at creating and using tons of power and creating large data sets. And how do you make AI economical for a business for not only now, but for scale, but then also how do you create ROI and like a better user experience down to your customer? So our business, it's very, very simple in terms of what we do.
The first thing is observability. We give full enterprise observability. It doesn't matter about the model. You don't migrate over. You can set budgets.
You can set off switches, you can track agent productivity, like a lot of things to kind of help you create those financial guardrails to its efficiency. It's how we can get you more utilization out of what you're doing today. So things like token compression and routing, which is a very hot topic in the space right now, caching, which is memory, a lot of savings within properly caching and then quality. Nobody wants to put something of quality into AI and get something back. That's that's crappy essentially.
So how do we maintain that while we're doing those efficiency that we're getting quality back out of, out of the AI stack as well. So really just helping companies to financially get control of AI and find ways to utilize more of that, that AI stack today.
Richard Walker: 05:51
That is awesome. And I want to break down some of these things you just said for the audience, because I'm not sure they caught the depth of what we're talking about here yet. So the first one I have talked to a lot of companies, I've been on roundtables, I have peers that run companies in tech, etc. and one of the dominant things is governance. How do I know what's happening? How do I know if I give out this tool, this license of ChatGPT or whatever to my team that they're getting value, that they're doing what they should be doing, that we're getting the most work, or we even have the right settings, like we've been burned on just not putting in the right settings before.
So what role do you think is playing to help customers have that with your observability layer and more?
Max Schuman: 06:32
Yeah. And that's a, that's a very, very interesting way to look at it because I think the word governance has two meanings. It's one, it's security governance over making sure that there's no shadow AI or anything like that. But two, it's like, you know, what is happening right now is, is my user at home going and spilling a bunch of intellectual property? How do you go and track that?
Well, the first thing is you need all the data. So what we have is our platform is deployed as a live proxy. So essentially we're seeing all the logs and data, which is huge. But something else is. It's kind of an interesting way to think about governance if it's really financial governance.
How do I know that we're getting ROI out of this project? One of the biggest things that we ask customers right now that we're running into is, for instance, you have a whole year planned. What AI products are you looking at building? And they list a laundry list of different things that they're and they're very progressive. They're great ideas that are very customized to their business.
It impacts the customer. It helps them scale. Like there's, there's so much value in that strategic roadmap for the customer. The second question, there is really a governance question. It's how, how much is it going to be to build all of this and how are you going to control it?
And what we've noticed is, is the control portion of that is typically like, we don't have an answer to it. Because if you go to a lot of these LMS, they believe that the usage portion of it is 100% built down to the user. So if you don't have a way of going and creating an off switch or having, you know, some type of a budget, it can get very, very crazy very, very quickly. The other thing is once you build those, how do you run them? And how do you make sure that those don't go off the rails?
Kind of like some, some history to put into here? I think it was like 2022, 2023. You started turning on the news and seeing every single customer. There's like a security breach, a ransomware. This company had to pay, you know, an absurd amount of money.
And like, that's a very, very scary thing for a business is like getting breached. But if you don't control your AI and it goes off the deep end. It's a financial breach to the business. And that's why that financial governance portion for AI is such a big deal. We talk to a customer.
It's like, you know, you never want to be, you know, the story. But they had an agent that did not understand how to turn itself off or it did not have an answer in it for a $40,000 bill in under two business days. So oh my goodness.
Richard Walker: 09:14
I hope the customer or that company could afford that because that would be devastating to most companies.
Max Schuman: 09:19
Yeah. I mean, think about it like you have this PNL item that you think is going to be $2,000 a month and, you know, goodbye to all of your AI budget over the last year. That sucks. And that's like you, you never unclog yourself to finding out the ROI of AI if you get burned by that. So like, so to your point of governance, it's like you have to govern the user, you have to govern the project, you have to govern the business unit, and you have to be able to have budget allocation to all of those different different units or there's it's just very hard to get to the other side of the ROI.
Richard Walker: 09:53
Yeah. And one of the things I like about what you're doing is you're giving companies the ability to put all of their AI channels through one proxy. Yeah. Therefore, whether I have users on Claude or Gemini or ChatGPT or an agent or whatever, I have this one place to observe all that. Right.
And today, that's really hard. I mean, if you give out two licenses, one to ChatGPT and one to Claude, do you know if you're getting things from it? Are people using it the right way? What if somebody in your org says, oh, guess what? I'm now using perplexity.
I didn't tell you. Right. Right.
Max Schuman: 10:25
You have to have that data, though. Or you don't know what's happening.
Richard Walker: 10:29
Yeah. So I think, I think you're creating the opportunity for companies to say, hey, we have a single place that you go to get AI now, and you filter it out to all the other things that they can be doing. There's another question I have in there. I mean, because you're creating this observability, you're creating reporting and the ability to control a lot better.
Max Schuman: 10:50
Yeah.
Richard Walker: 10:51
How much? This is more of an opinion question. How much do you think these LMS and these Ars are really commodities like the ability to switch between them? Is this just going to be like, I don't care which lm I use, I just want the cheapest.
Max Schuman: 11:05
Yeah, this is a deep question. And I think you asked this because, you know, there's so many layers to this one.
Richard Walker: 11:11
I know you know more about this than I do actually. So that's why I've asked.
Max Schuman: 11:15
So model routing is very, very, very interesting. And what you're seeing and like Rich, you're a good example of this, by the way. And you, you know, the AI game better than a lot of people do. So credit, credit where credit's due. But you think about these models, models are very subjective to workflows.
They react very similarly to humans. For instance, Gemini is awesome at photo generation as well as ChatGPT. Claude sucks at it, and that's just like a fact. And I love Claude, by the way. I'm like, you know, if you take my anthropic toy away, I'm going to go cry in a corner because I, I love it and it's so amazing.
And there's tons of business value for us, you know, by, by us using certain tools. But what we're seeing is that workflows change for businesses. What you originally selected is that one model might end up changing whether it has to think more, whether it has to reason more, whether it's agentic, whether it's all these different things, like it's always changing. And for customers.
Richard Walker: 12:25
They upgrade the model and you can't use it anymore. It's not doing what you wanted, right?
Max Schuman: 12:29
And it's reacting differently. You know, for instance, if you go from Claude Sonnet 4.6 to something like five, I think we've had this conversation before. Five, even though it thinks more, if it's a very predictable workflow and it's, it might think and burn more tokens. So it's like, is that the best model for your business. So what we've done is we've built out our own quality and benchmarking system.
So we have some of our own eval criteria. We're utilizing it from some of the benchmark providers, but it's very directional in terms of how models change and you have to be able to assess what's best for your business. And what we're seeing is part of that is most businesses that are on the progressive side are noticing that 5 or 6 different limes might fit their business better than just using one single one. We talked to a customer where it took them almost 100 hours to determine that Gemini was the best for this one specific workflow. So the question is, how do you get to that answer faster?
Like, how do you not make something that takes 100 hours down to like 5 or 6 hours? So something that's like newer in the model space is like all these open weight models are, are here and they're here to stay. And what's very, very interesting is you're seeing these large inference or GPU farms that are popping up here in the United States. So you have people like fireworks, for instance, that host GLM 5.2 when I think they're launching Kimi as well.
Richard Walker: 13:58
You got to clarify, people don't know these terms. Quinn GLM right. What is that?
Max Schuman: 14:03
All open weight models.
Richard Walker: 14:05
What's an open weight model first?
Max Schuman: 14:07
An open weight model is an open source model that can be deployed by a customer or a GPU farm. And what's interesting about this is like, let's use opus as a band or like, as a is like a measuring point. So opus we're talking.
Richard Walker: 14:21
About opus, which is a high model. Sorry. Yeah.
Max Schuman: 14:24
Oh no. All good, I appreciate it. And it's very high quality. It does things really well. It thinks it's an industry benchmark and is something that always delivers quality.
And it's always recognized as being one of the top tiers. What's interesting is you have these United States companies that have gone and launched these massive data centers and thrown a bunch of GPUs into them, and they're going and bringing these open weight, open source models into the United States for customers to access. What's very, very interesting about this is that, as good as Opus is, these open weight models are very, very, very close in terms of the overall quality. The difference is it's about five times cheaper to assess and to route to some of these open weight models. So wow.
Now do you go and trust an open weight model as much as you do a frontier model? That's more of the business discussion that we're having with customers is when do you start to evaluate that? Because one of the biggest misconceptions in the market today is like, I want to do token mixing or I want to like, you know, spend 60 quadrillion tokens. It's like, that's, that's fine, but what are you getting out of that? It's like the measure should be, what are you getting out of?
Like the value of the tokens that you're spending today. So if you can go and route, for instance, to an open weight model by somebody in the United States, like a fireworks or a base ten, and you can get five X, the amount of utilization. Some would say you could get to the ROI conversation faster. At Trimio, we can route to all those models instantaneously. You don't have to host it.
You don't have to go. All you need to do is bring an API key. We enter it, and then now we can statically route within minutes to some of those models to set that eval criteria for customers.
Richard Walker: 16:13
And some of the costs or price points of these models. Just make the AI decision a no brainer. Like who cares? Imagine if gas was $0.03 a gallon, you'd be like, I'll drive anywhere, I don't care. Yeah, exactly.
So that's part of it.
Max Schuman: 16:28
Right?
Richard Walker: 16:29
It does. Max, I gotta tell you, I just bought a new machine. It's like a week old sitting here on my desktop. And it has an AMD chip in it called Ryzen, which is for AI. It literally says it on the tag.
It's for AI. Like this is designed to run AI. And it's a super powerful machine. I can run my own LLMs on it, which I haven't started. Don't even think I'll do it yet, but you're in it.
Max Schuman: 16:53
I know you well enough.
Richard Walker: 16:54
I might, I might, but it's not my number one goal. My number one goal is just to be able to process more code faster, right? And offload it from other machines. Like especially my, my other machine that I'm talking to you on. And I'm, I'm looking at this going like, okay, now that it's so fast, can I afford all the tokens?
My gosh, I can run so many more tokens through it. Can I even afford it? So to be able to go to another model, that's a fifth of the cost, a 10th of the cost, that would change the dynamic of what I can do. And so through your product, then a lot of people have never heard of fireworks as an example.
Max Schuman: 17:26
Yeah.
Richard Walker: 17:26
How are you enabling people to discover this and then take advantage of it and try it out and benchmark it and see if it really does what they need it to do?
Max Schuman: 17:34
Yeah. No, that's a great point. So first thing is we have a quality catalog, so you don't even know where to start with evaluating some of these new options. Think of it as just like a live updated quality score. So we tell you the cost of, you know, how much it is to, to work with somebody like fireworks, we show you the value per workload.
So like, let's say you're looking at maybe building an agent or you want to code something or you have long contacts or whatever the use case is, you can kind of qualify that tag. And then we'll start to tell you what models perform, why they're performing that way and the associated price tag. But the big thing is, you know, we never force a customer into saying, hey, by the way, you're, you're suddenly routing to fireworks. Like that just doesn't work. So all they need to do is to go to a fireworks display or base ten or any of these companies and get an API key.
They come into Trimio, we give them a mask key, which is great. So if you're if you're focused on security, one of the big scary things right now is having an API key and having it leaked with ours since they're masked. We can set budgets on them. So although it is a live key behind, we can make sure that things don't go off the rails, which is a really, really big thing. If you have security concerns about, you know, like fireworks or a base ten or any of that.
So that's a big thing. Then we get into what can do for you in terms of a routing decision. We have two routers here at Trimio. The first is static. So let's say everything that would originally route to this model.
Now statically routes to this model. 100% of the time. But what's interesting is we allow you to set thresholds. So let's say you did want to start to evaluate one of these new open weight models with base ten. What you would do is we can set routing rules specific to the customer.
So 90% goes to the original model, 10% goes to the new model. So you can start to see in a very, very small concentrated workflow or in a very, very small concentrated way. How that model is performing off of what your super used to to see if that's a viable option, and if you're getting the same quality out of it. That's the first thing. The second is more where we think the market is going to, which is a dynamic router.
So we mentioned anthropic opus. Very high level of quality. Does a lot of things very, very well. How do we start to find more models that are inside of that kind of same quality threshold that are super high in terms of quality? They're good at thinking, they're good at reasoning, they're good at damn near everything.
But we pick 4 or 5 of those. But what if one of those open weight models is 80% cheaper? What we'll do is think of it as like a classifier. The request comes in to Trimio. Trimio looks at the request.
And inside of that band we can dynamically route to the correct model with no static routing protocols. So we're picking the best model based off of the task as well. So that's a little bit ahead of it.
Richard Walker: 20:43
Sounds crazy.
Max Schuman: 20:44
Yeah.
Richard Walker: 20:45
It's just I mean.
Max Schuman: 20:46
It's not it sounds great. Yeah. I mean the technology is but it's like, you know, people AI is predictable. And if you get a good and you're great, you know, use case of this, if it's predictable and you get high quality, it's like, well, do I want to potentially evaluate 4 or 5 different options.
Richard Walker: 21:03
Maybe look as far along the path as I am on AI? This still just sounds kind of crazy in a, in a very distinct way, crazy in the sense that won't this slow down all my requests by like 80s to do all this analysis and figure it all out and to like, how, how, how do you know all the context to be able to make that decision to route it to a different model or not. And what I'm sensing is the models are fairly equivalent. The context may live on my side more than anything else, or within the request itself. But I mean, it just seems.
Max Schuman: 21:35
More. Equivalent. And that's a super good pickup on your side there. Way more equivalent than people think they are. And we're a great use case.
I mean, we have like 15 agents. They're all named after NASCAR drivers, which I know you're a car person, so I know you would appreciate. We just launched a new one called Smokey. So we're very excited to have Smokey in the fleet, but we've changed our own models routinely. They started on Quinn, which is an open weight model.
They've been with GLM 5.2. We've now had them running through 5.6 Luna, which is an open AI model. It's super cheap. We've had really, really good productivity out of it, but like we're a great use case for some, for some business, by the way, that has gentrified a lot of processes. But we've also changed the model like six times because what we're noticing is.
There were eight frontier model updates last week alone. There was a fable that got launched, there was Astra, all these different things. So like, it's very overwhelming being on the customer side being like, oh, Astra came out. I'm just going to go and root there. But like, you just don't know.
It's so blind today. So it's like, how do we just get you a little bit closer to having the data to go and analyze these new models and make sure that it's economical, like one, you, you just have to have the data to even evaluate something like that. So until you know, these model updates start looking more like iPhone updates where like it's just, hey, you get 3% better battery, you're like, awesome, I'll probably save the 1500 bucks. It's just like you're always going to want the knowledge to like, you know, going in and seeing us something new is potentially better.
Richard Walker: 23:19
All right. I'm going to ask a very selfish question. I don't know if any listener cares about this question, but hopefully it'll help somebody. In your experience of switching models, is that also inclusive of software development coding like or have you. Have you found models that are better than anthropic for coding?
Max Schuman: 23:37
It depends on who you ask. So but I think that's like a great testament to where more businesses are probably at, which is like Lorenzo and Brad are much more on the engineering and coding side than I am, than I am. So their thresholds for what works are probably a little bit stricter than mine. I was super curious last week to see Codex has a bunch of available inferences. They're making their own chips now over at OpenAI. I went and got a Codex account, so I'm starting to run these models against each other because I want to know what's best for my specific use cases, you know, helping to run Trimio.
But you know.
Richard Walker: 24:20
It's actually benchmarked last night, three models and three variants of anthropic models, frankly. And I took one specific problem, and I had to create three effective copies of it, and I ran. Sonnet 4.6. Yeah. Sonnet five high and sonnet five medium, which.
If anybody's listening. Sonnet. These are all sonnet versions at anthropic, and they're just older versus new. And then. Thresholds of thinking, high versus medium thinking.
And I just wanted to see like, if. What I'm doing, could it be better with a different model or not? And here are the results. The medium sonnet five medium didn't do part of the job. It just said, I'm not going to finish.
So it looked like it used fewer tokens, but it actually didn't finish the job. The 4.6 versus five used almost identical amounts of tokens to do the same problem, but five did it way better. So then you're like, oh, and then what's the cost difference? Well, one's $3, one's $2. So I think I have my answer.
I go to five high schools to get a better outcome faster by the way, for less money. Now it makes sense to do it. And I don't know if people are thinking about how to do that for themselves. And I literally did it on the same code base, so it would be identical work. You know.
Max Schuman: 25:34
You just opened something super interesting and you need to go further and explain that to the guests, because what I just heard is something that we're working on. But you were ironically one of the people that kind of got us going down this dovetail. So what are you talking about? Because I think I just heard that you're running your own eval framework for more judging, maybe model efficiency for specific your business and your requests. That's starting to get very, very, that's a hot button that's happening.
So I'd love to hear kind of maybe why you're doing that.
Richard Walker: 26:11
I am an entrepreneur. I see a problem. I have to address it. And it's not for anybody else but my, by myself, I, I started looking at my costs and I started thinking, I've chosen 4.6 because my first assessment a month and a half ago was that five introduced three times as much time to complete a process and three times the cost because of overhead and thinking whatever. I've also seen refinements from emails from Claude saying, hey, we've got these refinements in the sonnet five model.
And I thought, well, maybe it's time to test this again. And honestly, tests are really hard if you don't have the same test over and over again. And while I'm doing software development, I don't want to do the same work three times. But I finally said, you know what? We're going to choose one of these tech debt problems that is complicated enough, and just do three different windows and have it all work at the same time.
And the fourth window is evaluating the other three. So I had, I had, I think I had like opus, a higher model doing the evaluation, by the way, but honestly, I was looking at this, can I save money? Can I go faster? Can I, can I use fewer tokens and spend my money? And I want to stay inside the anthropic family because I trust anthropic for software development.
And I haven't tried any other things for software development yet. So it really was just spur of the moment. I mean, this took ten minutes to set up and then whatever time it was like an hour for it to run on each of those sessions and, and then do some comparison. But I just, I was super curious and I didn't know how to do it any other way. Max.
Max Schuman: 27:38
Yeah. Yeah. You just hit on exactly what kind of is, is starting to become applicable, which is it's almost more yes, it's eval criteria because it's specific to you, but it's almost model efficiency and a super interesting thing that's kind of starting to come up that we've, you know, been playing with is that eval criteria is one, like, we can't judge what's important to you, but we think we can get you close, meaning we can be the harness or like, we can be the testing vertical because we can route to all these different models to be able to tell you how many turns it took to be able to tell you the total cost, to be able to tell you the time, the amount of turns that it took. So like, we're getting close enough for you to be like, hey, like, this is what's important to me and why. So, but you know, it is also something very, very interesting as well.
You're starting to go through this. If you go and run that same exact request, let's call it ten times over, you might get a drastically different answer from sonnet five because it is thinking so much more. So something that's also coming up in this eval criteria. We had a really, really interesting call with the head of ASU for AI Applied Science, and she's been in space since like 2023 when nobody knew what eval criteria was. And she was already starting to work towards that.
But they even think about it in terms of model consistency. If I go and run the same model ten times over, am I going to get a similar or a close-ish answer? So like in eval criteria, something that's also coming is like how consistent is the response of these answers? So if I am running a workflow called 100 different times, like how consistent is that output to the customer? It's a very, very, very hard thing to go and judge.
Richard Walker: 29:30
You know. And this goes back to the earlier test I did in August when I tested 4.6 versus five. I was using a very constrained approach because I have a very constrained methodology of how I use LMS in my software development process for a very particular purpose. And what I found was five was overkill and was just taking too long. And that's still, still true.
That is still true. That's because of how I've constrained the context and how I've constrained what it can and cannot do. It is still true. So a smaller model or lower end model works better in those circumstances. What's fascinating to me is how many people are just like, I'm going to hand it all over to fable.
And because fables are so smart. I'm just going to let it do it all. And I find that to be crazy because of the variability in where AI can land depending on when it's doing and thinking, etc..
Max Schuman: 30:20
Yeah. And like, that's a, that's a great point because most people don't sometimes understand that Sonic can be the better answer depending on what you're asking it. It's just, hey, so good. You know, and it works in both ways. People always assume that like, you know, sometimes if you have a, it's, it's price per task, but it's how quick can the LLM get to the output is what people need to measure.
Sometimes the cheapest model is by far and above the answer because it's just giving you what you need. Sometimes that more dense kind of fable or astro like model. Ironically, on a more complex test, even though on paper it's ten times more expensive, might be able to get you the answer faster. So it's like pricing is very, very subjective in AI based on specific UK use cases and evaluation criteria. That is 100%.
The hardest part about evaluating new models is how's it gonna affect my business and how do I do it with scale?
Richard Walker: 31:20
I think we're touching on a problem that deserves a lot of conversation, frankly. We give out these tools to our team. We didn't train them on which model to use. We're assuming they might think about it, but they may not think about it. Yeah, everybody might say everything goes through opus more expensive than sonnet versus when they only need haiku, the cheapest one at anthropic, etc..
I mean, you can extrapolate that against models. I think if you can show me, I have these users who are maxing out their usage on fables, then I can have this conversation with them. Like, is this paying off to use fables every single time? Why are you using it? And the answer might simply be, well, I thought it was the smartest and so I thought I should use it.
But I think you.
Max Schuman: 31:58
Just gave me a layup there.
Richard Walker: 32:00
No it is. I mean, this is what I'm hoping you guys can do.
Max Schuman: 32:03
Yeah we can. So we can break it down on one of the big things that's like a bit one of one of our strategic advisors for on our board. He's been a CFO for 30 years. And we, you know, we talk about AI FinOps, but like, what, what does that actually mean? Like, what does that get you?
So one of the things that we've decided to go much further in is like that very, very granular level of reporting. So for instance, people that are just using and abusing fables, we can not only track it down to the model and who's using it, but we can see down to a request or why, you know, kind of building justification. We can see exactly who's doing that. So, you know, in our belief too, it's very hard to educate like a non AI user into why they shouldn't use fables inside of Trimio. We can set very, very strict parameters around like, hey, X amount goes to fable X amount goes here inside of that router too.
So also it is an interesting decision because what we've noticed is like that back to the very, very first question about governance. Ironically, that right there is a governance issue and it's very, very hard to control if you don't know what's coming. It's like, for instance, if you go to, you know, dinner with a vendor or somebody like that, you might be a little bit more inclined to get something like the Wagyu tomahawk, right? Where if you go with your significant other, are you going to get it? Maybe.
But like, are you going to think about it twice? Probably like, so it's just very, very different. But like that example there is if you're paying the bill, you're typically a little smarter about maybe that model selection. So we've built in a lot of those routing decisions to say, hey, if things can go to a sonnet or a haiku, which are anthropic models, and they can obtain that same level of quality to the user, why wouldn't we choose the more economical path? Well, we can and we can report on all those data sets as well.
Richard Walker: 34:02
So I don't know if the audience is intuiting this and I'm, I have to wrap this up like we're getting over time here for my normal show length, but we could talk for hours, I think. Max, we can't. We already have. Actually, I want to leave the audience with this kind of thinking. When you look inside the cockpit of a jet plane.
You see knobs and switches and all sorts of equipment. But really all that matters is fuel speed up and down, right? There's only like 5 or 6 things that matter. That's Trimio. They're going to give you just the core basics you need, but the whole cockpit of Trimio and all the things you can do, it's so amazing.
For those who are technical and looking at how you can control things. So I mean, kudos to you and your team for what you're building there, Max. All right. Before I get to my very last question, what is the best way for people to find and connect with you?
Max Schuman: 34:51
Yeah. So we have www.trimio.ai. You can reach out to me. Max Schuman, it's SCHUMAN.
On LinkedIn, it's max.schuman@trimio.ai. You can go to our page. You can reach out. Being a small company, I'm about as connected to all the different marketing platforms. So if you hit one of them, I'll make sure to reach out.
But even if you just want, like, you know, one thing is, I believe this space is very, very hard to get honest feedback, reach out to rich people. Rich is like a very, very good litmus test towards like, you know, are you going crazy? Or like, what applies, you know, for like what you should be focusing on? Same with us, even if it's not directly, you know, in touch with what we do at Tromeo more than likely we can guide you on kind of what to do or maybe what to focus on, on where to start. Yeah.
Richard Walker: 35:48
And I would say go follow Max. He's got some great posts consistently going out about what's going on. You're going to learn a lot, if nothing else. All right, let's get to my last question. So Max, who has had the biggest impact on your leadership style and how you approach your role today?
Max Schuman: 36:05
Yeah. Loaded question. I mean, the first person that comes to my mind is my dad. He was an entrepreneur. I'm following in his footsteps.
I'm super grateful I got the opportunity to work with my dad and he just teaches me a lot. So always a big team, team person. And he took a gamble early as well. And you know, it worked out. So I wanted to be able to kind of show my family that I would be able to take a risk and do something like that as well.
And then, you know, professionally speaking, my dad, Chris Herpes, another one who is our president and our last company, Chris, is an awesome business operator. And then a big shout out to our board as well. We have Steve Bochner, Rob Chaplinsky, Kirk Robinson, Anthony Mackle. My dad's on our board, which is awesome. And they've been great because they've all been business operators and I'm learning how to be a business operator.
Like they have those battle scars, so maybe I'm going to have them. I just hopefully won't have as many because I've got, you know, an amazing group of people to run ideas by, but very, very grateful for, for our entire board and them kind of guiding you on, on what to do.
Richard Walker: 37:18
You're smart to surround yourself with great people. That is awesome to hear. All right, I'm going to wrap this up. I want to give a huge thank you to Max Schuman, president and co-founder of Trimio, for being on this episode of The Customer Wins. Go check out their website at trimio.ai.
And don't forget to check out Quik! quickforms.com where we make processing forms easier. Hey, I really hope you enjoyed this discussion. We'll click the like button, share this with someone, and subscribe to our channels for future episodes of The Customer Wins. Max, thank you so much for joining me today.
Max Schuman: 37:48
Thank you, Rich and I know we'll be talking soon.
Richard Walker: 37:51
Yes we will.
Max Schuman: 37:52
Thanks for having me on.
Outro: 37:55
Thanks for listening to The Customer Wins podcast. We'll see you again next time. And be sure to click subscribe to get future episodes.
Comments