Hermes (00:07)
Welcome to the Convo AI World Podcast, where we host the co-founders and leaders pushing the AI space forward.

For years, the AI race has been about building bigger models, but our guest today has bet in the other direction. I'd like to welcome Neal Patel from ScaleDown AI. Thanks for joining us, Neal.

Neal (00:26)
Thank you for having me.

Hermes (00:27)
So Neal, in a couple words, tell me what what does ScaleDown do? Yeah.

Neal (00:33)
So at ScaleDown, we're building task-specific small language models. So as opposed to the large general purpose models from Frontier AGI labs like OpenAI and Anthropic, we are building custom models for individual tasks. these These base models are really good at things like classification, summarization, and data extraction.

Hermes (00:56)
So what inspired you guys to start ScaleDown?

Neal (00:59)
Yeah. So I was at Accenture doing data center optimization. So I worked with a a lot of lot of big hyperscale customers that you can probably think of. And when I started, it was like 2020, 2021, and we were doing a lot of like cloud storage and in cloud storage data centers primarily. And we were tasked with the responsibility of thinking through building out the training infrastructure and the inference

that we needed to kind of scale AI to where we're at today. And I was the finance guy in charge of doing a lot of the modeling and forecasting to understand like how many GPUs do we need, what's our token demand gonna look like. And really what we noticed was no matter how far out we went, 10, 15, 25 years, there was no real kind of alignment of the supply and demand curves. And what that means is there was just never enough GPU or hardware supply

to serve all of the demand for inference and training that would be required to kind of satisfy the world's need for tokens.

Hermes (02:03)
Okay. So here you are, you're at Accenture. You're like, there's no way we're gonna make this work if the models just keep getting bigger, right? Exactly. So what was that kind of like aha moment for you and maybe your co-founder is like, okay, this isn't gonna work. How can we change it?

Neal (02:22)
Yeah. So like first I just like needed to get really smart on the space. So I just started reading a bunch of AI research papers. I read a decent amount like on my day-to-day job of like building the forecasts and the assumptions to build these four like models but yeah, just like started getting really fascinated by efficient AI inference. So like tiny ML, new model architectures, small language models in general, the open source world, all of that. And

really, that's where like it started to make sense to me. And like taking a step back to the to the supply and demand curves that like kind of like kickstarted this passion. I knew that everyone was working on AI from the hardware side, right? Like everyone was like Moore's Law, GPU, NVIDIA, AMD, like they're gonna solve all these problems for us and and build all the the hardware that we need. And so that like seemed like a relatively solved problem. It's like some of the greatest minds were working on that.

What didn't seem like a solved problem to me that I was very interested in was like the demand side of the equation. So like actually lowering the amount of tokens that we needed to serve while still creating all of that value and getting the job done that like the world needed through AI. So yeah, like I was really fascinated by token efficiency, tokenomics, like all of that stuff, and like the research that could could make that a reality. What really kind of like stuck and like like drove all of this was

my co-founder and CTO, Soham, he's an AI researcher. He had like a ton of pub like papers published on this topic. And like one of his papers in particular that he was working on was around persona vectors. And that was kind of the foundation of ScaleDown's task-specific model architectures. But like when I read that paper, I was kind of like jaw dropped and like read it like 10 times because you were able to do so much more

with so much less, right? And that was like really the the token efficiency. Like the models we're working on are like four to eight billion parameters and outperform models that are hundreds of billions of parameters, right? And so if you think about the hardware side of that, it's like 10x to 100x less GPU capacity necessary to to do the job.

Hermes (04:34)
So it's like I think I've heard you say it once as the

it's like getting the smarts of Opus at the cost of Haiku. Exactly.

Neal (04:44)
Cheaper than Haiku, more accurate than Opus for a specialized task, right? I'm not saying that like we are a generalized model that can do everything under the sun. Like you kind of, Instead of like when you go from Opus to Haiku, you're trading off accuracy for cheaper and faster. When you go from Haiku to ScaleDown, you're trading off generalizability, but you are getting that accuracy for that one specific task. So, like if you're summarizing documents or doing data extraction, like

we will beat the performance of Opus and be cheaper and faster than Haiku. But if you try to do anything outside of summarizing, it's not gonna work. If you try to do anything out outside of data abstraction, it's not gonna work. Right. So agenc workloads were were kind of the rise of of task-specific models or the enabler.

Hermes (05:29)
So when you say specialized models, right, for not only summarization but categorization and things like that.

What's that architecture look like? What's what's the difference between that generalized model and these small language models? And does it take specific training? Like are you training them for customers, or do you have more general models?

Neal (05:53)
Yeah. So like 80 to 90% of our customers are just using the the base models as we call them, where they're not fine-tuned and and not custom. We do kind of like some of our larger customers are fine-tuning the models for their particular domains and and workflows. But what I mean when I say a specialized or task-specific model, a lot of people just like automatically go to fine-tuning an open source model or distilling a larger model, quantizing sometimes.

That's not at all what we're doing. So what we're doing is we're going into the actual model architectures and adjusting how the generative decoder works. So taking a step back for for anyone new that doesn't understand how like LLM architectures work, the easiest way to think about this, and like if you've any of you have seen our our launch videos, we we kind of compare our models as like smart cars and mini coopers, and like the large language models as 18 wheeler semi trucks, right?

So that's like the easiest analogy that I can use to kind of explain this. So to spend a minute on this, if you think of an LLM as an 18-wheeler semi truck, there's two parts to an LLM. First, there's like the pre-fill stage, which is effectively the engine of this 18-wheeler, and then there's like the trailer and the body and the 18 wheels and all that kind of stuff that's in the back. And that's the generative decoder. So what we do and like

like just double clicking that pre-fill architecture is the ingestion. So basically like one forward pass over all of your tokens. So your context, your system prompt, basically like ingesting all the data that the user is putting in or the agent is putting into a model. So that's pre-fill. That stage is super powerful, right? It's like a lot of horsepower in that engine. It's very fast. On average, like 12k tokens per second, somewhere around there is pre-fill, just like for a sense of it. The generative decoder is the part that gives models their ability to do things that are general

like write code and write emails and tell you the temperature of chicken thighs for like the 15th time or whatever. Right. So that you can do anything under the sun because of the generative decoder. But it's very slow. And the reason for that is because it produces and generates one token at a time. Right. And every token that it generates, it now has to look back at every single token that came before it. So if you throw a 100k tokens in to the pre-fill,

when the generation happens, it looks at a hundred thousand tokens, generates one more. And then looks at a hundred thousand and one tokens, generates a second one. Hundred thousand and two tokens, generates a third one. And so that process is really slow, generating around like two hundred tokens per second, right? So a huge drop-off from the 12k of the pre-fill. So that's just like like model architecture 101.

Hermes (08:37)
So that's kinda like where you see that latency like, it read it in, now it's Yeah, now it's changing. It's processing, doing that kinda like that one token at a time prediction. Oh, this is the next thing. This is the next thing. Exactly.

Neal (08:48)
And that's why like input tokens are so cheap, output tokens are so expensive because all that horsepower is used from that pre-fill to lug that heavy trailer and those 18 wheels around. So that's like the generative decode part that makes models so powerful and like kind of general, like I mentioned many times. But it's the part that we can kind of drop. That's what we drop when we build our models. So that's just like one-on-one. Does that make sense? And I can tell you about ScaleDown more specifically.

Hermes (09:16)
Yeah, no, no, that totally makes sense. And because you're dropping this, you're obviously getting a smaller model. So you have the full power of the ingest. Yep. And then walk us through how that now differs from the large language models. Like you got rid of the decoder, what's in its place?

Neal (09:33)
Yeah, great question. So that's like the ScaleDown secret sauce, right? And I'll and I'm happy to share more about that. It's a really hard process to do. But basically what we do is we keep that pre-fill architecture from like open source large language models. And then what we do on the the generative decode side when we drop the trailer and the 18 wheels is we build our own decode architecture. And that for different models, you need different types of decode, right? For classification, you need different neural nets. And we call them heads

like architectural heads is like the common phrase in in the AI world. But basically we build our own heads that we attach to this pre-fill stage so that the output is much like more accurate for a particular task. So like let's take classification and extraction. Those are two types of tasks or use cases that you don't really want a model to generate anything, right? Like if you give it a document, you don't want it to go generate some new output.

Same thing with extraction, right? Like you just want it to pull out something from the document that you gave it, not write a new sentence that doesn't exist. And that's where like back in the day it was hallucinations. Now more people are thinking about it as like consistency is more of the the eval metric that people care about. And LLMs are really hard to get consistent or deterministic, right? And so what we do is we build, we test like 80 different pre-fill architectures to build our new model, and then we build

our own proprietary neural nets that basically do the decode step for a particular task. So our classification decode looks very different than the extraction decode, and that looks very different than the summarized, right? So all of these are task-specific decodes with a pre-fill that matches that task specifically. That's the long and short.

Hermes (11:22)
No, that makes complete sense. And now if you think about it from like that generalized decoder, because it's not generalized, it's not asking to predict which token to make up. It's saying like what tokens exist? How do I pull them in and put it to the format or how do I make a judgment call on the values, right? Exactly. Exactly. Okay. Do you ever see instances where

let's say for like a classifier, something's falling outside the bounds and it can't, your classifier doesn't pick it up. Does it then offload to a bigger model or is there a way that you guys kind of like tune that out of the system? Yeah.

Neal (11:58)
So I think the the easiest way is adding like a no-match or ambiguous label. So kind of like the the how to use the models differs quite a bit. And what I mean by that is like the system prompt. So for a general purpose LLM, AI engineers and and like even non-technical people in chat windows, are just like basically writing a system prompt when they write the instructions, right? Or if you're actually writing a system prompt, you like have

kind of some instructions for the for the agent or model to follow, as well as like some few-shot examples, et cetera. And that's like the classic architecture for a system prompt. For us, it's not a free text box that you can just like throw in anything and tell it all these instructions because anything you put into our summarized model is going to be summarized. So you kind of can skip over the instruction side of the house. Examples are always good for models to give them some direction of like what you want. And so like for classification

it's literally just class names and labels, right? So if you put in like let's say an email thread from a customer, and you're like, is this a technical inquiry, a billing inquiry, or a sales inquiry? Right. You throw in that whole email thread and then in the system prompt, all you're doing is defining what constitutes a technical email, what constitutes a sales email, and what constitutes a billing email, right? Okay. Like what are the terms that they should look for? What are the like kind of examples that that would

can constitute technical versus billing versus sales. So that's like how our system prompts work.

Hermes (13:32)
So you define the classifications that you want to kinda look for. Yeah, exactly.

Neal (13:35)
Yeah, exactly.

And then there are a lot of classifiers, and like that's been really popular as of late, which we can talk about more. But like that is one part of it is giving like the actual output of it's a technical email. That's great. But a lot of the customers that we work on want more than that, right? They don't want just like a number and an output of like a probability and the answer. They also want a reason, right? They want the ability to actually

understand why it was selected as a technical email or why it was selected as a sales email. So that's something that our models do, our classification model in particular does. And then for extraction, citations are really important, right? Like legal finance companies are using this. They need to be able to say where did we extract this this data point or this number from? So giving the string of where that actually came out of the context is also something that we've been able to do with our extract model. And that's all because of the decoders and the neural nets that we've built that are

task-specific.

Hermes (14:32)
Okay. So there to summarize, there's no real prompt. Like the models know the system instruction, whatever you give them or your users pass to them. Yeah. They know this is how we're acting upon it. So just look either for the success criteria, the classification criteria, or an example. Yeah.

Neal (14:52)
Yeah, and that's like for classify and then for extract, it's like entity name and entity definition, like what is it like you're looking for name or email address or bill like volume or total bill or whatever. So you can define the entities when you're doing extraction. So each model has a system prompt. It's not that it doesn't have a system prompt, it's just that you're not giving it the instruction like you would a general purpose LLM to do extraction because you're just using an extraction model, you're just telling it what to extract, right?

Hermes (15:23)
Makes sense. And then you also do summarization and kind of like compression. Yeah. Right?

Neal (15:28)
Yeah, those are two other big models that we have. Small models but

Hermes (15:31)
Small model, yeah. But powerful.

Neal (15:33)
Yeah, exactly. Small but mighty. That's right.

Hermes (15:35)
Do you guys ever layer the models? And like 'cause I would imagine a classifier that needs a reason kind of needs to be able to like use that citation type feature, right?

Neal (15:48)
Yeah, yeah. So we're working on a router, so that will kind of be a thing that we we release soon enough. But the the current way that like people are using these models is is in large part due to agentic workflows. So for those that are kind of thinking about agents or have heard the word agents but don't really like know why that's different than a chat bot or anything like that. I'll take a second to explain that. So like maybe a year or two ago, everyone was like using

chat windows and like ChatGPT and and like Claude Chat, maybe Co-work if they were crazy. The kind of progression to that was just like kind of like one-shotting agents. And what that means is like basically you give it a long system prompt with a bunch of guardrails and like the system prompt started to balloon into like thousands, if not tens of thousands or hundreds of thousands of tokens every every query. And basically

like that would be asking Opus to do 15 different things, like summarize this and classify it and extract it and write a new version of it, right? Something like that. What people have realized and what gave rise to agents is actually breaking it into different steps, right? So the aforementioned stacked prompt, where you have 15 different things going on in one query, is a lot less accurate than breaking that into 15 different steps

and 15 different LLM calls, right? And when you break that one stack prompt into 15 different steps, that is effectively an agent. you can also call it a workflow, but if it loops a bit, you can call it an agent. And so that gave rise to using task-specific models because now you're not forced to only use the biggest model that's the smartest and has the highest reasoning capabilities. You can understand what model is going to be good at this classification task and this template task

and this reasoning task, et cetera. And so a lot of what AI agent companies and and startups and enterprises are doing these days is they're actually thinking through like not just what is the right system prompt to give to this one step in a 15 step agent, but what do we actually use as the model to do that action? And so it gives them a lot more flexibility, obviously a lot more complex, but understanding like, is GPT 5.4 nano better at this? Or is

Sol better at this or is Haiku better? Like you can kind of change the models. Is a DeepSeek model better, et cetera. And so you can change and test different models to see across the 15 different steps. Can we call, is it 15 different models? Is it four models across the board and use them in different places, et cetera? So it gives a lot more choice for task-model fit.

Hermes (18:35)
No, that makes sense. Now with the agent, like you said, it could be a workflow or an agent workflow being something that's more automated, less reasoning. Yeah. It's not looping itself. Exactly. Really. Or maybe it is, but just on a classifier type basis. Exactly. That's

Neal (18:52)
dynamic. Like an agent I'd consider more dynamic, like has to kind of figure out what's going on. And like the the work is a little bit different because the inputs are different or the task is different, etc., versus a workflow is like a very defined kind of step-by-step process that just like goes over and over again, like a gum loop or an N8N or something like that.

Hermes (19:12)
Makes sense. And and that's kind of timely for when we think about things like voice agents because you have the one agent that's kind of like the orchestrator, the the one you're talking to, communicating with. But then would you give these task-specific models more like these are tools within that agent's tool belt that can send off this chunk of data to summarize and like how do you see that fitting together?

Neal (19:39)
Great question. So I think like per this progression it was like stack prompts, then workflows, then agents, and then now kind of the the 201 of agents is this architecture where you have an orchestrator agent, generally like the biggest, largest general purpose model that you can get, thinking through kind of the plan and the orchestration of how to use sub-agents. So it's like an orchestrator agent and then sub-agents, right? Sub-agents are

the models that you can have a lot more flexibility on, right? I generally recommend people that I talk to, especially if they're building voice agents, to get like the best agent that they can get, or the best model they can get for an orchestrator that has the right latency requirements for them, right? You want the most reasoning, the most host horsepower and generalizability to deal with a human on the other line or whoever it's calling, maybe another agent, who knows? But you want to have like a really good model and a big model for an orchestrator, assuming latency is covered

and requirements are met. Then you have sub-agents. And that's where it gets really interesting, right? So we work with a lot of voice AI companies with their sub-agents because that's like right 201 of this like kind of agentic workflow where you can pick the right model for a particular task. And so when you think about sub-agents, there are a lot of like you can spin up hundreds of sub-agents to work in the background of a call as you're talking to like a human, as an orchestrator is handling a human call.

And so that can be in the form of tool calls where you're hitting APIs for like EHR dot data or financial data or whatever, like you kind of need to pull in to kind of validate or get the information to the human through the orchestrator. So the orchestrator is kind of always in communication with the sub-agents and the sub-agents are spun up and there's so many of them because each of them is doing one specific thing really quickly and then feeding that to the orchestrator so it has the right information at the right time.

So most of our models are actually used at the sub-agents level because they're quick, fast, and and specialized. So similar like examples of how people are using our models for sub-agents are like patient name extraction or financial financial document classification or like referrals or like a lot of like sales companies and like receptionist companies are are using this to kind of like make sure that the patient or person that's calling

is the right person and then they're routed to the right documentation that they have in the back end. So the sub-agents are kind of the like connection to any data sources and are doing all these different tasks to then feed it to the orchestrator. So the orchestrator doesn't need to waste its tokens and its time accessing all this data in the back end.

Hermes (22:24)
Super interesting. Raises a whole set of questions around, so you have the orchestrator probably running in a cloud. Do customers for ScaleDown traditionally run your models in the cloud or do they self-host to keep it close to the data and you know, privacy requirements? How does that all work? Yeah,

Neal (22:43)
both. I would say a majority right now are using APIs, and are cloud hosted. Our like because our models are so small, the latency is is pretty negligible. We have pretty high throughput and and pretty low latency. And so like the APIs are are very functional for sub-agents and a lot easier to just like kind of test and swap and benchmark and do all that good stuff that that teams like to do. Obviously there are instances of larger enterprises that

are hosting these on their own hardware, close to the source in code or edge sites, et cetera. Those are generally not for latency requirements, generally more for like security and data protection and compliance requirements. But yeah, we offer licensing models like a licensing option for like kind of getting the the weights of our model and and hosting that on your own hardware. And then also offer like an API. The interesting thing that I mentioned before

was like in traditional LLMs, input tokens are relatively cheap. Still not that cheap, but that that's another point. Output tokens are more expensive, like significantly more expensive. For us, we've kind of flipped that narrative on its head where input tokens are for us really cheap, like five cents for a million input tokens. And then output tokens, we don't charge anything, right? Because we don't have a generative decoder.

And oftentimes in classification and extraction and summarization workloads, there isn't a huge output token volume that comes out. So like we kind of eat the cost of that, but for our customers, you don't need to worry about output token costs at all. So that's where like the significant cost savings comes in from using a task-specific model from ScaleDown. And yeah, I think like generally what we found with that is because that like frees up a lot of the like compute and like

GPU capacity. So you can kind of stretch your GPUs further. A lot of companies that are at like enterprise levels like to see that because now they can have 10 GPUs instead of a hundred or a thousand, for example.

Hermes (24:53)
And those GPUs are hard to come by now. Yeah.

Neal (24:55)
Exactly. Exactly. They're a hot commodity.

Hermes (24:57)
The the more use you can get out of what you got, probably the better. So we talked a lot about classification. I want to come back to it because that's really like a hot topic, especially with the recent releases this week. But I don't want to overlook like one of your really interesting models for compression. Because when we first got introduced, that was like your frontier, you're focused a lot on compression as well. Totally. And how does your compression really differ from

generic like summarize this document.

Neal (25:28)
Great. Yeah. Great, great question. So the architecture for compress, it's used differently, not like our internal architecture. I've kind of alluded to how that works, but how a company would use this within their own agent or their own workflow is very different. So kind of rewind to RAG and chatbots, that is where compress makes the most sense. And so how people use compress today is they will use this

compression SLM, TSLM, task-specific small language model, in front of a large language model. Right? And so this is kind of like a more traditional or or old school approach to RAG or chatbots where you're feeding a ton of context and like chunks from a vector database that you've retrieved through RAG to a large model, right? So let's say you're a finance company and you have a big chat bot that customers and finance professionals are using to

understand like a publicly traded company's data. You are then doing RAG to retrieve all the chunks about that company and then throwing that into an Opus or a Fable or a 5.6, whatever it may be, to answer that question and reason over all of that data. And that's really where input tokens balloon, right? At that point, like RAG is is a is a pretty good methodology for getting the most relevant stuff. And you can do re-ranking and things like that as well. But you still have

tons of tokens and the hundreds of thousands, if not like nearly a million tokens going into this large language model. And yeah, as you mentioned, like we started off as a context compression company because what we realized at that point was that the more information you throw into a model, the less accurate the output. And most people think like, let me throw everything I have at this model and it's smart enough, it'll figure out what's relevant or not.

Hermes (27:24)
But I've heard that before where like it generally what you send at the top and the bottom is really has a lot more weight than the middle and the middle tends to get lost, right? Yeah, yeah.

Neal (27:35)
Exactly. The very popular paper called Lost in the Middle that kind of outlines this. So yeah, great, great recall there. So yeah, when you throw in, let's say, a million tokens into an Opus or a Fable, it's generally reading the top 25% and the bottom 25%, and the middle 50% is kind of lost. And this is more so for like needle in a haystack type queries when you're looking for particular data points or particular numbers, aka needles in this long context input.

So that was one of the like pain points. Another way to frame it is like context raw. And what I mean by like throwing a lot of tokens into a model is bad. It's like if everything in there is super important and super like high signal, that's great. But oftentimes, even if you're doing like RAG and you're chunking well and you're pulling from the vector db well and you're doing re-ranking, there's still a lot of noise in that.

And so how our compression model works is it's all about relevance and and signal over noise, right? And so basically the the compression SLM will sit in front of the large language model that's gonna answer the question and it'll basically act as like the pre processing layer. So let's say you're asking about Agora's financials over the last 20 year, or like I don't even know how CSR, but like the last five, ten years, whatever. And understanding like what was the net income,

growth rate over these years and like where's it gonna go, right? Do some forecasting. And Opus can do that really well. If you are now chunking and pulling in all of the stuff from the 10K over the last five, 10 years, you have a lot of data flowing in there. That's not all relevant to this forecast and this net income calculation, right? And so the compressed model will understand the query. So it's query aware context compression. It'll understand the query that the user is asking.

And then use that to go through all of the context before Opus gets it and understand what is actually relevant, what is high signal, and what's low signal. Basically, it will prune out anything that's low signal and only keep the things that are high signal and then feed that to Opus. So it's like a post-processing of vector db and RAG and re-ranking, and it's a pre-processing of the the the context that's going into a large language model.

Hermes (29:56)
And so that also reduces the number of so that reduces both the number of tokens, but like if it's doing that constant look back on the generative decoder, it it's gotta make that part cheaper too, right?

Neal (30:09)
Yeah,

yeah, exactly. So the the two real value props of compression, of context compression before a large model gets it is one input token reduction, which is a direct correlation to cost, right? Like you can reduce input tokens 30% to 50% to sometimes even 80%. So your costs kind of kind of dramatically decrease on these like chatbot and kind of RAG type workflows. But the other big one that was super interesting was it increases the accuracy of Opus.

So we've done benchmarks on Finance Bench and a few other kind of popular like needle in a haystack chatbot type like benchmarks and we see about a 7 to 10% increase in accuracy when you reduce the the noise in the input, right? So if you cut out 30 to 50% of the tokens and and most of that is like lossy noise, you now have only really high quality signal input context

going into Opus. So it's able to answer the question much more accurately without getting lost in the sauce.

Hermes (31:14)
Makes sense. 'Cause its decoder has all the high signal stuff that it's comparing what's the next motion.

Neal (31:19)
Pre

fill. It's pre-fill, has all the good stuff and then when it's doing the generative decode, every time it looks back, it's looking back at all good quality tokens and then producing good new quality tokens.

Hermes (31:31)
So that's huge for RAG. Mm-hmm. Can it be used as like custom compaction? So let's say I don't want to rely on well, if I'm running my own API, yeah, I'm not gonna get like the the compaction that GPT or Claude ships with in the CLI, right? So I'd have to basically build my own. Could I use your compression models for that?

Neal (31:53)
Yeah, so compress is pretty good at like compacting previous conversations. We often recommend people to use compression when they are doing more needle in a haystack type workflows. So like you can, it's not just like a single needle, like three to five needles in a haystack is fine. And we've kind of built the compression model to be able to be a little bit more generalizable. If you're thinking about coding agents and like Claude Code and and CLIs,

the way to think about it is a mix of all of the models, right? So there's a lot of commands and and like actual requests that you send to Claude Code. They're very simple, right? But the way that you use these coding agents is you kind of pick one model and run all of your coding tasks with that model. So you can either pick like Fable or Opus or Sonnet or Haiku, and that's like the one default model that you use for all of Claude Code, right? So we actually are working on a product called Diet Code

that we that we named so aptly. But basically what that does is, it uses the whole host of different ScaleDown models that we have for summarization, classification, data extraction, and compression to actually do that automatically. So it's an effective router. it's it's an MCP server and Claude Code plugin that basically plugs into your Claude Code instance. And instead you can select Opus for all or Fable for all of your kind of like coding tasks. But when you have a simple

command like a bash or a grep or something like that, it will route to a ScaleDown model to do summarize for compression or for compaction or compress for compaction. Those are kind of interchangeable depending on if you're doing needle in a haystack or just full compaction. And then like if you're doing extraction or summarization, it'll kind of automatically route to those models for different commands. So basically, instead of using Opus for all the simple tasks and simple commands, you can now use a task-specific small language model.

And in a nutshell, the value prop of doing this is one we've seen about like four X cost savings. Because if you're using Opus for like compaction and all these like really high token workflows, you're gonna run out of your session limit if you're on like a monthly plan, or you're gonna pay a ton API costs if you're over 150 seats. So that's like one big use case, is obviously cost savings. But the other one is accuracy. So we've seen significant accuracy increases in like the double digit percentage points

of improving the the quality of this on like DeepSeek benchmarks and things like that. And then the final one is latency, right? Like you, because you're moving things through a smaller model that's faster and better at this job, you actually reduce your latency and getting your responses back from all code.

Hermes (34:35)
That makes sense. So like when you're

I'm gonna put it in the voice AI context because I'm thinking about this as like every time I do a turn within an LLM or within a voice AI agent, we've streamed that text. Now we have our turn ready to submit to the lan large language model, but I can't just submit my current turn, I have to submit the whole context of turns. Exactly. And I'm just paying this LLM to remember my history. Exactly. Whereas I could picture like, okay,

maybe every two or three turns I can compact down or maybe every five turns, whatever. Yeah. So it gets the most relevant pieces, but then has just the rich context. Exactly. Pulling out the ums and uhs. and

Neal (35:21)
Yeah. Exactly. Exactly. The the buzzwords, the filler words that are just so common. And also just like anything like any tangents that people go off of on voice calls, that's like that's a common hindrance that people get is like they'll switch the topic. They'll be talking about like scheduling an appointment and then they'll talk about, actually I have this knee problem or whatever. And so, being able to kind of like deduce and and remove any of the irrelevant tangents is also a really valu like big value prop of of summarizing and compacting and compressing.

And I think like, yeah, that's like we we did a demo to a lot of the Agora folks and and fans in New York City and I showed them kind of how compress can be used with voice agents and we saw I think like 97% cost savings and like 10 %, 15% or 15x latency improvements and and obviously significant cost savings as well. And it was just like kind of mind boggling to think about that in like a voice AI sense because latency and cost are like very important.

Obviously accuracy is like paramount across the board, but yeah, those two in particular for voice are are pretty salient.

Hermes (36:26)
No, it makes sense because human thought is messy, right? At the end of the day. So it's not just like, let me pull in all this RAG detail to kind of in interject into it. Yeah. Just the human thought alone is is messy enough to throw off the model. And you're talking about all right, we're not only re reducing token counts, we're reducing latency, but you're increasing accuracy. Yeah. Like that just makes the voice agent that much better. Exactly.

Neal (36:52)
Yeah, we've we've kind of come up with this this motto where accuracy is the priority and cost and latency are the passion, right? And so like we like very much with all of the folks that we work with and clients and customers is accuracy is the number one thing. We will at least meet or exceed the accuracy thresholds that you have with your your like prod deployments, whether it's a voice agent or an like a text agent or image or whatever.

We make sure the accuracy is at or above the standard that you've set out. And then it's kind of an inherent benefit of cost and latency because our models are smaller, right? And you can only get that with task-specific architectures instead of general purpose architectures.

Hermes (37:36)
Makes sense. Could you run these small language models in parallel? So let's say all right, I have the compaction one going, I have the classifier going, so that when they both return my agent can can hit on this like more accurately.

Neal (37:51)
A

100%. And that's kind of what we were talking about earlier with the like orchestrator and the sub-agents, right? Like a lot of people that are building voice agents today are moving more towards like an orchestrator that handles a call and a bunch of sub-agents that are running in the background concurrently. And so like that is like how voice agents are kind of being architected now, rather than the like retrieve everything, feed to orchestrator, orchestrator responds.

'Cause like it just brings a lot of like lag time in between calls and kind of ruins that like human like voice and and conversational ability. So yeah, a lot of people have kind of moved away from like RAG and and retrieval in voice agents and now have sub-agents kind of doing that stuff in the back end.

Hermes (38:38)
Makes sense. So we're talking a lot about like real-time voice agents, but this technology can also apply in post-conversational usage, right? So you have the agent, yes, it uses the ScaleDown models to compress and and classify, but then afterwards, usually as a as a developer, you're left with the transcript.

Yeah, you have the audio, but like generally you you act upon the transcript. So for that, you're gonna like review if and classify success or failure. Yep. Look for these key items in there and walk us through what that would look like. Like let's say I ran a call campaign with I don't know, 50,000 calls, right? I called 50,000 people. Maybe 10,000 or 20,000 didn't actually answer.

But I have 30,000 people that like answered, spoke to my agent, and I need to figure out like, did it book a call? Like why did it fail? Cause it's a lot of data to go through and it's really rich. Mm-hmm.

Neal (39:46)
Great question. So obviously like live voice agents are are where latency is like the most critical. As

soon as we get to post processing of call transcripts, like latency kind of flows out like throws out the window. So the way that we think about post processing at ScaleDown and with our our voice customers is really about accuracy and cost. Like those are the two most paramount metrics as we think through this. And really like you

do a lot of post-processing when you have like a lot of data, scale, and a specific thing that you're trying to learn, right? What do we need to change about how the voice agent is built or operates? Or what do we need to change about the humans that are running sales calls or doing any of that stuff? Right? And there's a lot of like really cool companies that are doing like sales coaching and stuff like that that we've been talking to and working with. And

I think the the way to think about it really is if you are doing analysis over 30,000 calls, right? That that seems like a large number. But then as soon as you multiply that by the number of tokens in each call, if the calls are five, ten, thirty, forty-five minutes, it balloons to billions, if not trillions, of tokens, right? And if you throw that into an LLM to do this analysis.

One, it's gonna be like super expensive, but two, the the inconsistencies at that scale are gonna be significant portions of the data. So like what we've seen generally is around a 10% to 15% inconsistency or variance rate when you throw data into an LLM.

Hermes (41:29)
And that's just between runs on the same data.

Neal (41:32)
Yeah, exactly. And so like, yeah, if you run the same like one call a thousand times, you're gonna get 10% to 15% of the outputs of that LLM with like let's say you're doing classification of if it was like a a a closed one or a closed lost, right? For like pickups. It's like 10% to 15% of that is gonna be said that it's a closed lost, even though it was a closed one, right?

And so it's like really hard to build off of that shaky foundation for post-processing when at scale the samples and the outputs are inconsistent or variable. And so that's really where like a lot of people have been interested in post-call QC and QA to understand like, okay, we are doing all this like kind of data science type work, and but now we're using a non-deterministic mechanism to do so, right? And it's like if you can use an ML model to do this, like I highly recommend, but like if that's

not possible or like it's not like actually like worth the effort to kind of build the ML architecture that you need to do that. That's really where the the task-specific small language models come in. Right. And so a lot of what we see is like large volumes of classification and data extraction to remove like what were some of the keywords that we said, what were some of the closing techniques that we used, what were some of the like actual like sentiment analysis of

this call and like how is the the caller acting, how was the person that answered acting, like those types of things are really good to be done with these these smaller models because obviously they're you're doing it at scale, you want it to be cheaper, but more importantly, you want to remove as much variability of that generative decoder as you can.

Hermes (43:17)
it's wild to think about. Like 10% to 15% is not like a small trivial number. Yeah. And then if you're thinking about like bringing this into the accuracy of enterprise, it has to be a 100%. Yeah. It can't be yeah, you know, it's like 95%, it's like 85% accurate. Yeah. Like that's not gonna fly in most enterprise businesses, right?

Neal (43:46)
Most businesses in general, whether you're an enterprise or startup or even kind of building your own tool, it's like, I I think a lot about like first principles and engineering principles and like this makes me think of tolerance, right? Like if you're doing hardware engineering, there's kind of this idea of the the more t the more tolerance you add to like building like like let's say this table or this microphone, right? When you're you're manufacturing these types of products, it's more expensive

to have something that has like one millimeter of tolerance, plus or minus, to like build the size of this circular table. Because if it goes out one inch, it's gonna look lopsided, right? And so like when you're buying manufacturing quotes or buying from manufacturers, you pay more for lower tolerance SLAs or requirements, right? And so it's kind of similar to that, where it's like if you're thinking about voice agents and like the postcall processing of the transcripts.

It's like any number that you get as an output if you use an LLM, you're gonna have to slap on like a plus or minus 10 to 15% to any analysis or insights that you generate. So let's say you say 20% of these calls were closed loss. It's like more like 5% to 35% of these calls were a closed loss. And it's like, all I what are we supposed to make decisions off of something that has a 10% plus or minus variance on it? Right. And so normally in engineering world, you have to pay more.

To have a lower tolerance and be, let's say, 18 to 22%, right? That's a much more like manageable number. And normally in like the real world, you'd have to pay more for that. In this case, we were able to kind of make it so that we've architected models to where you pay less and you can still get a a smaller tolerance window. That's that's a smaller plus or minus that you add to every number that you do you analyze and output.

Hermes (45:38)
So you're paying less, getting a better product. It feels like a no brainer on that.

Neal (45:44)
Yeah, just the generalizability. I will be upfront about that. Like if you try to make it do anything outside of those like those like sentiment analyses or data extractions or summarizations, like it's not gonna work. So it's kind of a pick and choose. And thankfully, like there are agents that you can build that have certain steps that you can use this like more deterministic. I don't I don't ever say that our models are deterministic. Like we still do have variants, right? Because of the pre-fill architecture and we do have our own neural nets. Like there is

a tolerance that we add, it's not like zero, but it is much closer to zero than the 10% to 15% a a large generative decoder model will get you.

Hermes (46:25)
Makes sense. So speaking of compression categorization, this whole post processing. Let's address the elephant in the room. Yeah, that's it. His name is JEV.

Neal (46:38)
My name is JEV.

Hermes (46:42)
I'd love to hear how does ScaleDown compare to something like a JEV? And obviously you guys have been doing it

for quite some time now, you have a lot of experience and an open source thing like JEV is not gonna like out compete you. But how do you see something like that in the industry?

Neal (46:59)
Yeah. I think like our thesis has always been that the future is multimodal. Like we've seen this from the team at Open Router, Alex A Atala has talked about this a lot. Like people at Fireworks have talked about this, Space 10, like all of these neo clouds and kind of like early movers on like the AI model races have have been very adamant about the future is multimodal. And we are a hundred percent in alignment with that.

And so we welcome like Typespace and JEV to this kind of like paradigm of task-specific models. I think the way that we think about classification is a little bit different than the way that they think about classification. Like the products are very similar, etc., but like how to actually use them in production is where we vary. So there's a couple of differences between us and JEV. And so I'll I'll take a step back for the folks that don't know about TypeSpace and JEV, first get out from under a rock. But two,

What what JEV and TypeSafe are doing is building kind of this this new model called a decision model, right? And it's basically a fancy way to say like kind of what I've talked about with dropping the generative decoder, thinking through calibrated decisions, and kind of like building YES, NO or classification outputs, right? Yeah. So JEV is really good at binary classification. So YES, NOs, they them nulls

as well as kind of like choice. So you give it like 5 options, 3 to 5 to 10 options, and then it will select kind of where your input is on that kind of spectrum or scoring. We have done both of those things. And I think they're doing a really good job at that as well. Where we differ are are twofold. One is modality. So originally we were text only as well, and and JEV is text-only to date, but we also

with customers kind of working with our design partners and our customers, we were able to build image capabilities into our models. So you can throw in, let's say, a bunch of tweets and a lot like a majority of tweets come with an image or a video or some some mechanism like that. So we're able to not only use the the text, but also the image to do the classification of sentiment analysis or what type of of kind of output you want there. So that's one is modality.

We're also working on voice. That's something that we'll be rolling out soon. So we'll be kind of multimodal in in the terms of text, image, and voice. Because we've been we've been working with some awesome voice companies. So that's one. I think modality is the is the big one. So like if you're doing text-only workflows, it it is kind of a choice between ScaleDown and JEV. And then the like if you're doing image, kind of we're the only model in the market that that does image or voice classification and extraction. and

Extraction in particular is important because like financial documents, PDFs, invoices, all that stuff require image capabilities. So that's something that we've been working really hard on to roll out image, and that's been released, so you can go check that out. The other big one is multi-label. Okay. So I'll take a step back and kind of explain the like three pillars of classification since like classification is very important right now. I gotcha.

Hermes (50:18)
I don't know man. You gotta you gotta explain the stuff.

Neal (50:21)
Gotcha.

So binary is like YES/NO or TRUE/FALSE or like those types of like pick one of two options. The the kind of next version of that is multi-class. So multi-class is basically when you have five different options and you're picking one of five or one of three or one of twelve, right? That's multi-class where you have multiple classifications.

Hermes (50:44)
Is it this, this, this, this, the like the giant IF

Neal (50:46)
If

statement. Yeah, exactly. It's an IF statement, but you're picking one out of that if

Hermes (50:50)
IF/OR. Or IF/ELSE. Yeah. It's a giant IF/ELSE. Okay.

Neal (50:53)
IF/ELSE. And then the the one that we've been working really hard on because like with legal and finance companies, they really need and like post call QA, like that's a that's an awesome example of this, where you need multi label. The other easy way to think about this is tagging, right? Where you have 15 different options or a 100 different options for what a call could be classified as

or what

Hermes (51:17)
It's not just clearly

Neal (51:18)
came.

Hermes (51:19)
this or this. It's

Neal (51:21)
It's multi-label, right? So you're adding three, five, ten, fifteen, a hundred different tags onto a document or onto a call transcript or onto any sort of input context, whether it's voice, image, or text. So that is something that we've been able to unlock that a lot of the other competitors haven't yet. And so that's like where the differentiation comes is both modality outside of just text and also this idea of like

doing multi label, not just binary and multi-class, where a. But

Hermes (51:57)
Still supporting those

Neal (51:58)
Yeah, we still do binary, we still do multi class, we also do multi label, which is like the the kind of the most complicated of the the three options of classification.

Hermes (52:07)
That makes sense for for certain industries, like they don't differentiate between text, they don't differentiate between images or voice, like they just have all this data that they have to collect, they have to store it. Yep. And now you don't need humans to sit there and label and go through and you you can actually like act upon it in almost real time, right?

Neal (52:28)
Exactly.

Yeah. And I think that's where like so those are just like mechanisms and use case type of like par parameters. Cost and latency are the other two. So like obviously accuracy, like we were very similar, if not better in some benchmarks, they're better in some benchmarks. It's like that depends on your use case. The the like important thing outside of those like most important ones on accuracy and and modality is around cost and latency. So

our cost is very similar, around five cents, four cents for a million input tokens. They also are doing the the no cost of output tokens similar to us, which is awesome. Like we want to see more of that. So super bullish on on like output tokens kind of being phased out for these more simple workflows that that both of us are doing. And then the other one is around like latency, right? And and I think everyone's really excited about how fast JEV and ScaleDown are.

For a lot of these workflows, because the way to think about it is like it's a different paradigm of models, right? Like, and I I tell this to all of our customers and people that we talk to is like you use an LLM when it makes sense to use an LLM. You use a decision model or a task-specific small language model when it makes sense to use that, right? They're not, they're they're mutually exclusive, right? They're not interchangeable, they're not and like.

That's kind of the education that we were kind of doing with our folks, and like the people we work with is like there are instances in which you're using an LLM and you're using like a Haiku or a GPT 5.4 nano or or like a "small" general purpose model, small in quotes because they're still very large. But be they still have that generative decoder, right? And so like you're using them in instances where you shouldn't have a generative decoder, you shouldn't have a small model in there at all.

Because it's a type of decision or a type of task that it doesn't make sense to have an LLM. And so I think TypeSafe has done a really good job of kind of educating people and showing the world that there is a different type of model category that's kind of being unlocked right now with us, them, Fastino and a couple other companies as well, that are doing a really good job at kind of like adding on to LLMs where LLMs are currently failing.

Hermes (54:49)
Yeah. I mean, and we need it. Like to get this space moving forward, we can't just be running large language models and su as the sub-agents. Like you need things that are closer to deterministic and closer to zero tolerance.

Neal (55:01)
Yeah. And I think the the two really, really interesting places that kind of like propel the the AI industry and development way faster and further is model routing and evals. So I'll I'll kind of double click on on some of the interesting use cases that we're seeing and that people have been kind of publishing is around model routing and around eval. So model routing you can think about as like the top of the sphere, the tip of the sphere, where

you are doing something with an agent or a workflow and you're trying to figure out what model should I actually use for this particular task, right? Open Router, Martian, a lot of these companies have kind of come to to be more not so diamond that have done these really cool jobs with routing the complex tasks to bigger models and the simple tasks to smaller, simple models, right? And that's a really good use case for a ScaleDown classify or a JEV or some one of these like kind of decision

or classification models because it's very fast, right? Like you don't want your agent spending turns and spending a bunch of a milliseconds kind of waiting to figure out what model should I even use to start the process. You want to just like quick in a couple milliseconds move to the right model and get that model going.

Hermes (56:16)
And that's kinda like that IF/ELSE statement, like it's either this, this or this.

Send

it in that direction.

Neal (56:21)
Exactly. So it's not binary. It can be if you're only using two models, but I highly recommend using more than one model more than two models. But yeah, so you you basically are using that like multi label or multi-class classification to kind of route to the right model out of a plethora of different options. So model routing is one really interesting use case where you can get a lot of the benefits of kind of the the model task fit

That I was talking about before, picking the right model for the right task. That's a really big way to like capitalize and make AI much more efficient and kind of like work in in the places that smaller general purpose models like Haiku are better at versus bigger, more like powerful reasoning models are good at. I think that's like one, and that's the the tip of the spear and kind of the start of the agentic workflow. And then similar to like voice agents and call transcripts and post processing.

There's these things called evals. Evals are like you want to be as deterministic, just like I mentioned with like the tolerance analogy. where if you're doing evals against kind of your traces and and what your agent is outputting, you want that to be as near determinism as possible. And also there's a lot of scale there. So you want it to be relatively cheap and effective. And then also you don't want it to take forever. Latency is less important with evals, but it is something that's interesting where

if you're trying to do real-time evals, that's like generally important as well. But mostly it's about accuracy, consistency, and reducing variability and variance. So a lot of people have been using ScaleDown classify and JEV and all these classification models to do instead of LLM as a judge, using classification as a judge, right? Because effectively what's happening when you're doing evals is how many times was this output

matching the golden data set or how many times was this output accurate. And so if you have a classification model that's running that, it'll be way faster, way cheaper, and much more consistent.

Hermes (58:23)
Makes sense. It's it's one of those areas where like it just touches upon the whole agentic flow. And I don't think people are even thinking in this direction yet. Like, yeah, JEV has woken up the community to this, but unless you're in one of these big businesses or enterprises that's trying to build an agent and run into this, like

I feel like some founders aren't even thinking of this piece of it yet when it's gonna be like a major pitfall for them five steps down the road.

Neal (58:54)
A hundred percent. we Yeah, we talk with a lot of folks that are still kind of focused on like product and priorities and roadmaps and those types of things. And I think like the best founders that I talk to understand that like having the the true foundations in place of using the right model for your agent in the right place, is really important. And I think a lot of what we've seen is like

quick iteration and and kind of like testing out things quickly to understand if it's worth your team's time are really important kind of attributes for a high hyper growth or fast growing company. Whether it's an enterprise or or a leaner startup, it's like how do we think through the the right decisions that are like lower priority but easy to implement

or higher priority and harder to implement and kind of balance those accordingly. So I think there's a lot of value to be had in like adding classification within agent loops to make sure that the agent, the right agent is being activated or you're activating the right like step in the agentic workflow and using the right model for these really high volume workflows because it's literally one line or like two line API switch to go from like a Haiku to a ScaleDown model.

Right. It's like super simple integration time. It's more the actual like testing and evals and benchmarks to make sure that it's like performant. Right. And one of the things that we've been working on that's that's really exciting is the ScaleDown agent. And what that is is basically an integration agent that basically plugs into your local Claude Code and will not only look at all of your AI API calls across your agents and workflows to understand like

which of like based off the input and output and maybe some traces, like what is the actual model that you should be using for all the steps of your workflow? Like it's very clearly will say, like, okay, for these reasoning steps, like stay using these models. For this one where you're doing high volumes of classification, you should test out the classify model. If you're doing some summarization at the end before it goes to the user, compaction, et cetera. It'll kind of give you a a markdown report of like you have these 15 different calls, 7 of them you should use like big frontier models.

3 of them you should use Haiku, and then 3 of them you should use ScaleDown, right? And so like it'll help you understand kind of what steps you should consider like exploring task-specific small language models. And then most importantly, it'll help you do the prompt tuning. Like I alluded to at the beginning, our system prompts are pretty different from the general purpose free text system prompts that general purpose models have. And so there is kind of that that transition of changing a system prompt from that long like guardrail full,

instructionful system prompt to something that's much more simple and just like defined with the examples that you need to kind of like overcome the lack of general generalization. And so the SLM agent, the ScaleDown SLM agent will help you do a lot of that testing and prompt tuning and evaluations when you when you go to go consider something like this.

Hermes (1:02:08)
That's super interesting. So it's like here's a tool that uses the technology to make recommendations on how to use the technology more efficiently within your own flow.

Neal (1:02:16)
And we're not we're not using ScaleDown models in the SLM agent. We're y'all we are using like Claude Code, it's an MCP server Claude Code plugin to do this locally so no data leaves your system. And so it's like we know that like LLMs are good at that. Like ScaleDown models are not good at assessing all of your inputs and outputs and traces and like schemas. But a model is good at that. Exactly, exactly, exactly. So you can use diet code and the SLM agent together.

Hermes (1:02:45)
That's super cool. Cool.

So you run your your agent flows, you run your voice agents, now you have not only in call, you have evals, like ScaleDown is kind of part of the conversation the whole way through. And it's it feels like it's that piece of it that you really need when you start running agents at scale anyway. Yeah. Right? Like whether it's

in the agent flow or afterwards, you need all these like key pieces and th those need to be deterministic. So that's today. And you mentioned, you know, you used to do forecasting and looking to the future. So let's put our our crystal ball out on the table and look into the future a little bit. Where do you think this kind of goes in the next like one to two years? We won't even go three to five. That's like

too far out. Like like like in the next 12 months, in the next 24 months, like what should founders be thinking about in this space?

Neal (1:03:48)
Yeah. So kind of I was alluding like I mean, three to five years back in the day is now twelve months, so like I'll kind of frame it as such. But I think the two things. One is this idea of like what changes make the most meaningful impact to your agent's performance is how founders should be thinking and how some of the best founders that we work with are thinking. Right. Like the ideal future state of the most

like ScaleDownified agent is using a ScaleDown classification model for routing, using an Opus or like whatever is existing in a year, the for the fables or the Astras, to do the orchestration. All the sub-agents are using either like the the small general purpose models or our fleet of ScaleDown models to do a lot of the summarization and classification and extraction. That's kind of happening behind the scenes. And this is all in the voice context, but it's like

model routing to the right orchestrator, the right orchestrator kicking off the right sub-agents. And then that's kind of how the like live workflow works. And then afterwards you have the post processing of the call transcript where you can use a lot of the ScaleDown models and larger models to do a lot of the synthesis. And then doing the evals on your like agent runs and your traces and your logs to understand like what is good, what is bad, and how can we make the agent better. And so this is kind of like

loop not within an agent, but like of building your agent and looking at the data and how it works to kind of continue to make it better. And so there's that iterative process. And so the best founders and the best enterprise teams are thinking through which of those steps actually makes us better? Is it the evals that makes us much better? And making evals more consistent with the classifier as as a judge rather than an LLM as a judge. Is that the first step we should take to explore this new technology?

Or is it the the post-processing of call transcripts and like actual analysis of human data? Or is it the model routing? Or is it the sub-agents, right? There's these like four big buckets of like evals, routing, sub-agents, and post-call processing that are like the four buckets where ScaleDown can really impact a a company in an agentic platform. And so just figuring out which one of those moves the needle the most, and then obviously kind of expanding and continuing to use that

in different places as you kind of hit the checklist of different things. I think that's like step number one, really. I don't know if that's gonna take a year. I think a lot of the companies that we're working with, as soon as they integrate ScaleDown in one place, quickly come back and they're like, we wanna use it for we used it for routing. Now we want to use it for evals. We used it for evals and routing, now we want to use it for sub-agents or vice versa. And they kind of they they change. But the main point to that is like companies and founders and

enterprises that know where they have the biggest gaps to close and where consistency and efficiency really move the needle is the most important. And maybe zooming out 18 months, 24 months. I would think through how the orchestrator and sub-agent architecture is going to change. Like I mentioned before, it was stacked prompts where you have one really long prompt with a bunch of different instructions.

That has now morphed into workflows. Workflows have now morphed into agents. Agents are now moving into orchestrator sub-agent. I have kind of foreseen, or what I think the future will look like is no longer like agents as the sub-agents, where you have like one orchestrator with a series of tools and a series of sub-agents that it can activate depending on what the the caller or the the voice kind of agent is doing. But

actually picking models and just letting the models do their job. Because we saw that with stacked prompts, where in a stacked prompt, it kind of morphed into just giving Opus the ability to cook. Like I did a lot of customer discovery last year, and like the amount of times I heard we threw away all of our guardrails and now we just let Opus cook was insane. And so the way that I think about it is like that's also gonna happen at the sub agent level.

You're gonna kind of like remove a lot of these steps and workflows and like actual agent loops that you've built and that you've manually painstakingly built, and you're gonna throw them all away and just let the small language models, whether they're the Haikus or the ScaleDown models, actually just do the job. Right. And that's why we're kind of focused more on like how do we bring in that generalizability and that like general intelligence to these task-specific models.

So that they can not only classify but understand: like, should this be a classification task, should this be a summarization task, and like what type of classification? Is it binary? Is it multi-class? Is it multi-label? And kind of let it decide what type of classification so that when the orchestrators no longer have as many guardrails and kind of like system prompt instructions, now that orchestrator can just pick the right model for the job rather than the right sub agent for the job.

And I think that's like the 18-month, 24-month future where we kind of don't think about agents as much and we just think about models and doing their thing. Just like right now, we think about Opus and just letting it do its thing.

Hermes (1:09:14)
Makes sense. And that's funny you say that because I've heard that also before. Like the the models are getting so much better that the prompt is almost like less important. Yeah, you kind of give it direction. Yeah. But like the more you constrain it, the less it can actually perform on its own because it's not this older class of model. This is like a new frontier model. It's exactly it's pushing pushing forward. So it's also interesting to hear like

do the same with the small language.

Neal (1:09:45)
Let

it cook. Yeah, they're both gonna keep getting better and more accurate. And and our our main challenge right now is how do we get that generalizability in while still keeping that accuracy, latency, and cost high for accuracy, low for the rest. And I think like the the pro tip for the people watching and and listening from home is like the the creator of Claude Code, Boris, was was talking about like every few months, as new models are released, delete all of your skills, delete all of your MD files.

Start from scratch and kind of reset your Claude Code and your cowork. Because when you have all of these skills that in the in your context that the the newer models are using, they are constrained, right? And that was like a really clear sign to me that if we're doing that for like our chat applications and our Claude Code instances, it's like, don't shouldn't we do that with our agents too? Like shouldn't we just like test out what happens if we just delete all those system prompts?

Hermes (1:10:43)
That's where you need the eval to make sure that it worked. Exactly.

Neal (1:10:46)
A consistent eval classify as an as a judge for sure. But yeah, that's the future that I see is like agents kinda get replaced for for better models instead of having all the the human garbage in there.

Hermes (1:11:01)
Don't let the human

Neal (1:11:04)
Impede the agent.

Hermes (1:11:05)
Yeah, I've heard it said. Yeah. Get get out, let the don't let the human impede the agent. Like don't let the human stand in the model's way. Just let it do its thing. Exactly.

Neal (1:11:15)
Exactly. Which is a scary thought, but I think it's it's it's the path to performant agents in the future.

Hermes (1:11:22)
Yeah, because those more generalized ones are actually gonna be able to do work versus w what we saw a few years ago where people are like, it's cool, but it's not there yet. Now it's like it's getting there. Mm-hmm.

Neal (1:11:33)
Yeah. 100%.

Hermes (1:11:36)
Neal, thank you so much for your time. I know you're busy. I'm gonna have one more question. It's kinda like my wild card that I like to ask all guests is if you weren't working in this specific part of AI in small language models, where do you think you'd see yourself?

Neal (1:11:53)
Great question. I think like this is funny. Some of my friends and I always talk about how like if we weren't in corporate America or if we weren't in the AI startup world, we would just work in like a blue-collar job and like work construction. And I think to that end, like I think there's an element of like working with your hands and like being on the ground and kind of having that like builder mindset.

So my answer is kind of like a data center technician. I think like back in the day when I would spend all my days in these large hyperscale data centers where I was like doing the forecasting and the the hardware like optimization and like all the stuff that kind of gave me the idea of ScaleDown. I spent a lot of time with the data center technicians. And they were awesome. Like they were super down to earth, solid to the grant, like they were just cool guys that like

were the the classic like high school like super like math whizzes that were like, I don't really want to work like as a mechanic, but I wanna work on like cool computers and like they built their own computers and their monitors and all this stuff. And I think that would be really cool to spend time like within the DCs, actually working on servers, maintaining the hardware, keeping everything up to date and like kind of net new. I mean like those those server trays are recycled and like updated every like

year to year and a half now and it's it's a lot of cool work. A lot of cool machinery, a lot of cool hardware. And I think obviously like cool companies that are doing like like etched and and stuff like that that are doing really cool hardware stuff for these trillion plus parameter models that are gonna be coming out in the in the next 18 to 24 months. I think that's also really cool. I unfortunately am not so good at soldering and and hardware stuff. So I don't know if I can be be building

new trays and DIMS and GPUs and stuff like that. So yeah, I think I would be in the data centers swapping out server racks and and trays and building new new new data centers.

Hermes (1:14:01)
Being one of the unsung heroes. Yeah, exactly. Those are the people you never really think about. Yeah, none of this would be possible without them. Yeah.

Neal (1:14:10)
100%. People boots on the ground, ESD boots. When you you the funny you have to wear these special electrostatic boots if you're in a data center so you don't get shocked. They're the ugliest shoes in the world, but they're pretty cool. So yeah, I'd love to love to wear those instead of the polo.

Hermes (1:14:30)
Thank you so much for joining us today and for talking to our audience. Thank you for everyone tuning in. Do that social media thing. Like, subscribe, reshare. We'll catch you on the next one. Thanks for joining us.

Neal (1:14:43)
Thank you for having me. I appreciate it. Cool.

Hermes (1:14:46)
Cool.

Neal (1:14:48)
Yeah.