AI Is Worth 5% of Its Hype? AGI Myths, Machine Bias & Predictive AI’s Untapped Value

Intro:
Hello everyone, and welcome to Secure Talk. I'm your host, Justin Beals. Here is a question for you. Do you want to solve problems at scale securely? Then you need predictive analytics, a risk score, a threshold, and a decision about when to pull a human into the loop. That is machine learning doing the unglamorous work our industry has relied on for years. And it is the security professional's native language. My guest today wrote the book on it.

Eric Siegel's The AI Playbook asks you to swap the word AI from machine learning almost every time you read it. Then strip away the science fiction, and you are left with something genuinely useful. Not a crystal ball, just better than guessing, which for security work is already enough to change your day. Now, what I loved about this conversation is where we landed. Predictive AI and generative AI are not competitors.

Put them together, and you get a real architecture. The generative model does the work. The predictive layer scores the risk and catches it when it is about to fail. That is what makes an agent reliable enough to deploy. And that is where resilient business value actually comes from. 

Eric Siegel is a former Columbia University professor who helps companies deploy machine learning. He is the co-founder and CEO of Gooder AI, founder of the Machine Learning Week conference series and author of the best-selling Predictive Analytics, which I have on my bookcase, and the AI Playbook. His work bridges the stubborn gap between the technology and the business that is supposed to use it. Join me today on Secure Talk for this conversation with Eric Siegel.
—-

Justin Beals:  Eric, thanks for joining us on Secure Talk today.

Eric Siegel:  My pleasure. Thanks for having me, Justin.

Justin Beals: Right on. I'm very excited about this conversation because we're gonna get to talk about something I have a love-hate relationship with lately, which is AI. I think all of us do. And I'll start off by mentioning your most recent book is called The AI Playbook. But inside it, you ask readers to really mentally swap the word AI for machine learning almost every time.

What made you keep the word AI in the title, though?

Eric Siegel: What you're referring to is that this is a book; The AI Playbook is about predictive analytics, right? There are two main kinds of AI: generative AI, which is mostly the use of large language models, and predictive AI. And you know, generative AI is new; predictive AI is older, but not old school. Most of its value is largely untapped. So, to answer your question, I first need to distinguish between the two because these days all the attention is going on to generative AI.

And when people say AI as of a few years ago, they're specifically referring to generative AI or the use of large language models in most contexts, right? Unless it's qualified depending, but in most contexts, that's sort of what the words become. Prior to that, it mostly referred to predictive AI, otherwise known as enterprise machine learning, also previously known as predictive analytics. 

And that's learning from data to predict the outcome for each individual: who's gonna click by, lie, or die, commit an act of fraud- any outcome or behavior for which there is value to the organization to drive individual, per-case decisions. So for targeting, marketing, fraud detection, credit scoring, all you know, whether to hold or investigate on potential security incidents, any network interaction on that level of detail. 

So, these days, so much on generative, but what I like to say is that most organizations , for most use cases and most functions, should  be investing at least as much in predictive AI as in generative AI. So AI, though, is the umbrella term; it was always the umbrella term. That's the word you need to use if you're writing a book that's meant to be accessible to newcomers and non-experts, as well as data scientists, because that's sort of the catch-all. So that's where we start. And then, we refer to it as machine learning for most of the book. More recently it's sort of being called predictive AI. We have a new preface to the book, the AI Playbook, in which I refer to it as predictive AI. And at the end of that new preface I say, okay, for the rest of the book, we're gonna call it machine learning.

Justin Beals:  That's really good. Yeah, I was watching a podcast this week with Corey Doctorow on it, the computer scientist and researcher, famous for creating the word in shittification. And he had mentioned that really what we call AI is just the next version of what people have been calling AI since, like, the 40s, as we were starting to even think about this as a concept. And it's almost a theme more than it talks about the real computer science innovation. And I agree, to me, the computer science innovation has been machine learning. Training an algorithm against a data set to predict an outcome on the other side.

Eric Siegel: Yeah, and that's what a large language model also is doing, right? At the per-word or per-token level. So instead of the enterprise element or unit, like a customer or a place you might drill for oil or a satellite that might run out of battery or a transaction that might be fraudulent, it's as you're creating text, what should the next word be? Right. And both of these areas, generative AI and predictive AI, are basically use cases or categories of use cases of machine learning. So both machine learning- the difference between the two is the type of project; the type, you know, predictive is what you turn to to improve your existing large-scale operations. But yes, I agree with that. And I actually heard Corey say that, maybe we will both listen to the same podcast interview, that hey, look, this term has been  sort of redefined and repurposed in so many different ways. I mean, it's always referred to; it's always alluded to a science fiction concept, which is somewhat problematic. 

The word intelligence is very much subjective. But these days, if it's just being used to refer to a large language model, which certainly is seemingly human-like in a way that we've never seen before, and does have a very particular set of use cases, some very clear-cut and clearly valuable, other ones a little bit more subjected to hype and overpromising, the whole agentic stuff to some degree. But look, if the word now is really just referring to that concretely, great, right? I mean, a rose by any other name. I mean, we call it a smartphone without necessarily being grandiose about what the thing is capable of. So there's it's not like there needs to be a religious outlawing of the word intelligence in the case in the context of technology.

It's just that it is gonna tend to continually, it's like the it's like the mafia. Every time you get out, they pull you back in. That word intelligence, and therefore the term AI, is always gonna be pulling the hype masters out there into that sci-fi narrative. And that's what AGI really is capturing the spirit of the notion of artificial general intelligence, the idea that we're replicating people, that we're creating virtual people. And that's where I think we cross the line into hype.

Justin Beals:  Yeah. I'm, you know, before we move on to some of the things that have happened around AGI here very recently, one of the things that I find so often when I talk to people and they mention AI is that they're talking about the user interface for predictive analytics. Yeah, you know, how they interact with it, like they're looking for a chatbot or something along those lines.

Eric Siegel:  Right. Well, to be clear, so a chatbot would be use of a large language model. That would be generative AI, right? It's generating language. Whereas predictive AI, well, you might have a chatbot slapped on the top. There's a lot of ways these these two areas of AI can complement and help one another, actually. perhaps the most significant is the other way around where predictive can help generative. the so the problem with the term agentic, just like the term intelligence or AI, is that again, it's gonna keep pulling us into overzealous narratives because what does it mean? It means we're granting the machine agency, right? We're talking about replacing roles rather than just help being a tool that helps with tasks, right? and that tends to be overpromising. But putting the semantics aside and even the religious d debates aside, I think there's something that everyone can agree with, whether you're extraordinarily bullish and I would consider a hype person, or you're super skeptical. 

The fact is, there's value to automation. that's why we build machines. And there's always gonna be a tendency to want the thing to take on a bigger role. Like, it could be do doing customer service agent-type interaction. It can answer questions about a million different topics. Can it also conduct a transaction? When you start asking questions like that, this you get this mental scope creep. We're gonna naturally our imagination is gonna go a little bit further about what our ambitions are for the project and the scope of what this so-called agent is meant to do, and very quickly get to the point where it's just going to fail too often, even if it's only failing four to six percent of the time, to be viable, won't get deployed at all. But if we can put a predictive layer to serve as reliability and therefore intervene with that more expensive human in the loop at some minority of the more highly risky cases, you're putting a risk score. 

That's, you know, in general, you can think of many or probably most use cases of predictive AI as essentially some kind of a risk management, right? Should I, if I risk $2 contacting this person by including them in my targeted marketing, am  I'm risking that they'll lose the two dollars because they'll ignore it and they'll consider it spam or you know junk mail. If I don't do anything, this subscriber might cancel, right? So, you know, or or if if I allow this network interaction, it may turn out to be malicious, right? So it turns out that you can really frame a lot of these use cases, if probably most of them, in terms of predicting a negative outcome in order to determine how to potentially intervene and not allow it to go unmitigated.

Justin Beals: Yeah. Let's talk a little bit about that multi-model scenario. So I think that you and I have worked with a lot of different types of modeling techniques, generative AI or neural nets, as they're you know, probably more commonly known, being things that write a paragraph, document for us, or produce an image. But I mean II've built models out of Bayesian statistics that were just quite simple algorithms or random forest is more complex. And each modeling technique is different and they're used as different tools. But we have this deep obsession with one type right now, you know, tensors, and we pretty much maximize their impact. And even us in our work, we're starting to say we need a model of this type here, checking this model over here for what it recommended. And as we build these things, I guess we think of them as automation or workflow, but we're building machines to check the machines,  because each machine is probabilistic. We're trying to drive down the probability of a false positive, false negative. Yeah.

Eric Siegel:  Yeah, exactly. in fact, you bring up false negatives and false positives, and that's sort of very concretely the kinds of things that can go wrong. The problem is that when people conceive of predictive AI projects, they often stop with that type of metric, that type of benchmark, as far as the performance and how well it predicts the pure predictive performance,


which in and of itself is arcane, right? It tells you it predicts better than guessing, which generally the most we can hope for. We don't have a magic crystal ball. Predicting better than guessing is generally more than sufficient to improve bottom lines of of large scale operational performance. But what we really care about is the business value of it, which for many businesses would be a monetary performance, the profit or the amount saved for avoiding fraud or avoiding security incidents or what have you. So put a dollar sign on it.

Take those false positive, false negative rates or take whatever technical metrics you're using and convert them, which actually is the focus of the startup that I co-founded, which is called Gooder AI. So it's for predictive AI projects and putting that business value on the potential model and how it might or could be deployed.

Justin Beals: One of the things I loved about your book is you especially the biz/bizml side as you describe it. You know, you're talking about machine learning, these computer science areas, even AI as we want to describe it from a business perspective. I think you tell tell help us describe how that kind of capturing the business side, not the computer science side, helps drive the success of an AI initiative at an organization.

Eric Siegel: Yeah. Of a predictive AI. Now there are similar themes. Both types of projects tend to fail to get to operationalization. You make a great pilot, you love the technology, you go, wow, this technology is awesome, but we don't get to wow number two, which is it's not just great on paper, but it's actually gonna improve performance of operations. And there's a similar phenomenon with both predictive AI and generative. Let's focus on predictive for a second to answer your question.

BizML is the titular playbook of my book, The AI Playbook, The Framework, The Paradigm, the Business Practice for Running a Predictive AI project end-to-end successfully from inception to deployment. So let's get these things actually operationalized where most models, machine learning models generated with the intention of being deployed, actually fail to achieve deployment. BizMl.com is the website, so I'll just promotionally mention.

If you go to the website for the book, bizml.com, we're about to have the paperback drop in late October. So you can get the audiobook for free if you pre-order the  paperback. So bizml is a six-step practice, which basically tacks on three pre-production steps before the post the sort of production tech technical steps that are intrinsic to any machine learning project, which is prep the data, train the model, learn from that data, and then deploy operationalized, at least hopefully. And pre-production has to do with getting everyone on the same page as far as what's predicted and defining these things. What's predicted, how well, what are the metrics- not just pure predictive performance arcane technical metrics, but those business metrics as well.

And how well is it predicted, and then what's done about it, and that's the operationalization, you know. If this is potentially a security incident, triage it in one way or one of a few ways or what whatever the deployment options are. So that is to say, bring that process, put it on the table, make it explicit, and engender and facilitate deep collaboration between the tech and bus side.

And that's the only antidote to this organizational dilemma that we still have. And thematically it's very similar with generative AI. In general, these pilots knock our socks off. The ability to learn from data, to to to model from d to create a model, to automatically derive generalizations that hold over new unforeseen cases, the ability for a large large language model to create a

A demo that knocks our socks off that operates really seemingly well on at least a handful of and sort of anecdotal examples and gets everybody excited. That's the first wow. But to get to the second wow, where it's not just great on paper, but it actually achieves value. It actually makes the world a better place, makes operations run more effectively, efficiently, im delivers an improvement in terms of business metrics, right? 

That second wow has largely to do with getting on the same page, speaking the same language, collaborating between tech and biz, the business side, the person in charge of the operations meant to be improved or expanded on with this core technology, so that they understand with a semi-technical, but concrete, accessible, interesting, pertinent way, exactly what's meant to be done so they can buy into it. And that includes the metrics, the benchmarks, right? Don't just stick with, you know, in the case of a predictive model.

The problem is, hey, we we stop with these technical metrics like you said, false positive and false negative. Those are the two main com or two of the four components, of what's called a confusion matrix, right? A really friendly name. Precision, recall, F-score, even accuracy, is only technical metrics. And these are never what the business actually cares about. Let's also estimate the monetary impact or whatever the KPI or business metric that matters to the organization.

In the case of generative AI, it's a very much analogous problem. We're hung up on these benchmarks that are generic. You know, how well does it do on these public test beds or on an aptitude test meant for people like the SAT or the LSATs? and that's definitely interesting. Interesting is kind of a bad word, right? It's an academic word. Interesting is the utmost compliment when you pass a colleague in the hallway at an academic research institution.

But in the real world, we don't want just the first wow, the interesting. The thing that really matters is the value, the deployment. It's like we can't just fetish and get so excited about this technology. That's ironic. It's like being more excited about the rocket science than the launch of the rocket. So in the case of a large language base mate system, if you want to call it agentic or what have you, the measure that matters is how well it does at the problem at hand, not how well the language model does at these sort of generic test beds.

Justin Beals: Yeah. It's kind of like building a faster computer but not knowing what software to run on it. You know. What was the point? Yeah.

Eric Siegel: Yeah, yeah, kind of. Well, I mean, that's an inch so what you bring up there, it brings up almost a philosophical question, which is a very applied philosophy question, which is sort of what is it what's the value of general purpose technology, right? Obviously the idea of improving computers, which are the ultimate general-purpose machines, right? They can do any algorithm, or that's the thesis that most computer scientists buy in on, Right?  So there's so general-purpose. So improving the technology is good. But in the case of something like AI, which is also obviously very general purpose, if it's being sold as solution and we're supposed to have a strategy around an AI, that's like having a strategy around Excel. It's solutionism, like if to a hammer the whole world's a nail. So how do you strike this balance between fostering and cultivating the value of something that's really general purpose?

But not just sort of buying it because it's supposed to be valuable and then we'll figure out what the value might be or what problem it might solve later.

Justin Beals: Yeah, exactly. And it feels like in our industry, I I'll just call it tech generally, that we're in a struggle to use the right words to talk about what's available and searching for the right business applications. And I think, you know, this fall, just recently, OpenAI released a model and they framed it as the arrival of the AGI era. They've been tossing this around. Jensen Wang agreed that AGI arrived. And, you know, you've written that ML gets misconstrued as intelligent when it's not. What's your reaction when, you know, a company valued in the trillions of dollars announces this, you know, AGI intelligence era that carries this sci-fi affect to it? 

Eric Siegel Well, AGI is the epitome of hype, right? It's a virtual human, you know, you should if they're able to do everything that a person can do or everything important that a person can do, however, you might qualify it a bit, but basically it basically just means virtual human. You could onboard it the same as a new human employee and just let it rip. Or let her rip, let him rip. Whatever it is, we might, as you know, might as well, if we're treating it like a person, let's refer to it as a person, but I don't think we should.

Now, saying that ML is when I said that ML is construed as intelligent when it's not, that's true. That I don't mean I said that before Gen AI. in the context of what it can do now, I'm not religiously against the word intelligent, as long as you sort of don't get too hyperbolic, right? I think the difference between what it can do and what people can do is gonna become increasingly well defined. It's very hard to put your finger on sometimes, in a clear, formal way that you can follow through on and that's credible. But I think there is a difference. I don't think that we're replacing people with machines, and for good reason. And I don't see a risk of that happening. I don't think there'll be a job apocalypse.

But I also think that even if it's even in cases where objectively almost everybody would agree, hey, this thing's not intelligent, like in predictive AI, right? Where I mean predictive AI, we don't have a magic crystal ball. Predicting better than guessing is the best we can hope for. And in fact, the next best thing to a crystal ball is probabilities, a number between zero and a hundred of what's the outcome is for this particular case. That turns out to be really valuable, but the precision of those predictions, right? has very much diminishing returns because we're not going to get to a magic crystal ball, no matter how sophisticated the technology. So even if you use a whole large language model to help improve a predictive AI project, It may make a little dent in how well it predicts that could turn out to be worthwhile in terms of value depending on the project, but it's still going to be far from a crystal ball for most of the things that are important. Do you want to predict, you know, which satellite's battery is going to run out, who's going to click by lie, die outcome or behaviors that pertain to complex systems or to people, right? That's just the nature; uncertainty is not going away. So with those types of projects, like the word intelligence probably shouldn't pertain, but people tended to use it even then. Now, with the thing that's so seemingly human-like and unprecedented and dumbfounding in its capabilities, at least relative to what tech had been until a few years ago, of course, people are gonna use the word intelligence.

But going back to your original question about AGI, you know, you started out this conversation by saying you have a love-hate relationship with AI. I wonder how much of that is from the hate part? Is it the hype or is it the way in which AI is being used? Or is it both?

Justin Beals: Yeah, I think I think the my primary concern is the hype, you know, and probably what I think is challenging from a business perspective, and one of the reasons your book really resonated with me, I think that getting value it's fun to build these tools and imagine what you can do with them. That that's enjoyable, right?. As a tinkerer, as someone that's interested in building things, that that's that's fun to do. But I also want those inventions to be sustainable. And that means they have to make more money than they cost. And when it feels like these businesses are being built- not yet businesses, not even able to find profit in a unit of cost execution.

And the hype engine is being built on venture capital subsidizing your ability to consume these tools. I think we're setting ourselves up for a resilience nightmare as businesses that are reliant on them and perhaps deciding to cut staff and then being in a situation where they're finding they can't afford it. And I'll turn that back to you, Eric. You must have gotten calls as as a as a service in your startup, as an agency, as a consultant, to be like, how are we being told to do this AI thing? But it doesn't seem to be moving the needle.

Eric Siegel: Yeah. I mean, for sure. so I mean, yes, it's definitely setting us up, right? We're in a bubble, presumably something of an economic bubble, and also just sort of a tech bubble in terms of ideation and ideology. It's gotta be setting us up for disillusionment. There's gotta be a place where the stuff hits the fan, the executives get wise, it's not gonna do what they thought it was gonna do.

But it's still valuable. So how valuable is it compared to the valuations implied by these investments? I think it's like five, maybe up to ten, probably closer to five per cent. And if it is, that's extremely valuable, but that's a lot of disillusionment. It's really, really hard.  I'm not an economist, so as far as what does that what will that dis how will that disillusionment creep in, how quickly and how we'll have what kind of economic effects it'll have. That's really hard to predict. That's why I like predictive AI, by the way. It's different than forecasting. You're trying to predict whether the economy is going to go up or down or how many ice cream cones you're going sell; you're going to generally be wrong, right? Who's going to win the presidential election? That's why Nate Silver got some unfair backlash. He put probabilities on it, but everyone took 80% as meaning confident. But 80% or 70%, you know, is closer to it's closer to 50-50 toss-up than a definite Hillary Clinton's gonna win kind of thing. So I don't have to deal with that when I do predictive AI projects because nobody does, because we understand it's not a magic crystal ball. We're playing the odds, we're tipping the balance in this numbers game, and we have the law of large numbers in our on our side. 

So we're no one's gonna get crucified for an individual wrong prediction, but rather overall it's gonna predict wrong less often than, you know, the legacy system that it's replacing. so with that, I will definitely refuse to make predictions about the economic outfall of of today's hype. So you say, so I find the hate the hype I find it hateful. This is just a personal comment. I find it hateful in my heart, but I would never say that out loud because part of me knows that really the only thing that really is about it for me is it's just really annoying because I just don't want people to be misled. You know, I want the truth out there.

Justin Beals: I think it makes it hard  for people to find the real value because they're so interested in the science fiction. To your point, I'd love to do more predictive analytics. I've also seen more reliable business value derived out of them. and I think they overlay in a generative environment quite nicely, actually working together. But a lot of budget, you know, if you're talking to a a business and budget is going to an AI initiative for a rollout of a large language model, I mean, I think I get frustrated with the waste. And probably I'm just jealous. I want them to buy a more fruitful solution from me or someone else at the end of the day. Yeah.


Eric Siegel: I mean, but there's a time and place for large language models for sure. I mean, there's lots of viable applications. They're really apples and oranges. They solve very different problems. They should, in principle, compete no more than a camera and a phone. And like the smartphone, right? The combination of these is more than the sum of their parts, it turns out, although that's really emerging only in its infancy, those types of hybrid approaches. But it's definitely happening. I believe it's totally inevitable.

They shouldn't compete, but they totally do, right? For attention and budgets. And these days, generative AI is definitely winning as far as venture investments and media attention. But I don't think that it's winning in terms of achieved value. And which one's gonna be more valuable for the enterprise? You know, that's definitely an impossible question to forecast. But one thing's for sure: to the degree that Gender of AI wipes out predictive AI budgets, that's a hype problem. And it looks like it's not really happening. Enterprises are more quiet about it, but they're doing predictive AI.

Justin Beals:  I agree. I think I probably use a strong language there, but it certainly it it's been frustrating, of course. And I think that it's soaked up a lot of attention and resources and not necessarily delivered. And I want, yeah, it does deliver a lot of value. Don't get me wrong. We use generative AI to really increase our productivity. It's been brilliant when well applied. And even inside of our own engineering of new features, outlining a more generative model that we want to put alongside a predictive checker or a predictive checker that's driving a generative model to develop something because the prediction had a probability that this would be the most successful tactic. You're right, those were better-together strategies, but it asked us as an enterprise to build them. Not to buy them off the shelf. Yeah.

Eric Siegel:  Yeah, exactly. I mean, typical usage of a chatbot or large language model-based product has you squarely in the loop. You're the human in the loop, right? And that's great. And there's so many ways in which that is more and more myself and everybody I know in my professional and personal life finds that valuable. 
But that's not the story that we're being sold, right? The story we're being told sold is one of autonomy, right? AGI is, by definition the epit the ultimate autonomous story. Because you don't need a human in the loop. It can do anything a human can do. So why would you need a human in the loop if it really meets that definition?

So, you know, we- we- so yeah, sorry. So I was distinguishing between sort of our experience with it being so impressive and often valuable to use as sort of a thought partner or ask questions and find out how do I use this piece of software, right? It seems to know so much about everything and can assimilate from so many different places. And if you're the human in the loop, if and when it does make a mistake, that's okay. You're there, right? It's not an untrained dog off the leash. But the promise is these systems that are going to be super autonomous and taking on human roles. And that's where society is believing it's more valuable than it has potential to be.

Justin Beals: Absolutely. Also, though, there's a danger in that, right? You're blunt a little bit about ethics theater, you know, firms reciting that their AI is fair responsible without committing to it, you know, platitudes. How are you perceiving, you know, ethics, and what would you want to see from an ethics perspective in the development of these models?

Eric Siegel:  Yeah, I mean, like most data scientists, most of my career has been involved with predictive AI. Right. 32:22There's some noise outside, so I'm gonna just close this window. the right, I've been in machine learning for 35 years, right? But nobody's been in generative AI for more than four years, or at least barely anybody has, right? Because that's when it really came. So my ethics focus has been on predictive use cases, and I've written a bunch of op-eds about discriminatory models, and I've put these op eds in the Scientific American blog and San Francisco Chronicle all in one place. There's like a dozen of them. You can go to civilrightsdata.com, and if you if you want to access that list of op-eds I've I've published. But yeah, I do feel that, these days that type of stuff is getting very little attention. And just like predictive AI deserves as much investment as generative AI and actually requires a lot less investment to do projects, it's much more lightweight, right? Most of these things, you might have a million rows of data to learn from, but you can just do it on your laptop. It doesn't have the same footprint, right? It's sort of a more normal technology in that respect.

But just like it should receive more attention and investment in that way, it also needs to receive more attention in terms of the ethical ramification because that's how organizations potentially are making automate automatic or semi-automatic decisions about resources, like whether you get housing, whether you get approved for a loan or health insurance, and these kinds of things. And they're determined largely by those risk scores.

And the way it scores you does have potential to either inadvertently, or in the case of some data scientists, which is a little less fringe than I might help hope, may explicitly make base those decisions in part on protected class. So that's a big part of and focus on it.

Justin Beals:  Yeah. I've had projects where I had to take into account protected class, especially when I worked in education or hiring. And I was kind of I thought we were traipsing into very new territory, but I was surprised to see that the Equal Employment Opportunity Commission had some very specific performance requirements around models that were used in hiring and was really glad to see it. Like we had, we had some overfitting things we had to check for and accuracy, and those might be baseline, but at least there was something. what role

Eric Siegel: I mean, there's there's lots of places where there's laws against discrimination based on even on in part on protected class, and that does extend to a machine that's automating that. But there's also loopholes in places where it doesn't apply. For example, when an immigration officer makes a decision overseas in a consulate about somebody's application for a visa, right? they're allowed to use whatever they want. I actually kind of compiled a list of places where there's condoned, or at least known, explicit institutionalized prejudice based on protected classes in one of those articles. The San Francisco one, the San Francisco Chronicle ones at in my op-eds there.

Justin Beals: So I think what you're pointing out is that it's a challenging area. You know, it it's it there's a philosophical discussion around not harming protected classes, but at the same time, as you dig into the nuance, the details, you'll find that some institutions are designed around affecting certain groups of people and not others.

Eric Siegel:
Yeah, right. So even if, in the majority of projects, right, it's prohibited to explicitly consider any protected class, and in that sense the model, the machine learning model is colorblind, right? Because you've kind of withheld that column of data, what have you, there's ways th that other variables act as proxies, and it will sort of find that out, even if the data scientists themselves


has every liberal intention in the book. So I refer to that as discriminatory no, excuse me- as machine bias and adopting the name given by a  famous ProPublica article entitled Machine Bias, where it turned out that the suggestion to judges about sentence for sentencing and for parole boards for early release were and it was trying to predict recidivism, which is if you release this inmate, will they end up being jailed again in the future? And those predictions had a higher false positive rate for black inmates than for whites. So it did it was calibrated, it was calibrated in general, but not if you dig one level down and you go into the false positive rate. And so that turns out to be pr very alarming, although there are people who say, well, that's just the way it is, right? I see that as well; it's the way it is because it's a reflection, manifestation of historical injustices. So that's what I argue: I argue that that should also be taken just as just as seriously, even if it's not done intentionally or explicitly.

Justin Beals: I think that that is something that we don't think about enough when we're using machine learning or AI tools. They can be a little bit of a mirror. I once had a company that wanted to use predictive analytics in hiring. And they wanted to hire people that were more creative and curious. They said they were looking for a trait. And when we went in and looked at the data, we showed them actually how depending on there was a correlation between the number of years an employee had been in the business and the amount of curiosity or creativity they self-represent as. And that they were literally weeding that out in their own working culture. And even if they hired more of those folks, the fundamental change wasn't in a model that would predict who would be more curious, but in a way of working together that allowed more curiosity to flourish. Yeah.

Eric Siegel: Interesting. Yeah. Yeah. Well, that's something that's hard to measure, right? A curiosity index. Yeah.

Justin Beals:  Very hard, yes. And to dig into the details a little bit and be technical, we were utilizing a form of data science called psychometrics, which happens in education a lot. And it, though, has its own amount of error, right? You you you have to understand actually, I think in a lot of these things, I'm sure you would agree, kind of the probability of error or the validity of the measurement tool on the other side, because you can't assume it's just correct. Yeah. It's it certainly answers like it is. Yeah.

Eric Siegel:  Well, then you now you're bringing up a big issue about advanced tech in general, right? To what degree do people oversubscribe to, give too much faith to the machine because it's advanced, because it's complex, because it's reputable in some way, right? So you believe the result; you know, that that's what large language models are not notorious about: being confident even when they're wrong. Similarly, right, this idea that predictive AI projects aren't predicting with extremely high confidence for every case, depending on the project, right? If it's like, is this a picture of a cat or a dog, right? At this point it can do that very well. And even for certain medical diagnosis things, but in general, when trying to predict someone's behavior outcome, their health gonna go up or down, are they gonna click by a lie or die, right? We don't have this high confidence, but the judges potentially who see a recidivism risk score will give it too much confidence. So there's a s there's a level of education that's not that deep. It's not rocket science, but it's still largely omitted.

Justin Beals: Yeah. And we're all users of these tools. We need to understand, you know, how they work. I struggled so often; some concepts were hard and new, like probability. Yeah. How do you tell someone, well, this is sixty per cent probably right, and they it is very hard for them to translate it into real-world day-to-day. Yeah.


Eric Siegel: Well, you know what though? I think it's hard if somebody says it's sixty per cent probable. I think the hard thing there is to just let yourself recog it to shrug your shoulders. To be like, well, what this mostly means, there may be a little bit of leaning one way or the other, but mostly means we just don't know. And people tend to be uncomfortable with that. I don't think the discomfort is so much because it's a very technical thing; probability itself is not technical at all. It's the calculation of the probability, the rocket science, the actual machine learning, the capability to learn from millions of examples to generate a model which can then calculate those probabilities on a per-case basis. That's the rocket science. That can be complex, but using it, leveraging it, understanding the value of an individual probability, I think it's just like a little cheerleading, like, it's okay to not know, right? It's a don't blame Nate Silver because, you know, the quote underdog won the presidential election when they, you know, when they were given a thirty per cent chance of winning. Thirty per cent is far, far, far from a long shot.

Justin Beals: Yeah. That in itself, Eric, is some great science and truth, I think, as we work on this stuff. That's right. Eric, thank you so much for joining us today on Secure Talk. We really appreciate you sharing your expertise and the work you've done to make machine learning meaningful, you know, from a business context and an outcomes perspective.


Eric Siegel:  Great talking to you, Justin. Thanks for having me.