The ROI of Security Tested: What a new paper reveals about security value | Secure Talk with Minh Nguyen and Thi Tran

Justin Beals:

Hello everyone, and welcome to SecureTalk. I'm your host, Justin Beals. Before we get started today, just a couple of quick things. First off, everyone we got a chance to meet with at CS5, the San Diego CMC Conference, was a pleasure. Really glad to connect with our colleagues, friends, and customers at the conference. Thanks for joining us there. Also, the subscriber base for SecureTalk has been growing rapidly. And so thanks for all our recent subscribers.

You really make this work feel worthwhile. Thanks for listening. Back in 1998, when I was working as a network security engineer at Concert, helping British Telecom build out their global data network, my boss told me something I've never forgotten. He said, Justin, if we don't have a breach this year, they'll cut our budget. And if we do, they'll increase it. But that always struck me as backwards. And almost 30 years later, I still hear versions of that same frustration  from nearly every security leader I talk to. How do you justify cybersecurity investment when the best possible outcome, nothing happens, looks identical to nothing at all? CISOs are asked to defend budgets against CFOs who want hard numbers. They sit across from boards that want simple answers about return on investment for work that is fundamentally about preventing things that haven't even happened yet.

And the industry has responded with a lot of what I'd call folklore, vague claims about staying ahead of threats, generic frameworks and measurement tools that were never validated through real research. Now this pressure is only getting sharper. Gartner projects global cybersecurity spending will reach $240 billion in 2026, up more than 12 % from the year before.

And yet in a recent survey of more than 300 CISOs, over half said their organizations still aren't investing enough to match the risks they face. The boards have stopped accepting breach headlines that tool refresh cycles as justification. They want quantified risk, measurable outcomes, and defensible numbers. The era of, trust me, we need this is over. That's why I was excited to come across the paper from our guest today.

Min Nguyen and Ti Tran, along with their collaborators, did something I don't see often enough in our industry. They built a rigorous, academically validated measurement tool for what they call cybersecurity readiness, and then they tested whether it actually correlates with firm financial performance. The methodology itself is fascinating. They built a cybersecurity dictionary combining nouns and verbs,  then analyzed the earning conference calls of hundreds of thousands of company observations, separating what insiders at the company said from what outsiders said. The outsiders being the journalists, analysts, and investors asking questions. Then they ran 30 different models to test the robustness of their findings. What they found has real implications for how security leaders should think about budget conversation.

When outsiders mentioned cybersecurity on earnings calls, it correlated with measurable increases in return on assets the following year. And the size of that effect from outsiders was roughly 100 times greater than when insiders talked about it. That's because outsider attention is harder to manipulate. It's a more objective signal of what the market actually cares about. Now for a CISO trying to build a business case, this is

Genuinely useful finding the level of external scrutiny your company faces on security topics from analysts journalists customers and the public is Itself a leading indicator of what your investment level probably ought to be It gives you something concrete to point to beyond threat reports and compliance checklists Now we get into their research methodology What they're working on next and the practical question of how security leaders can actually use this work to make the case for their budgets. Now, Minh Nguyen is an assistant professor in the Department of Information Technology and Operations Management at the College of Business, Florida Atlantic University. His research sits at the intersection of information systems, AI and business, causal inference and machine learning. Minh holds a PhD in economics from the University of Hawaii at Manoa. Before joining FAU, he completed a two-year postdoctoral fellowship at Michigan State University's Broad College of Business, where he collaborated with researchers from Carnegie Mellon and Arizona State on causal machine learning projects using social media data. And Thi Tran is an assistant professor of management information systems at Binghamton University, specializing in cybersecurity teaching and research.

His work focuses on both the technical and behavioral sides of cybersecurity studying the complex human interactions with technology that attackers exploit. Thi holds a PhD in information technology, specialized in cybersecurity from the University of Texas at San Antonio. And his research has been awarded a National Science Foundation grant for handling misinformation harms during the COVID-19 pandemic. Before academia, he spent six years founding and managing nonprofit charity organizations, leading teams of up to 100 people which shapes the distinctly human lens he brings to the technical security problems. Please join me in welcoming Min and Thi to Secure Talk today.

—--

Justin Beals: Minh, Thi, thank you for joining us today on SecureTalk. We're really grateful to have you with us today.

Minh Nguyen:Well, thank you so much. Glad to be here.

Thi Tran: Thank you, Justin.

Justin Beals: Yeah.

Thi Tran: Yeah, I'm glad to be here.

Justin Beals: Excellent. Min, I think I'll start with you a little bit with a question. Your academic path runs through economics and mathematics in Vietnam through Erasmus, Rotterdam and Hawaii. You also worked on causal machine learning at Michigan State. You know, I'm curious about in your career, what made you realize the intersection of AI and business and cybersecurity was an area that you wanted to focus in.

Minh Nguyen: Okay, thanks for the question, Thu Thien. So at the beginning, I studied economics in Vietnam and also a little bit math, but math I studied after Econ. But I mostly focused on the behavioral side of there's something, anything about, so for example, like cybersecurity, there are two aspects, right? One is technical, one, and mostly people from engineering come to science, they focus on that.

The other one we call behavioral cybersecurity. So Dr. Tran and I, focus on that. So based on my background in economics and then during my PhD, I did a research interaction between the AI and business. So I use the method that's really suitable for behavioral cybersecurity. So that's why it's very easy for, it's very natural and easy for me to move to the behavioral or cybersecurity area. So the point here is that the different in terms of the topics, but the methods is remain like econometrics, more certainly and something like that. The experimental data, observational data, they still exist in many field in business include the cybersecurity.

Justin Beals: And then Thi, you spent a number of years as a cybersecurity researcher, six years I think, and working for nonprofits in Vietnam as well. How did your experience with human behavior help shape these concepts of very technical things like security and your work with people? Yeah.

Thi Tran: Thanks for a very interesting question. Think something like, all of those have prepared me very well so that I can take some research and some teaching about position and cybersecurity as well. actually the point is that cybersecurity is a kind of the social technical issue. And because of that, it is not purely technical or it is not purely social. And I did change my background a little bit during my master degree and my PhD as well.

So I obtained a lot of technical background, on top of that, I built on top of all of the social interactions and behavioral interaction and so on. And because of that, the point that the majority of the public has not realized, it is that cybersecurity is not purely technical, but it is something like the, even when it can sound a little bit technical or it can look like a technical issue, but actually the majority of the issue is that the bad guys, like the hackers, they set up the trap, and that they wait for people to fall into that one. And all of those are behavioral. So the mechanism are still the same as the way that we interact between people and people, but just the way that they play the game, is different. They use the technical ways as the way that they, the new way that they can play the game. But the majority of the mechanism of the game stay the same. So that is why I would say something like that is a good and a very smooth transition from something from behavioral and social impact into something about the technical as well.

Justin Beals: So Min and Thi, how did you guys meet? Were you at a similar university? Was it in Vietnam? Because I think the two of you combined are quite the superpower here.

Minh Nguyen: So I remember like we, so first we met online, on actually Facebook, the social media. So, and we interact for a while and we started to collaborate. So we work for a while before we met at the conference.

Justin Beals: Epic. Yeah.

Thi Tran: Yeah, I remember something similar to that. And I think something like we got connected through some social media channels, something like the Viet PhD. And then in that one, of the doctoral students, the professors and researchers, they joined together and all of the Vietnamese guys as well. And then we found out, hey, we have similar research interests. And that is why we reached out to each other. And then we finally met. actually before that, we already collaborated in some of the research projects.

And that is the reason why we already met online before. Then we met each other at another conference, and then we talked more about all of the future collaborations. That's why.

Justin Beals: So I came across you guys because I kind of monitor some of the academic journals around cybersecurity. And a paper that you wrote recently was really intriguing to me because in my work, we help companies get through security compliance outcomes oftentimes. everyone I've talked to, there's always a discussion about the return on investment, the value of doing cybersecurity work. It feels very esoteric.

A long time ago when I was at British Telecom, my boss told me, hey, Justin, if we don't have a breach this year, they'll cut our budget. And if we do, they'll increase it. And it always seemed very backwards, you know, the economics, the investment, the capital, and your paper around the effects of cybersecurity readiness on firm performance, on a company performance, was like really on the nose.

Were you guys responding to a similar problem that you were seeing and deciding to write this paper or investigate this space?

Thi Tran: I like when it starts up.

Minh Nguyen: So this one is, so actually we're thinking about to build a measure that reflects how firms are not only aware about the cybersecurity, but how they are ready to respond to that. so the most difficult thing is we need to build that measure, how to make sure that measure is objective, not subjective.

That's why you see, we think about a number of data set, database that we have. So we have Form 10k, we have the other financial report and we have conference calls. And the point we focus on conference call because, so in the structure of conference call, are two sections. So first, there is the presentation of the people from companies.

So, we call that one insiders and the other one we call outsiders, includes a journalist, analyst, investor or any other not come from the company. And by focus on the measure, the text inside and outside, and we can combine the total, it's allow us to have many ways to construct and to build that novel cybersecurity readiness one. So that's what important because we took a look

the literature before. And we found that there are some people mentioned about the cybersecurity readiness, but they just mentioned about that course without any measure. They just mentioned that without any formal construction to build that. And we build that one. And after that one, have the measure. We try to measure whether that one affect different type of firm performance. So the most difficult thing is the first one, build a measure make sure that the objective enough for the research.

Thi Tran: Let me clarify one point, something like, because you asked about the motivation, why we get that one, right? So I would say something like the motivation is not from the kind of specific observation or something, but it is, I think it is mainly from the intellectual curiosity. And we have some backgrounds about cybersecurity, we have some backgrounds about performance, why don't we try to combine the two? And then because we know about how the interactions can actually work.

When people see something, they can try to react to something. And the reaction can affect the firm's performance as well, including all of the stock market and including all of the way that people try to become the of the loyal customer or they hate some firms and then they boycott the firms and something. Then because about it triggers all of the curiosity so that we run all of the analysis over there. And then as Minh already said, so the majority of the literature right now, they already figure out about the hate.

Cybersecurity readiness is something that we should talk about, but there is no study before that they already statistically and systematically measure it. And that is the reason why we want to measure it. And we want to run some tests on top of that. So measurement is just only the first step, but we want to increase all of the robustness and then increase the reliability of all of those. So that is the reason why we keep expanding all of our work. And then the concept of cybersecurity readiness, it is like the upgraded version of the cybersecurity awareness. So you can aware about something, but what about the action? So are you ready to fight it? Are you willing to talk about that and a lot of other things.

Justin Beals: Yeah, I complain many times that in this industry it feels like we have a lot of what I call folklore, like legend. I appreciate the intellectual curiosity and developing a reliable measurement tool is a process, Like it requires and I think my next question will be a good research methodology, which is why we need academic research into these spaces.

And so let me start with a little bit on your research methodology. I think one of the things you began with, and this is near and dear to my heart because I do a lot of natural language processing work, is you developed a cybersecurity dictionary. Tell me how that came about. Maybe, Thi, you'll start with us.

Thi Tran: I would say that because I did some similar studies before, we captured the public reaction related to the COVID-19 pandemic because they are panicked during the crisis, right? So they start talking to each other and they raise all of the concerns. So for example, the concern about the financial outcome, or the quarantine and something, right? So we try to capture all of those and then we try to extract something that will define the concept and then we extract something.

And the thing that we focused on at that time, it is the concept of uncertainty. Because actually we are facing something that we are not know what, that is not very well known. And because of that, when we move to this particular project, I feel something like, hey, why don't we try to do the same? And that is the reason why we have some technical guys in the team, and they are willing to try to do something like that. So they try to define the back of keywords related to cybersecurity readiness.

They try to extract all of the words from the text that we already have, from the conference call and some other data sets. Then we try to mismatch, and then we see all of the ways that we can extract the scores. And the scores, at first, we only focus on something like only the nouns. And because of the nouns, so it is like, I mentioned about that, but what about some actionable words? So that is the reason why we expanded a little bit more, and then we included all of the list of verbs.

And then we combine the two together so that we can have a little bit of a contextual understanding. And then from that, we can have more confidence when we say that, if we have the consistent findings from both of the verb and the noun list, so it is a little bit more confident that those guys actually talk about cybersecurity readiness, not mentioning simply about, I think that cybersecurity is important. because that kind of statement is a little bit generic. But if they say something like, hey, we need to do this, we need to do that, so the combination of verb and noun, it is stronger, and it is more confident for us to try to claim about something like

Minh Nguyen: Let me explain a little bit the way that we build a dictionary. So to explain something, Dr. Tran mentioned that. first, we take a look on not only some main of the glossary, the dictionary online out there in the government website, some IT company website. So we took a look at their website and we pick up all of their important words that we believe that link to the research. And the other way we also look at the previous studies related to cybersecurity, and then we read their paper and then we pick up some of the words that we believe that link to our data sets. And then we put previous one from the website government one, so that the two thing.

And then after that, so that's the first version. And then, so after that, use LLMs to try to find all of the different versions of a work. For example, like managed. They can have, like, maybe managed in the present tense, in the past tense, or maybe investment. It can be invest, it can be investment, can be invest with ED in the past tense, right? So we take all the different versions of that.

And then one more step is we are the cybersecurity expert. And I mentioned in the, we mentioned in the paper. So we consult independent cybersecurity experts to see do you think this list of words that include or mostly nouns, whether they are enough, do we need to add, do we need to add any more words or do we need to remove any word that you believe that's not relevant to cybersecurity readiness?

So that's mostly about now. For the list of verbs, we borrow from a really good paper published on the top of a journal a couple of years ago. And they have a list of verb is linked to the, also linked to the cyber security. And we also borrow LLMs and we created another version of a list of verb. And then we combine a list of noun and a list of verb to make sure we randomly take some conference calls and to make sure.

Dr. Tran mentioned, the work and the now, they make sense. So for example, like in the conference course, maybe the CEO, they mentioned that. Our company needs to invest, maybe the verb invest, in cybersecurity next year. So if the verb invest and the verb cybersecurity, they appear within, they close enough, we consider the window like 10 words, 20 words and 30 words.

So make sure they are close enough and then they make sense and we count that appear. So that we really not just focus on work out, but this is a co-occurrence of two words and it should make sense. And then we count that one pair is being one. If there's another pair, we count it twice too. And then we manage a column like that and we call that one cybersecurity readiness.

Justin Beals: Yeah, well, I find this fascinating deeply because I think when so much in my own data science work and as you guys are working too, it was a lot about being creative with the data that we had, right? To your point, man, you could be like, okay, the simple thing in natural language processing is what's the count of word occurrence? But the more precise instrument will say what is the relationship of words? You know, together and can we derive a greater mathematical meaning when they show up together? It's very powerful, yeah. And so you build this corpus of data in some level, a measurement instrument, and then you're applying it to both external and internal communications, right?

And I think one of the findings in the paper was that any mention or kind of scoring of this language, by outsiders on a conference call correlated to a 0.0013 % increase in return on assets in the following year. And I'm curious your intuition on the value of that return on assets and what it means, especially for a large business.

Minh Nguyen: So actually, we focus on both the direction, the size. So actually, we try to estimate the coefficient. And we estimate the size first. We need to see whether it's negative or positively affect any performance outcome. First, that's the size. And the second one, we focus on the magnitude of that effect, whether it's big or not. If you see in the paper, the coefficient, the size of coefficient is not so big. But remember that's only one word. If they mention about only one pair of word and that have to increase a little bit about the ROA, the return on total assets. But normally in conference call is really, really big one, right? It can be 10 pages, it can be more, sometimes it can be 20 pages. And if they mention about maybe 10 or maybe 100,

So it really depends on the level they mentioned about that. It's we measure whether they're ready for that or not. And then the coefficient you multiply by 10 or by 100 and it should be bigger.

Justin Beals: Yeah, yeah, I mean, please Thi,

Thi Tran: I just want to add a little bit to that. when we already talked about all of the empirical studies and methodologies for how we do that. So actually for your question about the motivation, why we consider the outsiders to to play some part over here. So the key point is that the point is when we should not only focus on the insiders, because those are the guys that will talk about something good about the firm or something, right? So the outsiders will be the ones that will challenge that.

And because it is challenging, so I would say something like, we are curious to try to explore whether the outsiders actually pay attention on all of the things related to cybersecurity readiness, and they try to raise the concern. And if they raise the concern, is that in line with what the CEO or the leaders of the firm already talk about, or that is totally mismatch.

And if it is mismatched or according to that one, so how all of these can drive towards all of the following return on investment and something else. That is the reflection of the performance. So in other words, I would say something like the public attention on the share more security issues of that particular firm doesn't matter. And then how the matter can actually trigger all of the market reaction on top of that. So all of those are the motivation. Why we think that? We need to separately consider something from the outsiders.

Justin Beals: I, please, yeah, well, I'm just gonna say I think that I think that a lot of the reason people do any security is because they had a customer that asked them if they did something they got a security questionnaire from a customer that asked them if they did something I think outsiders are driving your security roadmap a lot of times Please men. Yeah

Thi Tran: Yeah, especially because sorry, especially because sorry Yeah, so I just want to continue to one without one for one more point because especially we are living in the information age and Because of that information can quickly circulate especially on social media, right? so if one company got hacked or something or they got some vulnerability and that it got reviewed by the Forum of hackers or something so people know and then probably because of some concerns and then the worries from the firm, so probably the CEO tried to cover it up and that is not nice.

And because of that, the public, they know and they will challenge it. even if on the other hand, so if the firm already introduced something like the new cybersecurity solution or defense, so probably the public want to know why. Why did you suddenly try to invest on that? Is it because you cared about security or is it because you tried to fight against something that you already know and then you do not want to declare or something else? So it is very complex of all of the psychological thinking and the mechanism, how it drives everything. So that is the reason why we want to unbox that one. And because of that, the more that the public actually they raise the concern, they want to mention about something related to cybersecurity awareness.

So it means they know something or they want to know about something that they already see some signs, some signals and something as well.

Minh Nguyen: So I want to add two points. So for example, intuitively, if I have my son, and he's three years old, and if I tell everyone that my son is intelligent, let's say for example like that. And it's always subjective, right? Because I am like his father, right? But if the teacher or someone else,they mentioned about that, okay, maybe he's smart, he's quick learner, something like that. It's more objective. So that's the intuition that we think about. We need some more objective way to measure. Okay, that's the first. The second one. So in term of the methods, so we use something called econometrics method. And in that one, if a company, someone from the insider, If they talk a lot about the cybersecurity, and then they can manipulate that. So it means they can increase number of words they talk about that, or maybe less. It really depends on them. Nobody can control. And we call that one, so in research, we call that one a heterogeneous issue. So it means we try to measure the firm, but someone else in the firm, they manipulate that if they want. So it's not objective. In terms of the research, the result is not reliable.

Minh Nguyen: But if we focus on the outsiders, so it means someone from like investor, journalist, and something, we focus on the words are spoken by those outsiders. That's more objective. So we try to avoid the his it is reasonable issue and make sure our method is made sense, is solid enough. Of course, maybe the audience, the outsider industry, they don't care much about the method, but because we do a research and we write a paper, we want to publish that. And the audience, are all the academics, the reviewers, they focus on the method. And if the method does not make sense, and they reject the paper, and we need to focus on that. So that's why we focus on the eternal one.

Justin Beals: Yeah, so I think you're bringing up something Minh and Thi, that I feel very strongly about in the commercial space. I see a lot of organizations that will build a measurement tool without research or efficacy in the development of the measurement tool, right? That would be like, oh, I scored a vendor risk, A, B, C, or D, know, American style letter grades or something very esoteric, but I think in my work, some of my work was done in the education space and especially in any space, having a poorly performing measurement tool that's not well researched, validated and accurate, or at least we know the level of accuracy, is super dangerous. Because to your point, right, like an insider can skew a result on the word count that's happening inside an organization and that's an outlier. That's not something, and if you had a good research methodology around your measurement tool, you'd count out outliers because you'd recognize that they don't give you an accurate representation. And so I'm, I think I'm,appreciating your work in this realm to being accuracy to some of these measurement tools by using effective research methodologies. Yes.

Minh Nguyen: Let me ask one more point. Thanks, Justin, for that. So actually, when we look at the result that we found, it actually supports our intuition at the beginning. So one word mentioned by the outsider, have the effect, the side of the effect is about 100 times greater than the one mentioned by insider. So it means it's more objective. So it means if we consider the workout for the whole of the conference course. It does not make sense because maybe we need to give weight, the weight of the rate of weight for the words mentioned by the outsider, then the insider. And I think it's made sense in the term of the objective and the subjective that I mentioned at the beginning.

Justin Beals; Yeah. Okay, so with this research that you've done, this measurement tool that you've prepared, especially your understanding of how much the value of cybersecurity investment comes from the perception outside the business as opposed to inside the business, how would you recommend that a CISO or someone that's a security leader? help use your research in justifying budget spend or investment in the business. I'd love to hear some of your thoughts on pragmatically how you would like the broader practitioner community to kind of consume and utilize what you've found.

Minh Nguyen: And then you want to talk.

Thi Tran: Yeah, I think it is a very complex question and thanks for bringing it up. And I think it is a kind of complexity that deals with the way that we as humans, need to interact and then we need to communicate. And because of that, feel something like because we talk about all of the behavioral patterns or something, right?

So imagine something like if you are the chief information security officer and then you think about whether or not we should declare about everything, like the incident that we just got hacked, we just got a data breach and something. So first, the lower requires that you need to declare that one. You cannot hide it. And the other one is that because the sooner that you acknowledge something like that, the better because the public will know sooner or later. And then if you try to cover that one, it is like you are creating the very bad image that you are lying about something and then that is not good.

So it is better to admit that we have the problem and then we try to resolve the problem. It is much better than trying to cover the problem. And the other one is that probably we're supposed to be proactive rather than reactive. And whether or not you already face all of the cybersecurity incidents or not, so you're supposed to pay attention on all of these. So I keep bringing all of the issues related to some firms that they actually ignore all of those. So that is bad.

And they're supposed to be really proactive in another time that they assure to the public that they care about security, they care about all of the well-being of the ones that play with the information systems. And because of that, for example, for the incident that I keep bringing to my class, it happened like more than 10 years ago in 2013 that the target corporate got hacked. But the point is that anyone can get hacked, right? But they already have the intrusion detection system. They got it installed, and the system was so well, it notified them once. When the hacker entered the system, they ignored it, the admin ignored it. Then the hacker actually collected all of the data and then the hacker was about to quit the system. And then the system notified the admin again, but actually the admin ignored it twice. And that is the reason why after that, they face a really serious cybersecurity incident. because of that, so after that, they have to pay for more than $200 million to try to pay for all of the victims. And then on top of that, all of the related businesses, they need to reissue all of the cards for the affected accounts and something else. So for all of the total effects from all of those together, including all of the estimated and then the one that are already measured, so we have more than $500 million loss for one incident.

And then the weird point is that they already have something. They already have the cybersecurity defense, but they ignore it. So it seems to look like they just got it for decoration. They just got it to satisfy people. So they're like, hey, we care about it. And then we prove that we try to do something. But the point is that they do not leverage on the use of that one. So that is very serious.

So it is not only to show to the public that hey, I am working on the security of that particular company and I care about your information, but actually you need to actually care about it. So that is the reason why we want to emphasize on all of the actionable words, that people actually care about something and then we see how the public can react to that. And imagine that even when you can try to pretend that you care about it for now, but you ignore it by trying to save all of the money or something, by not wanting to look at all of the bad things or something.

But eventually the public will know. And because of that, it will backfire you later. And because of that, how can you feel that you can trust that company anymore if something like that can already happen? So the issue of trust is very big. And then the moment that you can trust the system, that you can trust the defense line, you can trust the people that handle all of things. And then imagine something like even when you can have really high positions, something like the information security officer or something, But you can be affected a lot by the public pressure, by the top leaders of the firm, when you ignore something like that and then it costs, let's say, half a billion dollars of loss for the company and for the reputation and for all of the related allies in business.

So it is a lot of things then that is only from one incident that happened more than 10 years ago. And if the same thing happened now for Target corporate, you can imagine that the size of the effect at the loss can be like 10 times bigger. Yeah.

Justin Beals: Minh, you talked a little bit about the coefficient and listening to T, something that I imagine could be useful from your research is being able to understand how much discussion is happening outside the firm as being a coefficient for how much investment should be going on inside the firm. Almost a predictor of you should probably invest this much amount of resources and time and energy, because that's how important it is to the outside the firm discussions.

Is that an appropriate way to think about how your measurement tool may illuminate how we want to manage an organization?

Minh Nguyen: Yeah, so I think there are two. So I'm thinking about two recommendations here, two implications based on your question. So first, the company needs to make everything transparent. So that's the first thing. So if they have any data breach, something, they should not hide it. So it means they should talk about that. And I think maybe you may know, like, at HEC, they also have requirement, right, for company to make that available for everyone to know about the summit shoot if they face. That's the first one. And the second implication is the company need to give some chance for people from outside, they can speak. It can be customer, it can be investor, it can be a journalist. I mean, if they have a chance to talk about the company and they can learn something from that.

So it's been both the people from inside and outside, if they talk about that. So that's why we need to keep something like conference calls. I think it's really good place and good data set that people look at that and find something that relates to the company. But the point is, I believe the conference call right now is not requirement for all of the public company in the US like Form 10k. So I don't know, maybe one day, maybe the government will require all the companies to do that. But right now it's like voluntary. So that's yeah, difficult.

Justin Beals: Yeah, think we're a patchwork of legal requirements, state by state and federally here in the United States. I do think that if you're traded publicly, there are some new requirements about disclosure. But that's a very few number of companies are publicly traded compared to the number of businesses that they're out there. And if you're privately held, really you can make up your own mind. That being said, the big revolution we've seen is in security compliance. So, you know, SOC 2s or ISO 27001s, and there's a whole bunch of other ones that have come out lately. And so then you start to get in contractual arrangements with your customers that you will do data breach notification, for example, and you have a methodology around it.

So I do think that's improving. So, I'd like to learn a little bit because I think that this was one paper, but you guys are continuing your research work in this space. Can you give us a little highlight of what's coming up next in your research and work around this cybersecurity work?

Minh Nguyen: So right now we have, so all of the results we saw in the paper so far have come from the conference calls. But we also have the Form 10-K data, and we already cleaned and have everything. And for conference call only, we run 30 models. So we have the 30 model code for 30 different versions. And all of the versions, all of the 30 versions, we see the

positive effect. It's really consistent. It's really robust. so our next step right now is to do the same for the Form 10k to see whether we can get the same result or not. But we need to be aware because the Form 10k and Conference calls are different. Form 10k are prepared by the company. So all of the work, even though that can be audited, but all of the work, or everything that prepared by the people from the insider, from the company. So it's been, we need to be careful with that. But we want to do to get some, we call that the robust, robust change. So we want to get something to see, because we want to test to make sure our measure is make sense, even though in the case of insider only at the point.

Thi Tran: Yeah. I want to add a little bit to that. Are you done? Yeah, yeah. So I want to add a little bit for that, for the plan of expansion for what we already have for the conference. So we are thinking and then we are working on some ways that we can expand it. So the first one is to add another dataset like we said about the 10K. And the second one is that we want to expand the examination about the effect of cybersecurity readiness to firm performance by considering the modulation effect from the experience of facing the data breach before, whether the firm already have the breach before. So at first we thought about doing something like only zero, one, yes or no, that they already faced the breach before. But then I proposed to them, hey, no, we need to increase the measurement and that we need to count for how many times that it already happened before for that firm. And then how big is the loss?

Something like sometimes, one time of having the breach can be more than 10 other times, right? So for the loss and for the total number of all of the victims and for all of the affected allies and something else, so a lot of angles that we need to consider. So that is the second expansion. The third expansion that we are considering is that, so instead of only relying on all of the ways that we can extract something from NLP when we try to use some mechanical way to try to extract something from the tax, from either inside or outside and probably from the 10K. So why don't we try to get something that is a little bit more concrete? And then we are thinking about getting all of the actual data that some firms that may actually acquire all of the patterns. And if we can get all of the pattern data, and I already searched and then I found a lot of pattern data, and then we can classify by using all of the machine learning as well. So we can use LLM or we can use the actual machine learning base our classification function so that we can classify it and then define which one is related to cybersecurity. And then from that, it is stronger because actually either the firm actually invest all of the effort and human resources and the financial resources so that they can invent something or they can pay so that they can buy or they can train some innovations like that.

So it's stronger when we can claim something like,that is the actual and then the concrete efforts that some form they actually pay attention on cybersecurity issues. And because of that, we see all of those. And the other one that can be considered is the total investment in IT. And then in IT, it can be anything, right? So we can try to see how many of the items in the list that they have, and then the amount that are tied to cybersecurity. So that is stronger, kind of the actionable part, when we think about self-sufficiency readiness, so that it can strengthen the world. So I would say that that can be tied together in one top paper or we can split it into multiple papers. That is like we are building the whole chain of research papers.

Minh Nguyen: I want to add one more point, Thi and Justin. So, besides we think about the Form 10k and the part and data, or maybe at the IT investment. That's one way it mean work with another data sets or some more data sets. There's one way that we are actually, we are doing right now. We try to extend our paper in the term that we focus on subset of the data. And we call that one a subgroup analysis.

For example, now, because we have, so we have like, so our dataset right now really big, I think like couple hundred thousand observation. And then we can focus on the S&P 500 company. So that's one way. And we might focus on like, focus on the big company, analyze that to see the effect compared to the small company. to see, so right now our result, the average result for all of the company.

But if we have different groups like that, let's see what's going on. And one thing is we can focus on young companies, maybe companies that are founded like a few years ago, compared to someone there for many decades. To see the young and the old companies, whether that's a fact or not. Or maybe, let's say, maybe right now, we actually we control for the state.

We have the information of the headquarter of all of the company, and we know when is the conference call happen. And we also have the real data bridge data. But right now in the model, we took control for that. We don't consider the real data bridge data as the main variable because many people did about that before. So we focus on the one based on the test. But we have around 20 other variables that we control for in the model.

Every time we run, we include the outcome variable, the main variable, and the other 20 variables. So make sure our result is, sense, if we remove some variable, the result still OK. But if, so in research, if we remove some variable, the result change completely, the size and the magnitude is not reliable. This means we try a lot of things. Right now, it's 30 models. But I think when we finish, we will run more than 100 models to make sure all of the results are consistent. If some results are not consistent, we need to explain that.

Justin Beals: I find this absolutely fascinating. I love the methodology and I think your passion in creating a more precise instrument, right? Because if you can bring more models, more classification like SMBs, you know, this is how the model works or for a large business, this is how the model works. At the end of the day, your instrument will be so much more precise. I also think about the expansion in data, you know, almost every sales call is recorded and we know who the salesperson was, and we know who the customer they're talking to was. And if we can take that style of lexical information and understand what's happening in the sales motion where a new customer is asking about security, I think we can understand what the level of investment should be at a business to win deals. Not even talking about breach, right? Just to grow the commercial interest of the business. Yeah.

Minh Nguyen: You make me think about some of the social media data. Sometimes it's not from the company or maybe some formal wildlife journalist, customer, anyone that can talk about on any social media, Instagram, something Reddit, we can get that data, and we can measure something. There's the cybersecurity readiness inside those copter online reviews or maybe comments or something. That should be another direction too.

Justin Beal: Yeah, well, Thi and Min, thank you so much for the work you and your team have already done. There are a couple of teammates on the paper that couldn't join us today. And thank you for the work you're continuing to do. I'm very grateful. I deeply appreciate the science that you bring to this work as opposed to the folklore that happens so often. And thank you for joining us today on SecureTalk.

Thi Tran: Thank you.

Minh Nguyen: Thank you, Justin, for giving us the time and opportunity to share our research to the audience. And I hope to talk with you in your next term.

Justin Beals: Wonderful.