Why you could fail your CMMC Level 2 C3PAO audit.

Justin Beals: Hello everyone, and welcome to SecureTalk. I'm your host, Justin Beals.

Before I built compliance software, I hacked my way into psychometrics. It's a strange and fascinating corner of data science, and I came at it sideways because I was building educational assessment products and needed to understand what the numbers I was generating actually meant. What I learned changed how I think about measurement in general.

Here is the uncomfortable truth psychometricians live with. Intelligence is not measurable. Not really. Not the way mass is measurable, or temperature, or the distance between two points. There is no independent physical quantity sitting inside a person called IQ that we are reading off a scale. What we have instead is a constructed score, built from complicated statistical analysis and, more recently, machine learning, that tries to capture something we can only observe in relative terms. One person's score only means something compared to another person's score, on the same instrument, administered under similar conditions. The whole edifice rests on whether the test is reliable, meaning it gives consistent results when you run it again, and whether it is valid, meaning the score actually corresponds to the thing you claim to be measuring.

If those two properties break down, you do not have a measurement. You have a number that masquerades as meaning.

I want you to hold that idea while I describe how CMMC actually works. At Level 2 and above, CMMC uses an additive scoring methodology. You start at 110 points and lose points for each requirement you fail to meet. The final number determines whether you can bid on Department of Defense contracts. It looks rigorous. It looks like a measurement. But those point values are derived from an estimation of an organization's relative maturity against the NIST 800-171 control set. Maturity, like intelligence, is not a physical quantity. You cannot weigh it. You cannot put it on a ruler. It is a construct that only has meaning when the instrument measuring it is reliable and valid across every administration.

Which brings me to our guest's research. Logan Therrien spent months collecting data from CMMC assessors across the Defense Industrial Base, and what he found was that some assessors were looking at a single piece of evidence to validate a control, while others were looking at one hundred percent of the available evidence. Same standard. Same framework. Wildly different yardsticks. If two assessors can examine the same contractor and arrive at meaningfully different scores, then the 110-point scale stops describing maturity and starts describing the assessor. That is the same failure mode a poorly designed psychometric test exhibits. It generates numbers without producing knowledge.

The stakes are not abstract. CMMC compliance is now written into federal law. As of November 10th, 2025, every new DoD solicitation carries the requirement. The score on a CMMC assessment now decides who gets to compete for defense work. If that score is not calibrated the same way across the C3PAO ecosystem, we have a problem that no policy memo can paper over.

Logan and I spend this episode digging into what a reliable assessment actually looks like, why NIST 800-171 left room for interpretation, how the ISO 17020 accreditation requirement may close the gap, and what contractors should expect from a well-prepared assessor. We also talk about why this matters beyond compliance, because trust between private industry and the DHS is itself a piece of national defense infrastructure.

Before we begin, I want to ask something of you. If you are an assessor, a contractor, or a consultant living inside this ecosystem, leave a comment on this episode telling us what you have seen. Where have the yardsticks varied? What has worked? Your experience is data, and Logan's research depends on more of it.

Now, about our guest.

Logan Therrien is Chief Strategy Officer and Lead CMMC Assessor at Kieri Solutions, one of the original C3PAOs in the U.S. Defense Industrial Base. A retired Navy Lieutenant Commander with 24 years of active duty, he managed information security for over 7,000 personnel and oversaw physical security for assets valued at more than 20 billion dollars. Since transitioning to the private sector, he has become one of the most recognized names in CMMC implementation, conducting assessments, training the next generation of assessors as a CMMC Provisional Instructor, and co-authoring research on standardized evidence sampling in CMMC assessments. He holds a Master of Science in Information Assurance, the CISSP certification, and is actively pursuing a Ph.D. in Cyber Defense.

Please join me in welcoming Logan to SecureTalk.

—--

Justin Beals: Logan, thank you for joining us today on SecureTalk.

Logan Therrien: Yes, sir. Hey, thanks, Justin, for having me. I really appreciate the opportunity. The conversation that we're going to have today and some of the topics are really not well known. And there's a lot of folks out there that need information. And so I really appreciate the opportunity, the platform and your time to be able to talk with you today.

Justin Beals: It's great. I came across your work, your published paper in the journal and we'll get to that in a bit, but it's just really exciting to talk to folks that are doing real research, peer-reviewed research in the space. I think that's the type of science that we need to bring to compliance and security broadly.

We're trying to make the right choices at the right time. Speaking of which, you have worked in security at a number of different tiers. You spent 24 years as a Navy submarine officer working in DOD information, personnel and physical security, and now you're a CMMC assessor and Cary's Chief Strategy Officer. I'm very curious about your experience in the Navy and kind of what that taught you about evaluating security.

Logan Therrien: Oh, certainly. So I have a pretty diverse background. It is easy to say I was in the submarine force 24 years, but even that I've always done things the hard way and found an interesting path. And so the first 10 years of that time was enlisted and very technically oriented. I was repairing things, not sleeping underwater, fixing things, making sure that the submarine kept on being able to operate its combat systems.

And then, you know, some of that time was spent instructing. So I found a passion for being able to teach people getting, getting, watching those light bulbs flip on over people's heads. But throughout that time, it was still security related because you know, I'm dealing with classified information and a classified world, lots of secrets to be had. And then, so I eventually got a commission. And even as my commission path, I had two different fields there. One was electronics summary maintenance. Another one was summary ordinance, all compliance-focused, all compliance-heavy, even when it was in the technical and listed side. And my first job as an officer was as a security officer for submarine school. I was just thrown into a field I wasn't too familiar with at the policy and direction and compliance overall level. And so I learned very quickly how things can fall apart, how to make things work, and ended up actually going right into education, get my master's degree, information assurance at that time, because I like, I just want to learn more.

But throughout the next 14 years, whatever I did, it was still heavily focused on the protection of information, people, places, and doing the best I could. And I had a lot of lessons learned throughout the process. But more specifically, throughout the submarine force, one thing that they do very well is the root cause of the analysis throughout whatever it is, whether it's fixing things, watching things, programs, policies, it is the, if something goes wrong, let's all sit down for some time until we figure out what the root cause is. And in many of those cases, it typically boils down to people. And then once it boils down to people, it's whether they cared or didn't know. So either way, helped me figure out how to identify why a process is broken or how to work at our process efficiencies.

Justin Beals: can't imagine a more risk-full environment, Logan. You know, I do a little bit of sailing and that's, that like feels dangerous enough and I'm floating on top of the water, let alone in an environment where there are enemy combatants potentially, even if you're not, you know, directly in conflict, you're certainly navigating a very challenging environment.

Logan Therrien: Three-dimensional environment, and you want to make sure that everybody's trained to go when they submerged that they come right back up to the surface when they want to.

Justin Beals: Your paper has a motivation section that's unusually candid for an academic paper, not a lot of emotion sometimes in these. And you wrote that, you you watched assessors applying divergent sampling thresholds and expectations under similar conditions. It is a challenge. know, we imagine an assessment as an instrument, like I do anyways, like as a measurement device. But if you're not measuring on the same scale, you can wind up starting to not trust the instrument. I think that's what I was most fearful of. I wonder if that's an emotion you considered as well as you were looking at trying to build a reliable tool.

Logan Therrien: Emotion. I haven't thought of it that way. That's, but that's not wrong. That's that's not too far off, and it's really the passion towards making sure we're doing this right.

The tool, mean, some of the knowledge is used throughout there, right? What is the yardstick? Are we all using the same yardstick or are they using a one inch string and I'm using a five foot steel rod, you know? And that's kind of where I stood is that the CMMC, so the Cybersecurity Maturity Model Certification Program that I'm involved in is pretty much based off of NIST 800 171 and 171 alpha.

And those two documents provide the standards on how to the implementation requirements for defense industrial base contractors, right? The the follow-up to that was the CMC assessment process and that was in the 32 CFR part 170 and all those talk about how the baselines you need to meet, but it doesn't talk about how to assess as an assessor. And even when it comes to the requirements for the different certification levels of an assessor, when it comes to a certified CMC professional, which is kind of the entry level, and then the CMC certified assessor, which is more experience, and there is experience for assessments, that you have to have a certain amount of years for assessing, you a certain amount of years of managing teams and a certain amount of years of cybersecurity.

But it doesn't really focus on what that assessment experience was. Was it internal, was it external? You just have to have a resume. It has to make sense. And I will say the Cyber AB is the accreditation body, and the CACO is the one that validates the certification for those personnel.

They're doing a good job, right? But there's a lot of work still to be done. ISACA actually just took over as Kaco, and I imagine there's gonna be some really significant differences based off of ISACA's experience doing this. But where we are right now is ISACA's not in place. CyberAV has done a great job. The 32CFR170 is only a year and a half old, and we've been doing this since maybe less than 30 days after 32 CFR was implemented, right? And so it didn't really address the experience of that assessment or the different types of measuring stick that you're supposed to use across assessments. So really the metrics for of training folks has been through the training providers, right?

And there is a level of educational requirements that have to be conveyed to the learner for them to go to their certification. But in that, it's not how to do sampling. It's not how to do assessments. The focus has really been on what does this requirement mean? How do you evaluate an assessment objective? Right. And the deepest it would go into what sampling would be required is there is this 800 171 talks about depth and coverage, but it's very top level, very vague.

It doesn't get into the specifics of if it has more risk, should you have a deeper dive? Do you need to look at more things? If you look at a single group policy object that can cover every asset in your information system, is that good enough? So again, people were taking those different aspects of their experience. And then when it came to the training that was being provided, the training is also, this is where it gets funky.

It was being provided off of draft information prior to 32 CFR 170. So when 32 CFR 170 came out, that training information was never updated. And as instructors, we have to go and say, this is where we're training to the exam. This is time now. And here's the differences. But even at that level, it just talked about adequacy and sufficiency. So am I looking at the right information? And have I seen enough of it, but it doesn't say what enough is or how to expand based on risk analysis. And so that's been an issue. And when it comes to my experience with assessments, we have different ways of making assessments happen.

And it's slowly if I'm going too deep to have a hear on you, but prior to 32 CFR 170 being published and enforced,we did something called a joint surveillance voluntary assessment. And C3PO's, this is where C3PO's would work with DIPCCAC, which is the Defense Industrial Base Cybersecurity Assessment Center. And they were the ones that typically do these assessments, but they're rolling C3PO's in for training, right? Let's see how the C3PO's are doing.

DIPCCAC has a pretty standard methodology for performing that, but does stop the moment the 32 CFR once heavy was published and then C3PO's went on their own. And so during that time, I got to see how folks did it differently. Meeting folks in the ecosystem itself, we have conversations and got to see how they would assess differently. And then we use to function on the earlier days a lot of C3PO's would use 1099, so subcontractors. They would bring them in and they would teach them how to operate with their policies and procedures. But that would also give me a perspective on how other folks who may have performed assessments with other C3PO's also operate. And the playing field was not the same. So at that point, the organization I'm with has a pretty high standard. We have developed procedures and policies on how we do this.

So we would just train those folks on how we do it. And then now we know that the method that we're using is measurable and a future state.

Justin Beals: The impact of what these assessments being rolled out has been really? The scale of it is hard to get my arms around sometimes, and I think You know from your perspective, Logan and helping to get good assessment methodologies in place as it rolls down through you know the what is the public side of companies supplying the DoD with services and needing to keep that information secure is really massive i've talked to folks that mow lawns on a base having to go through a cmc level two as well as folks that Won't tell me what they make they just say it flies going through a cmc level two assessment and It is a ginormous shift in in private markets and the way they contract with the Department of Defense. That, are you perceiving it the same way that I am from a scale of impact?

Logan Therrien: The scale is huge, right? And I think we're starting to identify how much the scale is growing. And so kind of some background, and this is where if you talk to folks, there's some passion in the ecosystem from the implementer side, from the consultant side, from the assessor side, from the policymaker side. They'll have their different spectrums of where they stand. And if you go back to where the policymaker stand, the Department of War, really, the requirement to do this thing has been in place since almost 10 years, right? And those contractors have had the requirement to go report. And so the expectation of the Department of Defense or Department of War has been that it's already done. UTOTA SIL is already done. It's an SPRS system, the supplier performance readiness system, that you have a score, right? But the score requirement hasn't been perfect.

The requirement is that you have a score to be eligible to bid. So some organizations are honest, we're good, and say, we're not fully at 110. So let's work on this contract. And really, what went into place over the last year and a half was the we're required to get a third party to come in and validate what you're saying.

There are some phased roll-ins. So this is a four-year phased roll-in. And really, the first year was self-assessment mostly. There were some third-party requirements to go have a third party come and assess you. And it's going to become more increased into the contract requirements that there's a third-party assessment for level two. And eventually at end of the four years in those contracts, will be that all third parties have been assessed by a third-party assessment organization.

Here's what I think we're getting at though is that even the wrong numbers, if you go back to 32 CFR 170, the preamble, there's a table that talks about how many organizations seeking certification, an OSC, are expected to be level two certified. And the number at that point was about, I think, 80,000. And I believe what happened over the next year was conversations happened because there's a requirement to flow this down to subcontractors, right? So the prime contractor, the big Boeing's, Lockheed's, they'll get a contract from the Department of Defense for war and to make a thing. And then those prime contractors need multiple other contractors, and they'll flow that down. But when they started pulling the string on where that CUI, the controlled unclassified information flow to, I think the numbers started growing. So, that organization that's doing Chrome plating on the side of the road, are they getting CUI? Well, maybe they are.

And so now we go look at the 48 CFR, which was the requirement that put this into contracts, right? So if you think of 32 CFR, that's a defense saying we need to do something. 48 CFR is how do we get that, need to do something as a requirement for contracts. And that's where the numbers shifted up to about 118,000, right? So now we're up almost 38,000 new organizations. And that is a significant impact.

So for those organizations that didn't have a clue that they needed this done. So the argument saying that they have had to have known that you had to do these 110 security requirements for years may not be true until they actually started pulling the string and they were notified by their prime contractor.

Justin Beals: Yeah, I think there was also a little bit of bravado where certain companies were like, make us. And then they did. And now there's like, now we're made to do it. A little bit of kvetching on the other side. I think that having operated without a standard for security and developing online software or needing to deal with security about the types of software that I build and then operating within a standard on the other side of operating within a standard.

I'm always much happier. I feel like I know what I should be doing, at least at a baseline level. And that makes me more confident in our work. And I think that's what I preach sometimes to people who are resistant about needing to do these things is like, honestly, this is good medicine. You'll feel better knowing that here's a list of things you need to operate under, and you can figure out how to meet those particular requirements, NIST 800171, setting up those expectations. I think it's healthy.

Logan Therrien: Absolutely, I do agree. you know, that's something, show me what I need to do and then I'll go do it, right? Let me figure out how much it's gonna cost, let me figure out who I need to involve, but at least make it where I don't have to overdo what I'm doing or underdo, just give me the standard. Problem that I've seen with this standard, that is, NIST 800171, is that it was designed to accommodate a lot of different types of organizations. And that's the, although the heart is there, I think it caused a lot of confusion. And, and most folks that were implemented at this, and we do, see a lot of folks from the RMF side, you know, the, NIST 853 requirements, which are pretty black and white. But the 171, even though fairly tied to RMF, it's not the same. And in a lot of cases, they loosened up the wording and that made it where people now had to figure it out again.

And here's the problem is that those organizations seeking certification, there's sometimes small mom-and-pop shops, right? They want to make doorknobs. They want to make a widget to go provide to the prime. fortunately, they have to get that CUI to make that widget. They may not have an IT person, right? So now they have, do I need to spend the money? Well, I already have the contract. How do I go back until the prime that I need to add another $100,000 for a salary? or another $200,000 for salary and some technology or five for MSP because I just can't do any of that stuff, right? I don't even have oversight to run an IT department, right? So there's huge differentials on that side and big obstacles for companies to overcome. with the 171, adding that unknown of how do I do this as an organization and then the variability of how to interpret 800, 171 requirements and assessment objectives. It's huge, right?

And so I do this, I talk about 171, I talk about CMMC, I talk about obstacles and how to implement a compliance program between 40 and 80 hours a week. And I still learn something every week, working with my teammates and the other folks in ecosystem. How is somebody from an organization that doesn't deal with security going to figure this out?

And so I see a lot of organizations that are really trying to do their best, but then when they get to the assessment table, they can't get past the pre-screening because they're not answering questions.

Justin Beals: Yeah. It's a challenging conversation. I agree with you. When I'm talking with someone, they're like, OK, so we're going to pick your flavor of framework. NIST 800171 is a great one for us to talk about. I'm like, it's not going to tell you what to do. It will ask what you do. And if you can't tell it what you do, like, do you keep data private? Well, yes, we encrypt it at rest.

Then you're gonna be caught unawares when you get into assessment. And so that is something that, and in a way, some people are flummoxed by that. They're like, no, how can I tell if I'm gonna pass the assessment, but won't tell me what to do? And I'm like, but they had to design it for a lot of different types of businesses. And the solution can be very different depending on the type of company you are. That was their challenge from a framework perspective, yeah.

Logan Therrien: And yes, so the, I think the Department of War, we'll go with the FW for the rest of the time to go.

Justin Beals: Sure, that sounds great, Logan. I'm flexible. Yeah. Yeah.

Logan Therrien: That depends on who I'm talking to. So I think they recognized the difference in understanding and the difference of maybe the level of effort that is desired to be expended in this process. And the variability that was afforded in this 800.171.REV2, which is what's being implemented and is dressed right now, is it's changed when it comes to REV3. And I think they spent a lot of time going, all right, understand that you maybe ignored all the best practices or you didn't understand best practices. So now let's define the organizational defined parameters for you. And so for example, maybe before we could have gotten away with the requirement that people are trained on risk associated with the system that handles that data. That was a requirement, right? So if the organization and OSC came and said, yeah, we did train on that.

Everybody goes through that training. All right, when was the last time you did that training? We did it six years ago. Okay, I don't think that's what the intention is or best practice. Well, what does it say the requirement is? Right, and so now we're having this argument back and forth. So if you go to Rev 3, it's people are trained when they onboard. It's called it specifically, and it's also they're trained every year, or they're trained when security risks are identified, you and then you have to make changes to the system, right? So they're very much more specific.

And in some cases where organizations would have gotten away with an extended period of a time to respond to something. And again, it was acceptable under REV2 and REV3, the DOW has now become very clear of you have 24 hours or you shall respond in 72 hours or one week. And it's just, think after all these organizations finally get through REV2 and then are now faced with Rev3, they're gonna have even more challenges with those organizational changes for 120,000 organizations.

Justin Beals: Yeah, I, you know, there's a balancing act there. I can see where not being strict enough in the specification or how you're tested would lead to people wanting to find the easiest route possible and not actually getting the outcome that's most successful. And then on the flip side, being a little bit more specific is going to drive people into certain habits that are probably good. Now, one thing about CMMC that I think a lot of people have when I've chatted with them and they're a little surprised by it or they act like it's a little new, but wasn't to me is, and this is the subject of your paper is the evidence testing.

Like I think CMMC or the way a lot of the cyber AB folks have described it to me is, is an evidence-tested type of security assessment. And, of course, evidence testing brings a lot of different questions about how often, with what rigor over what time period.

That seemed to be a lot of the subject of your analysis and paper.

Logan Therrien: The types of evidence that we're testing, and so I guess I'll start from the beginning, is that there's really a couple different things that we're looking at when we do assessments. And we'll go into, there's 110 requirements. And then out of those 110 requirements, there's 320 assessment objectives. And they're not equally distributed. One may just be that they do training, and then that training's done, right? So.

A lot of cases though, there is a definition defined something, right? So define how you do this thing. And then, or identify a time limit or something specific associated with how you do this thing. And then the related assessment objectives beyond that is that you do this thing as identified or you do this thing as defined. And so in some cases that will determine

the evidence sampling that we're gonna be looking at. In some cases, it will be determined off of, can I look at one piece of evidence and determine that everything is being met, right? Or do I need to look at, I guess a simple way to do it is if I have a GPO setting for a information system that uses Microsoft across the board, right?

Is there a way to validate in a GPO setting that all the devices in that system are being controlled the same way? If so, that may be, I need to look at one thing. There is, then let's go and maybe I have three different types of operating systems. Maybe I got some Linux and some Mac and some Mac. Maybe now if you can show me that you do something globally for those three different types of operating system, maybe I need to look at three different things, right? And that's it.

But if you can't, now we start expanding how we look at things. Maybe I need to just go start picking how many of those I need to see to call as an assessor that I've resolved that I trust what you're saying, right? That I have no doubt that what you've said is true, or at least a high percentage of no doubt. However, that's not well-defined as it is now, right?

Logan Therrien: And that's where we get into, again, how do I have to look at those numbers? What I did find through the research is that there are some folks out there that look at 100%, right? And there are some folks out there that look at maybe one, right? Without anything else, and that may be a part of how they're trained, but there was not a defined standard to go back to. Because as I mentioned before, the draft CMC assessment process called a CAP.

I just talked about adequacy and sufficiency that was kind of removed out of the cap that's published. And then the 171 just talks about depth and coverage. So if an assessor that has very little experience, maybe has assessment experience, but doesn't have sampling experience, they will do something that is not founded in a method that will achieve results that can be repeatable, measurable and that's the concern.

Justin Beals: Yeah. Okay. So let me try this out. Like with a feet on the ground, a little bit approach, right? Like, so sometimes I like what you're talking about where I have to describe how I do it and in what cycle I do it. I think that's absolutely valid by like we, you know, let's say that from an incident response perspective, you know, we're, operating incident response practice. I have some type of process or methodology about how we respond and we had two incidences in the last year. And so now I have two incident response things that happened, whether I followed that with a ticket or we wrote it up in a document or whatever my practices were, you know, in a way I'm starting to create evidence around, we said we were gonna do something, we're gonna have an incident response practice, we're gonna respond in the moment to the incident. We have a design of how we're gonna respond. And then we had two incidents and so we have the receipts from going after that incident when it took place. The assessor is trying to define how much testing they're gonna do to have that confidence that the test in that sector, that little focus was efficaciously executed. Like you can derive a pass fail or from that. Am I grabbing the information in the right way, Logan.?

Logan Therrien: Yeah, I so this is a great example on how I would look at things, right, from the assessment side. And there are requirements that we do look at. How do you track your incidents? Do you have an incident or response program in place? How do you evaluate it? Are they tracked from beginning to end, right? in fact, the incident response is very delineated on, I think it's NIST 861 for IR response. So it's tied to another framework that's well known.

But in this case, if you say that this is our incident response policy and process, and it talks about eradication, containment, recovery, those different phases, and then I see your policy and procedure that you actually address that, well, does it expand well? If I'm talking to, if I'm in the interview process with your IT manager or your incident response team or your MSP, can they, if I just ask them a few questions and they can say, yeah, that's how we do it and it checks with policies and procedures, you know, that's probably gonna enough, right? In some cases, again, if it's just identified that it's something there, I just go look at the policy or procedure, it's right there. And in some cases though, well, actually let's focus on incident response because it's a great one. I don't want you to have incident responses, right? I mean, I don't want you to have incidents that you have to respond to, but ultimately you probably do.

Here is a good reason to discuss this exact security requirement though is because up into the 30, so the 48 CFR that talks about the implementation of CMC, it's requirement now going into contracts that you have to be level two certified or assessed prior to consideration of award for the contract. Right, so let's say I'm a company, I'm just starting up, and I want to be eligible to bid for these contracts. I may have just hired an MSP to build me a Microsoft environment, or whatever environment works. And it is four months old, right? And I haven't even had it the whole time. And I only have three people in the infrastructure and the system. What's the expectation that I have any incidents? Maybe none, right? So this is as an assessor, I evaluate that, and I say, all right, well, show me your procedures and policies. Do they make sense?Do I believe that they're achievable based off of my experience?

And then, one thing that's important as an assessor is that we put our thoughts onto the paper. So if the string is ever pulled by the Department of Justice or if something goes bad, somebody in the investigative chain, that we can say how we got to the endpoint of achieving validation for their policies and processes, procedures, interviews. So it may not look the same in that case.

But what I just told you, do I think that 100 % of the assessors are doing it the same way? I don't. And I think that's where we're going to get to. And here's the thing, though, is I don't know that we'll ever get to the point where every assessor is going to be doing it the same way.

What I do think the most important thing is that when it comes to that investigative chain, if something would happen and these assessment results were ever pulled, that the Department of Justice or the Department of War can get to the same conclusion that the assessor did based off of our processes. So if I am doing something, how did I get there? How is my documentation as a C3PO or as an assessor for getting from point A to point C, right? Okay, I looked at this, and this is where findings and writing stuff down, I looked at, or there were 100 different sample points. I looked at five. At five, I felt that it met the intent of the security requirement, all right? And is that good enough, right? No, I don't, in some cases, it may not be, right? So if I wanna have a 90 % validation rate, maybe 5 % is not good enough, right? And so this is where I think it's important that we understand effective sampling. And we can talk a little bit more about what my thoughts are on how to get there and what I think the actual environment is doing right now to make that happen.

Justin Beals: Yeah, let's dig in because you know, you've mentioned you have a master's in information assurance. The science around this is not brand new. You know, I think about all the years of running businesses, I've been through a financial audit and how many transactions they reviewed. It wasn't every bank record, but it was a fair few. And the assessor designed the methodology for the sampling. But you know, when I think about a certified public accountant, that's part of their trainin is how to develop assurance, right? And I think for many in the C3PO space, and probably why you were interested in investigating this, researching it, this is some new concepts.

Logan Therrien: I think it's a new concept for most security folks, right?

Justin Beals: Yes, we're used to testing everything all the time whenever asked.

Logan Therrien: And so, know, come from my background back in the military is I had to report compliance all the time, or I was the person out there validating compliance. You know, I loved when somebody would say, " Here's a checklist, right?” Or here's a go validate state compliance. I would have like 300 plus checkpoints of going, is it this far from this line? Yep, it's not. Is it blue? Yeah, it's blue. And so it really helped having again some kind of baseline to formulate my assessment methodology off of. So let me, I guess I'll jump, let me jump to the end. I think this is where we're going to get it right. However, there's gonna be a time before between us getting from point A to point B. And that is the ISO 17.020 accreditation requirement for C3PO's.

However, there's again 27 months from the time they get authorized to the time they have to get 17.020 accredited as an inspection body. And one of those requirements,as through the accreditation process is that they have a sampling methodology documented, or how do they do sampling performance. It's not that they have to design one, it's that they have to show what they do. So if they could use a different framework that's actually working, and then incorporate that into their processes.

I tend to lean towards, because again, there's vagary in this, is that they have to have one. And now there could be some that say, sure, I developed mine and we do one. Okay, yeah. And I think this is where the Cyber AB and A-NAB are gonna do their diligence and their experience and go, that's maybe not good enough. I need something better than that.

And I'm hoping that's where it kicks in from my understanding. That's really where they're to be paying attention to. And I think that's going to make a world of difference again, but there's just, there's just time between that actually happening. And that's, that's why I felt as important to address now, because there's not much data in the CMC ecosystem. And it's kind of hard to have a discussion because until there's data on the table, it's all just feelings. And it's really hard to have a stance when everybody's talking about their feelings on some.

Justin Beals: Yeah, I think that was one of the major findings of your paper was that there's a variety of feelings out there right now. Is that true? Was that your takeaway having published, Logan?

Logan Therrien: Yes, yes. The community is pretty tight. I don't know who provided the results from those data collection, but I try to communicate with as many folks as I can professionally, because in the end, we're here for the Department of War to make sure they're getting a secure environment from the contractors that hire us to validate that they have a secure environment or hire us to go out there and consult them from the consultant side. We can't do both to make sure they're meeting the requirements. Right. So if it's a bad day when we have someone who's saying this is good enough, and then we have an assessment team come and say this is obviously not good enough, you fail. Right. That's not doing any good for our defense industry. Right. And ultimately, how's it going to affect the supply chain?

We don't want to impact the supply chain. We just want to make sure the data is secure. And if we are impacting the supply chain on false basis, that's not good. And ultimately, all this cost going back and forth is going to go back into the taxpayer's wallet. We are going to eventually pay for this somehow.

Justin Beals: Yeah. You know, I think back to my days of developing education assessments and even quality assurance work on software development. And there were two terms that we, we really focus on. And one is the reliability of the testing. Like when we rerun it, we get a similar result. And the second is the accuracy. Does the testing represent accurately false positive false negative a true positive true negative what we expected as well and I think to your point about the cost We really need to get this right and the way to get it right is to make the instrument the the assessment that the result both reliable and accurate because if it is and people will trust it and if it isn't then they won't and will have wasted a lot of time and energy

Logan Therrien: And we can't lose the trust.

Justin Beals: No, that's it's what it's designed for, right? So that we can trust who we're working with together and at least raise the bar a little bit. Yeah.

Logan Therrien:Right. And so it's a hard argument for anybody to say we should not do these security requirements because it costs money. But that argument is being had. It is being had at the congressional level that's that's it's a burden upon the supply chain. It's it's the argument that we as C3PO don't have enough folks to do this and we're we're coming up with programs that not we -I didn't come with this program I'm just part of the part of the program- but programs are being developed that is just a burden and we can't support it over time and and so this this Not having a standard format for this this assessment methodology is just not a good data point in their argument

Justin Beals: Yeah.

Yeah, but to me, what hinges in the balance is the ability for DOW and private industry to work together because without some amount of trust or assessment back and forth, and that trust comes from effective assessment at the end of the day, especially as private industry is so involved in nation-state secret and defense that if we wanna keep that going, we're gonna need to otherwise the most vulnerable part of our ability to defend is going to be commercial companies that are supplying it. Yeah.

I'm preaching to the choir a little bit Logan. I'm known for that from time to time. Good.

Logan Therrien: We agree

Justin Beals: Tell me what's next in your, in both this paper and things that you're thinking about from a research perspective as you go forward.

Logan Therrien: So, from the outcome of this paper, I've really, so let me kind of back up. The CyberAB has recognized along with the Department of War that there are a lot of questions. There are a lot of folks with input from both the defense industry, from consultants that are trying to work with OSCs, that are maybe a little bit more vocal,a nd then also from the C3PO as well.

And so in conjunction with both of those organizations, the DOW and the CyberAB, they set up committees, right? So there are committees that are now ingesting that input from the ecosystem and trying to address it, and then provide recommendations to the Department of War and then hopefully get that vetted out into either updated documentation, maybe future rules, a body of knowledge, and CyberAB actively talks about this. I'm part of one of the subcommittees and we actually ended up talking, now I have a method to actually address this. So in that paper, I did mention maybe a way to have some kind of CMMC assessment methodology or a way of doing things, but sometimes, if it's easier and still accomplishable.

That's probably the route I'd like to go. So I was able to provide my feedback into that forum and then have, again, peers in that community evaluate methods of doing this. But there's also, you know, there's also other ways. I think the bigger picture for this, though, is some kind of known sampling methodology that is communicated. is what Logan would like to see, is that there's a known sampling methodology that C3PO's that they can point to and say, " This is how I did it”. And that the OSC understands during and prior to the engagement that that's how they're going to do it. And then again, if something in the future happens, the Department of War knows that's how they did it. And so that's the ultimate outcome, right? Nothing crazy, just everybody communicating on what that standard was. It doesn't have to be the same standard as the next C3PO, but it still has to be understood, communicated, and recreatable.

Logan Therrien: So that's, think, between that input path, and that came after I published my research, so that it worked out well. The other path, again, is the ISO1720. We're gonna start, I think we're already starting to see C3POs work through that accreditation process with ANAP, and to have solidified ways of doing things.

My research that I'm wrapping up right now is for commonly misunderstood CMMC requirements. Again, I need data, right? And this, I think, is good data because I'm also an instructor. So I can take that back and go, here's some of the major obstacles you're going to face with your clients. Here's some of the major obstacles you're going to face as an assessor. How do you get that evidence? So this was a multi-part survey on, I really wanted the top three because it the survey has to end at some point there are so many things we could talk about. So I just wanted the top three. I would most commonly misunderstood and then I gave some opportunities most commonly misunderstood either how they present their evidence or the requirement itself and then opportunities for the respondents to enter data that maybe was not on that list. So I didn't try to make it simple. I got some pretty good data from there. It's just wrapping it up right now. I'm looking to get that published later in the summer.

And my goal is, and so I'm in a, I am in a PhD program, by the way, full disclosure. It's really helping me do this process. I have, you know, other folks that are research professionals that have helped me make sure I do this right so that the research results make sense. They're formatted in a way that is understood by the scientific community and can be used in future use. But my goal with that is, again, that we can make the OSCs and ecosystem better.

We can make assessors better and more prepared for, we are sitting about, the last time I got was maybe 1,300 OSCs have been assessed. So we only have 116,750 left to go over the next few years. So anything that I can do to help the ecosystem through this process, I'm trying my best because I'm just passionate about it, right? I was part of the military. I understand what it is to get parts to need that supply chain, right? I know what deployments delayed look like and the impact. And so that's my why for this process.

Justin Beals: I love it. I think, you know, me coming from the commercial side a little bit, what you're talking about helps them as well. OSCs or companies that are trying to go through the assessment, they just want to know a little bit what they're in store for. I like to tell them that getting into these assessments should feel like an open book test. You really should know what you're in for before you get there. And if you do and you do your homework and you do the work, you'll pass. It'll be okay.

So let's just do the work, and really it'll be successful on the other side.

Logan Therrien: Agree. So I will say that there are resources too, right? And some companies are just limited on the efforts that they can take. Some companies may just say, I want to bring them out, folks, well, what are you doing to train them? Are you putting them through a CCA course, a CCP course? That's about 40 hours per course.

But at the end, they're going to have people they can communicate with. They're going to have resources. They're going to have a body of knowledge that they can walk away from the table with. They're going to have just, I mean, resources is really important when it comes to this. It's constantly changing. There's FAQs being generated by the DOW on a quarterly basis that change things on how we knew it to be the week before. If they decide that they can't support that, or they don't have the time, or they don't want to expand their departments, right?

Am I looking for external support? So I have a consultant that deals with that 40 hours a week for the past three years, right? Do I just, I want to bother with my infrastructure build. I'll just hire an external service provider to do it for myself, right? So that's really evaluating the level of effort and how are they going to get that level of knowledge? I will say on the consulting side, what I commonly see is that they have somebody who's with the words IT associated with their name, who's working a 40 hour week, and that company says, hey, this says cybersecurity.

Unfortunately, only a portion of it, a portion of cybersecurity, there's also physical security. There's also training environments. There's HR, onboarding, right? There's all these other things. And so what typically happens is if the leadership, the executive side of that company doesn't recognize what they're putting onto their IT person, they may have, at the end of the day, someone who has CCP in front of their name, and then goes to get a better job that treats them better and have a better work life restructure. You know, just as an organization, again, understand it is a lot and how are they going to address that? Are they listening to their folks? Are they training their folks? It's important.

Justin Beals: Yeah. that exact experience inspired me to the work I do today because I was like, why am I solving for HR problems? You know, I'm, I'm the Chief Technology Officer and I get that this compliance thing had some IT stuff in there that I do need to solve for, but it is, it is a, I call it a horizontal practice. will touch every part of your organization and you just really need to be aware that it's not an IT managers little perp view. Yeah.

Logan Therrien:And then throw in some data labeling, and now we have legal involved.

Justin Beals: That's right. Yes, exactly. That's good. Logan, this has been incredibly fruitful for me from a conversation perspective, and I am deeply grateful for the work you're doing, both in your own research and at Cyber AB. I think that a healthy environment ecosystem for this change is a part of nation's defense. So we're thankful for joining us today on SecureTalk and thankful for it. I'm personally thankful for the work you're doing. Yeah.

Logan Therrien: Thanks, Justin. I really appreciate you me the time to be here with you today.