Share This Article
My son Charlie is 8. He’s an outgoing kid, loves to play, loves to tinker on devices, he’s me, but smaller. Getting a chance to firm up some first principles is both the scariest part of being a parent – what if they glom on to the wrong thing? – and the best part of being a parent.
He joined Scouts a few years back, and we’ve been going most weeks since then.
There’s a whole curriculum around preserving the outdoors and engaging in conservation.
As part of our weekly meeting, we recite the Outdoor code, and damned if it doesn’t stick with these boys and girls.
The Outdoor code says:
I will do my best to:
- Be clean in my outdoor manners.
- Be careful with fire.
- Be considerate in the outdoors.
- Be conservation-minded.
It’s a helpful reminder to our youngsters that being outdoors means taking responsibility for that space, being good stewards of the world in which we live, and giving the world around you due care and attention.
It’s awesome to watch this take hold with kids, and Charlie and his fellow scouts are slowly but surely becoming good conservators of the outdoors following that code.
I got to thinking recently about what it means to have an IT Code. If you’ve not yet seen my sessions from last summer and fall about having an IT Code of Ethics, and what it means to do good IT in 2022, then I’ll point you at those resources.
What does it mean to do IT “right” in 2022? In a post-pandemic world. In an ongoing pandemic world. In a remote-first world. In an office world.
Overall, I keep coming back to the conversations that I have with Charlie about being responsible stewards of the world in which we live. We talk a lot about what it means to be considerate in the outdoors, and to be careful with fire.
He’s a Bear Scout this year, and that meant working with his den to teach them how to use a pocket knife without ending up in the nearby Emergency Room.
I think that two decades of IT experience gave me a bit of a leg up there, because as much as IT professionals have to spend their time getting their coworkers not to run with metaphorical scissors, I could absolutely handle 10 8-year-olds with pocket knives, right?
Well, no one went to the ER, and the first aid kit got only a bit of use. But all that led me to: “What’s the IT Code for 2022?”
I came up with:
As an IT Professional, I will do my best to:
- Preserve company data from carelessness and compromise
- Protect my people from attack and their occasional inattention
- Defend the integrity of our organization against bad policies, poor security, and short-sightedness.
- Empower my people with knowledge.
This is the approach that we have to take as IT pros, to get the job done in 2022 as well as the start of the post-pandemic, yet still endemic, Covid times. These are principles that should apply to everything IT admins have to face, from interactions with coworkers, creation of systems lifecycles, and adoption of a new tool to data governance, IT audits and policy management.
Today, there’s a really critical thing I want to focus on: macOS patch management.
Patch Management and Your Organization
Staying on the most current version of the operating system is frequently called out as one of the best ways to avoid being breached by an attacker. When it comes to the process of staying patched, I tend to start with recommendations from experts. In the US that means the Cybersecurity & Infrastructure Security Agency, or CISA. In the UK, that would be National Cyber Security Centre, or NCSC, who release the Cyber Essentials security standards for any business that wants to work with a public entity.
CISA in the US has some clear guidelines on where to start:
- Enable automatic software updates whenever possible.
- Do not use unsupported EOL software.
- Always visit vendor sites directly rather than clicking on advertisements or email links.
- Avoid software updates while using untrusted networks.
If we take these one by one, there are some caveats that I can already hear you forming as IT professionals.
Enabling Automatic Software Updates
Automatic updates are great. When they work perfectly. Recent surprises in new versions of macOS have come with challenges related to the removal of the system python framework, and recent versions of applications like Google Chrome have changed codesigning certificates. This can result in unexpected behavior for your admins, and give you some frustration to balance out.
Of course, shortly after that, you may get a zero day in Chrome, and suddenly you’re making many more difficult decisions.
A fleet set to automatically upgrade may come with problems you wish you could work around with some warning.
Enabling automatic updates is definitely a choice you can make as an administrator, but there are tradeoffs. You may find out about bugs after they have reached the entirety of your production fleet, and that may have negative consequences for your department’s reputation in the eyes of your coworkers.
Apple’s own documentation – here I refer to last summer’s session on updating macOS – indicates that admins often need time to evaluate the state of a given update.
You’ll note here that Admin Testing begins during the beta cycle. Yours definitely should, too.
Catching changes that are in the release notes is one thing, but having a working device on the bleeding edge will help you understand the severity of changes, as well as provide necessary feedback, will keep you at least one step ahead of potential disruption.
Do not use End-of-Life (EOL) Software
Apple famously doesn’t declare a timeframe on the end-of-life of its major releases. One could ask: Is macOS Mojave end-of-life?
Well, the update schedule sure says so. The OS still functions, of course, but it hasn’t seen an update in more than 220 days. It’s dead, but Apple has definitely never said so.
Howard Oakley of the blog Eclectic Light Company has some good dissection on the end of life versions of various macOS releases. Since Apple never declares EOL, it only stops updating, and even then not all software updates make it to all release trains of macOS, so admins are left operating by rule of thumb rather than by specific manufacturer-lead declarations.
In addition, some common patch vulnerabilities that have been addressed on macOS Monterey have yet to be released for macOS Big Sur, and likely never will be.
The only thing we’re left to know is this: the most secure version of the Operating System is the latest one, followed by a fully patched version of the previous operating system. Two versions previous may get updates, but there’s no clear rationale behind the updates delivered to the n-2 version, and no clear vulnerability score at which we should expect an n-2 security update.
This leads us to the First Law of Patch Management:
“The second-most secure computer is one that is fully patched. The first is one that’s off and in your desk drawer”
Visiting Vendor Sites Directly, and Avoid Software Updates on Untrusted Networks
When it comes to patching macOS, the only place to get updates is directly through Apple, fortunately. With the system volume being signed & sealed starting in Big Sur, the only way to update the OS is via Software Update or a properly signed package from the App Store.
This is also fairly anodyne when it comes to macOS. Due to codesigning and integrity checking at multiple points throughout the process, Apple updates, while requiring network access, don’t necessarily require highly secured networks to download. Apple covers all of that work for you, thankfully. This advice is substantially less important for Mac Admins.
Why is patching so important?
I’m not here to scare you, because being scared leads to making decisions from fear and not strength. However, things are rough out there. This is the golden age of cyber crime, and it’s your job as an admin to do what you can. The number of vulnerabilities discovered in key software products is on the rise. 2017 and onward has shown a substantial increase in zero-day vulnerabilities that have been found in key software applications and operating systems.
In the States, we have a saying: “If you and your friend are walking through the woods and you come across a bear, you don’t have to be faster than the bear; you just have to be faster than your friend.”
The stakes are high, there’s no question. Worse, more than half of businesses don’t even know where to start in the event of a security problem. So you can see where this is going: where more businesses are being attacked internally and they’re not even sure where to start.
If you’re here to learn about Patch Management, you’re in the segment of businesses that know they need help, and you’ve come to see more about how JumpCloud can help you. Good. That’s a good start.
Before we talk about how JumpCloud’s patching solution works for macOS, we have to talk about how Apple intends for OS Patching to work.
How Does Apple Intend for Patching To Work?
At Worldwide Developers Conference 2021, Apple announced improvements to patching macOS via the Mobile Device Management protocol. This improved method would take advantage of the ScheduleOSUpdate MDM command with a new set of capabilities associated with the MaxDeferrals key. This would allow Mac Admins to send an update command, allow a number of deferrals, and then enforce the update.
Apple is depending on users acting on the small notifications that the operating system provides. They’re depending on their new mechanisms working each time. Unfortunately, there have been challenges with this feature working in the field.
Until the release of macOS 12.3.1, it was difficult for an MDM to understand fully where devices were in the deferral cycle, as well as getting those updates to apply correctly. Of course, if your organization can’t adopt Monterey immediately, or has good reasons to stay at the previous release for stability, none of these new features will work for you. You still need to stay up to date, though, so what are you supposed to do?
How Should Patching for macOS Actually Work?
There’s no denying that a mobile device management solution is required for this part of the admin’s job. You cannot manage Macs effectively without the MDM layer, and that’s been true for nearly half a decade at this point. Some profiles – especially those that delay the rollout of a given operating system release – require MDM in order to be installed. In addition, the commands that can act as a forcing function for upgrades require an MDM to deliver. There’s no question that MDM is the right way to approach this.
In addition to being driven by your MDM, admins need to push for user-driven updates. Your coworkers should get a say, if not the final one, in the patching of their devices. Updates are disruptive, lengthy, and not risk-free, so letting your users be in charge of the timing is important in getting your coworkers’ buy-in for a given update. That’s not to say they should be given the ability to skip necessary updates; your organization’s security policy still has to be given primary focus, and coworkers who fail to abide by those policies should face organizational consequences.
With these bounds set, we can start to look for a solution to handle patching macOS that aligns with our core principles:
- It needs to include a user’s consent
- It needs to be schedulable
- It needs to support delays; and, most importantly
- It needs to support automation
If you have to readjust the update policy every time Apple releases something new, you’re going to be spending a lot of time adjusting when you could just be testing the release.
More than what we’ve discussed thus far, though, your organizational culture needs to be tolerant of update delays when they occur. If someone’s late to a meeting because they’re applying the latest update, thank them for looking out for your company’s security rather than scolding them and thus sending the mixed message that they shouldn’t be doing these updates during the workday. Caring about security means making room for people to patch their devices.
Patching in Rings: Making Processes That Work
Two choices that you have to make as an IT admin is when to patch, and who goes first. You could decide that the most current operating system is where everyone belongs, all the time, and you accept the risk of deploying software that doesn’t operate properly. You could decide to delay everyone for at least 90 days to prevent messing with software that doesn’t operate properly, and accept that a missed security flaw could be exploited and allow an attacker to gain control of some or all of your devices.
It’s not a pleasant dilemma, is it? Do you prioritize security, or do you prioritize operations?
There’s an answer here, and it’s all about tiered access. We take our model from the Linux Kernel and protection rings. There’s a highly privileged group of skilled users who can handle some small amount of inconvenience and who have immense systems knowledge, and we can start there. That’s you and your team, as the IT admins. From there, we can broaden the approach and pick key users of line-of-business software packages that perhaps IT doesn’t have the specialized knowledge to test. Once they give the all clear, you can move to the vast majority of systems that represent general adoption of the new updates. And last but not least, you can patch the trailing devices who maybe shouldn’t be updated with rapidity.
Using a ring model will not only allow you to expose your organization to the new updates without risking your whole team’s productivity, but it is a great way to test the validity and functionality of software updates without – and I’m using a technical term here – YOLO’ing new software to production. Good testing rubrics will give you a pathway to success regarding the functionality of any given update. Building a list of tasks that your expert users can test for you, and that tests the bounds of line-of-business software activities, will help you identify which updates are safe to apply, which may require patches of other apps, and which aren’t ready for public distribution.
Making a strong written procedure guide is a great way to ensure that updates get reviewed in a timely manner, and that your organization is protected not just against security vulnerabilities, but also against problematic updates damaging your workflows.
Returning to the Code
At the beginning of this I talked about creating an IT Code for organizations to follow. Protecting your coworkers and your company is a delicate art, but it’s a necessary one. Stewarding your organization’s values into production policies will give you a chance to show your coworkers that you’re doing everything you can to keep them safe and balance their work experience with the operating conditions and security needs of your company.
Providing a good experience for patch management will simultaneously make your devices more secure and make your coworkers appreciate your attention to detail and execution. Treating their schedules with respect, as well as instilling the internal values related to patch management’s requirements, is an important thing for any systems admin to do.
Building your values into your policies is a cornerstone to managing IT well. It’s not something that some IT operations think about much, however. Blind adherence to individuated policies that are developed in a vacuum is a good way to run a department that your coworkers will do anything not to engage with. Running a human-centric IT department, however, pays dividends in other ways, such as enhanced trust, better cooperation, better communications, and more frictionless IT experiences. Which is why I’ll close with a good set of guideposts to get you there:
As an IT Professional, I will do my best to:
- Preserve company data from carelessness and compromise
- Protect my people from attack and their occasional inattention
- Defend the integrity of our organization against bad policies, poor security, and short-sightedness.
- Empower your people with knowledge.
Great experiences make IT a seamless experience, which everyone will appreciate.
