Rebecca's Journal
Zero Clients - Upgrade Automation

While working in government, I helped deploy the VMware Horizon Desktop and Application delivery platform. This is a bit of a look back on some issues we had with our zero clients and management.

The Firmware Frustration

Firmware was an absolute nightmare. Well, only partially. I can’t remember the exact firmware version we started on, but I want to say it was like 3.5 or something like that. These devices were designed to be managed by a management console, provided by a virtual appliance installed on our vCenter infrastructure. There was one problem - we were being encouraged to use the new version of the firmware (I think this was 4.0) - but it was incompatible with the old management console. We were running a small configuration of these clients, so we just deployed the new management virtual appliance, applied the upgrade through the old one, then there it goes, it’s in the new console. Alright, that was wonderful. We decommissioned the old management console and called it a week. We were certainly glad that at this point new clients would come with new firmware.

But, they didn’t.

The Deploy

We had about 10-15 of these clients already in inventory, for testing purposes, giving some of our users that we trust with new technology a chance to try out what we’ve been messing with. They were wonderfully happy with the implementation as we had, so we decided to go full deploy. Our deployment plan was small at first, including mostly publicly-accessible workstations, as well as a few systems that we considered “high risk” - the data on their PCs should be stored on a safe central location, and not on a tower that’s out in the open.

We ordered about 40 more of these clients. I was expecting to just connect them to the LAN, and they’d see my DNS entries that would point them right at the configuration server, they’d pull configuration and basically be a whole hands-off ordeal. I’d swap a PC with one, and no configuration required it just works.

Nope.

Remember me mentioning old firmware? Turns out, every single one of the 40 we just ordered came with old firmware. They weren’t compatible with our management tool. Uh oh. I didn’t want to go through the process of re-configuring the old management appliance just to push firmware updates. So I thought about it.

Wait. These things have a web interface that I can upload firmware packages to.

That’s it!

A Python Adventure

I spent a good half the day trying to figure out how I would automatically upgrade these clients in bulk. I knew it had to do with me uploading that firmware package to the client, but how would I do that? I was thinking, kinda reading over my inventory list having already enrolled these clients, and their MAC addresses into our asset management software. Then it hit me - These all have the same 3 bytes in their MAC address. Every single one of them. It got me thinking.

I realized that due to routing restrictions we had, I couldn’t actually resolve the MAC addresses anywhere on the network. But, I did have a testing VLAN that was available on a switch that sat off in the corner of my workspace. That testing VLAN even had a DHCP scope. 254 shiny addresses for my own use. Here’s how it went, at least in my mind before I implemented it.

I quickly stuck a network cable into my workstation and plugged it into the test VLAN switch off to my side. I hooked up a few zero clients, then went through the first two steps of my process manually. It worked! I could see the clients, and get their IP addresses without having to look at the leases on my DHCP server!

Alright, I knew that worked. Now, updates. I messed with the ones I had connected, eventually figuring out how to use cURL to push the package as needed. I bundled it all together into a neat little script, plugged in all of the remaining clients into my switch in my workspace, had a disastrous mess of power strips to make sure they were all plugged in and I ran my little script. Fingers crossed. It took roughly 10 minutes to push a firmware package and for the client to reboot afterwards. I made sure to parallelize this or I would’ve been sitting at my desk until late, wasting time when I could’ve made it just all go at once and take about 10 minutes.

I go off on my lunch break, I had a wonderful curry I had made the night before. I brought the container back to my desk and started to sift through my ticket queue. I kinda got a bit sidetracked, and ended up spending about 3 hours on ticket related stuff. It was nearly time to go home for the day, I didn’t want to stay in the office late troubleshooting these clients, and the next step required my co-worker to set up some routing rules - he left about 2 hours ago. Also, it would be a wonderful thing to demonstrate at our team meeting the following morning.

I came back in the morning, a bit early, hoping to solve any issues with the script before our meeting. Turns out, there weren’t any, and all of those little clients had been upgraded to the new firmware. I let out a giant sigh of relief, and started putting them back in their storage boxes.

I get into the meeting that morning, and my co-worker, who I had asked to contact the vendor to ask about why these came with old firmware on them, addressed me and told me that the vendor had just released a new version of the hardware that used the new firmware. Wonderful! But, a little bit bittersweet. I made a neat little tool to get around the requirement for the management interface we were provided, and was told that it would likely not be needed again.

I still documented it and uploaded it to our IT resources folder, in case future clients need it. I made sure the deployment went over well, and then went home for the weekend.

Some Takeaways

Don’t assume that just because a firmware package has been out for years that the hardware you order will be shipped with it. I made that mistake and ended up spending my days anxious about upgrading them all. Getting 40 clients that need upgrades that I had been doing manually was a little bit daunting. Imagine if it was hundreds. That would’ve been even more stressful.

Have a way to automate these devices, it’s a simple thing to do, but just write it, document it, and stash it somewhere in case you ever need it.