My Homelab! Pt. 3

My personal touches

Being a homelab, there are some parts of my server rack that you likely won’t find in a commercial application. The goal of most commercial operations is that the server, rack, networking equipment, are all commodities. Easily replaced, with a large pool of talent who knows how to replace it. My rack isn’t like that. The big box that sits behind me is full of “temporary” solutions. She’s customized to my particular use case, in the particular location that it sits in. Any change to any of this disrupts the delicate balance of function that I’ve set up.

The actual server rack itself is an old NetShelter from APC. I’m not sure it’s exact age, but it’s old enough for the foam to be falling apart if you touch it wrong. It’s heavily insulated on the inside with strategically placed vents that baffle the sound before it can escape. I got this on what I consider to be a sweet deal, $550 CAD, I just has to pick it up. This was just post COVID and offices were closing up their leases and getting rid of their old equipment, including their in-house server racks. I had to disassemble the rack at their office, load it into my Honda Civic, then clean the hell out of when I got home. The rack looked like it hadn’t been dusted in a decade. If you’re not running the rack in a clean air environment (an office space is most definitely not a clean air environment) then regular dust removable is a must. Dust is the enemy of heat transfer and airflow, two things desperately needed in a server rack. The only mistake I made in this process is not cleaning it out before getting it into my office. Made one hell of a mess.

?? Insulated Rack ?? Overkill much?

I know some people are able to sit in noisy environments all day, but having the equivalent of a hairdryer running behind you 24/7 is not my idea of a good time. Purchasing this rack was a requirement for running Heather, as boy does she ever scream. If you let Heather run the stock fan curve with the rack doors open, you can’t talk over her, even in the next room.

If you’re wondering why I keep an elderly white woman in my server rack, you should read the first few posts in this series! Heather is my Dell R730XD LFF.

The NetShelter was a must have, or I’d have to downsize my hardware (haha like that was an option). With the doors closed and the fan curve tuned, Heather is manageable. She still screams like a banshee from time to time, but only when she’s actually doing some work. The rest of the time her wails are contained by that sweet sweet polyethylene foam. With the front doors of the rack closed, the hum of the servers, the networking gear, and my desktop (it also lives in the rack) is barely audible. One of the things it isn’t though, is cold. In my last post I included a picture of my Home Assistant page that tracks the temperature of the rack, and at idle, without any sunlight the rack is 35C (95F). That is the coldest it typically gets. On a winter day, with direct sunlight, running a workload that hits all servers + desktop, I have to open the front doors of the rack.

Yes I did say winter. My condo doesn’t have AC during winter, but does have a lot of windows. In the winter we have to keep some windows partially open otherwise it can get easily get to 28-30C.

I have yet to install the front fans that would push air into rack. Without those, the interior will easily hit 45-50C (113-122F). That is no bueno for the servers. Especially Martha who’s GPU is in a less than ideal airflow configuration. That temperature means excessive thermal throttling as soon as the cooler’s thermal mass is saturated. To keep that from happening I open the front doors, put on noise cancelling headphones, and let my fan curve algo go to work.

There is a custom algorithm that sets the fan speed of the exhaust fan, which pulls from the rear of the rack and exhausts out the window. In the summer on a sunny day, when I’m working on an AI or other heavy GPU project the rack can easily reach 40-45 degrees, even with the fans running full bore. This is the upgraded set up after I realized that the reason some of my GPU jobs were dying was due to the GPU overheating. My next step here is

The noise

I mentioned the volume previously, but let’s talk numbers. These readings are taken with my phone sitting at my desk, from some app that was free and seemed to do that job, so give them a grain of salt. I’ve managed to get the noise down to 44dB when the door to the rack is closed, it’s cloudy, and there isn’t a huge load on the servers. That’s about the equivalent of standing next to your fridge when it’s running. It’s not unpleasant, especially as I’m normally playing music over my speakers and they drown out the fans anyway. When there is any load or some sun, the rack starts to make some noise. With the exhaust fan running full tilt I can read about 54 dB - the equivalent of a a semi-heavy rainfall on your roof. This is when I really start to notice it, and am conscious something is doing some work in there. The real threshold for me though is when I get an alert that the temperature has reached the “warning” level. I have this set at 40C at which point my Galatic Unicorn lights up and lets me know I should open the rack doors and dump heat into the room. Opening the doors really jumps the noise floor, especially as that’s the time when all the fans are running full bore. This is when it hits 65 dB, an annoying loud sound that I can hear over top my music. It makes being in the same room as the rack uncomfortable, and would be impossible to take a meeting with that in the background. If you’re going to start a homelab of your own, you should think about where the rack will sit and the noise and heat it will produce.

Comparing these readings to my lived experiences make me think the readings too low. For example,

The heat

Small spaces heat up quick. Small spaces where one wall is a west face

One thing that the decibel readings miss though is it’s important to know whether you’re particularly sensitive to a given frequency, and tune away from that. My exhaust fan is relatively high quality, and doesn’t produce higher pitched whines. The original fans I had in my network switch did have an incredibly high pitched whine that bothered me even at the lowest volume. I had to open the switch up and replace the fan in order to bring it to a reasonable level.