Homelab

Work in progress. The rack layout, servers and pools are still sample values.

This is my prod lab: the hardware I actually depend on day to day. I have plenty of other servers and drives for testing and tinkering, but they don’t make this list. Experiments get their own projects, and land in the changelog here once they go into service.

Now

Almost everything runs on a single, beefy Proxmox host. It manages the ZFS pools directly.

42

Patch panel24p Cat6 keystone

42

41

Switch24p PoE+ · 2× 10G SFP+

41

13

FirewallOPNsense

13

12

brush panel

12

11

Pi shelf4× Pi 5

11

10090807

JBOD36 bay

10090807

0504

Proxmox

0504

030201

UPS

030201

Servers

Storage

Most of the spinning drives are Seagate Exos, in a mix of sizes. The main SSD arrays are built from SAMSUNG MZILS3T8HCJM 3.84TB PM1633 12Gbps SAS 2.5” SSDs.

Power

A 3U UPS sits at the bottom of the rack, and the rack has its own dedicated 240 V, 30 A circuit.

Notes

• LXC containers get the ZFS pools through bind mounts. VMs mount them over NFS.
• Pool usage comes from zfs list -o name,used,avail. zpool list and the Proxmox ZFS panel show raw space, parity included.

Changelog

2024-2026: down to one host

Energy prices in my area kept climbing, so I spent a while consolidating a lot of dedicated servers into as few as possible, ending up with one Proxmox host. My dedicated TrueNAS box went too: its ZFS pools moved over to the Proxmox host, which now manages them directly.

I could cut the power draw further with a newer motherboard, CPU and RAM, but most of it actually comes from the hard drives spinning 24/7. Right now new hardware would cost more than it would save.

2022-now: the 42U rack

Back to a full-size 42U rack, this time on its own dedicated 240 V, 30 A circuit. At its peak, a good amount of it was filled.

The old 26U rack is still around. It holds servers I don’t actively use, which is a good way to store them without taking up more space.

2020-12: four servers and SnapRAID

From my post in HardForum’s storage showoff thread. Four servers by now, all on 10 Gb networking:

• FreeNAS, for my important data: Intel Xeon E5-2630 v3 @ 2.40GHz, 64GB memory, and a ZFS array of 8 x 8TB drives in mirrored pairs.
• Proxmox: Intel Xeon E3-1225 v6 @ 3.30GHz, 32GB memory, and a ZFS array of 4x 1TB SSDs in mirrored pairs.
• Proxmox: 2x Intel Xeon X5690 @ 3.47GHz, 96GB memory, and a ZFS array of 4x 1TB SSDs in mirrored pairs.
• OpenMediaVault, for media storage, playback and downloading: a Chenbro NR40700 4U 48-bay (top load) with an Intel i5-9600K @ 3.70GHz, a Quadro P2000 for transcoding, 32GB memory, a 100GB SSD for the OS, and 2x 1TB SSDs in a ZFS mirror for app data.

Both Proxmox servers ran a mix of LXCs and VMs, depending on the workload. The main media array was SnapRAID and MergerFS over 19x 16TB, 3x 10TB and 3x 8TB drives, 358 TB raw. On top of that I had 24x 4TB drives and a handful of 6TB drives waiting for something to do.

2014-2021: the 26U rack

I picked this rack up in early fall 2014 from someone who was getting rid of their home setup. It has a glass door, sound damping, and exhaust fans for cooling. It’s only 26U, but it weighs well over 100 lb empty, so it had to be fully taken apart every time I moved, starting with the move to another NYC apartment in January 2015. I still have it.

This is when I really ramped up. By then I was all in on Supermicro chassis, which made life easy: the drive sleds are all the same, so drives can move between servers as needed. The large storage arrays started here, along with real clusters to run storage and apps.

I’d started building clusters around 2015, and over these years I built and ran most of the usual ones: Proxmox clusters, Kubernetes, Ceph and Docker Swarms, as well as some dedicated SAN servers (I forget what those ran). I like overkill, but for a personal homelab even that was too much. They were fun to build and I learned a lot from each one. They were also completely unnecessary for what I actually run.

The numbers I have from this time come from old HardForum posts: the two below, from when the rack was new, and the 2020-12 one above. The years in between went mostly unrecorded.

2015-01: one pool with Drive Bender

From my post in the newer HardForum showoff thread, five days after the last one and in a new NYC apartment. Same server, reshuffled: 87T total, 63T of it in the one Supermicro case.

What changed from 2014-12:

• Memory: 24GB DDR3 1333 Ram
• Boot: the OS moved to a Samsung 850 250GB SSD
• Drives: 6 x Samsung Spinpoint F4EG 2T (HD204UI), 9 x Seagate Barracuda 3T (ST3000DM001) and 12 x WD Red 4T (WD40EFRX)
• Pooling: Drive Bender combined arrays 1 to 3 into a single pool, so expanding meant adding a new array to the pool.
• ESXi: a 2U Norco case became my ESXi server.

Array	Drives	Chassis
1, RAID 6	6 x 2T Samsung (HD204UI)	SC846E16
2, RAID 6	8 x 3T Seagate (ST3000DM001)	middle SE3016
3, RAID 6	12 x 4T WD Red (WD40EFRX)	SC846E16
Downloads	1 x 3T Seagate (ST3000DM001)	SC846E16

The bottom SE3016 was empty for now. I also kept an unused 2T, 3T and 4T drive on hand to replace any that failed in the arrays.

The post also had a video of the noise difference, but it was on vid.me, which is long gone.

2014-12: 86 TB in a 26U rack

From my post in HardForum’s storage showoff thread. One Windows Server 2012 box ran Plex and my Usenet downloads, with two SAS expander chassis for more drives: 86T raw, 59.073T usable. I posted it in December, but most of this setup dates from early 2014.

• Chassis: Supermicro SC846E16 & 2 x SE3016 SAS expanders
• Power: 2 x SuperMicro 900 Watt
• Board: Supermicro X8DTE-F
• CPU: 2 x Xeon 5570 @ 2.93GHz
• Memory: 4 x G.SKILL Ripjaws 4Gb sticks
• RAID: Adaptec 5805 HW Raid card
• UPS: Tripp Lite SMART1500LCDT 1500VA 900W UPS

The drives were split into three RAID 6 arrays, plus a RAID 1 to download to:

Array	Drives	Chassis
1, RAID 6	8 x Samsung Spinpoint F4EG 2T (HD204UI)	SC846E16
2, RAID 6	8 x Seagate Barracuda 3T (ST3000DM001)	middle SE3016
3, RAID 6	10 x WD Red 4T (WD40EFRX)	middle SE3016
RAID 1	2 x Seagate Barracuda 3T (ST3000DM001)	bottom SE3016

That small gray box on top of the rack is the WD ShareSpace from 2008, my first NAS. It was retired by then, but I kept it since it was my first. The closet behind the rack is where the DIY shelf rack lived.

2014: the closet rack

In early 2014, I moved into an apartment in NYC. The 42U rack stayed in my friend’s basement, not by choice: it physically wouldn’t fit.

The closet was weirdly deep and very narrow, so I built my own “rack”: a plywood base with casters bolted on, and one of those metal shelving units with wood shelves. The servers just sat on top of each other, and the whole thing rolled in and out of the closet. Sometimes you make do with what you have.

I ran that until early fall 2014, when I got a real rack again.

2009-2013: college

It didn’t take long before I had a 42U open-frame, four-post rack in my dorm room, with a few enterprise servers in it, some kind of Dell R-series I think. My roommate and I were both getting into self-hosting, and we each had our own servers in it. Did we need a 42U rack? No, but it was cheap, and light since it was open-frame. And yes, that means enterprise servers running almost 24/7 in a dorm room. It kept causing problems for the college’s network team; let’s just say we got to know each other pretty well over those years.

I also started getting into hardware RAID controllers, and we picked up some Rackable SE3016s: 3U, 16-bay JBODs that ran over SAS-1 links and could be daisy-chained together. Two of them ended up in the 26U rack, under the Supermicro in the 2014 photos. I think I still have those chassis.

Toward the end of college, my friend got a house, and we moved to two proper 42U racks with sides and doors in his basement. By then we each had a handful of servers and a few storage boxes.

2008: the first NAS

Before this, my data lived on a few external drives and random internal drives, swapped through eSATA and USB docks and a few internal bays. No backups, no RAID. Then a full 640 GB drive died. Nothing that couldn’t be “replaced”, thankfully, but at the internet speeds of the time it still hurt a lot.

At the time, a few WD 640 GB drives and a 750 GB Seagate Barracuda held all of my data. That was mostly “ripped DVDs”, plus a collection of rainbow tables I generated myself on a small cluster of old desktops back in the early 2000s. For educational purposes, of course.

So I started looking into better ways to store data, and learning about the different types of RAID. My first dedicated NAS was a WD WDA4NC40000N 4TB ShareSpace Network Storage, bought when it came out in 2008. It shipped with four 1 TB WD Caviar Green drives. About the only bigger drive you could buy was Seagate’s 1.5 TB Barracuda, which had come out a couple of months earlier. It retailed for $999.99.

I was already deep into computers by then, but if the homelab has a day one, it’s this NAS. It came with me to college as my first real RAID storage device, and it all grew from there.