The barrier to entry for machine learning experiments is pretty high – to run any model yourself, let alone train a model, one needs a decently modern CPU, ideally a CUDA-capable graphics card with at least 8GB of RAM, oodles of disk space and oodles of RAM (64GB is generally considered a minimum starting point). This kind of hardware should be sufficient to run large language models (LLMs) either from the command line or using one of a handful of web-server based front-ends such as [Oobabooga]. Whilst these needs can be met via online cloud providers, and indeed there are even brokerages that seek to link those with spare processor cycles and end-users, it all comes at a hefty cost and there are significant disadvantages with cloud based solutions given the size of machine learning models that we’ll be shuffling across the intertubes. So instead I decided to explore a less conventional route, and look for an ex-enterprise server that I could run from home.

Selecting the right server: A false start

My first try was a false start. An HP DL380 Gen 8, which had a couple of E5-2609 Xeon processors and 64GB of RAM for £75.

The immaculate hardware spun up perfectly (I love the way ex-data centre gear is always so clean and dust-free), and I opted to install Proxmox on an SSD to give me some flexibility in experimenting with multiple alternative configurations as either containers or virtual machines on the physical host. However, I rapidly came to realise the shortcomings of this configuration – much of the open source LLM related software requires additions to the CPU instruction set that only appeared relatively recently. It is possible to upgrade the CPUs, I think there is only one model that the server supports that has the elusive AVX2 support (E5-2690 v2 – the v2 bit is critical by the way), but given that 64GB of RAM was also clearly going to be a limiting factor, I decided to cut my losses and shop around again

Second Time Lucky?

Trawling the world’s largest online bootsale, I lucked out with a DL380 Gen 9 with a monster spec.: 180GB RAM, a pair of Xeon processors totalling 48 cores, modern enough to support AVX2 instructions, and a bonus dual port 10GB network card, all for the princely sum of £220.

As is common for these kind of purchases, the server came with no disks or caddies. Whilst caddies are readily available, I opted for a quick 3D print instead . Now, 2U form factor servers have a bit of a reputation for being noisy, but I’ve found that when using SSDs (nothing special, normal consumer grade SATA disks) in preference to spinning rust, the server is essentially silent.

I’ve decided to run with Proxmox to give me flexibility in terms of run-time OS environments, and to allow for throw-away experiments on VMs. Once I have a bit of a handle on the basics, I’ll think about adding a GPU or two.