Ian Webster

TrueNAS - Home Server

2023-12-16

Hardware

Note

This post assumes some rudimentary knowledge about computer hardware, servers, and TrueNAS!

Motivation

Short motivation - Project!

I had a Dell PowerEdge T330 lying around, and I wanted to build something! I had also been thinking about building a backup server for my family's huge collection of family videos on DVD and even VCR tapes.

Long motivation - Backups

There is a common phrase for the ideal backup strategy called '3-2-1':

keep 1 original copy of the data on your phone or computer,
a second copy on a separate backup device (e.g. external hard drive), and
a third copy residing on an off-site backup, e.g. Google Cloud or your friend's server

With more redundancy, the less likely it will be that you lose your most important data, such as thousands of photos of friends and family, or important documents.

With a server lying around, it made perfect sense to create a backup server for our family's humongous library of home DVDs and tapes.

Research

What are the Dell PowerEdge T330 Specs?

Nothing amazing, but good enough for a backup server:

Intel(R) Xeon(R) CPU E3-1220 v5 @ 3.00GHz (4 cores, 4 threads)
16GB DDR4 ECC UDIMM, 2R, 2400MT/s RAM
8x Hot-swap 3.5" or 2.5" drive bays
PERC H730 SAS/SATA Hardware RAID Controller
Broadcom® 5720 Network Controller with Dual 1G Ethernet

Home Server Requirements

Let's go over some requirements for the server.

Objectives

Should be cost-effective (although this is much more expensive than the NextCloudPi I'm running on a Raspberry Pi 3B+)
Should have 4TB space, expandable to 16TB
Should have a boot drive with at least 16GB of swap space
Should have data redundancy (at least RAID1 or Mirrored drives)
Should have power redundancy (battery backup for power failures)
Should have a GUI that is relatively easy to manage, in case I need to remotely aid my family in fixing things!

Constraints

Must have read and write file sharing for Mac and PC (e.g. SMB shares and/or NextCloud)
Must be able to stream home movies to Apple TV (e.g. via Jellyfin or Plex streaming clients)
Must be able to access files and home movies from home intranet via Mac, PC, iPhone, and Android (e.g. via the "connect to server" option in the Files app on iOS; Finder on Mac; File Explorer on Windows)

What is TrueNAS?

TrueNAS is an open-source operating system used to manage network-attached storage (NAS) servers. It is based on FreeBSD and the OpenZFS filesystem.

Why TrueNAS? Why not Proxmox, ESXi, UnRAID, or OpenMediaVault?

Proxmox and ESXi are good hypervisor options, but I didn't want this extra complication for a simple home server.
UnRAID is similar to TrueNAS, but it is paid.
OpenMediaVault looks to be a decent solution! I have not tried it.
I have used FreeNAS (the predecessor to TrueNAS) and I enjoyed the experience, so I wanted to try TrueNAS.

Does TrueNAS work well on the Dell PowerEdge T330?

To answer this, we have to understand how OpenZFS, the file system for TrueNAS, works. ZFS has features such as pooled storage and snapshots. ZFS can also configure software RAID configurations, such as RAID-Z (akin to RAID5). There have been many TrueNAS community posts advising against using hardware RAID controllers with ZFS, since this could obfuscate the status of hard drives to ZFS software. The PowerEdge T330 I have is configured with a PERC H730 Hardware RAID Controller attached to an 8-drive hot-swap SATA/SAS Backplane. There's our problem: how will I take advantage of this backplane if TrueNAS doesn't play nice with Hardware RAID Controllers?

Can't I connect drives to the motherboard instead?

If I want to take advantage of all the drive bays, I need to use the backplane and therefore the PERC H730. Without the backplane, I can only attach 2 SATA drives to the motherboard alone. This would not support my requirement to have 1 boot drive and 2 mirrored drives.

What about buying a non-RAID SATA card?

If I'm going to keep cost low, I do not want to have to purchase a new RAID card. To increase compliancy with ZFS, it is advised to use drives connected in HBA (host bus adapter) mode. The PERC H730 is capable of HBA mode. Some TrueNAS forum posts suggest buying a PERC H330 controller (around $100 used) and "cross-flashing" it to "IT-mode" to enable a true HBA mode. With a cursory glance, the H330 and H730 seem to be built from the same DNA, so the H730 should be fine. The next section addresses how to enable HBA mode on the H730.

Some TrueNAS forum posts suggests against using the H730, but my testing found it to be sufficiently stable.

Here are some additional posts to read, if you're interested:

Addressing the Hardware RAID Issue: Changing the PERC730 to HBA mode from the BIOS

There are two ways to enable HBA mode: 1) "IT-Mode", where you can cross-flash your H730 to IT-mode using LSI firmware, or 2) "BIOS HBA Mode", where you can enable HBA mode using the Device Settings in the BIOS. I decided to avoid cross-flashing as this TrueNAS post suggests that I would have to remove the H730's battery. Instead, I went with option 2), and enabled HBA mode using the BIOS. This took a few attempts since I got a warning message the first couple times.

This guide worked:

# 1. Boot into BIOS using the F2 key
# 2. Navigate to Device Settings > Integrated RAID Controller
# 3. Clear config
Configuration Management -> Clear Config -> Confirm -> Yes

# 4. Clear events
Controller Management -> Advanced Controller Management -> Clear Controller Events -> OK

# 5. Reset to defaults
Controller Management -> Advanced Controller Management -> Clear Controller Events -> Set Factory Default -> Set Factory Defaults -> confirm -> OK

# 6. Switch to HBA mode
Controller Management -> Advanced Controller Management -> Switch to HBA Mode you should be then ready to go.

Below is a summary of the aforementioned alternatives in addressing the Hardware RAID Issue with TrueNAS:

Summary Table of Alternatives

Rank	Alternative	Pros	Cons
1	H730 in BIOS HBA mode	No issues with ZFS, if tested properly	Requires stability tests
2	H730 Cross-flashed	ZFS may prefer this over BIOS HBA mode	Cross-flashing is risky. Can restore original H730 firmware using the tool, but the battery needs to be removed!
3	H310 Non-RAID SAS2 controller Cross-flashed	No issues with ZFS	Out of budget. $100 CAD
4	SATA-connected drives to Motherboard	No issues with ZFS	Can't use 8 drive bays. Must use internal drive bays. Only 2 supported SATA ports on motherboard.
5	H730 in RAID Mode	No changes to system	ZFS and hardware RAID could be bad - hardware RAID obfuscates the "true" status of physical drives from ZFS in TrueNAS. In the event of a failure, as in the H730 controller dies, it could be harder to recover the data than if the config were software raid.

Risks with BIOS HBA Mode?

Although it is possible that the H730 could still be writing to the drives in a proprietary format as determined by Dell, I accept the risk because:

I will only store backups of data on the drives
I will mirror the data
I have run failure scenarios and recovery testing, and was satisfied with the results

In the next section, I outline failure testing for the BIOS HBA mode, which suggested to me that the PERC H730 does not write to the disks in a proprietary format in HBA mode. (If I wanted to be 100% certain, I could connect the drives via a SATA to USB bridge to a different computer with ZFS support.)

Failure Testing the BIOS HBA mode

There are some posts suggesting that in the event of a drive failure, data recovery may be difficult using a RAID controller:

JBOD (just a bunch of disks) or HBA mode may still make recovery difficult, and ZFS may be told lies about what's really on the disk

I ran a test to recover data in the event that the RAID controller dies but the rest of the T330 server is still working.

TrueNAS Test Install Configuration

First I installed TrueNAS on the server. Skip to the next section to see the process I took for the final install.

Here is the TrueNAS configuration I used for testing:

All drives are connected to the H730 in BIOS HBA mode
70GB SATA boot drive
1 Pool - called "Mirror Pool" - comprised of 1 VDEV with:
2x 70GB SATA HDDs in Mirror (effectively RAID1) configuration
1 dataset with default permissions in the pool
SMB share setup to share the 1 dataset

I connected to the SMB share and copied over about 70GB of photos to the drives. Since the drives are in a mirror configuration, the files are exist on both drives.

OpenZFS Setup on Ubuntu for Testing

Next, I wanted to try removing one of the drives to access it on my Mac with a SATA to USB bridge. I connected one of the Mirrored 70GB drives from the T330 to my computer. Unfortunately, the setup (installer and homebrew cask) did not work, and after turning on the required kernel extensions in macOS, I decided to pivot. I instead installed Ubuntu Server on a spare drive in the T330, and installed openzfs there instead, see below:

Installing openzfs on Ubuntu Server and accessing TrueNAS pools

The following code block is from https://svennd.be/mount-unknown-filesystem-type-zfs_member/.

# After installing Ubuntu server from usb drive:
# Boot into Unbuntu Server

# install openzfs
sudo apt install zfsutils-linux

# check zpool
sudo zpool import

# mount disk with zpool
sudo zpool import "your pool name" -f

# find mountpoint
zfs get all "your pool name" | grep mountpoint

# mount
zfs mount "your pool name"

Test Scenarios

The following tests showed that I was able to access the pool I created on TrueNAS from a different OS, with and without the H730. I tested 3 scenarios: Scenario 1 assumes that no drives have failed in the mirror; scenarios 2 and 3 assume that 1 drive in the mirror has failed; and scenario 3 assumes that the H730 controller has failed.

Scenarios:

Accessing the Mirror Pool via the H730
With 1 of the Mirror Pool drives removed, and accessing the other drive via the H730
With 1 of the Mirror Pool drives removed from the H730, and accessing the other drive attached to the T330's motherboard via SATA

All scenarios yielded accessible original data. I was satisfied with this result, and moved on to a final install of TrueNAS.

Final Install for TrueNAS Scale on Dell PowerEdge T330

The following section outlines the OS I chose (TrueNAS Core vs TrueNAS Scale), the hardware configuration, and the software I chose to run on the server to satisfy the project requirements.

Choosing an OS Flavour

A little experimenting with TrueNAS Core revealed that the plugin support was lacking. I could not get Jellyfin (video streaming) working. I decided to try TrueNAS Scale's apps, which felt more polished compared to TrueNAS Core's plugins.

Hardware Configuration

Dual 1 gigabit connections (only into a gigabit switch; not directly to router, so may not be aggregate speeds)
1x 240GB Intel 540 Series SATA SSD (~500MB/s read) for Boot
2x 4TB Seagate IronWolf SATA HDD (5900 RPM 64MB cache) for Storage
All drives are connected to the H730 in BIOS HBA mode

Battery Backup (UPS) Configuration

Cyberpower UPS for Battery Backup (BRG1500AVRLCD)

To prevent data loss in the event of an unexpected power failure, I am using a battery backup or UPS (Uninterruptible Power Supply). Upon power failure, such as when a storm knocks down power lines, the UPS will keep the server powered and it will send a signal to TrueNAS to shut down the server safely. TrueNAS uses NUT to interface with the UPS.

To setup my battery backup, I connected it to the T330 via USB. Next, from the truenas.local web interface, I navigated to System Settings>Shell. I entered the commands below, as referenced from this guide:

# 1. List UPS connected
sudo nut-scanner -U

# Output:
Scanning USB bus.
[nutdev1]
        driver = "usbhid-ups"
        port = "auto"
        vendorid = "0764"
        productid = "0501"
        product = "BRG1500AVRLCD"
        serial = "xxxxxxx"
        vendor = "CPS"
        bus = "001"

# 2. Configure UPS Service: System Settings>Services>UPS>Configure

# Used defaults but added the following:
Driver: Cyber Power Systems ups 3 CP1500EPFCLCD USB (usbhid-ups)
Shutdown Command: /sbin/shutdown -h
Power Off UPS: True
# Left everything else as in this guide mentioned earlier.

# 3. Start UPS Service: System Settings>Services>UPS>Toggle to "Running"

# 4. Test the connection to the UPS
upsc ups@localhost

Now I can rest easy when initiating large transfers!

Software Configuration

TrueNAS Scale 23.10.0.1 running:
1 Pool comprised of 1 VDEV with: 2x 4TB HDDs in Mirror configuration
Datasets and SMB Shares for: General Backups, Time Machine, Jellyfin media, and NextCloud files
Apps: Jellyfin (from TrueCharts) and NextCloud (from TrueNAS)
Email Warnings: via Amazon SNS

Time Machine

Upon setting up a Time machine share, I had to set the "Purpose" to "Basic time machine share" instead of Multi-user time machine share. Multiple macs should be able to backup with the basic share. You can tweak the ACL to only allow certain users to backup to it.

Jellyfin

Jellyfin was tricky to setup.
I will make a separate post for this one!

NextCloud

Installed from TrueNAS (although I did not try the TrueCharts version)
The default username and password is in the app configuration itself. On TrueNAS Core, the username and password is in a dropdown called "post-install notes".

Email Warnings

Gmail was tricky to setup. Oauth did not work, and the SMTP login only worked when I generated an app password, which requires 2FA to be enabled.
I used Amazon SNS instead, very low cost. Setup alerts for logging events such as a hard drive failing.

SMB Share Benchmarks

I tested file transfer speeds by transferring 150GB of photos to the server. Speeds topped out and sustained around 45MB/s or 360Mbps on our network. Transferring a 21GB zip file sustained 120MB/s, which is line speed for 1Gb networking. With Jellyfin, I could stream videos on 2 concurrent devices with no issues.

Summary

This project took about a week to complete and I learned an incredible amount along the way. It was super fun! Below are some of my lessons learned and next steps.

Lessons Learned

You can use a PowerEdge T330 with the H730 RAID Controller in BIOS HBA mode for TrueNAS
Jellyfin on TrueNAS Scale is finicky, but it works nicely once setup correctly
NextCloud is much faster to use on the PowerEdge T330 compared to the NextCloudPi

Lessons to be Learned (Questions)

Does the H730 take on too much load from ZFS? Can this load be minimized if so?
How do I take advantage of the dual 1 gigabit connections on the T330? Can I make this aggregate if I connect it directly to our router?
How does the system perform with more than 2 concurrent users?
How could I setup a reverse proxy to allow internet access to this server, as well as the NextCloudPi I have running?

Thanks for reading through this long post! I hope you learned something about servers, RAID controllers, TrueNAS, or maybe just what a NAS is!