[web-hugo.git] / content / posts / nas1-build-and-borgbackup.md

---
title: "NAS Part I - The Build & Borgbackup Initial Impressions"
date: 2023-05-15T20:40:00+03:00
description: "Almost a year later, the rumours about untimely death of my NAS build have been greatly exaggarated"
tags: [backup, linux, review, opensource, opinion, nas]
type: blog
draft: false
---

# NAS - Part I

If you have been reading since last year July, you may know of my "Project ReuseNAS" - you may also
know that my build date estimate for "end of 2022" or "Q1 2023" was never met as per obvious lack of posts on that front.

When I was originally planning the NAS build, I was very much all over the idea, that I could reuse all of my old PC components.
Over time after discussing with members of the based.quest community, we came to the conclusion that I would be severly handicapping
myself with the hardware I already had laying around.

I was unable to find a satisfactory case for the NAS build that could hold the HDDs on them - apparently cases with 4x 3.5" + 2x 2.5" bays are quite uncommon,
as such I ended up buying a brand new case.

Further I did not trust getting an used power supply or used SSD/HDD drives. CPU cooler I most likely could have found used, but the project was starting to drag
on for far too long - so I ended up getting a brand new one. I know, impatient.

My original plan was also to buy 4x4TB IronWolf Pro drives - as of right now, I only got 2x4TB IronWolf (non-Pro) drives.

The final rig ended up being:

CPU: i5-2400 (Second-hand)\
CPU cooling: be quiet! Pure Rock 2 (Brand new)\
Motherboard: MSI H61MA-E35(b3) (Second-hand)\
RAM: 4x4GB DDR3 from my old "Celery Fujitsu" PC (Second-hand)\
PSU: Corsair CV550 (Brand new)\
Case: Krux Naos TG (Brand new)\
HDD: 2x4TB Seagate IronWolf (Brand new)\
Boot SSD: 2x120GB Patriot (Brand new)

As of writing this, a few of these are still being shipped, but that didn't stop me from starting the experimentation, using my current PC as the test-bench.

## The borgbackup
One simple Friday evening, as I had finished getting final parts of the NAS - I asked the matrix chat for suggestions regarding backup solutions I could use.
My criteria was simple - it must be able to do incremental backups and potentially deduplicate. I was thinking of some hacky rsync solution already until
Kerum stepped in with the idea of using [borgbackup](https://www.borgbackup.org) - I was intrigued, it offers space efficient storage of backups, LZ4 compression,
mountable backups with FUSE and claims to be easy to setup.

Without wasting further time - I decided to take my 2x4TB drives out, attach them to my main PC, build a software RAID 1 and install borgbackup.
What I discovered was everything I could have asked for from a free solution and beyond.

Let's start with the extras I wasn't expecting for -- the backups are all encrypted. If your HDDs get stolen for whatever reason, your backups are secure.
Further you are secure from malicious actors - you do NOT need borg on the remote site even, you do not need to store any passwords on the repository server.

About performance - your initial pass is always going to be slower as borgbackup has to make note of blocks for deduplication and also encrypt the data.
However, this is why I wanted the ability to do incremental backups - you only have to move what you need. Your next passes however, will be a lot quicker - depending on the
amount of inodes in your server. For me, an incremental backup takes about ~ minutes presently. Here's a summary from the last backup I just took (presently manually, going full-automatic
once the NAS final components have shipped!).

```
Mon 15 May 2023 07:10:29 PM CEST Starting backup

/path/to/<redacted>.log: file changed while we backed it up.log                                                                                                             
------------------------------------------------------------------------------                                                                                                          
Repository: ssh://<user>@<nas>:<port>/path/to/borg-repo
Archive name: cernodile-2023-05-15T19:10:29
Archive fingerprint: 42cc287287f3c2f9d34308844ad3f18b15f53333d9eeb3e9b71ad1c89c33b063
Time (start): Mon, 2023-05-15 19:10:30
Time (end):   Mon, 2023-05-15 19:16:53
Duration: 6 minutes 22.94 seconds
Number of files: 741792
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:              197.53 GB            149.67 GB              1.58 GB
All archives:              971.53 GB            754.30 GB            151.90 GB

                       Unique chunks         Total chunks
Chunk index:                  530571              4076474
------------------------------------------------------------------------------
terminating with warning status, rc 1

Mon 15 May 2023 07:16:59 PM CEST Pruning repository

Keeping archive (rule: daily #1):        cernodile-2023-05-15T19:10:29        Mon, 2023-05-15 19:10:30 [42cc287287f3c2f9d34308844ad3f18b15f53333d9eeb3e9b71ad1c89c33b063]
Pruning archive (1/1):                   cernodile-2023-05-15T06:15:29        Mon, 2023-05-15 06:15:31 [4266d525a871b036cb71f0ba0f8930e1dcd04499fa5293deba87305497fe957d]
Keeping archive (rule: daily #2):        cernodile-2023-05-14T01:08:46        Sun, 2023-05-14 01:08:48 [0dc46a3106d1df91170fb60d0661a1b6ca92fc50694acfa8833745dd302d1fd6]
Keeping archive (rule: daily #3):        cernodile-2023-05-13T09:09:41        Sat, 2023-05-13 09:09:41 [f0be5704c565202b405ac833e1256fd320ce0968946bfabfd3312d3a1aa3afc2]
Keeping archive (rule: daily[oldest] #4): cernodile-2023-05-13T00:51:50        Sat, 2023-05-13 00:51:51 [14be5350fa55603c65e897c5a6f37925943b7c5eed793db386d5090464d6a18a]
terminating with success status, rc 0

Mon 15 May 2023 07:17:18 PM CEST Compacting repository


Mon 15 May 2023 07:17:34 PM CEST Backup, Prune, and/or Compact finished
```

I seem to face a visual bug regarding the warning status (no actual warning is output!), but other than that, I've had no real issues so far - I have verified integrity of the backups (each one!), all match up.

Note the original and compressed sizes here. The original backed up data (aka actual FS usage) is 197 GB which gets compressed down to 149.67GB! With 3 other archives visible here, you can see that the incrementals have only added up about 3GB (which roughly aligns with the daily amount of data added too ~1GB/day).

## FUSE the backups!
As mentioned before, borgbackup supports mounting your backups with FUSE provided you have the space for it. Let's say I accidentally make changes I didn't really intend to today and I didn't make a backup of it.
So I want to naturally take my nginx.conf back from the previous day.

To do that, I go back to my PC terminal and check for my existing restore points with `borg list /path/to/repo`:
```
$ borg list /path/to/repo:
Enter passphrase for key /path/to/repo: 
cernodile-2023-05-13T00:51:50        Sat, 2023-05-13 01:51:51 [14be5350fa55603c65e897c5a6f37925943b7c5eed793db386d5090464d6a18a]
cernodile-2023-05-13T09:09:41        Sat, 2023-05-13 10:09:41 [f0be5704c565202b405ac833e1256fd320ce0968946bfabfd3312d3a1aa3afc2]
cernodile-2023-05-14T01:08:46        Sun, 2023-05-14 02:08:48 [0dc46a3106d1df91170fb60d0661a1b6ca92fc50694acfa8833745dd302d1fd6]
cernodile-2023-05-15T19:10:29        Mon, 2023-05-15 20:10:30 [42cc287287f3c2f9d34308844ad3f18b15f53333d9eeb3e9b71ad1c89c33b063]
```

As I said before, I want to see /etc/nginx from previous day, in this case, May 14th. To do that, I'll have to use `borg mount` as such:
```
$ borg mount /path/to/repo/::cernodile-2023-05-14T01:08:46 /mnt/ /etc/nginx
Enter passphrase for key /path/to/repo: 
$ cd /mnt/
$ ls -lh etc/nginx/nginx.conf 
-rw-r--r-- 1 root root 1.5K 26. apr   23:33 etc/nginx/nginx.conf
```

Voilah! We have retrieved the configuration file that was present that day! Now we can retrieve the contents of it using whichever preferred utility you wish or transport it directly
to the server with scp/ftp/sftp.
Once you're done, run `borg umount /mnt`.

## Future concepts to look into
I want to see if it's possible to make my backups immutable in event of cernodile.com or based.quest being hacked.

## Conclusion
Overall my first impressions so far are very positive, there is so much I haven't discovered yet with this backup solution and it is everything I could have asked for.
For my use case, this is ideal. The backup script I use is available in borgbackup's own documentation. I recommend you read up on that as well if you are as interested
in borgbackup as I am.

I am glad to finally have my NAS and backup solution set up, it's been a long time coming.

Thank you so much for reading, hopefully without this long of a delay next time,
- Cernodile
Commit	Line	Data
d5f70bed JM	1	---
	2	title: "NAS Part I - The Build & Borgbackup Initial Impressions"
	3	date: 2023-05-15T20:40:00+03:00
	4	description: "Almost a year later, the rumours about untimely death of my NAS build have been greatly exaggarated"
	5	tags: [backup, linux, review, opensource, opinion, nas]
	6	type: blog
	7	draft: false
	8	---
	9
	10	# NAS - Part I
	11
	12	If you have been reading since last year July, you may know of my "Project ReuseNAS" - you may also
	13	know that my build date estimate for "end of 2022" or "Q1 2023" was never met as per obvious lack of posts on that front.
	14
	15	When I was originally planning the NAS build, I was very much all over the idea, that I could reuse all of my old PC components.
	16	Over time after discussing with members of the based.quest community, we came to the conclusion that I would be severly handicapping
	17	myself with the hardware I already had laying around.
	18
	19	I was unable to find a satisfactory case for the NAS build that could hold the HDDs on them - apparently cases with 4x 3.5" + 2x 2.5" bays are quite uncommon,
	20	as such I ended up buying a brand new case.
	21
	22	Further I did not trust getting an used power supply or used SSD/HDD drives. CPU cooler I most likely could have found used, but the project was starting to drag
	23	on for far too long - so I ended up getting a brand new one. I know, impatient.
	24
	25	My original plan was also to buy 4x4TB IronWolf Pro drives - as of right now, I only got 2x4TB IronWolf (non-Pro) drives.
	26
	27	The final rig ended up being:
	28
	29	CPU: i5-2400 (Second-hand)\
	30	CPU cooling: be quiet! Pure Rock 2 (Brand new)\
	31	Motherboard: MSI H61MA-E35(b3) (Second-hand)\
	32	RAM: 4x4GB DDR3 from my old "Celery Fujitsu" PC (Second-hand)\
	33	PSU: Corsair CV550 (Brand new)\
	34	Case: Krux Naos TG (Brand new)\
	35	HDD: 2x4TB Seagate IronWolf (Brand new)\
	36	Boot SSD: 2x120GB Patriot (Brand new)
	37
	38	As of writing this, a few of these are still being shipped, but that didn't stop me from starting the experimentation, using my current PC as the test-bench.
	39
	40	## The borgbackup
	41	One simple Friday evening, as I had finished getting final parts of the NAS - I asked the matrix chat for suggestions regarding backup solutions I could use.
	42	My criteria was simple - it must be able to do incremental backups and potentially deduplicate. I was thinking of some hacky rsync solution already until
	43	Kerum stepped in with the idea of using [borgbackup](https://www.borgbackup.org) - I was intrigued, it offers space efficient storage of backups, LZ4 compression,
	44	mountable backups with FUSE and claims to be easy to setup.
	45
	46	Without wasting further time - I decided to take my 2x4TB drives out, attach them to my main PC, build a software RAID 1 and install borgbackup.
	47	What I discovered was everything I could have asked for from a free solution and beyond.
	48
	49	Let's start with the extras I wasn't expecting for -- the backups are all encrypted. If your HDDs get stolen for whatever reason, your backups are secure.
	50	Further you are secure from malicious actors - you do NOT need borg on the remote site even, you do not need to store any passwords on the repository server.
	51
	52	About performance - your initial pass is always going to be slower as borgbackup has to make note of blocks for deduplication and also encrypt the data.
	53	However, this is why I wanted the ability to do incremental backups - you only have to move what you need. Your next passes however, will be a lot quicker - depending on the
	54	amount of inodes in your server. For me, an incremental backup takes about ~ minutes presently. Here's a summary from the last backup I just took (presently manually, going full-automatic
	55	once the NAS final components have shipped!).
	56
	57	```
	58	Mon 15 May 2023 07:10:29 PM CEST Starting backup
	59
	60	/path/to/<redacted>.log: file changed while we backed it up.log
	61	------------------------------------------------------------------------------
	62	Repository: ssh://<user>@<nas>:<port>/path/to/borg-repo
	63	Archive name: cernodile-2023-05-15T19:10:29
	64	Archive fingerprint: 42cc287287f3c2f9d34308844ad3f18b15f53333d9eeb3e9b71ad1c89c33b063
65	Time (start): Mon, 2023-05-15 19:10:30
66	Time (end): Mon, 2023-05-15 19:16:53
67	Duration: 6 minutes 22.94 seconds
68	Number of files: 741792
69	Utilization of max. archive size: 0%
70	------------------------------------------------------------------------------
71	Original size Compressed size Deduplicated size
72	This archive: 197.53 GB 149.67 GB 1.58 GB
73	All archives: 971.53 GB 754.30 GB 151.90 GB
74
75	Unique chunks Total chunks
76	Chunk index: 530571 4076474
77	------------------------------------------------------------------------------
78	terminating with warning status, rc 1
79
80	Mon 15 May 2023 07:16:59 PM CEST Pruning repository
81
82	Keeping archive (rule: daily #1): cernodile-2023-05-15T19:10:29 Mon, 2023-05-15 19:10:30 [42cc287287f3c2f9d34308844ad3f18b15f53333d9eeb3e9b71ad1c89c33b063]
83	Pruning archive (1/1): cernodile-2023-05-15T06:15:29 Mon, 2023-05-15 06:15:31 [4266d525a871b036cb71f0ba0f8930e1dcd04499fa5293deba87305497fe957d]
84	Keeping archive (rule: daily #2): cernodile-2023-05-14T01:08:46 Sun, 2023-05-14 01:08:48 [0dc46a3106d1df91170fb60d0661a1b6ca92fc50694acfa8833745dd302d1fd6]
85	Keeping archive (rule: daily #3): cernodile-2023-05-13T09:09:41 Sat, 2023-05-13 09:09:41 [f0be5704c565202b405ac833e1256fd320ce0968946bfabfd3312d3a1aa3afc2]
86	Keeping archive (rule: daily[oldest] #4): cernodile-2023-05-13T00:51:50 Sat, 2023-05-13 00:51:51 [14be5350fa55603c65e897c5a6f37925943b7c5eed793db386d5090464d6a18a]
87	terminating with success status, rc 0
88
89	Mon 15 May 2023 07:17:18 PM CEST Compacting repository
90
91
92	Mon 15 May 2023 07:17:34 PM CEST Backup, Prune, and/or Compact finished
93	```
94
95	I seem to face a visual bug regarding the warning status (no actual warning is output!), but other than that, I've had no real issues so far - I have verified integrity of the backups (each one!), all match up.
96
97	Note the original and compressed sizes here. The original backed up data (aka actual FS usage) is 197 GB which gets compressed down to 149.67GB! With 3 other archives visible here, you can see that the incrementals have only added up about 3GB (which roughly aligns with the daily amount of data added too ~1GB/day).
98
99	## FUSE the backups!
100	As mentioned before, borgbackup supports mounting your backups with FUSE provided you have the space for it. Let's say I accidentally make changes I didn't really intend to today and I didn't make a backup of it.
101	So I want to naturally take my nginx.conf back from the previous day.
102
103	To do that, I go back to my PC terminal and check for my existing restore points with `borg list /path/to/repo`:
104	```
105	$ borg list /path/to/repo:
106	Enter passphrase for key /path/to/repo:
107	cernodile-2023-05-13T00:51:50 Sat, 2023-05-13 01:51:51 [14be5350fa55603c65e897c5a6f37925943b7c5eed793db386d5090464d6a18a]
108	cernodile-2023-05-13T09:09:41 Sat, 2023-05-13 10:09:41 [f0be5704c565202b405ac833e1256fd320ce0968946bfabfd3312d3a1aa3afc2]
109	cernodile-2023-05-14T01:08:46 Sun, 2023-05-14 02:08:48 [0dc46a3106d1df91170fb60d0661a1b6ca92fc50694acfa8833745dd302d1fd6]
110	cernodile-2023-05-15T19:10:29 Mon, 2023-05-15 20:10:30 [42cc287287f3c2f9d34308844ad3f18b15f53333d9eeb3e9b71ad1c89c33b063]
111	```
112
113	As I said before, I want to see /etc/nginx from previous day, in this case, May 14th. To do that, I'll have to use `borg mount` as such:
114	```
115	$ borg mount /path/to/repo/::cernodile-2023-05-14T01:08:46 /mnt/ /etc/nginx
116	Enter passphrase for key /path/to/repo:
117	$ cd /mnt/
118	$ ls -lh etc/nginx/nginx.conf
119	-rw-r--r-- 1 root root 1.5K 26. apr 23:33 etc/nginx/nginx.conf
120	```
121
122	Voilah! We have retrieved the configuration file that was present that day! Now we can retrieve the contents of it using whichever preferred utility you wish or transport it directly
123	to the server with scp/ftp/sftp.
124	Once you're done, run `borg umount /mnt`.
125
126	## Future concepts to look into
127	I want to see if it's possible to make my backups immutable in event of cernodile.com or based.quest being hacked.
128
129	## Conclusion
130	Overall my first impressions so far are very positive, there is so much I haven't discovered yet with this backup solution and it is everything I could have asked for.
131	For my use case, this is ideal. The backup script I use is available in borgbackup's own documentation. I recommend you read up on that as well if you are as interested
132	in borgbackup as I am.
133
134	I am glad to finally have my NAS and backup solution set up, it's been a long time coming.
135
136	Thank you so much for reading, hopefully without this long of a delay next time,
137	- Cernodile