Title is TLDR. More info about what I'm trying to do below.
My daily driver computer is Laptop with an SSD. No possibility to expand.
So for storage of lots n lots of files, I have an old, low resource Desktop with a bunch of HDDs plugged in (mostly via USB).
I can access Desktop files via SSH/SFTP on the LAN. But it can be quite slow.
And sometimes (not too often; this isn't a main requirement) I take Laptop to use elsewhere. I do not plan to make Desktop available outside the network so I need to have a copy of required files on Laptop.
Therefor, sometimes I like to move the remote files from Desktop to Laptop to work on them. To make a sort of local cache. This could be individual files or directory trees.
But then I have a mess of duplication. Sometimes I forget to put the files back.
Seems like Laptop could be a lot more clever than I am and help with this. Like could it always fetch a remote file which is being edited and save it locally?
Is there any way to have Laptop fetch files, information about file trees, etc, located on Desktop when needed and smartly put them back after editing?
Or even keep some stuff around. Like lists of files, attributes, thumbnails etc. Even browsing the directory tree on Desktop can be slow sometimes.
I am not sure what this would be called.
Ideas and tools I am already comfortable with:
-
rsync is the most obvious foundation to work from but I am not sure exactly what would be the best configuration and how to manage it.
-
luckybackup is my favorite rsync GUI front end; it lets you save profiles, jobs etc which is sweet
-
freeFileSync is another GUI front end I've used but I am preferring lucky/rsync these days
-
I don't think git is a viable solution here because there are already git directories included, there are many non-text files, and some of the directory trees are so large that they would cause git to choke looking at all the files.
-
syncthing might work. I've been having issues with it lately but I may have gotten these ironed out.
Something a little more transparent than the above would be cool but I am not sure if that exists?
Any help appreciated even just idea on what to web search for because I am stumped even on that.
You have two problems.
Transferring between your laptop and desktop is slow. There’s a bunch of reasons that this could be. My first thought is that the desktops got a slow 100mbps nic or not enough memory. You could also be using something that’s resource intensive and slow like zfs/zpools or whatever. It’s also possible your laptops old g WiFi is the bottleneck or that with everything else running at the same time it doesn’t have the memory to hold 40tb worth of directory tree.
Plug the laptop into the Ethernet and see if that straightens up your problems.
You want to work with the contents of desktop while away from its physical location. Use a vpn or overlay network for this. I have a complex system so I use nebula. If you just want to get to one machine, you could get away with just regular old openvpn or wireguard.
E: I just reread your post and the usb is likely the problem. Even over 2.0 it’s godawful. See if you can migrate some of those disks onto the sata connectors inside your desktop.
Thanks!
I elaborated on why I'm using USB HDDs in this comment. I have been a bit stuck knowing how to proceed to avoid these problems. I am willing to get a new desktop at some point but not sure what is needed and don't have unlimited resources. If I buy a new device, I'll have to live with it for a long time. I have about 6 or 8 external HDDs in total. Will probably eventually consolidate the smaller ones into a larger drive which would bring it down. Several are 2-4TB, could replace with 1x 12TB. But I will probably keep using the existing ones for backup if at all possible.
Re the VPN, people keep mentioning this. I am not understanding what it would do though? I mostly need to access my files from within the LAN. Certainly not enough to justify the security risk of a dummy like me running a public service. I'd rather just copy files to an encrypted disk for those occasions and feel safe with my ports closed to outsiders.
Is there some reason to consider a VPN for inside the LAN?
You’re getting a lot of advice in this thread and it’s all pretty good, but not all of it seems to answer the problems you have in your order of priority or under your constraints. I’ll try to give an explanation of why I think my advice will do so then give it.
Getting off usb will speed up file access and increase the number of operations you can do from the laptop on your lan. Some stuff will still need to be copied over locally, normal people like us just can’t afford the kind of infrastructure that lets you do everything over the lan. For those things, rsync is perfectly good, and they’re most likely going to be enough of an edge case that it won’t be very often.
When you’re ready, and from your responses in this thread I’d say you are, a vpn doesn’t expose you to much security risk if any. There are caveats to that, but if you’re doing something like openvpn or wireguard it’s all encrypted and key based and basically ain’t nobody getting into it unless they were to get a key off an old computer you use and didn’t wipe before throwing out or something. That would solve your remote access bonus problem. No pressure and in your own time, of course.
You are me twenty years ago.
Cobbling together solutions from what’s available at the cost of the parts from the hardware store. Serial experiments lain but shot in the trailer park boys set. Hackers with the cast of my name is earl.
Don’t ever change.
So you want to kick the bad habit but don’t have enough physical space in your desktops case or enough sata ports! You have a bigger tower case but don’t know if it’ll really hold the drives you want.
The best bet is to transplant the motherboard and power supply from your sff desktop into the big case. If the big case has at least three 5 1/4 bays you can use a bracket to go from 3 big bays to 4 or 5 smaller 3.5” hdd bays. I’d recommend 4 instead of 5, more on that later.
If the big tower case has the little dangly 2x 3.5 bay cage hanging down from its cd cage, you can use four strips of sheet metal and a carpenters square (or the square corner of some copy paper) to make a column of hdd mounting space all the way to the floor of the chassis. Just remember to use vibration damping grommets.
Make sure when you’re filling your tower up with drives to put some fans blowing on em. Drives need to be kept cool for maximum life. Those 3 cd bay to 4 hdd bay adapter brackets are nice for that because they usually have a fan mount or one included.
Now you need sata (or maybe ide) ports to plug all these in. Someone else already said to use those cheap little sata expanders and those are great. I used an old cheap pc mounted in a salvaged case just like you might with four of em back in the day.
You’ll actually want to use the towers power supply if it has one and it works and matches the sff desktops connectors because it probably has more power capacity than the sff desktops supply. You may need some molex or sata splitters to get power to all your drives.
Consider mergerfs and snapraid once you have this wheezing Frankensteins monster operational.
Mergerfs displays all the drives as one big file system to the system and to users. So if drive one had /pics/dog.jpg and drive two had /pics/cat.jpg then the mergerfs of the two of them would have both pictures in the /pics directory when you open it.
Snapraid is like if raid5 (really zpools because you can have many parity devices) but it only runs once a day or whatever and is basically a snapshot.
Anyway sorry for the tangent. Post some pics of the tower case or its model number or whatever and I can give better advice about filling it with drives.