Hey Folks, I have a bit of a conundrum that I’m hoping the hive mind can assist with.
I am in the process of learning docker to prep for my migration to Linux, but I have some questions about my filesystem structure. Currently my media files of all types live on a single file-based iSCSI LUN hosted on a QNAP which I connect to from a Windows machine. In my research to see if this would be consistent with best practice, I came to the conclusion that I should create independent NFS shares that the docker containers would connect to individually, rather than serving the files to the containers through the host and it’s iSCSI connection.
This leads to my problem.
I can’t seem to find any way to directly copy data from the LUN to one of my newly created NFS shares. With the volume of data I’ll need to copy I’m trying to avoid as much overhead as possible, and using my Windows machine to connect to the new NFS share, then transferring the files from the iSCSI share, would be ludicrously inefficient.
As I’m able to SSH into my NAS, my first thought was to try and mount the iSCSI file locally and rsync the contents directly to the NFS share. After finding the home of the iSCSI file in the NAS filesystem, I discovered that it is not stored as a single, mountable file, but broken up into 1TB chunks. This leaves me unable to mount it, even in part, as each of the files lack an identifiable filesystem. Further, this is my largest partition, and so I don’t (currently) have the space to attempt to concatenate the files into a single file (assuming that would even work, no idea).
After giving up on this approach, I decided to try and log into it’s own external iSCSI target (from the NAS), then mount the LUN as I would from an external client. I thought I might be in the clear, as the login was successful, and both iscsiadm and the NAS GUI showed the active session to itself. But no matter where I looked I could see no evidence of a newly available partition, only those that were there from before I connected to the iSCSI target.
At this point the next step seems to be shrinking the partition and trying to concatenate the iSCSI files as I mentioned earlier. I have the space to play with, but I’ll need to convert the volume to thin-provisioned, then shrink the volume, which would likely take foreverrrrrrr. But really, even this option sucks, because I’d prefer to avoid jeopardizing my primary storage volume in changing the provisioning style.
So anyway, after banging my head on it for the last few hours, I decided to step away and do some “rubber ducky debugging” with you guys.
So here are my questions: Is migrating to NFS worth the effort? Would the file concatenation method even work? COULD the loopback iSCSI method work if I do something differently? Any other tricks, or maybe something in the QNAP App Marketplace?
Any assistance welcome, thanks for reading!
You really painted yourself in a corner with the QNAP setup unfortunately. There’s no way to migrate over to NFS without headache. Moreover, if you want to go the Docker on Linux route, you’re headed for the same headache but different. You can’t really treat iSCSI/LUN as a run of the mill filesystem (as you’ve discovered).
The issue I see is you’re virtualizing everything. Docker was essentially built to negate the OS. iSCSI virtualizes the filesystem. Doesn’t quite matter what you end up running them on really, you’re a slave to each in a sense.
I’ve always favoured running on the rails with this type of thing to maximize compatibility and limit overhead and headache when it comes to potential migration and connectivity to your NAS. Ubuntu server on ZFS running SMB gives you the broadest compatibility and today probably the best performance even on Linux. Moving to NFS doesn’t seem worth at all imo, it’s just a lateral move with a costly bill at the end.
Are you using Trash Guides? https://trash-guides.info/Hardlinks/How-to-setup-for/Docker/
Mounting the iSCSI to the docker host and then sharing the volumes to docker containers might not be so crazy in terms of hard linking. Though I’ll admit I’m not too familiar with iSCSI and I’m assuming it’s a block device and you create a file system (e.g. ext4) on top, where the file system would support hard links.
I hadn’t heard of Trash Guides before I first set mine up and I did it all poorly (used 2x the storage than I needed) and realised too late, so hopefully this helps!
I guess I am not getting it.
If you can access your files, you can copy your files. If the concern is that you only know how to connect from a full PC, consider plugging a laptop into the switch (or even just set up a VM).
Hard to give much more help without knowing your actual setup. But one nasty solution is to ssh into the server then connect to the running container (or mount the same storage into a different one) if there are some shenanigans going on there.
But yeah. My general rule of thumb is that if something needs to outlive the life of a container then it is being stored on the local filesystem or a zfs/ceph pool.
What’s your filesystem? I’d use zfs send / receive. Worst case scenario, you can use dd.
