I have several file systems mounted on /mnt/hdd0, /mnt/hdd1, /mnt/hdd2, etc. on my Manjaro Linux system. Let's call these file systems fs0, fs1, fs2, and so on. Currently, all of these file systems have the same directory structure and contain files.
My goal is to reorganize and redistribute all the files across the drives based on the alphabetical ordering of their relative file paths, following these rules:
- The files with the earliest alphabetical relative paths should be moved to the first mounted file system in alphabetical order (e.g. /mnt/hdd0) which contains the appropriate folder structure.
- The files with the latest alphabetical relative paths should be moved to the last mounted file system in alphabetical order (e.g. /mnt/hddN) which contains the appropriate folder structure.
- The remaining files should be distributed across the remaining mounted file systems (/mnt/hdd1, /mnt/hdd2, etc.) in alphabetical order based on their relative paths.
The process should fill up each file system in alphabetical order (/mnt/hdd0, /mnt/hdd1, etc.) until the given free space threshold is reached before moving on. While reorganizing, I want to ensure each file system maintains a specified amount of free space remaining. If a file system does not have enough free space for the files being moved to it, I want to move enough existing files from that file system to another file system with the most available space to make room.
I need step-by-step instructions, including commands, scripts, or tools in Manjaro Linux, to automate this entire process of reorganizing and redistributing the files while following the rules mentioned above.
To accomplish this task, we can use a combination of Bash scripting and various Linux utilities such as
find
,mv
,du
, andsort
. Here's a step-by-step approach:reorganize.sh
.#!/bin/bash # Define the mount points and the desired free space threshold declare -a MOUNT_POINTS=("/mnt/hdd0" "/mnt/hdd1" "/mnt/hdd2" ...) FREE_SPACE_THRESHOLD=20 # In gigabytes # Function to get the available space on a file system get_available_space() { local mount_point="$1" df --output=avail "$mount_point" | tail -n 1 | awk '{print $1}' } # Create an associative array to keep track of the available space on each file system declare -A AVAILABLE_SPACE for mount_point in "${MOUNT_POINTS[@]}"; do AVAILABLE_SPACE["$mount_point"]=$(get_available_space "$mount_point") done # Function to find the mount point with the most available space find_mount_point_with_most_space() { local max_space=0 local max_mount_point="" for mount_point in "${!AVAILABLE_SPACE[@]}"; do if [ "${AVAILABLE_SPACE[$mount_point]}" -gt "$max_space" ]; then max_space="${AVAILABLE_SPACE[$mount_point]}" max_mount_point="$mount_point" fi done echo "$max_mount_point" } # Function to redistribute files based on alphabetical order redistribute_files() { local source_dir="$1" local target_dir="$2" # Get a list of files sorted alphabetically by relative path local files=$(find "$source_dir" -type f | sort) # Move files to the target directory while read -r file; do local relative_path=$(realpath --relative-to="$source_dir" "$file") local file_size=$(du -k "$file" | cut -f 1) # Update available space on source and target directories AVAILABLE_SPACE["$source_dir"]=$((AVAILABLE_SPACE["$source_dir"] + file_size)) AVAILABLE_SPACE["$target_dir"]=$((AVAILABLE_SPACE["$target_dir"] - file_size)) mkdir -p "$target_dir/$(dirname "$relative_path")" mv "$file" "$target_dir/$relative_path" done <<< "$files" } # Function to move files from one file system to another if the free space is below the threshold move_files_if_needed() { local source_dir="$1" local temp_dir=$(mktemp -d --tmpdir="$(find_mount_point_with_most_space)") if [ "${AVAILABLE_SPACE[$source_dir]}" -lt "$((FREE_SPACE_THRESHOLD * 1024 * 1024))" ]; then # If the available space on the source directory is below the threshold, # move the files to a temporary directory on the mount point with the most available space redistribute_files "$source_dir" "$temp_dir" rm -rf "$source_dir"/* # Move the files back to the source directory, preserving the alphabetical order redistribute_files "$temp_dir" "$source_dir" rmdir "$temp_dir" fi } # Main script for ((i=0; i<${#MOUNT_POINTS[@]}; i++)); do local mount_point="${MOUNT_POINTS[$i]}" # Check if the current mount point needs to be cleared to meet the free space threshold move_files_if_needed "$mount_point" # Redistribute files from the current mount point to the remaining mount points in alphabetical order for ((j=i+1; j<${#MOUNT_POINTS[@]}; j++)); do redistribute_files "$mount_point" "${MOUNT_POINTS[$j]}" done done
chmod +x reorganize.sh
This wouldn't work because of the way it deletes everything when making space on a filesystem. It should just move enough files to make space for the ones it's moving there. Even if this script worked, I would need a lot of testing before I were confident using it to reorganize my actual data.
Also, to copy the directory structure I use:
rsync -av -f"+ */" -f"- *" "/mnt/hdd1" "/mnt/hdd2" &