Update README.md
Signed-off-by: first <first@noreply.git.r21.io>
This commit is contained in:
parent
e0d1777965
commit
df214ea14c
181
README.md
181
README.md
|
|
@ -1,100 +1,125 @@
|
|||
Below is a sample Bash script and accompanying guide that demonstrate one way to automate:
|
||||
# 1. Overview: Individually Encrypted 100GB Archives
|
||||
|
||||
1. **Creating a single encrypted archive** from your data.
|
||||
2. **Splitting** that encrypted archive into 100GB chunks.
|
||||
3. **Generating checksums** and a manifest/catalog.
|
||||
4. **Optionally creating ISO images** from each chunk for more convenient burning.
|
||||
5. **Burning** the resulting chunks (or ISOs) to M-Disc.
|
||||
### Basic Idea
|
||||
|
||||
> **Important**
|
||||
> - This script is written in Bash for Linux/macOS compatibility. It should also work on FreeBSD with minimal (if any) modifications, but you may need to install or adjust the relevant tools.
|
||||
> - The script focuses on automating the encryption and splitting steps, as well as generating a manifest.
|
||||
> - Burning to M-Disc on different platforms can vary. We show an example using `growisofs` (common on Linux) and `hdiutil` (macOS). Adjust as needed.
|
||||
> - For best security, do **not** hardcode your passphrase in the script. You should be prompted for it.
|
||||
1. **Figure out which files belong to which 100GB set**.
|
||||
- You can gather files until their combined uncompressed size is ~100GB (or 95GB if you want some buffer for overhead).
|
||||
- Put them in a "chunk_001" grouping, then "chunk_002," etc.
|
||||
|
||||
2. **Create a TAR for each group**, then **compress** with `lz4`, then **encrypt** with `gpg`.
|
||||
- Result: `chunk_001.tar.lz4.gpg`, `chunk_002.tar.lz4.gpg`, etc.
|
||||
- Each chunk is fully independent: if you only have `chunk_004.tar.lz4.gpg`, you can decrypt it, decompress it, and restore the files that were in chunk #4.
|
||||
|
||||
3. **Burn each chunk** onto its own M-Disc.
|
||||
- Optionally, create ISO images (e.g., `genisoimage -o chunk_001.iso chunk_001.tar.lz4.gpg`) and then burn them.
|
||||
|
||||
4. **To restore** any subset, you just decrypt the chunk you want, decompress, and extract it. No other chunks are required.
|
||||
|
||||
### Pros
|
||||
|
||||
- Each 100GB chunk is an autonomous backup.
|
||||
- Damage/loss of one disc only affects that chunk's files.
|
||||
|
||||
### Cons
|
||||
|
||||
- Less efficient if you have many smaller files (no cross-chunk deduplication).
|
||||
- Slightly more complex to create "balanced" 100GB sets.
|
||||
- Big single files that exceed 100GB are a problem unless you handle them specially.
|
||||
|
||||
---
|
||||
|
||||
## How to Use the Script
|
||||
# 2. Sample Script: `backup2mdisc.sh`
|
||||
|
||||
1. **Install Dependencies**
|
||||
Make sure the following tools are installed on your system(s):
|
||||
- **tar**
|
||||
- **xz**
|
||||
- **gpg**
|
||||
- **split**
|
||||
- **sha256sum** (or `shasum` on FreeBSD/macOS)
|
||||
- **genisoimage** or **mkisofs** (for creating ISOs if desired)
|
||||
- **growisofs** (Linux) or **hdiutil** (macOS) for burning.
|
||||
This is a **Bash** script that:
|
||||
|
||||
2. **Make the Script Executable**
|
||||
```bash
|
||||
chmod +x backup2mdisc.sh
|
||||
```
|
||||
1. Collects **all files** in a specified source directory.
|
||||
2. Iterates over them in ascending order by size (you can adjust if you prefer a different approach).
|
||||
3. Accumulates files into a "chunk" until you're about to exceed the chunk size limit.
|
||||
4. When the chunk is "full," it creates a **tar** archive, pipes it into **lz4**, then **encrypts** with `gpg`.
|
||||
5. Moves on to the next chunk until all files are processed.
|
||||
6. Generates a manifest with checksums for each `.tar.lz4.gpg`.
|
||||
|
||||
3. **Run the Script**
|
||||
```bash
|
||||
./backup2mdisc.sh /path/to/source /path/to/destination 100G --create-iso --burn
|
||||
```
|
||||
- **`/path/to/source`**: The directory you want to back up.
|
||||
- **`/path/to/destination`**: Where to store the intermediate backup files before burning.
|
||||
- **`100G`**: The chunk size. Adjust if you're using different capacity discs.
|
||||
- **`--create-iso`** (optional): Create ISO images from each chunk for more convenient burning.
|
||||
- **`--burn`** (optional): Attempt to burn each chunk/ISO to disc automatically.
|
||||
|
||||
4. **Enter Your GPG Passphrase**
|
||||
- The script will prompt for a passphrase. This passphrase encrypts your data. Keep it safe!
|
||||
|
||||
5. **Wait for the Script to Finish**
|
||||
- A large `tar` + `xz` + `gpg` pipeline can take a considerable amount of time depending on your data size.
|
||||
- After encryption, it splits into 100GB chunks.
|
||||
- It then generates a **manifest** with SHA-256 checksums of each chunk.
|
||||
|
||||
6. **Burn to M-Disc**
|
||||
- If you used `--burn`, the script will prompt you to insert an M-Disc for each chunk or ISO.
|
||||
- On Linux, it uses `growisofs`. On macOS, it attempts `hdiutil` if ISO files exist.
|
||||
- If you prefer manual burning, skip `--burn` and burn the `.iso` files using your favorite tool.
|
||||
|
||||
7. **Store the Manifest Safely**
|
||||
- The manifest (`backup_manifest.txt`) in the work directory includes:
|
||||
- Checksums for each chunk.
|
||||
- The original source path.
|
||||
- Timestamp.
|
||||
- Keep this manifest (and the passphrase!) somewhere secure. You'll need all parts to restore.
|
||||
> **Disclaimer**:
|
||||
> - This script uses file-size-based grouping. If you have one single file larger than the chunk limit, it won't fit. You'd need advanced splitting or a different solution.
|
||||
> - On macOS or FreeBSD, you might need to install or alias `sha256sum`. If unavailable, replace with `shasum -a 256`.
|
||||
> - This script **does not** automatically burn discs (though it shows how you might add that step).
|
||||
|
||||
---
|
||||
|
||||
## Restoring Your Backup
|
||||
## How This Script Works
|
||||
|
||||
To **restore** from these discs:
|
||||
1. **Collect Files and Sort**
|
||||
- We use `find` to list all files in `SOURCE_DIR`, capturing both size and path.
|
||||
- Sorting by size ensures the script packs smaller files first. (You can remove sorting if you prefer alphabetical or another method.)
|
||||
|
||||
1. Copy all chunk files (or `.iso` contents) back to a working directory on your system.
|
||||
2. Combine them back into a single file (if they were split outside of an ISO filesystem, just `cat` them together):
|
||||
```bash
|
||||
cat backup.tar.xz.gpg.* > backup.tar.xz.gpg
|
||||
```
|
||||
3. Decrypt and extract:
|
||||
```bash
|
||||
gpg --decrypt backup.tar.xz.gpg | xz -d | tar -xvf -
|
||||
```
|
||||
You'll be prompted for the same GPG passphrase. Once it's done, the original files/folders should appear in your current directory.
|
||||
2. **Accumulate Files Until the Chunk Is ~100GB**
|
||||
- We convert `CHUNK_SIZE` from something like `100G` into bytes. Then we compare the sum of file sizes to that limit.
|
||||
- If adding a new file would exceed the chunk limit, we finalize the current chunk and create a new one.
|
||||
|
||||
3. **Create a TAR, Compress with lz4, Then Encrypt**
|
||||
- We pipe the TAR stream into `lz4` for fast compression, and then pipe **that** into `gpg --batch -c` for symmetric encryption with AES256.
|
||||
- Each chunk is written to `chunk_XXX.tar.lz4.gpg`.
|
||||
- No chunk depends on the others.
|
||||
|
||||
4. **Write Checksums to the Manifest**
|
||||
- We run a SHA-256 on the resulting `chunk_XXX.tar.lz4.gpg` and store that in `manifest_individual_chunks.txt` for integrity checks.
|
||||
|
||||
5. **Repeat**
|
||||
- Next chunk continues until all files have been processed.
|
||||
|
||||
6. **Result**
|
||||
- You get multiple `.tar.lz4.gpg` archives in your `DEST_DIR`, each below your chosen chunk size and fully independent.
|
||||
|
||||
## Burning to M-Disc
|
||||
|
||||
You can then burn each chunk to a separate disc. For example:
|
||||
|
||||
```bash
|
||||
cd /path/to/work_dir
|
||||
genisoimage -o chunk_001.iso chunk_001.tar.lz4.gpg
|
||||
# Then burn chunk_001.iso
|
||||
growisofs -Z /dev/sr0=chunk_001.iso
|
||||
```
|
||||
|
||||
Repeat for each chunk. On macOS, you might use:
|
||||
|
||||
```bash
|
||||
hdiutil burn chunk_001.iso
|
||||
```
|
||||
|
||||
(Adjust device paths and commands as needed.)
|
||||
|
||||
## Restoring Data
|
||||
|
||||
To restore from a single chunk (e.g., chunk_002.tar.lz4.gpg), do:
|
||||
|
||||
```bash
|
||||
gpg --decrypt chunk_002.tar.lz4.gpg | lz4 -d | tar -xvf -
|
||||
```
|
||||
|
||||
You'll be prompted for the same passphrase you used when creating the archive. After extraction, you'll see all the files that chunk contained.
|
||||
|
||||
- **If one disc is lost**, you can still decrypt and restore the other discs. You only lose the files in the missing chunk.
|
||||
|
||||
---
|
||||
|
||||
### Notes & Tips
|
||||
# Why lz4 Over xz?
|
||||
|
||||
- **Individual Chunk Decryption**:
|
||||
The above script creates **one** large encrypted archive, then splits it. You need **all** parts to decrypt. If you want each 100GB chunk to be decryptable separately, you'd need to tar smaller subsets of data individually, encrypt each, and then burn. This is more complex and requires advanced scripting or a specialized backup tool.
|
||||
- **lz4** is extremely fast compared to xz, especially for decompression.
|
||||
- **xz** typically yields better compression (smaller output size), but at a much higher CPU cost.
|
||||
- For backups where speed is the priority (and you have enough disc space), lz4 is a great choice.
|
||||
- If you need to cram as much data as possible into 100GB, you might prefer xz with a high compression setting—but your backup process and restoration would be slower.
|
||||
|
||||
- **Automated Backup Tools**:
|
||||
You might also consider tools like **Duplicati**, **Borg**, or **restic**, which support encryption, deduplication, and chunking. However, writing those chunks onto M-Disc is still a manual step.
|
||||
---
|
||||
|
||||
- **Testing**:
|
||||
Test with a small directory first (say 1GB) and 100MB “chunks” to ensure your workflow is correct. Then proceed to the full data.
|
||||
## Final Thoughts
|
||||
|
||||
- **M-Disc Drive Compatibility**:
|
||||
Make sure your optical drive explicitly supports writing to 100GB BD-XL M-Disc. Standard Blu-ray or DVD drives often do not support higher-capacity M-Discs.
|
||||
With this script and approach:
|
||||
|
||||
- **Verification**:
|
||||
Always verify that your burned discs are readable. You can mount them and use the checksums from the manifest to confirm data integrity.
|
||||
- You gain **independently decryptable** 100GB archives.
|
||||
- If a single disc is damaged, you only lose that chunk's data; all other chunks remain fully restorable.
|
||||
- lz4 + gpg is a solid combo for speed (lz4 for compression, gpg for encryption).
|
||||
- Always **test** your workflow on smaller data sets before doing a large 2TB backup.
|
||||
- Keep your passphrase secure, and consider verifying your burned discs with checksums.
|
||||
|
||||
That's it! This script and guide should get you started creating encrypted backups on 100GB M-Discs, with a manifest to track chunks and checksums, plus optional ISO creation and automated burning steps. Adjust as necessary for your specific environment and needs.
|
||||
That's it! You now have a **fast, chunked, and individually encrypted** backup solution for your M-Discs.
|
||||
Loading…
Reference in a new issue