Access Keys:
Skip to content (Access Key - 0)
Home (Access Key - 1)
All spaces... (Access Key - 3)
Log in (Access Key - 5)
Sign up (Access Key - 6)
Toggle Sidebar

Dealing with Large Backup Archive Files

(None)

When transferring large files over the internet, eg. via FTP or WebDAV, large files can often be cumbersome. This tutorial explains some options for splitting large files in to a series of smaller files that are often easier to transfer over the internet.

Windows

Tools like WinRAR, PowerArchiver and WinZIP often have features that allow large backups to be split in to smaller files and later merged back together.

If you're using Mac OS X then check out Stuffit, a popular file archiver for people using a "proper PC".

All of the Windows and Mac OS X applications are pretty trivial to use - for more information see their associated websites.

Many archive formats, associated applications or transfer protocols have inherent 2GB (or similar) file size limitations so always test heavily before trusting your backups with them.

Gzipped Tar Archives

The standard archive format on BSD, Unix and Linux based systems is called tar and is compressed using gzip.

The tutorial below shows how to split a compressed archive in to smaller chunks and then recombine them, all via the command line (so it's easy to script)...

Creating lots of smaller archive files

To create:

tar -cf - input | gzip -c - | split -b1024m - output.tgzs.

This command tars the input which is compressed and split into 1GB files. With the extension .tgzs.aa, tgzs.ab, tgzs.ac and so on.

To rejoin:

cat file[s] | gzip -cd - | tar -xf -

You can download a script that simplifies the command to this:

targzsplit input output

Splitting an existing large file

If you already have large backup archive file, you can do the following to split it in to smaller files.

To split a large tarball:

split -b1m mahoosive_file.tgz mahoosive_fragment_

The -b switch can be changed to your preference, it accepts b, k, or m. b represent bytes, k represent kilobytes, m represent megabytes. In this case we're splitting it into 1MB chunks.

These files can later be reassembled with:

cat mahoosive_fragment_* > mahoosive_file.tgz
Toggle Sidebar
Adaptavist Theme Builder Powered by Atlassian Confluence