Comment

Anonymous

Consider using bzip2, xz or something similar which can be both compressed and decompressed in parallel. See pbzip2 and pixz. (Zip does not lend itself to this, so pigz would not help you too much under most circumstances.)

Choose the right UNIX tool, subprocesses and pipes are the way to go here and pipes can go across network boundaries as well - rely on the deep history of computing.

Replies

Peter Bengtsson

Zip files can be decompressed in parallel too. You first ask the zip file for its file contents (fast), then you start parallel processes (or threads or whatever) that each get a file name to extract.

My multiprocess example above is a proof of that.