> I'd rather provide compilation time options for that. Variants are:
> 1. ./configure --with-compressor=bzip2=/etc/alternatives/bzip2-filter,gzip=...
> 2. ./configure --with-bzip2=/etc/alternatives/bzip2-filter --with-gzip=...
> 3. ./configure BZIP2_PROGRAM=/etc/alternatives/bzip2-filter GZIP_PROGRAM=...
> All of this is only being contemplated because of lbzip? No one has
> ever wanted "alternative" implementations of any other compressor,
No user has ever wanted a decompressor that would exercise all four cores
of his new quad-core computer when extracting the 300M openoffice tarball,
staring at the cpu load desktop applet and cursing about three quarters of
his CPU idling? No system administrator has ever wanted a bzip2 compressor
to compress his/her daily 7G of logs in 1/16th time on his/her 16 core
For a long time, bzip2 was the most space-efficient main-stream, free
software compressor, but it was relatively slow. Thus any speedup was
useful. *Lots* of parallel bzip2 compressors and some decompressors were
written before lbzip2, but in my interpretation, they never got the
multi-threaded decompression quite right. See  if you care.
And what did Mark Adler write pigz for, then? Why didn't he just extend
gzip? He's a big name, maybe you'll accept from him that parallelism by
way of multi-threading is not an additive property, you cannot just slap
it on a pre-existing bitstream format.
Why did Tim Cook write tamp?
I'm not pushing for lbzip2 to be integrated better with tar, or at least
not for myself. I asked first when I was asked to ask. I'm perfectly fine
with --use and I would be fine even without it.
> So how about working (Sergey, I don't mean you) on making lbzip an
> actual replacement for bzip?
If you mean me, I won't work on that, sorry. Please anybody feel free to
fork lbzip2. Nice move though trying to allocate my time. I worked my ass
of on lbzip2 after my day job, nights till 4 o'clock in the morning till I
was falling out of my chair, running kilometers in my room like a rat in a
maze thinking about the decompressor design. Don't care if it doesn't
show, if it's "trivial" or "convoluted" for some (yeah, why didn't anybody
implement it before, among the 3 or so pre-existing parallelizations?)
Lbip2 is done for me. I have some experiments in the queue if some really
nice people will help me out, I'll document those experiments, maybe lose
face on them, I'll release 1.00 then just abandon it.
> I know that isn't being done now, but that doesn't it couldn't be done.
> Free software and all that.
Sure. I'm not interested in toiling away on that shit for a year. You get
my code and the four freedoms with it, you don't get my time and effort. I
contacted Julian Seward, author of original bzip2, both when I had
questions about bzip2 itself and when the idea first emerged to extend
bzip2 with multi-threading, for example by merging lbzip2 into it. He
didn't seem interested. So what? Do it yourself, free software and all
that. Everybody will be thankful, me included, if you manage to do that.
> Finally, does lbzip offer any advantages over xz
> (http://tukaani.org/xz)? Which already compresses better and
> decompresses faster than bz2, in general.
I looked at their 1.0 file format when it came out and it was great, the
blocks are length-prefixed which makes it obvious that it was designed
with parallelism in mind. (The format is great for a whole series of other
reasons, too.) Once they get the multi-threading done, I'll be the first
to throw away lbzip2, use xz for efficiency-oriented compression and tamp
(multi-threaded QuickLZ) for speed-oriented compression. In the meantime,
there are *lots* of single-stream tar.bz2's on the net, and there are
people (I am for sure) issuing
$ wget -O - URL | tee -i f.tar.bz2 | tar -x --use=lbzip2
Just forget the damn thing, I'm fucking tired of it. Defending lbzip2 like
it was some crusade of mine, sending around the same old links a thousand
times, identifying the use cases where lbzip2 is "unique", proving I'm not
doing this only for "fame" or some shit like that. Forget it.