On Sat, Apr 28, 2012 at 12:36 PM, Alejandro Imass <aimass@yaba...> wrote:
> On Sat, Apr 28, 2012 at 11:39 AM, Robert Bonomi
> <bonomi@mail...> wrote:
>> Alejandro Imass <aimass@yaba...> wrote:
>>> On Sat, Apr 28, 2012 at 3:22 AM, Wojciech Puchar
>>> <wojtek@wojt...> wrote:
>>> >> I somewhat agree, but it wasn't a person. I am the only administrator,
>>> >> the only one with root access. The jails were effectively moved to the
>>> >> /usr/local/etc/apache22 of the single that survived at the top level.
>>> >> I'm thinking something between mount, EzJail, the journal and the way
>>> >> MySQL created a great deal of head contention, so something must have
>>> >> gotten corrupted at the directory level like you state, but the
>>> >> strange part is no _data_ corruption as such, because I was able to
>>> >> physically archive the jails, move them to the correct directory and
>>> > no matter what you do FreeBSD DOES NOT ramdomly move directories. if you are
>>> > sure you didn't move it yourself then it must be machine hardware problem
>>> > but still unlikely.
>>> After a little more research, ___it it NOT unlikely at all___ that
>>> under high distress and a hard boot, UFS could have somehow corrupted
>>> the directory structure, whilst maintaining the data intact.
>> This is techically accurate, *BUT* the specifics of the quote "corruption"
>> unquote in the case under discussion make it *EXTREMELY* unlikely that this
>> is what happened.
>> 99.99+++% of all UFS filesystem "corruption' issues are the result of a
>> system crash _between_ the time cached 'meta-data' is updated in memory
>> and that data is flushed to disk (a deferred write).
>> The second most common (and vanishingly rare) failure mode is a powerfail
>> _as_ a sector of disk is being written -- resulting in 'garbage data'
>> being written to disk.
>> The next possibility is 'cosmic rays'. If running on 'cheap' hardware (i.e.,
>> without 'ECC' memory), this can cause a *SINGLE-BIT* error in data being
>> The fact that the 'corrupted' filesystem passed fsck -without- any reported
>> errors shows that everything in the filesystem meta-data was consistent
>> Given *that*, there are precisely *TWO* ways that the 'results' that have
>> been reported could have happened.
>> 1) "Something" did a mv(2) of the various jail directories 'from' their
>> original location to the 'apache' diretory. This involves simply
>> *copying* the diretory entry from the jail's 'parent directory' to
>> the apache directory, and then marking the entry in the original
>> parent as 'unused'. Nothing other than the directory whre the jail
>> 'used to live', and the directory 'where it was found' are touched.
>> This occured _through_ the system 'mv' function, so all the normal
>> 'housekeeping' was done properly.
>> 2) it was -not- done though mv(2) -- but that requires that a whole
>> *series* of "corruptions" of the filesystem, _ALL_ of which had to
>> occur in 'exactly' the right way. They are:
>> I think it is safe to conclude that the probabilities -greatly- favor
>> alternative #1.
> OK. So after your comments and further research I concur with you on
> the mv but if it wasn't a human, then this might be exposing a serious
> security flaw in the jail system or the way EzJail implements it. The
> whole point of using jails is to protect things like this from
> happening. Given that the only jail that survived was the front-end
> Apache Web server/reverse proxy, then it is also safe to suspect the
> apache (or other) process running on it was able to perform a mv of
> the rest of the jails to it's own /usr/local/etc/apache22 directory.
> Is there no possibility is that after the system crash, the journal
> recocery process and/or fsck could have moved this directories ?
Also note that even the EzJail basejail was moved also, so it could be
a security hole in the way nullfs is used or in nullfs itself. but the
curious thing is that the basejail is supposed to be mounted read-only
so how did that get moved to the http-proxy jail??
That is why I suspect it could have been something in the boot process
like the journal recovery, fsck or something else with that kind of
privilege and when the EzJail filesystems were unmounted.