Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Fixed
-
Gump2-2.3
-
None
Description
FOG is defined right now as:
def getFOGFactor(self):
return round((float(self.successes) / (float(self.failures) + float(self.prereqs))), 2)
This is not helpful. Last month, jakart-oro
failed to build by coincidence around the time of a Gump
run and the problem was fixed before the next Gump run,
yet its FOG factor dropped in half. The responsiveness
to resolving build and integration problems is not taken
into account. It is likely that it is impossible for FOG
to mean anything because the factors that determine
a project's "health" (to borrow the term used by Adam Jack
in an email to the commons-dev and gump lists) are largely
orthogonal. FOG should probably be a vector and if you really
wanted a single number you could use the length of that vector.
One of the major problems with the current factor is that
a project's FOG factor continues to increase when no changes
have been made to the repository. It's not meaningful to
count only successful builds towards improving the factor
because no changes may have been made to the code base.
Ideally, you count only builds that cause no failures after
a change to the code base. FOG doesn't factor in how quickly
(or slowly) projects recover from build failures, and how often
changes to a code base cause failures (either in itself or in
other projects). Right now, a project with few code changes
(say once a month) that causes a failure 50% (1 every 2 months)
of the time the code is changed with a mean time of failure
correction of 1 week is deemed more reliable (higher FOG factor)
than a project that makes frequent changes (say every day) to
its code base and causes a failure 10% of the time (1 every 10
days) with a mean time of failure correction of 1 day. That's
because FOG is based almost entirely on consecutive successful
Gump builds, which is not useful.
Rather than propose a new meaningless way of computing FOG, I
suggest FOG stop being used temporarily. It would be more
useful to break out and report all of the independently useful
metrics and through experience over time determine how each
impacts the overall notion of FOG and eventually derive a
way of condensing the information (perhaps a FOG factor,
perhaps a vector) in a useful manner. Right now the state
of affairs is that someone decided FOG was a cool thing, so
it's being reported and people see this number and mistakenly
think a project with half the FOG factor of another is
less dependable and therefore avoid using it. Alternatively
an explanation of what FOG is intended to compute and how
it is computed should be placed on Gump reports that list
FOG so that consumers of the information will not be misled.