Bug That Took Down 8% of Ethereum’s Validators Sparks Worries About Even Bigger Outage

Ethereum
just
had
one
of
those
moments
where
the
big
story
is
what
didn’t
go
wrong.

A
bug
in
Ethereum’s
Nethermind
client
software
–
used
by
validators
of
the
blockchain
to
interact
with
the
network
–
knocked
out
a
chunk
of
the
chain’s
key
operators
on
Sunday.

It
was
a
manageable
incident,
but
the
episode
revived
a
long-simmering
debate
in
the
Ethereum
ecosystem
around
the
need
for
“client
diversity.”
Some
experts
took
the
opportunity
to
point
out
how
bad
things
could
have
been
if
another
client
software,
Geth,
the
chain’s
most
popular
execution
client,
had
gone
out;
the
question
is
whether
Ethereum
could
have
kept
going
since
Geth
stands
out
as
a
possible
single
point
of
failure
for
the
network.

Nethermind

powers
around
8%
of
the
validators
that
operate
Ethereum,
and
this
weekend’s
bug
was
critical
enough
to
pull
those
validators
offline.
Ethereum
stayed
up
and
running
despite
the
issue,
and
Nethermind’s
developers
released
a
patch
fixing
things
within
hours.
The
main
consequence
of
the
bug
was
that
modest
financial
penalties
fell
onto
some
Nethermind-based
validators,
but
the
Nethermind
incident
followed
a
similar
outage
earlier
in
January
that
impacted
Besu,
the
client
software
behind

around
5%
of
Ethereum’s
validators.

The
back-to-back
outages
have
reignited
a
spirited
discussion
on
X,
the
platform
formerly
known
as
Twitter,
around
Ethereum’s
persistent
problem
with
client
diversity.
The
idea
is
that
the
network
becomes
more
resilient
if
it’s
not
dependent
on
any
single
client
software.

Around

85%
of
Ethereum’s
validators
are
currently
powered
by
Geth,
and
the
recent
outages
to
smaller
execution
clients
have
renewed
concerns
that
Geth’s
dominant
market
position
could
pose
grave
consequences
if
there
were
ever
issues
with
its
programming.

Geth,
which
stands
for
“Go
Ethereum,”
is
primarily
developed
and
maintained
by
the
Ethereum
Foundation,
the
main
nonprofit
that
supports
Ethereum
development.

Geth
hasn’t
been
totally
immune
from
bugs
(no
software
is),
but
it
has
never
suffered
from
a
critical
outage
like
the
ones
that
hit
Nethermind
and
Besu.
If
it
did,
the
consequences
would
be
far
more
serious
for
Ethereum.

Depending
on
the
nature
of
the
bug,
a
Geth
glitch
could
halt
the
entire
network,
rendering
it
impossible
for
validators
to
add
new
blocks
to
the
blockchain.
Ethereum
is
also
programmed
to
penalize
validators
that
fall
offline
or
break
the
network’s
rules,
meaning
thousands
of
Geth-based
validators
could
be
held
financially
liable
in
the
event
of
a
bug,
and
the
penalties
could
get
even
bigger
if
the
bug
proves
difficult
to
patch.

Notably,
some
of
the
leading
services
that
stake
Ethereum
on
behalf
of
users
–
effectively
turning
people
into
validators
with
less
of
a
headache
–
rely
on
Geth
to
power
their
operations.

Cygaar,
a
crypto
educator,

noted
in
an
X
post
that
“Ethereum
has
terrible
client
diversity,”
adding
that,
“A
critical
issue
in
Geth
can
lead
to
potentially
millions
of
ETH
being
destroyed
from
validators
running
Geth.”

Cygaar
cited
data
from
the
website

execution-diversity.info
noting
that
popular
crypto
exchanges
like
Coinbase,
Binance
and
Kraken
all

rely
on
Geth
to
run
their
staking
services.
“Users
who
are
staked
in
protocols
that
run
Geth
would
lose
their
ETH”
in
the
event
of
a
critical
issue,”
Cygaar
wrote.

DCinvestor,
a
pseudonymous
crypto
investor
with
a
large
social
media
following,

claimed
in
an
X
post
that
they
were
pulling
their
staked
funds
from
Coinbase
until
the
company
switches
its
validator
operations
to
a
system
that
relies
less
on
the
Geth
client.
“I
can’t
ignore
the
risks
of
what
appears
to
be
a
single
client
staking
setup
(reliant
on
Geth)
at
this
time,”
DCinvestor
wrote,
adding
that
“I
could
stand
to
lose
a
large
percentage
of
my
deposit”
if
things
go
wrong.

For
Daniel
Hwang,
a
validator
expert
who
helms
the
Kintsugi
Tech
incubator,
the
attention
on
Ethereum’s
client
diversity
stems
in
part
from
the
fact
that
the
network
is
held
to
a
higher
standard
than
competing
chains.

“Almost
all
other
chains
don’t
have
the
type
of
client
diversity
that
Ethereum
has,”
Hwang
told
CoinDesk
in
an
interview.
“Most
are
just
running
on
one
client.”

“I
think
perhaps
the
bar
is
just
held
higher
for
Ethereum
because
it
is
the
dominant
smart
contract
chain,”
he
said.

While
Geth
has
a
strong
record
of
reliability,
Hwang
says
many
of
Ethereum’s
validators
just
default
to
using
it
(rather
than
alternatives
like
Nethermind)
out
of
laziness.
In
his
experience,
validators
are
“not
doing
their
own
research”
into
the
strengths
and
weaknesses
of
competing
client
software.

The
Ethereum
Foundation
urges
validators
to
help

improve
client
diversity,
and
Dankrad
Feist,
one
of
its
researchers,
was
widely-cited
this
week
for
a

2022
article
imploring
validators
not
to
use
majority
clients.
Nethermind’s
development

was
also
funded,
in
part,
by
a
2018
grant
from
the
Ethereum
Foundation.

Hwang
likened
Geth’s
dominance
despite
all
this
to
an
old
business
adage:
“Nobody
gets
fired
for
buying
IBM.”
In
other
words,
if
everyone
else
is
using
Geth,
then
it
would
be
hard
to
blame
an
upstart
validator
for
using
it,
too
–
even
if
things
eventually
go
awry.

Counterintuitively,
Hwang
sees
a
silver
lining
in
the
recent
Nethermind
and
Besu
bugs.

“I
don’t
want
to
say
it’s
great
that
a
client
suffered
a
bug,
but
I
think
it’s
great
if
it
begins
to
get
people
thinking
about
responsibilities,”
he
said.
“Validators
should
have
been
checking
this
s–t
for
themselves
instead
of
just
picking
it
off
a
supermarket
shelf,
and
then
raising
their
hands
when
things
go
wrong.”