[6bone] [NOTIFY] XS26 service/peering outage

[Jan Oravec]

On Tue, Dec 17, 2002 at 02:04:41PM -0500, John Fraizer wrote:
> 
> On Tue, 17 Dec 2002, Jan Oravec wrote:
> 
> > On Mon, Dec 16, 2002 at 09:26:00PM +0100, Stephane Bortzmeyer wrote:
> > > On Monday 16 December 2002, at 18 h 3, 
> > > Petr Baudis <pasky@xs26.net> wrote:
> > > 
> > > > and bgpd. Apart random crashes in various time periods (from few minutes to
> > > > weeks) 
> > > 
> > > A funny things about distributed systems is the difference in testimonies :-) 
> > > We never had a Zebra crash.
> > 
> > You were probably never running zebra on router with 2048 interfaces, having
> > 2k static routes redistributed into BGP, 10k internal BGP routes, about 200
> > prefixes in IGP and about 300 external BGP routes.
> 
> Find me, outside of 6bone, *ANY* quasi-production router, I'm talking
> about on the entire planet, that has 2048 interfaces.
> 
> This sounds like more a problem of you should be splitting that interface
> load between many routers than one of there being a problem with Zebra.

There were static routes only on these interfaces redistributed into BGP,
not IGP. We had at most 200 prefixes in IGP -- nexthops. 

> > The result: CPU time of ospf6d reached sometimes ~100%, zebra was unable to
> > save config files, zebra sometimes freezed for 5 minutes or so making ospf6d
> > and bgpd also freeze, sometimes something crashed and so on.
> 
> I don't doubt it.  Have you tried to do the same on a 7513?  I'll bet a
> dollar to a doughnut that it will croak on that interface count as well
> and that the SPF calculation will take forever, delaying convergence, and
> that it will burn proc like it was going out of style.

See above, it has nothing do to with IGP.

> > 
> > Zebra is not ready for production networks.
> > 
> 
> I beg to differ.  Your "network" from what ou've described, is
> under-engineered.  What was the purpose again of terminating 2000+
> endpoints on a single router again?  You can't seriously think that any
> true production (BTW: most of us consider production to be equal to
> billable) network architect would put that many eggs in one basket can
> you?

Well, we had the same problems even with few users connected to one router.
The fact that there were 2048 users terminated at one router did not
generate routing-based problems, just side-effects, like unable to save
config files, etc.

> > > > problems, especially on FreeBSD we had visibly more problems than on ie. Linux.
> > > 
> > > I do not have personal experience with Zebra on FreeBSD but, on the Zebra 
> > > mailing list, you can clearly see there are far more FreeBSD users than Linux 
> > > ones so I doubt that Zebra is much worse on FreeBSD.
> > 
> > e.g. this one FreeBSD-only bug: you create interface in the system and in
> > order to zebra know about it, you have to restart zebra completely...
> > imagine doing 100 such changes a day... your BGP peers won't like you :)...
> > fortunatelly we have found an ugly way how to solve this...
> > 
> 
> Jan, forgive me if I'm wrong but, I don't recall seeing you post about
> this problem on the Zebra mailing list.  If you did, can you reference the
> archive?  

Unfortunatelly I don't have enought old archive. The problem was that BSD
systems did not support interface add/remove notification. I have sent PR
(see http://www.freebsd.org/cgi/query-pr.cgi?pr=33747) and after adding
RTM_IFANNOUNCE event I have e-mailed notification about that to Kunihiro
and/or(?) zebra ML.

> > > > Basically, zebra looks not to be prepared for the networks which change very
> > > > dynamically (our iBGP table changes very frequently as user prefixes appear and
> > > > disappear; it's also relatively big (in the 6bone world, at least ;) 
> > > 
> > > We use Zebra for default-free routers on the IPv4 Internet. The 6 bone is a 
> > > very small experiment when you compare it to the always-changing 100k routes 
> > > of the IPv4 Internet.
> > 
> > We have 10k always-changing routes in the IPv6. BGP implementation is
> > relatively good if you don't dynamically add/remove interfaces.
> > 
> 
> Again, that sounds like an implementation issue in your network.
> 
> Let us assume you are routing out of 5 cities (example)
> 
> Router-1
> 3ffe:80e0:0000::/36
> 
> Router-2
> 3ffe:80e0:1000::/36
> 
> Router-3
> 3ffe:80e0:2000::/36
> 
> Router-4
> 3ffe:80e0:3000::/36
> 
> Router-5
> 3ffe:80e0:4000::/36
> 
> 
> All of these routers can announce your aggregate 3ffe:80e0::/28 to their
> eBGP peers while announcing the specific "pool" /36 they assing tunnel
> space from into iBGP/IGP.
> 
> An IGP route to 3ffe:80e0:0000::/36 will still attract traffic destined to
> 3ffe:80e0:0fff::/48 and there is no need for the more specific ::/48 route
> in the IGP or via iBGP announcement to the other routers in the AS.
> 
> If you are not assigning each router a "pool" from which you assign tunnel
> space, NLA assignments, etc from, you are making your network topology
> much more complicated than it needs to be.

We want our users to be able to move between PoPs without changing their
assignment. It is obviously more complicated, but possible. Better than
making simple solutions for us it to make a solution that is simple for
users. We have enought skills to do that and it is worth of month of
coding new routing software.

> 
> The interface problem is another story but again, as you stated, it is an
> issue with freebds and NOT one with Zebra. I the OS doesn't properly
> report interface changes, you can't expect Zebra to keep up.

It does. At least for 11 months.

> > We are not saying zebra is bad, we just say it is not usable in our
> > environment.
> 
> Actually, you did say that Zebra was bad.  You didn't use those words but,
> it was abundantly clear that you were implying that it was bad.
> 
> I would like to stress that I don't know of any routing suite that is
> going to be happy in the environment I'm picturing based on your
> description of your network topology.  Perhaps you might look into that a
> bit.

I do not know about any too, when we try our software, I will post a
message about results.


-- 
Jan Oravec                           XS26 coordinator
6COM s.r.o.                          'Access to IPv6'
http://www.6com.sk                   http://www.xs26.net
+421-903-316905