[6bone] 6bone-mess-01.txt

[Tim Chown]

> Comments, please.
> 
> (Anyone with more than 5 peers with transit should feel the sting of guilt
> now. :-)

Some comments on -01.txt.  I think the dcoument as is makes the point well.
All we need are some agreed answers :)  I notice for example that from the
IETF here my IPv6 RTT to home (UK) is 400ms, as opposed to 100ms via IPv4.  

(This is not a US-Europe thing per se as the Abilene-Europe link has near
identical RTT due to use of dual-stack links following a very similar path,
and no "errant" tunnels)

- Comments:

   Currently, IPv6 Internet is, globally considered, inseparable from
   the 6bone network.  The 6bone has been built as a tighly meshed
   tunneled topology.  As the number of participants has grown, it has
   become an untangible mess, hindering the real deployment of IPv6 due
   to low quality of connections.  This memo discusses the nature and

- The key problem is that many users wish to use IPv6 for daily, production
- use, and this is not possible due to the way 6bone prefixes are advertised
- and transit is too often given.

- We should also point out now that SubTLAs are easier to get in the 2001:
- space, we risk the 6boneisation of the so-called "production" IPv6 space.

- We could also argue that there is less genuine demand for 6bone pTLAs; indeed
- new SubTLA owners no longer require 6bone experience as a precondition for 
- that allocation(?)

1. Introduction

   Currently, IPv6 Internet is, globally considered, inseparable from
   the 6bone network.  The 6bone has been built as a tighly meshed
   tunneled topology.  As the number of participants has grown, it has
   become an untangible mess, hindering the real deployment of IPv6 due
   to low quality of connections.

- I think "low quality" doesn't perhaps state the problem strongly enough.
- perhaps mention the lack of stability and predictablity, hampering the use
- of IPv6 by *any* users, whether on a 6bone or production prefix site, on
- a day-to-day basis.

   This memo tries to outline some ways to gain better connectivity for
   the future IPv6 Internet.  This can be done by increasing the quality
   of 6bone (in some applicable parts) and moving the 6bone network to

- Indeed applying the principles that make the current IPv4 Internet usable
- on a day-to-day basis.

   It should be noted that as addressing and routing are quite
   independent, the same problems also exist, in addition to 6bone-
   allocated pTLA's, with RIR-allocated "sTLA" addresses too.  It can
   often be assumed, though, that those organizations are at least
   slightly more knowledgeable and serious than an average pTLA holder,
   due to the process it takes to get an sTLA if you're not an
   equivalent of Local Internet Registry (LIR).  Nevertheless, these are
   considered as one category.

- Yes, the slacker SubTLA rules now make this a concern.

   This is the root cause why 6bone network is a mess: as there is no
   hierarchy, most sites try to gain good connectivity by creating
   tighter mesh to other pTLA/sTLA's: increase the number of virtual
   peers by creating tunnels.

- And in many cases these tunnels are not optimal, instead often spanning
- a large number of IPv4 hops, with some pTLA owners collecting peerings as
- if they were "trophies".  You might expect a typical pTLA to have 5-10
- peerings, in many cases pTLAs have many times this, coupled with lack of
- controls on transit.

   Using IPv6-in-IPv4 tunneling is also not so bad in itself, if used
   properly; for example, tunneling over resilient IPv4 connectivity
   within one AS could be seen as highly advantageous to using
   dedicatede circuits for IPv6 under some circumstances.  However, the
   problem is that tunneling hides the natural, underlying topology: a
   link may actually consist of many hops, going through many different
   administrative and technical entities.  As such, it becomes very
   difficult to debug and notice the real problems when using "long"
   tunnels.  Tunneling can also be used to gain connectivity in the
   absence of native IPv6; as long as one does not provide transit
   through such "long tunnels", you can only shoot yourself in the foot.

- Indeed, it is the transit that causes the problem.

   Full transit, especially in conjunction with points above is the
   worst problem: there are no longer useful metrics to choose best
   paths (as AS-path length, the most imporant global metric in BGP path
   selection, becomes next to useless), and the network topology gets
   extremely complicated and unstable.  Some metric can naturally be
   assigned locally, with usually significant effort, but these are
   indeed often only locally useful.

- Witness the resulting mess of paths that cross the atlantic two or even four
- times between sites on the same continent.

4. Coping with the Administrative Problems

   Some of the problems are more of an administrative nature.  It is
   clearly visible that under current pTLA assignment policy
   [6BROUTING], there are many organizations which are not significant-
   enough ISP's, transits or such but are still getting pTLA's.  This is
   a double-edged sword: in the absence of IPv6 support by their
   upstreams, it could be argued it's good that at least someone will do
   something about IPv6 (note, extending this argument, they should also
   provide the same services as their ISP would -- ie. not only reap the
   benefits but also take up the responsibilities); on the other hand,
   these organizations usually have neither skills, equipment nor
   customers to understand the operational aspects of being part of a
   world-wide network, and are the ones worst burdening the 6bone.

- I think the number of providers is now growing in most regions to the extent
- that this is not the case in the same way as it was 2-3 years ago.

   Therefore, as one step, the requirements for a pTLA should be made
   stricter by revising [6BROUTING]; the revised edition should also
   take more stance on what kind of organization would be eligible for
   an allocation (a difficult thing to define precisely, to be sure).

- Agreed 100%, and this is now happening as a result of a recent pTLA case.

   Another thing to consider would be whether pTLA allocations could be
   revoked from "irresponsible" operators (even more difficult thing to
   define as before), possibly as a consequence to not conforming to
   possible new guidelines for pTLA's.  This might eliminate some
   specific problematic operators, but it might be that then the focus
   would just shift to the others; in general, removing existing
   allocations is a very harsh measure and it is unprobable that it
   could be done except for special reasons.

- There should be some way for pTLAs to be reviewed or revoked.  But in theory
- filters could also be applied - someone has to offer connectivity for the
- "rogues" to be a problem; the "blockade" method you describe below is also
- interesting for this.

5. Coping with the Connectivity Problems

   There are a few main approaches to the connectivity problems
   described above:

      1. Wait until a significant number of transit providers support
         IPv6 and only try to get anything stable then

- I think the path being pursued by the research networks (academic) is to
- look at policies, inparticular through use of community tagging, to establish
- a stable intrastructure between those networks (in Europe, the US and beyond).
- That stability is a prerequisite to using IPv6 applications (e.g. international
- conferencing) with confidence that the applications will run as desired.

   Also, this leads to waiting.  As it is, IPv6 connections around the
   world are, generally speaking, very poor.  It is irresponsible to
   turn on IPv6 by default on e.g. operating systems, as that would only
   lead to a lower quality for the user: for example, if a website is
   IPv4/6 dual-stack, trying to use IPv6 by default is often much
   slower, and should be avoided until the network is in a reasonably
   good shape.

- Well, it's deeper than that, if people go dual stack on web servers, and
- advertise AAAA's, and the browsers (as is I think universally the default)
- try AAAA's before A's, there will be delays for users where the IPv6
- connectivity is broken.   As you say, turning on IPv6 on the "clients" is
- currently a "brave" step for site administrators, and only likely to cause
- pain.

5.2. Adjust BGP Path Selection or Route Visibility

   Some might even suggest BGP specification modifications (e.g. taking
   a lower layer end-to-end latency into account), but those are
   definitely considered out of scope: only operational measures are
   discussed.

- Some level of network monitoring can give clues to at least perhaps tune
- general BGP/filter policies.   Indeed most 6bone-related problems are only
- spotted by user traceroutes in practice right now.

   MED is next to useless in itself: with "always-compare-med" -like
   option, it can be used to influence the path from between two routes
   of equally long AS-path -- but the path length was deemed a next-to-
   useless metric in 6bone, so this might only help if AS-paths are very
   short, e.g. 1 or 2 AS's.

- I suppose that the IPv6 BGP would like to know the number of hops in the
- underlying IPv4 tunnel, if a plain hop metric is being used, as a 2-hop
- AS path may indeed pass through many more IPv4 AS's due to tunnels.

5.2.1. Example of Route Visibility Control

   In addition to the implemented example of 6NET communities, see
   below, one could sketch additional ones, like often used in IPv4.
   Some possibilities are show below.

- 6NET is introduced rather suddenly here?

   (Note that there are several vendor-specific problems with no-export
   communities, see Appendix A for more information.)  There are many
   possibilities; these are indeed just examples.  It is not the purpose
   of this memo to try to standardize (requiring IANA action) "well-
   known" communities; but either to give ideas what could be
   implemented in a non-global scope, e.g. regionally or locally, or
   come up with simple "if you don't know what to do, you can do this"
   -rules.

- Well, common meaning of the values would be nice, to be able to look at
- other policies and understand them more readily.   But I think that is as
- "standardised" as much as DSCP 8 is standardised for Scavenger/LBE.

   It should also be noted that this kind of fine-grained policing could
   prove to be very useful in traditional 6bone routing context -- but
   that is what we try to escape: simplicity is very rarely gained by
   adding more complexity.

- True.

6.1. Guidelines for End-sites

   End-sites, usually obtaining a /48 assignment, do not really need to
   do much, as their possibilities in affecting 6bone routing are, by
   design, limited (common prefix filters do not allow them to advertise
   their specific routes).

- Assuming those filters are in place!   Many sites use 6bone and SubTLA
- space combined to try out multihoming, which is one reason why "Old" pTLAs
- don't get handed back.

   The most important things for the end-sites are to:

      1. Ask their local ISP(s), both marketing and technical people,
         for IPv6 if they do not yet provide service: ISP(s) will not
         usually react unless there is an observed demand, and this will
         help in making production networks closer to reality.

- NOte that tunnel brokers and 6to4 can reduce this demand, as users and sites
- can connect and not knock on their admin or upstream ISP door, so long as 
- Protocol 41 is allowed.

      3. Due to designed restrictions regarding advertising more
         specific routes, usually obtaining two connections doesn't
         really help much and may cause problems if the ISP(s) implement
         ingress filtering.  It's better to just use one for simplicity
         unless there are special reasons to do otherwise.

- Multihoming experiments?

   Therefore the benefit (for the whole Internet) of having such an
   organization establish peering relations with others all over the
   world, just for itself and the customers, is small.

   These kinds of organizations must seek transit providers to create
   hierarchy: when there is enough hierarchy, the branch can be
   considered globally significant.  The most natural action would be to
   find these from the IPv4 upstreams, but that may not always be
   possible.  If not, as above, it's always good to try to underline
   that IPv6 transit service is something that's needed.  There are some
   organizations in the community which are often willing to perform
   transit free of charge, though, too.

- Agreed 100% - this single recommendation is perhaps key.

      2. Disbanding especially "long tunnels", in particular if these
         include providing transit.  If they cannot be removed, some
         extra measures like AS-path prepending (outlined in sections
         above) should be implemented.

- Some spring to mind, but best left unnamed here :)

6.3. Guidelines for Transit sTLA/pTLA's

   "Real" transit-providing organizations should consider whether they
   could provide limited transit to other organizations too, at least
   initially.  For example, research organizations could be able to
   provide research-to-commercial and commercial-to-research transit for
   others too, but fully commercial might be unacceptable due to higher-
   level policies.

- Yes, this may be a bit of a mire.  But it is very hard for anyone to offer
- commercial IPv6 services with the current 6bone-ness situation.

   To re-iterate, especially when providing transit for others, care
   must be observed so that it is done in a controlled fashion.  Transit
   networks are in the key position when applying and forcing proper
   policies on 6bone: working together, they might be able for force
   some of the more "irresponsible" pTLA/sTA's to act in a more sensible
   manner: together they might have much more power in the arm-wrestling
   match against the 6bone routing mess.

- The second key recommendation, I think.

   The scheme can be extended to other networks too: to add proper
   certain communities and preferences to routes learned through other
   networks (like commercial transits, other research networks, etc.)
   and that way the benefits of good, native connections can be more
   widely realized.  Perhaps then commercial pTLA's/sTLA's would need a
   lot smaller number of tunnels and transit services (at least to
   research organizations).

- This is happening at the moment, bringing in other academic/research networks
- such as Abilene.

   Something similar could be done in other networks too.  The key is to
   build something that prefers local (hopefully native, but short
   tunnels are also okay) connections and disallows any uncontrolled
   transit.

- Key recommendation #3 :)

   So, the role should change from addressing and routing to only (or
   mainly) addressing.

   A good question is where these stub/non-transiting sites/pTLA's would
   get transit from -- who would offer them that.  There must be
   solution for this if this path is to be adopted.

   One solution would be getting that from one of their ISP's (or their
   ISPs' upstreams), or secondarily, from some willing third parties.
   In some cases this may not be possible though.

   Ideas for the ways to proceed would be appreciated.

- I think that sites that already have IPv4 transit need to look for IPv6 transit
- from the same supplier.  This is easier in the research network case, where
- for example in 6NET the universities go to their NRENs, who in turn will look
- to 6NET (or GEANT) for transit, or who will have existing peering agreements
- with international links (e.g. to the USA).  I don't see that IPv6 would demand
- any different structure to the connectivity - do we really foresee that new 
- carriers will emerge for IPv6 traffic, or that (much more likely) existing
- carriers will offer (dual-stack) IPv6?




tim