Cisco Router Commands Bail Me Out of BGP Issue

When I got to work this morning I was hearing the all to frequent "Internet is down."
My first knee-jerk reaction to this question is always "Does Al Gore know about this?" I digress. Email confirmed that our carrier had opened a ticket on our circuits (we have 2 T1's in a bundle) because they noticed an issue with an upstream DS3 card and replaced it. The ticket comments said they saw both circuits testing clean. Anyway, a quick console into our Cisco router showed me some interesting info...

2821# show int multilink 1
Multilink1 is up, line protocol is up
  Hardware is multilink group interface
  Description: Verizon MPLS BCFG3BK0001
  Internet address is 68.139.69.194/30
  MTU 1500 bytes, BW 3000 Kbit, DLY 100000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation PPP, LCP Open, multilink Open
  Listen: CDPCP
  Open: IPCP, loopback not set
  Keepalive set (10 sec)
  .....
  .....

2821# s ppp multilink

Multilink1
  Bundle name: blah1
  Remote Endpoint Discriminator: [1] blah1
  Local Endpoint Discriminator: [1] p1234567-1765432
  Bundle up for 1w0d, total bandwidth 1544, load 4/255
  Receive buffer limit 12000 bytes, frag timeout 1000 ms
    0/0 fragments/bytes in reassembly list
    18 lost fragments, 1627759 reordered
    0/0 discarded fragments/bytes, 1 lost received
    0x660BB1 received sequence, 0x77660C sent sequence
  Member links: 1 active, 1 inactive (max not set, min 1)
    Se0/2/0, since 00:05:56
    Se0/3/0 (inactive)
No inactive multilink interfaces

That's odd, the logical bundle is up, up.

2821# s int se0/2/0
Serial0/2/0 is up, line protocol is up
  Hardware is GT96K with integrated T1 CSU/DSU
  Description: M1/HCGS/415028/SC
  MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation PPP, LCP Open, multilink Open
  Link is a member of Multilink bundle Multilink1, loopback not set
  Keepalive set (10 sec)

2821# s int se0/3/0
Serial0/3/0 is up, line protocol is up
  Hardware is GT96K with integrated T1 CSU/DSU
  Description: M1/HCGS/415029/SC
  MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation PPP, LCP Open, multilink Open
  Link is a member of Multilink bundle Multilink1, loopback not set

Both physical T1's in the bundle are up, up.

2821# show service-module  
Interface Serial0/2/0
Module type is T1/fractional
    Hardware revision is 1.2, Software revision is 20090205,
    Image checksum is 0x4144A7, Protocol revision is 0.1
Receiver has no alarms.
Framing is ESF, Line Code is B8ZS, Current clock source is line,
Fraction has 24 timeslots (64 Kbits/sec each), Net bandwidth is 1536 Kbits/sec.
Last module self-test (done at startup): Passed
Last clearing of alarm counters 41w1d
    loss of signal        :    3, last occurred 24w6d
    loss of frame         :   77, last occurred 03:18:58
    AIS alarm             :   73, last occurred 03:18:58
    Remote alarm          :   74, last occurred 03:18:35
    Module access errors  :    0,
Total Data (last 96 15 minute intervals):
    0 Line Code Violations, 64 Path Code Violations
    4 Slip Secs, 415 Fr Loss Secs, 0 Line Err Secs, 25 Degraded Mins
    73 Errored Secs, 4 Bursty Err Secs, 415 Severely Err Secs, 0 Unavail Secs
Data in current interval (848 seconds elapsed):
    0 Line Code Violations, 0 Path Code Violations
    0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
    0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

Interface Serial0/3/0
Module type is T1/fractional
    Hardware revision is 1.2, Software revision is 20090205,
    Image checksum is 0x4144A7, Protocol revision is 0.1
Receiver has no alarms.
Framing is ESF, Line Code is B8ZS, Current clock source is line,
Fraction has 24 timeslots (64 Kbits/sec each), Net bandwidth is 1536 Kbits/sec.
Last module self-test (done at startup): Passed
Last clearing of alarm counters 41w1d
    loss of signal        :    3, last occurred 24w6d
    loss of frame         :   75, last occurred 03:18:59
    AIS alarm             :   70, last occurred 03:18:59
    Remote alarm          :   72, last occurred 03:18:36
    Module access errors  :    0,
Total Data (last 96 15 minute intervals):
    0 Line Code Violations, 42 Path Code Violations
    2 Slip Secs, 415 Fr Loss Secs, 0 Line Err Secs, 40 Degraded Mins
    70 Errored Secs, 1 Bursty Err Secs, 415 Severely Err Secs, 0 Unavail Secs
Data in current interval (810 seconds elapsed):
    0 Line Code Violations, 0 Path Code Violations
    0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins
    0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs


Lots of alarms and errors, but, that was several hours ago.
Can I ping our MPLS gateway?

2821# ping 46.144.32.271

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 46.144.32.271, timeout is 2 seconds:
!!!!!
Success rate is 100 per

Publish Post

cent (5/5), round-trip min/avg/max = 20/20/20 ms

Yep, that's really weird. Layer 2 is good. Time to check routing info...

2821# s ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       .....
       .....

Gateway of last resort is not set

"This is a problem!"

2821# show bgp neighbor
BGP neighbor is 46.144.32.271,  remote AS 65000, external link
  BGP version 4, remote router ID 0.0.0.0

"Zoiks!!! Carrier hasn't programmed their side of the circuit to advertise BGP info. Quick, to the Bat-phone, time to call the carrier. These simple commands can show you a wealth of information if you just slow down and look. Hope this helped!