Tuesday, January 17, 2023

An Old Sysadmin Tries Out ChatGPT

(Editorial note: Pretend this post is a Star Trek episode with folks spouting techno-babble, use the words as place holders, and feel the flow of the story...)

A while back, I built some anycasting DNS resolvers using bind on a loopback, and FRR talking OSPF to a router.  I finally decided I should hammer together a script to turn off FRR if there was something wrong with the bind process so the servers would pull themselves out of service.

Why not try this new ChatGPT thingee?

I won't bore you with the code that was created, but you'll see my thoughts, and the input I gave to ChatGPT as I worked through this problem until... the end?

Wednesday, September 29, 2021

The Ramp Room

If you've ever spent time doing telecommunication work in Albuquerque, NM, the odds are you've had to work at 505 Marquette NW.  If you're local, you just talk about "505" and everyone knows what you mean.

505 was built during 1965 and 1966, and isn't a bad looking place, but it is showing it's age. Wikipedia quotes a July 10, 1966 Albuquerque Journal article: "It has twelve floors of office space above a wider six-story base which incorporates a parking garage on floors 2–6."  The Wikipedia article has this  picture:

Now that Albuquerque has become a center for shooting TV and Movie productions, you will sometimes see this building in the background.  Perhaps my favorite is from the Netflix series Daybreak where the building gets an FX makeover to fit the apocalyptic theme of the show:

I can't find a picture on the interwebs, so you have to settle for the one I took while watching Daybreak on Netflix.  Note: Palm trees, background hills, and building damage are all simulated.  I do, however, get my home Internet connectivity from the antenna on the roof.

Did you note that the parking garage was on floors two through six?  That means you have to drive up a ramp to get to the parking levels.  This architectural need has lead to one of  Albuqueruque's telecom black holes known as the Ramp Room.

To enter the Ramp Room, you need to get the the garage ramp, where air conditioning condensers congregate to release the heat of the inner building.  In between the first and second floors you'll find a non-descript door:

Upon opening the door, you'll find that it immediately blocks your way.  It feels like you aren't really welcome, but you move forward anyway...

...And end up in a narrow hallway.

Behind the door at the end of the hall way you immediately see a rack of batteries that were put in twenty years ago and never maintained. 

One step into the room and you can see the building punch downs -- the metallic path over which the business conversations from fifty years ago used to pulse.  Yet there is also a veneer of modern fiber optic communications that reside here as well.  In fact, they are the mainstay of the Ramp Room.

In every city, there is a nexus of communication that isn't ensconced in a pristine data center.  A place where various players gather.  But not because they really want to do so.  Rather, their customers demand they work together.

The path of least resistance nets small, out-of-the-way rooms where fiber optics from multiple carriers co-exist.  A place that isn't official, but becomes important.

Some players have been around for a long time.   They've been sold, bought others, been forced to divest assets, and any number of other acquisition issues.  Sometimes you forget what they're called now, and just refer to them by their "old" names.  After a while it you stop worrying about what to call them.

They have, and will persist.

Other are younger.  More nimble.  Willing to do things out of the ordinary, and force the old guard to play their way.

For a time.

They will become the old guard for a new generation.

Years roll on, and old gear is left in rooms like this.  Forgotten.  Useless technology from the past interwoven with the modern.

Old gear isn't ever removed.  It sits and drinks power from the building.

No one owns it, so no one cleans it.

Everyone is afraid to shut anything off.  Few know how to tell if something is still in use or not.

Within this building are a multitude of other places to interconnect -- but none have the diversity of the Ramp Room.

Friday, January 22, 2021

ModemCast -- a podcast

I tend to babble a lot.  Sometimes I find people that babble about some of the same things.  My friend Nick has talked me into helping him babble as well.  His take on things is here:

You might even enjoy some of our mutual babbling:

If you find it to your liking, do not blame me.

Sunday, May 26, 2019

Joe's Internet -- an IGP adventure

Before we can setup inter-network links, Joe's Internet (J-Inet)  need to get the internal network in order.  We'll implement an Internal Gateway Protocol (IGP) so all of our infrastructure will be able communicate with other parts of our network.

The IGP's job is to advertise what it knows to all of it's neighbors.  If there are multiple paths to a destination, it will pick the best of those available, and pick alternates in case of a path failure.  The IGP ensures communication resiliency as long as there are redundant paths.

Quick network overview

We're going to make sure each router has a single IP address on a "loopback" interface.  Loopbacks give you a management interface on the router that is not dependant upon any given interconnection link's state.  As long as the IGP is running properly, every router will know how to get to every other router via their loopback regardless of what path is in use.

Our infrastructure addressing is going to include the networks that interconnect the routers, and the loopback addresses on each router:

Joe's Internet -- IGP Map
J-Inet's IGP Map.  Click on it for more detail.

IGP by psuedo-code

No matter the platform, we have a few things we need to do to ensure our goal.  Those steps come down to:
  1. setup loopback interface address
  2. set global IGP parameters
  3. get IGP to adopt the loopback
  4. for each infrastructure link (one or more)
    1. setup interface address
    2. get IGP to adopt the interface
Pretty simple, really.  We're going to use OSPF in this example.  I'll assume a passing familiarity with OSPF, and just point out that we will only be using Area 0.  There's no reason to do things in a fashion more complicated than a single area.  (If that doesn't make sense, find some background material on OSPF, and come back.)

Also, we're going to pretend these routers run something that looks a lot like Cisco IOS from a syntax standpoint. Take a look at a config for routers R1 and R2:

!  Router R1
! global IGP parameters
router ospf 1
! setup loopback, and add to IGP
interface Loopback20
 ip address
 ip ospf 1 area 0
! setup infrastructure link, and add to IGP
interface ether 1
  description Link to R2
  ip address
  ip ospf 1 area 0
! setup infrastructure link, and add to IGP
interface ether 2
  description Link to R3
  ip address
  ip ospf 1 area 0

! Router R2
! global IGP parameters
router ospf 1
! setup loopback, and add to IGP
interface Loopback20
 ip address
 ip ospf 1 area 0
! setup infrastructure link, and add to IGP
interface ether 1
  description Link to R1
  ip address
  ip ospf 1 area 0
! setup infrastructure link, and add to IGP
interface ether 2
  description Link to R3
  ip address
  ip ospf 1 area 0
! setup infrastructure link, and add to IGP
interface ether 3
  description Link to R3
  ip address
  ip ospf 1 area 0

I will leave the construction of configs for routers R3 and R4 to our readers.

While your setting things up on a Cisco router, you will find the following commands useful to help you track your progress:
  • show ip ospf neighbor
  • show ip route
  • show ip route ospf

Adding Complications to the IGP setup

Most of the time, I  tell people to stick to the simplest setup they can.  The configs above should let you build a simple IGP to keep your network running.  However, there are some things you may encounter that require a bit more thought into your setup and will force your to consult your platform's documentation.  A couple of the more common issues and a method to use them are:
  • password / encryption / message-digest for IGP on links
  • link type (point-to-point vs. point-to-multipoint, etc.)
  • changing link metrics
Consult your platform documentation, but to implement the above features, we could have added something like the following to our configs:
router ospf 1
   area 0 authentication message-digest
   auto-cost reference-bandwidth 10000
interface ether-whatever
   ip ospf network point-to-point
   ip ospf message-digest-key 1 md5 <some key/password>


All of our routers now know how to reach each other because the IGP keeps them all informed.  If we add new routers, we simply need to add them into the IGP, and they will be added to the reachability information that all of the routers share.

Our eventual goal is to ensure that the customers share this type of reachability.  To do that, we will use BGP to leverage the IGP information for anything else we need to reach on this network.  That will be the subject of the next episode of J-Inet...

Sunday, April 14, 2019

Joe's Internet -- Addressing and Interconnection

In the last installment, Routing Policy -- interconnecting with other networks, we talked about the business relationships Joe's Internet (J-Inet) might have with other networks.  Now we'll look a little closer at the technical side of how those relationships are enacted on the network itself.

Before we get too much closer to J-Inet, I want to point out that I'm going to be using a single network model for continuity between posts.  I've worked out IP addressing (v4 & v6), consistent routing policies, and a topology that should suffice for many posts. I've got several posts lined up that should help show at least one path to building a scalable network.

The Interconnects

J-Inet has two transit providers (Transit 1 & 2), three customers (Customer A, B, & C), and a single Peer (Peer 1).  Customer C is a bit of stretch for this conversation.  They aren't an autonomous system, but we'll look at them anyway, as it may lead some readers to revelations they might not have otherwise.

J-Inet has four Points of Presence (POPs) where equipment is housed.  Most customers of J-Inet are like Customer C.  They simply buy transit from J-Inet.  These customers actually make up the bulk of J-Inet's revenue.  But they're pretty boring from an network perspective.

The Network, Prefixes, and Policy

The network itself is pretty simple, and looks like this:
Joe's Internet -- Simple Map
Over view J-Inet's network and interconnects.  Click on it for more detail

J-Inet has two blocks of address space it uses internally, that we will advertise to anyone that interconnects.  Those prefixes are: and

Customer A uses  That block comes from J-Inet's prefixes, but they are allowed to use it as long as they are a customer of J-Inet.  J-Inet also allows Customer A to advertise this prefix across any interconnects they may have with other networks.  J-Inet will pass all of the prefixes it knows about to Customer A.

Customer B uses, which was allocated to them from their Regional Internet Registry.  They advertise this prefix to us, and any other interconnects they have according to their own routing policy. J-Inet will pass advertise all routes to Customer B.

Customer C uses  They don't have any other interconnects, so they aren't really an autonomous system (AS).  They're in this mix as a kind of anti-example -- a customer that really only acts as an extension of J-Inet's network and routing policy.  Neither Customer C nor J-Inet will advertise prefixes to one another.

Peer 1 is another ISP that happens to have equipment in the same building that houses POP 1.  J-Inet and the Peer 1 decided that they should interconnect directly.  They will be announcing to J-Inet.  J-Inet will pass internal and customer prefixes to Peer 1, but will not send any routes learned from transit providers.

Transit 1 and Transit 2 are networks that have customers and other interconnects.  Since they are transit providers, they have promised to get traffic to any destination. However, they bouth announce prefixes to us, J-Inet can make an informed decision about which network should be used for any given prefix.  J-Inet will announce internal routes and customer routes to both transit providers.  J-Inet will not advertise routes learned from Peer1.  Lastly, J-Inet will not advertise any transit routes back to either transit provider.  (i.e. J-Inet will not allow transit to the these providers.)

At this point, we have a pretty good idea of J-Inet's network topology and how it interconnects with customers, peers and transit providers.  In our next installment, we'll look at some of the internal housekeeping of J-Inet's network is put together, and later use this internal configuration to make our interconnects simpler.

Monday, April 1, 2019

Routing Policy -- interconnecting with other networks

The next few posts about Joe's Internet (J-Inet) will start with our routing policy, and refine it based on some ideas concerning how to describe the interconnections between networks.  We'll get a look at J-Inet's network soon, and eventually we'll see how the router configurations enact the routing policy.  For now, let's get a grounding in how we can describe links between networks, and how they relate with routing policy.

J-Inet has a simple management philosophy when it comes to interconnecting with other networks.  In summary:
  • keep things as simple as possible
  • minimize costs
  • maximize revenue

These thoughts are going to be the basis of our routing policy -- make money while keeping things simple and cheap.

Types of interconnection

The types of networks we are discussing are referred to as Autonomous Systems (ASes).  Within any AS are destinations referred to as prefixes or routes.  When an AS interconnects with another, they advertise their prefixes to the other -- essentially offering a path to that destination.

It turns out that if you learn about three simple types of business relationships, you can understand the vast majority of AS interconnection types:

  • Transit  -- If you buy Internet connectivity from someone, they are a transit provider.  You can hand them traffic, and they will deliver it to the desired recipient, or will pass that traffic across their network to a provider that can get the traffic closer to its destination.  For our purposes, if you pay people to take your traffic, you're buying transit.
  • Customer -- If you supply transit to another network, they are a customer.  The simplest customer relies on you completely, and more complex customers look at you in the same way you look at your transit provider(s).  For our purposes, customers pay you to be their transit provider.
  • Peering -- Sometimes, networks find that it is mutually advantageous to interconnect, and they decide to do so without either side paying for the privilege.  Common customers, shared geographic regions, and other factors may influence if networks will become peers.  We aren't going to worry about *why* some of our neighbors a peers -- we will just know that they are.

Egress Policy

As the number of interconnections increase, the odds of having more than a single path to a given prefix grows.  Our policy will give us guidance about which type of interconnection we wish to use, and we will rank prefix advertisements in this order:

  1. Customer -- Goal:  Maximize revenue.  Always use this path if available.
  2. Peers -- Goal: Minimize cost: Use this path before using one to a transit provider.
  3. Transit -- Goal: Minimize cost.  Use this path only if no better path exists.

Prefix Advertisement Policy

Business policy shaped how we prioritize the choice of multiple egress paths.  Likewise, what we choose to advertise across an interconnection point is shaped by our policy as well.  In this case the reasoning becomes:
  • Transit  -- Goal: Minimize costs.  Only advertise internal prefixes and those that your are paid to carry (e.g. prefixes learned from Customers), thus minimizing traffic.
  • Customer -- Goal: Maximize revenue.  Advertise every prefix you know about to maximize traffic.
  • Peering -- Goal: Minimize costs.  Advertise the same set of prefixes as sent to Transit providers, in the hope of reducing transit costs.

Parting thoughts and a glimpse of the future

By describing AS interconnections as one of three different types, we've been able to express some routing goals that will support our business goals.  In the next installment, we'll look at a map of J-Inet, start to see actual prefixes, and classify the inter-AS interconnects.

In other future installments, we will start to put together router configurations that will allow us to enact this routing policy.  At first, we'll use Cisco vocabulary, and stick with IPv4, but the plan is to use a single network to show the use of other vendor's gear as well. Stick around, this should be fun...

Monday, March 25, 2019

Project Hummingbird

Originally posted to Facebook on January 12, 2018 I thought this should be available outside of Facebook:

This may be long, but I came upon an old floppy (a floppy!) today, and found the drawing you see here. I'm about to turn 50, so let me spin a tale from when I wasn't even half my current age...
Project Hummingbird
Some friends and I had to build "something" for our engineering senior design course. A friend had a model airplane remote control setup and wanted to play with "ducted fans." We came up with an idea for an aircraft we had a hard time describing to people.
Rather than wings, we were going to generate lift with pure thrust from the ducted fans! It was going to be the prototype for vertical take off and landing craft, and be a darn cool model to fly around.
Today, you can go buy a drone, and fly something similar, but in 1990, this was something no one had seen in radio controlled aircraft.
We had *NO* idea how much we'd bitten off.
To fit our budget constraint, we could only afford two of the ducted fan assemblies and engines. So we had to design a set of ducts to split the two sets of thrust into four so that we could control the craft.
To generate the thrust required, we had to have ducts that were over five inches in diameter. To keep control, we needed the nozzles over 20 inches from the center of the craft. This means that the craft needed to be about a foot high, and nearly four feet across.
Naturally, we designed it as a saucer.
We managed to fabricate one set of ducts, and showed that it generated more thrust than we had calculated. We got prototype control circuits mocked up before we realized we couldn't afford the gyroscopes available at the time. We learned the types of things you're supposed to learn in a senior design course.
But I *still* want to see that thing fly...