GPS and the North American blackout of 1965

[SAR-EMT40]

This isn’t a rant but something that I find very interesting and hope maybe some others may find interesting also. There actually is a point.

NA blackout of 1965 Map

The cause of the failure was human error that happened days before the blackout, when maintenance personnel incorrectly set a protective relay on one of the transmission lines between the Niagara generating station Sir Adam Beck Station No. 2 in Queenston, Ontario and Southern Ontario. Instead of the relay being set to trip and protect the line if the flow of power exceeded the line's capacity, it was set for a much lower value.

As was common on a cold November evening, power for heating, lighting and cooking was pushing the electrical system to near its peak capacity, and the transmission lines heading into Southern Ontario were heavily loaded. At 5:16 p.m. Eastern Time a small surge of power coming from Lewiston, New York's Robert Moses generating plant caused the mis-set relay to trip at far below the line's rated capacity, disabling a main power line heading into Southern Ontario. Within seconds, the power that was flowing on the tripped line transferred to the other lines, causing them to become overloaded. Their protective relays, which are designed to protect the line if it became overloaded, tripped, isolating Adam Beck from all of Southern Ontario.

With no place else to go, the excess power from Beck then switched direction and headed east over the interconnected lines into New York State, overloading them as well and isolating the power generated in the Niagara region from the rest of the interconnected grid. The Beck and Moses generators, with no outlet for their power, were automatically shut down to prevent damage. Within five minutes the power distribution system in the northeast was in chaos as the effects of overloads and loss of generating capacity cascaded through the network, breaking it up into "islands". Plant after plant experienced load imbalances and automatically shut down. The affected power areas were the Ontario Hydro System, St Lawrence-Oswego, Western New York and Eastern New York-New England. Maine, with only limited electrical connection southwards, was not affected. The only part of the Ontario Hydro System not affected was the Fort Erie area next to Buffalo which was still powered by the old 25 Hz generators. Residents in Fort Erie were able to pick up a TV broadcast from New York where a local backup generator was being used for transmission purposes.”

The parts of the story that really fascinate me is not just the cause which is interesting but also the way people reacted. They all stayed put. The sat in the subways, some for 12 hours. Same with the elevators. It is our reaction to technology failing that really interests me. People couldn’t go to the bathroom, they were extremely uncomfortable because of the stale air and cold and couldn’t get water or food, but everyone was told what they were always told. Stay put, the power will be on in a couple of minutes. We are taught to rely on technology and it is expected and their experience told them that it would be on in a few minutes so they did what they were told. They overrode even the most basic human needs of food, water and comfort their belief was so strong.

But why bring this up? Well, I have another question. Do you know were you where 8/22/99?

I do. What is interesting isn’t what I was doing but why I remember what I was doing. I was in the northwest corner of CT between Bear mountain and Mt. Frissel. Now this was not critical navigation but more playing with my toy that I had been using for several years at that point. I pulled my shiny GPS out and nothing worked. Didn’t have a clue where it was. But how is it possible that billions of dollars of space based atomic clocks and all this technology could have stopped working? Well the real story is interesting only because it shows how complicated some of our technology has become. For people who program they understand that 1024 is a magic number. Back in the ‘70s when the GPS systems were actually designed memory was at a premium and programmers learned lots of creative ways to “shave bits” as it was called. One of the things they did was to only use enough bits to indentify 1024 weeks of the almanac information. That’s about 20 years which everyone knows is well past the useful life spans of computer stuff so everyone felt pretty safe with this. So we fast forward 1024 weeks to, you got it August 22nd 1999 and the firmware in my handheld GPS wasn’t ready for week 1024 and modulo math brings it back to week 0 (zero) which was January 6, 1980 for most systems. Now, for those that don’t know. A GPS only knows where it is because it knows what satellites are overhead. It only knows what satellites’ are overhead because it knows where in general it is in the world (or does an elaborate search of all possible satellites) and the date and time. Since it knows the orbits of the satellites it knows who is overhead to listen too. Mess with the GPS’s date and time and it doesn’t know where the satellites are because the ephemeris data doesn’t match up with any of the satellite orbits. That results in staring at a very shiny, expensive and clunky paper weight or a very large fishing lure.

There use to be many reasons to not fully trust that the data from GPS satellites was always going to be there. The reasons are fewer now but still relying on a high tech gadget that can have so much go wrong, even things that the experts didn’t consider could be dicey and I always like to have more than one option.

When I teach navigation classes and especially the introduction classes I teach on GPS the first things that I go over with students are the pages that reflect first, what the health and strength of reception of the satellites is and if it even exists. Next is the EPA or estimated positional accuracy. Knowing that my students are sending a waypoint for a search location to the base and that it has a positional estimated accuracy of 8 meters is considerably different than an EPA of 360 feet. They need to know useful from hurtful information. Searching for something in an area and assuming 20 foot accuracy is counterproductive to the effort when the actual EPA was 360 feet.

I also don’t teach staring at the compass page while walking. I usually have my students identify the location and the bearing and distance to the waypoint letting the GPS calculate that and then use their compass skills to walk the bearing and pace the distance. That allows the GPS to be the check for navigation and their compass skills to improve so that they are always comfortable with their "backup system".

The point was that some of these technologies are built on a house of cards. The blackout was caused by a simple trip breaker set incorrectly to a value actually less than it was capable of carrying. The other was an engineering design tradeoff decision that even though they knew about wasn't able to be universally corrected in time.

Just my $.02 with a little history,
Keith

 

[Kevin Rooney]

Interesting post, Keith. In terms of fragility, I understand that an EMP from a nuclear blast could have huge implications on lots of things electronic & mechanical. Not to mention human...

During the blackout of 1965 I was in my dentist's chair, having a tooth filled. He had just drilled it, was prepping it, and ... so, he packed it in cotton with instructions to return in the AM if the power was restored. I went to my girlfriend's house and had tomato soup and a balony sandwich. I was a junior in high school.

I looked up my hiking logs for the 8/22/99 date, and found I didn't hike that day. The date is sandwiched between here:

Aug 16, 1999 Mt St Helens, Washington
Aug 25, 1999 Washington via Huntington Ravine

and since that was a roadtrip year, most likely I was either still driving or just gotten home.

 

[carole]

I was hiking on 8/19/99 and 8/23/99. Didn't have a GPS then and don't have one now.

In '65 we built a fire in the fireplace and hoped my brother made it home safe from his new job at the mall.

 

[DougPaul]

A few technical nits:

Back in the ‘70s when the GPS systems were actually designed memory was at a premium and programmers learned lots of creative ways to “shave bits” as it was called. One of the things they did was to only use enough bits to indentify 1024 weeks of the almanac information. That’s about 20 years which everyone knows is well past the useful life spans of computer stuff so everyone felt pretty safe with this.

The issue was restricting the transmission bandwidth, not the memory requirements of the GPS satellites or receivers. Reducing the rate at which information has to be transmitted (bandwidth) makes the system more reliable. This is every bit as important now as it was back when the system was originally designed. The system still uses a 10 bit (1024) GPS week.

So we fast forward 1024 weeks to, you got it August 22nd 1999 and the firmware in my handheld GPS wasn’t ready for week 1024 and modulo math brings it back to week 0 (zero) which was January 6, 1980 for most systems. Now, for those that don’t know. A GPS only knows where it is because it knows what satellites are overhead. It only knows what satellites’ are overhead because it knows where in general it is in the world (or does an elaborate search of all possible satellites) and the date and time. Since it knows the orbits of the satellites it knows who is overhead to listen too. Mess with the GPS’s date and time and it doesn’t know where the satellites are because the ephemeris data doesn’t match up with any of the satellite orbits. That results in staring at a very shiny, expensive and clunky paper weight or a very large fishing lure.

You are referring to the almanac (a table of coarse orbits for all operational satellites), not the ephereris (the detailed orbit of a satellite). The almanac (with approximate time and approximate location) is only used to speed up acquisition (locking) of the satellites, not in the location computation. Each satellite transmits its own ephemeris, once the receiver has the ephemeris, the satellite can be used in the location computation.
Notes:
1. The almanac is transmitted by all satellites. It takes 10-30 min (IIRC) to download. The almanac is valid for 2? weeks.
2. The ephemeris is transmitted by each individual satellite, takes ~30 sec to download, and is valid for 4 hrs.

One can divide GPSRs into 3 generations:
1. Only a few channels (typ 5). (Up to ~12 satellites can be in view at one time.) These units probably required the almanac to startup and to operate. (I have never used one.)
2. 12 channel units. The almanac speeds startup, but is not required for operation. (Startup simply takes longer without a valid almanac.) Eg: Garmin eTrexes, 60CS.
3. Units with hundreds of thousands of search channels (eg SIRF StarIII GPS Chipset units such as the Garmin 60CSx.) These units don't even appear to use the almanac. The massive search capability allows them to find all satellites within view very quickly (with or without knowing the approximate time and location, often within 10 sec) and give the location as soon as the ephemeris is available.

The inability of your GPS to handle the GPS week wraparound was a bug in your GPS. (Judging by the date, I'm guessing your GPS might have been generation 1.) Some early units had such bugs, others had no problem. Some of the buggy units could be recovered with a long exposure to a clear sky, some with a reset and new almanac, and some others required a firmware update.

There use to be many reasons to not fully trust that the data from GPS satellites was always going to be there. The reasons are fewer now but still relying on a high tech gadget that can have so much go wrong, even things that the experts didn’t consider could be dicey and I always like to have more than one option.

When I teach navigation classes and especially the introduction classes I teach on GPS the first things that I go over with students are the pages that reflect first, what the health and strength of reception of the satellites is and if it even exists. Next is the EPA or estimated positional accuracy. Knowing that my students are sending a waypoint for a search location to the base and that it has a positional estimated accuracy of 8 meters is considerably different than an EPA of 360 feet. They need to know useful from hurtful information. Searching for something in an area and assuming 20 foot accuracy is counterproductive to the effort when the actual EPA was 360 feet.

I also don’t teach staring at the compass page while walking. I usually have my students identify the location and the bearing and distance to the waypoint letting the GPS calculate that and then use their compass skills to walk the bearing and pace the distance. That allows the GPS to be the check for navigation and their compass skills to improve so that they are always comfortable with their "backup system".

The point was that some of these technologies are built on a house of cards. The blackout was caused by a simple trip breaker set incorrectly to a value actually less than it was capable of carrying. The other was an engineering design tradeoff decision that even though they knew about wasn't able to be universally corrected in time.

The power grid is a commercially-run complex distributed system that is being run closer and closer to its limits (ie smaller and smaller safety factors). Control of such systems is very difficult and limiting the spread of failures is even more difficult. Its reliability may be getting worse.

The GPS control and satellites are also complex, but they have been qualified very carefully and are being managed and monitored very carefully with real-time status reporting (see RAIM and WAAS). Its accuracy and reliability have been increasing with time. However, consumer GPS receivers are not designed and checked nearly as well. (Aircraft-qualified GPSRs use WAAS and RAIM and are checked very well. Surveyors also check the current status of the GPS constellation.)

Complex systems have bugs and failure modes. In consumer use of GPSRs, there is also the additional problem that many (most?) users do not understand the units well enough to recognize and deal properly with malfunctions or often even normal operation.

In the case of EPA (or EPE=estmated position error), the GPS makes it an obvious issue by presenting a number to the user. (BTW, the number only takes some of the factors into account, so the error could be much larger. And it is only an estimate of the error statistics--not the actual error.) However, both maps and compasses have errors too but are much harder to estimate in the field. They also have failure modes. The wise navigator uses strategies that are tolerant of the errors in his instruments and keeps his eyes open for malfunctions.

Doug

 

[Remix]

From GPS World
-----------------------
A major GPS breakdown on one satellite occurred on January 1, 2004, as the result of an atomic clock failure on board SVN/PRN23. This caused transmission of erroneous data for more than a couple of hours, which clearly affected the vessels that where under Jobourg VTS surveillance inside Casquets traffic separation scheme (TSS) center that day.

According to the Jobourg VTS surveillance team who logged all vessel VHF reports during this breakdown, the following effects were observed: some GPS receivers were not able to provide a position for approximately one hour; some receivers transmitted position reports with very large errors (about 1 nm) during various periods of time; and finally some receivers, thanks to embedded anomaly detection processes or WAAS/EGNOS messages, were able to discard the information coming from the failed satellite quickly and provide good positions.

----------------------------
On Sunday, May 25, 2003, the atomic frequency standard on GPS satellite number 27 (SVN/PRN 27) became erratic. By Monday the range errors caused by the faulty clock had compelled the GPS satellite operators at Schriever Air Force Base to perform two contingency uploads in an attempt to correct the condition. Still, the atomic clock did not stabilize. Its effect on GPS users grew as signal ranging errors grew.

Finally, after satellite operators saw the user range error (URE) spike to approximately 38 meters at 1733 hours universal coordinated time (UTC) on May 26, they commanded the signal to "unhealthy," which effectively removed the satellite from service. (Information based on a presentation by Karl Kovach of ARINC Inc., "Two Recent GPS Integrity Anomalies," presentation to Interagency GPS Executive Board's Integrity Failure Modes and Effects Analysis team, July 29, 2003.)


----------------------------
On July 28, 2001, the ranging error on the broadcast signal of GPS SVN 22 reached 300 kilometers and remained in an HMI condition for almost two hours. Many aviation receivers detected this anomalous signal and either disabled their GPS navigation function or excluded the signal from their solution. Many other receivers, however, used the signal, generating erroneous position and timing solutions.

--------------------------------


Problems happen, so its important to have another means of navigating.

 

[Tobit]

In 1965, I wasn't born nor was I a twinkle in my fathers eye.

In 1999, not sure what I was doing but I wasn't hiking.

Interesting read though.

 

[The Hikers]

Now I know quite a lot more than I understand about how my GPS works.
Thank you for that.
This much I do know: When we are standing in the spruce and it is snowing heavily and my GPS says "Are you indoors?" ,I am in BIG trouble if I haven't kept track with my map and compass.

 

[DougPaul]

This much I do know: When we are standing in the spruce and it is snowing heavily and my GPS says "Are you indoors?" ,I am in BIG trouble if I haven't kept track with my map and compass.

Not necessarily.

From the viewpoint of your GPS, being under tree cover and being indoors are very similar. The GPS sees only the radio signals from the satellites. Trees and buildings both block signals, weaken signals, and reflect them around in complex patterns. Therefore the GPS is asking a reasonable question. (My GPSes just say something to the effect of weak or inadequate signals.)

One can often reacquire the signals by moving to a [more] open area, and holding the GPS stationary (properly oriented) above the local obstacles (eg above one's head) for up to a few minutes.

The recent generation of high-sensitivity GPSes (eg Garmin 60CSx) do far better under trees and in buildings than the older units.

And a hiker generally doesn't have to know exactly where he is all the time, whether he is using GPS or M&C.

Doug