NT: My time in CTS

As the title suggests, I'd like to relate a few stories from my time in Crisis Tech Services. As usual, names of places and people have been changed. All the stories are absolutely true.

The first comes to mind, in the spirit of Vikings and Misspeak, was my first or second time at bat. I'd been with the company no more than 3 months at this point. My English is very fluent but also very street. The switch - technical jargon for telephone central office - was still alive, but feeling none too worthy of keeping up the good fight. The first step, since it was available to me, was to make a backup copy of the data and operating system configuration. The technique is really to dump data to tape, or disk. In my work environment we called it something a little different, which, honestly, I'd never thought twice about.

With a calm and collected voice, I explain the procedure of producing a backup tape and with great alacrity I explain that this action is called "taking a dump".

And so began my career in emergency recovery.

Calm and collected was the defacto tactic by which we would talk to anyone on-site. It showed a measure of concern but at the same time it allowed us to maintain composure in the face of whatever happened at the far end. One such time had three of us working on a call. Usually only one was called out, but my friend received the page while another teammate and me were at his place, so we figured we'd help.

The telephone office which had failed was somewhere in the boonies, as they so often were. It was 10 or 11 pm local time. The technician calling in was very drunk, this in itself was not overly unusual at that time of night, but he was more interested in telling us stories than working on the issue at hand. He asked us all sorts of question from where we were located through to what kind of cars we drove. Our answers were short, his replies were not. Upon my teammates answer that he drove an Acura Integra, our man on site launched into a story of his brother in law driving such an auto, right under a moose crossing the highway. The car sustained little damage as the moose was that tall that the car passed almost complety underneath and between it's legs. Our good training took over, that we remained impassive in our voices, while our sides were splitting.

Some events were more of the ludicrous, although understandable. Working a broken component issue in northern Ontario, the technician on site had to retrieve a spare from the cabinet. Expecting the cabinet to be in the next room, I simply asked how long he would be gone. "About 1/2 hour" he answered, "I have to take the canoe across to the other office". Canoe?!@? As anyone in the computer field can attest, vast amounts of humidity and computer parts just don't mix well, and this replacement was about to take a 15 minute ride in a canoe!

Ghosts and other gremlins were often at work. These were the most interesting, albeit frustrating, problems by far. A telephone office in northern Quebec had recurring grounding problems since it's initial installation, power converters would fail on a regular basis. During the installation period, everything worked perfectly, but shortly after the installers left, the office would reboot for no obvious reason. I wasn't around at the time, but I was assured that all tests revealed nothing. As soon as a technician would show up on-site, all problems were miraculously cured. The grounding rod - a chunk of heavy duty steel or copper driven into the ground behind the building - was actually sitting in a hole. Said hole would dry up and the grounding would become faulty. But when a technician would arrive on site after several hours of travel, he would relieve himself into the hole behind the building, as there are not toilet facilities at these un-manned offices. In doing so, the grounding rod would become wet enough to re-establish good electrical conduit. Simply put: pissing on the rod fixed the problem.

I was very involved with another grounding problem. Or so we thought. Every night at around the same time, this older telephone office, which had no history of failure whatsoever, would suddenly reboot with every single eletrical diagnostic lighting up like a Christmas tree. The scraping of a chair on the floor would generate enough static electricity to send the machine into a frenzy. After a day or two of this, we prohibited anyone from even going into the office, but to no avail. It was significant that the problems would start at roughly the same time every day. What had changed? Was there something new in the environment? Indeed, there was a radio station going up soon and the broadcast technicians were running tests every night. We felt a little wary of this event and wondered how a radio station, operating on completely different electrical principles would damage the telephone central. The news came down from one of the irate telephone technicians that the telephone company had agreed to let the radio company use the microwave tower.

A little information is in order: a microwave dish is line-of-sight and it's tower is usually 10 or 20 feet above the telephone office, it is low wattage and is also pointed in one specific direction. This is for long-distance where wires would be too costly to install. A radio tower is hundreds of feet high, broadcasting in every available direction at extremely high wattage with radiated power reaching up to 200kilo-watts.

So here we had some 50k watt broadcast radiating the office, not 20 feet away! To give a mental picture, it's as if 40 or so microwave ovens with the doors open, were blasting away at the building. An honest mistake. We eventually calculated that more than a 10k watt broadcast was the max a telephone office would take. Eventually a proper radio tower was installed and all was well. In the meantime, the radio station was enjoined to conduct their tests at less than 5k watts.

Speaking of cooking things: an office that would experience a revolution at 4pm in the afternoon, but only on sunny days and then only after a tech had been on site for a few days. I cannot remember how long it took us to solve this one. The sun would come streaming into the building through a curtained window, but a technician had gone to site and pulled the curtains to see better, then left without thinking of the curtain since it was night time. This eventually allowed the 4pm sun to cook the machine enough to overheat one of the computer controllers.

These are very robust machines, military grade in fact, but direct sunlight just isn't good. Nor is ignorance. One of my teammates worked on a very small switch in the Great White North (tm), it had maybe 20 or 30 phone lines in all, but it kept getting overloaded. This is completely unheard of. This damned thing was designed to easily handle 10,000 lines! Ignorance being bliss, this new fangled telephone thingy was a mystery to the local populace who would call each other up and leave the phone off the hook. When Joe wanted to talk to Fred, he'd lean out his window and shout: Fred pick up your phone! The overload? If Sam wanted to talk to Joe, he was shit out of luck, since line was always busy!

On subject matter expertise, or lack thereof, and I say this with the utmost respect, I once had to talk the building janitor into emergency recovery of a switch. He was the only one available to work with me since all the technicians were 4 hours away in the next closest town. This man knew nothing of telephone office operations, and I do mean nothing. Nevertheless, he tried his best, was resolute and steadfast. Arguably the most dedicated person I ever worked with. It took us about 20 minutes to find the modem - flat little black or gray box - by which I'd be able to log into the phone office and begin troubleshooting. By god, he found the modem, reset it and then rebooted the telephone office. This was a very relieved man when I told him the phones were working again.

Sometimes the phones are down for so long that some people start to notice. At about 8pm one evening, I get paged by the Chief of a rather large telephone office nearby. I immediately recognized it as being the second biggest office of this type in North America. This means a lot of people are without phone service. It was going to take us several hours to recover this bastard no matter how creative I got. About an hour into the call, I hear some heated discussion in the background. Apparently, the police was outside, along with a mess of reporters. This was a little surprising to me since the location of telephone offices are not usually public knowledge. The chief of the C.O. - central office - was livid, as if his telephone switch having crashed wasn't enough, I could hear him yelling in the distance: "I swear if anybody opens that fucking door, I'm going to fucking kill him." O...kayyy. Keep composure.

C.O. people get upset. It happens. It's also somewhat related to the clients being served. I got my ass chewed out one day for not having the correct documentation on hand. Remember, this was in 1990-91, some 15 years ago now. The web was still text based and on-line documentation was barely an itch in somebody's shorts. The term Web had not yet been coined in the general populace. The word Nuclear on the other hand was very well known. A brand new feature card was available, so new in fact that I did not have the documentation covering it. This new card allowed for some PBX - private branch exchange - also known as business lines, to be installed on a regular phone line.

This was a single card failure, and was not normally handled by my Team, but as would happen every Friday, the Field Service Engineers in Toronto would sluff the call off to CTS when 4pm rolled around. This occurred often enough that the person on-call on a Friday would systematically not make any kind of plans for 4pm. This annoyed the shit out of us, but it was difficult to prove and Toronto being the larger F.S.E. organization in Canada, they certainly had the power to make life more miserable for us. There is no love lost between me and a couple of cretins in Toronto at that time. The situation thankfully got better over the years as those bad seeds left.

Back to the single card failure, I began by asking the basic questions for my troubleshooting and realized this was single line, single customer and clearly not of my purview. Not to be deterred, I asked who the customer was, that this was so important to solve. The answer: the local nuclear power plant. Re-establishing communications with a nuclear plant seemed important enough for me to keep working the issue.

Earlier, I mentioned microwaves: as anyone with a TV-LOOK system knows, the dish has to be pointed in a particular direction and be clear of obstacles between your dish and the broadcasting dish across town. In telephony, microwaves are often used to cover great distances where overhead or underground wires would be prohibitively expensive. If the distance is great enough, the curvature of the earth starts being a problem and taller towers are needed. One such site in the great plains of Labrador had tall towers, but not tall enough. Twice a year for about a week, all long-distance to or from this telephone central would simply not exist. Then the problem would sort itself out. The line of sight between the two microwave dishes came within a few feet of the ground, and every year, twice a year, for about a week, moose migration would cut the beam.

And then people look at me funny when I tell them I actually closed a customer service request with the cause being sun-spots.



Glossary
C.O. : Central Office, esp. building(s) containing telephony equipment.
C.T.S. : Crisis Technical Service, also a mess of other names over the years.
F.S.E : Field Service Engineers
Office : ref C.O.
PBX : private branch exchange, private multi-line business installations hosted by the local phone company, but with private business-specific phone numbers with operator facilities, etc.
Skivvies : Quebec F.S.E. name for an installation tool, that had the guys in the U.S. just about pissing themselves laughing.
Switch : the actual computer which processes the telephone calls, in the C.O.
Telephone central : ref. C.O.
Telephony: the science of all things related to telephones.

Note of Author: C.O., switch, office, central were terms often used interchangeably unless technical correctness was required.