I am new to this board and new to a position managing a small team of Network Engineers.

We’ve had an issue turning up a pair of T1s between our local and remote site. We’re having difficulty determining where and what the problem is.

The routers (Cisco 2621XM with internal CSU WIC) have their error counters (CRC, frame, aborts, interface resets) increment periodically. One interface will increment by thousands on a predictable basis (Tuesday and Friday mornings at 0000 - 0030).

The local router, WIC and cable to the patch panel have all been replaced, so we’re pretty confident that it’s not a customer (us) hardware issue. We’ve had the router configurations reviewed and they’re OK.

Our circuits are delivered to us in a patch panel in our network room. The smartjack is in our provider’s area and we cannot physically access or test the device.

We scheduled testing with our provider this morning (Friday – when a predictable error would occur). We had a hard loopback cable plugged into our local patch panel port. While “looped,” the errors on our remote router slowly incremented steadily (a few per minute). During the provider’s testing (I don’t know what tests), the error counters incremented over 1 million then slowed to the steady incrementing when the testing stopped. When we took the loop out this morning, the errors went back to periodic incrementing.

When we looped (line) at the remote router, our local interface began to increment steadily (a few per minute). When we unconfigured the remote loop, the errors went back to periodic incrementing.

My initial question is, would looping a circuit cause the errors? I would assume that looping a clean line would not inject errors.

Our provider says the line is clean.

Thanks in advance!

No looping will not cause errors. However if you had a hard loop not going through a CSU and a long circuit then you could get errors on testing. Testing to a looped CSU or NIC regenerates the signal, which is the way it should be tested if not doing head to head testing.

We had a similar post in Networking. Depending on which WIC card you have you may have end to end testing capabilities. I'm not familiar with this product only the information I've taken from Cisco's WEB site. You will be better off just testing the T-1 instead of going through all your other stuff to try to isolate where the problem is. You should have access to the 8pin jack on the back of the smart jack as that is your interface.

The errors you were seeing on your equipment while the phone company was testing could be caused by a lot of things, it wouldn't be a valid result in my opinion.

This should be your key! Get access to the telco areas. There are older facilities still out there that receive power locally instead of being line powered. Eliminate this as a possibility.

WearyTraveler -

Loop the CSU at the router. Check for errors. If you do not get any errors loop the CSU @ the smartjack, this will test your extended DMARC (layer 1). If you are getting errors at step 1 it is your WIC. If you are getting errors at step 2 it is probably your cable. A telcom T1 only uses 1-2, 4-5 which if you are using a 568B patch cable is usually the ORG and BLU pairs. If you are lazy like me cut the end off and use the GRN and BRN pairs (I will sometimes do this to run 2 T1s).

Run these tests and let us know.

Cisco's got a document that reasonably explains the various errors one might see on a T1/E1 interface. Not as practical as the advice given here, but it's a good reference to have none-the-less.

https://www.cisco.com/en/US/tech/tk713/tk628/technologies_tech_note09186a00800a758d.shtml

You guys are very helpful! Thanks - I will check these out.

This is what confuses me about the Cisco site. These are all DS0 terms, not T-1. So are they talking about the channels or the T-1? Why would they not use the same verbiage that's been used for decades? T-1 is T-1 and DS0 is DS0 and the two are no where near the same.

DTE = data terminal equipment

CD = Carrier Detect

CSU = channel service unit

DSU = digital service unit

DCE = data circuit-terminating equipment

CTS = clear-to-send

DSR = data-set ready

Bill -

This is exactly what I found in the Cisco VOIP classes.

Cisco is attempting to rewrite communications "in their own image". Why use terminology that's been around and tried and true for 40 years and more? Then Telephone people will be required (or available) to work on your equipment.

If you use "Cisco Speak" to describe something then only Cisco people can understand you or work on your equipment.

Not that I'm paranoid or a conspiracy theorist, but.....


Sam

Thanks for that Sam, I thought it was just me. As a pure phone guy it makes it hard to help the good folks that come around with these questions when a company isn't talking Telco language.

It makes for more robust communications via enterprise platforms while conversing by analog transmission that is digitized after being inputed to the handset.

Sorry to chime in so late, but I do have to ask something of you Charlie. You mentioned that there is a patch panel for your circuit. Is it a standard CAT5/5e patch panel or is it one that is wired for RJ48 circuits? The reason that I ask is that I suspect it is just a standard one. That being the case, chances are that the wiring between it and the serving end isn't proper either. If both the transmit and receive pairs are not effectively isolated as much as possible (under separate cable jackets or separate binders in a multi-pair cable), you will likely have problems.

Are you in a co-location facility? I wonder if this is a case of multi-pair cable feeding the patch panel where appropriate isolation isn't maintained. As far as time-of-day occurrences, could it possibly have something to do with energy management in the building or worse-yet, proximity to other circuits that are bustling at these times causing interference? I'm just grasping at straws at this point.

Ed,
I'm not sure about the patch panel. I'll check tomorrow. You're right, your assumption may be correct in that the pp is RJ45.
I'll be on site tomorrow.
Thanks again for all the information!
Charlie

Guys you only see those types of serial designations if you are looking at a T1 with a V.35 cable. The document someone posted is for troubleshooting using a WIC-1T or a WIC-2T which are V.35 WICs. You will NEVER see this if you are looking at a T1 with an integrated CSU/DSU (RJ-48). Not to ruffle anyones feathers but this is a IEEE standard no one made anything up. I even went out and found a non-Cisco site if you care to verify this. https://telecom.tbi.net/v35.htm. Here is a guide for a WIC-1DSU-T1-V2.
https://www.cisco.com/en/US/products/hw/routers/ps214/products_data_sheet09186a0080091ff6.html
Notice in the middle of the page all of the diagnostics you can run on it.

That may well be, but I don't care how you shake it that is not T1 terminology. You tell a T1 tech your having a problem with your T1 because you have no DCD and they'll just give you a glazed look and walk away. Tell a T1 tech you can't ping and you might get pinged. These things used to drive me nuts when I worked in a test center. I have worked with people who have told me their T1 was down and I could see a good signal bothways so it might be looped right? You split the circuit to see if one end is looped and you're asked just what the heck did you do all are terminals just went down. Uh, you said your T1 was down. Well it is, my ABC can't ping my XYZ and I've checked all my stuff so it has to be the T1. :read:

To the point at hand. Trans and Rec in the same sheath can cause problems. I've never seen inter office ties cause it though.

I have not seen a new install with an external CSU/DSU in at least 5 years. If I gave you the indication that a V.35 cable is part of a T1 circuit it was my mistake, I do not believe this. I was merely attempting to clarify why serial terminology would be used in a circuit. As I am sure you know on a external CSU/DSU the T1 terminates on the CSU/DSU (RJ48 or DB-15). The CSU/DSU "converts" the signal to a standard V.35 cable and passes to either a molex DB60 or a 26 PIN "smart" serial interface on the router. The serial from the router can be V.35, X.21, RS-232, RS-449, or RS-530. So the same interface that I can interface to a standard modem can interface to the CSU/DSU and run a T1. This is where the serial terminology comes in. If you have the right cable you just plug the DB25 into the modem and plug the 60 pin into the router go to the port and execute the command "physical-layer async", configure the line and do a reverse telnet session. At that point you can issue AT commands to the modem and also receive calls into the router to do backup or to backdoor into it for administration.

I kind'a went off. Just a pet peeve of mine. If you have the tools isolate the trouble and report the right thing. The point I'm trying to make is if you report a T-1 trouble use the right terminology when you report it, it will save you a lot of grief in the long run.

There are still external CSU's, which of course you don't need if you have a built in CSU in your equipment.

I will give up now.

Guys, I'm back. We've been working this issue for months now. When I got all the interested parties on several conference calls, the techs from the providers saw errors and eventually "redesigned" the circuits. Two days after the redesigned circuits came online, we received a boatload of errors on my routers' interfaces. After more conference calls and numerous "exercising" of jacks, the errors subsided. The lines went clean for 13 days, then received several hundred errors. We went through more conference calls and providers on line. The circuits went clean for about 13 days again. Over this weekend, I had:
THERE ARE ERRORS ON THIS CIRCUIT
3 INPUT errors out of 381171 packets ( 1 out of 127057)
1 CRC errors out of 381171 packets ( 1 out of 381171 )
2 ABORT errors out of 381171 packets ( 1 out of 190586)
on one and:
THERE ARE ERRORS ON THIS CIRCUIT
31 INPUT errors out of 8211069 packets ( 1 out of 264874)
29 FRAME errors out of 8211069 packets ( 1 out of 283141 )
2 ABORT errors out of 8211069 packets ( 1 out of 4105535)

Now I'm hearing (from my provider) that:
"I am not certain that amount of errors is statistically significant. If we weren't putting these two T1's under a microscope, I am not sure these would have been noticed and generally speaking T1's get a few errors over time."

This is my first exposure to T1 circuits. However, in a previous position I had 10 Gig circuits passing more bits across the Atlantic and Pacific getting ZERO errors. I want to respond that the statement from my provider is wrong, but not dealing with T1s before, I don't want to miss-speak.

From a data standpoint, do T1's get errors on a periodic basis? Or should these lines be clean? I'm leaning towards the latter, but again...

Thanks for your help in the past and hopefully on this question.
Charlie

No, a good T-1 won't have errors. Back to the original question way back when. Are you testing JUST the T-1? T-1's don't put out packets. The only things I recognize as T-1 terminology is the CRC and the Frame errors, but again the way your using it is not in T-1 lingo. If the T1 and I stress the T1 has CRC errors that is usually a connection (copper, jack etc) problem.

Have you ever taken a T-Berd or similar piece of test equipment and tested this end to end?

Like Bill said, a good T-1 should be error free, but if you happen to be in an area with a lot of new construction, there are a lot of hands in the telco/provider's network. As your provider said, if the ckts were not under a microscope, you probably wouldn't have noticed. All it would take to cause those errors, if they are coming from the t1, is for someone to cross you with another ckt for just a second while wiring up another customer at a dsx location (not much room between pins) or they could be from a mux in the telco/provider's network, switching paths for some reason.

Are both t-1s taking errors or just one of them? Also, you might want to double check the settings for timing on your csu at each end. It's possible that your csu is reporting what is known in t-1 language as frame slips, as frame errors, which are caused by the timing getting off.

Hope that helps some.