atcomsystems.ca/forum Forums TDM Business Telephone Systems Samsung Telephones & Systems Neat trouble 7100

I've got a neat trouble that i'm happy to share...
7100, 1 year old (scarily - almost to the day), busy doctors office. Config:
MP10 - 4DLM
Univ - 4DLM, 4SLM, 4TRM

System was installed with 4.04a which ran happily for a year. I go upgrade to 4.22... Runs a week.
Problem:
around 3:30pm, all sets crash (5 - 28 button falcon), showing boot "REV 2.3" or whatever they show when first powered up. Calls drop when it goes.
Interesting part: Switch hasnt fully crashed. I can still log in using installation tool - confirmed by Samsung Tech Support. System is up according to IT, but no sets or trunks work. Incoming calls RNA.

First trouble call - reboot, system recovers.
Second call - default system, hand program only using set, system recovers.
Third call - replaced MP10 (which happens to be on 4.14k), so far its still running.

I have the customers MP10 running in our test system here in the office - without a hitch. Running on customer database, same config. (my daughterboards and universal card/cabinet)

STA next to replace is cabinet (not daughter/univ cards) if the system crashes on my spare MP10.

No wierd environment, 70 (20c) temp controlled, system is on batteries, location is mini-mall. Anyone else experience this on any system?

Did you replace the 4.04a with 4.22 and do a complete memory erasure and start over or did you an upgrade procedure?

We have a 7100 that started with 4.04a, lost two mailboxes and button programming on only the phones attached to those mailboxes and after swapping out MP10 and memory we are now being required to put in new memory with an 811 to see if that will help.

Yes the 7100s do have ghosts.

did the upgrade procedure - remove software card, erase everything except '7100' folder, copy 4.22 to software card, upload database using IT (which defaults ksu - but not vmail).
We did default the ksu on the 2nd trouble call and hand program the whole database. Defaulted by changing country then changed back to USA.

If you are still having problems, I would do what we have to do next week. Default the system, power down, insert a new clean 4.22 card, power up and reprogram everything including voicemail.

The 7100 has bugs, that is a definitive answer. All these weird things cannot be defective technicians.

Question for Vernon. Is this 7100 connected to the LAN? When the system is in version, if you unplug the LAN connection, does it come back up?

System is on a LAN, and is accessable remotely over IP. I also thought of that T&R - it does not recover when the lan is unplugged... (i've had that happen with 4.04a, but not quite the same, the system didnt recover, but did stop crashing when connected to linksys router)
So the system has been running now almost a week on 4.14k on my spare MP10 card. Ultimately now the last thing to swap would be software cards - to put in the customers software card on my spare MP10 and see if its software related or the customers MP10. (and by software related, could be the database or the card itself) - I'm not sure I want to go there with the customer, we'll see if they are willing.

Marty, all you’re troubleshooting steps are right except for the voice mail. Take a look for the VM logs and compare with normal working OS 7100 VM. We need to default the VM and everything should by ok until next undocumented OS 7100 bug. Hope is helpful.

Second customer is now experiencing this problem...Different city, same card/module config. Again, was 4.04a upgraded following samsung proceedure to 4.22. This system ran 2 months on 4.22 before today - all sets die but still get power and only show REV: startup display. Reboot cures it - oddly, it happens at roughly the same TIME as the other system - around 3:30pm local. Today was the first for this customer, I'll go to site tomorrow at 3pm to watch and perhaps do a complete default after checking the vmail logs as mathrap indicated.
Update tomorrow if it crashes again...

*sigh* second verse, same as the first.
Customer called again, this time about an hour later - same thing. All sets crashed. I log in and reboot - going there first thing tomorrow AM. Doesnt seem to be a traffic issue, first site had HEAVY traffic (doctors office). This site has much lighter traffic... I can't log in to see vmail logs remotely as the ISP blocks 443...

The system crashed only twice last nite before the owner went home, he call forwarded his lines to his cell so the impact was less. the 7100 hasnt crashed since, i'm on site and there are no logs after Oct 14 (copied here:
NOTICE - Tue Oct 14 3:59:31 2008

NOTICE - Tue Oct 14 3:59:32 2008
System maintainance complete

NOTICE - Thu Dec 4 16:36:53 2008
Block table /os7100/vm/dta/BLOCK.TBL successfully loaded

NOTICE - Thu Dec 4 16:36:53 2008
Total voice ports available: 4

NOTICE - Thu Dec 4 16:36:57 2008
Clock set

NOTICE - Fri Dec 5 2:57:42 2008
Daily system maintainance

NOTICE - Fri Dec 5 2:57:53 2008
System maintainance complete

NOTICE - Fri Dec 5 2:59:59 2008
Daily system maintainance

NOTICE - Fri Dec 5 3:00:03 2008
System maintainance complete

The Dec 4th block table successfully loaded is when i rebooted the system the second time.

The KSU logs (below) are interesting, in that it shows BOTH reboot's by me - the first at 15:41 (which doesnt show up in the vmail logs) and the second at 16:37 which is also in the vmail logs.

MJA01 08/20/2008 00:00:21 Power On Restart Pwr reset
MNF35 08/20/2008 00:00:21 SVMi Request End C1-MP
MNF38 12/04/2008 15:41:29 Card Active C1-S01
MJA05 12/04/2008 15:41:29 LCP Restart CABINET:1
MJE10 12/04/2008 15:41:29 SVMi Restart C1-MP
MNF33 12/04/2008 15:41:29 SVMi Request C1-MP
MJE01 12/04/2008 15:41:29 MGI Restart C1-MP
MJA03 12/04/2008 15:41:29 MMC Restart Reset only
MNF35 12/04/2008 15:41:29 SVMi Request End C1-MP
MNF38 12/04/2008 16:37:13 Card Active C1-S01
MJA05 12/04/2008 16:37:13 LCP Restart CABINET:1
MJE10 12/04/2008 16:37:13 SVMi Restart C1-MP
MNF33 12/04/2008 16:37:13 SVMi Request C1-MP
MJE01 12/04/2008 16:37:13 MGI Restart C1-MP
MJA03 12/04/2008 16:37:13 MMC Restart Reset only
MNF35 12/04/2008 16:37:13 SVMi Request End C1-MP

At this point im torn between calling this a software bug and a hardware issue. With the Doctors office (the first customer with exactly the same symptoms and ksu card config as this), I've replaced the MP10 card with our in-house spare (which is on 4.14K), and that site has been running fine since Nov 20th. So is it S/W or Hardware? Both systems have been installed about a year, in different towns.

BOTH the Doctors office and this site (small car dealership) were running flawlessly on 4.04a, but then a month after 4.22 upgrade - start this fun.

I dont have another MP10 i can just swap in here, so instead, i've brought a copy of this customers 4.04a software card to further confirm/deny a software issue. Both systems are in controlled environmental telephone closets, stable temperatures, on surge supressors with battery backup, and grounded properly to supply panel.