Hi guys.
I’m testing the operation of the T3-LB controller with my big program on the latest firmware mini_arm_rev50.1_12.10.hex.
I observe very often controller reboots when I change the values of any variables or work in the GUI.
@maurice , I sent you my program, but I did not find out the reason for the controller reboot from you, I just received recommendations to reduce the program and reduce the number of points in the GUI.
I decided to test my program on the controller with earlier firmware and as a result, on the mini_arm_rev49.9_11.6.hex firmware, the controller reboots stopped.
But on this firmware and earlier there is a problem of periodically shutting down user programs for reasons unknown to me (switching from ON to OFF). Programs are rarely turned off, and not the same ones, sometimes for example, program number 3 will turn off, and sometimes number 5.
After turning on the programs (switching from OFF to ON), everything normalizes and can work for several days, then the situation may repeat again.
I suspect that @chelsea made some kind of algorithm in the firmware, which began to reboot the entire controller, and not turn off the programs: https://forums.temcocontrols.com/t/t3-lb-controller-writing-error/580/9. But this led to a big problem, the controller became almost uncontrollable.
I suggest that you recall what changes were released in the firmware and change it so that the programs do not turn off and the controller does not reboot.
This is very important, otherwise we will not be able to use your controllers in our commercial projects due to their unpredictability.
I hope you hear me as soon as possible.
Thanks for your detailed work on getting to the bottom of this Aleksei, we will support you well but need to ask for a few days till the local holiday is over, Feb2.
A few days ago we started having trouble communicating to one of our T3’s. The usual remedy is to go up to the Hot Water penthouse on the roof and reboot it. After doing that 2 days in a row, I decided to try a different approach. I simply unplugged the Ethernet connector from the unit, waited 30 seconds and reconnected it. The network connection re-established itself and the dreaded red X went away.
At the same time, I was surprised to see activity on the Main RS485 red & green lights. I checked the setup screen and it was NOT set to “Unused”. So I set it properly.
It’s been several days and we’ve no more issues. Maybe one of these actions helped, maybe not. But I thought it might be helpful for you to know.
Hi @tdawgtoo , it’s a really strange thing, but I think it has nothing to do with my question about spontaneous shutdown of programs in the T3-LB.
I look forward to February 3, when I get an answer from @maurice and @chelsea to my question and the controller finally works well.
Hi,
For some reason, no one answers, strange.
I had the idea to write watchdog in one of the programs, which would control the shutdown of the programs and turn them on again.
I managed to get a message about disabling a specific program PRG1, but when the PLC try to restart it from the watchdog program, nothing changes. Maybe I’m doing it wrong? Here is a sample wathdog code:
10 DALARM NOT PRG1 , 5 , PRG1 OFF
20 IF NOT PRG1 THEN START PRG1
I changed the mode of the controlled PRG1 program in both Auto and Manual, but without result.
Thanks for your input Aleksei. I could be wrong but going from memory, there’s no commands to turn a program on and off from within a program. If you have seen something in the documentation that says otherwise I will check into this more.
Tdawg: When you reset the T3 controller, is the heartbeat LED blinking normally about once per second? If so we need to look into this more, send on your program and I’ll see if we can repeat it. If the LED is NOT blinking normally then the controller is locked up and its supposed to reset itself, there are no know issues like that so let me know and we’ll work on this with you in depth. This thread is wandering around too many topics, please start a new thread if there’s something not directly to the main topic.
Hi Maurice, I did not find a specific example in the manual for turning one program ON or OFF from another program, but I think this is possible. In any case, we urgently need to do this or fix the problem with the sudden shutdown of programs in the firmware. It’s not the first month that I’ve been debugging your controller and we just have to take the last step.
Let’s do it, okay?
I can not swear that the heartbeat light was blinking normally, but I believe it was. The problem was related to communication, not T3 performance. Tasks assigned to the T3 that did not require communication with the other T3’s continued to be performed as programmed.
At the moment, my four T3-LB’s are performing and communicating like little angels with zero comm problems. Every network point has the correct info. My theory is that the problems I have are not directly due to the T3’s. But it’s their susceptibility to external factors that makes them misbehave. I will cease posting on this thread and post the results of testing for those factors if I find something of note.
The root cause of the above problem is that the program you write has a dead loop or some kind of exception.
mini_arm_reboot_close_program.hex (624.3 KB)
(For T3BB T3LB T3TB T3NB)
If others experience the same problem, use the attached firmware.
This version of firmware will shut down all the programs(1-16) when T3 restarted or powered on .
I will reproduce this fault as soon as possible and fix it.
Hi guys,
@maurice, @Fan_Du, tell me please, are there successes with the problem of rebooting the controller? Did you find the reason?
Also, last week I wrote to you email with my request to enable the software Wachdog in firmware 49.9 - run some programs from other programs if these programs suddenly shut down. Unfortunately, I have not received your answer.
All this has been going on for a very long time. How much more time will you need?
Reboot: Fandu worked all last week trying to repeat your issues but no luck. We will keep at it though. Please continue to contact him through Skype/Email/Teamviewer as you have been doing. He is off for a couple days though.
Watchdog: As I mentioned, we don’t have statements to turn programs on and off from within a program. You could use a variable and only execute certain logic based on that variable.
Hi Alexksei:
Your programs will cause the T3 to restart in some cases.Maybe it’s a memory leak. I retried your steps many times, we did see the issue but it can take hours to see phenomenon. Sometimes the problem is not repeated for days. We plan to spend more time on it but it will take time.
Your suggestion about letting one program restart another program is not implemented but we can consider it for an upcoming release. In the mean time as Maurice suggested you can use a variable to flag whether certain sections of code run or not.
For now your best option to get up & running is to simplify and break your application into smaller programs. You can also run some code in a separate controller, the new T3-Nano is only $39 for example and can run the same programs that the bigger T3-XX series can run.
Thanks for your work on this, we will be reporting back soon!
tagging @Aleksei,
Hi Maurice,
I asked you in the past if it’s possible to add at least a network watchdog.
Sometimes I loose connection between my T3 and my network (even the T3 is fully functionnal and the prgm is running smoothly) and I have to reboot it to restart everything. And I know that’s not related to my network. OK, maybe (I don’t want to give the T3 the full responsability).
But a network watchdog could be useful: if the T3-XX is no more into the network, just reboot it. Maybe add a counter or a timeline (reboot after X unsuccesful pings) to fine tune the feature and a custom IP address (could be internal like the router address or external like a Google ping). And memorize into a variable the number of reboots (to have a count of them).
With this “simple” watchdog, some unknown network/T3-XX errors could be solved quickly and automatically.
Is it feasible ?
Thank you
Mike
If I understand correctly, you would like a feature where if the dead master timer triggers (the unit is offline) it will reset itself a certain number of times. I can add this to our todo list though I would prefer to get at the root problem rather than mask it like this. The devices are supposed to, and generally DO run forever without resettting in normal use.
If you haven’t updated the firmware in a while please do that, we’re making continual tweaks and improvements to the firmware based on feedback such as yours so keep it coming.
Correct !
maybe the T3-XX are not the culprits, but this can be useful. Btw, I updated all my T3 with the latest firmware.
Thank you
Mike,
You are not alone. I have been battling the problem for a year now. Recently, with firmware 50.4 and higher, the T3’s tend to either stay connected via Ethernet or reboot themselves. It’s not perfect, but it’s better than having a T3 that never gets needed data from some other T3’s bacnet point. I am currently on firmware 50.4 and my four T3’s are pretty stable. Frankly, I am now at the point where I am a little reluctant to update because they are working pretty well. On average they reboot about every other day. I see that they’re up to 51.0 so I will probably bite the bullet and upgrade them in the hopes that the rebooting will calm down too.
Through all this, Maurice has been a saint and and patiently helped me navigate these problems.
Ted
Thanks for the continued feedback and efforts with our equipment fellows. We haven’t had a look at your program in a while Tdawg, why don’t you send your program here again and we’ll run it on our test setup to see what we can see. There’s a new feature you will appreciate, a timer so you can see how long its been since the last reboot.
We had a hard time copying this communication bug in the office .
In one case we did see a coupe of events where the ping response time will slow down and eventually the unit stops communicating. A reset of the controller was required. We’ll continue with the debugging and report back soon.
[Updated April 17] We are sending out a new release of firmware for all devices which fixes this issue. Turns out the Ethernet chip we’re using can get jammed up in certain busy environments. We added some code to detect the condition and re-initialize the chip when it happens. No reboot required.
thank you Fance. That’s exactly what I saw. slower communication and then stop. Hope that you’ll find the problem / find a solution / develop a network watchdog.
Mike
Update: Since I updated for the latest firmware, it seems that my T3-TB is more network-stable. I’ve no disconnection since 3 days (on the very same network environnement). Hope it will last connected “forever”.