Thanks for your input Aleksei. I could be wrong but going from memory, there’s no commands to turn a program on and off from within a program. If you have seen something in the documentation that says otherwise I will check into this more.
Tdawg: When you reset the T3 controller, is the heartbeat LED blinking normally about once per second? If so we need to look into this more, send on your program and I’ll see if we can repeat it. If the LED is NOT blinking normally then the controller is locked up and its supposed to reset itself, there are no know issues like that so let me know and we’ll work on this with you in depth. This thread is wandering around too many topics, please start a new thread if there’s something not directly to the main topic.
Hi Maurice, I did not find a specific example in the manual for turning one program ON or OFF from another program, but I think this is possible. In any case, we urgently need to do this or fix the problem with the sudden shutdown of programs in the firmware. It’s not the first month that I’ve been debugging your controller and we just have to take the last step.
Let’s do it, okay?
I can not swear that the heartbeat light was blinking normally, but I believe it was. The problem was related to communication, not T3 performance. Tasks assigned to the T3 that did not require communication with the other T3’s continued to be performed as programmed.
At the moment, my four T3-LB’s are performing and communicating like little angels with zero comm problems. Every network point has the correct info. My theory is that the problems I have are not directly due to the T3’s. But it’s their susceptibility to external factors that makes them misbehave. I will cease posting on this thread and post the results of testing for those factors if I find something of note.
If others experience the same problem, use the attached firmware.
This version of firmware will shut down all the programs(1-16) when T3 restarted or powered on .
I will reproduce this fault as soon as possible and fix it.
Hi guys, @maurice, @Fan_Du, tell me please, are there successes with the problem of rebooting the controller? Did you find the reason?
Also, last week I wrote to you email with my request to enable the software Wachdog in firmware 49.9 - run some programs from other programs if these programs suddenly shut down. Unfortunately, I have not received your answer.
All this has been going on for a very long time. How much more time will you need?
Reboot: Fandu worked all last week trying to repeat your issues but no luck. We will keep at it though. Please continue to contact him through Skype/Email/Teamviewer as you have been doing. He is off for a couple days though.
Watchdog: As I mentioned, we don’t have statements to turn programs on and off from within a program. You could use a variable and only execute certain logic based on that variable.
Hi Alexksei:
Your programs will cause the T3 to restart in some cases.Maybe it’s a memory leak. I retried your steps many times, we did see the issue but it can take hours to see phenomenon. Sometimes the problem is not repeated for days. We plan to spend more time on it but it will take time.
Your suggestion about letting one program restart another program is not implemented but we can consider it for an upcoming release. In the mean time as Maurice suggested you can use a variable to flag whether certain sections of code run or not.
For now your best option to get up & running is to simplify and break your application into smaller programs. You can also run some code in a separate controller, the new T3-Nano is only $39 for example and can run the same programs that the bigger T3-XX series can run.
Thanks for your work on this, we will be reporting back soon!
I asked you in the past if it’s possible to add at least a network watchdog.
Sometimes I loose connection between my T3 and my network (even the T3 is fully functionnal and the prgm is running smoothly) and I have to reboot it to restart everything. And I know that’s not related to my network. OK, maybe (I don’t want to give the T3 the full responsability).
But a network watchdog could be useful: if the T3-XX is no more into the network, just reboot it. Maybe add a counter or a timeline (reboot after X unsuccesful pings) to fine tune the feature and a custom IP address (could be internal like the router address or external like a Google ping). And memorize into a variable the number of reboots (to have a count of them).
With this “simple” watchdog, some unknown network/T3-XX errors could be solved quickly and automatically.
If I understand correctly, you would like a feature where if the dead master timer triggers (the unit is offline) it will reset itself a certain number of times. I can add this to our todo list though I would prefer to get at the root problem rather than mask it like this. The devices are supposed to, and generally DO run forever without resettting in normal use.
If you haven’t updated the firmware in a while please do that, we’re making continual tweaks and improvements to the firmware based on feedback such as yours so keep it coming.
Mike,
You are not alone. I have been battling the problem for a year now. Recently, with firmware 50.4 and higher, the T3’s tend to either stay connected via Ethernet or reboot themselves. It’s not perfect, but it’s better than having a T3 that never gets needed data from some other T3’s bacnet point. I am currently on firmware 50.4 and my four T3’s are pretty stable. Frankly, I am now at the point where I am a little reluctant to update because they are working pretty well. On average they reboot about every other day. I see that they’re up to 51.0 so I will probably bite the bullet and upgrade them in the hopes that the rebooting will calm down too.
Through all this, Maurice has been a saint and and patiently helped me navigate these problems.
Thanks for the continued feedback and efforts with our equipment fellows. We haven’t had a look at your program in a while Tdawg, why don’t you send your program here again and we’ll run it on our test setup to see what we can see. There’s a new feature you will appreciate, a timer so you can see how long its been since the last reboot.
We had a hard time copying this communication bug in the office .
In one case we did see a coupe of events where the ping response time will slow down and eventually the unit stops communicating. A reset of the controller was required. We’ll continue with the debugging and report back soon.
[Updated April 17] We are sending out a new release of firmware for all devices which fixes this issue. Turns out the Ethernet chip we’re using can get jammed up in certain busy environments. We added some code to detect the condition and re-initialize the chip when it happens. No reboot required.
thank you Fance. That’s exactly what I saw. slower communication and then stop. Hope that you’ll find the problem / find a solution / develop a network watchdog.
Mike
Update: Since I updated for the latest firmware, it seems that my T3-TB is more network-stable. I’ve no disconnection since 3 days (on the very same network environnement). Hope it will last connected “forever”.
Update 2: I discovered something different. My problem is now a self reboot like other users have.
I noticed that because the T3-TB is hooked to 2 pumps, one of them is a (relatively) big emergency pump and I hear it when it runs. For now, I hear randomly a noise (“the pump” is running) for less than a second. Typical of a reboot when all registers are checked (and relays opened for a quick). I also noticed (related ?) that a specific program line is not working as expected: when time is 00:00, a variable is set to 0.
Because I’m a night owl, I see now that variable is not reset at the right time, maybe later (I have to sleep a little bit).
Last remark to validate the self-reboot problem: I noticed that the T3-TB running time is now about 2 hours and I know it’s impossible (9PM here and I didn’t volontary rebooted the T3-TB since days).
I don’t know if all those problems are related to the last firmware but things are going worse
Hello Mike, could I ask you to send the program file, save the whole *.prog file and send that to us by email. We can try & duplicate the issue here. Save the prog file by connecting and going to T3000 → file → save as, give it a name and navigate to your work folder. Its a good idea to save often as you develop your application.
Hi Chelsea,
I sent you the whole prg of my T3-TB.
I also made a test today by collecting data from my T3-TB, every second, with another plc.
Look at my graph here: obviously some parameter (variable 1 in my case) is set to zero randomly and the running time of the unit is reset to zero.
Randomly ? At midnight (OK), 3:40 AM, 6:05 AM, 8:48 AM, 7:38 PM
The only time the reset of that variable should be 00:00:00
Let me know what you find and if you can reproduce the problem.
I can try a little bit more this week–end: remove totally the unit from the main and feed it with a 12V battery bank. I didn’t have power outages, but I want to be sure that micro-outages can’t be the possibility.
Update: firmware updated to 51.3 (was running 51.0) after those tests and beginning a new set at 22:22 PM After the update, the unit rebooted and launched the pump2 for a fraction of a second (and was recorded by my plc as a single pump2 run)
tagging @Fan_Du and @maurice,
I made tests this week-end. Before going with my battery bank, I just downgraded the firmware from 51.0 (and 51.3) to 49.0 and left all the parameters (I even did not touch the wiring and my network) as before.
The result: no more reboot (fingers crossed) since 17 hours. Previously, I had multiple reboots along the day. I’ll let my T3-TB running with this config for more days and will let you know the story.
Look yourself below
Mike
So, I understand that the T3-XX reboot itself (latest firmwares) when there is a network problem (ping). But we can’t manage it (like: IF TIME = 00:00:00 AND PING=0 THEN reboot). And the variables are reset to 0, whatever their previous values .
Is it possible to avoid this ? Because loosing variables values is also difficult (running time, states, etc) and not predictable with spontaneous reboot. Variables that hold values even after a reboot ?
After 1 day and 3 hours with Firmware 49.0 (no spontaneous reboot):
We loaded your program into a controller here but have not been able to duplicate the issue. We’ll keep the test running over the next while and update you in the forum shortly.
Update: Still no issues here, how about over there?
