Cannot load the program you emailed, looks like the file has become corrupted. Please save and send again.
I load your prg file in my T3-BB (version 50.1), and I also click the VAR113 ON ->OFF OFF->ON , and I can’t repeat the issue at your site.
I see your Graphics have more than 100 points , Please try to reduce the number of dots in graphics.
Many unexpected situations are caused by programming. You are using program 1-14 , please check all of this programs as expected.
1 Like
Maurice,
I have a similar problem. The latest T3-LB in my network hangs when it tries to retrieve data via Ethernet from the Chiller T3. I have tried so many things, to no avail. I thought it might be that I had some variables with the same name on both controllers. Fixed it, still hangs.
I thought it was because I was retrieving some data from the T3 via Bacnet style (instance & variable) and some via the new style that just requires the panel number. Made it all ethernet, still hangs
I put all four T3’s on one little ethernet switch. Pings a bit faster, but still hangs.
I tested the network by pinging each T3 50 times. No errors. So it’s not a network problem.
I am at the end of my rope. I hope you can help me.
1 Like
One further thing to note about my predicament. See the attached alarm list. I gen an alarm every time the chiller time and the VFD-T3 time vary by more than 40 seconds. Oddly, it always comes with a second alarm that means nothing to me. Maybe it will give you or Chelsea a clue as to what is going on.
T3000 is Dec 6, 2019 and all firmware is updated to 50.1.
Hope I understand your comments correctly, and here’s my two bits worth:
In general you should try to make each controller self sufficient so that if the network cable goes down equipment can still behave reasonably well. If you do detect a network outage then add a few lines to fail to a safe condition.
Time Sync Alarm:
Time sync is actually done automatically in the background so you shouldn’t have to manually be polling time from one controller to the other. The time sync alarm comes up when the T3 tries to reach the NTP time server but cannot get through. In our office lan the network is locked down so the T3 needs to sync to one of the PC’s in the office. You can check communications manually using the ‘update’ button.
Purple Line Hits 100:
This looks like you have a timer of some sort that is able to detect network outages. We’ll set up a test here and see what we can see. I will put a ramping program in one controller and log it from another controller and see what we can see.
10 IF INTERVAL 0:0:01 THEN VAR1 = VAR1 + 1
20 IF VAR1 > 100 THEN VAR1 = 0
1 Like
Eureka!.. I think. Somewhere on this troubleshooting blog I noticed that someone mentioned their T3 hung when frequently plotting points gathered from an RS485 network. My “Purple Line” was being plotted from a variable calculated by the absolute value of the difference between the system time from the programmed T3 and the system time on another T3 on the same ethernet network called the chiller T3. Whenever they’re off by more than 45 seconds, there must be a problem getting data over ethernet from the chiller T3. I logged that absolute value every 3 minutes. Based on the hint I got from the post with trouble on an RS485 network, I thought it might be hanging my system. So I changed the logging interval from every 3 minutes to every thirty minutes. It ran all night without any significant hangs. This morning I killed the graph info gathering altogether by turning the graph (trend log) off. It has run like a champ all day!
From now on, I will just rely on the Alarm log to let me know if there’s a problem.
1 Like
The network points get cached and polled automatically in the background, there’s no way right now at least, to adjust the interval.
Here’s the results of my test, a program in one controller ramps a variable. A second program in controller sets a local variable to the remote controller and both variables are logged. If there were disruptions in the network traffic they would show up as jagged horizontal lines in these graphs but I wasn’t able to see any jags going back a couple days now. This test just verifies that the network update mechanism is sound and working in this environment, your mileage may vary.
I will re-emphasize my earlier suggestion: If you are going to rely on the network for a particular process make sure you also have logic to make sure everything fails to safe mode when the network goes down, you can count on it to happen once in a while.
1 Like
Per your test below, I may not have all the answers. After working fine for 48 hours, my VFD-T3 lost contact over Ethernet with my Chiller-T3 at about 22:30 last night. The alarm log captured the situation and also had the companion “update time server fail” error at the same time. This has nothing to do with going out on the web to sync to a time server because all my T3’s are set to “Synchronize with Local PC”. Here’s a new piece of the puzzle. The Chiller-T3’s graph of compressor amps had a 2.5 hour gap in it. I have noticed that at other times and had not bothered to mention it because it’s a little bug I can live with. What’s interesting here is that it resumed the trend log at the same time the ethernet comm was lost between the two T3’s. Here’s something else. This is the way the Network Points table looked on VFD-T3 just before I rebooted it:
108362 is my chiller-T3. Point 14 is the time variable. Point 26 is the number of compressors running. We only have two.
Can I ask you to set up another test that more closely reflects our environment here?
Set-up 4 T3’s on an Ethernet. Set them all to “Synchronize with Local PC” on the time update. Have one of the 4 be the central T3 where the other 3 are all requesting info from it’s variables via ethernet. On the central T3, set one of the variables to the system TIME. Then run the following program on one of the other T3’s while running a trend log of the DIFERROR variable:
10 REM COMPARE TIME RETRIEVED FROM CHILLER VS VFD
20 REM IF THEY ARE TOO DIFFERENT, THEN THERE MUST BE
30 REM SOME COMMUNICATION ERROR. WHEN THAT HAPPENS
40 REM SPIN UP THE PUMPS AND THROW OUT AN ALARM
50 REM FIGHT CHILLER HIGH HEAD PRESSURE AT ALL COSTS
150 REM CHILTIME = 108362AV14 ( DONE IN ANOTHER PROGRAM )
200 VFDTIME = TIME
300 WAIT 5
400 DIFERROR = ABS ( CHILTIME - VFDTIME )
450 IF DIFERROR > 100 THEN DIFERROR = 99
500 ALARM DIFERROR > 60 , 10 , THE VFD PANEL IS NOT TALKING TO THE CHILLER PANEL
999 END
Setting up this test is a lot to ask, but having reliable Ethernet comms from one panel to another is worth it. It ought to bear fruit in about 48 hours.
Ted
1 Like
Hi guys.
I’m testing the operation of the T3-LB controller with my big program on the latest firmware mini_arm_rev50.1_12.10.hex.
I observe very often controller reboots when I change the values of any variables or work in the GUI.
@maurice , I sent you my program, but I did not find out the reason for the controller reboot from you, I just received recommendations to reduce the program and reduce the number of points in the GUI.
I decided to test my program on the controller with earlier firmware and as a result, on the mini_arm_rev49.9_11.6.hex firmware, the controller reboots stopped.
But on this firmware and earlier there is a problem of periodically shutting down user programs for reasons unknown to me (switching from ON to OFF). Programs are rarely turned off, and not the same ones, sometimes for example, program number 3 will turn off, and sometimes number 5.
After turning on the programs (switching from OFF to ON), everything normalizes and can work for several days, then the situation may repeat again.
I suspect that @chelsea made some kind of algorithm in the firmware, which began to reboot the entire controller, and not turn off the programs: https://forums.temcocontrols.com/t/t3-lb-controller-writing-error/580/9. But this led to a big problem, the controller became almost uncontrollable.
I suggest that you recall what changes were released in the firmware and change it so that the programs do not turn off and the controller does not reboot.
This is very important, otherwise we will not be able to use your controllers in our commercial projects due to their unpredictability.
I hope you hear me as soon as possible.
1 Like
Thanks for your detailed work on getting to the bottom of this Aleksei, we will support you well but need to ask for a few days till the local holiday is over, Feb2.
A few days ago we started having trouble communicating to one of our T3’s. The usual remedy is to go up to the Hot Water penthouse on the roof and reboot it. After doing that 2 days in a row, I decided to try a different approach. I simply unplugged the Ethernet connector from the unit, waited 30 seconds and reconnected it. The network connection re-established itself and the dreaded red X went away.
At the same time, I was surprised to see activity on the Main RS485 red & green lights. I checked the setup screen and it was NOT set to “Unused”. So I set it properly.
It’s been several days and we’ve no more issues. Maybe one of these actions helped, maybe not. But I thought it might be helpful for you to know.
Hi @tdawgtoo , it’s a really strange thing, but I think it has nothing to do with my question about spontaneous shutdown of programs in the T3-LB.
I look forward to February 3, when I get an answer from @maurice and @chelsea to my question and the controller finally works well.
1 Like
Hi,
For some reason, no one answers, strange.
I had the idea to write watchdog in one of the programs, which would control the shutdown of the programs and turn them on again.
I managed to get a message about disabling a specific program PRG1, but when the PLC try to restart it from the watchdog program, nothing changes. Maybe I’m doing it wrong? Here is a sample wathdog code:
10 DALARM NOT PRG1 , 5 , PRG1 OFF
20 IF NOT PRG1 THEN START PRG1
I changed the mode of the controlled PRG1 program in both Auto and Manual, but without result.
Thanks for your input Aleksei. I could be wrong but going from memory, there’s no commands to turn a program on and off from within a program. If you have seen something in the documentation that says otherwise I will check into this more.
Tdawg: When you reset the T3 controller, is the heartbeat LED blinking normally about once per second? If so we need to look into this more, send on your program and I’ll see if we can repeat it. If the LED is NOT blinking normally then the controller is locked up and its supposed to reset itself, there are no know issues like that so let me know and we’ll work on this with you in depth. This thread is wandering around too many topics, please start a new thread if there’s something not directly to the main topic.
Hi Maurice, I did not find a specific example in the manual for turning one program ON or OFF from another program, but I think this is possible. In any case, we urgently need to do this or fix the problem with the sudden shutdown of programs in the firmware. It’s not the first month that I’ve been debugging your controller and we just have to take the last step.
Let’s do it, okay?
1 Like