Rate Group Error

[sheershak-agarwal]

Hello,

I have initialized rate groups in my project but it always fails after some time. I am printing Scheduler for MSM ON in rate group 1 (every one second). But I get an FATAL 122 error. What does this number represent? Did I not connect it properly in Topology.xml?

EVENT: (121) (2:1630515645,339964) WARNING_HI: (health) HLTH_PING_WARN : Ping entry GPSUARTDRV late warning
EVENT: (121) (2:1630515645,340028) WARNING_HI: (health) HLTH_PING_WARN : Ping entry PAYLOADUARTDRV late warning
EVENT: (121) (2:1630515645,340047) WARNING_HI: (health) HLTH_PING_WARN : Ping entry WSUARTDRV late warning
EVENT: (121) (2:1630515645,340057) WARNING_HI: (health) HLTH_PING_WARN : Ping entry BDRV late warning
Scheduler for MSM ON
Scheduler for MSM ON
Scheduler for MSM ON
Scheduler for MSM ON
Scheduler for MSM ON
Scheduler for MSM ON
Scheduler for MSM ON
Scheduler for MSM ON
FATAL 122 handled.
Scheduler for MSM ON
Exiting.
Segmentation fault (core dumped)

[sheershak-agarwal]

Can the rate groups only be include in passive and queued components? Because I have added the SchedIn in an active component.

[timcanham]

You can connect an active component to the rate group, but you should make the input port asynchronous or guarded so you don't have multiple threads accessing data in the component at the same time.

[sheershak-agarwal]

I don't think that is the issue. I commented the SchedIn Pin from my component but I see the same error.

[timcanham]

The four components mentioned are not returning the ping code, that's why you are seeing the WARNINGs and the eventual FATAL.

Did you implement ping handlers for the components? Are they Active? Note that if they are Queued, you have to call doDispatch() in the scheduler handler to dispatch any pending messages including the pings.

[timcanham]

This is correct. If you don't have ping ports on your component, it can't echo back the ping response. What do you have the Health ping ports connected to?

[sheershak-agarwal]

So if I understand that correctly, I have PingSend and PingRecv port for each component in my Top design (not the UART Drivers). Though, I did not see any example where they are connecting components with health and I am misunderstanding something. And I should also remove the "{3,5,getHealthName(ttcUartDrv)},
{3,5,getHealthName(gpsUartDrv)}, // 13
{3,5,getHealthName(adcsUartDrv)},
{3,5,getHealthName(payloadUartDrv)}, // 13
{3,5,getHealthName(wsUartDrv)}," lines from my AppTopology.

I have listed all my health connections here: you can also see it here https://github.com/sheershak-agarwal/F-Prime-Debug/blob/main/App/Top/AppTopologyAppAi.xml ( line 834 -919)
`

`

[rawinza555]

@sheershak-agarwal Pretty much yes. I haven't used that much of Health myself. I happened to stumble upon this thing when I looked at the ComLogger. I believe what it does is like sending NO_OP Command but to a specific component,

If you want to have this pinging feature on those UART driver, you have to add Ping ports and write a input handler. You can take a look at components that utilize this feature and mimic them to your new UART driver.

[timcanham]

The components have to be active in order to respond to the ping in time. You shouldn't use them with queued components. The concept is that active components are pinged to make sure they are alive. If one doesn't respond within the time limit, Health will issue a FATAL assuming that something has gone wrong. It's an optional feature - it catches weird hangs and the like, but it's not required.

What puzzles me is that it is not asserting when trying to invoke ping output ports that are not connected, which is what should happen since you have entries in the table but no port connections corresponding to them.

BTW, FATAL 122 is the ID of the FATAL sent by Health when it stops getting ping replies.

[sheershak-agarwal]

I was getting assert but I didn't paste it with the log. So I removed the uarts drivers form health but now chanTlm is throwing an error.

EVENT: (1541) (2:1630600036,721359) DIAGNOSTIC: (RG1) RateGroupStarted : Rate group started.
EVENT: (1561) (2:1630600036,721430) DIAGNOSTIC: (RG2) RateGroupStarted : Rate group started.
EVENT: (1581) (2:1630600036,721493) DIAGNOSTIC: (RG3) RateGroupStarted : Rate group started.
Assert file "/home/sheershak/Desktop/F-Prime-Debug/App/build-fprime-automatic-raspberrypi/F-Prime/Svc/Ping/PingPortAc.c
EVENT: (121) (2:1630600048,725057) WARNING_HI: (health) HLTH_PING_WARN : Ping entry TLM late warning
FATAL 122 handled.
Exiting.
Segmentation fault

[timcanham]

You should make sure that the numbers here:

https://github.com/sheershak-agarwal/F-Prime-Debug/blob/main/App/Top/AppTopologyAppAi.xml#L844

Match the numbers in the table here:

https://github.com/sheershak-agarwal/F-Prime-Debug/blob/main/App/Top/Topology.cpp#L223

[timcanham]

It looks like 7 is missing from the topology. It makes your life easier to number them the same in both places and in order to make sure they match.

[sheershak-agarwal]

Thank you so much. That solved the issue.

[sheershak-agarwal]

One more question,

When I am adding my components to the health ping, object at number 13 throws this error.

Svc::HealthImpl::PingEntry pingEntries[] = { {3,5,getHealthName(rateGroup1Comp)}, // 0 {3,5,getHealthName(rateGroup2Comp)}, // 1 {3,5,getHealthName(rateGroup3Comp)}, // 2 {3,5,getHealthName(cmdDisp)}, // 3 {3,5,getHealthName(eventLogger)}, // 4 {3,5,getHealthName(cmdSeq)}, // 5 {3,5,getHealthName(chanTlm)}, // 6 {3,5,getHealthName(fileUplink)}, // 7 {3,5,getHealthName(fileDownlink)}, // 8 {3,5,getHealthName(blockDrv)}, // 9 {3,5,getHealthName(msmManager)}, // 10 {3,5,getHealthName(adcsManager)}, // 11 {3,5,getHealthName(cmdManager)}, // 12 {3,5,getHealthName(deployerManager)}, // 13 // {3,5,getHealthName(gps)}, // 14 // {3,5,getHealthName(plManager)}, // 15 // {3,5,getHealthName(telemetryManager)}, // 16 // {3,5,getHealthName(comManager)}, // 17 };

Hit Ctrl-C to quit
Assert file "/home/sheershak/Desktop/F-Prime-Debug/fprime/Fw/Obj/SimpleObjRegistry.cpp": Line: 63
Assert file "/home/sheershak/Desktop/F-Prime-Debug/fprime/Fw/Obj/SimpleObjRegistry.cpp": Line: 63
Segmentation fault

[sheershak-agarwal]

I doesn't matter what component is there. It always fails at 13. I tried to increase the base_id_window size of the health instance in AppTopologyAppAi.xml but that didn't help. is there a max limit to the ping entries? If not, then what can cause the error ? I am following the same Ping Send and Ping Recv method in adcsManager(number 11) and cmdManager (number 12)

[timcanham]

That assert shows you are running out of slots in the ObjectRegistry, which stores pointers to created objects.

There is a procedure for setting up a local config file to update the value to make more slots:

https://nasa.github.io/fprime/UsersGuide/dev/configuring-fprime.html

That should get rid of the asserts.

[sheershak-agarwal]

Ok I will read this. Make more slots of what? The file is huge so will come back to you with other questions

[sheershak-agarwal]

These are the steps I am following:

1. Adding ac_constants: ./Cfg/AcConstants.ini in settings.ini
2. Create a file AcConstants.ini in Cfg folder
3. Adding line HealthPingPorts = 50 to AcConstants.ini
4. compile and I saw thsi erroe

configparser.MissingSectionHeaderError: File contains no section headers.
file: '/home/sheershak/Desktop/F-Prime-Debug/App/Cfg/AcConstants.ini', line: 1
'HealthPingPorts: 50'

[timcanham]

I would copy the one from FPrime and use it as a starting point:

https://github.com/nasa/fprime/blob/devel/config/AcConstants.ini

[timcanham]

Copy FpConfig there too and modify:

 #define FW_OBJ_SIMPLE_REG_ENTRIES           500  //!< Number of objects stored in simple object registry

[sheershak-agarwal]

I see the same error after adding the line in Cfg/FpCongi.hpp file

raceback (most recent call last):
File "/home/sheershak/Desktop/F-Prime-Debug/fprime/Autocoders/Python/bin/codegen.py", line 1779, in
file: '/home/sheershak/Desktop/F-Prime-Debug/App/Cfg/AcConstants.ini', line: 1
main()
'HealthPingPorts: 50'
File "/home/sheershak/Desktop/F-Prime-Debug/fprime/Autocoders/Python/bin/codegen.py", line 1709, in main
make[3]: *** [App/CmdManager/CmdManagerComponentAc.cpp] Error 255
xml_filename
File "/home/sheershak/Desktop/F-Prime-Debug/fprime/Autocoders/Python/src/fprime_ac/parsers/XmlComponentParser.py", line 100, in init
self.__const_parser.read(constants_file)
File "/usr/lib/python3.6/configparser.py", line 697, in read
self._read(fp, filename)
File "/usr/lib/python3.6/configparser.py", line 1080, in _read
raise MissingSectionHeaderError(fpname, lineno, line)
configparser.MissingSectionHeaderError: File contains no section headers.
file: '/home/sheershak/Desktop/F-Prime-Debug/App/Cfg/AcConstants.ini', line: 1
'HealthPingPorts: 50'
App/CmdManager/CMakeFiles/App_CmdManager.dir/build.make:85: recipe for target 'App/CmdManager/CmdManagerComponentAc.cpp' failed
CMakeFiles/Makefile2:6239: recipe for target 'App/CmdManager/CMakeFiles/App_CmdManager.dir/all' failed
make[2]: *** [App/CmdManager/CMakeFiles/App_CmdManager.dir/all] Error 2
CMakeFiles/Makefile2:6422: recipe for target 'App/Top/CMakeFiles/App.dir/rule' failed
make[1]: *** [App/Top/CMakeFiles/App.dir/rule] Error 2
Makefile:2132: recipe for target 'App' failed
make: *** [App] Error 2
[ERROR] CMake erred with return code 2

[timcanham]

Can you push it to your branch? It looks like it is complaining about the syntax of your AcConstants.ini.

[sheershak-agarwal]

I added config_directory: Cfg in settings.ini
and then I get different error:

Building CXX object F-Prime/Fw/Cfg/CMakeFiles/Fw_Cfg.dir/ConfigCheck.cpp.obj
/home/sheershak/Desktop/F-Prime-Debug/fprime/Fw/Cfg/ConfigCheck.cpp:18:16: error: expected constructor, destructor, or type conversion before ‘(’ token
F-Prime/Fw/Cfg/CMakeFiles/Fw_Cfg.dir/build.make:75: recipe for target 'F-Prime/Fw/Cfg/CMakeFiles/Fw_Cfg.dir/ConfigCheck.cpp.obj' failed
FW_CONFIG_ERROR(FW_CMD_STRING_MAX_SIZE <= FW_CMD_ARG_BUFFER_MAX_SIZE,CMD_STRING_TOO_BIG);
               ^
/home/sheershak/Desktop/F-Prime-Debug/fprime/Fw/Cfg/ConfigCheck.cpp:19:16: error: expected constructor, destructor, or type conversion before ‘(’ token
FW_CONFIG_ERROR(FW_LOG_STRING_MAX_SIZE <= FW_LOG_BUFFER_MAX_SIZE,LOG_STRING_TOO_BIG);
               ^
/home/sheershak/Desktop/F-Prime-Debug/fprime/Fw/Cfg/ConfigCheck.cpp:20:16: error: expected constructor, destructor, or type conversion before ‘(’ token
FW_CONFIG_ERROR(FW_TLM_STRING_MAX_SIZE <= FW_TLM_BUFFER_MAX_SIZE,TLM_STRING_TOO_BIG);
               ^
/home/sheershak/Desktop/F-Prime-Debug/fprime/Fw/Cfg/ConfigCheck.cpp:21:16: error: expected constructor, destructor, or type conversion before ‘(’ token
FW_CONFIG_ERROR(FW_PARAM_STRING_MAX_SIZE <= FW_PARAM_BUFFER_MAX_SIZE,PRM_STRING_TOO_BIG);
               ^
/home/sheershak/Desktop/F-Prime-Debug/fprime/Fw/Cfg/ConfigCheck.cpp:25:16: error: expected constructor, destructor, or type conversion before ‘(’ token
FW_CONFIG_ERROR((FW_ENABLE_TEXT_LOGGING == 1) && (FW_SERIALIZABLE_TO_STRING == 1),FW_SERIALIZABLE_TO_STRING_not_enabled_for_FW_ENABLE_TEXT_LOGGING);
               ^
make[3]: *** [F-Prime/Fw/Cfg/CMakeFiles/Fw_Cfg.dir/ConfigCheck.cpp.obj] Error 1
CMakeFiles/Makefile2:2013: recipe for target 'F-Prime/Fw/Cfg/CMakeFiles/Fw_Cfg.dir/all' failed
make[2]: *** [F-Prime/Fw/Cfg/CMakeFiles/Fw_Cfg.dir/all] Error 2
CMakeFiles/Makefile2:6422: recipe for target 'App/Top/CMakeFiles/App.dir/rule' failed
make[1]: *** [App/Top/CMakeFiles/App.dir/rule] Error 2
Makefile:2132: recipe for target 'App' failed
make: *** [App] Error 2
[ERROR] CMake erred with return code 2

[sheershak-agarwal]

is there supposed to be any header in the ini file like [fprime] in settings.ini

[timcanham]

The expected pattern is that users will copy the entire F Prime config folder and modify it for your purposes.

Put the following in your settings.ini:

ac_constants: ./Cfg/AcConstants.ini/AcConstants.ini
config_directory: ./Cfg/AcConstants.ini

The issue you are having is with the FW_OBJ_SIMPLE_REG_ENTRIES setting above. Increase it to 600 or so.

[timcanham]

You do have to copy all the files with all their contents and modify what you need.

[sheershak-agarwal]

You are the best. I copied the whole config file and made #define FW_OBJ_SIMPLE_REG_ENTRIES 1000 and HealthPingPorts = 50 and works fine. Now I will add the components to health ping. Thank you so much

[timcanham]

Great! Good luck!