Problem with testing in another envrionment

[sattlelite]

Hello, Mr. Reiniscimurs, I have done the test after training, bleow is the average reward function generated by my training of the improved method.In addition, I conducted tests on the original map, and basically reached the target, although there was a collision in a few cases. I then created a 30x30 size map with some sparse and regular obstacles in it and replaced the TD3.world file, and I modified the velodyne_env.py file so that the goal would not fall in the obstacles.The question is, when everything is running properly, the robot will stay in place and rotate at a small Angle to the left and right, and will not reach its destination. The answer seems to be obvious that the robot has never been trained beyond the present situation, so,dose it needs to be trained in a new environment until the positive reward is stable, right? I also wonder if the new envrionment requires more parameters for the TD3 network? If I increase the number of states, do I need a large number of network parameters? Looking forward to your reply. Thank you so much!

[sattlelite]

Sorry, the top two curves in this graph are 2000 real_time_rate and the bottom is raw time.

[reiniscimurs]

Hi,
No there is no explicit need to train the model in the new environment. The issue here could be that either it has encountered local optima, or the input sensor values are outside of the trained range. For the latter, you could just cap the max sensor values to the max values in training. I.E. distance = min(distance, max_dist_in_training) and same with laser sensors. This should give you a reasonable performance in local planning setting. You of course can also re-train the model. But I am not sure if you would need to increase the model size. The only thing that would realistically change is the distance to the goal as max distance there would grow. And since it would take longer to actually get to the goal, it might be useful to figure out the proper discount factor and learning rate.

[sattlelite]

Hello Mr. reincimurs,
I agree with you, I hope to try in the next experiment, and plan to train the model in the new map.But at the moment I have two questions:
1.do you mean to only get the maximum when you get the state? Do I need to modify this part of the code in "velodyne_env"?

2.I am having trouble opening the world file while modifying the gazebo environment which I had changed from the original environment. I searched the Internet for a solution,and some said that gazebo needed to load materials online, and network issue were preventing them,so I installed the material pack locally. But it still reports an error. Here is the information running.

and my file is here.

[reiniscimurs]

1. I am not sure if I understand the questionn, but if you do not modify anything in laser sensor itself, then this should work just fine in larger environments as well.
2. This is outside the scope of this repo so i could not consult you with your issue here. I have no way to check your world file at the moment.

[sattlelite]

Thank you for your reply.
Fortunately, I have resolved the second question. I just carefully recreated another environment. I suspect I just broke the primal files.
And now I am training in the 30*30 map. Before this, I modify the hokuyo.xacro file and self.velodyne_data as follows:

So I let laser_state get a 20m range of data. After that, I design a tactic to force the robot to go straight when the robot sees the goal. The tactic will let the action consistent about 10 steps.
Then I train the robot on the new map. In the early times, it could achieve the goal, but now it just runs a circle after 10 minutes.

74167686953__237F55D0-173B-43F6-B4A1-624B59D41E45.MOV

A larger map needs more time to train, but could you please give me some suggestions?
Is there anything I dismiss?

[reiniscimurs]

Are you using the main branch? The laser sensor is only used to detect collisions. It will not affect how the state is represented.

[sattlelite]

Yes. I use the main branch, I use the state[8:-12].
Then I change two lines of code in velodyne_env.py, then I print the state[8:-12] and check the terminal:

Additionally, because the map I created have many cylinders, and everytime I train I find that after a while almost half of the posts fall down, I suspected that the posistion of the robots was reset to the cardboardbox, which caused them ot crash violently and affected the posts, so I set every one‘s tags
<self_collide>1</self_collide> <kinematic>1</kinematic>
on them in the world to fix them. Because I don't want a lot of changes in the envrionment to affect the training of the robot.
Thanks for your patience and reply, I just want it to train successfully on this new map.

[sattlelite]

Hello Mr. Reiniscimurs,
I have trained the model in the 30x30 map with 2x acceleration for 14 hours. And I found that the robot can’t find the right policy to get to the goal. And the average reward was very low in the end. I have added the OU noise with “sigma =0.7” and “dt=0.09””mu=0”.I have increase the goal_reach _dist from 0.3 to 0.6. I also multiplied the final noise by an attenuation factor (just use the expl_noise).

Should I need to adjust the max angular velocity？Could you please give some advice?

[reiniscimurs]

I think I am missing the overall picture of what changes you have made to the code here. Can you please explain all changes in detail is a structured way please? It is quite hard for me to follow all the details here.

For instance, you mention changing the max distance in laser sensor setup. However, laser sensor is not used to get state information. For that you would have to update the supporting velodyne puck sensor files, but that is not mentioned, so it is unclear to me if it was done or not. It is also unclear to me what you mean by OU noise.