Chatbot took long to respond and in the end "Oops! Unfortunately, I'm unable to answer right now."

[spencerwongfeilong]

I crawl an entire website. And i an running webwhiz on a ubuntu machine.

In the logs of front end, it shows the following.

frontend_1 | 2024/02/21 02:08:50 [error] 31#31: *7 upstream timed out (110: Operation timed out) while reading response header from upstream, client: 10.200.230.2, server: , request: "POST /api/chatbot/answer/?debug=true HTTP/1.1", upstream: "http://172.19.0.4:3000/chatbot/answer/", host: "192.168.0.124:3030", referrer: "http://192.168.0.124:3030/app/edit-chatbot/65d466ea2177ff2312d365a7/"
frontend_1 | 10.200.230.2 - - [21/Feb/2024:02:08:50 +0000] "POST /api/chatbot/answer/?debug=true HTTP/1.1" 504 569 "http://192.168.0.124:3030/app/edit-chatbot/65d466ea2177ff2312d365a7/" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36" "-"

[spencerwongfeilong]

It seems that the python worker took a long time to respond. While it did respond eventually, after 270 seconds, i guess it was too late. What can i do to help resolve this?

I would like to add that there are 1000 over pages crawled. This could be the underlying root cause.