Discussion:
Issues with command line
(too old to reply)
Mitchell Rathbun (BLOOMBERG/ 731 LEX)
2018-09-19 00:39:31 UTC
Permalink
As has been discussed before, when submitting a topology there are really two processes that are started up, with one being the logwriter process and the other being the worker process. When submitting our topology, we independently set TOPOLOGY_WORKER_CHILDOPTS to "-Xmx64m" and TOPOLOGY_WORKER_LOGWRITER_CHILDOPTS to "Xmx2g". When running ps -ef, the logwriter process command displays "-Xmx64m" followed by "-Xmx2g". Why is this? TOPOLOGY_WORKER_CHILDOPTS should be specific to the worker process and should not have any effect on the logwriter process. Is this assumption correct? If so, why does it appear that TOPOLOGY_WORKER_CHILDOPTS appears to be taking precedence over TOPOLOGY_WORKER_LOGWRITER_CHILDOPTS for the logwriter process?
Mitchell Rathbun (BLOOMBERG/ 731 LEX)
2018-09-19 14:51:30 UTC
Permalink
Made a small mistake when describing this. We independently set TOPOLOGY_WORKER_CHILDOPTS to "-Xmx2g" and TOPOLOGY_WORKER_LOGWRITER_CHILDOPTS to "Xmx64m". So the command line for the logwriter process should only have -Xmx64m show up, and definitely should not have -Xmx2g show up after -Xmx64m.

From: Mitchell Rathbun (BLOOMBERG/ 731 LEX) At: 09/18/18 20:39:31To: ***@storm.apache.org
Subject: Issues with command line
As has been discussed before, when submitting a topology there are really two processes that are started up, with one being the logwriter process and the other being the worker process. When submitting our topology, we independently set TOPOLOGY_WORKER_CHILDOPTS to "-Xmx64m" and TOPOLOGY_WORKER_LOGWRITER_CHILDOPTS to "Xmx2g". When running ps -ef, the logwriter process command displays "-Xmx64m" followed by "-Xmx2g". Why is this? TOPOLOGY_WORKER_CHILDOPTS should be specific to the worker process and should not have any effect on the logwriter process. Is this assumption correct? If so, why does it appear that TOPOLOGY_WORKER_CHILDOPTS appears to be taking precedence over TOPOLOGY_WORKER_LOGWRITER_CHILDOPTS for the logwriter process?
Mitchell Rathbun (BLOOMBERG/ 731 LEX)
2018-09-20 22:03:47 UTC
Permalink
I looked at this a little closer. -Xmx64m is passed as an option to the Logwriter process, while -Xmx2g is passed as an argument. This argument is then passed to the worker process, which is a child process of LogWriter. So the memory limits are being respected.

From: ***@storm.apache.org At: 09/19/18 10:51:42To: ***@storm.apache.org
Subject: Issues with command line

Made a small mistake when describing this. We independently set TOPOLOGY_WORKER_CHILDOPTS to "-Xmx2g" and TOPOLOGY_WORKER_LOGWRITER_CHILDOPTS to "Xmx64m". So the command line for the logwriter process should only have -Xmx64m show up, and definitely should not have -Xmx2g show up after -Xmx64m.

From: Mitchell Rathbun (BLOOMBERG/ 731 LEX) At: 09/18/18 20:39:31To: ***@storm.apache.org
Subject: Issues with command line
As has been discussed before, when submitting a topology there are really two processes that are started up, with one being the logwriter process and the other being the worker process. When submitting our topology, we independently set TOPOLOGY_WORKER_CHILDOPTS to "-Xmx64m" and TOPOLOGY_WORKER_LOGWRITER_CHILDOPTS to "Xmx2g". When running ps -ef, the logwriter process command displays "-Xmx64m" followed by "-Xmx2g". Why is this? TOPOLOGY_WORKER_CHILDOPTS should be specific to the worker process and should not have any effect on the logwriter process. Is this assumption correct? If so, why does it appear that TOPOLOGY_WORKER_CHILDOPTS appears to be taking precedence over TOPOLOGY_WORKER_LOGWRITER_CHILDOPTS for the logwriter process?
Loading...