ATS crashes when Metalink Enabled in plugins.config #1

Degreane · 2014-01-27T17:25:26Z

Hi
I downloaded several versions of ATS that are available (4.1.x ~ 4.2.x ) and all compiled with --enable-experimental-plugins as described in the manual ./configure --help shows

nevertheless whenever i enable the plugin metalink traffic_server crashes and the manager keeps restarting it

tried it with different versions of linux. (ubuntu 13.10 - fedora 18,19,20) same issue

Much Regards

[Jan 27 18:47:51.693] Manager {0xb7609780} NOTE: [ClusterCom::ClusterCom] Node running on OS: 'Linux' Release: '3.12.8-300.fc20.i686+PAE'
[Jan 27 18:47:51.693] Manager {0xb7609780} NOTE: [LocalManager::listenForProxy] Listening on port: 8080
[Jan 27 18:47:51.693] Manager {0xb7609780} NOTE: [TrafficManager] Setup complete
[Jan 27 18:47:52.712] Manager {0xb7609780} NOTE: [LocalManager::startProxy] Launching ts process
[Jan 27 18:47:52.717] Manager {0xb7609780} NOTE: [LocalManager::pollMgmtProcessServer] New process connecting fd '13'
[Jan 27 18:47:52.717] Manager {0xb7609780} NOTE: [Alarms::signalAlarm] Server Process born

[Jan 27 18:48:35.991] Manager {0xb7609780} FATAL: [LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
[Jan 27 18:48:35.991] Manager {0xb7609780} NOTE: [LocalManager::mgmtShutdown] Executing shutdown request.
[Jan 27 18:48:35.991] Manager {0xb7609780} NOTE: [LocalManager::processShutdown] Executing process shutdown request.
[Jan 27 18:48:35.991] Manager {0xb7609780} ERROR: [LocalManager::sendMgmtMsgToProcesses] Error writing message
[Jan 27 18:48:35.991] Manager {0xb7609780} ERROR: (last system error 32: Broken pipe)

hopefully you may check it out and get it fixed ..

antbryan · 2014-02-07T20:50:22Z

from @jablko

I pushed the updated plugin to GitHub (hopefully it will also be distributed with the next Traffic Server release).
To build it,

download the updated metalink.cc file
replace the plugins/experimental/metalink/metalink.cc file in your Traffic Server source tree with the updated file,
and rebuild Traffic Server by rerunning "make".

Degreane · 2014-02-07T20:51:36Z

excuse me but when did you update it ???

much regards

     Faysal Banna

Meteorological Services
Rafic Harriri International Airport
Beirut - Lebanon
Mob: +961-3-258043

On 02/07/2014 10:50 PM, antbryan wrote:

from @jablko https://github.com/jablko

I pushed the updated plugin to GitHub (hopefully it will also be
distributed with the next Traffic Server release).
To build it,

download the updated metalink.cc file

replace the plugins/experimental/metalink/metalink.cc file in your
Traffic Server source tree with the updated file,

and rebuild Traffic Server by rerunning "make".

—
Reply to this email directly or view it on GitHub
#1 (comment).

antbryan · 2014-02-07T21:01:37Z

if you look at the commit log, you can see that @jablko updated it a few days ago on 4 & 5 Feb.

have you tried the new code since then?

it's also been included upstream, so hopefully should be available w/ the next ATS release: https://git-wip-us.apache.org/repos/asf?p=trafficserver.git

Degreane · 2014-02-07T21:38:58Z

I already used this metalink.cc with ATS 5.0.x and 4.1.2 and both give this error below when i activate the metalink plugin...

did you compile it and test it yourself ?

anyhow the error is as shows
[TrafficServer] using root directory '/usr/local'
FATAL: InkAPI.cc:6248: failed assert sdk_sanity_check_iocore_structure(connp) == TS_SUCCESS
/usr/local/bin/traffic_server - STACK TRACE:
/usr/local/lib/libtsutil.so.4(+0xf36b)[0x4001136b]
/usr/local/lib/libtsutil.so.4(+0xe23d)[0x4001023d]
/usr/local/bin/traffic_server(TSVConnWrite+0x2c9)[0x80f4489]
/usr/local/libexec/trafficserver/metalink.so(+0x1d0f)[0x40047d0f]
/usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x15f)[0x83087bf]
/usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x7a3)[0x83093c3]
/usr/local/bin/traffic_server(main+0xc6a)[0x80bc04a]
/lib/libc.so.6(__libc_start_main+0xf3)[0x47794b73]
/usr/local/bin/traffic_server[0x80c134d]
[TrafficServer] using root directory '/usr/local'
FATAL: InkAPI.cc:6248: failed assert sdk_sanity_check_iocore_structure(connp) == TS_SUCCESS
/usr/local/bin/traffic_serverFATAL: InkAPI.cc:6248: failed assert sdk_sanity_check_iocore_structure(connp) == TS_SUCCESS

STACK TRACE:
/usr/local/bin/traffic_server - STACK TRACE:
/usr/local/lib/libtsutil.so.4(+0xf36b)[0x4001136b]
/usr/local/lib/libtsutil.so.4(+0xe23d)[0x4001023d]
/usr/local/bin/traffic_server(TSVConnWrite+0x2c9)[0x80f4489]
/usr/local/libexec/trafficserver/metalink.so(+0x1d0f)[0x40047d0f]
/usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x15f)[0x83087bf]
/usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x7a3)[0x83093c3]
/usr/local/bin/traffic_server[0x830830b]
/lib/libpthread.so.0[0x47948d8a]
/lib/libc.so.6(clone+0x5e)[0x47872a0e]
[TrafficServer] using root directory '/usr/local'
FATAL: InkAPI.cc:6248: failed assert sdk_sanity_check_iocore_structure(connp) == TS_SUCCESS
/usr/local/bin/traffic_server - STACK TRACE:
/usr/local/lib/libtsutil.so.4(+0xf36b)[0x4001136b]
/usr/local/lib/libtsutil.so.4(+0xe23d)[0x4001023d]
/usr/local/bin/traffic_server(TSVConnWrite+0x2c9)[0x80f4489]
/usr/local/libexec/trafficserver/metalink.so(+0x1d0f)[0x4004cd0f]
/usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x15f)[0x83087bf]
FATAL: InkAPI.cc:6248: failed assert sdk_sanity_check_iocore_structure(connp) == TS_SUCCESS
/usr/local/bin/traffic_server - STACK TRACE:
/usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x7a3)[0x83093c3]
/usr/local/lib/libtsutil.so.4(+0xf36b)[0x4001136b]
/usr/local/lib/libtsutil.so.4(+0xe23d)[0x4001023d]
/usr/local/bin/traffic_server(TSVConnWrite+0x2c9)[0x80f4489]
/usr/local/libexec/trafficserver/metalink.so(+0x1d0f)[0x4004cd0f]
/usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x15f)[0x83087bf]
/usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x7a3)[0x83093c3]
/usr/local/bin/traffic_server[0x830830b]
/lib/libpthread.so.0[0x47948d8a]
/lib/libc.so.6(clone+0x5e)[0x47872a0e]
[TrafficServer] using root directory '/usr/local'
[TrafficServer] using root directory '/usr/local'
FATAL: InkAPI.cc:6248: failed assert sdk_sanity_check_iocore_structure(connp) == TS_SUCCESS
/usr/local/bin/traffic_server - STACK TRACE:
/usr/local/lib/libtsutil.so.4(+0xf36b)[0x4001136b]
/usr/local/lib/libtsutil.so.4(+0xe23d)[0x4001023d]
/usr/local/bin/traffic_server(TSVConnWrite+0x2c9)[0x80f4489]
/usr/local/libexec/trafficserver/metalink.so(+0x1d0f)[0x40047d0f]
/usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x15f)[0x83087bf]
/usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x7a3)[0x83093c3]
/usr/local/bin/traffic_server[0x830830b]
/lib/libpthread.so.0[0x47948d8a]
/lib/libc.so.6(clone+0x5e)[0x47872a0e]
[TrafficServer] using root directory '/usr/local'
FATAL: InkAPI.cc:6248: failed assert sdk_sanity_check_iocore_structure(connp) == TS_SUCCESS
/usr/local/bin/traffic_server - STACK TRACE:
/usr/local/lib/libtsutil.so.4(+0xf36b)[0x4001136b]
/usr/local/lib/libtsutil.so.4(+0xe23d)[0x4001023d]
/usr/local/bin/traffic_server(TSVConnWrite+0x2c9)[0x80f4489]
/usr/local/libexec/trafficserver/metalink.so(+0x1d0f)[0x40047d0f]
/usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x15f)[0x83087bf]
/usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x7a3)[0x83093c3]
/usr/local/bin/traffic_server(main+0xc6a)[0x80bc04a]
/lib/libc.so.6(__libc_start_main+0xf3)[0x47794b73]
/usr/local/bin/traffic_server[0x80c134d]
[TrafficServer] using root directory '/usr/local'

Degreane · 2014-02-07T21:49:43Z

try to compile it yourself and when you get it running and tested please post your procedure here so we may get it running ....

tested with ATS 4.1.2 and 5.0.x and still same error

what debug procedure you wish me to do so we can ease up the issue of fixing it up ?

kieronrob · 2014-05-09T19:28:47Z

Hi,

I get these errors when the plugin is enabled on ATS 4.2.1:

FATAL: InkAPI.cc:6261: failed assert sdk_sanity_check_iocore_structure(connp) == TS_SUCCESS
/usr/local/bin/traffic_server - STACK TRACE:
/usr/local/lib/libtsutil.so.4(+0x1f5d8)[0x2b82d6c3c5d8]
/usr/local/lib/libtsutil.so.4(+0x1da3f)[0x2b82d6c3aa3f]
/usr/local/bin/traffic_server(TSVConnWrite+0x11f)[0x4b507f]
/usr/local/libexec/trafficserver/metalink.so(+0x1ffc)[0x2b82de5f4ffc]
/usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8f)[0x6ab7cf]
/usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x61b)[0x6ac2fb]
/usr/local/bin/traffic_server(main+0xe1b)[0x4c511b]
/lib64/libc.so.6(__libc_start_main+0xfd)[0x3d2921ed1d]
/usr/local/bin/traffic_server[0x487e69]

Please let me know if you need any additional info.

jablko · 2014-05-09T20:26:06Z

Hi, are you able to recompile Traffic Server?
If so, can you please try the following,

download the latest metalink.cc code [1]
replace the plugins/experimental/metalink/metalink.cc in your Traffic Server source tree with the latest metalink.cc code
and recompile Traffic Server by running "make"

[1] https://raw.github.com/jablko/dedup/master/metalink.cc

After recompiling and restarting Traffic Server, do you still get these errors?

kieronrob · 2014-05-11T04:22:48Z

Hi Jack,

Recompiled ATS with the updated metalink.cc and the erors are no longer coming up.

Thank you for the swift reply and resolution as this is a feature I really think is valuable.

Regards,

Kieron

jablko · 2014-05-11T21:12:45Z

Great!
If you encounter any more problems, please let me know.
The latest metalink.cc code should be included in the next major Traffic Server release (5.X)

kieronrob · 2014-05-21T14:48:28Z

Hi Jack,

I have setup a new box with ATS and included the new metalink.cc in the
/plugins/experimental/metalink directory before running ./configure
--enable--experimental-plugins.

I am still getting the same error:

FATAL: InkAPI.cc:4534: failed assert sdk_sanity_check_txn(txnp) == TS_SUCCESS
/usr/local/bin/traffic_server - STACK TRACE:
/usr/local/lib/libtsutil.so.4(+0x1a9d4)[0x4001c9d4]
/usr/local/lib/libtsutil.so.4(+0x191dd)[0x4001b1dd]
/usr/local/bin/traffic_server(TSHttpTxnClientReqGet+0x32)[0x80df972]
/usr/local/libexec/trafficserver/metalink.so(+0x18d6)[0x400348d6]
/usr/local/bin/traffic_server(_ZN7CacheVC8callcontEi+0x5a)[0x821ac9a]
/usr/local/bin/traffic_server(_ZN7CacheVC19openWriteStartBeginEiP5Event+0x288)[0x8297b38]
/usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8c)[0x82dbd9c]
/usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x5ec)[0x82dc87c]
/usr/local/bin/traffic_server[0x82db17c]
/lib/libpthread.so.0[0xbd1b39]
/lib/libc.so.6(clone+0x5e)[0xb0dd8e]

Are there any other things I can try as I would like to make use of the
plugin.

I am also using the cacheurl plugin and can send you a copy of my
records.config if it will help.

Kind regards,

Kieron

On Sun, May 11, 2014 at 11:12 PM, Jack Bates [email protected]:

Great!
If you encounter any more problems, please let me know.
The latest metalink.cc code should be included in the next major Traffic
Server release (5.X)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1#issuecomment-42783451
.

Kieron Robertson

Man and Machine

Mail: [email protected]
Cell: 083 309 3538

jablko · 2014-05-23T16:52:42Z

Hmm, first I'm curious if this crash is specific to the version of Traffic Server or not.
If you can recompile Traffic Server, can you please reproduce the crash with the latest code from the Traffic Server repository?

The following commands will check out and build the latest code from the repository:

$ git clone https://git-wip-us.apache.org/repos/asf/trafficserver.git
$ cd trafficserver
$ autoreconf -i
$ ./configure --enable-experimental-plugins
$ make

You don't need to replace the metalink.cc file in this case because the repository already contains the latest metalink.cc code.

Next, can you please run Traffic Server with the "http" debug tag and send me the output when it crashes?
Run traffic_server with the "-Thttp" switch or run traffic_manager with the "-debug http" option.

Thanks for reporting this crash. With the extra details above, I will work on fixing it.
If you have any questions or problems, please let me know.

kieronrob · 2014-06-03T08:12:26Z

Hi Jack,

Sorry for the long delay, but been a bit hectic. I have recompiled and
reinstalled Traffic Server as per your suggestions above and am now running
it with the trafficserver start -Thttp switch.

I will come back to you if it crashes and give you the logs.

Regards,

Kieron

On Fri, May 23, 2014 at 6:52 PM, Jack Bates [email protected]
wrote:

Hmm, first I'm curious if this crash is specific to the version of Traffic
Server or not.
If you can recompile Traffic Server, can you please reproduce the crash
with the latest code from the Traffic Server repository?

The following commands will check out and build the latest code from the
repository:

$ git clone https://git-wip-us.apache.org/repos/asf/trafficserver.git
$ cd trafficserver
$ autoreconf -i
$ ./configure --enable-experimental-plugins
$ make

You don't need to replace the metalink.cc file in this case because the
repository already contains the latest metalink.cc code.

Next, can you please run Traffic Server with the "http" debug tag and send
me the output when it crashes?
Run traffic_server with the "-Thttp" switch or run traffic_manager with
the "-debug http" option.

Thanks for reporting this crash. With the extra details above, I will work
on fixing it.
If you have any questions or problems, please let me know.

—
Reply to this email directly or view it on GitHub
#1 (comment).

Kieron Robertson

Man and Machine

Mail: [email protected]
Cell: 083 309 3538

kieronrob · 2014-06-03T09:20:37Z

Hi,

OK - got a crash. Here is the output on the CLI:

[root@ats ~]# traffic_manager -debug http
[E. Mgmt] log ==> [TrafficManager] using root directory '/usr/local'
[Jun 3 10:14:50.803] Manager {0x7f4b7a9cb7e0} DEBUG: (http_tproxy) Listen
port 8080 inbound transparency enabled.
[TrafficServer] using root directory '/usr/local'
FATAL: InkAPI.cc:4535: failed assert sdk_sanity_check_txn(txnp) == TS_SUCCESS
/usr/local/bin/traffic_server - STACK TRACE:
/usr/local/lib/libtsutil.so.5(+0x1f5b8)[0x2b486fd7e5b8]
/usr/local/lib/libtsutil.so.5(+0x1da7f)[0x2b486fd7ca7f]
/usr/local/bin/traffic_server(TSHttpTxnClientReqGet+0x26)[0x4b6526]
/usr/local/libexec/trafficserver/metalink.so(+0x1fd2)[0x2b487c203fd2]
/usr/local/bin/traffic_server(_ZN7CacheVC8callcontEi+0x53)[0x669583]
/usr/local/bin/traffic_server(_ZN7CacheVC19openWriteStartBeginEiP5Event+0x2d8)[0x6e2f08]
/usr/local/bin/traffic_server(_ZN7EThread13process_eventEP5Eventi+0x8f)[0x7296bf]
/usr/local/bin/traffic_server(_ZN7EThread7executeEv+0x663)[0x72a233]
/usr/local/bin/traffic_server[0x728a6a]
/lib64/libpthread.so.0(+0x38a88079d1)[0x2b48705079d1]
/lib64/libc.so.6(clone+0x6d)[0x38a84e8b6d]

And the second SSH session I had open:

[connect] ERROR (main_socket_fd 3): Connection refused
traffic_top:
stats.h:251: void Stats::getStats(): Assertion `TSRecordGetInt(item.name,
&value) == TS_ERR_OKAY' failed.
Aborted (core dumped)
[root@ats ~]#

Manager Log:

[Jun 3 11:02:16.279] Manager {0x7f4b7a9cb7e0} FATAL:
[LocalManager::pollMgmtProcessServer] Error in read (errno: 104)
[Jun 3 11:02:16.279] Manager {0x7f4b7a9cb7e0} NOTE:
[LocalManager::mgmtShutdown] Executing shutdown request.
[Jun 3 11:02:16.279] Manager {0x7f4b7a9cb7e0} NOTE:
[LocalManager::processShutdown] Executing process shutdown request.
[Jun 3 11:02:16.280] Manager {0x7f4b7a9cb7e0} ERROR:
[LocalManager::sendMgmtMsgToProcesses] Error writing message
[Jun 3 11:02:16.280] Manager {0x7f4b7a9cb7e0} ERROR: (last system error
32: Broken pipe)

Please let me know if you need anything else.

Regards,

Kieron

On Tue, Jun 3, 2014 at 10:12 AM, Kieron Robertson [email protected]
wrote:

Hi Jack,

Sorry for the long delay, but been a bit hectic. I have recompiled and
reinstalled Traffic Server as per your suggestions above and am now running
it with the trafficserver start -Thttp switch.

I will come back to you if it crashes and give you the logs.

Regards,

Kieron

On Fri, May 23, 2014 at 6:52 PM, Jack Bates [email protected]
wrote:

Hmm, first I'm curious if this crash is specific to the version of
Traffic Server or not.
If you can recompile Traffic Server, can you please reproduce the crash
with the latest code from the Traffic Server repository?

The following commands will check out and build the latest code from the
repository:

$ git clone https://git-wip-us.apache.org/repos/asf/trafficserver.git
$ cd trafficserver
$ autoreconf -i
$ ./configure --enable-experimental-plugins
$ make

You don't need to replace the metalink.cc file in this case because the
repository already contains the latest metalink.cc code.

Next, can you please run Traffic Server with the "http" debug tag and
send me the output when it crashes?
Run traffic_server with the "-Thttp" switch or run traffic_manager with
the "-debug http" option.

Thanks for reporting this crash. With the extra details above, I will
work on fixing it.
If you have any questions or problems, please let me know.

—
Reply to this email directly or view it on GitHub
#1 (comment).

Kieron Robertson

Man and Machine

Mail: [email protected]
Cell: 083 309 3538

Kieron Robertson

Man and Machine

Mail: [email protected]
Cell: 083 309 3538

jablko · 2014-06-03T20:21:42Z

No sweat, thanks for reproducing the crash with the latest Traffic Server code.
I was wrong about how to turn on the "http" debug tag, sorry!
Can you please run either "trafficserver start" or traffic_manager with the following two environment variables and send me the output again?

PROXY_CONFIG_DIAGS_DEBUG_ENABLED=1 PROXY_CONFIG_DIAGS_DEBUG_TAGS=http

Thanks for your help tracking down this crash. If you have any questions or problems, please let me know.

kieronrob · 2014-06-13T07:56:53Z

Hi Jack,

Just an update - I haven't had another crash since the last one, so looking
good for the re-compiled ats from source.

I will use this method on another test appliance and give you feedback if I
see the crash again.

Kind regards,

Kieron

On Tue, Jun 3, 2014 at 10:21 PM, Jack Bates [email protected]
wrote:

No sweat, thanks for reproducing the crash with the latest Traffic Server
code.
I was wrong about how to turn on the "http" debug tag, sorry!
Can you please run either "trafficserver start" or traffic_manager with
the following two environment variables and send me the output again?

PROXY_CONFIG_DIAGS_DEBUG_ENABLED=1 PROXY_CONFIG_DIAGS_DEBUG_TAGS=http

Thanks for your help tracking down this crash. If you have any questions
or problems, please let me know.

—
Reply to this email directly or view it on GitHub
#1 (comment).

Kieron Robertson

Man and Machine

Mail: [email protected]
Cell: 083 309 3538

jablko · 2014-06-13T15:52:50Z

Thanks for keeping me updated,
do let me know if you experience any more crashes.
I'll work on fixing them if you do.

yzlai · 2014-10-28T09:05:18Z

I got same crash report.
It happened while calling TSHttpTxnClientReqGet() from cache_open_write() function.

-Ethan

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ATS crashes when Metalink Enabled in plugins.config #1

ATS crashes when Metalink Enabled in plugins.config #1

Degreane commented Jan 27, 2014

antbryan commented Feb 7, 2014

Degreane commented Feb 7, 2014

antbryan commented Feb 7, 2014

Degreane commented Feb 7, 2014

Degreane commented Feb 7, 2014

kieronrob commented May 9, 2014

jablko commented May 9, 2014

kieronrob commented May 11, 2014

jablko commented May 11, 2014

kieronrob commented May 21, 2014

jablko commented May 23, 2014

kieronrob commented Jun 3, 2014

kieronrob commented Jun 3, 2014

jablko commented Jun 3, 2014

kieronrob commented Jun 13, 2014

jablko commented Jun 13, 2014

yzlai commented Oct 28, 2014

ATS crashes when Metalink Enabled in plugins.config #1

ATS crashes when Metalink Enabled in plugins.config #1

Comments

Degreane commented Jan 27, 2014

antbryan commented Feb 7, 2014

Degreane commented Feb 7, 2014

antbryan commented Feb 7, 2014

Degreane commented Feb 7, 2014

Degreane commented Feb 7, 2014

kieronrob commented May 9, 2014

jablko commented May 9, 2014

kieronrob commented May 11, 2014

jablko commented May 11, 2014

kieronrob commented May 21, 2014

jablko commented May 23, 2014

kieronrob commented Jun 3, 2014

kieronrob commented Jun 3, 2014

jablko commented Jun 3, 2014

kieronrob commented Jun 13, 2014

jablko commented Jun 13, 2014

yzlai commented Oct 28, 2014