xhgui: document to insert contains invalid key: keys cannot contain "."

[SimonDeconde]

Hi there

I'm trying to install xhprof +xhgui, but something is not working with mongoDB. After a full setup, when I browse a page on my project, I'm getting errors in apache logs saying:

xhgui - document to insert contains invalid key: keys cannot contain ".": "main()==>load::myprojecttoprofile/index.php"

(I haven't been able to find any similar ticket issues for xhgui on this specific topic)

I'm using:

• mongoDB 2.4.9
• the latest xhgui git code (master branch, but I've also tried release 0.8.1 and 0.8.0)
• xhprof 0.9.2
• Ubuntu Ubuntu 14.04.5 LTS (virtual machine)

Is there something wrong with my setup?

Thanks!

[felixkk]

Same error on a debian server...
In the requirements, it says we should use "Mongodb 2.2.0 or later".
Maybe 2.4.9 is too late ?

[markstory]

Sounds like at some point mongo has changed what is valid in keys. I'm still using an old mongo (2.2.4). If . isn't valid in new mongo, we'll likely need to rebuild xhgui 😢

[martins-olekss]

Ran in same issue just now. Everywhere I search, only workaround is to modify keys before saving - in this specific case doesn't sound too possible.

Using mongo 2.6.10

As far I have found there is no setting to turn this key check off.

[lauripiisang]

This is odd. I'm using MongoDB 3.0.14 and have no such issue.

Do note that top-level keys probably cannot contain such a profile, this must be stored in a lower level key!

e.g. you cannot have

{
  "a.b" : {...}
}

but

{
  "a" : {
    "a.b" : { ... }
  }

is valid, at least from what I understand and what I see in my database.

I encountered a similar error and it turned out I was creating the profile object wrong.

[lauripiisang]

Can you please post the full document at the time of insert (use debugger?)

[martins-olekss]

@lauripiisang do I understand correctly, you wish to see $data content just before
$saver->save($data); in external/header.php ?

Just as side-note - noticed that this exception happens only when I use auto_append in server (nginx) config. If I disable auto_append and try to encompass code with xhprof_enable and xhprof_disable functions - exception doesn't happen anymore. Even more - new records show up in xhgui interface. But records are without any measurements and if I go inside each request - theres xhgui error

#0 /var/www/xhgui/src/Xhgui/Profile.php(359): Slim\Slim::handleErrors(2, 'Invalid argumen...', '/var/www/xhgui/...', 359, Array)
#1 /var/www/xhgui/src/Xhgui/Controller/Run.php(69): Xhgui_Profile->calculateSelf()
#2 /var/www/xhgui/src/routes.php(30): Xhgui_Controller_Run->view()
#3 [internal function]: {closure}()
#4 /var/www/xhgui/vendor/slim/slim/Slim/Route.php(468): call_user_func_array(Object(Closure), Array)
#5 /var/www/xhgui/vendor/slim/slim/Slim/Slim.php(1355): Slim\Route->dispatch()
#6 /var/www/xhgui/vendor/slim/slim/Slim/Middleware/Flash.php(85): Slim\Slim->call()
#7 /var/www/xhgui/vendor/slim/slim/Slim/Middleware/MethodOverride.php(92): Slim\Middleware\Flash->call()
#8 /var/www/xhgui/vendor/slim/slim/Slim/Middleware/SessionCookie.php(110): Slim\Middleware\MethodOverride->call()
#9 /var/www/xhgui/src/Xhgui/Middleware/Render.php(11): Slim\Middleware\SessionCookie->call()
#10 /var/www/xhgui/vendor/slim/slim/Slim/Slim.php(1300): Xhgui_Middleware_Render->call()
#11 /var/www/xhgui/webroot/index.php(10): Slim\Slim->run()
#12 {main}

[lauripiisang]

do I understand correctly, you wish to see $data content just before
$saver->save($data); in external/header.php ?

That would do, but I'd prefer the document submitted to mongo (deeper down the call stack).

Just as side-note - noticed that this exception happens only when I use auto_append in server (nginx) config.

With header.php, you're supposed to be using auto_prepend, not append!

Can I ask you to try out https://github.com/perftools/xhgui-collector in your project and see if you still encounter problems?

I use the register_shutdown_function like provided in header.php to register a shutdown function which does:

$data = array('profile' => xhprof_disable());
// some filtering to prevent leakage of user passwords via server_http_password etc 
// filling $data['meta'] like in header.php
// $saver is instance of Xhgui_Saver, constructed beforehand.
$saver->save($data);

I think what's different here is that you don't have the xhprof_disable call under profile key.

[lauripiisang]

I think I can get the profiling library I developed open-sourced so you can have an easier time at using this.

[emilycraps]

I have the same issue. This is the output for the $data variable that will be saved to Mongo:

Array
(
    [profile] => Array
        (
            [main()==>load::shop/test.php] => Array
                (
                    [ct] => 1
                    [wt] => 3126
                    [cpu] => 1000
                    [mu] => 3320
                    [pmu] => 0
                )

            [main()==>???_op] => Array
                (
                    [ct] => 1
                    [wt] => 300
                    [cpu] => 0
                    [mu] => 904
                    [pmu] => 0
                )

            [main()==>{closure}] => Array
                (
                    [ct] => 1
                    [wt] => 167
                    [cpu] => 0
                    [mu] => 808
                    [pmu] => 0
                )

            [main()] => Array
                (
                    [ct] => 1
                    [wt] => 3866
                    [cpu] => 2000
                    [mu] => 5032
                    [pmu] => 0
                )

        )

    [meta] => Array
        (
            [url] => /shop/test.php
            [SERVER] => Array
                (
                    [SCRIPT_URL] => /shop/test.php
                    [SCRIPT_URI] => http://esales.dev/shop/test.php
                    [REMOTE_PORT] => 51476
                    [DATARETENTIONXFP] => 
                    [DATARETENTIONXFF] => 
                    [teleEnv] => vagrant
                    [HTTP_HOST] => esales.dev
                    [HTTP_CONNECTION] => keep-alive
                    [HTTP_CACHE_CONTROL] => max-age=0
                    [HTTP_USER_AGENT] => Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36
                    [HTTP_UPGRADE_INSECURE_REQUESTS] => 1
                    [HTTP_ACCEPT] => text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
                    [HTTP_ACCEPT_ENCODING] => gzip, deflate
                    [HTTP_ACCEPT_LANGUAGE] => en-GB,en-US;q=0.8,en;q=0.6
                    [HTTP_COOKIE] => XDEBUG_SESSION=debug;
                    [HTTP_X_DATARETENTION_SOURCEPORT] => 51476
                    [HTTP_X_DATARETENTION_XFP] => 
                    [HTTP_X_DATARETENTION_XFF] => 
                    [PATH] => /sbin:/usr/sbin:/bin:/usr/bin
                    [SERVER_SIGNATURE] => 
                    [SERVER_SOFTWARE] => Apache/2.2.15 (CentOS)
                    [SERVER_NAME] => esales.dev
                    [SERVER_ADDR] => 192.168.50.15
                    [SERVER_PORT] => 80
                    [REMOTE_ADDR] => 192.168.50.1
                    [DOCUMENT_ROOT] => /data/www/vhosts/esales.dev/htdocs/html
                    [SERVER_ADMIN] => root@localhost
                    [SCRIPT_FILENAME] => /data/www/vhosts/esales.dev/htdocs/html/shop/test.php
                    [GATEWAY_INTERFACE] => CGI/1.1
                    [SERVER_PROTOCOL] => HTTP/1.1
                    [REQUEST_METHOD] => GET
                    [QUERY_STRING] => 
                    [REQUEST_URI] => /shop/test.php
                    [SCRIPT_NAME] => /shop/test.php
                    [PHP_SELF] => /shop/test.php
                    [REQUEST_TIME_FLOAT] => 1507277429.582
                    [REQUEST_TIME] => 1507277429
                )

            [get] => Array
                (
                )

            [env] => Array
                (
                )

            [simple_url] => /shop/test.php
            [request_ts] => MongoDate Object
                (
                    [sec] => 1507277429
                    [usec] => 0
                )

            [request_ts_micro] => MongoDate Object
                (
                    [sec] => 1507277429
                    [usec] => 0
                )

            [request_date] => 2017-10-06
        )

)

Mongo version: v2.4.14
XHProf version: 0.9.4
PHP version: 5.6.16

[samt2497]

Had the issue, fixed it by encoding the stored profile.
It might not be good for performance but helps anyone having this issue currently.

#210

Current environment
Php 7.1.8
mongo extension: master
mongodb 3.4.4

[markstory]

Thanks @samt2497 Closing as there is a pull request open now.

[lauripiisang]

Well, I found the reason why it worked for some and didn't work for others:

https://jira.mongodb.org/browse/PHPC-712 seems to have been a bug in the adapters causing the ignorance of said dot-check. That is to say, the documents could still be inserted, just not updated afaik.
The newer versions of the adapters fix this by not allowing keys with dots.

While @samt2497 -s provided unicode-based encoding-decoding solution is a band-aid for now, I think a long term solution should include solving the problem by restructuring the data.

I propose moving from

{
  "profiles": { 
    "main()==>load::shop/test.php" : {
      "ct" : 1
    }
  }

to

{
  "profiles": { 
    "some-uuid-here-1234" : {
      "stackName" : "main()==>load::shop/test.php",
      "ct" : 1
    }
  }

This preserves the possibility to view a sub-key on its own and avoids the dots-in-keys problem.

Suggestions welcome for stackName

[lauripiisang]

Reopening issue to get some opinions on the data restructure solution.

[markstory]

If we have to restructure the data should we also consider moving away from mongodb entirely? A few features like multi domain support and ad hoc queries are more complex because of mongo. Furthermore the awkward datetime handling and limited set of aggregation functions limit our ability to offer dashboards and summary data.

[felixkk]

@markstory
That would be good news !
Everything around me is using MySQL, having to install mongodb is a bit painful when you don't know.

The sysadmin finally managed to get around the bug and install xhgui, and it looks excellent.
If it could just have a more "friendlier" install phase, it would be a new must have for us.

[michaelbutler]

For what it's worth, I have a private fork of XHGUI that uses MySQL. I'm storing the data in MySQL in a separate project, and XHGUI is just reading off that table. To do this I tried to abstract the data storage away as much as possible, so one could "switch" to a different storage driver using the config file. Not everything is supported with this MySQL abstraction though, such as some of the aggregation, searching & filtering tools. But it works great to just load up individual profiles.

Overall, I think having XHGUI be built in a way that developers can implement their own storage drivers (memcached, redis, S3, etc.) would be great.

I also added an authentication system. That might be best kept as a separate PR though.

If I get some time I can try to clean up & push my changes back to XHGUI. I would love for the project to be converted to full PSR-4 and PSR-2 coding standard as well just to keep consistent and have better class names.

[dhampik]

For anyone experiencing this issue, I was able to make it work by adding the following snippet of code to xhgui/external/header.php right before ignore_user_abort(true); call. Not sure how correct is this and if it could affect anything else, but it did the trick for me.

        $profile = [];
        foreach($data['profile'] as $key => $value) {
            $profile[strtr($key, ['.' => '_'])] = $value;
        }
        $data['profile'] = $profile;

[bogdan-dubyk]

@dhampik is your fix require some more manipulations? It showing all available profilings list as well, but when i click on some of them I'm getting error:

#0 /var/www/xhgui/src/Xhgui/Profile.php(361): Slim\Slim::handleErrors(8, 'Undefined index...', '/var/www/xhgui/...', 361, Array)
#1 /var/www/xhgui/src/Xhgui/Controller/Run.php(69): Xhgui_Profile->calculateSelf()
#2 /var/www/xhgui/src/routes.php(30): Xhgui_Controller_Run->view()
#3 [internal function]: {closure}()
#4 /var/www/xhgui/vendor/slim/slim/Slim/Route.php(468): call_user_func_array(Object(Closure), Array)
#5 /var/www/xhgui/vendor/slim/slim/Slim/Slim.php(1355): Slim\Route->dispatch()
#6 /var/www/xhgui/vendor/slim/slim/Slim/Middleware/Flash.php(85): Slim\Slim->call()
#7 /var/www/xhgui/vendor/slim/slim/Slim/Middleware/MethodOverride.php(92): Slim\Middleware\Flash->call()
#8 /var/www/xhgui/vendor/slim/slim/Slim/Middleware/SessionCookie.php(110): Slim\Middleware\MethodOverride->call()
#9 /var/www/xhgui/src/Xhgui/Middleware/Render.php(11): Slim\Middleware\SessionCookie->call()
#10 /var/www/xhgui/vendor/slim/slim/Slim/Slim.php(1300): Xhgui_Middleware_Render->call()
#11 /var/www/xhgui/webroot/index.php(11): Slim\Slim->run()
#12 {main}

[dhampik]

@bogdan-dubyk no, only the change that I wrote.

[tm1000]

@dhampik thank you. Worked for me.

[apuppy]

For anyone experiencing this issue, I was able to make it work by adding the following snippet of code to xhgui/external/header.php right before ignore_user_abort(true); call. Not sure how correct is this and if it could affect anything else, but it did the trick for me.

        $profile = [];
        foreach($data['profile'] as $key => $value) {
            $profile[strtr($key, ['.' => '_'])] = $value;
        }
        $data['profile'] = $profile;

It worked for me, thank you.

[matteo-gz]

For anyone experiencing this issue, I was able to make it work by adding the following snippet of code to xhgui/external/header.php right before ignore_user_abort(true); call. Not sure how correct is this and if it could affect anything else, but it did the trick for me.

        $profile = [];
        foreach($data['profile'] as $key => $value) {
            $profile[strtr($key, ['.' => '_'])] = $value;
        }
        $data['profile'] = $profile;

It worked for me, thank you.

this way is so cool

[glensc]

Now Ive encountered this myself as well. with ZF1 the autoloader entries look like this:

    "Zend_Controller_Dispatcher_Standard::loadClass==>run_init::controllers\/ArticleController.php": {
      "ct": 1,
      "wt": 4230,
      "cpu": 6667,
      "mu": 843328,
      "pmu": 436912
    },

[glensc]

For everybody there seems to be "load::" in call stack, I also have "run_init::" prefixes.

I wonder it's xhprof quirk or something?

My data is from PHP 5.3 and xhprof 0.9.4

[glensc]

indeed, it's present in xhprof 0.9.4 extension.

these code paths add filenames to call tree, and filenames obviously can contain the dot:

• run_init: https://github.com/phacility/xhprof/blob/0bbf2a2ac34f495e42aa852293fe0ed821659047/extension/xhprof.c#L1016
• ???_op: https://github.com/phacility/xhprof/blob/0bbf2a2ac34f495e42aa852293fe0ed821659047/extension/xhprof.c#L1002
• load: https://github.com/phacility/xhprof/blob/0bbf2a2ac34f495e42aa852293fe0ed821659047/extension/xhprof.c#L1792

later removed for 2.x (new maintainer):

• longxinH/xhprof@d341e22

however "load::" is still present:

• https://github.com/longxinH/xhprof/blob/v2.2.2-beta/extension/xhprof.c#L1213

[glensc]

Re: #209 (comment)

@michaelbutler still there?

have you considered sending PRs of your changes? over the time this project has got pdo support, but it's not complete either:

• Equalize data store backends #320

[michaelbutler]

@glensc I haven't updated the project in quite a while but if I get a chance I'll try the latest version with PDO and see if I have anything to add