Getting SyntaxError: Unterminated string literal error

[desmondtam]

Hi Andy,

Because of the 200 test cap per API key, I sometimes provision a series of tests and use the WPT-Bulk-Tester sheet to fetch the results by pasting the WPT results URL to the WPT ID column and running Get Results function. Recently, the sheet seems to be failing with the following error:

SyntaxError: Unterminated string literal

It correctly parses and fetches the results of other (older) WPT results though.

Here is one example WPT results ID where it fails:

https://www.webpagetest.org/result/161003_10_5327eec3e3713d1a1a3d75f233666a36/

Any idea why?

[andydavies]

Thanks for the report I'll take a look on the weekend

[andydavies]

The appscript command that parses the JSON is failing, but the JSON coming back from WPT looks valid so I need to dig in more.

[javiergarza]

I am getting the same errors. I believe the problem is the JSON returned is returned in chunks and the UrlFetchApp.fetch function may just "think" its done after receiving the first chunk and return the incomplete JSON body causing that error.

If you check the execution transcript you will see something like:

[17-07-28 18:26:29:043 PDT] Sheet.getRange([M1]) [0.053 seconds]
[17-07-28 18:26:29:044 PDT] Range.setValue([http://www.webpagetest.org/result/170728_S5_8ed3adcafa935c40f214e82b2bd6af0e/?f=json]) [0 seconds]
[17-07-28 18:26:30:385 PDT] UrlFetchApp.fetch([http://www.webpagetest.org/result/170728_S5_8ed3adcafa935c40f214e82b2bd6af0e/?f=json, {muteHttpExceptions=true}]) [1.34 seconds]
[17-07-28 18:26:30:398 PDT] HTTPResponse.getContentText() [0.012 seconds]
[17-07-28 18:26:31:003 PDT] Execution failed: SyntaxError: Unterminated string literal (line 186, file "webpagetest") [2.063 seconds total runtime]

Here is the code (marking in bold the code that triggers that error):
SpreadsheetApp.getActiveSheet().getRange('M1').setValue(wptAPI);
var response = UrlFetchApp.fetch(wptAPI);
var result = JSON.parse(response.getContentText());

I tried adding the optional parameter "muteHttpExceptions: true" to the UrlFetchApp.fetch call but didn't fix the issue
var response = UrlFetchApp.fetch(
wptAPI,
{
muteHttpExceptions: true,
}
);

Any ideas? (This error comes randomly on different rows, but once it triggers it does not go away)

[javiergarza]

I did some debugging (added the Logger line below) and I can confirm in the logs the JSON body is truncated at around 8KB (no idea why)

  var response = UrlFetchApp.fetch(wptAPI);
  Logger.log(response.getContentText());
  var result = JSON.parse(response.getContentText());

Logs:
[17-07-28 18:47:30:973 PDT] {"data":{"id":"170728_S5_8ed3adcafa935c40f214e82b2bd6af0e","url":"https://www.playstation.com/en-us/games/uncharted-the-lost-legacy-ps4/","summary":"http://www.webpagetest.org/results.php?test=170728_S5_8ed3adcafa935c40f214e82b2bd6af0e","testUrl":"https://www.playstation.com/en-us/games/uncharted-the-lost-legacy-ps4/","location":"ec2-us-west-1:Chrome","from":"California USA - EC2 - Chrome</b> - 3GFast</b>","connectivity":"3GFast","bwDown":1600,"bwUp":768,"latency":150,"plr":"0","mobile":0,"completed":1501283907,"tester":"i-72d75ac7","testerDNS":"172.31.0.2","runs":{"1":{"firstView":{"numSteps":1,"run":1,"tester":"i-72d75ac7","URL":"https://www.playstation.com/en-us/games/uncharted-the-lost-legacy-ps4/","loadTime":39820,"TTFB":1035,"bytesOut":422499,"bytesOutDoc":352961,"bytesIn":6387013,"bytesInDoc":5950643,"connections":87,"requests":[{"ip_addr":"23.34.148.155","method":"GET","host":"www.playstation.com","url":"/en-us/games/uncharted-the-lost-legacy-ps4/","responseCode":200,"load_ms":550,"ttfb_ms":360,"load_start":675,"bytesOut":347,"bytesIn":28640,"objectSize":28364,"cacheControl":"max-age=9526","contentType":"text/html","contentEncoding":"gzip","type":3,"socket":10,"score_cache":-1,"score_cdn":-1,"score_gzip":100,"score_cookies":-1,"score_keep-alive":100,"score_minify":-1,"score_combine":-1,"score_compress":-1,"score_etags":-1,"is_secure":1,"dns_ms":-1,"connect_ms":153,"ssl_ms":198,"gzip_total":28364,"gzip_save":0,"minify_total":0,"minify_save":0,"image_total":0,"image_save":0,"cache_time":-1,"dns_start":0,"dns_end":263,"connect_start":263,"connect_end":416,"ssl_start":417,"ssl_end":615,"server_count":1,"server_rtt":153,"client_port":61483,"jpeg_scan_count":0,"priority":"VeryHigh","request_id":1,"was_pushed":0,"initiator_type":"other","initiator_detail":"{"type":"other"}","protocol":"HTTP/2","http2_stream_id":1,"http2_stream_dependency":0,"http2_stream_weight":256,"http2_stream_exclusive":1,"certificate_bytes":4198,"objectSizeUncompressed":194759,"full_url":"https://www.playstation.com/en-us/games/uncharted-the-lost-legacy-ps4/","score_progressive_jpeg":-1,"load_end":1225,"ttfb_start":675,"ttfb_end":1035,"download_start":1035,"download_end":1225,"download_ms":190,"all_start":263,"all_end":1225,"all_ms":901,"headers":{"request":[":method: GET",":authority: www.playstation.com",":scheme: https",":path: /en-us/games/uncharted-the-lost-legacy-ps4/","upgrade-insecure-requests: 1","user-agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.78 Safari/537.36 PTST/387","accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8","accept-encoding: gzip, deflate, br","accept-language: en-US,en;q=0.8","pragma: akamai-x-cache-on, akamai-x-check-cacheable, akamai-x-get-true-cache-key, akamai-x-cache-remote-on"],"response":[":status: 200","server: Apache","content-type: text/html; charset=UTF-8","x-check-cacheable: YES","x-akamai-staging: ESSL","vary: Accept-Encoding","content-encoding: gzip","cache-control: max-age=9526","date: Fri, 28 Jul 2017 23:13:50 GMT","content-length: 28364","x-cache: TCP_HIT from a23-50-49-29.deploy.akamaitechnologies.com (AkamaiGHost/9.0.2.0.1-20438731) (-)","x-true-cache-key: /L/origin-us.playstation.com/en-us/games/uncharted-the-lost-legacy-ps4/"]},"index":0,"number":1},{"ip_addr":"23.34.148.155","method":"GET","host":"www.playstation.com","url":"/en-us/etc/designs/pdc/clientlibs_audiencemanager/js/sne_cloud_id.js","responseCode":200,"load_ms":246,"ttfb_ms":199,"load_start":1165,"bytesOut":124,"bytesIn":12138,"objectSize":11787,"expires":"Fri, 28 Jul 2017 23:43:50 GMT","cacheControl":"max-age=1800","contentType":"application/javascript","contentEncoding":"gzip","type":3,"socket":10,"score_cache":0,"score_cdn":0,"score_gzip":100,"score_cookies":-1,"score_keep-alive":100,"score_minify":-1,"score_combine":-1,"score_compress":-1,"score_etags":-1,"is_secure":1,"dns_ms":-1,"connect_ms":-1,"ssl_ms":-1,"gzip_total":11787,"gzip_save":0,"minify_total":0,"minify_save":0,"image_total":0,"image_save":0,"cache_time":1800,"dns_start":0,"dns_end":0,"connect_start":0,"connect_end":0,"ssl_start":0,"ssl_end":0,"initiator":"https://www.playstation.com/en-us/games/uncharted-the-lost-legacy-ps4/","initiator_line":7,"server_count":1,"server_rtt":153,"client_port":61483,"jpeg_scan_count":0,"priority":"High","request_id":2,"was_pushed":0,"initiator_type":"parser","initiator_detail":"{"lineNumber":7,"type":"parser","url":"https://www.playstation.com/en-us/games/uncharted-the-lost-legacy-ps4/"}","protocol":"HTTP/2","http2_stream_id":3,"http2_stream_dependency":1,"http2_stream_weight":220,"http2_stream_exclusive":1,"certificate_bytes":0,"objectSizeUncompressed":29970,"full_url":"https://www.playstation.com/en-us/etc/designs/pdc/clientlibs_audiencemanager/js/sne_cloud_id.js","score_progressive_jpeg":-1,"load_end":1411,"ttfb_start":1165,"ttfb_end":1364,"download_start":1364,"download_end":1411,"download_ms":47,"all_start":1165,"all_end":1411,"all_ms":246,"headers":{"request":[":method: GET",":authority: www.playstation.com",":scheme: https",":path: /en-us/etc/designs/pdc/clientlibs_audiencemanager/js/sne_cloud_id.js","user-agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.78 Safari/537.36 PTST/387","accept: /","referer: https://www.playstation.com/en-us/games/uncharted-the-lost-legacy-ps4/","accept-encoding: gzip, deflate, br","accept-language: en-US,en;q=0.8","pragma: akamai-x-cache-on, akamai-x-check-cacheable, akamai-x-get-true-cache-key, akamai-x-cache-remote-on"],"response":[":status: 200","accept-ranges: bytes","content-type: application/javascript","last-modified: Fri, 21 Jul 2017 19:30:42 GMT","server: Apache","vary: Accept-Encoding","content-encoding: gzip","cache-control: max-age=1800","expires: Fri, 28 Jul 2017 23:43:50 GMT","date: Fri, 28 Jul 2017 23:13:50 GMT","content-length: 11787","x-cache: TCP_MEM_HIT from a23-50-49-29.deploy.akamaitechnologies.com (AkamaiGHost/9.0.2.0.1-20438731) (-)","x-true-cache-key: /L/origin-us.playstation.com/en-us/etc/designs/pdc/clientlibs_audiencemanager/js/sne_cloud_id.js","x-akamai-staging: ESSL","x-check-cacheable: YES"]},"index":1,"number":2},{"ip_addr":"23.34.148.155","method":"GET","host":"www.playstation.com","url":"/en-us/etc/clientlibs/granite/jquery.min.js","responseCode":200,"load_ms":1232,"ttfb_ms":309,"load_start":1168,"bytesOut":52,"bytesIn":38183,"objectSize":37851,"expires":"Fri, 28 Jul 2017 23:43:50 GMT","cacheControl":"max-age=1800","contentType":"application/javascript","contentEncoding":"gzip","type":3,"socket":10,"score_cache":0,"score_cdn":0,"score_gzip":100,"score_cookies":-1,"score_keep-alive":100,"score_minify":-1,"score_combine":-1,"score_compress":-1,"score_etags":-1,"is_secure":1,"dns_ms":-1,"connect_ms":-1,"ssl_ms":-1,"gzip_total":37851,"gzip_save":0,"minify_total":0,"minify_save":0,"image_total":0,"image_save":0,"cache_time":1800,"dns_start":0,"dns_end":0,"connect_start":0,"connect_end":0,"ssl_start":0,"ssl_end":0,"initiator":"https://www.playstation.com/en-us/games/uncharted-the-lost-legacy-ps4/","initiator_line":165,"server_count":1,"server_rtt":153,"client_port":61483,"jpeg_scan_count":0,"priority":"High","request_id":3,"was_pushed":0,"initiator_type":"parser","initiator_detail":"{"lineNumber":165,"type":"parser","url":"https://www.playstation.com/en-us/games/uncharted-the-lost-legacy-ps4/"}","protocol":"HTTP/2","http2_stream_id":5,"http2_stream_dependency":3,"http2_stream_weight":220,"http2_stream_exclusive":1,"certificate_bytes":0,"objectSizeUncompressed":110890,"full_url":"https://www.playstation.com/en-us/etc/clientlibs/granite/jquery.min.js","score_progressive_jpeg":-1,"load_end":2400,"ttfb_start":1168,"ttfb_end":1477,"download_start":1477,"download_end":2400,"download_ms":923,"all_start":1168,"all_end":2400,"all_ms":1232,"headers":{"request":[":method: GET",":authority: www.playstation.com",":scheme: https",":path: /en-us/etc/clientlibs/granite/jquery.min.js","use

[andydavies]

@javiergarza Thanks for digging into this - I'll try to take a look at it over the next week or so

[JasonCoe-FL]

I'm curious if this issue has been looked into any further as I'm running into the same thing frequently but only on certain tests. As @javiergarza stated, the response is limited to 8kb which sticks out to me only because I know it's a common practice by CDN's, servers, and hosting providers to limit the response headers to a max of 8kb. Not sure if it's related but I'm curious if there's a correlation.