I/O improvements release

I/O Improvements Release
- snap pdf (phantomjs / headless chrome)
- write step to append new line to file
- table step to save table (no css selector)
- auto transpose datatable csv file
- support datatable cells with commas
- present() function to check element
- chrome extension note down element

Been holding back implementing some of these changes in order to avoid
attracting users. I’m happy with current user base, growing more and
faster will spread time thinly across much more edge cases needs.

Google just launched Puppeteer, I think most developers would be busy
exploring it than other web automation tool now, so I feel safe to add
these features. Outstanding - updating packaged installation,
publishing to npm and Chrome web store.

README.md

@@ -46,7 +46,7 @@ If you know JavaScript and want to be more expressive, you can even use JavaScri
 There is automatic waiting for web elements to appear + error-checking + nesting of JavaScript code blocks. Not forgetting the option to run automation flows hosted online or auto-upload run results online for sharing. TagUI also supports visual automation of website and desktop through built-in integration with Sikuli. Instead of using element identifiers, images can be used to identify user interface elements to interact with.
 
 # Set Up
-TagUI is in v2.3 and runs on macOS, Linux, Windows ([link to release notes](https://github.com/tebelorg/TagUI/releases))
+TagUI is in **process of updating to v2.4** and runs on macOS, Linux, Windows ([link to release notes](https://github.com/tebelorg/TagUI/releases))
 
 ### PACKAGED INSTALLATION
 Easiest way to use TagUI - no setup is needed, all dependencies are packaged in
@@ -156,7 +156,10 @@ show / print|element to read (page = webpage, ie raw html) |print element text t
 save|element (page = webpage) ***to*** optional filename|save element text to file
 echo|text (in quotation marks) and variables|print text/variables to output
 dump|text and variables ***to*** optional filename|save text/variables to file
+write|text and variables ***to*** optional filename|append text/variables to file
 snap|element (page = webpage) ***to*** optional filename|save screenshot to file
+snap (pdf)|page ***to*** filename.pdf (headless Chrome / PhantomJS)|save webpage to basic pdf
+table|element (non-CSS selector) ***to*** optional filename.csv|save basic html table to csv
 upload|element (CSS selector only) ***as*** filename to upload|upload file to website
 download|url to download ***to*** filename to save|download from url to file
 receive|url keyword to watch ***to*** filename to save|receive resource to file
@@ -198,7 +201,7 @@ less than or equal to / lesser than or equal to / lower than or equal to|<=
 and|&&
 or|&#124;&#124;
 
-Tip - use { and } step to define step/code blocks for powerful repetitive automation with for loop. conversion to CasperJS blocks syntax happens automatically. when using contain / equal, you can write with or without s behind. you can use if check_tx("element") to check if the element exists, before doing the step on next line
+Tip - use { and } step to define step/code blocks for powerful repetitive automation with for loop. conversion to CasperJS blocks syntax happens automatically. when using contain / equal, you can write with or without s behind. you can use if present('element') to check if the element exists, before doing the step on next line
 
 ### REPOSITORIES
 - Repositories help to make objects or steps reusable and improve readability
@@ -218,6 +221,7 @@ type email|type \`email\` as [email protected]
 - TagUI loops through each column to automate using values from different datasets
 - Eg, echo "TESTCASE - \`testname\`" in your flow shows TESTCASE - Trade USDSGD
 - Data-centric approach with rows representing data fields (usually row = test case)
+- To auto-transpose conventional datatable, save as flow filename plus _transpose.csv
 
 TEST TRADES|TEST #1|TEST #2|TEST #3
 :----------|:------|:------|:------
@@ -239,7 +243,7 @@ Automation flows can also be triggered via API URL. TagUI has an API service and
 0,15,30,45 * * * * /full_path_on_your_server/tagui_crontab
 ```
 
-To call an automation flow from your application or web browser, use below API syntax. Custom input(s) supported. Automation flows can also be triggered from emails using the API. For email integration, [install Tmail](https://github.com/tebelorg/Tmail). It's an open-source mailbot to act on incoming emails or perform mass emailing; it also delivers emails by API. Emails with run-time variables can be sent directly from your flow with a single line (see flow sample 6C_datatables).
+To call an automation flow from your application or web browser, use below API syntax. Custom input(s) supported. Automation flows can also be triggered from emails using the API. For email integration, [check out Tmail](https://github.com/tebelorg/Tmail). It's an open-source mailbot to act on incoming emails or perform mass emailing; it also delivers emails by API. Emails with run-time variables can be sent directly from your flow with a single line (see flow sample 6C_datatables). If you have data transformation in your process pipeline [check out TLE](https://github.com/tebelorg/TLE), which can help with converting data.
 ```
 your_website_url/tagui_service.php?SETTINGS="flow_filename option(s)"
 ```
@@ -256,15 +260,15 @@ For advanced API calls, you can set above api_config variable which defaults as
 ### CHROME
 TagUI has built-in integration with Chrome web browser to run web automation in visible or headless mode. It uses a websocket connection to directly communicate automation JavaScript code and information to Chrome.
 
-To develop new custom methods for Chrome integration, see this [TagUI issue](https://github.com/tebelorg/TagUI/issues/24#issuecomment-312361674) and [tagui_header.js](https://github.com/tebelorg/TagUI/blob/master/src/tagui_header.js) for examples of websocket calls from TagUI to Chrome (via Chrome Debugging Protocol). The function `chrome_step(method, params)` sends message to Chrome and returns the response. You will see examples from simple websocket calls such as getting webpage title to stacked ones such as handling of frame or popup window. To tweak how TagUI launches / kills Chrome and the integration PHP process, see TagUI runner script for [macOS/Linux](https://github.com/tebelorg/TagUI/blob/master/src/tagui) or [Windows](https://github.com/tebelorg/TagUI/blob/master/src/tagui.cmd). 
+To develop new custom methods for Chrome integration, see this [TagUI issue](https://github.com/tebelorg/TagUI/issues/24#issuecomment-312361674) and [tagui_header.js](https://github.com/tebelorg/TagUI/blob/master/src/tagui_header.js) for examples of websocket calls from TagUI to Chrome (via Chrome DevTools Protocol). The function `chrome_step(method, params)` sends message to Chrome and returns the response. You will see examples from simple websocket calls such as getting webpage title to stacked ones such as handling of frame or popup window. To tweak how TagUI launches / kills Chrome and the integration PHP process, see TagUI runner script for [macOS/Linux](https://github.com/tebelorg/TagUI/blob/master/src/tagui) or [Windows](https://github.com/tebelorg/TagUI/blob/master/src/tagui.cmd). 
 
-Probably the best way to see the websocket communication in action is to enter TagUI live mode (add live step in your automation flow), then `tail -f tagui_chrome.log` in another terminal to see the Chrome Debugging Protocol messages going to and fro as you enter TagUI steps or JavaScript code. If you are running on Windows, you can click on the PHP process window directly to see the messages.
+Probably the best way to see the websocket communication in action is to enter TagUI live mode (add live step in your automation flow), then `tail -f tagui_chrome.log` in another terminal to see the Chrome DevTools Protocol messages going to and fro as you enter TagUI steps or JavaScript code. If you are running on Windows, you can click on the PHP process window directly to see the messages.
 
 At run-time TagUI will start a PHP thread in the background to manage the integration with Chrome for concurrent communication. The [Textalk PHP websocket](https://github.com/Textalk/websocket-php) is used as it is super-light and most importantly, it works even without any update for 2 years. The normal approach to integrate with Chrome is through [chrome-remote-interface](https://github.com/cyrus-and/chrome-remote-interface) project or tools [such as Chromy](https://github.com/OnetapInc/chromy) which is based on chrome-remote-interface. However, that approach introduces Node.js dependency which means users without a Node.js development environment cannot run TagUI with Chrome. Outside of JavaScript ecosystem, there are also tools like [chromedp in Go language](https://github.com/knq/chromedp) to integrate with Chrome.
 
 In order to retain TagUI unzip and run functionality, the approach of launching a [separate PHP thread](https://github.com/tebelorg/TagUI/blob/master/src/tagui_chrome.php) is chosen. Since the TagUI [natural language interpreter](https://github.com/tebelorg/TagUI/blob/master/src/tagui_parse.php) is already written in PHP, there is no new dependency. Also, doing websocket communication within the single-threaded JavaScript environment used by CasperJS is not possible as it involves a redesign of fundamental CasperJS methods such as casper.exists to support async/await or use JavaScript promises which are not yet supported by CasperJS (for compatibility with [latest PhantomJS](https://github.com/casperjs/casperjs/issues/1663#issuecomment-285952446)).
 
-Like chrome-remote-interface, TagUI communicates with Chrome through [Chrome Debugging Protocol](https://chromedevtools.github.io/devtools-protocol/). The protocol is primarily designed for debugging the web browser instead of web automation, so many methods are still in experimental status. However, the API is stable enough for TagUI steps to work with Chrome. When chrome or headless option is used, TagUI replaces CasperJS methods it uses with custom methods to talk to Chrome instead of PhantomJS. When firefox option is used or by default, TagUI doesn't invoke custom methods and PHP process.
+Like chrome-remote-interface, TagUI communicates with Chrome through [Chrome DevTools Protocol](https://chromedevtools.github.io/devtools-protocol/). The protocol is primarily designed for debugging the web browser instead of web automation, so many methods are still in experimental status. However, the API is stable enough for TagUI steps to work with Chrome. When chrome or headless option is used, TagUI replaces CasperJS methods it uses with custom methods to talk to Chrome instead of PhantomJS. When firefox option is used or by default, TagUI doesn't invoke custom methods and PHP process.
 
 ### TESTING
 The step check allows simple testing of conditions. For professional test automation, CasperJS comes with a tester module for unit and functional testing purpose. To use advanced testing features, run TagUI with the test option. Note that CasperJS is not yet [supporting Chrome](https://github.com/casperjs/casperjs/issues/1825), below won't work when chrome or headless option is used.
@@ -313,6 +317,7 @@ erina.cmd|same as above but for Windows platform
 tagui_helper.php|command line natural language parser
 tagui_helper|generated normal TagUI command to run
 tagui_helper.cmd|same as above but for Windows platform
+transpose.php|transpose conventional datatable csv
 
 # Be a Force for Good
 TagUI default config does not hide identity as an automated user

package.json

@@ -1,6 +1,6 @@
 {
   "name": "tagui",
-  "version": "2.3.1",
+  "version": "2.4.0",
   "description": "General purpose tool for automating web interactions",
   "keywords": [
     "tagui"

src/chrome/background.js

@@ -24,7 +24,7 @@ chrome.runtime.onMessage.addListener(function(request, sender, sendResponse) {
   	    testcase_items = new Array();
   	    tab_id = request.recorded_tab;
   	    chrome.tabs.update(tab_id, {url: request.start_url}, function(tab) {
-          alert("\n - Click OK to start recording your actions\n\n - Right-click for shortcuts to TagUI steps\n\n");
+          alert("\n▹   click OK to start recording your actions\n▹   right-click for shortcuts to TagUI steps\n");
           chrome.tabs.sendMessage(tab_id, {action: "open", 'url': request.start_url});
           sendResponse({start: true});
   	    });

src/chrome/manifest.json

@@ -1,6 +1,6 @@
 {
     "manifest_version": 2,
-    "name": "TagUI Web Automation", "short_name": "TagUI Tool", "version": "2.3.0",
+    "name": "TagUI Web Automation", "short_name": "TagUI Tool", "version": "2.4.0",
     "description": "Create TagUI automation flows by recording your actions",
     "homepage_url": "https://github.com/tebelorg/TagUI",
     "browser_action": {

src/chrome/recorder.js

@@ -298,6 +298,7 @@ TestRecorder.EventTypes.ExplicitWait = 28;
 TestRecorder.EventTypes.FetchElementText = 29;
 TestRecorder.EventTypes.SelectElementOption = 30;
 TestRecorder.EventTypes.CancelLastStep = 31;
+TestRecorder.EventTypes.NoteDownElement = 32;
 TestRecorder.EventTypes.Cancel = 99;
 TestRecorder.EventTypes.MouseDown = 19;
 TestRecorder.EventTypes.MouseUp = 20;
@@ -477,6 +478,10 @@ TestRecorder.ElementScreenShotEvent = function() {
     this.type = TestRecorder.EventTypes.ElementScreenShot;
 }
 
+TestRecorder.NoteDownElement = function() {
+    this.type = TestRecorder.EventTypes.NoteDownElement;
+}
+
 TestRecorder.MoveCursorToElementEvent = function() {
     this.type = TestRecorder.EventTypes.MoveCursorToElement;
 }
@@ -617,16 +622,17 @@ TestRecorder.ContextMenu.prototype.build = function(t, x, y) {
 //        menu.appendChild(this.item("Screenshot", this.doScreenShot));
 //    }
 
-    menu.appendChild(this.item("Save webpage screenshot", this.doScreenShot));
-    menu.appendChild(this.item("Save element screenshot", this.doElementScreenShot));
+    menu.appendChild(this.item("Note down element", this.doNoteDownElement));
+    menu.appendChild(this.item("Move cursor to element", this.doMoveCursorToElement));
     menu.appendChild(this.item("Fetch element text", this.doFetchElementText));
     menu.appendChild(this.item("Print element text", this.doPrintElementText));
     menu.appendChild(this.item("Save element text", this.doSaveElementText));
-    menu.appendChild(this.item("Move cursor to element", this.doMoveCursorToElement));
+    menu.appendChild(this.item("Save webpage screenshot", this.doScreenShot));
+    menu.appendChild(this.item("Save element screenshot", this.doElementScreenShot));
     menu.appendChild(this.item("Wait for a few seconds", this.doExplicitWait));
-//    menu.appendChild(this.item("Cancel the last step", this.doCancelLastStep));
     menu.appendChild(this.item("Close shortcuts menu", this.cancel));
-
+//    menu.appendChild(this.item("Cancel the last step", this.doCancelLastStep));
+
     b.insertBefore(menu, b.firstChild);
     return menu;
 }
@@ -740,6 +746,13 @@ TestRecorder.ContextMenu.prototype.doElementScreenShot = function() {
     contextmenu.record(e);
 }
 
+TestRecorder.ContextMenu.prototype.doNoteDownElement = function() {
+    var t = contextmenu.target;
+    var et = TestRecorder.EventTypes;
+    var e = new TestRecorder.ElementEvent(et.NoteDownElement, t);
+    contextmenu.record(e);
+}
+
 TestRecorder.ContextMenu.prototype.doMoveCursorToElement = function() {
     var t = contextmenu.target;
     var et = TestRecorder.EventTypes;

src/chrome/style.css

@@ -3,7 +3,7 @@ body {
   font-size: 100%;
   text-decoration: none;
   color: black; 
-  background: #ffffff;
+  background: #FFFFFF;
   width: auto;
   margin: auto;
 }
@@ -13,11 +13,12 @@ body {
 }
 
 input[type=submit], input[type=button] {
-	border: 0;
-	color: #FFFFFF;
+	border: 3px solid #1E82C9;
+	color: #1E82C9;
 	padding: 1em 1em;
-	background: #1E82C9;
+	background: #FFFFFF;
 	font-size: 1em;
+	font-weight: bold;
 	border-radius: 0em;
 	-webkit-border-radius: 0em;
 	-moz-border-radius: 0em;

src/chrome/tagui.js

@@ -33,6 +33,7 @@ EventTypes.ExplicitWait = 28;
 EventTypes.FetchElementText = 29;
 EventTypes.SelectElementOption = 30;
 EventTypes.CancelLastStep = 31;
+EventTypes.NoteDownElement = 32;
 EventTypes.Cancel = 99;
 EventTypes.MouseDown = 19;
 EventTypes.MouseUp = 20;
@@ -114,6 +115,7 @@ d[EventTypes.CheckImageSrc] = "checkImageSrc";
 d[EventTypes.PageLoad] = "pageLoad";
 d[EventTypes.ScreenShot] = "screenShot";
 d[EventTypes.ElementScreenShot] = "elementScreenShot";
+d[EventTypes.NoteDownElement] = "noteDownElement";
 d[EventTypes.MoveCursorToElement] = "moveCursorToElement";
 d[EventTypes.PrintElementText] = "printElementText";
 d[EventTypes.SaveElementText] = "saveElementText";
@@ -372,6 +374,12 @@ CasperRenderer.prototype.elementScreenShot = function(item) {
   this.stmt('snap ' + selector);
 }
 
+CasperRenderer.prototype.noteDownElement = function(item) {
+  var selector; selector = this.getControl(item);
+  if (selector.charAt(0) == '#') {selector = selector.substring(1);}
+  this.stmt('// element is ' + selector);
+}
+
 CasperRenderer.prototype.moveCursorToElement = function(item) {
   var selector; selector = this.getControl(item);
   if (selector.charAt(0) == '#') {selector = selector.substring(1);}

src/tagui

@@ -6,7 +6,7 @@ chrome_command="google-chrome"
 
 if [ -z "$1" ]
 then
-echo "tagui v2.3: use following syntax and below options to run ./tagui flow_filename option(s)"
+echo "tagui v2.4: use following syntax and below options to run ./tagui flow_filename option(s)"
 echo
 echo "IMPORTANT: SAVE YOUR WORK BEFORE USING CHROME OR HEADLESS, TAGUI WILL RESTART CHROME"
 echo "headless - run on invisible Chrome web browser instead of default PhantomJS (first install Chrome)"
@@ -191,13 +191,17 @@ if [ -f "tagui_chrome.out" ]; then rm tagui_chrome.out; fi
 # default exit code 0 to mean no error parsing automation flow file
 tagui_error_code=0
 
+# transpose datatable csv file if file to be transposed exists
+if [ -f "$1_transpose".csv ]; then php -q transpose.php "$1_transpose".csv | tee -a "$1".log; fi
+
 # check datatable csv file for batch automation
 tagui_data_set_size=1
 if [ -f "$1".csv ]; then
 min_column=`awk -F',' '{print NF}' "$1".csv | sort -nu | head -n 1`
 max_column=`awk -F',' '{print NF}' "$1".csv | sort -nu | tail -n 1`
-if [ "$min_column" != "$max_column" ]; then echo "ERROR - $1.csv has inconsistent # of columns" | tee -a "$1".log
-elif [ "$min_column" -lt 2 ]; then echo "ERROR - $1.csv has lesser than 2 columns" | tee -a "$1".log
+# comment off sanity check for columns consistency as cells with , will trigger false-positive
+# if [ "$min_column" != "$max_column" ]; then echo "ERROR - $1.csv has inconsistent # of columns" | tee -a "$1".log
+if [ "$min_column" -lt 2 ]; then echo "ERROR - $1.csv has lesser than 2 columns" | tee -a "$1".log
 else tagui_data_set_size="$(($min_column-1))"; fi; fi
 
 # big loop for managing multiple data sets in datatable 

src/tagui.cmd

@@ -8,7 +8,7 @@ rem enable windows for loop advanced flow control
 setlocal enableextensions enabledelayedexpansion
 
 if "%~1"=="" (
-echo tagui v2.3: use following syntax and below options to run - tagui flow_filename option^(s^)
+echo tagui v2.4: use following syntax and below options to run - tagui flow_filename option^(s^)
 echo.
 echo IMPORTANT: SAVE YOUR WORK BEFORE USING CHROME OR HEADLESS, TAGUI WILL RESTART CHROME
 echo headless - run on invisible Chrome web browser instead of default PhantomJS ^(first install Chrome^)
@@ -424,15 +424,20 @@ if exist "tagui_chrome.out" del "tagui_chrome.out"
 rem default exit code 0 to mean no error parsing automation flow file
 set tagui_error_code=0
 
+rem transpose datatable csv file if file to be transposed exists
+if exist "%flow_file%_transpose.csv" php -q transpose.php "%flow_file%_transpose.csv" | tee -a "%flow_file%.log"
+
 rem check datatable csv file for batch automation
 set tagui_data_set_size=1 
 if not exist "%flow_file%.csv" goto no_datatable
 	for /f "tokens=* usebackq" %%c in (`gawk -F"," "{print NF}" "%flow_file%.csv" ^| sort -nu ^| head -n 1`) do set min_column=%%c
 	for /f "tokens=* usebackq" %%c in (`gawk -F"," "{print NF}" "%flow_file%.csv" ^| sort -nu ^| tail -n 1`) do set max_column=%%c
-
-	if %min_column% neq %max_column% (
-		echo ERROR - %flow_file%.csv has inconsistent # of columns | tee -a "%flow_file%.log"
-	) else if %min_column% lss 2 (
+
+rem comment off sanity check for columns consistency as cells with , will trigger false-positive
+rem	if %min_column% neq %max_column% (
+rem		echo ERROR - %flow_file%.csv has inconsistent # of columns | tee -a "%flow_file%.log"
+rem	)
+	if %min_column% lss 2 (
 		echo ERROR - %flow_file%.csv has has lesser than 2 columns | tee -a "%flow_file%.log"
 	) else (
 		set /a tagui_data_set_size=%min_column% - 1