Skip to content

Commit

Permalink
Release 0.8.0
Browse files Browse the repository at this point in the history
Added in more help information at top of cron-shim.sh script as well as offered outputting to stdout.
Fixed requriement for bc and only if available otherwise use awk
Updated GridPane Nginx log format, missed request time which broke everything after words which included OS and referrer.
Nuked git commit history as rebase would have taken ages.
jordantrizz committed May 26, 2023
0 parents commit 3d7b9ae
Showing 40 changed files with 5,511 additions and 0 deletions.
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
.development
.gptoken
.debug
.test
103 changes: 103 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
<!--ts-->
Table of Contents
=================

* [Table of Contents](#table-of-contents)
* [wp-shelltools](#wp-shelltools)
* [Rename from gp-tools to wp-shelltools](#rename-from-gp-tools-to-wp-shelltools)
* [Donate](#donate)
* [Install](#install)
* [Requirements](#requirements)
* [Help](#help)
* [Help Command](#help-command)
* [Advanced Documentation](#advanced-documentation)
* [<a href="docs/gp-goaccess.md">gp-goaccess</a>](#gp-goaccess)
* [<a href="docs/gp-api.md">gp-api</a>](#gp-api)
* [<a href="docs/gp-plugins.md">gp-plugins</a>](#gp-plugins)
* [<a href="docs/attackscan.md">attackscan.sh</a>](#attackscansh)
* [Debugging](#debugging)
* [Future Features](#future-features)
* [Donate](#donate-1)
* [Resources](#resources)
<!--te-->

# wp-shelltools
The goal of this project is to make it easy to manage and maintain WordPress servers using shell scripting. This repository was original named gp-tools, but decided to make it provider agnostic

You might find that there are still tools and functions that have gridpane or gp in their name. This will change when it becomes a priority.

# Donate
[!["Buy Me A Coffee"](https://www.buymeacoffee.com/assets/img/custom_images/orange_img.png)](https://wpinfo.net/sponsor/)

# Install
## Requirements
* You need root, otherwise this makes no sense.
* You need git (apt-get install git)

It's simple, make sure you have git installed (apt-get install git) and then run the following
```
git clone https://github.com/jordantrizz/gpcli.git ~/gpcli;~/gpcli/gpcli -c install
```

# Help
Simply run the following command to get a list of available tools
```
wp-shelltools help
```
Here's a list of all commands
```
-- Loading wp-shelltools - v0.0.1
wp-shelltools help
-----------------------------------
help goaccess - Process GridPane logs with goaccess
help backups - List backups for all sites on the server.
help plugins - Lists WordPress plugins on all websites on a GridPane Server
help logcode - Look for specifc HTTP codes in web server logfiles and return top hits.
help gpcron - List sites using GP Cron
help mysqlmem - GridPane monit memory calculation
help logs - tail or show last lines on all GridPane logs.
Examples:
--
wp-shelltools goaccess
wp-shelltools log
```

# Directories
* dawn-mover - GridPane's prime-mover recoded but broken, it's a mess.
* docs - Advanced documentation.
* tests - any test data.
* unfinished - code that is untested or needs to be moved into core.

# Advanced Documentation
Below is help for the advanced tools.
* [wpst-goaccess](docs/wpst-goaccess.md)
* [gp-api](docs/gp-api.md)
* [gp-plugins](docs/gp-plugins.md)
* [attackscan.sh](docs/attackscan.md)
* [dawn-mover](dawn-mover/README.md)

# Debugging
Add a .debug file to any directory to get debug information.

# Future Features
* Check Log Sizes - debug.log is a culprit in disk usage. Find all occurances and print out size.
* MySQL Database Checks - Check for anomiles, myisam, large table sizes, large databases.
* MySQL Database Report - Largest Databases + more.
* CLI to API - A CLI to the GridPane API.

# ToDo
* Go through unfinished directory code.
* Better documentation
* Place some scripts into functions or separate files not inside root for a cleaner directory structure and code management.

# Donate
[!["Buy Me A Coffee"](https://www.buymeacoffee.com/assets/img/custom_images/orange_img.png)](https://managingwp.io/sponsor/)

# Resources
* [Managing WP](https://mangingwp.io) - WordPress Articles and more!
* [Managing WordPress](https://www.facebook.com/groups/managingwordpress) - Created and managed by me.
* [Managing WordPress Discord](https://discord.gg/QCsHM234zh) - Come and chat about WordPress
* [GridPane Facebook Group](https://www.facebook.com/groups/selfmanagedwordpress) - Managed by GridPane and full of customers and WordPress community members.
1 change: 1 addition & 0 deletions VERSION
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
0.8.0
118 changes: 118 additions & 0 deletions attackscan.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
#!/bin/bash
# -- Variables
VERSION="0.0.3"
SCRIPT="attackscan.sh"
OLS_LOG_DIR="/var/log/ols"
NGINX_LOG_DIR="/var/log/nginx"

# -- _error
_error () {
echo "## ERROR $@"
}

# -- usage
usage () {
echo "$SCRIPT [-top <lines>|-scan]"
echo "Version: $VERSION"
echo ""
echo "Parses Nginx and OLS web server access logs to find top number of requests and common attack requests for WordPress."
echo ""
echo "Commands:"
echo " -top -List top number of requests from the webserver access log."
echo " -scan -List common attack requests that return a 200 status code, by IP address."
echo ""
echo "Options:"
echo " <lines> -How many lines to show, if not specified defaults to 10"
echo ""
echo "Examples"
echo " $SCRIPT -top 20"
echo " $SCRIPT -scan"
echo ""
}

# -- top-ols $SERVER $LOGS $LINES
top () {
SERVER=$1
LOGS=$2
LINES=$3
echo "Server: $SERVER - Logs: $LOGS - Lines: $LINES"
if [[ $SERVER == "ols" ]]; then
LOG_FILES=($(ls ${LOGS}/*.access.log))
for SITE in "${LOG_FILES[@]}"; do
echo "** Parsing ${SITE} for top ${LINES} requests"
cat ${SITE} | awk {' print $7 '} | sort | uniq -c | sort -nr | head -n ${LINES}
echo "======================"
done
elif [[ $SERVER == "nginx" ]]; then
LOG_FILES=($(ls ${LOGS}/*.access.log))
for SITE in "${LOG_FILES[@]}"; do
echo "** Parsing ${SITE} for top ${LINES} requests"
cat ${SITE} | awk {' print $8 '} | sort | uniq -c | sort -nr | head -n ${LINES}
echo "======================"
done
else
_error "Something went wrong"
fi
}

scan () {
SERVER=$1
LOGS=$2
LINES=$3

echo "Server: $SERVER - Logs: $LOGS - Lines: $LINES"
if [[ $SERVER == "ols" ]]; then
LOG_FILES=($(ls ${LOGS}/*.access.log))
for SITE in "${LOG_FILES[@]}"; do
echo "** Parsing ${SITE}"
cat ${SITE} | awk {' print $7 "," $1 "," $9 '} | sed 's/"//' | grep -v "redirect_to" | grep "200" | egrep -e "xmlrpc|wp-login" | sort | uniq -c | sort -nr | head -n ${LINES}
echo "======================"
done
elif [[ $SERVER == "nginx" ]]; then
LOG_FILES=($(ls ${LOGS}/*.access.log))
for SITE in "${LOG_FILES[@]}"; do
echo "** Parsing ${SITE}"
cat ${SITE} | awk {' print $3 "," $8 "," $10 '} | sed 's/"//' | grep -v "redirect_to" | grep "200" | egrep -e "xmlrpc|wp-login" | sort | uniq -c | sort -nr | head -n ${LINES}
echo "======================"
done
fi
}

# -- main
if [[ -z $1 ]]; then
usage
exit 1
fi

# -- logs to process
if [[ -d $OLS_LOG_DIR ]]; then
echo " -- Found OLS logs at /var/log/ols"
SERVER="ols"
LOGS="/var/log/ols"
elif [[ -d /var/log/nginx ]]; then
echo " -- Found Nginx logs at /var/log/nginx"
SERVER="nginx"
LOGS="/var/log/nginx"
else
_error "Didn't find any logs"
exit 1
fi

# -- lines
LINES=""
if [[ -n $2 ]]; then
LINES="$2"
else
LINES="10"
fi

# -- top command
if [[ $1 == "-top" ]]; then
echo " -- Showing top $LINES requests per log file"
top $SERVER $LOGS $LINES
# -- scan command
elif [[ $1 == "-scan" ]]; then
echo " - Running a scan for common attack requests"
scan $SERVER $LOGS $LINES
fi

1 change: 1 addition & 0 deletions dawn-mover
Submodule dawn-mover added at 269ee6
95 changes: 95 additions & 0 deletions docs/attackscan.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
# attackschan.sh
```
attackscan.sh [-top <lines>|-scan]
Version: 0.0.3
Parses Nginx and OLS web server access logs to find top number of requests and common attack requests for WordPress.
Commands:
-top -List top number of requests from the webserver access log.
-scan -List common attack requests that return a 200 status code, by IP address.
Options:
<lines> -How many lines to show, if not specified defaults to 10
Examples
attackscan.sh -top 20
attackscan.sh -scan
```
# Commands
## -top <lines>
* List top number of requests from the webserver access log.
### Example -top Output
```
** Parsing /var/log/ols/domain.com.access.log for top 10 requests
467 /wp-login.php
353 /
97 /xmlrpc.php
93 /robots.txt
78 //xmlrpc.php
63 /robots.txt/
32 /favicon.ico
30 //wp-json/wp/v2/users/
30 //wp-json/oembed/1.0/embed?url=https://workingefficiently.com/
30 //?author=1
```
## -scan <lines>
* List common attack requests that return a 200 status code, by IP address.
### Example -scan Output
```
** Parsing /var/log/ols/domain.com.access.log for top common attack requests serving 200 status code
9 //xmlrpc.php,143.198.225.22,200
9 /wp-login.php,185.119.81.105,200
8 //xmlrpc.php,20.168.90.200,200
8 /wp-login.php,185.119.81.108,200
6 //xmlrpc.php,20.168.90.200,503
6 /wp-login.php,185.119.81.97,200
6 /wp-login.php,185.119.81.102,200
6 /wp-login.php,185.119.81.100,200
4 //xmlrpc.php,79.110.62.40,200
4 //xmlrpc.php,208.67.105.86,200
```
# Running attackscan.sh
* Log into your server via SSH
* Run one of the following commands
## Run one time via bash+curl
You can run attachscan.sh one time using the following shell command, it won't download anything to your server.
```
bash <(curl -sL https://raw.githubusercontent.com/managingwp/gp-tools/main/attackscan.sh)
```
You should see the help page output, now you can append a command to the end of the shell command.
```
bash <(curl -sL https://raw.githubusercontent.com/managingwp/gp-tools/main/attackscan.sh) -top
```
## Download and Run
If you want to keep a copy on your server.
```
cd /usr/local/sbin; wget "https://raw.githubusercontent.com/managingwp/gp-tools/main/attackscan.sh"; chmod u+x /usr/local/sbin/attackscan.sh
```
You can now simply type the following at anytime.
```
attackscan.sh
```

# FAQ
## Why would I use this script?
This should will help identify requests that could potentially be causing an increase in resource usage on your server. Or highlight potential common attacks that spike resource usage, effectively causing your server to be at capacity prematurely.
## Why did you create this script?
I required a quick way to see what were the top requests being served, and to confirm if common resource based attacks on WordPress were being carried out due to no preventative measures being in place.
## Do you support other platforms beside GridPane
Yes, simply create an issue with the platform and we'll look into support it.

# Documentation
## Parsing Method for -scan command
The parsing method for -scan is rather simple using grep. Here is the full command.
```
grep -v "redirect_to" | grep "200" | egrep -e "xmlrpc|wp-login"
```
As you can see "redirect_to" is parsed out because the request was answered and redirecting, versus a direct POST request to the wp-login page.
## Log File Formats
### GP OLS
* ```logFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"```
* ```"192.168.1.1 - - [01/Sep/2022:20:16:55 -0700] "GET /favicon.ico HTTP/1.1" 200 103639 "https://domain.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"```
### GP Nginx
* ```'[$time_local] $remote_addr $upstream_response_time $upstream_cache_status $http_host "$request" $status $body_bytes_sent $request_time "$http_referer" "$http_user_agent" "$http3"';```
* ```[01/Sep/2022:20:20:07 -0700] 192.168.1.1 1.360 - domain.com "GET /favicon.ico HTTP/2.0" 302 0 1.362 "https://domain.com" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"```
Empty file added docs/gp-api.md
Empty file.
57 changes: 57 additions & 0 deletions docs/gp-plugins.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# gpplugin.sh
The following plugin will allow you accomplish two tasks
* List all plugins for all sites.
* List the status of a specific plugin.

# Donate
Coffee, Beer or steaped Tea! Feel free to buy me one :0

<a href="https://wpinfo.net/sponsor/" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/default-orange.png" alt="Buy Me A Coffee" height="41" width="174"></a>

# Install
You can refer to [README.md](README.md) to install all the tools.

or

Run the following!

```
wget https://raw.githubusercontent.com/jordantrizz/gpcli/main/gpplugin.sh
```

# Usage
You can run the following command to see the help screen.
```
./gpplugin.sh
```

You'll be presented with the following

```
❯ ./gpplugin.sh
Lists WordPress plugins on all websites on a GridPane Server
Syntax: gpplugin -a <plugin>
options
-a List all plugins
-p Status on specific plugin
Notes:
* Find out which sites have which plugins + search for specific plugins and print wp-cli plugin status
* Excludes canary + staging + 22222 + nginx + *.gridpanevps.com sites
* --skip-plugins is run as to not fail potentially due to an error with a plugin
Updates: https://github.com/jordantrizz/gpcli
```

## List all Plugins
If you want to list all plugins simply run the following.

```
gpplugin.sh -a
```
## Statis on Specific Plugin
If you want the status on a specific plugin!
```
gpplugin.sh -p elementor
```
2 changes: 2 additions & 0 deletions docs/netdata.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Setting up Netdata Properly on GridPane.
See wpinfo.net
2 changes: 2 additions & 0 deletions docs/phpfpm-monitoring.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
# Ideas
* https://bidhankhatri.com.np/monitoring/monitoring-php-fpm-pools-through-prometheus-and-grafana-on-centos-6-and-7/
26 changes: 26 additions & 0 deletions docs/wpst-goaccess.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
Welcome GridPane'rs!

<a href="https://www.buymeacoffee.com/jordantrask" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/default-orange.png" alt="Buy Me A Coffee" height="41" width="174"></a>

# How to use goacecss with GridPane
## Installation
```
echo "deb https://deb.goaccess.io/ $(lsb_release -cs) main" | sudo tee -a /etc/apt/sources.list.d/goaccess.list
wget -O - https://deb.goaccess.io/gnugpg.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/goaccess.gpg add -
sudo apt-get update
sudo apt-get install goaccess
```

## Log Format
```
goaccess --log-format='[%d:%t %^] %h %^ - %v "%r" %s %b "%R" "%u"' --date-format='%d/%b/%Y' --time-format='%H:%M:%S %Z'
```

## Configuration
If you don't want to use the command above, you can configure goaccess globally to use the abouve settings
* Edit /etc/goaccess/goaccess.conf and update the following lines.
```
date-format %d/%b/%Y
time-format %H:%M:%S %Z
log-format [%d:%t %^] %h %^ - %v "%r" %s %b "%R" "%u"
```
221 changes: 221 additions & 0 deletions functions.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,221 @@
# ---------------
# -- functions.sh
# ---------------

# -----------
# -- Includes
# -----------

# ------------
# -- Variables
# ------------
SCRIPT=$(readlink -f "$0")
SCRIPTPATH=$(dirname "$SCRIPT")
REQUIRED_APPS=("jq" "column")
if [[ $DEBUG_ON="1" ]]; then DEBUG="1"; fi

# -- Colors
RED="\033[0;31m"
GREEN="\033[0;32m"
CYAN="\033[0;36m"
BLUEBG="\033[0;44m"
YELLOWBG="\033[0;43m"
GREENBG="\033[0;42m"
DARKGREYBG="\033[0;100m"
ECOL="\033[0;0m"

# ----------------
# -- Core Functions
# ----------------

_error () { echo -e "${RED}** ERROR ** - ${*} ${ECOL}"; }
_warning () { echo -e "${RED}** WARNING ** - ${*} ${ECOL}"; }
_notice () { echo -e "${BLUE}** NOTICE ** - ${*} ${ECOL}"; }
_success () { echo -e "${GREEN}** SUCCESS ** - ${*} ${ECOL}"; }
_running () { echo -e "${BLUEBG}${@}${ECOL}"; }
_creating () { echo -e "${DARKGREYBG}${@}${ECOL}"; }
_separator () { echo -e "${YELLOWBG}****************${ECOL}"; }

# -- _debug
_debug () {
if [[ $DEBUG_ON == "1" ]]; then
echo -e "${CYAN}** DEBUG: ${*}${ECOL}"
fi
}

# -- show debug information
_debug_all () {
_debug "--------------------------"
_debug "arguments: ${*}"
_debug "funcname: ${FUNCNAME[@]}"
_debug "basename: $SCRIPTPATH"
_debug "sourced files: ${BASH_SOURCE[@]}"
_debug "--------------------------"
}

# -- debug curl
_debug_curl () {
if [[ $DEBUG == "2" ]]; then
echo -e "${CCYAN}**** DEBUG ${*}${NC}"
fi
}

# -- _cexists -- Returns 0 if command exists or 1 if command doesn't exist
_cexists () {
if [[ "$(command -v $1)" ]]; then
_debug $(which $1)
if [[ $ZSH_DEBUG == 1 ]]; then
_debug "${*} is installed";
fi
CMD_EXISTS="0"
else
if [[ $ZSH_DEBUG == 1 ]]; then
_debug "${*} not installed";
fi
CMD_EXISTS="1"
fi
return $CMD_EXISTS
}

# -- Check root
_checkroot () {
if [ ! -f .debug ]; then
if [ "$EUID" -ne 0 ]
then echo "Please run as root"
exit
fi
fi
}

# ------------------------------
# -- GridPane specific functions
# ------------------------------

# - logs
help_cmd[logs]='tail or show last lines on all GridPane logs.'
tool_logs () {
gp-logs.sh ${*}
}

# - logcode
help_cmd[logcode]='Look for specifc HTTP codes in web server logfiles and return top hits.'
tool_logcode () {
gp-logcode.sh ${*}
}

# - mysqlmem
help_cmd[mysqlmem]='GridPane monit memory calculation'
tool_mysqlmem () {
gp-mysqlmem.sh ${*}
}

# - plugins
help_cmd[plugins]='Lists WordPress plugins on all websites on a GridPane Server'
tool_plugins () {
gp-plugins.sh ${*}
}

# - goaccess - execute log functions
help_cmd[goaccess]='Process GridPane logs with goaccess'
tool_goaccess () {
_debug_all ${*}

if [[ -z $2 ]]; then
echo "Usage: goaccess -c [domain.com|all]"
echo " Commands"
echo " domain - Domain name of log files to process"
echo " all-domains - Go through all the logs versus a single domain"
echo ""
echo " Options:"
echo " -c - Process compressed log files"
echo ""
return
fi

_debug "Checking if goaccess is installed"
_cexists goaccess
_debug "\$CMD_EXISTS: $CMD_EXISTS"
if [[ $CMD_EXISTS == "1" ]]; then
_error "goaccess is not installed"
return 1
else
_debug "Confirmed goaccess is installed"
fi

# Usage
if [ -v $2 ]; then
echo "Usage: $SCRIPT_NAME gp-goaccess [<domain.com>|-a]"
return 1
fi

# Formats for goaccess
# OLS
# logformat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"
# "192.168.0.1 - - [13/Sep/2022:16:28:40 -0400] "GET /request.html HTTP/2" 200 46479 "https://domain.com/referer.html" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"
OLS_LOG_FORMAT='\"%h - - [%d:%t %^] \"%r\" %s %b \"%R\" \"%u\"\'
OLS_DATE_FORMAT='%d/%b/%Y\'
OLS_TIME_FORMAT='%H:%M:%S %Z\'

# NGINX
# log_format we_log '[$time_local] $remote_addr $upstream_response_time $upstream_cache_status $http_host "$request" $status $body_bytes_sent $request_time "$http_referer" "$http_user_agent" "$http3"';
# [14/Sep/2022:10:12:55 -0700] 129.168.0.1 - domain.com "GET /request.html HTTP/1.1" 200 47 1.538 "https://domain.com/referer.html" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"
NGINX_LOG_FORMAT='[%d:%t %^] %h %^ - %v \"%r\" %s %b \"%R\" \"%u\"\'
NGINX_DATE_FORMAT='%d/%b/%Y\'
NGINX_TIME_FORMAT='%H:%M:%S %Z\'

_debug "goaccess arguments - $ACTION"
_debug "goaccess LOG_FORMAT = $LOG_FORMAT ## DATE_FORMAT = $DATE_FORMAT ## TIME_FORMAT = $TIME_FORMAT"
# -- Check args.

_debug "Detecting log files"
if [[ -d /usr/local/lsws ]]; then
WEB_SERVER="OLS"
LOG_FORMAT=$OLS_LOG_FORMAT
DATE_FORMAT=$OLS_DATE_FORMAT
TIME_FORMAT=$OLS_TIME_FORMAT
if [[ $2 == "all" ]]; then
LOG_FILE_LOCATION="/var/www/*/logs"
LOG_FILTER="*.access.log"
else
LOG_FILE_LOCATION="/var/www/$2/logs"
LOG_FILTER="*.access.log"
fi
elif [[ -d /var/log/nginx ]]; then
WEB_SERVER="NGINX"
LOG_FORMAT=$NGINX_LOG_FORMAT
DATE_FORMAT=$NGINX_DATE_FORMAT
TIME_FORMAT=$NGINX_TIME_FORMAT
LOG_FILE_LOCATION="/var/log/nginx"

if [[ $2 == "all" ]]; then
LOG_FILTER="*.access.log"
else
LOG_FILTER="$2.access.log"
fi
else
_error "Can't detect webserver logs"
fi

_debug "Webserver detected as $WEB_SERVER"

# Main
cat ${LOG_FILE_LOCATION}/${LOG_FILTER} | goaccess --log-format="$LOG_FORMAT" --date-format="$DATE_FORMAT" --time-format="$TIME_FORMAT"
}

# - gpcron
help_cmd[gpcron]='List sites using GP Cron'
tool_gpcron () {
grep 'cron:true' /var/www/*/logs/*.env
}

# - backups - execute log functions
help_cmd[backups]='List backups for all sites on the server.'
tool_backups () {
ls -aL /home/*/sites/*/logs/backups.env | xargs -l -I {} sh -c "echo {} | awk -F/ '{print \$5}'|tr '\n' '|'; tr '\n' '|' < {};echo \n"
}

# - api - GridPane api
help_cmd[api]='Interact with the GridPane API'
tool_api () {
gp-api.sh ${*}
}
80 changes: 80 additions & 0 deletions scripts/cron-shim.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
#!/bin/bash
# -- Created by Jordan - hello@managingwp.io - https://managingwp.io
# -- Version 1.0.0 -- Last Updated: 2020-05-09
# -- Purpose: Run WordPress crons via wp-cli and log the output to stdout, syslog, or a file.
# -- Usage: Add the following to your crontab (replacing /path/to/wordpress with the path to your WordPress install):
# */5 * * * * /home/systemuser/cron-shim.sh

# Set up the necessary variables
WP_CLI="/usr/local/bin/wp" # - Location of wp-cli
WP_ROOT="/path/to/wordpress" # - Path to WordPress
LOG_TO_STDOUT="1" # - Log to stdout? 0 = no, 1 = yes
LOG_TO_SYSLOG="1" # - Log to syslog? 0 = no, 1 = yes
LOG_TO_FILE="0" # - Log to file? 0 = no, 1 = yes
LOG_FILE="${WP_ROOT}/wordpress-crons.log" # Location for wordpress cron.
HEARTBEAT_URL="https://betteruptime.com/api/v1/heartbeat/v25v234v4634b636v3" # - Heartbeat monitoring URL
POST_CRON_CMD="" # - Command to run after cron completes
SCRIPT_DIR=$(dirname "$(realpath "$0")")

# Check if wp-cli is installed
[[ $(command -v $WP_CLI) ]] || { echo 'Error: wp-cli is not installed.' >&2; exit 1; }

# Check if $WP_ROOT exists
if [[ ! -d "$WP_ROOT" ]]; then
if [[ -d $SCRIPT_DIR/htdocs ]]; then
WP_ROOT=$SCRIPT_DIR/htdocs
elif [[ -d $SCRIPT_DIR/public_html ]]; then
WP_ROOT=$SCRIPT_DIR/public_html
else
echo "Error: $WP_ROOT does not exist." >&2; exit 1;
fi
fi

# Check if $WP_ROOT contains a WordPress install
[[ WP_ROOT_INSTALL=$(wp --path=$WP_ROOT --skip-plugins --skip-themes core is-installed 2> /dev/null) ]] || { echo "Error: $WP_ROOT is not a WordPress install.\n$WP_ROOT_INSTALL " >&2; exit 1; }

# Get the domain name of the WordPress install
DOMAIN_NAME=$($WP_CLI --path=$WP_ROOT --skip-plugins --skip-themes option get siteurl 2> /dev/null | grep -oP '(?<=//)[^/]+')
[[ $? -ne 0 ]] && { echo "Error: Could not get domain name.\n$DOMAIN_NAME"; exit 1; }

# Log the start time
START_TIME=$(date +%s.%N)

# Run WordPress crons due now and log the output
CRON_OUTPUT=$($WP_CLI --skip-plugins --skip-themes cron event run --due-now --path=$WP_ROOT 2>&1)

# Check if there was an error running wp-cli command
if [[ $? -ne 0 ]]; then
echo "Error: wp-cli command failed" >&2
echo "$CRON_OUTPUT"
if [[ -n "$POST_CRON_CMD" ]]; then
eval "$POST_CRON_CMD"
fi
exit 1
fi

# Check if heartbeat monitoring is enabled and send a request to the heartbeat URL if it is and there are no errors
if [[ -n "$HEARTBEAT_URL" ]] && [[ $? -eq 0 ]] ; then
curl -I -s "$HEARTBEAT_URL" > /dev/null
fi

# Log the end time and CPU usage
END_TIME=$(date +%s.%N)

# check if bc installed otherwise use awk
if [[ $(command -v bc) ]]; then
TIME_SPENT=$(echo "$END_TIME - $START_TIME" | bc)
else
TIME_SPENT=$(echo "$END_TIME - $START_TIME" | awk '{printf "%f", $1 - $2}')
fi
CPU_USAGE=$(ps -p $$ -o %cpu | tail -n 1)

# Check if logging to syslog is enabled
if [[ $LOG_TO_STDOUT == "1" ]]; then
echo -e "Cron job completed in $TIME_SPENT seconds with $CPU_USAGE% CPU usage. \nOutput: $CRON_OUTPUT"
elif [[ $LOG_TO_SYSLOG == "1" ]]; then
echo -e "Cron job completed in $TIME_SPENT seconds with $CPU_USAGE% CPU usage. \nOutput: $CRON_OUTPUT" | logger -t "wordpress-crons-$DOMAIN_NAME"
elif [[ $LOG_TO_FILE == "1" ]]; then
# Log to file in the WordPress install directory
echo "$(date +"%Y-%m-%d %H:%M:%S") - WordPress crons completed in $TIME_SPENT seconds with $CPU_USAGE% CPU usage. \nOutput: $CRON_OUTPUT" >> $LOG_FILE
fi
7 changes: 7 additions & 0 deletions tests/test.log
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
This is a test log!
This is a test log!
This is a test log!
This is a test log!
This is a test log!
This is a test log!
This is a test log!
24 changes: 24 additions & 0 deletions tests/wpst-goaccess-gpnginx.log
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
[14/Apr/2023:06:30:29 -0500] 192.168.1.1 - - plantsbymail.com "GET /product-category/trees/birch-trees/ HTTP/2.0" 200 31812 0.000 "https://plantsbymail.com/shop/delta-moonlight-crapemyrtle/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36"
[14/Apr/2023:06:30:29 -0500] 192.168.1.1 - - plantsbymail.com "GET /wp-content/uploads/2021/06/plants-by-mail-logo-tm.png HTTP/2.0" 304 0 0.000 "https://plantsbymail.com/product-category/trees/birch-trees/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36"
[14/Apr/2023:06:30:29 -0500] 192.168.1.1 - - plantsbymail.com "GET /wp-content/uploads/2023/02/sunshine-ligustrum-clipped1-445x500.jpg HTTP/2.0" 304 0 0.000 "https://plantsbymail.com/product-category/trees/birch-trees/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36"
[14/Apr/2023:06:30:29 -0500] 192.168.1.1 - - plantsbymail.com "GET /wp-content/uploads/2021/08/river-birch-landscape-1k.jpg HTTP/2.0" 200 327526 0.002 "https://plantsbymail.com/product-category/trees/birch-trees/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36"
[14/Apr/2023:06:30:29 -0500] 192.168.1.1 - - plantsbymail.com "GET /wp-content/plugins/woocommerce-google-merchant-center-customer-reviews/assets/image/google-reviews-badge-dark-background.png HTTP/2.0" 304 0 0.000 "https://plantsbymail.com/product-category/trees/birch-trees/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36"
[14/Apr/2023:06:30:29 -0500] 192.168.1.1 - - plantsbymail.com "GET /wp-content/uploads/2018/12/Payment-Icons.png HTTP/2.0" 304 0 0.000 "https://plantsbymail.com/product-category/trees/birch-trees/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36"
[14/Apr/2023:06:30:30 -0500] 192.168.20.1 1.584 - plantsbymail.com "GET /product-category/shrubs/camellias/?_pagination_alt=1 HTTP/2.0" 200 45246 1.584 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/103.0.5060.134 Safari/537.36"
[14/Apr/2023:06:30:31 -0500] 10.0.0.1 0.244 - plantsbymail.com "POST /wp-cron.php?doing_wp_cron=1681471831.1777219772338867187500 HTTP/2.0" 200 20 0.245 "-" "WordPress/42; https://plantsbymail.com"
[14/Apr/2023:06:30:32 -0500] 192.168.20.1 1.732 - plantsbymail.com "GET /?pwgc=1628918241 HTTP/2.0" 200 39563 1.731 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/103.0.5060.134 Safari/537.36"
[14/Apr/2023:06:30:32 -0500] 10.0.0.1 0.972 - plantsbymail.com "GET /wp-json/woofunnels/v1/worker?1681471831 HTTP/2.0" 499 0 0.970 "-" "WordPress/42; https://plantsbymail.com"
[14/Apr/2023:06:30:37 -0500] fc00::1 0.744 - plantsbymail.com "POST /?wpgb-ajax=refresh&_sort_by=title_asc&_stock_status=1 HTTP/2.0" 200 7374 0.743 "https://plantsbymail.com/brand/southern-living-plants/?_sort_by=title_asc&_stock_status=1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.3 Safari/605.1.15"
[14/Apr/2023:06:30:38 -0500] 192.168.1.1 - - plantsbymail.com "GET /shop/river-birch-tree/ HTTP/2.0" 200 43547 0.001 "https://plantsbymail.com/product-category/trees/birch-trees/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36"
[14/Apr/2023:06:30:38 -0500] 192.168.1.1 - - plantsbymail.com "GET /wp-content/uploads/2021/06/plants-by-mail-logo-tm.png HTTP/2.0" 304 0 0.000 "https://plantsbymail.com/shop/river-birch-tree/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36"
[14/Apr/2023:06:30:38 -0500] fc00::1 - - plantsbymail.com "GET /wp-content/uploads/2021/08/river-birch-3fl-dimensions-510x510.jpg HTTP/2.0" 200 21429 0.000 "https://plantsbymail.com/shop/river-birch-tree/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36"
[14/Apr/2023:06:30:38 -0500] 192.168.1.1 - - plantsbymail.com "GET /wp-content/uploads/2021/08/birch-river-birch-3fl-dimensions-510x510.jpg HTTP/2.0" 200 42353 0.000 "https://plantsbymail.com/shop/river-birch-tree/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36"
[14/Apr/2023:06:30:38 -0500] 192.168.1.1 - - plantsbymail.com "GET /wp-content/uploads/2021/08/river-birch-landscape-1k-510x510.jpg HTTP/2.0" 200 118038 0.002 "https://plantsbymail.com/shop/river-birch-tree/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36"
[14/Apr/2023:06:30:38 -0500] 192.168.1.1 - - plantsbymail.com "GET /wp-content/uploads/2021/08/river-birch-landscape-1k.jpg HTTP/2.0" 304 0 0.000 "https://plantsbymail.com/shop/river-birch-tree/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36"
[14/Apr/2023:06:30:38 -0500] 192.168.1.1 - - plantsbymail.com "GET /wp-content/uploads/2021/08/River-Birch-3FL-Clipped-510x510.jpg HTTP/2.0" 200 16648 0.000 "https://plantsbymail.com/shop/river-birch-tree/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36"
[14/Apr/2023:06:30:38 -0500] 172.16.0.1 - - plantsbymail.com "GET /product-category/trees/spruce/ HTTP/2.0" 200 32571 0.001 "https://plantsbymail.com/brand/first-editions/?gclid=CjwKCAjw8-OhBhB5EiwADyoY1ansy-J2_fcbxMpfN1E2wlN3vZqNIXMbCRtQSs83a6HcRxy04gTh8hoCfaIQAvD_BwE" "Mozilla/5.0 (iPhone; CPU iPhone OS 16_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/112.0.5615.46 Mobile/15E148 Safari/604.1"
[14/Apr/2023:06:30:38 -0500] 172.16.0.1 - - plantsbymail.com "GET /wp-content/uploads/2021/06/plants-by-mail-logo-tm.png HTTP/2.0" 304 0 0.000 "https://plantsbymail.com/product-category/trees/spruce/" "Mozilla/5.0 (iPhone; CPU iPhone OS 16_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/112.0.5615.46 Mobile/15E148 Safari/604.1"
[14/Apr/2023:06:30:38 -0500] 192.168.1.1 - - plantsbymail.com "GET /wp-content/uploads/2021/08/river-birch-landscape-1k-100x100.jpg HTTP/2.0" 200 4419 0.000 "https://plantsbymail.com/shop/river-birch-tree/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36"
[14/Apr/2023:06:30:38 -0500] 192.168.1.1 - - plantsbymail.com "GET /wp-content/uploads/2021/08/river-birch-3fl-dimensions-100x100.jpg HTTP/2.0" 200 2536 0.000 "https://plantsbymail.com/shop/river-birch-tree/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36"
[14/Apr/2023:06:30:38 -0500] 192.168.1.1 - - plantsbymail.com "GET /wp-content/uploads/2021/08/River-Birch-3FL-Clipped-100x100.jpg HTTP/2.0" 200 2009 0.000 "https://plantsbymail.com/shop/river-birch-tree/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36"
[14/Apr/2023:06:30:38 -0500] 192.168.1.1 - - plantsbymail.com "GET /wp-content/uploads/2021/08/river-birch-bark-1k-100x100.jpg HTTP/2.0" 200 3747 0.000 "https://plantsbymail.com/shop/river-birch-tree/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36"
1 change: 1 addition & 0 deletions unfinished/gp-api
125 changes: 125 additions & 0 deletions unfinished/gp-api.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,125 @@
#!/usr/bin/env bash
# - gp-logcode
. $(dirname "$0")/functions.sh
_debug "Loading functions.sh"

# --
# -- Variables
# --
API_URL="https://my.gridpane.com/oauth/api/v1/$gp_api_endpoint"
API_TOKEN=.gptoken
gp_api_cmd=$(curl -s $gp_api_url --header 'Authorization: Bearer $gp_api_access_token')

## - Color
RED='\033[97;41m'
GREEN='\033[97;42m'
YELLOW='\033[30;103m'
NC='\033[0m' # No Color
SUCCESS="${GREEN}* SUCCESS *${NC}"
ERROR="${RED}* ERROR *${NC}"

# -- Variable Ingestion
POSITIONAL=()
while [[ $# -gt 0 ]]
do
key="$1"

case $key in
-v|--value)
VALUE="$2"
shift # past argument
shift # past value
;;
-c|--command)
COMMAND="$2"
shift # past argument
shift # past value
;;
*) # unknown option
POSITIONAL+=("$1") # save it in an array for later
shift # past argument
;;
esac
done
set -- "${POSITIONAL[@]}" # restore positional parameters

if [[ -n $1 ]]; then
unknown_options=$1
fi

# --
# -- Functions
# --

# -- usage
usage () {
echo "Usage: $SCRIPT_NAME <command> <variables>"
echo ""
echo " Commands"
echo ""
echo "servers -List servers"
echo "sites -List sites"
echo "test - Test if your token works"
echo ""
echo "version: $VERSION"
echo""
}


run_api () {
api_output=$(eval curl -s --location --request $@ --header \'Authorization: Bearer $gp_api_access_token\')
_debug "Running curl --location --request $@ --header 'Authorization: Bearer $gp_api_access_token'"
if [ $? -ne 0 ]; then
echo $api_output
exit
fi
}

check_gptoken () {
if [ -f "$gp_api_token" ]; then
echo -e "$SUCCESS GP Token file found $gp_api_access_token"
source $gp_api_token
_debug "GP Token: $gp_api_access_token"
else
echo -e "$ERROR GP Token file missing, please place your GridPane API Token in the file named .gptoken"
exit
fi
}

test_gptoken () {
test_path="GET 'https://my.gridpane.com/oauth/api/v1/user'"
run_api $test_path
_debug $api_output
api_output_test=$(echo $api_output | cut -c1-6)
if [[ $api_output_test == '{"id":' ]]; then
echo -e "$SUCCESS API Success! $NC"
else
echo -e "$ERROR API Failure...something is wrong! $NC"
exit
fi
#api_test_output=$(echo $api_output | jq ' . | {name: .name, id: .id, email: .email}')
}

send_command () {
echo "Send command"
}

# -- Start

#if [[ -z $COMMAND ]]; then help;exit;fi
#if [[ $COMMAND == 'test' ]]; then check_gptoken;test_gptoken; fi
#if [[ -z $VALUE ]]; then echo "Specify a value"; exit; fi


#if [[ $COMMAND == 'd2s' ]]; then
# check_gptoken
# run_api "GET https://my.gridpane.com/oauth/api/v1/site"
# _debug $api_output
# d2s_domain_id=$(echo $d2s_list)
# _debug $d2s_domain_id
#fi

if [[ -z $1 ]]; then
usage
exit
fi
104 changes: 104 additions & 0 deletions unfinished/gp-backupsites.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
#!/usr/bin/env bash

# -- Intro
# Created by Jordan v0.1

# -- Variables
BACKUP_DIR="/var/www/backups/gpbksite"

## - Color
RED='\033[97;41m'
GREEN='\033[97;42m'
YELLOW='\033[30;103m'
NC='\033[0m' # No Color
SUCCESS="${GREEN}* SUCCESS *${NC}"
ERROR="${RED}* ERROR *${NC}"

# -- debug command line
_debug () {
if [ -f .debug ]; then
echo -e "\n${YELLOW} Debug:${NC} $@"
fi
}
_debug "Running $@"

# -- Variable Ingestion
POSITIONAL=()
while [[ $# -gt 0 ]]
do
key="$1"

case $key in
-s|--site)
SITE="$2"
shift # past argument
shift # past value
;;
-a|--all)
ALLSITES="true"
shift # past argument
shift # past value
;;
-f|--files)
DATABASE="true"
shift # past argument
shift # past value
;;
-d|--database)
DATABASE="true"
shift # past argument
shift # past value
;;
*) # unknown option
POSITIONAL+=("$1") # save it in an array for later
shift # past argument
;;
esac
done
set -- "${POSITIONAL[@]}" # restore positional parameters

if [[ -n $1 ]]; then
unknown_options=$1
fi

# -- Functions
# echo commands run using | exe eval ls -al
exe() { echo "\$ $@" ; "$@" ; }

help () {
echo "Syntax: gpbksite.sh [ -s <site> | -a ] [ -d | -f ]"
echo ""
echo "gpbksite.sh - Allows you to backup a website on GridPane to a .tar file"
echo ""
echo "Commands:"
echo "-s <site> - Website to backup"
echo "-a - Backup all sites"
echo ""
echo "Options:"
echo "-f - Files Only"
echo "-d - Databases Only"
echo "-l - Location, defaults to /var/backups/gpbksite"
echo ""
}

if [[ ! -z $SITE ]]; then
echo "Backing up $SITE"
elif [[ ! -z $ALLSITES ]]; then
echo "Backing up all sites"
else
help;exit;
fi



#if [[ $COMMAND == 'test' ]]; then check_gptoken;test_gptoken; fi
#if [[ -z $VALUE ]]; then echo "Specify a value"; exit; fi


#if [[ $COMMAND == 'd2s' ]]; then
# check_gptoken
# run_api "GET https://my.gridpane.com/oauth/api/v1/site"
# _debug $api_output
# d2s_domain_id=$(echo $d2s_list)
# _debug $d2s_domain_id
#fi
29 changes: 29 additions & 0 deletions unfinished/gp-goaccess.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
#/usr/bin/env bash
# Run gp-go-access <domain.com>
# Example gp-goaccess testing.com

# Variables
LOG_FORMAT='[%d:%t %^] %h %^ - %v \"%r\" %s %b \"%R\" \"%u\"\'
DATE_FORMAT='%d/%b/%Y\'
TIME_FORMAT='%H:%M:%S %Z\'


# -- Check args.
if [[ -v $1 ]]; then
echo "usage: gp-goaccess <domain.com>"
return
fi

# -- check for goaccess binary
if [[ _cexists goaccess == "1" ]]; then
echo " -- goaccess installed"
else
echo " -- goaccess not installed"
fi

# Main
if [ $1 = "-a" ]; then
zcat /var/log/nginx/$2.access.log.*.gz | goaccess --log-format="$LOG_FORMAT" --date-format="$DATE_FORMAT" --time-format="$TIME_FORMAT"
else
cat /var/log/nginx/$1.access.log | goaccess --log-format="$LOG_FORMAT" --date-format="$DATE_FORMAT" --time-format="$TIME_FORMAT"
fi
1 change: 1 addition & 0 deletions unfinished/gp-logcode
193 changes: 193 additions & 0 deletions unfinished/gp-logcode.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,193 @@
#!/usr/bin/env bash
# - gp-logcode
. $(dirname "$0")/functions.sh
_debug "Loading functions.sh"

# -- getopts
while getopts "c:m:l:d:r:ezx" option; do
case ${option} in
c) LOGCODE=$OPTARG ;;
m) MODE=$OPTARG ;;
l) LOGFILE=$OPTARG ;;
d) DOMAIN=$OPTARG ;;
r) RESULTS=$OPTARG ;;
e) GP_EXCLUDE=1 ;;
z) Z_INCLUDE=1 ;;
x) DEBUG=1 ;;
?) usage ;;
esac
done

_debug "args: $@"
_debug "getopts: \$LOGCODE=$LOGCODE \$MODE=$MODE \$LOGFILE=$LOGFILE \$DOMAIN=$DOMAIN \$RESULTS=$RESULTS \$GP_EXCLUDE=$GP_EXCLUDE \$Z_INCLUDE=$Z_INCLUDE"
_debug_all


# Usage
usage () {
echo "Usage: $SCRIPT_NAME -c <code> -m [all|domain|log] (-l <logfilename>|-d <domain>) [-r results] [-e]"
echo ""
echo "Commands:"
echo " -c <code> = the http status code to locate number 4 = 4xx, 5 = 5xx, 404"
echo " -m <mode> - Log Method"
echo " - all = all logs"
echo " - domain = domain logs"
echo " - log = single log, requires -l"
echo " -l <logfilename> - Specific log file when using -m log"
echo " -d <domainname> - Domain logs when using -m domain"
echo ""
echo " Options"
echo " -r [results] = number of results, optional, default 40"
echo " -e = exclude GridPane, staging and canary."
echo " -z = include compressed files"
echo " -x = debug"
}

# -- check_logs
check_logs () { # Nginx or OLS?
_debug "check_logs()"
_debug "Checking if logs exist"

# Paths
nginxlogs=/var/log/nginx
olslogs=/var/log/ols

if [ -d $nginxlogs ]; then
_debug "Found nginx logs under $nginxlogs"
LOGFILEDIR=$nginxlogs
elif [ -d $olslogs ]; then
_debug "Found ols logs under $olslogs"
LOGFILEDIR=$olslogs
elif [[ $TEST ]]; then
_debug "In test mode, using test data"
LOGFILEDIR=$SCRIPTPATH/tests
else
_error "No $nginxlogs or $olslogs directory"
exit
fi
}

process_options () {
# Confirm exclude
if [[ $GP_EXCLUDE == "-e" ]]; then
_debug "argument -e set - exclude gp, staging and canary sites in $LOGFILEDIR/*"
EXCLUDE="| egrep -v '/access.log$|staging|canary|gridpane|.gz'"
else
EXCLUDE=""
fi

# Set results if not defined.
if [ -z $RESULTS ];then
RESULTS="40"
fi
_debug "RESULTS=$RESULTS"
}

# -- main
main () {
_debug_function

# Process options
process_options

if [[ $1 == "all" ]]; then
# Mode all
_debug "All logs mode"
if [[ $Z_INCLUDE ]]; then
_debug "Include compressed files"
PROCESS_FILES=$(ls -aSd $LOGFILEDIR/* $EXCLUDE)
_debug "PROCESS_FILES: $PROCESS_FILES"
else
_debug "Exclude compressed files"
PROCESS_FILES=$(ls -aSd $LOGFILEDIR/*.log $EXCLUDE)
_debug "PROCESS_FILES: $PROCESS_FILES"
fi
elif [[ $1 == "domain" ]]; then
# Mode domain
_debug "Domain logs mode"
if [[ $Z_INCLUDE ]]; then
_debug "Include compressed files"
PROCESS_FILES=$(ls -aSd $LOGFILEDIR/$DOMAIN* $EXCLUDE)
_debug "PROCESS_FILES: $PROCESS_FILES"4
else
_debug "Exclude compressed files"
_debug "ls -aSd $LOGFILEDIR/$DOMAIN*.log $EXCLUDE"
PROCESS_FILES=$(ls -aSd $LOGFILEDIR/$DOMAIN*.log $EXCLUDE)
_debug "PROCESS_FILES: $PROCESS_FILES"
fi
elif [[ $1 == "log" ]]; then
# Mode log
_debug "Single logfile mode - $LOGFILE"
if [[ -f $LOGFILEDIR/$LOGFILE ]]; then
_success "Log file exists - $logfile"
PROCESS_FILES=$LOGFILEDIR/$LOGFILE
else
_error "Log file $LOGFILEDIR/$logfile doesn't exist"
exit
fi
fi

# logcode
if [[ $LOGCODE -lt 10 ]]; then
_debug "logcode is less than 10"
logcode="$logcode[0-9][0-9]"
fi

for file in $PROCESS_FILES; do
_debug "Processing $file"
if [[ $Z_INCLUDE == "1" ]]; then
_debug "Running -- zgrep \" $LOGCODE \" $file | awk '{ print $6" - "$10" - "$7" "$8" "$9}' | sort | uniq -c | sort -nr | head -$RESULTS)"
content=$(zgrep "\" $LOGCODE " $file | awk '{ print $6" - "$10" - "$7" "$8" "$9}' | sort | uniq -c | sort -nr | head -$RESULTS)
else
_debug "Running -- grep \" $LOGCODE \" $file | awk '{ print $6" - "$10" - "$7" "$8" "$9}' | sort | uniq -c | sort -nr | head -$RESULTS)"
content=$(grep "\" $LOGCODE " $file | awk '{ print $6" - "$10" - "$7" "$8" "$9}' | sort | uniq -c | sort -nr | head -$RESULTS)
fi
# Print content
if [[ $content ]]; then
echo "##### File: $file"
echo "$content"
echo "...more lines but limited to top $RESULTS"
else
_debug "No data in $file"
fi
done
}

# -------
# -- Main
# -------

check_logs

if [[ -z $LOGCODE ]] || [[ -z $MODE ]]; then
usage
_error "Need to specify -c <code> and -m <method>"
exit
fi

if [[ $MODE == "all" ]]; then
# Log all
_debug "mode: $MODE"
main all
elif [[ $MODE == "domain" ]]; then
# Log domain
_debug "mode: $MODE"
if [[ $DOMAIN ]]; then
_debug "domain: $DOMAIN"
main domain
else
_error "Need to specify domain via -d <domain>"
fi
elif [[ $MODE == "log" ]]; then
# Log mode
_debug "mode: $MODE"
if [[ $LOGFILE ]]; then
_debug "logfile: $LOGFILE"
main log $LOGFILE
else
_error "Need to specify logfile via -l <logfile>"
fi
else
usage
fi

1 change: 1 addition & 0 deletions unfinished/gp-logs
326 changes: 326 additions & 0 deletions unfinished/gp-logs.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,326 @@
#!/usr/bin/env bash
# ----------
# -- gp-logs
# ----------

# ------------
# -- functions
# ------------
. $(dirname "$0")/functions.sh
_debug "Loading functions.sh"

# -- getopts
while getopts ":el:c:" option; do
case ${option} in
l) LOGS=$OPTARG ;;
c) COMMAND=$OPTARG ;;
e) GP_EXCLUDE=1 ;;
?) usage ;;
esac
done
_debug "args: $@"
_debug "getopts: \$LOGS=$LOGS \$COMMAND=$COMMAND \$GP_EXCLUDE=$GP_EXCLUDE"

# ------------
# -- Variables
# ------------
GPOPT="/opt/gridpane"
GPLP="/opt/gridpane/logs"
NGINX_LOG_PATH="/var/log/nginx"
OLS_LOG_PATH="/usr/local/lsws/logs"

(( TEST >= "1" )) && LOG_FILES+=("$SCRIPTPATH/tests/test.log")
SYS_LOGS=("/var/log/syslog")
NGINX_SERVER_LOGS=("$NGINX_LOG_PATH/error.log" "$NGINX_LOG_PATH/access.log")
OLS_SERVER_LOGS=("$OLS_LOG_PATH/stderr.log" "$OLS_LOG_PATH/error.log" "$OLS_LOG_PATH/lsrestart.log")
GP_LOGS=("$GPOPT/gpclone.log" "$GPOPT/gpdailyworker.log" "$GPOPT/gphourlyworker.log" "$GPOPT/gpworker.log")
GP_BACKUP_LOGS=("$GPOPT/backup.log" "$GPOPT/backup.error.log" "$GPOPT/backups.monitoring.log")
PHP_FPM_LOGS=("/var/log/php")

# ------------
# -- debug all
# ------------
_debug_all $@

# ------------
# -- functions
# ------------

# -- Usage
usage () {
echo "Usage: $SCRIPT_NAME -c [tail|last|test] -l [all|web|system|gp|fpm] (-e)"
echo ""
echo "Command (-c):"
echo " tail - Tail all logs"
echo " last - Last 10 lines of all logs"
echo " test - Test what logs will be processed"
echo ""
echo "Logs (-l):"
echo " all - Include all logs."
echo " web - Web logs only"
echo " system - System log files only"
echo " gp - GridPane core log files only"
echo " gpbackup - GridPane backup logs only"
echo ""
echo "Options:"
echo " -e - Exclude GridPane, staging, and canary"
echo ""
echo ""
echo "Log files checked:"
echo " System - ${SYS_LOGS[@]}"
echo " OLS - ${OLS_LOGS[@]}"
echo " NGINX - ${NGINX_LOGS[@]}"
echo " GP - ${GP_LOGS[@]}"
echo " GP Backup - ${GP_BACKUP_LOGS[@]}"
echo ""
}

# -- start_logs
start_logs () {
echo ""
read -p " -- Press [Enter] to start"


}

# -- collect_logs
collect_logs () {
_debug_function
# all = Include all logs.
# web = Web logs only
# system = System log files only
# gp = GridPane core log files only"
# fpm = PHP-FPM log files only"

if [[ $LOGS == "all" ]]; then
_debug "Collect logs: all"
collect_system_logs
collect_webserver-logs
collect_web-logs both
collect_php-fpm-logs
collect_gridpane
collect_gridpane_backup
elif [[ $LOGS == "web-access" ]];then
collect_webserver-logs
collect_web-logs access
elif [[ $LOGS == "web-error" ]]; then
collect_web-logs error
collect_php-fpm-logs
elif [[ $LOGS == "web-all" ]]; then
collect_webserver-logs
collect_web-logs both
collect_php-fpm-logs
elif [[ $LOGS == "system" ]]; then
collect_system_logs
elif [[ $LOGS == "gp" ]]; then
collect_gridpane
collect_gridpane_backup
elif [[ $LOGS == "gpbackup" ]]; then
collect_gridpane_backup
else
_error "Something borked... :("
fi

}


# -- system logs
collect_system_logs () {
# System log files
echo "-- Checking System log files"
for SYS_LOG in "${SYS_LOGS[@]}"; do
echo -n " -- Checking $SYS_LOG"
if [ -f $SYS_LOG ]; then
_success " -- Found $SYS_LOG"
LOG_FILES+=("$SYS_LOG")
else
_error " -- Didn't find $SYS_LOG"
fi
done
}

# -- collect_web-logs errors|access|both
collect_web-logs () {
_debug_function
# Find logs
if [ -d $NGINX_LOG_PATH ]; then
_debug "Found nginx log directory"
collect_nginx $1
elif [ -d $OLS_LOG_PATH ]; then
_debug "Found OLS log directory"
collect_ols $1
else
_error "-- Didn't find access logs directory for OLS or Nginx"
fi
}

# -- collect_access-logs-nginx -- grab Nginx log files.
collect_nginx () {
# Choose what logs to return based on $1
if [[ $1 == "error" ]] || [[ $1 == "access" ]]; then
LOG_GREP="$1"
elif [[ $1 == "both" ]]; then
LOG_GREP="error|access"
fi
_debug "\$LOG_GREP:$LOG_GREP"

if [[ $GP_EXCLUDE ]]; then
_debug "Exclude GridPane access logs '/access.log$|staging|canary|gridpane|.gz'"
NGINX_LOGS=$(ls -aSd $NGINX_LOG_PATH/* | egrep "$LOG_GREP" | egrep -v '/access.log$|staging|canary|gridpane|.gz' | tr '\n' ' ')
else
_debug "Including GridPane access logs '/access.log$|staging|canary|gridpane|.gz'"
NGINX_LOGS=$(ls -aSd $NGINX_LOG_PATH/* | egrep "$LOG_GREP" | grep -v '.gz' | tr '\n' ' ')
fi
_debug "\$NGINX_LOGS = ${NGINX_LOGS}"

if [[ -f $NGINX_LOGS ]]; then
_error "Didn't find any Nginx Access logs"
else
_success " -- Found $NGINX_LOGS"
fi
LOG_FILES+=("$NGINX_LOGS")

}

collect_ols () {
_debug "Not completed"
echo "-- Not completed"
}


# -- collect_webserver-logs
collect_webserver-logs () {
_debug_function

# Nginx logs
if [[ -d $NGINX_LOG_PATH ]]; then
echo " -- Checking for NGINX webserver logs"
for LOG in "${NGINX_SERVER_LOGS[@]}"; do
echo -n " -- Checking $LOG"
if [ -f $LOG ]; then
_success " -- Found $LOG"
LOG_FILES+=("LOG")
else
_error " -- Didn't find $LOG"
fi
done
else
_error "-- Didn't find Nginx webserver logs"
fi

# OLS logs
if [[ -d $OLS_LOG_PATH ]]; then
echo "-- Checking for OLS web server logs"
for LOG in "${OLS_SERVER_LOGS[@]}"; do
echo -n " -- Checking $LOG"
if [ -f $LOG ]; then
_success " -- Found $LOG"
LOG_FILES+=("$LOG")
else
_error " -- Didn't find $LOG"
fi
done
else
_error "-- Didn't find OLS web server logs"
fi
}

collect_gridpane () {
# GridPane specific log files
echo "-- Checking for GridPane log files"
for GP_LOG in "${GP_LOGS[@]}"; do
echo -n " -- Checking $GP_LOG"
if [ -f $GP_LOG ]; then
_success " -- Found $GP_LOG"
LOG_FILES+=("$GP_LOG")
else
_error " -- Didn't find $GP_LOG"
fi
done
}


collect_gridpane_backup () {
# GridPane Backup specific log files
echo "-- Checking for GridPane Backup log files"
for GP_BACKUP_LOG in "${GP_BACKUP_LOGS[@]}"; do
echo -n " -- Checking $GP_BACKUP_LOG"
if [ -f $GP_BACKUP_LOG ]; then
_success " -- Found $GP_BACKUP_LOG"
LOG_FILES+=("$GP_BACKUP_LOG")
else
_error " -- Didn't find $GP_BACKUP_LOG"
fi
done
}

collect_php-fpm-logs () {
# PHP FPM log files
echo "-- Checking PHP FPM log files"

if [ "$(find "$PHP_FPM_LOGS" -mindepth 1 -maxdepth 1 -not -name '.*')" ]; then
_debug "Found PHP log directories"
for name in $PHP_FPM_LOGS/*/*; do
_debug "Found $name"
done
else
_debug "Not including PHP FPM logs"
fi
# Old Method
# Locate FPM logs.
#for file in /var/log/php/*/fpm.log; do
# _debug "fpm file - $file"
# FPM_LOGS+=("$file")
#done
}

# -- check_logs -- most likely not needed anymore
check_logs () {
# -- Check if there are any logs to run against.
if [[ -z $LOG_FILES ]]; then
_debug "\$LOG_FILES = $LOG_FILES"
_error "No logs files found at all, exiting"
exit 1
else
_success " -- Found log files, continuing"
_debug " -- Found log files - ${LOG_FILES[*]}"
fi
}


# -------
# -- Main
# -------

if [[ -z $COMMAND ]] || [[ -z $LOGS ]]; then
_error "Requires -c and -l to proceed"
usage
exit 1
fi

if [[ $COMMAND = 'tail' ]]; then
echo " -- Starting to tail logs"
collect_logs
echo " -- Tailing files ${LOG_FILES[@]}"
start_logs
tail -f "${LOG_FILES[@]}"
elif [[ $COMMAND = 'last' ]]; then
echo " -- Starting to tail logs"
collect_logs
echo " -- Tailing last 50 lines of files $LOG_FILES"
start_logs
tail -n 50 "${LOG_FILES[@]}" | less
elif [[ $COMMAND = 'test' ]]; then
echo " -- Starting test"
collect_logs
echo " -- Running test to confirm log files to process"
start_logs
echo ${LOG_FILES[@]} | tr ' ' '\n'
else
_debug "args: $@"
_error "No option provided to print out logs, choose either tail or last"
usage
exit 1
fi

_success "Test!"
23 changes: 23 additions & 0 deletions unfinished/gp-multisite-cron.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
#!/bin/bash
#
# This one liner should work, you would need to replace your.site with your actual multisite primary domain:
# sudo su - "$(/usr/local/bin/gp conf read sys-user -site.env "your.site" -q)" -c "/usr/local/bin/wp site list --field=url --path=/var/www/your.site/htdocs | xargs -i -n1 /usr/local/bin/wp cron event run --due-now --url={} --path=/var/www/your.site/htdocs"
#
# You would save that in some filepath and make it executable:
# chmod +x /path/to/your/multisite-cron-script.sh
# The script accepts two args,
# multisite-cron-script.sh {primary.domain:string} {seconds:integer}
#
# Then create a cronjob in your cron tab, passing in your site domain and seconds to sleep.
# It defaults to 1 second if you omit the second arg. Here is an example to run every 5 minutes, with 5 seconds delay between subsites crons, and outputting to a log.
#
# 5 * * * * /path/to/your/multisite-cron-script.sh your.site 2 >>/var/log/multisite-cron.log

default_sleep=1
multi_site="$1"
sleep_for_seconds="${2:-$default_sleep}"
sys_user=$(/usr/local/bin/gp conf read sys-user -site.env "${multi_site}" -q)
for site in $(sudo su - "${sys_user}" -c "/usr/local/bin/wp site list --field=url --path=/var/www/${multi_site}/htdocs"); do
sudo su - "${sys_user}" -c "/usr/local/bin/wp cron event run --due-now --url=${site} --path=/var/www/${multi_site}/htdocs"
sleep ${sleep_for_seconds}
done
1 change: 1 addition & 0 deletions unfinished/gp-mysqlmem
63 changes: 63 additions & 0 deletions unfinished/gp-mysqlmem.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
#!/usr/bin/env bash
# - gp-logcode
. $(dirname "$0")/functions.sh
_debug "Loading functions.sh"

# --
# -- Variables
# --
ram_KB=$(grep MemTotal /proc/meminfo | awk '{print $2}')
ram_MB=$(echo "scale=0; ${ram_KB} / 1000" | bc)
ten_percent_ram_MB=$(echo "scale=0; ${ram_KB} / 10000" | bc)
twenty_percent_ram_MB=$(echo "scale=0; ${ram_KB} / 5000" | bc)
thirty_percent_ram_MB=$(echo "scale=0; ${ram_KB} / 3000" | bc)
fourty_percent_ram_MB=$(echo "scale=0; ${ram_KB} / 2500" | bc)
fifty_percent_ram_MB=$(echo "scale=0; ${ram_KB} / 2000" | bc)
sixty_percent_ram_MB=$((thirty_percent_ram_MB + thirty_percent_ram_MB))
seventy_percent_ram_MB=$((fifty_percent_ram_MB + twenty_percent_ram_MB))
eighty_percent_ram_MB=$((fifty_percent_ram_MB + thirty_percent_ram_MB))
ninety_percent_ram_MB=$((fifty_percent_ram_MB + fourty_percent_ram_MB))

# --
# -- Start
# --

echo "gpmonit mysql algorithm"
echo "--"
echo "Current memory in MB: $ram_MB"
echo "--"

echo "If ram_MB > 1500 then kill at 500MB"
if [[ ${ram_MB} -lt 1500 ]]; then
mysqlRAM="500"
fi

echo "If ram_MB =< 1500 then kill at $(echo "scale=0; ${seventy_percent_ram_MB} / 2" | bc)MB (70% Memory in MB/2)"
if [[ ${ram_MB} -ge 1500 ]]; then
mysqlRAM="$(echo "scale=0; ${seventy_percent_ram_MB} / 2" | bc)"
fi

echo "If ram_MB =< 8000 then kill at $(echo "scale=0; ${ninety_percent_ram_MB} / 2" | bc)MB (80% memory in MB/2)"
if [[ ${ram_MB} -ge 8000 ]]; then
mysqlRAM="$(echo "scale=0; ${ninety_percent_ram_MB} / 2" | bc)"
fi

echo "If ram_MB =< 1600 then kill at ${fifty_percent_ram_MB}MB (50% memory in MB)"
if [[ ${ram_MB} -ge 16000 ]]; then
mysqlRAM="${fifty_percent_ram_MB}"
fi

# -- print out details
echo "gpmonit mysql would set the restart threshold at ${mysqlRAM}MB memory usage for MySQL"
echo ""
echo "Suggested Changes"
echo "--"
echo "If ram_MB =< 4000 then kill at ${fifty_percent_ram_MB} (50% Memory in MB)"
echo "or"
echo "Regardless of memory kill at ${fifty_percent_ram_MB} (50% Memory in MB)"
echo "--"
echo ""
echo "To fix this issue run the following command:"
echo " 50% memory - gpmonit mysql -mem-high-mb ${fifty_percent_ram_MB} -mem-restart-mb ${fifty_percent_ram_MB}"
echo " 60% memory - gpmonit mysql -mem-high-mb ${sixty_percent_ram_MB} -mem-restart-mb ${sixty_percent_ram_MB}"
echo " 705 memory - gpmonit mysql -mem-high-mb ${seventy_percent_ram_MB} -mem-restart-mb ${seventy_percent_ram_MB}"
1 change: 1 addition & 0 deletions unfinished/gp-plugins
61 changes: 61 additions & 0 deletions unfinished/gp-plugins.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
#!/usr/bin/env bash
# - gp-logcode
. $(dirname "$0")/functions.sh
_debug "Loading functions.sh"

# Variables
GP_WWW_PATH="/var/www/"
WP_CLI_OPTIONS="--allow-root --skip-plugins --skip-themes --skip-packages"

# Functions
usage () {
echo "Lists WordPress plugins on all websites on a GridPane Server"
echo ""
echo "Syntax: gpplugin -a <plugin>"
echo " options"
echo " -a List all plugins"
echo " -p Status on specific plugin"
echo ""
echo "Notes:"
echo " * Find out which sites have which plugins + search for specific plugins and print wp-cli plugin status"
echo " * Excludes canary + staging + 22222 + nginx + *.gridpanevps.com sites"
echo " * --skip-plugins is run as to not fail potentially due to an error with a plugin"
echo ""
echo "Updates: https://github.com/jordantrizz/gpcli"
echo ""
exit
}

POSITIONAL=()
while [[ $# -gt 0 ]]
do
key="$1"

case $key in
-a|--all)
ALL="YES"
shift # past argument
shift # past value
;;
-p|--plugin)
PLUGIN="$2"
shift # past argument
shift # past value
;;
*) # unknown option
POSITIONAL+=("$1") # save it in an array for later
shift # past argument
;;
esac
done
set -- "${POSITIONAL[@]}" # restore positional parameters

if [ ! -z $PLUGIN ]; then
echo "Searching for $PLUGIN on all sites"
ls -1 $GP_WWW_PATH | grep -vE 'canary|staging' | grep -vE '22222|nginx|gridpanevps.com' | xargs -i sh -c "echo '\033[1;42m$GP_WWW_PATH{}\e[0m';wp $WP_CLI_OPTIONS --path=$GP_WWW_PATH{}/htdocs plugin status $PLUGIN"
elif [ ! -z $ALL ]; then
echo "Searching for all plugins on all sites!"
ls -1 $GP_WWW_PATH | grep -vE 'canary|staging' | grep -vE '22222|nginx|gridpanevps.com' | xargs -i sh -c "echo '\033[1;42m$GP_WWW_PATH{}\e[0m';wp $WP_CLI_OPTIONS --path=$GP_WWW_PATH{}/htdocs plugin status"
else
usage
fi
1 change: 1 addition & 0 deletions unfinished/gp-tune
15 changes: 15 additions & 0 deletions unfinished/gp-tune.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
#!/usr/bin/env bash
# - gp-logcode
. $(dirname "$0")/functions.sh
_debug "Loading functions.sh"

usage () {
echo "Usage: nothing yet!"
}

tuneswap () {
# Tune swap
##ISSUE need to check if swappiness is already tuned.
echo " -- Tuning /proc/sys/vm/swappiness"
echo 1 > /proc/sys/vm/swappiness
}
17 changes: 17 additions & 0 deletions unfinished/gp-tunemysql.pl
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
#!/usr/bin/env bash
# -- Command to set profiles for specific server sizes
# -- Usage: gptunemysql (small|medium|large)

# -- Thread Pools
# Use pool-of-threads because https://mariadb.com/kb/en/thread-pool-in-mariadb/
# gp stack mysql -thread-handling pool-of-threads

# -- Thread Pool Size
# thread_pool_size = Number of CPU's
# gp stack mysql -thread-pool-size 4

# -- Thread Pools +
# The rest of the thread_pool_* options are fine as default for now.

# -- Innodb
# innodb_flush_method=O_DIRECT
76 changes: 76 additions & 0 deletions wp-create.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
#!/bin/bash
#
# USE FOR LOCAL DEVELOPMENT ONLY!!!
#
EMAIL='your-email@gmail.com'
MYSQL=`which mysql`

echo -n "DB user/name for project (lowercase) [ENTER]:"
read DB

if [ -z "$DB" ]; then
echo "User must define a DB Name for your project."
exit;
fi

echo -n "Local URL eg. local.test.com [ENTER]:"
read URL

if [ -z "$URL" ]; then
echo "You must enter a valid url."
exit;
fi

#
# Start WP Installation
#

random-string() {
LENGTH=$1
if [ -z "$LENGTH" ]; then
LENGTH='20'
fi

cat /dev/urandom | env LC_CTYPE=C tr -dc 'a-zA-Z0-9' | fold -w $LENGTH | head -n 1
}

# Generate random password (mac friendly)
PASS=`random-string 20`

PREFIX=''
# No prefix? Then just comment out the line below
PREFIX="--dbprefix=`random-string 6`_"

Q1="CREATE DATABASE IF NOT EXISTS $DB;"
Q2="GRANT ALL ON $DB.* TO '$DB'@'localhost' IDENTIFIED BY '$PASS';"
Q3="FLUSH PRIVILEGES;"
SQL="${Q1}${Q2}${Q3}"

# Set up a login path for creating your dbs and users (must have global privileges)
# mysql_config_editor set --login-path=local --host=localhost --user=username --password
$MYSQL --login-path=local -e "$SQL"
echo "MYSQL user/db '$DB' created."

# Install latest WP
wp core download
wp core config --dbname="$DB" --dbuser="$DB" --dbpass="$PASS" $PREFIX
wp core install --url="$URL" --title='New WP Build' --admin_user='jason' --admin_password='local' --admin_email="$EMAIL" --skip-email

# Install but leave inactive

# Remove old default themes
wp theme delete twentyfifteen
wp theme delete twentythirteen
wp theme delete twentyfourteen

# Remove default posts, widgets, comments etc.
wp site empty --yes

# Move the config a directory above
mv wp-config.php ../

# Make uploads writable
chmod -R 777 wp-content/uploads

# Making a dumping folder
mkdir etc#
1 change: 1 addition & 0 deletions wp-migrate-ssh
3,177 changes: 3,177 additions & 0 deletions wp-migrate-ssh.sh

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions wpst
1 change: 1 addition & 0 deletions wpst-goaccess
374 changes: 374 additions & 0 deletions wpst-goaccess.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,374 @@
#!/bin/bash
# ------------------
# -- wpst-goacces.sh
# ------------------

# -----------
# -- Includes
# -----------
. $(dirname "$0")/functions.sh

# ------------
# -- Variables
# ------------
DEBUG_ON="0"

# ------------
# -- Functions
# ------------

usage () {
USAGE=\
"Usage: wpst-goaccess [-h|-d|-c|-dr|-f (nginx|ols)|-p [(gridpane|runcloud)] [-ld <log directory>|-l <log file>|-time <timerange>] [-domain domain.com|--all|-file <filename> [-p (gridpane|runcloud)]
This script will try and detect log files in common locations, you can also
specify the platform and format using the options below
Commands
-domain <domain> - Domain name of log files to process
-all - Go through all the logs versus a single log file
-file <filename> - Process a single file
-time <timerange> - Specify a time range to process dates in the format yyyy-mm-dd-hh-mm-ss,yyyy-mm-dd-hh-mm-ss (e.g. 2017-01-01-00-00-00,2017-01-01-23-59-59)
- First portion is start time and second part is end time.
Options:
-h - Help
-d - Debug
-dr - Dry Run
-p - Specify platform (gridpane|runcloud)
-f - Override detected format, (nginx|ols)
-t - Use test log files
"
echo "$USAGE"
}

# -- check_goaccess
check_goaccess () {
_debug "Checking if goaccess is installed"
_cexists goaccess
_debug "\$CMD_EXISTS: $CMD_EXISTS"
if [[ $CMD_EXISTS == "1" ]]; then
_error "goaccess is not installed"
exit
else
_debug "Confirmed goaccess is installed"
fi
}

# -- set_format
set_format () {
# Formats for goaccess
# OLS
# logformat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"
# "192.168.0.1 - - [13/Sep/2022:16:28:40 -0400] "GET /request.html HTTP/2" 200 46479 "https://domain.com/referer.html" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"
if [[ $FORMAT == "OLS" ]]; then
LOG_FORMAT='\"%h - - [%d:%t %^] \"%r\" %s %b \"%R\" \"%u\"'
DATE_FORMAT='%d/%b/%Y'
TIME_FORMAT='%H:%M:%S %Z'
fi

# NGINX
# log_format we_log '[$time_local] $remote_addr $upstream_response_time $upstream_cache_status $http_host "$request" $status $body_bytes_sent $request_time "$http_referer" "$http_user_agent" "$http3"';
# [14/Sep/2022:10:12:55 -0700] 129.168.0.1 - domain.com "GET /request.html HTTP/1.1" 200 47 1.538 "https://domain.com/referer.html" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"
if [[ $FORMAT == "NGINX" ]]; then
LOG_FORMAT='[%d:%t %^] %h %^ - %v \"%r\" %s %b \"%R\" \"%u\"\'
DATE_FORMAT='%d/%b/%Y'
TIME_FORMAT='%H:%M:%S %Z'
fi

# GRIDPANE-NGINX
# ./common/logging.conf:log_format we_log '[$time_local] $remote_addr $upstream_response_time $upstream_cache_status $http_host "$request" $status $body_bytes_sent $request_time "$http_referer" "$http_user_agent"';
# $request_time = seconds with a milliseconds resolution
# [14/Apr/2023:06:30:32 -0500] 127.0.0.1 1.732 - domain.com "GET /?pwgc=1628918241 HTTP/2.0" 200 39563 1.731 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/103.0.5060.134 Safari/537.36"
# [27/Apr/2023:06:38:01 -0500] 2600:1700:4dce:28f0:ed74:ea35:b8e5:ece6 - - domain.com "GET /image.jpg HTTP/2.0" 304 0 0.000 "https://yahoo.com" "Mozilla/5.0 (iPhone; CPU iPhone OS 16_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/107.0.5304.66 Mobile/15E148 Safari/604.1"
if [[ $FORMAT == "GPNGINX" ]]; then
LOG_FORMAT='[%d:%t %^] %h %^ - %v \"%r\" %s %b %T \"%R\" \"%u\"\'
DATE_FORMAT='%d/%b/%Y'
TIME_FORMAT='%H:%M:%S %Z'
fi

# GRIDPANE-OLS
if [[ $FORMAT == "GPOLS" ]]; then
LOG_FORMAT='[%d:%t %^] %h %^ - %v \"%r\" %s %b \"%R\" \"%u\"\'
DATE_FORMAT='%d/%b/%Y'
TIME_FORMAT='%H:%M:%S %Z'
fi

_debug "goaccess LOG_FORMAT = $LOG_FORMAT ## DATE_FORMAT = $DATE_FORMAT ## TIME_FORMAT = $TIME_FORMAT"
}

detect_logs () {
echo "Detecting log files"
# TODO figure out how to test and set the format better.

# GRIDPANE-OLS
if [[ -d /usr/local/lsws ]] && [[ -d /var/www/ ]]; then
echo "Detected GridPane OLS logs"
FORMAT="GPOLS"
if [[ $ACTION == "ALL" ]]; then
LOG_FILE_LOCATION="/var/www/*/logs"
LOG_FILTER="*.access.log*gz"
elif [[ $ACTION == "DOMAIN" ]]; then
LOG_FILE_LOCATION="/var/www/$DOMAIN/logs"
LOG_FILTER="*.access.log"
elif [[ $ACTION == "FILE" ]]; then
LOG_FILE_LOCATION="$FILE"
fi
elif [[ -d /var/log/nginx ]] && [[ -d /var/www/ ]]; then
echo "Detected GridPane NGINX logs"
FORMAT="GPNGINX"
if [[ $ACTION == "ALL" ]]; then
LOG_FILE_LOCATION="/var/log/nginx"
LOG_FILTER="*.access.log*gz"
elif [[ $ACTION == "DOMAIN" ]]; then
LOG_FILE_LOCATION="/var/log/nginx"
LOG_FILTER="*${DOMAIN}*.access.log"
elif [[ $ACTION == "FILE" ]]; then
LOG_FILE_LOCATION="$FILE"
fi
# OLS
elif [[ -d /usr/local/lsws ]]; then
echo "Detected OLS logs"
FORMAT="OLS"
if [[ $ACTION == "ALL" ]]; then
LOG_FILE_LOCATION="/var/www/*/logs"
LOG_FILTER="*.access.log*gz"
elif [[ $ACTION == "DOMAIN" ]]; then
LOG_FILE_LOCATION="/var/www/$DOMAIN/logs"
LOG_FILTER="*.access.log"
elif [[ $ACTION == "FILE" ]]; then
LOG_FILE_LOCATION="$FILE"
fi
# NGINX
elif [[ -d /var/log/nginx ]]; then
echo "Detected GridPane Nginx logs"
FORMAT="OLS"
LOG_FILE_LOCATION="/var/log/nginx"

if [[ $ACTION == "ALL" ]]; then
LOG_FILTER="*.access.log"
elif [[ $ACTION == "DOMAIN" ]]; then
LOG_FILTER="$DOMAIN.access.log"
elif [[ $ACTION == "FILE" ]]; then
LOG_FILE_LOCATION="$FILE"
fi
else
_error "Can't detect webserver logs"
exit
fi

_debug "FORMAT:$FORMAT LOG_FILE_LOCATION:$LOG_FILE_LOCATION LOG_FILTER:$LOG_FILTER"
}

# -- collect_logs
collect_logs () {
local CATCMD="cat"
_debug "Format: $FORMAT Log File Location:$LOG_FILE_LOCATION Log Filter: $LOG_FILTER"
LOG_COLLECT_DATA=$(mktemp)
if [[ $ACTION == "DOMAIN" ]]; then
if [[ $DRY_RUN == "1" ]]; then
echo "cat ${LOG_FILE_LOCATION}/${LOG_FILTER} > $LOG_COLLECT_DATA"
else
cat ${LOG_FILE_LOCATION}/${LOG_FILTER} > $LOG_COLLECT_DATA
fi
elif [[ $ACTION == "ALL" ]]; then
if [[ $DRY_RUN == "1" ]]; then
echo "cat ${LOG_FILE_LOCATION}/*.access.log > $LOG_COLLECT_DATA; zcat ${LOG_FILE_LOCATION}/${LOG_FILTER} >> $LOG_COLLECT_DATA"
else
cat ${LOG_FILE_LOCATION}/*.access.log > $LOG_COLLECT_DATA; zcat ${LOG_FILE_LOCATION}/${LOG_FILTER} >> $LOG_COLLECT_DATA
fi
elif [[ $ACTION == "FILE" ]]; then
if [[ $DRY_RUN == "1" ]]; then
echo "$CATCMD ${LOG_FILE_LOCATION} > $LOG_COLLECT_DATA"
else
[[ $LOG_FILE_LOCATION == "*.gz" ]] && CATCMD="zcat"
$CATCMD ${LOG_FILE_LOCATION} > $LOG_COLLECT_DATA
fi
fi
LOG_DATA_FILE=$LOG_COLLECT_DATA
}

# -- do_goaccess
do_goaccess () {
_debug "Proceeding with d_goaccess"
if [[ -n $TIME_SPEC ]]; then
GOACCESS_EXTRA+="--hour-spec=$TIME_SPEC"
fi

if [[ $ACTION == "DOMAIN" ]]; then
if [[ $DRY_RUN == "1" ]]; then
echo "cat ${LOG_FILE_LOCATION}/${LOG_FILTER} | goaccess ${GOACCESS_EXTRA} --log-format='$LOG_FORMAT' --date-format='$DATE_FORMAT' --time-format='$TIME_FORMAT'"
else
cat $LOG_DATA_FILE | goaccess ${GOACCESS_EXTRA} --log-format="${LOG_FORMAT}" --date-format="$DATE_FORMAT" --time-format="$TIME_FORMAT"
fi
elif [[ $ACTION == "ALL" ]]; then
if [[ $DRY_RUN == "1" ]]; then
echo "cat $LOG_DATA_FILE | goaccess ${GOACCESS_EXTRA} --log-format='$LOG_FORMAT' --date-format='$DATE_FORMAT' --time-format='$TIME_FORMAT'"
else
cat $LOG_DATA_FILE | goaccess ${GOACCESS_EXTRA} --log-format="$LOG_FORMAT" --date-format="$DATE_FORMAT" --time-format="$TIME_FORMAT"
fi
elif [[ $ACTION == "FILE" ]]; then
[[ $LOG_FILE_LOCATION == "*.gz" ]] && CATCMD="zcat"
if [[ $DRY_RUN == "1" ]]; then
echo "cat $LOG_DATA_FILE | goaccess ${GOACCESS_EXTRA} --log-format='$LOG_FORMAT' --date-format='$DATE_FORMAT' --time-format='$TIME_FORMAT'"
else
cat $LOG_DATA_FILE | goaccess --log-format="$LOG_FORMAT" --date-format="$DATE_FORMAT" --time-format="$TIME_FORMAT"
echo "Command Ran $(history | tail -n 1)"
fi
elif [[ $ACTION == "TEST" ]]; then
[[ $LOG_FILE_LOCATION == "*.gz" ]] && CATCMD="zcat"
if [[ $DRY_RUN == "1" ]]; then
echo "cat ${LOG_FILE_LOCATION} | goaccess ${GOACCESS_EXTRA} --log-format='$LOG_FORMAT' --date-format='$DATE_FORMAT' --time-format='$TIME_FORMAT'"
else
$CATCMD ${LOG_FILE_LOCATION} | goaccess ${GOACCESS_EXTRA} --log-format="$LOG_FORMAT" --date-format="$DATE_FORMAT" --time-format="$TIME_FORMAT"
fi
else
echo "No action specified"
exit
fi
}

function sed_logs() {
_debug "Processing logs using custom time - $CUSTOM_TIME"
SED_LOG=$(mktemp)
if ! [[ $CUSTOM_TIME =~ ^[0-9]{2}/(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)/[0-9]{4}:[0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{2}/(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)/[0-9]{4}:[0-9]{2}:[0-9]{2}:[0-9]{2}$ ]]; then
echo "Error: Please provide dates in the format dd/Mon/yyyy:hh:mm:ss"
exit 1
fi

local EDATE=$(echo $CUSTOM_TIME | cut -d, -f1)
local SDATE=$(echo $CUSTOM_TIME | cut -d, -f2)
local START_DATE=$(date -d "$(echo "$SDATE" | sed 's/\// /g;s/:/ /')" +%s | xargs -I{} date -d "@{}" +'%d\/%b\/%Y:%H:%M:%S')
local END_DATE=$(date -d "$(echo "$EDATE" | sed 's/\// /g;s/:/ /')" +%s | xargs -I{} date -d "@{}" +'%d\/%b\/%Y:%H:%M:%S')

if [[ $DRY_RUN == "1" ]]; then
echo "sed -n \"/$START_DATE/,/$END_DATE/ p\" $LOG_DATA_FILE > $SED_LOG"
else
sed -n "/$START_DATE/,/$END_DATE/ p" $LOG_DATA_FILE > $SED_LOG
fi
LOG_DATA_FILE=$SED_LOG
}

# ---------------
# -- Main Program
# ---------------
DCMD=""
POSITIONAL=()
while [[ $# -gt 0 ]]
do
key="$1"
case $key in
-h)
ACTION="HELP"
DCMD+="ACTION=HELP "
shift # past argument
;;
-platform)
PLATFORM="$2"
DCMD+="PLATFORM=$2 "
shift # past argument
shift # past value
;;
-f|--format)
FORMAT="$2"
DCMD+="FORMAT=$2 "
shift # past argument
shift # past value
;;
-d|--debug)
DEBUG_ON="1"
DCMD+="DEBUG_ON=1 "
shift # past argument
;;
-dr)
DRY_RUN="1"
DCMD+="DRY_RUN=1 "
shift # past argument
;;
-domain)
ACTION="DOMAIN"
DOMAIN="$2"
DCMD+="ACTION=DOMAIN DOMAIN=$2 "
shift # past argument
shift # past value
;;
-all)
ACTION="ALL"
DCMD+="ACTION=ALL "
shift # past argument
;;
-file)
ACTION="FILE"
FILE="$2"
DCMD+="ACTION=FILE FILE=$2 "
shift # past argument
shift # past value
;;
-t)
ACTION="TEST"
FILE="$2"
DCMD+="ACTION=TEST FILE=$2 "
shift # past argument
shift # past value
;;
-time)
CUSTOM_TIME="$2"
DCMD+="CUSTOM_TIME=$2 "
shift # past argument
shift # past value
;;
-timespec)
TIME_SPEC="$2"
DCMD+="TIME_SPEC=$2 "
shift # past argument
shift # past value
;;
*) # unknown option
POSITIONAL+=("$1") # save it in an array for later
shift # past argument
;;
esac
done
set -- "${POSITIONAL[@]}" # restore positional parameters



# ----------------------------#
# -- Debugging
# ----------------------------#
_debug "Running wpst-goaccess $DCMD"
_debug_all "${*}"

# ----------------------------#
# --- Check Arguments
# ----------------------------#
if [[ -n $TIME_SPEC ]]; then
if [[ $TIME_SPEC != "min" && $TIME_SPEC != "hour" ]]; then
echo "Error: Invalid time specification"
exit
fi
fi

# ----------------------------#
# --- Main
# ----------------------------#

if [[ -z $ACTION ]] || [[ $ACTION == "HELP" ]]; then
usage
echo "Error: No action specified"
exit
else
if [[ $ACTION == "DOMAIN" ]] && [[ -z $DOMAIN ]]; then
usage
echo "Error: specify a domain"
exit
fi
_debug "goaccess arguments - ${*}"
check_goaccess
detect_logs
set_format
collect_logs
[[ ! -z $CUSTOM_TIME ]] && sed_logs $CUSTOM_TIME
do_goaccess $ACTION $LOG_DATA_FILE
fi
1 change: 1 addition & 0 deletions wpst-sql
131 changes: 131 additions & 0 deletions wpst-sql.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
#!/bin/bash
# ------------------
# -- wpst-goacces.sh
# ------------------

# -----------
# -- Includes
# -----------
. $(dirname "$0")/functions.sh

# ------------
# -- Variables
# ------------
DEBUG_ON="0"
CMD_VERSION="0.0.1"
CMD_SCRIPT="wpst-sql"

# ------------
# -- Functions
# ------------

USAGE=\
"Usage: $CMD_SCRIPT [-d] [command]
Commands
post-meta <post-id> - Get wp-postmeta data for <post-id>
woo-order-meta <post-id> - Get woocomerce_order_items dat for <post-id>
wp-autoload-options - List autoload options.
Options:
-d|--debug - Debug
CMD Version $CMD_VERSION
WPST Version $WPST_VERSION
"

usage () {
echo "$USAGE"
}

wpst_init () {
if ! command -v wp > /dev/null;then
echo "wp-cli not installed"
exit 1
else
_debug "wp-cli is installed"
fi
}

get_table_prefix () {
WP_TABLE_PREFIX=$(wp db prefix)
_debug "WP_TABLE_PREFIX=$WP_TABLE_PREFIX"
}

sql_post_meta () {
get_table_prefix
POST_ID=$1
_debug "SQL: wp sql query \" SELECT * from ${WP_TABLE_PREFIX}postmeta where post_id = '$POST_ID'"
wp sql query "SELECT * from ${WP_TABLE_PREFIX}postmeta where post_id = '$POST_ID'\""
}

sql_woo_order_meta () {
get_table_prefix
POST_ID=$1
#wp sql query "SELECT * FROM wp_bspr_woocommerce_order_items where order_id = '106818'" | less
#wp sql query "SELECT * FROM wp_bspr_woocommerce_order_itemmeta where order_item_id = '78807'" | less
_debug "SQL: wp sql query \"SELECT * from ${WP_TABLE_PREFIX}postmeta where post_id = '$POST_ID'\" | tr '\t' ','"
OUTPUT=$(wp sql query "SELECT * from ${WP_TABLE_PREFIX}postmeta where post_id = '$POST_ID'" | tr '\t' ',')
HEADERS=$(head -1 $OUTPUT)
echo $HEADERS
}
sql_wp_autoload_options () {

_debug "wp sql query \"SELECT * from ${WP_TABLE_PREFIX}options where autoload = 'yes' and option_name like '_transient%';\""
wp sql query "SELECT * from ${WP_TABLE_PREFIX}options where autoload = 'yes' and option_name like '_transient%'l"
}

poop () {
PANTS=$1
echo "Found poop in $PANTS"
}

# ---------------
# -- Main Program
# ---------------
DCMD=""
POSITIONAL=()
while [[ $# -gt 0 ]]
do
key="$1"
case $key in
-d|--debug)
DEBUG_ON="1"
DCMD+="DEBUG_ON=1 "
shift # past argument
;;
*) # unknown option
POSITIONAL+=("$1") # save it in an array for later
shift # past argument
;;
esac
done
set -- "${POSITIONAL[@]}" # restore positional parameters

_debug "Running wpst-goaccess $DCMD"
_debug_all $@


ACTION=$1
EXTRA=$2
_debug "ACTION=$ACTION EXTRA=$EXTRA"

wpst_init

if [[ -z $ACTION ]]; then
usage
else
case $ACTION in
post-meta)
if [[ -z $EXTRA ]]; then _error "Missing post_id";usage;exit 1;fi
sql_post_meta $EXTRA
;;
woo-order-meta)
if [[ -z $EXTRA ]]; then _error "Missing post_id";usage;exit 1;fi
sql_woo_order_meta $EXTRA
;;
wp-autoload-options)
sql_wp_autoload_options
;;
esac
fi
46 changes: 46 additions & 0 deletions wpst.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
#!/bin/bash
# --------------------------
# Lazy way to provide help and a wrapper to related tools.
# -------------------------
WPST_VERSION=0.7.1
SCRIPT_NAME=wpst
DEBUG_ON="0"

# -----------
# -- Includes
# -----------
. $(dirname "$0")/functions.sh

# ------------
# -- Variables
# ------------

# -- USAGE
USAGE=\
"WordPress Shell Tools - $VERSION
Commands:
wpst-goaccess - Run goaccess on web server logs
wpst-sql - Run common WordPress SQL Commands
wpst help - This help.
WPSET Version $WPST_VERSION
"

# -- usage
usage () {
echo "$USAGE"
}

# ------------
# -- Main Loop
# ------------
ACTION="$2"
if [[ ! $ACTION ]]; then
usage
exit 1
elif [[ $ACTION = "help" ]]; then
usage
exit 1
fi

0 comments on commit 3d7b9ae

Please sign in to comment.