-
Notifications
You must be signed in to change notification settings - Fork 1
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Added in more help information at top of cron-shim.sh script as well as offered outputting to stdout. Fixed requriement for bc and only if available otherwise use awk Updated GridPane Nginx log format, missed request time which broke everything after words which included OS and referrer. Nuked git commit history as rebase would have taken ages.
0 parents
commit 3d7b9ae
Showing
40 changed files
with
5,511 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,4 @@ | ||
.development | ||
.gptoken | ||
.debug | ||
.test |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,103 @@ | ||
<!--ts--> | ||
Table of Contents | ||
================= | ||
|
||
* [Table of Contents](#table-of-contents) | ||
* [wp-shelltools](#wp-shelltools) | ||
* [Rename from gp-tools to wp-shelltools](#rename-from-gp-tools-to-wp-shelltools) | ||
* [Donate](#donate) | ||
* [Install](#install) | ||
* [Requirements](#requirements) | ||
* [Help](#help) | ||
* [Help Command](#help-command) | ||
* [Advanced Documentation](#advanced-documentation) | ||
* [<a href="docs/gp-goaccess.md">gp-goaccess</a>](#gp-goaccess) | ||
* [<a href="docs/gp-api.md">gp-api</a>](#gp-api) | ||
* [<a href="docs/gp-plugins.md">gp-plugins</a>](#gp-plugins) | ||
* [<a href="docs/attackscan.md">attackscan.sh</a>](#attackscansh) | ||
* [Debugging](#debugging) | ||
* [Future Features](#future-features) | ||
* [Donate](#donate-1) | ||
* [Resources](#resources) | ||
<!--te--> | ||
|
||
# wp-shelltools | ||
The goal of this project is to make it easy to manage and maintain WordPress servers using shell scripting. This repository was original named gp-tools, but decided to make it provider agnostic | ||
|
||
You might find that there are still tools and functions that have gridpane or gp in their name. This will change when it becomes a priority. | ||
|
||
# Donate | ||
[](https://wpinfo.net/sponsor/) | ||
|
||
# Install | ||
## Requirements | ||
* You need root, otherwise this makes no sense. | ||
* You need git (apt-get install git) | ||
|
||
It's simple, make sure you have git installed (apt-get install git) and then run the following | ||
``` | ||
git clone https://github.com/jordantrizz/gpcli.git ~/gpcli;~/gpcli/gpcli -c install | ||
``` | ||
|
||
# Help | ||
Simply run the following command to get a list of available tools | ||
``` | ||
wp-shelltools help | ||
``` | ||
Here's a list of all commands | ||
``` | ||
-- Loading wp-shelltools - v0.0.1 | ||
wp-shelltools help | ||
----------------------------------- | ||
help goaccess - Process GridPane logs with goaccess | ||
help backups - List backups for all sites on the server. | ||
help plugins - Lists WordPress plugins on all websites on a GridPane Server | ||
help logcode - Look for specifc HTTP codes in web server logfiles and return top hits. | ||
help gpcron - List sites using GP Cron | ||
help mysqlmem - GridPane monit memory calculation | ||
help logs - tail or show last lines on all GridPane logs. | ||
Examples: | ||
-- | ||
wp-shelltools goaccess | ||
wp-shelltools log | ||
``` | ||
|
||
# Directories | ||
* dawn-mover - GridPane's prime-mover recoded but broken, it's a mess. | ||
* docs - Advanced documentation. | ||
* tests - any test data. | ||
* unfinished - code that is untested or needs to be moved into core. | ||
|
||
# Advanced Documentation | ||
Below is help for the advanced tools. | ||
* [wpst-goaccess](docs/wpst-goaccess.md) | ||
* [gp-api](docs/gp-api.md) | ||
* [gp-plugins](docs/gp-plugins.md) | ||
* [attackscan.sh](docs/attackscan.md) | ||
* [dawn-mover](dawn-mover/README.md) | ||
|
||
# Debugging | ||
Add a .debug file to any directory to get debug information. | ||
|
||
# Future Features | ||
* Check Log Sizes - debug.log is a culprit in disk usage. Find all occurances and print out size. | ||
* MySQL Database Checks - Check for anomiles, myisam, large table sizes, large databases. | ||
* MySQL Database Report - Largest Databases + more. | ||
* CLI to API - A CLI to the GridPane API. | ||
|
||
# ToDo | ||
* Go through unfinished directory code. | ||
* Better documentation | ||
* Place some scripts into functions or separate files not inside root for a cleaner directory structure and code management. | ||
|
||
# Donate | ||
[](https://managingwp.io/sponsor/) | ||
|
||
# Resources | ||
* [Managing WP](https://mangingwp.io) - WordPress Articles and more! | ||
* [Managing WordPress](https://www.facebook.com/groups/managingwordpress) - Created and managed by me. | ||
* [Managing WordPress Discord](https://discord.gg/QCsHM234zh) - Come and chat about WordPress | ||
* [GridPane Facebook Group](https://www.facebook.com/groups/selfmanagedwordpress) - Managed by GridPane and full of customers and WordPress community members. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
0.8.0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,118 @@ | ||
#!/bin/bash | ||
# -- Variables | ||
VERSION="0.0.3" | ||
SCRIPT="attackscan.sh" | ||
OLS_LOG_DIR="/var/log/ols" | ||
NGINX_LOG_DIR="/var/log/nginx" | ||
|
||
# -- _error | ||
_error () { | ||
echo "## ERROR $@" | ||
} | ||
|
||
# -- usage | ||
usage () { | ||
echo "$SCRIPT [-top <lines>|-scan]" | ||
echo "Version: $VERSION" | ||
echo "" | ||
echo "Parses Nginx and OLS web server access logs to find top number of requests and common attack requests for WordPress." | ||
echo "" | ||
echo "Commands:" | ||
echo " -top -List top number of requests from the webserver access log." | ||
echo " -scan -List common attack requests that return a 200 status code, by IP address." | ||
echo "" | ||
echo "Options:" | ||
echo " <lines> -How many lines to show, if not specified defaults to 10" | ||
echo "" | ||
echo "Examples" | ||
echo " $SCRIPT -top 20" | ||
echo " $SCRIPT -scan" | ||
echo "" | ||
} | ||
|
||
# -- top-ols $SERVER $LOGS $LINES | ||
top () { | ||
SERVER=$1 | ||
LOGS=$2 | ||
LINES=$3 | ||
echo "Server: $SERVER - Logs: $LOGS - Lines: $LINES" | ||
if [[ $SERVER == "ols" ]]; then | ||
LOG_FILES=($(ls ${LOGS}/*.access.log)) | ||
for SITE in "${LOG_FILES[@]}"; do | ||
echo "** Parsing ${SITE} for top ${LINES} requests" | ||
cat ${SITE} | awk {' print $7 '} | sort | uniq -c | sort -nr | head -n ${LINES} | ||
echo "======================" | ||
done | ||
elif [[ $SERVER == "nginx" ]]; then | ||
LOG_FILES=($(ls ${LOGS}/*.access.log)) | ||
for SITE in "${LOG_FILES[@]}"; do | ||
echo "** Parsing ${SITE} for top ${LINES} requests" | ||
cat ${SITE} | awk {' print $8 '} | sort | uniq -c | sort -nr | head -n ${LINES} | ||
echo "======================" | ||
done | ||
else | ||
_error "Something went wrong" | ||
fi | ||
} | ||
|
||
scan () { | ||
SERVER=$1 | ||
LOGS=$2 | ||
LINES=$3 | ||
|
||
echo "Server: $SERVER - Logs: $LOGS - Lines: $LINES" | ||
if [[ $SERVER == "ols" ]]; then | ||
LOG_FILES=($(ls ${LOGS}/*.access.log)) | ||
for SITE in "${LOG_FILES[@]}"; do | ||
echo "** Parsing ${SITE}" | ||
cat ${SITE} | awk {' print $7 "," $1 "," $9 '} | sed 's/"//' | grep -v "redirect_to" | grep "200" | egrep -e "xmlrpc|wp-login" | sort | uniq -c | sort -nr | head -n ${LINES} | ||
echo "======================" | ||
done | ||
elif [[ $SERVER == "nginx" ]]; then | ||
LOG_FILES=($(ls ${LOGS}/*.access.log)) | ||
for SITE in "${LOG_FILES[@]}"; do | ||
echo "** Parsing ${SITE}" | ||
cat ${SITE} | awk {' print $3 "," $8 "," $10 '} | sed 's/"//' | grep -v "redirect_to" | grep "200" | egrep -e "xmlrpc|wp-login" | sort | uniq -c | sort -nr | head -n ${LINES} | ||
echo "======================" | ||
done | ||
fi | ||
} | ||
|
||
# -- main | ||
if [[ -z $1 ]]; then | ||
usage | ||
exit 1 | ||
fi | ||
|
||
# -- logs to process | ||
if [[ -d $OLS_LOG_DIR ]]; then | ||
echo " -- Found OLS logs at /var/log/ols" | ||
SERVER="ols" | ||
LOGS="/var/log/ols" | ||
elif [[ -d /var/log/nginx ]]; then | ||
echo " -- Found Nginx logs at /var/log/nginx" | ||
SERVER="nginx" | ||
LOGS="/var/log/nginx" | ||
else | ||
_error "Didn't find any logs" | ||
exit 1 | ||
fi | ||
|
||
# -- lines | ||
LINES="" | ||
if [[ -n $2 ]]; then | ||
LINES="$2" | ||
else | ||
LINES="10" | ||
fi | ||
|
||
# -- top command | ||
if [[ $1 == "-top" ]]; then | ||
echo " -- Showing top $LINES requests per log file" | ||
top $SERVER $LOGS $LINES | ||
# -- scan command | ||
elif [[ $1 == "-scan" ]]; then | ||
echo " - Running a scan for common attack requests" | ||
scan $SERVER $LOGS $LINES | ||
fi | ||
|
Submodule dawn-mover
added at
269ee6
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,95 @@ | ||
# attackschan.sh | ||
``` | ||
attackscan.sh [-top <lines>|-scan] | ||
Version: 0.0.3 | ||
Parses Nginx and OLS web server access logs to find top number of requests and common attack requests for WordPress. | ||
Commands: | ||
-top -List top number of requests from the webserver access log. | ||
-scan -List common attack requests that return a 200 status code, by IP address. | ||
Options: | ||
<lines> -How many lines to show, if not specified defaults to 10 | ||
Examples | ||
attackscan.sh -top 20 | ||
attackscan.sh -scan | ||
``` | ||
# Commands | ||
## -top <lines> | ||
* List top number of requests from the webserver access log. | ||
### Example -top Output | ||
``` | ||
** Parsing /var/log/ols/domain.com.access.log for top 10 requests | ||
467 /wp-login.php | ||
353 / | ||
97 /xmlrpc.php | ||
93 /robots.txt | ||
78 //xmlrpc.php | ||
63 /robots.txt/ | ||
32 /favicon.ico | ||
30 //wp-json/wp/v2/users/ | ||
30 //wp-json/oembed/1.0/embed?url=https://workingefficiently.com/ | ||
30 //?author=1 | ||
``` | ||
## -scan <lines> | ||
* List common attack requests that return a 200 status code, by IP address. | ||
### Example -scan Output | ||
``` | ||
** Parsing /var/log/ols/domain.com.access.log for top common attack requests serving 200 status code | ||
9 //xmlrpc.php,143.198.225.22,200 | ||
9 /wp-login.php,185.119.81.105,200 | ||
8 //xmlrpc.php,20.168.90.200,200 | ||
8 /wp-login.php,185.119.81.108,200 | ||
6 //xmlrpc.php,20.168.90.200,503 | ||
6 /wp-login.php,185.119.81.97,200 | ||
6 /wp-login.php,185.119.81.102,200 | ||
6 /wp-login.php,185.119.81.100,200 | ||
4 //xmlrpc.php,79.110.62.40,200 | ||
4 //xmlrpc.php,208.67.105.86,200 | ||
``` | ||
# Running attackscan.sh | ||
* Log into your server via SSH | ||
* Run one of the following commands | ||
## Run one time via bash+curl | ||
You can run attachscan.sh one time using the following shell command, it won't download anything to your server. | ||
``` | ||
bash <(curl -sL https://raw.githubusercontent.com/managingwp/gp-tools/main/attackscan.sh) | ||
``` | ||
You should see the help page output, now you can append a command to the end of the shell command. | ||
``` | ||
bash <(curl -sL https://raw.githubusercontent.com/managingwp/gp-tools/main/attackscan.sh) -top | ||
``` | ||
## Download and Run | ||
If you want to keep a copy on your server. | ||
``` | ||
cd /usr/local/sbin; wget "https://raw.githubusercontent.com/managingwp/gp-tools/main/attackscan.sh"; chmod u+x /usr/local/sbin/attackscan.sh | ||
``` | ||
You can now simply type the following at anytime. | ||
``` | ||
attackscan.sh | ||
``` | ||
|
||
# FAQ | ||
## Why would I use this script? | ||
This should will help identify requests that could potentially be causing an increase in resource usage on your server. Or highlight potential common attacks that spike resource usage, effectively causing your server to be at capacity prematurely. | ||
## Why did you create this script? | ||
I required a quick way to see what were the top requests being served, and to confirm if common resource based attacks on WordPress were being carried out due to no preventative measures being in place. | ||
## Do you support other platforms beside GridPane | ||
Yes, simply create an issue with the platform and we'll look into support it. | ||
|
||
# Documentation | ||
## Parsing Method for -scan command | ||
The parsing method for -scan is rather simple using grep. Here is the full command. | ||
``` | ||
grep -v "redirect_to" | grep "200" | egrep -e "xmlrpc|wp-login" | ||
``` | ||
As you can see "redirect_to" is parsed out because the request was answered and redirecting, versus a direct POST request to the wp-login page. | ||
## Log File Formats | ||
### GP OLS | ||
* ```logFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"``` | ||
* ```"192.168.1.1 - - [01/Sep/2022:20:16:55 -0700] "GET /favicon.ico HTTP/1.1" 200 103639 "https://domain.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"``` | ||
### GP Nginx | ||
* ```'[$time_local] $remote_addr $upstream_response_time $upstream_cache_status $http_host "$request" $status $body_bytes_sent $request_time "$http_referer" "$http_user_agent" "$http3"';``` | ||
* ```[01/Sep/2022:20:20:07 -0700] 192.168.1.1 1.360 - domain.com "GET /favicon.ico HTTP/2.0" 302 0 1.362 "https://domain.com" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"``` |
Empty file.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,57 @@ | ||
# gpplugin.sh | ||
The following plugin will allow you accomplish two tasks | ||
* List all plugins for all sites. | ||
* List the status of a specific plugin. | ||
|
||
# Donate | ||
Coffee, Beer or steaped Tea! Feel free to buy me one :0 | ||
|
||
<a href="https://wpinfo.net/sponsor/" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/default-orange.png" alt="Buy Me A Coffee" height="41" width="174"></a> | ||
|
||
# Install | ||
You can refer to [README.md](README.md) to install all the tools. | ||
|
||
or | ||
|
||
Run the following! | ||
|
||
``` | ||
wget https://raw.githubusercontent.com/jordantrizz/gpcli/main/gpplugin.sh | ||
``` | ||
|
||
# Usage | ||
You can run the following command to see the help screen. | ||
``` | ||
./gpplugin.sh | ||
``` | ||
|
||
You'll be presented with the following | ||
|
||
``` | ||
❯ ./gpplugin.sh | ||
Lists WordPress plugins on all websites on a GridPane Server | ||
Syntax: gpplugin -a <plugin> | ||
options | ||
-a List all plugins | ||
-p Status on specific plugin | ||
Notes: | ||
* Find out which sites have which plugins + search for specific plugins and print wp-cli plugin status | ||
* Excludes canary + staging + 22222 + nginx + *.gridpanevps.com sites | ||
* --skip-plugins is run as to not fail potentially due to an error with a plugin | ||
Updates: https://github.com/jordantrizz/gpcli | ||
``` | ||
|
||
## List all Plugins | ||
If you want to list all plugins simply run the following. | ||
|
||
``` | ||
gpplugin.sh -a | ||
``` | ||
## Statis on Specific Plugin | ||
If you want the status on a specific plugin! | ||
``` | ||
gpplugin.sh -p elementor | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
# Setting up Netdata Properly on GridPane. | ||
See wpinfo.net |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
# Ideas | ||
* https://bidhankhatri.com.np/monitoring/monitoring-php-fpm-pools-through-prometheus-and-grafana-on-centos-6-and-7/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,26 @@ | ||
Welcome GridPane'rs! | ||
|
||
<a href="https://www.buymeacoffee.com/jordantrask" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/default-orange.png" alt="Buy Me A Coffee" height="41" width="174"></a> | ||
|
||
# How to use goacecss with GridPane | ||
## Installation | ||
``` | ||
echo "deb https://deb.goaccess.io/ $(lsb_release -cs) main" | sudo tee -a /etc/apt/sources.list.d/goaccess.list | ||
wget -O - https://deb.goaccess.io/gnugpg.key | sudo apt-key --keyring /etc/apt/trusted.gpg.d/goaccess.gpg add - | ||
sudo apt-get update | ||
sudo apt-get install goaccess | ||
``` | ||
|
||
## Log Format | ||
``` | ||
goaccess --log-format='[%d:%t %^] %h %^ - %v "%r" %s %b "%R" "%u"' --date-format='%d/%b/%Y' --time-format='%H:%M:%S %Z' | ||
``` | ||
|
||
## Configuration | ||
If you don't want to use the command above, you can configure goaccess globally to use the abouve settings | ||
* Edit /etc/goaccess/goaccess.conf and update the following lines. | ||
``` | ||
date-format %d/%b/%Y | ||
time-format %H:%M:%S %Z | ||
log-format [%d:%t %^] %h %^ - %v "%r" %s %b "%R" "%u" | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,221 @@ | ||
# --------------- | ||
# -- functions.sh | ||
# --------------- | ||
|
||
# ----------- | ||
# -- Includes | ||
# ----------- | ||
|
||
# ------------ | ||
# -- Variables | ||
# ------------ | ||
SCRIPT=$(readlink -f "$0") | ||
SCRIPTPATH=$(dirname "$SCRIPT") | ||
REQUIRED_APPS=("jq" "column") | ||
if [[ $DEBUG_ON="1" ]]; then DEBUG="1"; fi | ||
|
||
# -- Colors | ||
RED="\033[0;31m" | ||
GREEN="\033[0;32m" | ||
CYAN="\033[0;36m" | ||
BLUEBG="\033[0;44m" | ||
YELLOWBG="\033[0;43m" | ||
GREENBG="\033[0;42m" | ||
DARKGREYBG="\033[0;100m" | ||
ECOL="\033[0;0m" | ||
|
||
# ---------------- | ||
# -- Core Functions | ||
# ---------------- | ||
|
||
_error () { echo -e "${RED}** ERROR ** - ${*} ${ECOL}"; } | ||
_warning () { echo -e "${RED}** WARNING ** - ${*} ${ECOL}"; } | ||
_notice () { echo -e "${BLUE}** NOTICE ** - ${*} ${ECOL}"; } | ||
_success () { echo -e "${GREEN}** SUCCESS ** - ${*} ${ECOL}"; } | ||
_running () { echo -e "${BLUEBG}${@}${ECOL}"; } | ||
_creating () { echo -e "${DARKGREYBG}${@}${ECOL}"; } | ||
_separator () { echo -e "${YELLOWBG}****************${ECOL}"; } | ||
|
||
# -- _debug | ||
_debug () { | ||
if [[ $DEBUG_ON == "1" ]]; then | ||
echo -e "${CYAN}** DEBUG: ${*}${ECOL}" | ||
fi | ||
} | ||
|
||
# -- show debug information | ||
_debug_all () { | ||
_debug "--------------------------" | ||
_debug "arguments: ${*}" | ||
_debug "funcname: ${FUNCNAME[@]}" | ||
_debug "basename: $SCRIPTPATH" | ||
_debug "sourced files: ${BASH_SOURCE[@]}" | ||
_debug "--------------------------" | ||
} | ||
|
||
# -- debug curl | ||
_debug_curl () { | ||
if [[ $DEBUG == "2" ]]; then | ||
echo -e "${CCYAN}**** DEBUG ${*}${NC}" | ||
fi | ||
} | ||
|
||
# -- _cexists -- Returns 0 if command exists or 1 if command doesn't exist | ||
_cexists () { | ||
if [[ "$(command -v $1)" ]]; then | ||
_debug $(which $1) | ||
if [[ $ZSH_DEBUG == 1 ]]; then | ||
_debug "${*} is installed"; | ||
fi | ||
CMD_EXISTS="0" | ||
else | ||
if [[ $ZSH_DEBUG == 1 ]]; then | ||
_debug "${*} not installed"; | ||
fi | ||
CMD_EXISTS="1" | ||
fi | ||
return $CMD_EXISTS | ||
} | ||
|
||
# -- Check root | ||
_checkroot () { | ||
if [ ! -f .debug ]; then | ||
if [ "$EUID" -ne 0 ] | ||
then echo "Please run as root" | ||
exit | ||
fi | ||
fi | ||
} | ||
|
||
# ------------------------------ | ||
# -- GridPane specific functions | ||
# ------------------------------ | ||
|
||
# - logs | ||
help_cmd[logs]='tail or show last lines on all GridPane logs.' | ||
tool_logs () { | ||
gp-logs.sh ${*} | ||
} | ||
|
||
# - logcode | ||
help_cmd[logcode]='Look for specifc HTTP codes in web server logfiles and return top hits.' | ||
tool_logcode () { | ||
gp-logcode.sh ${*} | ||
} | ||
|
||
# - mysqlmem | ||
help_cmd[mysqlmem]='GridPane monit memory calculation' | ||
tool_mysqlmem () { | ||
gp-mysqlmem.sh ${*} | ||
} | ||
|
||
# - plugins | ||
help_cmd[plugins]='Lists WordPress plugins on all websites on a GridPane Server' | ||
tool_plugins () { | ||
gp-plugins.sh ${*} | ||
} | ||
|
||
# - goaccess - execute log functions | ||
help_cmd[goaccess]='Process GridPane logs with goaccess' | ||
tool_goaccess () { | ||
_debug_all ${*} | ||
|
||
if [[ -z $2 ]]; then | ||
echo "Usage: goaccess -c [domain.com|all]" | ||
echo " Commands" | ||
echo " domain - Domain name of log files to process" | ||
echo " all-domains - Go through all the logs versus a single domain" | ||
echo "" | ||
echo " Options:" | ||
echo " -c - Process compressed log files" | ||
echo "" | ||
return | ||
fi | ||
|
||
_debug "Checking if goaccess is installed" | ||
_cexists goaccess | ||
_debug "\$CMD_EXISTS: $CMD_EXISTS" | ||
if [[ $CMD_EXISTS == "1" ]]; then | ||
_error "goaccess is not installed" | ||
return 1 | ||
else | ||
_debug "Confirmed goaccess is installed" | ||
fi | ||
|
||
# Usage | ||
if [ -v $2 ]; then | ||
echo "Usage: $SCRIPT_NAME gp-goaccess [<domain.com>|-a]" | ||
return 1 | ||
fi | ||
|
||
# Formats for goaccess | ||
# OLS | ||
# logformat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" | ||
# "192.168.0.1 - - [13/Sep/2022:16:28:40 -0400] "GET /request.html HTTP/2" 200 46479 "https://domain.com/referer.html" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" | ||
OLS_LOG_FORMAT='\"%h - - [%d:%t %^] \"%r\" %s %b \"%R\" \"%u\"\' | ||
OLS_DATE_FORMAT='%d/%b/%Y\' | ||
OLS_TIME_FORMAT='%H:%M:%S %Z\' | ||
|
||
# NGINX | ||
# log_format we_log '[$time_local] $remote_addr $upstream_response_time $upstream_cache_status $http_host "$request" $status $body_bytes_sent $request_time "$http_referer" "$http_user_agent" "$http3"'; | ||
# [14/Sep/2022:10:12:55 -0700] 129.168.0.1 - domain.com "GET /request.html HTTP/1.1" 200 47 1.538 "https://domain.com/referer.html" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" | ||
NGINX_LOG_FORMAT='[%d:%t %^] %h %^ - %v \"%r\" %s %b \"%R\" \"%u\"\' | ||
NGINX_DATE_FORMAT='%d/%b/%Y\' | ||
NGINX_TIME_FORMAT='%H:%M:%S %Z\' | ||
|
||
_debug "goaccess arguments - $ACTION" | ||
_debug "goaccess LOG_FORMAT = $LOG_FORMAT ## DATE_FORMAT = $DATE_FORMAT ## TIME_FORMAT = $TIME_FORMAT" | ||
# -- Check args. | ||
|
||
_debug "Detecting log files" | ||
if [[ -d /usr/local/lsws ]]; then | ||
WEB_SERVER="OLS" | ||
LOG_FORMAT=$OLS_LOG_FORMAT | ||
DATE_FORMAT=$OLS_DATE_FORMAT | ||
TIME_FORMAT=$OLS_TIME_FORMAT | ||
if [[ $2 == "all" ]]; then | ||
LOG_FILE_LOCATION="/var/www/*/logs" | ||
LOG_FILTER="*.access.log" | ||
else | ||
LOG_FILE_LOCATION="/var/www/$2/logs" | ||
LOG_FILTER="*.access.log" | ||
fi | ||
elif [[ -d /var/log/nginx ]]; then | ||
WEB_SERVER="NGINX" | ||
LOG_FORMAT=$NGINX_LOG_FORMAT | ||
DATE_FORMAT=$NGINX_DATE_FORMAT | ||
TIME_FORMAT=$NGINX_TIME_FORMAT | ||
LOG_FILE_LOCATION="/var/log/nginx" | ||
|
||
if [[ $2 == "all" ]]; then | ||
LOG_FILTER="*.access.log" | ||
else | ||
LOG_FILTER="$2.access.log" | ||
fi | ||
else | ||
_error "Can't detect webserver logs" | ||
fi | ||
|
||
_debug "Webserver detected as $WEB_SERVER" | ||
|
||
# Main | ||
cat ${LOG_FILE_LOCATION}/${LOG_FILTER} | goaccess --log-format="$LOG_FORMAT" --date-format="$DATE_FORMAT" --time-format="$TIME_FORMAT" | ||
} | ||
|
||
# - gpcron | ||
help_cmd[gpcron]='List sites using GP Cron' | ||
tool_gpcron () { | ||
grep 'cron:true' /var/www/*/logs/*.env | ||
} | ||
|
||
# - backups - execute log functions | ||
help_cmd[backups]='List backups for all sites on the server.' | ||
tool_backups () { | ||
ls -aL /home/*/sites/*/logs/backups.env | xargs -l -I {} sh -c "echo {} | awk -F/ '{print \$5}'|tr '\n' '|'; tr '\n' '|' < {};echo \n" | ||
} | ||
|
||
# - api - GridPane api | ||
help_cmd[api]='Interact with the GridPane API' | ||
tool_api () { | ||
gp-api.sh ${*} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,80 @@ | ||
#!/bin/bash | ||
# -- Created by Jordan - hello@managingwp.io - https://managingwp.io | ||
# -- Version 1.0.0 -- Last Updated: 2020-05-09 | ||
# -- Purpose: Run WordPress crons via wp-cli and log the output to stdout, syslog, or a file. | ||
# -- Usage: Add the following to your crontab (replacing /path/to/wordpress with the path to your WordPress install): | ||
# */5 * * * * /home/systemuser/cron-shim.sh | ||
|
||
# Set up the necessary variables | ||
WP_CLI="/usr/local/bin/wp" # - Location of wp-cli | ||
WP_ROOT="/path/to/wordpress" # - Path to WordPress | ||
LOG_TO_STDOUT="1" # - Log to stdout? 0 = no, 1 = yes | ||
LOG_TO_SYSLOG="1" # - Log to syslog? 0 = no, 1 = yes | ||
LOG_TO_FILE="0" # - Log to file? 0 = no, 1 = yes | ||
LOG_FILE="${WP_ROOT}/wordpress-crons.log" # Location for wordpress cron. | ||
HEARTBEAT_URL="https://betteruptime.com/api/v1/heartbeat/v25v234v4634b636v3" # - Heartbeat monitoring URL | ||
POST_CRON_CMD="" # - Command to run after cron completes | ||
SCRIPT_DIR=$(dirname "$(realpath "$0")") | ||
|
||
# Check if wp-cli is installed | ||
[[ $(command -v $WP_CLI) ]] || { echo 'Error: wp-cli is not installed.' >&2; exit 1; } | ||
|
||
# Check if $WP_ROOT exists | ||
if [[ ! -d "$WP_ROOT" ]]; then | ||
if [[ -d $SCRIPT_DIR/htdocs ]]; then | ||
WP_ROOT=$SCRIPT_DIR/htdocs | ||
elif [[ -d $SCRIPT_DIR/public_html ]]; then | ||
WP_ROOT=$SCRIPT_DIR/public_html | ||
else | ||
echo "Error: $WP_ROOT does not exist." >&2; exit 1; | ||
fi | ||
fi | ||
|
||
# Check if $WP_ROOT contains a WordPress install | ||
[[ WP_ROOT_INSTALL=$(wp --path=$WP_ROOT --skip-plugins --skip-themes core is-installed 2> /dev/null) ]] || { echo "Error: $WP_ROOT is not a WordPress install.\n$WP_ROOT_INSTALL " >&2; exit 1; } | ||
|
||
# Get the domain name of the WordPress install | ||
DOMAIN_NAME=$($WP_CLI --path=$WP_ROOT --skip-plugins --skip-themes option get siteurl 2> /dev/null | grep -oP '(?<=//)[^/]+') | ||
[[ $? -ne 0 ]] && { echo "Error: Could not get domain name.\n$DOMAIN_NAME"; exit 1; } | ||
|
||
# Log the start time | ||
START_TIME=$(date +%s.%N) | ||
|
||
# Run WordPress crons due now and log the output | ||
CRON_OUTPUT=$($WP_CLI --skip-plugins --skip-themes cron event run --due-now --path=$WP_ROOT 2>&1) | ||
|
||
# Check if there was an error running wp-cli command | ||
if [[ $? -ne 0 ]]; then | ||
echo "Error: wp-cli command failed" >&2 | ||
echo "$CRON_OUTPUT" | ||
if [[ -n "$POST_CRON_CMD" ]]; then | ||
eval "$POST_CRON_CMD" | ||
fi | ||
exit 1 | ||
fi | ||
|
||
# Check if heartbeat monitoring is enabled and send a request to the heartbeat URL if it is and there are no errors | ||
if [[ -n "$HEARTBEAT_URL" ]] && [[ $? -eq 0 ]] ; then | ||
curl -I -s "$HEARTBEAT_URL" > /dev/null | ||
fi | ||
|
||
# Log the end time and CPU usage | ||
END_TIME=$(date +%s.%N) | ||
|
||
# check if bc installed otherwise use awk | ||
if [[ $(command -v bc) ]]; then | ||
TIME_SPENT=$(echo "$END_TIME - $START_TIME" | bc) | ||
else | ||
TIME_SPENT=$(echo "$END_TIME - $START_TIME" | awk '{printf "%f", $1 - $2}') | ||
fi | ||
CPU_USAGE=$(ps -p $$ -o %cpu | tail -n 1) | ||
|
||
# Check if logging to syslog is enabled | ||
if [[ $LOG_TO_STDOUT == "1" ]]; then | ||
echo -e "Cron job completed in $TIME_SPENT seconds with $CPU_USAGE% CPU usage. \nOutput: $CRON_OUTPUT" | ||
elif [[ $LOG_TO_SYSLOG == "1" ]]; then | ||
echo -e "Cron job completed in $TIME_SPENT seconds with $CPU_USAGE% CPU usage. \nOutput: $CRON_OUTPUT" | logger -t "wordpress-crons-$DOMAIN_NAME" | ||
elif [[ $LOG_TO_FILE == "1" ]]; then | ||
# Log to file in the WordPress install directory | ||
echo "$(date +"%Y-%m-%d %H:%M:%S") - WordPress crons completed in $TIME_SPENT seconds with $CPU_USAGE% CPU usage. \nOutput: $CRON_OUTPUT" >> $LOG_FILE | ||
fi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,7 @@ | ||
This is a test log! | ||
This is a test log! | ||
This is a test log! | ||
This is a test log! | ||
This is a test log! | ||
This is a test log! | ||
This is a test log! |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,24 @@ | ||
[14/Apr/2023:06:30:29 -0500] 192.168.1.1 - - plantsbymail.com "GET /product-category/trees/birch-trees/ HTTP/2.0" 200 31812 0.000 "https://plantsbymail.com/shop/delta-moonlight-crapemyrtle/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" | ||
[14/Apr/2023:06:30:29 -0500] 192.168.1.1 - - plantsbymail.com "GET /wp-content/uploads/2021/06/plants-by-mail-logo-tm.png HTTP/2.0" 304 0 0.000 "https://plantsbymail.com/product-category/trees/birch-trees/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" | ||
[14/Apr/2023:06:30:29 -0500] 192.168.1.1 - - plantsbymail.com "GET /wp-content/uploads/2023/02/sunshine-ligustrum-clipped1-445x500.jpg HTTP/2.0" 304 0 0.000 "https://plantsbymail.com/product-category/trees/birch-trees/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" | ||
[14/Apr/2023:06:30:29 -0500] 192.168.1.1 - - plantsbymail.com "GET /wp-content/uploads/2021/08/river-birch-landscape-1k.jpg HTTP/2.0" 200 327526 0.002 "https://plantsbymail.com/product-category/trees/birch-trees/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" | ||
[14/Apr/2023:06:30:29 -0500] 192.168.1.1 - - plantsbymail.com "GET /wp-content/plugins/woocommerce-google-merchant-center-customer-reviews/assets/image/google-reviews-badge-dark-background.png HTTP/2.0" 304 0 0.000 "https://plantsbymail.com/product-category/trees/birch-trees/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" | ||
[14/Apr/2023:06:30:29 -0500] 192.168.1.1 - - plantsbymail.com "GET /wp-content/uploads/2018/12/Payment-Icons.png HTTP/2.0" 304 0 0.000 "https://plantsbymail.com/product-category/trees/birch-trees/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" | ||
[14/Apr/2023:06:30:30 -0500] 192.168.20.1 1.584 - plantsbymail.com "GET /product-category/shrubs/camellias/?_pagination_alt=1 HTTP/2.0" 200 45246 1.584 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/103.0.5060.134 Safari/537.36" | ||
[14/Apr/2023:06:30:31 -0500] 10.0.0.1 0.244 - plantsbymail.com "POST /wp-cron.php?doing_wp_cron=1681471831.1777219772338867187500 HTTP/2.0" 200 20 0.245 "-" "WordPress/42; https://plantsbymail.com" | ||
[14/Apr/2023:06:30:32 -0500] 192.168.20.1 1.732 - plantsbymail.com "GET /?pwgc=1628918241 HTTP/2.0" 200 39563 1.731 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/103.0.5060.134 Safari/537.36" | ||
[14/Apr/2023:06:30:32 -0500] 10.0.0.1 0.972 - plantsbymail.com "GET /wp-json/woofunnels/v1/worker?1681471831 HTTP/2.0" 499 0 0.970 "-" "WordPress/42; https://plantsbymail.com" | ||
[14/Apr/2023:06:30:37 -0500] fc00::1 0.744 - plantsbymail.com "POST /?wpgb-ajax=refresh&_sort_by=title_asc&_stock_status=1 HTTP/2.0" 200 7374 0.743 "https://plantsbymail.com/brand/southern-living-plants/?_sort_by=title_asc&_stock_status=1" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.3 Safari/605.1.15" | ||
[14/Apr/2023:06:30:38 -0500] 192.168.1.1 - - plantsbymail.com "GET /shop/river-birch-tree/ HTTP/2.0" 200 43547 0.001 "https://plantsbymail.com/product-category/trees/birch-trees/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" | ||
[14/Apr/2023:06:30:38 -0500] 192.168.1.1 - - plantsbymail.com "GET /wp-content/uploads/2021/06/plants-by-mail-logo-tm.png HTTP/2.0" 304 0 0.000 "https://plantsbymail.com/shop/river-birch-tree/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" | ||
[14/Apr/2023:06:30:38 -0500] fc00::1 - - plantsbymail.com "GET /wp-content/uploads/2021/08/river-birch-3fl-dimensions-510x510.jpg HTTP/2.0" 200 21429 0.000 "https://plantsbymail.com/shop/river-birch-tree/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" | ||
[14/Apr/2023:06:30:38 -0500] 192.168.1.1 - - plantsbymail.com "GET /wp-content/uploads/2021/08/birch-river-birch-3fl-dimensions-510x510.jpg HTTP/2.0" 200 42353 0.000 "https://plantsbymail.com/shop/river-birch-tree/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" | ||
[14/Apr/2023:06:30:38 -0500] 192.168.1.1 - - plantsbymail.com "GET /wp-content/uploads/2021/08/river-birch-landscape-1k-510x510.jpg HTTP/2.0" 200 118038 0.002 "https://plantsbymail.com/shop/river-birch-tree/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" | ||
[14/Apr/2023:06:30:38 -0500] 192.168.1.1 - - plantsbymail.com "GET /wp-content/uploads/2021/08/river-birch-landscape-1k.jpg HTTP/2.0" 304 0 0.000 "https://plantsbymail.com/shop/river-birch-tree/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" | ||
[14/Apr/2023:06:30:38 -0500] 192.168.1.1 - - plantsbymail.com "GET /wp-content/uploads/2021/08/River-Birch-3FL-Clipped-510x510.jpg HTTP/2.0" 200 16648 0.000 "https://plantsbymail.com/shop/river-birch-tree/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" | ||
[14/Apr/2023:06:30:38 -0500] 172.16.0.1 - - plantsbymail.com "GET /product-category/trees/spruce/ HTTP/2.0" 200 32571 0.001 "https://plantsbymail.com/brand/first-editions/?gclid=CjwKCAjw8-OhBhB5EiwADyoY1ansy-J2_fcbxMpfN1E2wlN3vZqNIXMbCRtQSs83a6HcRxy04gTh8hoCfaIQAvD_BwE" "Mozilla/5.0 (iPhone; CPU iPhone OS 16_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/112.0.5615.46 Mobile/15E148 Safari/604.1" | ||
[14/Apr/2023:06:30:38 -0500] 172.16.0.1 - - plantsbymail.com "GET /wp-content/uploads/2021/06/plants-by-mail-logo-tm.png HTTP/2.0" 304 0 0.000 "https://plantsbymail.com/product-category/trees/spruce/" "Mozilla/5.0 (iPhone; CPU iPhone OS 16_3 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/112.0.5615.46 Mobile/15E148 Safari/604.1" | ||
[14/Apr/2023:06:30:38 -0500] 192.168.1.1 - - plantsbymail.com "GET /wp-content/uploads/2021/08/river-birch-landscape-1k-100x100.jpg HTTP/2.0" 200 4419 0.000 "https://plantsbymail.com/shop/river-birch-tree/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" | ||
[14/Apr/2023:06:30:38 -0500] 192.168.1.1 - - plantsbymail.com "GET /wp-content/uploads/2021/08/river-birch-3fl-dimensions-100x100.jpg HTTP/2.0" 200 2536 0.000 "https://plantsbymail.com/shop/river-birch-tree/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" | ||
[14/Apr/2023:06:30:38 -0500] 192.168.1.1 - - plantsbymail.com "GET /wp-content/uploads/2021/08/River-Birch-3FL-Clipped-100x100.jpg HTTP/2.0" 200 2009 0.000 "https://plantsbymail.com/shop/river-birch-tree/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" | ||
[14/Apr/2023:06:30:38 -0500] 192.168.1.1 - - plantsbymail.com "GET /wp-content/uploads/2021/08/river-birch-bark-1k-100x100.jpg HTTP/2.0" 200 3747 0.000 "https://plantsbymail.com/shop/river-birch-tree/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
gp-api.sh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,125 @@ | ||
#!/usr/bin/env bash | ||
# - gp-logcode | ||
. $(dirname "$0")/functions.sh | ||
_debug "Loading functions.sh" | ||
|
||
# -- | ||
# -- Variables | ||
# -- | ||
API_URL="https://my.gridpane.com/oauth/api/v1/$gp_api_endpoint" | ||
API_TOKEN=.gptoken | ||
gp_api_cmd=$(curl -s $gp_api_url --header 'Authorization: Bearer $gp_api_access_token') | ||
|
||
## - Color | ||
RED='\033[97;41m' | ||
GREEN='\033[97;42m' | ||
YELLOW='\033[30;103m' | ||
NC='\033[0m' # No Color | ||
SUCCESS="${GREEN}* SUCCESS *${NC}" | ||
ERROR="${RED}* ERROR *${NC}" | ||
|
||
# -- Variable Ingestion | ||
POSITIONAL=() | ||
while [[ $# -gt 0 ]] | ||
do | ||
key="$1" | ||
|
||
case $key in | ||
-v|--value) | ||
VALUE="$2" | ||
shift # past argument | ||
shift # past value | ||
;; | ||
-c|--command) | ||
COMMAND="$2" | ||
shift # past argument | ||
shift # past value | ||
;; | ||
*) # unknown option | ||
POSITIONAL+=("$1") # save it in an array for later | ||
shift # past argument | ||
;; | ||
esac | ||
done | ||
set -- "${POSITIONAL[@]}" # restore positional parameters | ||
|
||
if [[ -n $1 ]]; then | ||
unknown_options=$1 | ||
fi | ||
|
||
# -- | ||
# -- Functions | ||
# -- | ||
|
||
# -- usage | ||
usage () { | ||
echo "Usage: $SCRIPT_NAME <command> <variables>" | ||
echo "" | ||
echo " Commands" | ||
echo "" | ||
echo "servers -List servers" | ||
echo "sites -List sites" | ||
echo "test - Test if your token works" | ||
echo "" | ||
echo "version: $VERSION" | ||
echo"" | ||
} | ||
|
||
|
||
run_api () { | ||
api_output=$(eval curl -s --location --request $@ --header \'Authorization: Bearer $gp_api_access_token\') | ||
_debug "Running curl --location --request $@ --header 'Authorization: Bearer $gp_api_access_token'" | ||
if [ $? -ne 0 ]; then | ||
echo $api_output | ||
exit | ||
fi | ||
} | ||
|
||
check_gptoken () { | ||
if [ -f "$gp_api_token" ]; then | ||
echo -e "$SUCCESS GP Token file found $gp_api_access_token" | ||
source $gp_api_token | ||
_debug "GP Token: $gp_api_access_token" | ||
else | ||
echo -e "$ERROR GP Token file missing, please place your GridPane API Token in the file named .gptoken" | ||
exit | ||
fi | ||
} | ||
|
||
test_gptoken () { | ||
test_path="GET 'https://my.gridpane.com/oauth/api/v1/user'" | ||
run_api $test_path | ||
_debug $api_output | ||
api_output_test=$(echo $api_output | cut -c1-6) | ||
if [[ $api_output_test == '{"id":' ]]; then | ||
echo -e "$SUCCESS API Success! $NC" | ||
else | ||
echo -e "$ERROR API Failure...something is wrong! $NC" | ||
exit | ||
fi | ||
#api_test_output=$(echo $api_output | jq ' . | {name: .name, id: .id, email: .email}') | ||
} | ||
|
||
send_command () { | ||
echo "Send command" | ||
} | ||
|
||
# -- Start | ||
|
||
#if [[ -z $COMMAND ]]; then help;exit;fi | ||
#if [[ $COMMAND == 'test' ]]; then check_gptoken;test_gptoken; fi | ||
#if [[ -z $VALUE ]]; then echo "Specify a value"; exit; fi | ||
|
||
|
||
#if [[ $COMMAND == 'd2s' ]]; then | ||
# check_gptoken | ||
# run_api "GET https://my.gridpane.com/oauth/api/v1/site" | ||
# _debug $api_output | ||
# d2s_domain_id=$(echo $d2s_list) | ||
# _debug $d2s_domain_id | ||
#fi | ||
|
||
if [[ -z $1 ]]; then | ||
usage | ||
exit | ||
fi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,104 @@ | ||
#!/usr/bin/env bash | ||
|
||
# -- Intro | ||
# Created by Jordan v0.1 | ||
|
||
# -- Variables | ||
BACKUP_DIR="/var/www/backups/gpbksite" | ||
|
||
## - Color | ||
RED='\033[97;41m' | ||
GREEN='\033[97;42m' | ||
YELLOW='\033[30;103m' | ||
NC='\033[0m' # No Color | ||
SUCCESS="${GREEN}* SUCCESS *${NC}" | ||
ERROR="${RED}* ERROR *${NC}" | ||
|
||
# -- debug command line | ||
_debug () { | ||
if [ -f .debug ]; then | ||
echo -e "\n${YELLOW} Debug:${NC} $@" | ||
fi | ||
} | ||
_debug "Running $@" | ||
|
||
# -- Variable Ingestion | ||
POSITIONAL=() | ||
while [[ $# -gt 0 ]] | ||
do | ||
key="$1" | ||
|
||
case $key in | ||
-s|--site) | ||
SITE="$2" | ||
shift # past argument | ||
shift # past value | ||
;; | ||
-a|--all) | ||
ALLSITES="true" | ||
shift # past argument | ||
shift # past value | ||
;; | ||
-f|--files) | ||
DATABASE="true" | ||
shift # past argument | ||
shift # past value | ||
;; | ||
-d|--database) | ||
DATABASE="true" | ||
shift # past argument | ||
shift # past value | ||
;; | ||
*) # unknown option | ||
POSITIONAL+=("$1") # save it in an array for later | ||
shift # past argument | ||
;; | ||
esac | ||
done | ||
set -- "${POSITIONAL[@]}" # restore positional parameters | ||
|
||
if [[ -n $1 ]]; then | ||
unknown_options=$1 | ||
fi | ||
|
||
# -- Functions | ||
# echo commands run using | exe eval ls -al | ||
exe() { echo "\$ $@" ; "$@" ; } | ||
|
||
help () { | ||
echo "Syntax: gpbksite.sh [ -s <site> | -a ] [ -d | -f ]" | ||
echo "" | ||
echo "gpbksite.sh - Allows you to backup a website on GridPane to a .tar file" | ||
echo "" | ||
echo "Commands:" | ||
echo "-s <site> - Website to backup" | ||
echo "-a - Backup all sites" | ||
echo "" | ||
echo "Options:" | ||
echo "-f - Files Only" | ||
echo "-d - Databases Only" | ||
echo "-l - Location, defaults to /var/backups/gpbksite" | ||
echo "" | ||
} | ||
|
||
if [[ ! -z $SITE ]]; then | ||
echo "Backing up $SITE" | ||
elif [[ ! -z $ALLSITES ]]; then | ||
echo "Backing up all sites" | ||
else | ||
help;exit; | ||
fi | ||
|
||
|
||
|
||
#if [[ $COMMAND == 'test' ]]; then check_gptoken;test_gptoken; fi | ||
#if [[ -z $VALUE ]]; then echo "Specify a value"; exit; fi | ||
|
||
|
||
#if [[ $COMMAND == 'd2s' ]]; then | ||
# check_gptoken | ||
# run_api "GET https://my.gridpane.com/oauth/api/v1/site" | ||
# _debug $api_output | ||
# d2s_domain_id=$(echo $d2s_list) | ||
# _debug $d2s_domain_id | ||
#fi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,29 @@ | ||
#/usr/bin/env bash | ||
# Run gp-go-access <domain.com> | ||
# Example gp-goaccess testing.com | ||
|
||
# Variables | ||
LOG_FORMAT='[%d:%t %^] %h %^ - %v \"%r\" %s %b \"%R\" \"%u\"\' | ||
DATE_FORMAT='%d/%b/%Y\' | ||
TIME_FORMAT='%H:%M:%S %Z\' | ||
|
||
|
||
# -- Check args. | ||
if [[ -v $1 ]]; then | ||
echo "usage: gp-goaccess <domain.com>" | ||
return | ||
fi | ||
|
||
# -- check for goaccess binary | ||
if [[ _cexists goaccess == "1" ]]; then | ||
echo " -- goaccess installed" | ||
else | ||
echo " -- goaccess not installed" | ||
fi | ||
|
||
# Main | ||
if [ $1 = "-a" ]; then | ||
zcat /var/log/nginx/$2.access.log.*.gz | goaccess --log-format="$LOG_FORMAT" --date-format="$DATE_FORMAT" --time-format="$TIME_FORMAT" | ||
else | ||
cat /var/log/nginx/$1.access.log | goaccess --log-format="$LOG_FORMAT" --date-format="$DATE_FORMAT" --time-format="$TIME_FORMAT" | ||
fi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
gp-logcode.sh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,193 @@ | ||
#!/usr/bin/env bash | ||
# - gp-logcode | ||
. $(dirname "$0")/functions.sh | ||
_debug "Loading functions.sh" | ||
|
||
# -- getopts | ||
while getopts "c:m:l:d:r:ezx" option; do | ||
case ${option} in | ||
c) LOGCODE=$OPTARG ;; | ||
m) MODE=$OPTARG ;; | ||
l) LOGFILE=$OPTARG ;; | ||
d) DOMAIN=$OPTARG ;; | ||
r) RESULTS=$OPTARG ;; | ||
e) GP_EXCLUDE=1 ;; | ||
z) Z_INCLUDE=1 ;; | ||
x) DEBUG=1 ;; | ||
?) usage ;; | ||
esac | ||
done | ||
|
||
_debug "args: $@" | ||
_debug "getopts: \$LOGCODE=$LOGCODE \$MODE=$MODE \$LOGFILE=$LOGFILE \$DOMAIN=$DOMAIN \$RESULTS=$RESULTS \$GP_EXCLUDE=$GP_EXCLUDE \$Z_INCLUDE=$Z_INCLUDE" | ||
_debug_all | ||
|
||
|
||
# Usage | ||
usage () { | ||
echo "Usage: $SCRIPT_NAME -c <code> -m [all|domain|log] (-l <logfilename>|-d <domain>) [-r results] [-e]" | ||
echo "" | ||
echo "Commands:" | ||
echo " -c <code> = the http status code to locate number 4 = 4xx, 5 = 5xx, 404" | ||
echo " -m <mode> - Log Method" | ||
echo " - all = all logs" | ||
echo " - domain = domain logs" | ||
echo " - log = single log, requires -l" | ||
echo " -l <logfilename> - Specific log file when using -m log" | ||
echo " -d <domainname> - Domain logs when using -m domain" | ||
echo "" | ||
echo " Options" | ||
echo " -r [results] = number of results, optional, default 40" | ||
echo " -e = exclude GridPane, staging and canary." | ||
echo " -z = include compressed files" | ||
echo " -x = debug" | ||
} | ||
|
||
# -- check_logs | ||
check_logs () { # Nginx or OLS? | ||
_debug "check_logs()" | ||
_debug "Checking if logs exist" | ||
|
||
# Paths | ||
nginxlogs=/var/log/nginx | ||
olslogs=/var/log/ols | ||
|
||
if [ -d $nginxlogs ]; then | ||
_debug "Found nginx logs under $nginxlogs" | ||
LOGFILEDIR=$nginxlogs | ||
elif [ -d $olslogs ]; then | ||
_debug "Found ols logs under $olslogs" | ||
LOGFILEDIR=$olslogs | ||
elif [[ $TEST ]]; then | ||
_debug "In test mode, using test data" | ||
LOGFILEDIR=$SCRIPTPATH/tests | ||
else | ||
_error "No $nginxlogs or $olslogs directory" | ||
exit | ||
fi | ||
} | ||
|
||
process_options () { | ||
# Confirm exclude | ||
if [[ $GP_EXCLUDE == "-e" ]]; then | ||
_debug "argument -e set - exclude gp, staging and canary sites in $LOGFILEDIR/*" | ||
EXCLUDE="| egrep -v '/access.log$|staging|canary|gridpane|.gz'" | ||
else | ||
EXCLUDE="" | ||
fi | ||
|
||
# Set results if not defined. | ||
if [ -z $RESULTS ];then | ||
RESULTS="40" | ||
fi | ||
_debug "RESULTS=$RESULTS" | ||
} | ||
|
||
# -- main | ||
main () { | ||
_debug_function | ||
|
||
# Process options | ||
process_options | ||
|
||
if [[ $1 == "all" ]]; then | ||
# Mode all | ||
_debug "All logs mode" | ||
if [[ $Z_INCLUDE ]]; then | ||
_debug "Include compressed files" | ||
PROCESS_FILES=$(ls -aSd $LOGFILEDIR/* $EXCLUDE) | ||
_debug "PROCESS_FILES: $PROCESS_FILES" | ||
else | ||
_debug "Exclude compressed files" | ||
PROCESS_FILES=$(ls -aSd $LOGFILEDIR/*.log $EXCLUDE) | ||
_debug "PROCESS_FILES: $PROCESS_FILES" | ||
fi | ||
elif [[ $1 == "domain" ]]; then | ||
# Mode domain | ||
_debug "Domain logs mode" | ||
if [[ $Z_INCLUDE ]]; then | ||
_debug "Include compressed files" | ||
PROCESS_FILES=$(ls -aSd $LOGFILEDIR/$DOMAIN* $EXCLUDE) | ||
_debug "PROCESS_FILES: $PROCESS_FILES"4 | ||
else | ||
_debug "Exclude compressed files" | ||
_debug "ls -aSd $LOGFILEDIR/$DOMAIN*.log $EXCLUDE" | ||
PROCESS_FILES=$(ls -aSd $LOGFILEDIR/$DOMAIN*.log $EXCLUDE) | ||
_debug "PROCESS_FILES: $PROCESS_FILES" | ||
fi | ||
elif [[ $1 == "log" ]]; then | ||
# Mode log | ||
_debug "Single logfile mode - $LOGFILE" | ||
if [[ -f $LOGFILEDIR/$LOGFILE ]]; then | ||
_success "Log file exists - $logfile" | ||
PROCESS_FILES=$LOGFILEDIR/$LOGFILE | ||
else | ||
_error "Log file $LOGFILEDIR/$logfile doesn't exist" | ||
exit | ||
fi | ||
fi | ||
|
||
# logcode | ||
if [[ $LOGCODE -lt 10 ]]; then | ||
_debug "logcode is less than 10" | ||
logcode="$logcode[0-9][0-9]" | ||
fi | ||
|
||
for file in $PROCESS_FILES; do | ||
_debug "Processing $file" | ||
if [[ $Z_INCLUDE == "1" ]]; then | ||
_debug "Running -- zgrep \" $LOGCODE \" $file | awk '{ print $6" - "$10" - "$7" "$8" "$9}' | sort | uniq -c | sort -nr | head -$RESULTS)" | ||
content=$(zgrep "\" $LOGCODE " $file | awk '{ print $6" - "$10" - "$7" "$8" "$9}' | sort | uniq -c | sort -nr | head -$RESULTS) | ||
else | ||
_debug "Running -- grep \" $LOGCODE \" $file | awk '{ print $6" - "$10" - "$7" "$8" "$9}' | sort | uniq -c | sort -nr | head -$RESULTS)" | ||
content=$(grep "\" $LOGCODE " $file | awk '{ print $6" - "$10" - "$7" "$8" "$9}' | sort | uniq -c | sort -nr | head -$RESULTS) | ||
fi | ||
# Print content | ||
if [[ $content ]]; then | ||
echo "##### File: $file" | ||
echo "$content" | ||
echo "...more lines but limited to top $RESULTS" | ||
else | ||
_debug "No data in $file" | ||
fi | ||
done | ||
} | ||
|
||
# ------- | ||
# -- Main | ||
# ------- | ||
|
||
check_logs | ||
|
||
if [[ -z $LOGCODE ]] || [[ -z $MODE ]]; then | ||
usage | ||
_error "Need to specify -c <code> and -m <method>" | ||
exit | ||
fi | ||
|
||
if [[ $MODE == "all" ]]; then | ||
# Log all | ||
_debug "mode: $MODE" | ||
main all | ||
elif [[ $MODE == "domain" ]]; then | ||
# Log domain | ||
_debug "mode: $MODE" | ||
if [[ $DOMAIN ]]; then | ||
_debug "domain: $DOMAIN" | ||
main domain | ||
else | ||
_error "Need to specify domain via -d <domain>" | ||
fi | ||
elif [[ $MODE == "log" ]]; then | ||
# Log mode | ||
_debug "mode: $MODE" | ||
if [[ $LOGFILE ]]; then | ||
_debug "logfile: $LOGFILE" | ||
main log $LOGFILE | ||
else | ||
_error "Need to specify logfile via -l <logfile>" | ||
fi | ||
else | ||
usage | ||
fi | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
gp-logs.sh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,326 @@ | ||
#!/usr/bin/env bash | ||
# ---------- | ||
# -- gp-logs | ||
# ---------- | ||
|
||
# ------------ | ||
# -- functions | ||
# ------------ | ||
. $(dirname "$0")/functions.sh | ||
_debug "Loading functions.sh" | ||
|
||
# -- getopts | ||
while getopts ":el:c:" option; do | ||
case ${option} in | ||
l) LOGS=$OPTARG ;; | ||
c) COMMAND=$OPTARG ;; | ||
e) GP_EXCLUDE=1 ;; | ||
?) usage ;; | ||
esac | ||
done | ||
_debug "args: $@" | ||
_debug "getopts: \$LOGS=$LOGS \$COMMAND=$COMMAND \$GP_EXCLUDE=$GP_EXCLUDE" | ||
|
||
# ------------ | ||
# -- Variables | ||
# ------------ | ||
GPOPT="/opt/gridpane" | ||
GPLP="/opt/gridpane/logs" | ||
NGINX_LOG_PATH="/var/log/nginx" | ||
OLS_LOG_PATH="/usr/local/lsws/logs" | ||
|
||
(( TEST >= "1" )) && LOG_FILES+=("$SCRIPTPATH/tests/test.log") | ||
SYS_LOGS=("/var/log/syslog") | ||
NGINX_SERVER_LOGS=("$NGINX_LOG_PATH/error.log" "$NGINX_LOG_PATH/access.log") | ||
OLS_SERVER_LOGS=("$OLS_LOG_PATH/stderr.log" "$OLS_LOG_PATH/error.log" "$OLS_LOG_PATH/lsrestart.log") | ||
GP_LOGS=("$GPOPT/gpclone.log" "$GPOPT/gpdailyworker.log" "$GPOPT/gphourlyworker.log" "$GPOPT/gpworker.log") | ||
GP_BACKUP_LOGS=("$GPOPT/backup.log" "$GPOPT/backup.error.log" "$GPOPT/backups.monitoring.log") | ||
PHP_FPM_LOGS=("/var/log/php") | ||
|
||
# ------------ | ||
# -- debug all | ||
# ------------ | ||
_debug_all $@ | ||
|
||
# ------------ | ||
# -- functions | ||
# ------------ | ||
|
||
# -- Usage | ||
usage () { | ||
echo "Usage: $SCRIPT_NAME -c [tail|last|test] -l [all|web|system|gp|fpm] (-e)" | ||
echo "" | ||
echo "Command (-c):" | ||
echo " tail - Tail all logs" | ||
echo " last - Last 10 lines of all logs" | ||
echo " test - Test what logs will be processed" | ||
echo "" | ||
echo "Logs (-l):" | ||
echo " all - Include all logs." | ||
echo " web - Web logs only" | ||
echo " system - System log files only" | ||
echo " gp - GridPane core log files only" | ||
echo " gpbackup - GridPane backup logs only" | ||
echo "" | ||
echo "Options:" | ||
echo " -e - Exclude GridPane, staging, and canary" | ||
echo "" | ||
echo "" | ||
echo "Log files checked:" | ||
echo " System - ${SYS_LOGS[@]}" | ||
echo " OLS - ${OLS_LOGS[@]}" | ||
echo " NGINX - ${NGINX_LOGS[@]}" | ||
echo " GP - ${GP_LOGS[@]}" | ||
echo " GP Backup - ${GP_BACKUP_LOGS[@]}" | ||
echo "" | ||
} | ||
|
||
# -- start_logs | ||
start_logs () { | ||
echo "" | ||
read -p " -- Press [Enter] to start" | ||
|
||
|
||
} | ||
|
||
# -- collect_logs | ||
collect_logs () { | ||
_debug_function | ||
# all = Include all logs. | ||
# web = Web logs only | ||
# system = System log files only | ||
# gp = GridPane core log files only" | ||
# fpm = PHP-FPM log files only" | ||
|
||
if [[ $LOGS == "all" ]]; then | ||
_debug "Collect logs: all" | ||
collect_system_logs | ||
collect_webserver-logs | ||
collect_web-logs both | ||
collect_php-fpm-logs | ||
collect_gridpane | ||
collect_gridpane_backup | ||
elif [[ $LOGS == "web-access" ]];then | ||
collect_webserver-logs | ||
collect_web-logs access | ||
elif [[ $LOGS == "web-error" ]]; then | ||
collect_web-logs error | ||
collect_php-fpm-logs | ||
elif [[ $LOGS == "web-all" ]]; then | ||
collect_webserver-logs | ||
collect_web-logs both | ||
collect_php-fpm-logs | ||
elif [[ $LOGS == "system" ]]; then | ||
collect_system_logs | ||
elif [[ $LOGS == "gp" ]]; then | ||
collect_gridpane | ||
collect_gridpane_backup | ||
elif [[ $LOGS == "gpbackup" ]]; then | ||
collect_gridpane_backup | ||
else | ||
_error "Something borked... :(" | ||
fi | ||
|
||
} | ||
|
||
|
||
# -- system logs | ||
collect_system_logs () { | ||
# System log files | ||
echo "-- Checking System log files" | ||
for SYS_LOG in "${SYS_LOGS[@]}"; do | ||
echo -n " -- Checking $SYS_LOG" | ||
if [ -f $SYS_LOG ]; then | ||
_success " -- Found $SYS_LOG" | ||
LOG_FILES+=("$SYS_LOG") | ||
else | ||
_error " -- Didn't find $SYS_LOG" | ||
fi | ||
done | ||
} | ||
|
||
# -- collect_web-logs errors|access|both | ||
collect_web-logs () { | ||
_debug_function | ||
# Find logs | ||
if [ -d $NGINX_LOG_PATH ]; then | ||
_debug "Found nginx log directory" | ||
collect_nginx $1 | ||
elif [ -d $OLS_LOG_PATH ]; then | ||
_debug "Found OLS log directory" | ||
collect_ols $1 | ||
else | ||
_error "-- Didn't find access logs directory for OLS or Nginx" | ||
fi | ||
} | ||
|
||
# -- collect_access-logs-nginx -- grab Nginx log files. | ||
collect_nginx () { | ||
# Choose what logs to return based on $1 | ||
if [[ $1 == "error" ]] || [[ $1 == "access" ]]; then | ||
LOG_GREP="$1" | ||
elif [[ $1 == "both" ]]; then | ||
LOG_GREP="error|access" | ||
fi | ||
_debug "\$LOG_GREP:$LOG_GREP" | ||
|
||
if [[ $GP_EXCLUDE ]]; then | ||
_debug "Exclude GridPane access logs '/access.log$|staging|canary|gridpane|.gz'" | ||
NGINX_LOGS=$(ls -aSd $NGINX_LOG_PATH/* | egrep "$LOG_GREP" | egrep -v '/access.log$|staging|canary|gridpane|.gz' | tr '\n' ' ') | ||
else | ||
_debug "Including GridPane access logs '/access.log$|staging|canary|gridpane|.gz'" | ||
NGINX_LOGS=$(ls -aSd $NGINX_LOG_PATH/* | egrep "$LOG_GREP" | grep -v '.gz' | tr '\n' ' ') | ||
fi | ||
_debug "\$NGINX_LOGS = ${NGINX_LOGS}" | ||
|
||
if [[ -f $NGINX_LOGS ]]; then | ||
_error "Didn't find any Nginx Access logs" | ||
else | ||
_success " -- Found $NGINX_LOGS" | ||
fi | ||
LOG_FILES+=("$NGINX_LOGS") | ||
|
||
} | ||
|
||
collect_ols () { | ||
_debug "Not completed" | ||
echo "-- Not completed" | ||
} | ||
|
||
|
||
# -- collect_webserver-logs | ||
collect_webserver-logs () { | ||
_debug_function | ||
|
||
# Nginx logs | ||
if [[ -d $NGINX_LOG_PATH ]]; then | ||
echo " -- Checking for NGINX webserver logs" | ||
for LOG in "${NGINX_SERVER_LOGS[@]}"; do | ||
echo -n " -- Checking $LOG" | ||
if [ -f $LOG ]; then | ||
_success " -- Found $LOG" | ||
LOG_FILES+=("LOG") | ||
else | ||
_error " -- Didn't find $LOG" | ||
fi | ||
done | ||
else | ||
_error "-- Didn't find Nginx webserver logs" | ||
fi | ||
|
||
# OLS logs | ||
if [[ -d $OLS_LOG_PATH ]]; then | ||
echo "-- Checking for OLS web server logs" | ||
for LOG in "${OLS_SERVER_LOGS[@]}"; do | ||
echo -n " -- Checking $LOG" | ||
if [ -f $LOG ]; then | ||
_success " -- Found $LOG" | ||
LOG_FILES+=("$LOG") | ||
else | ||
_error " -- Didn't find $LOG" | ||
fi | ||
done | ||
else | ||
_error "-- Didn't find OLS web server logs" | ||
fi | ||
} | ||
|
||
collect_gridpane () { | ||
# GridPane specific log files | ||
echo "-- Checking for GridPane log files" | ||
for GP_LOG in "${GP_LOGS[@]}"; do | ||
echo -n " -- Checking $GP_LOG" | ||
if [ -f $GP_LOG ]; then | ||
_success " -- Found $GP_LOG" | ||
LOG_FILES+=("$GP_LOG") | ||
else | ||
_error " -- Didn't find $GP_LOG" | ||
fi | ||
done | ||
} | ||
|
||
|
||
collect_gridpane_backup () { | ||
# GridPane Backup specific log files | ||
echo "-- Checking for GridPane Backup log files" | ||
for GP_BACKUP_LOG in "${GP_BACKUP_LOGS[@]}"; do | ||
echo -n " -- Checking $GP_BACKUP_LOG" | ||
if [ -f $GP_BACKUP_LOG ]; then | ||
_success " -- Found $GP_BACKUP_LOG" | ||
LOG_FILES+=("$GP_BACKUP_LOG") | ||
else | ||
_error " -- Didn't find $GP_BACKUP_LOG" | ||
fi | ||
done | ||
} | ||
|
||
collect_php-fpm-logs () { | ||
# PHP FPM log files | ||
echo "-- Checking PHP FPM log files" | ||
|
||
if [ "$(find "$PHP_FPM_LOGS" -mindepth 1 -maxdepth 1 -not -name '.*')" ]; then | ||
_debug "Found PHP log directories" | ||
for name in $PHP_FPM_LOGS/*/*; do | ||
_debug "Found $name" | ||
done | ||
else | ||
_debug "Not including PHP FPM logs" | ||
fi | ||
# Old Method | ||
# Locate FPM logs. | ||
#for file in /var/log/php/*/fpm.log; do | ||
# _debug "fpm file - $file" | ||
# FPM_LOGS+=("$file") | ||
#done | ||
} | ||
|
||
# -- check_logs -- most likely not needed anymore | ||
check_logs () { | ||
# -- Check if there are any logs to run against. | ||
if [[ -z $LOG_FILES ]]; then | ||
_debug "\$LOG_FILES = $LOG_FILES" | ||
_error "No logs files found at all, exiting" | ||
exit 1 | ||
else | ||
_success " -- Found log files, continuing" | ||
_debug " -- Found log files - ${LOG_FILES[*]}" | ||
fi | ||
} | ||
|
||
|
||
# ------- | ||
# -- Main | ||
# ------- | ||
|
||
if [[ -z $COMMAND ]] || [[ -z $LOGS ]]; then | ||
_error "Requires -c and -l to proceed" | ||
usage | ||
exit 1 | ||
fi | ||
|
||
if [[ $COMMAND = 'tail' ]]; then | ||
echo " -- Starting to tail logs" | ||
collect_logs | ||
echo " -- Tailing files ${LOG_FILES[@]}" | ||
start_logs | ||
tail -f "${LOG_FILES[@]}" | ||
elif [[ $COMMAND = 'last' ]]; then | ||
echo " -- Starting to tail logs" | ||
collect_logs | ||
echo " -- Tailing last 50 lines of files $LOG_FILES" | ||
start_logs | ||
tail -n 50 "${LOG_FILES[@]}" | less | ||
elif [[ $COMMAND = 'test' ]]; then | ||
echo " -- Starting test" | ||
collect_logs | ||
echo " -- Running test to confirm log files to process" | ||
start_logs | ||
echo ${LOG_FILES[@]} | tr ' ' '\n' | ||
else | ||
_debug "args: $@" | ||
_error "No option provided to print out logs, choose either tail or last" | ||
usage | ||
exit 1 | ||
fi | ||
|
||
_success "Test!" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,23 @@ | ||
#!/bin/bash | ||
# | ||
# This one liner should work, you would need to replace your.site with your actual multisite primary domain: | ||
# sudo su - "$(/usr/local/bin/gp conf read sys-user -site.env "your.site" -q)" -c "/usr/local/bin/wp site list --field=url --path=/var/www/your.site/htdocs | xargs -i -n1 /usr/local/bin/wp cron event run --due-now --url={} --path=/var/www/your.site/htdocs" | ||
# | ||
# You would save that in some filepath and make it executable: | ||
# chmod +x /path/to/your/multisite-cron-script.sh | ||
# The script accepts two args, | ||
# multisite-cron-script.sh {primary.domain:string} {seconds:integer} | ||
# | ||
# Then create a cronjob in your cron tab, passing in your site domain and seconds to sleep. | ||
# It defaults to 1 second if you omit the second arg. Here is an example to run every 5 minutes, with 5 seconds delay between subsites crons, and outputting to a log. | ||
# | ||
# 5 * * * * /path/to/your/multisite-cron-script.sh your.site 2 >>/var/log/multisite-cron.log | ||
|
||
default_sleep=1 | ||
multi_site="$1" | ||
sleep_for_seconds="${2:-$default_sleep}" | ||
sys_user=$(/usr/local/bin/gp conf read sys-user -site.env "${multi_site}" -q) | ||
for site in $(sudo su - "${sys_user}" -c "/usr/local/bin/wp site list --field=url --path=/var/www/${multi_site}/htdocs"); do | ||
sudo su - "${sys_user}" -c "/usr/local/bin/wp cron event run --due-now --url=${site} --path=/var/www/${multi_site}/htdocs" | ||
sleep ${sleep_for_seconds} | ||
done |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
gp-mysqlmem.sh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,63 @@ | ||
#!/usr/bin/env bash | ||
# - gp-logcode | ||
. $(dirname "$0")/functions.sh | ||
_debug "Loading functions.sh" | ||
|
||
# -- | ||
# -- Variables | ||
# -- | ||
ram_KB=$(grep MemTotal /proc/meminfo | awk '{print $2}') | ||
ram_MB=$(echo "scale=0; ${ram_KB} / 1000" | bc) | ||
ten_percent_ram_MB=$(echo "scale=0; ${ram_KB} / 10000" | bc) | ||
twenty_percent_ram_MB=$(echo "scale=0; ${ram_KB} / 5000" | bc) | ||
thirty_percent_ram_MB=$(echo "scale=0; ${ram_KB} / 3000" | bc) | ||
fourty_percent_ram_MB=$(echo "scale=0; ${ram_KB} / 2500" | bc) | ||
fifty_percent_ram_MB=$(echo "scale=0; ${ram_KB} / 2000" | bc) | ||
sixty_percent_ram_MB=$((thirty_percent_ram_MB + thirty_percent_ram_MB)) | ||
seventy_percent_ram_MB=$((fifty_percent_ram_MB + twenty_percent_ram_MB)) | ||
eighty_percent_ram_MB=$((fifty_percent_ram_MB + thirty_percent_ram_MB)) | ||
ninety_percent_ram_MB=$((fifty_percent_ram_MB + fourty_percent_ram_MB)) | ||
|
||
# -- | ||
# -- Start | ||
# -- | ||
|
||
echo "gpmonit mysql algorithm" | ||
echo "--" | ||
echo "Current memory in MB: $ram_MB" | ||
echo "--" | ||
|
||
echo "If ram_MB > 1500 then kill at 500MB" | ||
if [[ ${ram_MB} -lt 1500 ]]; then | ||
mysqlRAM="500" | ||
fi | ||
|
||
echo "If ram_MB =< 1500 then kill at $(echo "scale=0; ${seventy_percent_ram_MB} / 2" | bc)MB (70% Memory in MB/2)" | ||
if [[ ${ram_MB} -ge 1500 ]]; then | ||
mysqlRAM="$(echo "scale=0; ${seventy_percent_ram_MB} / 2" | bc)" | ||
fi | ||
|
||
echo "If ram_MB =< 8000 then kill at $(echo "scale=0; ${ninety_percent_ram_MB} / 2" | bc)MB (80% memory in MB/2)" | ||
if [[ ${ram_MB} -ge 8000 ]]; then | ||
mysqlRAM="$(echo "scale=0; ${ninety_percent_ram_MB} / 2" | bc)" | ||
fi | ||
|
||
echo "If ram_MB =< 1600 then kill at ${fifty_percent_ram_MB}MB (50% memory in MB)" | ||
if [[ ${ram_MB} -ge 16000 ]]; then | ||
mysqlRAM="${fifty_percent_ram_MB}" | ||
fi | ||
|
||
# -- print out details | ||
echo "gpmonit mysql would set the restart threshold at ${mysqlRAM}MB memory usage for MySQL" | ||
echo "" | ||
echo "Suggested Changes" | ||
echo "--" | ||
echo "If ram_MB =< 4000 then kill at ${fifty_percent_ram_MB} (50% Memory in MB)" | ||
echo "or" | ||
echo "Regardless of memory kill at ${fifty_percent_ram_MB} (50% Memory in MB)" | ||
echo "--" | ||
echo "" | ||
echo "To fix this issue run the following command:" | ||
echo " 50% memory - gpmonit mysql -mem-high-mb ${fifty_percent_ram_MB} -mem-restart-mb ${fifty_percent_ram_MB}" | ||
echo " 60% memory - gpmonit mysql -mem-high-mb ${sixty_percent_ram_MB} -mem-restart-mb ${sixty_percent_ram_MB}" | ||
echo " 705 memory - gpmonit mysql -mem-high-mb ${seventy_percent_ram_MB} -mem-restart-mb ${seventy_percent_ram_MB}" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
gp-plugins.sh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,61 @@ | ||
#!/usr/bin/env bash | ||
# - gp-logcode | ||
. $(dirname "$0")/functions.sh | ||
_debug "Loading functions.sh" | ||
|
||
# Variables | ||
GP_WWW_PATH="/var/www/" | ||
WP_CLI_OPTIONS="--allow-root --skip-plugins --skip-themes --skip-packages" | ||
|
||
# Functions | ||
usage () { | ||
echo "Lists WordPress plugins on all websites on a GridPane Server" | ||
echo "" | ||
echo "Syntax: gpplugin -a <plugin>" | ||
echo " options" | ||
echo " -a List all plugins" | ||
echo " -p Status on specific plugin" | ||
echo "" | ||
echo "Notes:" | ||
echo " * Find out which sites have which plugins + search for specific plugins and print wp-cli plugin status" | ||
echo " * Excludes canary + staging + 22222 + nginx + *.gridpanevps.com sites" | ||
echo " * --skip-plugins is run as to not fail potentially due to an error with a plugin" | ||
echo "" | ||
echo "Updates: https://github.com/jordantrizz/gpcli" | ||
echo "" | ||
exit | ||
} | ||
|
||
POSITIONAL=() | ||
while [[ $# -gt 0 ]] | ||
do | ||
key="$1" | ||
|
||
case $key in | ||
-a|--all) | ||
ALL="YES" | ||
shift # past argument | ||
shift # past value | ||
;; | ||
-p|--plugin) | ||
PLUGIN="$2" | ||
shift # past argument | ||
shift # past value | ||
;; | ||
*) # unknown option | ||
POSITIONAL+=("$1") # save it in an array for later | ||
shift # past argument | ||
;; | ||
esac | ||
done | ||
set -- "${POSITIONAL[@]}" # restore positional parameters | ||
|
||
if [ ! -z $PLUGIN ]; then | ||
echo "Searching for $PLUGIN on all sites" | ||
ls -1 $GP_WWW_PATH | grep -vE 'canary|staging' | grep -vE '22222|nginx|gridpanevps.com' | xargs -i sh -c "echo '\033[1;42m$GP_WWW_PATH{}\e[0m';wp $WP_CLI_OPTIONS --path=$GP_WWW_PATH{}/htdocs plugin status $PLUGIN" | ||
elif [ ! -z $ALL ]; then | ||
echo "Searching for all plugins on all sites!" | ||
ls -1 $GP_WWW_PATH | grep -vE 'canary|staging' | grep -vE '22222|nginx|gridpanevps.com' | xargs -i sh -c "echo '\033[1;42m$GP_WWW_PATH{}\e[0m';wp $WP_CLI_OPTIONS --path=$GP_WWW_PATH{}/htdocs plugin status" | ||
else | ||
usage | ||
fi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
gp-tune.sh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,15 @@ | ||
#!/usr/bin/env bash | ||
# - gp-logcode | ||
. $(dirname "$0")/functions.sh | ||
_debug "Loading functions.sh" | ||
|
||
usage () { | ||
echo "Usage: nothing yet!" | ||
} | ||
|
||
tuneswap () { | ||
# Tune swap | ||
##ISSUE need to check if swappiness is already tuned. | ||
echo " -- Tuning /proc/sys/vm/swappiness" | ||
echo 1 > /proc/sys/vm/swappiness | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,17 @@ | ||
#!/usr/bin/env bash | ||
# -- Command to set profiles for specific server sizes | ||
# -- Usage: gptunemysql (small|medium|large) | ||
|
||
# -- Thread Pools | ||
# Use pool-of-threads because https://mariadb.com/kb/en/thread-pool-in-mariadb/ | ||
# gp stack mysql -thread-handling pool-of-threads | ||
|
||
# -- Thread Pool Size | ||
# thread_pool_size = Number of CPU's | ||
# gp stack mysql -thread-pool-size 4 | ||
|
||
# -- Thread Pools + | ||
# The rest of the thread_pool_* options are fine as default for now. | ||
|
||
# -- Innodb | ||
# innodb_flush_method=O_DIRECT |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,76 @@ | ||
#!/bin/bash | ||
# | ||
# USE FOR LOCAL DEVELOPMENT ONLY!!! | ||
# | ||
EMAIL='your-email@gmail.com' | ||
MYSQL=`which mysql` | ||
|
||
echo -n "DB user/name for project (lowercase) [ENTER]:" | ||
read DB | ||
|
||
if [ -z "$DB" ]; then | ||
echo "User must define a DB Name for your project." | ||
exit; | ||
fi | ||
|
||
echo -n "Local URL eg. local.test.com [ENTER]:" | ||
read URL | ||
|
||
if [ -z "$URL" ]; then | ||
echo "You must enter a valid url." | ||
exit; | ||
fi | ||
|
||
# | ||
# Start WP Installation | ||
# | ||
|
||
random-string() { | ||
LENGTH=$1 | ||
if [ -z "$LENGTH" ]; then | ||
LENGTH='20' | ||
fi | ||
|
||
cat /dev/urandom | env LC_CTYPE=C tr -dc 'a-zA-Z0-9' | fold -w $LENGTH | head -n 1 | ||
} | ||
|
||
# Generate random password (mac friendly) | ||
PASS=`random-string 20` | ||
|
||
PREFIX='' | ||
# No prefix? Then just comment out the line below | ||
PREFIX="--dbprefix=`random-string 6`_" | ||
|
||
Q1="CREATE DATABASE IF NOT EXISTS $DB;" | ||
Q2="GRANT ALL ON $DB.* TO '$DB'@'localhost' IDENTIFIED BY '$PASS';" | ||
Q3="FLUSH PRIVILEGES;" | ||
SQL="${Q1}${Q2}${Q3}" | ||
|
||
# Set up a login path for creating your dbs and users (must have global privileges) | ||
# mysql_config_editor set --login-path=local --host=localhost --user=username --password | ||
$MYSQL --login-path=local -e "$SQL" | ||
echo "MYSQL user/db '$DB' created." | ||
|
||
# Install latest WP | ||
wp core download | ||
wp core config --dbname="$DB" --dbuser="$DB" --dbpass="$PASS" $PREFIX | ||
wp core install --url="$URL" --title='New WP Build' --admin_user='jason' --admin_password='local' --admin_email="$EMAIL" --skip-email | ||
|
||
# Install but leave inactive | ||
|
||
# Remove old default themes | ||
wp theme delete twentyfifteen | ||
wp theme delete twentythirteen | ||
wp theme delete twentyfourteen | ||
|
||
# Remove default posts, widgets, comments etc. | ||
wp site empty --yes | ||
|
||
# Move the config a directory above | ||
mv wp-config.php ../ | ||
|
||
# Make uploads writable | ||
chmod -R 777 wp-content/uploads | ||
|
||
# Making a dumping folder | ||
mkdir etc# |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
wp-migrate-ssh.sh |
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
wpst.sh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
wpst-goaccess.sh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,374 @@ | ||
#!/bin/bash | ||
# ------------------ | ||
# -- wpst-goacces.sh | ||
# ------------------ | ||
|
||
# ----------- | ||
# -- Includes | ||
# ----------- | ||
. $(dirname "$0")/functions.sh | ||
|
||
# ------------ | ||
# -- Variables | ||
# ------------ | ||
DEBUG_ON="0" | ||
|
||
# ------------ | ||
# -- Functions | ||
# ------------ | ||
|
||
usage () { | ||
USAGE=\ | ||
"Usage: wpst-goaccess [-h|-d|-c|-dr|-f (nginx|ols)|-p [(gridpane|runcloud)] [-ld <log directory>|-l <log file>|-time <timerange>] [-domain domain.com|--all|-file <filename> [-p (gridpane|runcloud)] | ||
This script will try and detect log files in common locations, you can also | ||
specify the platform and format using the options below | ||
Commands | ||
-domain <domain> - Domain name of log files to process | ||
-all - Go through all the logs versus a single log file | ||
-file <filename> - Process a single file | ||
-time <timerange> - Specify a time range to process dates in the format yyyy-mm-dd-hh-mm-ss,yyyy-mm-dd-hh-mm-ss (e.g. 2017-01-01-00-00-00,2017-01-01-23-59-59) | ||
- First portion is start time and second part is end time. | ||
Options: | ||
-h - Help | ||
-d - Debug | ||
-dr - Dry Run | ||
-p - Specify platform (gridpane|runcloud) | ||
-f - Override detected format, (nginx|ols) | ||
-t - Use test log files | ||
" | ||
echo "$USAGE" | ||
} | ||
|
||
# -- check_goaccess | ||
check_goaccess () { | ||
_debug "Checking if goaccess is installed" | ||
_cexists goaccess | ||
_debug "\$CMD_EXISTS: $CMD_EXISTS" | ||
if [[ $CMD_EXISTS == "1" ]]; then | ||
_error "goaccess is not installed" | ||
exit | ||
else | ||
_debug "Confirmed goaccess is installed" | ||
fi | ||
} | ||
|
||
# -- set_format | ||
set_format () { | ||
# Formats for goaccess | ||
# OLS | ||
# logformat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\" | ||
# "192.168.0.1 - - [13/Sep/2022:16:28:40 -0400] "GET /request.html HTTP/2" 200 46479 "https://domain.com/referer.html" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" | ||
if [[ $FORMAT == "OLS" ]]; then | ||
LOG_FORMAT='\"%h - - [%d:%t %^] \"%r\" %s %b \"%R\" \"%u\"' | ||
DATE_FORMAT='%d/%b/%Y' | ||
TIME_FORMAT='%H:%M:%S %Z' | ||
fi | ||
|
||
# NGINX | ||
# log_format we_log '[$time_local] $remote_addr $upstream_response_time $upstream_cache_status $http_host "$request" $status $body_bytes_sent $request_time "$http_referer" "$http_user_agent" "$http3"'; | ||
# [14/Sep/2022:10:12:55 -0700] 129.168.0.1 - domain.com "GET /request.html HTTP/1.1" 200 47 1.538 "https://domain.com/referer.html" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36" | ||
if [[ $FORMAT == "NGINX" ]]; then | ||
LOG_FORMAT='[%d:%t %^] %h %^ - %v \"%r\" %s %b \"%R\" \"%u\"\' | ||
DATE_FORMAT='%d/%b/%Y' | ||
TIME_FORMAT='%H:%M:%S %Z' | ||
fi | ||
|
||
# GRIDPANE-NGINX | ||
# ./common/logging.conf:log_format we_log '[$time_local] $remote_addr $upstream_response_time $upstream_cache_status $http_host "$request" $status $body_bytes_sent $request_time "$http_referer" "$http_user_agent"'; | ||
# $request_time = seconds with a milliseconds resolution | ||
# [14/Apr/2023:06:30:32 -0500] 127.0.0.1 1.732 - domain.com "GET /?pwgc=1628918241 HTTP/2.0" 200 39563 1.731 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/103.0.5060.134 Safari/537.36" | ||
# [27/Apr/2023:06:38:01 -0500] 2600:1700:4dce:28f0:ed74:ea35:b8e5:ece6 - - domain.com "GET /image.jpg HTTP/2.0" 304 0 0.000 "https://yahoo.com" "Mozilla/5.0 (iPhone; CPU iPhone OS 16_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/107.0.5304.66 Mobile/15E148 Safari/604.1" | ||
if [[ $FORMAT == "GPNGINX" ]]; then | ||
LOG_FORMAT='[%d:%t %^] %h %^ - %v \"%r\" %s %b %T \"%R\" \"%u\"\' | ||
DATE_FORMAT='%d/%b/%Y' | ||
TIME_FORMAT='%H:%M:%S %Z' | ||
fi | ||
|
||
# GRIDPANE-OLS | ||
if [[ $FORMAT == "GPOLS" ]]; then | ||
LOG_FORMAT='[%d:%t %^] %h %^ - %v \"%r\" %s %b \"%R\" \"%u\"\' | ||
DATE_FORMAT='%d/%b/%Y' | ||
TIME_FORMAT='%H:%M:%S %Z' | ||
fi | ||
|
||
_debug "goaccess LOG_FORMAT = $LOG_FORMAT ## DATE_FORMAT = $DATE_FORMAT ## TIME_FORMAT = $TIME_FORMAT" | ||
} | ||
|
||
detect_logs () { | ||
echo "Detecting log files" | ||
# TODO figure out how to test and set the format better. | ||
|
||
# GRIDPANE-OLS | ||
if [[ -d /usr/local/lsws ]] && [[ -d /var/www/ ]]; then | ||
echo "Detected GridPane OLS logs" | ||
FORMAT="GPOLS" | ||
if [[ $ACTION == "ALL" ]]; then | ||
LOG_FILE_LOCATION="/var/www/*/logs" | ||
LOG_FILTER="*.access.log*gz" | ||
elif [[ $ACTION == "DOMAIN" ]]; then | ||
LOG_FILE_LOCATION="/var/www/$DOMAIN/logs" | ||
LOG_FILTER="*.access.log" | ||
elif [[ $ACTION == "FILE" ]]; then | ||
LOG_FILE_LOCATION="$FILE" | ||
fi | ||
elif [[ -d /var/log/nginx ]] && [[ -d /var/www/ ]]; then | ||
echo "Detected GridPane NGINX logs" | ||
FORMAT="GPNGINX" | ||
if [[ $ACTION == "ALL" ]]; then | ||
LOG_FILE_LOCATION="/var/log/nginx" | ||
LOG_FILTER="*.access.log*gz" | ||
elif [[ $ACTION == "DOMAIN" ]]; then | ||
LOG_FILE_LOCATION="/var/log/nginx" | ||
LOG_FILTER="*${DOMAIN}*.access.log" | ||
elif [[ $ACTION == "FILE" ]]; then | ||
LOG_FILE_LOCATION="$FILE" | ||
fi | ||
# OLS | ||
elif [[ -d /usr/local/lsws ]]; then | ||
echo "Detected OLS logs" | ||
FORMAT="OLS" | ||
if [[ $ACTION == "ALL" ]]; then | ||
LOG_FILE_LOCATION="/var/www/*/logs" | ||
LOG_FILTER="*.access.log*gz" | ||
elif [[ $ACTION == "DOMAIN" ]]; then | ||
LOG_FILE_LOCATION="/var/www/$DOMAIN/logs" | ||
LOG_FILTER="*.access.log" | ||
elif [[ $ACTION == "FILE" ]]; then | ||
LOG_FILE_LOCATION="$FILE" | ||
fi | ||
# NGINX | ||
elif [[ -d /var/log/nginx ]]; then | ||
echo "Detected GridPane Nginx logs" | ||
FORMAT="OLS" | ||
LOG_FILE_LOCATION="/var/log/nginx" | ||
|
||
if [[ $ACTION == "ALL" ]]; then | ||
LOG_FILTER="*.access.log" | ||
elif [[ $ACTION == "DOMAIN" ]]; then | ||
LOG_FILTER="$DOMAIN.access.log" | ||
elif [[ $ACTION == "FILE" ]]; then | ||
LOG_FILE_LOCATION="$FILE" | ||
fi | ||
else | ||
_error "Can't detect webserver logs" | ||
exit | ||
fi | ||
|
||
_debug "FORMAT:$FORMAT LOG_FILE_LOCATION:$LOG_FILE_LOCATION LOG_FILTER:$LOG_FILTER" | ||
} | ||
|
||
# -- collect_logs | ||
collect_logs () { | ||
local CATCMD="cat" | ||
_debug "Format: $FORMAT Log File Location:$LOG_FILE_LOCATION Log Filter: $LOG_FILTER" | ||
LOG_COLLECT_DATA=$(mktemp) | ||
if [[ $ACTION == "DOMAIN" ]]; then | ||
if [[ $DRY_RUN == "1" ]]; then | ||
echo "cat ${LOG_FILE_LOCATION}/${LOG_FILTER} > $LOG_COLLECT_DATA" | ||
else | ||
cat ${LOG_FILE_LOCATION}/${LOG_FILTER} > $LOG_COLLECT_DATA | ||
fi | ||
elif [[ $ACTION == "ALL" ]]; then | ||
if [[ $DRY_RUN == "1" ]]; then | ||
echo "cat ${LOG_FILE_LOCATION}/*.access.log > $LOG_COLLECT_DATA; zcat ${LOG_FILE_LOCATION}/${LOG_FILTER} >> $LOG_COLLECT_DATA" | ||
else | ||
cat ${LOG_FILE_LOCATION}/*.access.log > $LOG_COLLECT_DATA; zcat ${LOG_FILE_LOCATION}/${LOG_FILTER} >> $LOG_COLLECT_DATA | ||
fi | ||
elif [[ $ACTION == "FILE" ]]; then | ||
if [[ $DRY_RUN == "1" ]]; then | ||
echo "$CATCMD ${LOG_FILE_LOCATION} > $LOG_COLLECT_DATA" | ||
else | ||
[[ $LOG_FILE_LOCATION == "*.gz" ]] && CATCMD="zcat" | ||
$CATCMD ${LOG_FILE_LOCATION} > $LOG_COLLECT_DATA | ||
fi | ||
fi | ||
LOG_DATA_FILE=$LOG_COLLECT_DATA | ||
} | ||
|
||
# -- do_goaccess | ||
do_goaccess () { | ||
_debug "Proceeding with d_goaccess" | ||
if [[ -n $TIME_SPEC ]]; then | ||
GOACCESS_EXTRA+="--hour-spec=$TIME_SPEC" | ||
fi | ||
|
||
if [[ $ACTION == "DOMAIN" ]]; then | ||
if [[ $DRY_RUN == "1" ]]; then | ||
echo "cat ${LOG_FILE_LOCATION}/${LOG_FILTER} | goaccess ${GOACCESS_EXTRA} --log-format='$LOG_FORMAT' --date-format='$DATE_FORMAT' --time-format='$TIME_FORMAT'" | ||
else | ||
cat $LOG_DATA_FILE | goaccess ${GOACCESS_EXTRA} --log-format="${LOG_FORMAT}" --date-format="$DATE_FORMAT" --time-format="$TIME_FORMAT" | ||
fi | ||
elif [[ $ACTION == "ALL" ]]; then | ||
if [[ $DRY_RUN == "1" ]]; then | ||
echo "cat $LOG_DATA_FILE | goaccess ${GOACCESS_EXTRA} --log-format='$LOG_FORMAT' --date-format='$DATE_FORMAT' --time-format='$TIME_FORMAT'" | ||
else | ||
cat $LOG_DATA_FILE | goaccess ${GOACCESS_EXTRA} --log-format="$LOG_FORMAT" --date-format="$DATE_FORMAT" --time-format="$TIME_FORMAT" | ||
fi | ||
elif [[ $ACTION == "FILE" ]]; then | ||
[[ $LOG_FILE_LOCATION == "*.gz" ]] && CATCMD="zcat" | ||
if [[ $DRY_RUN == "1" ]]; then | ||
echo "cat $LOG_DATA_FILE | goaccess ${GOACCESS_EXTRA} --log-format='$LOG_FORMAT' --date-format='$DATE_FORMAT' --time-format='$TIME_FORMAT'" | ||
else | ||
cat $LOG_DATA_FILE | goaccess --log-format="$LOG_FORMAT" --date-format="$DATE_FORMAT" --time-format="$TIME_FORMAT" | ||
echo "Command Ran $(history | tail -n 1)" | ||
fi | ||
elif [[ $ACTION == "TEST" ]]; then | ||
[[ $LOG_FILE_LOCATION == "*.gz" ]] && CATCMD="zcat" | ||
if [[ $DRY_RUN == "1" ]]; then | ||
echo "cat ${LOG_FILE_LOCATION} | goaccess ${GOACCESS_EXTRA} --log-format='$LOG_FORMAT' --date-format='$DATE_FORMAT' --time-format='$TIME_FORMAT'" | ||
else | ||
$CATCMD ${LOG_FILE_LOCATION} | goaccess ${GOACCESS_EXTRA} --log-format="$LOG_FORMAT" --date-format="$DATE_FORMAT" --time-format="$TIME_FORMAT" | ||
fi | ||
else | ||
echo "No action specified" | ||
exit | ||
fi | ||
} | ||
|
||
function sed_logs() { | ||
_debug "Processing logs using custom time - $CUSTOM_TIME" | ||
SED_LOG=$(mktemp) | ||
if ! [[ $CUSTOM_TIME =~ ^[0-9]{2}/(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)/[0-9]{4}:[0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{2}/(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)/[0-9]{4}:[0-9]{2}:[0-9]{2}:[0-9]{2}$ ]]; then | ||
echo "Error: Please provide dates in the format dd/Mon/yyyy:hh:mm:ss" | ||
exit 1 | ||
fi | ||
|
||
local EDATE=$(echo $CUSTOM_TIME | cut -d, -f1) | ||
local SDATE=$(echo $CUSTOM_TIME | cut -d, -f2) | ||
local START_DATE=$(date -d "$(echo "$SDATE" | sed 's/\// /g;s/:/ /')" +%s | xargs -I{} date -d "@{}" +'%d\/%b\/%Y:%H:%M:%S') | ||
local END_DATE=$(date -d "$(echo "$EDATE" | sed 's/\// /g;s/:/ /')" +%s | xargs -I{} date -d "@{}" +'%d\/%b\/%Y:%H:%M:%S') | ||
|
||
if [[ $DRY_RUN == "1" ]]; then | ||
echo "sed -n \"/$START_DATE/,/$END_DATE/ p\" $LOG_DATA_FILE > $SED_LOG" | ||
else | ||
sed -n "/$START_DATE/,/$END_DATE/ p" $LOG_DATA_FILE > $SED_LOG | ||
fi | ||
LOG_DATA_FILE=$SED_LOG | ||
} | ||
|
||
# --------------- | ||
# -- Main Program | ||
# --------------- | ||
DCMD="" | ||
POSITIONAL=() | ||
while [[ $# -gt 0 ]] | ||
do | ||
key="$1" | ||
case $key in | ||
-h) | ||
ACTION="HELP" | ||
DCMD+="ACTION=HELP " | ||
shift # past argument | ||
;; | ||
-platform) | ||
PLATFORM="$2" | ||
DCMD+="PLATFORM=$2 " | ||
shift # past argument | ||
shift # past value | ||
;; | ||
-f|--format) | ||
FORMAT="$2" | ||
DCMD+="FORMAT=$2 " | ||
shift # past argument | ||
shift # past value | ||
;; | ||
-d|--debug) | ||
DEBUG_ON="1" | ||
DCMD+="DEBUG_ON=1 " | ||
shift # past argument | ||
;; | ||
-dr) | ||
DRY_RUN="1" | ||
DCMD+="DRY_RUN=1 " | ||
shift # past argument | ||
;; | ||
-domain) | ||
ACTION="DOMAIN" | ||
DOMAIN="$2" | ||
DCMD+="ACTION=DOMAIN DOMAIN=$2 " | ||
shift # past argument | ||
shift # past value | ||
;; | ||
-all) | ||
ACTION="ALL" | ||
DCMD+="ACTION=ALL " | ||
shift # past argument | ||
;; | ||
-file) | ||
ACTION="FILE" | ||
FILE="$2" | ||
DCMD+="ACTION=FILE FILE=$2 " | ||
shift # past argument | ||
shift # past value | ||
;; | ||
-t) | ||
ACTION="TEST" | ||
FILE="$2" | ||
DCMD+="ACTION=TEST FILE=$2 " | ||
shift # past argument | ||
shift # past value | ||
;; | ||
-time) | ||
CUSTOM_TIME="$2" | ||
DCMD+="CUSTOM_TIME=$2 " | ||
shift # past argument | ||
shift # past value | ||
;; | ||
-timespec) | ||
TIME_SPEC="$2" | ||
DCMD+="TIME_SPEC=$2 " | ||
shift # past argument | ||
shift # past value | ||
;; | ||
*) # unknown option | ||
POSITIONAL+=("$1") # save it in an array for later | ||
shift # past argument | ||
;; | ||
esac | ||
done | ||
set -- "${POSITIONAL[@]}" # restore positional parameters | ||
|
||
|
||
|
||
# ----------------------------# | ||
# -- Debugging | ||
# ----------------------------# | ||
_debug "Running wpst-goaccess $DCMD" | ||
_debug_all "${*}" | ||
|
||
# ----------------------------# | ||
# --- Check Arguments | ||
# ----------------------------# | ||
if [[ -n $TIME_SPEC ]]; then | ||
if [[ $TIME_SPEC != "min" && $TIME_SPEC != "hour" ]]; then | ||
echo "Error: Invalid time specification" | ||
exit | ||
fi | ||
fi | ||
|
||
# ----------------------------# | ||
# --- Main | ||
# ----------------------------# | ||
|
||
if [[ -z $ACTION ]] || [[ $ACTION == "HELP" ]]; then | ||
usage | ||
echo "Error: No action specified" | ||
exit | ||
else | ||
if [[ $ACTION == "DOMAIN" ]] && [[ -z $DOMAIN ]]; then | ||
usage | ||
echo "Error: specify a domain" | ||
exit | ||
fi | ||
_debug "goaccess arguments - ${*}" | ||
check_goaccess | ||
detect_logs | ||
set_format | ||
collect_logs | ||
[[ ! -z $CUSTOM_TIME ]] && sed_logs $CUSTOM_TIME | ||
do_goaccess $ACTION $LOG_DATA_FILE | ||
fi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
wpst-sql.sh |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,131 @@ | ||
#!/bin/bash | ||
# ------------------ | ||
# -- wpst-goacces.sh | ||
# ------------------ | ||
|
||
# ----------- | ||
# -- Includes | ||
# ----------- | ||
. $(dirname "$0")/functions.sh | ||
|
||
# ------------ | ||
# -- Variables | ||
# ------------ | ||
DEBUG_ON="0" | ||
CMD_VERSION="0.0.1" | ||
CMD_SCRIPT="wpst-sql" | ||
|
||
# ------------ | ||
# -- Functions | ||
# ------------ | ||
|
||
USAGE=\ | ||
"Usage: $CMD_SCRIPT [-d] [command] | ||
Commands | ||
post-meta <post-id> - Get wp-postmeta data for <post-id> | ||
woo-order-meta <post-id> - Get woocomerce_order_items dat for <post-id> | ||
wp-autoload-options - List autoload options. | ||
Options: | ||
-d|--debug - Debug | ||
CMD Version $CMD_VERSION | ||
WPST Version $WPST_VERSION | ||
" | ||
|
||
usage () { | ||
echo "$USAGE" | ||
} | ||
|
||
wpst_init () { | ||
if ! command -v wp > /dev/null;then | ||
echo "wp-cli not installed" | ||
exit 1 | ||
else | ||
_debug "wp-cli is installed" | ||
fi | ||
} | ||
|
||
get_table_prefix () { | ||
WP_TABLE_PREFIX=$(wp db prefix) | ||
_debug "WP_TABLE_PREFIX=$WP_TABLE_PREFIX" | ||
} | ||
|
||
sql_post_meta () { | ||
get_table_prefix | ||
POST_ID=$1 | ||
_debug "SQL: wp sql query \" SELECT * from ${WP_TABLE_PREFIX}postmeta where post_id = '$POST_ID'" | ||
wp sql query "SELECT * from ${WP_TABLE_PREFIX}postmeta where post_id = '$POST_ID'\"" | ||
} | ||
|
||
sql_woo_order_meta () { | ||
get_table_prefix | ||
POST_ID=$1 | ||
#wp sql query "SELECT * FROM wp_bspr_woocommerce_order_items where order_id = '106818'" | less | ||
#wp sql query "SELECT * FROM wp_bspr_woocommerce_order_itemmeta where order_item_id = '78807'" | less | ||
_debug "SQL: wp sql query \"SELECT * from ${WP_TABLE_PREFIX}postmeta where post_id = '$POST_ID'\" | tr '\t' ','" | ||
OUTPUT=$(wp sql query "SELECT * from ${WP_TABLE_PREFIX}postmeta where post_id = '$POST_ID'" | tr '\t' ',') | ||
HEADERS=$(head -1 $OUTPUT) | ||
echo $HEADERS | ||
} | ||
sql_wp_autoload_options () { | ||
|
||
_debug "wp sql query \"SELECT * from ${WP_TABLE_PREFIX}options where autoload = 'yes' and option_name like '_transient%';\"" | ||
wp sql query "SELECT * from ${WP_TABLE_PREFIX}options where autoload = 'yes' and option_name like '_transient%'l" | ||
} | ||
|
||
poop () { | ||
PANTS=$1 | ||
echo "Found poop in $PANTS" | ||
} | ||
|
||
# --------------- | ||
# -- Main Program | ||
# --------------- | ||
DCMD="" | ||
POSITIONAL=() | ||
while [[ $# -gt 0 ]] | ||
do | ||
key="$1" | ||
case $key in | ||
-d|--debug) | ||
DEBUG_ON="1" | ||
DCMD+="DEBUG_ON=1 " | ||
shift # past argument | ||
;; | ||
*) # unknown option | ||
POSITIONAL+=("$1") # save it in an array for later | ||
shift # past argument | ||
;; | ||
esac | ||
done | ||
set -- "${POSITIONAL[@]}" # restore positional parameters | ||
|
||
_debug "Running wpst-goaccess $DCMD" | ||
_debug_all $@ | ||
|
||
|
||
ACTION=$1 | ||
EXTRA=$2 | ||
_debug "ACTION=$ACTION EXTRA=$EXTRA" | ||
|
||
wpst_init | ||
|
||
if [[ -z $ACTION ]]; then | ||
usage | ||
else | ||
case $ACTION in | ||
post-meta) | ||
if [[ -z $EXTRA ]]; then _error "Missing post_id";usage;exit 1;fi | ||
sql_post_meta $EXTRA | ||
;; | ||
woo-order-meta) | ||
if [[ -z $EXTRA ]]; then _error "Missing post_id";usage;exit 1;fi | ||
sql_woo_order_meta $EXTRA | ||
;; | ||
wp-autoload-options) | ||
sql_wp_autoload_options | ||
;; | ||
esac | ||
fi |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,46 @@ | ||
#!/bin/bash | ||
# -------------------------- | ||
# Lazy way to provide help and a wrapper to related tools. | ||
# ------------------------- | ||
WPST_VERSION=0.7.1 | ||
SCRIPT_NAME=wpst | ||
DEBUG_ON="0" | ||
|
||
# ----------- | ||
# -- Includes | ||
# ----------- | ||
. $(dirname "$0")/functions.sh | ||
|
||
# ------------ | ||
# -- Variables | ||
# ------------ | ||
|
||
# -- USAGE | ||
USAGE=\ | ||
"WordPress Shell Tools - $VERSION | ||
Commands: | ||
wpst-goaccess - Run goaccess on web server logs | ||
wpst-sql - Run common WordPress SQL Commands | ||
wpst help - This help. | ||
WPSET Version $WPST_VERSION | ||
" | ||
|
||
# -- usage | ||
usage () { | ||
echo "$USAGE" | ||
} | ||
|
||
# ------------ | ||
# -- Main Loop | ||
# ------------ | ||
ACTION="$2" | ||
if [[ ! $ACTION ]]; then | ||
usage | ||
exit 1 | ||
elif [[ $ACTION = "help" ]]; then | ||
usage | ||
exit 1 | ||
fi |