Fixing #1, updating postgres driver and support for multiple dates in…

… s3 driver (#2)

* Updated postgres driver and README

* Added support for multiple dates

* Updated docker containers to point to mine

* Oops 'i' floating

* Added date parameter to postgres driver

* Dates didnt have a value, oops

* Support uncompressed files and fixes (wow so many fixes)

* Updated readme for compressed flag

.gitignore

@@ -0,0 +1,4 @@
+run_remote.sh
+docker_clean.sh
+build_n_push.sh
+.idea/

README.mdown

@@ -99,7 +99,7 @@ Create a global config in `/etc/uphold/uphold.yml` (even if you leave it empty),
   * Generally only overridden in development on OSX when you need to mount your own src directory
 * `docker_url` (default: `unix:///var/run/docker.sock`)
   * If you connect to Docker via a TCP socket instead of a Unix one, then you would supply `tcp://example.com:5422` instead (untested)
-* `docker_container` (default: `forward3d/uphold-tester`)
+* `docker_container` (default: `vjeeva/uphold-tester`)
   * If you need to customize the docker container and use a different one, you can override it here
 * `docker_tag` (default: `latest`)
   * Can override the Docker container tag if you want to run from a specific version
@@ -116,7 +116,7 @@ If you change the global config you will need to restart the UI docker container
     config_path: /etc/uphold
     docker_log_path: /var/log/uphold
     docker_url: unix:///var/run/docker.sock
-    docker_container: forward3d/uphold-tester
+    docker_container: vjeeva/uphold-tester
     docker_tag: latest
     docker_mounts:
       - /var/my_backups
@@ -140,10 +140,14 @@ Each config is in YAML format, and is constructed of a transport, an engine and
         access_key_id: your-access-key-id
         secret_access_key: your-secret-access-key
         bucket: your-backups
-        path: mongodb/systemx/{date}
-        filename: mongodb.tar
-        date_format: '%Y.%m.%d'
-        date_offset: 0
+        path: mongodb/systemx/{date0}
+        filename: mongodb_{date1}.tar
+        dates:
+         - date_format: '%Y.%m.%d'
+           date_offset: 0
+         - date_format: '%Y%m%d'
+           date_offset: 0
+         # ... add as many as needed
         folder_within: mongodb/databases/MongoDB
     tests:
       - test_structure.rb
@@ -179,6 +183,8 @@ Transports all inherit these generic parameters...
   * When using dates the code starts at `Date.today` and then subtracts this number, so for checking a backup that exists for yesterday, you would enter `1`
 * `folder_within`
   * Once your backup has been decompressed it may have folders inside, if so, you need to provide where the last directory is, this generally can't be programmatically found as some database backups may contain folders in their own structure.
+* `compressed`
+  * Specify if your path leads to an compressed archive with a boolean value (`true` or `false`).
 
 ##### S3 (type: `s3`)
 
@@ -308,16 +314,18 @@ Unless you need to change any of the defaults, a standard configuration for Mong
 
 ###### Full `postgresql` Engine Example
 
-The `database` also becomes your username for when you run the tests.
+You can specify the username for accessing the postgres server (other than the default `postgres`) for when you run the tests. Furthermore, you can specify the file type to be `sql` or `dump`. If `sql`, then the restore will be done with the `psql` command, which takes a `.sql` file. If `dump`, then the restore will be done with the `pg_restore` command, which takes a `.pg_dump` file (usually, or whatever file is obtained using the pg_dump command).
 
     engine:
       type: postgresql
       settings:
         database: your_database_name
         docker_image: postgres
         docker_tag: 9.5.0
+        file_type: sql
         port: 5432
         sql_file: PostgreSQL.sql
+        username: postgres
 
 #### Tests
 
@@ -351,14 +359,14 @@ Obviously this is just a simple test, but you can write any number of tests you
 
 Once you have finished your configuration, to get the system running you only need to start the Docker container called 'uphold-ui'.
 
-    docker pull forward3d/uphold-ui:latest
+    docker pull vjeeva/uphold-ui:latest
     docker run \
       --rm \
       -p 8079:8079 \
       -v /var/run/docker.sock:/var/run/docker.sock \
       -v /etc/uphold:/etc/uphold \
       -v /var/log/uphold:/var/log/uphold \
-      forward3d/uphold-ui:latest
+      vjeeva/uphold-ui:latest
 
 * You must make sure that you mount in the `docker.sock` from wherever it resides.
 * Feel free to change the port number if you don't want it to start up on port 8079.

lib/config.rb

@@ -41,7 +41,7 @@ def self.load_global
       yaml = deep_convert(yaml)
       yaml[:log_level] ||= 'DEBUG'
       yaml[:docker_url] ||= 'unix:///var/run/docker.sock'
-      yaml[:docker_container] ||= 'forward3d/uphold-tester'
+      yaml[:docker_container] ||= 'vjeeva/uphold-tester'
       yaml[:docker_tag] ||= 'latest'
       yaml[:docker_mounts] ||= []
       yaml[:config_path] ||= '/etc/uphold'

lib/engine.rb

@@ -47,6 +47,7 @@ def start_container
         Docker::Image.create('fromImage' => @docker_image, 'tag' => @docker_tag)
       end
 
+      logger.debug @docker_image + ":" + @docker_tag
       @container = Docker::Container.create(
         'Image' => "#{@docker_image}:#{@docker_tag}",
         'Env' => @docker_env

lib/engines/postgresql.rb

@@ -5,15 +5,31 @@ class Postgresql < Engine
       def initialize(params)
         super(params)
         @docker_image ||= 'postgres'
-        @docker_tag ||= '9.5.0'
-        @docker_env ||= ["POSTGRES_USER=#{@database}", "POSTGRES_DB=#{@database}"]
+        @docker_tag ||= '9.5'
+        @file_type ||= 'sql'
         @port ||= 5432
+        @username ||= 'postgres'
+        @docker_env ||= ["POSTGRES_USER=#{@username}", "POSTGRES_DB=#{@database}"]
         @sql_file = params[:sql_file] ||  'PostgreSQL.sql'
+        @dates = params[:dates]
+
+        @dates.each_with_index do |date_settings, index|
+	  date_format = date_settings[:date_format] || '%Y-%m-%d'
+	  date_offset = date_settings[:date_offset] || 0
+	  date_string = '{date' + index.to_s + '}'
+	  @sql_file.gsub!(date_string, (Date.today - date_offset).strftime(date_format))
+	end
       end
 
       def load_backup(path)
         Dir.chdir(path) do
-          run_command("psql --no-password --set ON_ERROR_STOP=on --username=#{@database} --host=#{container_ip_address} --port=#{@port} --dbname=#{@database} < #{@sql_file}")
+          if @file_type.include?("sql")
+            run_command("psql --no-password --set ON_ERROR_STOP=on --username=#{@username} --host=#{container_ip_address} --port=#{@port} --dbname=#{@database} < #{@sql_file}")
+          elsif @file_type.include?("dump")
+             run_command("pg_restore --no-password --set ON_ERROR_STOP=on --username=#{@username} --host=#{container_ip_address} --port=#{@port} --dbname=#{@database} < #{@sql_file}")
+          else
+            raise 'File Type parameter in Postgresql Driver is invalid.'
+          end
         end
       end
     end

lib/transport.rb

@@ -10,11 +10,17 @@ def initialize(params)
       @path = params[:path]
       @filename = params[:filename]
       @folder_within = params[:folder_within]
+      @dates = params[:dates]
+      @compressed = params[:compressed]
+
+      @dates.each_with_index do |date_settings, index|
+        date_format = date_settings[:date_format] || '%Y-%m-%d'
+        date_offset = date_settings[:date_offset] || 0
+	date_string = "{date" + index.to_s + "}"
+        @path.gsub!(date_string, (Date.today - date_offset).strftime(date_format))
+        @filename.gsub!(date_string, (Date.today - date_offset).strftime(date_format))
+      end
 
-      @date_format = params[:date_format] || '%Y-%m-%d'
-      @date_offset = params[:date_offset] || 0
-      @path.gsub!('{date}', (Date.today - @date_offset).strftime(@date_format))
-      @filename.gsub!('{date}', (Date.today - @date_offset).strftime(@date_format))
     end
 
     def fetch

lib/transports/local.rb

@@ -11,11 +11,16 @@ def fetch_backup
           tmp_path = File.join(@tmpdir, File.basename(file_path))
           logger.info "Copying '#{file_path}' to '#{tmp_path}'"
           FileUtils.cp(file_path, tmp_path)
-          decompress(tmp_path) do |_b|
-          end
-          File.join(@tmpdir, @folder_within)
+	  if @compressed
+            decompress(tmp_path) do |_b|
+            end
+            File.join(@tmpdir, @folder_within)
+	  else 
+	    @tmpdir
+	  end
         else
           logger.fatal "No file exists at '#{file_path}'"
+	  nil
         end
       end
 

lib/transports/s3.rb

@@ -17,10 +17,16 @@ def fetch_backup
         File.open(File.join(@tmpdir, File.basename(matching_file)), 'wb') do |file|
           logger.info "Downloading '#{matching_file}' from S3 bucket #{@bucket}"
           s3.get_object({ bucket: @bucket, key: matching_file }, target: file)
-          decompress(file) do |_b|
+          if @compressed
+            decompress(file) do |_b|
+            end
           end
+	      end
+        if @compressed  
+	        File.join(@tmpdir, @folder_within)
+        else
+          @tmpdir
         end
-        File.join(@tmpdir, @folder_within)
       end
     end
   end

views/index.erb

@@ -8,9 +8,9 @@
     <tr>
       <td class="config_name"><%= c[:name] %> <a class="run" href="run/<%= c[:file] %>"></a></td>
       <td>
-        <% unless @logs[c[:name]].nil? %>
+        <% unless @logs[c[:file]].nil? %>
         <ul>
-          <% @logs[c[:name]].each do |log| %>
+          <% @logs[c[:file]].each do |log| %>
             <li>
               <a title="<%= log[:state] %>" class="<%= log[:state] %>" href="logs/<%= log[:filename] %>"></a> <%= epoch_to_datetime(log[:epoch]) %>
             </li>