Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sift supports specifying types of scope arguments #48

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .rubocop.yml
Original file line number Diff line number Diff line change
Expand Up @@ -372,7 +372,7 @@ Style/StringLiterals:
Style/TrailingCommaInArguments:
Description: 'Checks for trailing comma in argument lists.'
StyleGuide: 'https://github.com/bbatsov/ruby-style-guide#no-trailing-array-commas'
EnforcedStyleForMultiline: comma
EnforcedStyleForMultiline: no_comma
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was driving me nuts, but I'm definitely willing to revert this.

SupportedStylesForMultiline:
- comma
- consistent_comma
Expand Down
57 changes: 57 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,63 @@ end
Passing `?filters[user_posts_on_date]=10/12/20` will call the `user_posts_on_date` scope with
`10/12/20` as the the first argument, and will grab the `user_id` and `blog_id` out of the params and pass them as a hash, as the second argument.

### Scope Arguments with Types

Sometimes you need the argument to a scope to be parsed using Sift types. Scope argument types can be specified by passing a hash to the `type:` keyword,
as follows:

```ruby
class Post < ActiveRecord::Base
scope :with_priority, ->(int_array) { where(priority: int_array) }
end

class PostsController < ApplicationController
include Sift

filter_on :with_priority, type: { scope: :int }

def index
render json: filtrate(Post.all)
end
end
```
Passing `?filters[with_priority]=[1,2]` will call the `with_priority` scope with the array, `[1, 2]`, instead of `"[1,2]"`
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There doesn't seem to be much consensus over how arrays should be used in query parameters.
https://medium.com/raml-api/arrays-in-query-params-33189628fa68

But =1,2 appears to be a more common pattern than =[1, 2]

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sift uses the JSON standard for array parameters. My(unpopular?) opinion is that we should use either the Rails standard filters[foo][]=1&filters[foo][]=2 or POST JSON and use the X-http-method-override header to specify GET simulation.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do think this PR is not affected by that decision, however. We could change this code to parse however we want:

https://github.com/procore/sift/blob/master/lib/sift/value_parser.rb#L24-L33

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a fan of filters[foo][]=1&filters[foo][]=2 is that really a Rails standard? Can you explain X-http-method-override with GET simulation in more detail?

Copy link
Author

@andrew-wheeler andrew-wheeler May 13, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@njbbaer
By setting a header with key X-Http-Method-Override and value get in a JSON POST request (for example) you can specify that the request should be handled as a GET in servers that support this header (Rails servers).

So you can POST with a body of {"filters": {"foo": [1,2]}} with rails treating this as a GET with params of { filters: {foo: [1,2]}}. This sidesteps all issues with URL length, and allows your request to use actual JSON instead of mixing URL parameters with JSON. In practice it works really well. Here is example code from the vector_processing repo:

      def large_get(path, params, timeout: 180)
        url = vapid_uri_base + path
        retries = 0
        begin
          raw_response = RestClient::Request.execute(
            url: url,
            method: :post,
            headers: {'Accept' => 'application/json;charset=utf-8', 'Content-Type' => 'application/json', 'Authorization' => authorization.authorization_header, 'X-Http-Method-Override' => 'get'},
            open_timeout: 3,
            timeout: timeout,
            verify_ssl: VectorProcessing::Vapid::Authorization.verify_ssl?,
            payload: params.to_json
          )
          JSON.parse(raw_response.body.force_encoding('UTF-8'))
        rescue RestClient::Unauthorized
          retry if authorization.refresh_success? && (retries+=1) < 5
        rescue RestClient::ExceptionWithResponse => rcewr
          raise ErrorResponseException.new("Error Response #{rcewr.response&.code || 'missing'} to X-Http-Method-Override 'get' on POST '#{url}'")
        end
      end

This requires no change to Vapid; it's supported by Rails out-of-the-box.

`filters[with_priority]` can also be validated with the `:int` type using `filters_valid?` (see below).

Types can also be specified for `scope_params` like this:

```ruby
class Post < ActiveRecord::Base
scope :user_posts_on_date, ->(date, options) {
where(user_id: options[:user_id], blog_id: options[:blog_id], date: date)
}
end

class UsersController < ApplicationController
include Sift

filter_on :user_posts_on_date,
type: {
scope: [
:datetime,
{ blog_id: :int }
]
},
scope_params: [:user_id, :blog_id]

def show
render json: filtrate(Post.all)
end
end
```
Passing `?filters[user_posts_on_date]=2010-12-20&user_id=3blog_id=[4,5]` will result in the filter
`Post.all.user_posts_on_date(DateTime.parse('2010-12-20'), user_id: "3", blog_id: [4, 5])`.
Unfortunately, validation with `filters_valid?` is not yet supported on params used by scope options.

Note that if `scope_params` is omitted it will default to the keys provided the scope option types.
For example, `filter_on :foo, type: {scope: [:int, { bar: :int, baz: :int }]}` implicitly defines `scope_params`
of `[:bar, :baz]`

### Renaming Filter Params

A filter param can have a different field name than the column or scope. Use `internal_name` with the correct name of the column or scope.
Expand Down
9 changes: 8 additions & 1 deletion lib/procore-sift.rb
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
require "sift/filter"
require "sift/filter/scope"
require "sift/filter/where"
require "sift/filter_validator"
require "sift/filtrator"
require "sift/sort"
Expand Down Expand Up @@ -66,7 +68,12 @@ def sort_fields

class_methods do
def filter_on(parameter, type:, internal_name: parameter, default: nil, validate: nil, scope_params: [])
filters << Filter.new(parameter, type, internal_name, default, validate, scope_params)
filters <<
if Sift::Filter.scope_type?(type)
Sift::Filter::Scope.new(parameter, type, internal_name, default, validate, scope_params)
else
Sift::Filter::Where.new(parameter, type, internal_name, default, validate)
end
end

def filters
Expand Down
49 changes: 25 additions & 24 deletions lib/sift/filter.rb
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,16 @@ module Sift
# Filter describes the way a parameter maps to a database column
# and the type information helpful for validating input.
class Filter
attr_reader :parameter, :default, :custom_validate, :scope_params
attr_reader :parameter, :default, :custom_validate

def initialize(param, type, internal_name, default, custom_validate = nil, scope_params = [])
def self.scope_type?(type)
type == :scope || type.respond_to?(:key?)
end

def initialize(param, type, internal_name, default, custom_validate = nil)
@parameter = Parameter.new(param, type, internal_name)
@default = default
@custom_validate = custom_validate
@scope_params = scope_params
raise ArgumentError, "scope_params must be an array of symbols" unless valid_scope_params?(scope_params)
raise "unknown filter type: #{type}" unless type_validator.valid_type?
end

Expand All @@ -21,10 +23,8 @@ def validation(_sort)
def apply!(collection, value:, active_sorts_hash:, params: {})
if not_processable?(value)
collection
elsif should_apply_default?(value)
default.call(collection)
else
handler.call(collection, parameterize(value), params, scope_params)
default_or_handler(value, params).call(collection)
end
end
# rubocop:enable Lint/UnusedMethodArgument
Expand All @@ -49,10 +49,21 @@ def param
parameter.param
end

protected

# Subclasses should override. Default behavior is to return none
def handler(_value, _params)
->(collection) { collection.none }
end

private

def parameterize(value)
Copy link
Author

@andrew-wheeler andrew-wheeler May 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The handlers take over responsibility for parsing the raw values from the request. This allows the scope handler to use its types to choose the right parsing strategy.

ValueParser.new(value: value, type: parameter.type, options: parameter.parse_options).parse
def default_or_handler(value, params)
if should_apply_default?(value)
default
else
handler(value, params)
end
end

def not_processable?(value)
Expand All @@ -63,22 +74,12 @@ def should_apply_default?(value)
value.nil? && !default.nil?
end

def mapped_scope_params(params)
scope_params.each_with_object({}) do |scope_param, hash|
hash[scope_param] = params.fetch(scope_param)
def validation_type
if type != :scope || scope_types.empty?
type
else
scope_types.first
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only a single scope type is allowed right? If thats the case should the argument be changed to scope_type with a default value of nil?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another thought is that scope_type param is unrelated to other filters. Should we create a different filter class to have better argument cohesion for scopes and so that way we aren't complicating other filters with logic specific to scopes?

Copy link
Author

@andrew-wheeler andrew-wheeler May 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

scope_types can have something like [:int, {foo: :datetime}] or [:int]. I can definitely see making some kind of inheritance / mixin relationship between new classes WhereFilter and ScopeFilter. I think they both support the default: option so there's some overlap. I totally agree that the frequently unused scope_params and scope_types arguments are a smell.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What do you think of this @amayer171 ?
#49

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

merged...

end
end

def valid_scope_params?(scope_params)
scope_params.is_a?(Array) && scope_params.all? { |symbol| symbol.is_a?(Symbol) }
end

def handler
parameter.handler
Copy link
Author

@andrew-wheeler andrew-wheeler May 5, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can't have Parameter be responsible for the handler, because scope filters need more than just the "chief" parameter to handle.

end

def supports_ranges?
parameter.supports_ranges?
end
end
end
63 changes: 63 additions & 0 deletions lib/sift/filter/scope.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
# frozen_string_literal: true

module Sift
class Filter
class Scope < Sift::Filter
attr_reader :scope_params, :scope_types

def initialize(param, raw_type, internal_name, default, custom_validate = nil, scope_params = [])
super(param, :scope, internal_name, default, custom_validate)
@scope_types = []
@scope_params = scope_params || []
raise ArgumentError, "scope_params must be an array of symbols" unless valid_scope_params?(scope_params)

if raw_type.respond_to?(:key?)
@scope_types = Array(raw_type[:scope]).compact
validate_scope_types!

@type_validator = Sift::TypeValidator.new(param, @scope_types.first || :scope)
end
end

protected

def handler(value, params)
Sift::ScopeHandler.new(value, mapped_scope_params(params), parameter, scope_types)
end

private

def mapped_scope_params(params)
scope_params.each_with_object({}) do |scope_param, hash|
hash[scope_param] = params.fetch(scope_param)
end
end

def valid_scope_params?(scope_params)
scope_params.is_a?(Array) && scope_params.all? { |symbol| symbol.is_a?(Symbol) }
end

def validate_scope_types!
return if scope_types.empty?

unless Sift::TypeValidator.new(parameter.param, scope_types.first).valid_type?
raise ArgumentError, "scope_types must contain a valid filter type for the scope parameter"
end
return if scope_types.size == 1

if scope_types.size > 2 || !valid_scope_option_types!(scope_types[1])
raise ArgumentError, "type: scope: expected to have this structure: [type, {#{scope_params.map { |sp| "#{sp}: type" }.join(', ')}}]"
end
end

def valid_scope_option_types!(hash)
valid_form = hash.respond_to?(:keys) && hash.all? { |key, value| Sift::TypeValidator.new(key, value).valid_type? }
if valid_form && scope_params.empty?
# default scope_params
@scope_params = hash.keys
end
valid_form && (hash.keys - scope_params).empty?
end
end
end
end
13 changes: 13 additions & 0 deletions lib/sift/filter/where.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# frozen_string_literal: true

module Sift
class Filter
class Where < Sift::Filter
protected

def handler(value, _params)
Sift::WhereHandler.new(value, parameter)
end
end
end
end
30 changes: 0 additions & 30 deletions lib/sift/parameter.rb
Original file line number Diff line number Diff line change
Expand Up @@ -8,35 +8,5 @@ def initialize(param, type, internal_name = param)
@type = type
@internal_name = internal_name
end

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parameter loses responsibility for parsing values because we want to parse things other than just the "chief" parameter.

def parse_options
{
supports_boolean: supports_boolean?,
supports_ranges: supports_ranges?,
supports_json: supports_json?
}
end

def handler
if type == :scope
ScopeHandler.new(self)
else
WhereHandler.new(self)
end
end

private

def supports_ranges?
![:string, :text, :scope].include?(type)
end

def supports_json?
type == :int
end

def supports_boolean?
type == :boolean
end
end
end
34 changes: 23 additions & 11 deletions lib/sift/scope_handler.rb
Original file line number Diff line number Diff line change
@@ -1,24 +1,36 @@
module Sift
class ScopeHandler
def initialize(param)
@param = param
def initialize(raw_value, raw_scope_options, parameter, scope_types)
@value = ValueParser.new(value: raw_value, type: scope_types.first || parameter.type).parse
@param = parameter
@scope_options = parsed_scope_options(raw_scope_options, scope_types)
end

def call(collection, value, params, scope_params)
collection.public_send(@param.internal_name, *scope_parameters(value, params, scope_params))
def call(collection)
collection.public_send(@param.internal_name, *scope_parameters)
end

def scope_parameters(value, params, scope_params)
if scope_params.empty?
[value]
private

def scope_parameters
if @scope_options.present?
[@value, @scope_options]
else
[value, mapped_scope_params(params, scope_params)]
[@value]
end
end

def mapped_scope_params(params, scope_params)
scope_params.each_with_object({}) do |scope_param, hash|
hash[scope_param] = params.fetch(scope_param)
def parsed_scope_options(raw_scope_options, scope_types)
return nil if raw_scope_options.empty?

scope_option_types = scope_types[1] || {}
raw_scope_options.each_with_object({}) do |(key, raw_param), hash|
hash[key] =
if scope_option_types[key].present?
ValueParser.new(value: raw_param, type: scope_option_types[key]).parse
else
raw_param
end
end
end
end
Expand Down
29 changes: 19 additions & 10 deletions lib/sift/value_parser.rb
Original file line number Diff line number Diff line change
@@ -1,11 +1,8 @@
module Sift
class ValueParser
def initialize(value:, type: nil, options: {})
@value = value
@supports_boolean = options.fetch(:supports_boolean, false)
@supports_ranges = options.fetch(:supports_ranges, false)
@supports_json = options.fetch(:supports_json, false)
def initialize(value:, type: nil)
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ValueParser can be responsible for determining supports_boolean, supports_ranges, and supports_json from the type, so it doesn't need to be passed in. This helps us parse values for things other than the "chief" parameter.

@value = normalized_value(value, type)
@type = type
end

def parse
Expand Down Expand Up @@ -34,22 +31,34 @@ def array_from_json

private

attr_reader :value, :type, :supports_boolean, :supports_json, :supports_ranges
def supports_ranges?
![:string, :text, :scope].include?(type)
end

def supports_json?
type == :int
end

def supports_boolean?
type == :boolean
end

attr_reader :value, :type

def parse_as_range?(raw_value=value)
supports_ranges && raw_value.to_s.include?("...")
def parse_as_range?(raw_value = value)
supports_ranges? && raw_value.to_s.include?("...")
end

def range_value
Range.new(*value.split("..."))
end

def parse_as_json?
supports_json && value.is_a?(String)
supports_json? && value.is_a?(String)
end

def parse_as_boolean?
supports_boolean
supports_boolean?
end

def boolean_value
Expand Down
2 changes: 1 addition & 1 deletion lib/sift/version.rb
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
module Sift
VERSION = "0.13.0".freeze
VERSION = "0.14.0".freeze
Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm bumping the minor here because if anything is using the Sift internals (Sift::ValueParser, Sift::Parameter) it could be broken.

end
Loading