Skip to content

Commit dd8bd8d

Browse files
author
tomas statter
committed
more work on reading config from yaml
1 parent 1ec99f7 commit dd8bd8d

11 files changed

+182
-254
lines changed

CHANGELOG.md

+88
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
# Change Log
2+
3+
## [Unreleased](https://github.com/autotelik/datashift/tree/HEAD)
4+
5+
[Full Changelog](https://github.com/autotelik/datashift/compare/v0.16.0...HEAD)
6+
7+
**Closed issues:**
8+
9+
- Issues with image import? [\#42](https://github.com/autotelik/datashift/issues/42)
10+
- Error when custom delimeter assigned [\#40](https://github.com/autotelik/datashift/issues/40)
11+
- FR: Admin gui for import/export [\#39](https://github.com/autotelik/datashift/issues/39)
12+
- Not working with Spree 3.0.5 stable [\#36](https://github.com/autotelik/datashift/issues/36)
13+
- uninitialized constant Thor::Sandbox::Datashift::Generate::ModelMapper in datashift:generate:csv [\#33](https://github.com/autotelik/datashift/issues/33)
14+
- Create has\_many records on import [\#32](https://github.com/autotelik/datashift/issues/32)
15+
16+
**Merged pull requests:**
17+
18+
- \#40 - Fix for custom delimiter set for csv files [\#41](https://github.com/autotelik/datashift/pull/41) ([wilgoszpl](https://github.com/wilgoszpl))
19+
- Complete rewrite and Restructure API [\#38](https://github.com/autotelik/datashift/pull/38) ([autotelik](https://github.com/autotelik))
20+
21+
## [v0.16.0](https://github.com/autotelik/datashift/tree/v0.16.0) (2016-03-10)
22+
[Full Changelog](https://github.com/autotelik/datashift/compare/v0.11.0...v0.16.0)
23+
24+
**Fixed bugs:**
25+
26+
- default available\_on in config file [\#4](https://github.com/autotelik/datashift/issues/4)
27+
28+
**Closed issues:**
29+
30+
- Uninitialized constant Logging \(NameError\) [\#29](https://github.com/autotelik/datashift/issues/29)
31+
- uninitialized constant DataShift::SpreeHelper::SpreeBaseLoader \(NameError\) [\#28](https://github.com/autotelik/datashift/issues/28)
32+
- Map records with ID [\#27](https://github.com/autotelik/datashift/issues/27)
33+
- More Complex Examples [\#26](https://github.com/autotelik/datashift/issues/26)
34+
- Can't link the github repo in the Gemfile when deploying on Heroku [\#23](https://github.com/autotelik/datashift/issues/23)
35+
- uninitialized constant DataShift::Logging [\#21](https://github.com/autotelik/datashift/issues/21)
36+
- All exports fail with "Please supply array of records to export" [\#20](https://github.com/autotelik/datashift/issues/20)
37+
- How can i add image with product when import csv with datashift? [\#19](https://github.com/autotelik/datashift/issues/19)
38+
- Setting specific values on variants [\#13](https://github.com/autotelik/datashift/issues/13)
39+
40+
**Merged pull requests:**
41+
42+
- Rails 4.2.0 changed 'type\_cast' name to type\_cast\_from\_database [\#31](https://github.com/autotelik/datashift/pull/31) ([redglory](https://github.com/redglory))
43+
- CSV support [\#25](https://github.com/autotelik/datashift/pull/25) ([pnomolos](https://github.com/pnomolos))
44+
- Travis Support [\#24](https://github.com/autotelik/datashift/pull/24) ([pnomolos](https://github.com/pnomolos))
45+
- Replace call to trim! with strip! [\#18](https://github.com/autotelik/datashift/pull/18) ([swrobel](https://github.com/swrobel))
46+
- Fixed file descriptors leak in paperclip\_loader [\#16](https://github.com/autotelik/datashift/pull/16) ([arr-ee](https://github.com/arr-ee))
47+
48+
## [v0.11.0](https://github.com/autotelik/datashift/tree/v0.11.0) (2012-10-23)
49+
[Full Changelog](https://github.com/autotelik/datashift/compare/v0.10.0...v0.11.0)
50+
51+
## [v0.10.0](https://github.com/autotelik/datashift/tree/v0.10.0) (2012-09-25)
52+
[Full Changelog](https://github.com/autotelik/datashift/compare/v0.9.0...v0.10.0)
53+
54+
**Closed issues:**
55+
56+
- spec/fixtures/spree not available [\#15](https://github.com/autotelik/datashift/issues/15)
57+
- Regression on image loading when using master [\#14](https://github.com/autotelik/datashift/issues/14)
58+
- Spree import does not allow sub categories of same name [\#12](https://github.com/autotelik/datashift/issues/12)
59+
- thor csv processing options are missing? [\#8](https://github.com/autotelik/datashift/issues/8)
60+
61+
## [v0.9.0](https://github.com/autotelik/datashift/tree/v0.9.0) (2012-09-03)
62+
[Full Changelog](https://github.com/autotelik/datashift/compare/v0.6.0...v0.9.0)
63+
64+
**Closed issues:**
65+
66+
- Not recognizing Jruby [\#11](https://github.com/autotelik/datashift/issues/11)
67+
- doesn't scan directory when loading images [\#9](https://github.com/autotelik/datashift/issues/9)
68+
- `perform\_load': .rb files not supported [\#7](https://github.com/autotelik/datashift/issues/7)
69+
- How to attach Images to product? [\#6](https://github.com/autotelik/datashift/issues/6)
70+
- spree variants and multiple option\_types [\#5](https://github.com/autotelik/datashift/issues/5)
71+
- What is your environment? [\#3](https://github.com/autotelik/datashift/issues/3)
72+
73+
## [v0.6.0](https://github.com/autotelik/datashift/tree/v0.6.0) (2012-05-18)
74+
[Full Changelog](https://github.com/autotelik/datashift/compare/v0.5.0...v0.6.0)
75+
76+
## [v0.5.0](https://github.com/autotelik/datashift/tree/v0.5.0) (2012-05-08)
77+
[Full Changelog](https://github.com/autotelik/datashift/compare/v0.4.1...v0.5.0)
78+
79+
## [v0.4.1](https://github.com/autotelik/datashift/tree/v0.4.1) (2012-03-19)
80+
[Full Changelog](https://github.com/autotelik/datashift/compare/v0.4.0...v0.4.1)
81+
82+
## [v0.4.0](https://github.com/autotelik/datashift/tree/v0.4.0) (2012-03-12)
83+
[Full Changelog](https://github.com/autotelik/datashift/compare/v0.0.2...v0.4.0)
84+
85+
## [v0.0.2](https://github.com/autotelik/datashift/tree/v0.0.2) (2012-02-12)
86+
87+
88+
\* *This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)*

lib/datashift/header.rb

+4-4
Original file line numberDiff line numberDiff line change
@@ -3,21 +3,21 @@
33
# Date :: Feb 2016
44
# License:: MIT
55
#
6-
# Details:: A header
7-
# The source text
6+
# Details:: A column header
87
#
98
module DataShift
109

1110
class Header
1211

13-
attr_accessor :source
12+
attr_accessor :source, :presentation
1413

1514
def initialize(source:)
1615
@source = source
16+
@presentation = source
1717
end
1818

1919
def to_s
20-
source
20+
presentation
2121
end
2222

2323
end

lib/datashift/mapping/data_flow_schema.rb

+30-70
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,7 @@ class DataFlowSchema
4949

5050
include DataShift::Logging
5151

52-
attr_reader :configuration, :nodes
52+
attr_reader :nodes, :raw_data, :yaml_data
5353

5454
def initialize
5555
@nodes = DataShift::NodeCollection.new
@@ -59,16 +59,19 @@ def sources
5959
@nodes.collect(&:method_binding).collect(&:source)
6060
end
6161

62-
def prepare_from_klass( klass )
62+
# Build the node collection from a Class, that is for each operator in scope
63+
# create a method binding and a node context, and add to collection.
64+
#
65+
def prepare_from_klass( klass, doc_context = nil )
66+
67+
context = doc_context || DocContext.new(klass)
6368

6469
@nodes = DataShift::NodeCollection.new
6570

6671
klass_to_model_methods( klass ).each_with_index do |mm, i|
67-
puts "DataFlowSchema - #{mm.operator}"
68-
6972
binding = MethodBinding.new(mm.operator, i, mm)
7073

71-
@nodes << DataShift::NodeContext.new( DocContext.new(klass), binding, i, nil)
74+
@nodes << DataShift::NodeContext.new(context, binding, i, nil)
7275
end
7376

7477
@nodes
@@ -97,32 +100,44 @@ def klass_to_model_methods(klass)
97100
end
98101

99102
model_methods
100-
101103
end
102104

103105
def prepare_from_file(file_name, locale_key = 'data_flow_schema')
104-
yaml = YAML.load( File.read(file_name) )
106+
@raw_data = File.read(file_name)
107+
yaml = YAML.load(raw_data)
105108

106109
prepare_from_yaml(yaml, locale_key)
107110
end
108111

109112
def prepare_from_string(text, locale_key = 'data_flow_schema')
113+
@raw_data = text
110114
yaml = YAML.load(text)
111115

112116
prepare_from_yaml(yaml, locale_key)
113117
end
114118

115119
def prepare_from_yaml(yaml, locale_key = 'data_flow_schema')
116120

121+
@yaml_data = yaml
122+
117123
@nodes = NodeCollection.new
118124

119125
raise RuntimeError.new("Bad YAML syntax - No key #{locale_key} found in #{yaml}") unless yaml[locale_key]
120126

121-
klass_section = yaml[locale_key]
127+
locale_section = yaml[locale_key]
122128

123-
klass = klass_section.keys.first
129+
klass = locale_section.keys.first
130+
131+
klass_section = locale_section[klass]
132+
133+
# The over all doc context
134+
doc = DocContext.new(MapperUtils.class_from_string_or_raise(klass))
124135

125-
yaml_nodes = klass_section[klass]['nodes']
136+
nodes.doc_context = doc
137+
138+
DataShift::Transformer.factory { |f| f.configure_from_yaml(klass, klass_section) }
139+
140+
yaml_nodes = klass_section['nodes']
126141

127142
logger.info("Read Data Schema Nodes: #{yaml_nodes.inspect}")
128143

@@ -131,11 +146,6 @@ def prepare_from_yaml(yaml, locale_key = 'data_flow_schema')
131146
raise 'Bad syntax in flow schema YAML - Nodes should be a sequence'
132147
end
133148

134-
nodes = []
135-
136-
# The over all doc context
137-
doc = DocContext.new(MapperUtils.class_from_string_or_raise(klass))
138-
139149
yaml_nodes.each_with_index do |keyed_node, i|
140150

141151
unless(keyed_node.keys.size == 1)
@@ -149,28 +159,23 @@ def prepare_from_yaml(yaml, locale_key = 'data_flow_schema')
149159
# - project:
150160
# heading:
151161
# source: "title"
152-
# destination: "Title"
162+
# presentation: "Title"
153163
# operator: title
154164
# operator_type: has_many
155-
#
156-
# - project_owner_budget:
157-
# heading:
158-
# destination: "Budget"
159-
# operator: owner.budget
160-
#
161165
logger.info("Node Data: #{keyed_node.inspect}")
162166

163167
# type one of ModelMethod.supported_types_enum
164-
165168
section = keyed_node.values.first
166169

167170
operator = Operator.new(section['operator'], :method) if(section['operator'])
168171

169172
model_method = ModelMethod.new( klass, operator, section['operator_type']) if(section['operator_type'])
170173

171-
method_binding = MethodBinding.new(section['heading']['source'], i, model_method)
174+
source = section.fetch('heading', {}).fetch('source', nil)
172175

173-
doc.headers.add( section['heading']['source'] )
176+
method_binding = MethodBinding.new(source, i, model_method)
177+
178+
doc.headers.add( source )
174179

175180
node = DataShift::NodeContext.new(doc, method_binding, i, nil)
176181

@@ -180,50 +185,5 @@ def prepare_from_yaml(yaml, locale_key = 'data_flow_schema')
180185
nodes
181186
end
182187

183-
private
184-
185-
attr_accessor :current_review_object
186-
187-
attr_accessor :model_object
188-
189-
def row_to_node_collection(review_object, row)
190-
# The section name, can be used as the state, for linking whole section, rather than at field level
191-
link_state = row[:link_state] || current_section
192-
link_title = row[:link_title]
193-
194-
@current_review_object = review_object
195-
196-
# The review partial can support whole objects, or low level data from method call defined in the DSL
197-
if(row[:method].blank?)
198-
node_collection.add(row[:title], review_object, link_state.to_s, link_title)
199-
else
200-
# rubocop:disable Style/IfInsideElse
201-
if(review_object.respond_to?(:each))
202-
review_object.each do |o|
203-
@current_review_object = o
204-
node_collection.add(row[:title], send_chain(row[:method]), link_state.to_s, link_title)
205-
end
206-
else
207-
node_collection.add(row[:title], send_chain(row[:method]), link_state.to_s, link_title)
208-
end
209-
210-
end
211-
end
212-
213-
def find_association(method_chain)
214-
arr = method_chain.to_s.split('.')
215-
216-
arr.inject(model_object) { |o, a| o.send(a) }
217-
end
218-
219-
def send_chain(method_chain)
220-
arr = method_chain.to_s.split('.')
221-
begin
222-
arr.inject(current_review_object) { |o, a| o.send(a) }
223-
rescue => e
224-
Rails.logger.error("Failed to process method chain #{method_chain} : #{e.message}")
225-
return I18n.t('.enrollment_review.missing_data')
226-
end
227-
end
228188
end
229189
end

lib/datashift/mapping/mapping_file_definitions.rb

-97
This file was deleted.

0 commit comments

Comments
 (0)