Implement generic time series data model

[grahamcrowell]

analytical model

Subject

• Has attributes
• Has state processes
• Has event processes

Attribute

• Has a label
• Has a value

State

• Has a label
• Has a value
• Has a time span

Event

• Has a label
• Has a value
• Has a time point

Process

• Has a label
• Has a time granularity
• Is a map from a date to a state

Time Granularity

• label

Sent from my Samsung SM-J120W using FastHub

[grahamcrowell]

rationale

generic/abstract model

• reduces amount of code.
• increases flexibility

analysis

• is define metrics with interface something like:

trait Metric {
  def calculate(process: Process, dateId: Int) : Double
}

Metrics basically take a time series and a perform some calculation and return value of the metric for a given date.

Concrete examples of model components

Subjects

• Asset
  • symbol = "MSFT" (attribute)
  • industry = "Software" (attribute)
  • price process = ... (state process)
  • stock splits process = ... (event process)
  • dividend ex-date process = ... (event process)
• Portfolio
  • start date = 1999-10-23 (attribute)
  • name = ... (attribute)
  • market value process = ... (state process)
  • return process = ... (state process)
  • holdings process = ... (state process)
  • cash flows process = ... (event process)
  • trade process = ... (event process)
• (Patient)
• ... anything which has a State that changes over time

[grahamcrowell]

Process as spark dataset

Denormalize price data into Spark Dataset[State]

case class State(objectLabel, processLabel, dateId, value) 

• objectLabel id's the parent object which has this state at the time
  -- stock symbol, portfolio
• processLabel id's the process
  -- close price, market value

Start with strings determine if performance blocks progress

[grahamcrowell]

analytical data structure

single time series

• Subject
• Time axis ie vector of date intervals or points
• State process ie vector of state values

system of time synchronized processes

• Each process is associated with a single subject instance
• Process instances are indexed by standard interface id (symbol)
• Each process shares a single time axis
• Each process is assumed to have a accessible state value for each interval/point in the time axis

Map symbol to asset

[grahamcrowell]

@grahamcrowell this is basically the same as Spark's machine learning pipeline. Spark ml lib pipeline docs

[grahamcrowell]

setup aws for hadoop
http://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-gs-prerequisites.html