random_notes.txt

class ExecutionManager
  - local variables
    - idtothreadmanager = map from threadmanager ID to threadmanager object
    - HIDcounter = value of next threadmanager ID for generation
    - idtopipeline = which pipeline each threadmanager is running on
    - idtostatus = one of "new" "waitingfortaskfunction" "waitingforchildren" "finished"
    - idtowaiting = the ids (either threadmanager ids or queue ids) that the threadmanager is waiting on
    - QIDcounter = values of next queue ID
    - qidqueue = the QIDs in order of request time
    - funcqueue = the taskfunction to perform for each respective entry in queue
    - envqueue = the variable set for each taskfunction in queue
    - taskfunctionresults = taskfunctions that have finished execution, map from QID to variable set
    - workerrefs = object ids for current worker functions, map from oid to the qid it's processing
  - spawnThreadManager(self,pipeline,environment) = spawns a new ThreadManager for a given pipeline and given initial variables
    - returns HID
  - spawnThreadManagers(self,pipelines,environments) = spawns a set of threadmanagers
    - returns list of HIDs
  - cyclethrough(self) = goes through set of all threads and determines if any of the waiting conditions have been met. If so, runs the threadmanager with the results to get a new request
  - processThreadManagerRequest(self,HID,request) = generate the processes/tasks that the thread is requesting
  - addToQueue(self,taskfunction,variables) = adds a particular task taskfunction to the queue
    - returns the ID
  - consumeQueue(self) = spawns ray processes to handle tasks in queue
    - if there's tasks in queue, always waits for at least 1 task to finish execution
    - continues spawning / waiting on processes until the number of tasks remaining is less than the number of max
  - run(self,pipeline) = run a pipeline
    - create a thread 0, then alternate between cyclethrough() and consumeQueue() until the thread 0 is finished
    - return variable set of primary thread


class ThreadManager
  - local variables
    - ID
    - location = index within pipeline
    - variables = dict of var names to values
    - special = a special state, usually None but used for ParallelModules to keep track of control/body
  - run(self, pipeline, taskfunctionresult = None, childresults = None) = provides the data that was previously requested (if any) by this threadmanager
  - returns a new request, which of 3 possible types
    - finished
    - waitingforblock: taskfunction + variable set
    - waitingforchildren: list of pipelines + initial variable sets for each pipeline


class Pipeline
  - local variables
    - modulelist = list of modules in
    - varplaceholdermap = dictionary from where placeholders come from
  - add(self,module) = add a module to pipeline
  - getModuleByIndex(self, index) = return the module in a given index or None if none found
  - apply(self,func,args...) = create a TaskFunctionModule from a given function with certain inputs
    - returns a VariablePlaceholder
  - placeholdersplit(self,parent,children) = update the

class VariablePlaceholder
  - local variables
    - pipeline = the pipeline this was created from
    - indexlist = list of indices to indicate the splits necessary to arrive at this variable
    - uniquename = an internal, meaningless name for this variable
  - split(self,number) = return a new set of variableplaceholders based on a "split" operation so that functions which return a tuple can have te components separated

class Module
  - just a wrapper class for other module types


class TaskFunctionModule
  - core module type
  - local variables
    - taskfunction = the function for the module
  - getTaskFunction(self) = returns the function / code for the module
    - the function takes a dictionary (representing the variable set) and returns a new dictionary (the new variable set)
  - taskfromfunction(func,inputvars,outputvars) = given a function, and what to pass in/out of it, create a taskfunction that performs it
  - taskfromcodeblock(codeblock,...) = given a code block, turn it into a taskfunction

class PipelineModule
  - core module type
  - local variables
    - pipeline = the pipeline to perform when this module is reached
  - getPipeline(self) = return the child pipeline
    - this pipeline's variable set is initialized to the parent threadmanager's variable set, and then the parent's variable set is updated based on results
  - setPipeline(...) = set the child to a given pipeline

class ParallelModule
  - core module type
  - local variables
    - controlpipeline = pipeline that is run to determine how many body processes need to be run
    - bodypipeline = pipeline that comprises the "body" of the loop, where multiple copies are executed in parallel
  - getControl(...) = return control pipeline
  - getBody(...) = return body pipeline
  - setControl(...) = set control pipeline
  - setBody(...) = set body pipeline


---------------------------------------

class Trace
  - local variables
    - val = anything
    - meta = a set of strings (labels)
  - overrides for essentially all object functions


class TraceDtype(ExtensionDtype)
  - helper class to end pandas datatypes

class TraceArray(ExtensionArray, ExtensionScalarOpsMixin)
  - helper class for Traced dataframe

----------------------------------------

helper functions

DataFrameFromDict(data) = conversion from a dictionary of column:values
  - returns a dataframe

ParamSet(orig,df) = labeling a dataframe according to an origin

note: a Selector is a function that takes a single row from a dataframe and returns True or False

ModifyViaSelector(df,template,selector=None,orig='mvs') = use a selector to select specific lines from a dataframe, and then replace those lines with new lines based on a template
  - can insert multiple new lines for each original line
  - template dataframe can specific unique modified values for each row
  - returns the new dataframe

OptSelector(optinfo,extra=None) = generate a selector based on keys to lists of options
  - the selector is such that it returns true for any row in a dataframe where the value in each column in a set of columns is a value from a list of options (a different list for each column)
  - return the selector

GenSinglePropertySelector(prop) = given a name of a property (column name), produce a SinglePropertySelector for that column
  - SinglePropertySelector(opts) = given a list of options, returns an OptSelector for the column to those options
    - returns a Selector
  - returns the SinglePropertySelector

constructSquareDf(ids) = from list of IDs, create a square dataframe with those rows and columns

untrace(data) = ...


note: a biselector is a function that takes two rows from two dataframes (a source and destination) and returns either True or False

FullBiselector(selectsrc, selectdest) = return a biselector that's an all-to-all mapping between two selectors

MatchBiselector(selectsrc, selectdest, matchprops=[]) = return a biselector similar to a FullBiselector but requiring that the source and destination both have equal values at a list of column names

NamePathwaySelector(srcname, destname, matchprops=[])

FillGridSelection(grid,df,biselector,fillvalue) = fill a grid dataframe with a specific value when the row/column fulfills the biselector's condition
- used to fill in efficacies and connection probabilities between source and destination populations

--------------------------

need a helper function to convert dictionary-of-dictionary data structures to dataframes for better ease-of-use

helper module type for ParamSets?

lots of code needs to be converted to modules

----------------------------------

pipeline for population-level parameters

  module
    celldefaults = ParamSet('celldef',{...})

  module
    popspecific = a dictionary of dictionaries, giving certain population names and which properties need to have the default values overrided

  module
    Receptordefaults = ParamSet('receptordef',{...})

  module
    basestim = dictionary of dictionaries, giving certain populations and which external stimulation parameters need to be set to which values

  module
    dpmndefaults = ParamSet('dpmndef',{...})

  module
    d1defaults = ParamSet('d1def',{...})

  module
    d2defaults = ParamSet('d2def',{...})

  module
    channels = ParamSet('actionchannel',{...})
    initialize action channel numbers

  - some way to create helper modules

  module
    popdata = pd.DataFrame()
    popdata['name'] = ['GPi','STNE','GPeP','D1STR','D2STR','LIP','Th','FSI','LIPI']
    popdata = trace(popdata,'init')

  module
    popdata = ModifyViaSelector(popdata,channels,SelName(['GPi','STNE','GPeP','D1STR','D2STR','LIP','Th']))

  module
    popdata = ModifyViaSelector(popdata,celldefaults)

  module
    for key,data in popspecific.items():
        params = ParamSet('popspecific',data)
        popdata = ModifyViaSelector(popdata,params,SelName(key))

  module
   popdata = ModifyViaSelector(popdata,receptordefaults)

  module
    for key,data in basestim.items():
       params = ParamSet('basestim',data)
       popdata = ModifyViaSelector(popdata,params,SelName(key))

  module
    popdata = ModifyViaSelector(popdata,dpmndefaults,SelName(['D1STR','D2STR']))
    popdata = ModifyViaSelector(popdata,d1defaults,SelName('D1STR'))
    popdata = ModifyViaSelector(popdata,d2defaults,SelName('D2STR'))

-----------------------------------------

pipeline for population-level connection data

  module
    simplepathways = pd.DataFrame(data for all connections)
    simplepathways = trace(simplepathways,'init')

  module
    create a new dataframe 'pathways' containing the individual copies of all pathways
    initialize 6 grids (3 neurotransmitters x 2 values (efficacy and connection probability))
    code is written

----------------------------------------------

pipeline for simulation control

  module: