Roadmap

[hauntsaninja]

1. Allow use of magic variables in config
   This would essentially allow user defined magic variables / give pyp more macro like abilities, e.g, you could have j = json.loads(stdin) in your config and then run j[0]["whatever"]. An implementation that mostly works is pretty straightforward. The only tricky part in getting something that works for most use cases is that we wouldn't want to define other need parts of the config after magic variables are defined (e.g., inside a loop). We should also allow config definitions to depend on config definitions that depend on magic variables.
2. Allow fancy use of magic variables in config
   The suggestion in 1 could be more complicated. Since we're now introducing two locations for config definitions (previously only after missing imports, now also before processing input), we could allow config definitions to depend on --before definitions. This could unlock some cool possibilities.
   It would also be cool to allow custom magic variables that loop over some custom object (e.g., read in CSV and do something for each record). I'm not sure what the syntax for this should be, but since top-level loops aren't really useful in a config, we could maybe invent some special semantics for them. It would be a nice unification if the out of the box magic variables were essentially "default config".
3. Allow pyp flags to be set in config
   Currently pyp only has --define-pypprint, but we could allow that to be configured in pyp's config. This should be very straightforward.
4. (not currently merged) Transparently optimise lines to a generator, if possible
   Currently lines is always a list. It's impossible to do this right in general, but one way that could work well is to optimise if we detect it's only mentioned once and that mention is either in a comprehension, loop or the second argument to map.
   Possible implementations were investigated in if lines is used once, define it as a generator #23 and Optimise lines to a generator, if possible #25. The version in if lines is used once, define it as a generator #23 is probably better opted in to by using the new user defined magic variables feature implemented in 1; I didn't really see wins from Optimise lines to a generator, if possible #25 that justified the change. Willing to reconsider if this is an actual user painpoint.
5. Improve support for conditional definitions in config.
   We have some basic support, but it's broken. I have some patches that fix things, but complicate the code in ways that I'm not sure I want to commit to.
   (edit: conditional definitions are no longer fairly broken and are instead just sometimes surprising in a bad way)
6. Improve find_names to take scopes into account.
   This could help simplify config parsing, especially regarding the above point, if we broaden our return value to separate out maybe defined variables. Two notes, a) currently we only use self.defined in get_valid_name, which we only use in one place, so that's the only thing defined variables definitely needs to support (i.e., make sure output doesn't end up in a weird scope), and b) we should make sure we're optimistic about user code definitions and pessimistic about config definitions, so we don't weirdly break things.
   We now have scope support in name finding, which has allowed unification with the config parsing code that used to be.
   Things that can be further improved: a) conditional definitions (somewhat done), b) scoping semantics for comprehensions (done), late binding, nonlocal / global

I am open to contributions (maybe other than for 1 and 2), but please ask me first. Also fair warning, for this project, I will likely be a little picky / may refactor completely fine contributions as I see fit.

[wangjiawen2013]

I am using bash shell and xonsh shell. But I hate the commands of sed/awk, I cannot remember them. Pyp is a good complementation of xonsh, because xonsh is lack of a oneliner.