Loopy Clauses Without Loopy Loop

[Luis-Henriquez-Perez]

Table of Contents

1. Loopy Clauses Without Loopy Loop
   1. finding myself yearning for normal loops often
   2. loopy's convenience comes at the cost of increased syntax
   3. proposed solution: iter-block
      1. very rough sketch of iter block implementation
      2. example
   4. applications of iter-block
      1. iter-defun
      2. iter-defun example
      3. iter-defmacro
   5. conclusion

Loopy Clauses Without Loopy Loop

This issue concerns the idea of creating a macro iter-block that is similar to
loopy-iter except it does not put its body in a while loop by default.

finding myself yearning for normal loops often

Not that loopy is bad (it is an excellent package), but sometimes for very
simple loops the increased syntax is a bit overkill. That in an of itself is not
a problem, you might think. I can just use dolist or some other
seq-doseq. However, often I'm faced with a predicament–I want to use the
minimal syntax of a loop such as dolist or while, but I also want the power
of the loopy commands–usually loopy accumulation commands such as collect or
appending.

Ultimately, I end up doing something like in the example below. I use iter
only for the access it provides to loopy commands and exit it on its first
iteration, using my own loops inside of it.

(iter (dolist (a '(1 2 3))
	(sum sums a))
      (dolist (b '(dog cat mouse))
	(collect animals b))
      (return (list sums animals)))

loopy's convenience comes at the cost of increased syntax

Sometimes it's a pain to have finally-return, initially or after-do. Of
course we need them in loopy and loopy iter, because then there's no
distinguishing between something that's run before/after the loop or something
in it. But it is wordier than just plain lisp.

Nested loopy and loopy-iter are also typically very syntax heavy. Having to
to use loopy or loopy-iter subloops when you can have all variables on one
level can also be inconvenient.

In situations where you don't need control flow between nested loops (via
return-from or using something like (at loop-name (leave))) and where you
also don't need a specific iteration command that only loopy provides, I find
the combination of normal lisp expressions and loopy clauses very powerful. You
get the benefit of loopy commands without paying for it with increased syntax.

proposed solution: iter-block

iter-block or loopy-block would be the same as loopy-iter except it
wouldn't loop by default (or even at all). It would just allow you to use all
the loopy commands that it would make sense to use in it (since it doesn't loop
continue and skip may not make sense). You'd be able to initialize variables
with with declaratively instead of having to wrap all your code. And you'd be
able to use accumulation commands throughout your code.

very rough sketch of iter block implementation

It occured to me that perhaps the best way to nest loops is to use existing
loops combined with the power of loopy's syntax.

This is a very rough sketch of what I mean. Some problems are that using with
and without won't work in my example because they need to be in the top level.

(defmacro iter-block! (name &rest body)
  (declare (indent defun))
  (mmt-with-gensyms (result)
    `(iter (each ,(gensym) '(1))
	   (expr ,result (progn ,@body))
	   (finally-return ,result))))

example

The loopy "environment" is still useful even without using an iteration
command. The most powerful thing about it in my opinion is the ability to create
variables on the fly implicity via with and the acumulation commands instead of
having to wrap a large let around the body.

(iter-block! my-block
  (with (a 1) (b 2) (c 3))
  (while t
    (expr key (pop clause))
    (dolist (a '(1 2 3 4))
      (sum a))
    (while (and clause (not (keywordp it)) (funcall take-fn it))
      (collect taken (pop clause)))
    (prepending processed (funcall transform-fn (list (cons key taken) rest)))))

applications of iter-block

These are some cool ideas I had using iter. The ideas are not so novel as I see
you've have similar ones by creating loopy-let* and loopy-lambda. However,
they have my own spin added to them.

iter-defun

We can use this to extend other macros like defun. In the example below I add
destructuring to defun by using iter's destructuring and body of defun!
would be able to use the loopy clauses in its body without having to declare a
loopy loop.

(defmacro loopy-defun (name args &rest body)
  (iter-block! nil
    (with (result (gensym "result")))
    (dolist (arg args)
      (if (listp arg)
	  (collect with (list arg (gensym)))
	(collect without arg)
	(collect with (list arg arg))))
    `(defun ,name ,(mapcar #'cadr with)
       (iter-block! ,@with
	 ,@without
	 ,@body))))

iter-defun example

In this example I have a few nested loops. The advantage is apparent: we save
some text by just using while for both loops as opposed to (loopy (while...)) and we can still use the rich set of loopy commands. Another plus
is since the iter-block is not looping by default we could have it return
something normally–just by putting it at the end of the form (see processed
in my example) as opposed to using (finally-return).

Of course if we need specific loops like looping through the cons of a list with
cons or even if we need more complex loops, we'd use loopy or iter. But
for the looping of a list, when where we're not declaring variables with with,
it's nice to use while or dolist because its syntax is more concise.

The drawback we lose the ability to abort from specific loops or skip specific
loops. skip won't work properly. Using it would be equivalent to (return nil). So we lose the capability to use return-from and skip.

(defun! oo-split-processed (clause)
  (with (a 1) (b 2) (c 3))
  (while clause
    (expr key (pop clause))
    (expr (key-fn take-fn transform-fn) (--first (funcall (car it) key) oo-bind-processers))
    (while (and clause (not (keywordp it)) (funcall take-fn it))
      (collect taken (pop clause)))
    (prepending processed (funcall transform-fn (list (cons key taken) rest))))
  processed)

;; Comparison.
(defun oo-split-processed (clause)
  (loopy main
	 (with (a 1) (b 2) (c 3))
	 (while clause)
	 (expr key (pop clause))
	 (expr (key-fn take-fn transform-fn) (--first (funcall (car it) key) oo-bind-processers))
	 (loopy (while (and clause (not (keywordp it)) (funcall take-fn it)))
		(at main (collect taken (pop clause))))
	 (expr new (funcall transform-fn (list (cons key taken) rest)))
	 (prepending processed (funcall transform-fn (list (cons key taken) rest)))
	 (finally-return processed)))

iter-defmacro

This would be the same as iter-defun except for defmacro.

conclusion

I think such a feature would be useful by providing an interesting alternative
to loopy and iter. All in all, a useful tool for particular cases.

[okamsn]

Sometimes it's a pain to have finally-return, initially or after-do. Of course
we need them in loopy and loopy iter, because then there's no distinguishing
between something that's run before/after the loop or something in it. But it
is wordier than just plain lisp.

Not that it affects your point, but I'll state for the record that after-do
and finally-do aren't quite the same as just using a progn. after-do only
runs if the loop completes and finally-do always runs, but neither affect the
loop's return value so that they don't override the effect of a command like
return. Their use creates code that works a bit like prog1.

These things have their place, but yes, I agree that loopy can be
noticeably wordier, especially for the simple cases. I think what you propose
makes sense.

For clarity, in case I missed it:

1. The implicit return value that comes from using the accumulation commands
   would be ignored, yes?

2. Returning from the loop/block early with cl-return prevents optimized
   accumulation variables from being finalized. This can be worked around with
   leave and after-do/finally-do, but it sounds like you want to avoid
   them. Are you planning on foregoing the optimized accumulations?

   For example an optimized collect builds lists in reverse, but the
   correct-order version is much slower (though still a bit faster than
   repeated nconc for very large lists).

3. For the examples where you do use a loop, where do you find loopy-iter
   inadequate (besides the finally-return)?

   For example, in

   (defun! oo-split-processed (clause)
      (with (a 1) (b 2) (c 3))
      (while clause
        (expr key (pop clause))
        (expr (key-fn take-fn transform-fn) (--first (funcall (car it) key) oo-bind-processers))
        (while (and clause (not (keywordp it)) (funcall take-fn it))
          (collect taken (pop clause)))
        (prepending processed (funcall transform-fn (list (cons key taken) rest))))
      processed)
   
    ;; Comparison.
    (defun oo-split-processed (clause)
      (loopy main
    	 (with (a 1) (b 2) (c 3))
    	 (while clause)
    	 (expr key (pop clause))
    	 (expr (key-fn take-fn transform-fn) (--first (funcall (car it) key) oo-bind-processers))
    	 (loopy (while (and clause (not (keywordp it)) (funcall take-fn it)))
    		(at main (collect taken (pop clause))))
    	 (expr new (funcall transform-fn (list (cons key taken) rest)))
    	 (prepending processed (funcall transform-fn (list (cons key taken) rest)))
    	 (finally-return processed)))

   would

   ;; Didn't test these.
   
   ;; This expansion doesn't seem to optimize the `collect' uses,
   ;; so I need to fix that.  EDIT: Should be fixed.
   (loopy-iter
    main
    (flag lax-naming)
    (with (a 1) (b 2) (c 3))
    (while clause)
    (loopy-let* ((key (pop clause))
                 ((key-fn take-fn transform-fn) (--first (funcall (car it) key)
                                                         oo-bind-processers))
                 (taken (loopy (while (and clause
                                           (not (keywordp it))
                                           (funcall take-fn it)))
                               (collecting (pop clause)))))
      (collecting (funcall transform-fn (list (cons key taken) rest))
                  :at start)))
   
   ;; This version optimizes both `collect's correctly.
   (loopy main
          (with (a 1) (b 2) (c 3))
          (while clause)
          (set key (pop clause))
          (set (key-fn take-fn transform-fn) (--first (funcall (car it) key)
                                                      oo-bind-processers))
          (set taken (loopy (while (and clause
                                        (not (keywordp it))
                                        (funcall take-fn it)))
                            (collect (pop clause))))
          (collect (funcall transform-fn (list (cons key taken) rest))
                   :at start))

   be sufficient for your needs for now?

[Luis-Henriquez-Perez]

The implicit return value that comes from using the accumulation commands
would be ignored, yes?

Yes. As I imagined, loopy-block would not use iteration clauses because it's not a loop, only an enviroment where you can use loopy accumulation clauses (and others like `expr). So it wouldn't make as much sense to have an implicit return value.

Returning from the loop/block early with cl-return prevents optimized
accumulation variables from being finalized. This can be worked around with
leave and after-do/finally-do, but it sounds like you want to avoid
them. Are you planning on foregoing the optimized accumulations?

Is this example I gave what you're referring to with returning from loop/block early? I don't want to forego the optimizations. I returned because it was the first thing I though of to stop the loop and return the return value I wanted. But now that you mention it, you're right that a more precise choice would have been a combination of leave and after-do/finally-do here. The point of the example was more to convey what I wish I could have written.

I also want to clarify I wouldn't say I want to avoid them. I wouldn't go out of my way not to use them in a situation when they are useful. However, I want to use them if they are necessary.

(iter (dolist (a '(1 2 3))
	(sum sums a))
      (dolist (b '(dog cat mouse))
	(collect animals b))
      (return (list sums animals)))

However, with loopy-block I would use return because it would not be a loop like loopy and loopy-iter. I'd do this, instead.

(loopy-block nil
  (dolist (a '(1 2 3))
    (sum sums a))
  (dolist (b '(dog cat mouse))
    (collect animals b))
  (list sums animals))    

For the examples where you do use a loop, where do you find loopy-iter
inadequate (besides the finally-return)?

I would not say I found loopy inadequate per se. It is adequate in the sense of it can get the job done. The purpose of my example was not so much about solving that particular problem. It was more of commenting on my general feeling when writing a loop. I intuitively wanted to write example below to solve the problem. It felt natural, easy to write--it practically wrote itself. Whereas, I had to think more for concocting the loopy example.

 (with (a 1) (b 2) (c 3))
   (while clause
     (expr key (pop clause))
     (expr (key-fn take-fn transform-fn) (--first (funcall (car it) key) oo-bind-processers))
     (while (and clause (not (keywordp it)) (funcall take-fn it))
       (collect taken (pop clause)))
     (prepending processed (funcall transform-fn (list (cons key taken) rest))))
   processed

On a sidenote, since loopy is already providing macros such as loopy-let* and loopy-lambda; perhaps it should provide convenient wrappers on some existing loops. For example, loopy-dolist, loopy-doseq, and loopy-while. Unlike the originals they'd have destructuring support you could have some clauses in it. Of course, they would not provide the same functionality as the loopy macro itself. Don't know if this is a good idea.

[Luis-Henriquez-Perez]

I found an example where I use the return in my code. Thought you might be interested in a real life example I have in my config--it should be better than a made up one. Had I recalled this at the time I would have definitely used this.

Some context. I liked the let-alist macro but it had some problems.

1 . One is that it is not general enough to work for other common key-value structures like plists.

2. The other is that it has a problem with backquoted forms.

Specifically, the . character the macro uses was a bad choice because it conflicts with the cons pair syntax. Though for some reason I use it in with-map! still, don't remember why--but at least its simple to change.

3. Often times keys in a key-val structure uses keywords as keys instead of symbols. And to specify keywords I'd have to do .:keyname--so I needed the extra : character. I found this bit annoying. So I made the asterix character refer to these. So you could write *keyname instead of .:keyname.

(defun oo-keyword-intern (&rest args)
  "Return ARGS as a keyword."
  (declare (pure t) (side-effect-free t))
  (apply #'oo-symbol-intern ":" args))

(defun oo-anaphoric-symbol-p (obj regexp)
  "Return non-nil if OBJ is a special macro symbol matching REGEXP."
  (and (symbolp obj) (s-matches-p regexp (symbol-name obj))))

;; I want to use loopy for this in the future. Would probably be more efficient.
(defun oo-anaphoric-symbols (form regexp &optional contains-p)
  "Return the special macro symbols in FORM that match REGEXP."
  (-uniq (--select (or (oo-anaphoric-symbol-p it regexp)
		       (when contains-p (s-matches-p regexp (symbol-name it))))
		   (-flatten form))))

(defun oo-dot-symbols (body)
  "Return a list of dot symbols in BODY."
  (oo-anaphoric-symbols body (rx "." (group (1+ (not white))))))

(defun oo-dot-symbol-name (symbol)
  "Return name of dot symbol."
  (oo-anaphoric-symbol-name symbol (rx "." (group (1+ (not white))))))

(defun oo-anaphoric-symbol-name (symbol regexp)
  "Return first group for the REGEXP matching SYMBOL."
  (awhen (nth 1 (s-match regexp (symbol-name symbol)))
    (intern it)))

(defmacro with-map! (map &rest body)
  "Let-bind dotted symbols to their values in BODY.
This is similar to `let-alist' but map can be any key value structure."
  (declare (indent 1))
  (iter (dolist (dot-sym (oo-dot-symbols body))
	  (expr name (macroexp-quote (oo-dot-symbol-name dot-sym)))
	  (collect let-binds (list name dot-sym)))
	(dolist (asterix-sym (oo-asterix-symbols body))
	  (expr name (oo-keyword-intern (oo-asterix-symbol-name asterix-sym)))
	  (collect let-binds (list name asterix-sym)))
	(return `(map-let ,let-binds ,map ,@body))))

Example usages.

;; an alist
(with-map! '((a . 1) (b . 2) (c . 3))
  (+ .a .b .c))

;; => 6

;; symbols as keys
(with-map! '(a 1 b 2 c 3)
  (+ .a .b .c))

;; => 6

;; keywords as keys
(with-map! '(:a 1 :b 2 :c 3)
  (+ *a *b *c))

;; => 6

;; Keys not found are bound to nil.
(with-map! '(a 1 c 3)
  (list .a .b .c))

;; => (1 nil 3)

I think what I can divine from my usage is that sometimes I tend to use iter for it's clauses. Or that sometimes I'd like to have the option of not looping and just using the clauses.

For curiosity's sake I'll include another one too. I have tons of crazy macros in my config. This is similar to the above except when a key is not present, the body is not evaluated at all. Same thing, I kind of wanted to have the environment. I'm a bit lazy to post the helper functions here. But let me know if you want to see them too.

(defmacro! when-map! (!map &rest body)
  "Same as `with-map!' but if any key is not in MAP ignore body and return nil."
  (declare (indent 1))
  (iter (with (dots (->> (oo-dot-symbols body)
			 (-map #'oo-dot-symbol-name)))
	      (astx (->> (oo-asterix-symbols body)
			 (-map #'oo-asterix-symbol-name)
			 (-map #'oo-keyword-intern))))

	(dolist (sym (append dots astx))
	  (collect preds `(member ',sym ,!map-keys)))

	(return `(wrap! ((with-map! ,!map)
			 (let ((,!map-keys (map-keys ,!map))))
			 (when (and ,@preds)))
		   ,@body))))

(when-map! '(a 1 c 3)
  (list .a .b .c))

; => nil