Parallelize referred-syms-by-file&fullname (#320)

Part of #230

Author: vemv

CHANGELOG.md

@@ -3,6 +3,9 @@
 ## Unreleased
 
 #### Changes
+
+* [(Part of #230)](https://github.com/clojure-emacs/refactor-nrepl/issues/230): Parallelize various functionality
+  * This will have a noticeable improvement in e.g. clj-refactor.el's `cljr-slash` performance.
 * [#291](https://github.com/clojure-emacs/refactor-nrepl/issues/291): The `:ignore-errors` option will be honored in more places, making refactor-nrepl more robust in face of files not particularly meant to be part of the AST corpus.
   * Examples: WIP files, Moustache template files, scripts.
 * Upgrade Orchard
@@ -15,6 +18,7 @@
 * Honor internal `future-cancel` calls, improving overall responsiveness and stability.
 
 ### Bugs fixed
+
 * [#289](https://github.com/clojure-emacs/refactor-nrepl/issues/289): Fix an edge-case with involving keywords that caused find-symbol to crash.
 * [#305](https://github.com/clojure-emacs/refactor-nrepl/issues/305): Don't put `:as` or `:refer` on their own lines in the ns form, when the libspec is so long it causes the line to wrap.
 * [clojure-emacs/clj-refactor.el#459](https://github.com/clojure-emacs/clj-refactor.el/issues/459): `clean-ns` should conform to the style guide: `(:require` in the ns form should be followed by a newline.

src/refactor_nrepl/core.clj

@@ -99,15 +99,20 @@
 (defn find-in-dir
   "Searches recursively under dir for files matching (pred ^File file).
 
-  Note that files which are non-existant, hidden or build-artifacts
+  Note that files which are non-existent, hidden or build-artifacts
   are pruned by this function."
   [pred dir]
-  (->>  dir
-        file-seq
-        (filter (every-pred fs/exists?
-                            (complement fs/hidden?)
-                            pred
-                            (complement build-artifact?)))))
+  (->> dir
+       file-seq
+       ;; `pmap` performs better in large projects.
+       (pmap (fn [f]
+               (when ((every-pred fs/exists?
+                                  (complement fs/hidden?)
+                                  pred
+                                  (complement build-artifact?))
+                      f)
+                 f)))
+       (filter identity)))
 
 (defn read-ns-form
   ([path]
@@ -184,7 +189,10 @@
 (defn find-in-project
   "Return the files in the project satisfying (pred ^File file)."
   [pred]
-  (-> find-in-dir (partial pred) (mapcat (dirs-on-classpath)) distinct))
+  (->> (dirs-on-classpath)
+       (pmap (partial find-in-dir pred))
+       (apply concat)
+       distinct))
 
 (defn throw-unless-clj-file [file-path]
   (when-not (re-matches #".+\.clj$" file-path)

src/refactor_nrepl/ns/libspecs.clj

@@ -77,21 +77,22 @@
 
 (defn referred-syms-by-file&fullname
   "Return a map of filename to a map of sym fullname to sym
-   the sym itself
+   the sym itself.
 
    Example:
    {:clj  {\"/home/someuser/projects/some.clj\" [\"example.com/foobar\" foobar]}
     :cljs}"
   ([]
    (referred-syms-by-file&fullname false))
   ([ignore-errors?]
+   ;; `pmap` is used as it has proved to be more efficient, both for cached and non-cached cases.
    {:clj  (->> (core/find-in-project (util/with-suppressed-errors
                                        (some-fn core/clj-file? core/cljc-file?)
                                        ignore-errors?))
-               (map (juxt identity (partial get-libspec-from-file-with-caching :clj)))
+               (pmap (juxt identity (partial get-libspec-from-file-with-caching :clj)))
                sym-by-file&fullname)
     :cljs (->> (core/find-in-project (util/with-suppressed-errors
                                        (some-fn core/cljs-file? core/cljc-file?)
                                        ignore-errors?))
-               (map (juxt identity (partial get-libspec-from-file-with-caching :cljs)))
+               (pmap (juxt identity (partial get-libspec-from-file-with-caching :cljs)))
                sym-by-file&fullname)}))