In brief: a technical post describing the challenges and methods for integrating clojurescript with node_modules, particularly for front end development.

There have been some really exciting developments in the javascript<>clojurescript scene lately regarding ecosystem integration and interop. A few months ago I stumbled across the post: “Clojurescript is not an island” that described some of the new capabilities for clojurescript to interop with node_modules. Gonna build a bridge from CLJS to NPM!

I’ve been interested in this since using older methods of pulling in external libraries. For example, I’ve used Antizer to pull in the great AntD react component library via CLJSJS. What annoys me, however, is that when it comes time to compile the app, it will include the entire Ant Design component library, even if I’ve used only a single component!

Before we get to looking at using the :npm-deps feature of the cljs compiler, let’s overview the history of including external libs with Clojurescript.

Terms

Cljs<>Js History

Having come into Clojure/script only within the last year, I don’t have a ton of experience with the history of different mechanisms for setting up ecosystem-interop. Rather than attempt to describe the history of CLJS<>JS ecosystem stuff, I’d recommend this post or at least it’s sections on the following:

Current Build Tools

So, at this point, I’d been made aware of Cljs being capable of interoping with node_modules. But! But! I had no idea how to do this. Before reading “Clojurescript is not an island” I didn’t realize that clojurescript code was simply compiled by… well…Clojure. Up till this post, I’ve been using leiningen and project templates for everything Clojure and Clojurescript related and so I didn’t really know what was going on under the hood. As someone coming to Clojure/script from the world of Javascript the following may not be super obvious:

Back to Interop

And now? Now it appears that it’s possible to use external node_modules without needing a CLJSJS type situation, or to fiddle with the compiler externs options. I would have tried getting these new features working with my work projects but:

  1. I have more priority work to do, and ;
  2. fiddling with build tools will almost certainly cause breaking changes when I upgrade deps, and ;
  3. I’ll start yak shaving (trying to get the repl working with my editor, etc).

To be honest it’s a bit overwhelming to try and figure some of these things out; despite being a niche language, there are still several ways to do things here, and it’s difficult to dedicate the time to a project that’s already in motion.

I’ve tried starting a few projects from scratch recently and my curiosity and interest in better ecosystem interop has been enough to drive me to write this post and try and figure out a working solution using a scratch project. At the very least - I want to try new features that are coming out with CLJS.

cljs.build.api

The cljs quickstart is a great way to get exposed to doing things the “vanilla” way (ie, compiling using Clojure without a build tool like Lein or Boot). I followed the quick start guide, and created a deps.edn file.

{:deps {org.clojure/clojurescript {:mvn/version "1.10.238"}
        reagent {:mvn/version "0.8.1"}
        re-frame {:mvn/version "0.10.5"}}}

And then run:

clj --main cljs.main --compile cli-approach.core --repl

This will download all your dependencies and start a local server + repl. Sweet! But! According to the island post, I need some kind of build.clj that uses cljs.build.api. 10 Minutes of googling didn’t reveal much in the CLJS docs about using the build api, but this post/blog/book thing seems useful.


30 Mins later and I’ve landed on the compiler options page. It looks like I need to pass an .edn file configuration to the cli build command. The page isn’t clear about how to do that, so I’ll look at the command line help:

clj --main cljs.main --help

# ...
init options:
  -co, --compile-opts edn     Options to configure the build, can be an EDN
                              string or system-dependent path-separated list of
                              EDN files / classpath resources. Options will be
                              merged left to right.
# ...

And so I try:

clj --main cljs.main -co build.edn

Where the build edn is:

;; build.edn:
{:output-dir "out"
   :output-to "out/main.js"
   :optimizations :none
   :main 'cli-approach.core
   :install-deps true
   :npm-deps {:antd "3.6.1"}
 }

Success! all of a sudden I have a node_modules folder, meaning that the npm-deps have been fetched. Now then, I want to try and use some of these fully sick new npm deps.

I was getting a little ahead of myself, it seems that when compiling, my namespace that goog.require is trying to fetch can’t be found: after compiling I’m getting this error:

Uncaught Error: goog.require could not find: _SINGLEQUOTE_cli_approach.core
    at Object.goog.require [as require__] (base.js:711)
    at Object.clojure.browser.repl.bootstrap.goog.require (repl.cljs:215)
    at ?rel=1528377228906:38

I’m starting to hesitate with the new clj cli tools . So I’m going to switch over to trying to do things with a build.clj file as noted here.

Create a build.clj in my root:

(require 'cljs.build.api)

(cljs.build.api/build "src" {:output-to "out/main.js"})

And run: java -cp cljs.jar:src clojure.main build.clj.

At first I got an error looking for the cljs.jar, so I promptly teleported to the Clojurescript website and downloaded the latest jar. Re-running the command… worked, but no out folder, and no sign of success. Maybe I need to have the Clojure jar in my directory as well?

I’ve lost steam for trying to get the new cli tools working. I was close! I think! I’m not totally sure if I’m using the cli properly (I’m guessing not). As for these new clj cli tools, Mr. Oakes, druid-wizard of Clojure, has a pertinent gist on the topic.

Leiningen + Reframe Template

Now onto using a template to try and get some node_modules up in a front-end project. I’ve used the re-frame template for several projects, and it really is lovely; a real feather in my cap! Hats off! But let’s cut the flowery shit and get moving:

❯ lein new re-frame rf-approach +cider +less +routes +10x
Generating re-frame project.

❯ lein deps
❯ lein clean
❯ lein figwheel dev

I’m up and running!

Now to get some npm-modules in this sweet little hatched cottage that is my new re-frame app. I pop open project.clj in the folder root and scan for a place where things can go. Ah, :cljsbuild:

  :cljsbuild
  {:builds
   [{:id           "dev"
     :source-paths ["src/cljs"]
     :figwheel     {:on-jsload "rf-approach.core/mount-root"}
     :compiler     {:main                 rf-approach.core
                    :output-to            "resources/public/js/compiled/app.js"
                    :output-dir           "resources/public/js/compiled/out"
                    :asset-path           "js/compiled/out"
                    :source-map-timestamp true
                    :preloads             [devtools.preload
                                           day8.re-frame-10x.preload]
                    :closure-defines      {"re_frame.trace.trace_enabled_QMARK_" true
                                           "day8.re_frame.tracing.trace_enabled_QMARK_" true}
                    :external-config      {:devtools/config {:features-to-install :all}}
                    }}

    {:id           "min"
     :source-paths ["src/cljs"]
     :compiler     {:main            rf-approach.core
                    :output-to       "resources/public/js/compiled/app.js"
                    :optimizations   :advanced
                    :closure-defines {goog.DEBUG false}
                    :pretty-print    false}}


    ]}

I don’t see any other places where I can put clojurescript related dependencies in. In fact, it bugs me like a caterpillar that I still don’t really know what :cljsbuild is. Off to google… Looks like we’re dealing with a leiningen plugin:

This is a Leiningen plugin that makes it quick and easy to automatically compile your ClojureScript code into Javascript whenever you modify it. It’s simple to install and allows you to configure the ClojureScript compiler from within your project.clj file.

Very nice! There are a lot of key/vals up in that previous code block for the :cljsbuild map, but the :compiler key/map looks to be the most useful, resembling build.edn I was trying to work with before. According to this issue It shouldn’t be a problem to map the cljs compiler build keys to the lein-cljsbuild plugin - it appears to just be a wrapper.

After running figwheel with the new :npm-deps added, I didn’t see any results. I read some more github issues and tried trashing previous compiled files. No dice! Now to try running npm manually, because just running lein figwheel doesn’t seem do that for you (I don’t know why I would expect that either.) So I go ahead and run

❯ npm init -y
❯ npm install --save react

I try and require react like so:

(ns rf-approach.views
  (:require
   [re-frame.core :as re-frame]
   [react :refer [createElement]]
   [rf-approach.subs :as subs]
   ))

And get this response in the console:

async.cljs?rel=1528380841030:24 Uncaught Error: Undefined nameToPath for react
    at visitNode (base.js:1357)
    at Object.goog.writeScripts_ (base.js:1369)
    at Object.goog.require [as require_figwheel_backup_] (base.js:706)
    at figwheel$client$file_reloading$figwheel_require (file_reloading.cljs?rel=1528380857719:179)
    at figwheel$client$file_reloading$require_with_callback (file_reloading.cljs?rel=1528380857719:330)
    at figwheel$client$file_reloading$js_reload (file_reloading.cljs?rel=1528380857719:355)
    at figwheel$client$file_reloading$reload_js_file (file_reloading.cljs?rel=1528380857719:362)
    at file_reloading.cljs?rel=1528380857719:376
    at file_reloading.cljs?rel=1528380857719:374
    at figwheel$client$file_reloading$load_all_js_files_$_state_machine__35231__auto____1 (file_reloading.cljs?rel=1528380857719:374)
    at figwheel$client$file_reloading$load_all_js_files_$_state_machine__35231__auto__ (file_reloading.cljs?rel=1528380857719:374)
    at cljs$core$async$impl$ioc_helpers$run_state_machine (ioc_helpers.cljs?rel=1528380855277:35)
    at cljs$core$async$impl$ioc_helpers$run_state_machine_wrapped (ioc_helpers.cljs?rel=1528380855277:39)
    at file_reloading.cljs?rel=1528380857719:374
    at cljs$core$async$impl$dispatch$process_messages (dispatch.cljs?rel=1528380843430:19)

I start to lose hope! I’m creeping the clojure repos of people who have reported success in hopes of finding a config that will satisfy the incantations that I may have mispoken. I’ve trashed my resources/public/js/compiled and re-ran lein clean, lein deps and lein figwheel. Still no luck. But wait! I change songs and scroll back in my terminal as the coals of hope and love in my heart slowly warm again:

❯ lein figwheel dev
Figwheel: Cutting some fruit, just a sec ...
Figwheel: Validating the configuration found in project.clj
Figwheel: Configuration Valid ;)
Figwheel: Starting server at http://0.0.0.0:3455
Figwheel: Watching build - dev
Compiling build :dev to "resources/public/js/compiled/app.js" from ["src/cljs"]...
module.js:487
    throw err;
    ^

Error: Cannot find module '@cljs-oss/module-deps'
    at Function.Module._resolveFilename (module.js:485:15)
    at Function.Module._load (module.js:437:25)
    at Module.require (module.js:513:17)
    at require (internal/module.js:11:18)
    at [eval]:3:13
    at ContextifyScript.Script.runInThisContext (vm.js:44:33)
    at Object.runInThisContext (vm.js:116:38)
    at Object.<anonymous> ([eval]-wrapper:6:22)
    at Module._compile (module.js:569:30)
    at evalScript (bootstrap_node.js:432:27)

What is this! It looks like I’m missing a module that could be very useful for… modules. In fact, now that I think of it, I’m pretty sure I saw this @cljs-oss/module-deps get installed when I was originally working with the build.edn file. Let’s try installing it manually. I do so. I kick resources/public/js/compiled to the curb yet again, and re-run lein figwheel, feeling like I have narrowly dodged a loosed pen of build-tooling ghosts over the cafe that I currently find myself in.

What! it works! I drown myself in coffee as I stare in disbelief at some code/console:

(ns rf-approach.views
  (:require
   [re-frame.core :as re-frame]
   [react :refer [createElement]]
   [rf-approach.subs :as subs]
   ))

(prn "react is " createElement)
util.cljs?rel=1528381588581:187 Installing CLJS DevTools 0.9.10 and enabling features :formatters :hints :async
async.cljs?rel=1528381589117:50 cljs-devtools: the :async feature is no longer needed since Chrome 65.0.3321, see https://github.com/binaryage/cljs-devtools/issues/20
core.cljs:192 "react is " #object[M$$module$Users$tees$Desktop$cljs_npm$rf_approach$node_modules$react$cjs$react_production_min] << Womp womp
core.cljs:192 dev mode

Sickening. Let’s get antd, my favourite front-end component library UP IN THIS THATCHED HOUSE. It doesn’t work on first install, because it relies on reactDOM. I install that.

Things are not looking promising. AntD is a complex component library, and while it’s been packaged extremely well (both with es import/export syntax AND common js require syntax) I can’t seem to require the library. There’s a good chance I just haven’t figure out the proper path to the component I want to require – after all I definitely don’t want to pull in the whole library to use a single button component in ns:

123456789
(ns rf-approach.views
  (:require
   [re-frame.core :as re-frame]
   [react :refer [createElement]]
   ["antd/es/button/index" :as button] ;; does not work
   ["antd" :refer [button]] ;; does not work
   [antd :as ant] ;; nope.
   [rf-approach.subs :as subs]
   ))

Unfortunate. I think I’ve hit my time cap on this one. So close too. I would really like this to work because:

  1. It’s quick to boot up the re-frame template.
  2. Figwheel is lovely (and the new readline functionality is sweet)

It feels like giving up, but I’m hoping at this point that someone else might have a helpful answer and I can return to this solution. Time to move on and try shadow-cljs.

Shadow-cljs + Reframe

Shadow-cljs is a (relatively) newer build tool, as far as I understand, specifically for clojurescript. The author/maintainer seems to be hard at work on this project and is often around in the Clojure communities, offering help and suggestions for people using Shadow. Still that means there are a few less docs for newer projects, or at least less google results, perhaps (although to be fair, I couldnt’ glean much when I tried using the Clojure cljs build api…).

Let’s get into it. I’m following this quickstart guide mixed with some of the docs on the shadow cljs home page.

I’ve gotten a basic clojurescript compiling + server running fairly easily. It’s probably easier to just post a screenshot of my editor with the file tree on the left and each piece of the plumbing in view:

Now, to add in some dependencies.

 1 2 3 4 5 6 7 8 91011
{:source-paths ["src"]
 :dependencies [[reagent "0.7.0"]
                [re-frame "0.10.5"]
                ]
 :builds {:app {:output-dir "public/js"
                :asset-path "/js"
                :target :browser
                :modules {:main {:entries [app.main]}}
                :devtools {:after-load app.main/reload!
                           :http-root "public"
                           :http-port 8080}}}}

Since reagent relies on react, Shadow actually told me that it wasn’t available after installing reagent/re-frame previously. That’s pretty nice!

shadow-cljs - watching build :app
[:app] Configuring build.
[:app] Compiling ...
[:app] Build completed. (136 files, 2 compiled, 0 warnings, 2.16s)
[:app] Compiling ...
[:app] Build failure:
The required JS dependency "react" is not available, it was required by "cljsjs/react.cljs".

Searched in:/Users/tees/Desktop/cljs_npm/shadow/node_modules

You probably need to run:
  npm install react

See: https://shadow-cljs.github.io/docs/UsersGuide.html#npm-install

You can read more details about detecting missing packages with shadow-cljs here.

So after installing react, antd, and dropping in the day 8 simple example I managed to require the AntD component. It took about 20 seconds to load in all the dependencies at first, and then it worked!

(ns app.main
  (:require [reagent.core :as reagent]
            [re-frame.core :as rf]
            [antd :refer [Input Button]] ;; nice, easy clojure style requires.
            [clojure.string :as str]))

(prn "Button is" Button) ;; Ant component exists!
(def T-Button (reagent/adapt-react-class Button)) ;; wrapping it for reagent-use
(prn "Button is " T-Button) ;; It exists!

;; ...

;; View code

(defn ui
  []
  [:div {:style {:padding "32px"}}
   [:h1 "Hello world, it is now"]
   [T-Button {:icon "home" :type "primary"} "Hello!"] ;; Success!
   [clock]
   [color-input]])

So it worked pretty much out of the box. I think I figured out how to get figwheel-esque code reloading like so:

 1 2 3 4 5 6 7 8 910
;; -- Entry Point -------------------------------------------------------------

(defn mount-root []
  (rf/clear-subscription-cache!)
  (reagent/render [ui] (.getElementById js/document "app")))

(defn ^:export run
  []
  (rf/dispatch-sync [:initialize]) 
  (mount-root))

^ Create a function specifically for remounting the entire app - which seems to be what the re-frame + figwheel setup does. I did pretty much the same thing with the :devtools :after-load val.

 1 2 3 4 5 6 7 8 91011
{:source-paths ["src"]
 :dependencies [[reagent "0.7.0"]
                [re-frame "0.10.5"]
                ]
 :builds {:app {:output-dir "public/js"
                :asset-path "/js"
                :target :browser
                :modules {:main {:entries [app.main]}}
                :devtools {:after-load app.main/mount-root
                           :http-root "public"
                           :http-port 8080}}}}

Ok. I got it working. But I haven’t tried actually compiling the code with “:advanced” compilation with the cljs compiler. The whole point of looking into the npm-deps feature was to be able to have fine grain control of the external libraries I’m using. At this point, the amount of work I’ve had to do to figure this out makes the whole thing pretty pedantic and pointless.

Anyway, let’s try and run a compile.

...

Here’s my main.js - 1.7mb. Doesn’t sound quite right. I also see some antd modules in there that I didn’t use (modals, datepicker, etc).

According to the shadow-cljs wiki:

Since release builds by default run through :advanced compilation by the Closure Compiler there may be a few things that we need to take care of.

The –check command compiles your files in :release mode but only runs the code through the Closure Compiler checks. Its a quick way to discover missing externs.

So I suppose that was advanced compilation, but maybe I need to do something with externs.

Shadow-cljs seems really promising and quick to get moving. I like being able to actually use npm install and a package json for managing the libraries I want to use, rather than a wrapper for it that gets placed into a `project.clj** – I suppose I’d like to have just one dependencies file, but I have a hard time imagining how we can have strong Javascript environment interop without including package.json in your project folder.

Edit: After posting this log on r/clojure, I got feedback from the author of Shadow-Cljs. It turns out that I just had to change my name space require to be like so:

123456
(ns app.main
  (:require [reagent.core :as reagent]
            [re-frame.core :as rf]
            ;; [antd :refer [Input Button]]
            ["antd/lib/button" :as Button] ;; Properly requires JUST the button!
            [clojure.string :as str]))

My build size is down to 347kb from 1.7 - a definite improvement.

Javascript World

If we flip over to JS world and clone AntD’s create-react-app-port we can do a bit of comparing.

Cloning, installing, and running the project serves up an empty react app with a hello world AntD Button. Running npm run build spits out a build that is 142kb in size, and 800~kb with a source map.

Summing up

Original Notes:

If this post was helpful for you at all, let me know. If I missed something, or you know how to get npm-deps working definitely get in touch.

Updated:

After getting some tips about getting shadow-cljs working I’m pretty keen to use it when starting future projects. It was the only tool I could figure out to get the npm-deps working with proper singular import/requires.

Cljs NPM examples repo