xscDev

Something Something Development

[Release] Recursively Check/Upgrade your Project Dependencies using lein-ancient 0.5.0-RC1

without comments

I just released lein-ancient 0.5.0-RC1, incorporating the following neat features and enhancements:

  • recursively traverse your directories and upgrade all “project.clj” files that are found,
  • run user-specified regression tests after upgrading to ensure compatibility with the upgraded artifacts,
  • print warnings for unrecognized and no-longer-supported options,
  • apply artifact check/upgrade to a given file/directory, not only the current one.

This probably doesn’t look a lot but it prompted the restructuring of the codebase which should make it easier to incorporate new features and options in future releases. And here’s a short peek of what it looks like:

$ lein ancient upgrade :recursive :interactive :no-tests libraries
-- libraries/ancient-clj/project.clj

[clj-aws-s3 "0.3.7"] is available but we use "0.3.6"
Do you want to upgrade? [yes/no] yes

1 artifact was upgraded.

-- libraries/pandect/project.clj
Nothing was upgraded.

-- libraries/panoptic/project.clj

[com.taoensso/timbre "2.6.2"] is available but we use "2.1.2"
Do you want to upgrade? [yes/no] yes

[potemkin "0.3.3"] is available but we use "0.3.0"
Do you want to upgrade? [yes/no] yes

[pandect "0.3.0"] is available but we use "0.2.3"
Do you want to upgrade? [yes/no] no

2 artifacts were upgraded.
...

Please check the above features out and let me know if you run into any problems! It would be much appreciated!

Written by Yannick

September 28th, 2013 at 11:20 pm

[PreRelease] lein-ancient 0.5.0-alpha2

without comments

Today, I released lein-ancient 0.5.0-alpha2 incorporating some features and adjustments I’ve been working on. For those that don’t know, lein-ancient is a Leiningen plugin that let’s you check your projects for outdated dependencies and, if desired, upgrade them automatically.

I’m not quite happy with automatic regression testing yet (I started lein-testem to address that point), since a) only very few frameworks are supported and b) even those that are supported might not always behave as expected. Maybe I’ll drop automatic test call generation in favor of the also-implemented use of a “test-ancient” alias in your project.clj, though.

Anyways, this is how it should look:

$ lein ancient upgrade
[org.clojure/tools.reader "0.7.7"] is available but we use "0.7.5"
[potemkin "0.3.3"] is available but we use "0.3.2"
2 artifacts were upgraded.

WARN: lein-testem will only work correctly with lein-midje >= 3.1.3
Running 3 [:midje] Test Cycle(s) ...
All checks (306) succeeded.
All checks (306) succeeded.
All checks (306) succeeded.

I actually contributed to the lein-midje plugin to be usable with lein-testem but as it turns out I broke it instead. Oops…

Written by Yannick

September 25th, 2013 at 7:00 pm

Posted in Clojure,Projects

Tagged with

[ANN/WIP] lein-testem: Test your Projects with reduced Configuration Effort

without comments

I’m currently working on lein-testem, a Leiningen plugin designed to automatically detect how to run tests on a project. It will decide which test framework(s) to use and what profile combinations to examine. Head over to the GitHub page for a small example, or – if you dare – try it out by including the following into your ~/.lein/profiles.clj file:

[lein-testem "0.1.0-alpha2"]

You can then run:

$ lein testem

Currently supported are clojure.test and Midje, and maybe Speclj. But don’t misunderstand: You still have to include all plugins/dependencies associated with your test framework – you just don’t have to call them directly any more.

Why?

I want to include regression testing in lein-ancient, a plugin that enables you to check for outdated dependencies and to upgrade those in need. Upgrading should only be finalized if tests pass so I was faced with the problem of how to run existing tests without having to explicitly configure which framework they are based upon. lein-testem’s core functionality will thus end up in lein-ancient at some point.

Other Plans

I want to include common test scenarios (e.g. “test against different Clojure versions”, …) in the future. Ideas are always welcome, so fire away in the comment section!

Written by Yannick

September 5th, 2013 at 3:17 am

Posted in Clojure,Projects

Removing Nodes in Zippers or How To Get Lost

with 5 comments

Update (07.08.2013): I wrote some additional removal operations for fast-zip which can be found in this Gist.

I’ve recently had a fair share of zipper usage in Clojure whilst working on rewrite-clj (which utilizes zippers to traverse Clojure code/EDN trees). One thing that really causes tons and tons of problems for me – or at least unpleasant surprises – is the design of remove (doesn’t matter whether in Clojure’s standard zippers or fast-zip‘s) whose documentation reads as follows:

Removes the node at loc, returning the loc that would have preceded
it in a depth-first walk.

Okay, so when I remove a node I’ll end up somewhere before it. But where exactly? Well, it might be directly left (on the same level) of the removed node in some cases but it might also happen that removal takes you deep (actually as deep as possible) into a left-adjacent branch. Let’s have some XML (which can be traversed using zippers) to illustrate the two cases:

<list>
  <first>1</first>
  <second>2</second>
</list>

Let’s say the resulting zipper is stored in xml-loc and tag returns the XML tag name as a keyword. We would observe the following:

(-> xml-loc  ;; at 'list'
  z/down     ;; at 'first'
  z/right    ;; at 'second'
  z/remove   ;; where are we?
  tag)       ;;   => :first

That matches a sane person’s expectations. Even the following does not surprise too much:

(-> xml-loc  ;; at 'list'
  z/down     ;; at 'first'
  z/remove   ;; where are we?
  tag)       ;;   => :list

We removed the leftmost element (which we can know beforehand), so going up one level in the tree is reasonable since we can predict that perfectly. Let’s alter the XML document a little bit:

<list>
  <first>
    <a>0</a>
    <b>
      <c>0.5</c>
      <d>1</d>
    </b>
  </first>
  <second>2</second>
</list>

The same traversal as before will now let us end up deep inside the document. Read the rest of this entry »

Written by Yannick

August 6th, 2013 at 3:47 pm

Posted in Clojure

Tagged with , ,

[Release] lein-ancient 0.4.3: Automatically upgrade Project Dependencies and Plugins

with 9 comments

I just released version 0.4.3 of lein-ancient, a plugin that checks your projects for outdated dependencies and plugins. This release includes two useful new modes:

  • :upgrade, which takes your “project.clj” and tries to replace outdated versions with current ones, and
  • :upgrade-global, which does the same for your global user profile in “~/.lein/profiles.clj”.

Both offer options to tweak their behaviour, most prominently :interactive (which let’s you choose which artifacts to update) and :print (which writes the result to the console instead of back to disc). Let’s see how it’s supposed to work:

$ cd panoptic                         # an older project
$ git diff                            # no changes so far
$ lein ancient                        # standard functionality
[com.taoensso/timbre "2.4.1"] is available but we use "2.1.2"
[potemkin "0.3.1"] is available but we use "0.3.0"
[pandect "0.3.0"] is available but we use "0.2.3"

# okay, let's get down to (interactive) business
$ lein ancient :upgrade :interactive
[com.taoensso/timbre "2.4.1"] is available but we use "2.1.2"
Do you want to upgrade? [yes/no] yes
Upgrade to [com.taoensso/timbre "2.4.1"] from "2.1.2"

[potemkin "0.3.1"] is available but we use "0.3.0"
Do you want to upgrade? [yes/no] no

[pandect "0.3.0"] is available but we use "0.2.3"
Do you want to upgrade? [yes/no] yes
Upgrade to [pandect "0.3.0"] from "0.2.3"

# something should have changed
$ git diff
diff --git a/project.clj b/project.clj
index 18a408a..3031031 100644
--- a/project.clj
+++ b/project.clj
@@ -4,9 +4,9 @@
   :license {:name "Eclipse Public License"
             :url "http://www.eclipse.org/legal/epl-v10.html"}
   :dependencies [[org.clojure/clojure "1.5.1"]
-                 [com.taoensso/timbre "2.1.2"]
+                 [com.taoensso/timbre "2.4.1"]
                  [potemkin "0.3.0"]
-                 [pandect "0.2.3"]]
+                 [pandect "0.3.0"]]
   :repositories  {"sonatype-oss-public" "https://oss.sonatype.org/content/groups/public/"}
   :exclusions [org.clojure/clojure]
   :profiles {:test {:dependencies [[midje "1.5.1"]]

Voilà, all the desired dependencies have been upgraded while the rest (and especially the formatting of your project file) has been left untouched. Similarly, :upgrade-global will only alter what’s necessary in your global user profile, e.g.:

$ lein ancient :upgrade-global :allow-snapshots
Upgrade to [lein-marginalia "0.8.0-SNAPSHOT"] from "0.7.1"
Upgrade to [lein-pprint "1.1.2-SNAPSHOT"] from "1.1.1"

$ cat ~/.lein/profiles.clj 
{ :user { :plugins [[lein-marginalia "0.8.0-SNAPSHOT"]
                    [lein-pprint "1.1.2-SNAPSHOT"] 
                    [lein-ancient "0.4.3"]
                    [lein-try "0.3.0"]] } }

See here for installation instructions. Because once it is installed you might never have to manually upgrade your global plugins again. :)

Written by Yannick

July 30th, 2013 at 3:26 am

lein-try 0.3.0: Trying Clojure libraries becomes even easier

with 3 comments

A few days ago, I wrote about lein-try, a useful Leiningen plugin that let’s you start a REPL including any Clojure libraries you desire. Now, with the latest release the whole thing becomes even easier since you no longer have to provide the version number of those artifacts you want lein-try to offer you. Simply run it using:

$ lein try com.taoensso/nippy
Retrieving com/taoensso/nippy/2.0.0/nippy-2.0.0.pom from clojars
Retrieving expectations/expectations/1.4.49/expectations-1.4.49.pom from clojars
...
nREPL server started on port 59952
REPL-y 0.2.0
Clojure 1.5.1
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)

user=> (require '[taoensso.nippy :as nippy])
nil

See here on how to install the plugin correctly.

Written by Yannick

July 29th, 2013 at 4:42 pm

Try Clojure libraries with ease using lein-try

with 3 comments

Update (26.07.2013): Added installation to user profile to post (at the bottom).

In this post I’ll not advertise one of my own projects but instead direct your attention towards the awesomeness that is Ryan Neufeld’s lein-try. Imagine you’re browsing the intertubes and you discover a blog post/GitHub page/etc… outlining an interesting library, say Peter Taoussanis’ nippy. You feel you want to try it – you need to try it – and you want to do that immediately.

This is how you’d probably do it, having never heard of lein-try:

  1. Create a new project, e.g. “nippy-test”:

    lein new nippy-test
  2. Modify the “project.clj” to include nippy:
    ...
      :dependencies [[org.clojure/clojure "1.5.1"]
                     [com.taoensso/nippy "2.0.0"]]
    ...
  3. Start a REPL:
    lein repl
  4. Use nippy.

Behold, lein-try can speed things up drastically. You do not have to create a project, you do not have to modify your project.clj – and the REPL is part of the deal. Just pass any artifacts you’d like to try to it (either with “[...]” or without):

$ lein try com.taoensso/nippy "2.0.0"
Retrieving com/taoensso/nippy/2.0.0/nippy-2.0.0.pom from clojars
Retrieving expectations/expectations/1.4.49/expectations-1.4.49.pom from clojars
...
nREPL server started on port 59952
REPL-y 0.2.0
Clojure 1.5.1
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)

user=>

Fire away!

(require '[taoensso.nippy :as nippy])
(def s (nippy/freeze {:hello "world"})) ;; => #'user/s
(map int s)                             ;; => (78 80 89 1 25 96 ...)
(nippy/thaw s)                          ;; => {:hello "world"}

That’s it, from zero to nippy in one line. Try it out by adding the plugin to your ~/.lein/profiles.clj:

{:user {...
        :plugins [... [lein-try "0.2.0"] ... ]
        ...}
 ...}

Written by Yannick

July 26th, 2013 at 1:36 pm

Upgrade your Project’s Dependencies using lein-ancient (wip)

with 2 comments

Following an idea by Mayank Jain, I let lein-ancient meet my new Clojure rewrite library rewrite-clj to be able to automatically upgrade the dependencies/plugins in a project’s “project.clj”. They got along quite well and the work is done – however, seeing as there is a variety of Clojure project files out there and I’m not 100% sure that everything always works correctly, I’d very much like for as many people as possible to try out the new feature and tell me if they encounter any problems. I’d really appreciate that!

This comment on GitHub sums up how to retrieve and run the latest version of lein-ancient. If everything works smoothly you might get an output similar to the following:

$ lein ancient :upgrade :interactive

[com.taoensso/timbre "2.4.1"] is available but we use "2.1.2"
Do you want to upgrade? [yes/no] yes
Upgrade to [com.taoensso/timbre "2.4.1"] from "2.1.2"

[potemkin "0.3.1"] is available but we use "0.3.0"
Do you want to upgrade? [yes/no] no

[pandect "0.3.0"] is available but we use "0.2.3"
Do you want to upgrade? [yes/no] yes
Upgrade to [pandect "0.3.0"] from "0.2.3"

And to verify that the right changes where made you can, for example, use Git’s “diff” command (if your project.clj is under version control):

$ git diff project.clj
diff --git a/project.clj b/project.clj
index 18a408a..3031031 100644
--- a/project.clj
+++ b/project.clj
@@ -4,9 +4,9 @@
   :license {:name "Eclipse Public License"
             :url "http://www.eclipse.org/legal/epl-v10.html"}
   :dependencies [[org.clojure/clojure "1.5.1"]
-                 [com.taoensso/timbre "2.1.2"]
+                 [com.taoensso/timbre "2.4.1"]
                  [potemkin "0.3.0"]
-                 [pandect "0.2.3"]]
+                 [pandect "0.3.0"]]
   :repositories  {"sonatype-oss-public" "https://oss.sonatype.org/content/groups/public/"}
   :exclusions [org.clojure/clojure]
   :profiles {:test {:dependencies [[midje "1.5.1"]]

Thanks in any case!

Written by Yannick

July 24th, 2013 at 4:51 pm

rewrite-clj: Rewrite Clojure Code and EDN!

without comments

I just pushed the first release of rewrite-clj, a library trying to offer ways to easily manipulate Clojure and EDN documents in a whitespace- and comment-preserving way. It is similar in its basics to sjacket which I found really, really hard to use – plus, I didn’t really get along with zippers that well, so I saw the project as a simple learning-by-doing kind of thing. At first.

rewrite-clj solves a particular problem of mine: If I have a project’s “project.clj” file, how can I automatically update all its dependencies to the latest versions? Well, sure, that’s what clojure.tools.reader is for, right? Just read the project map, update the version strings and write it back. Hmm, problems:

  • the project map might not be pure EDN,
  • the reader might expand some things (e.g. 'a to (quote a)).
  • whitespace (indentation, linebreaks, …) is lost,
  • comments are lost.

Ideally, I should be able to read the project map, jump to the right position within, update stuff and then write it back, leaving everything I did not touch as it was before. It shouldn’t be too hard to write a parser for Clojure data structures that preserves whitespace and comments, so the only thing I really have to think about is how to traverse the resulting tree. I mean, of course I could just manually operate on nested vectors using Clojure’s basic seq functions à la map but this can get rather tedious and, to be frank, terrifying. Zippers to the rescue!

A zipper is a data structure that wraps tree-like structures and offers facilities to navigate the tree and to modify single nodes within. Zippers operate on immutable data structures, so every movement/edit/replace/remove/… operation always returns a completely new immutable zipper instance. This is necessary in functional environments like Clojure’s. Check out this short introduction to get started if you have no idea what I’m talking about.

The problem with a naive zipper implementation in our domain (and sjacket’s for that matter) is that navigating the tree means navigating over tons and tons of whitespace nodes that could mostly be ignored. Also, once you have the zipper structure (e.g. over a map) you cannot operate on the contained Clojure data anymore, and vice versa.

rewrite-clj tries to make working with zippers over Clojure/EDN data intuitive. Whitespace is ignored when moving from one node to another; edit, insert and replace transparently convert between the tree and the Clojure representation of values; and every insert makes sure that there is enough whitespace between the inserted value and the next or previous one. Of course, there are still some TODOs like automatic indentation of multi-line seqs if the beginning of the seq moves (one of sjacket’s cooler features if I saw that correctly). Also, the parser might not be the most robust one, but time and opened issues will tell.

So, head over to the GitHub repository and checkout the examples in the README or this minimalistic implementation of Kevin Lynagh’s cljx using rewrite-clj. And, of course, feel free to complain!

Written by Yannick

July 21st, 2013 at 12:10 am

Posted in Clojure,Projects

Tagged with , , , , ,

[Release] pandect 0.3.0

without comments

pandect, a message digest and checksum library for Clojure, can now compute hash-based message authentication codes (HMACs), wrapping Java’s javax.crypto.Mac class. Just wanted to announce that.

Also, I’m thinking about including SHA-3, but I’ll probably wait until it’s officially included in NIST’s Secure Hash Standard.

Written by Yannick

July 10th, 2013 at 2:46 pm