xscDev

Something Something Development

thrift-clj: Using Thrift from Clojure

without comments

Update: Please be aware that the examples given below use thrift-clj “0.1.0-alpha1″. There have been some API changes since.

I’ve had my eye on Apache Thrift for quite some while now but never got around to trying it. Now, with Clojure residing on the JVM and Thrift IDL files being compilable to Java code, I saw the possibility of finally diving into this great (albeit very badly documented) framework.

Now, a quick Google Search for “thrift” and “clojure” revealed:

  • a short interaction about Clojure code generation from Thrift IDLs (which would be great but two years later there still doesn’t seem to be anything in that direction);
  • small examples for Thrift serialization, wrapping existing Java classes, …;
  • a Leiningen Plugin.

Now, while it is certainly feasible to create Thrift Java code and access it (after all, Clojure’s interop is magnificient) there’s always a thought in the back of my head: This could be easier. And if not that, at least prettier.

Example

If you really want to try the following pieces of code, you might have to create a new Clojure project, but you will be fine just reading along. We will use the following simple example (test.thrift) to show what Thrift/Clojure is about:

namespace java org.example

struct Person {
  1: optional string firstName,
  2: string lastName,
  3: byte age
}

service PersonIndex {
    bool store(1:Person p)
}

This can be compiled using Thrift:

thrift --gen java -out <Path> test.thrift

It has to be put on Leiningens classpath since we want to access it via the REPL:

:java-source-paths ["<Path>"]

And finally, start the REPL:

lein repl

Clojure/Java Interop

Note that I will now demonstrate how to access Thrift from Clojure manually; if you don’t care and can’t wait to see sweet wrappers and pure Clojure instead of Java interop, feel free to skip this section.

What was notably created by the Thrift compiler is the following:

  • org.example.Person: the Person type
  • org.example.PersonIndex$Client: the Client class
  • org.example.PersonIndex$Iface: the Service interface
  • org.example.PersonIndex$Processor: the Processor to be used by Thrift servers

As illustrated here, we can directly implement the Service Processor using Clojure’s proxy:

(import '(org.example Person PersonIndex PersonIndex$Processor 
                      PersonIndex$Client PersonIndex$Iface))
(def person-index-processor
  (PersonIndex$Processor. 
    (proxy [PersonIndex$Iface] [] 
      (store [p] 
        (println "Storing Person:")
        (println "  First Name:" (.getFirstName p))
        (println "  Last Name:" (.getLastName p))
        (println "  Age:" (.getAge p))
        true))))

Creating a Server is done like this:

(import '(org.apache.thrift.server TServer TServer$Args TSimpleServer))
(import '(org.apache.thrift.transport TSocket TServerSocket))

(defn person-index-server
  [port]
  (let [transport (TServerSocket. port)
        args (TServer$Args. transport)
        processor (.processor args person-index-processor)]
    (TSimpleServer. processor)))

And this can now be run:

(def server (person-index-server 7007))
(future (.serve server))

And accessed by a Client:

(import '(org.apache.thrift.protocol TBinaryProtocol))
(defn person-index-client
  [host port]
  (let [transport (TSocket. host port)
        protocol (TBinaryProtocol. transport)
        client (PersonIndex$Client. protocol)]
    (.open transport)
    [transport client]))
(def client-data (person-index-client "localhost" 7007))
(def client (second client-data))
(defn stop-client [] (.close (first client-data)))

Now, data to store:

(def p (Person. "Yannick" "Scherer" 24))
;; => CompilerException java.lang.IllegalArgumentException: 
;;    No matching ctor found for class org.example.Person, 
;;    compiling:(NO_SOURCE_PATH:1:1)

What happened here? If you define a field as optional Thrift will not include it into its constructor. Thus, we have to omit firstName and set it later on:

(def p (Person. "Scherer" 24))
(.setFirstName p "Yannick")
(println p) ;; => #<Person Person(firstName:Yannick, lastName:Scherer, age:24)>

Looks good, let’s send it:

(.store client p)
;; Storing Person:
;;   First Name: Yannick
;;   Last Name: Scherer
;;   Age: 24
;; => true

(stop-client)

And that’s it. You can find the full code here.

So, how was that?

This actually works rather well but one cannot deny that there are some points a Clojure developer could wish for:

  • Clojure data structures for Client and Server. (.getFirstName p) is (subjectively) uglier than (:firstName p),
  • Wrappers around Iface and Processor for service implementation
  • Wrappers around Server and Client creation
  • Importing multiple Types/Services at once (as in Java’s “import some.package.*”)

We wrote a lot of boilerplate. And we shouldn’t have to.

Introducing thrift-clj

thrift-clj is a small library that tries to facilitate Thrift usage and development. We will now build the same example but using new tools (restart your REPL if you are working on one):

(require '[thrift-clj.core :as thrift])
(thrift/import
  (:types org.example.Person)
  (:services org.example.PersonIndex)
  (:clients [org.example.PersonIndex :as PIClient]))

(thrift/defservice person-index-service
  PersonIndex
  (store [{:keys[firstName lastName age]}]
    (println "Storing Person:")
    (println "  First Name:" firstName)
    (println "  Last Name:" lastName)
    (println "  Age:" age)
    true))

(def server (thrift/single-threaded-server person-index-service :socket 7007))
(def client (thrift/create-client PIClient :socket "localhost" 7007))
(future (thrift/start-server! server))
(thrift/start-client! client)

(def p (Person. "Yannick" "Scherer" 24))
(thrift/->thrift p) ;; => org.example.Person<...>
(PIClient/store client p)
;; Storing Person:
;;   First Name: Yannick
;;   Last Name: Scherer
;;   Age: 24
;; => true

(thrift/stop-client! client)
(thrift/stop-server! server)

Step by Step:

  1. The thrift/import macro loads different Thrift entities and wraps them to be accessible via Clojure. To prevent name clashes, it supports require/use-like :as specifications.
  2. thrift/defservice defines a service implementation. Its syntax is similar to defprotocol/deftype. As you can see, the full arsenal of Clojure tools is at your disposal, e.g. destructuring.
  3. Using thrift/single-threaded-server one can create a TSimpleServer using a previously defined service implementation. The server transport to be used is identified by a keyword (e.g. :socket) and additional options. (see namespace thrift-clj.server).
  4. Similarly, using thrift/create-client one can create a client.
  5. thrift/start-server! and thrift/start-client! destroy your hard disk. Just kidding, they start stuff.
  6. What was a simple Java Object before is now a Clojure record. You can convert between the two representations using thrift/->thrift and thrift/->clj – but most of the time you won’t need to.
  7. Behind the scenes a new namespace was created and aliased with PIClient; that’s why you can access the client methods using the “/” notation. By the way, they transparently convert between Thrift and Clojure data. (The old client methods, like (.store …) are still available, of course. Just be careful with the types of parameters.)
  8. The service did its job, shut it down using two exoticly named functions.

Some Notes:

  • Client methods (like PIClient/store) work directly with service implementations (like person-index-service); this enable testing of service logic separately from any server transport.
  • Using defservice you can create a number of different implementations for the same service, choosing the one you want to use at runtime.

Final Words

Please keep in mind that I started this yesterday. It’s not even 30hrs old, so give me a few more days to implement unit tests and create documentation. I’d really like some feedback on this project, though. It’s in the early stages, so a lot can still be changed. Any opinions and suggestions are very welcome!

Thanks for reading!

Written by Yannick

April 10th, 2013 at 6:49 pm

Posted in Clojure,Projects

Tagged with , , ,