libpython-clj2.java-api

A java api is exposed for libpython-clj2. The methods below are statically callable without the leading '-'. Note that returned python objects implement the respective java interfaces so a python dict will implement java.util.Map, etc. There is some startup time as Clojure dynamically compiles the source code but this binding should have great runtime characteristics in comparison to any other java python engine.

Note that you can pass java objects into python. An implementation of java.util.Map will appear to python as a dict-like object an implementation of java.util.List will look like a sequence and if the thing is iterable then it will be iterable in python. To receive callbacks from python you can provide an implementation of the interface clojure.lang.IFn

Performance:

There are a two logical ways to repeatedly invoke python functionality. The first is to set some globals and repeatedly runStringAsInput a script. The second is 'exec'ing a script, finding an exported function in the global dict and calling that. With libpython-clj, the second pathway -- repeatedly calling a function -- is going to be faster than the first if the user makes a fastcallable out of the function to be invoked. Here are some sample timings for an extremely simple function with two arguments:

Python fn calls/ms 1923.2490094806021
Python fastcallable calls/ms 3776.767751742239
Python eval pathway calls/ms 2646.0478013509883
  • makeFastcallable - - For the use case of repeatedly calling a single function - this will cache the argument tuple for repeated use as opposed to allocating the argument tuple every call. This can be a surprising amount faster -- 2x-3x -- than directly calling the python callable. Once a fastcallable object is made you can either cast it to a clojure.lang.IFn or call it via the provided call static method.

  • HAIR ON FIRE MODE - If you are certain you are correctly calling lockGIL and unlockGIL then you can define a variable, -Dlibpython_clj.manual_gil=true that will disable automatic GIL lock/unlock system and gil correctness checking. This is useful, for instance, if you are going to lock libpython-clj to a thread and control all access to it yourself. This pathway will get at most 10% above using fastcall by itself.

Example:

  java_api.initialize(null);
  try (AutoCloseable locker = java_api.GILLocker()) {
    np = java_api.importModule("numpy");
    Object ones = java_api.getAttr(np, "ones");
    ArrayList dims = new ArrayList();
    dims.add(2);
    dims.add(3);
    Object npArray = java_api.call(ones, dims); //see fastcall notes above
    ...
  }

-arrayToJVM

(-arrayToJVM pyobj)

Copy (efficiently) a numeric numpy array into a jvm map containing keys "datatype", "shape", and a jvm array "data" in flattened row-major form.

-call

(-call item)(-call item arg1)(-call item arg1 arg2)(-call item arg1 arg2 arg3)(-call item arg1 arg2 arg3 arg4)(-call item arg1 arg2 arg3 arg4 arg5)(-call item arg1 arg2 arg3 arg4 arg5 arg6)

Call a clojure IFn object. Python callables implement this interface so this works for python objects. This is a convenience method around casting implementation of clojure.lang.IFn and calling invoke directly.

-callKw

(-callKw pyobj pos-args kw-args)

Call a python callable with keyword arguments. Note that you don't need this pathway to call python methods if you do not need keyword arguments; if the python object is callable then it will implement clojure.lang.IFn and you can use invoke.

-copyData

(-copyData from to)

Copy data from a jvm container into a numpy array or back. This allows you to use a fixed preallocated set of numpy (and potentially jvm arrays) to transfer data back and forth efficiently. The most efficient transfer will be from a java primitive array that matches the numeric type of the numpy array. Also note the element count of the numpy array and the jvm array must match.

Note this copies from the first argument to the second argument -- this is reverse the normal memcpy argument order!!. Returns the destination (to) argument.

Not this does not work with array-of-arrays. It will work with, for instance, a numpy matrix of shape 2, 2 and an double array of length 4.

-copyToJVM

(-copyToJVM object)

Copy a python object such as a dict or a list into a comparable JVM object.

-copyToPy

(-copyToPy object)

Copy a basic jvm object, such as an implementation of java.util.Map or java.util.List to a python object.

-createArray

(-createArray datatype shape data)

Create a numpy array from a tuple of string datatype, shape and data.

  • datatype - One of "int8" "uint8" "int16" "uint16" "int32" "uint32" "int64" "uint64" "float32" "float64".
  • shape - integer array of dimension e.g. [2,3].
  • data - list or array of data. This will of course be fastest if the datatype of the array matches the requested datatype.

This does work with array-of-array structures assuming the shape is correct but those will be slower than a single primitive array and a shape.

-getAttr

(-getAttr pyobj attname)

Get a python attribute. This corresponds to the python '.' operator.

-getGlobal

(-getGlobal varname)

Get a value from the global dict. This function expects the GIL to be locked - it will not lock/unlock it for you.

-getItem

(-getItem pyobj itemName)

-GILLocker

(-GILLocker)

Lock the gil returning an AutoCloseable that will unlock the gil upon close.

Example:

  try (AutoCloseable locker = java_api.GILLocker()) {
    ... Do your python stuff.
  }

-hasAttr

(-hasAttr pyobj attname)

Returns true if this python object has this attribute.

-hasItem

(-hasItem pyobj itemName)

Return true if this pyobj has this item

-importModule

(-importModule modname)

Import a python module. Module entries can be accessed via getAttr.

-initialize

(-initialize options)

See options for libpython-clj2.python/initialize!. Note that the keyword option arguments can be provided by using strings so :library-path becomes the key "library-path". Also note that there is still a GIL underlying all of the further operations so java should access python via single-threaded pathways.

-initializeEmbedded

(-initializeEmbedded)

Initialize python when this library is being called from a python program. In that case the system will look for the python symbols in the current executable. See the embedded topic

-lockGIL

(-lockGIL)

Attempt to lock the gil. This is safe to call in a reentrant manner. Returns a long representing the gil state that must be passed into unlockGIL.

See documentation for pyGILState_Ensure.

Note that the API will do this for you but locking the GIL before doing a string of operations is faster than having each operation lock the GIL individually.

-makeFastcallable

(-makeFastcallable item)

Given a normal python callable, make a fastcallable object that needs to be closed. This should be seen as a callsite optimization for repeatedly calling a specific python function in a tight loop with positional arguments. It is not intended to be used in a context where you will then pass this object around as this is not a reentrant optimization.

  try (AutoCloseable fastcaller = java_api.makeFastcallable(pycallable)) {
     tightloop: {
       java_api.call(fastcaller, arg1, arg2);
     }
  }

-runStringAsFile

(-runStringAsFile strdata)

Run a string returning the result of the last expression. Strings are compiled and live for the life of the interpreter. This is the equivalent to the python exec all.

The global context is returned as a java map.

This function expects the GIL to be locked - it will not lock/unlock it for you.

Example:

   Map globals = java_api.runStringAsFile("def calcSpread(bid,ask):
	return bid-ask

");
   Object spreadFn = globals.get("calcSpread");
   java_api.call(spreadFn, 1, 2); // Returns -1

-runStringAsInput

(-runStringAsInput strdata)

Run a string returning the result of the last expression. Strings are compiled and live for the life of the interpreter. This is the equivalent to the python eval call.

This function expects the GIL to be locked - it will not lock/unlock it for you.

Example:

  java_api.setGlobal("bid", 1);
  java_api.setGlobal("ask", 2);
  java_api.runStringAsInput("bid-ask"); //Returns -1

-setAttr

(-setAttr pyobj attname objval)

Set an attribute on a python object. This corresponds to the __setattr python call.

-setGlobal

(-setGlobal varname varval)

Set a value in the global dict. This function expects the GIL to be locked - it will not lock/unlock it for you.

In addition to numbers and strings, this method can take an implementation of clojure.lang.IFn that will be converted to a python callable, arrays and lists will be converted to python lists. If you would like numpy arrays use -createArray.

-setItem

(-setItem pyobj itemName itemVal)

-unlockGIL

(-unlockGIL gilstate)

Unlock the gil passing in the gilstate returned from lockGIL. Each call to lockGIL must be paired to a call to unlockGIL.