libpython-clj2.python

Python bindings for Clojure. This library dynamically finds the installed python, loads the shared library and allows Clojure users to use Python modules as if they were Clojure namespaces.

Example:

user> (require '[libpython-clj2.python :as py])
nil
user> (py/initialize!)
;;  ... (logging)
:ok
user> (def np (py/import-module "numpy"))
#'user/np
user> (py/py. np linspace 2 3 :num 10)
[2.         2.11111111 2.22222222 2.33333333 2.44444444 2.55555556
 2.66666667 2.77777778 2.88888889 3.        ]

$a

macro

($a item attr & args)

Call an attribute of an object using automatic detection of the python kwargs. Keywords must be compile time constants. So this won't work with 'apply'. On the other hand, building the positional and kw argmaps happens at compile time as opposed to at runtime. The attr name can be a symbol.

$c

macro

($c item & args)

Call an object using automatic detection of the python kwargs. Keywords must be compile time constants. So this won't work with 'apply'. On the other hand, building the positional and kw argmaps happens at compile time as opposed to at runtime.

->jvm

(->jvm v & [opts])

Copy a python value into java datastructures

->py-dict

(->py-dict v)

Copy v into a python dict

->py-list

(->py-list v)

Copy the data into a python list

->py-tuple

(->py-tuple v)

Copy v into a python tuple

->python

(->python v)

Copy a jvm value into a python object

add-module

(add-module modname)

Add a python module. This can create a module if it doesn't exist.

afn

(afn item attr & args)

Call an attribute of an object. Arguments are passed in positionally. Any keyword arguments are paired with the next arg, gathered, and passed into the system as *kwargs.

Not having an argument after a keyword is an error.

as-jvm

(as-jvm v & [opts])

Copy a python value into java datastructures

as-list

(as-list pobj)

Make a python object appear as a list

as-map

(as-map pobj)

Make a python object appear as a map of it's items

as-python

(as-python v)

Bridge a jvm value into a python object

attr-type-map

(attr-type-map pyobj)

Return a map of attr name to python-type of the attribute

call-attr

(call-attr pyobj attname & args)

Call an attribute on a python object using only positional arguments

call-attr-kw

(call-attr-kw pyobj attname args kw-list)

Call an attribute passing in both positional and keyword arguments.

callable?

(callable? pyobj)

Return true if python object is callable.

cfn

(cfn item & args)

Call an object. Arguments are passed in positionally. Any keyword arguments are paired with the next arg, gathered, and passed into the system as *kwargs.

Not having an argument after a keyword argument is an error.

create-class

(create-class name bases cls-hashmap)

Create a new class object. Any callable values in the cls-hashmap will be presented as instance methods. If you want access to the 'this' object then you must use make-instance-fn.

Example:

user> (require '[libpython-clj2.python :as py])
nil
user> (def cls-obj (py/create-class
                    "myfancyclass"
                    nil
                    {"__init__" (py/make-instance-fn
                                 (fn [this arg]
                                   (py/set-attr! this "arg" arg)
                                   ;;If you don't return nil from __init__ that is an
                                   ;;error.
                                   nil))
                     "addarg" (py/make-instance-fn
                               (fn [this otherarg]
                                 (+ (py/get-attr this "arg")
                                    otherarg)))}))
#'user/cls-obj
user> cls-obj
__no_module__.myfancyclass
user> (def inst (cls-obj 10))
#'user/inst
user> (py/call-attr inst "addarg" 10)
20

def-unpack

macro

(def-unpack symbols input)

Unpack a set of symbols into a set of defs. Useful when trying to match Python idioms - this is definitely not idiomatic Clojure.

Example:

user> (py/def-unpack [a b c] (py/->py-tuple [1 2 3]))
#'user/c
user> a
1
user> b
2
user> c
3

dir

(dir pyobj)

from-import

macro

(from-import module-path item & args)

Support for the from a import b,c style of importing modules and symbols in python. Documentation is included.

get-attr

(get-attr pyobj attname)

Get an attribute from a python object

get-item

(get-item pyobj item-name)

Get an item from a python object using getitem

has-attr?

(has-attr? pyobj att-name)

Return true if this python object has this attribute.

has-item?

(has-item? pyobj item-name)

Return true if the python object has an item. Calls hasitem.

import-as

macro

(import-as module-path varname)

Import a module and assign it to a var. Documentation is included.

import-module

(import-module modname)

Import a python module. Module entries can be accessed via get-attr.

initialize!

(initialize! & {:keys [windows-anaconda-activate-bat library-path no-io-redirect?], :as options})

Initialize the python library. If library path is not provided, then the system attempts to execute a simple python program and have python return system info.

Note: all of the options passed to initialize! may now be provided in a root-level python.edn file. Example:

;; python.edn
{:python-executable   "/usr/bin/python3.7"
 :python-library-path "/usr/lib/libpython3.7m.so"
 :python-home         "/usr/lib/python3.7"
 :python-verbose      true}

or, using a local virtual environment:

;; python.edn
{:python-executable   "env/bin/python"}

Additionaly the file can contain two keys which can can refer to custom hooks to run code just before and just after python is initialised. Typical use case for this is to setup / verify the python virtual enviornment to be used.

:pre-initialize-fn my-ns/my-venv-setup-fn!
:post-initialize-fn my-ns/my-venv-validate-fn!

A :pre-initialize-fn could for example shell out and setup a python virtual enviornment.

The :post-initialize-fn can use all functions from ns libpython-clj2.python as libpython-clj is initialised alreday and could for example be used to validate that later needed libraries can be loaded via calling import-module.

The file MUST be named python.edn and be in the root of the classpath. With a python.edn file in place, the initialize! function may be called with no arguments and the options will be read from the file. If arguments are passed to initialize! then they will override the values in the file.

Returns either :ok in which case the initialization completed successfully or :already-initialized in which case we detected that python has already been initialized via Py_IsInitialized and we do nothing more.

Options:

  • :library-path - Library path of the python library to use.
  • :program-name - Optional -- will show up in error messages from python.
  • :no-io-redirect? - True if you don't want python stdout and stderr redirection to out and err.
  • :python-executable - The python executable to use to find system information.
  • :python-home - Python home directory. The system first uses this variable, then the environment variable PYTHON_HOME, and finally information returned from python system info.
  • :signals? - defaults to false - true if you want python to initialized signals. Be aware that the JVM itself uses quite a few signals - SIGSEGV, for instance - during it's normal course of operation. For more information see:

is-instance?

(is-instance? py-inst py-cls)

Return true if inst is an instance of cls. Note that arguments are reversed as compared to instance?

make-callable

(make-callable ifn options)(make-callable ifn)

Make a python callable object from a clojure function. This is called for you if you use as-python on an implementation of IFn.

Options:

  • :arg-converter - Function called for each function argument before your ifn gets access to it. Defaults to ->jvm.
  • :result-converter - Function called on return value before it gets returned to python. Must return a python object. Defaults to ->python; the result will get an extra incref before being returned to Python to account for the implied tracking of as-python or ->python.
  • :name - Name of the python method. This will appear in stack traces.
  • :doc - documentation for method.

make-fastcallable

(make-fastcallable item)

Wrap a python callable such that calling it in a tight loop with purely positional arguments is a bit (2x-3x) faster.

Example:

user> (def test-fn (-> (py/run-simple-string "def spread(bid,ask):
	return bid-ask

")
                       (get :globals)
                       (get "spread")))
#'user/test-fn
user> test-fn
<function spread at 0x7f330c046040>
user> (py/with-gil (time (dotimes [iter 10000]
                           (test-fn 1 2))))
"Elapsed time: 85.140418 msecs"
nil
user> (py/with-gil (time (dotimes [iter 10000]
                           (test-fn 1 2))))
"Elapsed time: 70.894275 msecs"
nil
user> (with-open [test-fn (py/make-fastcallable test-fn)]
        (py/with-gil (time (dotimes [iter 10000]
                             (test-fn 1 2)))))

"Elapsed time: 39.442622 msecs"
nil
user> (with-open [test-fn (py/make-fastcallable test-fn)]
        (py/with-gil (time (dotimes [iter 10000]
                             (test-fn 1 2)))))

"Elapsed time: 35.492965 msecs"
nil

make-instance-fn

(make-instance-fn ifn options)(make-instance-fn ifn)

Make an callable instance function - a function which will be passed the 'this' object as it's first argument. In addition, this function calls make-callable with a arg-converter defaulted to as-jvm. See documentation for make-callable and libpython-clj2.python.class/make-tuple-instance-fn.

make-kw-instance-fn

(make-kw-instance-fn ifn options)(make-kw-instance-fn ifn)

Make an kw callable instance function - function by default is passed 2 arguments, the positional argument vector and a map of keyword arguments. Results are marshalled back to python using libpython-clj2.python.fn/bridged-fn-arg->python which is also used when bridging an object into python. See documentation for make-callable libpython-clj2.python.class/make-kw-instance-fn.

module-dict

(module-dict mod)

Get the module dictionary.

path->py-obj

(path->py-obj item-path & {:keys [reload?]})

Given a string such as "builtins" or "builtins.list", load the module or the class object in the module.

Options:

  • :reload - Reload the module.

py*

macro

(py* x method args)(py* x method args kwargs)

Special syntax for passing along *args and **kwargs style arguments to methods.

Usage:

(py* obj method args kwargs)

Example:

(def d (python/dict)) d ;;=> {} (def iterable [:a 1 :b 2]) (def kwargs {:cat "dog" :name "taco"}) (py* d update iterable kwargs) d ;;=> {"a": 1, "b": 2, "cat": "dog", "name": "taco"}

py**

macro

(py** x method kwargs)(py** x method arg & args)

Like py*, but it is assumed that the LAST argument is kwargs.

py.

macro

(py. x method-name & args)

Class/object method syntax. (py. obj method arg1 arg2 ... argN) is equivalent to Python's obj.method(arg1, arg2, ..., argN) syntax.

py.-

macro

(py.- x arg)

Class/object getter syntax. (py.- obj attr) is equivalent to Python's obj.attr syntax.

py..

macro

(py.. x & args)

Extended accessor notation, similar to the .. macro in Clojure.

(require-python 'sys) (py.. sys -path (append "/home/user/bin"))

is equivalent to Python's

import sys sys.path.append('/home/user/bin')

SPECIAL SYNTAX for programmatic *args and **kwargs

Special syntax is provided to meet the needs required by Python's *args and **kwargs syntax programmatically.

(= (py.. obj (method args)) (py obj methods args))

(= (py.. obj (method args kwargs)) (py obj method args kwargs))

(= (py.. obj (method kwargs)) (py obj method kwargs))

(= (py.. obj (method arg1 arg2 arg3 ... argN kwargs)) (py obj method arg1 arg2 arg3 ... argN kwargs) (py* obj method arg1 arg2 arg3 ... argN kwargs))

These forms exist for when you need to pass in a map of options in the same way you would use the f(*args, **kwargs) forms in Python.

python-type

(python-type v)

Get the type (as a keyword) of a python object

run-simple-string

(run-simple-string program & {:keys [globals locals]})

Run a string expression returning a map of {:globals :locals}. This uses the global main dict under the covers so it matches the behavior of the cpython implementation with the exception of returning the various maps used.

Note this will never return the result of the expression: https://mail.python.org/pipermail/python-list/1999-April/018011.html

Globals, locals may be provided but are not necessary.

Implemented in cpython as:

PyObject *m, *d, *v; m = PyImport_AddModule("main"); if (m == NULL) return -1; d = PyModule_GetDict(m); v = PyRun_StringFlags(command, Py_file_input, d, d, flags); if (v == NULL) { PyErr_Print(); return -1; } Py_DECREF(v); return 0;

set-attr!

(set-attr! pyobj attname attval)

Set an attribute on a python object. Returns pyobj.

set-attrs!

(set-attrs! pyobj att-seq)

Set a sequence of name value attributes. Returns pyobj.

set-item!

(set-item! pyobj item-name item-val)

Set an item on a python object using setitem

set-items!

(set-items! pyobj item-seq)

Set a sequence of name value. Returns pyobj

stack-resource-context

macro

(stack-resource-context & body)

Create a stack-based resource context. All python objects allocated within this context will be released at the termination of this context. !!This means that no python objects can escape from this context!! You must use copy semantics (->jvm) for anything escaping this context. Furthermore, if you are returning generic python objects you may need to call (into {}) or something like that just to ensure that absolutely everything is copied into the jvm.

with

macro

(with bind-vec & body)

Support for the 'with' statement in python: (py/with item (py/call-attr testcode-module "WithObjClass" true fn-list) (py/call-attr item "doit_err"))

with-gil

macro

(with-gil & body)

Capture the gil for an extended amount of time. This can greatly speed up operations as the mutex is captured and held once as opposed to fine grained grabbing/releasing of the mutex.

with-gil-stack-rc-context

macro

(with-gil-stack-rc-context & body)

Capture the gil, open a resource context. The resource context is released before the gil is leading to much faster resource collection. See documentation on stack-resource-context for multiple warnings; the most important one being that if a python object escapes this context your program will eventually, at some undefined point in the future crash. That being said, this is the recommended pathway to use in production contexts where you want defined behavior and timings related to use of python.

with-manual-gil

macro

(with-manual-gil & body)

When running with -Dlibpython_clj.manual_gil=true, you need to wrap all accesses to the python runtime with this locker. This includes calls to require-python or any other pathways.

  (with-manual-gil
    ...)

with-manual-gil-stack-rc-context

macro

(with-manual-gil-stack-rc-context & body)

When running with -Dlibpython_clj.manual_gil=true, you need to wrap all accesses to the python runtime with this locker. This includes calls to require-python or any other pathways. This macro furthermore defines a stack-based gc context to immediately release objects when the stack frame exits.