as-jvm failed for a returned list

[kenhuang-io]

Hi,

I experienced an issue while I was using libpython-clj to encode the returned list as json via cheshire, it threw an exception with this snippet:

  (println "Just something interesting from Python:"
           (-> (py/py. simple a_list)
               py/as-list
               json/generate-string))

The exception:

Execution error (ClassCastException) at cheshire.generate/generate$fn (generate.clj:135).
class java.lang.String cannot be cast to class java.util.Map$Entry (java.lang.String and java.util.Map$Entry are in module java.base of loader 'bootstrap')

And it worked fine for a map.

I've created a demo repo for this: https://github.com/whatacold/libpython-clj-as-jvm-issue, hopefully I've put all necessary info there.

Was there something I was missing? Thanks.

[cnuernber]

I think what you are missing is that as-jvm creates bridge objects that implement a subset of the interfaces in java.util -- ->jvm creates copies of the data. Passing a bridge object into cheshire isn't a use case we support as the resulting pathway is going to call into the python interpreter in a fine grained fashion and is thus very likely to perform quite badly.

Python has json libraries - using their libraries and then only marshalling the string itself across language boundaries is likely to be faster and more correct than as-jvm -> cheshire.

That all being said - this is interesting to me in that it is something that should work -- albeit there are better ways -- and this may represent an area where we could have better support for the jvm interfaces. ham-fisted library interface defaults could be used here.

[kenhuang-io]

I think what you are missing is that as-jvm creates bridge objects that implement a subset of the interfaces in java.util -- ->jvm creates copies of the data. Passing a bridge object into cheshire isn't a use case we support as the resulting pathway is going to call into the python interpreter in a fine grained fashion and is thus very likely to perform quite badly.

Thanks for the explanations, I had a feeling that maybe it was wrong to feed it into cheshire directly.

Python has json libraries - using their libraries and then only marshalling the string itself across language boundaries is likely to be faster and more correct than as-jvm -> cheshire.

Yes, that was what I did, by using json.dumps to encode it to a string for clojure. It's even faster by doing this way? I didn't expect that.

That all being said - this is interesting to me in that it is something that should work -- albeit there are better ways -- and this may represent an area where we could have better support for the jvm interfaces. ham-fisted library interface defaults could be used here.

I'm wondering, is it rare to return a list/map/set, etc. from python? What do you usually use libpython-clj for? For me, I was rewriting a webapp that was in python, so I had some logic using python regex to do the trick, hence I needed to feed a python function a map and then get back a list.

[cnuernber]

The libpython-clj default is to use ->jvm via copy which I think is usually faster than converting to json. Bridging is used for object that have behavior on them and ->jvm is used for datastructures.