top button
Flag Notify
    Connect to us
      Facebook Login
      Site Registration Why to Join

Facebook Login
Site Registration

Parallel python in the cloud

+1 vote
41 views

We were happily using PiCloud for several long calculations and we very happy with with it. With their real-time cores, we could take really large calculations set and run through fairly quickly.

Now that PiCloud is going away, we ran a few tests on Mutlyvac but so far, we are struggling to accomplish the same thing we had on PiCloud.

I have several "pieces" of my puzzle but can't seem to be able to put it together. I've seen and tried StarCluster and also various parallel python options but all options seem challenging to put together.

The goal is to mimic PiCloud, ie. loop through a function:

def some_NP_func(x, y):
 ...
 return z

some_cloud.call(some_NP_func, a1, a2)

Which computes the function on the cloud. We use this often in for loops with arrays of arguments. The other scenario is:

some_cloud.map(some_NP_intense_func, [...], [...])

Which iterates through and returns results. We need to run a lot of this in batch from a scheduler so I always try to avoid interactive environment (how does iPython parallel work in batch?).

What is the preferred approach or method right now for heavy parallel computation like this?

posted May 24, 2014 by Gurminder

Share this question
Facebook Share Button Twitter Share Button Google+ Share Button LinkedIn Share Button Multiple Social Share Button

1 Answer

+1 vote

IPython parallel works just fine "in batch". As far as your client code (i.e. what you wrote above) is concerned, it's just another library. E.g.

https://github.com/ipython/ipython/blob/master/examples/Parallel%20Computing/nwmerge.py
https://github.com/ipython/ipython/blob/master/examples/Parallel%20Computing/itermapresult.py

answer May 24, 2014 by Honey
Similar Questions
+2 votes

Operators look for OSS on cloud.
How can I move my solution to cloud and how do I choose a cloud platform?

+1 vote

Please give me a valuable inputs..
most of the applications like real estate apps n android apps to be hosted onto the server..

+5 votes

What are the security measures one should take before adopting a cloud-based test environment?

Contact Us
+91 9880187415
sales@queryhome.net
support@queryhome.net
#280, 3rd floor, 5th Main
6th Sector, HSR Layout
Bangalore-560102
Karnataka INDIA.
QUERY HOME
...