I have recently been working on version 0.6.2. I am excited about the changes.
The biggest one is that I have made the queue system definition fully modular
and moved it to the batch_systems sub-package, so adding a new queue system
is now very easy.
I also completely changed the local queue system and made it a database using daemon process instead. This adds a few new requirements, but it is worth it, the local queue is now extremely fast and stable.
Also, on a whim I added the ability to submit files and shell scripts with the fyrd console script, so you can now do things like::
fyrd run --extra_vars 'output:input:\(.*\).txt:\1.out' "my_algorithm {input} > {output}" "data/{input}"
and:
fyrd submit --wait script.qsub
Finally, the other big change is the appearance of a decorator called jobify that
massively simplifies submitting functions:
>>> import fyrd
>>> @fyrd.jobify(name='test_job', mem='1GB')
... def test(string, iterations=4):
... """This does basically nothing!"""
... outstring = ""
... for i in range(iterations):
... outstring += "Version {0}: {1}".format(i, string)
... return outstring
...
>>> j = test('hi')
>>> j.get()
'Version 0: hiVersion 1: hiVersion 2: hiVersion 3: hi'
>>> @fyrd.jobify()
... def raise_me(num, power=10):
... return num**power
...
>>> raise_me(10).get()
10000000000
I hope to get an alpha version of 0.6.2 out shortly.