Questions tagged [concurrent.futures]
concurrent.futures is a Python module which provides a high-level interface for asynchronously executing callables.
concurrent.futures
984
questions
0
votes
1
answer
53
views
Getting the result from a future in Python
I have the following code which executes a process and calls a callback function when the process is done
import os
import subprocess
import tempfile
def callback(future):
print(future....
- 1,192
0
votes
0
answers
27
views
When using the Python Concurrent Futures should I use ALL the vCores?
Is it safe when using the Python Concurrent Futures to use ALL the vCores available in the machine?
My code currently uses the vCores available minus 1 as I want to leave one vCore "free" ...
- 678
0
votes
1
answer
54
views
Python Concurrent Futures not taking the data out of memory
I am working on Python code that copies tables of 50+ GB from PostgreSQL to SQL server and creates tables dynamically. Copies the data using SQLAlchemy Streaming and batch insert using Concurrent ...
0
votes
0
answers
9
views
Can processing of request/grequest responses be run concurrently in Python2?
I'm writing a Python2 program that pulls a large amount of JSON data from a remote CouchDB database. The data is indexed by timestamp: I pass a startkey and an endkey, and the database returns all ...
- 303
0
votes
0
answers
25
views
Passing Spark Session variable as a parameter while executing a function using ProcessPoolExecutor is not working
Pyspark version: 3.3.0-amzn-0
Python: 3.7.16
I am using the below code snippet where I am trying to use spark session as a parameter while calling a function by name test using ProcessPoolExecutor.
...
- 99
0
votes
0
answers
21
views
Improve Parallelising *Reading, Cropping and Patching* individual .RT-H5 files (using Python **H5py**)
I have a data processing pipeline that is a perfect candidate for parallelisation but I can't seem to get "good" speed ups.
The process that I need to complete is:
Read in a .RT-H5 file ...
- 131
0
votes
0
answers
20
views
How can I transform a petl table across all CPUs?
I created an ETL pipeline using Python ETL PETL.
It works as expected, but my current transformation part is a bit slow.
I'm utilizing ProcessPoolExecutor to transform each petl data table in separate ...
- 246
1
vote
1
answer
53
views
Can I create a global ThreadPoolExecutor in a Flask application?
I am using a concurrent.futures.ThreadPoolExecutor in a WSGI (Flask) Python REST service to send queries in parallel. I use the code below to instantiate one executor per request. There is one ...
- 609
1
vote
0
answers
30
views
How to capture printed messages in parallel process?
I have a function parallel_run which print out the diagnostic message during the run and then return a result. I want to capture the printed message and also the returned value. However, somehow I got ...
0
votes
0
answers
11
views
Skipping any unreachable network device using concurrent.futures.ThreadPoolExecutor in python
With my portion of the code below using concurrent.futures.ThreadPoolExecutor
if __name__ == "__main__":
xw.Book("Main_Template.xlsm").set_mock_caller()
test1 = ...
0
votes
0
answers
112
views
Finding the valid tetrahedrons with smallest volume for the large input file size
I have two files points_small.txt and points_large.txt which contain the list of points on a 3D plane. Each point is defined by its coordinates and an associated number and is presented in the ...
- 139
0
votes
1
answer
56
views
Multithreading stuck at last future
def download_files_from_folder(base_url, folder_name):
folder_url = f"{base_url}{folder_name}/"
response = requests.get(folder_url)
soup = BeautifulSoup(response.content, "...
- 11
0
votes
1
answer
42
views
How to close multiprocessing pool inside process?
I am wondering how to make multiprocessing in python (3.11) with asynchronous calls (not asyncio lib) and automatically close processes when they are finished?
Below I wrote a simple code and the ...
- 1,713
1
vote
1
answer
36
views
How do you parallelize access to a shared array in python using concurrent.futures?
I have the following piece of code to illustrate my problem:
Each thread calculates a value locs and then updates the result array, assume that that update (result[locs] += mask[locs] ) is a very slow ...
- 25.7k
0
votes
1
answer
52
views
concurrent.futures and Pandas DataFrame
I tried use multiprocessing to create 2 dataframes at the same time. The result of 2 functions are Pandas Dataframe. When I tried use 'concurrent.futures.ProcessPoolExecutor', but I can't extrat this ...
- 1