Questions tagged [multiprocessing]
Multiprocessing is the use of two or more central processing units (CPUs) within a single computer system. Relevant implementation and usage details vary per operating system and programming language. So always add tags for both the OS and language when using this tag.
multiprocessing
14,724
questions
-2
votes
0
answers
17
views
What exactly happens when I spawn 5 workers using Gunicorn in PROD for Fast API? [duplicate]
So my understanding is that workers should usually be 2 * cores + 1. Lets suppose I have a 2 core machine and spawned 5 workers in a PROD instance.
How does it work under the hood ?
Does Fast API ...
- 125
0
votes
0
answers
38
views
Python multiprocessing pool with class objects
This code is runnable
I get an empty array "link"
although I should get an array with objects in it,
when I print link in func_b it is not empty
Self should refer to the object, but I guess ...
- 150
0
votes
0
answers
30
views
Why I am facing [Errno 32] Broken pipe when I start new process?
I am using confluent-kafka to process messages from kafka topic.
Here is the code template:
from multiprocessing import Process, Manager
def handle_records(records):
for record in records:
...
- 53
2
votes
1
answer
55
views
Multiprocessing shared memory to pass large arrays between processes
Context: I need to analyse some weather data for every hour of the year. For each hour, I need to read in some inputs for each hour before performing some calculation. One of these inputs is a very ...
- 21
0
votes
0
answers
38
views
GUI unable to send messages through Manager().Queue() to the processes
Please guide me in the right direction if this is already answered. I have a GUI script that has several buttons and when I click the start button, it needs to send a message to a process which is ...
3
votes
1
answer
117
views
Why does iterating break up my text file lines while a generator doesn't?
For each line of a text file I want to do heavy calculations. The amount of lines can be millions so I'm using multiprocessing:
num_workers = 1
with open(my_file, 'r') as f:
with multiprocessing....
- 7,518
-7
votes
0
answers
41
views
How to use python multiprocessing? [closed]
I want to use multiprocessing to improve the time execution of the following function:
def verif(mat1,tab):
import numpy as np
mat=np.matrix(mat1)
t=1
leng=len(mat)
mm=mat+np.eye(...
- 1
0
votes
0
answers
31
views
Issue with running flask as a new process through a customtkinter GUI
My GUI lets the user play a crossword online using Flask. When it is used as a package and initialised through an executable (like the one defined in a setup.py in the entry points), it works fine. ...
- 51
1
vote
1
answer
58
views
Trouble Capturing All Tick Data with Concurrent.futures in Python [closed]
I am developing a program to insert tick-level data for 300+ stocks received in a webstream dictionary format from a broker API. I successfully have a version of the program running without concurrent....
1
vote
0
answers
29
views
Trying to replace a for loop with pool to write data to a xlsx file with multiple processes
I have a program that reads a csv file with Pandas and then writes the data to an xlsx file with XLSXWriter in order to make plots of the data. I am trying to change the for loop which is used to ...
- 23
0
votes
2
answers
46
views
Multiprocessing sharing c pointer in Python
I tried implementing tree sitter support in Python using multiprocessing but the Python process is crashing. Further below is a Minimum Reproducible Example.
My needs are to be able to send the ...
- 213
-4
votes
0
answers
38
views
I'm trying to use the multiprocessing module on my MacBook to parallelize some computations. On macOS, it throws a RuntimeError [closed]
My code works perfectly on Windows, but on macOS, it throws a RuntimeError. Here’s a simplified version of my code (my python version is : 3.10.9 ):
import multiprocessing as mp
def worker(num):
...
- 1
2
votes
0
answers
30
views
Python Multiprocessing with Pipe: Subprocesses sleep forever when training deep reinforcement learning agent with no error raised
I was using Pytorch-based Deep Reinforcement Learning (DRL) agent to train the environment with Multiprocessing package to train in multiple processes (thanks to the renowned elegantrl framework).
To ...
- 21
0
votes
1
answer
16
views
TypeError: cannot pickle 'builtins.DT' object - multiprocessing, pickle, pathos
I'm using "startinpy" library (https://startinpy.readthedocs.io/en/0.10.2/) in my project. It generally works great, but when I want to use it in second process, the pickle error shows up:
...
- 359
0
votes
0
answers
56
views
Python fastapi asyncio or threading + multiprocessing, which is better approach [closed]
Please help me find out a best (better) approach to use multiprocessing with fastapi
I have a project where i have one endpoint. Where heavy ocr model(easyocr) is imported and initialized in different ...
