askvity

How Do I Run Multiple Python Programs?

Published in Python Programming 4 mins read

There are several ways to run multiple Python programs, depending on whether you need them to run simultaneously (concurrently) or sequentially (one after another). Here are the most common methods:

1. Running Programs Sequentially:

This is the simplest approach. You execute one Python script, and after it completes, you execute the next one.

  • From the Command Line/Terminal:
Open your command line or terminal and execute each script one by one:

```bash
python script1.py
python script2.py
python script3.py
```

This will run `script1.py` first, then `script2.py`, and finally `script3.py`.
  • Using a Driver Script:
You can create a Python script (e.g., `run_all.py`) that executes the other scripts sequentially:

```python
import subprocess

def run_script(script_name):
    """Runs a Python script using subprocess."""
    try:
        subprocess.run(['python', script_name], check=True) # check=True raises an exception for non-zero exit codes
    except subprocess.CalledProcessError as e:
        print(f"Error running {script_name}: {e}")

if __name__ == "__main__":
    run_script("script1.py")
    run_script("script2.py")
    run_script("script3.py")
```

Save this as `run_all.py` and execute it: `python run_all.py`

2. Running Programs Concurrently (Simultaneously):

This allows multiple Python scripts to run at the same time, potentially speeding up tasks that can be parallelized. Python provides several modules for concurrency:

  • Using threading (for I/O-bound tasks):
The `threading` module is suitable for tasks that spend a lot of time waiting for input/output (I/O), such as network requests or file operations. Due to the Global Interpreter Lock (GIL) in Python, true parallel execution of CPU-bound tasks is limited with threads.

```python
import threading
import subprocess

def run_script(script_name):
    """Runs a Python script using subprocess in a thread."""
    try:
        subprocess.run(['python', script_name], check=True)
    except subprocess.CalledProcessError as e:
        print(f"Error running {script_name}: {e}")


if __name__ == "__main__":
    threads = []
    scripts = ["script1.py", "script2.py", "script3.py"]

    for script in scripts:
        t = threading.Thread(target=run_script, args=(script,))
        threads.append(t)
        t.start()

    for t in threads:
        t.join()  # Wait for all threads to complete

    print("All scripts finished.")
```

*   **Using `multiprocessing` (for CPU-bound tasks):**

The `multiprocessing` module creates separate processes, bypassing the GIL and allowing true parallel execution on multi-core systems. Use this for CPU-intensive tasks.

```python
import multiprocessing
import subprocess

def run_script(script_name):
    """Runs a Python script using subprocess in a process."""
    try:
        subprocess.run(['python', script_name], check=True)
    except subprocess.CalledProcessError as e:
        print(f"Error running {script_name}: {e}")


if __name__ == "__main__":
    processes = []
    scripts = ["script1.py", "script2.py", "script3.py"]

    for script in scripts:
        p = multiprocessing.Process(target=run_script, args=(script,))
        processes.append(p)
        p.start()

    for p in processes:
        p.join()  # Wait for all processes to complete

    print("All scripts finished.")
```

Example script1.py (and similarly for script2.py and script3.py):

import time

print("Script 1 starting...")
time.sleep(2)  # Simulate some work
print("Script 1 finished.")

Important Considerations:

  • Error Handling: The subprocess.run function with check=True will raise an exception if a script returns a non-zero exit code (indicating an error). The try...except blocks in the examples handle these errors. Adapt the error handling to your specific needs.
  • Resource Management: When running scripts concurrently, be mindful of resource usage (CPU, memory, file handles). Avoid overwhelming the system.
  • Data Sharing: If your scripts need to share data, use appropriate inter-process communication (IPC) mechanisms like queues or shared memory when using multiprocessing. For threading, be careful with shared resources due to potential race conditions. Consider using locks.
  • Output Handling: Concurrent scripts might write to the console simultaneously, leading to interleaved output. Consider redirecting output to files or using logging to organize the output.

In summary, choose sequential execution for simple tasks or when order matters. Use threading for I/O-bound concurrency and multiprocessing for CPU-bound concurrency. Always handle errors and manage resources appropriately.

Related Articles