Python multiprocessing library in Azure Functions

Question

I have python code that uses multiprocessing to speed up some processing. We are considering starting using Azure Functions. How multiprocessing library would behave in an Azure Function? If we deploy our current code will Azure (with multiprocessing) do the split over cores the same way as it's done on a standard machine?

Answer 1

I agree with @Ziya Mert Karakas -

Azure Functions can execute Python code. By design, a Python host instance can only handle one function call at once. The FUNCTIONS_WORKER_PROCESS_COUNT - 1-10 in application setting allows you to increase the number of worker processes per host (up to 10). Python versions 3.10, 3.9, and are supported by Azure Functions.

Note- FUNCTIONS_WORKER_PROCESS_COUNT does not work with Python v2 function

To increase throughput, Azure Functions also enables memory sharing between your out-of-process Python language worker and the Functions host process. By adding an application setting with the name FUNCTIONS_WORKER_SHARED_MEMORY_DATA_TRANSFER_ENABLED and a value of 1, you can enable shared memory when your function app has bottlenecks. Once shared memory is enabled, you may adjust the shared memory to a size of 256 MB by using the DOCKER_SHM_SIZE parameter.

In conclusion, you can run Python code using Azure Functions, but you should avoid utilising the multiprocessing library and instead use asyncio for parallelism. Additionally, you can boost performance by raising the number of worker processes per host.

Sample http trigger code with async method & asyncio library:-

import logging
import asyncio
import azure.functions as func

async def main(req: func.HttpRequest) -> func.HttpResponse:
    logging.info('Python HTTP trigger function processed a request.')

    name = req.params.get('name')
    if not name:
        try:
            req_body = await req.get_json()
        except ValueError:
            pass
        else:
            name = req_body.get('name')

    if name:
        return func.HttpResponse(f"Hello, {name}. This HTTP triggered function executed asynchronously.")
    else:
        return func.HttpResponse(
             "Query String",
             status_code=400
        )

Output:-

Sample function multiprocessing module with spawn start method:-

import multiprocessing as mp
import azure.functions as func

def worker():
    # your multiprocessing code here
    pass

def main(req: func.HttpRequest) -> func.HttpResponse:
    # create a new process for each function invocation
    ctx = mp.get_context('spawn')
    p = ctx.Process(target=worker)
    p.start()
    p.join()

    return func.HttpResponse("Hello from Azure Functions!")

Output:-

Reference:-

azure-docs/articles/azure-functions/functions-reference-python.md at main · MicrosoftDocs/azure-docs (github.com)

azure-docs/articles/azure-functions/functions-reference-python.md at main · MicrosoftDocs/azure-docs (github.com)