At Function, we've been building a code generation platform for running Python code natively, cross-platform. AI inference is our highest-traffic use case, and we're now serving thousands of devices with inference binaries every month. Today, I'm pleased to announce that we're launching public access for compiling AI inference functions using the Function SDK for Python. Let's walk through a concrete example:

Writing an Image Classifier

Let's build an Android application that classifies what is seen by the device camera in realtime. For this, let's use the MobileNet v2 image classifier model from the PyTorch torchvision package:

from PIL import Image
from torch import argmax, softmax, randn
from torchvision.models import mobilenet_v2, MobileNet_V2_Weights
from torchvision.transforms import functional as F

weights = MobileNet_V2_Weights.DEFAULT
model = mobilenet_v2(weights=weights).eval()

def predict (image: Image.Image) -> tuple[str, float]:
    """
    Classify an image.
    """
    # Preprocess
    image = image.convert("RGB")
    image = F.resize(image, 224)
    image = F.center_crop(image, 224)
    image_tensor = F.to_tensor(image)
    normalized_tensor = F.normalize(
        image_tensor,
        mean=[0.485, 0.456, 0.406],
        std=[0.229, 0.224, 0.225]
    )
    # Run model
    logits = model(normalized_tensor[None])
    scores = softmax(logits, dim=1)
    idx = argmax(scores, dim=1)
    score = scores[0,idx].item()
    label = weights.meta["categories"][idx]
    # Return
    return label, score

This code is 100% idiomatic PyTorch: it's the kind of code your AI research scientists would write; and the kind that you would find in open-source AI models on GitHub or HuggingFace. Let's run the function to ensure that it classifies an input image correctly:

Generating Native Code with Function

Up till now, deploying AI inference into applications has been a treacherous journey: you would need to figure which inference library to use (e.g. ONNXRuntime, Llama.cpp); how to write code to integrate, compile, and link against said library; how to convert the AI model to the library's preferred format; and how to translate required pre- and post-processing code.

All of this ends up being incredibly arduous but undifferentiated work, taking time away from building the higher-impact aspects of your application. Let's see how Function simplifies this entire workflow:

from fxn import compile, Sandbox
from fxn.beta import ONNXInferenceMetadata
...

@compile(
    tag="@yusuf/mobilenet-v2",
    description="Image classifier trained on ImageNet 1k.",
    sandbox=Sandbox().pip_install("torch==2.6.0", "torchvision==0.21"),
    targets=["android", "macos"],
    metadata=[
        ONNXInferenceMetadata(
          model=model,
          model_args=[randn(1, 3, 224, 224)]
        ),
    ]
)
def predict (image: Image.Image) -> tuple[str, float]:
    ...

First, we use the @compile decorator to decorate the prediction function, adding metadata to specify how we want to run the classifier model at runtime (here we use ONNXRuntime). Next, we'll use the Function CLI to generate native C++ code from our Python code, then compile the generated code to run on Android, macOS, and in the browser (with WebAssembly):

# 🔥 Compile our image classifier with the Function CLI
$ fxn compile mobilenet_v2.py

Over the next few minutes, Function will upload our code into a cloud sandbox, generate native C++ code from our Python code, then compile the generated native code for our requested targets (Android and macOS):

You can use the Function CLI to download and inspect the generated native code.

Deploying the Classifier in our App

Once compiled, using the image classifier function in our app is trivial: First, install the Function SDK. Next, create a prediction using the Function client:

// Handle camera preview images
val onCameraImage: (ImageProxy) -> Unit = remember {
    { imageProxy: ImageProxy ->
        // Create a prediction
        val prediction = fxn.predictions.create(
            "@yusuf/mobilenet-v2",
            mapOf("image" to Image.fromImage(
                imageProxy.image,
                imageProxy.imageInfo.rotationDegrees
            ))
        )
        // Display the results
        labelText = prediction.results!![0] as String
        scoreText = (prediction.results!![1] as Float).toString()
    }
}

That's it! Check out the full example here:

Exciting Times Ahead

Teams deploying AI inference into their applications—whether in their VPC, on-prem, or on-device—have had to spend countless hours and resources building and maintaining software to simply take their models from prototype to production. For our design partners, this has amounted to weeks of roadmap delays, and tens of thousands of dollars worth of engineering time.

With Function, all that effort is reduced to a diff as little as 5 lines of code: no more wrangling inference libraries, developing and maintaining support functions, debugging, or profiling performance regressions. Instead, you can now keep your entire inference pipeline in a single Python function, then use Function to compile and run it everywhere.

We're working on adding support for LLM inference, starting with Llama.cpp. Keep up with the latest by joining our community on Discord. And if you find this work interesting, we're hiring!

Compiling and Deploying AI Inference with Function