OG
Om Ghante
•FOCUSCurrently working on ScheduleSomething and LayerdEvents algorithmsLearn More
•FOCUSCurrently working on ScheduleSomething and LayerdEvents algorithmsLearn More
•FOCUSCurrently working on ScheduleSomething and LayerdEvents algorithmsLearn More
•FOCUSCurrently working on ScheduleSomething and LayerdEvents algorithmsLearn More
Back to Projects
AI & MLProduction

Paper Ecosystem

Multi-repo Agentic AI & LLM Inference platform featuring a central gRPC token-streaming inference gateway (paper-core), real-time voice agent runtime (paper-caller), multi-agent tools, and MCP integrations.

⚡ Impact Outcome: Abstracts multi-provider LLM inference (MiniMax, OpenAI, Ollama) behind a high-performance token-streaming gRPC gateway.
View GitHub Repository Live Demo Read GitHub Docs

Tech Stack

Python 3.12gRPCProtobufFastAPIMiniMax / OpenAI / OllamaDockerPrometheus

Architecture Concepts

gRPC Inference GatewayToken StreamingMulti-Agent RuntimeModel Context Protocol (MCP)

Live GitHub Repository README

Live Synced

Paper Core (
paper-core
)

Central LLM Inference Gateway and gRPC Service for the Paper AI Ecosystem.
paper-core
is the core inference gateway in the Paper multi-repo ecosystem. It abstracts underlying LLM provider APIs (OpenAI, MiniMax, Ollama, Anthropic, Groq) behind a high-performance gRPC interface, managing request routing, token streaming, rate limiting, and circuit breaking centrally for all callers.

Role in the Ecosystem

Caller Services (paper-caller / paper-mcp / paper-cli)
                         |
                         | gRPC Generate() or StreamGenerate()
                         v
                    paper-core (Port 50051)
                         |
           +-------------+-------------+
           |             |             |
        MiniMax       OpenAI        Ollama
  • What it owns: LLM provider adapters, gRPC server lifecycle (
    AIService
    , 
    AIStreamService
    ), rate limiting per API key, circuit breaker state tracking, and request scheduling.
  • What it does NOT do: 
    paper-core
    does not handle audio processing (owned by 
    paper-caller
    ), session memory (owned by 
    paper-memory
    ), or tool execution (owned by 
    paper-tools
    ).

Key Features

  • Provider Abstraction: Unified interface for cloud and local language models (MiniMax, OpenAI, Anthropic, Groq, Ollama).
  • Token-by-Token gRPC Streaming: Server-streaming gRPC interface for sub-second first-token delivery.
  • Resilience Controls: Built-in circuit breakers and exponential backoff retries with jitter via 
    paper-common
    .
  • Token Bucket Rate Limiting: Configurable rate limiting per API key or tenant ID.
  • Prometheus Telemetry: Native metrics export on port 9090 for inference latency, token counts, and error rates.

Repository Structure

paper-core/
├── app/
│   ├── main.py                  # Service entry point and signal handling
│   ├── config.py                # Pydantic environment configuration
│   ├── engine/                  # AIEngine orchestrator
│   ├── grpc_server/             # gRPC servicers, interceptors, and channels
│   ├── providers/               # LLM provider adapters (MiniMax, OpenAI, etc.)
│   ├── scheduler/               # Priority request queue and scheduler
│   └── throttling/              # Token bucket rate limiter
├── tests/                       # Unit and integration test suite
├── Dockerfile                   # Production container definition
├── pyproject.toml               # Python dependencies and build config
└── README.md

Quickstart & Installation

Prerequisites

  • Python 3.12+
  • Docker & Docker Compose (optional for containerized setup)

Local Setup

bash
# Clone repository
git clone https://github.com/artificialpaper/paper-core.git
cd paper-core

# Create virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install dependencies
pip install -e .

# Copy environment template
cp .env.example .env

Running the gRPC Server

bash
# Start gRPC inference server on port 50051
python -m app.main

Configuration Reference

paper-core
is configured via environment variables or a 
.env
file:
VariableTypeDefaultDescription
PAPER_PORT
int
50051
gRPC server listening port
PAPER_METRICS_PORT
int
9090
Prometheus metrics HTTP port
LLM_PROVIDER
string
minimax
Active provider key (
minimax
, 
openai
, 
ollama
)
MINIMAX_API_KEY
string
""
MiniMax API secret key
OPENAI_API_KEY
string
""
OpenAI API secret key
OLLAMA_HOST
string
http://localhost:11434
Local Ollama server address
RATE_LIMIT_TOKENS_PER_SEC
int
100
Token bucket refill rate

gRPC API Reference

paper-core
implements contracts defined in 
paper-proto
:

Unary Generation (
AIService.Generate
)

protobuf
rpc Generate(GenerateRequest) returns (GenerateResponse);

Server Streaming (
AIStreamService.StreamGenerate
)

protobuf
rpc StreamGenerate(GenerateRequest) returns (stream StreamChunk);

Testing & Quality

bash
# Run pytest test suite
pytest

# Run linter and type checker
ruff check .
mypy app

Related Repositories & Documentation

License

This project is licensed under the MIT License - see the LICENSE file for details.