Efemel

Python as a functional markup language. Solves YAML scaling issues.

---

🎯 Overview

Efemel replaces markup templating with native Python. Solves configuration scaling issues without DSLs.

Why Python:

• Native syntax instead of templating DSLs
• Built-in validation and IDE support
• Full programming capabilities
• Testable configurations

Use Cases:

• CI/CD pipelines
• Infrastructure as Code
• Workflow definitions
• Docker Compose configs
• Application settings

---

✨ Features

Core

• Python-native configuration
• Multi-format export (JSON/YAML/TOML)
• Directory structure preservation
• Parallel processing
• Glob pattern support
• IDE integration

Advanced

• Environment-specific processing
• Extensible hook system
• Auto-validation
• Testable configurations
• Live reload

---

📦 Installation

To be filled out once compiled and released.

---

🚀 Usage

Basic

# Single file
efemel process config.py --out output/

# Batch processing
efemel process "**/*.py" --out exported_configs/

# Pick specific keys
efemel process "**/*.py" --out output/ --pick result

# Unwrap values
efemel process "**/*.py" --out output/ --unwrap result
### Advanced Usage

#### Environment-Specific Processing
```bash
# Production environment
efemel process "config/**/*.py" --out prod_config/ --env prod

# Development environment  
efemel process "config/**/*.py" --out dev_config/ --env dev

# Staging with custom working directory
efemel process "*.py" --cwd /app/configs --out staging/ --env staging

Advanced

Performance Tuning

# Control parallel workers
efemel process "**/*.py" --out output/ --workers 4

# Single-threaded processing
efemel process "**/*.py" --out output/ --workers 1

Hook-Based Transformations

# Use hook directory
efemel process "**/*.py" --out output/ --hooks hooks/

Parameter Injection

# Pass parameters to Python scripts during processing
efemel process "config.py" --out output/ --param app_name=myapp --param version=2.0.0 --param debug_mode=true

# Use JSON for complex parameters
efemel process "config.py" --out output/ --param 'database_config={"host":"prod-db","port":5432}'

# Multiple parameters
efemel process "**/*.py" --out output/ \
  --param app_name=myapp \
  --param version=2.0.0 \
  --param debug_mode=true \
  --param port=8080 \
  --param environment=production

# Use params file for complex configurations
efemel process "config.py" --out output/ --params-file params.py

# Combine params file with individual params (individual params override file values)
efemel process "config.py" --out output/ --params-file params.py --param app_name=override-app

Core Patterns & Examples

Pattern 1: Parameter Injection

Input (config.py):

# Parameters are injected as global variables
app_name = globals().get("app_name", "default-app")
port = globals().get("port", 3000)
debug_mode = globals().get("debug_mode", False)

Command:

efemel process config.py --out output/ --param app_name=myapp --param port=8080 --param debug_mode=true

Output:

{
  "app_name": "myapp",
  "port": 8080,
  "debug_mode": true
}

Pattern 2: Basic Data Extraction

Input (app_config.py):

app_name = "my-app"
version = "1.0.0"
port = 8080
debug_mode = True

# Private variables (underscore prefix) are ignored
_internal_secret = "hidden"

Command:

efemel process app_config.py --out output/

Output:

{
  "app_name": "my-app",
  "version": "1.0.0",
  "port": 8080,
  "debug_mode": true
}

Pattern 3: Environment-Specific File Overrides

Input Files:

config.py (default):

workers = 2
timeout = 30

config.prod.py (production override):

workers = 8
timeout = 60

main.py (imports the config):

from config import workers, timeout

server_config = {
    "workers": workers,
    "timeout": timeout
}

Default Output (efemel process main.py --out output/):

{
  "server_config": {
    "workers": 2,
    "timeout": 30
  }
}

Production Output (efemel process main.py --out output/ --env prod):

{
  "server_config": {
    "workers": 8,
    "timeout": 60
  }
}

Pattern 4: Composable Configuration Parts

Input (docker_config.py):

# Base configuration
base_service = {
    "restart": "unless-stopped",
    "networks": ["app-network"]
}

# Specific services using base
web_service = {
    **base_service,
    "image": "nginx:alpine",
    "ports": ["80:80"]
}

api_service = {
    **base_service,
    "image": "python:3.12",
    "ports": ["8080:8080"]
}

# Final composition
services = {
    "web": web_service,
    "api": api_service
}

Command:

efemel process docker_config.py --out output/ --pick services

Output:

{
  "services": {
    "web": {
      "restart": "unless-stopped",
      "networks": ["app-network"],
      "image": "nginx:alpine",
      "ports": ["80:80"]
    },
    "api": {
      "restart": "unless-stopped",
      "networks": ["app-network"],
      "image": "python:3.12",
      "ports": ["8080:8080"]
    }
  }
}

Pattern 5: Parameter Files

Params File (params.py):

app_name = "my-app"
version = "2.0.0"
debug_mode = False
port = 8080

database_config = {
    "host": "prod-db.example.com",
    "port": 5432
}

Config File (config.py):

# Use parameters from params file
app_config = {
    "name": app_name,
    "version": version,
    "debug": debug_mode,
    "port": port
}

db_config = database_config

Command:

efemel process config.py --out output/ --params-file params.py

Output:

{
  "app_config": {
    "name": "my-app",
    "version": "2.0.0",
    "debug": false,
    "port": 8080
  },
  "db_config": {
    "host": "prod-db.example.com",
    "port": 5432
  }
}

---

⚙️ Configuration

Command-Line Options

Option	Short	Type	Required	Default	Description
FILE_PATTERN	-	str	Yes	-	Glob pattern for Python files
--out	-o	str	Yes	-	Output directory path
--cwd	-c	str	No	"."	Working directory for file operations
--env	-e	str	No	None	Environment name for imports
--workers	-w	int	No	CPU_COUNT	Number of parallel workers
--hooks	-h	str	No	None	Path to hooks file or directory
--flatten	-f	flag	No	False	Flatten directory structure
--clean	-	flag	No	False	Clean (delete) the output directory before processing
--dry-run	-	flag	No	False	Show what would be processed without writing files
--pick	-p	str	No	None	Pick specific keys from the extracted data (can be used multiple times)
--unwrap	-u	str	No	None	Extract specific values from the processed data, merging them (can be used multiple times)
--param	-P	str	No	None	Pass custom parameters to processed scripts in key=value format (can be used multiple times)
--params-file	-	str	No	None	Path to a Python file that will be processed to extract parameters for other files

Hook Configuration

Create custom transformation hooks in Python:

# hooks/output_filename.py
def add_timestamp(context):
    """Add timestamp to output filenames"""
    from datetime import datetime
    output_path = context['output_file_path']
    timestamp = datetime.now().strftime('%Y%m%d')
    new_name = f"{output_path.stem}_{timestamp}{output_path.suffix}"
    context['output_file_path'] = output_path.with_name(new_name)

---

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

🚀 Built With

• Python 3.12+ - Core language
• UV Package Manager - Dependency management
• Click - CLI framework
• Ruff - Code formatting and linting
• Pytest - Testing framework
• DevContainers - Consistent development environment
• GitHub Actions - CI/CD automation

---

⭐ Star this repository if Efemel helps your workflow!