Frame Extractor is an extension of the thumbnail generation feature that enables real-time frame extraction from live streams. When thumbnail.decode.all is set to true, the server decodes all frames and can push them to a WebSocket endpoint or upload them to S3 as JPEG images.

This feature enables AI and computer vision applications by providing real-time access to video frames from live streams.

For basic thumbnail generation and configuration options, see the Thumbnails documentation.

Enabling Frame Extraction

Frame extraction builds on the existing thumbnail configuration. To enable it, set the following properties in conf/red5.properties:

thumbnails.enabled=true
thumbnails.decode.all=true
thumbnails.websocket.endpoint=ws://your-server:8080
thumbnails.frequency=0

Setting thumbnail.frequency=0 extracts every frame. Adjust this value to control extraction rate. Frequency in seconds. E.g set to 7 means it will extract a frame every 7 seconds and push it to configured websocket endpoint.
After editing the conf file, save and restart red5pro for configuration to apply.

Settings Per Stream

Frame extraction can be configured per stream using connection params or query params.

Example:

ffmpeg -re -stream_loop -1 -i example.mp4 -c:v libx264 -c:a aac -f flv "rtmp://10.10.10.10/live/stream1?thumbnails.active=true&thumbnails.decodeAll=true&thumbnails.websocketEndpoint=wss://your-server.example.com&thumbnails.frequency=0"

WebSocket Frame Format

The server sends frames to the WebSocket endpoint as binary messages with the following structure:

The metadata JSON includes:

{
  "filename": "{STREAM_NAME}_{TIMESTAMP}.jpeg"
}

Example WebSocket Receiver

The following Node.js example demonstrates how to receive and save frames from the server:

const WebSocket = require('ws');
const fs = require('fs');
const path = require('path');

const outputDir = path.join(__dirname, 'received_frames');
if (!fs.existsSync(outputDir)) {
    fs.mkdirSync(outputDir, { recursive: true });
}

const wss = new WebSocket.Server({ port: 8080 });

console.log('WebSocket server started on ws://localhost:8080');
console.log(`Saving images to: ${outputDir}`);

wss.on('connection', (ws, req) => {
    const clientIp = req.socket.remoteAddress;
    console.log(`Client connected from ${clientIp}`);

    ws.on('message', (data, isBinary) => {
        if (isBinary && data.length > 4) {
            try {
                // Read metadata length (first 4 bytes, big-endian)
                const metadataLength = data.readInt32BE(0);
                
                // Extract and parse metadata JSON
                const metadataBytes = data.slice(4, 4 + metadataLength);
                const metadata = JSON.parse(metadataBytes.toString('utf8'));
                
                // Extract image binary data
                const imageData = data.slice(4 + metadataLength);
                
                // Use filename from metadata
                const filename = metadata.filename || `frame_${Date.now()}.jpeg`;
                const filepath = path.join(outputDir, filename);

                fs.writeFile(filepath, imageData, (err) => {
                    if (err) {
                        console.error('Error saving image:', err);
                    } else {
                        console.log(`Saved: ${filename} (${imageData.length} bytes)`);
                    }
                });
            } catch (e) {
                console.error('Error processing frame:', e);
            }
        }
    });

    ws.on('close', () => {
        console.log(`Client disconnected from ${clientIp}`);
    });

    ws.on('error', (error) => {
        console.error('WebSocket error:', error);
    });

    ws.send(JSON.stringify({ status: 'connected', message: 'Ready to receive frames' }));
});

S3 Upload

When the CloudStorage plugin is enabled, extracted frames can be automatically uploaded to S3. See the Thumbnails documentation for cloud storage configuration.