Graphics Rendering

IronRDP provides comprehensive graphics decoding support for various RDP bitmap codecs and update types. This guide covers image processing, codec support, and rendering patterns.

Graphics Pipeline Overview

Receive PDU - Graphics update arrives from server
Decode - Decompress and decode using appropriate codec
Update Image - Apply decoded pixels to framebuffer
Render - Display the updated framebuffer

Supported Codecs

IronRDP supports these graphics codecs (defined in ironrdp-graphics):

Uncompressed Bitmap - Raw pixel data
RLE (Run-Length Encoding) - Interleaved RLE compression
RDP 6.0 Bitmap Compression - Advanced RDP compression
RemoteFX (RFX) - Microsoft’s advanced codec with DWT
ZGFX - Bulk compression for graphics

Image Buffer Management

Creating a Decoded Image

use ironrdp::session::image::DecodedImage;
use ironrdp_graphics::image_processing::PixelFormat;

let width = 1920;
let height = 1080;

let mut image = DecodedImage::new(
    PixelFormat::RgbA32,  // 32-bit RGBA
    width,
    height,
);

Pixel Formats

use ironrdp_graphics::image_processing::PixelFormat;

// Available formats:
let rgba32 = PixelFormat::RgbA32;  // 32-bit RGBA (recommended)
let bgra32 = PixelFormat::BgrA32;  // 32-bit BGRA
let rgb24 = PixelFormat::Rgb24;    // 24-bit RGB
let bgr24 = PixelFormat::Bgr24;    // 24-bit BGR

Accessing Image Data

// Get raw pixel data
let pixels: &[u8] = image.data();

// Get dimensions
let width = image.width();
let height = image.height();

// Convert to image crate format
let img_buffer: image::ImageBuffer<image::Rgba<u8>, _> =
    image::ImageBuffer::from_raw(
        u32::from(width),
        u32::from(height),
        pixels.to_vec(),
    )?;

// Save to disk
img_buffer.save("screenshot.png")?;

Processing Graphics Updates

Active Stage Integration

The ActiveStage automatically handles graphics decoding:

use ironrdp::session::{ActiveStage, ActiveStageOutput};

let mut active_stage = ActiveStage::new(connection_result);
let mut image = DecodedImage::new(
    PixelFormat::RgbA32,
    connection_result.desktop_size.width,
    connection_result.desktop_size.height,
);

loop {
    let (action, payload) = framed.read_pdu()?;

    // Process PDU and update image
    let outputs = active_stage.process(&mut image, action, &payload)?;

    for output in outputs {
        match output {
            ActiveStageOutput::GraphicsUpdate => {
                // Image has been updated, trigger redraw
                render_image(&image);
            }
            ActiveStageOutput::ResponseFrame(frame) => {
                framed.write_all(&frame)?;
            }
            ActiveStageOutput::Terminate(reason) => {
                println!("Session terminated: {:?}", reason);
                break;
            }
            _ => {}
        }
    }
}

Manual Graphics Decoding

For advanced scenarios, decode graphics manually:

use ironrdp_graphics::image_processing::{PixelFormat, decode_bitmap};
use ironrdp_pdu::rdp::bitmap::BitmapData;

fn decode_bitmap_update(
    image: &mut DecodedImage,
    bitmap: &BitmapData,
) -> anyhow::Result<()> {
    let decoded = decode_bitmap(
        &bitmap.bitmap_data,
        bitmap.width,
        bitmap.height,
        bitmap.bits_per_pixel,
        PixelFormat::RgbA32,
    )?;

    // Apply to framebuffer at specified position
    image.apply_rectangle(
        bitmap.dest_left,
        bitmap.dest_top,
        bitmap.width,
        bitmap.height,
        &decoded,
    );

    Ok(())
}

RemoteFX Decoding

RemoteFX uses wavelet-based compression for high-quality graphics:

use ironrdp_graphics::rfx::{RfxDecoder, RfxContext};

let mut rfx_decoder = RfxDecoder::new();
let mut rfx_context = RfxContext::new();

// Decode RFX message
let tiles = rfx_decoder.decode(
    &rfx_pdu_data,
    &mut rfx_context,
)?;

// Apply tiles to image
for tile in tiles {
    image.apply_rectangle(
        tile.x,
        tile.y,
        tile.width,
        tile.height,
        &tile.pixels,
    );
}

Enabling RemoteFX on Windows Server

Run these PowerShell commands and reboot:

Set-ItemProperty -Path 'HKLM:\Software\Policies\Microsoft\Windows NT\Terminal Services' -Name 'ColorDepth' -Type DWORD -Value 5
Set-ItemProperty -Path 'HKLM:\Software\Policies\Microsoft\Windows NT\Terminal Services' -Name 'fEnableVirtualizedGraphics' -Type DWORD -Value 1

Or use Group Policy Editor (gpedit.msc):

Enable: Computer Configuration/Administrative Templates/Windows Components/Remote Desktop Services/Remote Desktop Session Host/Remote Session Environment/RemoteFX for Windows Server 2008 R2/Configure RemoteFX
Enable: Computer Configuration/Administrative Templates/Windows Components/Remote Desktop Services/Remote Desktop Session Host/Remote Session Environment/Enable RemoteFX encoding for RemoteFX clients designed for Windows Server 2008 R2 SP1
Reboot

Compression Types

Configuring Client Compression

use ironrdp_pdu::rdp::client_info::CompressionType;
use ironrdp::connector::Config;

let config = Config {
    // Level 0 (least compression)
    compression_type: Some(CompressionType::K8),

    // Level 1
    // compression_type: Some(CompressionType::K64),

    // Level 2
    // compression_type: Some(CompressionType::Rdp6),

    // Level 3 (best compression)
    // compression_type: Some(CompressionType::Rdp61),

    // Disabled
    // compression_type: None,

    // ... other config
};

Rendering Backends

Software Rendering (softbuffer)

IronRDP’s ironrdp-client uses softbuffer for portable software rendering:

use softbuffer::{Context, Surface};
use winit::window::Window;

// Create softbuffer context
let context = Context::new(&window)?;
let mut surface = Surface::new(&context, &window)?;

// Render image
surface.resize(
    NonZeroU32::new(width).unwrap(),
    NonZeroU32::new(height).unwrap(),
)?;

let mut buffer = surface.buffer_mut()?;

// Copy pixels from DecodedImage to buffer
for (i, pixel) in image.data().chunks_exact(4).enumerate() {
    let r = pixel[0] as u32;
    let g = pixel[1] as u32;
    let b = pixel[2] as u32;
    buffer[i] = (r << 16) | (g << 8) | b;
}

buffer.present()?;

GPU Rendering (custom)

For GPU-accelerated rendering, upload the decoded image to a texture:

// Example with wgpu (pseudocode)
fn upload_to_gpu(image: &DecodedImage, queue: &wgpu::Queue, texture: &wgpu::Texture) {
    queue.write_texture(
        wgpu::ImageCopyTexture {
            texture,
            mip_level: 0,
            origin: wgpu::Origin3d::ZERO,
            aspect: wgpu::TextureAspect::All,
        },
        image.data(),
        wgpu::ImageDataLayout {
            offset: 0,
            bytes_per_row: Some(image.width() as u32 * 4),
            rows_per_image: Some(image.height() as u32),
        },
        wgpu::Extent3d {
            width: image.width() as u32,
            height: image.height() as u32,
            depth_or_array_layers: 1,
        },
    );
}

HTML5 Canvas (WebAssembly)

For web clients, render to HTML5 Canvas:

use web_sys::{CanvasRenderingContext2d, ImageData};

fn render_to_canvas(
    image: &DecodedImage,
    ctx: &CanvasRenderingContext2d,
) -> Result<(), JsValue> {
    let width = image.width() as u32;
    let height = image.height() as u32;

    // Create ImageData from decoded pixels
    let image_data = ImageData::new_with_u8_clamped_array_and_sh(
        wasm_bindgen::Clamped(image.data()),
        width,
        height,
    )?;

    // Draw to canvas
    ctx.put_image_data(&image_data, 0.0, 0.0)?;

    Ok(())
}

Optimizing Graphics Performance

Dirty Rectangle Tracking

Only redraw changed regions:

struct DirtyTracker {
    dirty_regions: Vec<Rectangle>,
}

impl DirtyTracker {
    fn mark_dirty(&mut self, x: u16, y: u16, width: u16, height: u16) {
        self.dirty_regions.push(Rectangle { x, y, width, height });
    }

    fn render_dirty(&mut self, image: &DecodedImage, surface: &mut Surface) {
        for rect in &self.dirty_regions {
            self.render_region(image, surface, rect);
        }
        self.dirty_regions.clear();
    }
}

Frame Rate Limiting

use std::time::{Duration, Instant};

let mut last_frame = Instant::now();
let frame_time = Duration::from_millis(16); // ~60 FPS

loop {
    let (action, payload) = framed.read_pdu()?;
    let outputs = active_stage.process(&mut image, action, &payload)?;

    for output in outputs {
        if matches!(output, ActiveStageOutput::GraphicsUpdate) {
            let now = Instant::now();
            if now.duration_since(last_frame) >= frame_time {
                render_image(&image);
                last_frame = now;
            }
        }
    }
}

Double Buffering

struct DoubleBuffer {
    front: DecodedImage,
    back: DecodedImage,
}

impl DoubleBuffer {
    fn swap(&mut self) {
        std::mem::swap(&mut self.front, &mut self.back);
    }

    fn update(&mut self) {
        // Decode into back buffer
        // ...

        // Swap buffers when ready
        self.swap();
    }

    fn render(&self) {
        // Render from front buffer while back buffer is being updated
        render_image(&self.front);
    }
}

Color Space Conversion

use ironrdp_graphics::color_conversion::{rgb_to_bgr, bgr_to_rgb};

// Convert RGB to BGR
let mut pixels = vec![255, 0, 0, 255]; // Red in RGBA
rgb_to_bgr(&mut pixels);
// Now pixels = [0, 0, 255, 255] // Red in BGRA

// Convert BGR to RGB
bgr_to_rgb(&mut pixels);

Server-side Graphics

When building an RDP server, generate graphics updates:

use ironrdp::server::{BitmapUpdate, PixelFormat};
use core::num::{NonZeroU16, NonZeroUsize};

fn create_bitmap_update() -> BitmapUpdate {
    let width = NonZeroU16::new(100).unwrap();
    let height = NonZeroU16::new(100).unwrap();

    // Generate RGBA pixels
    let mut pixels = Vec::new();
    for y in 0..100 {
        for x in 0..100 {
            pixels.push(255); // B
            pixels.push(0);   // G
            pixels.push(0);   // R
            pixels.push(255); // A
        }
    }

    BitmapUpdate {
        x: 0,
        y: 0,
        width,
        height,
        format: PixelFormat::BgrA32,
        data: pixels.into(),
        stride: NonZeroUsize::new(100 * 4).unwrap(),
    }
}

Troubleshooting

Graphics Not Updating

Check ActiveStageOutput::GraphicsUpdate is being handled
Verify image buffer is being rendered after updates
Enable logging: IRONRDP_LOG=trace

Color Corruption

Verify pixel format matches between decoder and renderer
Check byte order (RGB vs BGR)
Ensure stride is calculated correctly

Performance Issues

Use appropriate compression level
Implement dirty rectangle tracking
Limit frame rate to display refresh rate
Consider GPU acceleration for large resolutions

Next Steps

Building a Client - Integrate graphics rendering
Building a Server - Generate graphics updates
WebAssembly - Render in web browsers

Documentation Index

​Graphics Pipeline Overview

​Supported Codecs

​Image Buffer Management

​Creating a Decoded Image

​Pixel Formats

​Accessing Image Data

​Processing Graphics Updates

​Active Stage Integration

​Manual Graphics Decoding

​RemoteFX Decoding

​Enabling RemoteFX on Windows Server

​Compression Types

​Configuring Client Compression

​Rendering Backends

​Software Rendering (softbuffer)

​GPU Rendering (custom)

​HTML5 Canvas (WebAssembly)

​Optimizing Graphics Performance

​Dirty Rectangle Tracking

​Frame Rate Limiting

​Double Buffering

​Color Space Conversion

​Server-side Graphics

​Troubleshooting

​Graphics Not Updating

​Color Corruption

​Performance Issues

​Next Steps