# Virtual Gimbal

**Repository Path**: LJN2017117574/virtual_gimbal

## Basic Information

- **Project Name**: Virtual Gimbal
- **Description**: No description available
- **Primary Language**: Unknown
- **License**: BSD-3-Clause
- **Default Branch**: master
- **Homepage**: None
- **GVP Project**: No

## Statistics

- **Stars**: 0
- **Forks**: 0
- **Created**: 2026-07-26
- **Last Updated**: 2026-07-26

## Categories & Tags

**Categories**: Uncategorized

**Tags**: None

## README

# virtualgimbal_ros
VirtualGimbal ROS stabilizes video by correcting shake based on angular velocity measured by an Inertial Measurement Unit (IMU).
VirtualGimbal ROS was developed to improve visibility during robot operation through camera images, enhance image recognition and object tracking performance, and improve video recording quality.
VirtualGimbal ROS operates at high speed using OpenCL GPU processing. For example, we have confirmed that it can stabilize 1920x1080 pixel, 30 fps video from Intel RealSense D435i in real-time on a laptop (Core i7-8550U).

# 1. Overview
This package consists of three types of nodes:

1. `virtualgimbal_ros_node` - Stabilizes video using angular velocity measured by a gyro sensor
1. `synchronizer_node` - Precisely measures the timestamp offset between video and gyro sensor
1. `line_delay_estimator_node` - Precisely measures different readout delay times for each row of a rolling shutter CMOS sensor

[![](https://github.com/yossato/images/blob/master/youtube.png?raw=true)](https://www.youtube.com/watch?v=ft6v7h5kN6g&feature=youtu.be)

# 1.1 Install dependencies
This package has been tested on Ubuntu 20.04 and ROS Noetic.
The following OpenCL-related libraries are required:

```
# apt install ocl-icd-libopencl1 opencl-headers clinfo ocl-icd-opencl-dev
```
Additionally, you need to install the appropriate driver depending on your GPU type.

### 1.1.1 NVIDIA GPU
If you are using an NVIDIA GPU, please install the [display driver](https://www.nvidia.com/download/index.aspx).

### 1.1.2 Intel HD Graphics
If you are using Intel's integrated GPU, install [Intel(R) Graphics Compute Runtime for OpenCL(TM)](https://github.com/intel/compute-runtime/blob/master/documentation/Neo_in_distributions.md), the OpenCL driver, with the following commands:
```
# add-apt-repository ppa:intel-opencl/intel-opencl
# apt update
# apt install intel-opencl-icd
```

### 1.1.3 AMD
T.B.D.

# 2. Tutorials
## 2.1 Stabilizer Tutorial
Let's experience the stabilization operation with a sample video. First, download the [rosbag of video recorded with RealSense D435i](https://www.dropbox.com/s/43ucvmjfjhxeyzg/soleil.bag?dl=1).
Run the following commands in separate terminals:
```
$ roslaunch virtualgimbal_ros stabilize_realsense_rgb.launch
$ rosbag play soleil.bag --clock  # Run in the directory where the rosbag is placed
```
Two rqt image_view windows will launch.
Input Image is the video before stabilization, and Stabilized Image is the video after stabilization. After stabilization, the image is zoomed in with edges cropped, but the movement within the image is reduced and motion is stabilized.

![Input Image ](https://github.com/yossato/images/blob/master/Screenshot%20from%202020-02-29%2023-10-14.png?raw=true)
![Stabilized Image ](https://github.com/yossato/images/blob/master/Screenshot%20from%202020-02-29%2023-10-10.png?raw=true)

## 2.2 Line Delay Estimation Tutorial
Let's experience the line delay estimation of a rolling shutter CMOS camera with a sample video. The line delay value is required for high-precision video stabilization. Download the [rosbag for calibration recorded with RealSense D435i](https://www.dropbox.com/s/e4onsul90wjbsad/aruco_board_d435i.bag?dl=1).
Run the following commands in separate terminals:
```
$ roslaunch virtualgimbal_ros estimate_line_delay_d435i.launch
$ rosbag play aruco_board_d435i.bag --clock -s 10 # Run in the directory where the rosbag is placed
```
The line delay estimation screen will launch.
![Line delay estimation](https://github.com/yossato/images/blob/master/line_delay_estimation.png?raw=true)
From the rotating ArUco board, markers are detected individually, and the change in angle is calculated. Finally, for example, when `Inlier:7999 / 10198 Line_delay:0.00003025 [second]` is displayed, the line delay estimation is complete. The value varies slightly each time. This line delay value should be set to the `line_delay` parameter of `virtualgimbal_ros_node` described later.

# 3. Nodes
## 3.1 virtualgimbal_ros_node
`virtualgimbal_ros_node` generates stabilized video from camera-recorded video and IMU-measured angular velocity.
![nodes](https://github.com/yossato/images/blob/master/nodes.png?raw=true)

### 3.1.1 Subscribe Topics
#### image_raw (sensor_msgs/Image)
Rectified image stream from the camera driver.

#### camera_info (sensor_msgs/CameraInfo)
Camera metadata.

#### imu_data (sensor_msgs/Imu)
Angular velocity.

## 3.1.2 Publish Topics
#### stabilized/image_rect (sensor_msgs/Image)
Stabilized image stream.

#### stabilized/camera_info (sensor_msgs/CameraInfo)
Stabilized camera metadata.

## 3.1.3 Parameters
|Parameter|Type|Default|Description|
|:---|:---|:---|:---|
|image|string|image|Input image topic|
|imu_data|string|imu_data|Input angular velocity topic|
|zoom_factor|float|1.3|Image zoom factor. Set a value of 1 or higher. Larger values zoom in the image and improve shake correction capability, but increase the amount of cropping at the four corners of the image.|
|enable_trimming|bool|true|When set to true, the output image size is reduced by the amount cropped from the four corners. When set to false, the input and output image sizes are equal. The output image is stretched. Setting to true reduces topic data volume, but the image size becomes non-standard.|
|offset_time|double|0|Offset time (seconds) between gyro sensor and image stream timestamps. Can be measured with `synchronizer_node`. Values vary by camera.|
|line_delay|double|0.0|Line delay (seconds) of a rolling shutter camera. For a rolling shutter image sensor where exposure timing differs for each row, this represents the delay time of exposure timing per row. This value is estimated by `line_delay_estimator_node`. Set to 0 for global shutter image sensors.|
|verbose|bool|false|Provides debug functionality. When true, various logs are enabled and stabilization status is displayed as graphs. When false, logging is disabled.|
|allow_blue_space|bool|false|Setting to true allows blue areas to appear in the image, significantly improving shake correction capability. However, when camera shake becomes too large and reaches its limit, blue areas will appear at the edges of the screen. Setting to false reduces shake correction capability but prevents blue areas from appearing at screen edges.|
|lsm_period|double|1.5|VirtualGimbal's video stabilization algorithm calculates image shift amount from the difference between time-series smoothed camera angle and unsmoothed raw camera angle. For smoothing the time-series camera angle, VirtualGimbal ROS uses the least squares method. The least squares method allows selection of any order and fitting time range. This `lsm_period` is the time range (seconds) for calculating the least squares method. Larger values calculate smoothed camera angle from longer-term camera movement, improving stabilization capability but losing ability to follow sudden camera movements. Smaller values calculate smoothed camera angle from shorter-term camera movement, reducing stabilization capability but enabling following of sudden camera movements.|
|lsm_order|double|1|Order of the curve for least squares fitting. 1 fits with a first-order linear equation. 2 fits with a second-order parabolic equation. Higher orders improve tracking but reduce stabilization capability. First-order fitting provides high stability but low tracking.|

## 3.1.4 Launch files
- stabilize.launch
  - Stabilizes an image stream.
- stabilize_realsense_rgb.launch
  - Stabilizes the RGB image stream of Intel RealSense D435i using IMU angular velocity. Launches `stabilize.launch` with D435i topic names as parameters. Assumes an image size of 1920x1080 pixels.
- stabilize_realsense_ir.launch
  - Stabilizes the left IR image stream of Intel RealSense D435i using IMU angular velocity. Launches `stabilize.launch` with D435i topic names as parameters.

## 3.2 synchronizer_node
`synchronizer_node` is a node that precisely measures the timestamp offset between video and IMU. Generally, there is a slight time difference (offset) between camera video and IMU angular velocity timestamps. Since video stabilization requires synchronization accuracy of less than a millisecond, failing to correctly set this offset will degrade video stabilization quality.

VirtualGimbal ROS uses the exposure center of the camera shutter as the timestamp reference. `synchronizer_node` calculates the correlation between video and IMU angular velocity using Sum of Absolute Differences (SAD) and determines the offset value with the best correlation. More specifically, this node performs the following processing: 1. Calculates optical flow from video, 2. Estimates angular velocity from optical flow, 3. Calculates the correlation between estimated angular velocity and IMU-measured angular velocity while gradually changing the offset. The offset with the best correlation has the smallest SAD value, allowing offset estimation. The obtained offset should be set to the `offset_time` parameter in `stabilize.launch`.

### 3.2.1 Subscribe Topics
#### image_raw (sensor_msgs/Image)
Rectified image stream from the camera driver.

#### camera_info (sensor_msgs/CameraInfo)
Camera metadata.

#### imu_data (sensor_msgs/Imu)
Angular velocity.

### 3.2.2 Parameters
|Parameter|Type|Default|Description|
|:---|:---|:---|:---|
|image|string|image|Input image topic|
|imu_data|string|imu_data|Input angular velocity topic|
|maximum_offset_time|float|0.5|Maximum offset value (seconds) for SAD calculation. Specify the maximum timestamp difference between video and IMU. Larger values allow offset estimation even with large timestamp errors, but computation takes longer.|
|correlation_time|float|15.0|Duration (seconds) for SAD calculation. Longer durations improve offset estimation accuracy but require longer video for estimation and take longer to compute.|

### 3.2.3 Launch files
- synchronizer.launch
  - Estimates the offset time required for time synchronization between image stream and IMU.
- synchronize_realsense_rgb.launch
  - Synchronizes Intel RealSense D435i RGB image stream with IMU.
- synchronize_realsense_ir.launch
  - Similarly synchronizes IR image stream with IMU.

## 3.3 line_delay_estimator_node
`line_delay_estimator_node` estimates the line delay of a general rolling shutter CMOS image sensor. A rolling shutter CMOS image sensor reads data line by line during capture, resulting in different capture timing for each row of the obtained image. This node can estimate the capture timing difference for each row. This node displays an ArUco marker board on the screen. To estimate line delay, rotate the camera around its optical axis while capturing this ArUco marker board. When imaging this rotating board, the capture timing differs for each row, causing slight angle variations of markers between the top and bottom of the captured image. The line delay per row can be estimated from these angle changes. The unit of line delay is seconds.

When the camera rotation speed is sufficiently fast and line delay is being measured, yellow phase and green fitted line are displayed as shown below. When the camera rotation speed is slow, grayed-out phase is displayed.
![Line delay estimation](https://github.com/yossato/images/blob/master/line_delay_estimation.png?raw=true)
After execution, when sufficient data has been obtained, the final result is displayed in the terminal. This node terminates automatically.

### 3.3.1 Subscribe Topics
#### image (sensor_msgs/Image)
Rectified image stream from the camera driver.

#### camera_info (sensor_msgs/CameraInfo)
Camera metadata.

### 3.3.2 Parameters
|Parameter|Type|Default|Description|
|:---|:---|:---|:---|
|image|string|image|Input image topic|
|minimum_angle_thresh|float|0.05|Threshold for minimum angle difference [radian] between frames accepted as estimation data when rotating the camera. Larger values require faster camera rotation.|
|maximum_relative_delay_ransac|float|0.01|For the RANSAC algorithm that removes outlier effects during line delay estimation, the difference between the predicted value of the hypothetical model considered as inlier and the sample. Set in the range (0,1]. Larger values make it more susceptible to outliers.|
|maximum_iteration_ransac|float|10000|Maximum number of RANSAC iterations. More iterations take longer but produce more refined results.|
|minimum_number_of_data_ransac|float|10000|Number of data points input to RANSAC. Larger values produce more stable estimation results but take longer to collect data.|
|generate_aruco_board|bool|false|When set to true, generates a PNG format ArUco board for printing. Set to true and regenerate the PNG image when changing ArUco board parameters written in `marker_params.yaml` described later.|
|show_gui|bool|true|When set to false, does not display the GUI window for checking calibration status.|

Parameters for line delay estimation are recorded in `params/line_delay_estimation_params.yaml`.
This node uses OpenCV's ArUco markers, so there are many additional detection parameters stored in `params/detector_params.yaml` and `params/marker_params.yaml`. For details, refer to [Detection of ArUco Markers](https://docs.opencv.org/4.5.2/d5/dae/tutorial_aruco_detection.html). These parameters are loaded by the `rosparam` command when launching with ROS launch.

### 3.3.3 Launch files
- estimate_line_delay.launch
  - Estimates CMOS image sensor line delay from image stream.
- estimate_line_delay_d435i.launch
  - Estimates CMOS image sensor line delay from RealSense D435i image stream.