feature-extraction-opencl/README.md

# Feature Extraction & Sparse Image Alignment with OpenCL

A GPU-accelerated Visual Inertial Odometry (VIO) system implementing sparse image alignment and feature detection algorithms using OpenCL.

This project is based on the [SVO (Semi-direct Visual Odometry)](http://rpg.ifi.uzh.ch/svo/) framework from the Robotics and Perception Group at the University of Zurich, adapted for GPU acceleration using OpenCL.

## Features

- **GPU-Accelerated Image Processing**: Utilizes OpenCL for parallel computation of image features and alignment
- **FAST Corner Detection**: GPU-optimized FAST (Features from Accelerated Segment Test) algorithm for corner detection
- **FREAK Descriptors**: GPU implementation of FREAK (Fast Retina Keypoint) feature descriptors
- **Sparse Image Alignment**: Photometric error minimization using Gauss-Newton optimization
- **Pose Optimization**: SE(2) transformation estimation for camera pose refinement
- **Multi-level Pyramid Processing**: Coarse-to-fine image pyramid for robust alignment

## Requirements

### System Dependencies
- CMake >= 3.7
- C++14 compiler (g++, clang, or MSVC)
- OpenCL headers and libraries
- Intel GPU drivers (for Intel GPU support) or other OpenCL-compatible GPU drivers

### Libraries
- **OpenCV** (>= 3.0): Image processing and manipulation
- **Eigen** (>= 3.2): Linear algebra computations
- **Sophus** (>= 1.0): Lie group operations for SE(2) transformations

## Installation

### Prerequisites (Ubuntu/Debian)

```bash
# Install basic build tools
sudo apt-get update
sudo apt-get install -y cmake build-essential git

# Install OpenCL headers and runtime
sudo apt-get install -y opencl-headers ocl-icd-opencl-dev

# Install OpenCV
sudo apt-get install -y libopencv-dev

# Install Eigen
sudo apt-get install -y libeigen3-dev

# Install Sophus
sudo apt-get install -y libsophus-dev
# Or build from source: https://github.com/strasdat/Sophus
```

### For Intel GPUs (Optional)

```bash
# Add Intel graphics repository
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys F42ED6FBAB17C654
sudo apt-get install -y software-properties-common
sudo apt-add-repository 'deb [arch=amd64] https://repositories.intel.com/graphics/ubuntu bionic main'
sudo apt-get update

# Install Intel GPU drivers and OpenCL support
sudo apt-get install -y intel-level-zero-gpu level-zero intel-opencl intel-igc-opencl-devel level-zero-devel
```

### Build

```bash
mkdir build
cd build
cmake ..
make
```

The executable `Vio_OpenCL` will be created in the build directory.

## Usage

### Basic Execution

```bash
./Vio_OpenCL
```

The program will:
1. Initialize OpenCL context and compile kernels
2. Load test data (images, poses, features)
3. Execute sparse image alignment iterations
4. Output residual and pose estimates

### Configuration

Key parameters can be modified in the source code:

- **Pyramid Levels**: `max_level`, `min_level` - control coarse-to-fine processing
- **Iterations**: `n_iter` - number of optimization iterations per level
- **Optimization Method**: `GaussNewton`, `LevenbergMarquardt` - solver selection
- **Patch Size**: `patch_halfsize_` - correlation patch size (currently 12×12)
- **Feature Count**: `FAST_COUNT` - maximum number of detected corners

### Input Data Format

The program expects:
- **Images**: Grayscale images with known intrinsic parameters
- **Features**: Corner positions and 3D world coordinates
- **Poses**: Initial camera pose estimates (SE2 transformation)

Test data location: `test/data/`

## Project Structure

```
.
├── CMakeLists.txt                 # CMake build configuration
├── main2.cpp                      # Entry point
├── include/
│   ├── cl_class.h                # OpenCL wrapper classes
│   ├── sparse_img_align_gpu.h    # Image alignment algorithm header
│   ├── math_utils.h              # Mathematical utilities
│   └── robust_cost.h             # Robust cost functions
├── src/
│   ├── cl_class.cpp              # OpenCL context and kernel management
│   ├── sparse_img_align_gpu.cpp  # Sparse alignment implementation
│   ├── math_utils.cpp            # Math utility implementations
│   ├── robust_cost.cpp           # Robust cost function implementations
│   ├── FastGrayStep.cc           # FAST detector preprocessing
│   └── fastcl.cc                 # FAST detector wrapper
├── kernel/                        # OpenCL kernels
│   ├── compute-residual.cl       # Photometric residual computation
│   ├── fast-gray.cl              # FAST corner detection
│   ├── fast-best.cl              # FAST non-maximum suppression
│   ├── freak.cl                  # FREAK descriptor computation
│   ├── integral_image.cl         # Integral image computation
│   ├── is_corner_10_*.cl         # FAST corner test variants
│   └── prefast-gray.cl           # Preprocessing for FAST
└── test/
    ├── data/                      # Test images and feature data
    └── res/                       # Result output directory
```

## Algorithm Overview

### Sparse Image Alignment

The algorithm optimizes camera pose by minimizing the photometric error:

```
minimize: Σ ρ(I(x+∇x) - I'(x))²
```

Where:
- `I` and `I'` are reference and current frame intensities
- `∇x` is the image warp from pose change
- `ρ()` is a robust weighting function

Optimization uses:
- **Gauss-Newton** or **Levenberg-Marquardt** solvers
- **Coarse-to-fine** multi-level pyramid for robustness
- **Huber/Tukey** robust cost functions for outlier rejection

### Feature Detection

- **FAST** algorithm for rapid corner detection with automatic scale adaptation
- **FREAK** descriptors for compact feature representation
- GPU parallel processing across multiple image regions

## Performance Notes

- Kernel compilation happens at runtime; first execution will be slower
- Memory transfers between CPU and GPU are optimized using pinned memory
- Floating-point precision: Single precision (32-bit) for GPU kernels

## Limitations & Known Issues

- Currently supports single-level (SE2) 2D transformation
- Limited to single-threaded execution on CPU side
- Requires compatible OpenCL-capable GPU for optimal performance
- Memory access patterns require careful alignment (see error handling in code)

## References

- **SVO**: Forster, C., Pizzoli, M., & Scaramuzza, D. (2014). "SVO: Semi-direct visual odometry for monocular and stereo cameras." ICRA.
- **FAST**: Rosten, E., & Drummond, T. (2006). "Machine learning for high-speed corner detection." ECCV.
- **FREAK**: Alahi, A., Ortiz, R., & Vandergheynst, P. (2012). "FREAK: Fast retina keypoint." CVPR.
- **OpenCL**: https://www.khronos.org/opencl/

## License

This project is licensed under the GNU General Public License v3.0 (GPLv3), consistent with the original SVO framework.

See `LICENSE` file for details.

## Contributing

Bug reports and improvements are welcome. Please ensure:
- Code follows C++14 standard
- OpenCL kernels are optimized for parallel execution
- GPU memory constraints are considered
- Changes include corresponding kernel updates if needed

## Authors

Original SVO Framework: Christian Forster, Robotics and Perception Group, University of Zurich

GPU Acceleration: Feature-extraction team (gfuse-show-cases project)

## Support

For issues or questions:
- Check test data compatibility
- Verify OpenCL drivers and hardware support
- Review kernel compilation logs for runtime errors
- Ensure sufficient GPU memory (minimum 1GB recommended)