Candle - Minimalist ML framework for Rust - Step-by-Step Tutorial - Run Models Locally

 This video installs Candle locally and run local models. Candle is a minimalist ML framework for Rust with a focus on performance (including GPU support) and ease of use. 

Code:

-- Make sure nvcc shows CUDA version 12.3 and nvidia-smi shows 8.9 for compute capacity.

nvcc --version should print information about your Cuda compiler driver.

nvidia-smi --query-gpu=compute_cap --format=csv

Make sure that that openssl and cc1plus are on the following path by finding them :  sudo find / -name cc1plus

-- export PATH=$PATH:/usr/lib/gcc/x86_64-linux-gnu/11/   //This is for cc1plus  (cc1plus is the C++ compiler driver )

-- export OPENSSL_LIB_DIR=/usr/lib/x86_64-linux-gnu      //This is for openssl

-- export PATH=/usr/local/cuda/bin${PATH:+:${PATH}}

-- export LD_LIBRARY_PATH=/usr/local/cuda-12.2/lib64\${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}

-- If you cannot find openssl, then run following :

sudo find / -name cc1plus

sudo apt install libssl-dev

sudo apt install pkg-config

-- Create a new app and add candle-core with Cuda support.

cargo new myapp

cd myapp

cargo add --git https://github.com/huggingface/candle.git candle-core --features "cuda"

cargo build

-- Run Inference with Phi-2 model

git clone https://github.com/huggingface/candle.git

cd candle/candle-examples

cargo run --example phi --features cuda --release -- --model 2 --prompt "What is happiness in 3 lines?"