In order to get a more accurate depth map, I had to get the matrices for our cameras by calibrating both cameras using a chessboard. Since the relative setup of the cameras will always be constant, the calibration matrices can be saved and used for future parts of the project.
The python script uses the frames from both the left and right cameras to identify the checkerboard squares. When using the script make sure to configure the main parameters in the beginning of the script such as the square size, number of columns and both camera indices. Also make sure to plug the USB cameras directly into your device and not through a USB hub as that might cause the frame to not be in sync between both cameras.
Calibration with SGBM
After getting a sufficient amount of checkerboard recognitions, I found that capturing distant checkerboards at all locations within the frame really helps give a good calibration, the matrices will be printed out as well as saved in the same directory as the file.
There might still be some room for improving the result such as having a higher resolution which our project can’t afford since we will be running this on the Raspberry Pi’s weaker hardware and StereoSGBM is more computationally intensive than StereoBM. In addition, we can also tune StereoSGBM parameters which might help us.