In order to get a more accurate depth map, I had to get the matrices for our cameras by calibrating both cameras using a chessboard. Since the relative setup of the cameras will always be constant, the calibration matrices can be saved and used for future parts of the project.
