In our experience with a lot of users, we see that there are couple of places where things can go wrong in evaluation. We published a guide that outlines how to use the tools we built to do this evaluation: Hailo guide: Evaluating model accuracy after compilation - Guides / Hailo Guides - DeGirum
Regarding quantizing on your local system: yes, it is possible and you do not need a hailo device to compile.