Yolov8 Porting to DeGirum Model Zoo and Evaluation

Documentation User Guides
1

PyTorch YOLOv8 Checkpoint Porting Guide to DeGirum Model Zoo

Introduction

This guide is designed to assist you in porting your PyTorch YOLOv8 checkpoint to various models in the DeGirum Model Zoo. It is prerequisite that you have a PyTorch checkpoint (.pt file) ready for this process.

Step 1: Evaluate .pt File

  • Objective: Evaluate the .pt file using an evaluation dataset along with an annotation JSON file. This step aims to obtain COCO format accuracies within the ultralytics_yolov8 repository.
  • Command:
    python dg_val.py --weights path_to_pt_file --data path_to_data.yaml --annotations path_to_ground_truth.json

Step 2: Quantize to TFLite

  • Objective: Convert the .pt file to a quantized TFLite format using the dg_export command in the ultralytics_yolov8 repository.

  • Command:

    python dg_export.py --weights path_to_pt_file --data path_to_data.yaml --quantize --format tflite

  • Storage Detail: After executing the command, the fully quantized model will be stored at the following location:

    • {pt_file_name}_saved_model/{pt_file_name}_full_integer_quant.tflite

  • Versatility of Command: It’s important to note that the same dg_export command can be utilized for exporting to various formats. This includes not only TFLite but also ONNX and OpenVINO formats, providing flexibility in how you choose to deploy your model.

Step 2.5: Quantize to OpenVINO

  • Objective: Perform the same quantization process as in Step 2, but this time for converting the model to the OpenVINO format.
  • Command:
    python dg_export.py --weights path_to_pt_file --data path_to_data.yaml --quantize --format openvino

Step 3: Evaluate Quantized TFLite File

  • Objective: Evaluate the performance of the quantized TFLite file. This step is crucial for verifying the accuracy of the model post-quantization.
  • Command:
    python dg_val.py --weights path_to_tflite_file --data path_to_data.yaml --annotations path_to_ground_truth.json

Step 3.5: Evaluate Quantized OpenVINO File

  • Objective: Conduct an evaluation of the quantized OpenVINO file. This step is essential to assess the model’s accuracy after it has been converted to the OpenVINO format.
  • Command:
    python dg_val.py --weights path_to_openvino_folder --data path_to_data.yaml --annotations path_to_ground_truth.json

Step 4:Compiling for PySDK

  • Objective: With the PyTorch floating point accuracy, along with the quantized TFLite and OpenVINO accuracies now gathered, the next step is to compile the PyTorch file to all runtime formats acceptable by PySDK.
  • Process:
    • Utilize the hosted compiler to convert the PyTorch model into formats compatible with PySDK. This step is crucial for ensuring that your model is ready for deployment using PySDK.
  • Hosted Compiler Link: DeGirum Hosted Compiler

This phase is integral for preparing the model for its final deployment, ensuring that it is compatible with various runtime environments supported by PySDK.

Note on Evaluation Images

  • Calibration images for Quantization: For the purpose of quantizing your model, you have three options:
    1. Use a maximum of 100 images from your existing validation dataset.
    2. Alternatively, you can utilize a set of 100 images from the following repository:
      Franklin DeGirum’s Calibration Images Repository
    3. If no images are uploaded, 30 default images are used for quantization.

Step 5: Evaluation in PySDK

  • Objective: After uploading your model to the DeGirum model zoo, it’s essential to evaluate it within the PySDK environment.

  • Method: Utilize the benchmark branch in the PySDKExamples repository for this evaluation.

  • Command for Model Evaluation:

    python benchmark_scripts/detection_eval_zoo.py --annotations path_to_ground_truth.json --data path_to_validation_folder --model {Model Name Prefix} --cfg benchmark_scripts/eval_yaml/default.yaml

Optionally, you can use a specific configuration file instead of the default provided.

Important Note: Setting Environment Variables

  • Before proceeding with the evaluation in PySDK, it is crucial to configure certain environment variables in your env.ini file. Ensure the following settings are in place:

    • DEGIRUM_CLOUD_TOKEN: Set this variable with your DeGirum cloud token.
    • CLOUD_ZOO_URL: Specify the URL for the DeGirum hosted zoo.

Accuracy Matching Between PyTorch and PySDK

  • Objective: It’s crucial to ensure that the accuracies between PyTorch and PySDK models match. Pay special attention to the following comparisons:

    • Orca and Edge-TPU Accuracy: These should match the accuracy of the PyTorch quantized TFLite model.
    • OpenVINO Float Accuracy: This should be equivalent to the accuracy of the original PyTorch .pt file.
    • OpenVINO Quant Accuracy in PySDK: This must align with the accuracy of the quantized OpenVINO file in PyTorch.

  • Error Identification: If there is a discrepancy in any of these accuracy comparisons, investigate to determine the source of the error. Such mismatches might indicate issues in the model conversion or quantization process.

Important Note on Quantized Bounding Box Accuracy

  • Accuracy Loss Threshold: In my experiments, the quantized bounding box accuracy loss should be less than 1%. If the loss exceeds this threshold, it may indicate a potential error in the quantization process.

Ensuring these accuracies align is critical for validating the integrity and performance of your model across different platforms and formats.

2

@khatami.mehrdad Thanks for this detailed guide. Can you please add onnx and/or openvino float evaluation steps (maybe Step2). Also, can you clarify if dg_val.py performs evaluation at some specific size and if the expectation is that the results to match with onnx float/openvino float? If dg_export can take an img_size as input, it would be better to specify in the guide.