Correctness checking for security_barrier_camera_demo w/ 1 network multi channels with inputting 1 image #3392
Conversation
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
…_demo. Signed-off-by: Wang, Yang <yang4.wang@intel.com>
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
…_damo. Signed-off-by: Wang, Yang <yang4.wang@intel.com>
…ty_barrier_camera_dem. Signed-off-by: Wang, Yang <yang4.wang@intel.com>
…video. 2 Update case parameters in order to capturing ROI data from this demo. 3 Enable parse function for AUTO device. Signed-off-by: Wang, Yang <yang4.wang@intel.com>
…ted. Signed-off-by: Wang, Yang <yang4.wang@intel.com>
…nputing single iamge for each channel. Signed-off-by: Wang, Yang <yang4.wang@intel.com>
…UTO_plugin Signed-off-by: Wang, Yang <yang4.wang@intel.com>
…era_demo. Signed-off-by: Wang, Yang <yang4.wang@intel.com>
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
|
there are some basic pre-commit checks failed: 06:15:26 + ./venv/bin/python ./src/ci/check-basics.py |
|
I'm generally fine with the changes, yet the correctness checking seems to happen on the OMZ side? how do we get the results of such correctness, is there a weekly/nightly etc report or what? also the question to @vladimir-dudnik |
|
@myshevts the idea is to extend OMZ tests with inference results correctness check (currently we do not have this). OMZ tests currently used in automatic public runs, but there is no any reports on this. |
There was a problem hiding this comment.
I would suggest to refer my branch with scratch validation scenario and simplify the implementation of parser() for multi channel correctness checking.
Since in parser() the results are parsed from log file, these results obtained from runtime should be saved in the parser while run_tests.py running.
The main loop in run_tests.py consists of 3 loops: iteration over demos -> iteration over test cases -> iteration over devices. While the third loop, the output log might be saved in the parser directly (just need to fill self.results[device][test_case]).
Finally, after all combinations of devices + test cases are ended, checker() might be called for the demo having parsed results.
And if parser() with checker() are transformed into some class MultiChannelCorrectnessChecker, the list of demos will be (it's aligned to the implementation of performance_scope, validation_scope):
DEMOS = [
deepcopy(BASE['security_barrier_camera_demo/cpp'])
.update_option({'-r': None})
.add_parser(MultiChannelCorrectnessChecker)
]
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
| @@ -42,7 +42,7 @@ | |||
|
|
|||
| log.basicConfig(format='[ %(levelname)s ] %(message)s', level=log.DEBUG, stream=sys.stdout) | |||
|
|
|||
| DEVICE_KINDS = ['CPU', 'GPU', 'MYRIAD', 'HETERO', 'HDDL'] | |||
| DEVICE_KINDS = ['CPU', 'GPU', 'AUTO', 'AUTO:CPU,GPU', 'MULTI:CPU,GPU', 'MYRIAD', 'HETERO', 'HDDL'] | |||
There was a problem hiding this comment.
This should not be required for security_barrier_camera_demo ?
|
there are still some precommit checks failed 09:06:39 running documentation checks... |
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
…inputting video. 2. Update the correctness checker that allows insignificant difference for the ROI. Signed-off-by: Wang, Yang <yang4.wang@intel.com>
…urity_barrier_camera_demo. Signed-off-by: Wang, Yang <yang4.wang@intel.com>
…2. Update the raw data format for the demo. Signed-off-by: Wang, Yang <yang4.wang@intel.com>
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
…ty_barrier_camera demo. Signed-off-by: Wang, Yang <yang4.wang@intel.com>
… to avoid changing. Signed-off-by: Wang, Yang <yang4.wang@intel.com>
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
…rred frames. 2. update correctness checking logic that could validate the correctness the multi results within 1 frame individually. Signed-off-by: Wang, Yang <yang4.wang@intel.com>
…alue assign logic for the output port name of vehicle attribution. Signed-off-by: Wang, Yang <yang4.wang@intel.com>
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
…l_zoo into ywang2/analysis_result_automatically_for_AUTO_plugin
Signed-off-by: Wang, Yang <yang4.wang@intel.com>
…eout_occurs' into ywang2/analysis_result_automatically_for_AUTO_plugin
…eout_occurs' into ywang2/analysis_result_automatically_for_AUTO_plugin
Correctness checking for demo security_barrier_camera_demo with single image inputting for each channel.
JIRA:
https://jira.devtools.intel.com/browse/CVS-74969