Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: [benchmark][cluster][LRU] queryNode panic context deadline exceeded in dml & dql scene #32987

Closed
1 task done
wangting0128 opened this issue May 11, 2024 · 1 comment
Closed
1 task done

[Bug]: [benchmark][cluster][LRU] queryNode panic context deadline exceeded in dml & dql scene #32987

wangting0128 opened this issue May 11, 2024 · 1 comment
Assignees
@chyezh
Labels
kind/bug Issues or changes related a bug test/benchmark benchmark test triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone
2.4.lru

Comments

@wangting0128
Copy link
Contributor

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version:master-20240510-c0e62e66-amd64
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):pulsar    
- SDK version(e.g. pymilvus v2.0.0rc2):2.4.0rc66
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

argo task: lru-fouramf-d5jpz-verify-32801

server:

NAME                                                              READY   STATUS             RESTARTS          AGE     IP              NODE         NOMINATED NODE   READINESS GATES
lru-verify-32801-cluster-etcd-0                                   1/1     Running            0                 31m     10.104.20.151   4am-node22   <none>           <none>
lru-verify-32801-cluster-etcd-1                                   1/1     Running            0                 31m     10.104.23.94    4am-node27   <none>           <none>
lru-verify-32801-cluster-etcd-2                                   1/1     Running            0                 31m     10.104.21.233   4am-node24   <none>           <none>
lru-verify-32801-cluster-milvus-datacoord-86f498d87f-2kr7f        1/1     Running            3 (30m ago)       31m     10.104.24.19    4am-node29   <none>           <none>
lru-verify-32801-cluster-milvus-datanode-7bbf577884-sbsl6         1/1     Running            3 (30m ago)       31m     10.104.13.71    4am-node16   <none>           <none>
lru-verify-32801-cluster-milvus-indexcoord-58669fb8c4-k9zbj       1/1     Running            0                 31m     10.104.21.227   4am-node24   <none>           <none>
lru-verify-32801-cluster-milvus-indexnode-59645889c9-tbhfj        1/1     Running            3 (30m ago)       31m     10.104.23.90    4am-node27   <none>           <none>
lru-verify-32801-cluster-milvus-indexnode-59645889c9-tv84d        1/1     Running            3 (30m ago)       31m     10.104.25.44    4am-node30   <none>           <none>
lru-verify-32801-cluster-milvus-proxy-5b7b7cb494-zwgmt            1/1     Running            3 (30m ago)       31m     10.104.24.20    4am-node29   <none>           <none>
lru-verify-32801-cluster-milvus-querycoord-5bf6785d45-vwbzl       1/1     Running            3 (30m ago)       31m     10.104.24.17    4am-node29   <none>           <none>
lru-verify-32801-cluster-milvus-querynode-84bc9c849d-wqfnh        1/1     Running            4 (6m9s ago)      31m     10.104.25.42    4am-node30   <none>           <none>
lru-verify-32801-cluster-milvus-rootcoord-58675fbf87-6rff7        1/1     Running            3 (30m ago)       31m     10.104.24.18    4am-node29   <none>           <none>
lru-verify-32801-cluster-minio-0                                  1/1     Running            0                 31m     10.104.23.93    4am-node27   <none>           <none>
lru-verify-32801-cluster-minio-1                                  1/1     Running            0                 31m     10.104.20.148   4am-node22   <none>           <none>
lru-verify-32801-cluster-minio-2                                  1/1     Running            0                 31m     10.104.21.230   4am-node24   <none>           <none>
lru-verify-32801-cluster-minio-3                                  1/1     Running            0                 31m     10.104.24.25    4am-node29   <none>           <none>
lru-verify-32801-cluster-pulsar-bookie-0                          1/1     Running            0                 31m     10.104.20.149   4am-node22   <none>           <none>
lru-verify-32801-cluster-pulsar-bookie-1                          1/1     Running            0                 31m     10.104.24.26    4am-node29   <none>           <none>
lru-verify-32801-cluster-pulsar-bookie-2                          1/1     Running            0                 31m     10.104.21.234   4am-node24   <none>           <none>
lru-verify-32801-cluster-pulsar-bookie-init-vvvnw                 0/1     Completed          0                 31m     10.104.13.70    4am-node16   <none>           <none>
lru-verify-32801-cluster-pulsar-broker-0                          1/1     Running            0                 31m     10.104.30.229   4am-node38   <none>           <none>
lru-verify-32801-cluster-pulsar-proxy-0                           1/1     Running            0                 31m     10.104.5.142    4am-node12   <none>           <none>
lru-verify-32801-cluster-pulsar-pulsar-init-9wb27                 0/1     Completed          0                 31m     10.104.20.141   4am-node22   <none>           <none>
lru-verify-32801-cluster-pulsar-recovery-0                        1/1     Running            0                 31m     10.104.24.21    4am-node29   <none>           <none>
lru-verify-32801-cluster-pulsar-zookeeper-0                       1/1     Running            0                 31m     10.104.20.147   4am-node22   <none>           <none>
lru-verify-32801-cluster-pulsar-zookeeper-1                       1/1     Running            0                 30m     10.104.30.231   4am-node38   <none>           <none>
lru-verify-32801-cluster-pulsar-zookeeper-2                       1/1     Running            0                 30m     10.104.24.29    4am-node29   <none>           <none>
截屏2024-05-11 17 21 08 截屏2024-05-11 17 21 20

client pod name: lru-fouramf-d5jpz-verify-32801-1684395872
client log:

截屏2024-05-11 17 22 15

client config:

{
     "dataset_params": {
          "dataset_name": "laion1b_nolang",
          "column_name": "float32_vector",
          "dim": 768,
          "dataset_size": "10w",
          "ni_per": 10000,
          "metric_type": "L2",
          "scalars_params": {
               "int64_1": {
                    "params": {
                         "is_partition_key": true
                    }
               }
          }
     },
     "collection_params": {
          "other_fields": [
               "int64_1"
          ],
          "num_partitions": 64
     },
     "index_params": {
          "index_type": "HNSW",
          "index_param": {
               "M": 30,
               "efConstruction": 360
          }
     },
     "concurrent_tasks": [
          {
               "type": "insert",
               "weight": 1,
               "params": {
                    "nb": 64,
                    "timeout": 3000,
                    "random_vector": true
               }
          },
          {
               "type": "delete",
               "weight": 1,
               "params": {
                    "delete_length": 64,
                    "timeout": 3000
               }
          },
          {
               "type": "flush",
               "weight": 0,
               "params": {
                    "timeout": 3000
               }
          },
          {
               "type": "load",
               "weight": 1,
               "params": {
                    "timeout": 3000
               }
          },
          {
               "type": "search",
               "weight": 1,
               "params": {
                    "top_k": 1,
                    "nq": 10,
                    "search_param": {
                         "ef": 64
                    },
                    "expr": "int64_1 >= 0 && int64_1 <= 4",
                    "timeout": 3000,
                    "random_data": true
               }
          },
          {
               "type": "query",
               "weight": 1,
               "params": {
                    "expr": "int64_1 != 2",
                    "timeout": 3000,
                    "offset": 0,
                    "limit": 10
               }
          }
     ],
     "concurrent_params": {
          "interval": 20,
          "during_time": "12h",
          "concurrent_number": 10
     }
}

Expected Behavior

No response

Steps To Reproduce

1. create a collection with 3 fields: id(primaryKey, autoID), float_vector(768dim), int64_1(partitionKey=64)
2. build HNSW index
3. prepare 10w data
4. flush collection
5. build index again with the same params
6. load collection
7. concurrent requests:
   - insert
   - delete
   - load
   - search
   - query

Milvus Log

No response

Anything else?

No response
@wangting0128 wangting0128 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. test/benchmark benchmark test labels May 11, 2024
@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels May 13, 2024
@yanliang567 yanliang567 added this to the 2.4.lru milestone May 13, 2024
@yanliang567 yanliang567 removed their assignment May 13, 2024
@wangting0128
Copy link
Contributor Author

verification passed

image: master-20240514-f48a7ff8
argo task: lru-fouramf-wx7vx

test result:

{'server': {'deploy_tool': 'helm',
            'deploy_mode': 'standalone',
            'config_name': 'standalone_8c16m',
            'config': {'standalone': {'resources': {'limits': {'cpu': '2',
                                                               'memory': '8Gi',
                                                               'ephemeral-storage': '70Gi'},
                                                    'requests': {'cpu': '2',
                                                                 'memory': '8Gi'}},
                                      'messageQueue': 'pulsar',
                                      'extraEnv': [{'name': 'LOCAL_STORAGE_SIZE',
                                                    'value': '70'}],
                                      'disk': {'size': {'enabled': True}}},
                       'cluster': {'enabled': False},
                       'etcd': {'replicaCount': 1,
                                'metrics': {'enabled': True,
                                            'podMonitor': {'enabled': True}}},
                       'minio': {'mode': 'standalone',
                                 'metrics': {'podMonitor': {'enabled': True}},
                                 'persistence': {'size': '320Gi'}},
                       'pulsar': {'enabled': True},
                       'metrics': {'serviceMonitor': {'enabled': True}},
                       'log': {'level': 'debug'},
                       'extraConfigFiles': {'user.yaml': 'queryNode:\n'
                                                         '  '
                                                         'diskCacheCapacityLimit: '
                                                         '51539607552\n'
                                                         '  mmap:\n'
                                                         '    mmapEnabled: '
                                                         'true\n'
                                                         '  lazyload:\n'
                                                         '    enabled: true\n'
                                                         '    waitTimeout: '
                                                         '300000\n'
                                                         '  '
                                                         'useStreamComputing: '
                                                         'true\n'
                                                         '  cache:\n'
                                                         '    warmup: off\n'},
                       'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus',
                                         'tag': 'master-20240514-f48a7ff8'}}},
            'host': 'lru-32987-dml-dql-milvus.qa-milvus.svc.cluster.local',
            'port': '19530',
            'uri': ''},
 'client': {'test_case_type': 'ConcurrentClientBase',
            'test_case_name': 'test_concurrent_locust_custom_parameters',
            'test_case_params': {'dataset_params': {'metric_type': 'L2',
                                                    'column_name': 'float32_vector',
                                                    'dim': 768,
                                                    'scalars_params': {'int64_1': {'params': {'is_partition_key': True}}},
                                                    'dataset_name': 'laion1b_nolang',
                                                    'dataset_size': '10w',
                                                    'ni_per': 10000},
                                 'collection_params': {'other_fields': ['int64_1'],
                                                       'num_partitions': 64},
                                 'index_params': {'index_type': 'HNSW',
                                                  'index_param': {'M': 30,
                                                                  'efConstruction': 360}},
                                 'concurrent_params': {'concurrent_number': 10,
                                                       'during_time': '12h',
                                                       'interval': 20,
                                                       'spawn_rate': None},
                                 'concurrent_tasks': [{'type': 'insert',
                                                       'weight': 1,
                                                       'params': {'nb': 64,
                                                                  'timeout': 3000,
                                                                  'random_vector': True}},
                                                      {'type': 'delete',
                                                       'weight': 1,
                                                       'params': {'delete_length': 64,
                                                                  'timeout': 3000}},
                                                      {'type': 'flush',
                                                       'weight': 0,
                                                       'params': {'timeout': 3000}},
                                                      {'type': 'load',
                                                       'weight': 1,
                                                       'params': {'timeout': 3000}},
                                                      {'type': 'search',
                                                       'weight': 1,
                                                       'params': {'top_k': 1,
                                                                  'nq': 10,
                                                                  'search_param': {'ef': 64},
                                                                  'expr': 'int64_1 '
                                                                          '>= '
                                                                          '0 '
                                                                          '&& '
                                                                          'int64_1 '
                                                                          '<= '
                                                                          '4',
                                                                  'timeout': 3000,
                                                                  'random_data': True}},
                                                      {'type': 'query',
                                                       'weight': 1,
                                                       'params': {'expr': 'int64_1 '
                                                                          '!= '
                                                                          '2',
                                                                  'timeout': 3000,
                                                                  'offset': 0,
                                                                  'limit': 10}}]},
            'run_id': 2024051412144173,
            'datetime': '2024-05-14 12:53:34.093352',
            'client_version': '2.2'},
 'result': {'test_result': {'index': {'RT': 63.8284},
                            'insert': {'total_time': 27.8717,
                                       'VPS': 3587.8687,
                                       'batch_time': 2.7872,
                                       'batch': 10000},
                            'flush': {'RT': 4.9282},
                            'load': {'RT': 3.5367},
                            'Locust': {'Aggregated': {'Requests': 624242,
                                                      'Fails': 0,
                                                      'RPS': 14.45,
                                                      'fail_s': 0.0,
                                                      'RT_max': 68522.33,
                                                      'RT_avg': 689.36,
                                                      'TP50': 98,
                                                      'TP99': 15000.0},
                                       'delete': {'Requests': 124461,
                                                  'Fails': 0,
                                                  'RPS': 2.88,
                                                  'fail_s': 0.0,
                                                  'RT_max': 20965.18,
                                                  'RT_avg': 48.13,
                                                  'TP50': 34,
                                                  'TP99': 280.0},
                                       'insert': {'Requests': 124810,
                                                  'Fails': 0,
                                                  'RPS': 2.89,
                                                  'fail_s': 0.0,
                                                  'RT_max': 20390.22,
                                                  'RT_avg': 348.05,
                                                  'TP50': 260.0,
                                                  'TP99': 1800.0},
                                       'load': {'Requests': 125020,
                                                'Fails': 0,
                                                'RPS': 2.89,
                                                'fail_s': 0.0,
                                                'RT_max': 20068.64,
                                                'RT_avg': 75.72,
                                                'TP50': 53,
                                                'TP99': 430.0},
                                       'query': {'Requests': 125121,
                                                 'Fails': 0,
                                                 'RPS': 2.9,
                                                 'fail_s': 0.0,
                                                 'RT_max': 68510.11,
                                                 'RT_avg': 1502.58,
                                                 'TP50': 130.0,
                                                 'TP99': 24000.0},
                                       'search': {'Requests': 124830,
                                                  'Fails': 0,
                                                  'RPS': 2.89,
                                                  'fail_s': 0.0,
                                                  'RT_max': 68522.33,
                                                  'RT_avg': 1469.38,
                                                  'TP50': 110.0,
                                                  'TP99': 23000.0}}}}}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@chyezh chyezh

Labels
kind/bug Issues or changes related a bug test/benchmark benchmark test triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
2.4.lru
Development

No branches or pull requests
3 participants
@chyezh @wangting0128 @yanliang567