pomelo-nwu's dogfooding #3397

pomelo-nwu · 2023-12-04T03:00:19Z

pomelo-nwu
Dec 4, 2023
Collaborator

Task 1: 在本地试用 GraphScope

安装 python3 ，安装 graphscope

pip install graphscope

根据 docs 文档，使用 graphscope ： https://graphscope.io/docs/zh/getting_started

感觉文档需要把 docker 这部分再详细介绍下

启动完docker，把数据文件下载在 ~/Desktop/GS/ogbn_mag_small/

import os

import graphscope
from graphscope.dataset import load_ogbn_mag

# assume we mount `~/test_data` to `/testingdata` in pods, and `ogbn_mag_small` within.
k8s_volumes = {
    "data": {
        "type": "hostPath",
        "field": {
            "path": os.path.expanduser("~/Desktop/GS/ogbn_mag_small/"),
            "type": "Directory"
        },
        "mounts": {
            "mountPath": "/testingdata"
        }
    }
}

sess = graphscope.session(k8s_volumes=k8s_volumes, k8s_service_type="LoadBalancer",cluster_type="hosts",num_workers=1)


g = sess.g()
g = (
    g.add_vertices("paper.csv", label="paper")
    .add_vertices("author.csv", label="author")
    .add_vertices("institution.csv", label="institution")
    .add_vertices("field_of_study.csv", label="field_of_study")
    .add_edges(
        "author_affiliated_with_institution.csv",
        label="affiliated",
        src_label="author",
        dst_label="institution",
    )
    .add_edges(
        "paper_has_topic_field_of_study.csv",
        label="hasTopic",
        src_label="paper",
        dst_label="field_of_study",
    )
    .add_edges(
        "paper_cites_paper.csv",
        label="cites",
        src_label="paper",
        dst_label="paper",
    )
    .add_edges(
        "author_writes_paper.csv",
        label="writes",
        src_label="author",
        dst_label="paper",
    )
)

# get the entrypoint for submitting Gremlin queries on graph g.
interactive = sess.gremlin(g)

# check the total node_num and edge_num
node_num = interactive.execute("g.V().count()").one()
edge_num = interactive.execute("g.E().count()").one()

print(f"node_num: {node_num}, edge_num: {edge_num}")

运行报错

Loading empty graph:   0%|          | 0/10 [00:00<?, ?it/s]
Loading empty graph: 100%|██████████| 10/10 [00:00<00:00, 74764.78it/s]
E20231204 11:27:02.474931 816159 /usr/local/include/graphscope/frame/property_graph_frame.cc:440] graphscope error in frame: code = 9 at /usr/local/include/graphscope/frame/property_graph_frame.cc:440: AddLabelsToGraph -> basic_string, backtrace: 
E20231204 11:27:02.476377 816159 /private/var/folders/76/zy5ktkns50v6gt5g8r0sf6sc0000gn/T/cirrus-ci-build/analytical_engine/core/server/dispatcher.cc:153] Worker 0: Unmatched error detected: leaf::error_info: Error ID = 1
2023-12-04 11:27:02,485 [ERROR][rpc:188]: Runstep failed with code: ANALYTICAL_ENGINE_INTERNAL_ERROR, message: Error occurred during RunStep, The traceback is: Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.11/site-packages/gscoordinator/op_executor.py", line 102, in run_step
    for response in responses:
  File "/opt/homebrew/lib/python3.11/site-packages/grpc/_channel.py", line 541, in __next__
    return self._next()
           ^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/grpc/_channel.py", line 967, in _next
    raise self
grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
        status = StatusCode.INTERNAL
        details = "Unmatched error detected: leaf::error_info: Error ID = 1

"
        debug_error_string = "UNKNOWN:Error received from peer ipv6:%5B::1%5D:36465 {created_time:"2023-12-04T11:27:02.476625+08:00", grpc_status:13, grpc_message:"Unmatched error detected: leaf::error_info: Error ID = 1\n\n"}"
>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/lib/python3.11/site-packages/gscoordinator/coordinator.py", line 316, in _RunStep
    head, bodies = self._operation_executor.run_on_analytical_engine(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/gscoordinator/monitor.py", line 191, in runOnAnalyticalEngineWarp
    res = func(instance, dag_def, dag_bodies, loader_op_bodies)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/gscoordinator/op_executor.py", line 174, in run_on_analytical_engine
    response_head, response_bodies = self.run_step(dag_def, dag_bodies)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.11/site-packages/gscoordinator/op_executor.py", line 116, in run_step
    raise AnalyticalEngineInternalError(msg)
graphscope.framework.errors.AnalyticalEngineInternalError: Unmatched error detected: leaf::error_info: Error ID = 1

pomelo-nwu · 2023-12-11T03:53:47Z

pomelo-nwu
Dec 11, 2023
Collaborator Author

Python 环境安装，建议使用 Conda

这里面有一个小坑，在 Python 官网安装的版本是最新的 3.12.0 这个时候再 pip3 install graphscope 就会报错。

经过@元斯大佬排查，应该是 python 的版本太新导致的，建议使用更专业的包管理工具 Conda 来安装3.9这个版本

pip / conda / miniconda 的关系

pip 是 Python 的默认包管理器安装和管理 Python 包。
conda提供了环境管理系统，可以创建独立的虚拟环境，可以和pip组合使用。
miniconda 是 conda 的免费最小安装程序。它是 Anaconda 的一个小型引导版本，仅包含 conda、Python、它们都依赖的包以及少量其他有用的包

使用 Conda 安装 Python 3.9

访问 miniConda 官网：https://docs.conda.io/projects/miniconda/en/latest/
用 pkg 安装的时候，建议修改安装位置在 ~目录，不要无脑下一步
查看版本：conda --version
安装 python 3.9 ：conda create --name py39 python=3.9
查看当前环境：conda env list
激活 python 3.9 环境：conda activate py39

安装 graphscope

pip install graphscope

按照 Readme 实操

import graphscope
from graphscope.dataset import load_ogbn_mag

g = load_ogbn_mag()

# get the endpoint for submitting Gremlin queries on graph g.
interactive = graphscope.gremlin(g)

# count the number of papers two authors (with id 2 and 4307) have co-authored
papers = interactive.execute("g.V().has('author', 'id', 2).out('writes').where(__.in('writes').has('id', 4307)).count()").one()

print(papers)

运行 python gs.py 如下报错，看错误提示似乎是 graphscope.gremlin 换成了 graphscope.interactive

0 replies

pomelo-nwu · 2023-12-11T03:59:01Z

pomelo-nwu
Dec 11, 2023
Collaborator Author

基于上述报错，Readme文档需要更新了，现在按照 Docs 最新文档使用：https://graphscope.io/docs/overview/getting_started

import graphscope
from graphscope.dataset import load_ogbn_mag

g = load_ogbn_mag()

# get the endpoint for submitting interactive queries on graph g.
interactive = graphscope.interactive(g)

# Gremlin query for counting the number of papers two authors (with id 2 and 4307) have co-authored
papers = interactive.execute("g.V().has('author', 'id', 2).out('writes').where(__.in('writes').has('id', 4307)).count()").one()

# Cypher query for counting the number of papers two authors (with id 2 and 4307) have co-authored
# Note that for Cypher query, the parameter of lang="cypher" is mandatory
papers = interactive.execute( \
    "MATCH (n1:author)-[:writes]->(p:paper)<-[:writes]-(n2:author) \
        WHERE n1.id = 2 AND n2.id = 4307 \
        RETURN count(DISTINCT p)", \
    lang="cypher")

print(papers)

抛出错误如下

python gs.py 
Loading empty graph:   0%|          | 0/10 [00:00<?, ?it/s]
Loading empty graph: 100%|██████████| 10/10 [00:00<00:00, 82565.04it/s]
Loading vertex labeled paper and:   0%|          | 0/10 [00:00<?, ?it/s]
Loading vertex labeled paper and:  20%|██        | 2/10 [00:00<00:00, 12.25it/s]
Loading vertex labeled paper and: 100%|██████████| 10/10 [00:00<00:00, 50.22it/s]
Loading vertex labeled author and:   0%|          | 0/10 [00:00<?, ?it/s]
Loading vertex labeled author and: 100%|██████████| 10/10 [00:00<00:00, 361.53it/s]
Loading vertex labeled institution and:   0%|          | 0/10 [00:00<?, ?it/s]
Loading vertex labeled institution and: 100%|██████████| 10/10 [00:00<00:00, 634.36it/s]
Loading vertex labeled field_of_study and:   0%|          | 0/10 [00:00<?, ?it/s]
Loading vertex labeled field_of_study and: 100%|██████████| 10/10 [00:00<00:00, 531.91it/s]
Loading edge labeled affiliated:   0%|          | 0/10 [00:00<?, ?it/s]
Loading edge labeled affiliated: 100%|██████████| 10/10 [00:00<00:00, 423.55it/s]
Loading edge labeled hasTopic:   0%|          | 0/10 [00:00<?, ?it/s]
Loading edge labeled hasTopic: 100%|██████████| 10/10 [00:00<00:00, 297.99it/s]
Loading edge labeled cites:   0%|          | 0/10 [00:00<?, ?it/s]
Loading edge labeled cites: 100%|██████████| 10/10 [00:00<00:00, 526.84it/s]
Loading edge labeled writes:   0%|          | 0/10 [00:00<?, ?it/s]
Loading edge labeled writes: 100%|██████████| 10/10 [00:00<00:00, 282.05it/s]
/Users/pomelo/miniconda3/envs/py39/lib/python3.9/site-packages/graphscope.runtime/bin/giectl: line 454: kill: (55296) - No such process
/Users/pomelo/miniconda3/envs/py39/lib/python3.9/site-packages/graphscope.runtime/bin/giectl: line 454: kill: (55296) - No such process
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/pomelo/Desktop/github/gs-test/gs.py:7 in <module>                                         │
│                                                                                                  │
│    4 g = load_ogbn_mag()                                                                         │
│    5                                                                                             │
│    6 # get the endpoint for submitting interactive queries on graph g.                           │
│ ❱  7 interactive = graphscope.interactive(g)                                                     │
│    8                                                                                             │
│    9 # Gremlin query for counting the number of papers two authors (with id 2 and 4307) have     │
│   10 papers = interactive.execute("g.V().has('author', 'id', 2).out('writes').where(__.in('wr    │
│                                                                                                  │
│ /Users/pomelo/miniconda3/envs/py39/lib/python3.9/site-packages/graphscope/client/session.py:1794 │
│ in interactive                                                                                   │
│                                                                                                  │
│   1791 │   assert (                                                                              │
│   1792 │   │   graph._session is not None                                                        │
│   1793 │   ), "The graph object is invalid"  # pylint: disable=protected-access                  │
│ ❱ 1794 │   return graph._session.interactive(graph, params)  # pylint: disable=protected-access  │
│   1795                                                                                           │
│   1796                                                                                           │
│   1797 def graphlearn(graph, nodes=None, edges=None, gen_labels=None):                           │
│                                                                                                  │
│ /Users/pomelo/miniconda3/envs/py39/lib/python3.9/site-packages/graphscope/client/session.py:1373 │
│ in interactive                                                                                   │
│                                                                                                  │
│   1370 │   │   (                                                                                 │
│   1371 │   │   │   gremlin_endpoint,                                                             │
│   1372 │   │   │   cypher_endpoint,                                                              │
│ ❱ 1373 │   │   ) = self._grpc_client.create_interactive_instance(                                │
│   1374 │   │   │   object_id, schema_path, params                                                │
│   1375 │   │   )                                                                                 │
│   1376 │   │   interactive_query = InteractiveQuery(graph, gremlin_endpoint, cypher_endpoint)    │
│                                                                                                  │
│ /Users/pomelo/miniconda3/envs/py39/lib/python3.9/site-packages/graphscope/client/rpc.py:216 in   │
│ create_interactive_instance                                                                      │
│                                                                                                  │
│   213 │   │   │   for k, v in params.items():                                                    │
│   214 │   │   │   │   request.params[str(k)] = str(v)                                            │
│   215 │   │                                                                                      │
│ ❱ 216 │   │   response = self._stub.CreateInteractiveInstance(request)                           │
│   217 │   │   return response.gremlin_endpoint, response.cypher_endpoint                         │
│   218 │                                                                                          │
│   219 │   def create_learning_instance(self, object_id, handle, config):                         │
│                                                                                                  │
│ /Users/pomelo/miniconda3/envs/py39/lib/python3.9/site-packages/grpc/_channel.py:1160 in __call__ │
│                                                                                                  │
│   1157 │   │   ) = self._blocking(                                                               │
│   1158 │   │   │   request, timeout, metadata, credentials, wait_for_ready, compression          │
│   1159 │   │   )                                                                                 │
│ ❱ 1160 │   │   return _end_unary_response_blocking(state, call, False, None)                     │
│   1161 │                                                                                         │
│   1162 │   def with_call(                                                                        │
│   1163 │   │   self,                                                                             │
│                                                                                                  │
│ /Users/pomelo/miniconda3/envs/py39/lib/python3.9/site-packages/grpc/_channel.py:1003 in          │
│ _end_unary_response_blocking                                                                     │
│                                                                                                  │
│   1000 │   │   else:                                                                             │
│   1001 │   │   │   return state.response                                                         │
│   1002 │   else:                                                                                 │
│ ❱ 1003 │   │   raise _InactiveRpcError(state)  # pytype: disable=not-instantiable                │
│   1004                                                                                           │
│   1005                                                                                           │
│   1006 def _stream_unary_invocation_operations(                                                  │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.ABORTED
        details = "Create interactive instance failed: $Gremlin check query failed: Cannot connect to host 127.0.0.1:8235 ssl:default 
[Connect call failed ('127.0.0.1', 8235)]. The traceback is: Traceback (most recent call last):
  File "/Users/pomelo/miniconda3/envs/py39/lib/python3.9/site-packages/gscoordinator/coordinator.py", line 488, in CreateInteractiveInstance
    if check_server_ready(
  File "/Users/pomelo/miniconda3/envs/py39/lib/python3.9/site-packages/gscoordinator/utils.py", line 2085, in check_server_ready
    raise TimeoutError(
TimeoutError: Gremlin check query failed: Cannot connect to host 127.0.0.1:8235 ssl:default [Connect call failed ('127.0.0.1', 8235)]
"
        debug_error_string = "UNKNOWN:Error received from peer ipv4:127.0.0.1:42854 {created_time:"2023-12-11T11:59:38.776607+08:00", 
grpc_status:10, grpc_message:"Create interactive instance failed: $Gremlin check query failed: Cannot connect to host 127.0.0.1:8235 
ssl:default [Connect call failed (\'127.0.0.1\', 8235)]. The traceback is: Traceback (most recent call last):\n  File 
\"/Users/pomelo/miniconda3/envs/py39/lib/python3.9/site-packages/gscoordinator/coordinator.py\", line 488, in CreateInteractiveInstance\n   
if check_server_ready(\n  File \"/Users/pomelo/miniconda3/envs/py39/lib/python3.9/site-packages/gscoordinator/utils.py\", line 2085, in 
check_server_ready\n    raise TimeoutError(\nTimeoutError: Gremlin check query failed: Cannot connect to host 127.0.0.1:8235 ssl:default 
[Connect call failed (\'127.0.0.1\', 8235)]\n"}"
>
Exception ignored in: <function AiohttpTransport.__del__ at 0x139262af0>
Traceback (most recent call last):
  File "/Users/pomelo/miniconda3/envs/py39/lib/python3.9/site-packages/gremlin_python/driver/aiohttp/transport.py", line 61, in __del__
    self.close()
  File "/Users/pomelo/miniconda3/envs/py39/lib/python3.9/site-packages/gremlin_python/driver/aiohttp/transport.py", line 135, in close
    self._loop.close()
  File "/Users/pomelo/miniconda3/envs/py39/lib/python3.9/asyncio/unix_events.py", line 58, in close
    super().close()
  File "/Users/pomelo/miniconda3/envs/py39/lib/python3.9/asyncio/selector_events.py", line 87, in close
    self._close_self_pipe()
  File "/Users/pomelo/miniconda3/envs/py39/lib/python3.9/asyncio/selector_events.py", line 94, in _close_self_pipe
    self._remove_reader(self._ssock.fileno())
  File "/Users/pomelo/miniconda3/envs/py39/lib/python3.9/asyncio/selector_events.py", line 272, in _remove_reader
    key = self._selector.get_key(fd)
  File "/Users/pomelo/miniconda3/envs/py39/lib/python3.9/selectors.py", line 191, in get_key
    return mapping[fileobj]
  File "/Users/pomelo/miniconda3/envs/py39/lib/python3.9/selectors.py", line 72, in __getitem__
    fd = self._selector._fileobj_lookup(fileobj)
  File "/Users/pomelo/miniconda3/envs/py39/lib/python3.9/selectors.py", line 226, in _fileobj_lookup
    return _fileobj_to_fd(fileobj)
  File "/Users/pomelo/miniconda3/envs/py39/lib/python3.9/selectors.py", line 42, in _fileobj_to_fd
    raise ValueError("Invalid file descriptor: {}".format(fd))
ValueError: Invalid file descriptor: -1
Exception ignored in: <function BaseEventLoop.__del__ at 0x104f468b0>
Traceback (most recent call last):
  File "/Users/pomelo/miniconda3/envs/py39/lib/python3.9/asyncio/base_events.py", line 688, in __del__
    self.close()
  File "/Users/pomelo/miniconda3/envs/py39/lib/python3.9/asyncio/unix_events.py", line 58, in close
    super().close()
  File "/Users/pomelo/miniconda3/envs/py39/lib/python3.9/asyncio/selector_events.py", line 87, in close
    self._close_self_pipe()
  File "/Users/pomelo/miniconda3/envs/py39/lib/python3.9/asyncio/selector_events.py", line 94, in _close_self_pipe
    self._remove_reader(self._ssock.fileno())
  File "/Users/pomelo/miniconda3/envs/py39/lib/python3.9/asyncio/selector_events.py", line 272, in _remove_reader
    key = self._selector.get_key(fd)
  File "/Users/pomelo/miniconda3/envs/py39/lib/python3.9/selectors.py", line 191, in get_key
    return mapping[fileobj]
  File "/Users/pomelo/miniconda3/envs/py39/lib/python3.9/selectors.py", line 72, in __getitem__
    fd = self._selector._fileobj_lookup(fileobj)
  File "/Users/pomelo/miniconda3/envs/py39/lib/python3.9/selectors.py", line 226, in _fileobj_lookup
    return _fileobj_to_fd(fileobj)
  File "/Users/pomelo/miniconda3/envs/py39/lib/python3.9/selectors.py", line 42, in _fileobj_to_fd
    raise ValueError("Invalid file descriptor: {}".format(fd))
ValueError: Invalid file descriptor: -1

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pomelo-nwu's dogfooding #3397

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Select a reply

pomelo-nwu's dogfooding #3397

pomelo-nwu Dec 4, 2023 Collaborator

Task 1: 在本地试用 GraphScope

Replies: 2 comments

pomelo-nwu Dec 11, 2023 Collaborator Author

Python 环境安装，建议使用 Conda

pip / conda / miniconda 的关系

使用 Conda 安装 Python 3.9

安装 graphscope

pomelo-nwu Dec 11, 2023 Collaborator Author

pomelo-nwu
Dec 4, 2023
Collaborator

pomelo-nwu
Dec 11, 2023
Collaborator Author

pomelo-nwu
Dec 11, 2023
Collaborator Author