graphFun: A Graph Machine Learning Experiment Platform for High-Performance Computing

graphFun is an open-source experimental environment focused on machine learning for graph-structured data, supporting deployment on high-performance computing clusters and providing a flexible testing platform for the research and development of graph neural networks, graph embeddings, and graph analysis algorithms. It aims to lower the threshold for developing and testing graph ML algorithms while addressing engineering challenges in graph machine learning such as scalability and parallel computing complexity.

Graph data has a non-Euclidean structure, making traditional CNNs difficult to apply directly. GNNs solve this problem through message-passing mechanisms, but in practice, they face three major challenges: scalability (memory/time pressure from large-scale graph data), parallel computing complexity (difficult load balancing due to sparse connections), and algorithmic heterogeneity (different tasks/models require different optimization strategies).

graphFun is positioned as a "graph ML experiment playground" with core goals of lowering development thresholds and supporting HPC scalability. Its features include: modular component design (replaceable data loading/sampling, etc.); compatibility with mainstream HPC environments (MPI/OpenMP); and efficient graph partitioning strategies to minimize cross-node communication overhead.

The underlying layer uses PyG/DGL as the computing engine; the data layer supports formats such as NetworkX, CSR/CSC, OGB/SNAP, and various sampling algorithms; distributed training supports parameter server and all-reduce paradigms to optimize communication efficiency.

Academic scenarios: standardized experimental environment for model reproduction; industrial scenarios: prototype development for recommendation systems, drug discovery, and fraud detection; HPC scenarios: handling ultra-large-scale graph tasks such as astronomy/social networks to shorten experiment cycles.

Data preprocessing: node sorting to improve cache hit rate; sampling strategy: balancing convergence and overhead (e.g., importance sampling); distributed partitioning: selecting algorithms like METIS to minimize edge cuts and balance loads.

Higher abstraction compared to PyG/DGL; lightweight and open-source compared to commercial platforms (Neptune/Neo4j GDS); more general-purpose compared to specialized tools (DGL-KE), allowing users to choose as needed.

Plans to support the latest GNN variants, dynamic graphs/heterogeneous graphs; community participation (bug reports, code contributions, etc.) is crucial, aiming to lower the technical threshold in the field of graph intelligence.