[Tool] Add comprehensive GitHub Copilot instructions for StarRocks (#62136)
Signed-off-by: Seaven <seaven_7@qq.com>
This commit is contained in:
Copilot
GitHub
parent
2f101c3d88
commit
df147a4ee7
|
|
@ -0,0 +1,258 @@
|
||||||
|
# GitHub Copilot Instructions for StarRocks
|
||||||
|
|
||||||
|
## Project Overview
|
||||||
|
|
||||||
|
StarRocks is a high-performance, cloud-native analytical database system designed for real-time analytics and ad-hoc queries. It features a streamlined architecture with both shared-nothing and shared-data deployment modes, supporting sub-second query performance for complex analytical workloads.
|
||||||
|
|
||||||
|
**Key Technologies:**
|
||||||
|
- **Backend (BE)**: C++ - Core analytical engine, storage layer, and query execution
|
||||||
|
- **Frontend (FE)**: Java - SQL parsing, query planning, metadata management, and coordination
|
||||||
|
- **Java Extensions**: Java - External connectors and UDF framework
|
||||||
|
- **Testing**: Python - Integration tests and SQL test framework
|
||||||
|
|
||||||
|
## Architecture Components
|
||||||
|
|
||||||
|
### Backend (be/) - C++
|
||||||
|
The core analytical engine responsible for data storage, processing, and query execution:
|
||||||
|
|
||||||
|
**Core Components:**
|
||||||
|
- `be/src/exec/` - Query execution operators (scan, join, aggregate, etc.)
|
||||||
|
- `be/src/storage/` - Storage engine (tablets, rowsets, segments, compaction)
|
||||||
|
- `be/src/exprs/` - Expression evaluation and vectorized computation
|
||||||
|
- `be/src/formats/` - Data format support (Parquet, ORC, CSV, JSON)
|
||||||
|
- `be/src/runtime/` - Runtime services (memory management, load balancing, stream processing)
|
||||||
|
- `be/src/connector/` - External data source connectors (Hive, Iceberg, Delta Lake)
|
||||||
|
- `be/src/service/` - RPC services and BE coordination
|
||||||
|
- `be/src/common/` - Shared utilities and common data structures
|
||||||
|
|
||||||
|
**Performance Focus:**
|
||||||
|
- Vectorized query execution
|
||||||
|
- Columnar storage format
|
||||||
|
- Memory-efficient algorithms
|
||||||
|
- SIMD optimizations where applicable
|
||||||
|
|
||||||
|
📋 **Note:** See `be/.cursorrules` for detailed backend component breakdown
|
||||||
|
|
||||||
|
### Frontend (fe/) - Java
|
||||||
|
SQL interface and query coordination layer:
|
||||||
|
|
||||||
|
**Core Components:**
|
||||||
|
- `fe/fe-core/src/main/java/com/starrocks/`
|
||||||
|
- `sql/` - SQL parser, analyzer, and AST
|
||||||
|
- `planner/` - Query planning and optimization (CBO)
|
||||||
|
- `catalog/` - Metadata management (tables, partitions, statistics)
|
||||||
|
- `scheduler/` - Query scheduling and execution coordination
|
||||||
|
- `load/` - Data loading coordination (Broker Load, Stream Load, etc.)
|
||||||
|
- `backup/` - Backup and restore functionality
|
||||||
|
- `privilege/` - Authentication and authorization
|
||||||
|
- `qe/` - Query execution coordination and session management
|
||||||
|
- `fe/fe-common/` - Common frontend utilities
|
||||||
|
- `fe/plugin-common/` - Plugin framework common components
|
||||||
|
- `fe/spark-dpp/` - Spark data preprocessing integration
|
||||||
|
- `fe/hive-udf/` - Hive UDF compatibility layer
|
||||||
|
|
||||||
|
**Key Responsibilities:**
|
||||||
|
- Parse and validate SQL statements
|
||||||
|
- Generate optimized query plans using Cost-Based Optimizer (CBO)
|
||||||
|
- Manage cluster metadata and coordination
|
||||||
|
- Handle user sessions and security
|
||||||
|
|
||||||
|
📋 **Note:** See `fe/.cursorrules` for detailed frontend component breakdown
|
||||||
|
|
||||||
|
### Java Extensions (java-extensions/) - Java
|
||||||
|
External connectivity and extensibility:
|
||||||
|
|
||||||
|
**Data Source Connectors:**
|
||||||
|
- `hive-reader/` - Apache Hive integration
|
||||||
|
- `iceberg-metadata-reader/` - Apache Iceberg support
|
||||||
|
- `hudi-reader/` - Apache Hudi integration
|
||||||
|
- `paimon-reader/` - Apache Paimon support
|
||||||
|
- `jdbc-bridge/` - JDBC connectivity for external databases
|
||||||
|
- `odps-reader/` - Alibaba ODPS integration
|
||||||
|
|
||||||
|
**Extension Framework:**
|
||||||
|
- `udf-extensions/` - User-Defined Function framework
|
||||||
|
- `common-runtime/` - Shared runtime for extensions
|
||||||
|
- `hadoop-ext/` - Hadoop ecosystem integration
|
||||||
|
|
||||||
|
📋 **Note:** See `java-extensions/.cursorrules` for detailed extensions breakdown
|
||||||
|
|
||||||
|
### Additional Important Directories
|
||||||
|
|
||||||
|
**Generated Sources (gensrc/):**
|
||||||
|
- `gensrc/proto/` - Protocol buffer definitions
|
||||||
|
- `gensrc/thrift/` - Thrift interface definitions
|
||||||
|
- `gensrc/script/` - Code generation scripts
|
||||||
|
|
||||||
|
**Testing Framework (test/):**
|
||||||
|
- `test/sql/` - SQL test cases organized by functionality
|
||||||
|
- `test/common/` - Common test utilities
|
||||||
|
- `test/lib/` - Test libraries and helpers
|
||||||
|
|
||||||
|
**Tools and Utilities:**
|
||||||
|
- `tools/` - Diagnostic tools, benchmarks, and utilities
|
||||||
|
- `bin/` - Binary executables and scripts
|
||||||
|
- `conf/` - Configuration files and templates
|
||||||
|
- `build-support/` - Build system support files
|
||||||
|
- `docker/` - Docker build configurations
|
||||||
|
|
||||||
|
**Other Key Directories:**
|
||||||
|
- `thirdparty/` - External dependencies and patches
|
||||||
|
- `fs_brokers/` - File system broker implementations
|
||||||
|
- `webroot/` - Web UI static files
|
||||||
|
- `format-sdk/` - Format SDK for data interchange
|
||||||
|
|
||||||
|
## Coding Guidelines
|
||||||
|
|
||||||
|
### C++ (Backend)
|
||||||
|
```cpp
|
||||||
|
// Use modern C++ features (C++17/C++20)
|
||||||
|
// Follow Google C++ Style Guide conventions
|
||||||
|
// Use RAII for resource management
|
||||||
|
// Prefer smart pointers over raw pointers
|
||||||
|
// Use const-correctness
|
||||||
|
|
||||||
|
// Example: Vectorized processing pattern
|
||||||
|
Status ColumnProcessor::process_batch(const ChunkPtr& chunk) {
|
||||||
|
const auto& column = chunk->get_column_by_name("column_name");
|
||||||
|
auto result_column = std::make_shared<Column>();
|
||||||
|
|
||||||
|
// Vectorized operation on entire column
|
||||||
|
for (size_t i = 0; i < chunk->num_rows(); ++i) {
|
||||||
|
// Process element
|
||||||
|
}
|
||||||
|
|
||||||
|
return Status::OK();
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Java (Frontend)
|
||||||
|
```java
|
||||||
|
// Follow Java coding conventions
|
||||||
|
// Use dependency injection where appropriate
|
||||||
|
// Implement proper exception handling
|
||||||
|
// Use builder patterns for complex objects
|
||||||
|
// Follow existing naming conventions
|
||||||
|
|
||||||
|
// Example: Query planning pattern
|
||||||
|
public class ScanNodePlanner extends PlanFragment {
|
||||||
|
@Override
|
||||||
|
public PlanFragment visitLogicalScanOperator(
|
||||||
|
OptExpression optExpression, ExecPlan context) {
|
||||||
|
LogicalScanOperator scanOperator =
|
||||||
|
(LogicalScanOperator) optExpression.getOp();
|
||||||
|
|
||||||
|
// Create physical scan node
|
||||||
|
ScanNode scanNode = createScanNode(scanOperator);
|
||||||
|
return new PlanFragment(scanNode);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
## ⚠️ CRITICAL BUILD SYSTEM WARNING
|
||||||
|
**DO NOT attempt to build or run unit tests (UT) for this project unless explicitly requested by the user.**
|
||||||
|
|
||||||
|
The build system is extremely resource-intensive and time-consuming. Building the full project can take hours and requires significant system resources.
|
||||||
|
|
||||||
|
**Specific commands and files to AVOID:**
|
||||||
|
- `build.sh` - Main build script (extremely resource intensive)
|
||||||
|
- `build-in-docker.sh` - Docker-based build
|
||||||
|
- `run-be-ut.sh` / `run-fe-ut.sh` / `run-java-exts-ut.sh` - Unit test runners
|
||||||
|
- `docker-compose` commands - Heavy resource usage
|
||||||
|
- `Makefile*` - Make build files
|
||||||
|
- `pom.xml` - Maven build files (for Java components)
|
||||||
|
|
||||||
|
**Focus on code analysis and targeted changes instead of full builds.**
|
||||||
|
|
||||||
|
## Important Guidelines
|
||||||
|
|
||||||
|
### Pull Request Requirements
|
||||||
|
|
||||||
|
**PR Title Format:**
|
||||||
|
Must include category prefix:
|
||||||
|
- `[BugFix]` - Bug fixes and error corrections
|
||||||
|
- `[Feature]` - New features and capabilities
|
||||||
|
- `[Enhancement]` - Improvements to existing functionality
|
||||||
|
- `[Refactor]` - Code refactoring without functional changes
|
||||||
|
- `[Test]` - Test-related changes
|
||||||
|
- `[Doc]` - Documentation updates
|
||||||
|
- `[Build]` - Build system and CI/CD changes
|
||||||
|
- `[Performance]` - Performance optimizations
|
||||||
|
|
||||||
|
**Example:** `[Feature] Add Apache Paimon table format support`
|
||||||
|
|
||||||
|
|
||||||
|
### Code Review Focus Areas
|
||||||
|
|
||||||
|
**Performance Considerations:**
|
||||||
|
- Query execution efficiency
|
||||||
|
- Memory usage patterns
|
||||||
|
- Lock contention in concurrent scenarios
|
||||||
|
- Network I/O optimization
|
||||||
|
|
||||||
|
**Correctness Priorities:**
|
||||||
|
- SQL standard compliance
|
||||||
|
- Data type handling accuracy
|
||||||
|
- Transaction consistency
|
||||||
|
- Error handling completeness
|
||||||
|
|
||||||
|
**Security Considerations:**
|
||||||
|
- Input validation and sanitization
|
||||||
|
- Authentication and authorization
|
||||||
|
- Resource usage limits
|
||||||
|
- Information leak prevention
|
||||||
|
|
||||||
|
## Common Development Patterns
|
||||||
|
|
||||||
|
### Adding New SQL Functions
|
||||||
|
1. Define function signature in `fe/fe-core/src/main/java/com/starrocks/catalog/FunctionSet.java`
|
||||||
|
2. Implement evaluation logic in `be/src/exprs/`
|
||||||
|
3. Add comprehensive tests in `test/sql/test_functions/`
|
||||||
|
|
||||||
|
### Adding New Data Source Connectors
|
||||||
|
1. Implement connector interface in `java-extensions/`
|
||||||
|
2. Add metadata reader and schema handling
|
||||||
|
3. Integrate with query planner in `fe/fe-core/src/main/java/com/starrocks/connector/`
|
||||||
|
4. Add integration tests
|
||||||
|
|
||||||
|
### Query Optimization Improvements
|
||||||
|
1. Analyze optimizer rules in `fe/fe-core/src/main/java/com/starrocks/sql/optimizer/rule/`
|
||||||
|
2. Update cost model if needed in `fe/fe-core/src/main/java/com/starrocks/sql/optimizer/cost/`
|
||||||
|
3. Add test cases in `test/sql/test_optimizer/`
|
||||||
|
|
||||||
|
## Documentation References
|
||||||
|
|
||||||
|
- **Contributing Guide**: [`CONTRIBUTING.md`](../CONTRIBUTING.md)
|
||||||
|
- **Development Setup**: [StarRocks Documentation](https://docs.starrocks.io/docs/developers/)
|
||||||
|
- **Architecture Overview**: [README.md](../README.md#architecture-overview)
|
||||||
|
- **PR Template**: [`.github/PULL_REQUEST_TEMPLATE.md`](.github/PULL_REQUEST_TEMPLATE.md)
|
||||||
|
|
||||||
|
## Quick Reference
|
||||||
|
|
||||||
|
**Key File Extensions:**
|
||||||
|
- `.cpp`, `.h`, `.cc` - C++ backend code
|
||||||
|
- `.java` - Java frontend/extensions code
|
||||||
|
- `.sql` - SQL test cases
|
||||||
|
- `.py` - Python test scripts
|
||||||
|
- `.proto` - Protocol buffer definitions
|
||||||
|
- `.thrift` - Thrift interface definitions
|
||||||
|
|
||||||
|
**Important Configuration:**
|
||||||
|
- `conf/` - Runtime configuration templates
|
||||||
|
- `gensrc/` - Auto-generated code from IDL definitions
|
||||||
|
- `thirdparty/` - External dependencies
|
||||||
|
|
||||||
|
**Testing Structure:**
|
||||||
|
- `test/sql/` - SQL correctness tests organized by functionality
|
||||||
|
- `be/test/` - C++ unit tests
|
||||||
|
- `fe/fe-core/src/test/` - Java unit tests
|
||||||
|
|
||||||
|
**Build System Files to Avoid:**
|
||||||
|
- `build.sh` - Main build script (very resource intensive)
|
||||||
|
- `build-in-docker.sh` - Docker-based build
|
||||||
|
- `run-*-ut.sh` - Unit test runners
|
||||||
|
- `Makefile*` - Make build files
|
||||||
|
- `pom.xml` - Maven build files (for Java components)
|
||||||
|
|
||||||
|
This project prioritizes **performance**, **correctness**, and **scalability**. When contributing, consider the impact on query performance and ensure changes maintain SQL standard compliance.
|
||||||
Loading…
Reference in New Issue