8d83da4fdb
This PR adds scripting ability to BE, as a way to increase system observability, making testing, debugging and online hot fix issues easier. The script engine uses wren script language and wrenbind17 binding util(it's a small OO language).
Currently, only a small faction of BE objects & methods are bonded, mainly focusing on StorageEngine(Tablets, Rowsets, TabletUpates).
Will add more binding for rocksdb metadata, thread/stack dumps, memory/cpu profiling utils in the future.
Sample uses cases:
```
mysql> admin execute on 10004 '
'> var ts = StorageEngine.get_tablet_infos(-1,-1)
'> for (t in ts) {
'> System.print("%(t.tablet_id) %(t.data_size)")
'> }
'> ';
Query OK, 0 rows affected (0.01 sec)
10009 0
10011 0
10013 0
...
mysql> admin execute on 10004 '
'> var t = StorageEngine.get_tablet(11005)
'> System.print("info: %(t.updates().toPB().toString())")
'> ';
Query OK, 0 rows affected (0.00 sec)
info: {"versions":[{"version":{"major":"3","minor":"1"},"creationTime":"1679897711","rowsets":[2],"compaction":{"startVersion":{"major":"3","minor":"0"},"inputs":[0,1],"outputs":[2]}},{"version":{"major":"4","minor":"0"},"creationTime":"1679900665","rowsets":[2,3],"deltas":[3]},{"version":{"major":"5","minor":"0"},"creationTime":"1679900666","rowsets":[2,3,4],"deltas":[4]},{"version":{"major":"6","minor":"0"},"creationTime":"1679900667","rowsets":[2,3,4,5],"deltas":[5]},{"version":{"major":"7","minor":"0"},"creationTime":"1679900667","rowsets":[2,3,4,5,6],"deltas":[6]},{"version":{"major":"8","minor":"0"},"creationTime":"1679900667","rowsets":[2,3,4,5,6,7],"deltas":[7]},{"version":{"major":"9","minor":"0"},"creationTime":"1679900667","rowsets":[2,3,4,5,6,7,8],"deltas":[8]},{"version":{"major":"10","minor":"0"},"creationTime":"1679900668","rowsets":[2,3,4,5,6,7,8,9],"deltas":[9]},{"version":{"major":"11","minor":"0"},"creationTime":"1679900668","rowsets":[2,3,4,5,6,7,8,9,10],"deltas":[10]},{"version":{"major":"12","minor":"0"},"creationTime":"1679900668","rowsets":[2,3,4,5,6,7,8,9,10,11],"deltas":[11]},{"version":{"major":"13","minor":"0"},"creationTime":"1679900668","rowsets":[2,3,4,5,6,7,8,9,10,11,12],"deltas":[12]},{"version":{"major":"13","minor":"1"},"creationTime":"1679900708","rowsets":[13],"compaction":{"startVersion":{"major":"13","minor":"0"},"inputs":[2,3,4,5,6,7,8,9,10,11,12],"outputs":[13]}}],"applyVersion":{"major":"13","minor":"1"},"nextRowsetId":14,"nextLogId":"28"}
mysql> admin execute on 10004 '
'> for (dir in StorageEngine.get_data_dirs()) {
'> System.print("dir: %(dir.path()) rocksdb: %(dir.get_meta().sst_file_size())")
'> var result = dir.get_meta().compact().toString()
'> System.print("compact %(result) size: %(dir.get_meta().sst_file_size())")
'> }
'> ';
Query OK, 0 rows affected (0.00 sec)
dir: /home/decster/projects/starrocks/localrun/be/storage rocksdb: 156484
compact OK size: 12708
```
Signed-off-by: Binglin Chang <decstery@gmail.com>
|
||
|---|---|---|
| .github | ||
| be | ||
| bin | ||
| build-support | ||
| community | ||
| conf | ||
| contrib | ||
| docker | ||
| docs | ||
| fe | ||
| fs_brokers/apache_hdfs_broker | ||
| gensrc | ||
| images | ||
| java-extensions | ||
| licenses | ||
| licenses-binary | ||
| test | ||
| thirdparty | ||
| tools | ||
| webroot | ||
| .clang-format | ||
| .clang-tidy | ||
| .dockerignore | ||
| .gitignore | ||
| .mergify.yml | ||
| CODE_OF_CONDUCT.md | ||
| CONTRIBUTING.md | ||
| LICENSE.txt | ||
| NOTICE.txt | ||
| README.md | ||
| build.sh | ||
| env.sh | ||
| run-be-ut.sh | ||
| run-fe-ut.sh | ||
README.md
Download | Docs | Benchmarks | Demo
Learn more 👉🏻 Introduction to StarRocks
Features
- 🚀 Native vectorized SQL engine: StarRocks adopts vectorization technology to make full use of the parallel computing power of CPU, achieving sub-second query returns in multi-dimensional analyses, which is 5 to 10 times faster than previous systems.
- 📊 Standard SQL: StarRocks supports ANSI SQL syntax (fully supported TPC-H and TPC-DS). It is also compatible with the MySQL protocol. Various clients and BI software can be used to access StarRocks.
- 💡 Smart query optimization: StarRocks can optimize complex queries through CBO (Cost Based Optimizer). With a better execution plan, the data analysis efficiency will be greatly improved.
- ⚡ Real-time update: The updated model of StarRocks can perform upsert/delete operations according to the primary key, and achieve efficient query while concurrent updates.
- 🪟 Intelligent materialized view: The materialized view of StarRocks can be automatically updated during the data import and automatically selected when the query is executed.
- ✨ Querying data in data lakes directly: StarRocks allows direct access to data from Apache Hive™, Apache Iceberg™, and Apache Hudi™ without importing.
- 🎛️ Resource management: This feature allows StarRocks to limit resource consumption for queries and implement isolation and efficient use of resources among tenants in the same cluster.
- 💠 Easy to maintain: Simple architecture makes StarRocks easy to deploy, maintain and scale out. StarRocks tunes its query plan agilely, balances the resources when the cluster is scaled in or out, and recovers the data replica under node failure automatically.
Architecture Overview
StarRocks’s streamlined architecture is mainly composed of two modules：Frontend (FE) and Backend (BE). The entire system eliminates single points of failure through seamless and horizontal scaling of FE and BE, as well as replication of metadata and data.
Resources
📚 Read the docs
| Section | Description |
|---|---|
| Deploy | Learn how to run and configure StarRocks. |
| Docs | Full documentation. |
| Blogs | StarRocks deep dive and user stories. |
❓ Get support
- Slack community: join technical discussions, ask questions, and meet other users!
- YouTube channel: subscribe to the latest video tutorials and webcasts.
- GitHub issues: report an issue with StarRocks.
Contributing to StarRocks
We welcome all kinds of contributions from the community, individuals and partners. We owe our success to your active involvement.
- See Contributing.md to get started.
- Set up StarRocks development environment:
- Understand our GitHub workflow for opening a pull request; use this PR Template when submitting a pull request.
- Pick a good first issue and start contributing.
📝 License: StarRocks is licensed under Apache License 2.0.
👥 Community Membership: Learn more about different contributor roles in StarRocks community.
Used By
This project is used by the following companies. Learn more about their use cases:
Acknowledgment
StarRocks is built upon Apache® Doris™ 0.13 in early 2020. We have recreated many important parts of the database including a full vectorized execution engine, a brand new CBO optimizer, a novel real-time update engine, and query federation for data lakes.
Today, only less than 20% of the code in StarRocks is from Apache® Doris™.