Commit Graph

22203 Commits

Author SHA1 Message Date
Yixin Luo bb7a7ace8c
[BugFix] Fix pk index cumulative compaction strategy when max_rss_rowid is same (#63277)
Signed-off-by: luohaha <18810541851@163.com>
2025-09-22 11:48:12 +08:00
Harbor Liu 19bc2cc92b
[Refactor] Add mechanism to normalize names during parsing (#63309) 2025-09-22 10:15:25 +08:00
shuming.li df70c18210
[Enhancement] Enhance iceberg incremental scan read (#63290)
Signed-off-by: shuming.li <ming.moriarty@gmail.com>
2025-09-22 10:02:48 +08:00
before-Sunrise e0ce1d4b42
[BugFix] fix non-deterministic predicate push down problem (#62827)
Signed-off-by: before-Sunrise <unclejyj@gmail.com>
2025-09-22 09:49:23 +08:00
Kevin Cai bfb83e7e9c
[BugFix] fix duplicate key table delete issue in shared-data mode (#63296)
Signed-off-by: Kevin Cai <kevin.cai@celerdata.com>
2025-09-22 09:29:03 +08:00
絵空事スピリット da9c0e84ed
[Doc] 3.5.6 Release Notes (#63348) 2025-09-22 09:28:04 +08:00
Murphy f16b69ee5b
[Enhancement] remove the adaptive zonemap creation (#63297) 2025-09-21 19:31:46 +08:00
Dan Roscigno 42055ff3da
[Doc] add video (#63330)
Signed-off-by: DanRoscigno <dan@roscigno.com>
2025-09-20 21:09:32 +08:00
meegoo af7052ff34
[Enhancement] Make BE reject multi-statement transaction stream load (#63242)
Signed-off-by: meegoo <meegoo.sr@gmail.com>
2025-09-19 10:33:40 -07:00
Dan Roscigno 064cde829d
[Doc] Add tip about TPC-DS benchmark query example (#63203)
Signed-off-by: Dan Roscigno <dan@roscigno.com>
Signed-off-by: DanRoscigno <dan@roscigno.com>
2025-09-19 09:14:16 -04:00
Murphy f6f67a1af8
[Tool] meta_tool: dump_zonemap (#63292)
Signed-off-by: Murphy <mofei@starrocks.com>
2025-09-19 16:55:59 +08:00
Yixin Luo 2981ce997a
[BugFix] fix delvec no found issue when drop tablet and queries run concurrently (#63291)
Signed-off-by: luohaha <18810541851@163.com>
2025-09-19 06:20:44 +00:00
Harbor Liu 348cd8f151
[Enhancement] Support for `IF NOT EXISTS` and `IF EXISTS` clauses to the GROUP PROVIDER (#63248)
What I'm doing:
This pull request adds support for IF NOT EXISTS and IF EXISTS clauses to the CREATE GROUP PROVIDER and DROP GROUP PROVIDER SQL statements, ensuring these operations are idempotent and error-tolerant. It also introduces comprehensive unit tests to verify the correct behavior of these clauses, and updates the parser and semantic analyzer accordingly.

Enhancements to Group Provider DDL Statements:

Added support for IF NOT EXISTS to CREATE GROUP PROVIDER and IF EXISTS to DROP GROUP PROVIDER, allowing these statements to succeed silently if the provider already exists or does not exist, respectively. The implementation includes proper error handling and DDL exception messages for cases where these clauses are not specified. [1] [2]

Updated the SQL parser (AstBuilder) to recognize and propagate the IF NOT EXISTS and IF EXISTS flags in CreateGroupProviderStmt and DropGroupProviderStmt objects.

Enhanced the semantic analyzer to respect the new flags, throwing exceptions only when appropriate (i.e., when the flags are not set and the provider's existence does not match the operation). [1] [2]

Testing Improvements:

Added a new test class GroupProviderStatementTest with comprehensive unit tests covering all scenarios for the new IF NOT EXISTS and IF EXISTS functionality, including creation, dropping, legacy constructors, and sequential operations.
Error Handling:

Wrapped the execution of dropGroupProviderStatement in the DDL statement executor with error reporting to ensure runtime exceptions are properly managed.
2025-09-19 14:01:30 +08:00
Harbor Liu cfe33249d4
[Refactor] Remove some AST class from fe-core to fe-parser (#63260)
What I'm doing:
This pull request primarily focuses on cleaning up and simplifying the AST (Abstract Syntax Tree) visitor interface in the SQL parser by removing deprecated code and unnecessary method overrides. The changes also include the removal of the deprecated AlterLoadErrorUrlClause class. These updates help reduce code redundancy and improve maintainability.

Key changes:

1. Removal of Deprecated and Redundant Code:

Deleted the deprecated AlterLoadErrorUrlClause class from the codebase, as it is no longer needed.
Removed numerous default method overrides for various visit*Statement methods in the AstVisitorExtendInterface that simply delegated to existing base methods, reducing boilerplate and simplifying the interface. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18] [19]
2. Cleanup of Unused Imports:

Removed import statements for classes that are no longer referenced due to the method removals, further streamlining the code. [1] [2]
These changes collectively make the AST visitor interface easier to maintain and extend in the future by eliminating unnecessary code and focusing on essential logic.
2025-09-19 13:50:56 +08:00
PengFei Li b9fdfcd6c0
[BugFix] Fix incompatible zonemap reuse for fast schema evolution in shared-data (#63143)
Signed-off-by: PengFei Li <lpengfei2016@gmail.com>
2025-09-19 13:49:04 +08:00
zihe.liu 1ffaebab6f
[Enhancement] Optimize trunc_to_month and trunc_to_quarter for date (#63245)
## Why I'm doing:

Given a Julian date, our current approach to obtaining `trunc_to_quarter` or `trunc_to_month` is:
1. First decode it into year, month, and day. 
2. Then, we adjust the corresponding month and day, 
3. and finally compute the new Julian date from the adjusted year, month, and day.  

However, in practice, it is unnecessary to compute a new Julian date from the adjusted values. Instead, we can directly calculate the offset between the target Julian and the current Julian based on the decoded year, month, and day.

benchmark: speedup around 50%
- trunc_to_month: https://quick-bench.com/q/fZiRaC7U1pHODHsWQuO1cvFN2qM
- trunc_to_quarter: https://quick-bench.com/q/Idw7wLB74k-5PNW2u4xcXNsoeKA

Signed-off-by: zihe.liu <ziheliu1024@gmail.com>
2025-09-19 13:46:55 +08:00
zhanghe 5c2aaaf717
[BugFix] Fix ApplyCommitTask loss in concurrent scenarios. (#60633)
Signed-off-by: edwinhzhang <edwinhzhang@tencent.com>
Signed-off-by: sevev <qiangzh95@gmail.com>
Signed-off-by: zhangqiang <qiangzh95@gmail.com>
Co-authored-by: sevev <qiangzh95@gmail.com>
2025-09-19 10:10:25 +08:00
xiangguangyxg 2fc227d19a
[BugFix] Fix BE crash in tracer when jaeger_endpoint is invalid (#63257)
## Why I'm doing:
When BE config jaeger_endpoint is invalid, BE will crash in tracer:
```
*** Aborted at 1757906071 (unix time) try "date -d @1757906071" if you are using GNU date ***
PC: @          0xbe8d0af starrocks::Tracer::start_trace_or_add_span(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
*** SIGSEGV (@0x0) received by PID 3328297 (TID 0x7f30f7d2c640) LWP(3328810) from PID 0; stack trace: ***
    @     0x7f31e6d41ee8 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x99ee7)
    @         0x109a4ae8 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*)
    @     0x7f31e6cea520 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x4251f)
    @          0xbe8d0af starrocks::Tracer::start_trace_or_add_span(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
    @          0xcc1383c starrocks::OlapTableSink::init(starrocks::TDataSink const&, starrocks::RuntimeState*)
    @          0xcc09965 starrocks::DataSink::create_data_sink(starrocks::RuntimeState*, starrocks::TDataSink const&, std::vector<starrocks::TExpr, std::allocator<starrocks::TExpr> > const&, starrocks::TPlanFragmentExecParams const&, int, starrocks::RowDescriptor const&, std::uniq@^A
    @          0xcc5475f starrocks::pipeline::FragmentExecutor::_prepare_pipeline_driver(starrocks::ExecEnv*, starrocks::pipeline::UnifiedExecPlanFragmentParams const&)
    @          0xcc55a96 starrocks::pipeline::FragmentExecutor::prepare(starrocks::ExecEnv*, starrocks::TExecPlanFragmentParams const&, starrocks::TExecPlanFragmentParams const&)
    @          0xcdea3f5 starrocks::PInternalServiceImplBase<starrocks::PInternalService>::_exec_plan_fragment_by_pipeline(starrocks::TExecPlanFragmentParams const&, starrocks::TExecPlanFragmentParams const&)
    @          0xcde9873 starrocks::PInternalServiceImplBase<starrocks::PInternalService>::_exec_plan_fragment(brpc::Controller*, starrocks::PExecPlanFragmentRequest const*)
    @          0xcde9403 starrocks::PInternalServiceImplBase<starrocks::PInternalService>::_exec_plan_fragment(google::protobuf::RpcController*, starrocks::PExecPlanFragmentRequest const*, starrocks::PExecPlanFragmentResult*, google::protobuf::Closure*)
    @          0xcb1e547 starrocks::PriorityThreadPool::work_thread(int)
    @         0x1095100b thread_proxy
    @     0x7f31e6d3cac3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
    @     0x7f31e6dce850 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x12684f)
```

Invalid jaeger_endpoint example: 
`jaeger_endpoint = http://localhost:14268`

Valid jaeger_endpoint example:
`jaeger_endpoint = localhost:14268 `

## What I'm doing:
Create a no-op tracer to protect BE from crash when jaeger_endpoint is invalid.

Signed-off-by: xiangguangyxg <xiangguangyxg@gmail.com>
2025-09-19 10:02:46 +08:00
SevenJ c014cdb711
[BugFix] to fix manual compact test case (#63194)
Signed-off-by: SevenJ <wenjun7j@gmail.com>
2025-09-19 09:56:59 +08:00
wyb d5f8c1f57d
[BugFix] Fix get max compaction score NullPointerException bug (#63268)
Signed-off-by: wyb <wybb86@gmail.com>
2025-09-18 21:20:04 +08:00
shuming.li 6a63db6441
[BugFix] Remove unused codes (#63261) 2025-09-18 19:14:59 +08:00
Youngwb 42c1bda6e2
[BugFix] Fix ranger hive service alter privilege not work (#63251) 2025-09-18 09:15:41 +00:00
zihe.liu 82e2559533
[UT] Fix timeout SQL Test (#63246)
Signed-off-by: zihe.liu <ziheliu1024@gmail.com>
2025-09-18 15:06:34 +08:00
zhangqiang dc5d828118
[BugFix] Unblock repair when source replica is marked DECOMMISSION during migration (#62942)
## Why I'm doing:
A tablet becomes unrepaired (stuck in VERSION_INCOMPLETE) after balance-triggered migration when the only valid source replica is marked DECOMMISSION.

### Scenario
1. Tablet x created on BE1 (single-replica table); ingestion starts.
2. Cluster balance migrates the tablet to BE2; old replica on BE1 becomes DECOMMISSION after migration.
3. During migration, ingestion on BE1 was still running; max committed version at that moment was 1, so the replica on BE2 is version 1.
4. Ingestion then finishes successfully on BE1 and advances its replica to version 2.
5. Replica on BE2 remains version 1 and is marked VERSION_INCOMPLETE.
6. Repair on BE2 fails because its only source (BE1) is in DECOMMISSION, so no eligible source is available → tablet cannot repair.


## What I'm doing:

Allow DECOMMISSION replicas to serve as temporary repair sources while they are healthy and accessible, until the target catches up or the replica is actually dropped.
Tighten source selection predicates: prefer latest, healthy replicas; include DECOMMISSION replicas as sources only when they are the sole candidates and have newer versions.
Ensure version reconciliation runs before finalizing decommission of the source replica.

Signed-off-by: sevev <qiangzh95@gmail.com>
2025-09-18 15:01:35 +08:00
stephen 3d394db3e6
[Enhancement] Push down ASOF JOIN condition projection to child operator (#63200)
Signed-off-by: stephen <stephen5217@163.com>
2025-09-18 12:20:52 +08:00
zhangqiang 7a67b9bdf6
[BugFix] Fix use-after-free on pipeline timer during shutdown (#63189)
Signed-off-by: sevev <qiangzh95@gmail.com>
2025-09-18 11:20:54 +08:00
meegoo 2c759218fe
[Enhancement] Update default bucket size of random distribution (#63168)
Signed-off-by: meegoo <meegoo.sr@gmail.com>
2025-09-18 10:32:03 +08:00
Harbor Liu 0c0cb63472
[Refactor] Refactor ShowGrant Executor logic and add more UT test (#63196) 2025-09-17 17:33:03 +08:00
shuming.li 7c8a29660a
[Enhancement] Optimize parsing predicates with large number of CompoundPredicates (#63139)
Signed-off-by: shuming.li <ming.moriarty@gmail.com>
2025-09-17 16:37:29 +08:00
Shawn 40f0eca6ed
[Enhancement] Emit routine load lag time (#62048)
Signed-off-by: coderfreestyle <shawn_00852@hotmail.com>
Signed-off-by: Shawn Lyu <shawn_00852@hotmail.com>
Signed-off-by: Shawn <shawn_00852@hotmail.com>
Signed-off-by: lichaoyong <lichaoyong158@gmail.com>
Signed-off-by: lichaoyong <lichaoyong121@qq.com>
Co-authored-by: wyb <wybb86@gmail.com>
Co-authored-by: lichaoyong <lichaoyong158@gmail.com>
Co-authored-by: lichaoyong <lichaoyong121@qq.com>
2025-09-17 15:12:23 +08:00
Harbor Liu 9843efbb7c
[Refactor] Add EnhancedShowStmt provides advanced feature for show statement (#63197)
What I'm doing:
This pull request refactors the handling of SQL SHOW statements to introduce a new abstract class, EnhancedShowStmt, which provides advanced query features such as filtering, sorting, and limiting results. All SHOW statement classes that support these features now extend EnhancedShowStmt instead of the original ShowStmt. The original ShowStmt class has been removed, and relevant code has been updated to use the new class. This change improves code clarity and extensibility for SHOW statements with advanced capabilities.

Key changes include:

Core Refactoring:

Introduced a new abstract class EnhancedShowStmt that extends the capabilities of SHOW statements with support for WHERE (predicate), ORDER BY, and LIMIT clauses, as well as transformation to SELECT statements. (fe/fe-core/src/main/java/com/starrocks/sql/ast/EnhancedShowStmt.java)
Removed the original ShowStmt abstract class, consolidating its logic into EnhancedShowStmt and updating the visitor interface accordingly. (fe/fe-core/src/main/java/com/starrocks/sql/ast/ShowStmt.java, fe/fe-core/src/main/java/com/starrocks/sql/ast/AstVisitorExtendInterface.java) [1] [2]
SHOW Statement Implementations:

Updated all SHOW statement classes that support advanced features (such as ShowTableStmt, ShowDbStmt, ShowColumnStmt, etc.) to extend EnhancedShowStmt instead of ShowStmt. [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12]
Analyzer and Executor Updates:

Refactored analyzer and executor logic to check for EnhancedShowStmt instead of ShowStmt, ensuring that only enhanced statements receive advanced analysis and execution logic. (fe/fe-core/src/main/java/com/starrocks/qe/StmtExecutor.java, fe/fe-core/src/main/java/com/starrocks/qe/ShowExecutor.java, fe/fe-core/src/main/java/com/starrocks/sql/analyzer/ShowStmtAnalyzer.java) [1] [2] [3] [4]
Imports and Test Adjustments:

Updated imports throughout the codebase and in tests to use EnhancedShowStmt where applicable. (fe/fe-core/src/main/java/com/starrocks/qe/ShowExecutor.java, fe/fe-core/src/main/java/com/starrocks/sql/analyzer/ShowStmtAnalyzer.java, fe/fe-core/src/test/java/com/starrocks/sql/analyzer/AnalyzeShowTest.java) [1] [2] [3]

Signed-off-by: HangyuanLiu <460660596@qq.com>
2025-09-17 14:49:57 +08:00
Drake Wang 8736a9d386
[BugFix] Fix version check failed while appling replication txn with compaction enabled (#62663)
Signed-off-by: sunny.xl <wxl24life@gmail.com>
Signed-off-by: Drake Wang <wxl24life@gmail.com>
Co-authored-by: Kevin Cai <caixh.kevin@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-09-17 14:01:41 +08:00
Harbor Liu b70c85739c
[BugFix] Fix the bug where UserProperty priority is lower than Session Variable (#63173) 2025-09-17 11:33:53 +08:00
絵空事スピリット ace222ba95
[Doc] Add references for sys.policy_references (#63183) 2025-09-17 09:49:23 +08:00
Dan Roscigno bc2d5e3d88
[Doc] Add keywords for query tuning introduction (#63204)
Signed-off-by: Dan Roscigno <dan@roscigno.com>
2025-09-17 07:55:52 +08:00
stephen 9e641d9eaf
[Feature] Introduce asof join (FE Part) (#63070)
Signed-off-by: stephen <stephen5217@163.com>
2025-09-16 19:57:33 +08:00
meegoo 6e32a078e3
[BugFix] Fix transaction write deadlock due to set error message (#62961)
Signed-off-by: meegoo <meegoo.sr@gmail.com>
2025-09-16 19:48:40 +08:00
Yixin Luo a5e2e085dc
[Enhancement] publish sst files generated during data loading and compaction (#63005)
Signed-off-by: luohaha <18810541851@163.com>
2025-09-16 11:13:30 +00:00
wyb 0925f39e61
[Enhancement] Improve fragment instance exec state report (#63132)
Signed-off-by: wyb <wybb86@gmail.com>
2025-09-16 17:04:02 +08:00
satanson ad6d9af81a
[UT] Fix german_string UT fail in ASAN mode (#63185)
Signed-off-by: satanson <ranpanf@gmail.com>
2025-09-16 08:09:56 +00:00
0aKarmA_骅文 1859c8263c
[Enhancement] Support type hints and dataclass compatibility for Python under 3.10 (#63131)
Why I'm doing:
starrocks-python-client support most of the python3 version, but there are some new features that is not compatibility for Python under 3.10

What I'm doing:
use Union for supporting type hints in python under 3.10
not setting kw_only for dataclasses in python under 3.10
2025-09-16 15:43:02 +08:00
Seaven 0682fbf3a7
[BugFix] Fix cte reuse plan extract error (#62784)
Signed-off-by: Seaven <seaven_7@qq.com>
2025-09-16 15:35:49 +08:00
Misha 1698c0910a
[Feature] configurable cipher-suite filtering (#62603)
Signed-off-by: sviatahorau <mikhail.sviatahorau@deep.bi>
2025-09-16 15:05:20 +08:00
zhangqiang 3c9b9c5b3a
[BugFix] Fix null exception during remove expired load job (#63042)
Signed-off-by: sevev <qiangzh95@gmail.com>
2025-09-16 14:13:54 +08:00
絵空事スピリット dfb587126c
[Doc] Fix Stream Load Param Desc (#63172)
Signed-off-by: 絵空事スピリット <wanglichen@starrocks.com>
2025-09-16 14:06:22 +08:00
He Zhao a79d85173b
[Enhancement] Implement ProfileActionV2 and QueryDetailActionV2 to obtain query information across all FEs (#61345)
Signed-off-by: zhaohehuhu <luoyedeyi@163.com>
2025-09-16 13:47:50 +08:00
Harbor Liu 4bb51cbd9e
[Refactor] Refactors the Frontend (FE) module structure versioning management (#63086) 2025-09-16 10:42:43 +08:00
shuming.li d96e17baa8
[UT] [BugFix] Fix FineGrainedRangePredicateRule rule bug (#63148)
Signed-off-by: shuming.li <ming.moriarty@gmail.com>
2025-09-16 10:32:51 +08:00
絵空事スピリット 960bc35302
[Doc] Update Colocate Join Principles (#63153)
Signed-off-by: 絵空事スピリット <wanglichen@starrocks.com>
2025-09-16 10:00:49 +08:00
PengFei Li e0cf9111e1
[BugFix] Fix secondary replicas can continue waiting because of wrong timestamp (#62805)
Signed-off-by: PengFei Li <lpengfei2016@gmail.com>
2025-09-15 18:59:08 -07:00