Commit Graph

1114 Commits

Author SHA1 Message Date
lyw b05e7ded50
[Feature](function) support sec_to_time (#62797) 2025-10-11 17:28:48 +08:00
Yixin Luo ee93eaef39
[Enhancement] Improve delvec file cleanup logic to properly handle pk index references (#63384)
Signed-off-by: luohaha <18810541851@163.com>
2025-09-26 12:13:16 +00:00
stephen 9e641d9eaf
[Feature] Introduce asof join (FE Part) (#63070)
Signed-off-by: stephen <stephen5217@163.com>
2025-09-16 19:57:33 +08:00
satanson b956f11544
[Enhancement] Full sort use german string for comparison (#62929)
Signed-off-by: satanson <ranpanf@gmail.com>
2025-09-15 20:23:40 +08:00
Murphy 3688bc4bd4
[BugFix] fix hour_from_unixtime rule (#63006) 2025-09-12 13:57:33 +08:00
He Zhao 0035df76db
[Enhancement] add catalog and queryId info in ShowProcessList (#62552)
Signed-off-by: zhaohehuhu <luoyedeyi@163.com>
2025-09-11 11:04:48 +00:00
Yixin Luo ab2730d86a
[Enhancement] generate cloud native pk index sst files when tablet write (#62768)
Signed-off-by: luohaha <18810541851@163.com>
2025-09-10 20:31:39 +08:00
Rohit Satardekar 6b3d00852b
[Enhancement] add command to show the dropped meta information that can be recovered (#51007)
Signed-off-by: Rohit Satardekar <rohitrs1983@gmail.com>
2025-09-05 10:23:19 +08:00
Xu Bai e970bfd5de
[Feature] Introduce Variant column definition (#61099)
Signed-off-by: Xu Bai <xuba@cisco.com>
2025-09-03 13:22:19 +08:00
meegoo 7e26e9ee0d
[Enhancement] Support multi statements transaction (part2) (#62019)
Signed-off-by: meegoo <meegoo.sr@gmail.com>
2025-09-02 08:40:06 +08:00
Gavin 63ff16d528
[BugFix] Change the `tuple_id` field in `TIcebergTableSink` to optional to fix the compatibility issues with historical versions. (#62593)
Signed-off-by: GavinMar <yangguansuo@starrocks.com>
2025-09-01 16:32:57 +00:00
lyw 2cc7a0547d
[Enhancement] fix null_or_empty can support array type (#62207)
Signed-off-by: liyanwei14 <liyanwei1@jd.com>
Signed-off-by: stdpain <34912776+stdpain@users.noreply.github.com>
Co-authored-by: stdpain <34912776+stdpain@users.noreply.github.com>
2025-08-29 11:57:02 +08:00
eyes_on_me 6c220d0dac
[Enhancement] support expr reuse in outer join where predicates (#62139)
Signed-off-by: silverbullet233 <3675229+silverbullet233@users.noreply.github.com>
2025-08-29 11:19:32 +08:00
hmx 7a3e5e2cef
[Enhancement] add ExecState into /current_queries cmd result to distinguish running/pending query (#62261)
Signed-off-by: MatthewH00 <1639097204@qq.com>
2025-08-27 14:02:07 +08:00
zihe.liu 3b5872dfad
[Enhancement] Use linear-chained to optimize hash join (#61429)
Signed-off-by: zihe.liu <ziheliu1024@gmail.com>
2025-08-25 16:40:11 +08:00
yan zhang 8bbb2a3305
[Enhancement] short circuit optimization on select limit case (on Scan Node) (#62188)
Signed-off-by: yan zhang <dirtysalt1987@gmail.com>
2025-08-25 11:01:44 +08:00
Gavin 6172f84e4f
[Enhancement] Optimize the iceberg sink local sorting based on the spill partition writer (#62096)
Signed-off-by: GavinMar <yangguansuo@starrocks.com>
2025-08-23 00:20:56 +08:00
zombee0 96a37ca97b
[BugFix]runtime filter partition are bucket aware (#62191)
Signed-off-by: zombee0 <ewang2027@gmail.com>
2025-08-22 11:21:05 +08:00
wyb c1e1b1fdb4
[Enhancement] Improve fe tablet schedules system table (#62073)
Signed-off-by: wyb <wybb86@gmail.com>
2025-08-19 23:22:38 +08:00
wangshifa.1 d0c5c0cc33
[Feature] Add date function of weekday (#61865)
Why I'm doing:
Fixes #61695

What I'm doing:
Fixes #61695 , add WEEKDAY function similar to the dayofweek_iso function, but it returns numbers from 0 to 6 (corresponding to Monday to Sunday)

Signed-off-by: wangshifa.1 <wangshifa.1@jd.com>
2025-08-19 16:16:45 +08:00
Hongkun Xu 72fe131ace
[Feature] Support MATCH_ALL operator (#61906)
Signed-off-by: Hongkun Xu <xuhongkun666@163.com>
2025-08-19 15:46:49 +08:00
Gavin 14a8c1415a
[Enhancement] Introduce a connector partition chunk writer to support spilling chunk data for iceberg table sink. (#61963)
Signed-off-by: GavinMar <yangguansuo@starrocks.com>
2025-08-19 11:38:35 +08:00
Yixin Luo 463b29f8bd
[BugFix] avoid get file size in report tablet stat thread (#61901)
Signed-off-by: luohaha <18810541851@163.com>
2025-08-18 15:35:39 +08:00
srlch 441f126793
[Enhancement] VacuumFull Implementation (#61602)
Signed-off-by: srlch <linzichao@starrocks.com>
Co-authored-by: Connor Brennan <cbrennan@pinterest.com>
2025-08-18 10:37:20 +08:00
Murphy 413d6b9651
[Enhancement] support encode_sort_key function (#61781)
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2025-08-15 15:26:22 +08:00
meegoo 84243343b8
[Feature] Support multi statement transaction (part1) - stream load (#61362)
Signed-off-by: meegoo <meegoo.sr@gmail.com>
2025-08-14 10:17:58 +08:00
stdpain 5f6cdde3a0
[Enhancement] support group by compressed key (#61632)
Signed-off-by: stdpain <drfeng08@gmail.com>
2025-08-14 09:32:20 +08:00
xiangguangyxg 4ef246685f
[Enhancement] Separate path id from physical partition id (#61854)
Signed-off-by: xiangguangyxg <xiangguangyxg@gmail.com>
2025-08-13 17:01:36 +08:00
Murphy 561f99eeac
[Enhancement] Implement function json_contains (#61403)
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2025-08-13 10:42:27 +08:00
Kevin Cai e5116ba026
[Tool] separate gen_build_version.py, only build java for fe and cpp for be (#61775)
gen_build_version.py allows generating only java or cpp code
skip maven clean phase in fe build

Signed-off-by: Kevin Cai <kevin.cai@celerdata.com>
2025-08-12 12:29:13 +08:00
Murphy cc7c240861
[Enhancement] collect global dict for flatjson (#61680)
Signed-off-by: Murphy <mofei@starrocks.com>
2025-08-12 11:00:56 +08:00
Hongkun Xu f8e7371c48
[Feature] Support MATCH_ANY operator (#60986)
Signed-off-by: Hongkun Xu <xuhongkun666@163.com>
2025-08-11 22:24:13 +08:00
Murphy bf792a6455
[BugFix] fix builtin_function fuzzy test (#61530)
What I'm doing:
bits_function: the implementation is wrong
change the static DCHECK to dynamic argument validation for some functions
fix some type mapping error in the logical_type.cpp

Signed-off-by: Murphy <mofei@starrocks.com>
2025-08-11 13:39:06 +08:00
SevenJ 677dfcdef0
[Feature] support iceberg manual compaction (#61240)
Signed-off-by: SevenJ <wenjun7j@gmail.com>
2025-08-08 16:45:16 +08:00
PengFei Li e8982e7797
[Enhancement] Add prepared_timeout configuration for transaction stream load (#61539)
## Why I'm doing:

Currently, users can only configure the timeout for prepared transactions through the global FE configuration `prepared_transaction_default_timeout_second`. This approach lacks flexibility as it requires all transactions to use the same timeout value. Users need the ability to specify different timeout values for different transactions based on their specific requirements, especially in production environments where precise control over transaction lifecycle is crucial.

## What I'm doing:

This PR adds support for the `prepared_timeout` configuration in transaction stream load, allowing users to specify a timeout period for transactions from PREPARED to COMMITTED state. The implementation includes:

**Backend Changes:**
- Added `HTTP_PREPARED_TIMEOUT` constant in `be/src/http/http_common.h`
- Extended `StreamLoadContext` with `prepared_timeout_second` field
- Modified `TransactionMgr` to parse `prepared_timeout` HTTP header
- Updated `StreamLoadExecutor::prepare_txn` to pass timeout to FE
- Enhanced `TransactionState` with `preparedTimeoutMs` field and timeout detection logic
- Updated Thrift interface `TLoadTxnCommitRequest` with `prepared_timeout_second` field

**Frontend Changes:**
- Modified `TransactionLoadAction` to parse `prepared_timeout` parameter
- Updated `TransactionState` with `setPreparedTimeAndTimeout` method
- Enhanced `DatabaseTransactionMgr` and `GlobalTransactionMgr` to handle prepared timeout
- Updated transaction timeout detection logic in `TransactionState::isTimeout`

**Usage Example:**
```bash
# Begin transaction
curl --location-trusted -u root: -H "label:test_txn" -H "timeout:300" -H "db:test_db" -H "table:test_table" \
    -XPOST http://fe_host:8030/api/transaction/begin

# Load data
curl --location-trusted -u root: -H "label:test_txn" -H "db:test_db" -H "table:test_table" \
    -d '1' -XPUT http://fe_host:8030/api/transaction/load

# Prepare transaction with custom timeout (60 seconds)
curl --location-trusted -u root: -H "label:test_txn" -H "db:test_db" \
    -H "prepared_timeout:60" -XPOST http://fe_host:8030/api/transaction/prepare

# Commit transaction
curl --location-trusted -u root: -H "label:test_txn" -H "db:test_db" \
    -XPOST http://fe_host:8030/api/transaction/commit

# View transaction details including PreparedTime and PreparedTimeoutMs
SHOW TRANSACTION WHERE id = <transaction_id>;
+---------------+--------+---------------+-------------------+-------------------+---------------------+---------------------+---------------------+---------------------+---------------------+--------+--------------------+------------+-----------+-------------------+--------+
| TransactionId | Label  | Coordinator   | TransactionStatus | LoadJobSourceType | PrepareTime         | PreparedTime        | CommitTime          | PublishTime         | FinishTime          | Reason | ErrorReplicasCount | ListenerId | TimeoutMs | PreparedTimeoutMs | ErrMsg |
+---------------+--------+---------------+-------------------+-------------------+---------------------+---------------------+---------------------+---------------------+---------------------+--------+--------------------+------------+-----------+-------------------+--------+
| 1633          | test_txn | BE: 127.0.0.1 | VISIBLE           | BACKEND_STREAMING | 2025-08-03 11:02:54 | 2025-08-03 11:03:10 | 2025-08-03 11:03:14 | 2025-08-03 11:03:14 | 2025-08-03 11:03:14 |        | 0                  | [12237]    | 300000    | 60000             |        |
+---------------+--------+---------------+-------------------+-------------------+---------------------+---------------------+---------------------+---------------------+---------------------+--------+--------------------+------------+-----------+-------------------+--------+
```

**Documentation:**
- Updated `Stream_Load_transaction_interface.md` with `prepared_timeout` usage instructions
- Modified `SHOW_TRANSACTION.md` to document new `PreparedTime` and `PreparedTimeoutMs` fields
- Added version information indicating support from 4.0.0 onwards

The feature provides backward compatibility by using the FE configuration `prepared_transaction_default_timeout_second` as the default value when `prepared_timeout` is not specified.

Signed-off-by: PengFei Li <lpengfei2016@gmail.com>
Signed-off-by: 絵空事スピリット <wanglichen@starrocks.com>
Co-authored-by: 絵空事スピリット <wanglichen@starrocks.com>
2025-08-08 09:55:08 +08:00
wyb 0bca11047d
[Enhancement] Reorder tablet_balanced column of partitions_meta system table for better compatibility (#61665)
Signed-off-by: wyb <wybb86@gmail.com>
2025-08-07 18:52:11 +08:00
Murphy 0a9618db4c
[Feature] implement json_remove (#61394)
Signed-off-by: Murphy <mofei@starrocks.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2025-08-07 14:50:13 +08:00
liubotao 44c52750a2
[Enhancement] Shared Data Mode Support Flat Json parameters table-level configs (#61160)
Signed-off-by: liubotao <316945435@qq.com>
2025-08-07 14:08:47 +08:00
wyb 3caf4c6f5d
[Enhancement] Show tablet distribution balance statistic (#61549)
Signed-off-by: wyb <wybb86@gmail.com>
2025-08-07 10:57:50 +08:00
srlch 3e1ff80062
[Enhancement] Support System table for dynamic tablet jobs (#61152)
Signed-off-by: srlch <linzichao@starrocks.com>
2025-08-07 09:54:32 +08:00
zombee0 f9d77d4014
[Enhancement]exchange node support bucket aware execution (#61554)
Signed-off-by: zombee0 <ewang2027@gmail.com>
2025-08-05 15:39:57 +08:00
Mathieu Baurin a5dbbc17cd
[Feature] Add format_bytes function for human-readable byte formatting (#61535)
Signed-off-by: Mathieu Baurin <1mathieu.baurin@gmail.com>
Signed-off-by: stdpain <34912776+stdpain@users.noreply.github.com>
Co-authored-by: stdpain <34912776+stdpain@users.noreply.github.com>
2025-08-05 12:07:38 +08:00
xiangguangyxg 6ff7a42f75
[Feature] Implement cross publish version in FE for tablet splitting and merging (#60944)
Signed-off-by: xiangguangyxg <xiangguangyxg@gmail.com>
2025-08-04 20:13:36 +08:00
Murphy e88244ec37
[Enhancement] FlatJSON-V2 part 1: BE code (#61447)
Signed-off-by: Murphy <mofei@starrocks.com>
2025-08-04 19:21:22 +08:00
zombee0 48c3b47415
[Enhancement]scan range schedule for bucket-aware execution (#61424)
Signed-off-by: zombee0 <ewang2027@gmail.com>
2025-08-01 16:26:43 +08:00
Murphy af49488e6f
[UT] Fuzz test built-in functions with type coverage (#61303)
Signed-off-by: Murphy <mofei@starrocks.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
2025-07-29 11:20:47 +00:00
Yixin Luo 0f1deef421
[BugFix] fix missing partition id in combine txnlog (#61207)
Signed-off-by: luohaha <18810541851@163.com>
2025-07-29 16:25:12 +08:00
zombee0 e29df24895
[Enhancement]bucket property to do bucket aware execution (#61288)
## Why I'm doing:

## What I'm doing:
the overall design is waiting to review, but some basic functions are ready and can be shipped.

Fixes #61287
Signed-off-by: zombee0 <ewang2027@gmail.com>
2025-07-29 15:49:02 +08:00
stdpain 70a7f618d5
[Refactor] Refactor scalar function registration to speed up compilation (#61358)
Signed-off-by: stdpain <drfeng08@gmail.com>
2025-07-29 09:45:15 +08:00
starrocks-xupeng b0f5cbbbb1
[Enhancement] add segment write time in lake compaction (#60891)
Signed-off-by: starrocks-xupeng <xupeng@starrocks.com>
2025-07-28 17:35:11 +08:00