Why I'm doing:
jindo sdk was removed from FE in pr #46297
What I'm doing:
hadoop-aliyun is already in broker-core/pom.xml, remove jindo sdk from broker
remove useless hadoop-cos dependency description in pom.xml
Signed-off-by: wyb <wybb86@gmail.com>
If there is no properties in the broker load command, it should roll
back to read the core-site.xml to apply.
Signed-off-by: miomiocat <284487410@qq.com>
Currently, we will decompress jindosdk-4.6.2.tar.gz when building broker,
this is very time consuming.
In this pr, we optimize this procedure, jindosdk-4.6.2.tar.gz will
only be decompressed in the first time it downloads.
Signed-off-by: xyz <a997647204@gmail.com>
In this pr, we support broker load from azure abfs.
We remove the check of path prefix in getFileSystem, if the path prefix is not hdfs/s3a/oss/cos...,
we will try to create a generic file system. The reason why we can do this is because hadoop/s3 SDK
is compatible with all file/object storage system.
Then if in future, if there is a request for supporting new file system,
we only need to add the settings in hdfs-site.xml, core-site.xml of BE/FE, and the jar lib in the specific place.
Signed-off-by: xyz <a997647204@gmail.com>
In some cases, broker may hang when closing oss file,
this will lead to all following open/write requests stuck.
Here we disable the check_client_expiration_task temporarily,
after oss bug is fixed, we can turn on the flag again.
This problem may be introduced by #15606
The current broker has two types of kerberos token, one is the global, one unique to each HDFS client, access to HDFS will use each client unique token, and access to other components (such as kms) use the global. HDFS client unique token is not automatically refreshed, but HDFS client in the broker has an expiration time, if more than 5min is not used, the HDFS client will be destroyed. If there is a request the next time, a new client will be created, the kerberos login will be executed again, this is actually an indirect refresh token. But if there are access continuously, HDFS client will never be rebuilt, then the token will not be refreshed, it will time out after 1 day.
To fix this bug, destroy the client before token is expired.
Add a new broker config option `"fs.s3a.path.style.access"` to allow toggling the path style. The option value must either be "true" or "false". Default value is "false".
```
WITH BROKER "broker1"
(
"fs.s3a.access.key" = "myaccesskey",
"fs.s3a.secret.key" = "mysecretkey",
"fs.s3a.endpoint" = "host.testcontainers.internal:49651"
"fs.s3a.path.style.access" = "true"
)
```
Kevin Cai
Binglin Chang
gengjun-git
wyb
Felix Li
Vikas Attiguppa
ricky
Letian Jiang
Zhenxiao Luo
BlankLin
miomiocat
jay
xyz
zuyu
lichaoyong
Jiacheng Yang