lanyuanxiaoyao/hudi
1
0
Fork 0
Code Activity

Commit Graph

  • 8869b3b418 [HUDI-1902] Clean the corrupted files generated by FlinkMergeAndReplaceHandle (#2949) Danny Chan 2021-05-14 15:43:37 +08:00
  • 12443e4187 [HUDI-1446] Support skip bootstrapIndex's init in abstract fs view init (#2520) xoln ann 2021-05-14 12:29:26 +08:00
  • ad77cf42ba [HUDI-1900] Always close the file handle for a flink mini-batch write (#2943) Danny Chan 2021-05-14 10:25:18 +08:00
  • 6f7ff7e8ca [HUDI-1722]Fix hive beeline/spark-sql query specified field on mor table occur NPE (#2722) xiarixiaoyao 2021-05-12 20:52:37 +08:00
  • b98c9ab439 [HUDI-1895] Close the file handles gracefully for flink write function to avoid corrupted files (#2938) Danny Chan 2021-05-12 18:44:10 +08:00
  • 5a8b2a4f86 [HUDI-1768] add spark datasource unit test for schema validate add column (#2776) lw0090 2021-05-12 04:49:18 +08:00
  • be9db2c4f5 [HUDI-1055] Remove hardcoded parquet in tests (#2740) TeRS-K 2021-05-11 13:01:45 -04:00
  • ac72470e10 [HUDI-1851] Adding test suite long running automate scripts for docker (#2880) Sivabalan Narayanan 2021-05-11 04:26:01 -04:00
  • 7a5af806cf [HUDI-1818] Validate required fields for Flink HoodieTable (#2930) hiscat 2021-05-11 11:11:19 +08:00
  • 42ec7e30d7 [HUDI-1890] FlinkCreateHandle and FlinkAppendHandle canWrite should always return true (#2933) Danny Chan 2021-05-11 09:14:51 +08:00
  • aa398f77f1 [HUDI-1789] Support reading older snapshots (#2809) jsbali 2021-05-11 03:56:49 +05:30
  • 8a48d16e41 [HUDI-1707] Reduces log level for too verbose messages from info to debug level. (#2714) Volodymyr Burenin 2021-05-10 09:16:02 -05:00
  • 511ac4881d [MINOR] optimize FilePathUtils (#2931) hiscat 2021-05-10 21:47:56 +08:00
  • c1b331bcff [HUDI-1886] Avoid to generates corrupted files for flink sink (#2929) Danny Chan 2021-05-10 10:43:03 +08:00
  • bfbf993cbe [HUDI-1878] Add max memory option for flink writer task (#2920) Danny Chan 2021-05-08 14:27:56 +08:00
  • 2c5a661a64 [HUDI-1759] Save one connection retry to hive metastore when hiveSyncTool run with useJdbc=false (#2759) li36909 2021-05-08 06:30:26 +08:00
  • 528f4ca988 [HUDI-1880] Support streaming read with compaction and cleaning (#2921) Danny Chan 2021-05-07 20:04:35 +08:00
  • 0a5863939b [HUDI-1821] Remove legacy code for Flink writer (#2868) hiscat 2021-05-07 10:58:49 +08:00
  • 0284cdecce [HUDI-1876] wiring in Hadoop Conf with AvroSchemaConverters instantiation (#2914) Sivabalan Narayanan 2021-05-06 00:31:44 -04:00
  • 1db904a12e [HUDI-1718] When query incr view of mor table which has Multi level partitions, the query failed (#2716) xiarixiaoyao 2021-05-05 12:34:20 +08:00
  • c5220b96e9 [HUDI-1781] Fix Flink streaming reader throws ClassCastException (#2900) dijie 2021-05-01 19:13:15 +08:00
  • ea14d687da [HUDI-1852] Add SCHEMA_REGISTRY_SOURCE_URL_SUFFIX and SCHEMA_REGISTRY_TARGET_URL_SUFFIX property (#2884) Nick Young 2021-05-01 10:02:00 +08:00
  • 3418a92de8 [HUDI-1620] Fix Metrics UT (#2894) Raymond Xu 2021-04-30 11:20:41 -07:00
  • 46de9e0f3f [HUDI-1810] Fix azure setting for integ tests (#2889) Raymond Xu 2021-04-30 11:17:36 -07:00
  • faf3785a2d [HUDI-1811] Fix TestHoodieRealtimeRecordReader (#2873) Raymond Xu 2021-04-30 11:16:55 -07:00
  • 929eca43fe [HUDI-1817] Fix getting incorrect partition path while using incr query by spark-sql (#2858) xiarixiaoyao 2021-04-30 14:57:52 +08:00
  • 6848a683bd [HUDI-1867] Streaming read for Flink COW table (#2895) Danny Chan 2021-04-29 20:44:45 +08:00
  • 6e9c5dd765 [HUDI-1863] Add rate limiter to Flink writer to avoid OOM for bootstrap (#2891) Danny Chan 2021-04-29 20:32:10 +08:00
  • c9bcb5e33f [HUDI-1845] Exception Throws When Sync Non-Partitioned Table To Hive With MultiPartKeysValueExtractor (#2876) pengzhiwei 2021-04-29 10:11:46 +08:00
  • 3ca9030256 [HUDI-1858] Fix cannot create table due to jar conflict (#2886) dijie 2021-04-28 14:10:04 +08:00
  • 386767693d [HUDI-1833] rollback pending clustering even if there is greater commit (#2863) satishkotha 2021-04-27 14:21:42 -07:00
  • e4fd195d9f [MINOR] Refactor method up to parent-class (#2822) Roc Marshal 2021-04-27 21:32:32 +08:00
  • 2999586509 [HUDI-1690] use jsc union instead of rdd union (#2872) satishkotha 2021-04-26 23:35:01 -07:00
  • 63fa2b6186 [HUDI-1836] Logging consuming instant to StreamReadOperator#processSplits (#2867) hiscat 2021-04-27 14:00:59 +08:00
  • 5be3997f70 [HUDI-1841] Tweak the min max commits to keep when setting up cleaning retain commits for Flink (#2875) Danny Chan 2021-04-27 10:58:06 +08:00
  • 9bbb458e88 [MINOR] Remove redundant method-calling. (#2881) Roc Marshal 2021-04-27 09:34:09 +08:00
  • f4e3b94971 [HUDI-1742] Improve table level config priority for HoodieMultiTableDeltaStreamer (#2744) Nick Young 2021-04-26 22:05:06 +08:00
  • d047e91d86 [HUDI-1837] Add optional instant range to log record scanner for log (#2870) Danny Chan 2021-04-26 16:53:18 +08:00
  • 3e4fa170cf [HUDI-1835] Fixing kafka native config param for auto offset reset (#2864) Sivabalan Narayanan 2021-04-25 12:16:09 -04:00
  • 1b27259b53 [HUDI-1844] Add option to flush when total buckets memory exceeds the threshold (#2877) Danny Chan 2021-04-25 23:06:53 +08:00
  • a5789c4067 [HUDI-1829] Use while loop instead of recursive call in MergeOnReadInputFormat to avoid StackOverflow (#2862) Danny Chan 2021-04-23 09:59:36 +08:00
  • a1e636dc6b [HUDI-1551] Add support for BigDecimal and Integer when partitioning based on time. (#2851) Chanh Le 2021-04-22 21:56:20 +08:00
  • 4a3431866d [HUDI-1746] Added support for replace commits in commit showpartitions, commit show_write_stats, commit showfiles (#2678) jsbali 2021-04-21 23:01:35 +05:30
  • b31c520c66 [HUDI-1714] Added tests to TestHoodieTimelineArchiveLog for the archival of compl… (#2677) jsbali 2021-04-21 22:57:43 +05:30
  • c24d90d25a [MINOR] Expose the detailed exception object (#2861) vinoyang 2021-04-21 22:41:42 +08:00
  • cc81ddde01 [HUDI-1812] Add explicit index state TTL option for Flink writer (#2853) hiscat 2021-04-21 20:13:30 +08:00
  • ac3589f006 [HUDI-1814] Non partitioned table for Flink writer (#2859) Danny Chan 2021-04-21 20:07:27 +08:00
  • aacb8be521 [HUDI-1415] Read Hoodie Table As Spark DataSource Table (#2283) pengzhiwei 2021-04-21 05:21:38 +08:00
  • 3253079507 [HUDI-1764] Add Hudi-CLI support for clustering (#2773) Jintao Guan 2021-04-20 09:46:42 -07:00
  • d6d52c6063 [HUDI-1809] Flink merge on read input split uses wrong base file path for default merge type (#2846) Danny Chan 2021-04-20 21:27:09 +08:00
  • 9a288ccbeb [MINOR] Added metric reporter Prometheus to HoodieBackedTableMetadataWriter (#2842) Sebastian Bernauer 2021-04-20 01:04:59 +02:00
  • 6b4b878d08 [HUDI-1744] rollback fails on mor table when the partition path hasn't any files (#2749) li36909 2021-04-20 06:44:11 +08:00
  • d21753d903 [HUDI-1802] Timeline Server Bundle need to include com.esotericsoftware package (#2835) Thinking Chen 2021-04-20 00:27:58 +08:00
  • ec2334ceac [HUDI-1716]: Resolving default values for schema from dataframe (#2765) Aditya Tiwari 2021-04-19 19:35:20 +05:30
  • dab5114f16 [HUDI-1804] Continue to write when Flink write task restart because of container killing (#2843) Danny Chan 2021-04-19 19:43:41 +08:00
  • f7b6b68063 [MINOR][hudi-sync] Fix typos (#2844) Roc Marshal 2021-04-19 16:27:13 +08:00
  • 4e050cc2ba [MINOR] Add jackson module to presto bundle (#2816) satishkotha 2021-04-17 13:26:07 -07:00
  • 1d53d6e6c2 [HUDI-1803] Support BAIDU AFS storage format in hudi (#2836) Xu Guang Lv 2021-04-16 16:43:14 +08:00
  • 62b8a341dd [HUDI-1792] flink-client query error when processing files larger than 128mb (#2814) hj2016 2021-04-16 13:59:19 +08:00
  • b6d949b48a [HUDI-1801] FlinkMergeHandle rolling over may miss to rename the latest file handle (#2831) Danny Chan 2021-04-16 11:40:53 +08:00
  • 191470d1fc [HUDI-1797] Remove the com.google.guave jar from hudi-flink-bundle to avoid conflicts. (#2828) MINCWANG 2021-04-15 15:16:33 +08:00
  • 6d1aec604f [HUDI-1798] Flink streaming reader should always monitor the delta commits files (#2825) hiscat 2021-04-15 13:50:17 +08:00
  • 62bb9e10d9 [Hotfix][utilities] Optimized codes (#2821) Roc Marshal 2021-04-15 09:40:14 +08:00
  • 8d29863c86 [HUDI-1615] Fixing usage of NULL schema for delete operation in HoodieSparkSqlWriter (#2777) Sivabalan Narayanan 2021-04-14 03:35:39 -04:00
  • ab4a7b0b4a [HUDI-1788] Insert overwrite (table) for Flink writer (#2808) Danny Chan 2021-04-14 10:23:37 +08:00
  • 65844a8d29 [HUDI-1720] Fix RealtimeCompactedRecordReader StackOverflowError (#2721) xiarixiaoyao 2021-04-13 18:23:26 +08:00
  • e16d31dce2 [HUDI-1787] Remove the rocksdb jar from hudi-flink-bundle (#2807) hiscat 2021-04-13 10:31:16 +08:00
  • 1ff99ca7d7 [HUDI-1786] Add option for merge max memory (#2805) Danny Chan 2021-04-12 17:03:58 +08:00
  • 040756d8c0 [HUDI-1785] Move OperationConverter to hudi-client-common for code reuse (#2798) wangxianghu 2021-04-12 16:22:33 +08:00
  • 1da16dfd2e [HUDI-1784] Added print detailed stack log when hbase connection error (#2799) hj2016 2021-04-12 13:46:06 +08:00
  • f3777f44fe [MINOR] Remove unused imports and some other checkstyle issues (#2800) wangxianghu 2021-04-11 21:42:34 +08:00
  • b554835053 [MINOR] fix typo. (#2804) Roc Marshal 2021-04-11 10:31:07 +08:00
  • 8d4a7fe33e [HUDI-1783] Support Huawei Cloud Object Storage (#2796) xiarixiaoyao 2021-04-10 13:02:11 +08:00
  • 6786581c48 [HUDI-1775] Add option for compaction parallelism (#2785) Danny Chan 2021-04-09 13:46:19 +08:00
  • 08e82c469c [HUDI-1762] Added HiveStylePartitionExtractor to support Hive style partitions (#2769) Vinoth Govindarajan 2021-04-08 22:00:11 -07:00
  • cf3d2e21eb [MINOR] Update doap with 0.8.0 release (#2772) Gary Li 2021-04-08 08:06:13 -07:00
  • 5b3608f149 [HUDI-1778] Add setter to CompactionPlanEvent and CompactionCommitEvent to have better SE/DE performance for Flink (#2789) hiscat 2021-04-08 19:40:37 +08:00
  • ecdbd2517f [HUDI-699] Fix CompactionCommand and add unit test for CompactionCommand (#2325) hongdd 2021-04-08 15:35:33 +08:00
  • 18459d4045 [MINOR] Some unit test code optimize (#2782) Simon 2021-04-08 13:35:03 +08:00
  • 3a926aacf6 [HUDI-1773] HoodieFileGroup code optimize (#2781) hiscat 2021-04-07 18:16:03 +08:00
  • f4f9dd9d83 [HUDI-1772] HoodieFileGroupId compareTo logical error(fileId self compare) (#2780) hiscat 2021-04-07 18:10:38 +08:00
  • dadd081d45 [HUDI-1751] DeltaStreamer print many unnecessary warn log (#2754) li36909 2021-04-07 15:47:03 +08:00
  • d035fcbb3c [HUDI-1767] Add setter to HoodieKey and HoodieRecordLocation to have better SE/DE performance for Flink (#2779) hiscat 2021-04-07 14:13:31 +08:00
  • 8527590772 [HUDI-1750] Fail to load user's class if user move hudi-spark-bundle jar into spark classpath (#2753) li36909 2021-04-07 10:33:32 +08:00
  • e692c704da [MINOR] Fix deprecated build link for travis (#2778) Harshit Mittal 2021-04-06 17:57:10 -07:00
  • 9c369c607d [HUDI-1757] Assigns the buckets by record key for Flink writer (#2757) Danny Chan 2021-04-06 19:06:41 +08:00
  • 920537cac8 [HUDI-1749] Clean/Compaction/Rollback command maybe never exit when operation fail (#2752) li36909 2021-04-06 14:23:15 +08:00
  • e970e1f483 [HUDI-1696] add apache commons-codec dependency to flink-bundle explicitly (#2758) Harshit Mittal 2021-04-01 23:07:30 -07:00
  • 94a5e72f16 [HUDI-1737][hudi-client] Code Cleanup: Extract common method in HoodieCreateHandle & FlinkCreateHandle (#2745) Roc Marshal 2021-04-02 11:39:05 +08:00
  • 684622c7c9 [HUDI-1591] Implement Spark's FileIndex for Hudi to support queries via Hudi DataSource using non-globbed table path and partition pruning (#2651) pengzhiwei 2021-04-02 02:12:28 +08:00
  • 9804662bc8 [HUDI-1738] Emit deletes for flink MOR table streaming read (#2742) Danny Chan 2021-04-01 15:25:31 +08:00
  • fe16d0de7c [MINOR] Delete useless UpsertPartitioner for flink integration (#2746) vinoyang 2021-03-31 16:36:42 +08:00
  • aa0da72c59 Preparation for Avro update (#2650) Sebastian Bernauer 2021-03-31 06:50:17 +02:00
  • 8bc65b9318 [HUDI-1731] Rename UpsertPartitioner in hudi-java-client (#2734) leo-Iamok 2021-03-31 11:06:04 +08:00
  • 3cab928b50 [HUDI-1735] Add hive-exec dependency for hudi-examples (#2737) vinoyang 2021-03-30 21:35:16 +08:00
  • 050626ad6c [MINOR] Add Missing Apache License to test files (#2736) Gary Li 2021-03-29 07:17:23 -07:00
  • e069b64e10 [HOTFIX] fix deploy staging jars script garyli1019 2021-03-28 22:10:00 +08:00
  • 4db970dc8a [HOTFIX] Disable ITs for Spark3 and scala2.12 (#2733) Gary Li 2021-03-28 01:07:57 -07:00
  • 452f5e2d66 [HOTFIX] close spark session in functional test suite and disable spark3 test for spark2 (#2727) Gary Li 2021-03-26 20:58:29 -07:00
  • d415d45416 [HUDI-1729] Asynchronous Hive sync and commits cleaning for Flink writer (#2732) Danny Chan 2021-03-29 10:47:29 +08:00