apache spark - Structured Streaming 2.1.0 stream to Parquet creates many small files -


i running structured streaming (2.1.0) on 3-node yarn cluster , stream json records parquet. code fragment looks this:

  val query = ds.writestream         .format("parquet")         .option("checkpointlocation", "/data/kafka_streaming.checkpoint")         .start("/data/kafka_streaming.parquet")

i notice creates thousands of small files 1,000 records. suspect has frequency of trigger. changed it:

  val query = ds.writestream         .format("parquet")         .option("checkpointlocation", "/data/kafka_streaming.checkpoint")         .**trigger(processingtime("60 seconds"))**         .start("/data/kafka_streaming.parquet")

the difference obvious. can see smaller number of files created same number of records.

my question: there way have low latency trigger , keep smaller number of larger output files?


Comments

Post a Comment

Popular posts from this blog

javascript - Clear button on addentry page doesn't work -

this addentry blog page have made scratch , there underlying problems code particularly clear button supposed clear text entered when button clicked on doesn't work have saved html file problem. also 1 other problem submit button works doesn't pass of text index file have written php , reason whenever click on submit button redirects me index page without posting entry have made. <!doctype html> <html> <head> <script> function message() { alert("clear form?"); } </script> <style> body {background-color: beige;} h1 {color: black;} p {color: black;} textarea { width: 50%; height: 150px; padding: 12px 20px; box-sizing: border-box; border: 2px solid #ccc; border-radius: 4px; background-color: white; font-size: 16px; resize: none; } div { padding-left: 20px; } </style> </head> <meta charset="utf-8"> <link rel="stylesheet" href="....
Read more

python - Error: Unresolved reference 'selenium' What is the reason? -

# -*- coding: utf-8 -*- selenium import selenium import unittest, time, re class rc(unittest.testcase): def setup(self): self.verificationerrors = [] self.selenium = selenium("localhost", 4444, "*chrome", "http://stackoverflow.com/") self.selenium.start() def test_rc(self): sel = self.selenium sel.open("/") sel.type("id=kw", "selenium") sel.click("id=container") sel.click("id=su") def teardown(self): self.selenium.stop() self.assertequal([], self.verificationerrors) if __name__ == "__main__": unittest.main() this recorded selenium ide, selenium remote control code. in ide test passed.   in pycharm open, first line ( from selenium import selenium ) there error: unresolved reference 'selenium' what reason? my environment is python3.5 selenium3.3.1 selenium-server-standalone-...
Read more

asp.net ajax - Jquery scroll to element just goes to top of page -

i have jquery updates page content via ajax (paging through sub table content) , want page "jump" (scroll) top of section content has been updated. found code try , perform scroll ever goes right top of page , not specific element : <script type="text/javascript"> $(document).ready(function() { $(".page-number").on("click", function() { var page = parseint($(this).html()); $.ajax({ url: '@url.action("productreview")', datatype: 'html', data: { "id": @model.product.productid, "page": page }, success: function (data) { $("#review-list").html(data); $("#page-number-"+ page).addclass("selected"); //location.hash = "#prodreviews"; ...
Read more