How can I read output from multiple Linux commands as input streams in a Python script

90
April 16, 2022, at 00:50 AM

I'm trying to write a Python program that takes 2 very large files as input. They are both currently compressed on disk in .zst format, and I don't want to (don't have space to) decompress them on disk. I need to pipe the input directly from the decompression into my python script.

If there were just one file, I'd do something like:

zstd -d -c myfile.zst | python myscript.py

and have myscript read from sys.stdin

What can I do if I have 2 compressed files I need myscript to have access to?

I think I can use process substitution to pass the 2 streams in, but I'm not sure how to take streams as arguments in python. So if I do this:

myscript.py <(zstd -d -c myfile1.zst) <(zstd -d -c myfile2.zst)

What is the analog (in my python script) to for line in sys.stdin for reading from each of those input streams?

Or if I'm not on the right track with process substitution, what's the right way to do this? Thanks!

Edited to add: I need the two input streams to be separate parameters (not concatenated into a single stream).

Rent Charter Buses Company
READ ALSO
Python Selenium on Chrome | find active element name

Python Selenium on Chrome | find active element name

I’m trying to get the name of the active element through selenium

84
Apache Beam Streaming from Kafka into Files over Fixed Window not working

Apache Beam Streaming from Kafka into Files over Fixed Window not working

In python, I am trying to have a streaming pipeline that ingests data from Kafka, and then writes that data in 30 second intervals into a file at the end of each intervalI am using the FlinkRunner

123
Writing to a file with mode &quot;w&quot; versus &quot;a&quot; - I can&#39;t get &quot;w&quot; to work properly

Writing to a file with mode "w" versus "a" - I can't get "w" to work properly

Let me start out by saying I am very new to PythonI'm taking a Udemy bootcamp, so I am not by any means condoning my solution as the best solution, but for the life of me, I cannot figure out why, when I write to the output files, I have to use mode "a"...

89
Selenium python: iterating through dropdown element - number of items varies on page

Selenium python: iterating through dropdown element - number of items varies on page

I have a repository of pages accessible from a Home pageI have a selenium script that currently loops through each of these (a set number of pages) as I would like, telling me if an iframe on the respective page loads certain elements or not

126