Pdf2csv python github code#

I don't think that splitting this work onto multiple threads is going to provide much improvement, and may in fact make the problem worse by consuming even more memory. Do you know of a different approach to split at comas and replace the double quotes in each CSV line ? Would StringBuilder be of any healp here ? What about StringTokenizer ?

How could I reduce the amount of heap memory used in the process ? Is the multithread implementation with Callable correct ? How could I improve the speed of the CSV reading ? Other than that the api is running out of heap memory when running on the server, I know that a solution would be to enhance the amount of available memory but I suspect that the replace() and split() operations on strings made in the Callable(s) are responsible for consuming a large amout of heap memory. To improve speed processing, I tried to implement multithreading with Callable(s) but I am not familiar with that kind of concept, so the implementation might be wrong.

Pdf2csv python github code#

The code works fine on my local machine but it is very slow : it takes about 20 seconds to process 450 columns and 40 000 lines. I am not guaranteed to have the same header between files (each file can have a completly different header than another), so I have no way to create a dedicated class which would provide mapping with the CSV headers.Ĭurrently the api controller is calling a csv service which reads the CSV data using a BufferReader. It has to read big CSV files which will contain more than 500 columns and 2.5 millions lines each. I am currently working on a spring based API which has to transform csv data and to expose them as json.