Memory Profiler causes extreme memory usage during profiling. #78

Activity

1. SamSaffron commented on Jul 30, 2019

   

   SamSaffron

   on Jul 30, 2019

   Owner

   More actions

   its is very very tricky, collection all this data comes at a cost and we have no way of simply streaming this data to disk and then running analysis on data stored on disk.

   Is there any way you can simply collect less info?
2. envygeeks commented on Jul 30, 2019

   

   envygeeks

   on Jul 30, 2019

   Author

   More actions

   Kinda we did, it was a bit of a mess, but memory profiling acting crazy prompted me to look closer at other stuff with a manual audit, which helped me notice an allocation leak in one of the libraries we use, where the library was allocating millions of objects over and over again.
3. jgmontoya commented on Jan 2, 2020

   

   jgmontoya

   on Jan 2, 2020

   More actions

   I'm having the same issue, to the point where I can't really measure the job because I'm running out of memory... How can I reduce the amount of info collected? I'm already trying to evaluate a single class (using trace: [TargetClass])
4. GMolini commented on Mar 14, 2022

   

   GMolini

   on Mar 14, 2022

   More actions

   This is happening to me when Im trying to measure the amount of memory consumed while reading from a stream
   Ive got a zip file that has a CSV in it. I was trying to see how much memory would be used if I processed the file as a stream

   report = MemoryProfiler.report do
   
     file_path =  "/tmp/myzip"
     zip_file = Zip::File.open(file_path)
     zip_file.entries.each do |entry|
      
       Rails.logger.info "Count lines"
       total_lines = 0
       entry.get_input_stream.each do |line|
         total_lines += 1
       end
     end
   end
   

   If I execute that code without the memory profiler, using htop I can see that the memory usage barely changes. But If I put the block inside the MemoryProfiler.report do statement, the memory usage absolutely explodes (I stopped the console after it had risen from 6GB of used RAM to 50GB)

   👍React with 👍3Reacted by Oscar Rivas, Jeremy Prevost and Mikhail Varabyou
5. kadru commented on Oct 18, 2022

   

   kadru

   on Oct 18, 2022

   More actions

   I have a similar issue, I'm trying to profile a sidekiq job trough a middleware with memory profiler (sidekiq is running only with 1 thread). When I execute the code without the profiler the memory use is 2GB aprox, with the profiler it reaches up to 28GB until is killed by the OOM. The background job is using CSV to read a cvs file line per line to generate another csv file.
6. segiddins commented on Sep 20, 2023

   

   segiddins

   on Sep 20, 2023

   More actions

   https://github.com/ruby/ruby/blob/0a630fa461a7260235842e482f682deca30172d6/ext/objspace/object_tracing.c#L445C40-L445C40 looks like it allocates a new string each time we get the allocation source file, which definitely could be an fstring (my guess is its coming from the iseq, which already should have an associated fstring for the file name?)

   profile with rbspy is telling me that the CPU time is being spent in allocation_sourcefile, allocation_generation, __id__, []=, and maybe some class's new ?
7. segiddins commented on Sep 20, 2023

   

   segiddins

   on Sep 20, 2023

   More actions

   I think another thing that might help would be an option to skip retained and only look at allocated, then we can skip the __id__ call and associated hash?