ComponentSpace

Forums



High memory usage when importing federation metadata


High memory usage when importing federation metadata

Author
Message
cdimitroulas
cdimitroulas
New Member
New Member (15 reputation)New Member (15 reputation)New Member (15 reputation)New Member (15 reputation)New Member (15 reputation)New Member (15 reputation)New Member (15 reputation)New Member (15 reputation)New Member (15 reputation)

Group: Forum Members
Posts: 9, Visits: 117
Hi there,

We use the `ImportUrlAsync` method to pull down the federation metadata for both UK Federation and OpenAthens on a daily basis (as per the recommendation of the UK Federation). We're implementing the AbstractSamlConfigurationResolver interface so that we can store the metadata in our database.

We noticed that this process consumes a lot of memory and that the memory usage does not drop after the metadata import is complete. After some debugging it doesn't seem that the memory issues are related to our own code. I have a suspicion that it may be related to some kind of internal caching behaviour of the Component Space library which is holding all of the IDP metadata in memory and not allow it to be garbage collected.

 Any advice or idea how we can alleviate these problems would be much appreciated!  
ComponentSpace
ComponentSpace
ComponentSpace Development
ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)

Group: Administrators
Posts: 2.9K, Visits: 9.6K
We don't cache the metadata etc so I'm not sure what could be causing the high memory usage.

Please see if you can reproduce the issue using the ImportMetadata project. You might want to change this code to loop around several times. This will help to narrow down the issue.

I'll see if I can replicate it here as well.
 

Regards
ComponentSpace Development
ComponentSpace
ComponentSpace
ComponentSpace Development
ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)

Group: Administrators
Posts: 2.9K, Visits: 9.6K
I modified the ImportMetadata project to loop around a few hundred times importing the 71 MB UK federation metadata.

I saw no increase in memory usage over time and, looking at the heap at various times, I saw nothing unusual.

Just as an aside, if you aren't already you might want to use an ETag and check for the 304 return status to save downloading the metadata unnecessarily.

The CheckForMetadataUpdates example project we include under the Examples\Metadata folder demonstrates this and other mechanisms to minimize the processing required.

 

Regards
ComponentSpace Development
cdimitroulas
cdimitroulas
New Member
New Member (15 reputation)New Member (15 reputation)New Member (15 reputation)New Member (15 reputation)New Member (15 reputation)New Member (15 reputation)New Member (15 reputation)New Member (15 reputation)New Member (15 reputation)

Group: Forum Members
Posts: 9, Visits: 117
ComponentSpace - 5/6/2022
I modified the ImportMetadata project to loop around a few hundred times importing the 71 MB UK federation metadata.

I saw no increase in memory usage over time and, looking at the heap at various times, I saw nothing unusual.

Just as an aside, if you aren't already you might want to use an ETag and check for the 304 return status to save downloading the metadata unnecessarily.

The CheckForMetadataUpdates example project we include under the Examples\Metadata folder demonstrates this and other mechanisms to minimize the processing required.

 

Thank you for the suggestion, I will take a look at that
ComponentSpace
ComponentSpace
ComponentSpace Development
ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)

Group: Administrators
Posts: 2.9K, Visits: 9.6K
You're welcome.

Regards
ComponentSpace Development
cdimitroulas
cdimitroulas
New Member
New Member (15 reputation)New Member (15 reputation)New Member (15 reputation)New Member (15 reputation)New Member (15 reputation)New Member (15 reputation)New Member (15 reputation)New Member (15 reputation)New Member (15 reputation)

Group: Forum Members
Posts: 9, Visits: 117
I did some more tests. Indeed the ImportMetadata project does not exhibit any kind of memory issues.

I tried to setup a minimal reproduction of this issue by creating a new web project and adding an endpoint which triggers the import of the UK federation metadata file via the url. This project does exhibit the problems that I described earlier. Each time the endpoint is called, the memory usage of the app goes up and does not come back down. I was able to reach several GB of memory usage in this manner.

Could you please take a look and see whether you can determine why this might happen? I have attached the project where I was able to reproduce the issue as a zip file.



Attachments
ComponentSpaceMemoryTest.zip (2 views, 650.00 KB)
ComponentSpace
ComponentSpace
ComponentSpace Development
ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)

Group: Administrators
Posts: 2.9K, Visits: 9.6K
Thanks for the test app. I've run it fifty or more times and don't believe there's a memory leak although memory usage is high.

If I keep hitting the endpoint, process memory increases until it plateaus at around 1.5GB. Looking at the heap, there's about 24K objects consuming 133MB. I suspect these numbers will vary depending on how much system memory is available but that's what I see on my machine. By comparison, when the app starts and prior to hitting the endpoint, process memory usage is 68MB and the heap has 20K objects consuming 2MB.

As an experiment, I added another endpoint which simply called GC.Collect().

I tried various combinations of importing the metadata and calling the garbage collector by browsing to the respective endpoints. For example, sometimes I called the garbage collector after each import. Other times I did ten or twenty imports and then ran the garbage collector a few times. In all cases, after garbage collection completed the process memory usage dropped to 640MB and the heap had 24K objects consuming 2MB.

I'm not suggesting you call the garbage collection in a production application. I merely did this as an experiment.

I suggest doing something similar to confirm the results I saw.

Also, looking at the heap, there are just a couple ComponentSpace objects consuming a few dozen bytes. This makes sense as we're not keeping references to any of the objects associated with the metadata import.

Given the large size of the metadata document being downloaded and imported, this generates a large number of objects and consumes significant memory. However, the .NET garbage collector will eventually clean things up.

As I mentioned earlier, I suggest employing some of the techniques used by the CheckForMetadataUpdates project to minimize the number of downloads.

Let me know if you still have any concerns.

Regards
ComponentSpace Development
cdimitroulas
cdimitroulas
New Member
New Member (15 reputation)New Member (15 reputation)New Member (15 reputation)New Member (15 reputation)New Member (15 reputation)New Member (15 reputation)New Member (15 reputation)New Member (15 reputation)New Member (15 reputation)

Group: Forum Members
Posts: 9, Visits: 117
Thank you for taking the time to investigate
ComponentSpace
ComponentSpace
ComponentSpace Development
ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)ComponentSpace Development (4K reputation)

Group: Administrators
Posts: 2.9K, Visits: 9.6K
You're welcome. Let me know if you have any concerns.

Regards
ComponentSpace Development
GO


Similar Topics


Execution: 0.000. 2 queries. Compression Enabled.
Login
Existing Account
Email Address:


Password:


Social Logins

Select a Forum....









Forums, Documentation & Knowledge Base - ComponentSpace


Search